Geiping writes about a language model architecture that scales test-time compute by iterating a recurrent block to reason implicitly in latent space, unrolling to arbitrary depth at inference time rather than generating more tokens. Unlike chain-of-thought methods, this approach needs no specialized training data, operates with small context windows, and can represent reasoning that is difficult to express linguistically. The authors scale a proof-of-concept to 3.5 billion parameters trained on 800 billion tokens, showing dramatic benchmark improvements at computation loads equivalent to 50 billion parameters.
- Hugging Face "huginn-0125"
- GitHub com/seal-rg/huginn-0125
- Hugging Face "huginn-0125"
- GitHub com/seal-rg/huginn-0125
Iliya Mirzaei writes that test-time methods which make a language model critique, rewrite or reflect on its own output do not beat simple repeated sampling when both are matched on generated tokens. Re-running the budget-matched comparison of Wang et al. as a designed experiment with paired bootstrap intervals and Holm correction on Qwen2.5-1.5B, 3B and 7B for GSM8K and MATH-500 with 150 questions each, no method is reliably better than self-consistency at equal cost; ten comparisons are significantly worse, all involving self-inspection. Best-of-N selection by the model loses to majority voting on identical samples below 7B, while rewriting methods such as Self-Refine and forced Reflexion remain below the equal-cost baseline even at 7B.
- Reflexion as implemented never triggered a retry on the 1.5B model, judging every answer correct and collapsing to a single chain of thought.
- Best-of-N vs counting penalty shrinks with scale but stays negative at 7B; parity is approached, not crossed.
- Cost is generated tokens only; accounting for input tokens would make self-assessing methods look worse.
- All generations, prompts and checking scripts are released for reproducibility.
- Reflexion as implemented never triggered a retry on the 1.5B model, judging every answer correct and collapsing to a single chain of thought.
- Best-of-N vs counting penalty shrinks with scale but stays negative at 7B; parity is approached, not crossed.
- Cost is generated tokens only; accounting for input tokens would make self-assessing methods look worse.
- All generations, prompts and checking scripts are released for reproducibility.
