Iliya Mirzaei writes that test-time methods which make a language model critique, rewrite or reflect on its own output do not beat simple repeated sampling when both are matched on generated tokens. Re-running the budget-matched comparison of Wang et al. as a designed experiment with paired bootstrap intervals and Holm correction on Qwen2.5-1.5B, 3B and 7B for GSM8K and MATH-500 with 150 questions each, no method is reliably better than self-consistency at equal cost; ten comparisons are significantly worse, all involving self-inspection. Best-of-N selection by the model loses to majority voting on identical samples below 7B, while rewriting methods such as Self-Refine and forced Reflexion remain below the equal-cost baseline even at 7B.
- Reflexion as implemented never triggered a retry on the 1.5B model, judging every answer correct and collapsing to a single chain of thought.
- Best-of-N vs counting penalty shrinks with scale but stays negative at 7B; parity is approached, not crossed.
- Cost is generated tokens only; accounting for input tokens would make self-assessing methods look worse.
- All generations, prompts and checking scripts are released for reproducibility.
- Reflexion as implemented never triggered a retry on the 1.5B model, judging every answer correct and collapsing to a single chain of thought.
- Best-of-N vs counting penalty shrinks with scale but stays negative at 7B; parity is approached, not crossed.
- Cost is generated tokens only; accounting for input tokens would make self-assessing methods look worse.
- All generations, prompts and checking scripts are released for reproducibility.
A new paper by researchers from Google Research and UC Berkeley shows that a simple sampling-based search approach can enhance the reasoning abilities of large language models (LLMs) without needing specialized training or complex architectures.
Deep learning has been deployed in many tasks in NLP, such as translation, image captioning, and dialogue systems. In machine translation, it is used to read source language (input) and generate the desired language (output). Similarly in a dialogue system, it is used to generate a response given a context. This is also known as Natural Language Generation (NLG).
