Iliya Mirzaei writes that test-time methods which make a language model critique, rewrite or reflect on its own output do not beat simple repeated sampling when both are matched on generated tokens. Re-running the budget-matched comparison of Wang et al. as a designed experiment with paired bootstrap intervals and Holm correction on Qwen2.5-1.5B, 3B and 7B for GSM8K and MATH-500 with 150 questions each, no method is reliably better than self-consistency at equal cost; ten comparisons are significantly worse, all involving self-inspection. Best-of-N selection by the model loses to majority voting on identical samples below 7B, while rewriting methods such as Self-Refine and forced Reflexion remain below the equal-cost baseline even at 7B.
- Reflexion as implemented never triggered a retry on the 1.5B model, judging every answer correct and collapsing to a single chain of thought.
- Best-of-N vs counting penalty shrinks with scale but stays negative at 7B; parity is approached, not crossed.
- Cost is generated tokens only; accounting for input tokens would make self-assessing methods look worse.
- All generations, prompts and checking scripts are released for reproducibility.
- Reflexion as implemented never triggered a retry on the 1.5B model, judging every answer correct and collapsing to a single chain of thought.
- Best-of-N vs counting penalty shrinks with scale but stays negative at 7B; parity is approached, not crossed.
- Cost is generated tokens only; accounting for input tokens would make self-assessing methods look worse.
- All generations, prompts and checking scripts are released for reproducibility.
