Pith. sign in

REVIEW 2 major objections 6 minor 37 references

No method that makes a model inspect its own answers beats simply sampling more times once every generated token is counted the same way.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 03:17 UTC pith:XOVNTSKR

load-bearing objection Solid statistical re-run of Wang et al.: no method beats cost-matched sampling, and self-inspection stays negative through 7B, with unusually clean artifacts. the 2 major comments →

arxiv 2607.28576 v1 pith:XOVNTSKR submitted 2026-07-30 cs.CL cs.AIcs.LG

Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B

Iliya Mirzaei This is my paper
classification cs.CL cs.AIcs.LG
keywords test-time computeself-consistencyself-refinereflexionbudget-matched evaluationself-verificationmathematical reasoninglanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Many popular test-time tricks—self-critique, reflection, best-of-N selection, multi-agent debate—also make the model write far more text than one chain of thought. Extra text alone raises accuracy, so a win over a single sample does not prove the trick’s idea helped. This paper re-runs that comparison as a controlled experiment on open 1.5B–7B models and two math benchmarks: every method is measured by total generated tokens and scored against repeated sampling at that same cost, with paired bootstrap intervals and multiplicity correction. None of the thirty-six comparisons finds a reliable win for the fancy method; all eighteen that involve the model judging or rewriting its own output come out negative. As models grow, letting the model pick among its samples stops hurting relative to a majority vote, but forcing rewrite-and-retry loops still wastes tokens compared with drawing another independent attempt.

Core claim

At equal generated-token cost, none of seven common test-time reasoning methods is reliably better than repeated sampling with majority vote on GSM8K and MATH-500 for Qwen2.5 models from 1.5B to 7B. Every comparison in which the model assesses or rewrites its own output is negative; Self-Refine and forced Reflexion remain several points below the matched baseline even at 7B. Holding the same eight samples fixed, counting the most common answer beats asking the model to choose, by large margins below 7B and by amounts no longer distinguishable from zero at 7B.

What carries the argument

Cost-matched self-consistency curve: generate a pool of independent chains once, read majority-vote accuracy at every sample count from the same pool, then place each competing method against the curve at that method’s own measured completion-token cost.

Load-bearing premise

That counting generated tokens on exact-match math problems, with fixed untuned prompts and hyperparameters, is a fair test of whether self-inspection helps in the settings people actually care about.

What would settle it

Rerun the identical paired, cost-matched protocol on a larger model or with a trained verifier and show any self-inspection method landing reliably above the interpolated majority-vote curve on the same questions.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A reported gain over one chain of thought is not evidence that planning, critique, reflection, or debate caused the gain unless an equal-token sampling baseline is beaten.
  • With a fixed generation budget on checkable math, spending tokens on another independent attempt is a better default than spending them on self-critique or self-rewrite.
  • Adaptive methods must report how often their self-triggered loop actually fires; otherwise a method can score well by silently collapsing into a cheap single sample.
  • Untrained same-model selection among samples reaches parity with majority vote near 7B by agreeing more often with the tally, not by becoming a better override judge.
  • Future method papers can settle the budget question cheaply by releasing one repeated-sampling accuracy-versus-cost curve on the same model and questions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • On open-ended tasks where answers cannot be majority-voted, the practical rival to self-critique shrinks to “pick one sample,” so the same mechanisms could look better there without contradicting these math results.
  • If override accuracy of an untrained self-verifier never crosses 50% on disagreements, scaling alone will not make judging beat counting; training or external verifiers would be required.
  • Evaluations that charge for input tokens or wall-clock latency would likely widen the gap against sequential self-refinement and debate, because those methods re-read long contexts and cannot parallelize rounds.
  • Silent control-flow collapse (a method that stops acting while keeping its name) is a general evaluation hazard for any agent loop gated on the model’s own correctness judgment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper re-runs budget-matched evaluation of seven test-time reasoning methods (CoT, Plan-and-Solve, Self-Refine, Reflexion and a forced variant, Best-of-N with self-verify, multi-agent debate) against self-consistency on Qwen2.5 models (1.5B/3B, with selected methods at 7B) and two math benchmarks (GSM8K, MATH-500; 150 questions each). Cost is measured as all generated tokens; each method is compared to the self-consistency curve interpolated at its own measured cost, with paired bootstrap CIs and Holm correction. No method is reliably better than equal-cost repeated sampling; all 18 self-inspection comparisons are negative. Holding the same eight samples fixed, model selection loses to majority vote below 7B and is indistinguishable at 7B, while Self-Refine and forced Reflexion remain below baseline at 7B. Reflexion as published never retries on 1.5B. Code, prompts, generations, and verification scripts are released.

Significance. If the result holds inside its stated scope, it is a high-value corrective for the test-time compute literature: gains over single CoT are not evidence that planning, self-critique, or self-selection mechanisms help once token budget is controlled. Strengths that raise the contribution above a routine bake-off include (i) the identical-sample judges-vs-counting design that isolates selection from sampling, (ii) proper paired inference with multiplicity control and an explicit detectability statement, (iii) per-method realized-cost matching rather than a shared cap, (iv) reporting of Reflexion’s engagement rate (exposing silent collapse to CoT), and (v) full artifact release plus an independent re-derivation script and documented scoring/context bugs. The work is a careful statistical re-examination and extension of Wang et al., not a new phenomenon, but that is the right contribution for the claim.

major comments (2)
  1. [Table 2, §5, Abstract] Table 2 and §5: the full seven-method, cost-matched design is reported for 1.5B and 3B only. At 7B the table covers CoT, Self-Refine, forced Reflexion, and Best-of-N; Plan-and-Solve, published Reflexion, and debate are absent. The abstract and title frame the result as holding “from 1.5B to 7B” for the comparison as a whole. That framing is accurate for rewriting and for judges-vs-counting, but overstates coverage for the full method suite. Please state explicitly in the abstract and §5 which claims are supported at 7B and which stop at 3B, so the scale claim cannot be read as a complete six-setting × seven-method grid.
  2. [§5.1, §6.1, Abstract] §5.1 and §6.1: two groupings of “self-inspection” / “repeated passes over own work” are used to organize the negative pattern. The manuscript correctly labels one grouping as formed after the Best-of-N scoring fix and reports both a naive sign test and a clustered setting-level exact test (p=0.0156 / p=0.06). Because the central claim is already carried by the per-comparison table and the sample-fixed Best-of-N contrast, the post-hoc grouping should not be presented as confirmatory structure in the abstract (“all 18 self-inspection comparisons”). Keep the descriptive grouping in the body; lead the abstract with the pre-specified cost-matched tests and the fixed-sample selection result.
minor comments (6)
  1. [§4.1] §4.1 vs §5: the setup section introduces 1.5B and 3B, while results add 7B for a subset of methods. A single sentence in §4.1 stating the 7B extension and which methods were affordable would remove the discontinuity.
  2. [Figure 1] Figure 1 is dense (many rows). Consider splitting by model size or ordering methods consistently within each setting so significant (coloured) intervals are easier to scan.
  3. [§4.7, §5.3] §4.7 / input-token reconstruction: the total-token re-analysis for Best-of-N is valuable and strengthens the paper. A short pointer in the main results (not only threats) that the Best-of-N gap widens under total tokens would help readers who only skim §5.
  4. [Appendix A.4–A.5] Appendix A.4–A.5: the context-truncation bias and Best-of-N scoring bug are exemplary scientific hygiene. Consider a one-sentence forward reference in §4 or §5 so readers know the main numbers are post-correction without waiting for the appendix.
  5. [§1, §4] Typos / polish: “ap-value” spacing in the introduction; “V alidating” and “T able” broken initial capitals in §4; “MA TH-500” spacing artifacts appear in several places—likely PDF ligature/encoding issues worth a pass.
  6. [§3, §7] Related work: the discussion of trained verifiers (Zhang et al.) and imperfect-verifier limits (Stroebl et al.) is well placed. A brief explicit contrast with process-reward / trained self-verifiers in the conclusion would further prevent readers from over-generalizing to trained selectors.

Circularity Check

0 steps flagged

No significant circularity: empirical bake-off on external benchmarks with an independently defined baseline.

full rationale

This paper is a controlled empirical comparison of prompting/test-time methods against cost-matched self-consistency on GSM8K and MATH-500. Accuracy is graded against gold answers; token costs are measured from model generations; the baseline (majority vote over independent samples) is defined independently of the methods under test. The sharpest mechanism contrast—Best-of-N model choice versus counting on identical fixed samples—holds the candidates constant and only changes the selection rule, so the gap is not forced by construction or by a fitted parameter. There is no theoretical ansatz, no uniqueness theorem imported from the authors, and no self-citation that carries the central claim (the sole self-citation is an analogy to consensus-protocol measurement in §6.2 and is not load-bearing). The post-hoc descriptive grouping of ‘self-inspection’ methods is explicitly labeled as non-hypothesis-testing after a scoring bug fix and does not make any reported accuracy difference equal its inputs by definition. The derivation chain is measurement and paired bootstrap inference, not a closed definitional loop.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

Load-bearing content is experimental protocol, not a formal derivation. The claim rests on standard statistical assumptions, domain choices (math exact-match, generated-token cost), and faithful-but-fixed implementations of published methods. No new physical entities; free parameters are design knobs fixed a priori rather than fit to optimize the headline gap.

free parameters (4)
  • N_pool / subsample grid (K=16; N in {1,2,3,4,6,8,12,16}; 200 draws) = K=16, 200 draws
    Defines the baseline curve resolution and Monte-Carlo error; fixed by design, not fit to favor methods.
  • Method hyperparameters (Self-Refine R=3; Best-of-N N=8; Debate A=3,R=2; max 1024 tokens/call) = R=3, N=8, A=3, R_debate=2, cap=1024
    Fixed in advance to avoid answer leakage from tuning; may understate each method's best case.
  • Sampling temperature split (0 vs 0.7) = T=0 or 0.7
    Deterministic methods at T=0, diversity methods at T=0.7; controlled via SC@1 comparison but still a design choice.
  • Question subsample size and seed (150 per benchmark, Random(1234)) = n=150, seed=1234
    Fixes the experimental units; power stated (median SE ~2.6 pp).
axioms (6)
  • standard math Paired bootstrap over questions with Holm–Bonferroni multiplicity control yields valid frequentist inference for method-vs-baseline accuracy differences.
    Invoked in §4.5–4.6 for all 36 comparisons and detectability claims.
  • domain assumption Generated (completion) tokens are the primary cost unit for fair method comparison.
    §2.3 and §6.4; input tokens and latency discussed as alternatives that would not help the tested methods.
  • domain assumption Exact-match grading on GSM8K/MATH-500 after deterministic parse/normalization is a valid success criterion for the methods under test.
    §4.2–4.4; authors note this may disadvantage critique methods designed for open-ended text.
  • domain assumption Implementations from paper descriptions with no ground-truth stop signals are faithful enough to support claims about those methods (with forced Reflexion labeled as authors' variant).
    §4.3 and Appendix A prompts; engagement rates reported when adaptive control flow collapses.
  • ad hoc to paper Linear interpolation of the self-consistency curve in (cost, accuracy) space is acceptable for cost matching; bias direction favors methods.
    §4.7; concave-up baseline means chord underestimates baseline accuracy.
  • domain assumption Qwen2.5 Instruct checkpoints at Q8_0 on CPU are adequate proxies for the small open-model regime claimed.
    §4.1, §6.5 limitations on scale and quantization.

pith-pipeline@v1.2.0-daily-grok45 · 28657 in / 3736 out tokens · 70807 ms · 2026-07-31T03:17:06.737464+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B." pith.science (2026). https://pith.science/paper/XOVNTSKR

@misc{pith2026260728576,
  author       = {Pith},
  title        = {Pith review of: Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOVNTSKR}},
  note         = {Machine review of arXiv:2607.28576}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Methods that make a language model plan, criticise and rewrite its own answer, reflect on mistakes, pick the best of several attempts, or debate with copies of itself nearly all make it generate far more text than a single chain of thought. Because generating more text raises accuracy by itself, a gain over one chain of thought does not show the method's idea is what helped. Wang et al. (2024) reported that a simple baseline, sampling the same question repeatedly and keeping the most common answer, often wins once budgets are comparable, but gave point estimates with no confidence intervals or significance tests. We rerun that comparison as a designed experiment: seven methods, open models of 1.5B, 3B and 7B parameters, two mathematics benchmarks, 150 questions each. We count every generated token, including those spent on critiques, reflections, debate turns and checking, and compare each method against repeated sampling at its own measured cost. All 36 comparisons are paired by question, with bootstrap intervals and multiplicity correction. No method is reliably better than repeated sampling at equal cost anywhere. Ten are reliably worse, all of them methods where the model inspects its own output, and all 18 self-inspection comparisons are negative. The two kinds of self-inspection part company as models grow. Choosing stops hurting: taking Best-of-N's eight samples and just counting the most common answer beats letting the model pick by 8.0 and 11.3 points at 1.5B, but only 2.0 and 1.3 at 7B, no longer distinguishable from zero. Rewriting does not recover: Self-Refine and a forced Reflexion stay 3.6 to 10.1 points below baseline at 7B. Reflexion as published never triggered its own retry on the smallest model. It judged itself correct every time and silently became a single chain of thought. We release code, prompts, all generations, and our verification scripts.

Figures

Figures reproduced from arXiv: 2607.28576 by Iliya Mirzaei.

Figure 1
Figure 1. Figure 1: Difference in accuracy between each method and self-consistency at equal generated-token cost, for every method and setting. Points left of the dashed line mean the method did worse than simply drawing more samples for the same tokens. Bars are 95% paired bootstrap intervals over questions. Coloured intervals are the comparisons that remain significant after Holm correction within their setting; grey inter… view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy against mean generated tokens per question. The shaded band is a 95% interval on the self-consistency curve; error bars on the markers are 95% intervals on both axes. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The cost of asking the model to judge, against model size. Both quantities are measured on the same sampled solutions, so the vertical axis isolates the value of the model’s selection step from everything else. Points below the dashed line mean counting votes would have been better. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 28 linked inside Pith

  1. [1]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gi- aninazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. InAAAI Conference on Artificial Intelligence, 2024. arXiv:2308.09687

    Pith/arXiv arXiv 2024

  2. [2]

    Le, Christopher R´ e, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher R´ e, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Debate or vote: Which yields better decisions in multi-agent large lan- guage models? InAdvances in Neural Information Processing Systems (NeurIPS), Spotlight,

    Hyeong Kyu Choi et al. Debate or vote: Which yields better decisions in multi-agent large lan- guage models? InAdvances in Neural Information Processing Systems (NeurIPS), Spotlight,

  4. [4]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  5. [5]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. InInternational Conference on Machine Learning (ICML), 2024. arXiv:2305.14325

    Pith/arXiv arXiv 2024

  6. [6]

    Bootstrap methods: Another look at the jackknife.The Annals of Statistics, 7(1):1–26, 1979

    Bradley Efron. Bootstrap methods: Another look at the jackknife.The Annals of Statistics, 7(1):1–26, 1979

    1979

  7. [7]

    llama.cpp, 2026.https://github.com/ ggml-org/llama.cpp

    Georgi Gerganov and llama.cpp contributors. llama.cpp, 2026.https://github.com/ ggml-org/llama.cpp

    2026

  8. [8]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. InNeurIPS Datasets and Benchmarks Track, 2021. arXiv:2103.03874

    Pith/arXiv arXiv 2021

  9. [9]

    A simple sequentially rejective multiple test procedure.Scandinavian Journal of Statistics, 6(2):65–70, 1979

    Sture Holm. A simple sequentially rejective multiple test procedure.Scandinavian Journal of Statistics, 6(2):65–70, 1979

    1979

  10. [10]

    V-star: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457, 2024

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners.arXiv preprint arXiv:2402.06457, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. InInterna- tional Conference on Learning Representations (ICLR), 2024. arXiv:2310.01798

    Pith/arXiv arXiv 2024

  12. [12]

    When can llms actu- ally correct their own mistakes? a critical survey of self-correction of llms.arXiv preprint arXiv:2406.01297, 2024

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can llms actu- ally correct their own mistakes? a critical survey of self-correction of llms.arXiv preprint arXiv:2406.01297, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Scalable best-of-n selection for large language models via self-certainty.arXiv preprint arXiv:2502.18581, 2025

    Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty.arXiv preprint arXiv:2502.18581, 2025. 24

    arXiv 2025

  14. [14]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2205.11916

    Pith/arXiv arXiv 2022

  15. [15]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In International Conference on Learning Representations (ICLR), 2024. arXiv:2305.20050

    Pith/arXiv arXiv 2024

  16. [16]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. InAdvances in Neural Infor- mation Processi...

    Pith/arXiv arXiv 2023

  17. [17]

    Adding error bars to evals: A statistical approach to language model evaluations

    Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Fair on the surface: Transaction-ordering bias and mev in mysticeti dag-based bft protocol.arXiv preprint arXiv:2607.13378, 2026

    Iliya Mirzaei and Mohammad Javad Amiri. Fair on the surface: Transaction-ordering bias and mev in mysticeti dag-based bft protocol.arXiv preprint arXiv:2607.13378, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand` es, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Simplicity paradox: Debunking myths about prompting and datasets for llm evaluation.arXiv preprint arXiv:2607.14109, 2026

    Inder Preet et al. Simplicity paradox: Debunking myths about prompting and datasets for llm evaluation.arXiv preprint arXiv:2607.14109, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    Qwen2.5 technical report, 2024

    Qwen Team. Qwen2.5 technical report, 2024. arXiv:2412.15115

    Pith/arXiv arXiv 2024

  22. [22]

    The sequential edge: Inverse-entropy voting beats parallel self-consistency at matched compute.arXiv preprint arXiv:2511.02309, 2025

    Aman Sharma et al. The sequential edge: Inverse-entropy voting beats parallel self-consistency at matched compute.arXiv preprint arXiv:2511.02309, 2025

    arXiv 2025

  23. [23]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2303.11366

    Pith/arXiv arXiv 2023

  24. [24]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Inference scaling flaws: The limits of llm resampling with imperfect verifiers.arXiv preprint arXiv:2411.17501, 2024

    Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling flaws: The limits of llm resampling with imperfect verifiers.arXiv preprint arXiv:2411.17501, 2024

    arXiv 2024

  26. [26]

    Single-agent llms outperform multi-agent systems on multi-hop reasoning under equal thinking token budgets.arXiv preprint arXiv:2604.02460, 2026

    Dat Tran et al. Single-agent llms outperform multi-agent systems on multi-hop reasoning under equal thinking token budgets.arXiv preprint arXiv:2604.02460, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Reasoning in token economies: Budget-aware evaluation of llm reasoning strategies

    Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, and Ben Athi- waratkun. Reasoning in token economies: Budget-aware evaluation of llm reasoning strategies. arXiv preprint arXiv:2406.06461, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. InAnnual Meeting of the Association for Computational Linguistics (ACL),

  29. [29]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations (ICLR), 2023. arXiv:2203.11171

    Pith/arXiv arXiv 2023

  30. [30]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2201.11903

    Pith/arXiv arXiv 2022

  31. [31]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2305.10601

    Pith/arXiv arXiv 2023

  33. [33]

    Incentivizing llms to self-verify their answers

    Fuxiang Zhang, Jiacheng Xu, et al. Incentivizing llms to self-verify their answers. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2506.01369

    arXiv 2025

  34. [34]

    Progressive-hint prompt- ing improves reasoning in large language models.arXiv preprint arXiv:2304.09797, 2023

    Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompt- ing improves reasoning in large language models.arXiv preprint arXiv:2304.09797, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2023. arXiv:2306.05685

    Pith/arXiv arXiv 2023

  36. [36]

    Let’s first understand the problem and devise a plan to solve it. Then carry out the plan step by step

    Denny Zhou, Nathanael Sch¨ arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. InInternational Conference on Learning Representations (ICLR), 2023. arXiv:2205.10625. 26 A Reproducibility details A.1 Hardw...

    Pith/arXiv arXiv 2023

  37. [2023]

    arXiv:2305.04091. 25

    Pith/arXiv arXiv