Pith. sign in

REVIEW 3 major objections 4 minor 19 references

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The best LLM on a reasoning benchmark depends on how many tokens it is allowed to generate.

desk verdict A real measurement phenomenon—model rankings shift with max_tokens—but the router section overstates its case and the abstract overclaims one oracle finding. read the letter →

arxiv 2608.12150 v1 pith:YGGXKUIE submitted 2026-08-12 cs.AI cs.CL

Rodrigo Guedes de Souza , Alison R. Panisson This is my paper · ORCID
classification cs.AIcs.CL
keywords tokengenerationbudgetbudget-dependentevaluationmodelrankingreversaloverthinkingLLMroutingnon-monotoneaccuracyreasoningbenchmarkstest-timecompute
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Standard evaluations assume that which LLM is best does not depend on how many tokens it is allowed to generate. This paper tests that assumption by running four open-weight models on three reasoning benchmarks at seven max-token budgets, from 64 to 4,096 tokens, for a total of 56,476 inferences. It finds that the best model changes with the budget on every benchmark: for example, the 70B model leads GSM8K at 256 tokens while the 20B model dominates at 4,096 tokens. It also finds that 3-19% of items show accuracy decreasing as budget grows, even after removing truncation, and that these overthinking items barely overlap across models. The upshot is that a benchmark ranking is not a single ordering but a family of orderings parameterized by the token budget.

What carries the argument

The organizing device is a token generation budget b, defined as the max_tokens parameter and treated as an experimental axis. Each model-item pair is scored at seven budgets, producing a seven-bit correctness trajectory that is classified as always-correct, monotone-increasing, non-monotone, or always-wrong. To separate genuine reasoning failures from truncation artifacts, the analysis is run three ways: on all items, on stop-only items, and on common non-truncated items; ranking reversals are tested with paired significance tests on identical item sets. Complementarity is measured through an oracle ensemble and pairwise set-overlap similarity, while routing is implemented with per-model gradient-boosted classifiers that take log2(b) plus text and embedding-derived features as input.

What would settle it

Run the same four models on GSM8K at b=256 and b=4096, then repeat the comparison with outputs truncated by visible character count or by estimated internal reasoning time instead of by token count; if the 70B model remains best at both budgets under those controls, the ranking reversal is a token-measurement artifact rather than a true budget effect.

Watch

Extended reading notes

Core claim

The paper claims that model rankings are not invariant to the token generation budget. On GSM8K, LLaMA-3.3 70B is the best model at b=256 with 62.4% accuracy, while GPT-OSS 20B takes the lead at b=4096 with 94.8%. Similar reversals occur on MATH-500 and GPQA, with the smallest model ranking first at low budgets on GPQA before being overtaken by larger models. After controlling for truncation, 3-19% of items still show non-monotone behavior, meaning accuracy drops as more tokens are allowed, and 86-94% of these overthinking items are flagged for only one model. An oracle ensemble shows complementarity of up to +27.8 percentage points over the best single model, and a budget-aware router captures 14.1% of that gap in a cross-domain test.

Load-bearing premise

The argument assumes that setting the same max_tokens value gives every model the same amount of thinking; if one model spends tokens on internal tags while another spends them on visible reasoning, some of the ranking flips could be an artifact of how the budget is counted.

Editorial extensions

If this is right

  • A single accuracy number per benchmark underspecifies model quality; any reported ranking should name the token budget at which it was measured.
  • Deployers with tight latency or cost constraints should evaluate models at their operating budget, not at the generous budgets used by many leaderboards.
  • Routing systems can gain by adding budget as a feature within a domain, with the paper reporting +1.6 to +5.7 percentage points from budget features in within-domain settings.
  • Because models complement each other most at constrained budgets, ensemble and selection methods have the largest headroom where compute is scarce.
  • Truncation-adjusted analyses preserve the main findings, but some apparent reversals are truncation artifacts, so budget studies need a three-tier reporting protocol.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to normalize the budget by estimated reasoning tokens rather than emitted tokens; models that hide thinking inside internal tokens would then be comparable, and the non-monotone rates might shrink further.
  • Because overthinking items hardly overlap across models, a multi-model second-opinion router that lets a second model re-answer at high budgets could reclaim more of the oracle gap than the single-model router tested here.
  • The domain-specificity of budget features predicts that a router trained separately per domain, or adapted with a small target-domain set, would beat both the static router and the no-budget router by more than the reported margins.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates how the token generation budget (max_tokens) affects LLM evaluation results. It evaluates four open-weight models (8B–70B) on GSM8K, MATH-500, and GPQA at seven budgets (64–4,096), totaling 56,476 greedy-decoded inferences. Four findings are claimed: (i) 3–19% of items show non-monotone accuracy trajectories (accuracy drops with more budget) even after truncation control, with low cross-model overlap; (ii) the best-performing model changes across budgets on all three benchmarks, with several reversals significant by McNemar's test; (iii) an oracle ensemble exceeds the best single model by up to +27.8pp, with complementarity varying by budget; and (iv) a budget-aware XGBoost router captures 14.1% of the oracle gap in a cross-domain setting, with budget features helping within-domain but hurting transfer. The paper concludes that evaluation rankings are parameterized by token budget, and proposes budget-conditioned evaluation protocols.

Significance. If the central empirical claim holds, this is a consequential finding for LLM evaluation practice: leaderboard rankings would need to be reported with an explicit inference budget, and model selection would need to treat budget as a first-class variable. The paper supplies a large, deterministic dataset (56,476 inferences) and includes several protective controls: paired McNemar tests on common non-truncated item sets, a stop-only behavioral taxonomy, and a direct search for correct-at-low-budget/wrong-at-high-budget pairs with finish_reason='stop' at both levels. The discovery that overthinking is largely model-specific rather than item-inherent, and the documentation of non-monotone accuracy curves, are potentially useful empirical contributions that go beyond the existing 'overthinking' literature. The routing experiment, however, contains a comparison problem that weakens the practical contribution (see Major Comment 1), and the abstract overstates the oracle-gap claim (see Major Comment 2). With those issues fixed, the paper would be a solid, if not revolutionary, empirical study.

major comments (3)
  1. [§5, Table 2] The practical routing claim is not supported by the reported numbers because the proposed Router-Scoring is outperformed by the Largest-Always baseline. In Table 2, Largest-Always achieves 23.4% accuracy on the cross-domain GPQA test, while Router-Scoring achieves 22.9%; on the discriminative subset, Largest-Always reaches 57.7% versus Router-Scoring's 56.5%. The text reports only the comparison to Best-Per-Budget (+2.67pp) and states that the router 'captures 14.1% of the oracle gap,' which is misleading when a trivial baseline captures a larger share. The routing contribution should be re-evaluated against Largest-Always, and the claims in the abstract and Section 5 should be corrected or the router strengthened.
  2. [Abstract, §4.3, Figure 5a] The abstract states that the oracle gap is 'most pronounced at constrained budgets,' but this is internally inconsistent with the reported data. On GPQA, the largest absolute oracle gap is +27.8pp at b=4096, the largest budget in the study; on MATH-500 the gap increases monotonically to +12.8pp at b=4096; only on GSM8K does the gap peak at an intermediate budget (b=256, +16.9pp). The text in Section 4.3 correctly notes 'even larger relative gains at lower budgets' for GPQA, but the abstract's phrasing is wrong and should be amended.
  3. [§3, §4.2, Limitations (vi)] The paper's independent variable, max_tokens, is not a commensurable measure of reasoning effort across the evaluated models, and the paper's own data illustrate this: Qwen-3 32B uses internal <think> tokens and remains truncated on 59.7% of GPQA items at b=4096, while LLaMA-3.3 70B drops to 0.5% at the same budget. The paper excludes dedicated reasoning models (o1, DeepSeek-R1) because their dual-stream architecture changes the semantics of max_tokens, but Qwen-3 32B exhibits the same property in a milder form. The title's framing 'how long you let them think' therefore overstates the control that max_tokens provides over the reasoning process. The authors should either exclude Qwen-3 from the main claims or explicitly reframe the independent variable as 'maximum visible output tokens' and temper the title and abstract accordingly; the central ranking-reversal result may survive, but the interpretation needs to be made precise.
minor comments (4)
  1. [§5, Table 2] When reporting the router's improvement, the paper should also report the comparison to Largest-Always in the main text, since that baseline outperforms the router in both columns of Table 2.
  2. [Abstract] The range '3–19% of items' for non-monotone behavior after truncation control is not explicitly tied to a table; please refer to the stop-only rates in Section 4.1 (e.g., 19.1% for LLaMA-3 8B on GPQA, 3.3% for Qwen-3 32B on GSM8K) so readers can verify the range.
  3. [§7, Conclusion] The final sentence should be completed with proper punctuation: 'The question "which model is best?" has no single answer; it depends on how long you let them think.'
  4. [Figure 1] The heatmap labels for GSM8K at b=4096 show 95 for GPT-OSS while Table 6 reports 94.8±1.2; this is a rounding artifact, but a note about rounding would prevent confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all central claims are direct empirical measurements with explicit controls and no load-bearing self-citations.

full rationale

The paper's central claims are all direct empirical observations made under a fixed experimental protocol, not derived quantities that reduce to their own inputs. The ranking reversals, non-monotone taxonomy, oracle gaps, and within-domain/cross-domain router results are computed from the recorded accuracy trajectories and held-out evaluations; no fitted parameter is renamed as a prediction. The router is trained on GSM8K+MATH-500 and tested on GPQA, with ablations reported for budget features, so the routing result is a genuine held-out evaluation rather than a restatement of training data. There are no load-bearing self-citations: all references are to external prior work, and no uniqueness theorem or prior result by the authors is invoked to force a conclusion. The only substantive weakness is conceptual rather than circular: the max_tokens budget may not be commensurable across models with different tokenizers or internal thinking tokens, which threatens the interpretation of the independent variable but does not make any step definitionally equivalent to its inputs. The paper explicitly acknowledges this limitation (Limitations vi) and attempts to control for truncation via the three-tier analysis. Accordingly, no circular step meets the evidentiary standard required by the review rules, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of max_tokens as a comparable budget, the accuracy of answer parsing, and the representativeness of the model and benchmark sample. No free parameters are fitted to derive the central result; the router experiments use trained XGBoost classifiers, but these do not underlie the budget-dependence finding.

assumptions (4)
  • domain assumption The max_tokens parameter is a valid and comparable operationalization of token generation budget across the four models.
    The whole study varies max_tokens; Section 3 defines budgets as max_tokens. Qwen-3 32B's internal <think> tokens (Section 4.2, Figure 8) make this comparability questionable, as the paper itself acknowledges for reasoning models in Limitations (vi).
  • domain assumption Regex-based parsing with exact-match scoring correctly captures model correctness.
    Section 3 and Appendix H describe the parsing, but there is no validation of parse accuracy across the 56,476 outputs; mis-parsed answers would bias the taxonomy and ranking results.
  • domain assumption The four models and three benchmarks are representative enough to support general claims about budget-dependent rankings.
    The paper's Limitations (i) admit that only four models were used; the generalization of the phenomenon is therefore assumed, not established.
  • standard math McNemar's test assumptions are satisfied for the paired binary outcomes.
    Used in Section 4.2 for ranking reversals; the paper does not apply multiple-comparison correction across the many tests.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation." pith.science (2026). https://pith.science/paper/YGGXKUIE

@misc{pith2026260812150,
  author       = {Pith},
  title        = {Pith review of: Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGGXKUIE}},
  note         = {Machine review of arXiv:2608.12150}
}
abstract

Standard evaluation of large language models assumes stable model rankings across inference conditions. We challenge this assumption by varying the token generation budget, i.e., the maximum tokens a model may produce, across seven levels (64--4,096), evaluating four models on three reasoning benchmarks (56,476 inferences). We report four findings: (i) 3--19% of items exhibit non-monotone behavior (accuracy decreasing with more budget), even after controlling for truncation, and this phenomenon is model-specific (cross-model overlap: 6--14%). (ii) Model rankings reverse across budgets on all benchmarks ($p {<} 0.01$, McNemar). (iii) Oracle analysis reveals model complementarity up to $+27.8$pp, most pronounced at constrained budgets. (iv) A budget-aware router captures 14.1% of the oracle gap cross-domain; budget features help within-domain ($+1.6$ to $+5.7$pp) but are domain-specific and hurt transfer ($-1.2$pp). These results argue for budget-conditioned evaluation protocols.

Figures

Figures reproduced from arXiv: 2608.12150 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Behavioral taxonomy across models and benchmarks. Each bar decomposes items into four categories based on how correctness evolves with budget. Non-monotone items (red), indicating “overthinking,” are non-trivial across all settings and most pronounced on GPQA. 4 Budget-Dependent Model Behavior 4.1 Item-Level Behavioral Taxonomy For each model m and item i, we observe a binary correctness trajectory across seven budg… view at source ↗
Figure 3
Figure 3. Non-monotone (overthinking) rates: all budgets vs. stop-only trajectories. Rates decrease after excluding truncated budgets but remain substantial, particularly on GPQA, confirming that overthinking is a genuine reasoning failure. a 3.3% non-monotone rate after filtering (down from 11.7%, where most were truncation artifacts from its verbose <think> tokens). These results confirm that overthinking is a genuine pheno… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: All-items vs. stop-only accuracy on GPQA. Left: standard scoring (truncated = incorrect). Right: accuracy restricted to items where each model completed its generation. Parenthesized values indicate small sample sizes (N < 30). Black borders mark the best model per bud…
Figure 5
Figure 5. Figure 5: Model complementarity varies with budget. (a) The oracle gap (oracle ensemble minus best single model) peaks at low-to-moderate budgets for GSM8K but grows monotonically for harder benchmarks. (b) Mean pairwise Jaccard similarity between models’ correct-answer sets sta…
Figure 6
Figure 6. Figure 6: Router accuracy by budget level (cross-domain, GPQA). The router’s advantage is concentrated at moderate budgets (b=512–2048), where model complementarity is highest and the oracle gap is largest. the budget used. This is especially critical for constrained deployment …
Figure 7
Figure 7. Figure 7: Budget feature impact across evaluation settings. Within￾domain, budget is the most valuable feature (+1.6 to +5.7 pp). Cross-domain, budget features overfit to the training domain and hurt transfer (−1.2 pp). This contrast reveals that budget–accuracy patterns are dom…
Figure 8
Figure 8. Figure 8: Truncation rate by model, budget, and dataset. Qwen-3 32B experiences the highest truncation rates across all budgets, while LLaMA-3.3 70B shows the fastest decline. At b=4096, truncation rates are near zero for all models except Qwen-3 32B (15% on GSM8K, higher on MAT…
Figure 9
Figure 9. Figure 9: Distribution of non-monotone transitions by budget level. The heatmap shows at which budget level non-monotone items first lose a previously correct answer. Most transitions are concentrated at high budgets (b ≥ 1024), especially for GSM8K. 16 [PITH_FULL_IMAGE:figures…
Figure 10
Figure 10. Figure 10: SHAP feature importance for the per-model scoring router. Budget (log2 b) dominates with a mean absolute SHAP value of 2.21, approxi￾mately 6× larger than the next feature (presence of LaTeX: 0.36). Text features and embeddings contribute modestly. F IRT Difficulty An…
Figure 11
Figure 11. Figure 11: IRT-inspired difficulty distribution across benchmarks. Eas￾iness is computed as the fraction of model–budget combinations that answer each item correctly. GSM8K items cluster around easiness 0.5 (mean: 0.507), MATH-500 around 0.25 (mean: 0.253), and GPQA around 0.17 …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 10 canonical work pages

  1. [1]

    https://huggingface.co/spaces/ open-llm-leaderboard/open_llm_leaderboard, 2024

    Open LLM Leaderboard. https://huggingface.co/spaces/ open-llm-leaderboard/open_llm_leaderboard, 2024

    work page 2024

  2. [2]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. InNeurIPS, 2023

    work page 2023

  3. [3]

    Snell, J

    C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling LLM Test-Time Com- pute Optimally Can be More Effective than Scaling Model Parameters. arXiv:2408.03314, 2024

    arXiv 2024

  4. [4]

    Muennighoff, Z

    N. Muennighoff, Z. Yang, et al. s1: Simple Test-Time Scaling. arXiv:2501.19393, 2025

    arXiv 2025

  5. [5]

    Aggarwal, Y

    S. Aggarwal, Y. Arora, and A. Goyal. L1: Controlling How Long A 13 Reasoning Model Thinks With Reinforcement Learning.arXiv:2503.04697, 2025

    arXiv 2025

  6. [6]

    J. Wei, X. Wang, D. Schuurmans, et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InNeurIPS, 2022

    work page 2022

  7. [7]

    X. Chen, Z. Xu, et al. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs.arXiv:2412.21187, 2024

    arXiv 2024

  8. [8]

    Y. Sui, H. Yu, et al. Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models.arXiv:2503.16419, 2025

    arXiv 2025

Show all 19 references
  1. [9]

    Jiang, X

    D. Jiang, X. Ren, and B. Y. Lin. LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. InACL, 2023

    2023

  2. [10]

    K. Lu, H. Yuan, et al. Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models. InNAACL, 2024

    2024

  3. [11]

    Shnitzer, A

    T. Shnitzer, A. Ou, et al. Large Language Model Routing with Benchmark Datasets.arXiv:2309.15789, 2023

    2023 arXiv

  4. [12]

    Sainz, J

    O. Sainz, J. Campos, et al. NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for each Benchmark. InEMNLP Findings, 2023

    2023

  5. [13]

    Kiela, M

    D. Kiela, M. Bartolo, et al. Dynabench: Rethinking Benchmarking in NLP. InNAACL, 2021

    2021

  6. [14]

    Liang, R

    P. Liang, R. Bommasani, et al. Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences, 2023

    2023

  7. [15]

    The LLaMA 3 Herd of Models.arXiv:2407.21783, 2024

    Meta AI. The LLaMA 3 Herd of Models.arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Qwen3 Technical Report.arXiv:2505.09388, 2025

    Qwen Team. Qwen3 Technical Report.arXiv:2505.09388, 2025

    2025 arXiv

  9. [17]

    Cobbe, V

    K. Cobbe, V. Kosaraju, et al. Training Verifiers to Solve Math Word Problems.arXiv:2110.14168, 2021

    2021 arXiv

  10. [18]

    Hendrycks, C

    D. Hendrycks, C. Burns, et al. Measuring Mathematical Problem Solving With the MATH Dataset. InNeurIPS, 2021

    2021

  11. [19]

    D. Rein, B. L. Hou, et al. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. InICLR, 2024. 14 A Additional Experimental Details Model details.All models are evaluated via their standard chat/instruct variants through a unified API. Table 5 provides full identification for all...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.