{"id":"735b1fdf-38dd-4de4-ae6b-23d6622b7e6a","arxiv_id":"2608.12150","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Model rankings in LLM evaluation are budget-dependent: the best-performing model changes with the token generation budget on all three benchmarks tested.","lead":"By varying the maximum token budget from 64 to 4,096 for four large language models on three reasoning benchmarks, this paper shows that which model performs best changes with the budget. The authors document ranking reversals, analyze budget-dependent overthinking behavior, and test a budget-aware routing system.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"max_tokens is not a commensurable 'thinking budget' across models; tokenizer and internal-think-token differences can drive apparent ranking reversals, so the abstract's 'how long you let them think' framing overstates the control the independent variable provides.","rationale":"The reader identified the same load-bearing premise: max_tokens is not a commensurable budget across tokenizers and internal-thinking architectures. I agree this is the most serious threat to the central claim because it undermines the independent variable itself. If the concern lands, the empirical ranking reversals are still real as statements about the max_tokens parameter, but the paper's conceptual contribution—'who thinks best depends on how long you let them'—is not established; rankings could be driven by verbosity and tokenizer compression rather than by the amount of reasoning. The paper's own truncation controls address completion artifacts but not token-semantics differences, and the inclusion of Qwen-3 despite its <think> tokens is inconsistent with the exclusion rationale in Limitations (vi). The proposed test directly isolates the confound: toggling Qwen-3's thinking mode and measuring tokens-per-character would show whether ranking reversals are robust to a more commensurable notion of budget. Since the paper already received a CONDITIONAL verdict and this concern is the same one the reader flagged, the verdict should remain UNCHANGED; the authors should be required to address the commensurability issue or soften the 'thinking' framing.","tokens_in":13611,"tokens_out":7966,"duration_ms":70557,"concrete_test":"Evaluate Qwen-3 32B on GPQA and GSM8K with thinking mode disabled (if the API supports toggling) at all seven budgets and recompute Table 1. Additionally, sample 200 outputs per model-benchmark pair and compute each model's mean characters-per-token; if Qwen-3's participation changes the ranking reversals when think tokens are removed, or if cross-model tokens-per-character differ by more than roughly 20%, the max_tokens budget is not commensurable and the headline reversals are at least partly a token-semantics artifact. A complementary check: re-rank models at budgets rescaled to equal predicted generated characters and see whether any reversals survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that model rankings are parameterized by the token generation budget requires max_tokens (Section 3) to be a commensurable measure of how much reasoning each model is allowed. The paper's own Limitations (vi) concedes this fails for dedicated reasoning models, but Qwen-3 32B is still evaluated despite using internal <think> tokens (Section 4.2); Figure 8 shows it remains truncated on 59.7% of GPQA items at b=4096 while LLaMA-3.3 70B is at 0.5%. More broadly, the four models use different tokenizers, so the same budget b corresponds to different amounts of generated text, reasoning steps, and effective compute. The three-tier truncation controls condition on completion, not on the semantic content of a token, so they cannot remove this bias. Consequently, headline reversals (e.g., GSM8K b=256: LLaMA-3.3 62.4% vs GPT-OSS 48.6%) could reflect each model's token consumption style—LLaMA-3.3's lower truncation at b=256 means it fits more reasoning into the visible budget—rather than a genuine change in relative reasoning ability. The abstract's interpretation 'how long you let them think' overstates what a max_tokens parameter controls.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how the token generation budget (max_tokens) affects LLM evaluation results. It evaluates four open-weight models (8B–70B) on GSM8K, MATH-500, and GPQA at seven budgets (64–4,096), totaling 56,476 greedy-decoded inferences. Four findings are claimed: (i) 3–19% of items show non-monotone accuracy trajectories (accuracy drops with more budget) even after truncation control, with low cross-model overlap; (ii) the best-performing model changes across budgets on all three benchmarks, with several reversals significant by McNemar's test; (iii) an oracle ensemble exceeds the best single model by up to +27.8pp, with complementarity varying by budget; and (iv) a budget-aware XGBoost router captures 14.1% of the oracle gap in a cross-domain setting, with budget features helping within-domain but hurting transfer. The paper concludes that evaluation rankings are parameterized by token budget, and proposes budget-conditioned evaluation protocols.","tokens_in":1607,"tokens_out":1723,"duration_ms":59067,"significance":"If the central empirical claim holds, this is a consequential finding for LLM evaluation practice: leaderboard rankings would need to be reported with an explicit inference budget, and model selection would need to treat budget as a first-class variable. The paper supplies a large, deterministic dataset (56,476 inferences) and includes several protective controls: paired McNemar tests on common non-truncated item sets, a stop-only behavioral taxonomy, and a direct search for correct-at-low-budget/wrong-at-high-budget pairs with finish_reason='stop' at both levels. The discovery that overthinking is largely model-specific rather than item-inherent, and the documentation of non-monotone accuracy curves, are potentially useful empirical contributions that go beyond the existing 'overthinking' literature. The routing experiment, however, contains a comparison problem that weakens the practical contribution (see Major Comment 1), and the abstract overstates the oracle-gap claim (see Major Comment 2). With those issues fixed, the paper would be a solid, if not revolutionary, empirical study.","major_comments":[{"comment":"The practical routing claim is not supported by the reported numbers because the proposed Router-Scoring is outperformed by the Largest-Always baseline. In Table 2, Largest-Always achieves 23.4% accuracy on the cross-domain GPQA test, while Router-Scoring achieves 22.9%; on the discriminative subset, Largest-Always reaches 57.7% versus Router-Scoring's 56.5%. The text reports only the comparison to Best-Per-Budget (+2.67pp) and states that the router 'captures 14.1% of the oracle gap,' which is misleading when a trivial baseline captures a larger share. The routing contribution should be re-evaluated against Largest-Always, and the claims in the abstract and Section 5 should be corrected or the router strengthened.","section":"§5, Table 2"},{"comment":"The abstract states that the oracle gap is 'most pronounced at constrained budgets,' but this is internally inconsistent with the reported data. On GPQA, the largest absolute oracle gap is +27.8pp at b=4096, the largest budget in the study; on MATH-500 the gap increases monotonically to +12.8pp at b=4096; only on GSM8K does the gap peak at an intermediate budget (b=256, +16.9pp). The text in Section 4.3 correctly notes 'even larger relative gains at lower budgets' for GPQA, but the abstract's phrasing is wrong and should be amended.","section":"Abstract, §4.3, Figure 5a"},{"comment":"The paper's independent variable, max_tokens, is not a commensurable measure of reasoning effort across the evaluated models, and the paper's own data illustrate this: Qwen-3 32B uses internal <think> tokens and remains truncated on 59.7% of GPQA items at b=4096, while LLaMA-3.3 70B drops to 0.5% at the same budget. The paper excludes dedicated reasoning models (o1, DeepSeek-R1) because their dual-stream architecture changes the semantics of max_tokens, but Qwen-3 32B exhibits the same property in a milder form. The title's framing 'how long you let them think' therefore overstates the control that max_tokens provides over the reasoning process. The authors should either exclude Qwen-3 from the main claims or explicitly reframe the independent variable as 'maximum visible output tokens' and temper the title and abstract accordingly; the central ranking-reversal result may survive, but the interpretation needs to be made precise.","section":"§3, §4.2, Limitations (vi)"}],"minor_comments":[{"comment":"When reporting the router's improvement, the paper should also report the comparison to Largest-Always in the main text, since that baseline outperforms the router in both columns of Table 2.","section":"§5, Table 2"},{"comment":"The range '3–19% of items' for non-monotone behavior after truncation control is not explicitly tied to a table; please refer to the stop-only rates in Section 4.1 (e.g., 19.1% for LLaMA-3 8B on GPQA, 3.3% for Qwen-3 32B on GSM8K) so readers can verify the range.","section":"Abstract"},{"comment":"The final sentence should be completed with proper punctuation: 'The question \"which model is best?\" has no single answer; it depends on how long you let them think.'","section":"§7, Conclusion"},{"comment":"The heatmap labels for GSM8K at b=4096 show 95 for GPT-OSS while Table 6 reports 94.8±1.2; this is a rounding artifact, but a note about rounding would prevent confusion.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core empirical observation—that rankings change with the max_tokens setting—appears sound and is supported by careful truncation controls. The main issues are localizable: the router evaluation is stacked against a weak baseline, the abstract misstates the oracle-gap budget trend, and the token-commensurability caveat is not applied to Qwen-3. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper reports something the community should take seriously: model rankings on reasoning benchmarks shift as you vary max_tokens, and the effect survives truncation controls. It is not a perfect paper—the router section oversells a baseline it actually loses to, and the abstract overstates one oracle claim—but the core empirical finding is the first systematic multi-model, multi-budget comparison I know of, and it deserves referee time.\n\nWhat is new: prior work on test-time compute scaling and overthinking mostly studies single models or does not compare rankings across budgets. Here they evaluate four models at seven budgets (56,476 deterministic inferences at T=0), compute item-level trajectories, build a behavioral taxonomy, and find 3–19% non-monotone items after truncation control, with low cross-model overlap. They also document oracle gaps up to 27.8pp and provide a budget-aware routing proof of concept. The truncation controls are thoughtful: stop-only and common non-truncated item sets with paired McNemar tests. That is real work.\n\nSoft spots, in proportion: \n\n1. The abstract says oracle gains are “most pronounced at constrained budgets,” but the largest absolute gap (+27.8pp) is at b=4096 on GPQA. The paper’s own Figure 5a shows the GPQA gap growing with budget. That is an internal inconsistency that needs fixing. \n\n2. The router: Router-Scoring reaches 22.9%, below Largest-Always at 23.4%, and the cross-domain ablation shows that removing budget features improves accuracy (22.9% full vs. 24.2% no budget). The paper reports these numbers honestly but emphasizes the +2.67pp over Best-Per-Budget, which is a weak baseline. The claim that budget is the dominant routing signal is only true within-domain; cross-domain, budget actually hurts transfer.\n\n3. Commensurability: max_tokens is not the same “thinking budget” across models. Qwen-3 32B uses internal think tokens and stays truncated on 59.7% of GPQA items at b=4096, while LLaMA-3.3 70B is at 0.5%. Different tokenizers also change what one token means. The ranking-reversal finding remains operationally valid—max_tokens is a real deployment constraint—but the abstract’s “how long you let them think” framing overstates the control the independent variable provides.\n\n4. No code or data are released, XGBoost hyperparameters are not specified, and many McNemar tests are run without multiple-comparison correction. The GPQA sample is only 198 items, so several headline reversals on that benchmark are not significant; the paper does flag this.\n\nOverall, the central empirical claim holds up in substance. This paper is for anyone who builds or reads leaderboards. A serious referee should engage with it; the needed fixes are mostly reporting discipline—correct the abstract, add the missing baseline comparison, release data/code, and discuss token-semantics more carefully. I would bring it to reading group and would cite it as evidence that leaderboard rankings are budget-conditioned.","headline":"A real measurement phenomenon—model rankings shift with max_tokens—but the router section overstates its case and the abstract overclaims one oracle finding.","tokens_in":14383,"tokens_out":2396,"would_cite":true,"duration_ms":21366,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The best LLM on a reasoning benchmark depends on how many tokens it is allowed to generate.","keywords":["token generation budget","budget-dependent evaluation","model ranking reversal","overthinking","LLM routing","non-monotone accuracy","reasoning benchmarks","test-time compute"],"falsifier":"Run the same four models on GSM8K at b=256 and b=4096, then repeat the comparison with outputs truncated by visible character count or by estimated internal reasoning time instead of by token count; if the 70B model remains best at both budgets under those controls, the ranking reversal is a token-measurement artifact rather than a true budget effect.","tokens_in":1444,"feed_emoji":"🧠","tokens_out":1619,"duration_ms":68913,"temperature":0.7,"pith_summary":"Standard evaluations assume that which LLM is best does not depend on how many tokens it is allowed to generate. This paper tests that assumption by running four open-weight models on three reasoning benchmarks at seven max-token budgets, from 64 to 4,096 tokens, for a total of 56,476 inferences. It finds that the best model changes with the budget on every benchmark: for example, the 70B model leads GSM8K at 256 tokens while the 20B model dominates at 4,096 tokens. It also finds that 3-19% of items show accuracy decreasing as budget grows, even after removing truncation, and that these overthinking items barely overlap across models. The upshot is that a benchmark ranking is not a single ordering but a family of orderings parameterized by the token budget.","feed_headline":"LLM rankings flip as the token budget grows","feed_subtitle":"Across three benchmarks the winner changes between 64 and 4,096 max tokens, so no single best model exists.","key_machinery":"The organizing device is a token generation budget b, defined as the max_tokens parameter and treated as an experimental axis. Each model-item pair is scored at seven budgets, producing a seven-bit correctness trajectory that is classified as always-correct, monotone-increasing, non-monotone, or always-wrong. To separate genuine reasoning failures from truncation artifacts, the analysis is run three ways: on all items, on stop-only items, and on common non-truncated items; ranking reversals are tested with paired significance tests on identical item sets. Complementarity is measured through an oracle ensemble and pairwise set-overlap similarity, while routing is implemented with per-model gradient-boosted classifiers that take log2(b) plus text and embedding-derived features as input.","core_discovery":"The paper claims that model rankings are not invariant to the token generation budget. On GSM8K, LLaMA-3.3 70B is the best model at b=256 with 62.4% accuracy, while GPT-OSS 20B takes the lead at b=4096 with 94.8%. Similar reversals occur on MATH-500 and GPQA, with the smallest model ranking first at low budgets on GPQA before being overtaken by larger models. After controlling for truncation, 3-19% of items still show non-monotone behavior, meaning accuracy drops as more tokens are allowed, and 86-94% of these overthinking items are flagged for only one model. An oracle ensemble shows complementarity of up to +27.8 percentage points over the best single model, and a budget-aware router captures 14.1% of that gap in a cross-domain test.","pith_inferences":["A natural extension is to normalize the budget by estimated reasoning tokens rather than emitted tokens; models that hide thinking inside internal tokens would then be comparable, and the non-monotone rates might shrink further.","Because overthinking items hardly overlap across models, a multi-model second-opinion router that lets a second model re-answer at high budgets could reclaim more of the oracle gap than the single-model router tested here.","The domain-specificity of budget features predicts that a router trained separately per domain, or adapted with a small target-domain set, would beat both the static router and the no-budget router by more than the reported margins."],"forward_implications":["A single accuracy number per benchmark underspecifies model quality; any reported ranking should name the token budget at which it was measured.","Deployers with tight latency or cost constraints should evaluate models at their operating budget, not at the generous budgets used by many leaderboards.","Routing systems can gain by adding budget as a feature within a domain, with the paper reporting +1.6 to +5.7 percentage points from budget features in within-domain settings.","Because models complement each other most at constrained budgets, ensemble and selection methods have the largest headroom where compute is scarce.","Truncation-adjusted analyses preserve the main findings, but some apparent reversals are truncation artifacts, so budget studies need a three-tier reporting protocol."],"supporting_citations":[{"why":"Supplies evidence that additional test-time compute can improve performance, motivating the token budget as an evaluation axis.","marker":"[3]"},{"why":"Provides a simple test-time scaling recipe, supporting the premise that changing token budget changes outcomes.","marker":"[4]"},{"why":"Documents overthinking as a failure mode, the phenomenon the paper quantifies at the item level.","marker":"[7]"},{"why":"Proposes early termination for reasoning, providing evidence that more tokens can hurt accuracy.","marker":"[8]"},{"why":"Defines the LLaMA family of models whose rankings are compared across budgets.","marker":"[15]"},{"why":"Defines Qwen-3, the model with internal thought tokens that drives the truncation analyses.","marker":"[16]"},{"why":"Supplies the GSM8K dataset used for the grade-school math experiments.","marker":"[17]"},{"why":"Supplies the MATH-500 benchmark used for the competition-level math experiments.","marker":"[18]"},{"why":"Supplies the GPQA-Diamond benchmark used for the graduate-level science experiments.","marker":"[19]"}],"fun_headline_variants":["Token budget flips LLM rankings on reasoning benchmarks","Winner depends on token budget in LLM evaluation","More tokens can make LLMs worse, new study shows","LLM rankings are budget-dependent, study concludes","Budget-dependent rankings: no universal best LLM"],"cache_read_input_tokens":16384,"weakest_assumption_plain":"The argument assumes that setting the same max_tokens value gives every model the same amount of thinking; if one model spends tokens on internal tags while another spends them on visible reasoning, some of the ranking flips could be an artifact of how the budget is counted.","fun_headline_variants_meta":{"raw":{"variants":["Token budget flips LLM rankings on reasoning benchmarks","Winner depends on token budget in LLM evaluation","More tokens can make LLMs worse, new study shows","LLM rankings are budget-dependent, study concludes","Budget-dependent rankings: no universal best LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1639,"prompt_tokens":931,"completion_tokens":708,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":634}},"tokens_in":547,"tokens_out":708,"duration_ms":6760,"temperature":1.0,"reasoning_tokens":634,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:14:33.609066+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four models on GSM8K at b=256 and b=4096, then repeat the comparison with outputs truncated by visible character count or by estimated internal reasoning time instead of by token count; if the 70B model remains best at both budgets under those controls, the ranking reversal is a token-measurement artifact rather than a true budget effect.","supporting_citations":[{"cited_title":"Hendrycks, C","cited_arxiv_id":null,"evidence_quote":"Supplies the MATH-500 benchmark used for the competition-level math experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPQA-Diamond benchmark used for the graduate-level science experiments."}],"review_version":1}