Pith. sign in

REVIEW 3 major objections 6 minor 79 references

Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reasoning models ration token budgets by position, not value.

desk verdict A well-designed new benchmark and a credible finding that models allocate shared reasoning budgets by position rather than value, but the per-question effort measurements hinge on an unvalidated trace-segmentation and the token budget is unreported. read the letter →

arxiv 2608.07968 v2 pith:6AYVX2RC submitted 2026-08-08 cs.CL cs.AI

Chenrui Fan , Yize Cheng , Ming Li , Yongyuan Liang , Tianyi Zhou , Soheil Feizi This is my paper · ORCID
classification cs.CLcs.AI
keywords test-timecomputebudgetallocationreasoningmodelsshared-budgetevaluationmetacognitivecontrolpositionbiastokeneffortknapsack-styleselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether reasoning models can ration a fixed pool of thinking tokens across several questions at once, the way a test-taker must divide limited time among exam items. It introduces an exam-style evaluation in which a model sees all questions, their point values, and one shared token budget, then must maximize total score. Across several open-weight and API reasoning models, the paper finds the same answer: models behave as greedy sequential solvers, working through questions in presentation order, spending progressively less on later items, and barely responding to stated point values. The pattern sharpens as exams grow longer and extends from mathematics to code reasoning. The conclusion is that strong per-question reasoning does not imply the ability to allocate compute strategically across questions.

What carries the argument

The machinery is a controlled exam-style task with per-question point values and one global token cap, plus a trace-based measurement pipeline. Each model's single free-form reasoning trace is split at the Q1, Q2 markers; a question enters the work set only if it receives at least 200 attributed tokens or two distinct segments, and its solving position is the rank of the token-weighted centroid of its segments. Partial Spearman correlations separate presentation position from difficulty as predictors of effort and order, while a knapsack-style value-density reference, point value divided by the model's own isolated token cost with a correctness indicator, provides the benchmark for whether the model picks the right questions.

What would settle it

Run the same exams with high-value questions placed last and low-value questions first under an explicit planning prompt: if the model's solving order shifts to follow value, with order-value correlation rising well above zero while order-position correlation falls, or if the effort-position gradient disappears under a marker-free attribution method, the paper's central claim would be refuted.

Watch

Extended reading notes

Core claim

The central discovery is that shared-budget allocation is governed by input position, not expected value. Averaged over models, token effort correlates negatively with presentation position, strengthening from about -0.17 at five questions to -0.48 at twenty, while the order in which models substantively work on questions tracks presentation order closely. Correlations with stated point values are near zero, and the work set covers only about 40 percent of questions at twenty questions, with an average 51 percent of questions receiving zero tokens. When the same questions are presented hard-first under reversed scoring, models keep following the prompt order and lose substantial score. Explicit planning prompts spread effort more evenly but do not make allocation value- or difficulty-aware.

Load-bearing premise

The load-bearing assumption is that a single free-form reasoning trace can be split reliably into per-question tokens by matching the Q1, Q2, ... labels from the prompt; the paper itself flags that this segmentation is only an approximation, and if models do not consistently organize their traces around those markers, the measured effort, order, and coverage numbers could be artifacts of the splitting rather than true allocation behavior.

Editorial extensions

If this is right

  • Per-question evaluation misses a distinct capability: a model can solve single problems well yet fail to ration compute when problems compete for one budget.
  • As the number of competing questions grows, allocation becomes more positional, with effort concentrating on early items and an increasing fraction of questions receiving no substantive work.
  • Explicit planning instructions change the spread of effort but not the basis of prioritization, so prompt engineering alone does not fix allocation.
  • The same position-driven, value-insensitive pattern appears in code reasoning, suggesting the failure is general across task domains.
  • A meaningful share of the shared budget goes to questions the same model cannot solve even with an isolated high-budget attempt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if the position-driven pattern is causal, presenting high-value questions first and low-value questions last should recover most of the lost score without any change to the model, a cheap testable design rule for deployments.
  • The framework suggests a training-side fix: reinforcement learning with a global token cap and score objective, evaluated against uniform per-question allocation, could convert object-level reasoning strength into strategic rationing, something the paper does not attempt.
  • The results imply that model rankings from single-question token-scaling studies may not transfer to batch or multi-query serving settings where latency and cost budgets are shared.
  • A minimal diagnostic for future reasoning models would be the effort-position and order-position correlations at twenty questions; values near zero would indicate allocation is not purely sequential.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces a shared-budget multi-question evaluation setting in which a reasoning LLM must distribute one finite token budget across N scored questions, then uses strict Qn-marker segmentation of a single free-form reasoning trace to recover per-question token effort, a work set, and a solving order. Across Omni-MATH and CRUXEval-O, with several open-weight and API reasoning models, it reports that models solve largely in presentation order, front-load effort on early questions, respond weakly to point values, and that these tendencies become more pronounced as N grows; explicit planning prompts improve coverage but not value- or difficulty-aware prioritization. The paper concludes that global budget allocation is a distinct capability not captured by per-question evaluation.

Significance. If the result holds, the paper identifies a genuinely new failure mode for reasoning models: allocation of test-time compute across questions is driven by presentation position rather than expected score. The study has real strengths: matched exams that reuse the same questions across scoring schemes, orders, and prompts; partial Spearman correlations that separate position from difficulty; several converging analyses (work-set selection against chance overlap, a uniform-allocation baseline, a code-domain replication); and model-adaptive value-density references that are not fitted to the phenomena being explained. However, the central quantitative story rests on an unvalidated trace-segmentation convention and on an unreported shared budget for the main domain, so the paper is not yet at the standard required for publication.

major comments (3)
  1. [Section 3.5, Equations (5)-(7), Limitations] All of the paper's headline quantities—token effort t_i, the work set W, the solving-order centroid c_i, and therefore every correlation in Figure 2 and Tables 1-5—are computed by splitting the Phase 1 trace at strict 'Qn:' markers. The Limitations paragraph concedes that this attribution 'remains an approximation,' but no validation is reported: there is no manually inspected sample of traces, no agreement rate, and no audit of traces in which the model discusses a question before its marker, interleaves questions, or abandons the marker convention under budget pressure. If such traces are common, the zero-token rates, coverage, and effort-position/order-position correlations could be systematically biased toward the sequential story. Appendix F partially addresses the mention-order artifact, but that is not a substitute for validating the parser itself. I ask for (i) a quantitative validation of the marker segmentation on a sample of traces and (ii) a robustness check that recomputes the headline correlations after excluding or re-attributing anomalous traces.
  2. [Sections 3.1-3.2; Appendix D.1] The shared budget B for the Omni-MATH experiments is never stated. Section 3.1 defines B and Section 3.2 describes exam construction, but neither gives a value; the only concrete budget reported in the paper is B=3,000 for CRUXEval-O in Section 4.5 and Appendix D.1. This matters because the paper's central 'budget pressure magnifies the failure' claim compares behavior across N=5, 10, and 20: without B, the reader cannot determine whether the per-question allocation B/N is actually shrinking, whether the budget is exhausted in most runs, or whether B was scaled with N. Please report B for each Omni-MATH condition, ideally along with the fraction of runs reaching the token limit, or the budget-pressure comparisons cannot be audited.
  3. [Equation (6); Table 1] The work-set threshold of at least 200 tokens or at least two segments, and the zero-token rate, are used to define coverage and support the claim that coverage collapses as the exam grows. These thresholds are arbitrary, and no sensitivity analysis is reported. Because Table 1 is one of the main pieces of evidence for the claim that 'an increasing fraction of questions is never meaningfully considered,' please report coverage and zero-token rates under a range of thresholds (for example, 100, 200, and 400 tokens) or show the full per-question token distributions, so that the pattern is not an artifact of the chosen cutoffs.
minor comments (6)
  1. [Figure 1] The caption contains garbled text ('score= are visible to LLMs ,!--score='); this should be fixed.
  2. [Equation (8), Section 4.1] The partial Spearman formula should use parentheses around the numerator for readability, and the text has 'an Spearman correlation' where it should read 'a Spearman correlation.'
  3. [Section 4.1, abstract] The abstract says the tendencies 'become more pronounced as the number of questions grows,' but the order-position correlation is reported as stable across N (0.68, 0.66, 0.69); the text should clarify that the strengthening effect is mainly in effort-position correlation and coverage, not in order-position correlation.
  4. [Tables 2, 10] The 'By chance' overlap should state explicitly that it is k/N computed per exam, and the tables should report the mean k (or work-set size) for the conditions shown; as printed, the reader cannot reproduce the chance values from the reported quantities.
  5. [Appendix D.1] The calibration sentence 'approximately match B/N × average reference cost per question' is not clearly connected to the actual computation (B=3,000, described as roughly three times the median reference cost of about 995 tokens); please rephrase so the budget-calibration procedure is unambiguous.
  6. [Tables 3, 4] The asterisked claims of statistical significance based on a 95% confidence interval are not accompanied by the interval definitions or any multiple-comparison correction; please provide the details in the appendix or in the table notes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the findings are empirical measurements with independent controls; the acknowledged trace-segmentation limitation is a validity risk, not a circular derivation.

full rationale

The paper is an empirical study; its central claims are direct measurements of token allocation from model traces, not quantities derived by fitting a model to the conclusion. Token effort (Eq. 5), the work set (Eq. 6), and solving order (Eq. 7) are computed from the Qn-marker segmentation described in Section 3.5; the paper itself flags this attribution as an approximation in Limitations, making it a measurement-validity caveat rather than a circular step, because the segmentation is not fitted to or defined by the reported behavior (models can and do depart from prompt order, e.g., QW order-position correlations of 0.40-0.55). The value-density analysis in Section 4.2 (Eq. 9) uses an independent high-budget reference condition to estimate per-question cost w_i and solvability; that reference is external to the shared-budget runs and is used only to interpret selection, not to construct the position-driven finding. Comparisons against chance overlap, uniform allocation, and perturbed order/scoring conditions are external baselines. Self-citations to prior work by the same authors (Fan et al. 2025; Li et al. 2025d; Li et al. 2026) appear only as related-work context and are not load-bearing support for the paper's conclusions. No equation or claim reduces by construction to its inputs; the central result is an observed behavioral pattern, with the stated marker-segmentation limitation being a potential bias rather than a circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on methodological assumptions about trace segmentation, difficulty labels, LLM judging, and the cost proxy, plus hand-chosen experimental constants. None of these is a fitted parameter in the sense of a model trained to reproduce the phenomenon; the free parameters are experimental thresholds and budgets.

free parameters (3)
  • shared math budget B = not reported in paper
    The global token cap for Omni-MATH exams is never specified in the main text or appendices; only the CRUXEval-O value B=3000 is given. The comparisons across N=5, 10, and 20 rest on how B behaves as N grows, making this a load-bearing experimental constant.
  • work-set thresholds = 200 tokens; at least 2 segments
    Equation 6 defines substantive work as t_i>=200 or |S_i|>=2. Coverage, work-set size, and solving order all depend on these hand-chosen thresholds, and no threshold ablation is reported.
  • CRUXEval-O budget multiplier = 3 times median reference cost, B=3000
    Appendix D.1 calibrates the code budget as approximately three times the median high-budget reference cost per correctly answered question; the multiplier 3 is a hand choice.
assumptions (4)
  • domain assumption Per-question effort can be recovered by splitting the single reasoning trace at Qn markers.
    Introduced in Section 3.5 and used throughout Section 4; the Limitations section admits this is an approximation and that a fully rigorous notion of effort remains open.
  • domain assumption Omni-MATH difficulty labels are valid for constructing exams and as an analysis signal.
    Sections 3.2 and 4.1 use benchmark difficulty labels to build aligned and reversed scoring and to compute difficulty correlations, without validating the labels against actual model costs.
  • domain assumption The GPT-5 LLM judge accurately assesses mathematical answer equivalence.
    Appendix A describes an LLM-as-judge for math answers but reports no accuracy or agreement analysis.
  • domain assumption Independent high-budget token usage is a valid model-adaptive proxy for per-question cost w_i in Equation 9.
    Section 4.2 uses isolated 40,960-token attempts to estimate value density; the paper notes w_i is not the minimum cost, but the selection conclusions depend on this proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions." pith.science (2026). https://pith.science/paper/6AYVX2RC

@misc{pith2026260807968,
  author       = {Pith},
  title        = {Pith review of: Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6AYVX2RC}},
  note         = {Machine review of arXiv:2608.07968}
}
read the original abstract

Reasoning language models increasingly use test-time compute to improve performance, but existing evaluations typically study this compute one question at a time. Yet when multiple problems share an end-to-end cost or latency constraint, models must decide how to divide limited inference compute among them. We introduce an exam-style evaluation framework for studying this setting, in which a model must distribute one shared token budget across questions with different difficulty and point values to maximize its total score. Across several open and frontier reasoning models, we find that models fail to allocate a shared budget strategically across questions of varying difficulties and values. Models behave largely as greedy sequential solvers: they prioritize questions by presentation order, front-load effort on early questions, and remain insensitive to value, with these tendencies becoming more pronounced as the number of questions grows. Explicit planning prompts spread compute more evenly but do not produce value- or difficulty-aware prioritization. The same behavioral pattern extends from mathematical to code reasoning. These findings establish global budget allocation as a distinct capability that is not captured by conventional per-question evaluation and remains a challenge for current reasoning models.

Figures

Figures reproduced from arXiv: 2608.07968 by the authors.

Figure 1
Figure 1. Three evaluation regimes. (a) Standard: one question with an unrestricted budget. (b) A separate budget cap for each question. (c) Our setting: N scored questions compete for one global budget, which tests cross-question allocation. model must know not only how to solve a prob￾lem, but also which problems are worth attempting, when to give up, and when to return. Conven￾tional one-question-at-a-time evaluation conce… view at source ↗
Figure 2
Figure 2. Relationships between allocation behavior and presentation position, difficulty, and point value under the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Score rate (%, also shown by shading) and, in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The prompt used during inference. The exact wording varies slightly depending on the experimental [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The instruction to GPT-5 for it to serve as an LLM-as-a-judge. cost of a single question, spreading it more evenly is largely unavailable even when instructed [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 35 canonical work pages

  1. [1]

    Pranjal Aggarwal and Sean Welleck. 2025. https://arxiv.org/abs/2503.04697 L1: Controlling how long a reasoning model thinks with reinforcement learning . Preprint, arXiv:2503.04697

    arXiv 2025

  2. [3]

    Chi, Xuezhi Wang, and Denny Zhou

    Xinyun Chen, Ryan A. Chi, Xuezhi Wang, and Denny Zhou. 2024. https://arxiv.org/abs/2402.08939 Premise order matters in reasoning with large language models . Preprint, arXiv:2402.08939

    arXiv 2024

  3. [4]

    Zhoujun Cheng, Jungo Kasai, and Tao Yu. 2023. Batch prompting: Efficient inference with large language model apis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 792--810

    work page 2023

  4. [5]

    DeepSeek-AI, Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyu Hou, Chenhao Xu, Chenze Shao, Chong Ruan, Conner Sun, and 300 others. 2026. https://arxiv.org/abs/2606.19348 Deepseek-v4: Towards highly efficient million-token context i...

    arXiv 2026

  5. [6]

    Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. 2025. https://arxiv.org/abs/2504.06514 Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill? Preprint, arXiv:2504.06514

    arXiv 2025

  6. [7]

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. 2024. https://arxiv.org/abs/2410.07985 Omni-math: A universal olympiad level mathematic benchmark for l...

    arXiv 2024

  7. [8]

    Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. https://proceedings.mlr.press/v235/gu24c.html CRUXE val: A benchmark for code reasoning, understanding and execution . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p...

    2024

  8. [9]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025 a . https://doi.org/10.1038/s41586-025-09422-z Deepseek- R1 incentivizes reasoning in LLMs through reinforcement ...

    doi:10.1038/s41586-025-09422-z 2025

Show all 79 references
  1. [13]

    Kellerer, U

    H. Kellerer, U. Pferschy, and D. Pisinger. 2004. https://books.google.com/books?id=u5DB7gck08YC Knapsack Problems . Springer

    2004

  2. [14]

    Woosuk Kwon. 2025. vLLM: An Efficient Inference Engine for Large Language Models. Ph.D. thesis, UC Berkeley

    2025

  3. [15]

    Junyan Li, Wenshuo Zhao, Yang Zhang, and Chuang Gan. 2025 a . Steering LLM thinking with budget guidance. arXiv preprint arXiv:2506.13752

    2025 arXiv

  4. [16]

    Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, Yupeng Hou, Fuxiao Liu, and Tianyi Zhou. 2025 b . Mosaic-it: Cost-free compositional data synthesis for instruction tuning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 25287--25318

    2025

  5. [20]

    Siyuan Ma, Bo Gao, Zikai Xiao, Hailong Wang, Xinlei Yu, Rui Qian, Jiayu Qian, Luqi Gong, and Yang Liu. 2026. https://arxiv.org/abs/2603.28135 Cot2-meta: Budgeted metacognitive control for test-time reasoning . Preprint, arXiv:2603.28135

    2026

  6. [21]

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. 2025. https://arxiv.org/abs/2504.09858 Reasoning models can be effective without thinking . Preprint, arXiv:2504.09858

    2025 arXiv

  7. [22]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori B Hashimoto. 2025. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural...

    2025

  8. [23]

    Zabir Al Nazi and Shubhashis Roy Dipta. 2026. Triage: Evaluating prospective metacognitive control in LLMs under resource constraints. arXiv preprint arXiv:2605.13414

    2026 arXiv

  9. [24]

    OpenAI. 2024. https://cdn.openai.com/o1-system-card-20241205.pdf OpenAI o1 System Card

    2024

  10. [25]

    Vicky Zhao, Conghui He, and Lijun Wu

    Zhuoshi Pan, Qizhi Pei, Yu Li, Qiyao Sun, Zinan Tang, H. Vicky Zhao, Conghui He, and Lijun Wu. 2025. REST : Stress testing large reasoning models by asking multiple problems at once. arXiv preprint arXiv:2507.10541

    2025 arXiv

  11. [27]

    Nicolò De Sabbata, Theodore R

    C. Nicolò De Sabbata, Theodore R. Sumers, Badr AlKhamissi, Antoine Bosselut, and Thomas L. Griffiths. 2025. https://arxiv.org/abs/2410.05563 Rational metareasoning for large language models . Preprint, arXiv:2410.05563

    2025 arXiv

  12. [31]

    Xiangqi Wang, Yue Huang, Yanbo Wang, Xiaonan Luo, Kehan Guo, Yujun Zhou, and Xiangliang Zhang. 2025 a . https://arxiv.org/abs/2505.17312 Adareasoner: Adaptive reasoning enables more flexible thinking in large language models . Preprint, arXiv:2505.17312

    2025

  13. [32]

    Zhengxiang Wang, Jordan Kodner, and Owen Rambow. 2025 b . Evaluating llms with multiple problems at once. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM ^2 ), pages 178--199

    2025

  14. [36]

    Zhiyuan Zhai, Bingcong Li, Bingnan Xiao, Ming Li, and Xin Wang. 2026. https://arxiv.org/abs/2604.14853 Adaptive test-time compute allocation for reasoning llms via constrained policy optimization . Preprint, arXiv:2604.14853

    2026 arXiv

  15. [38]

    Muyang Zhao, Qi Qi, and Hao Sun. 2026. ROI -reasoning: Rational optimization for inference via pre-computation meta-cognition. arXiv preprint arXiv:2601.03822

    2026

  16. [39]

    MathArena: Evaluating

    Mislav Balunović and Jasper Dekoninck and Ivo Petrov and Nikola Jovanović and Martin Vechev , year=. MathArena: Evaluating. 2505.23281 , archivePrefix=

    arXiv

  17. [40]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and Zhang, Xiaokang and Yu, Xingkai and Wu, Yu and Wu, Z. F. and Gou, Zhibin and Shao, Zhihong and Li, Zhuoshu and Gao, Zi...

  18. [41]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    s1: Simple test-time scaling , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  19. [42]

    The Effect of Question Order on Evaluations of Test Performance: Can the Bias Dissolve? , volume =

    Bard, Gabriele and Weinstein-Jones, Jude , year =. The Effect of Question Order on Evaluations of Test Performance: Can the Bias Dissolve? , volume =. The Quarterly Journal of Experimental Psychology , doi =

  20. [43]

    Understanding performance in test taking: The role of question difficulty order , journal =

    Lina Anaya and Nagore Iriberri and Pedro Rey-Biel and Gema Zamarro , keywords =. Understanding performance in test taking: The role of question difficulty order , journal =. 2022 , issn =. doi:https://doi.org/10.1016/j.econedurev.2022.102293 , url =

    2022

  21. [44]

    arXiv preprint arXiv:2412.16720 , year=

    Openai o1 system card , author=. arXiv preprint arXiv:2412.16720 , year=

    arXiv

  22. [45]

    Bowman , year=

    David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman , year=. 2311.12022 , archivePrefix=

    arXiv

  23. [46]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    2021

  24. [47]

    Measuring Ability, Speed, or Both?

    Goldhammer, Frank , journal =. Measuring Ability, Speed, or Both?. 2015 , doi =

    2015

  25. [48]

    Psychometrika , volume =

    A Hierarchical Framework for Modeling Speed and Accuracy on Test Items , author =. Psychometrika , volume =. 2007 , doi =

    2007

  26. [49]

    2025 , eprint=

    Reasoning Models Can Be Effective Without Thinking , author=. 2025 , eprint=

    2025

  27. [50]

    2025 , eprint=

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. 2025 , eprint=

    2025

  28. [51]

    2020 , eprint=

    Scaling Laws for Neural Language Models , author=. 2020 , eprint=

    2020

  29. [52]

    Charlie Snell and Jaehoon Lee and Kelvin Xu and Aviral Kumar , year=. Scaling. 2408.03314 , archivePrefix=

    arXiv

  30. [53]

    Evolving Deeper

    Kuang-Huei Lee and Ian Fischer and Yueh-Hua Wu and Dave Marwood and Shumeet Baluja and Dale Schuurmans and Xinyun Chen , year=. Evolving Deeper. 2501.09891 , archivePrefix=

    arXiv

  31. [54]

    2024 , eprint=

    A Simple Model of Inference Scaling Laws , author=. 2024 , eprint=

    2024

  32. [55]

    2023 , eprint=

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. 2023 , eprint=

    2023

  33. [56]

    Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =

    Kojima, Takeshi and Gu, Shixiang Shane and Reid, Machel and Matsuo, Yutaka and Iwasawa, Yusuke , title =. Proceedings of the 36th International Conference on Neural Information Processing Systems , articleno =. 2022 , isbn =

    2022

  34. [57]

    Schoenfeld ' s Anatomy of Mathematical Reasoning by Language Models

    Li, Ming and Fan, Chenrui and Cheng, Yize and Feizi, Soheil and Zhou, Tianyi. Schoenfeld ' s Anatomy of Mathematical Reasoning by Language Models. Proceedings of the 64th Annual Meeting of the A ssociation for C omputational L inguistics (Volume 1: Long Papers). 2026. doi:10.1...

    2026 doi

  35. [58]

    Understanding the Thinking Process of Reasoning Models: A Perspective from Schoenfeld ' s Episode Theory

    Li, Ming and Zhang, Nan and Fan, Chenrui and Jiao, Hong and Fu, Yanbin and Peters, Sydney and Xu, Qingshu and Lissitz, Robert and Zhou, Tianyi. Understanding the Thinking Process of Reasoning Models: A Perspective from Schoenfeld ' s Episode Theory. Proceedings of the 2025 Con...

    2025 doi

  36. [59]

    2025 , eprint=

    Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill? , author=. 2025 , eprint=

    2025

  37. [60]

    arXiv preprint arXiv:2506.06998 , year=

    What makes reasoning models different? follow the reasoning leader for efficient decoding , author=. arXiv preprint arXiv:2506.06998 , year=

    arXiv

  38. [61]

    Do NOT Think That Much for 2+3=?

    Xingyu Chen and Jiahao Xu and Tian Liang and Zhiwei He and Jianhui Pang and Dian Yu and Linfeng Song and Qiuzhi Liu and Mengfei Zhou and Zhuosheng Zhang and Rui Wang and Zhaopeng Tu and Haitao Mi and Dong Yu , year=. Do NOT Think That Much for 2+3=?. 2412.21187 , archivePrefix=

    arXiv

  39. [62]

    2025 , eprint=

    Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models , author=. 2025 , eprint=

    2025

  40. [63]

    2025 , eprint=

    Efficient Reasoning Models: A Survey , author=. 2025 , eprint=

    2025

  41. [64]

    2025 , eprint=

    L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning , author=. 2025 , eprint=

    2025

  42. [65]

    Bairu Hou and Yang Zhang and Jiabao Ji and Yujian Liu and Kaizhi Qian and Jacob Andreas and Shiyu Chang , year=. Think. 2504.01296 , archivePrefix=

    arXiv

  43. [66]

    2024 , eprint=

    Toward Adaptive Reasoning in Large Language Models with Thought Rollback , author=. 2024 , eprint=

    2024

  44. [67]

    2025 , eprint=

    AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Models , author=. 2025 , eprint=

    2025

  45. [68]

    2505.20258 , archivePrefix=

    Siye Wu and Jian Xie and Yikai Zhang and Aili Chen and Kai Zhang and Yu Su and Yanghua Xiao , year=. 2505.20258 , archivePrefix=

  46. [69]

    2024 , eprint=

    Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models , author=. 2024 , eprint=

    2024

  47. [70]

    Steering

    Li, Junyan and Zhao, Wenshuo and Zhang, Yang and Gan, Chuang , journal=. Steering

  48. [71]

    Hao Wen and Xinrui Wu and Yi Sun and Feifei Zhang and Liye Chen and Jie Wang and Yunxin Liu and Yunhao Liu and Ya-Qin Zhang and Yuanchun Li , year=. Budget. 2508.17196 , archivePrefix=

    arXiv

  49. [72]

    Token-Budget-Aware

    Tingxu Han and Zhenting Wang and Chunrong Fang and Shiyu Zhao and Shiqing Ma and Zhenyu Chen , year=. Token-Budget-Aware. 2412.18547 , archivePrefix=

    arXiv

  50. [73]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  51. [74]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  52. [75]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  53. [76]

    2025 , school=

    vLLM: An Efficient Inference Engine for Large Language Models , author=. 2025 , school=

    2025

  54. [77]

    2024 , editor =

    Gu, Alex and Roziere, Baptiste and Leather, Hugh James and Solar-Lezama, Armando and Synnaeve, Gabriel and Wang, Sida , booktitle =. 2024 , editor =

    2024

  55. [78]

    Vicky and He, Conghui and Wu, Lijun , journal=

    Pan, Zhuoshi and Pei, Qizhi and Li, Yu and Sun, Qiyao and Tang, Zinan and Zhao, H. Vicky and He, Conghui and Wu, Lijun , journal=

  56. [79]

    Zhao, Muyang and Qi, Qi and Sun, Hao , journal=

  57. [80]

    Triage: Evaluating Prospective Metacognitive Control in

    Nazi, Zabir Al and Dipta, Shubhashis Roy , journal=. Triage: Evaluating Prospective Metacognitive Control in

  58. [81]

    2026 , eprint=

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=. 2026 , eprint=

    2026

  59. [82]

    2004 , publisher=

    Knapsack Problems , author=. 2004 , publisher=

    2004

  60. [83]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track , pages=

    Batch prompting: Efficient inference with large language model apis , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track , pages=

    2023

  61. [84]

    Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM ^2 ) , pages=

    Evaluating llms with multiple problems at once , author=. Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM ^2 ) , pages=

  62. [85]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Mosaic-IT: Cost-free compositional data synthesis for instruction tuning , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  63. [86]

    2024 , eprint=

    Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies , author=. 2024 , eprint=

    2024

  64. [87]

    Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies

    Wang, Junlin and Jain, Siddhartha and Zhang, Dejiao and Ray, Baishakhi and Kumar, Varun and Athiwaratkun, Ben. Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process...

    2024 doi

  65. [88]

    and Ballesteros, Miguel and Chilton, Lydia and Yu, Zhou and Roth, Dan

    Zhang, Xuanming and Ashrafi, Shwan and Mirsaidova, Aziza and Rezaeian, Amir H. and Ballesteros, Miguel and Chilton, Lydia and Yu, Zhou and Roth, Dan. Budget-Aware Anytime Reasoning with LLM -Synthesized Preference Data. Findings of the A ssociation for C omputational L inguist...

    2026 doi

  66. [89]

    2026 , eprint=

    CoT2-Meta: Budgeted Metacognitive Control for Test-Time Reasoning , author=. 2026 , eprint=

    2026

  67. [90]

    2024 , eprint=

    Premise Order Matters in Reasoning with Large Language Models , author=. 2024 , eprint=

    2024

  68. [91]

    Characterizing Positional Bias in Large Language Models: A Multi-Model Evaluation of Prompt Order Effects

    Schilcher, Patrick and Karasin, Dominik and Sch. Characterizing Positional Bias in Large Language Models: A Multi-Model Evaluation of Prompt Order Effects. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.1124

    2025 doi

  69. [92]

    Principles of metareasoning , journal =

    Stuart Russell and Eric Wefald , abstract =. Principles of metareasoning , journal =. 1991 , issn =. doi:https://doi.org/10.1016/0004-3702(91)90015-C , url =

    1991 doi

  70. [93]

    2025 , eprint=

    Rational Metareasoning for Large Language Models , author=. 2025 , eprint=

    2025

  71. [94]

    2026 , eprint=

    Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization , author=. 2026 , eprint=

    2026

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.