{"id":"bfa47e09-10cc-4dc4-b747-bd4735cc172a","arxiv_id":"2608.07968","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across seven reasoning models and two domains, shared test-time compute is allocated by presentation position rather than by question value or difficulty, and planning prompts widen coverage without improving prioritization.","lead":"This paper tests whether reasoning models can divide one shared token budget across multiple exam questions with different point values, instead of spending all effort on one question at a time. It finds that models mostly work through questions in the order presented, ignore point values, and leave many questions untouched as exams grow.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Every headline statistic in §§4.1–4.5 is computed from Qn-marker segmentation that the paper itself calls approximate; without a segmentation validity check, the position-driven allocation result may be an artifact of trace parsing.","rationale":"The paper is a well-controlled empirical study: same exams reused across orderings, scorings, and prompts; partial correlations separate position from difficulty; a second code domain and a uniform-allocation baseline in Appendix B provide independent support for the qualitative direction. But the central measures are not directly observable; they are reconstructions from marker-based trace segmentation. The reader's weakest assumption correctly identifies this. I agree. The concern is empirically resolvable and the paper itself conditions its conclusions on the approximation, so the appropriate verdict is unchanged: conditional acceptance pending segmentation validation and the missing budget report. I do not see an internal inconsistency in the derivations; the issue is external validity of the measurement pipeline. If the proposed annotation test shows large disagreement, the verdict should move toward rejection; if it confirms the parser, the paper's main claim stands. I therefore leave the reader's conditional verdict in place.","tokens_in":17456,"tokens_out":4750,"duration_ms":55097,"concrete_test":"Take a stratified sample of 100 Phase-1 traces (e.g., 20 per model family across N=5/10/20), and have two independent annotators label each contiguous reasoning span with the question it actually addresses (or 'none'), using the full trace and a protocol that does not reveal whether the span appears before or after a marker. Measure token-level agreement between the manual labels and the strict Qn-marker segmentation of §3.5. Then recompute the four headline quantities—effort–position Spearman ρ, order–position ρ, coverage |W|/N, and zero-token rate—using the manual labels. If the order–position correlation falls below 0.5 or the effort–position correlation weakens by more than 0.15, the central 'greedy sequential solver' claim should be revised; if the recomputed values are within 0.05 of the reported ones, the segmentation concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is the reconstruction of per-question effort and solving order from the single free-form trace by strict Q1/Q2 marker segmentation (§3.5). Equations (5)–(7), the work set W, token effort t_i, centroid c_i, and all correlations in Figure 2, Tables 1–5, and the score-rate grids in §4.4 rest on that attribution. The Limitations section concedes the attribution 'remains an approximation' and that a rigorous notion of effort is open. No validation rate, inter-annotator agreement, or sample of manually inspected traces is reported. If models interleave questions without markers, discuss a later question before its marker, or abandon marker conventions under budget pressure, then zero-token rates, coverage, effort–position, and order–position correlations could be systematically biased toward the sequential story. The paper's mitigation (work-set threshold and token centroid) reduces mention artifacts but cannot repair a segmentation that misassigns whole spans. The unreported math budget B compounds this: B is never stated for the Omni-MATH runs, so the budget-pressure comparisons across N cannot be audited. The concern is not that traces are certainly mis-segmented; it is that the paper's central claim is conditional on an unvalidated parser, and the authors acknowledge this.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a shared-budget multi-question evaluation setting in which a reasoning LLM must distribute one finite token budget across N scored questions, then uses strict Qn-marker segmentation of a single free-form reasoning trace to recover per-question token effort, a work set, and a solving order. Across Omni-MATH and CRUXEval-O, with several open-weight and API reasoning models, it reports that models solve largely in presentation order, front-load effort on early questions, respond weakly to point values, and that these tendencies become more pronounced as N grows; explicit planning prompts improve coverage but not value- or difficulty-aware prioritization. The paper concludes that global budget allocation is a distinct capability not captured by per-question evaluation.","tokens_in":17708,"tokens_out":6690,"duration_ms":71260,"significance":"If the result holds, the paper identifies a genuinely new failure mode for reasoning models: allocation of test-time compute across questions is driven by presentation position rather than expected score. The study has real strengths: matched exams that reuse the same questions across scoring schemes, orders, and prompts; partial Spearman correlations that separate position from difficulty; several converging analyses (work-set selection against chance overlap, a uniform-allocation baseline, a code-domain replication); and model-adaptive value-density references that are not fitted to the phenomena being explained. However, the central quantitative story rests on an unvalidated trace-segmentation convention and on an unreported shared budget for the main domain, so the paper is not yet at the standard required for publication.","major_comments":[{"comment":"All of the paper's headline quantities—token effort t_i, the work set W, the solving-order centroid c_i, and therefore every correlation in Figure 2 and Tables 1-5—are computed by splitting the Phase 1 trace at strict 'Qn:' markers. The Limitations paragraph concedes that this attribution 'remains an approximation,' but no validation is reported: there is no manually inspected sample of traces, no agreement rate, and no audit of traces in which the model discusses a question before its marker, interleaves questions, or abandons the marker convention under budget pressure. If such traces are common, the zero-token rates, coverage, and effort-position/order-position correlations could be systematically biased toward the sequential story. Appendix F partially addresses the mention-order artifact, but that is not a substitute for validating the parser itself. I ask for (i) a quantitative validation of the marker segmentation on a sample of traces and (ii) a robustness check that recomputes the headline correlations after excluding or re-attributing anomalous traces.","section":"Section 3.5, Equations (5)-(7), Limitations"},{"comment":"The shared budget B for the Omni-MATH experiments is never stated. Section 3.1 defines B and Section 3.2 describes exam construction, but neither gives a value; the only concrete budget reported in the paper is B=3,000 for CRUXEval-O in Section 4.5 and Appendix D.1. This matters because the paper's central 'budget pressure magnifies the failure' claim compares behavior across N=5, 10, and 20: without B, the reader cannot determine whether the per-question allocation B/N is actually shrinking, whether the budget is exhausted in most runs, or whether B was scaled with N. Please report B for each Omni-MATH condition, ideally along with the fraction of runs reaching the token limit, or the budget-pressure comparisons cannot be audited.","section":"Sections 3.1-3.2; Appendix D.1"},{"comment":"The work-set threshold of at least 200 tokens or at least two segments, and the zero-token rate, are used to define coverage and support the claim that coverage collapses as the exam grows. These thresholds are arbitrary, and no sensitivity analysis is reported. Because Table 1 is one of the main pieces of evidence for the claim that 'an increasing fraction of questions is never meaningfully considered,' please report coverage and zero-token rates under a range of thresholds (for example, 100, 200, and 400 tokens) or show the full per-question token distributions, so that the pattern is not an artifact of the chosen cutoffs.","section":"Equation (6); Table 1"}],"minor_comments":[{"comment":"The caption contains garbled text ('score= are visible to LLMs ,!--score='); this should be fixed.","section":"Figure 1"},{"comment":"The partial Spearman formula should use parentheses around the numerator for readability, and the text has 'an Spearman correlation' where it should read 'a Spearman correlation.'","section":"Equation (8), Section 4.1"},{"comment":"The abstract says the tendencies 'become more pronounced as the number of questions grows,' but the order-position correlation is reported as stable across N (0.68, 0.66, 0.69); the text should clarify that the strengthening effect is mainly in effort-position correlation and coverage, not in order-position correlation.","section":"Section 4.1, abstract"},{"comment":"The 'By chance' overlap should state explicitly that it is k/N computed per exam, and the tables should report the mean k (or work-set size) for the conditions shown; as printed, the reader cannot reproduce the chance values from the reported quantities.","section":"Tables 2, 10"},{"comment":"The calibration sentence 'approximately match B/N × average reference cost per question' is not clearly connected to the actual computation (B=3,000, described as roughly three times the median reference cost of about 995 tokens); please rephrase so the budget-calibration procedure is unambiguous.","section":"Appendix D.1"},{"comment":"The asterisked claims of statistical significance based on a 95% confidence interval are not accompanied by the interval definitions or any multiple-comparison correction; please provide the details in the appendix or in the table notes.","section":"Tables 3, 4"}],"recommendation":"major_revision","confidential_remarks":"This is a well-scoped empirical study with converging evidence, and the two load-bearing gaps—the unvalidated trace segmentation and the unreported Omni-MATH budget—appear fixable without changing the experimental design. I would therefore view a major-revision decision as appropriate rather than a rejection. The authors should also ensure that the quantitative claims in the abstract are matched by reported numbers, particularly the budget values and segmentation validation statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: this paper is worth reading. It defines a new evaluation axis—shared-budget allocation across questions—and reports a consistent, cross-model failure: reasoning models spend the budget by presentation order, not by question value or difficulty, and the effect strengthens as the exam grows. The design is genuinely controlled: the same question sets are reused across ordering, scoring, and prompting conditions, position effects are separated from difficulty with partial Spearman correlations, and the code-domain replication includes a deterministic judge. The work-set and token-centroid definitions are a sensible response to the obvious 'mention vs attempt' problem, and Appendix F shows the aggregate correlations are not an artifact of the work-set threshold. Credit is due for the triangulation: seven models, three exam lengths, four scoring schemes, and a second domain.\n\nThe soft spot is the one the authors admit: per-question effort is recovered by splitting the free-form trace at Q1/Q2 markers, and that attribution is unvalidated. The paper calls it an approximation but reports no check—no manual inspection rate, no agreement measure, no alternative attribution comparison. If models interleave questions or stop following the marker convention under budget pressure, then the effort, coverage, and order correlations could be systematically biased toward the sequential narrative. Appendix F addresses mention artifacts within the segmentation, not mis-segmentation of whole spans. This is fixable but load-bearing. Also, the Omni-MATH token budget B is never reported; only CRUXEval-O gives its calibration. Without B, the budget-pressure comparisons across N cannot be audited.\n\nThe central qualitative pattern is plausible and well triangulated, so I would not reject the paper on these grounds. But I would want the segmentation validated and B reported before trusting the headline numbers. The framework will be reused, and the finding is decision-relevant for anyone setting multi-question inference budgets. A serious referee should engage with it; my recommendation is conditional acceptance with those checks.","headline":"A well-designed new benchmark and a credible finding that models allocate shared reasoning budgets by position rather than value, but the per-question effort measurements hinge on an unvalidated trace-segmentation and the token budget is unreported.","tokens_in":18250,"tokens_out":2731,"would_cite":true,"duration_ms":29164,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reasoning models ration token budgets by position, not value.","keywords":["test-time compute","budget allocation","reasoning models","shared-budget evaluation","metacognitive control","position bias","token effort","knapsack-style selection"],"falsifier":"Run the same exams with high-value questions placed last and low-value questions first under an explicit planning prompt: if the model's solving order shifts to follow value, with order-value correlation rising well above zero while order-position correlation falls, or if the effort-position gradient disappears under a marker-free attribution method, the paper's central claim would be refuted.","tokens_in":17271,"feed_emoji":"🧠","tokens_out":7971,"duration_ms":83217,"temperature":0.7,"pith_summary":"The paper asks whether reasoning models can ration a fixed pool of thinking tokens across several questions at once, the way a test-taker must divide limited time among exam items. It introduces an exam-style evaluation in which a model sees all questions, their point values, and one shared token budget, then must maximize total score. Across several open-weight and API reasoning models, the paper finds the same answer: models behave as greedy sequential solvers, working through questions in presentation order, spending progressively less on later items, and barely responding to stated point values. The pattern sharpens as exams grow longer and extends from mathematics to code reasoning. The conclusion is that strong per-question reasoning does not imply the ability to allocate compute strategically across questions.","feed_headline":"Reasoning models ration token budgets by position, not value","feed_subtitle":"When questions share one token cap, even frontier models front-load early questions and ignore point values.","key_machinery":"The machinery is a controlled exam-style task with per-question point values and one global token cap, plus a trace-based measurement pipeline. Each model's single free-form reasoning trace is split at the Q1, Q2 markers; a question enters the work set only if it receives at least 200 attributed tokens or two distinct segments, and its solving position is the rank of the token-weighted centroid of its segments. Partial Spearman correlations separate presentation position from difficulty as predictors of effort and order, while a knapsack-style value-density reference, point value divided by the model's own isolated token cost with a correctness indicator, provides the benchmark for whether the model picks the right questions.","core_discovery":"The central discovery is that shared-budget allocation is governed by input position, not expected value. Averaged over models, token effort correlates negatively with presentation position, strengthening from about -0.17 at five questions to -0.48 at twenty, while the order in which models substantively work on questions tracks presentation order closely. Correlations with stated point values are near zero, and the work set covers only about 40 percent of questions at twenty questions, with an average 51 percent of questions receiving zero tokens. When the same questions are presented hard-first under reversed scoring, models keep following the prompt order and lose substantial score. Explicit planning prompts spread effort more evenly but do not make allocation value- or difficulty-aware.","pith_inferences":["Beyond the paper, if the position-driven pattern is causal, presenting high-value questions first and low-value questions last should recover most of the lost score without any change to the model, a cheap testable design rule for deployments.","The framework suggests a training-side fix: reinforcement learning with a global token cap and score objective, evaluated against uniform per-question allocation, could convert object-level reasoning strength into strategic rationing, something the paper does not attempt.","The results imply that model rankings from single-question token-scaling studies may not transfer to batch or multi-query serving settings where latency and cost budgets are shared.","A minimal diagnostic for future reasoning models would be the effort-position and order-position correlations at twenty questions; values near zero would indicate allocation is not purely sequential."],"forward_implications":["Per-question evaluation misses a distinct capability: a model can solve single problems well yet fail to ration compute when problems compete for one budget.","As the number of competing questions grows, allocation becomes more positional, with effort concentrating on early items and an increasing fraction of questions receiving no substantive work.","Explicit planning instructions change the spread of effort but not the basis of prioritization, so prompt engineering alone does not fix allocation.","The same position-driven, value-insensitive pattern appears in code reasoning, suggesting the failure is general across task domains.","A meaningful share of the shared budget goes to questions the same model cannot solve even with an isolated high-budget attempt."],"supporting_citations":[{"why":"Supplies the Omni-MATH problems and difficulty labels from which the exams are constructed.","marker":"(Gao et al., 2024)"},{"why":"Supplies the CRUXEval-O code-reasoning questions used to test generalization of the allocation pattern.","marker":"(Gu et al., 2024)"},{"why":"Provides the DeepSeek-R1-Distill-Qwen open-weight reasoning models evaluated in the study.","marker":"(Guo et al., 2025b)"},{"why":"Provides the Qwen3 open-weight reasoning models evaluated in the study.","marker":"(Yang et al., 2025)"},{"why":"Provides the DeepSeek-V4 Flash and Pro API models evaluated in the study.","marker":"(DeepSeek-AI et al., 2026)"},{"why":"Defines the knapsack problem used to frame value-density analysis of the models' question selection.","marker":"(Kellerer et al., 2004)"},{"why":"Provides the REST multi-problem stress-testing setting that this paper contrasts with its joint-execution allocation study.","marker":"(Pan et al., 2025)"},{"why":"Provides the concurrent TRIAGE plan-based allocation evaluation that this paper contrasts with its emergent allocation analysis.","marker":"(Nazi and Dipta, 2026)"}],"fun_headline_variants":["Reasoning models spend tokens by position, not value","Models front-load compute, ignore question points","Shared token budgets: LLMs favor early questions","Test-time compute: position overrides value in LLMs","LLMs ration thinking by order, not importance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a single free-form reasoning trace can be split reliably into per-question tokens by matching the Q1, Q2, ... labels from the prompt; the paper itself flags that this segmentation is only an approximation, and if models do not consistently organize their traces around those markers, the measured effort, order, and coverage numbers could be artifacts of the splitting rather than true allocation behavior.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning models spend tokens by position, not value","Models front-load compute, ignore question points","Shared token budgets: LLMs favor early questions","Test-time compute: position overrides value in LLMs","LLMs ration thinking by order, not importance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1571,"prompt_tokens":872,"completion_tokens":699,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":626}},"tokens_in":488,"tokens_out":699,"duration_ms":8153,"temperature":1.0,"reasoning_tokens":626,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:36:02.877703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same exams with high-value questions placed last and low-value questions first under an explicit planning prompt: if the model's solving order shifts to follow value, with order-value correlation rising well above zero while order-position correlation falls, or if the effort-position gradient disappears under a marker-free attribution method, the paper's central claim would be refuted.","supporting_citations":[],"review_version":1}