{"id":"62d7298b-13b9-4f66-bd6a-f075bffa1a54","arxiv_id":"2608.03401","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Matched-horizon comparisons show that a numeric/concision prompt shortens Qwen3 reasoning without consistent accuracy gains, while lower gpt-oss effort wins early mainly by finishing sooner, not by reasoning better.","lead":"Large AI models often 'think' for thousands of tokens before answering; this paper tests prompts and effort settings that shorten that thinking. On Qwen3 and gpt-oss benchmarks, shorter traces mostly reflect earlier stopping rather than better reasoning, and the authors propose an evaluation protocol that separates finishing early from reasoning well.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gpt-oss 'mostly earlier completion' claim depends on a termination-stratum decomposition that mixes natural terminal answers with forced-answer probes; a common-readout re-analysis is needed.","rationale":"The reader's weakest assumption correctly identifies the forced-answer readout for unfinished prefixes as the load-bearing measurement. I agree that this is the key vulnerability, but I localize it more sharply: the readout asymmetry is most consequential in the fixed B=512 termination-stratum decomposition (Table A16), where the dominant lower-complete/high-active stratum compares a natural terminal answer (lower effort) with a forced-answer probe (high effort). The reader's acceptance is well supported by the extensive validation, token-exact replay, positive control, and honest limitations, and I do not think the paper's central conceptual claim—that matched-horizon evaluation separates completion timing from prefix quality—is undermined. However, the quantitative headline 'Most of the 512-token advantage comes from lower effort finishing earlier' is a statement about the size of a decomposition contribution, and that specific estimate has not been shown to be invariant to the choice of readout for the unfinished arm. A symmetric-readout re-analysis is a small, feasible check given the released token-exact data, and it would settle whether the claim is robust or merely an artifact of mixing readout types. I therefore recommend CONDITIONAL rather than UNCHANGED: accept the paper's methodology and the qualitative distinction, but require the common-readout decomposition before the 'most of the advantage' phrasing is treated as established.","tokens_in":53761,"tokens_out":8083,"duration_ms":81331,"concrete_test":"Recompute Table A16's termination-stratum contributions at B=512 using a symmetric readout: for every lower-complete/high-active pair, replace the lower-effort natural terminal answer with the common-context candidate-logit readout of the replayed completed lower-effort block (Table A4 indicates this should agree ~100%), while keeping the high-effort forced-answer probe; equivalently, run the greedy generated-answer continuation for both arms within each stratum. If the lower-complete/high-active contribution remains positive and dominant in all eight model–benchmark–effort comparisons, the 'mostly earlier completion' claim is robust. If it shrinks materially or changes sign in some comparisons, the headline claim needs to be softened to a readout-dependent statement.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline gpt-oss claim in Section 5 ('Earlier completion explains most of the gpt-oss advantage at the fixed B=512 checkpoint') rests on the four-stratum decomposition of Table A16. In the largest positive stratum (lower-complete/high-active), the lower-effort outcome is the model's natural terminal answer, while the high-effort outcome is a candidate-logit forced-answer readout of an artificially truncated, unfinished prefix. The forced-answer probe is only validated against natural answers for completed reasoning blocks (Table A4, 98.3–100% agreement); its behavior on mid-reasoning artificial cuts is unvalidated, and the paper explicitly acknowledges that 'forced-answer readout affects unfinished-prefix accuracy' (Discussion, Limitations). The both-active stratum uses the probe on both arms, which can cancel a shared readout bias, but the lower-complete/high-active stratum cannot cancel it: only the high-effort arm is probed. Because that stratum contributes +3.4 to +15.8 points and is the largest positive component in all eight comparisons, an asymmetric readout artifact at artificial cut points could inflate the 'most of the advantage' conclusion. The matched-horizon continuation checks (Table A14) validate the all-replayable ordering, but they do not re-express the B=512 termination-stratum decomposition under a symmetric readout. The central methodological contribution survives, but the specific quantitative claim about the source of the fixed-checkpoint gap needs this robustness check.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a matched-horizon evaluation protocol for reasoning interfaces that separates whether an intervention shortens a trace by making the model stop sooner or by changing the answer available from an unfinished prefix. It applies the protocol to Qwen3-14B numeric/concision prompting and to gpt-oss-20b/-120b effort settings on GPQA Diamond and a stratified MMLU-Pro subset, with additional calibration on Qwen3-4B/-8B and Omni-MATH-2. The central empirical claims are that the numeric/concision prompt shortens reasoning by 12–17% without a consistent matched-horizon accuracy gain; a concise/early-answer instruction gives a +3.8 point gain at B=512 including +2.7 points for jointly unfinished runs; and lower gpt-oss effort is 14.5–26.3 points more accurate at its own stopping point, with most of the fixed B=512 advantage attributable to earlier completion. The paper includes extensive validation of the forced-answer readout, proper-score analyses, and a scripted positive control.","tokens_in":54017,"tokens_out":11918,"duration_ms":102643,"significance":"If the results hold, the paper makes a valuable methodological contribution by showing that accuracy-versus-trace-length evaluations conflate stopping policy with prefix quality, and it provides a replicable template for separating these factors. The manuscript is unusually careful: it uses paired item–replicate designs, token-exact replay, multiple readout contexts, bootstrap and crossed-intercept inference, validation against natural answers and generated continuations, and a scripted positive control. The claims are presented with appropriate scope restrictions, and the limitations are honestly stated. The paper also ships reproducible code and data, which strengthens the empirical contribution.","major_comments":[],"minor_comments":[{"comment":"The fixed-checkpoint decomposition supporting 'Earlier completion explains most of the gpt-oss advantage at the fixed B=512 checkpoint' mixes natural terminal answers for the lower-complete arm with candidate-logit forced-answer readouts for the high-active arm. The forced-answer readout is validated against natural answers for completed blocks (Table A4) and against generated continuations for the matched-horizon all-replayable comparison (Table A14), but not explicitly for this stratum. Because the lower-complete/high-active stratum is the largest positive component in all eight comparisons, I recommend adding a sensitivity analysis that expresses the B=512 decomposition under a symmetric readout (e.g., common-context replay of completed lower-effort blocks) or uses greedy continuations for the active high arm. The both-active stratum and the matched-horizon continuation checks provide indirect support, so this is a robustness improvement rather than a blocking defect.","section":"Section 5, Table A16"},{"comment":"The sentence 'Most of the 512-token advantage comes from lower effort finishing earlier' could be qualified to note that it holds under the candidate-logit readout and depends on the forced-answer readout's fidelity for unfinished prefixes. The Limitations section acknowledges this, but the abstract states the claim unconditionally; a brief qualifier would align the headline with the measurement caveat.","section":"Abstract and Section 5"}],"recommendation":"minor_revision","confidential_remarks":"The paper is a strong empirical contribution that is likely to be influential for reasoning-interface evaluation. The only substantive concern is the asymmetric-readout decomposition in Table A16; the reader's stress-test is partially mitigated by the matched-horizon continuation checks in Table A14 and by the both-active stratum showing small differences. A sensitivity analysis or a softened claim would fully resolve the issue. I recommend minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nThis is a careful empirical paper worth a serious referee slot. The genuinely new thing is the matched-horizon protocol: it separates \"finishing earlier\" from \"having a better answer available in an unfinished prefix,\" which most earlier work conflates by plotting accuracy against completed trace length. The implementation is unusually rigorous: token-exact replay, candidate-logit scoring, multiple readout contexts, proper-score diagnostics, a scripted positive control, and a full code/data release. The Qwen results are clean: the numeric/concision prompt shortens traces by 12–17% while same-horizon accuracy changes are small and mixed, and the concise/early-answer instruction gives a real 512-token gain driven by both earlier completion and better unfinished prefixes. Those claims look solid.\n\nThe gpt-oss story is mostly solid but has one soft spot that deserves attention. The primary matched-horizon result—lower effort is more accurate than high effort at the same realized length—uses a symmetric readout on both arms and is robust. But the headline quantitative claim that \"most of the fixed B=512 advantage comes from earlier completion\" rests on the termination-stratum decomposition in Table A16. In the largest positive stratum (lower complete / high active), the lower-effort arm's score is its natural terminal answer while the high-effort arm is a forced-answer probe of an artificially truncated prefix. The probe is only validated on completed reasoning; the paper itself says the forced-answer readout affects unfinished-prefix accuracy. If that probe understates what the unfinished high-effort prefix would have answered, the stratum contribution is inflated. The both-active strata, which probe both arms symmetrically, show smaller and mixed differences, so the qualitative direction probably survives—but the specific \"most of the advantage\" quantification needs a common-readout re-analysis of that stratum before I'd trust it as a headline number.\n\nMinor caveats: three global replicates, two model families, multiple-choice tasks for the primary claims, and the text-reconstructed cohorts are properly kept out of the token-exact primary analysis. The limitations section is honest about all of this.\n\nWho it's for: anyone evaluating reasoning interfaces or building anytime/deadline-aware LLM systems. The protocol itself is the takeaway, more than any single benchmark result.\n\nRecommendation: send it to peer review. If I were referee, I'd ask for the symmetric-readout robustness check on the fixed-checkpoint decomposition before accepting the \"mostly earlier completion\" phrasing as stated, but the central methodological contribution stands without it.\n\nBest,","headline":"Careful matched-horizon evaluation that cleanly separates stopping earlier from reasoning better; the gpt-oss 'mostly earlier completion' headline needs a symmetric-readout robustness check before the exact number is trusted.","tokens_in":54549,"tokens_out":3029,"would_cite":true,"duration_ms":26127,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In these tests, a shorter reasoning trace mostly means the model stopped sooner, not that its early reasoning is better.","keywords":["large language models","reasoning interfaces","reasoning budgets","matched-horizon evaluation","candidate-logit probe","reasoning effort","Qwen3","gpt-oss"],"falsifier":"Take a sample of unfinished high-effort gpt-oss prefixes at the lower-effort stopping point, record the candidate-logit forced answer, then let each high-effort run finish naturally and compare the two answers; if agreement on these unfinished prefixes drops far below the 98.3-100% agreement the paper reports for completed prefixes, the matched-horizon gap is an artifact of the probe rather than a property of the reasoning.","tokens_in":53518,"feed_emoji":"⏱️","tokens_out":4432,"duration_ms":37037,"temperature":0.7,"pith_summary":"The paper argues that a shorter reasoning trace is not evidence of better early reasoning: a model may simply stop sooner. It introduces matched-horizon probes that cut paired runs at the same token budget and separately record whether each run has finished, what answer a stopped unfinished run gives, and how much probability the run places on the correct option. Across Qwen3-14B and gpt-oss models on GPQA Diamond and MMLU-Pro, the numeric/concision prompt shortens typical reasoning by 12-17% without a consistent same-horizon accuracy gain, while lower gpt-oss effort beats high effort at a 512-token checkpoint mainly because it finishes earlier, not because its unfinished reasoning is better. The paper concludes that reasoning-interface evaluations should report completion by a deadline and the stopped-prefix answer as separate quantities.","feed_headline":"Shorter reasoning mostly means stopping sooner, not better answers","feed_subtitle":"Matched-horizon tests on Qwen3 and gpt-oss separate early completion from what an unfinished prefix actually knows.","key_machinery":"The matched-horizon probe: paired runs of the same item are cut at a common token budget $B$ (for Qwen) or at the lower-effort run's realized terminal length $L_c$ (for gpt-oss); a token-exact replay reconstructs the stopped prefix without retokenizing, and a candidate-logit readout closes the reasoning block and scores every valid one-token spelling of each answer option from full-vocabulary logits. Completion is tracked separately, giving the terminal-aware policy accuracy $Y^{z,T}_{ir}(B)$, the both-active common-context contrast, and a four-stratum decomposition. This machinery is what lets the paper attribute an early-accuracy difference to finishing sooner versus carrying a better unfinished prefix.","core_discovery":"In the tested settings, a reasoning control that produces a shorter trace and a control that produces a better answer at a fixed horizon are empirically distinct. For Qwen3-14B the numeric/concision prompt reduces geometric-mean reasoning length by 12-17% while same-horizon accuracy deltas sit within about two percentage points of zero; the only consistent early gain comes from a separate concise/early-answer instruction (+3.8 points at 512 tokens, +2.7 points among pairs where both runs are unfinished). For gpt-oss-20b and gpt-oss-120b, completed low- and medium-effort reasoning blocks beat matched-horizon high-effort prefixes by 14.5 to 26.3 points, but at the fixed 512-token checkpoint the dominant contribution is the lower-effort policy having already stopped and committed, while jointly unfinished prefixes differ by small and mixed amounts. The paper claims that accuracy-versus-trace-length curves conflate stopping policy with prefix quality; the right question is whether an answer is available at a horizon, whether it is correct, and whether it came from stopping or from the reasoning itself.","pith_inferences":["If the matched-horizon decomposition is adopted by benchmark suites, reported speedups from reasoning controls will need to be re-derived as stopping-policy effects plus prefix-quality effects, which would change how reasoning-effort settings are compared.","The same probe could be applied to other controls, such as trained token budgets, control tokens, or draft chains, to test whether their early gains are mostly early stopping; the paper's case-study design is a template rather than a proof for other models.","A direct testable extension would measure whether the candidate-logit forced answer on an unfinished prefix matches the answer the model gives after a small number of extra continuation tokens; the paper's generated-answer continuation partially checks this, and a larger unfinished-only cohort would tighten or weaken the central decomposition."],"forward_implications":["Accuracy plotted against trace length should be split into correct completion by a deadline and correctness of the stopped-prefix answer, because the two move independently.","For gpt-oss under a tight 512-token budget, lower effort is often the better deployment choice, but the advantage shrinks or reverses when both policies remain unfinished, and high effort can recover higher accuracy if allowed to finish.","A prompt that announces a token budget can shorten reasoning without improving the early answer, and matching the announced number to the stop does not by itself help.","Proper scores qualify accuracy: lower-effort terminal answers often concentrate probability on the chosen option, so Brier score and conditional log loss diverge, and evaluations that report only accuracy or only log loss will mislead."],"supporting_citations":[{"why":"Supplies the gpt-oss model family and the low/medium/high reasoning-effort settings that define the gpt-oss comparison.","marker":"[19]"},{"why":"Supplies the GPQA Diamond benchmark used for the main paired comparisons.","marker":"[31]"},{"why":"Supplies the MMLU-Pro benchmark and the stratified 500-item subset used for the main and mechanism experiments.","marker":"[32]"},{"why":"Supplies the Qwen3 model family and its thinking-mode generation used in the prompt comparisons.","marker":"[35]"},{"why":"Documents Qwen's external stop-then-answer thinking-budget procedure that the candidate-logit readout emulates.","marker":"[36]"},{"why":"Prior work on probing stopped reasoning trajectories that the matched-horizon forced-answer method builds on.","marker":"[28]"},{"why":"Prior work on early answer commitment that motivates comparing unfinished prefixes at matched horizons.","marker":"[29]"},{"why":"Prior work on interruptibility of reasoning models that grounds the stopped-run readout.","marker":"[30]"},{"why":"Frames the distinction between answer value and computation allocation that the matched-horizon evaluation formalizes.","marker":"[13]"}],"fun_headline_variants":["Shorter reasoning rarely means better answers at the same token budget","Shorter reasoning is mostly an early stop, not a quality win","Matching horizons shows shorter reasoning comes from stopping sooner","A tighter deadline favors early finishers, not better reasoning","Short reasoning often just means the model quit earlier"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the candidate-logit forced-answer readout on an unfinished reasoning prefix reveals the answer that would actually be available if the run were stopped there; if this proxy misreads unfinished prefixes, the paper's separation between completion timing and prefix quality is mis-measured.","fun_headline_variants_meta":{"raw":{"variants":["Shorter reasoning rarely means better answers at the same token budget","Shorter reasoning is mostly an early stop, not a quality win","Matching horizons shows shorter reasoning comes from stopping sooner","A tighter deadline favors early finishers, not better reasoning","Short reasoning often just means the model quit earlier"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000688,"raw_usage":{"total_tokens":3186,"prompt_tokens":1078,"completion_tokens":2108,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":2027}},"tokens_in":694,"tokens_out":2108,"duration_ms":12615,"temperature":1.0,"reasoning_tokens":2027,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:50:04.851609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of unfinished high-effort gpt-oss prefixes at the lower-effort stopping point, record the candidate-logit forced answer, then let each high-effort run finish naturally and compare the two answers; if agreement on these unfinished prefixes drops far below the 98.3-100% agreement the paper reports for completed prefixes, the matched-horizon gap is an artifact of the probe rather than a property of the reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPQA Diamond benchmark used for the main paired comparisons."},{"cited_title":"Qwen3 thinking budget","cited_arxiv_id":null,"evidence_quote":"Documents Qwen's external stop-then-answer thinking-budget procedure that the candidate-logit readout emulates."},{"cited_title":"Probing the trajectories of reasoning traces in large language models.arXiv preprint arXiv:2601.23163, 2026","cited_arxiv_id":null,"evidence_quote":"Prior work on probing stopped reasoning trajectories that the matched-horizon forced-answer method builds on."},{"cited_title":"Are Large Reasoning Models Interruptible?","cited_arxiv_id":"2510.11713","evidence_quote":"Prior work on interruptibility of reasoning models that grounds the stopped-run readout."},{"cited_title":"An analysis of time-dependent planning","cited_arxiv_id":null,"evidence_quote":"Frames the distinction between answer value and computation allocation that the matched-horizon evaluation formalizes."}],"review_version":2}