{"id":"bc1e5f9b-40a9-4a93-9581-31e8e9904c8b","arxiv_id":"2505.00127","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMs exhibit a non-monotonic length-accuracy pattern, overthinking easy questions and underthinking hard ones, and simple length-preference tuning reduces output length without major accuracy loss.","lead":"This paper measures how the length of an LLM's reasoning steps relates to whether its final answer is correct, using two small reasoning models on math word problems. It finds that the models write too much for easy questions, too little for hard ones, and that training them to prefer shorter answers can cut length while keeping accuracy mostly intact.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Recompute the headline length-accuracy curve on the Medium subset: the claimed fixed-question non-monotonicity may be an artifact of averaging across questions of different difficulty.","rationale":"The reader's verdict of CONDITIONAL already identifies the correlational nature of the study and the under/overthinking interpretation as the weakest spot. My concern is more specific and more upstream: the headline sample-level evidence itself, the Acc_r curves, is an aggregate over questions with strongly correlated baseline difficulty and length. Because of this, the curve does not isolate a within-question effect, despite the paper's claim in Section 4 that sample-level analysis holds the question fixed. This is a load-bearing issue because the abstract and conclusion present the non-monotonic fixed-question accuracy as the paper's main empirical finding, and the Section 6 length-preference experiments are motivated by it. The paper does include a Medium-question correlation analysis in Appendix B, which is a step in the right direction, but it does not recompute the headline Acc_r curves on that subset. My proposed check is direct and inexpensive: use the already-defined Medium set and accuracy bins to see whether the non-monotonicity is a within-question phenomenon or a between-question artifact. Given the current evidence, the CONDITIONAL verdict is appropriate and should not be changed until this check is run; the concern strengthens the conditionality but does not, by itself, move the verdict to REJECT or UNVERDICTED.","tokens_in":12261,"tokens_out":6904,"duration_ms":79482,"concrete_test":"Re-run the Figure 1 analysis (L_r and Acc_r over the N=10 length-ranked samples) restricted to the Medium questions defined in Section 5.1, i.e., questions with both correct and incorrect samples, and additionally stratified by baseline accuracy bins (e.g., exactly k = 1..9 correct out of 10). Add bootstrap confidence intervals clustered by question. If the non-monotonic peak persists within these strata, the fixed-question claim survives; if the curve flattens or becomes monotone decreasing, the headline finding is a composition artifact rather than a within-question property.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in Section 4.1 is that, for a fixed question, accuracy is non-monotonic in reasoning length. The evidence is Figure 1, which plots Acc_r: the fraction of questions whose r-th shortest sampled response is correct, averaged over all dataset questions. This aggregate mixes Easy questions (usually correct and short), Hard questions (usually incorrect and long), and Medium questions, so the curve can exhibit a peak even if no single question has the claimed non-monotonic relationship. The early rise can be driven by Medium questions where correct samples are comparatively longer, and the later decline by Easy questions where correct samples are the shortest; the last rank also tends to be an extreme order statistic, especially with the 8k token cap. Thus the statement that for a fixed question accuracy first rises then falls with length is not directly tested by the reported aggregate. The authors explicitly note in Section 5 that the causal reading is ambiguous, but the aggregate Acc_r curve inherits the same difficulty-composition confound and needs to be conditioned on within-question outcome variability before the over/underthinking interpretation can be supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically studies how reasoning length relates to answer correctness for two 1.5B reasoning models, DeepSeek-R1-Distill and DeepScaler-R1-Preview, on GSM8K and MATH. For each question it samples N=10 responses, ranks them by length, and analyzes the averaged accuracy at each length rank. The authors claim three main findings: (1) for a fixed question, accuracy is non-monotonic in reasoning length, rising then falling; (2) models overthink easy questions and underthink hard ones, indicating poor calibration of reasoning length to perceived difficulty; and (3) applying SimPO with a length-only preference signal on unlabeled data reduces generation length by roughly 30-60% while maintaining acceptable accuracy. The paper includes sample-level rank statistics, question-level difficulty splits with t-tests, and a preference-optimization intervention.","tokens_in":12462,"tokens_out":5537,"duration_ms":60665,"significance":"If the central claims were fully supported, the paper would make a useful contribution to the growing literature on reasoning-length calibration in LLMs. The strengths include a transparent sampling protocol with stated hyperparameters, rank-based metrics that are easy to reproduce, explicit t-test tables, and an honest acknowledgment in Section 5 that the causal interpretation of length-correctness correlations is ambiguous. The SimPO intervention is a concrete, falsifiable test of whether length-only preferences can reduce cost without large accuracy loss. However, the headline fixed-question non-monotonicity claim is not directly tested by the reported aggregated curves, and the under/overthinking interpretation rests on correlational and between-set comparisons that do not establish the claimed difficulty-miscalibration mechanism. These issues are load-bearing for the paper's main conclusions, so the manuscript needs additional analysis before the claims can be accepted.","major_comments":[{"comment":"The claim that 'for a fixed question, accuracy is non-monotonic in reasoning length' is not established by the reported Acc_r curve. Acc_r is the fraction of questions whose r-th shortest sample is correct, averaged over all questions. This is an aggregate over questions of widely different difficulty, so the curve can be non-monotonic even if every individual question has a monotone or flat length-accuracy relationship. For example, the early rise can be driven by Medium questions whose correct samples are comparatively longer, and the later decline can be driven by Easy questions whose shortest samples are already correct. The final rank is also an extreme order statistic and may be affected by the 8k token cap. To support the fixed-question claim, the authors should condition on within-question outcome variability: for instance, plot Acc_r restricted to Medium questions, compute a per-question measure of whether correctness improves or degrades with length rank, or fit a mixed-effects logistic regression with a question-level random intercept. The existing correlations in Appendix B.2 on Medium questions are related but do not directly replace this analysis.","section":"Section 4.1, Figure 1, and Appendix A.2"},{"comment":"The text states that 'a consistent non-monotonic trend' is observed across models and datasets, but the same paragraph reports that R1-Preview on MATH achieves its best accuracy at r*=0 with accuracy declining at higher ranks. That is a monotonically decreasing pattern, not a non-monotonic one. The blanket claim should be qualified to the model-dataset pairs where a rise-then-fall pattern is actually observed, and the single decreasing case should be discussed as an important exception rather than folded into the general statement.","section":"Section 4.1, text after Figure 1"},{"comment":"The underthinking/overthinking interpretation is not supported by the t-test comparisons as presented. For hard questions, Table 3 shows several non-significant or directionally inconsistent comparisons, and the conclusion that models 'do not consistently recognize or reflect increased difficulty' is drawn from the absence of a predicted pattern across sets that differ in intrinsic difficulty. Since the sets are defined by N=10 all-correct/all-incorrect outcomes, the split is arbitrary and the 'needed length' for a question is never measured. The paper itself acknowledges in Section 5 that longer responses may make a question appear more difficult than it truly is, which undermines the causal story. I recommend reframing these results as exploratory evidence or hypotheses, and adding a direct within-question test: for Medium questions with at least one correct sample, compare the lengths of correct versus incorrect samples, and check whether the probability of error increases with relative length after controlling for question difficulty.","section":"Section 5.1, Tables 2 and 3"},{"comment":"The claim that length-only SimPO preference optimization 'significantly reduces' length while maintaining 'acceptable accuracy' is based on a single training run without confidence intervals, multiple seeds, or a precise definition of acceptable accuracy. Figure 4 shows accuracy fluctuating during training, so it is unclear which training step is used for the headline 30-60% reduction and what the corresponding accuracy is. The authors should report exact numbers at a selected checkpoint, include error bars over at least three seeds, and state the accuracy drop relative to the base model. Without this, the practical recommendation that length-only preferences can replace correctness-based preferences is not robustly supported.","section":"Section 6, Figures 4 and 5"}],"minor_comments":[{"comment":"The 'Mean ± Std' entries, such as 1374.82 ± 92.34, appear to be standard errors rather than standard deviations given the sample sizes; if so, please relabel the column as 'Mean ± SE' or report standard deviations.","section":"Table 1"},{"comment":"The main text says that for over 60% of questions across all model-dataset combinations the shortest sample is correct, but Appendix B.1 reports over 80% for R1-Preview on both datasets and over 60% for R1-Distill. Please reconcile these numbers.","section":"Section 4.1 vs. Appendix B.1"},{"comment":"The citation 'Fu et al. proposes Dynasor-CoT...' is missing a year and is incomplete; please provide the full reference.","section":"Related Work"},{"comment":"The caption contains a typo: 'shared-sard set' should be 'shared-hard set'.","section":"Table 3 caption"},{"comment":"The naming of 'M1-Advantage' and 'M2-Advantage' sets is confusing, especially in the hard-question table where a set that is hard for one model but not the other is labeled as the other model's advantage set. Consider using explicit set definitions such as Q^hard_i \\ Q^hard_j in each panel.","section":"Section 5.1, Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has potentially useful empirical material and an honest limitations section, but the central fixed-question non-monotonicity claim needs a within-question analysis, and the under/overthinking interpretation needs to be reframed as a hypothesis. If the authors can supply the Medium-subset Acc_r curves, per-question monotonicity tests, and a small multi-seed SimPO experiment, the revised version would be much stronger. I do not see evidence of circularity; the issues are missing support rather than forced conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper is worth a read, but the headline result—that accuracy is non-monotonic in reasoning length for a fixed question—is not backed by the evidence as presented. The aggregate curve in Figure 1 mixes questions of different difficulty, so it can produce a peak even if no single question behaves that way. What survives the critique is the length-only preference optimization experiment, which is practical and reasonably executed.\n\nThe paper does a few things well. It tries to separate sample-level from question-level effects, which is the right frame. The cross-model comparison on shared-easy and advantage sets is a clever way to get at perceived difficulty. The SimPO experiment is a genuine intervention with clear hyperparameters, and the finding that you can cut length 30–60% without much accuracy loss, using only unlabeled data, is useful for people doing inference-cost work. The authors also clearly flag the ambiguity in their own interpretation in Section 5, and the limitations section is honest.\n\nThe soft spots are real. The central non-monotonicity claim relies on Acc_r, which is the average over all questions of whether the r-th shortest response is correct. As the stress-test note says, this aggregate confounds question difficulty: easy questions tend to be correct and short, hard ones incorrect and long, and medium ones somewhere in between. The early rise could be medium questions where correct samples are slightly longer; the later fall could be easy questions where the longest samples are errors or extreme order statistics. The paper doesn't show a within-question curve conditioned on difficulty, so \"for a fixed question\" is not actually tested. The Easy/Medium/Hard split itself is based on ten samples per question, which is noisy. There are also no confidence intervals or repeated runs on the trade-off curves. And the over/underthinking language goes beyond the correlational evidence, which the authors half-admit but still use in framing.\n\nI agree with the reader's conditional take, and I think the stress-test concern genuinely lands on the paper. That said, this is not a desk-reject case. The topic is timely, the length-preference experiment is a useful contribution, and the flaws are fixable. A serious referee could push the authors to condition the analysis on question difficulty, report per-question evidence, add error bars, release code and data, and soften the causal claims.","headline":"A useful empirical study whose headline fixed-question non-monotonicity claim is not supported by the aggregate curve, but whose length-only preference optimization experiment is solid and deserves a serious referee.","tokens_in":12966,"tokens_out":3394,"would_cite":false,"duration_ms":34310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For a fixed question, accuracy rises with reasoning length up to a point, then falls; models overthink easy problems, underthink hard ones, and length-only preference optimization cuts output by 30–60%.","keywords":["reasoning length","overthinking","underthinking","preference optimization","SimPO","chain-of-thought","math reasoning","length-accuracy tradeoff"],"falsifier":"Truncate long responses on solvable questions to the median length of correct responses; if accuracy does not fall, extra reasoning is not the cause of the decline. Conversely, force additional reasoning steps on hard questions that currently get short wrong answers; if accuracy does not rise, those failures are not caused by insufficient thinking.","tokens_in":12075,"feed_emoji":"🧠","tokens_out":5781,"duration_ms":58586,"temperature":0.7,"pith_summary":"This paper asks whether more reasoning tokens always make a language model more accurate, and answers no. Across two math benchmarks and two small reasoning models, accuracy first rises as responses get longer, then falls once reasoning becomes excessively long. The authors interpret the rise-and-fall as two failure modes: models overthink easy problems they can solve quickly, and underthink hard problems they cannot solve, apparently misjudging difficulty. They also show that preference-tuning a model to prefer shorter responses—with no correctness labels—reduces generation length by roughly 30–60% while keeping accuracy at an acceptable level. If the findings hold, length itself is a useful control knob for reasoning efficiency.","feed_headline":"Longer LLM reasoning helps, then hurts accuracy","feed_subtitle":"A length-only preference cut output by 30–60% with little accuracy loss, revealing overthinking and underthinking.","key_machinery":"The central tool is a length-ranking construction: for each question, ten sampled responses are sorted by token length, and average length and accuracy are computed at each rank, isolating the effect of length from question difficulty. At the question level, the machinery is a cross-model difficulty partition into shared-easy and shared-hard sets plus model-specific advantage sets, compared with t-tests; this separates 'the model sees difficulty and adapts' from 'the model fails to recognize difficulty.' The optimization machinery is SimPO, a reference-free preference optimization algorithm that uses average sequence log-probability as an implicit reward, applied here to preference pairs that prefer shorter responses without checking correctness.","core_discovery":"At the sample level, when a single question is answered multiple times, accuracy is non-monotonic in response length: the shortest response is often already correct, accuracy peaks early, and the longest samples are less accurate. At the question level, incorrect answers are on average far longer than correct ones—on the harder benchmark, more than twice as long—and length correlates negatively with accuracy. The authors partition questions into easy, medium, and hard by a model's own consistency, and use cross-model comparisons to show that models detect small difficulty increases on questions they can solve, generating longer responses, but on questions beyond their ability they often produce shorter-than-needed responses, a pattern they call underthinking. Finally, using preference optimization that always prefers the shorter of two unlabeled responses, they cut average generation length by roughly 30–60% with only mild accuracy loss, with most of the reduction coming from incorrect responses.","pith_inferences":["A length-aware self-consistency vote that discards the longest samples could improve accuracy and cut cost; the paper only notes this direction as motivation.","The peaked length–accuracy curve implies a model- and dataset-specific optimal length; an unlabeled-data controller that predicts the plateau would be a natural next step beyond the paper's experiments.","Because length diverges from perceived difficulty mainly on questions beyond a model's ability, calibration or perplexity outliers might flag when self-correction is unlikely to help—a connection the paper raises but does not test."],"forward_implications":["For a fixed question, accuracy peaks at an interior response length, so both minimal and maximal reasoning are risky; a token-budget-aware sampler that avoids the longest chains should match or beat unconstrained sampling.","Because over 60% of questions are solved by the shortest sampled response when they are solvable at all, much of the compute spent generating long chains is wasted.","Preference optimization with only length preferences, and no ground-truth correctness, reduces generation length by 30–60% while keeping accuracy in an acceptable range; most savings come from shortening incorrect responses, but correct responses also shrink by 10–25%.","Length reduction does not require labeled preference pairs, so unlabeled corpora can be used to make reasoning models cheaper at inference time.","Incorrect responses are markedly longer than correct ones and correlate negatively with accuracy, making length a usable signal for detecting likely-failed reasoning."],"supporting_citations":[{"why":"Supplies the GSM8K math word-problem benchmark used for both the length–accuracy analysis and the length-preference training.","marker":"(Cobbe et al., 2021)"},{"why":"Supplies the MATH benchmark, where the incorrect-response length gap is largest, often exceeding 6,000 tokens.","marker":"(Hendrycks et al., 2021)"},{"why":"Provides one of the two reasoning models analyzed, contributing the sample-level and question-level generation data.","marker":"(DeepSeek-AI, 2025)"},{"why":"Provides the other reasoning model, a 1.5B reinforcement-learning-trained model, used across both benchmarks.","marker":"(Luo et al., 2025b)"},{"why":"Introduces SimPO, the reference-free preference optimization algorithm used in the length-reduction experiments.","marker":"(Meng et al., 2024)"},{"why":"Supplies the earlier theoretical result that accuracy initially improves then declines with reasoning steps, which the paper's non-monotonic finding builds on.","marker":"(Wu et al., 2025)"},{"why":"Provides prior overthinking analysis that this paper extends into a systematic length–correctness study.","marker":"(Chen et al., 2024)"}],"fun_headline_variants":["LLMs overthink easy questions and underthink hard ones","Shorter LLM answers often correct, longer ones hurt accuracy","Preference for shorter responses trims LLM output 30-60%","Accuracy peaks early in LLM response length, not at maximum"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The interpretation that short incorrect answers are 'underthinking' assumes the model's response length tracks its perceived difficulty of the question, an assumption the paper does not directly test and whose alternative—longer responses themselves introducing errors—it explicitly acknowledges.","fun_headline_variants_meta":{"raw":{"variants":["LLMs overthink easy questions and underthink hard ones","Shorter LLM answers often correct, longer ones hurt accuracy","Preference for shorter responses trims LLM output 30-60%","Accuracy peaks early in LLM response length, not at maximum"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000824,"raw_usage":{"total_tokens":3573,"prompt_tokens":883,"completion_tokens":2690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2617}},"tokens_in":499,"tokens_out":2690,"duration_ms":19002,"temperature":1.0,"reasoning_tokens":2617,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:50:10.621665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Truncate long responses on solvable questions to the median length of correct responses; if accuracy does not fall, extra reasoning is not the cause of the decline. Conversely, force additional reasoning steps on hard questions that currently get short wrong answers; if accuracy does not rise, those failures are not caused by insufficient thinking.","supporting_citations":[],"review_version":1}