{"id":"36670985-1a8d-4585-b719-a2f42061de94","arxiv_id":"2502.07266","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM accuracy follows an inverted U in chain-of-thought length, with an optimal length that grows with task difficulty and shrinks with model capability.","lead":"This paper shows that longer chain-of-thought reasoning is not always better: accuracy rises then falls as reasoning steps increase, so there is an optimal length. It offers a simple theory and practical tips for choosing that length during training and inference.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The formal scaling laws are conditional on the linear ansatz E=T/(NM); the general extension only bounds N* from below, so the claimed monotonicity of the optimum is not actually established, and the printed Lambert-W argument appears mistyped.","rationale":"The paper's central claim is that accuracy is inverted-U in CoT length and that the optimal length increases with task difficulty and decreases with model capability. The real-world observations and controlled arithmetic experiments provide independent support for the inverted-U shape, and the RL demonstrations are suggestive. The weak point is the formal derivation: Theorem 4.3 and Corollary 4.4 are the load-bearing theoretical statements, and they are derived only under the exact linear forms E=T/(NM) and sigma=T/C. These forms already encode the inverse dependencies on N and M that the corollaries conclude, so the 'proof' of the scaling laws is largely a restatement of the ansatz. The paper gestures at generalization in Appendix F, but Theorem F.3 only gives a lower bound on any interior maximum, and the subsequent corollaries infer monotonicity of the maximum from monotonicity of that lower bound, which is not sufficient. I therefore agree with the reader's weakest-assumption identification and with the conditional verdict. I would not reject the paper: the empirical evidence is meaningful, and the theoretical model may be a useful toy even if the exact functional form is wrong. However, the abstract's claim that the paper 'formally proves' the scaling laws overstates what is actually shown; the theorem should be presented as conditional on the linear error model, and the general-extension section should either prove monotonicity of N* directly or be reframed as a robustness conjecture. The Lambert W typo in Theorem 4.3 should also be corrected. These are fixable issues, so the verdict remains conditional rather than accept or reject.","tokens_in":23641,"tokens_out":10658,"duration_ms":97811,"concrete_test":"Run a direct empirical check in the Section 3 synthetic setup: for each layer count M, difficulty T, and forced step length N (via the <t> control token), decode a large sample and measure the per-step answer error e(N,M,T). Fit e as a power law a T^b / (M^c N^d) and test whether b=c=d=1. Then recompute N*(M,T) = argmax_N [((1-T/C)(1-e(N,M,T)))^N] using the fitted e; if the argmax does not move monotonically right in T and left in M across the measured grid, the scaling laws in Corollary 4.4 are not robust and the linear ansatz in Theorem 4.3 is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central formal result, Theorem 4.3, is proved from Eq. (1) using the ansatz E(N,M,T)=T/(NM) and sigma(T)=T/C. This ansatz is doing essentially all of the work: the resulting optimum N* = T Z/(M(Z+1)) has Z independent of M, so N* is proportional to 1/M by construction, and the T-dependence is inserted through the numerator T and through Z=W(-(1-T/C)/e). Corollary 4.4 then 'derives' exactly those dependencies. Appendix E.2 reports only training-loss ratios and does not fit E or justify the linear form; indeed, its statement that the subtask error rate is taken constant with t sits uneasily with E=T/(NM), which varies with t=T/N. The general extension does not repair this: Theorem F.3 establishes lim_{N->inf} A(N)=0 and a lower bound NLB(M,T) on an interior optimum, but Corollaries F.4 and F.5 infer monotonicity of N* from monotonicity of the lower bound. That inference is not logically valid: a lower bound can move monotonically while the maximizer itself does not. Thus, beyond the linear ansatz, the monotonic scaling laws are not actually proven. Separately, the printed Z=W_{-1}(-1-T/(Ce)) lies outside the real domain of the Lambert W function; the appendix proof uses W_{-1}(-(1-T/C)/e), so the theorem statement needs correction. These issues do not undercut the empirical inverted-U observations or the controlled synthetic demonstrations, but they mean the formal scaling claims should be read as conditional on an unvalidated functional form.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies how chain-of-thought (CoT) length affects LLM accuracy. It argues that accuracy follows an inverted U-shaped curve in CoT length, that the optimal length increases with task difficulty and decreases with model capability (simplicity bias), and that RL training shifts models toward shorter chains. The evidence is threefold: real-world evaluations on Qwen and Llama models over MATH, MMLU, and GPQA; controlled synthetic arithmetic experiments with GPT-2 models of varying depth; and a theoretical model in which total accuracy is A(N)=alpha[(1-E)(1-sigma)]^N. Under the linear ansatz sigma(T)=T/C and E(N,M,T)=T/(NM), Theorem 4.3 gives an explicit optimum N*=T Z/(M(Z+1)), and Corollary 4.4 states the scaling laws. The paper also proposes training with optimal-length CoT data and a length-filtered majority vote. I agree with the reader that the controlled synthetic experiments cleanly demonstrate the inverted U and the scaling trends, but the formal scaling laws are conditional on an unvalidated functional ansatz, the printed Lambert-W formula in Theorem 4.3 is mistyped, and the general extension in Appendix F does not logically establish monotonicity of the maximizer from monotonicity of a lower bound.","tokens_in":24012,"tokens_out":9109,"duration_ms":81685,"significance":"If fully established, the paper would be a useful contribution: it challenges the default assumption that longer CoT is always better, offers a transparent mechanism (the trade-off between task decomposition and error accumulation), and gives practical guidance for adaptive CoT calibration. The controlled synthetic experiments are a genuine strength: they systematically vary task difficulty and model capability, show the inverted U, and demonstrate that RL converges to the optimal length. The theoretical model is simple and the empirical inverted-U does not depend on the theory for its support. The significance is currently limited because the scaling laws are largely derived from assumed functional forms rather than independently confirmed, and the real-world length estimates are noisy. With the theory either validated or explicitly presented as a conditional heuristic, the paper would be a solid empirical and conceptual contribution.","major_comments":[{"comment":"The central scaling laws are largely a restatement of the linear ansatz. With sigma(T)=T/C and E(N,M,T)=T/(NM), the maximizer of Eq. (1) is N*=T Z/(M(Z+1)) with Z independent of M, so N* is proportional to 1/M by construction and the T-dependence is inserted through the numerator T and through Z. Corollary 4.4 therefore derives dependencies that are already encoded in the assumptions; no independent evidence is provided for the specific linear forms. Appendix E.2, the only empirical support cited for the model, reports training-loss ratios rather than estimating E or sigma, and its statement that the subtask error rate is taken constant with t is in tension with E=T/(NM), which varies with the per-step size t=T/N. The formal claims should be reframed as conditional on the ansatz, or the ansatz should be validated.","section":"Section 4.1, Theorem 4.3, Corollary 4.4"},{"comment":"The theorem statement defines Z=W_{-1}(-1-T/(Ce)), but for 0<T/C<0.9 this argument is less than -1, which is outside the real domain of the Lambert W function. The proof in Appendix G.2 correctly uses W_{-1}(-(1-T/C)/e). The theorem statement must be corrected; as printed, the formula in Eq. (2) is not meaningful for the stated regime.","section":"Theorem 4.3"},{"comment":"Theorem F.3 establishes only a lower bound NLB(M,T) on any interior maximizer N*; it does not establish that A(N) has a maximum at all, nor that the maximizer is unique. Corollaries F.4 and F.5 infer monotonicity of N* from monotonicity of the lower bound, which is not logically valid: a lower bound can move monotonically while the maximizer itself moves in the opposite direction or ceases to exist. The claimed qualitative scaling laws under general error functions therefore remain unproved. Additionally, Eq. (19) in the proof of Theorem F.3 misprints the derivative: it should be ln((1-sigma)(1-E(x))) - xE'(x)/(1-E(x)), not ln(1-sigma(1-E(x))) - xE'(x)/(1-E(x)); the printed inequality does not follow from the written expression.","section":"Appendix F, Corollaries F.4 and F.5"},{"comment":"The real-world estimate of optimal CoT length is fragile. For each question, the optimal length is the best length bin among 60 sampled solutions, which is a noisy quantity, and the analysis uses coarse bin widths (5 in the cross-model comparison, 2 in the difficulty analysis) and excludes questions with accuracy below 0.01 or above 0.99 after the fact. No error bars or confidence intervals are provided for the estimated optimal lengths themselves, only for the regression line in Figure 2b. Figure 2a uses only four model sizes without repeated seeds. This weakens the real-world evidence for the scaling laws, although the controlled synthetic experiments stand independently.","section":"Section 2.1 and Appendix C.1"}],"minor_comments":[{"comment":"The heading contains a duplicated word: 'Scaling Laws of the Optimal CoT Length and and Practical Insights'. It should be 'and Practical Insights'.","section":"Section 3.2 heading"},{"comment":"There are typographical artifacts in the text, including 'Length-aware Voteto' in the abstract and 'Length-Filtered V ote' in Section 5.2 and Figure 4c; these should be cleaned up.","section":"Abstract and Section 5.2"},{"comment":"The sentence 'the loss of subtask generation tokens for the easiest subtask (t=1) is about 3 times larger than the hardest subtask (t=12)' appears to contradict the intended claim that harder subtasks have larger loss; please clarify whether this is a ratio reversal or a typo.","section":"Appendix E.2"},{"comment":"In Length-Filtered Vote, if a length bin L_j is empty, the computation of p_j[a] divides by |L_j|=0; the algorithm should specify how empty bins are handled.","section":"Appendix H, Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The controlled empirical study is the strongest part of the paper and could support a revised version focused on the inverted-U phenomenon and its practical implications rather than on overclaimed formal scaling laws. I would ask the authors to either validate the linear ansatz or explicitly state that the scaling laws are consequences of the assumed functional forms, and to correct the Lambert-W formulas and the Appendix F proof before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the empirical inverted-U and the optimal-length scaling with task difficulty and model capability are real and worth knowing. The theoretical proof of the scaling laws, however, is not load-bearing—it rests on an unverified linear error ansatz. Still worth engaging.\n\nWhat's actually new: controlled synthetic arithmetic experiments (GPT-2 with variable depth) cleanly show the inverted U, with the optimum shifting right as task difficulty grows and left as model size grows. The synthetic RL demonstration that policies converge to the optimal length is a nice touch, and length-filtered vote is a simple, plausible inference-time idea. Real-world Qwen/Llama observations support the qualitative trends, even if they are rough. The core empirical phenomena are likely correct.\n\nSoft spots, in proportion:\n\n1. Theorem 4.3 and Corollary 4.4 are proved only under the linear forms E = T/(NM) and sigma = T/C, which already encode the claimed inverse dependence on N and M. Appendix E.2 does not fit or justify those forms and even says subtask loss is roughly constant in t—awkward next to E = T/(NM).\n\n2. The general extension in Appendix F only yields a lower bound on the maximizer N*. Corollaries F.4 and F.5 infer monotonicity of N* from monotonicity of that lower bound, which is not logically valid. The general scaling claim is therefore not established.\n\n3. The theorem statement has a Lambert-W typo: W_{-1}(-1 - T/(Ce)) is outside the real domain; the proof correctly uses W_{-1}(-(1 - T/C)/e). Easy fix.\n\n4. Figure 2b's correlation analysis excludes questions with accuracy below 0.01 or above 0.99. That exclusion is disclosed but post hoc, and it weakens the difficulty-correlation claim.\n\n5. The GPQA filtered-vote result has no error bars and uses only 100 questions; no code or data is provided.\n\nNone of this undercuts the controlled experiments or the qualitative real-world pattern. It just means the theory should be read as a toy model illustrating a plausible trade-off, not as a proof of the scaling laws.\n\nWho it's for: anyone working on CoT efficiency, overthinking, or RL length control. It deserves a serious referee. I'd recommend conditional acceptance: the empirical core is solid and citable, but the theory needs either an honest reframing as a heuristic model or real justification of the ansatz, and the real-world statistics need to be tightened.","headline":"Empirical inverted-U is solid and useful; the scaling-law proof is conditional on an unverified linear ansatz, and the general extension does not prove what it claims.","tokens_in":24518,"tokens_out":1881,"would_cite":true,"duration_ms":18511,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chain-of-thought accuracy peaks at an intermediate length: longer is not better.","keywords":["chain-of-thought","optimal CoT length","inverted U-shape","simplicity bias","error accumulation","task decomposition","reinforcement learning","length-filtered voting"],"falsifier":"Take a reasoning task with verification steps that catch earlier errors, so per-step error decays faster than linearly with $N$, and measure accuracy against CoT length over a wide range; if accuracy keeps rising monotonically instead of peaking, the inverted-U claim and its linear error model are falsified. A second check is to measure the optimal length for one fixed difficulty across a fine grid of model sizes; if the optimum does not decrease monotonically with capability, Corollary 4.4 is contradicted.","tokens_in":23425,"feed_emoji":"🧠","tokens_out":8557,"duration_ms":69519,"temperature":0.7,"pith_summary":"Task accuracy follows an inverted U-shape as chain-of-thought (CoT) length grows: very short chains under-decompose the problem and very long chains accumulate errors. The paper argues this curve holds across real models and controlled arithmetic tasks, and that the peak location follows a scaling law: harder tasks push the peak to longer chains, while stronger models peak with shorter chains. This 'simplicity bias' also shows up during reinforcement learning, where models converge to the optimal (often shorter) length. If correct, the result turns CoT length from a free parameter into a quantity that should be calibrated per task difficulty and per model.","feed_headline":"Accuracy peaks at a middle CoT length — longer is not better.","feed_subtitle":"Harder tasks demand more steps, but stronger models peak with fewer; RL finds the same optimum.","key_machinery":"The load-bearing object is the per-step success product in Proposition 4.2: each CoT step must both state the right sub-question and answer it correctly, so $N$ steps compound the per-step success probability $(1-\\sigma(T))(1-E(N,M,T))$ into $A(N)$. Longer chains make each sub-question easier (lower $E$) but multiply more factors, and they also increase the chance of a sub-question error. The explicit linear forms $\\sigma(T)=T/C$ and $E(N,M,T)=T/(NM)$ turn this tradeoff into the closed-form optimum of Theorem 4.3, whose derivative condition is solved by the Lambert W function (the function $w$ with $w e^w = x$).","core_discovery":"The paper's central claim is formalized as a product decomposition of final accuracy: $A(N) = \\alpha [(1-\\sigma(T))(1-E(N,M,T))]^N$, where $N$ is the number of reasoning steps, $T$ is task difficulty, $M$ is model capability, $\\sigma(T)$ is the error rate of generating a sub-question, and $E(N,M,T)$ is the error rate of answering each sub-question. Under the linear-error model $\\sigma(T)=T/C$ and $E(N,M,T)=T/(NM)$, the accuracy function has an interior maximum at $N^*=T Z/(M(Z+1))$ with $Z=W_{-1}(-1-T/(Ce))$; this optimum increases with $T$ and decreases with $M$. Thus the author's intended claim is not merely that long CoTs fail, but that there is a provable, predictable peak length that explains both overthinking and the preference of stronger models for shorter reasoning paths.","pith_inferences":["If the inverted-U law holds across real tasks, then test-time compute is not monotone: for each question there is a compute budget beyond which extra tokens actively hurt accuracy, so overthinking is a measurable phenomenon rather than just a cost concern.","The theory predicts that when two models of different sizes are fine-tuned on the same fixed-length CoT corpus, the larger model will be further from its potential until the chain lengths in the data are re-matched to its capability; this is a direct comparison a user could run.","Length-aware filtering could be extended to reward shaping during RL by explicitly penalizing chains far from the estimated optimal length, which may speed convergence to the same peak that outcome-only rewards reach more slowly."],"forward_implications":["Models trained on chains of their own optimal length outperform models trained on uniformly mixed chain lengths, and a smaller model with correctly sized chains can beat a larger model with poorly sized chains.","Filtering out very long chains before majority voting (Length-Filtered Vote) improves accuracy over plain majority voting because very short and very long chains contribute noisier votes.","Reinforcement learning with outcome rewards converges to the optimal chain length, so the observed shrinking of CoT during RL is the expected consequence of reward optimization, not a separate anomaly.","Optimal per-step complexity rises with task difficulty, so forcing every problem into the same step granularity is suboptimal for hard tasks."],"supporting_citations":[{"why":"Supplies the complexity-based prompting scheme used to generate real-model solutions with varied step lengths.","marker":"[12]"},{"why":"Introduces chain-of-thought prompting, the reasoning format whose length is the paper's object of study.","marker":"[38]"},{"why":"Provides the MATH dataset on which real-world optimal-length versus model-size and difficulty trends are measured.","marker":"[18]"},{"why":"Provides the Qwen2.5 Instruct model family whose different sizes generate the model-capability trend.","marker":"[27]"},{"why":"Supplies the GPQA benchmark used to evaluate Length-Filtered Vote against vanilla majority vote.","marker":"[29]"},{"why":"Supplies the self-consistency majority-vote baseline that Length-Filtered Vote modifies.","marker":"[37]"},{"why":"Provides the GPT-2 architecture used in the controlled arithmetic experiments and in the synthetic RL study.","marker":"[28]"},{"why":"Supplies the reasoning-boundary concept used to define model capability $M$ in the theoretical model.","marker":"[5]"},{"why":"Supplies the GRPO algorithm used for the real-world reinforcement-learning simplicity-bias experiment.","marker":"[31]"},{"why":"Supplies the LeetCode-2K dataset used for real-world RL training and evaluation.","marker":"[40]"}],"fun_headline_variants":["CoT accuracy peaks at an optimal length, not max length","Optimal chain-of-thought length shrinks as models grow","The 'more is less' CoT curve: accuracy peaks early","RL also favors shorter chains once accuracy improves","Theory proves optimal CoT length: harder needs more, bigger needs less"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theory assumes the two error rates take the exact linear forms $\\sigma(T)=T/C$ and $E(N,M,T)=T/(NM)$, and these forms already build in the inverse dependence on $N$ and $M$ that the optimal-length formula outputs; if the true per-step error rates are not linear in $T/(NM)$ and $T/C$, the explicit formula for $N^*$ and the monotonicity results do not follow. The empirical optimal-length measurements additionally depend on coarse length binning and on excluding questions whose accuracy is near zero or near one.","fun_headline_variants_meta":{"raw":{"variants":["CoT accuracy peaks at an optimal length, not max length","Optimal chain-of-thought length shrinks as models grow","The 'more is less' CoT curve: accuracy peaks early","RL also favors shorter chains once accuracy improves","Theory proves optimal CoT length: harder needs more, bigger needs less"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2512,"prompt_tokens":999,"completion_tokens":1513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1428}},"tokens_in":615,"tokens_out":1513,"duration_ms":9962,"temperature":1.0,"reasoning_tokens":1428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:17:43.777835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a reasoning task with verification steps that catch earlier errors, so per-step error decays faster than linearly with $N$, and measure accuracy against CoT length over a wide range; if accuracy keeps rising monotonically instead of peaking, the inverted-U claim and its linear error model are falsified. A second check is to measure the optimal length for one fixed difficulty across a fine grid of model sizes; if the optimum does not decrease monotonically with capability, Corollary 4.4 is contradicted.","supporting_citations":[{"cited_title":"Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of- thought","cited_arxiv_id":null,"evidence_quote":"Supplies the reasoning-boundary concept used to define model capability $M$ in the theoretical model."}],"review_version":1}