Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RL-trained LLMs converge to an optimal chain-of-thought length because thinking too briefly underfits and thinking too long overfits.

desk verdict A clever and timely framing of CoT length as an under/overfitting tradeoff, but the key theorems are either forced by normalization or rest on a false mutual-information bound, so the theoretical claim doesn't hold as stated. read the letter →

arxiv 2509.04027 v4 pith:YXYJDKPH submitted 2025-09-04 cs.AI cs.CL

Zeyu Gan , Hao Yi , Yong Liu This is my paper
classification cs.AIcs.CL
keywords chain-of-thoughtreinforcementlearningoptimalreasoninglengthoverthinkingunderfitting-overfittingtrade-offgeneralizationboundsemanticmanifoldtest-timescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to explain a well-known empirical regularity: when language models are fine-tuned with reinforcement learning on reasoning tasks, their chain-of-thought length reliably converges to a stable value rather than growing without bound. It argues that this convergence is a natural consequence of a classical underfitting-overfitting trade-off, once reasoning is viewed as descent over a continuous semantic manifold rather than as token prediction. Too short a chain-of-thought cannot reach the answer (underfitting), while too long a one inflates the information capacity of the policy and worsens generalization (overfitting). The paper derives bounds for both effects and concludes that the optimal length is determined only by task difficulty and model capacity, not by the RL algorithm that discovers it. If true, this turns overthinking from a nuisance into a predicted equilibrium point and gives a theoretical reason to expect test-time scaling to have a compute-optimal length.

What carries the argument

CoT-Space is the central object: a reasoning-level state space in which a state is a query plus an incomplete sequence of thoughts, and a minimum is a state reaching the gold answer. A reasoning loss C(s) decreases with distance to the nearest reachable minimum, and the discrete state space is approximated by a continuous semantic manifold whose density grows exponentially with the token budget. On that manifold, the two load-bearing tools are sharpness-aware minimization (CoT length acting as a learning rate, with an optimal noise scale) and information-theoretic generalization bounds (mutual information between the trained policy and the training set bounded by expected output entropy, giv

What would settle it

Estimate mutual information between an RL-trained policy and its training set directly, for example by membership-inference reveal rate or leave-one-out performance, while measuring the policy's mean CoT token count. If a policy with a short mean chain nonetheless encodes extensive training-data information, Lemma B.11 is violated and Theorem 3.2's overfitting bound no longer follows; conversely, if doubling training-set duplicates raises generalization error without lengthening chains, the paper's mechanism would be incomplete.

Watch

Extended reading notes

Core claim

The central claim is that convergence to an optimal chain-of-thought length Lopt follows from two opposing bounds. When a reasoning chain ends before the task's minimum required depth, the empirical loss is at least the failure probability times a positive failure loss (Theorem 3.3); when chains grow long, the expected generalization error grows with the expected total output tokens times log|A| (Theorem 3.2). Between these bounds sits a unique minimizing length, intrinsic to the task and model and independent of the RL algorithm. The supporting geometric picture: as the token budget grows, reasoning states become dense enough to form a continuous semantic manifold (Theorem 2.10), where each

Load-bearing premise

The overfitting half of the trade-off rests on an unproven bound (Lemma B.11) claiming that a learned policy's mutual information with its training set is at most its expected output length times log|A|; if a policy can memorize training data in its weights while emitting short chains, the bound and the longer-CoT-causes-overfitting conclusion are unsupported.

Editorial extensions

If this is right

  • Harder reasoning tasks should converge to longer optimal CoT lengths, because the minimum required depth is larger and underfitting dominates if the policy stops early.
  • Larger or higher-capacity models should converge to shorter CoT lengths, because their larger information capacity increases the generalization-error penalty for long outputs.
  • Different RL algorithms trained on the same task and model should converge to roughly the same final CoT length, differing only in how quickly they reach it.
  • Noisier optimization (proxied by KL loss) should push the converged length down, since longer trajectories become more fragile under noise.
  • Overlong chain-of-thought should be treated as a generalization risk, not merely a compute waste, because expected test error explicitly grows with expected output length.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A corollary the paper leaves implicit: if Lopt is determined by task difficulty and model capacity before training, then one could predict the converged length from a small probe set and either stop training early or set rollout budgets to Lopt without waiting for convergence.
  • The noise-length relation suggests a testable intervention: injecting controlled entropy into reasoning, or penalizing length in a way that mimics noise, might move the policy toward Lopt or shorten it without sacrificing accuracy, in the spirit of flat-minima optimization.
  • The continuum approximation predicts that surface-level paraphrases of the same reasoning step should behave as nearly the same state, so adversarial paraphrasing of a single step should barely change future states.
  • The same underfitting-overfitting geometry likely applies to other test-time computation budgets, not only tokens: number of search steps, tool calls, or self-correction iterations should each have their own Lopt for the same reasons.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CoT-Space, a reasoning-level continuous semantic state-space framework for Chain-of-Thought reasoning in LLMs. It claims that RL-trained LLMs converge to an optimal CoT length because of an underfitting-overfitting trade-off: Theorem 3.1 connects optimization noise to 1/L, Theorem 3.2 gives an information-theoretic generalization bound that grows with CoT length, and Theorem 3.3 gives an underfitting lower bound on empirical loss. Experiments with GRPO/PPO/DAPO/Reinforce++/RLOO on GSM8K and MATH are presented in support of four remarks on task difficulty, model capacity, RL-algorithm independence, and noise scale. Proofs are collected in Appendix B.

Significance. The question addressed is timely and important, and the empirical setup is broad: multiple RL algorithms, model families, and task difficulties are covered, and the code is promised open-source. If the central results were correct, the paper would provide a mechanistic explanation for the observed convergence of CoT length and a principled basis for regularizing reasoning length. However, the core theoretical contribution is not sound as written. Theorem 3.1 is an artifact of the normalization chosen in Eq. (29); Theorem 3.2 rests on an unproved and generally false Lemma B.11; and the two risk bounds do not by themselves imply the existence of a finite optimal length. The experiments, while suggestive, are correlational and do not directly test the predicted U-shaped total-error curve. The paper's analogy is stimulating, but the claimed theoretical foundation is not established.

major comments (4)
  1. [Appendix B.5, Lemma B.11 and Theorem 3.2] Lemma B.11 asserts I(π;S) ≤ E_{S,π}[Σ_i |ξ_i|]·log|A|. This is not proved and is false in general. I(π;S) is the mutual information between the trained policy parameters and the training set, not the information conveyed through emitted text. A policy can memorize the training labels in its weights while emitting a constant single-token CoT; then I(π;S) ≈ H(S) ≈ n log|A|, whereas the right-hand side is O(log|A|) if E[L]=1. Conversely, a policy can emit long but uninformative text while carrying almost no information about S. Consequently Theorem 3.2, the overfitting half of the central trade-off, is unsupported. Additionally, the equality E[Σ_i |ξ_i|] = (E[L]·E[|ξ|]) used in the lemma requires an independence assumption that is not stated.
  2. [Appendix B.4, Eq. (29)-(36)] Theorem 3.1 is built into the definition of the update. Eq. (29) sets ∆s = -(1/L) dĈ/ds, so the per-step gradient-noise variance is F(s)/L² by construction. The SDE step over Δt=1/L then has noise variance gF(s)/L. Equating these two variances yields g=1/L. If the update had been written with step size 1, or with an independent step-size parameter, the relation would change. No independent definition of g is given, so the claimed inverse proportionality is a normalization artifact rather than a derived result.
  3. [Section 3.2 and Remarks 1-4] The two bounds do not imply a finite L_opt. Theorem 3.2 gives an upper bound that increases with expected length; Theorem 3.3 gives only a lower bound on the empirical loss that is positive for too-short CoTs. A lower bound on one component and an upper bound on another do not produce a U-shaped total error; the total error could decrease monotonically with L. The experiments report the final mean length L_opt under different conditions, not a measured U-shaped error-vs-L curve, so they do not fill this gap. The claim in Remark 2 that higher-capacity models face greater generalization error is also not a consequence of Theorem 3.2, which contains no model-capacity term.
  4. [Appendix B.1, Theorem 2.10] The Θ(c^K) claim is not justified. Assumption 2.9 provides only a lower bound |V(ξ,k)| ≥ c^k. The proof lower-bounds N_realize for a single path and then asserts |S_reasoning^(K)| = Θ(c^K) without an upper bound on the number of distinct valid reasoning paths or a matching upper construction. Moreover, Lemma B.2 is stated for a homogeneous Poisson point process, but no Poisson or uniform-density assumption is established for reasoning states. This weakens the continuum-manifold foundation on which the rest of the paper relies.
minor comments (4)
  1. [Section 3.2] Typo: 'virance' should be 'variance'.
  2. [Figures 5 and 6] The plots lack error bars or confidence intervals, and the use of KL-loss as a direct proxy for the noise scale g in Theorem 3.1 is not justified in the text.
  3. [Appendix B.5, Theorem B.8] The proof is mostly a restatement of the PAC-Bayes theorem; the normalization step C/C_max is not carried through, and the C_max^2 factor in Theorem 3.2 appears without a derivation from the normalized loss.
  4. [Definition 2.6] The reasoning loss C(·) is asserted to exist with a monotonicity property, but no construction or uniqueness is provided, and the definition is ambiguous for states with no reachable minimum.

Circularity Check

3 steps flagged · score 8.0 of 10

The central underfitting/overfitting tradeoff is assembled from a normalization that forces g=1/L, a false/unproved lemma that already assumes length bounds information, and a theorem that restates the failure assumption; the derivation reduces to its own inputs.

  1. self definitional [Section 3.1 / Appendix B.4, Eq. (29) and Eqs. (35)-(36)]
    "For an L-step CoT reasoning process, this can be viewed as an L-step optimization based on the gradient of the reasoning loss. A single step of this optimization can be written as: ∆s = − 1/L d/ds Ĉ ... By equating the variance from the discrete update (Eq. 32) and the SDE model (Eq. 34), we can solve for the noise scale g: F(s)/L^2 = gF(s)/L. Solving for g, we find: g = 1/L."

    The 1/L factor in Eq. (29) is inserted by hand as the step size of an L-step rollout, and the SDE noise variance over the interval 1/L is gF(s)/L. Equating the two variances algebraically forces g=1/L. The theorem is the normalization, not an independent derived relation; the later claim L_opt ∝ 1/g_opt inherits this chosen scaling rather than being an empirical or first-principles fact.

  2. other [Appendix B.5, Lemma B.11 (used in Theorem 3.2)]
    "Lemma B.11. In LLM reasoning scenarios, if the policy π is trained on a training set S, the mutual information I(π;S) is upper bounded as follows: I(π;S) ≤ E_{S,π}[ Σ_{i=1}^L |ξ_i| ]·log|A|. Lemma B.11 formalizes the intuition that the information a policy learns from data cannot exceed the information content it is capable of expressing in its output."

    This lemma is the overfitting half of the tradeoff in miniature: it asserts that a policy's information about the training set is bounded by the token length of its CoT. Combined with the Xu–Raginsky bound (Lemma B.10), it yields Theorem 3.2's conclusion that generalization error grows with E[L]·E[|ξ|]. But I(π;S) is the mutual information between policy parameters and training data, not output length; a policy can memorize the training labels in its weights while emitting a fixed one-token CoT, making the bound false in general. The lemma therefore assumes the length-overfitting link it is used to establish, so Theorem 3.2 is unsupported.

1 more flagged steps
  1. self definitional [Appendix B.6, Assumption B.13 and Theorem 3.3]
    "Assumption B.13. ... if the length is insufficient to solve the problem, i.e., L_i < L∗(q_i), the policy is guaranteed to fail. The resulting reasoning loss C(s^π_{q_i}) is therefore bounded below by a significant positive constant, which we denote as C_fail > 0. ... Theorem 3.3. ... ˆR_n(π) ≥ P_π(L < L∗)·C_fail."

    The proof partitions the empirical loss into failed and non-failed queries and applies Assumption B.13 verbatim to the failed subset. The theorem is just that assumption averaged over the training set (n_fail/n = Pπ(L < L∗)). It contributes no independent content to the tradeoff: the 'underfitting risk' is the same statement as the assumption that too-short CoTs fail, renamed as a lower bound.

full rationale

The paper's central theoretical claim is that an optimal CoT length emerges from an underfitting–overfitting tradeoff. The noise/optimization half (Theorem 3.1) is not derived: the authors insert a 1/L normalization into the discrete update (Eq. 29) and then equate the variance of this update with an SDE noise term, which algebraically yields g=1/L. This is a definitional relation, not a theorem about LLM reasoning. The overfitting half (Theorem 3.2) depends entirely on Lemma B.11, which asserts without proof that I(π;S) is bounded by the expected token length of the output. That lemma is false in general and already contains the conclusion that longer CoTs carry more training-data information; hence generalization error growing with L is not established. The underfitting half (Theorem 3.3) is a direct restatement of Assumption B.13 (insufficient depth ⇒ failure); it adds no independent derivation. Because both sides of the tradeoff reduce to either definitions or assumptions that contain the conclusion, the existence of L_opt as a 'natural consequence' is not supported by the theoretical chain. The experiments show correlations (task difficulty, model capacity, etc.) but do not directly measure a U-shaped total-error curve, so they do not break the circularity. We rate 8 rather than 10 because some parts (continuum approximation, empirical findings) are independent, but the central tradeoff derivation is forced by construction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 4 invented entities

The paper introduces several abstract constructs (CoT-Space, reasoning loss, semantic equivalence sets, maximal voids) and relies on strong assumptions, most notably exponential redundancy and a false information-capacity lemma. The constants c, D, Cmax, C_fail and Rmin are not measured but are necessary for the stated bounds. This ledger shows the central claim rests on substantial unverified scaffolding.

free parameters (5)
  • c (exponential redundancy base)
    Assumption 2.9 posits |V(xi_l,k)| >= c^k for some c>1. The base c is never measured or estimated; the exponential density claim depends on it.
  • D (semantic space dimension)
    Appears in rho(K)^{-1/D} bounds. The dimension is never specified or estimated, though asymptotic convergence holds for any fixed D.
  • Cmax (maximum reasoning loss)
    Appears in Theorem 3.2 as the loss range. Since the reasoning loss C is only assumed to exist, Cmax is not computable.
  • C_fail (failure loss lower bound)
    Introduced in Assumption B.13 as a 'significant positive constant'. No estimate is given and it is not tied to any measurement.
  • Rmin (minimum step radius)
    Assumption B.3 uses Rmin to bound the angular error. It is chosen by hand and never estimated.
assumptions (6)
  • domain assumption Exponential expressive redundancy of language: |V(xi_l,k)| >= c^k for some c>1 (Assumption 2.9).
    Central to Theorem 2.10's continuum convergence. No empirical evidence is provided for exponential growth.
  • ad hoc to paper Existence of a reasoning loss C with monotonicity to distance to nearest reachable minimum (Definition 2.6).
    All risk analysis operates on this abstract C. No construction or measurability argument is given.
  • domain assumption Bounded step optimization: each reasoning step has effective radius between Rmin and Rmax (Assumption B.3).
    Used in Theorem 2.12 to bound continuum errors. Plausible but unverified.
  • ad hoc to paper Mutual information bounded by output entropy: I(pi;S) <= (E[L] E[|xi|]) log|A| (Lemma B.11).
    Asserted without proof and false in general, since model weights can encode training information independently of generated length. This is the load-bearing assumption for the overfitting half of the tradeoff.
  • ad hoc to paper Policy failure on insufficient depth: L_i < L* implies guaranteed failure with loss >= C_fail (Assumption B.13).
    Close to definitional given L*, but introduces the failure-loss constant needed for the underfitting lower bound.
  • standard math Standard PAC-Bayes and information-theoretic generalization bounds (McAllester 1998; Xu and Raginsky 2017).
    Used as black-box lemmas. Accepted as standard background.
invented entities (4)
  • CoT-Space (continuous reasoning-level semantic manifold)
    purpose: Recasts discrete CoT as continuous optimization to apply classical learning theory.
    A modeling abstraction with no direct empirical handle beyond correlations.
  • Reasoning loss C(.)
    purpose: Proxy for distance to nearest solution; drives all risk analysis.
    Assumed to exist by Definition 2.6; not observable or constructed.
  • Semantic equivalence set V(xi,k) and Decode(.)
    purpose: Quantifies the exponential redundancy of language.
    Abstract constructs; Decode is never specified.
  • Maximal void B^hole_R(s)
    purpose: Bounds continuum approximation errors via largest empty region.
    Theoretical tool for error analysis; not measurable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought." pith.science (2026). https://pith.science/paper/YXYJDKPH

@misc{pith2026250904027,
  author       = {Pith},
  title        = {Pith review of: Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YXYJDKPH}},
  note         = {Machine review of arXiv:2509.04027}
}
read the original abstract

Test-time scaling, primarily manifested through multi-step Chain-of-Thought (CoT) reasoning via Reinforcement Learning (RL), has emerged as a pivotal paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). However, a significant theoretical gap persists: traditional token-level analysis fails to capture the macroscopic dynamics of reasoning-level scaling. To address this, we introduce CoT-Space, a novel theoretical framework that recasts the reasoning process from a discrete token-prediction task to an optimization process within a continuous, reasoning-level semantic space. By modeling the reasoning trajectory from both noise and risk perspectives and revitalizing foundational principles from classical learning theory, we demonstrate that the observed convergence to an optimal CoT length is a natural consequence of the fundamental trade-off between underfitting and overfitting. We further utilize RL as a tool to elicit and verify these results in our experiments. Our findings provide a mechanistic explanation for the internal test-time scaling via RL, offering a principled theoretical foundation to optimize reasoning trajectories in modern LLMs.

Figures

Figures reproduced from arXiv: 2509.04027 by the authors.

Figure 1
Figure 1. Analogy between strategy discovery in classical RL and LLM reasoning. (a) In classical RL, an agent with a high-level goal (e.g., break more bricks) discovers an effective strategy through exploration to maximize its reward. (b) Similarly, an LLM policy with the goal of providing correct answers autonomously learns that generating a suitable CoT is an effective strategy. behavior is a common outcome in classical RL.… view at source ↗
Figure 2
Figure 2. Token-level vs. Reasoning-level per￾spectives. (a) The token-level view treats the gen￾eration process as a path through discrete states. (b) The reasoning-level view zooms out, approxi￾mating the dense state space as continuous. Building upon the discussion above, we now de￾velop a reasoning-level theoretical framework designed to bridge the analytical gap. This al￾ternative perspective draws an analogy to in￾terst… view at source ↗
Figure 3
Figure 3. Illustration of CoT-Space. (a) Different reasoning steps (ξi , ξj , ξk) lead to different minimums (mi , mj , mk), while one step can have multiple token-level realizations (ξi1 , ξi2 , ξi3 ). (b) The discrete state space S (dots) is approximated as a continuous manifold S˜ (blue area), where the real optimization vector (⃗vreal) approximates the ideal one (⃗videal). posits that as the reasoning length increases, th… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Modeling LLM reasoning as an optimization process. (a) Traditional ML performs parameter optimization by updating model weights. (b) Our framework models reasoning as state optimization, where new reasoning steps iteratively update the current state. (c) This panel vis…
Figure 5
Figure 5. Figure 5: Empirical validation of factors governing optimal CoT length (Lopt). (a) Task difficulty: Lopt increases with task complexity. (b) Model capacity: More powerful models converge to a shorter Lopt to mitigate overfitting. (c) RL algorithm: The converged Lopt is largely a…
Figure 6
Figure 6. Figure 6: Supplementary validation of the relationship between task difficulty and optimal CoT length (Lopt) on additional models. The plots illustrate that the converged mean response length (Lopt) consistently increases with task difficulty across three different models: (a) Q…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Survey of Reinforcement Learning for Large Reasoning Models

    cs.CL 2025-09 accept novelty 3.0 of 10

    A survey compiling RL methods, challenges, data resources, and applications for enhancing reasoning in large language models and large reasoning models since DeepSeek-R1.

Reference graph

Works this paper leans on

53 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \"U st \"u n, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024

    arXiv 2024

  3. [3]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  4. [4]

    Towards reasoning era: A survey of long chain-of-thought for reasoning large language models

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025 a

    arXiv 2025

  5. [5]

    Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022

    arXiv 2022

  6. [6]

    An empirical study on eliciting and improving r1-like reasoning models

    Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving r1-like reasoning models. arXiv preprint arXiv:2503.04548, 2025 b

    arXiv 2025

  7. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    arXiv 2025

Show all 53 references
  1. [9]

    Towards a theoretical understanding of synthetic data in llm post-training: A reverse-bottleneck perspective

    Zeyu Gan and Yong Liu. Towards a theoretical understanding of synthetic data in llm post-training: A reverse-bottleneck perspective. arXiv preprint arXiv:2410.01720, 2024

    2024 arXiv

  2. [10]

    Rethinking external slow-thinking: From snowball errors to probability of correct reasoning, 2025

    Zeyu Gan, Yun Liao, and Yong Liu. Rethinking external slow-thinking: From snowball errors to probability of correct reasoning, 2025. URL https://arxiv.org/abs/2501.15602

    2025 arXiv

  3. [11]

    On distances in uniformly random networks

    Martin Haenggi. On distances in uniformly random networks. IEEE Transactions on Information Theory, 51 0 (10): 0 3584--3586, 2005

    2005

  4. [12]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  5. [13]

    Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models

    Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  6. [14]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55 0 (12): 0 1--38, 2023

    2023

  7. [15]

    Enhancing llm reasoning with reward-guided tree search

    Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, et al. Enhancing llm reasoning with reward-guided tree search. arXiv preprint arXiv:2411.11694, 2024

    2024 arXiv

  8. [16]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016

    2016 arXiv

  9. [17]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  10. [18]

    Some pac-bayesian theorems

    David A McAllester. Some pac-bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory, pp.\ 230--234, 1998

    1998

  11. [19]

    The llama 3 herd of models, 2024

    Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  12. [20]

    Nearest neighbor distance in three-dimensional space

    Masashi Miyagawa. Nearest neighbor distance in three-dimensional space. Forma, 33 0 (1): 0 7--11, 2018

    2018

  13. [21]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/learning-to-reason-with-llms/. Accessed: September 12, 2024

    2024

  14. [22]

    Introducing openai o3 and o4-mini, 2025

    OpenAI. Introducing openai o3 and o4-mini, 2025. URL https://openai.com/index/introducing-o3-and-o4-mini/. Accessed: April 16, 2025

    2025

  15. [23]

    Qwq: Reflect deeply on the boundaries of the unknown, November 2024

    Team Qwen. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwenlm.github.io/blog/qwq-32b-preview/

    2024

  16. [24]

    Qwen2.5 technical report, 2025 a

    Team Qwen. Qwen2.5 technical report, 2025 a . URL https://arxiv.org/abs/2412.15115

    2025 arXiv

  17. [25]

    Qwen3 technical report, 2025 b

    Team Qwen. Qwen3 technical report, 2025 b . URL https://arxiv.org/abs/2505.09388

    2025 arXiv

  18. [26]

    Benchmarking prompt sensitivity in large language models

    Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. Benchmarking prompt sensitivity in large language models. In European Conference on Information Retrieval, pp.\ 303--313. Springer, 2025

    2025

  19. [27]

    How much does your data exploration overfit? controlling bias via information usage

    Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory, 66 0 (1): 0 302--323, 2019

    2019

  20. [28]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  21. [29]

    Scaling test-time compute without verification or rl is suboptimal

    Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint arXiv:2502.12118, 2025

    2025 arXiv

  22. [30]

    Spurious rewards: Rethinking training signals in rlvr

    Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.10947, 2025

    2025 arXiv

  23. [31]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  24. [32]

    A bayesian perspective on generalization and stochastic gradient descent

    Samuel L Smith and Quoc V Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018

    2018

  25. [33]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  26. [34]

    Stop overthinking: A survey on efficient reasoning for large language models

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025

    2025 arXiv

  27. [35]

    Understanding chain-of-thought in llms through information theory

    Jean-Francois Ton, Muhammad Faaiz Taufiq, and Yang Liu. Understanding chain-of-thought in llms through information theory. arXiv preprint arXiv:2411.11984, 2024

    2024 arXiv

  28. [36]

    Alphazero-like tree-search can guide large language model decoding and training

    Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus Mcaleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. In International Conference on Machine Learning, pp.\ 49890--49920. PMLR, 2024

    2024

  29. [37]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  30. [38]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022 a

    2022 arXiv

  31. [39]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022 b

    2022

  32. [40]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024

    2024 arXiv

  33. [41]

    Information-theoretic analysis of generalization capability of learning algorithms

    Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in neural information processing systems, 30, 2017

    2017

  34. [42]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 0 11809--11822, 2023

    2023

  35. [43]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  36. [44]

    What's behind ppo's collapse in long-cot? value optimization holds the secret

    Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.01491, 2025

    2025 arXiv

  37. [45]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025 a

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025 a

    2025 arXiv

  38. [46]

    Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025 b

    2025 arXiv

  39. [47]

    Quiet-star: Language models can teach themselves to think before speaking

    Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024

    2024 arXiv

  40. [48]

    Rest-mcts*: Llm self-training via process reward guided tree search

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37: 0 64735--64772, 2024

    2024

  41. [49]

    Echo chamber: Rl post-training amplifies behaviors learned in pretraining

    Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025

    2025 arXiv

  42. [50]

    Prosa: Assessing and understanding the prompt sensitivity of llms

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. Prosa: Assessing and understanding the prompt sensitivity of llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 1950--1976, 2024

    2024

  43. [51]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  44. [52]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  45. [53]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.