Pith. sign in

REVIEW 3 major objections 5 minor 43 references

Iterative self-improvement provably helps only when the model starts in a moderate difficulty window, and easy-to-hard curricula beat fixed mixtures under moderate difficulty gaps and sufficient sample budgets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:37 UTC pith:7B5AZIYW

load-bearing objection Finite-sample lower-bound framework with real novelty, but Assumption 4.3 is impossible as stated and the main theorems need restriction/validation. the 3 major comments →

arxiv 2602.10014 v3 pith:7B5AZIYW submitted 2026-02-10 cs.LG stat.ML

A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula

Chenruo Liu , Yijun Dong , Yiqiu Shen , Qi Lei This is my paper
classification cs.LG stat.ML
keywords iterative self-improvementLLM reasoningeasy-to-hard curriculumfinite-sample guaranteesreward filteringexpected rewardrejection samplingmodel collapse
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish a finite-sample theory for iterative self-improvement—where a language model is fine-tuned on its own reward-filtered outputs—and to identify when an easy-to-hard curriculum across iterations provably beats training on a fixed mixture of tasks. The central result is that self-improvement is governed by an iterated map on the expected reward: better models accept more data, which sustains improvement, but only for initial performances in an intermediate window; too-easy and too-hard tasks both fail. For multiple difficulty levels, the paper derives explicit feasibility and improvement conditions on initialization, difficulty separation, and sample budget under which the easy-to-hard curriculum's lower bound strictly exceeds the baseline's. A sympathetic reader would care because these are finite-sample, task-level conditions that predict the empirically observed saturation, collapse, and curriculum gains in mathematical reasoning pipelines.

Core claim

The paper's central claim is that one step of self-improvement, modeled as maximum-likelihood fine-tuning on accept/reject filtered samples, yields an expected-reward lower bound V_{p0}(θ_{t+1}) ≥ F(V_{p0}(θ_t)), where F(x)=1−γ−c_δν/(c√x−c'_δν). Iterating F explains sustained improvement as a feedback loop—higher reward means more accepted samples, which tightens the MLE bound—and it shows the loop is monotone only for initial rewards inside an interval I(1,ν); outside it, improvement stalls or collapses, and inside it the lower bound saturates below 1−γ. The paper then proves that for L difficulty levels obeying a power-law separation, if the initialization lies in a feasibility interval an

What carries the argument

The load-bearing object is the iterated map F(x)=1−γ−c_δν/(c√x−c'_δν), together with its easy-to-hard variants H_t(x)=F(a_t x). F is derived from a finite-sample MLE bound combined with an assumption that couples the per-question acceptance rate α(θ,q) to the global expected reward V_p(θ) (all but a γ-fraction of questions satisfy α(θ,q) ≥ c·V_p(θ)). The map acts as a one-dimensional dynamical system: its fixed-point interval I(a,ν) describes the initial-performance window where iterates increase monotonically. The easy-to-hard comparison is carried by difficulty-separation exponents β', β and the constants a_0, a_t, a_L that rescale V between adjacent difficulty levels and at the final aver

Load-bearing premise

The whole proof rests on an assumption that for almost every question, the probability the model generates an acceptable answer is at least a constant c times the model's expected reward on that question distribution, and the constants c and γ are never measured or verified.

What would settle it

Measure per-question acceptance rate and expected reward on a real model-task pair and check the inequality Pr_q[α(θ,q) < c·V_p(θ)] ≤ γ; if the empirical distribution of α/V is not concentrated above a single fixed c, or if the estimated γ is large, the F-map guarantee does not hold. Alternatively, sweep the per-iteration question budget n on a fixed curriculum and look for the predicted sharp collapse of the easy-to-hard advantage at a critical n; failure to find any such threshold would contradict the phase-transition prediction.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the theory is right, self-improvement produces no gains for tasks that are too easy or too hard relative to the pretrained model; practitioners should first estimate the initial expected reward and locate it inside the moderate window before running repeated self-improvement rounds.
  • The saturating upper bound below 1−γ explains why pure self-improvement plateaus and why successful pipelines add a different phase (e.g., reinforcement learning) after the plateau.
  • Increasing the question budget n expands the monotonicity interval and thus the set of initializations for which self-improvement is guaranteed to help—a concrete finite-sample effect.
  • For curricula, there is a critical sample budget n_c: above it, easy-to-hard is provably better for a range of initializations; near n_c the advantageous region collapses sharply, limiting when curriculum scheduling helps.
  • Moderate difficulty separation between adjacent tasks is best; both too-similar and too-separated tasks shrink the provably advantageous regime.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The theory suggests a diagnostic: measure the acceptance-rate-versus-reward coupling constants c and γ for a given model and task set; if they are not favorable, one should expect self-improvement to fail regardless of iteration count.
  • The sharp phase transition in n predicts that curriculum benefits should be most visible in an intermediate compute regime; at very large budgets the baseline may catch up, and at very small budgets neither scheme improves. This is testable by sweeping n and measuring the gap.
  • The power-law separation condition is stronger than what experiments directly verify; a natural extension is to test whether real difficulty ratios need only hold on average over θ rather than uniformly over the post-training neighborhood.
  • The framework suggests a concrete curriculum design rule: choose adjacent difficulty ratios such that the initialization lies in the feasible and improving regions, and tune task pacing to keep the evolving lower-bound sequence inside the expanding invariant intervals.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper develops a finite-sample, task-centric theory of iterative LLM self-improvement. Each round is modeled as maximum-likelihood fine-tuning on reward-filtered, self-generated data. For a single task, Theorem 4.1 gives a finite-sample lower bound on expected reward, and under Assumption 4.3 the authors derive an iterated lower-bound map F whose monotone regime defines a moderate-difficulty window and a saturation ceiling (Corollary 4.4, Proposition 4.5). For multiple tasks, the paper compares an easy-to-hard curriculum with a fixed-mixture baseline under Assumption 5.1, and Theorem 5.2 gives sufficient feasibility and improvement conditions M_i<0 and N<0 under which the easy-to-hard lower bound strictly dominates. The predictions are supported by Monte-Carlo simulations, a synthetic shortest-path task, and experiments on GSM8K and DeepMind Mathematics.

Significance. If the stated results were valid, the paper would provide a valuable finite-sample account of self-improvement, connecting initialization quality, sample budget, task difficulty, and curriculum scheduling. The proof structure is detailed, the F-map analysis is coherent, and the experiments span both controlled synthetic tasks and standard benchmarks. The key caveat is that the central conclusions are conditional on Assumptions 4.3 and 5.1, whose current statements are not validated and, in the case of Assumption 4.3, are internally inconsistent under the stated quantifier. The paper would be much stronger if these assumptions were restricted to the distributions actually used and if the constants c and gamma were measured or calibrated.

major comments (3)
  1. [Assumption 4.3 / Section 4.2, Corollary 4.4] The universal quantifier in Assumption 4.3 makes the assumption impossible in the regime the paper targets. Fix any theta in Theta with V:=V_p0(theta)<1-gamma and choose epsilon < cV. Let p put mass (1-V)/(1-epsilon) on a question with alpha=epsilon and the remaining mass on a question with alpha=1. Then E_q alpha = V, but Pr_q[alpha < cV] = 1-V > gamma. Since Proposition 4.5 defines the self-improvement interval I(1,nu) subset of (c_delta' nu, 1-gamma), every initialization in the claimed improvement window violates the assumption. This is load-bearing: Corollary 4.4 and Theorem 5.2 use F via Assumption 4.3. If the intended assumption was p-specific (only p0 and the curriculum distributions p_i), that restriction must be explicit, and c,gamma must be empirically verified; the current text neither restricts nor measures them.
  2. [Assumption 5.1 / Section 5.2, Theorem 5.2] Assumption 5.1 requires a uniform power-law bound on V_pi(theta)/V_p{i+1}(theta) over all theta in Theta. The empirical section estimates beta and beta' only from the initialization model (Figure 5 and Section 6.2.2), so it does not verify the uniform-in-theta condition. Moreover, the feasibility and improvement conditions M_i<0 and N<0 in Theorem 5.2 depend on unmeasured constants c,gamma and on worst-case exponents beta,beta'. The synthetic experiments partly enforce the difficulty separation by construction (Appendix D.1), and the benchmark experiments do not report estimates of c,gamma at all. At a minimum, the paper should state a verifiable trajectory-level condition and provide measurements of the postulated constants.
  3. [Sections 4-5 overall] The main theoretical conclusions are conditional on free parameters c,gamma,beta,beta',|Pi| that are never instantiated or estimated. This does not, by itself, invalidate a qualitative theorem, but the paper's narrative treats Corollary 4.4 and Theorem 5.2 as predictive statements. Since the key assumptions are not checked, the empirical agreements in Section 6 are not direct confirmations of the theorem's hypotheses. The authors should either estimate these parameters from the synthetic/benchmark data or state more modestly that the theory provides qualitative trends under unverified conditions.
minor comments (5)
  1. [Section 4.1 / Notation] The notation n_t and n_t^(m) is sometimes used interchangeably in the main text; Corollary 4.4 uses the m=1 special case but this is stated only implicitly. A short remark clarifying the m dependence would help.
  2. [Section 6.2 / Table 4] Table 4 excludes modules whose canonical difficulty split yields beta'<0.1 or whose gap variation is below 1%. This selection criterion is not justified and could bias the empirical support. Please discuss the excluded modules and show their results in the appendix.
  3. [Figure 5] The estimates of beta and beta' are plotted as single curves without standard errors or seeds. Since these quantities are central to Assumption 5.1, report their variability.
  4. [Appendix B.1 / Lemma B.1] The Hellinger-to-total-variation step has a factor that is not tracked precisely (the definition of D_H^2 and the bound in Lemma B.1 differ by a factor of two from the standard Hellinger inequality). This does not affect the qualitative conclusions because c absorbs constants, but the definitions should be aligned.
  5. [Remark 4.6.3] The remark says the lower bound cannot exceed x_+(1,nu), which is strictly below 1-gamma. This is correct for the lower-bound map, but the wording 'inherently bounded' could be read as a statement about actual performance; clarify that saturation is a property of the proved bound, not a proven upper bound on the true reward.

Circularity Check

0 steps flagged

No circularity found: the F-map and the N<0 condition are derived from stated assumptions and concentration inequalities, not fitted to or defined as the target inequality.

full rationale

The derivation chain is self-contained in the relevant sense. Theorem 4.1 obtains its finite-sample lower bound from a standard MLE/Hellinger concentration lemma (Lemma B.1, citing Geer, Wong–Shen, Zhang, Huang et al.), with the finite model class Pi as an explicit assumption. Corollary 4.4 does not fit c, gamma, or nu to force the target inequality: Assumption 4.3 is a stated tail/coupling condition, and the F-map is obtained by a Hoeffding bound on n_t plus the same Hellinger bound; the constants c, gamma, c_delta, c_delta' are assumed or derived from failure probabilities, not calibrated to make V_{t+1} >= F(V_t) true. The 'saturation' below 1-gamma is literally the assumed bad-question fraction gamma in the proof's bound delta_bar <= gamma + (1/c)E[delta_t]; it is presented as a bound on the lower-bound guarantee, not as an empirical law, so it is not an input renamed as a prediction. In Theorem 5.2, the feasibility conditions M_i<0 and the improvement condition N<0 are introduced in Definition A.2 and proven (Corollary 5.3, Proposition 5.5) to be equivalent to explicit intervals for V_{p0}(theta_hat_0); N<0 is constructed in the proof as a sufficient algebraic condition for the comparison gap to be positive, not defined as the gap itself. No load-bearing self-citation or imported uniqueness/ansatz is used: citations to [16] and [31] support the finite-Pi modeling choice but are not the argument that produces the main inequalities. The skeptical concern that Assumption 4.3's universal quantifier may be internally inconsistent for V_p(theta)<1-gamma is a correctness/validity objection, not a demonstration that a prediction reduces to an input by construction; the paper would still fail or succeed on that basis independently of circularity. Experiments use controlled synthetic data and benchmarks to compare qualitative trends and do not fit c, gamma, beta', or beta to the target inequalities. Therefore no circular step meeting the quoting standard was identified.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The main theorems are conditional on c and gamma from Assumption 4.3 and on beta' and beta from Assumption 5.1, none of which are assigned values or independently verified; beta' and beta are estimated only at the initialization model, not over the required neighborhood Theta. These constants determine all intervals, the saturation ceiling, and the E2H improvement condition, so the practical scope of the guarantees is proportional to the plausibility of these assumptions. No new entities are invented.

free parameters (5)
  • c
    Coupling constant in Assumption 4.3: per-question acceptance is assumed to be at least c times global expected reward except on a gamma-mass of questions. No value or estimation procedure is given; all F-map bounds and the saturation ceiling depend on it.
  • gamma
    Exceptional-mass constant in Assumption 4.3. It appears additively in F and sets the 1-gamma saturation ceiling; it is never measured.
  • beta_prime
    Lower difficulty-exponent in Assumption 5.1. Figure 5 estimates it only at the initialization model, but the assumption requires the bound to hold for all theta in Theta.
  • beta
    Upper difficulty-exponent in Assumption 5.1. It enters a0, aL, at, and the improvement condition N; again verified only at initialization in experiments, not over the whole parameter neighborhood.
  • |Pi|
    Size of the finite model class in Theorem 4.1 and Lemma B.1. It enters c_delta and therefore all finite-sample gaps, but no bound or estimate is given.
axioms (6)
  • domain assumption The population optimizer of the filtered MLE objective belongs to the finite model class Pi.
    Invoked in Lemma B.1 and Theorem 4.1; without this realizability assumption the MLE concentration bound does not apply.
  • domain assumption alpha(theta,q) > 0 for all q and theta, i.e., every question has nonzero probability of passing the reward filter.
    Stated in the footnote after Eq. (3.1); needed to define the filtered distribution D' and to use ess-inf lower bounds.
  • domain assumption Assumption 4.3: there exist c in (0,1) and gamma >= 0 such that for all question distributions p and all theta in Theta, Pr_q[alpha(theta,q) < c V_p(theta)] <= gamma.
    Load-bearing: it converts the MLE error over filtered answers into a bound depending only on V_p(theta), producing the F-map used throughout Sections 4 and 5.
  • domain assumption Assumption 5.1: for all theta in Theta and all adjacent difficulty levels, V_{p_i}(theta)/V_{p_{i+1}}(theta) lies between i^{-beta'}/(i+1)^{-beta'} and i^{-beta}/(i+1)^{-beta}, with 0 < beta' < beta.
    Enables the chaining of easy-to-hard lower bounds and the final a_L conversion. The experiments estimate beta, beta' only at the initialization model, not over the full neighborhood Theta.
  • domain assumption Binary reward with threshold: 1{s(q,a) >= tau} = s(q,a) for s in {0,1}.
    Identifies acceptance rate with expected reward and yields the identity Z_p(theta)=V_p(theta) used in Corollary 4.4.
  • standard math Standard MLE concentration bounds from Geer, Wong-Shen, Zhang, used as Lemma B.1.
    Imported concentration inequalities for finite model classes; the paper does not re-derive them.

pith-pipeline@v1.3.0-alltime-deepseek · 56954 in / 21408 out tokens · 211045 ms · 2026-08-03T02:37:13.746494+00:00 · methodology

0 comments
read the original abstract

Iterative self-improvement fine-tunes an autoregressive large language model (LLM) on reward-verified outputs generated by the LLM itself. In contrast to the empirical success of self-improvement, the theoretical foundation of this generative, iterative procedure in a practical, finite-sample setting remains limited. We make progress toward this goal by modeling each round of self-improvement as maximum-likelihood fine-tuning on a reward-filtered distribution and deriving finite-sample guarantees for the expected reward. Our analysis reveals an explicit feedback loop where better models accept more data per iteration, supporting sustained self-improvement while explaining eventual saturation of such improvement. Adopting a task-centric view by considering reasoning tasks with multiple difficulty levels, we further prove quantifiable conditions on model initialization, task difficulty, and sample budget where easy-to-hard curricula provably achieve better guarantees than training on fixed mixtures of tasks. Our analyses are validated through Monte-Carlo simulations and experiments spanning a synthetic graph-based reasoning task and multiple standard mathematical reasoning benchmarks.

Figures

Figures reproduced from arXiv: 2602.10014 by Chenruo Liu, Qi Lei, Yijun Dong, Yiqiu Shen.

Figure 1
Figure 1. Figure 1: Feasible initialization region. Panels (a,c) report Monte-Carlo estimates of the length of the initialization interval Vp0 ( ˆθ0) for which {F ◦t (Vp0 ( ˆθ0))}t≥0 and {(Ht ◦ · · · ◦ H0)(Vp0 ( ˆθ0))}t≥0 are both monotonically increasing in t, under different (β ′ , β, ν) settings. Panels (b,d) show the length of the feasibility interval IM(β ′ , β, ν) in Corollary 5.3. Panels (a,b): fix β ′ = 0.1 and vary (… view at source ↗
Figure 2
Figure 2. Figure 2: Improvement initialization region. Panels (a)-(c) report Monte-Carlo estimates of the length of the initialization interval Vp0 ( ˆθ0) for which (G ◦ HL−1 ◦ HL−2 ◦ · · · ◦ H0) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Iterative self-improvement results on the synthetic shortest path task. Panel (a) shows the self-improvement trajectories of a fixed ˆθ0 across tasks with different initial Pass@1 accuracies; hollow markers and faded line segments indicate model collapse (Pass@1= 0 for at least one target distance l). Panel (b) shows the performance under different question budgets n, with ˆθ0 and the initial Pass@1 fixed.… view at source ↗
Figure 4
Figure 4. Figure 4: Iterative self-improvement with easy-to-hard curriculum on the synthetic shortest path task. Panel (a) fixes ∆ = 0.04 and ˆθ0, and shows for different initial Pass@1 accuracies, the final Pass@1 gap between easy-to-hard and the baseline (i.e., Vp0 ( ˆθ E2H L ) − Vp0 ( ˆθ B L )) as a function of the adjacent task difficulty ratio (captured by β ′ ). Panel (b) fixes ∆ = 0.04 and β ′ = 0.25, and shows for dif… view at source ↗
Figure 5
Figure 5. Figure 5: The left panel plots the estimate of β ′ as a function of ρ, and the right panel plots the estimate of β as a function of ρ, for all modules reported in [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 26 linked inside Pith

  1. [1]

    Intrinsic dimensionality explains the effectiveness of language model fine-tuning

    Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. InProceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 7319–7328, 2021

    2021

  2. [2]

    Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling.arXiv preprint arXiv:2408.16737, 2024

    Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling.arXiv preprint arXiv:2408.16737, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. 15

    Pith/arXiv arXiv 2021

  4. [4]

    Understanding self-distillation in the presence of label noise

    Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. InInternational Conference on Machine Learning, pages 7102–7140. PMLR, 2023

    2023

  5. [5]

    Beyond model collapse: Scaling up with synthesized data requires verification.arXiv preprint arXiv:2406.07515, 2024

    Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model collapse: Scaling up with synthesized data requires verification.arXiv preprint arXiv:2406.07515, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Self-consuming generative models with curated data provably optimize human preferences.arXiv preprint arXiv:2407.09499, 2024

    Damien Ferbach, Quentin Bertrand, Avishek Joey Bose, and Gauthier Gidel. Self-consuming generative models with curated data provably optimize human preferences.arXiv preprint arXiv:2407.09499, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Towards theoretical understandings of self-consuming generative models.arXiv preprint arXiv:2402.11778, 2024

    Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, and Dacheng Tao. Towards theoretical understandings of self-consuming generative models.arXiv preprint arXiv:2402.11778, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Self-verification provably prevents model collapse in recursive synthetic training

    Shi Fu, Yingjie Wang, Yuzhu Chen, Li Shen, and Dacheng Tao. Self-verification provably prevents model collapse in recursive synthetic training. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  9. [9]

    A theoretical perspective: Howtopreventmodelcollapseinself-consumingtrainingloops.arXiv preprint arXiv:2502.18865, 2025

    Shi Fu, Yingjie Wang, Yuzhu Chen, Xinmei Tian, and Dacheng Tao. A theoretical perspective: Howtopreventmodelcollapseinself-consumingtrainingloops.arXiv preprint arXiv:2502.18865, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Cambridge university press, 2000

    Sara A Geer.Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000

    2000

  11. [11]

    Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data

    Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv preprint arXiv:2404.01413, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Self-correcting self-consuming loops for generative model training.arXiv preprint arXiv:2402.07087, 2024

    Nate Gillman, Michael Freeman, Daksh Aggarwal, Chia-Hong Hsu, Calvin Luo, Yonglong Tian, and Chen Sun. Self-correcting self-consuming loops for generative model training.arXiv preprint arXiv:2402.07087, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519, 2025

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Self-improvement in language models: The sharpening mechanism.arXiv preprint arXiv:2412.01951, 2024

    Audrey Huang, Adam Block, Dylan J Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism.arXiv preprint arXiv:2412.01951, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Adastar: Adaptive data sampling for training self-taught reasoners.arXiv preprint arXiv:2505.16322, 2025

    Woosung Koh, Wonbeen Oh, Jaein Jang, MinHyung Lee, Hyeongjin Kim, Ah Yeon Kim, Joonkee Kim, Junghyun Lee, Taehyeon Kim, and Se-Young Yun. Adastar: Adaptive data sampling for training self-taught reasoners.arXiv preprint arXiv:2505.16322, 2025. 16

    arXiv 2025

  18. [18]

    Self-improving transformers overcome easy-to-hard and length generalization challenges.arXiv preprint arXiv:2502.01612, 2025

    Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, and Dimitris Papailiopoulos. Self-improving transformers overcome easy-to-hard and length generalization challenges.arXiv preprint arXiv:2502.01612, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Goedel-prover: A frontier model for open-source automated theorem proving.arXiv preprint arXiv:2502.07640, 2025

    Yong Lin, Shange Tang, Bohan Lyu, Jiayun Wu, Hongzhou Lin, Kaiyu Yang, Jia Li, Mengzhou Xia, Danqi Chen, Sanjeev Arora, et al. Goedel-prover: A frontier model for open-source automated theorem proving.arXiv preprint arXiv:2502.07640, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Goedel-prover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction.arXiv preprint arXiv:2508.03613, 2025

    Yong Lin, Shange Tang, Bohan Lyu, Ziran Yang, Jui-Hui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng, Jiawei Ge, Jingruo Sun, et al. Goedel-prover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction.arXiv preprint arXiv:2508.03613, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Self-distillation amplifies regular- ization in hilbert space.Advances in Neural Information Processing Systems, 33:3351–3361, 2020

    Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regular- ization in hilbert space.Advances in Neural Information Processing Systems, 33:3351–3361, 2020

    2020

  22. [22]

    Coherence mechanisms for provable self- improvement.arXiv preprint arXiv:2511.08440, 2025

    Mehryar Mohri, Jon Schneider, and Yifan Wu. Coherence mechanisms for provable self- improvement.arXiv preprint arXiv:2511.08440, 2025

    arXiv 2025

  23. [23]

    Understanding the gains from repeated self-distillation.Advances in Neural Information Processing Systems, 37:7759–7796, 2024

    Divyansh Pareek, Simon S Du, and Sewoong Oh. Understanding the gains from repeated self-distillation.Advances in Neural Information Processing Systems, 37:7759–7796, 2024

    2024

  24. [24]

    Deepseek-prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition.arXiv preprint arXiv:2504.21801, 2025

    ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, et al. Deepseek-prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition.arXiv preprint arXiv:2504.21801, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Analysing mathematical reasoning abilities of neural models.arXiv preprint arXiv:1904.01557, 2019

    David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models.arXiv preprint arXiv:1904.01557, 2019

    Pith/arXiv arXiv 1904

  26. [26]

    A mathematical theory of communication.The Bell system technical journal, 27(3):379–423, 1948

    Claude E Shannon. A mathematical theory of communication.The Bell system technical journal, 27(3):379–423, 1948

    1948

  27. [27]

    Beyond human data: Scaling self-training for problem-solving with language models.arXiv preprint arXiv:2312.06585, 2023

    Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models.arXiv preprint arXiv:2312.06585, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Mind the gap: Examining the self-improvement capabilities of large language models.arXiv preprint arXiv:2412.02674, 2024

    Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities of large language models.arXiv preprint arXiv:2412.02674, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Theoretical modeling of llm self- improvement training dynamics through solver-verifier gap.arXiv preprint arXiv:2507.00075, 2025

    Yifan Sun, Yushan Liang, Zhen Zhang, and Jiaye Teng. Theoretical modeling of llm self- improvement training dynamics through solver-verifier gap.arXiv preprint arXiv:2507.00075, 2025

    arXiv 2025

  30. [30]

    Canlanguagemodelssolvegraphproblemsinnaturallanguage?Advances in Neural Information Processing Systems, 36:30840–30861, 2023

    Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Canlanguagemodelssolvegraphproblemsinnaturallanguage?Advances in Neural Information Processing Systems, 36:30840–30861, 2023

    2023

  31. [31]

    Huxley-gödel machine: Human-level coding agent development by an approximation of the optimal self-improving machine, 2025

    Wenyi Wang, Piotr Piękos, Li Nanbo, Firas Laakom, Yimeng Chen, Mateusz Ostaszewski, Mingchen Zhuge, and Jürgen Schmidhuber. Huxley-gödel machine: Human-level coding agent development by an approximation of the optimal self-improving machine, 2025. 17

    2025

  32. [32]

    Propose, solve, verify: Self-play through formal verification.arXiv preprint arXiv:2512.18160, 2025

    Alex Wilf, Pranjal Aggarwal, Bryan Parno, Daniel Fried, Louis-Philippe Morency, Paul Pu Liang, and Sean Welleck. Propose, solve, verify: Self-play through formal verification.arXiv preprint arXiv:2512.18160, 2025

    arXiv 2025

  33. [33]

    Probability inequalities for likelihood ratios and convergence rates of sieve mles.The Annals of Statistics, pages 339–362, 1995

    Wing Hung Wong and Xiaotong Shen. Probability inequalities for likelihood ratios and convergence rates of sieve mles.The Annals of Statistics, pages 339–362, 1995

    1995

  34. [34]

    Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data.arXiv preprint arXiv:2405.14333, 2024

    Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data.arXiv preprint arXiv:2405.14333, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Deepseek-prover-v1

    HuajianXin, ZZRen, JunxiaoSong, ZhihongShao, WanjiaZhao, HaochengWang, BoLiu, Liyue Zhang, Xuan Lu, Qiushi Du, et al. Deepseek-prover-v1. 5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search.arXiv preprint arXiv:2408.08152, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    Pith/arXiv arXiv 2024

  37. [37]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  38. [38]

    Spendwisely: Maximizing post-training gains in iterative synthetic data bootstrapping.arXiv preprint arXiv:2501.18962, 2025

    PuYang, YunzhenFeng, ZiyuanChen, YuhangWu, andZhuoyuanLi. Spendwisely: Maximizing post-training gains in iterative synthetic data bootstrapping.arXiv preprint arXiv:2501.18962, 2025

    arXiv 2025

  39. [39]

    Optimizing chain-of-thought reasoners via gradient variance minimization in rejection sampling and rl.arXiv preprint arXiv:2505.02391, 2025

    Jiarui Yao, Yifan Hao, Hanning Zhang, Hanze Dong, Wei Xiong, Nan Jiang, and Tong Zhang. Optimizing chain-of-thought reasoners via gradient variance minimization in rejection sampling and rl.arXiv preprint arXiv:2505.02391, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

    2022

  41. [41]

    B-star: Monitoring and balancing exploration and exploitation in self-taught reasoners.arXiv preprint arXiv:2412.17256, 2024

    Weihao Zeng, Yuzhen Huang, Lulu Zhao, Yijun Wang, Zifei Shan, and Junxian He. B-star: Monitoring and balancing exploration and exploitation in self-taught reasoners.arXiv preprint arXiv:2412.17256, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Leanabell-prover: Posttraining scaling in formal reasoning.arXiv preprint arXiv:2504.06122, 2025

    Jingyuan Zhang, Qi Wang, Xingguang Ji, Yahui Liu, Yang Yue, Fuzheng Zhang, Di Zhang, Guorui Zhou, and Kun Gai. Leanabell-prover: Posttraining scaling in formal reasoning.arXiv preprint arXiv:2504.06122, 2025. 18

    Pith/arXiv arXiv 2025

  43. [43]

    question distribution

    Tong Zhang. Fromε-entropy to kl-entropy: Analysis of minimum information complexity density estimation.The Annals of Statistics, pages 2180–2210, 2006. 19 Appendix: Proofs A. Notation and Deferred Definitions A.1 Notation Summary Symbol Meaning qquestion aanswer s(q,a)reward;s(q,a)∈[0,1], where larger values indicate a better answerato the questionq τacce...

    2006