Pith. sign in

REVIEW 4 major objections 6 minor 51 references

This paper claims that after reinforcement learning, a model's final performance is set by its pretraining loss and data scale, and that these two quantities predict both the level and the slope of the RL improvement curve.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:21 UTC pith:FP6BSUP7

load-bearing objection A carefully done chess testbed that gives a new joint pretraining–RL scaling law and a clean mechanism story; the headline frontier numbers rest on an extrapolation to Cref=10^20 that the paper never checks against its own asymptote fits. the 4 major comments →

arxiv 2607.16097 v1 pith:FP6BSUP7 submitted 2026-07-17 cs.LG cs.AIcs.CL

Understanding Reasoning from Pretraining to Post-Training

Jingyan Shen , Ang Li , Salman Rahman , Yifan Sun , Micah Goldblum , Matus Telgarsky , Pavel Izmailov This is my paper
classification cs.LG cs.AIcs.CL
keywords pretrainingreinforcement learningscaling lawchess testbedchain-of-thoughtpass@1compute allocationverifiable rewards
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper studies how pretraining choices shape the returns to reinforcement learning, using chess as a controlled stand-in for the full LLM pipeline: pretrain on human games, fine-tune on synthetic reasoning traces, then run RL on verifiable puzzles. It tries to establish a joint scaling law: post-RL pass@1 at any given RL compute budget is a function of pretraining loss, while the rate of RL improvement grows roughly linearly with the log of pretraining tokens. If this law holds, a practitioner can decide how to split compute between pretraining and RL before training, and the paper's extrapolations suggest the optimal RL share rises from about 20% to 28% as total compute grows. Mechanistically, RL does not uniformly sharpen the SFT policy: on easy puzzles it amplifies already-preferred correct moves, while on hard puzzles it surfaces correct moves that were nearly absent and also reinforces some incorrect ones. The same predictive pattern appears in a 1B model pretrained on math text, suggesting the phenomenon is not chess-specific.

Core claim

The central claim is a joint pretraining–RL scaling law, R(CRL,N,T) = f(Lpt(N,T)) + g(N,T)(log10 CRL − log10 Cref). The reward level at reference RL compute is a decreasing exponential function of pretraining validation loss, and the slope g(N,T) is approximately linear in log10 T with a weaker positive dependence on log10 N. Combining this with a Chinchilla-style pretraining loss function lets the authors score hypothetical recipes (N, T, CRL) and trace a compute-optimal frontier: along that frontier the RL share of total compute increases from ~20% at 50M parameters to ~28% at 680M, while the pretraining token allocation stays close to Chinchilla-optimal. The paper also claims that RL resh

What carries the argument

The load-bearing object is the joint scaling law equation, which decomposes post-RL reward into a pretraining-loss-dependent level f(Lpt) plus a slope term g(N,T) times log-RL compute. The slope term is parameterized as g(N,T) = αg + βg log10 T + γg log10 N, and f is an offset exponential in pretraining loss. This machinery converts a two-stage training pipeline into a single predictive function, allowing compute-optimal allocation between pretraining and RL through grid search over (N, T, CRL).

Load-bearing premise

The law assumes that the RL reward curve is a straight line in log-compute over the measured range and that this line can be extrapolated to the top of the sweep and beyond; if RL begins to saturate inside the extrapolated range, the fitted slopes and the 20–28% optimal RL-share frontier are systematically wrong.

What would settle it

Take one mid-size checkpoint (e.g., 200M) that the law predicts has a steep slope, run RL long enough to reach a clear pass@1 plateau on B3–B4, and compare the actual final reward and the shape of the curve to the log-linear extrapolation; if the curve bends downward within one decade of RL compute past Cref, or if the plateau is reached well before the extrapolated line, the local log-linear approximation fails. A second check: hold out a 1B–2B model size entirely, fit the law on smaller sizes, and see whether the predicted RL trajectory and optimal RL share match the measured run.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the law holds, post-RL pass@1 can be predicted from pretraining loss alone, so RL sweeps become unnecessary for choosing among pretrained checkpoints.
  • Longer pretraining does more than improve the starting point: it also increases the rate at which RL compute pays off, so data scale and RL compute are complements, not substitutes.
  • The optimal use of a growing compute budget shifts toward RL: the frontier's RL share rises from roughly 20% at 50M parameters to 28% at 680M, implying pretraining should not be scaled alone.
  • RL's limited pass@16 gains follow from its mixed policy effects: amplification of preferred moves on easy puzzles, tail discovery on hard ones, and wrong-mode amplification, so improving large-k coverage likely requires targeting wrong-mode amplification.
  • The same predictive pattern appears in a 1B model pretrained on math text, suggesting the law may transfer to natural-language reasoning tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence of the law is that any intervention that lowers pretraining validation loss at the same (N, T) — better data quality, better curriculum, or better optimization — should raise the post-RL reward level in a predictable way; this could be checked directly by varying pretraining data quality while holding loss fixed.
  • The law implies a sharper allocation rule: if f is exponential in loss and loss follows Chinchilla scaling, then there is an optimal pretraining-loss 'handoff point' at which RL becomes more valuable than further pretraining; the paper's per-model frontiers are consistent with this, and a direct test would run RL from checkpoints on both sides of the handoff.
  • The mechanism analysis suggests that RL's inability to improve pass@16 is not a fundamental ceiling but a consequence of reward pressure toward high-probability correct moves; adding a reward or curriculum that penalizes wrong-mode amplification could plausibly convert tail discovery into large-k gains without more pretraining compute.
  • The chess testbed makes the law falsifiable at moderate cost: a re-run with verifiable rewards in another discrete domain (for example, theorem proving with a verifier) at 50M–1B scale would tell whether the f(Lpt) + g(N,T)·log CRL decomposition is a general property of the pretraining-to-RL interface or an artifact of chess-like search spaces.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a controlled chess testbed to study how pretraining shapes reinforcement-learning (RL) post-training, and vice versa. It claims a joint pretraining–RL scaling law, Eq. (10), in which post-RL pass@1 at RL-compute level C_RL is predicted by the pretraining validation loss through an exponential map f(L_pt) plus a local log-linear slope g(N,T) in log10 C_RL. The slope is reported to increase approximately linearly with log10 T and weakly with log10 N on the B3–B4 benchmark. Combining this law with a Chinchilla loss surface yields a compute-optimal frontier on which the RL share rises from roughly 20% at 50M parameters to 28% at 680M as total compute grows. The paper also contributes a mechanistic analysis: on easy puzzles RL amplifies already-preferred correct moves, while on hard puzzles it both discovers low-probability correct moves and reinforces incorrect modes, explaining why RL improves pass@1 without consistently improving pass@16. A qualitative math-domain experiment with one 1B OLMo-2 run is offered as evidence of transfer beyond chess.

Significance. If the joint law holds, this would be a useful quantitative bridge between pretraining compute and post-training RL scaling, and the chess testbed itself is a valuable, controlled infrastructure for studying pipeline-level phenomena. The paper is exemplary in transparency: code and models are released, LOO and LMSO validations are reported, the local linearity is explicitly labeled an approximation, and App. G.8 states limitations. The mechanism analysis is a clear strength: move-level policy categorization and structured-Co T tree metrics give a concrete answer to the debated 'sharpening vs. discovery' question. The main caveat is that the law's predictive reach is currently supported mostly by in-sample interpolation and an extrapolated anchor point, so the headline frontier and RL-share trend should be treated as a diagnostic until validated on genuinely held-out configurations.

major comments (4)
  1. [Eq. (1), §G.1, §G.4, §G.6, Fig. 4] The law anchors f(L) at Cref=10^20, but the per-run RL curves used to fit the local lines typically cover only 1–2 decades (e.g., 20M runs in Fig. 15 span log10 C from ~16–18; 680M runs from ~18–19). Eq. (1) is a first-order Taylor expansion of the sigmoid (5), so the truncation error grows quadratically in (log10 C − log10 Cref). If any saturation begins inside the extrapolated range, the fitted Rref overestimates the achievable reward and Eq. (10) inherits a systematic bias that propagates into Fig. 4's RL-share trend. The authors actually fit finite logistic asymptotes A∞ for the 20M family in App. G.6 but never compare the log-linear extrapolation at 10^20 against those asymptotes. The 'agreement' between the law frontier and the empirical frontier in Fig. 4a cannot validate the extrapolated regime because the empirical frontier lies near the observed RL-compute range. I request: (i)
  2. [§3.3, §G.3, Fig. 4] The central claim that Eq. (10) predicts post-RL performance is validated by leave-one-out on the same 36 runs used to fit f, g, and the Chinchilla loss. This measures interpolation error, not predictive extrapolation to hypothetical (N,T,C_RL) recipes. Fig. 4's frontiers are evaluations of the fitted surfaces, so their closeness to the empirical frontier is partly by construction. The LMSO analysis (G.3.1) is a useful step, but it still draws all data from the same sweep and does not include new pretraining/RL configurations. An independent validation set—for example, a few held-out (N,T,C_RL) combinations trained after the law is fixed—is needed to support the paper's predictive language. Until then, the 20–28% RL-share claim should be framed as an interpolation/extrapolation diagnostic rather than a verified prediction.
  3. [§3.3, Figs. 13–14] The slope law g(N,T) is fit only on the B3–B4 benchmark, where models do not saturate. On the easier B1 and B2 benchmarks the association degrades sharply: for B1 the token-only slope fit has R²=0.158 and the joint fit gives a negative log10 N coefficient; for B2 the loss-based fit has R²=0.020. This is acknowledged in §3.3, but the abstract and intro state without qualification that 'the slope of the RL reward curves improves approximately linearly with pretraining tokens.' The domain restriction to non-saturated, intermediate-difficulty benchmarks should appear in the headline claims and in the abstract. This is not a fatal flaw, but it changes the scope of the claimed law and should be explicit wherever Eq. (10) is summarized.
  4. [§5, Tables 14–15] The math transfer experiment uses a single 1B model, a single pretraining run, and 14 checkpoints taken along that same run. The correlations in Fig. 6 therefore use highly autocorrelated checkpoints as independent points; the reported R² values do not reflect independent replication. The authors call this a 'qualitative case study,' but the abstract says the findings 'transfer beyond chess.' I recommend either adding independent runs (different seeds or model sizes) or softening the transfer claim to 'preliminary evidence in a single math-domain run.' The current wording overstates the evidential weight of one trajectory.
minor comments (6)
  1. [Abstract and §3.3] Specify that the joint law applies to pass@1 on the non-saturated B3–B4 benchmark and to the locally log-linear regime; the current phrasing 'post-RL performance' is broader than what is established.
  2. [§3.3] The text says 'our RL compute spans 10^16 to 10^20,' but the per-run fitted ranges shown in Fig. 15 are narrower (e.g., 10^16–10^18 for 20M, 10^18–10^19 for 680M). Clarify whether 10^20 is an observed endpoint or only the reference anchor.
  3. [Fig. 4 caption] The caption includes hypothetical sizes up to 2B, outside the empirical support of the fit. Add a caution that this portion is extrapolation under the fitted law, consistent with G.7.
  4. [Eq. (10)] The law is written as log10 CRL − 20 without defining the implicit Cref=10^20 in the main text; state the value and its role as an anchor explicitly near Eq. (10).
  5. [Table 9] The LOO RMSE columns mix two validation modes (Chinchilla-L and observed-L). Define these abbreviations in the table caption and report the same set of anchors for both modes.
  6. [§5] Typo: 'pretraing-to-post-training' should be 'pretraining-to-post-training' in the opening sentence.

Circularity Check

0 steps flagged

No significant circularity: the joint pretraining-RL law is an empirical fit with LOO/LMSO out-of-sample checks and an independent math-domain replication; the Figure 4 frontier is an explicitly simulated extrapolation, not a hidden refit of the same data.

full rationale

The paper's derivation chain is an empirical scaling-law analysis, not a first-principles derivation, and I found no step where an output is identical to an input by construction. The per-run coefficients (Rref, B) are fitted by OLS to observed RL trajectories (Eq. 1, App. G.1); the maps f(L) and g(N,T) are then fitted to those coefficients, and Eq. (10) composes them. This is a standard two-stage regression, and 'prediction' in the abstract means interpolation/extrapolation from the fitted surface, not a claim that the law was derived independently of the data. The Fig. 4 frontier is explicitly computed by plugging hypothetical (N,T,CRL) into the fitted law and the fitted Chinchilla loss surface and optimizing the split (App. G.7); the agreement with the empirical frontier is an in-sample consistency check, which the paper presents as confirmation rather than as independent validation. Crucially, the paper provides genuine out-of-sample support: leave-one-out validation (App. G.3), leave-one-model-size-out validation that refits the entire pipeline on held-out sizes (App. G.3.1), and a separate 1B math-domain experiment reproducing the same qualitative pattern (Sec. 5). These checks give the central claim independent content beyond the training runs. The paper also clearly flags the local log-linearity assumption and the weak identifiability of the RL asymptote (App. G.1, G.8); those are extrapolation/correctness risks, not circular reductions. I found no load-bearing self-citation (the cited RL sigmoid is from Khatri et al., an external group), no imported uniqueness theorem, and no ansatz smuggled in via self-citation; the exponential form of f and the linear form of g are selected by explicit fit comparisons on the authors' own data. The derivation chain is therefore self-contained with respect to circularity concerns.

Axiom & Free-Parameter Ledger

14 free parameters · 6 axioms · 0 invented entities

No new physical entities are introduced. The law is an empirical fit: 14 fitted surface coefficients plus anchor and threshold choices. Predictive power rests on in-sweep leave-one-out validation and one small transfer experiment, so the central claim's contribution is interpolation-plus-extrapolation of fitted surfaces rather than a derivation.

free parameters (14)
  • Irreducible loss E (Chinchilla) = ≈0.414 (chess)
    Fitted to held-out human-game validation loss in the 11×10 pretraining sweep; sets the floor of L(N,T) in Eq. (3).
  • Chinchilla A (model-scale coefficient) = ≈95.4
    Fitted coefficient in L(N,T)=E+A N^{-α}+B D^{-β}; used to predict pretraining loss for hypothetical (N,T).
  • Chinchilla B (data-scale coefficient) = ≈1.62×10^5
    Fitted coefficient in the Chinchilla loss; with α and β defines the compute-optimal pretraining locus.
  • Chinchilla α (model exponent) = ≈0.404
    Fitted model-size exponent; Table 5 reports 0.401±0.040.
  • Chinchilla β (data exponent) = ≈0.679
    Fitted token exponent; more than twice the language-modeling value (≈0.28).
  • f(L) offset α_f = ≈0.0314
    Intercept in f(L)=α_f+β_f exp(-γ_f L), fit to the 36 observed Rref values.
  • f(L) amplitude β_f = ≈exp(4.87)≈130.4
    Scaling of the exponential dependence of post-RL reference reward on pretraining loss.
  • f(L) decay γ_f = ≈12.85
    Sharpness of the exponential drop in Rref as pretraining loss increases.
  • g(N,T) intercept = ≈0.216
    Intercept in the linear slope law g=α_g+β_g log10 T+γ_g log10 N (Fig. 3c/Fig. 4).
  • g(N,T) log10T coefficient = ≈0.0172
    RL improvement per decade of RL compute grows with log10 pretraining tokens.
  • g(N,T) log10N coefficient = ≈0.0098
    Weaker positive correction from model size.
  • Reference RL compute Cref = 10^20 FLOPs
    Anchor chosen in Table 9 as a compromise; shifts Rref but not the slope; selected after looking at validation.
  • Tail threshold ϵ_tail = 0.05
    Hand-set probability below which a correct move counts as 'nearly absent'; defines the tail-discovery category.
  • Top-k threshold = k=3
    Hand-set size of the candidate set in the policy categorization (Table 13).
axioms (6)
  • domain assumption Pretraining validation loss follows the parametric Chinchilla form L(N,D)=E+A N^{-α}+B D^{-β}.
    Eq. (3). The functional form is taken from Hoffmann et al. and refit to chess; if the form is wrong, predicted L for hypothetical (N,T) is wrong.
  • domain assumption The RL reward curve is sigmoid in log compute and locally log-linear over the measured range.
    Eq. (5) and App G.1. The joint law is a first-order Taylor expansion of Khatri et al.'s sigmoid; extrapolation assumes the local slope persists.
  • domain assumption Chess puzzle-solving with unique solution lines is a faithful controlled proxy for LLM reasoning.
    Sections 1–2. The transfer of conclusions to language models depends on this analogy.
  • domain assumption Pretraining loss is a sufficient scalar predictor of post-RL reward level Rref.
    Eq. (10). The law has no term for data composition, SFT recipe, or other run-level factors beyond L and (N,T).
  • ad hoc to paper The slope parameterization g is linear in log10 T and log10 N.
    Eq. (9). Chosen among candidates by lowest RMSE; not derived from mechanism.
  • standard math FLOP accounting C=6NT and C_rl=10NT_rollout approximates compute.
    App D. Standard dense-transformer estimate; wall-clock differences are intentionally ignored.

pith-pipeline@v1.3.0-alltime-deepseek · 40588 in / 16858 out tokens · 147607 ms · 2026-08-01T21:21:14.654749+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 25 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2410.03893 , year=

    Human-aligned chess with a bit of search , author=. arXiv preprint arXiv:2410.03893 , year=

    Pith/arXiv arXiv

  2. [2]

    Advances in Neural Information Processing Systems , volume=

    Amortized planning with large-scale transformers: A case study on chess , author=. Advances in Neural Information Processing Systems , volume=

  3. [3]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2409.19256 , year=

    Hybridflow: A flexible and efficient rlhf framework , author=. arXiv preprint arXiv:2409.19256 , year=

    Pith/arXiv arXiv

  5. [5]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    Pith/arXiv arXiv 2001

  6. [6]

    arXiv preprint arXiv:2203.15556 , volume=

    Training compute-optimal large language models , author=. arXiv preprint arXiv:2203.15556 , volume=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:2509.21016 , year=

    RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs? , author=. arXiv preprint arXiv:2509.21016 , year=

    arXiv

  8. [8]

    arXiv preprint arXiv:2407.20311 , year=

    Physics of language models: Part 2.1, grade-school math and the hidden reasoning process , author=. arXiv preprint arXiv:2407.20311 , year=

    Pith/arXiv arXiv

  9. [9]

    arXiv preprint arXiv:2509.25123 , year=

    From f (x) and g (x) to f (g (x)) : LLMs Learn New Skills in RL by Composing Old Ones , author=. arXiv preprint arXiv:2509.25123 , year=

    arXiv

  10. [10]

    arXiv preprint arXiv:2510.13786 , year=

    The art of scaling reinforcement learning compute for llms , author=. arXiv preprint arXiv:2510.13786 , year=

    Pith/arXiv arXiv

  11. [11]

    arXiv preprint arXiv:2512.07783 , year=

    On the interplay of pre-training, mid-training, and rl on reasoning language models , author=. arXiv preprint arXiv:2512.07783 , year=

    arXiv

  12. [12]

    arXiv preprint arXiv:2506.16029 , year=

    EvoLM: In Search of Lost Language Model Training Dynamics , author=. arXiv preprint arXiv:2506.16029 , year=

    arXiv

  13. [13]

    Forty-first International Conference on Machine Learning , year=

    Mobilellm: Optimizing sub-billion parameter language models for on-device use cases , author=. Forty-first International Conference on Machine Learning , year=

  14. [14]

    arXiv preprint arXiv:2604.18574 , year=

    When Can LLMs Learn to Reason with Weak Supervision? , author=. arXiv preprint arXiv:2604.18574 , year=

    Pith/arXiv arXiv

  15. [15]

    Nature , volume=

    DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning , author=. Nature , volume=. 2025 , publisher=

    2025

  16. [16]

    arXiv preprint arXiv:2305.08291 , year=

    Large language model guided tree-of-thought , author=. arXiv preprint arXiv:2305.08291 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:1712.01815 , year=

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm , author=. arXiv preprint arXiv:1712.01815 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2504.13837 , year=

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? , author=. arXiv preprint arXiv:2504.13837 , year=

    Pith/arXiv arXiv

  19. [19]

    Advances in Neural Information Processing Systems , volume=

    Star: Bootstrapping reasoning with reasoning , author=. Advances in Neural Information Processing Systems , volume=

  20. [20]

    arXiv preprint arXiv:2510.15020 , year=

    The Coverage Principle: How Pre-Training Enables Post-Training , author=. arXiv preprint arXiv:2510.15020 , year=

    arXiv

  21. [21]

    arXiv preprint arXiv:2510.14901 , year=

    Reasoning with sampling: Your base model is smarter than you think , author=. arXiv preprint arXiv:2510.14901 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2603.24844 , year=

    Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models , author=. arXiv preprint arXiv:2603.24844 , year=

    arXiv

  23. [23]

    International Conference on Learning Representations , volume=

    When scaling meets llm finetuning: The effect of data, model and finetuning method , author=. International Conference on Learning Representations , volume=

  24. [24]

    arXiv preprint arXiv:2510.03264 , year=

    Front-loading reasoning: The synergy between pretraining and post-training data , author=. arXiv preprint arXiv:2510.03264 , year=

    arXiv

  25. [25]

    arXiv preprint arXiv:2606.04272 , year=

    RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training , author=. arXiv preprint arXiv:2606.04272 , year=

    Pith/arXiv arXiv

  26. [26]

    arXiv preprint arXiv:2503.19206 , year=

    Overtrained language models are harder to fine-tune , author=. arXiv preprint arXiv:2503.19206 , year=

    Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    Pith/arXiv arXiv

  28. [28]

    2003 , publisher=

    Nonlinear programming: analysis and methods , author=. 2003 , publisher=

    2003

  29. [29]

    Measuring Mathematical Problem Solving With the MATH Dataset , author=

  30. [30]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2503.21878 , year=

    Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment , author=. arXiv preprint arXiv:2503.21878 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2512.13961 , year=

    Olmo 3 , author=. arXiv preprint arXiv:2512.13961 , year=

    Pith/arXiv arXiv

  33. [33]

    Notion Blog , volume=

    Deepscaler: Surpassing o1-preview with a 1.5 b model by scaling rl , author=. Notion Blog , volume=

  34. [34]

    arXiv preprint arXiv:2501.00656 , year=

    2 OLMo 2 Furious , author=. arXiv preprint arXiv:2501.00656 , year=

    Pith/arXiv arXiv

  35. [35]

    Hugging Face repository , howpublished =

    Jia LI and Edward Beeching and Lewis Tunstall and Ben Lipkin and Roman Soletskyi and Shengyi Costa Huang and Kashif Rasul and Longhui Yu and Albert Jiang and Ziju Shen and Zihan Qin and Bin Dong and Li Zhou and Yann Fleureau and Guillaume Lample and Stanislas Polu , title =. Hugging Face repository , howpublished =. 2024 , publisher =

    2024

  36. [36]

    arXiv preprint arXiv:2512.15489 , year=

    Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision , author=. arXiv preprint arXiv:2512.15489 , year=

    arXiv

  37. [37]

    International Conference on Learning Representations , volume=

    Language models scale reliably with over-training and on downstream tasks , author=. International Conference on Learning Representations , volume=

  38. [38]

    Proceedings of the 41st International Conference on Machine Learning , articleno =

    Sardana, Nikhil and Portes, Jacob and Doubov, Sasha and Frankle, Jonathan , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  39. [39]

    arXiv preprint arXiv:2604.01411 , year=

    Test-Time Scaling Makes Overtraining Compute-Optimal , author=. arXiv preprint arXiv:2604.01411 , year=

    arXiv

  40. [40]

    arXiv preprint arXiv:2503.19551 , year=

    Scaling laws of synthetic data for language models , author=. arXiv preprint arXiv:2503.19551 , year=

    arXiv

  41. [41]

    arXiv preprint arXiv:2603.12151 , year=

    IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL , author=. arXiv preprint arXiv:2603.12151 , year=

    arXiv

  42. [42]

    How Reasoning Evolves from Post-Training Data in Sequential Decision-Making Domains , author=

  43. [43]

    arXiv preprint arXiv:2411.15124 , year=

    Tulu 3: Pushing frontiers in open language model post-training , author=. arXiv preprint arXiv:2411.15124 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2503.14476 , year=

    Dapo: An open-source llm reinforcement learning system at scale , author=. arXiv preprint arXiv:2503.14476 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2503.18892 , year=

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild , author=. arXiv preprint arXiv:2503.18892 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  47. [47]

    Google AI , volume=

    Welcome to the era of experience , author=. Google AI , volume=

  48. [48]

    nature , volume=

    Mastering the game of Go with deep neural networks and tree search , author=. nature , volume=. 2016 , publisher=

    2016

  49. [49]

    Advances in Neural Information Processing Systems , volume=

    Transcendence: Generative models can outperform the experts that train them , author=. Advances in Neural Information Processing Systems , volume=

  50. [50]

    arXiv preprint arXiv:2312.09390 , year=

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision , author=. arXiv preprint arXiv:2312.09390 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2508.15096 , year=

    Nemotron-cc-math: A 133 billion-token-scale high quality math pretraining dataset , author=. arXiv preprint arXiv:2508.15096 , year=

    Pith/arXiv arXiv