Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that the reasoning gains of R1-style RL are reproducible by fine-tuning on about one thousand responses sampled from the RL-trained policy, a procedure it calls re-distillation, and that this works because RL raises the…

desk verdict Re-distillation is a real and useful empirical finding, but the K&K 'matches RL' claim is undercut by unequal evaluation token budgets and a numeric inconsistency, and the theory is largely definitional. read the letter →

arxiv 2505.17988 v3 pith:WTCXU5CR submitted 2025-05-23 cs.LG cs.AI

Yutong Chen , Jiandong Gao , Ji Wu This is my paper · ORCID
classification cs.LGcs.AI
keywords re-distillationR1-stylereinforcementlearningsmall-scaleSupervisedFine-TuningsampleeffectGRPOreasoninggeneralizationpolicydistillationLLMpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the gains of R1-style reinforcement learning (RL) for LLM reasoning are not intrinsic to the RL loop: they can be transferred to a base model by supervised fine-tuning on about one thousand responses sampled from the converged RL policy. It calls this re-distillation. On the Knight & Knave puzzle, a 1.5B model fine-tuned on 1K re-distilled samples matches the RL policy's accuracy (reported as 78.8% versus 82.0% in one comparison, and 0.820 average in Table 2) and surpasses DeepSeek-V3-0324. The paper accounts for this with an analytic 'sample effect' that measures each training sample's contribution to test-accuracy growth, proving that the optimal distillation target over-weights high-effect samples and that RL raises the sample effect of its own outputs. If correct, this means expensive RL runs can be compressed into cheap, re-usable SFT datasets, and that SFT's poor sample efficiency is a property of the data, not a fixed limit of the method.

What carries the argument

The carrying object is the sample effect, $V(a,s,\theta)$—for a prompt-response pair $(a,s)$, the inner product of its policy-gradient direction with the test-set reward gradient. It lets the paper write the growth rate of test accuracy during SFT or RL as a weighted average of sample effects, ignoring gradient noise via an SDE/drift-term approximation. Theorem 5.1, whose proof follows the closed-form derivation pattern of DPO, gives the optimal distillation target as a policy whose log-odds are shifted proportionally to the sample-effect difference, so high-effect correct answers receive more probability mass. Theorem 5.2 shows the dataset effect of a policy trained by RL is bounded below by the square of its accuracy growth rate, which is why the RL policy's own outputs are unusually good SFT data. Re-distillation is the practical recipe that samples correct responses from the RL replay buffer and fine-tunes the base model on them.

What would settle it

A direct test is to compute the linearized sample effect of rollout data from early and late checkpoints of a single RL run and compare with the actual SFT gains from distilling each checkpoint: the theory predicts late-checkpoint data loses effectiveness as dataset-effect growth decays, so if late-checkpoint re-distillation transfers undiminished, the explanation fails. A second observation, noted by the paper itself, is that the framework misorders long-CoT versus short-CoT SFT data on MATH, so any experiment where a predicted sample-effect ranking reverses observed SFT efficiency would also falsify the claim that sample effect carries the effect.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a small supervised fine-tuning set drawn from an RL-trained policy can stand in for the entire RL run. On K&K, RD-rl-kk—1,000 correct responses sampled from the RL replay buffer at step 125—brings the base model to the same average accuracy as the RL policy (0.820, and 78.8% vs 82.0% in the step-0 comparison), beating DeepSeek-V3-0324's 0.808; on MATH, RD-sft-math with 496 samples matches the 50-step RL checkpoint (54.4% vs 56.0%) and the Instruct baseline (54.4% vs 55.0%); on REASONING GYM, re-distilled models keep most of the RL improvement while losing only about 1.5–1.7% accuracy on average. The explanation offered is that RL-trained policies have high output sample effect: Theorem 5.1 shows the optimal distillation target shifts probability mass toward correct responses with high sample effect, and Theorem 5.2 guarantees the dataset effect of an RL-trained policy grows whenever training accuracy grows, under the assumption that linearized sample effect explains the non-linear training dynamics.

Load-bearing premise

The load-bearing premise is that a linearized 'sample effect'—computed while ignoring gradient noise and nonlinear interactions—can explain how multi-step, non-linear LLM training improves accuracy; if nonlinear effects dominate in real SFT or RL, the theoretical prescriptions do not follow.

Editorial extensions

If this is right

  • A single converged RL run can be compressed into roughly 1K supervised pairs, enabling transfer of the RL policy's reasoning ability to a base model by SFT alone.
  • Data quality for SFT should be judged by sample effect rather than human-perceived reasoning quality; on identical questions, re-distilled responses beat DeepSeek-R1-distilled ones (RD-rl-kk vs RD-rl-baseline-kk).
  • Re-distillation is model-specific; training Llama on Qwen-generated re-distilled data degrades markedly, so RL gains are tied to the source policy's own parameter trajectory.
  • RL policies trained for different goals can be merged by concatenating their re-distilled datasets, offering a cheap path to a Pareto frontier (RD-combined).
  • SFT's poor sample efficiency is not intrinsic; it reflects the sample effect of the data, so the analytic framework gives a principle for constructing more efficient SFT datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linearized sample-effect story survives contact with larger models, sample effect could be computed cheaply and used to select SFT data before any RL run, turning re-distillation from a post-hoc recipe into a data-selection principle.
  • The success of re-distillation implies RL's role in R1-style training may be to generate a better target distribution rather than to provide a uniquely necessary optimization path; a testable extension is whether the same compression works for safety or instruction-following RL.
  • The paper's own failure to predict long-CoT vs short-CoT ordering on MATH suggests non-linear effects matter; a natural next experiment is measuring whether sample-effect rankings align better with observed SFT efficiency as model size grows.
  • Since cross-model transfer fails, a testable hypothesis for why is capacity: re-distilling Qwen samples into a larger Llama may recover the transfer that failed at 3B.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies small-scale supervised fine-tuning (SFT, at most 1K examples) in R1-style reinforcement learning (RL). It first reports that SFT with about 1K examples substantially influences subsequent RL but that simple data-selection principles do not explain the observed patterns. The authors then propose a linearized analytical framework based on a per-sample 'sample effect' V(a,s,θ), defined so that the approximated reward-growth rate is an expectation of sample effects. They use this framework to argue that distillation policies should favor high-sample-effect responses and that RL training increases the dataset effect of RL-generated rollouts. Guided by this analysis, they introduce Re-distillation: sampling correct responses from an RL-trained policy and using them as SFT data for the base model. Experiments on K&K, MATH, and REASONING GYM with Qwen-2.5-1.5B and Llama-3.2-3B report that re-distilled models nearly match RL-trained policies using only SFT, that RD-rl-kk reaches 0.820 average accuracy on K&K and surpasses DeepSeek-V3-0324, and that re-distillation can merge two RL-trained policies. The paper also includes hyperparameter ablations, an interpolation study, and an analysis of position-wise token probabilities to support the proposed mechanism.

Significance. If the empirical result survives controlled evaluation, the claim that roughly 1K SFT samples can reproduce most of an RL-trained policy's performance is practically important and scientifically interesting. The paper has clear strengths: the RD-rl-baseline-kk control uses the same 1K questions with DeepSeek-R1 samples, providing a direct comparison that isolates the RL-trained policy as the data source; the method is tested on three datasets and two model families; the code is released; and the theoretical section is explicitly framed as hypothetical, with a limitations paragraph acknowledging the linearization assumption. At the same time, the theoretical contribution is largely definitional under the linearized SDE approximation, and the flagship K&K comparison is complicated by an evaluation-token asymmetry and by inconsistencies between the table and the text. The empirical claim therefore carries most of the weight, which is why experimental protocol issues are the central concern.

major comments (3)
  1. [Section E and Table 2] The headline K&K comparison is confounded by unequal evaluation token budgets. Section E states that RD-rl-kk is evaluated with Lmax=8192, while 'other trained models', including the RL policy long-CoT-kk it is claimed to match, are capped at 4096 tokens. Since the reward function in Section D.2 assigns zero reward to responses exceeding the length limit, a policy evaluated at 8K can receive credit for longer, slower reasoning that the same policy at 4K would lose. The apparent tie in Table 2 (0.820 vs 0.820) and the text's 78.8% vs 82.0% therefore do not establish that re-distillation matches RL performance. Please re-evaluate both models under identical max-token budgets, at minimum at 4K and 8K, and report the results with confidence intervals. This is load-bearing for the paper's central empirical claim.
  2. [Section 5.3 and Eq. (43)-(50)] The theoretical explanation is partly definitional. The sample effect V(a,s,θ) is defined so that the approximated RL growth rate is exactly an expectation of sample effects (Eq. 3), and Theorem 5.2 then reduces, after linearization and noise-neglect, to an application of E[X^2] >= E[X]^2. The paper itself flags in Limitations that 'linearized sample effect may not be able to fully explain the non-linear effect in SFT and RL' and lists this as the core assumption in Appendix C. Because the theorem's conclusion follows from the definition under that assumption, the analytical framework does not by itself provide independent evidence for the mechanism; the empirical re-distillation results are the main support. I therefore recommend that the paper either soften the causal language around Theorem 5.2 or provide a direct empirical test that distinguishes sample effect from other correlates, such as response length or simple correctness.
  3. [Section 6.1, Figure 3, and Table 2] The reported numbers for RD-rl-kk are inconsistent across the text and table. The text says 'RD-rl-kk is close to long-CoT-kk through SFT alone (78.8% vs 82.0%)', while Table 2 lists both RD-rl-kk and long-CoT-kk as 0.820. Similarly, Figure 3 appears to show step-0 SFT-only accuracy for RD-rl-kk, but the table appears to report a later or averaged value. Please clarify exactly which checkpoint or evaluation each number refers to and ensure the '1K SFT samples surpass DeepSeek-V3-0324' claim (0.820 vs 0.808) is based on the same protocol for both models. Without this clarification and without confidence intervals for the table averages, the headline result cannot be reliably assessed.
minor comments (5)
  1. [Section A] RD-transfer-gym is described as fine-tuning 'Llama-3.2-1.5B Instruct', but elsewhere the Llama experiments use Llama-3.2-3B; please correct this typo.
  2. [Section E] The sentence beginning 'For all Llama-3.2-3B based models...' is incomplete and grammatically garbled; it should specify the max response token setting for those models.
  3. [Section F and Figure 6] The interpolation study would be clearer if the direction of λ were stated consistently in the caption and text; currently the caption says 'interpolating from long-CoT-math initial policy to the 50-step checkpoint', while the text defines π(λθold + (1−λ)θ), which for λ=1 gives the initial policy and for λ=0 gives the checkpoint.
  4. [Table 2] The footnote '†: We use Lmax = 8192 in these models' evaluation' does not specify whether it applies to RD-rl-baseline-kk as well as RD-rl-kk; please clarify the evaluation budget of every row.
  5. [Title page] There is a typographical error in the corresponding author email ('wuji ee@tsinghua.edu.cn'); please fix the spacing.

Circularity Check

1 steps flagged · score 4.0 of 10

Theoretical explanation of re-distillation is partly self-definitional (SFT growth rate is defined as expected sample effect), but the central empirical matching result is independently tested; the K&K comparison is confounded by unequal evaluation token budgets.

  1. self definitional [Section 5.2, Definition 5.1, Theorem 5.1 and Eqs. (3)-(5)]
    "By introducing sample effect, the approximated growth rate ˆµ(t, X) can be written as a weighted average of sample effect in Equation 3. ... Theorem 5.1. ... For any a1, a2 and s ∈ {Dt} that satisfies r(a1, s) = r(a2, s) = 1, we have: ln π∗θ(a1,s)/π∗θ(a2,s) = 1/βp∗θ(s) (V (a1, s, θ) − V (a2, s, θ)) + ln πθ(a1,s)/πθ(a2,s)."

    Equation 4 defines the SFT growth rate as the expected sample effect, E[V r]. The objective in Theorem 5.1 is exactly that same expected sample effect (reweighted by π*/p*πθ) minus a KL term. Maximizing an objective that is literally a weighted sum of sample effects must, by construction, assign more probability mass to high-effect samples; the theorem's conclusion is contained in the definition of the objective. The paper presents this as an explanatory discovery ('samples with higher effect will obtain more probability mass') and then uses it to justify re-distillation, but no independent empirical content is added at this step.

full rationale

The central empirical claim—that SFT on re-distilled RL-policy samples matches RL performance with far fewer samples—is tested against external baselines (DeepSeek-R1, DeepSeek-V3-0324, Logic-RL-7B, Qwen/Instruct baselines) and against the RD-rl-baseline control, so it does not reduce to a fit or to the definition of sample effect. The theoretical explanation, however, is partially self-definitional: Equation 4 defines the SFT growth rate as the expected sample effect, and Theorem 5.1 maximizes exactly this quantity (reweighted, minus KL). Consequently, the conclusion that optimal distillation data have high sample effect is contained in the objective's definition, and the later explanation of re-distillation's success rests on this analytic step rather than on an independent empirical test. The paper openly labels the analysis 'hypothetical' and its core assumption is stated in Appendix C, which softens but does not eliminate the definitional reduction. The K&K flagship comparison also suffers an evaluation-protocol confound: Table 2 footnotes give RD-rl-kk 8K response tokens while the compared RL policy long-CoT-kk is evaluated at 4K; this threatens the empirical matching claim but is a protocol confound, not circularity under the stated rubric. The self-citation to Wang et al. (2025), whose author list overlaps with the present paper via J. Gao, is background and not load-bearing. Overall score 4: one explanatory 'prediction' reduces by construction, while the main empirical result retains independent content.

Assumptions & free parameters 2 free parameters · 3 assumptions · 2 invented entities

The empirical re-distillation method relies on a trained RL policy and manual dataset construction choices. The theoretical framework introduces two new quantities, sample effect and dataset effect, both defined in terms of model gradients. The key axioms are the linearization assumption and the equal-distribution assumption in Theorem 5.2.

free parameters (2)
  • Adam beta1 = 0.5
    Set after ablation in Section D.2; all RL runs use this custom value, and the re-distillation results depend on the RL policy quality.
  • Task sampling proportions for RD-qwen-gym and RD-llama-gym = listed percentages in Appendix A
    Manually chosen per task to create re-distillation datasets; these proportions affect which tasks contribute samples and thus the re-distilled model's accuracy.
assumptions (3)
  • ad hoc to paper Linearized sample effect explains multi-step, non-linear LLM training dynamics.
    Stated as the core assumption in Appendix C; the entire theoretical framework depends on it.
  • domain assumption Small learning rate, large batch, and noise term negligible in initial steps justify the SDE drift approximation.
    Appendix C assumptions for Equation 2; standard simplifications but unverified for real LLM training.
  • ad hoc to paper For Theorem 5.2, Dt = De and theta* = theta (training set same distribution as evaluation, source and target model identical).
    The proof's lower bound requires these equalities; the paper acknowledges they hold only approximately in practice (Section 5.3).
invented entities (2)
  • sample effect V(a,s,theta) independent evidence
    purpose: Quantifies the contribution of a training sample to the growth rate of test accuracy; used to rank data and justify re-distillation.
    Measurable from model gradients and parameter deltas; the paper computes it in Section F. However, the empirical test failed to order long-CoT vs short-CoT, so independent evidence is partial.
  • dataset effect V(theta,theta*,Dt)
    purpose: Aggregate sample effect of a filtered target policy; used in Theorem 5.2 to argue RL improves distillation data.
    Derived from sample effect and not directly measured; no external falsifiable handle beyond the sample effect itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning." pith.science (2026). https://pith.science/paper/WTCXU5CR

@misc{pith2026250517988,
  author       = {Pith},
  title        = {Pith review of: Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WTCXU5CR}},
  note         = {Machine review of arXiv:2505.17988}
}
read the original abstract

R1-style Reinforcement Learning (RL) significantly enhances Large Language Models' reasoning capabilities, yet the mechanism behind rule-based RL remains unclear. We found that small-scale SFT has substantial influence on RL but shows poor efficiency. To explain our observations, we propose an analytical framework and compare the efficiency of SFT and RL by measuring \textbf{sample effect}. Our hypothetical analysis shows the potential to improve SFT efficiency. Guided by our analysis, we propose \textbf{Re-distillation}, a technique that aims to boost the effectiveness of small-scale distillation by sampling from the RL-trained policy. Re-distillation shows consistent surprising efficiency on three datasets and both Qwen\&Llama models: Re-distilled models matched RL performance with far fewer samples and less computation. As a result, on K\&K dataset, our re-distilled Qwen-2.5-1.5B model surpasses DeepSeek-V3-0324 with only 1K SFT samples. We demonstrate that re-distillation can be used to efficiently balance multiple goals in RL. Our work explains several interesting phenomena in R1-style RL, shedding light on the mechanisms behind its empirical success. Code is available at: https://github.com/on1262/deep-reasoning.

Figures

Figures reproduced from arXiv: 2505.17988 by the authors.

Figure 1
Figure 1. Main experiments overview. Left (A): We investigate which method produces better samples for small-scale SFT be￾fore RL. Pipeline(A) shows samples distilled from RL-trained policy lead to the fastest convergence in RL stage. Common approaches, such as distilling from stronger model or using synthetic solutions, failed to maximize RL efficiency as depicted in (C). Left (B): It is possible to reproduce RL-trained poli… view at source ↗
Figure 2
Figure 2. Small-scale SFT affects RL convergence (Left: K&K dataset, Right: MATH dataset). The RL test accuracy shows that small-scale SFT has substantial influence but lacks of clear pattern. For example, although long-CoT-kk demonstrates superior performance to the other models, long-CoT-math is slightly slower than base-no-sft-math. Shaded area: 95% Confidence Interval. though the pattern remains unclear: long-CoT-kk achie… view at source ↗
Figure 3
Figure 3. Re-distillation matches RL performance through SFT alone. (Left: K&K dataset, Right: MATH dataset) At step 0, RD-sft-math matches Qwen-2.5-Instruct by only SFT on 496 samples. RD-rl-kk approaches optimal performance without RL. Re-distillation also boost RL efficiency, as RD-sft-kk converges significantly faster than long-CoT-kk. Shaded area: 95% Confidence Interval [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Re-distillation is generalizable on variant tasks and models (Left: Qwen, Right: Llama): RL from Instruct models gains substantial improvement(Instruct to rl-). The re-distilled model from RL-trained policy retains most perfor￾mance improvement(RD- vs inst-). However, …
Figure 5
Figure 5. Figure 5: Test accuracy for hyper-parameter ablation study: [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Left: Test accuracy smoothly changes when interpolating from long-CoT-math initial policy to the 50-step check￾point. Middle: Average sample effect(dashed green line: SGD effect, bar: Adam effect, error bar: 95% CI) of different SFT dataset measured on Qwen-2.5-1.5B ba…
Figure 7
Figure 7. Figure 7: Divergent Exploration Patterns in RL and SFT. Left: Position-wise logprobs from long-CoT-math RL rollouts evaluated by its initial policy. Right: Logprob distributions before/after SFT on identical data, evaluated by SFTed model of long-CoT-math. For contrast, we exami…
Figure 8
Figure 8. Figure 8: Left: Increasing temperature will not bring substantial improvement. Two models share the same GRPO recipe except different rollout temperatures. Right: Increasing temperature leads to significantly higher entropy in initial steps. H Details in Empirical Verification W…
Figure 9
Figure 9. Figure 9: Statistics of K&K experiments. Top Left: Test accuracy in RL. Top Middle: Train accuracy in RL. Top Right: Train￾ing response length(token) in RL. Bottom Left: Training entropy in RL. Bottom Middle: Training clip ratio(exceeding max response length in RL). Bottom Right…
Figure 10
Figure 10. Figure 10: Statistics of MATH experiments. Top Left: Test accuracy in RL. Top Middle: Train accuracy in RL. Top Right: Training response length(token) in RL. Bottom Left: Training entropy in RL. Bottom Middle: Training clip ratio(exceeding max response length in RL). Bottom Righ…
Figure 11
Figure 11. Figure 11: RL stage test accuracy for rl-llama-gym and rl-qwen-gym [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Statistics of REASONING GYM experiments. Left side of black dash line is in-domain task; Right side is out-of [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance

    cs.AI 2025-08 conditional novelty 6.0 of 10

    EAG-RL improves LLM performance on EHR mortality and readmission prediction by training on expert-generated reasoning traces and an attention-alignment RL reward.

Reference graph

Works this paper leans on

39 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    On exact computation with an infinitely wide neural net

    Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019

    2019

  2. [2]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics , pages 4447--4455. PMLR, 2024

    work page 2024

  3. [3]

    Quantifying memorization across neural language models

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations , 2022

    work page 2022

  4. [4]

    The Hyperfitting Phenomenon : Sharpening and Stabilizing LLMs for Open - Ended Text Generation

    Fredrik Carlsson, Fangyu Liu, Daniel Ward, Murathan Kurfali, and Joakim Nivre. The Hyperfitting Phenomenon : Sharpening and Stabilizing LLMs for Open - Ended Text Generation . arXiv preprint arXiv:2412.04318, 2024

    arXiv 2024

  5. [5]

    Generative AI for Math : Abel , 2023

    Ethan Chern, Haoyang Zou, Xuefeng Li, Jiewen Hu, Kehua Feng, Junlong Li, and Pengfei Liu. Generative AI for Math : Abel , 2023. URL https://github.com/GAIR-NLP/abel. Publication Title: GitHub repository

    work page 2023

  6. [6]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025

    arXiv 2025

  7. [7]

    Kto: Model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024

    arXiv 2024

  8. [8]

    Open R1 : A fully open reproduction of DeepSeek - R1 , January 2025

    Hugging Face. Open R1 : A fully open reproduction of DeepSeek - R1 , January 2025. URL https://github.com/huggingface/open-r1

    work page 2025

Show all 39 references
  1. [9]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307, 2025

    2025 arXiv

  2. [10]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and others . The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and others . Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  4. [12]

    SoK : Memorization in General - Purpose Large Language Models , 2023

    Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. SoK : Memorization in General - Purpose Large Language Models , 2023. URL https://arxiv.org/abs/2310.18362. \_eprint: 2310.18362

    2023 arXiv

  5. [13]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  6. [14]

    Reinforce++: A simple and efficient approach for aligning large language models

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  7. [15]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018

    2018

  8. [16]

    Towards efficient exact optimization of language model alignment

    Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient exact optimization of language model alignment. arXiv preprint arXiv:2402.00856, 2024

    2024 arXiv

  9. [17]

    Kimi k1.5: Scaling reinforcement learning with llms

    Kimi Team , Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and others . Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  10. [18]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  11. [19]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles , ...

    2023

  12. [20]

    NuminaMath , 2024

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. NuminaMath , 2024. Publication Title: Hugging Face r...

    2024

  13. [21]

    Limr: Less is more for rl scaling

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886, 2025

    2025 arXiv

  14. [22]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  15. [23]

    OpenAI , Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett,...

    2024 arXiv

  16. [24]

    Competitive Programming with Large Reasoning Models , 2025

    OpenAI , Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contribut...

    2025 arXiv

  17. [25]

    QwQ - 32B : Embracing the Power of Reinforcement Learning , March 2025

    Qwen Team . QwQ - 32B : Embracing the Power of Reinforcement Learning , March 2025. URL https://qwenlm.github.io/blog/qwq-32b/

    2025

  18. [26]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023

    2023

  19. [27]

    Learning dynamics of llm finetuning

    Yi Ren and Danica J Sutherland. Learning dynamics of llm finetuning. arXiv preprint arXiv:2407.10490, 2024

    2024 arXiv

  20. [28]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  21. [29]

    Policy Gradient Methods for Reinforcement Learning with Function Approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation . In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems , volume 12. MIT Press, 1999...

    1999

  22. [30]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and others . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  23. [31]

    On memorization of large language models in logical reasoning

    Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024

    2024 arXiv

  24. [32]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, 2025

    2025 arXiv

  25. [33]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and others . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024 a

    2024 arXiv

  26. [34]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and others . Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024 b

    2024 arXiv

  27. [35]

    LIMO : Less is More for Reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. LIMO : Less is More for Reasoning . arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  28. [36]

    Demystifying long chain-of-thought reasoning in llms, 2025

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms, 2025. URL https://arxiv. org/pdf/2502.03373, 2025

    2025 arXiv

  29. [37]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, and others . Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  30. [38]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892, 2025

    2025 arXiv

  31. [39]

    Echo chamber: Rl post-training amplifies behaviors learned in pretraining

    Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: Rl post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.