Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A small model’s RL-induced policy shift can improve a stronger student when used as a dense on-policy reward, without imitating the weak final policy or re-running sparse-reward RL on the target.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 07:01 UTC pith:RL5QHOM3

load-bearing objection Clean weak-to-strong post-training idea: transfer the small model’s RL log-ratio as a dense on-policy reward, not its final policy—and the AIME numbers back it up. the 3 major comments →

arxiv 2607.05394 v2 pith:RL5QHOM3 submitted 2026-07-06 cs.LG cs.AIcs.CL

Weak-to-Strong Generalization via Direct On-Policy Distillation

Shiyuan Feng , Huan-ang Gao , Haohan Chi , Hanlin Wu , Zhilong Zhang , Zheng Jiang , Bingxiang He , Wei-Ying Ma
show 2 more authors
Ya-Qin Zhang Hao Zhou
This is my paper
classification cs.LG cs.AIcs.CL
keywords weak-to-strong generalizationon-policy distillationreinforcement learningpolicy shiftimplicit rewardlanguage model reasoningknowledge distillationRLVR
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the costly loop of reinforcement learning with verifiable rewards need not be repeated on every new strong reasoning model. Run that RL where rollouts are cheap—on a smaller model—then transfer only the change RL induced, not the small model’s final answers. Direct On-Policy Distillation reads the log-ratio between the post-RL and pre-RL small checkpoints as a dense implicit reward and applies it on the stronger student’s own sampled states, while a KL term anchors the student to its own start. Empirically the method improves students that already beat the post-RL teacher, lifts a 1.7B model from 48.3% to 58.3% on AIME 2024 in about four hours on eight A100 GPUs, beats step-matched direct RL on a larger target, and can stack successive policy shifts. A sympathetic reader cares because post-training cost itself is becoming a scale bottleneck, and the work reframes finished RL runs as reusable improvement signals rather than finished models to copy.

Core claim

The paper’s central claim is that the RL-induced policy shift of a weak teacher pair—log π_T minus log π_Tref—acts as a transferable dense reward: applied on a stronger student’s own on-policy states under a KL anchor to the student initialization, it improves that student without imitating the weak final policy and without running sparse-reward RL on the target. The same shift can raise students that already exceed the post-RL teacher, outperform matched-step direct RL on the large model in accuracy and wall-clock, and be composed sequentially across independent teacher pairs.

What carries the argument

Direct-OPD: the teacher policy shift Δ_T(y|x) = log π_T(y|x) − log π_Tref(y|x), which under KL-regularized RL recovers the teacher’s implicit reward up to scale and a per-prompt constant, turned into a top-k analytical per-token policy gradient on student-visited prefixes plus adaptive KL to the student start.

Load-bearing premise

The method needs the small teacher’s before-versus-after log-ratio to stay a trustworthy improvement direction on the states the stronger student actually visits under short training rollouts and a workable KL setting.

What would settle it

Swap the post-RL teacher and pre-RL reference in the log-ratio and re-run the same student training: if accuracy still rises instead of falling, the claimed directionality of the policy-shift reward is false. Separately, force student rollouts into prefixes where both teacher checkpoints assign near-zero probability and check whether AIME gains collapse while the dense reward magnitude grows.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Small-model RL can serve as a cheap generator of transferable improvement directions for larger models, not only as a way to ship small models.
  • Under a fixed RL-step budget, RL on a small model plus short Direct-OPD transfer can beat direct RL on the large target in both accuracy and wall-clock time.
  • Independently trained policy shifts can be applied in sequence to accumulate gains in one student.
  • Useful transfer does not require high teacher–student top-k token overlap or matching thinking patterns.
  • Response length and KL strength control whether the dense reward stays informative rather than becoming large off-support noise.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism holds, a single small-model RL campaign could be amortized across many later larger models without re-running sparse-reward RL on each.
  • The same checkpoint-pair log-ratio might transfer non-math behaviors shaped by KL-regularized RL (format, tool use, safety) when evaluated on the student’s own states.
  • A sharp stress test is transfer when teacher and student come from fully disjoint pretraining families, not only different sizes within related lines.
  • Driving batch-mean dense reward toward zero with adaptive KL may be a general on-distribution diagnostic for any implicit-reward distillation, not only this recipe.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Direct On-Policy Distillation (Direct-OPD): after running RLVR on a small teacher, it treats the log-ratio Δ_T = log π_T − log π_Tref (Eqs. 5–7) as a dense implicit reward and optimizes a stronger student against that signal on the student’s own on-policy top-k states, with a KL anchor to the student initialization (Eqs. 8–15) and an optional sign-driven adaptive KL controller (Eq. 16). The central claim is that this transfers the RL-induced policy shift rather than the weak teacher’s final policy, so students that already match or exceed the post-RL teacher still improve, and the route can beat step-matched direct RL on the large target at lower compute. Empirically, two teacher pairs (JustRL and QuestA) improve R1-Distill-7B, Qwen3-1.7B, and Qwen3-4B on AIME 2024/2025 (Table 1, Fig. 2), including sequential composition (Fig. 4) and wall-clock comparisons to direct RL (Fig. 3); analyses address top-k overlap, short-horizon generalization, and KL reliability (Secs. 4.1–4.3).

Significance. If the result holds, the paper offers a practical weak-to-strong post-training path that reuses small-model RL as a transferable dense reward instead of re-running sparse-reward RL on every new large model. The DPO-style reverse use of the policy-as-reward identity is standard but cleanly applied; the empirical package is substantial (two teacher pipelines, three students, sequential composition, wall-clock vs direct RL, and ablations on overlap, horizon, and KL). Strengths include explicit failure of vanilla OPD when the student already exceeds the teacher (Fig. 1a), consistent gains above the post-RL teacher, and a clear compute argument (e.g., Qwen3-1.7B 48.3 o58.3 AIME24 in ~4h on 8 A100s). That combination is of real interest for scalable post-training of reasoning models.

major comments (3)
  1. [Sections 4.2–4.3, Figs. 7–9] Secs. 4.2–4.3 and Figs. 7–9: the load-bearing assumption is that Δ_T evaluated on student top-k tokens under ~2k-token rollouts remains a reliable improvement direction for long AIME generations, even when top-k overlap stays low and the student already exceeds the teacher. The paper itself shows that 6k horizons move the cumulative gap G_T further yet validate worse (Fig. 8), and that fixed KL is pair-dependent while adaptive KL only pulls mean dense reward toward zero (Fig. 9). That documents sensitivity, not robustness. A targeted stress test is needed: e.g., (i) late-prefix or off-support corruption of the teacher/reference log-ratio with held-out AIME, and/or (ii) transfer of a math teacher shift to a non-math or deliberately mismatched prompt set. Without isolating when early-prefix Δ_T becomes large but untrustworthy, the central reliability claim remains incompletely supported.
  2. [Section 3.2, Figure 3, Appendix Tables 3–4] Sec. 3.2 / Fig. 3: the claim that weak-to-strong beats direct RL under matched steps and wall-clock is important but under-specified. Direct R1-Distill-7B RL and the small-teacher RL runs use the GRPO settings in Appendix Table 4 (including KL coefficient 0.0), while Direct-OPD uses a different data/prompt template (Skywork-OR1 math + DAPO-style prompt) and a short 2k horizon. Please report matched data/prompt for the direct-RL baseline, variance over seeds or multiple teacher checkpoints, and a clearer accounting of total FLOPs/GPU-hours so the compute advantage is not confounded by recipe differences.
  3. [Section 2.3, Equations 10–15] Sec. 2.3, Eqs. 10–15: the idealized sequence-level objective is replaced by a zero-discount token-level top-k surrogate with stop-gradient weights A^w_t(v). This is a substantial approximation (no future return, restricted action set, detached student probabilities). The manuscript should either (a) give a short argument or small controlled experiment that the surrogate preserves the ranking induced by Δ_T on student states, or (b) clearly label the gap between the sequence-level optimum (Eq. 9) and the implemented gradient (Eq. 15) as an empirical design choice rather than a derived equivalence.
minor comments (5)
  1. [Figure 1, Table 1] Fig. 1 caption and main text: teacher ref is given as 28.5 while post-RL JustRL is 51.3; ensure all panels and Table 1 use the same evaluation protocol (ave@32, temperature, max length) so the weak-to-strong comparison is unambiguous.
  2. [Section 2.4, Equation 16] Eq. 16: the adaptive KL rule uses sgn of the batch mean dense reward with fixed ε=0.01 and clip bounds; a one-sentence motivation for the sign-driven (vs target-KL) form would help readers who know standard adaptive KL controllers.
  3. [Appendix A] Appendix A notes that the Direct-OPD prompt differs from the teacher RL prompt; a short ablation or statement that transfer is robust to this mismatch would reduce a minor confound.
  4. [Section 5] Related work is thorough; a brief explicit contrast with weak-to-strong preference optimization [93] and proxy tuning [105] in one paragraph would sharpen the novelty claim without lengthening the section much.
  5. [Throughout] Typos / polish: “Direct On-Policy Distillation” is sometimes hyphenated inconsistently; “R1-Distill” vs “R1-7B” labels in figures should be unified for readability.

Circularity Check

0 steps flagged

No circularity: Direct-OPD applies a fixed teacher-pair log-ratio as an on-policy reward and measures gains on external AIME benchmarks; the DPO-style identity is a standard derivation, not a fit renamed as prediction.

full rationale

The load-bearing chain is: (i) under KL-regularized RL, log(π*/π_ref) recovers the reward up to scale and a per-prompt constant (Eqs. 6–7, standard DPO identity used in reverse); (ii) Direct-OPD therefore treats Δ_T = log π_T − log π_Tref as a dense token-level reward on the student’s own on-policy top-k states, with a KL anchor to the student init (Eqs. 8–15); (iii) success is claimed only via post-training accuracy on AIME 2024/2025 (Table 1, Figs. 2–4), not by algebraic identity with the inputs. The teacher checkpoints are fixed before student training; nothing in the objective forces AIME gains—vanilla OPD on the same post-RL teacher can degrade a stronger student (Fig. 1a), which would be impossible if the claimed improvement were true by construction. Adaptive KL (Eq. 16) and response-length choices are ordinary hyperparameter control validated on held-out curves (Secs. 4.2–4.3), not parameters fitted to the reported accuracy and then re-presented as predictions. Self-citations (JustRL, OPD phenomenology, related RLVR) supply experimental teachers or background; none is a uniqueness theorem that forbids alternatives or makes the AIME lifts tautological. The paper is self-contained against external benchmarks; score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on the standard KL-regularized policy-as-reward identity plus several engineering choices (top-k support, short training horizon, adaptive KL bounds) that control whether the transferred log-ratio stays informative on student states. No new physical entities; free parameters are training hyperparameters that the paper shows are pair-dependent.

free parameters (5)
  • KL coefficient α (fixed or adaptive bounds)
    α scales the student KL anchor relative to the unobservable teacher reward scale; defaults and clip range [0.5, 2.5] (or pair-specific fixed values) are chosen by validation, and Section 4.3 shows best fixed α is pair-dependent.
  • Adaptive KL step size ε
    Default ε=0.01 multiplies α by (1±ε) from the sign of batch mean dense reward; a hand-set controller gain not derived from theory.
  • Student top-k support size
    Default k=16 restricts the dense reward and gradient to student top-k tokens; changes the effective action space of the surrogate.
  • Training response length (horizon)
    Default 2k tokens; Section 4.2 shows 512/2k/4k/6k change validation, so the reported gains depend on this choice.
  • Learning rate and training steps
    lr 1e-6 and ~300 Direct-OPD steps (plus teacher RL step selection T300–T1500) are fixed experimental knobs that select which policy shift is transferred.
axioms (4)
  • standard math Under KL-regularized RL, log(π*/π_ref) recovers the reward up to positive scale and a per-prompt constant (policy-as-reward / DPO identity).
    Invoked in Section 2.2 Eqs. 6–7 to interpret Δ_T as an implicit reward; standard in preference optimization literature.
  • ad hoc to paper A zero-discount token-level top-k surrogate with stop-gradient weights is an adequate practical stand-in for the sequence-level Direct-OPD objective.
    Sections 2.3–2.4 replace sequence returns and Monte Carlo token gradients with immediate r_t and Rao–Blackwellized top-k weights without a formal approximation bound.
  • domain assumption RL-induced shifts from small math teachers remain directionally useful on stronger students’ on-policy states, including cross-family and cross-thinking-pattern transfers.
    Load-bearing for weak-to-strong claims in Sections 3.1–3.2 and analysis 4.1; empirically supported in the paper but not guaranteed in general domains.
  • domain assumption Verifiable outcome rewards used to train the small teachers encode transferable reasoning improvements rather than only teacher-specific artifacts.
    Background of the whole RLVR setup; if teacher RL mainly exploits small-model quirks, Δ_T would not help stronger students.
invented entities (2)
  • Direct-OPD objective (policy-shift dense reward on student on-policy states) independent evidence
    purpose: Defines the training signal that transfers only RL-induced change rather than the weak teacher’s absolute policy.
    New method object of the paper; evaluated empirically, not an unobserved physical entity.
  • Sign-driven adaptive KL controller for Direct-OPD dense reward no independent evidence
    purpose: Adjusts α from the sign of batch mean student-weighted shift to keep the dense reward in a balanced regime.
    Paper-specific controller (Eq. 16); independent evidence is only the reported training curves, not an external law.

pith-pipeline@v1.1.0-grok45 · 27139 in / 3809 out tokens · 32685 ms · 2026-07-11T07:01:00.230633+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

    cs.LG 2026-07 conditional novelty 5.0

    Relative policy-improvement signals from a weak proxy model, after simple calibration, can be transferred to improve stronger primary LLMs without re-exploring on the primary.

Reference graph

Works this paper leans on

105 extracted references · 65 linked inside Pith · cited by 1 Pith paper

  1. [1]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Nature, 645:633–638, 2025

    2025

  2. [2]

    JustRL: Scaling a 1.5B LLM with a simple RL recipe

    Bingxiang He, Zekai Qu, Zeyuan Liu, Yinghao Chen, Yuxin Zuo, Cheng Qian, Kaiyan Zhang, Weize Chen, Chaojun Xiao, Ganqu Cui, Ning Ding, and Zhiyuan Liu. JustRL: Scaling a 1.5B LLM with a simple RL recipe. arXiv preprint arXiv:2512.16649, 2025

    arXiv 2025

  3. [3]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    POLARIS: A post-training recipe for scaling reinforcement learning on reasoning models

    Chenxin An et al. POLARIS: A post-training recipe for scaling reinforcement learning on reasoning models. https://github.com/ChenxinAn-fdu/POLARIS, 2025

    2025

  5. [5]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.arXiv preprint arXiv:2305.18290, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    QuestA: Expanding reasoning capacity in LLMs via question augmentation.arXiv preprint arXiv:2507.13266, 2025

    Jiazheng Li, Hongzhou Lin, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Yi Wu, and Jingzhao Zhang. QuestA: Expanding reasoning capacity in LLMs via question augmentation.arXiv preprint arXiv:2507.13266, 2025

    arXiv 2025

  8. [8]

    Lyng, Sanjit Singh Batra, and Robert E

    Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D. Lyng, Sanjit Singh Batra, and Robert E. Tillman. Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026, pages 25553–25569, 2026

    2026

  9. [9]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  11. [11]

    Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    QwQ-32B: Embracing the power of reinforcement learning.https://qwenlm.github.io/blog/qw q-32b/, 2025

    Qwen Team. QwQ-32B: Embracing the power of reinforcement learning.https://qwenlm.github.io/blog/qw q-32b/, 2025

    2025

  13. [13]

    Open-reasoner- zero: An open source approach to scaling reinforcement learning on the base model.https://github.com/Ope n-Reasoner-Zero/Open-Reasoner-Zero, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner- zero: An open source approach to scaling reinforcement learning on the base model.https://github.com/Ope n-Reasoner-Zero/Open-Reasoner-Zero, 2025

    2025

  14. [14]

    DAPO: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Skywork open reasoner 1 technical report.arXiv preprint arXiv:2505.22312, 2025

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner 1 technical report.arXiv preprint arXiv:2505.22312, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    How far can unsupervised rlvr scale llm training?arXiv preprintarXiv:2603.08660, 2026

    Bingxiang He, Yuxin Zuo, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang, Yuchen Fan, Ganqu Cui, et al. How far can unsupervised rlvr scale llm training?arXiv preprintarXiv:2603.08660, 2026

    arXiv 2026

  18. [18]

    DeepSeek V4 preview release.https://api-docs.deepseek.com/news/news260424, 2026

    DeepSeek-AI. DeepSeek V4 preview release.https://api-docs.deepseek.com/news/news260424, 2026

    2026

  19. [19]

    GLM-5.2: Built for long-horizon tasks.https://z.ai/blog/glm-5.2, 2026

    Z.ai. GLM-5.2: Built for long-horizon tasks.https://z.ai/blog/glm-5.2, 2026. 16

    2026

  20. [20]

    AIMO-2 winning solution: Building state-of-the-art mathematical reasoning models with OpenMathReasoning dataset

    Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. AIMO-2 winning solution: Building state-of-the-art mathematical reasoning models with OpenMathReasoning dataset. arXiv preprint arXiv:2504.16891, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178, 2025

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. 2024

    2024

  23. [23]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  24. [24]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of EMNLP, 2016

    2016

  25. [25]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Pith/arXiv arXiv 1910

  26. [26]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. 2020

    2020

  27. [27]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. 2020

    2020

  28. [28]

    Improved knowledge distillation via teacher assistant

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. 2020

    2020

  29. [29]

    On the efficacy of knowledge distillation

    Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. 2019

    2019

  30. [30]

    Distillation scaling laws.arXiv preprint arXiv:2502.08606, 2025

    Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws.arXiv preprint arXiv:2502.08606, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    MiniLLM: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    f-Divergence Minimization for Sequence-Level Knowledge Distillation

    Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-Divergence Minimization for Sequence-Level Knowledge Distillation. Proceedings of ACL, 2023. URLhttps://arxiv.org/abs/2307.15190

    Pith/arXiv arXiv 2023

  33. [33]

    Revisiting Knowledge Distillation for Autoregressive Language Models.Proceedings of ACL, 2024

    Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao. Revisiting Knowledge Distillation for Autoregressive Language Models.Proceedings of ACL, 2024. URLhttps://arxiv.org/abs/2402.11890

    Pith/arXiv arXiv 2024

  34. [34]

    Rethinking Kullback- Leibler Divergence in Knowledge Distillation for Large Language Models.Proceedings of COLING, 2025

    Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. Rethinking Kullback- Leibler Divergence in Knowledge Distillation for Large Language Models.Proceedings of COLING, 2025. URL https://arxiv.org/abs/2404.02657

    Pith/arXiv arXiv 2025

  35. [35]

    DistiLLM: Towards Streamlined Distillation for Large Language Models.Proceedings of ICML, 2024

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. DistiLLM: Towards Streamlined Distillation for Large Language Models.Proceedings of ICML, 2024. URLhttps://arxiv.org/abs/2402.03898

    Pith/arXiv arXiv 2024

  36. [36]

    DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

    Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. InProceedings of ICML, 2025. URL https://arxiv.org/abs/2503.07067

    Pith/arXiv arXiv 2025

  37. [37]

    MiniPLM: Knowledge Distillation for Pre-Training Language Models.Proceedings of ICLR, 2025

    Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. MiniPLM: Knowledge Distillation for Pre-Training Language Models.Proceedings of ICLR, 2025. URLhttps://arxiv.org/abs/2410.17215

    Pith/arXiv arXiv 2025

  38. [38]

    DistillSpec: Improving Speculative Decoding via Knowledge Distillation

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. DistillSpec: Improving Speculative Decoding via Knowledge Distillation. Proceedings of ICLR, 2024. URLhttps://arxiv.org/abs/2310.08461

    Pith/arXiv arXiv 2024

  39. [39]

    Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister

    Wenda Xu, Rujun Han, Zifeng Wang, Long T. Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling.Proceedings of ICLR, 2025. URLhttps://arxiv.org/abs/2410.11325. 17

    Pith/arXiv arXiv 2025

  40. [40]

    On-policy distillation of language models: Learning from self-generated mistakes.arXiv preprint arXiv:2306.13649, 2023

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes.arXiv preprint arXiv:2306.13649, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    On-policy distillation

    Kevin Lu and Thinking Machines Lab. On-policy distillation. 2025

    2025

  42. [42]

    A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026

    Pith/arXiv arXiv 2026

  43. [43]

    Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562, 2026

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562, 2026

    Pith/arXiv arXiv 2026

  44. [44]

    Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079, 2026

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079, 2026

    Pith/arXiv arXiv 2026

  45. [45]

    Stable on-policy distillation through adaptive target reformulation.arXiv preprint arXiv:2601.07155, 2026

    Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation.arXiv preprint arXiv:2601.07155, 2026

    Pith/arXiv arXiv 2026

  46. [46]

    Unifying group-relative and self-distillation policy optimization via sample routing

    Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing. arXiv preprint arXiv:2604.02288, 2026

    arXiv 2026

  47. [47]

    On-policy context distillation for language models

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275, 2026

    Pith/arXiv arXiv 2026

  48. [48]

    Online experiential learning for language models

    Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Online experiential learning for language models. arXiv preprint arXiv:2603.16856, 2026

    Pith/arXiv arXiv 2026

  49. [49]

    Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Pith/arXiv arXiv 2026

  50. [50]

    Self-distillation for multi-token prediction, 2026.arXiv preprint arXiv:2603.23911, 2026

    Guoliang Zhao, Ruobing Xie, An Wang, Shuaipeng Li, Huaibing Xie, and Xingwu Sun. Self-distillation for multi-token prediction, 2026.arXiv preprint arXiv:2603.23911, 2026

    arXiv 2026

  51. [51]

    Reinforcement Learning via Self-Distillation

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement Learning via Self-Distillation. arXiv preprint arXiv:2601.20802, 2026. URLhttps://arxiv.org/abs/2601.20802

    Pith/arXiv arXiv 2026

  52. [52]

    Self-distillation enables continual learning

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026

    Pith/arXiv arXiv 2026

  53. [53]

    PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence.arXiv preprint arXiv:2603.11178, 2026

    Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence.arXiv preprint arXiv:2603.11178, 2026. URL https: //arxiv.org/abs/2603.11178

    Pith/arXiv arXiv 2026

  54. [54]

    Scaling reasoning efficiently via relaxed on-policy distillation.arXiv preprint arXiv:2603.11137, 2026

    Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation.arXiv preprint arXiv:2603.11137, 2026

    arXiv 2026

  55. [55]

    Why does self-distillation (sometimes) degrade the reasoning capability of llms?arXiv preprint arXiv:2603.24472, 2026

    Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms?arXiv preprint arXiv:2603.24472, 2026

    Pith/arXiv arXiv 2026

  56. [56]

    Black-Box On-Policy Distillation of Large Language Models.arXiv preprint arXiv:2511.10643, 2025

    Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. Black-Box On-Policy Distillation of Large Language Models.arXiv preprint arXiv:2511.10643, 2025. URLhttps://arxiv.org/abs/2511.10643

    arXiv 2025

  57. [57]

    Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026

    Pith/arXiv arXiv 2026

  58. [59]

    URLhttps://arxiv.org/abs/2505.16142

    arXiv

  59. [60]

    KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning.arXiv preprint arXiv:2506.02208, 2025

    Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, and Fei Mi. KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning.arXiv preprint arXiv:2506.02208, 2025. URLhttps://arxiv.org/abs/2506.02208. 18

    Pith/arXiv arXiv 2025

  60. [61]

    Reinforcement-aware Knowledge Distillation for LLM Reasoning.arXiv preprint arXiv:2602.22495, 2026

    Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto. Reinforcement-aware Knowledge Distillation for LLM Reasoning.arXiv preprint arXiv:2602.22495, 2026. URLhttps://arxiv.org/abs/2602.22495

    Pith/arXiv arXiv 2026

  61. [62]

    AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation

    Songming Zhang, Xue Zhang, Tong Zhang, Bojie Hu, Yufeng Chen, and Jinan Xu. AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation. In Proceedings of ACL, 2025. URL https: //arxiv.org/abs/2503.02832

    Pith/arXiv arXiv 2025

  62. [63]

    Self-distilled rlvr.arXiv preprint arXiv:2604.03128, 2026

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr.arXiv preprint arXiv:2604.03128, 2026

    Pith/arXiv arXiv 2026

  63. [64]

    Small models struggle to learn from strong reasoners

    Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. 2025

    2025

  64. [65]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    Pith/arXiv arXiv 2026

  65. [66]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023

    Pith/arXiv arXiv 2023

  66. [67]

    Introducing Superalignment.OpenAI Blog, 2023

    Jan Leike and Ilya Sutskever. Introducing Superalignment.OpenAI Blog, 2023

    2023

  67. [68]

    Semi-supervised learning by entropy minimization.Advances in neural information processing systems, 17, 2004

    Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization.Advances in neural information processing systems, 17, 2004

    2004

  68. [69]

    Semi-supervised sequence learning.Advances in neural information processing systems, 28, 2015

    Andrew M Dai and Quoc V Le. Semi-supervised sequence learning.Advances in neural information processing systems, 28, 2015

    2015

  69. [70]

    Temporal ensembling for semi-supervised learning.arXiv preprintarXiv:1610.02242, 2016

    Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning.arXiv preprintarXiv:1610.02242, 2016

    Pith/arXiv arXiv 2016

  70. [71]

    There are many consistent explanations of unlabeled data: Why you should average.arXiv preprint arXiv:1806.05594, 2018

    Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many consistent explanations of unlabeled data: Why you should average.arXiv preprint arXiv:1806.05594, 2018

    Pith/arXiv arXiv 2018

  71. [72]

    Co- teaching: Robust training of deep neural networks with extremely noisy labels.Advances in neural information processing systems, 31, 2018

    Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co- teaching: Robust training of deep neural networks with extremely noisy labels.Advances in neural information processing systems, 31, 2018

    2018

  72. [73]

    Mixmatch: A holistic approach to semi-supervised learning.Advancesin neural information processing systems, 32, 2019

    David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning.Advancesin neural information processing systems, 32, 2019

    2019

  73. [74]

    Dividemix: Learning with noisy labels as semi-supervised learning

    Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394, 2020

    Pith/arXiv arXiv 2002

  74. [75]

    Big self-supervised models are strong semi-supervised learners.Advancesin neural information processing systems, 33:22243–22255, 2020

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners.Advancesin neural information processing systems, 33:22243–22255, 2020

    2020

  75. [76]

    Self-training avoids using spurious features under domain shift

    Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. Advancesin Neural Information Processing Systems, 33:21061–21071, 2020

    2020

  76. [77]

    Debiased self-training for semi-supervised learning.Advances in Neural Information Processing Systems, 35:32424–32437, 2022

    Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning.Advances in Neural Information Processing Systems, 35:32424–32437, 2022

    2022

  77. [78]

    Supervising strong learners by amplifying weak experts

    Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018

    Pith/arXiv arXiv 2018

  78. [79]

    Scalable agent alignment via reward modeling: a research direction.arXiv preprint arXiv:1811.07871, 2018

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction.arXiv preprint arXiv:1811.07871, 2018

    Pith/arXiv arXiv 2018

  79. [80]

    AI safety via debate.arXiv preprint arXiv:1805.00899, 2018

    Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate.arXiv preprint arXiv:1805.00899, 2018. 19

    Pith/arXiv arXiv 2018

  80. [81]

    Measuring progress on scalable oversight for large language models

    Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022

    Pith/arXiv arXiv 2022

Showing first 80 references.