The reviewed record of science sign in
Pith

arxiv: 2607.05394 · v1 · pith:RL5QHOM3 · submitted 2026-07-06 · cs.LG · cs.AI· cs.CL

Weak-to-Strong Generalization via Direct On-Policy Distillation

Shiyuan Feng , Huan-ang Gao , Haohan Chi , Hanlin Wu , Zhilong Zhang , Zheng Jiang , Bingxiang He , Wei-Ying Ma
show 2 more authors
Ya-Qin Zhang Hao Zhou
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-07 12:22 UTCglm-5.2pith:RL5QHOM3record.jsonopen to challenge →

classification cs.LG cs.AIcs.CL
keywords modeldirect-opdtargetteacherdirectmodelson-policypolicy
0
0 comments X

The pith

Small model's RL checkpoint pair doubles as reward for bigger model

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes Direct On-Policy Distillation (Direct-OPD), a method that reuses reinforcement learning gains from a small, cheap-to-train model to improve a larger, stronger model. The key insight is that the log-ratio between a weak model's pre-RL and post-RL checkpoints constitutes a dense implicit reward signal encoding what the RL run actually learned, separate from the weak model's capacity limits. By evaluating this log-ratio on the stronger student's own sampled states and using it as a per-token reward within a KL-regularized objective, Direct-OPD transfers the improvement direction without forcing the student to imitate the weaker teacher's final policy. The method requires no explicit reward model and no sparse-reward RL on the target model. Empirically, a 1.5B-parameter teacher's RL shift lifts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in about 4 hours on 8 GPUs, outperforming step-matched direct RL on the larger model at a fraction of the compute, and the approach works even when the student already surpasses the post-RL teacher.

Core claim

The central discovery is that a pair of model checkpoints from before and after a reinforcement learning run stores the RL supervision signal directly in policy space: the log-ratio of the post-RL policy to the pre-RL reference policy is mathematically equivalent to the reward that trained the model (up to a positive scale and per-prompt constant), under the standard KL-regularized RL objective. This checkpoint-pair log-ratio can be decomposed into dense per-token rewards and applied to a different, stronger model's own on-policy states, transferring the RL-induced direction of improvement across model scales without imitating the weaker model's final distribution.

What carries the argument

The teacher policy shift ΔT(y|x) = log πT(y|x) − log πTref(y|x), which decomposes into per-token rewards rt(v) = log πT(v|st) − log πTref(v|st). This shift is evaluated on the student's top-k candidate tokens at each visited prefix, weighted by the student's own renormalized probabilities (with stop-gradient applied to the weights), and used as the advantage in a policy-gradient objective regularized by a KL term anchoring the student to its own initialization. An adaptive KL controller adjusts the regularization strength based on the batch-level sign of the student-weighted reward.

If this is right

  • Post-training pipelines could run expensive RL once on a small model and cheaply transfer the learned improvement direction to many larger model variants, reducing the compute cost of reasoning-model post-training by an order of magnitude or more.
  • Multiple independently trained small models could each contribute different RL-discovered capabilities (e.g., different math strategies, coding approaches) that are sequentially composed into a single strong student via Direct-OPD.
  • The policy-as-reward identity could be used to audit or interpret what a given RL run actually learned, by reading off the implicit reward function from the checkpoint pair.
  • The method suggests a decoupling of model scale from RL cost: the model that discovers the improvement direction need not be the model that benefits from it, opening a design space where small models serve as RL surrogates for large ones.

Load-bearing premise

The method assumes that the teacher's implicit reward signal, extracted from the log-ratio of its pre-RL and post-RL checkpoints, remains meaningful and reliable when evaluated on the stronger student's own sampled states. If the student drifts into regions where both teacher checkpoints assign low probability, the log-ratio becomes large in magnitude but noisy, and the dense reward degrades into unhelpful signal.

What would settle it

Find a teacher-student pair where the teacher's RL produces a clear improvement on the teacher's own distribution, the student's on-policy states have reasonable overlap with the teacher's support, and yet applying Direct-OPD with the adaptive KL controller fails to improve the student or actively degrades it compared to its initialization.

read the original abstract

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without training an explicit reward model or running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 7 minor

Summary. The paper proposes Direct On-Policy Distillation (Direct-OPD), a weak-to-strong post-training method that transfers a small RL teacher's policy shift (the log-ratio between its post-RL and pre-RL checkpoints) as a dense implicit reward for a stronger student. The core identity—reading a reward-like signal from a checkpoint pair via the KL-regularized RL closed-form optimum—is standard from the DPO literature but applied here in a novel direction: instead of fitting a policy from preferences, the authors read the implicit reward back out of an existing RL checkpoint pair and apply it on the student's own on-policy states. Empirically, the method improves multiple students (R1-Distill-7B, Qwen3-1.7B, Qwen3-4B) using two different teacher pairs, including students that already surpass the post-RL teacher, and does so at a fraction of the compute cost of direct RL on the large model.

Significance. The paper addresses a practically important problem: reducing the cost of RLVR post-training for strong models by reusing RL outcomes from smaller models. The conceptual contribution—transferring the RL-induced policy shift rather than the teacher's final policy—is clean and well-motivated. The mathematical derivation in Section 2.2 is correct and self-contained. The empirical results are substantial: consistent gains across two teacher pairs and three student families, a compute-matched comparison against direct RL (Section 3.2), sequential composition of multiple policy shifts (Section 3.3), and informative analysis of transfer dynamics (Section 4). The method is falsifiable: Section 4.3 explicitly identifies conditions under which the signal degrades, and Figure 1a provides a control showing that vanilla OPD (imitating the post-RL teacher) degrades the student while Direct-OPD improves it. The project page and hyperparameter tables support reproducibility.

major comments (1)
  1. Missing no-teacher-signal control. The central claim is that the teacher's policy shift ΔT = log πT − log πTref serves as a useful dense reward for the student. The vanilla OPD comparison in Figure 1a shows that the *type* of signal matters (imitating the post-RL teacher degrades performance), but neither condition isolates the teacher signal's contribution from the effect of KL-regularized fine-tuning on math data (Skywork-OR1) with a KL anchor to the student's initial checkpoint. A control with r_t(v) = 0 (i.e., KL-regularized fine-tuning on the same data, same prompt template, same training steps, but no teacher signal) would directly test whether the teacher's implicit reward contributes beyond simple task exposure. This is most important for the headline Qwen3-1.7B result (+14.1%, Table 1a), where the baseline is 48.3% and task-exposure gains are plausible. For Qwen3-4B (72.5% base,
minor comments (7)
  1. Section 3.2: The compute-matched comparison between weak-to-strong transfer and direct RL on R1-Distill-7B uses different training data (DAPO for direct RL vs. Skywork-OR1 for Direct-OPD transfer, per Appendix A). The paper notes that 'similar transfer trends' hold with DAPO-Math-17K, but the main comparison in Figure 3 is not data-matched. This should be acknowledged more prominently as a caveat on the compute-advantage claim.
  2. Eq. (16): The adaptive KL controller uses sgn(r̄_m) to adjust α, but the rationale for why the *sign* of the batch-mean reward is the right control signal is only informally motivated. The paper would benefit from a brief discussion of failure modes (e.g., oscillation when the sign flips frequently). That said, Figure 9 shows that fixed KL coefficients also work, so this is not load-bearing.
  3. Table 1: The QuestA teacher reference baseline (61.77 on AIME24) is higher than the Qwen3-1.7B student baseline (49.1). The text should clarify whether the teacher reference here is Nemotron-1.5B or a different model, as the numbers seem inconsistent with the 1.5B scale suggested elsewhere.
  4. Figure 2: The y-axis ranges vary across subplots (e.g., 0.375–0.475 for Qwen3-1.7B on AIME 2025 vs. 0.65–0.688 for Qwen3-4B). While this is standard for showing relative improvement, a note or shared baseline annotation would help readers compare across students.
  5. Section 2.3, Eq. (13): The Rao–Blackwellization replaces the single-token MC estimate with an expectation over the top-k distribution. It would help to state explicitly that this does not change the gradient in expectation (since the trajectory is still sampled from π_θ), only reduces variance—this is implied but not stated.
  6. References [55] and [56] appear to be duplicate entries (same title, same arXiv ID). References [57] and [58] are also duplicates.
  7. Appendix A mentions that the DAPO-style prompt template 'gives slightly better transfer' than the boxed-answer prompt used during teacher RL. Since prompt template can affect results, it would be useful to report the magnitude of this difference.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful and constructive report. The referee correctly identifies the core contribution, confirms the mathematical derivation, and acknowledges the breadth of the empirical results. The one major comment requests a no-teacher-signal control (r_t(v)=0) to isolate the teacher's implicit reward from the effect of KL-regularized fine-tuning on math data. We agree this control is valuable and will add it to the revised manuscript.

read point-by-point responses

  1. Referee: Missing no-teacher-signal control. The central claim is that the teacher's policy shift ΔT = log πT − log πTref serves as a useful dense reward for the student. The vanilla OPD comparison in Figure 1a shows that the *type* of signal matters (imitating the post-RL teacher degrades performance), but neither condition isolates the teacher signal's contribution from the effect of KL-regularized fine-tuning on math data (Skywork-OR1) with a KL anchor to the student's initial checkpoint. A control with r_t(v) = 0 (i.e., KL-regularized fine-tuning on the same data, same prompt template, same training steps, but no teacher signal) would directly test whether the teacher's implicit reward contributes beyond simple task exposure. This is most important for the headline Qwen3-1.7B result (+14.1%, Table 1a), where the baseline is 48.3% and task-exposure gains are plausible. For Qwen3-4B (72.5% base,

    Authors: We agree that this control is important and will include it in the revision. The referee is correct that the vanilla OPD comparison in Figure 1a isolates the *type* of signal (teacher imitation vs. policy shift) but does not isolate the teacher signal's contribution from the effect of KL-regularized fine-tuning on math data alone. A r_t(v)=0 control—same Skywork-OR1 data, same DAPO-style prompt template, same training steps, same KL anchor to the student's initial checkpoint, but with the teacher reward zeroed out—directly tests whether the teacher's implicit reward contributes beyond simple task exposure. We will run this control for the headline Qwen3-1.7B result (baseline 48.3%) and for Qwen3-4B (baseline 72.5%), and add the results to the revised manuscript. We expect the control to show limited or no gain over the baseline, which would confirm that the teacher's policy shift is the active ingredient, but we will report whatever the data shows. We note that several existing results partially address this concern: (1) Figure 1a shows that vanilla OPD (which uses the same data and training infrastructure but a different teacher signal) *degrades* the student, demonstrating that not just any fine-tuning signal helps; (2) the cross-pattern transfer results in Section 4.1 show gains without progressive teacher imitation, suggesting the reward signal rather than distribution matching drives improvement; and (3) the KL-coefficient sweep in Section 4.3 shows that validation accuracy is sensitive to how the teacher signal is calibrated, which would not be expected if gains came primarily from task exposure. Nevertheless, the r_t(v)=0 control is the cleanest isolation and we will add it. revision: yes

Circularity Check

0 steps flagged

No circularity found: the derivation is self-contained and rests on an external identity (DPO, Rafailov et al. [6])

full rationale

The paper's core derivation chain is not circular. The central identity (Eq. 6, log(π*/π_ref) = (1/β)r − log Z) is a standard result from KL-regularized RL, attributed to Rafailov et al. [6] (DPO), an external citation with no author overlap. Eq. 7 applies this identity to the teacher checkpoint pair to read out an implicit reward — this is a straightforward instantiation, not a self-defitional step. The Direct-OPD objective (Eq. 8) is defined independently using ΔT as a reward signal with KL regularization to the student's own initialization; its optimum (Eq. 9) is derived, not assumed. The token-level decomposition (Eq. 10) follows from the autoregressive factorization of both teacher policies. The gradient estimator (Eqs. 11–15) is a Rao–Blackwellized policy gradient with a stop-gradient coefficient — a standard construction, not circular. The adaptive KL controller (Eq. 16) is a heuristic that adjusts α based on the sign of the batch-mean reward; it does not fit a parameter to the target result and then call it a prediction. No self-citation is load-bearing for the mathematical derivation: reference [5] (shared authors) is cited for the OPD framework and top-k overlap diagnostics, but the core policy-as-reward identity comes from [6]. The empirical claims (improvements across teacher pairs and students) are validated against external benchmarks (AIME 2024/2025) and compared against direct RL baselines. The skeptic's concern about a missing no-teacher-signal control is an experimental-design issue, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The method introduces no new entities (particles, forces, dimensions). It reuses existing models and mathematical identities. The free parameters are standard hyperparameters, not ad hoc constants introduced to make a derivation work.

free parameters (3)
  • α (KL coefficient) = [0.5, 2.5] range, pair-dependent
    The KL penalty coefficient α is a free parameter that must be tuned per teacher-student pair. The paper introduces an adaptive controller (Eq. 16) with its own hyperparameters (ε=0.01, αmin=0.5, αmax=2.5) because a single fixed α does not transfer robustly.
  • Response length = 2048 tokens
    The training response length is a hyperparameter that affects results. Section 4.2 shows 2k validates best, but 4k and 6k do not, indicating this is a tuned choice rather than a robust default.
  • Top-k support = 16
    The student top-k support size is fixed at 16 (Table 3) without sensitivity analysis.
axioms (3)
  • domain assumption The post-RL teacher πT is the KL-regularized optimum of some latent reward rT anchored at πTref.
    Section 2.2 states this assumption to apply the policy-as-reward identity. If the teacher's RL did not converge to or near this optimum, the extracted reward signal is invalid.
  • domain assumption The teacher's RL-induced policy shift is meaningful on the student's on-policy states.
    Section 4.3 explicitly states this is required for the method to work. If the student visits states where the teacher/reference log-ratio is noisy or uninformative, the dense reward degrades.
  • standard math The Rao-Blackwellized per-step estimator (Eq. 13) with stop-gradient (Eq. 14) is a valid surrogate for the policy gradient.
    The stop-gradient is applied to the weighted reward to prevent extra Jacobian terms. This is a standard technique but introduces an approximation.

pith-pipeline@v1.1.0-glm · 24949 in / 2529 out tokens · 191408 ms · 2026-07-07T12:22:06.016759+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

106 extracted references · 106 canonical work pages · 65 internal anchors

  1. [1]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. Nature, 645:633–638, 2025

    work page 2025

  2. [2]

    JustRL: Scaling a 1.5B LLM with a simple RL recipe

    Bingxiang He, Zekai Qu, Zeyuan Liu, Yinghao Chen, Yuxin Zuo, Cheng Qian, Kaiyan Zhang, Weize Chen, Chaojun Xiao, Ganqu Cui, Ning Ding, and Zhiyuan Liu. JustRL: Scaling a 1.5B LLM with a simple RL recipe. arXiv preprint arXiv:2512.16649, 2025

    work page arXiv 2025

  3. [3]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    POLARIS: A post-training recipe for scaling reinforcement learning on reasoning models

    Chenxin An et al. POLARIS: A post-training recipe for scaling reinforcement learning on reasoning models. https://github.com/ChenxinAn-fdu/POLARIS, 2025

    work page 2025

  5. [5]

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  6. [6]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.arXiv preprint arXiv:2305.18290, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  7. [7]

    QuestA: Expanding reasoning capacity in LLMs via question augmentation.arXiv preprint arXiv:2507.13266, 2025

    Jiazheng Li, Hongzhou Lin, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Yi Wu, and Jingzhao Zhang. QuestA: Expanding reasoning capacity in LLMs via question augmentation.arXiv preprint arXiv:2507.13266, 2025

    work page arXiv 2025

  8. [8]

    Lyng, Sanjit Singh Batra, and Robert E

    Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D. Lyng, Sanjit Singh Batra, and Robert E. Tillman. Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistics: ACL 2026, pages 25553–25569, 2026

    work page 2026

  9. [9]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  11. [11]

    Kimi k1.5: Scaling Reinforcement Learning with LLMs

    Kimi Team. Kimi k1.5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    QwQ-32B: Embracing the power of reinforcement learning.https://qwenlm.github.io/blog/qw q-32b/, 2025

    Qwen Team. QwQ-32B: Embracing the power of reinforcement learning.https://qwenlm.github.io/blog/qw q-32b/, 2025

    work page 2025

  13. [13]

    Open-reasoner- zero: An open source approach to scaling reinforcement learning on the base model.https://github.com/Ope n-Reasoner-Zero/Open-Reasoner-Zero, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner- zero: An open source approach to scaling reinforcement learning on the base model.https://github.com/Ope n-Reasoner-Zero/Open-Reasoner-Zero, 2025

    work page 2025

  14. [14]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [16]

    DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [17]

    How far can unsupervised rlvr scale llm training?arXiv preprintarXiv:2603.08660, 2026

    Bingxiang He, Yuxin Zuo, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang, Yuchen Fan, Ganqu Cui, et al. How far can unsupervised rlvr scale llm training?arXiv preprintarXiv:2603.08660, 2026

    work page arXiv 2026

  17. [18]

    DeepSeek V4 preview release.https://api-docs.deepseek.com/news/news260424, 2026

    DeepSeek-AI. DeepSeek V4 preview release.https://api-docs.deepseek.com/news/news260424, 2026

    work page 2026

  18. [19]

    GLM-5.2: Built for long-horizon tasks.https://z.ai/blog/glm-5.2, 2026

    Z.ai. GLM-5.2: Built for long-horizon tasks.https://z.ai/blog/glm-5.2, 2026. 16

    work page 2026

  19. [20]

    AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset

    Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. AIMO-2 winning solution: Building state-of-the-art mathematical reasoning models with OpenMathReasoning dataset. arXiv preprint arXiv:2504.16891, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [21]

    OpenThoughts: Data Recipes for Reasoning Models

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [22]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. 2024

    work page 2024

  22. [23]

    Distilling the Knowledge in a Neural Network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  23. [24]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of EMNLP, 2016

    work page 2016

  24. [25]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  25. [26]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. 2020

    work page 2020

  26. [27]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. 2020

    work page 2020

  27. [28]

    Improved knowledge distillation via teacher assistant

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. 2020

    work page 2020

  28. [29]

    On the efficacy of knowledge distillation

    Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. 2019

    work page 2019

  29. [30]

    Distillation Scaling Laws

    Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws.arXiv preprint arXiv:2502.08606, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  30. [31]

    MiniLLM: On-Policy Distillation of Large Language Models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  31. [32]

    f-Divergence Minimization for Sequence-Level Knowledge Distillation

    Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-Divergence Minimization for Sequence-Level Knowledge Distillation. Proceedings of ACL, 2023. URLhttps://arxiv.org/abs/2307.15190

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [33]

    Revisiting Knowledge Distillation for Autoregressive Language Models

    Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao. Revisiting Knowledge Distillation for Autoregressive Language Models.Proceedings of ACL, 2024. URLhttps://arxiv.org/abs/2402.11890

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  33. [34]

    Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models

    Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. Rethinking Kullback- Leibler Divergence in Knowledge Distillation for Large Language Models.Proceedings of COLING, 2025. URL https://arxiv.org/abs/2404.02657

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [35]

    DistiLLM: Towards Streamlined Distillation for Large Language Models

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. DistiLLM: Towards Streamlined Distillation for Large Language Models.Proceedings of ICML, 2024. URLhttps://arxiv.org/abs/2402.03898

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [36]

    DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

    Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se-Young Yun. DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. InProceedings of ICML, 2025. URL https://arxiv.org/abs/2503.07067

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [37]

    MiniPLM: Knowledge Distillation for Pre-Training Language Models

    Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. MiniPLM: Knowledge Distillation for Pre-Training Language Models.Proceedings of ICLR, 2025. URLhttps://arxiv.org/abs/2410.17215

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [38]

    DistillSpec: Improving Speculative Decoding via Knowledge Distillation

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. DistillSpec: Improving Speculative Decoding via Knowledge Distillation. Proceedings of ICLR, 2024. URLhttps://arxiv.org/abs/2310.08461

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [39]

    Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling

    Wenda Xu, Rujun Han, Zifeng Wang, Long T. Le, Dhruv Madeka, Lei Li, William Yang Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling.Proceedings of ICLR, 2025. URLhttps://arxiv.org/abs/2410.11325. 17

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  39. [40]

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes.arXiv preprint arXiv:2306.13649, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  40. [41]

    On-policy distillation

    Kevin Lu and Thinking Machines Lab. On-policy distillation. 2025

    work page 2025

  41. [42]

    A Survey of On-Policy Distillation for Large Language Models

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  42. [43]

    Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  43. [44]

    Entropy-Aware On-Policy Distillation of Language Models

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  44. [45]

    Stable On-Policy Distillation through Adaptive Target Reformulation

    Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation.arXiv preprint arXiv:2601.07155, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  45. [46]

    Unifying group-relative and self-distillation policy optimization via sample routing.arXiv preprint arXiv:2604.02288, 2026a

    Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing. arXiv preprint arXiv:2604.02288, 2026

    work page arXiv 2026

  46. [47]

    On-Policy Context Distillation for Language Models

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. arXiv preprint arXiv:2602.12275, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  47. [48]

    Online Experiential Learning for Language Models

    Tianzhu Ye, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Online experiential learning for language models. arXiv preprint arXiv:2603.16856, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  48. [49]

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  49. [50]

    Self-distillation for multi-token prediction, 2026.arXiv preprint arXiv:2603.23911, 2026

    Guoliang Zhao, Ruobing Xie, An Wang, Shuaipeng Li, Huaibing Xie, and Xingwu Sun. Self-distillation for multi-token prediction, 2026.arXiv preprint arXiv:2603.23911, 2026

    work page arXiv 2026

  50. [52]

    Self-Distillation Enables Continual Learning

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  51. [53]

    Lyng, Sanjit Singh Batra, and Robert E

    Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler, Qian Qian, Gregory D. Lyng, Sanjit Singh Batra, and Robert E. Tillman. Fast and Effective On-policy Distillation from Reasoning Prefixes. arXiv preprint arXiv:2602.15260, 2026. URLhttps://arxiv.org/abs/2602.15260

    work page arXiv 2026

  52. [54]

    PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence

    Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence.arXiv preprint arXiv:2603.11178, 2026. URL https: //arxiv.org/abs/2603.11178

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  53. [56]

    Scaling reasoning efficiently via relaxed on-policy distillation.arXiv preprint arXiv:2603.11137, 2026

    Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation.arXiv preprint arXiv:2603.11137, 2026

    work page arXiv 2026

  54. [58]

    Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

    Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?arXiv preprint arXiv:2603.24472, 2026. URLhttps://arxiv.org/abs/2603.24472

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  55. [59]

    Black-box on-policy distillation of large language models.arXiv preprint arXiv:2511.10643,

    Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. Black-Box On-Policy Distillation of Large Language Models.arXiv preprint arXiv:2511.10643, 2025. URLhttps://arxiv.org/abs/2511.10643. 18

    work page arXiv 2025

  56. [60]

    MiMo-V2-Flash Technical Report

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  57. [62]

    URLhttps://arxiv.org/abs/2505.16142

    work page arXiv

  58. [63]

    KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning

    Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, and Fei Mi. KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning.arXiv preprint arXiv:2506.02208, 2025. URLhttps://arxiv.org/abs/2506.02208

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  59. [64]

    Reinforcement-aware Knowledge Distillation for LLM Reasoning

    Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto. Reinforcement-aware Knowledge Distillation for LLM Reasoning.arXiv preprint arXiv:2602.22495, 2026. URLhttps://arxiv.org/abs/2602.22495

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  60. [65]

    AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation

    Songming Zhang, Xue Zhang, Tong Zhang, Bojie Hu, Yufeng Chen, and Jinan Xu. AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation. In Proceedings of ACL, 2025. URL https: //arxiv.org/abs/2503.02832

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  61. [66]

    Reinforcement Learning via Self-Distillation

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  62. [67]

    Self-Distilled RLVR

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr.arXiv preprint arXiv:2604.03128, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  63. [68]

    Small models struggle to learn from strong reasoners

    Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. 2025

    work page 2025

  64. [69]

    Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  65. [70]

    Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision.arXiv preprint arXiv:2312.09390, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  66. [71]

    Introducing Superalignment.OpenAI Blog, 2023

    Jan Leike and Ilya Sutskever. Introducing Superalignment.OpenAI Blog, 2023

    work page 2023

  67. [72]

    Semi-supervised learning by entropy minimization.Advances in neural information processing systems, 17, 2004

    Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization.Advances in neural information processing systems, 17, 2004

    work page 2004

  68. [73]

    Semi-supervised sequence learning.Advances in neural information processing systems, 28, 2015

    Andrew M Dai and Quoc V Le. Semi-supervised sequence learning.Advances in neural information processing systems, 28, 2015

    work page 2015

  69. [74]

    Temporal Ensembling for Semi-Supervised Learning

    Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning.arXiv preprintarXiv:1610.02242, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  70. [75]

    There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average

    Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many consistent explanations of unlabeled data: Why you should average.arXiv preprint arXiv:1806.05594, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  71. [76]

    Co- teaching: Robust training of deep neural networks with extremely noisy labels.Advances in neural information processing systems, 31, 2018

    Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co- teaching: Robust training of deep neural networks with extremely noisy labels.Advances in neural information processing systems, 31, 2018

    work page 2018

  72. [77]

    Mixmatch: A holistic approach to semi-supervised learning.Advancesin neural information processing systems, 32, 2019

    David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning.Advancesin neural information processing systems, 32, 2019

    work page 2019

  73. [78]

    DivideMix: Learning with Noisy Labels as Semi-supervised Learning

    Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.07394, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2002

  74. [79]

    Big self-supervised models are strong semi-supervised learners.Advancesin neural information processing systems, 33:22243–22255, 2020

    Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners.Advancesin neural information processing systems, 33:22243–22255, 2020

    work page 2020

  75. [80]

    Self-training avoids using spurious features under domain shift

    Yining Chen, Colin Wei, Ananya Kumar, and Tengyu Ma. Self-training avoids using spurious features under domain shift. Advancesin Neural Information Processing Systems, 33:21061–21071, 2020. 19

    work page 2020

  76. [81]

    Debiased self-training for semi-supervised learning.Advances in Neural Information Processing Systems, 35:32424–32437, 2022

    Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning.Advances in Neural Information Processing Systems, 35:32424–32437, 2022

    work page 2022

  77. [82]

    Supervising strong learners by amplifying weak experts

    Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  78. [83]

    Scalable agent alignment via reward modeling: a research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction.arXiv preprint arXiv:1811.07871, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  79. [84]

    AI safety via debate

    Geoffrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate.arXiv preprint arXiv:1805.00899, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  80. [85]

    Measuring Progress on Scalable Oversight for Large Language Models

    Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

Showing first 80 references.