Pith. sign in

REVIEW 3 major objections 5 minor 37 references

Distilled RL folds teacher token preferences into the RL gradient, so a teacher can guide a student step-by-step without unconditional imitation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:35 UTC pith:G6HLRYCY

load-bearing objection A practical and honest post-training recipe that reweights RL gradients with teacher token preferences; the cross-family gains are the main result, but the single-seed evidence and unvalidated teacher-competence assumption keep me from fully buying the averages. the 3 major comments →

arxiv 2607.17247 v1 pith:G6HLRYCY submitted 2026-07-19 cs.LG cs.AI

Distilled Reinforcement Learning for LLM Post-training

Chen Wang , Zhaochun Li , Jionghao Bai , Yining Zhang , Hexuan Deng , Ge Lan , Yue Wang This is my paper
classification cs.LG cs.AI
keywords reinforcement learningknowledge distillationon-policy distillationLLM post-trainingcredit assignmentteacher reweightingreasoningtoken-level guidance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that LLM post-training improves when teacher supervision enters the reinforcement learning gradient itself, rather than as a separate KL-based imitation loss. The proposed method, Distilled RL, reweights the RL advantage at the token level using a clipped, normalized teacher-to-student probability ratio, and it resets those weights on negative-advantage responses. The authors argue this gives the student fine-grained credit assignment while preserving the reward-driven optimization of RL, and they report that it outperforms both standard RL and on-policy distillation, especially in cross-family teacher-student settings. A controlled entropy-transfer experiment is offered as evidence that the student acquires information beyond the outcome reward.

Core claim

The central claim is that teacher supervision should be coupled into the RL surrogate objective through a reverse importance ratio rather than added as an unconditional imitation term. The objective is min(r_{i,t} w_{i,t} A_i, clip(r_{i,t}) w_{i,t} A_i), where r is the ordinary student-policy ratio, A_i is the group-normalized sequence advantage, and w_{i,t} is a teacher-derived weight: the clipped teacher-to-old-student token probability ratio, normalized per sequence so its geometric mean is one, and set to one whenever A_i is non-positive. On successful responses, tokens the teacher prefers are amplified and tokens it disprefers are suppressed; on failed responses, the update reduces to t

What carries the argument

The carrying object is the reverse importance ratio rho_{i,t} = pi_teacher(oi_t | q, o_{i,<t}) / pi_theta_old(oi_t | q, o_{i,<t}), evaluated on tokens the student itself generated. The ratio is clipped to a bounded interval, divided by the per-sequence geometric mean of clipped ratios, and multiplied into the clipped policy-gradient objective; for negative-advantage responses the weight is reset to 1. This converts the teacher's relative token preferences into a redistribution of the sequence-level RL advantage, so the teacher guides which tokens to reinforce without globally amplifying or suppressing the response.

Load-bearing premise

The method assumes that whenever a student response is correct, the teacher is competent enough on that prompt that its per-token preferences are useful guidance; Appendix A notes that the teacher is never rolled out, so the training procedure never checks this assumption.

What would settle it

Train Distilled RL on a set of prompts where the teacher provably fails on many prompts the student solves, and compare against plain RL: if teacher weighting on correct student responses hurts or offers no gain, the teacher-competence assumption is violated. A more direct check is to record, on correct student responses, whether the tokens the teacher prefers lead to later failure in the same rollout; if such reversed preferences are frequent, the token-level guidance is misleading.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the central claim is correct, teacher guidance no longer needs to be framed as distribution matching, so the teacher can be used selectively without dragging the student into a KL-induced local optimum.
  • Cross-family distillation becomes viable: the largest reported gains appear precisely where the teacher and student differ in architecture or reasoning distribution, the regime where KL-based on-policy distillation tends to degrade.
  • The student can acquire knowledge beyond the outcome reward signal, as the entropy-control case study demonstrates by transferring a prescribed distributional property that RL alone does not provide.
  • Pass@k improves as well as pass@1, implying the method strengthens the whole response distribution, not just the single most probable answer.
  • Because the update reduces to standard RL on negative-advantage trajectories, the method preserves the reward-driven penalty structure and avoids counterproductive imitation of bad responses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same teacher-weighting mechanism could be gated by an explicit per-prompt estimate of teacher competence, such as occasional teacher rollouts or a verifier check, potentially extending gains to cases where the teacher is locally weaker than the student.
  • Beyond the paper: the sequence-level geometric normalization is a generic device that could also be applied to token-level process rewards or advantage estimation, not only to teacher ratios.
  • Beyond the paper: the entropy-transfer result suggests Distilled RL could be used to steer distributional properties such as output entropy, response length, or style, by constructing teachers with prescribed properties rather than relying solely on correctness rewards.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Distilled RL, a post-training objective that injects token-level teacher preferences into a GRPO-style RL update. The method replaces unconditional KL-based imitation (OPD) with three components: reverse importance sampling with clipping of the teacher-to-student token ratio, a negative-sample reset that disables teacher reweighting on non-positive advantages, and sequence-level geometric normalization of the clipped ratios. The central empirical claim is that Distilled RL outperforms standard RL, OPD, and a simple OPD+RL combination on pass@1 and pass@k across within-family and cross-family distillation settings, with the largest gains in cross-family distillation. A controlled entropy-based case study is offered as evidence that the method transfers teacher-side distributional knowledge beyond the task reward.

Significance. If the empirical claims hold, Distilled RL would be a useful and simple recipe for combining teacher supervision with outcome-based RL in LLM post-training. The paper has clear strengths: the evaluation uses external benchmarks, the teacher (Qwen3-8B-GRPO) is external to the student family in the cross-family setting, the ablations in Table 4 support the two principal design components, and the authors provide a code link. The method is also conceptually appealing because it avoids unconditional imitation while retaining dense token-level feedback. However, the central empirical claim currently rests on point estimates without error bars or multiple seeds, and the key teacher-competence assumption is acknowledged but not validated. The entropy case study, as constructed, is a feedback-control demonstration rather than independent evidence of knowledge transfer. These issues are load-bearing for the paper's contributions and require additional empirical support or careful reframing.

major comments (3)
  1. [§5.1, Tables 2 and 3] The main empirical claim is that Distilled RL 'consistently outperforms' RL, OPD, and OPD+RL. However, the paper reports no error bars, no multiple seeds, and no statistical tests. Several reported improvements are small or negative at the individual-benchmark level; for example, Qwen3-1.7B shows -0.11 on HMMT25 and an average gain of only +1.16 over OPD. Since pass@1 is computed from 32 sampled responses, sampling noise is non-negligible. Without variance estimates or at least bootstrap confidence intervals, the central 'consistent improvement' claim is not yet established. Please add multiple runs (or seed-level results) and report variability, especially for the smaller gains.
  2. [§3.2 / Appendix A] Appendix A explicitly states the implicit teacher-competence assumption: when a student response has positive advantage, the teacher's token-level preferences are assumed to be useful guidance. The training loop never checks this: the teacher is only scored on student-generated tokens and is never rolled out on the prompt. As the student improves, it may locally surpass the fixed teacher on some prompts, making the teacher's reweighting misleading even on correct student responses. This is not merely a theoretical concern; it directly affects the positive-advantage reweighting in Eq. (3). The paper provides no per-prompt teacher-correctness statistics, no diagnostic of teacher/student token agreement, and no ablation that gates teacher weights on teacher competence. Without such evidence, the reported gains could be concentrated on prompts where the teacher is competent while the method
  3. [§4, Figure 4] The entropy case study is presented as evidence that Distilled RL can transfer 'previously unavailable knowledge' from the teacher. But the teacher is constructed from the student itself, with temperature chosen from the student's current entropy: low temperature when entropy is above 0.5 and high temperature when entropy is below 0.5. This is a feedback controller, not an independent teacher. The observed trajectory toward the entropy threshold is therefore expected from the sign of the control signal and does not demonstrate that Distilled RL acquires information beyond the reward. This claim is a listed contribution ('transfer previously unavailable knowledge'), so it needs either a genuinely independent teacher property or a reframing as a synthetic sanity check of the weighting mechanism rather than evidence of knowledge transfer.
minor comments (5)
  1. [§3.2, Eq. (3)] Notation is inconsistent: the objective defines r_{i,t}(θ) but the clipped expression and surrounding text use r_{i,t}. The min expression is also visually dense; adding parentheses around the clipped term would improve readability.
  2. [Table 4] The layout of Table 4 is very hard to read: benchmark values and parenthetical deltas run together in a single row, especially for DSQW-1.5B. Please align columns by benchmark and separate the deltas clearly, ideally in a separate row or column.
  3. [§4] The entropy threshold of 0.5 is introduced without units or justification. State whether this is natural-log entropy, and explain how the threshold was chosen.
  4. [Appendix B] The SuperGPQA subset is described as 'fixed' but no seed or construction procedure is given. Please specify how the 500 examples were selected.
  5. [§5.1] Figure 2 and Figure 5 report training dynamics for a single run. Adding light smoothing and a second seed would improve interpretability, especially for the OPD+RL 'performance decline' claim.

Circularity Check

1 steps flagged

External benchmark results are independent, but the entropy case study's 'teacher' is the student's own distribution with temperature set from the student's current entropy, making the demonstrated entropy tracking self-referential; Appendix A candidly admits the untested teacher-competence assumption.

specific steps
  1. self definitional [Section 4 (Interpretable Teacher-Side Information Transfer), Figure 4]
    "Instead of using a larger model as the teacher, we construct a temperature-controlled teacher from the student model itself. Specifically, when the student entropy is larger than 0.5, we use a low-temperature teacher with temperature 0.8 to provide a sharper distributional signal; otherwise, we use a high-temperature teacher with temperature 1.2 to provide a softer distributional signal."

    The 'teacher' is the student's own policy evaluated at a temperature that is a function of the student's current entropy H. If H > 0.5 the teacher is sharper (temperature 0.8), which tends to push entropy down; if H < 0.5 the teacher is softer (temperature 1.2), which tends to push entropy up. The observed convergence of student entropy toward the 0.5 threshold is therefore a built-in consequence of this negative-feedback construction, not evidence that an external teacher's knowledge was transferred. Thus the case study's claim to demonstrate transfer of 'previously unavailable knowledge' is self-referential rather than an independent test.

full rationale

The main benchmark results are self-contained: Distilled RL is evaluated on external reasoning benchmarks (AIME24/25/26, CMIMC25, HMMT25, AMC23, GSM8K, MATH500, Minerva, Olympiad, MMLU-Pro, SuperGPQA) using an external teacher (Qwen3-8B-GRPO), and no test-set statistic is fitted into Eq. (3). The GRPO and OPD baselines are also standard external methods. The central comparison therefore does not reduce to the method's own inputs. The one genuinely circular element is the 'interpretable case study' in Section 4: the teacher is the student's own distribution with temperature chosen as a function of the student's current entropy (0.8 when H > 0.5, 1.2 when H < 0.5). Under this feedback construction, entropy moving toward the 0.5 threshold is a built-in consequence of the control rule, so it cannot by itself support the abstract's claim that Distilled RL transfers 'previously unavailable knowledge' from a teacher. This is a supporting demonstration, not the main benchmark claim, so it does not invalidate the external results. Appendix A explicitly acknowledges the untested teacher-competence assumption; that is a correctness/robustness limitation, not circularity. Self-citations to the authors' prior entropy-RL papers are present but not load-bearing, and no uniqueness theorem is imported from them. Overall, the central empirical claim is independently grounded, with partial circularity confined to the entropy case study.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on the standard GRPO baseline, on the admissibility of using teacher/student token ratios as per-token weights, and on the unverified teacher-competence assumption. The hand-chosen hyperparameters (epsilon_rho=3 and the entropy threshold 0.5) are the main free parameters; no final result is defined by them, so the method is not a hidden fit, but they weaken the case study.

free parameters (4)
  • ratio clipping threshold epsilon_rho = 3
    Chosen by hand (Table 5); no sensitivity analysis is reported, and it directly controls how strongly teacher ratios can affect the gradient.
  • entropy threshold in case study = 0.5
    The temperature-controlled teacher switches at student entropy 0.5 (Sec. 4); this hand-chosen threshold defines the 'target' the experiment claims the student tracks.
  • policy clip range epsilon_low/epsilon_high = 0.2/0.2
    Standard PPO/GRPO clipping; included in Table 5 and used in Eq. (3).
  • rollout group size G = 8
    Rollout group for reward normalization and advantages (Table 5); Fig. 3 shows insensitivity for OPD but no such study for Distilled RL.
axioms (4)
  • standard math Group-normalized reward difference is a valid baseline/advantage estimate (GRPO).
    Invoked in Eq. (3) and Table 5; standard in GRPO, but no proof is given for the teacher-weighted objective.
  • domain assumption Autoregressive token-level ratios can be treated as importance weights and clipped/normalized without invalidating the policy update.
    Central to Eq. (3) and Sec. 3.2; no variance or bias analysis is provided.
  • domain assumption Teacher is competent on student-correct problems so its token preferences are useful.
    Explicitly acknowledged as an implicit assumption in Appendix A; load-bearing for knowledge transfer.
  • standard math The PPO min-clip surrogate with positive multiplier w is a safe proxy for improving expected reward.
    Because min(a,c)b = min(ab,cb) for b>0, Eq. (3) is the GRPO surrogate multiplied by a teacher weight; this standard clipping is used without new proof.

pith-pipeline@v1.3.0-alltime-deepseek · 13297 in / 14433 out tokens · 137577 ms · 2026-08-01T18:35:13.833803+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Distilled Reinforcement Learning for LLM Post-training." pith.science (2026). https://pith.science/paper/G6HLRYCY

@misc{pith2026260717247,
  author       = {Pith},
  title        = {Pith review of: Distilled Reinforcement Learning for LLM Post-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6HLRYCY}},
  note         = {Machine review of arXiv:2607.17247}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.

Figures

Figures reproduced from arXiv: 2607.17247 by Chen Wang, Ge Lan, Hexuan Deng, Jionghao Bai, Yining Zhang, Yue Wang, Zhaochun Li.

Figure 1
Figure 1. Figure 1: Overview of Distilled RL. determine whether the final answer is correct (Shao et al., 2024). Such coarse-grained supervision creates a severe credit-assignment problem and provides limited information about which intermediate reasoning steps should be encouraged or corrected (Lightman et al., 2023). Moreover, because the learning signal is derived mainly from task rewards, RL is effective at reinforcing be… view at source ↗
Figure 2
Figure 2. Figure 2: Training dynamics of OPD, RL, and OPD+RL on five competition reasoning benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics of OPD with different rollout group sizes. The teacher model is Qwen3- [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Entropy trajectory of Dis￾tilled RL with a temperature-controlled teacher. Instead of using a larger model as the teacher, we construct a temperature-controlled teacher from the student model itself. Specifically, when the student entropy is larger than 0.5, we use a low-temperature teacher with temperature 0.8 to provide a sharper distributional signal; otherwise, we use a high-temperature teacher with te… view at source ↗
Figure 5
Figure 5. Figure 5: Training dynamics of Distilled RL and baseline methods. The panels show reward, policy [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: further provides an interpretable demonstration. Under the same entropy-control setting as [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 28 linked inside Pith

  1. [1]

    On-policy distillation of language models: Learning from self- generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pp. 21246–21263,

    2024

  2. [4]

    Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758,

    Pith/arXiv arXiv

  3. [5]

    13 Preprint Table 5: Detailed configurations in the experiments. Configuration Setting Training Settings Global batch size|B|128 Mini-batch size 64 Micro-batch size per GPU 2 Rollout group sizeG8 Maximum prompt length 2048 Maximum response length 8192 Sampling temperature 1 Top-p0.99 Learning rate 1e-6 Number of training steps 160 Policy Optimization Adva...

    2048

  4. [7]

    Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,

    Pith/arXiv arXiv

  5. [8]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793,

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793,

    Pith/arXiv arXiv

  6. [9]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pp. 32694–32717,

    2024

  7. [10]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196,

    10 Preprint Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196,

    Pith/arXiv arXiv

  8. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025a

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025a. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang...

    Pith/arXiv arXiv

  9. [13]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 conference on empirical methods in natural language processing, pp. 1317–1327,

    2016

  10. [15]

    Treepo: Bridging the gap of policy optimiza- tion and efficacy and inference efficiency with heuristic tree-based modeling.arXiv preprint arXiv:2508.17445,

    Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, et al. Treepo: Bridging the gap of policy optimiza- tion and efficacy and inference efficiency with heuristic tree-based modeling.arXiv preprint arXiv:2508.17445,

    Pith/arXiv arXiv

  11. [16]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Pith/arXiv arXiv

  12. [18]

    Length-unbiased sequence policy optimization: Revealing and controlling response length variation in rlvr.arXiv preprint arXiv:2602.05261,

    Fanfan Liu, Youyang Yin, Peng Shi, Siqi Yang, Zhixiong Zeng, and Haibo Qiu. Length-unbiased sequence policy optimization: Revealing and controlling response length variation in rlvr.arXiv preprint arXiv:2602.05261,

    arXiv

  13. [19]

    Improve mathematical reasoning in language models by automated process supervision.arXiv preprint arXiv:2406.06592,

    Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision.arXiv preprint arXiv:2406.06592,

    Pith/arXiv arXiv

  14. [20]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  15. [22]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599,

    Pith/arXiv arXiv

  16. [23]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    Pith/arXiv arXiv

  17. [24]

    Arbitrary entropy policy optimization breaks the exploration bottleneck of reinforcement learning

    Chen Wang, Zhaochun Li, Jionghao Bai, Yuzhi Zhang, Shisheng Cui, Zhou Zhao, and Yue Wang. Arbitrary entropy policy optimization breaks the exploration bottleneck of reinforcement learning. arXiv preprint arXiv:2510.08141,

    Pith/arXiv arXiv

  18. [25]

    Implicit compression regularization: Concise reasoning via internal shorter distributions in rl post-training.arXiv preprint arXiv:2605.07316, 2026a

    Chen Wang, Hexuan Deng, Yining Zhang, Yuchen Zhang, Jionghao Bai, Zhaochun Li, Ge Lan, and Yue Wang. Implicit compression regularization: Concise reasoning via internal shorter distributions in rl post-training.arXiv preprint arXiv:2605.07316, 2026a. Chen Wang, Lai Wei, Yanzhi Zhang, Chenyang Shao, Zedong Dan, Weiran Huang, Ge Lan, and Yue Wang. Targeted ...

    Pith/arXiv arXiv 2026

  19. [26]

    Rethinking kullback-leibler divergence in knowledge distillation for large language models

    Taiqiang Wu, Chaofan Tao, Jiahao Wang, Runming Yang, Zhe Zhao, and Ngai Wong. Rethinking kullback-leibler divergence in knowledge distillation for large language models. InProceedings of the 31st International Conference on Computational Linguistics, pp. 5737–5755, 2025a. Yecheng Wu, Song Han, and Hai Cai. Lightning opd: Efficient post-training for large ...

    Pith/arXiv arXiv

  20. [27]

    On the generalization of sft: A reinforcement learning perspective with reward rectification.arXiv preprint arXiv:2508.05629, 2025b

    Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification.arXiv preprint arXiv:2508.05629, 2025b. Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, and Fe...

    arXiv

  21. [28]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115,

    Pith/arXiv arXiv

  22. [29]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    12 Preprint An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  23. [30]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125,

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125,

    Pith/arXiv arXiv

  24. [31]

    Shorterbetter: Guiding reasoning models to find optimal inference length for efficient reasoning.arXiv preprint arXiv:2504.21370,

    Jingyang Yi, Jiazheng Wang, and Sida Li. Shorterbetter: Guiding reasoning models to find optimal inference length for efficient reasoning.arXiv preprint arXiv:2504.21370,

    arXiv

  25. [32]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  26. [33]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    Pith/arXiv arXiv

  27. [34]

    Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922,

    Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922,

    Pith/arXiv arXiv

  28. [35]

    Consequently, the training procedure cannot directly determine whether the teacher is capable of solving a given problem

    Appendix A LIMITATION A fundamental limitation of on-policy distillation is that the teacher is only queried to evaluate student-generated trajectories, without sampling complete responses from the teacher. Consequently, the training procedure cannot directly determine whether the teacher is capable of solving a given problem. Distilled RL mitigates much ...

    2025

  29. [36]

    using the same prompt distribution, rollout budget, optimization schedule, and evaluation protocol. We use Qwen3-8B-GRPO as the teacher model and evaluate two student models: Qwen3-4B and DeepSeek-R1-Distill-Qwen-1.5B, abbreviated as DSQW-1.5B (Yang et al., 2024; 2025; Guo et al., 2025b). Qwen3-4B provides a relatively compatible teacher–student setting, ...

    2024

  30. [2015]

    Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079,

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079,

    Pith/arXiv arXiv

  31. [2017]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    11 Preprint Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  32. [2021]

    Process reinforcement through implicit rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025a. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao ...

    Pith/arXiv arXiv

  33. [2022]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

    Pith/arXiv arXiv

  34. [2023]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  35. [2024]

    Matharena: Evaluating llms on uncontaminated math competitions.arXiv preprint arXiv:2505.23281,

    Mislav Balunovi´c, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovi´c, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions.arXiv preprint arXiv:2505.23281,

    Pith/arXiv arXiv

  36. [2025]

    Howard Chen, Noam Razin, Karthik Narasimhan, and Danqi Chen

    doi: 10.48550/arXiv.2505.23281. Howard Chen, Noam Razin, Karthik Narasimhan, and Danqi Chen. Retaining by doing: The role of on-policy data in mitigating forgetting.arXiv preprint arXiv:2510.18874,

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.23281

  37. [2026]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv