Pith. sign in

REVIEW 4 major objections 3 minor 65 references

Simple-OPD: Demystifying Warm-up for On-policy Distillation

T0 review · 4 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the warm-up stage before on-policy distillation works by transferring the teacher's reasoning pattern—not correct answers—and that a LoRA-warmed, teacher-compatible chain-of-thought initialization consistently…

desk verdict A credible warm-up recipe whose LoRA-over-SFT claim is confounded by an unequal learning-rate comparison; worth serious review but needs a sweep and seeds. read the letter →

arxiv 2608.06802 v1 pith:XKBBPEUZ submitted 2026-08-07 cs.CL

Tao Liu , Taiqiang Wu , Mao Zheng , Xuan Luo , Runming Yang , Xuewei Yang , Junjie Wang , Yujiu Yang This is my paper · ORCID
classification cs.CL
keywords on-policydistillationwarm-upchain-of-thoughtlow-rankadaptationLoRAin-domainout-of-distributiongeneralizationreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On-policy distillation trains a student on its own rollouts under token-level supervision from a teacher, and this paper argues that the warm-up preceding it does one specific job: it imprints the teacher's reasoning pattern into the student before distillation begins. The paper establishes three facts about that warm-up: it needs chain-of-thought supervision generated by the same teacher that will supervise OPD, not by a stronger external model; the correctness of those teacher rollouts barely matters; and a low-rank LoRA adapter trained to near saturation balances in-domain adaptation against out-of-distribution generalization better than full-parameter fine-tuning. Together these facts yield Simple-OPD, a plug-and-play warm-up recipe that improves in-domain reasoning while preserving overall generalization across OPD objectives, thinking and non-thinking models, and same-size teacher-student consolidation. A sympathetic reader should care because warm-up is currently treated as an unexamined heuristic, and this paper turns it into a principled initialization with a cheap implementation.

What carries the argument

The load-bearing object is the teacher-compatible chain-of-thought rollout used as warm-up data together with the low-rank adapter that constrains how the student absorbs it. LoRA freezes the pretrained weights and learns only $A$ and $B$ low-rank matrices ($W = W_0 + \frac{\alpha}{r} BA$), so the student acquires the teacher's reasoning distribution without large changes to its pretrained capabilities. During warm-up, the student is trained on teacher-generated CoT with this low-rank constraint; during OPD, the student's rollouts are scored token-wise by the teacher's reverse KL. The mechanism's job is to move the student's rollout distribution into the teacher's generation space before distillation starts, which is why teacher-compatible reasoning matters more than answer correctness.

What would settle it

Run the same teacher-CoT warm-up with full-parameter SFT while sweeping learning rates across at least an order of magnitude, including $5\times10^{-5}$, and trace ID and OOD curves through OPD. If any full-SFT checkpoint reaches the same ID-OOD Pareto frontier as the LoRA checkpoints, the paper's central training-recipe claim is falsified; if no full-SFT setting does, the low-rank constraint is doing the work.

Watch

Extended reading notes

Core claim

The central discovery is that the warm-up stage transfers a teacher-compatible thinking pattern rather than correct answers, and that the training recipe should be constrained accordingly. Using rollouts that include chain-of-thought from the OPD teacher consistently beats using CoT from a much stronger external model, and paired correct and wrong teacher rollouts produce nearly identical SFT and OPD trajectories, with final scores within one point of each other. Full-parameter SFT warm-up accelerates in-domain convergence but permanently lowers out-of-domain performance, whereas LoRA warm-up reaches comparable in-domain performance while keeping OOD scores substantially higher, with a smaller rank (16) doing best. The recommended point is a near-saturation LoRA warm-up: enough steps to learn the teacher-aligned pattern, not so many that in-domain overfitting erodes generalization. The resulting Simple-OPD recipe—teacher CoT, LoRA, near-saturation duration, then standard OPD—improves ID performance consistently while leaving average OOD performance essentially unchanged.

Load-bearing premise

The training-recipe claim assumes that the comparison between LoRA and full-parameter SFT is fair, but the paper's own hyperparameter table reports different learning rates (LoRA at $5\times10^{-5}$, full SFT at $5\times10^{-6}$), so the conclusion that the low-rank parameterization—not the optimization budget—causes the better ID/OOD balance could fail if a differently tuned full SFT matched it.

Editorial extensions

If this is right

  • Warm-up data for OPD should be sampled from the OPD teacher's own rollouts with CoT; buying CoT from a stronger external model is not just wasteful but harmful.
  • Distillation pipelines can skip filtering warm-up rollouts for correctness, since wrong teacher rollouts with intact reasoning structure give nearly the same boost.
  • A low-rank LoRA warm-up (rank 16 to 32, trained to near saturation) is a cheap and Pareto-better alternative to full-parameter SFT warm-up.
  • Simple-OPD improves in-domain reasoning under vanilla OPD, G-OPD, and PowerOPD, and in thinking and non-thinking settings, so it can be layered onto existing OPD objectives without changing them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An inference the authors leave implicit: if warm-up is really distributional imprinting, the same recipe should transfer to other token-level on-policy imitation tasks beyond math reasoning, such as code or tool use, wherever the teacher's reasoning trace is available.
  • The correctness-insensitivity result suggests that answer-accuracy filtering in distillation data curation may be less valuable than style- and compatibility-filtering; a testable extension is whether systematically biased wrong rollouts (for example, all failing at the same reasoning step) eventually break the student, unlike the locally diverse errors studied here.
  • The authors' implicit claim is that warm-up acts as an initialization in function space, not just parameter space; this predicts that the overlap ratio between student and teacher top-k token sets should jump immediately after warm-up and stay high, which the reported overlap curve already hints at and could be measured precisely.
  • Because the main experiments use a math-reasoning curriculum, an open question is whether near-saturation LoRA warm-up remains optimal when the teacher's CoT is much longer or multi-step; the answer may depend on a rank-to-duration tradeoff not explored in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper studies the warm-up stage preceding on-policy distillation (OPD). Through ablations on the Qwen3-1.7B student / Qwen3-8B teacher setup, it reports three data-side findings (CoT supervision is essential, CoT from the OPD teacher is better than from a stronger external model, and incorrect teacher rollouts are nearly as useful as correct ones) and a training-side finding (LoRA warm-up gives a better ID/OOD balance than full-parameter SFT, with a near-saturation duration preferred). It then proposes Simple-OPD, which warms up the student on teacher-generated CoT with LoRA before the normal OPD stage, and evaluates this recipe across three OPD objectives, thinking and non-thinking models, and a same-size teacher-student consolidation setup.

Significance. If the central claims hold, Simple-OPD is a practical, plug-and-play initialization recipe that could improve OPD training in a variety of settings without modifying the distillation objective. The paper's strengths are its systematic ablation of the warm-up data and training choices, its coverage of multiple OPD objectives and model configurations, and its detailed appendix with benchmark-level results and example rollouts. The code release and the transparent reporting of hyperparameters are also valuable. However, the headline training-recipe claim rests on a comparison that is not fully controlled, and the robustness claims would be stronger with multiple seeds; these issues affect the confidence one can place in the specific design recommendation.

major comments (4)
  1. [§4.1, Appendix A, Table 3] The central claim that LoRA warm-up balances ID adaptation and OOD generalization better than full-parameter SFT is based on a comparison in which the two methods use different learning rates (full FT uses 5e-6, LoRA uses 5e-5) and different parameterizations. Since full-parameter fine-tuning and LoRA have very different numbers of trainable parameters and optimization geometry, the observed differences in ID/OOD trajectories could be due to the learning rate or effective update magnitude rather than to the parameterization itself. Please add a learning-rate sweep for both methods (or otherwise match the effective update scale, e.g., by equivalent step sizes or update norms) and report the resulting ID/OOD trajectories. Without this, the design choice of LoRA over full SFT is not identified from the presented experiments.
  2. [§4 (OOD evaluation) and Tables 1 and 2] Section 4 states that the OOD result is the average over six benchmarks, listing IFEval, GPQA-Diamond, HumanEval, and the Chemistry, Physics, and History subsets of MMLU-Pro. However, Tables 1 and 2 report only five OOD benchmarks (IFEval, GPQA-Diamond, Physics, Chemistry, History) and do not show HumanEval in the average. This is an inconsistency in the reported evaluation protocol. Please clarify whether HumanEval was evaluated and included in the OOD averages or correct the stated count, since the claim of "preserving overall OOD generalization" depends on which benchmarks are actually in the aggregate.
  3. [§5.1, §5.2, §5.3 (Tables 1, 2, 6) and §4.1 (Figure 5)] All reported results appear to come from single runs, with no error bars, confidence intervals, or multiple seeds. Differences as small as 0.95 points (PowerOPD in Table 1) and 1.35 points (vanilla OPD in Table 1) are used to claim consistent improvements, but without variance estimates these differences may not be significant. Please provide at least three seeds for the core comparisons, or explicitly state that results are single-run and temper the language accordingly.
  4. [§3.3 and Table 7] The conclusion that "incorrect teacher rollouts can provide comparable benefits" is supported by paired correct/wrong rollouts that are almost identical in reasoning structure, differing only in a local arithmetic error and its propagation. This is visible in Table 7. As presented, the finding supports the narrower claim that process-structurally similar incorrect traces still transfer the teacher's thinking pattern; it does not show that incorrectness per se is irrelevant. Please clarify how the paired incorrect rollouts were selected and whether the result holds for arbitrary incorrect teacher rollouts, or restrict the claim accordingly.
minor comments (3)
  1. [§4.2] The "near-saturation" warm-up duration is identified from the training curves without a formal criterion. Please state an operational rule (e.g., a loss plateau or a threshold on improvement) or at least acknowledge that the choice is heuristic.
  2. [Figures 2, 3, and 4] The multi-part bar figures contain a large amount of textual annotation that is difficult to parse at small print. Please consider separating the SFT and OPD trajectories into distinct panels or using a table for the numerical values.
  3. [§3.3] The text says that "a stronger SFT checkpoint does not necessarily provide a better initialization," but this claim is inferred from comparing different training configurations rather than from a controlled experiment that holds all else fixed. Please rephrase to reflect the correlational nature of the observation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Simple-OPD's claims are direct empirical comparisons against external benchmarks, not reductions to the paper's own definitions or fitted parameters.

full rationale

Simple-OPD is an empirical recipe paper. Its central claims—teacher CoT helps, teacher source matters more than answer correctness, LoRA warm-up preserves OOD better than full SFT, and a near-saturation warm-up is best—are established by controlled ablations on external benchmarks (MATH-500, AIME24/25, AMC23, IFEval, GPQA, MMLU-Pro, HumanEval) and are reported as measured trajectories, not derived from an equation that already contains the outcome. The formal definitions in Sec. 2 (SFT loss, OPD reverse-KL, LoRA update) are standard and none is defined in terms of the target result. The only self-citation, Yang et al. (2026b), provides the teacher checkpoint for the experimental setup; it is an externally reproducible artifact and does not carry the paper's conclusions, so it is not load-bearing. The LoRA-versus-full-SFT comparison uses different learning rates (Appendix A, Table 3), which is an experimental confound and a correctness risk, but it is not a circular reduction: the comparison's outcome is not forced by the paper's equations or by self-citation. Similarly, the correct-versus-wrong-rollout ablation is an empirical manipulation, and the teacher-compatibility interpretation is separately supported by the GPT-5.5 comparison. No step in the paper's argument reduces by construction to its own input.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest mainly on domain assumptions about representative models, benchmarks, and training setups, plus one ad hoc recipe choice (near-saturation duration). No new scientific entities or fitted physical parameters are introduced.

free parameters (3)
  • Warm-up duration (steps) = 150 or 175 steps (near saturation)
    Selected post hoc from observed ID/OOD trade-off in Figure 6; Simple-OPD main experiments do not state the exact number of warm-up steps.
  • LoRA rank = 16, 32, or 64; unspecified for Simple-OPD
    Hand-chosen hyperparameter; Section 4.1 shows ranks 16-64 behave similarly, but the final method does not fix a rank for the Section 5 experiments.
  • LoRA learning rate = 5e-5
    Chosen manually and differs from full-parameter SFT learning rate (5e-6), creating a potential confound in the LoRA versus full SFT comparison.
assumptions (5)
  • standard math Token-level reverse KL to the teacher policy is an appropriate OPD objective (Eq. 2).
    This is the standard OPD objective from prior work (Gu et al. 2024; Agarwal et al. 2024), used to define the distillation stage.
  • domain assumption The Qwen3-0.6B/1.7B/8B and DeepSeek-1.5B setups are representative of OPD in general.
    The paper generalizes findings from two model families and math-focused benchmarks to OPD broadly; the authors acknowledge this in the Limitation section.
  • domain assumption The chosen benchmarks measure in-domain reasoning and out-of-domain generalization.
    MATH-500, AIME, and AMC are treated as in-domain; IFEval, GPQA, HumanEval, and MMLU-Pro subsets are treated as out-of-domain (Section 4.1).
  • domain assumption Randomly sampled teacher rollouts from the OPD training set are sufficient warm-up data.
    The SFT dataset is built by random sampling from the OPD training prompts with no filtering (Appendix A); no data selection is studied.
  • ad hoc to paper The 'near-saturation' warm-up duration is identifiable from training curves without a formal criterion.
    The recommendation is based on observed ID/OOD trajectories in Figure 6 rather than a principled stopping rule.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Simple-OPD: Demystifying Warm-up for On-policy Distillation." pith.science (2026). https://pith.science/paper/XKBBPEUZ

@misc{pith2026260806802,
  author       = {Pith},
  title        = {Pith review of: Simple-OPD: Demystifying Warm-up for On-policy Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKBBPEUZ}},
  note         = {Machine review of arXiv:2608.06802}
}
read the original abstract

On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models, but its effectiveness can depend strongly on the warm-up stage before OPD. In this paper, we demystify warm-up for OPD from both data and training perspectives. For data, we find that effective warm-up relies on teacher-compatible chain-of-thought supervision, and that even incorrect teacher rollouts can provide comparable benefits to correct ones. This suggests that warm-up primarily transfers a teacher-compatible thinking pattern rather than merely correct answers. For training, we show that low-rank adaptation (LoRA) with a near-saturation training duration better balances in-domain adaptation and out-of-distribution generalization than full-parameter SFT. Based on these findings, we propose Simple-OPD, a plug-and-play initialization method that warms up the student on teacher-generated CoT with LoRA before OPD. Experiments across diverse settings demonstrate the effectiveness and robustness of Simple-OPD.

Figures

Figures reproduced from arXiv: 2608.06802 by the authors.

Figure 1
Figure 1. Overview of vanilla OPD and Simple-OPD. Simple-OPD first warms up the student with teacher￾generated CoT using LoRA, followed by standard OPD. mainly as an initialization heuristic, where the stu￾dent is fine-tuned on teacher responses. In this way, the overlap between student and teacher is higher, leading to better OPD performance (Li et al., 2026). However, the role of warm-up remains underex￾plored, especially t… view at source ↗
Figure 2
Figure 2. Effect of CoT supervision during warm-up. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of CoT source, i.e., [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Effect of correctness for teacher’s rollouts. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparison of full SFT and LoRA warm-up during OPD. Full SFT accelerates in-domain convergence but [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Effect of warm-up duration with the LoRA rank fixed to 32. A short warm-up remains close to direct [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Performance with a same-size student-trained [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: OPD training dynamics with and without SFT warm-up. From left to right, we report training reward, [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Detailed performance of SFT with and without CoT supervision across different SFT checkpoints. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Performance comparison between SFT models trained with CoT responses generated by the OPD teacher [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison between SFT models trained with correct and incorrect CoT responses across [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: ID and OOD performance during OPD training under different SFT warm-up configurations. We [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 18 canonical work pages

  1. [1]

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In International Conference on Learning Representations, volume 2024, pages 21246--21263

    work page 2024

  2. [2]

    AI-MO. 2024 a . Aime 2024. https://huggingface.co/datasets/AI-MO/aimo-validation-aime

    work page 2024

  3. [3]

    AI-MO. 2024 b . Amc 2023. https://huggingface.co/datasets/AI-MO/aimo-validation-amc

    work page 2024

  4. [5]

    LMDeploy Contributors. 2023 a . Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy

    2023

  5. [6]

    OpenCompass Contributors. 2023 b . Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass

    2023

  6. [9]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In Proceedings of ICLR

    work page 2024

  7. [12]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  8. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3

    2022

Show all 65 references
  1. [14]

    Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggyu Lim, and Taesup Kim. 2026. Stable on-policy distillation through adaptive target reformulation. In Findings of the Association for Computational Linguistics: ACL 2026, pages 42217--42227

    2026

  2. [20]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  3. [22]

    Kevin Lu and Thinking Machines Lab. 2025. https://doi.org/10.64434/tml.20251026 On-policy distillation . Thinking Machines Lab: Connectionism. Https://thinkingmachines.ai/blog/on-policy-distillation

    2025 doi

  4. [25]

    OpenAI . 2026. https://deploymentsafety.openai.com/gpt-5-5/gpt-5-5.pdf GPT-5.5 System Card . Technical report, OpenAI

    2026

  5. [26]

    OpenCompass. 2025. Aime 2025. https://huggingface.co/datasets/opencompass/AIME2025

    2025

  6. [28]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  7. [29]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath : Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  8. [30]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    2024 arXiv

  9. [42]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, and 1 others. 2026. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222--113244

    2026

  10. [43]

    Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, and Robert E Tillman. 2026. Fast and effective on-policy distillation from reasoning prefixes. In Findings of the Association for Computational Linguistic...

    2026

  11. [47]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. ICLR , volume=

  12. [48]

    Mixture-of-Subspaces in Low-Rank Adaptation

    Wu, Taiqiang and Wang, Jiahao and Zhao, Zhe and Wong, Ngai. Mixture-of-Subspaces in Low-Rank Adaptation. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.450

    2024 doi

  13. [49]

    Proceedings of the 2016 conference on empirical methods in natural language processing , pages=

    Sequence-level knowledge distillation , author=. Proceedings of the 2016 conference on empirical methods in natural language processing , pages=

    2016

  14. [50]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Rethinking kullback-leibler divergence in knowledge distillation for large language models , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  15. [51]

    Proceedings of ICLR , year=

    MiniLLM: Knowledge Distillation of Large Language Models , author=. Proceedings of ICLR , year=

  16. [52]

    International Conference on Learning Representations , volume=

    On-policy distillation of language models: Learning from self-generated mistakes , author=. International Conference on Learning Representations , volume=

  17. [53]

    arXiv preprint arXiv:2602.12125 , year=

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation , author=. arXiv preprint arXiv:2602.12125 , year=

    arXiv

  18. [54]

    arXiv preprint arXiv:2603.07079 , year=

    Entropy-aware on-policy distillation of language models , author=. arXiv preprint arXiv:2603.07079 , year=

    arXiv

  19. [55]

    arXiv preprint arXiv:2605.06387 , year=

    Asymmetric on-policy distillation: Bridging exploitation and imitation at the token level , author=. arXiv preprint arXiv:2605.06387 , year=

    arXiv

  20. [56]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    Fast and effective on-policy distillation from reasoning prefixes , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    2026

  21. [57]

    arXiv preprint arXiv:2605.07804 , year=

    Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning , author=. arXiv preprint arXiv:2605.07804 , year=

    arXiv

  22. [58]

    arXiv preprint arXiv:2605.07865 , year=

    KL for a KL: On-Policy Distillation with Control Variate Baseline , author=. arXiv preprint arXiv:2605.07865 , year=

    arXiv

  23. [59]

    arXiv preprint arXiv:2604.08527 , year=

    Demystifying opd: Length inflation and stabilization strategies for large language models , author=. arXiv preprint arXiv:2604.08527 , year=

    arXiv

  24. [60]

    arXiv preprint arXiv:2603.25562 , year=

    Revisiting on-policy distillation: Empirical failure modes and simple fixes , author=. arXiv preprint arXiv:2603.25562 , year=

    arXiv

  25. [61]

    arXiv preprint arXiv:2604.13016 , year=

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe , author=. arXiv preprint arXiv:2604.13016 , year=

    arXiv

  26. [62]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  27. [63]

    arXiv preprint arXiv:2604.13010 , year=

    Lightning opd: Efficient post-training for large reasoning models with offline on-policy distillation , author=. arXiv preprint arXiv:2604.13010 , year=

    arXiv

  28. [64]

    arXiv preprint arXiv:2605.12483 , year=

    Beyond GRPO and on-policy distillation: An empirical sparse-to-dense reward principle for language-model post-training , author=. arXiv preprint arXiv:2605.12483 , year=

    arXiv

  29. [65]

    arXiv preprint arXiv:2605.31159 , year=

    Trust-Region Behavior Blending for On-Policy Distillation , author=. arXiv preprint arXiv:2605.31159 , year=

    arXiv

  30. [66]

    arXiv preprint arXiv:2606.09304 , year=

    SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling , author=. arXiv preprint arXiv:2606.09304 , year=

    arXiv

  31. [67]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , journal =

  32. [68]

    Advances in Neural Information Processing Systems , volume=

    Dapo: An open-source llm reinforcement learning system at scale , author=. Advances in Neural Information Processing Systems , volume=

  33. [69]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  34. [70]

    NeurIPS , year=

    Measuring Mathematical Problem Solving With the MATH Dataset , author=. NeurIPS , year=

  35. [71]

    2024 , howpublished =

    AI-MO , title =. 2024 , howpublished =

    2024

  36. [72]

    2025 , howpublished =

    OpenCompass , title =. 2025 , howpublished =

    2025

  37. [73]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    arXiv

  38. [74]

    First Conference on Language Modeling , year=

    Gpqa: A graduate-level google-proof q&a benchmark , author=. First Conference on Language Modeling , year=

  39. [75]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

    arXiv

  40. [76]

    arXiv preprint arXiv:2406.01574 , year=

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. arXiv preprint arXiv:2406.01574 , year=

    arXiv

  41. [77]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  42. [78]

    OpenCompass: A Universal Evaluation Platform for Foundation Models , author=

  43. [79]

    LMDeploy: A Toolkit for Compressing, Deploying, and Serving LLM , author=

  44. [80]

    arXiv preprint arXiv:2605.27028 , year=

    Less is more: Early stopping rollout for on-policy distillation , author=. arXiv preprint arXiv:2605.27028 , year=

    arXiv

  45. [81]

    arXiv preprint arXiv:2603.11137 , year=

    Scaling reasoning efficiently via relaxed on-policy distillation , author=. arXiv preprint arXiv:2603.11137 , year=

  46. [82]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    Stable on-policy distillation through adaptive target reformulation , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    2026

  47. [83]

    arXiv preprint arXiv:2603.24472 , year=

    Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? , author=. arXiv preprint arXiv:2603.24472 , year=

    arXiv

  48. [84]

    arXiv preprint arXiv:2607.05394 , year=

    Weak-to-Strong Generalization via Direct On-Policy Distillation , author=. arXiv preprint arXiv:2607.05394 , year=

    arXiv

  49. [85]

    arXiv preprint arXiv:2607.05184 , year=

    Rethinking On-Policy Self-Distillation for Thinking Models , author=. arXiv preprint arXiv:2607.05184 , year=

    arXiv

  50. [86]

    Thinking Machines Lab: Connectionism , year =

    Kevin Lu and Thinking Machines Lab , title =. Thinking Machines Lab: Connectionism , year =

  51. [87]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  52. [88]

    arXiv preprint arXiv:2606.17199 , year=

    PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation , author=. arXiv preprint arXiv:2606.17199 , year=

    arXiv

  53. [89]

    arXiv preprint arXiv:2512.16649 , year=

    Justrl: Scaling a 1.5 b llm with a simple rl recipe , author=. arXiv preprint arXiv:2512.16649 , year=

  54. [90]

    arXiv preprint arXiv:2606.00755 , year=

    Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning , author=. arXiv preprint arXiv:2606.00755 , year=

    arXiv

  55. [91]

    arXiv preprint arXiv:2601.02780 , year=

    Mimo-v2-flash technical report , author=. arXiv preprint arXiv:2601.02780 , year=

    arXiv

  56. [92]

    arXiv preprint arXiv:2606.19348 , year=

    Deepseek-v4: Towards highly efficient million-token context intelligence , author=. arXiv preprint arXiv:2606.19348 , year=

  57. [93]

    2026 , month = apr, url =

    2026

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.