Pith. sign in

REVIEW 3 major objections 5 minor 25 references

$\beta$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

T0 review · 3 major / 5 minor · reviewed 2026-07-31 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vanilla on-policy self-distillation is just the β=1 case of a KL-regularized family whose optimum is a geometric mix of reference and teacher—and training toward that mix beats direct teacher imitation.

desk verdict Useful OPSD stabilization recipe with clean KL-family framing; the PO-to-distillation story is partly heuristic because the practical target is a local logit mix, not the trajectory optimum. read the letter →

arxiv 2607.28582 v1 pith:ZHQL7OFG submitted 2026-07-30 cs.LG

Jiawei Xu , Minghui Liu , Juzheng Zhang , Tom Goldstein , Furong Huang This is my paper
classification cs.LG
keywords on-policyself-distillationKL-regularizedpolicyoptimizationgeometricinterpolationlogitmixingreturn-to-goreasoninglanguagemodelsmathematical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On-policy self-distillation improves reasoning models by having a student learn from its own rollouts under a privileged teacher, but it is often brittle. This paper argues the brittleness has a structural cause: standard OPSD is exactly the β=1 member of a broader KL-regularized policy-optimization family, where β controls how hard the student is pulled toward a reference policy versus the teacher. The optimal policy of that family is a geometric interpolation between reference and teacher. Instead of running expensive RL, the authors turn that closed-form optimum into a cheap distillation target by mixing token-level logits along a scheduled path from reference toward teacher, and they weight token updates with return-to-go so early tokens get credit for later mismatch. On math reasoning benchmarks with open models from 1.7B to 8B, the method consistently beats vanilla OPSD, with the largest gains on the smallest model. A sympathetic reader cares because the paper gives a single dial and a practical recipe that keep OPSD’s efficiency while making the target and the credit assignment match the underlying sequence-level objective.

What carries the argument

The β-OPSD optimal policy (geometric interpolant between reference and privileged teacher), realized in practice as scheduled token-level logit mixing plus return-to-go weights on the reverse-KL mismatch. It converts an expensive RL objective into a cheap distillation target while aligning token updates with the sequence-level loss.

What would settle it

Hold the return-to-go estimator and training budget fixed, and compare scheduled logit interpolants against an equally smooth but non-geometric curriculum (or against direct teacher targets); if the geometric/RL-derived path no longer wins on AIME/HMMT avg@12, the central equivalence-to-practice claim fails.

Watch

Extended reading notes

Core claim

Vanilla OPSD is precisely the β=1 member of a KL-regularized family whose reward is the teacher-to-reference log-ratio; the family’s optimal policy is the geometric interpolant π★_β ∝ π_ref^{1−1/β} p_T^{1/β}. Training the student toward scheduled token-level logit interpolants of that path, with return-to-go credit assignment, consistently improves optimization stability and math-reasoning performance over direct teacher imitation.

Load-bearing premise

That mixing and renormalizing logits independently at each token is a faithful enough stand-in for the true full-trajectory optimum that the claimed policy-optimization link, not just smoother curriculum and REINFORCE-style weighting, is what drives the gains.

Editorial extensions

If this is right

  • β becomes an explicit curriculum dial: early targets stay near the student, later targets admit more teacher guidance without an abrupt jump.
  • OPSD training can stay token-level and cheap while still targeting the closed-form solution of a sequence-level KL-regularized objective.
  • Return-to-go on teacher–student log-ratios gives an unbiased (when undiscounted) sequence-level reverse-KL gradient under on-policy sampling.
  • Gains should be largest where the student–teacher gap is large (e.g., smaller models), which the reported Qwen3-1.7B results match.
  • The same derive-with-PO / train-with-distillation pattern extends in principle beyond math reasoning whenever a privileged teacher and a reference policy are available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Adaptive or performance-triggered β schedules, rather than fixed linear ones, are a natural next control knob the ablations already hint at.
  • If the local logit mixture is the weak link, any tighter approximation to the sequence normalizer Z_β (or a learned residual) would be a direct test of how much of the theory is doing real work.
  • The same geometric path could be used only for guided sampling with importance weights, trading on-policy simplicity for denser teacher influence on hard prefixes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reinterprets vanilla on-policy self-distillation (OPSD) as the β=1 case of a KL-regularized policy-optimization family whose reward is the teacher-to-reference log-ratio. It derives the family’s closed-form optimum as a geometric interpolant π*_β ∝ π_ref^{1−1/β} p_T^{1/β}, then proposes β-OPSD: schedule β (equivalently a teacher weight w_k), replace the sequence-level target by per-prefix logit mixtures of a (stop-gradient) student/reference and a privileged teacher, and train with reverse-KL distillation using return-to-go (RTG) token weights. On Qwen3-1.7B/4B/8B math reasoning (AIME 2024/2025, HMMT 2025), β-OPSD improves average avg@12 over vanilla OPSD, with the largest gains at 1.7B; ablations attribute benefits to both the interpolant target and RTG credit assignment.

Significance. If the framing holds, the paper gives a clean, useful bridge between OPSD and KL-regularized RL: β becomes an explicit regularization/curriculum knob, the geometric path motivates scheduled targets, and RTG restores sequence-level credit without full RL machinery. The Propositions 2.1–2.2 and Appendix B.3 RTG unbiasedness argument are standard and internally consistent for the stated objectives; the practical algorithm is simple (logit mix + discounted RTG) and shows consistent average gains over vanilla OPSD, SFT, and GRPO on competition math. That combination—principled reinterpretation plus an efficient training recipe with ablations—is a meaningful contribution to on-policy distillation for reasoning LMs, even if the local target is only an approximation to the trajectory optimum.

major comments (3)
  1. [§2.2–2.3, Eqs. (5), (10)–(11)] §2.2–2.3, Eqs. (5) vs (10)–(11): The central claim that distillation “approximates the solution of expensive policy optimization” rests on identifying the training target with π*_β. Eq. (5) is a trajectory-level geometric mixture with global normalizer Z_β; Eqs. (10)–(11) are an autoregressive product of independently renormalized per-prefix logit mixtures. These are not the same object (a geometric mixture of sequence laws is not the product of geometric mixtures of conditionals). The manuscript should either (i) state this gap explicitly and reframe the logit mix as a principled local/curriculum surrogate rather than “the” RL-derived solution, or (ii) provide analysis/evidence that the local target is close enough to π*_β for the PO→distillation story to drive the gains. As written, the title/abstract overclaim the equivalence.
  2. [§4.3, Tables 2–3] Tables 2–3 and §4.3: Ablations show that scheduled interpolants beat teacher+RTG and that RTG beats local token KL under a fixed interpolant—useful, but they do not isolate whether gains come from approximating Eq. (5) versus generic target smoothing/curriculum and REINFORCE-style weighting. A load-bearing check would compare the β-path interpolant to non-geometric mixtures (e.g., linear probability mix, temperature annealing of the teacher only, or unmatched schedules) under matched RTG. Without that, the causal link “PO optimum → logit mix → gains” remains under-supported even if the method is empirically helpful.
  3. [Table 1, §4.2] Table 1: Main results are reported at the 100-step checkpoint of a 200-step schedule, and several cells favor vanilla OPSD on individual benchmarks (e.g., AIME 2024 on Qwen3-4B; HMMT 2025 on Qwen3-8B). The average wins are real, especially at 1.7B (+5.74), but “consistently outperforms” should be qualified. Please report the full checkpoint sweep (or justify 100-step a priori), include variance/seeds, and align the claim language with mixed per-benchmark outcomes so stability/robustness is not overstated relative to the table.
minor comments (5)
  1. [§2.1, §4.1, Eq. (22)] Notation drift: π_ref in the theory vs stop-gradient current student z_θ̄ in Eq. (22) and dynamic-student+fixed-teacher in §4.3.3. State early which reference is used in the main algorithm and that Prop. 2.2 still applies with a frozen copy at each step.
  2. [§4.1, Table 4] Default schedule w_start=0.5, w_end=0.8 and γ=0.99 are free hyperparameters; Table 4 shows schedule sensitivity. A short sensitivity paragraph or recommended default range would help reproducibility.
  3. [Figure 2] Figure 2 y-axes are rescaled per panel; note that in the caption so absolute differences are not misread across benchmarks.
  4. [Remark 4.1, Appendix D] Appendix D mixed-sampling results are interesting but dense; a one-sentence pointer in the main text that off-policy mixture sampling is comparable yet costlier is enough—consider moving full Table 6 detail fully to the appendix discussion.
  5. [§2.2, §4.3.2] Minor typos/spacing: “Locallogitrealization”, “Wenextisolatethecontribution”, and similar concatenated words in §2.2 and §4.3.2; clean for camera-ready.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: β-family equivalence and geometric optimum are algebraic identities from a deliberately chosen objective; empirical gains are on external benchmarks.

full rationale

The paper’s load-bearing derivation is self-contained and non-circular. Proposition 2.1 follows by expanding J_β after the authors deliberately set R = log(p_T/π_ref) so that β=1 cancels reference terms and recovers reverse KL to the teacher; that is an intentional embedding/generalization, not a fitted or smuggled identity presented as an independent discovery. Proposition 2.2 is the standard closed-form maximizer of a KL-regularized objective (geometric interpolant), proved in Appendix B.2 by completing the KL. The practical algorithm then openly approximates the intractable sequence-level π*_β by per-prefix logit mixtures (Eq. 10–11) and adds return-to-go weights; those are engineering surrogates with stated approximations, not predictions forced by construction from benchmark fits. Schedule endpoints (w_start, w_end) and γ are hyperparameters. Empirical claims are measured on external contest sets (AIME/HMMT) against baselines. No load-bearing self-citation uniqueness theorem, no fitted-input-called-prediction, and no renaming of a known empirical law as a derived optimum. The skeptic’s concern that the local logit product is not equal to the trajectory-level geometric mixture is a faithfulness/correctness issue about the approximation gap, not circularity of the derivation chain.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

Load-bearing content is mostly standard KL-regularized RL plus modeling choices for privileged-teacher distillation. Free knobs are the interpolation schedule and discount; axioms are support compatibility, the local-prefix surrogate for the sequence optimum, and the usual on-policy reverse-KL training setup. No new physical entities; ‘invented’ items are algorithmic constructs.

free parameters (4)
  • teacher-weight schedule endpoints w_start, w_end = default w_start=0.5, w_end=0.8 over K=200 steps
    Default 0.5→0.8 (and other linear schedules in Table 4) chosen by hand; performance depends on these endpoints and they are not derived from the closed form.
  • return-to-go discount γ = 0.99
    Practical stability knob; unbiasedness holds at γ=1, but experiments use γ=0.99 without a derived optimum.
  • training checkpoint used for main table = 100-step checkpoint
    Main avg@12 reported at step 100 of a 200-step budget; checkpoint choice affects claimed margins.
  • generation/eval sampling hyperparameters = as in App. C
    Train temp 1.1 / top-p 0.95 / top-k 20 and eval temp 0.6 / k=12 affect measured reasoning scores.
assumptions (5)
  • domain assumption Compatible support of π_ref and p_T so the geometric interpolant and Z_β are well-defined.
    Stated in Prop. 2.2; needed for the closed-form optimum.
  • ad hoc to paper Per-prefix softmax of mixed logits is an adequate local approximation to the sequence-level geometric optimum with global normalizer Z_β.
    §2.2 explicitly replaces intractable trajectory normalization by independent token-level normalization; central practical bridge.
  • domain assumption Privileged teacher p_T(·|x,c) with ground-truth solution c provides useful denser supervision than outcome-only rewards on student prefixes.
    Setup in §4.1 and throughout OPSD literature the method extends.
  • standard math Score-function / REINFORCE identities yield unbiased gradients of sequence reverse KL under on-policy sampling when γ=1 and the target is stop-grad.
    Appendix B.3; classical policy-gradient fact applied to token log-ratios.
  • ad hoc to paper Reference endpoint may be a stop-gradient current student (dynamic) while teacher endpoint is fixed initial privileged model.
    Default D+F construction in §4.3.3; ablation-selected implementation choice.
invented entities (2)
  • β-OPSD objective family J_β with teacher-to-reference log-ratio reward
    purpose: Unify vanilla OPSD with tunable KL-regularized policy optimization and define intermediate targets.
    Named formulation in §2.1; standard math objects repackaged as a method family.
  • Scheduled token-level logit-interpolant distillation target p̃_βk
    purpose: Cheaply realize the geometric path without sequence-level Z_β or full RL.
    Eq. 10–12; algorithmic surrogate, validated only via downstream ablations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of $\beta$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation." pith.science (2026). https://pith.science/paper/ZHQL7OFG

@misc{pith2026260728582,
  author       = {Pith},
  title        = {Pith review of: $\beta$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHQL7OFG}},
  note         = {Machine review of arXiv:2607.28582}
}
abstract

On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the $\beta=1$ member of a broader policy-optimization family, where $\beta$ weights the KL penalty anchoring the student to a reference policy. This equivalence turns $\beta$ from an implicit value fixed at one into a controllable regularization parameter, yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce $\beta$-OPSD and derive its optimal policy as a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing this objective with reinforcement learning, however, would be costly and high-variance. Rather than optimize the RL objective directly, we turn its closed-form solution into a distillation target. Each value of $\beta$ selects a target along the reference-to-teacher path, which we implement efficiently by mixing their token-level logits. In this way, inexpensive distillation approximates the solution of expensive policy optimization. Return-to-go credit assignment further aligns token updates with the sequence-level objective while retaining the simplicity of OPSD. Experiments on mathematical reasoning benchmarks show that $\beta$-OPSD consistently outperforms vanilla OPSD, improving optimization stability and downstream reasoning performance. Our results provide a principled route from self-distillation to policy optimization and back without sacrificing the efficiency that makes OPSD practical.

Figures

Figures reproduced from arXiv: 2607.28582 by the authors.

Figure 1
Figure 1. Overview of β-OPSD. Vanilla OPSD is the β = 1 case of a broader KL-regularized objective, whose optimal policy is a geometric interpolant between the reference policy and the privileged teacher. β-OPSD turns this optimal policy into a practical distillation target through scheduled logit interpolation, while retaining efficient token-level training with return-to-go credit assignment. * Equal contribution and joint … view at source ↗
Figure 2
Figure 2. Effect of interpolant reference choices. All variants use the same return-to-go estimator and fixed teacher weight wk = 1/βk = 0.5. We report avg@12. Each subplot uses a benchmark-specific y-axis range to highlight relative differences. The dynamic student + fixed teacher construction performs best across all benchmarks [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 18 linked inside Pith

  1. [1]

    Reinforcement learning from rich feedback with distributional dagger.arXiv preprint arXiv:2606.05152,

    Rishabh Agrawal, Jacob Fein-Ashley, and Paria Rashidinejad. Reinforcement learning from rich feedback with distributional dagger.arXiv preprint arXiv:2606.05152,

    arXiv

  2. [3]

    This proves the proposition. B.3. Proof of the Return-to-Go Gradient Estimator Proof. We prove that whenγ=1 , the return-to-go estimator in Eq. 18 gives an unbiased estimator of the sequence-level reverse-KL gradient. Proposition B.1(Unbiased sequence-level gradient estimator).Fix a training stepk and treat the interpolant target ˜pβk (· |x,c)as fixed dur...

    2024

  3. [4]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al

    URLhttps://arxiv.org/abs/2605.00674. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  4. [7]

    Stable on-policy distillation through adaptive target reformulation

    Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggyu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation. InFindings of the Association for Computational Linguistics: ACL 2026, pages 42217–42227,

    2026

  5. [8]

    Why does self-distillation (sometimes) degrade the reasoning capability of llms?arXiv preprint arXiv:2603.24472,

    Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms?arXiv preprint arXiv:2603.24472,

    arXiv

  6. [9]

    Rethinkingon-policydistillationoflargelanguagemodels: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, WenkaiYang,ZhiyuanLiu,etal. Rethinkingon-policydistillationoflargelanguagemodels: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

    arXiv

  7. [10]

    Self-distilled policy gradient.arXiv preprint arXiv:2606.04036,

    Yifeng Liu, Shiyuan Zhang, Yifan Zhang, and Quanquan Gu. Self-distilled policy gradient.arXiv preprint arXiv:2606.04036,

    arXiv

  8. [17]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    arXiv

Show all 25 references
  1. [18]

    Self-distilled rlvr.arXiv preprint arXiv:2604.03128,

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr.arXiv preprint arXiv:2604.03128,

    arXiv

  2. [19]

    Opsdl: On-policy self-distillation for long-context language models.arXiv preprint arXiv:2604.17535,

    Xinsen Zhang, Zhenkai Ding, Tianjun Pan, Run Yang, Chun Kang, Xue Xiong, and Jingnan Gu. Opsdl: On-policy self-distillation for long-context language models.arXiv preprint arXiv:2604.17535,

    arXiv

  3. [20]

    American invitational mathematics examination (aime) 2024,

    Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024,

    2024

  4. [21]

    American invitational mathematics examination (aime) 2025,

    Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025,

    2025

  5. [22]

    Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    arXiv

  6. [23]

    Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

    16 β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation DanielMZiegler, NisanStiennon, JeffreyWu, TomBBrown, AlecRadford, DarioAmodei, PaulChristiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,

    1909 arXiv

  7. [25]

    For each problem, we generate k=12 independent candidate solutions using temperature0.6, top-p sampling with p=0.95 , and top-k=50

    Decoding configuration.We use stochastic decoding with thinking mode enabled. For each problem, we generate k=12 independent candidate solutions using temperature0.6, top-p sampling with p=0.95 , and top-k=50 . The evaluation batch size is32. We setmax_new_tokens=0, correspond...

    2024

  8. [26]

    Given its additional dual-model decoding cost and the variance introduced by importance sampling, we use student on-policy sampling as the default implementation ofβ-OPSD

    However, no single proposal schedule dominates across all benchmarks, indicating that the off-policy variant is sensitive to both the mixture coefficient and the training checkpoint. Given its additional dual-model decoding cost and the variance introduced by importance sampli...

    2024

  9. [1992]

    Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling

    Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. InInternational Conference on Learning Representat...

    2025

  10. [2011]

    High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438,

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438,

    arXiv

  11. [2015]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    arXiv

  12. [2017]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  13. [2022]

    Reinforcement learning via self-distillation

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802,

    arXiv

  14. [2023]

    Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897,

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897,

    arXiv

  15. [2024]

    Anti- self-distillation for reasoning rl via pointwise mutual information.arXiv preprint arXiv:2605.11609,

    Guobin Shen, Xiang Cheng, Chenxiao Zhao, Lei Huang, Jindong Li, Dongcheng Zhao, and Xing Yu. Anti- self-distillation for reasoning rl via pointwise mutual information.arXiv preprint arXiv:2605.11609,

    arXiv

  16. [2025]

    Self-distillation zero: Self-revision turns binary rewards into dense supervision.arXiv preprint arXiv:2604.12002,

    Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and Sanjeev Arora. Self-distillation zero: Self-revision turns binary rewards into dense supervision.arXiv preprint arXiv:2604.12002,

    arXiv

  17. [2026]

    Unmasking on-policy distillation: Where it helps, where it hurts, and why.arXiv preprint arXiv:2605.10889,

    Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc NM Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, and Mehrdad Farajtabar. Unmasking on-policy distillation: Where it helps, where it hurts, and why.arXiv preprint arXiv:2605.10889,

    arXiv

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.