Pith. sign in

REVIEW 4 major objections 4 minor 34 references

SAF shows that stable advantage fusion—sparsifying, bounding, warming up, and annealing the teacher signal—avoids entropy collapse and improves aggregate accuracy across every tested model–domain setting.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:36 UTC pith:3K3DZYG5

load-bearing objection A practical, honestly-reported fix for entropy collapse in RLVR+OPD fusion; the gains are small and the key threshold is selected on the same benchmarks, but the work is worth a serious referee. the 4 major comments →

arxiv 2607.29209 v1 pith:3K3DZYG5 submitted 2026-07-31 cs.LG cs.AI

SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

Yifan Ding , Xincheng Wei , Yoshua Y. Li , Ziheng Li , Yuquan Lu , Siyu Zhang , Dongsheng Ma , Rongxiang Weng
show 2 more authors
Xunliang Cai Yun Chen
This is my paper
classification cs.LG cs.AI
keywords reinforcement learning with verifiable rewardson-policy distillationadvantage fusionentropy collapsemagnitude mismatchtemporal mismatchKL-triggered annealingpolicy entropy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that combining a verified, response-level reinforcement signal with dense token-level teacher feedback works only if the teacher term is separately controlled in scale and in time. It diagnoses two failures of fixed-coefficient fusion: token-level teacher advantages are unbounded and can dwarf the normalized verifier advantage, and full-strength teacher guidance outlives its usefulness once the student nears the teacher. The proposed solution, SAF, applies a four-stage transform to the teacher advantage alone—sparsify to the top-k tokens per response, compress with a bounded tanh, warm up from zero, then anneal the coefficient down—leaving the verifier advantage untouched. Across three student scales and two domains, SAF avoids the entropy collapse of fixed fusion and improves aggregate accuracy in every setting by 0.51–2.70%. If correct, this makes SAF a drop-in stabilizer for pipelines that want both sparse verified rewards and dense distillation.

Core claim

On the paper's own terms: adding GRPO's bounded, group-normalized response-level advantage to OPD's unbounded token-level teacher gap fails because of two miscalibrations. A few tokens carry OPD advantages an order of magnitude larger than the verifier signal and dominate the update (magnitude mismatch), while sustained full-strength teacher guidance collapses entropy early and caps accuracy (temporal mismatch). SAF replaces the single mixing coefficient with Atotal = A_GRPO + opd_coef(s)·scale(s)·A_OPD,tanh, where the OPD term is sparsified per sequence, bounded by tanh, ramped up with a KL-triggered early stop, and annealed over the remaining steps. The paper reports this avoids entropy co

What carries the argument

The load-bearing object is the fused advantage identity Atotal_i,t = A_GRPO_i + opd_coef(s)·scale(s)·A_OPD,tanh_i,t, built from four switchable stages applied only to the OPD branch: (1) per-sequence top-k% sparsification keeps only tokens whose absolute OPD advantage exceeds the sequence's (1−k%) quantile; (2) bounded tanh compression maps survivors into (−c, c) while preserving near-linear order; (3) a KL-triggered warm-up ramps scale(s) from 0 to 1 and stops early once (KL0−KLs)/KL0 ≥ δ; (4) linear annealing decays the coefficient from 1 to c_min over the remaining steps. Disabling all four stages recovers the fixed fusion that fails.

Load-bearing premise

The load-bearing premise is that the measured drop in student-to-teacher divergence reliably marks when full-strength teacher guidance stops helping; if that estimate is noisy or the 0.2 trigger is overfit to one configuration, the annealing schedule is miscalibrated and the gains could fail to transfer.

What would settle it

Run SAF on a held-out model-task pair with several early-stop thresholds (0.0, 0.1, 0.2, 0.3) and record the training entropy and final accuracy. If SAF's entropy is not higher than fixed fusion's, or its accuracy does not beat fixed fusion in every setting, or the early-stop fires at erratic steps across seeds, the temporal controller is not the mechanism behind the gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fixed-coefficient fusion fails from two distinct, coexisting miscalibrations, so no single tuning of the mixing coefficient can fix it; separate magnitude and temporal controls are needed.
  • SAF can be inserted into an existing GRPO+OPD loop as a drop-in replacement: no extra model, loss, or forward pass, just a re-weighting of the sampled teacher advantage.
  • If SAF's claim holds, practitioners can keep dense teacher feedback without sacrificing exploration: entropy stays at an intermediate level and final accuracy keeps improving late in training.
  • On mathematics, SAF's advantage over fixed fusion grows as the student shrinks (0.97% at the largest, 1.51% at the middle, 1.85% at the smallest), consistent with smaller students depending more on dense guidance once it is controlled.
  • The diagnostic finding that the lowest student-teacher divergence (closest imitation) coincides with the lowest accuracy implies that imitation quality is not the training objective; the verifiable reward is.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the sparsify-then-compress pattern is not specific to distillation; any dense token-level signal (for example a process reward or a token-level bonus) added to a bounded response-level advantage should be sparsified and bounded before fusion, because the failure mode is unbounded magnitude, not the teacher source.
  • Editorial inference: the KL-triggered warm-up embodies a general 'teacher guidance as a depleting resource' principle; a testable extension is replacing the fixed δ=0.2 with an online stopping rule that watches validation reward, which could remove the need to tune δ per domain.
  • Editorial inference: because SAF preserves more entropy than fixed fusion, it should compose well with explicit entropy-regularized RLVR variants; testing whether SAF's gains are additive with such regularizers would separate the entropy-preservation effect from the magnitude-control effect.
  • Editorial inference: the paper's monotone math margins across student scales suggest the temporal controller's value increases with student-teacher gap; a testable prediction is that SAF's advantage over fixed fusion is larger when the teacher is much stronger than the student and shrinks (but does not reverse) with closer teacher-student pairs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies the fusion of a response-level GRPO advantage with a token-level on-policy distillation (OPD) advantage in RLVR training. It identifies two failure modes of fixed-coefficient fusion: a magnitude mismatch, where token-level OPD advantages are heavy-tailed and can exceed the bounded GRPO advantage, and a temporal mismatch, where full-strength OPD guidance becomes harmful as the student converges. SAF is proposed as a four-stage pipeline applied only to the OPD advantage: per-sequence top-k sparsification, tanh compression, a KL-triggered warm-up, and linear annealing. Using Qwen3-1.7B/4B/8B students with a Qwen3-30B teacher, the paper reports that SAF improves aggregate scores over fixed-coefficient GRPO+OPD by 0.51–2.70% across all six model–domain settings and avoids entropy collapse.

Significance. If the empirical results are reliable, SAF is a useful, low-overhead stabilizer for RLVR+OPD pipelines. The diagnosis of magnitude and temporal miscalibration is clearly articulated, and the training-dynamics plots support the entropy-collapse mechanism. The four-stage design is simple, switchable, and introduces no auxiliary model or loss, which are strengths. However, the central empirical claim currently rests on single runs without error bars, and the key temporal-controller hyperparameter δ is selected on the same benchmarks used for the headline results. The reported margins are small enough that statistical noise and selection bias are plausible alternative explanations. The significance is therefore conditional on additional robustness evidence.

major comments (4)
  1. [§4.2, Table 1] No seed variance, confidence intervals, or statistical tests are reported. Per-benchmark differences are often below 1% (e.g., Qwen3-1.7B code: 51.39 vs 50.88), and the largest aggregate gain is 2.70%. Since the central claim is consistent improvement in all six settings, single runs are insufficient to separate the reported margins from run-to-run noise. Please report at least three seeds per configuration, with means and standard deviations or confidence intervals, and a significance test for the aggregate comparisons.
  2. [§4.3, Table 2; §3.3] The temporal controller's threshold δ=0.2 is selected using the same Qwen3-4B mathematics benchmark that appears in Table 1. No validation split is used. The ablation shows a sharp dependence: SAF with δ=0.2 beats fixed fusion by 1.51% (45.89 vs 44.38), while δ=0.3 reduces the margin to 0.13% (44.51 vs 44.38). This means the reported main result depends on a hyperparameter that is neither robustly demonstrated across settings nor protected against selection bias. Please provide δ (and ideally k, c, S_warmup, c_min) sensitivity for all six model–domain settings, or use a held-out validation set for hyperparameter selection.
  3. [Appendix C, Eqs. (9)–(11); §3.3] The KL-trigger uses a clipped sampled-token estimator of the reverse KL. Its finite-sample noise propagates directly into the warm-up endpoint s_w, which determines the annealing schedule in Eq. (3). The paper does not report the variance of the KL estimate or the distribution of s_w across runs or settings. If the trigger is noisy, the entropy-stability and accuracy benefits may not transfer even when δ is held fixed. Please provide evidence on the stability of the trigger, e.g., the observed KL trajectory and s_w values across seeds.
  4. [§1, §4.4] The paper claims that the two mismatches 'cannot be resolved by simply retuning that coefficient,' but the only fixed-coefficient baseline considered is λ=1 in A_total = A_GRPO + λ·A_OPD. A grid over λ (with all SAF controls disabled) is needed to support this claim. If some intermediate λ, e.g., λ=0.3, already avoids entropy collapse and matches SAF's final accuracy, then the temporal controller's added value is not established. This is load-bearing for the motivation and should be tested.
minor comments (4)
  1. [§4.1, Tables 3–4] OPD-only uses a separate configuration (batch size 1024, 100/50 steps) from the GRPO-based methods (batch size 128, 300/200 steps). OPD-only results are therefore not directly comparable to SAF; this is acceptable for the main fixed-fusion comparison but should be stated more prominently.
  2. [§4.2, Table 1] The 'aggregate score' is the unweighted mean over six model–domain averages, each of which combines four math or three code benchmarks with different metrics (average accuracy over 32 samples vs pass@1 over 4 samples). Reporting per-benchmark averages and the aggregation formula explicitly would improve transparency.
  3. [Appendix B, Figures 5–7] The parameter-space update geometry is clearly labeled as exploratory and not used to design SAF; this is appreciated. In Figure 5, panels (b) and (c) are plotted over steps 20–100 while panel (a) extends to 300; a note in the caption that the overlap panels are restricted to the first 100 steps would prevent confusion.
  4. [Eq. (4) and Figure 2] The fusion rule is written as opd_coef(s)·scale(s)·A_OPD,tanh in Eq. (4), which is consistent with Figure 2. However, the warm-up stage in §3.3 says scale(s) is 'freezing at its current value' after the KL trigger; specifying whether the frozen value is min(s_w/S_warmup, 1) or the raw pre-min value would remove ambiguity.

Circularity Check

0 steps flagged

No significant circularity: the fused-advantage construction is not defined in terms of the benchmark outcomes it claims, and the reported gains are empirical measurements rather than predictions forced by the fit.

full rationale

I walked the derivation chain from Eq. (1) through Eq. (4). The OPD advantage A_OPD is defined as a token-level log-probability gap; the fused advantage (Eq. 4) is a composition of that quantity with a GRPO term and multiplicative magnitude/temporal controllers. Nothing in that construction is defined in terms of the AIME/HMMT/MBPP/LiveCodeBench scores used to support the headline improvement. The temporal controller's early-stop criterion (KL0-KLs)/KL0 >= delta depends on online student-teacher KL statistics and step counts, not on benchmark accuracies, so the entropy-stability and accuracy claims are not equivalent to the controller's input by construction. The one methodological concern is that delta=0.2 is selected in Table 2 on Qwen3-4B mathematical reasoning and then reused in the same setting in Table 1; this is a validation/protocol weakness that could inflate the reported margin, but it is not a circular derivation: Table 2 itself shows delta=0.3 produces a different, smaller margin, so the improvement is not statistically forced by the fitted threshold. No load-bearing self-citation appears: the only overlapping-author citation (Lu et al. 2026) is a related-work contrast, not the argument supporting SAF's design or results. No uniqueness theorem, imported ansatz, or renamed known result is used as the load-bearing step. The central claim therefore has independent empirical content, and the paper is best scored as having no significant circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on two ad hoc design assumptions: that low-magnitude OPD tokens are noise, and that a sampled-KL drop of δ marks the end of useful teacher guidance. The first is built into Stage 1; the second is the temporal early-stop in Stage 3. No new physical or architectural entities are introduced; SAF is a transform of existing advantage signals.

free parameters (5)
  • k (topk_percent) = 20%
    Per-sequence retention ratio in Stage 1; hand-set, no sensitivity study reported.
  • c (tanh compression coefficient) = 0.1
    Bounds OPD advantage contribution in Stage 2; hand-set without sweep.
  • S_warmup = 100
    Maximum warm-up steps in Stage 3; hand-set, interacts with the KL trigger.
  • δ (relative KL-drop threshold) = 0.2
    Early-stop threshold for warm-up; selected by comparing δ=0.3 on the same Qwen3-4B math benchmarks (Table 2), so it is fitted to the evaluation target.
  • c_min (annealing floor) = 0.0
    Residual OPD coefficient at the end of annealing in Stage 4; hand-set.
axioms (5)
  • domain assumption A^OPD_i,t = log π_T(y_i,t|s_i,t) − log π_θ(y_i,t|s_i,t) is a valid token-level advantage and can be added to the GRPO advantage.
    Eq. 1, §3.1; grounded in cited G-OPD equivalence; if the equivalence fails, the fused update is not the KL-constrained RL objective the paper assumes.
  • standard math The k3 sampled estimator D_i,t = clip(exp(d_i,t)−d_i,t−1, −10, 10) estimates the reverse KL D_KL(π_θ||π_T) on student-generated tokens.
    Appendix C, Eq. 10; unbiased without clipping, but the clips at ±10 and the sampled-token mean introduce bias the paper does not quantify.
  • ad hoc to paper Zeroing low-magnitude OPD tokens destroys no useful learning signal.
    Stage 1 top-k filter assumes per-sequence quantile τ_k(i) separates noise from salience; no ablation removes only this assumption.
  • ad hoc to paper A relative decrease of δ in sampled reverse KL indicates that full-strength teacher guidance has become harmful.
    Stage 3 early-stop in §3.3; this is the paper's weakest load-bearing premise because δ is chosen on the evaluation set.
  • domain assumption The teacher Qwen3-30B-A3B-Instruct-2507 is a sufficiently strong and reliable supervisor for base students at 1.7B–8B scale.
    Used throughout; the paper itself argues the teacher is fallible, so the signal's usefulness is assumed to persist through early training.

pith-pipeline@v1.3.0-daily-deepseek · 25744 in / 13608 out tokens · 136264 ms · 2026-08-03T11:36:48.083818+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 28 linked inside Pith

  1. [3]

    doi: 10.1038/s41586-025-09422-z

    ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL http://dx.doi.org/10.1038/s41586-025-09422-z. Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes,

    doi:10.1038/s41586-025-09422-z

  2. [4]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang

    URL https://arxiv.org/abs/2306.13649. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. InThe Twelfth International Conference on Learning Representations,

    Pith/arXiv arXiv

  3. [6]

    RuiWang,HongruWang,YiChen,BoyangXue,TianqingFang,WenhaoYu,andKam-FaiWong

    URLhttps://arxiv.org/abs/2601.02780. RuiWang,HongruWang,YiChen,BoyangXue,TianqingFang,WenhaoYu,andKam-FaiWong. Demystifyingon- policy distillation: Roles, pathologies, and regulations, 2026b. URLhttps://arxiv.org/abs/2607.13399. Xingrun Xing, Haoqing Wang, Boyan Gao, Ziheng Li, and Yehui Tang. Trust region on-policy distillation,

    Pith/arXiv arXiv

  4. [7]

    Anhao Zhao, Junlong Tong, Yingqi Fan, Ping Nie, Wenjie Li, and Xiaoyu Shen

    URLhttps://arxiv.org/abs/2606.01249. Anhao Zhao, Junlong Tong, Yingqi Fan, Ping Nie, Wenjie Li, and Xiaoyu Shen. Poweropd: Stabilizing on-policy distillation with bounded power transformation,

    Pith/arXiv arXiv

  5. [8]

    Feng Luo, Yu-Neng Chuang, Guanchu Wang, Zicheng Xu, Xiaotian Han, Tianyi Zhang, and Vladimir Braverman

    URLhttps://arxiv.org/abs/2606.17199. Feng Luo, Yu-Neng Chuang, Guanchu Wang, Zicheng Xu, Xiaotian Han, Tianyi Zhang, and Vladimir Braverman. Demystifying opd: Length inflation and stabilization strategies for large language models,

    Pith/arXiv arXiv

  6. [9]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    URL https://arxiv.org/abs/2604.08527. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms,

    Pith/arXiv arXiv

  7. [11]

    10 SAF-OPD Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al

    URL https://arxiv.org/abs/2503.14476. 10 SAF-OPD Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization,

    Pith/arXiv arXiv

  8. [12]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe

    URLhttps://arxiv.org/abs/2507.18071. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step,

    Pith/arXiv arXiv

  9. [14]

    org/abs/2312.08935

    URLhttps://arxiv. org/abs/2312.08935. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network,

    Pith/arXiv arXiv

  10. [17]

    Bowen Zhang

    URL https://arxiv.org/abs/2604.00626. Bowen Zhang. A formula-driven survey and research agenda for on-policy distillation,

    Pith/arXiv arXiv

  11. [18]

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun

    URLhttps: //arxiv.org/abs/2606.22793. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models,

    Pith/arXiv arXiv

  12. [19]

    Qiyong Zhong, Mao Zheng, Mingyang Song, Xin Lin, Jie Sun, Houcheng Jiang, Xiang Wang, and Junfeng Fang

    URLhttps://arxiv.org/abs/2402.03898. Qiyong Zhong, Mao Zheng, Mingyang Song, Xin Lin, Jie Sun, Houcheng Jiang, Xiang Wang, and Junfeng Fang. Sod: Step-wise on-policy distillation for small language model agents.arXiv preprint arXiv:2605.07725,

    Pith/arXiv arXiv

  13. [20]

    Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron

    URLhttps://arxiv.org/ abs/2603.07079. Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation,

    Pith/arXiv arXiv

  14. [21]

    Shenzhi Yang, Guangcheng Zhu, Bowen Song, Haobo Wang, Mingxuan Xia, Xing Zheng, Yingfan Ma, Zhongqi Chen, Weiqiang Wang, Junbo Zhao, and Gang Chen

    URLhttps://arxiv.org/abs/2603.11137. Shenzhi Yang, Guangcheng Zhu, Bowen Song, Haobo Wang, Mingxuan Xia, Xing Zheng, Yingfan Ma, Zhongqi Chen, Weiqiang Wang, Junbo Zhao, and Gang Chen. Oprd: On-policy representation distillation, 2026b. URLhttps://arxiv.org/abs/2606.06021. Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, and Yohan Jo. Kl for a kl: On-pol...

    arXiv

  15. [22]

    Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, and Fei Mi

    URLhttps://arxiv.org/abs/2605.07865. Hongling Xu, Qi Zhu, Heyuan Deng, Jinpeng Li, Lu Hou, Yasheng Wang, Lifeng Shang, Ruifeng Xu, and Fei Mi. Kdrl: Post-training reasoning llms via unified knowledge distillation and reinforcement learning,

    Pith/arXiv arXiv

  16. [23]

    Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan

    URL https://arxiv.org/abs/2506.02208. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr, 2026c. URLhttps://arxiv.org/abs/2604.03128. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yon...

    Pith/arXiv arXiv

  17. [24]

    QitaiTan, ZefangZong, YangLi, andPengChen

    URL https://arxiv.org/abs/2605.15155. QitaiTan, ZefangZong, YangLi, andPengChen. Atod: Annealedturn-awareon-policydistillationformulti-turn autonomous agents,

    Pith/arXiv arXiv

  18. [25]

    Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, and Lijie Wen

    URLhttps://arxiv.org/abs/2606.27814. Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, and Lijie Wen. Rlcsd: Reinforcement learning with contrastive on-policy self-distillation,

    Pith/arXiv arXiv

  19. [26]

    Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Zheng Pan, Xin Li, and Lan-Zhe Guo

    URLhttps://arxiv.org/ abs/2606.11709. Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Zheng Pan, Xin Li, and Lan-Zhe Guo. Trace: Distilling where it matters via token-routed self on-policy alignment, 2026c. URLhttps://arxiv.org/abs/2605. 10194. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan ...

    Pith/arXiv arXiv

  20. [27]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al

    doi: 10.1145/3689031.3696075. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report,

    arXiv

  21. [28]

    11 SAF-OPD Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al

    URLhttps://arxiv.org/abs/2505.09388. 11 SAF-OPD Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456,

    Pith/arXiv arXiv

  22. [29]

    Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025b

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025b. AI-MO. Aime 2024.https://huggingface.co/datasets/AI-MO/aimo-validation-aime,

    Pith/arXiv arXiv 2024

  23. [30]

    Aime 2025.https://huggingface.co/datasets/opencompass/AIME2025,

    OpenCompass. Aime 2025.https://huggingface.co/datasets/opencompass/AIME2025,

    2025

  24. [31]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Armando Solar-Lezama, Koushik Sen, and Ion Stoica

    URLhttps://openreview.net/forum?id=1qvx610Cu7. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Pith/arXiv arXiv

  25. [32]

    12 SAF-OPD A Detailed Experiment Settings We use Qwen3-30B-A3B-Instruct-2507 as the teacher and initialize the student from Qwen3-8B, Qwen3-4B, or Qwen3-1.7B

    URLhttps://arxiv.org/abs/2605.11739. 12 SAF-OPD A Detailed Experiment Settings We use Qwen3-30B-A3B-Instruct-2507 as the teacher and initialize the student from Qwen3-8B, Qwen3-4B, or Qwen3-1.7B. Math training uses the filtered DeepMath split containing 57K problems of difficulty level at least 6, while code training uses the 25K-problem Eurus-RL-Code dat...

    Pith/arXiv arXiv

  26. [33]

    write” directions), and (c) the cross-method overlap of their right singular subspaces (input/“read

    we sample four solutions per problem and report pass@1, i.e., a problem counts as solved if at least one sampled solution passes. Mathematical answers are checked with Math-Verify, while code submissions are scored by the benchmark-provided unit tests. These decoding and scoring settings are held fixed across Base, all baselines, and SAF. Computing infras...

    2026

  27. [34]

    panels, since GRPO-only, fixed fusion, and SAF are shown over their full 300-step budget while OPD-only is shown over its own full 100-step training horizon

    Each surface plots drift magnitude against transformer layer index and optimization step; note that the z-axis and step-axis ranges differ across 15 SAF-OPD 0 1 2 3 4 5 6 7 8 9 1011121314151617181920212223242526272829303132333435 Layer 50 100 150 200 250 300 Step 0.000 0.005 0.010 0.015 0.020 0.025 0.030 L2 Drift 0.0150 0.0175 0.0200 0.0225 0.0250 0.0275 ...

    2098

  28. [2015]

    Yoon Kim and Alexander M

    URL https://arxiv.org/abs/1503.02531. Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation,

    Pith/arXiv arXiv

  29. [2016]

    Mingyang Song and Mao Zheng

    URLhttps://arxiv.org/ abs/1606.07947. Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models,

    Pith/arXiv arXiv

  30. [2017]

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin

    URLhttps://arxiv.org/abs/1707.06347. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective,

    Pith/arXiv arXiv

  31. [2023]

    Peiyi Wang, Lei Li, Zhihong Shao, R

    URLhttps://arxiv.org/abs/ 2305.20050. Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math- shepherd: Verify and reinforce llms step-by-step without human annotations,

    Pith/arXiv arXiv

  32. [2024]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

    URLhttps://arxiv.org/abs/2402.03300. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Rua...

    Pith/arXiv arXiv

  33. [2025]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y

    URLhttps://arxiv.org/abs/2509.08827. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Pith/arXiv arXiv

  34. [2026]

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin

    URL https://arxiv.org/abs/2606.04036. Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026a. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize ...

    Pith/arXiv arXiv