Pith. sign in

REVIEW 3 major objections 6 minor 54 references

Cross-teacher on-policy distillation works once you subtract recurring style disagreement from the teacher signal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 07:00 UTC pith:QMMK7OFV

load-bearing objection Solid systems fix for cross-teacher OPD: residualize recurring teacher–reference disagreement and the offline gains come back; proxy is operational but the empirics hold up. the 3 major comments →

arxiv 2607.28449 v1 pith:QMMK7OFV submitted 2026-07-30 cs.CL

Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models

Yecheng Wu , Song Han , Han Cai This is my paper
classification cs.CL
keywords on-policy distillationcross-teacher OPDstyle biasstyle residualizationLightning OPDmathematical reasoningcode generationoffline distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

On-policy distillation gives a student dense token-level feedback from a teacher, but it usually works best when that teacher is the same model that wrote the supervised fine-tuning demonstrations. In practice that match is often missing: SFT data may come from mixed sources, or teams may want a strong generator for SFT and a different teacher for distillation. The paper shows that under this mismatch the raw teacher–reference disagreement mixes useful, problem-specific corrections with a recurring bias about wording, formatting, and reasoning cadence, so even a stronger teacher can barely beat the SFT baseline. Lightning OPD 2.0 estimates that recurring component from other cached rollouts with simple token and context lookup tables, subtracts it, and trains on the residual. On math and code benchmarks in two cross-teacher setups, the residualized update consistently beats uncorrected Lightning OPD and lets the SFT generator and distillation teacher be chosen independently.

Core claim

In cross-teacher OPD, raw teacher–reference disagreement contains both context-specific teacher evidence and a recurring component tied to style. Estimating that recurring component with rollout-level cross-fitted token and context lookups and subtracting it before the Lightning OPD update restores effective distillation without requiring the OPD teacher to match the SFT data generator.

What carries the argument

Cross-fitted style residualization: split cached rollouts into folds; on other folds build response-balanced lookup tables of mean teacher–reference disagreement by token identity and by coarse bins of normalized position and reference surprisal; average the two lookups as a style-bias proxy; subtract it from the raw disagreement and use the residual in the reference-anchored OPD advantage.

Load-bearing premise

Disagreement that can be predicted across unrelated rollouts from token identity and coarse position/surprisal bins is mostly harmful style bias, while what is left is useful problem-specific teacher signal.

What would settle it

If residualization were removed or replaced by random/in-sample corrections and cross-teacher Lightning OPD 2.0 no longer beat uncorrected Lightning OPD on AIME and LiveCodeBench—or if the residual moved farther from a teacher-consistent diagnostic signal rather than closer—the central claim would fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Teacher consistency is no longer required for effective offline OPD: SFT data generator and distillation teacher can be chosen independently.
  • Existing mixed-provenance SFT checkpoints can be improved with a stronger external OPD teacher without regenerating demonstrations.
  • Practitioners can keep a high-quality SFT reference while swapping in a preferred distillation teacher and still expect gains on math and code.
  • Competing OPD fixes aimed at position or teachability do not substitute for removing recurring cross-teacher style bias in these settings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same residualization idea may transfer to other teacher–student mismatches beyond style, such as length or format preferences in instruction following.
  • If lookup features were enriched (for example discourse-role tags) without leaking problem-specific reasoning, more bias could be removed with less residual noise.
  • Online OPD with a live mismatched teacher may benefit from the same cross-fitted correction if a replay buffer is large enough to support fold statistics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies cross-teacher on-policy distillation (OPD), where the OPD teacher differs from the model that generated the SFT demonstrations. It argues that raw teacher–reference log-probability disagreement mixes useful context-specific evidence with a recurring component tied to wording, formatting, and reasoning cadence, and that this recurring component can make even a stronger teacher unhelpful. Lightning OPD 2.0 estimates that component via rollout-level cross-fitting of response-balanced token-identity and coarse-context (normalized position × reference surprisal) lookup tables, subtracts the estimate from the disagreement, and applies the residual in the frozen-replay Lightning OPD objective (Eqs. 5–11, Algorithm 1). On math and code benchmarks in two cross-teacher settings (Qwen3-4B-SFT and Klear-Reasoner-8B-SFT, both with Qwen3-30B-A3B-Thinking-2507 as teacher), the method consistently beats uncorrected Lightning OPD and two adapted OPD baselines (Table 1), with supporting mechanism analysis against a diagnostic teacher-consistent signal (Figs. 3–4) and component ablations (Table 2).

Significance. If the result holds, the paper removes a practically important constraint on OPD pipelines: practitioners can keep a strong existing SFT reference and still distill from a separately chosen teacher, which matters when SFT provenance is mixed/unknown or when the best demonstration generator and best distillation teacher differ. The contribution is concrete and implementable—cross-fitted residualization on a frozen cache, fixed effective teacher scores, and a clear algorithm—rather than a purely conceptual reframing. Strengths include matched frozen-replay comparisons, an explicit operational (not semantic) framing of style bias, diagnostic evidence that residualization moves the signal toward a teacher-consistent reference, and ablations of the main design pieces. Code release is promised. The work is incremental relative to Lightning OPD but addresses a failure mode that the prior work itself identified as central.

major comments (3)
  1. [§3.2 Eqs. (5)–(7); §4.3–4.4] §3.2 Eqs. (5)–(7): The central modeling step treats disagreement predictable from token identity and coarse (position, reference-surprisal) bins as removable style-token bias, with the residual carrying useful teacher evidence. The authors correctly label this operational, but it remains load-bearing. Table 2 shows both lookups help, and §4.3 shows residualization reduces deviation from a diagnostic consistent teacher, which supports the proxy in these settings. Still, the manuscript should more directly test whether residualization discards shared reasoning preference (e.g., by reporting residual vs. removed mass correlated with answer correctness, or a controlled consistent-teacher control where residualization should be near-neutral). Without that, the claim that the residual is preferentially “reasoning evidence” is only indirectly supported.
  2. [Table 1; §4.1 Evaluation] Table 1: All Pass@1 numbers are point estimates with no seed/run variance, confidence intervals, or repeated training. Several headline margins are modest (e.g., Klear math Avg. 73.6 → 74.6; code gains of ~1.4 over Lightning OPD). Given temperature sampling (0.6) and multi-sample Pass@1, evaluation noise alone could move some comparisons. At minimum, report evaluation uncertainty (bootstrap over problems or repeated decoding) and, ideally, 2–3 training seeds for the main Lightning OPD vs. 2.0 comparison so the “consistently outperforms” claim is statistically grounded.
  3. [§4.1 Baselines; Table 1] §4.1 / Table 1 baselines: IW-OPD and TA-OPD are adapted from online formulations to frozen replay. That is reasonable for control, but the paper should state more carefully what is preserved vs. lost in the adaptation (especially TA-OPD’s “teachability” predictor and IW-OPD’s accumulated-discrepancy weighting when rollouts are fixed). If the adaptations weaken those methods, the relative ranking overstates Lightning OPD 2.0’s advantage over the broader OPD literature. A short appendix describing the exact adapted objectives would resolve this.
minor comments (6)
  1. [§3.3; §4.1] Free parameters (K=5 folds, equal 1/2–1/2 lookup weights, B_pos/B_ref/ξ_max, rare-group smoothing) are listed in the method but not sensitivity-tested beyond the three ablations in Table 2. A brief sensitivity note on K and bin granularity would strengthen reproducibility.
  2. [Figure 1] Figure 1 caption and bars are useful, but the y-axis breaks and dual panels make effect sizes harder to read than Table 1; consider aligning scales or annotating deltas.
  3. [§4.3] §4.3: For Klear-Reasoner-8B-SFT the diagnostic uses Klear-Reasoner-8B as a proxy rather than the true demonstration generator (DeepSeek-R1-0528). This is disclosed, but the text should stress that Fig. 3 (right) is weaker evidence than the Qwen3-4B panel where the diagnostic is exact.
  4. [Figure 2] Notation: ℓ vs ω appear in figure LaTeX fragments; keep a single symbol for log-probabilities in figures and main text.
  5. [§5; §3.1] Conclusion correctly limits scope to math/code and Qwen-family tokenization; a sentence on whether incompatible tokenizers break the cache alignment assumption would help practitioners.
  6. [§2; §3.1; §4.1; §4.3] Typos/style: “Post-T raining”, “Cross-T eacher”, “Style-T oken” in headings look like line-break artifacts; “T raining Settings” similarly.

Circularity Check

0 steps flagged

No significant circularity: residualization is an operational preprocessing step; student gains are external benchmark results, not reconstructions of the fit.

full rationale

Lightning OPD 2.0 defines an operational decomposition d_it = b(z_it) + v_it and estimates b via cross-fitted token/context lookups, then subtracts it before the Lightning OPD update. That construction makes the subtracted term equal to cross-rollout-predictable disagreement by design, but the authors explicitly label this an operational proxy rather than a semantic or first-principles derivation of style, and they do not present the lookup fit itself as a scientific prediction to be verified. The load-bearing claims—improved Pass@1 on AIME, HMMT, and LiveCodeBench versus uncorrected Lightning OPD and other OPD baselines—are evaluated on held-out external benchmarks after training the student, not on recovering the fitted b. Cross-fitting only prevents self-fitting of the correction tables; it does not force downstream accuracy. Self-citation of Lightning OPD (Wu et al., 2026) supplies the offline OPD substrate and is not used as a uniqueness theorem or load-bearing external fact that closes the argument. No fitted parameter is renamed as an independent prediction of the same quantity, and no ansatz is smuggled in via citation. The derivation chain is therefore self-contained engineering plus empirical evaluation, not circular.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

Empirical ML methods paper. Load-bearing commitments are modeling choices about what recurring disagreement means, frozen on-policy replay from the SFT reference, compatible tokenization, and several estimation hyperparameters—not physical axioms. No new particles or forces; invented entities are operational constructs (style-token bias proxy, lookup residualization).

free parameters (5)
  • Number of cross-fitting folds K = 5
    Set to 5 prompt-level folds; affects bias/variance of style estimates and is not derived.
  • Equal weights on token vs context lookups (1/2, 1/2) = 0.5 / 0.5
    Eq. 7 fixes equal averaging without learned or validated optimal mix; ablation shows each alone is weaker but weights are hand-chosen.
  • Position and surprisal bin counts (B_pos, B_ref) and ξ_max
    Coarse context coordinates depend on binning hyperparameters that control support vs specificity; exact values not fully reported as a sensitivity study.
  • PPO clipping range and training schedule = clip=0.2, 150 steps
    Clip 0.2 and 150 steps with Lightning OPD surrogate are training choices that can move Pass@1.
  • Rare-group smoothing toward global mean
    Lookup smoothing strength/threshold is an estimation regularizer chosen for stability.
axioms (6)
  • domain assumption Teacher consistency is often violated in practice and naive cross-teacher OPD degrades relative to consistent OPD.
    Motivating premise from Lightning OPD and §1; treated as established problem setting rather than re-proved.
  • ad hoc to paper d_it = b(z_it) + v_it where b is recurring/style-like and v carries context-specific teacher evidence, estimable from token and coarse context coordinates shared across rollouts.
    §3.2 operational decomposition (Eq. 5); authors disclaim semantic ground truth but training depends on this proxy.
  • domain assumption Frozen single-sample rollouts from π_R with precomputed teacher log-probs suffice for OPD updates (Lightning OPD offline surrogate).
    Inherited from Lightning OPD; entire method is built on this replay (§3.1).
  • domain assumption Teacher and reference scores are aligned on the same tokens (compatible Qwen-family tokenization).
    Stated in §3.1; required for d_it to be well-defined.
  • domain assumption Advantage-weighted policy gradient with fixed scalar advantages (standard OPD practice) is a valid learning update.
    §3.1 cites Agarwal et al. / Lu & Lab / Wu et al.; not re-derived.
  • standard math Cross-fitting prevents self-fitting of lookup tables and yields a usable held-out style estimate.
    Invokes sample-splitting practice (Chernozhukov et al. 2018) as estimation device in §3.3.
invented entities (2)
  • Style-token bias (operational recurring component b(z)) no independent evidence
    purpose: Name and target the harmful cross-teacher component of teacher–reference disagreement to be removed.
    Not independently measured style labels; defined as cross-rollout predictable disagreement from chosen coordinates.
  • Cross-fitted token and context lookup residualization (Lightning OPD 2.0) no independent evidence
    purpose: Concrete estimator producing residualized advantages A_res for offline OPD.
    New method object; validated only via downstream benchmarks and diagnostic deviation plots in this paper.

pith-pipeline@v1.2.0-daily-grok45 · 23477 in / 3884 out tokens · 81033 ms · 2026-07-31T07:00:38.264562+00:00 · methodology

0 comments
read the original abstract

On-policy distillation (OPD) provides dense token-level supervision from a teacher, but its effectiveness can depend on teacher consistency, meaning that the model providing OPD supervision should also have generated the demonstrations used to train the supervised fine-tuning (SFT) reference. However, this condition is frequently violated in practice when SFT data have mixed or unknown provenance or when different models are preferred for SFT data generation and subsequent distillation. In such cross-teacher settings, even a stronger OPD teacher can yield little improvement over the SFT reference. We find that raw teacher--reference disagreement contains potentially useful context-specific teacher evidence as well as a recurring component associated with differences in wording, formatting, and reasoning cadence. We introduce Lightning OPD 2.0 with cross-fitted style residualization, which uses rollout-level cross-fitting to estimate this recurring component as an operational proxy for style-token bias and subtracts it before constructing the token-level OPD update. Across mathematical reasoning and code generation benchmarks, Lightning OPD 2.0 consistently outperforms Lightning OPD in cross-teacher settings. Starting from Klear-Reasoner-8B-SFT, Lightning OPD 2.0 reaches 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5. Together, these results establish Lightning OPD 2.0 as a practical approach to cross-teacher OPD, relaxing teacher consistency as a prerequisite and allowing the SFT data generator and distillation teacher to be selected independently. Code will be released soon.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 47 linked inside Pith

  1. [2]

    AIME 2024.https://huggingface.co/datasets/AI-MO/aimo-validation-aime,

    AI-MO. AIME 2024.https://huggingface.co/datasets/AI-MO/aimo-validation-aime,

    2024

  2. [3]

    Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N. M. Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, and Mehrdad Farajtabar. Unmasking on-policy distillation: Where it helps, where it hurts, and why.arXiv preprint arXiv:2605.10889,

    Pith/arXiv arXiv

  3. [8]

    Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178,

    Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models.arXiv preprint arXiv:2506.04178,

    Pith/arXiv arXiv

  4. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  5. [11]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290,

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290,

    Pith/arXiv arXiv

  6. [12]

    Ponti, and Ivan Titov

    Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling.arXiv preprint arXiv:2507.01679,

    Pith/arXiv arXiv

  7. [13]

    Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation.arXiv preprint arXiv:2601.20802,

    Pith/arXiv arXiv

  8. [14]

    LiveCodeBench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974,

    Pith/arXiv arXiv

  9. [15]

    Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079,

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079,

    Pith/arXiv arXiv

  10. [16]

    VinePPO: Refining credit assignment in RL training of LLMs.arXiv preprint arXiv:2410.01679,

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in RL training of LLMs.arXiv preprint arXiv:2410.01679,

    Pith/arXiv arXiv

  11. [17]

    Respecting self-uncertainty in on-policy self-distillation for efficient LLM reasoning.arXiv preprint arXiv:2605.13255,

    Junlong Ke, Zichen Wen, Weijia Li, Conghui He, and Linfeng Zhang. Respecting self-uncertainty in on-policy self-distillation for efficient LLM reasoning.arXiv preprint arXiv:2605.13255,

    Pith/arXiv arXiv

  12. [18]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,

    2016

  13. [19]

    Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

    Pith/arXiv arXiv

  14. [20]

    ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models.arXiv preprint arXiv:2310.10505,

    Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models.arXiv preprint arXiv:2310.10505,

    Pith/arXiv arXiv

  15. [21]

    Orbit: On-policy exploration-exploitation for controllable multi-budget reasoning.arXiv preprint arXiv:2601.08310,

    Kun Liang, Clive Bai, Xin Xu, Chenming Tang, Sanwoo Lee, Weijie Liu, Saiyong Yang, and Yunfang Wu. Orbit: On-policy exploration-exploitation for controllable multi-budget reasoning.arXiv preprint arXiv:2601.08310,

    Pith/arXiv arXiv

  16. [22]

    DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    11 Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chengqi Zhao, Chenggang Deng, Chengpeng Zhang, et al. DeepSeek-V3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  17. [23]

    UFT: Unifying supervised and reinforcement fine-tuning.arXiv preprint arXiv:2505.16984, 2025a

    Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. UFT: Unifying supervised and reinforcement fine-tuning.arXiv preprint arXiv:2505.16984, 2025a. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025b. Kevin Lu and ...

    arXiv

  18. [24]

    https://thinkingmachines.ai/blog/on-policy-distillation

    doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, Bin Cui, and Wentao Zhang. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions.arXiv preprint arXiv:2506.07527,

    arXiv

  19. [25]

    MiniMax-M1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585,

    MiniMax, Aonian Shan, Bangwei Gong, Bo Yang, et al. MiniMax-M1: Scaling test-time compute efficiently with lightning attention.arXiv preprint arXiv:2506.13585,

    Pith/arXiv arXiv

  20. [26]

    Nvidia nemotron 3: Efficient and open intelligence.arXiv preprint arXiv:2512.20856,

    NVIDIA. Nvidia nemotron 3: Efficient and open intelligence.arXiv preprint arXiv:2512.20856,

    Pith/arXiv arXiv

  21. [27]

    AIME 2025.https://github.com/open-compass/opencompass,

    OpenCompass. AIME 2025.https://github.com/open-compass/opencompass,

    2025

  22. [29]

    Revealing the power of post-training for small language models via knowledge distillation.arXiv preprint arXiv:2509.26497,

    Miao Rang, Zhenni Bi, Hang Zhou, Hanting Chen, An Xiao, Tianyu Guo, Kai Han, Xinghao Chen, and Yunhe Wang. Revealing the power of post-training for small language models via knowledge distillation.arXiv preprint arXiv:2509.26497,

    arXiv

  23. [30]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  24. [32]

    Rl’s razor: Why online reinforcement learning forgets less.arXiv preprint arXiv:2509.04259,

    Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. Rl’s razor: Why online reinforcement learning forgets less.arXiv preprint arXiv:2509.04259,

    Pith/arXiv arXiv

  25. [33]

    Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897,

    Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning.arXiv preprint arXiv:2601.19897,

    Pith/arXiv arXiv

  26. [34]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267,

    Pith/arXiv arXiv

  27. [35]

    A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

    Pith/arXiv arXiv

  28. [36]

    Self-supervised on-policy distillation for reasoning language models.arXiv preprint arXiv:2605.17497,

    Zhiquan Tan and Yinrong Hong. Self-supervised on-policy distillation for reasoning language models.arXiv preprint arXiv:2605.17497,

    Pith/arXiv arXiv

  29. [37]

    Kimi k1.5: Scaling reinforcement learning with LLMs.arXiv preprint arXiv:2501.12599,

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, et al. Kimi k1.5: Scaling reinforcement learning with LLMs.arXiv preprint arXiv:2501.12599,

    Pith/arXiv arXiv

  30. [38]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276,

    Pith/arXiv arXiv

  31. [39]

    Not all disagreement is learnable: Token teachability in on-policy distillation.arXiv preprint arXiv:2605.26844,

    12 Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, and Hongxia Yang. Not all disagreement is learnable: Token teachability in on-policy distillation.arXiv preprint arXiv:2605.26844,

    Pith/arXiv arXiv

  32. [40]

    Yecheng Wu, Song Han, and Han Cai

    doi: 10.1162/tacl_a_00612. Yecheng Wu, Song Han, and Han Cai. Lightning OPD: Efficient post-training for large reasoning models with offline on-policy distillation.arXiv preprint arXiv:2604.13010,

    Pith/arXiv arXiv

  33. [41]

    Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780,

    Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780,

    Pith/arXiv arXiv

  34. [42]

    On the position bias of on-policy distillation.arXiv preprint arXiv:2606.22600,

    Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, and Yifei Wang. On the position bias of on-policy distillation.arXiv preprint arXiv:2606.22600,

    Pith/arXiv arXiv

  35. [43]

    SG-OPD: Sign-gated on-policy distillation via sign-consistency gating and phased teacher sampling.arXiv preprint arXiv:2606.09304,

    Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, and Xiaosong Yuan. SG-OPD: Sign-gated on-policy distillation via sign-consistency gating and phased teacher sampling.arXiv preprint arXiv:2606.09304,

    Pith/arXiv arXiv

  36. [44]

    Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945,

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945,

    Pith/arXiv arXiv

  37. [45]

    Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122,

    Pith/arXiv arXiv

  38. [46]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  39. [47]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026a

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026a. Yuxiao Yang, Xiaoyun Wang, and Weitong Zhang. OGLS-SD: On-policy self-distillation with outcome-guided logit steering for LLM reasoning.arXiv preprint ar...

    Pith/arXiv arXiv

  40. [48]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  41. [49]

    What’s behind PPO’s collapse in long-CoT? value optimization holds the secret.arXiv preprint arXiv:2503.01491,

    Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind PPO’s collapse in long-CoT? value optimization holds the secret.arXiv preprint arXiv:2503.01491,

    Pith/arXiv arXiv

  42. [50]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025a

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025a. Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, Tiantian Fan, Zhengyin Du, Xiangpeng Wei,...

    Pith/arXiv arXiv

  43. [51]

    On-policy RL meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting.arXiv preprint arXiv:2508.11408,

    Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy RL meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting.arXiv preprint arXiv:2508.11408,

    arXiv

  44. [52]

    Reinforcement-aware knowledge distillation for LLM reasoning.arXiv preprint arXiv:2602.22495,

    13 Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto. Reinforcement-aware knowledge distillation for LLM reasoning.arXiv preprint arXiv:2602.22495,

    Pith/arXiv arXiv

  45. [53]

    Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

    Pith/arXiv arXiv

  46. [54]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    Pith/arXiv arXiv

  47. [2015]

    REINFORCE++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,

    Jian Hu. REINFORCE++: A simple and efficient approach for aligning large language models.arXiv preprint arXiv:2501.03262,

    Pith/arXiv arXiv

  48. [2017]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  49. [2018]

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al

    doi: 10.1111/ectj.12097. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456,

    Pith/arXiv arXiv

  50. [2022]

    Privileged information distillation for language models.arXiv preprint arXiv:2602.04942,

    Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models.arXiv preprint arXiv:2602.04942,

    Pith/arXiv arXiv

  51. [2023]

    Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,

    Pith/arXiv arXiv

  52. [2024]

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs.arXiv preprint arXiv:2402.14740,

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs.arXiv preprint arXiv:2402.14740,

    Pith/arXiv arXiv

  53. [2025]

    Beyond two-stage training: Cooperative SFT and RL for LLM reasoning.arXiv preprint arXiv:2509.06948, 2025a

    Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. Beyond two-stage training: Cooperative SFT and RL for LLM reasoning.arXiv preprint arXiv:2509.06948, 2025a. Yang Chen, Zhuolin Yang, Zihan Liu, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nemotron-cascade: Scaling cascaded reinforceme...

    Pith/arXiv arXiv

  54. [2026]

    MathArena: Evaluating LLMs on uncontaminated math competitions.arXiv preprint arXiv:2505.23281,

    Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. MathArena: Evaluating LLMs on uncontaminated math competitions.arXiv preprint arXiv:2505.23281,

    Pith/arXiv arXiv