Pith. sign in

REVIEW 4 major objections 5 minor 76 references

Information asymmetry—not teacher quality—drives the 'privilege illusion' in self-distillation, and the paper's dual-anchored distillation (DAPD) removes it by aligning reference and rollout behavior only under matched information availabil

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DAPD uses a self-conditioned bridge and bidirectional anchoring to match information between teacher and student during on-policy self-distillation, improving reasoning, coding, and instruction-following benchmarks over OPSD.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A well-engineered distillation recipe with consistent gains, but the root-cause claim about information asymmetry is stronger than the evidence supports. the 4 major comments →

arxiv 2608.01735 v1 pith:4GBUZ74G submitted 2026-08-03 cs.AI

DAPD: Dual-Anchored Policy Distillation

Jianyu Wu , Yizhou Wang , Encheng Su , Chen Tang , Shixiang Tang This is my paper
classification cs.AI
keywords privilege illusionon-policy self-distillationinformation asymmetrypolicy distillationlanguage model post-trainingreference-conditioned teacherself-conditioned bridgereasoning models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On-policy self-distillation lets a model train by imitating its own heavily conditioned teacher, but the teacher sees a reference answer the student will never have at inference. The paper argues that this information gap, not teacher quality, is what makes students behave as if privileged knowledge were available—asserting unsupported answers and degrading scores as training proceeds. To test this diagnosis, it conditions the student on the full completion it is generating, matching the teacher's information; wrong claims fall by 45% and reasoning scores rise by 6.22 points. DAPD generalizes that intervention into a training loss with two anchoring paths (with and without privileged information) applied in both reference-to-rollout and rollout-to-reference directions. Across six tasks on a 4B-scale model the method gains 2.00 points over standard on-policy self-distillation, and the gain persists across five model scales from 1.7B to 32B.

Core claim

Stated on the paper's own terms, the discovery is that privilege illusion in on-policy self-distillation is caused by information asymmetry between the privileged teacher distribution and the student distribution at inference, not by teacher quality alone. The key evidence is an isolated intervention: keep the privileged teacher unchanged but replace the student's unconditioned distribution with a self-conditioned distribution pθ(· | x, y_<t, y) that conditions on the full completion being predicted. Because both teacher and student now receive a full completion, the asymmetry disappears, and wrong claims drop by 45% while Avg@12 rises by +6.22. DAPD turns this into a general objective: the

What carries the argument

The central object is the self-conditioned distribution pθ(· | x, s_<t, s): the model predicting the next token while conditioned on the rest of the completion s it is producing. This distribution is trainable like the inference-time None distribution (the student's distribution without privileged information), but it is information-matched to the privileged teacher Cross, which is conditioned on the other completion. That combination makes Self a bridge: dual-path anchoring constructs an unconditioned path (Entangled Distillation + Inference Anchor) to align the two None distributions, and a privileged path (Privileged Anchor) to align Self with Cross; dual-source anchoring then applies bot

Load-bearing premise

The load-bearing premise is that conditioning the student on the full completion it is generating gives it the same information as conditioning the teacher on the reference completion—so that matching the presence of a full completion, rather than matching its content, removes the asymmetry. If the content of the completion matters, the matched-information mechanism may not be the true cause of DAPD's gains.

What would settle it

Run DAPD with the privileged teacher conditioned on a full completion that is deliberately wrong or taken from a different problem, while keeping the same length and position. If wrong-claim counts stay low and reasoning gains persist, then 'having a full completion' is doing the work; if the gains disappear, content mismatch is the real driver and the information-asymmetry diagnosis fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At inference time DAPD adds no cost; the anchor distributions exist only during training, so the deployed policy is unchanged in compute but should make fewer unsupported claims.
  • Existing fixes that filter or reweight the privileged teacher signal are treating symptoms: as long as the teacher sees the reference and the student does not, the illusion can re-emerge; DAPD changes the supervision structure itself.
  • Reference and rollout guidance are complementary: using both beats either source alone, which implies the optimal distillation signal shifts with model scale (smaller models need more reference guidance, larger models more rollout guidance).
  • A second on-policy rollout can partially replace a curated reference: the dual-rollout variant still outperforms OPSD at 1.7B–8B, and a verifier-selected rollout raises this further.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One inference the paper leaves implicit is that the mechanism may really be about semantic content match rather than merely the presence of a completion: Proposition 1's bridge-consistency assumption requires Self and Cross to be close, i.e., the two completions to be similar. If content mismatch drives the illusion, then any teacher conditioned on a different full completion would behave differen
  • Because the optimal reference-guidance weight falls from λ=0.5 at 1.7B to λ=0.2 at larger scales, a per-token or per-example adaptive weighting policy should improve on the paper's fixed scale-level coefficients; the paper lists this as future work.
  • The anchor design should transfer to other privileged signals such as tool traces, retrieved documents, or verifier feedback, but 'self-conditioned' would need a non-textual definition; the paper's reference-free experiments hint at this direction without testing it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper argues that the privilege illusion in on-policy self-distillation (OPSD) is caused by information asymmetry: the teacher is conditioned on a reference completion, while the student at inference is not. To address this, the authors propose DAPD, which introduces a self-conditioned distribution (Self) as a bridge, and constructs two alignment paths: an unconditioned path (Entangled Distillation plus Inference Anchor) and a privileged path (Privileged Anchor), applied in both reference-to-rollout and rollout-to-reference directions. Experiments on Qwen3 models from 1.7B to 32B report consistent gains over OPSD, including +2.00 points on Qwen3-4B, reductions in a hand-defined 'wrong claims' probe, and a reference-free variant. The paper includes a theoretical analysis (Proposition 1) that bounds the distance between two None distributions under a bridge-consistency assumption, plus an appendix with implementation and probing details.

Significance. If the information-asymmetry diagnosis is correct, DAPD is a principled and broadly applicable modification to OPSD, and the paper would make a useful contribution to post-training methodology. The manuscript has several strengths: a clearly described method, a reproducible experimental appendix, a behavioral probe for privilege illusion, controlled ablations of the path and source components, and a scale study spanning five model sizes. The reference-free variant is an interesting extension. However, the central causal claim is not yet established: the key intervention in Section 2.2 changes multiple factors simultaneously, the theoretical bridge assumption in Proposition 1 is unmeasured, and the headline numerical results rest on a single seed with per-scale hyperparameter selection. These issues are fixable but require additional experiments and a more cautious interpretation.

major comments (4)
  1. [Section 2.2 and Section 3.1, Eq. (4)] The paper concludes that 'matching the information available to Cross and Self reduces privilege illusion' and that information asymmetry is the root cause. But the Privileged-Anchor intervention replaces the trainable None distribution with Self = p_theta(·|x, y_<t, y), which conditions on the full rollout y. At inference the student is None and does not have access to y, so Self does not match the inference-time student context. The intervention therefore also changes the training distribution and adds a self-conditioned objective, not just the information available to teacher and student. Without a control that varies the content of the privileged completion while holding the presence of a full completion fixed (e.g., conditioning Self on an independent rollout, or on a random reference from another example), the observed improvement cannot be uniquely attributed to 'information match
  2. [Appendix B.1, Eq. (12), Proposition 1] Proposition 1 bounds E_TV(n*, n_y) by sqrt(E_infer/2) + epsilon + sqrt(E_ent/2), but this is conditional on the bridge-consistency assumption epsilon_bridge <= epsilon. This epsilon is not measured anywhere in the paper. The empirical check in Appendix B.3 uses Eq. (20), which measures a gradient-alignment scalar between Self and Cross, not the TV distance between s* and c_y used in Eq. (11). A positive gradient dot product does not imply small total variation. Thus the paper provides no evidence that the bridge term is small, and the claimed empirical support for the unconditioned path is not established. If rollouts diverge from references, epsilon_bridge could be large and the bound would be vacuous. The authors should either measure epsilon_bridge directly on paired prefixes or present Proposition 1 as a purely conditional statement without the empirical endorsement in Section 3.2.
  3. [Section A.6 and Tables 1, 6] All training jobs use a single seed (seed 42), and the matched-seed protocol only aligns initialization, data order, and rollout streams across methods. The main results compare single runs of each method, while the evaluation itself is stochastic (Avg@12 at temperature 1.0). The reported gains (+2.00 overall, +2.69 at 4B, +2.78 at 32B) may be within run-to-run or evaluation noise. Please provide multiple seeds or bootstrap confidence intervals over the 12 samples per problem for the headline comparisons and the scale-wise results. Without this, the persistence-of-gain claim is not statistically grounded.
  4. [Section 4.4, Table 3(d), Table 5] The scale-wise DAPD configurations are selected per scale by evaluating Avg@12 on AIME24/AIME25/HMMT25; Table 3(d) explicitly reports the 'preferred' lambda and anchor weights, and Table 5 lists the final configurations. The OPSD baseline is not similarly tuned per scale. The claim that 'gains persist across scales' is therefore a comparison between a per-scale tuned DAPD and a fixed OPSD, which can inflate the apparent improvement. Please also report DAPD under a single fixed hyperparameter configuration, or select hyperparameters on a separate validation split, to demonstrate that the cross-scale robustness is not an artifact of test-set selection.
minor comments (5)
  1. [Section 2.1, Behavioral probe] The definition of 'wrong claims' is rule-based but somewhat subjective (e.g., 'attributes a concrete answer to memory, a reference or official solution'). Providing the exact matching pattern or an annotation agreement measure would make the probe reproducible.
  2. [Section A.6, Optimization] The text says 'linear 500-step schedule without warmup.' It is unclear whether this is a linear decay or a linear increase in learning rate, and how the schedule is applied. Please clarify.
  3. [Eq. (8), Divergence implementation] The component cap c=0.05 is introduced without explanation of how it was chosen. Table 8 shows sensitivity, but a rationale for the value (e.g., based on token-level clipping behavior) would help.
  4. [Table 3(e) caption] The phrase 'Verified applies a correctness verifier to the reference-side rollout' is ambiguous: in the verified-rollout variant, u is selected by the verifier while v is unfiltered. Clarify which side is 'reference-side' in the caption.
  5. [Figure 3] The figure uses notation L^y_ent, L^{y*}_infer, etc. without a legend. Adding a short legend or expanding the caption would improve readability.

Circularity Check

0 steps flagged

No significant circularity; the paper is self-contained and the theoretical bound is conditional, not a prediction forced by construction.

full rationale

The paper's derivation chain does not reduce to its own inputs. Proposition 1 bounds the None-to-None total-variation distance by the two training losses and an explicitly stated bridge-consistency assumption (ε_bridge ≤ ε). This is a conditional consistency theorem for the proposed objective, not a prediction that is equivalent to the objective by construction. The bridge assumption is unverified and could be false, but that is a correctness or robustness concern, not circularity. The empirical claims are external benchmark comparisons against OPSD and other baselines; hyperparameters are tuned on those benchmarks, which is overfitting risk, but the OPSD comparison is an independent measurement and not forced by the method's definition. There are no load-bearing self-citations: OPSD, PurifiedOPSD, DOPD, SDFT, and SDPO are all external prior works. The 'information asymmetry' diagnosis is an empirical hypothesis supported by the Privileged Anchor intervention and ablations, not a definitional equivalence. The dual-rollout and verified-rollout experiments in Section 4.5 further separate the 'presence of a full completion' from 'content of the reference,' indicating that the paper does not merely rename a known pattern. Overall, no circular step can be exhibited from the text.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The method relies on several free parameters tuned per scale and on explicit, unproven assumptions in the theoretical analysis. No new physical entities are introduced; the 'Self distribution' is a training construct, not a postulated external entity.

free parameters (3)
  • lambda (reference-guidance weight) = 0.2 (4B, 8B, 14B, 32B), 0.5 (1.7B)
    Chosen per model scale, decreasing as scale increases; likely selected on validation or test performance.
  • beta_ref_infer, beta_ref_priv, beta_roll_infer, beta_roll_priv (loss redistribution weights) = Vary by scale; see Table 5 (e.g., 4B: 1, 1, 1, 2)
    Ad hoc weights tuned per scale to balance reference and rollout guidance.
  • Component cap c for forward KL = 0.05
    Selected as best in Table 8 ablation; affects gradient magnitude for large divergences.
axioms (5)
  • domain assumption Bridge-consistency assumption: epsilon_bridge <= epsilon for coupled reference and rollout prefixes (Eq. 12)
    Assumed to hold for the coupling of reference and rollout prefixes; used in Proposition 1 to bound None-None distance. Not proven, only motivated by shared parameters.
  • domain assumption Local compatibility condition: E[g_C^T J_C J_S^T g_S] >= gamma > 0 (Eq. 20)
    Assumed to hold for the shared-parameter transfer; empirically checked on 128 positions from 32 math examples, but not guaranteed generally.
  • standard math Pinsker's inequality, Jensen's inequality, Cauchy-Schwarz, triangle inequality for TV distance
    Used in the proofs of Proposition 1 and Lemma 1.
  • domain assumption Student and teacher share parameters (p_theta)
    Core to self-distillation and required for the parameter-sharing transfer in Appendix B.3.
  • domain assumption Privileged information is a reference completion inserted into the prompt (reference-injection format)
    The entire analysis assumes the privileged information is a reference solution; other forms (tool traces, plans) are left to future work.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DAPD: Dual-Anchored Policy Distillation." pith.science (2026). https://pith.science/paper/4GBUZ74G

@misc{pith2026260801735,
  author       = {Pith},
  title        = {Pith review of: DAPD: Dual-Anchored Policy Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GBUZ74G}},
  note         = {Machine review of arXiv:2608.01735}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.

Figures

Figures reproduced from arXiv: 2608.01735 by Chen Tang, Encheng Su, Jianyu Wu, Shixiang Tang, Yizhou Wang.

Figure 1
Figure 1. Figure 1: DAPD consistently improves over OPSD across [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training dynamics for OPSD, Privileged Anchor, and DAPD, averaged across five scales: (a) wrong claims and (b) Avg@12 over AIME24, AIME25, and HMMT25. ilege illusion, DAPD improves over OPSD by +2.00 points on average on Qwen3-4B. Notably, while the gains from OPSD largely disappear as model scale increases, DAPD maintains stable improvements over OPSD across Qwen3-1.7B–32B, with gains of +2.69 points at 4… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of DAPD. Left: Dual-Source Anchoring uses both reference-to-rollout and rollout-to-reference guidance to balance correctness and student reachability. Right: Dual-Path Anchoring details the rollout-to-reference guidance direction by introducing Self as a trainable bridge. The unconditioned path pairs Entangled Distillation with the Inference Anchor to align the two None distributions through Self … view at source ↗
Figure 4
Figure 4. Figure 4: Visualization of inference-time results. On the same [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 36 canonical work pages · 3 internal anchors

  1. [1]

    Agarwal, R.; Vieillard, N.; Zhou, Y.; Stanczyk, P.; Ramos, S.; Geist, M.; and Bachem, O. 2024. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. In International Conference on Learning Representations

    work page 2024

  2. [4]

    Guha, E.; Marten, R.; Keh, S.; et al. 2025. OpenThoughts : Data Recipes for Reasoning Models. arXiv preprint arXiv:2506.04178

    Pith/arXiv arXiv 2025

  3. [5]

    Harvard-MIT Mathematics Tournament . 2025. HMMT February 2025 Archive

    work page 2025

  4. [6]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Advances in Neural Information Processing Systems, volume 34

    work page 2021

  5. [7]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop

    work page 2015

  6. [8]

    Hou, W.; Peng, S.; Wang, W.; Ruan, Z.; Zhang, Y.; Zhou, Z.; Gao, M.; Chen, Y.; Wang, K.; Yang, H.; et al. 2026. Uni-OPD : Unifying On-Policy Distillation with a Dual-Perspective Recipe. arXiv preprint arXiv:2605.03677

    Pith/arXiv arXiv 2026

  7. [9]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

    2022

  8. [11]

    Jain, N.; Han, K.; Gu, A.; Li, W.-D.; Yan, F.; Zhang, T.; Wang, S.; Solar-Lezama, A.; Sen, K.; and Stoica, I. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In International Conference on Learning Representations

    work page 2025

  9. [14]

    Kim, Y.; and Rush, A. M. 2016. Sequence-Level Knowledge Distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 1317--1327

    work page 2016

  10. [16]

    S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

    Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems, volume 35

    work page 2022

  11. [17]

    H.; Gonzalez, J

    Kwon, W.; Li, Z.; Zhuang, S.; Sheng, Y.; Zheng, L.; Yu, C. H.; Gonzalez, J. E.; Zhang, H.; and Stoica, I. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM Symposium on Operating Systems Principles

    work page 2023

  12. [18]

    J.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V

    Lewkowycz, A.; Andreassen, A. J.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V. V.; Slone, A.; Anil, C.; Schlag, I.; Gutman-Solo, T.; et al. 2022. Solving Quantitative Reasoning Problems with Language Models. In Advances in Neural Information Processing Systems, volume 35

    work page 2022

  13. [20]

    Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2024. Let's Verify Step by Step. In International Conference on Learning Representations

    work page 2024

  14. [21]

    Mathematical Association of America . 2026. MAA Invitational Competitions

    work page 2026

  15. [22]

    Nguyen, D.; Xiao, H.; Prasad, A.; Khan, Z.; Das, A.; Zhang, A.; Sahu, S.; Lee, H.; Stengel-Eskin, E.; and Bansal, M. 2026. AVSD : Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signals. arXiv preprint arXiv:2605.20643

    Pith/arXiv arXiv 2026

  16. [23]

    G.; Mao, H.; Yan, F.; Ji, C

    Patil, S. G.; Mao, H.; Yan, F.; Ji, C. C.-J.; Suresh, V.; Stoica, I.; and Gonzalez, J. E. 2025. The Berkeley Function Calling Leaderboard ( BFCL ): From Tool Use to Agentic Evaluation of Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267, 48371--48392

    work page 2025

  17. [25]

    Pyatkin, V.; Malik, S.; Graf, V.; Ivison, H.; Huang, S.; Dasigi, P.; Lambert, N.; and Hajishirzi, H. 2025. Generalizing Verifiable Instruction Following. In Advances in Neural Information Processing Systems, volume 38

    work page 2025

  18. [26]

    K.; Wu, Y.; and Guo, D

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath : Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  19. [27]

    Shen, Z.; Tong, J.; Yan, S.; Shen, C.; Chen, H.; Ye, W.; Hu, X.; Miao, R.; Wang, H.; Zhao, J.; Chen, G.; and Ye, J. 2026. Purified OPSD : On-Policy Self-Distillation Without Losing How to Think. arXiv preprint arXiv:2607.02234

    Pith/arXiv arXiv 2026

  20. [29]

    Thinking Machines Lab . 2025. On-Policy Distillation. Thinking Machines Lab: Connectionism

    work page 2025

  21. [30]

    Tu, S.; Xu, C.; Zhang, Q.; Ma, Y.; Zhang, Y.; Li, L.; Li, D.; Lan, X.; and Zhao, D. 2026. UCOB : Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation. arXiv preprint arXiv:2606.29502

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [31]

    Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In International Conference on Learning Representations

    work page 2023

  23. [32]

    V.; and Zhou, D

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; and Zhou, D. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, volume 35

    work page 2022

  24. [33]

    Xu, Y.; Sang, H.; Zhou, Z.; He, R.; Wang, Z.; and Geramifard, A. 2026. TIP : Token Importance in On-Policy Distillation. arXiv preprint arXiv:2604.14084

    Pith/arXiv arXiv 2026

  25. [34]

    Yang, A.; et al. 2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  26. [37]

    Yu, X.; Li, G.; Si, Q.; Zhang, G.; Xu, Y.; Wang, C.; Dong, S.; Tuo, K.; Zeng, X.; Feng, K.; Wang, Q.; Shi, Y.; Hu, X.; Yue, X.; Wang, J.; and Yan, S. 2026. DOPD : Dual On-Policy Distillation. arXiv preprint arXiv:2606.30626

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  27. [38]

    Zelikman, E.; Wu, Y.; Mu, J.; and Goodman, N. D. 2022. STaR : Bootstrapping Reasoning With Reasoning. In Advances in Neural Information Processing Systems, volume 35

    work page 2022

  28. [40]

    Zhao, Z.; Ma, X.; Yang, L.; Feng, Y.; Shi, D.; He, J.; Xin, X.; Ren, Z.; and Wu, X.-M. 2026 b . ROSD : Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains. arXiv preprint arXiv:2605.28014

    Pith/arXiv arXiv 2026

  29. [41]

    Zheng, B.; Ma, X.; Liang, Y.; Ruan, J.; Fu, X.; Lin, K.; Zhu, B.; Zeng, K.; and Cai, X. 2026. SCOPE : Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting. arXiv preprint arXiv:2604.10688

    Pith/arXiv arXiv 2026

  30. [42]

    International Conference on Learning Representations , year=

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes , author=. International Conference on Learning Representations , year=

  31. [43]

    arXiv preprint arXiv:2108.07732 , year=

    Program Synthesis with Large Language Models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  32. [44]

    arXiv preprint arXiv:2204.05862 , year=

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback , author=. arXiv preprint arXiv:2204.05862 , year=

    Pith/arXiv arXiv

  33. [45]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  34. [46]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  35. [47]

    Thinking Machines Lab: Connectionism , year=

    On-Policy Distillation , author=. Thinking Machines Lab: Connectionism , year=. doi:10.64434/tml.20251026 , url=

    doi:10.64434/tml.20251026

  36. [48]

    arXiv preprint arXiv:2601.18734 , year=

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models , author=. arXiv preprint arXiv:2601.18734 , year=

    Pith/arXiv arXiv

  37. [49]

    Zhao, Ziqi and Ma, Xinyu and Yang, Liu and Feng, Yujie and Shi, Daiting and He, Jingzhou and Xin, Xin and Ren, Zhaochun and Wu, Xiao-Ming , journal=

    work page

  38. [50]

    Purified

    Shen, Zhanming and Tong, Jintao and Yan, Shaotian and Shen, Chen and Chen, Hao and Ye, Wentao and Hu, Xiaomeng and Miao, Rui and Wang, Haobo and Zhao, Junbo and Chen, Gang and Ye, Jieping , journal=. Purified

    work page

  39. [51]

    arXiv preprint arXiv:2602.04942 , year=

    Privileged Information Distillation for Language Models , author=. arXiv preprint arXiv:2602.04942 , year=

    Pith/arXiv arXiv

  40. [52]

    NIPS Deep Learning and Representation Learning Workshop , year=

    Distilling the Knowledge in a Neural Network , author=. NIPS Deep Learning and Representation Learning Workshop , year=

  41. [53]

    Measuring Mathematical Problem Solving With the

    Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , booktitle=. Measuring Mathematical Problem Solving With the

  42. [54]

    Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages=

    Sequence-Level Knowledge Distillation , author=. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages=

    2016

  43. [55]

    Advances in Neural Information Processing Systems , volume=

    Large Language Models are Zero-Shot Reasoners , author=. Advances in Neural Information Processing Systems , volume=

  44. [56]

    Advances in Neural Information Processing Systems , volume=

    Solving Quantitative Reasoning Problems with Language Models , author=. Advances in Neural Information Processing Systems , volume=

    work page

  45. [57]

    , booktitle=

    Zelikman, Eric and Wu, Yuhuai and Mu, Jesse and Goodman, Noah D. , booktitle=

    work page

  46. [58]

    Yu, Xinlei and Li, Gen and Si, Qingyi and Zhang, Guibin and Xu, Yuqi and Wang, Congcong and Dong, Shuai and Tuo, Kaiwen and Zeng, Xiangyu and Feng, Kaituo and Wang, Qunzhong and Shi, Yang and Hu, Xiaobin and Yue, Xiangyu and Wang, Jiaqi and Yan, Shuicheng , journal=

    work page

  47. [59]

    arXiv preprint arXiv:2602.12275 , year=

    On-Policy Context Distillation for Language Models , author=. arXiv preprint arXiv:2602.12275 , year=

    Pith/arXiv arXiv

  48. [60]

    arXiv preprint arXiv:2602.12125 , year=

    Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation , author=. arXiv preprint arXiv:2602.12125 , year=

    Pith/arXiv arXiv

  49. [61]

    Hou, Wenjin and Peng, Shangpin and Wang, Weinong and Ruan, Zheng and Zhang, Yue and Zhou, Zhenglin and Gao, Mingqi and Chen, Yifei and Wang, Kaiqi and Yang, Hongming and others , journal=

    work page

  50. [62]

    arXiv preprint arXiv:2601.19897 , year=

    Self-Distillation Enables Continual Learning , author=. arXiv preprint arXiv:2601.19897 , year=

    Pith/arXiv arXiv

  51. [63]

    International Conference on Learning Representations , year=

    Let's Verify Step by Step , author=. International Conference on Learning Representations , year=

  52. [64]

    Advances in Neural Information Processing Systems , volume=

    Training Language Models to Follow Instructions with Human Feedback , author=. Advances in Neural Information Processing Systems , volume=

  53. [65]

    Advances in Neural Information Processing Systems , volume=

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author=. Advances in Neural Information Processing Systems , volume=

  54. [66]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal Policy Optimization Algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  55. [67]

    arXiv preprint arXiv:2601.20802 , year=

    Reinforcement Learning via Self-Distillation , author=. arXiv preprint arXiv:2601.20802 , year=

    Pith/arXiv arXiv

  56. [68]

    arXiv preprint arXiv:2603.07079 , year=

    Entropy-Aware On-Policy Distillation of Language Models , author=. arXiv preprint arXiv:2603.07079 , year=

    Pith/arXiv arXiv

  57. [69]

    Xu, Yuanda and Sang, Hejian and Zhou, Zhengze and He, Ran and Wang, Zhipeng and Geramifard, Alborz , journal=

    work page

  58. [70]

    Nguyen, Duy and Xiao, Hanqi and Prasad, Archiki and Khan, Zaid and Das, Anirban and Zhang, Austin and Sahu, Sambit and Lee, Hyunji and Stengel-Eskin, Elias and Bansal, Mohit , journal=

    work page

  59. [71]

    Tu, Songjun and Xu, Chengdong and Zhang, Qichao and Ma, Yiwen and Zhang, Yaocheng and Li, Linjing and Li, Dong and Lan, Xiangyuan and Zhao, Dongbin , journal=

    work page

  60. [72]

    arXiv preprint arXiv:2211.14275 , year=

    Solving Math Word Problems with Process- and Outcome-Based Feedback , author=. arXiv preprint arXiv:2211.14275 , year=

    Pith/arXiv arXiv

  61. [73]

    International Conference on Learning Representations , year=

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author=. International Conference on Learning Representations , year=

  62. [74]

    Advances in Neural Information Processing Systems , volume=

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author=. Advances in Neural Information Processing Systems , volume=

  63. [75]

    Zheng, Binbin and Ma, Xing and Liang, Yiheng and Ruan, Jingqing and Fu, Xiaoliang and Lin, Kepeng and Zhu, Benchang and Zeng, Ke and Cai, Xunliang , journal=

    work page

  64. [76]

    Rethinking On-Policy Self-Distillation for Thinking Models

    Rethinking On-Policy Self-Distillation for Thinking Models , author=. arXiv preprint arXiv:2607.05184 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  65. [77]

    arXiv preprint arXiv:2604.13016 , year=

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe , author=. arXiv preprint arXiv:2604.13016 , year=

    Pith/arXiv arXiv

  66. [78]

    arXiv preprint arXiv:2603.11137 , year=

    Scaling Reasoning Efficiently via Relaxed On-Policy Distillation , author=. arXiv preprint arXiv:2603.11137 , year=

    arXiv

  67. [79]

    Yang, An and others , journal=

  68. [80]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle=

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle=

  69. [81]

    Proceedings of the ACM Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM Symposium on Operating Systems Principles , year=

    work page

  70. [82]

    International Conference on Learning Representations , year=

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code , author=. International Conference on Learning Representations , year=

    work page

  71. [83]

    and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E

    Patil, Shishir G. and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , booktitle=. The Berkeley Function Calling Leaderboard (

    work page

  72. [84]

    Advances in Neural Information Processing Systems , volume=

    Generalizing Verifiable Instruction Following , author=. Advances in Neural Information Processing Systems , volume=

    work page

  73. [85]

    2026 , url=

    MAA Invitational Competitions , author=. 2026 , url=

    work page 2026

  74. [86]

    Guha, Etash and Marten, Ryan and Keh, Sedrick and others , journal=

    work page

  75. [87]

    arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  76. [88]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , journal=

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.