REVIEW 3 major objections 4 minor 27 references
Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read On-policy self-distillation improves language models even when the teacher never sees the target problem's verified solution.
desk verdict A cleanly designed intervention that reframes OPSD's privileged-information story, but the single-run-per-condition design leaves the effect sizes too shaky to take the magnitudes on faith. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is OP2SD (On-Policy Self-Distillation from Other Problems), an intervention that isolates the teacher-only context. Formally, for target $x_i$, OPSD's teacher conditions on $(x_i, y_i^*)$, while OP2SD's teacher conditions on $(x_i, x_j, y_j^*)$ for $j \neq i$, with the prompt stating that $(x_j, y_j^*)$ is unrelated. The student sees only $x_i$ in both cases, rolls out $\hat{y}_i \sim p_\theta(\cdot | x_i)$, and both methods optimize the same clipped Forward-KL token-level distillation loss. This one-change design identifies whether the paired reference is doing causal work or merely shaping the teacher's next-token distribution.
What would settle it
Run OPSD and OP2SD with, say, ten independent training runs each (changing the data order or initialization) and compare the distribution of Avg@12. If the two distributions overlap substantially, the claim that paired solutions are unnecessary collapses. A complementary test is to log the teacher's next-token distributions on the same student prefixes under both contexts: if they are nearly identical, the context explanation cannot be right.
Extended reading notes
Core claim
The central discovery is that the standard interpretation of OPSD as privileged-answer transfer is at least incomplete. The paper constructs a controlled intervention, OP2SD, in which the only change is the teacher-only context: instead of the verified solution to the target problem $A$, the teacher receives the problem and solution of a different example $B$, explicitly marked as unrelated. With student prompt, on-policy rollout, frozen teacher, Forward-KL objective, and optimization unchanged, OP2SD improves Avg@12 over the base model on all nine model–benchmark groups and attains the highest point estimate in eight of nine. For Qwen3-8B it exceeds OPSD by roughly 10 points on each benchmark. The paper also shows the effect is not from arbitrary context: a single fixed worked example suffices, a locally corrupted solution does not hurt, a verbose solution hurts, and cross-subject physics contexts degrade performance below base. This pattern implies the distilled signal is not the answer itself but a teacher behavior elicited by a substantive mathematical worked-solution context.
Load-bearing premise
Each method is trained once, so the measured gaps between OPSD and OP2SD could be caused by which random training run happened to happen; the four decoding seeds do not capture that training-run variation.
Editorial extensions
If this is right
- Paired verified solutions can be dropped from OPSD-style training; any well-formed worked example from the same subject suffices to retain most of the gain.
- Teacher-context design becomes a first-order lever: concise, correct mathematical worked examples help; verbose, trivial, or cross-subject contexts hurt or reverse the gain.
- Because a single fixed example works, the data-collection cost for self-distillation could be far lower than curating per-problem reference solutions.
- Comparisons between OPSD variants should control for context properties (length, correctness, domain, style) and not only for whether the reference is target-aligned.
- The same Forward-KL on-policy objective with a frozen teacher can produce large gains or large degradations depending on what the teacher sees, so context should be reported alongside accuracy numbers.
Reading between the lines
- A sharper test of the context-behavior mechanism would compare the teacher's next-token distributions under the two contexts on identical student prefixes, checking whether the OP2SD teacher is already steering toward the correct answer before any privileged information could matter.
- The single-fixed-example result suggests a possible 'prompt-style transfer' effect; one could test whether a fixed, well-structured worked example from another reasoning domain (e.g., code or logic) produces a similar gain on math benchmarks.
- The physics-context failure may stem from surface-level differences (notation, units, style) rather than subject matter; a controlled extension would rewrite physics solutions in mathematical step-by-step form to isolate content from format.
- Because each condition is one training run, the reported OPSD-vs-OP2SD ordering could reverse under training-run variance; multi-seed training runs would tell whether the advantage is stable.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether On-Policy Self-Distillation (OPSD) improves reasoning because the teacher has access to the verified solution of the target problem, or because the additional worked-solution context changes the teacher's token-level behavior. To separate these, it introduces OP2SD, which replaces the teacher-only reference solution with a problem-and-solution pair drawn from a different training instance, while keeping the student rollout, teacher model, and distillation objective unchanged. Experiments on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across AIME 2024, AIME 2025, and HMMT 2025 report that OP2SD improves over the base model and is competitive with, or better than, standard OPSD. Additional controls vary the worked example's diversity, correctness, verbosity, mathematical domain, and subject matter. The authors conclude that OPSD's gains do not necessarily require the paired target solution and that context-induced teacher behavior is an important factor.
Significance. If the empirical claims are robust, this is a valuable diagnostic result: it reframes OPSD not as privileged-answer transfer but as a more general context-induced teacher phenomenon, with direct implications for how on-policy distillation methods are designed and interpreted. The paper's strengths are its controlled intervention, the range of models and benchmarks, the thoughtful ablation controls (fixed example, corrupted solution, verbose solution, domain and subject mismatch), the explicit statement of limitations, and the public implementation. The main claims are empirical rather than derived, so the study's value depends on the reliability of the training and evaluation comparisons. The single-training-run design and the multiple simultaneous changes in the intervention currently limit the strength of the central conclusion.
major comments (3)
- [Section 6, Table 1, Appendix F] The central comparison between OPSD and OP2SD rests on a single training run per condition. The four decoding seeds used for the reported error bars quantify sampling variation for a fixed checkpoint, not variation across optimization seeds, as the authors acknowledge in Section 6. This is load-bearing: at Qwen3-1.7B several OPSD/OP2SD differences are within the reported Monte Carlo standard error (e.g., AIME 2025: 40.35 versus 40.69), while at Qwen3-8B the 10-plus-point gaps in favor of OP2SD have no training-seed error bars. Appendix F shows that the same OP2SD condition moves 3.96 Avg@12 points between the 100-update and 400-update checkpoints, demonstrating real optimization-trajectory sensitivity. Without repeated training seeds, the observed gaps, especially the 8B results, could be reversed on replication, and the claim that OPSD gains do not come from the target reference solution is not yet robustly established. The limitation is acknowledged but not mitigated; I would require either replicated training runs for the main OPSD/OP2SD comparison or a substantially weakened conclusion.
- [Figure 2 and Section 3.1] The OP2SD intervention changes more than the target-reference pairing. Compared with the OPSD teacher prompt, OP2SD adds the problem statement of the auxiliary example, inserts an explicit "not a solution or a hint" disclaimer, changes the final instruction from "derive the same answer" to "solve the target problem independently," and lengthens the prompt. These are separate dimensions of the teacher context, so the experiment does not isolate the removal of target-specific privilege from prompt-format and instruction effects. Figure 1 labels this as "ONE CHANGE," which is misleading. A control that preserves the OPSD prompt template while substituting an unrelated solution, or that varies the disclaimer and instruction wording one at a time, would be needed to attribute the observed outcome specifically to the identity of the reference solution rather than to other context changes. As written, the results support the weaker statement that some other-problem context can work, but not the stronger conclusion that the teacher's context-induced behavior is the factor replacing the privileged solution.
- [Section 3.2 and Section 7] The paper does not directly measure teacher behavior; it infers "context-induced teacher behavior" from downstream student accuracy. The teacher's next-token distributions under OPSD, OP2SD, and Base are never compared, so alternative explanations remain plausible: the different prompt format could change teacher output length, formatting, or termination behavior, and the Forward-KL objective could interact with those changes in ways not captured by the accuracy metric. The conclusion in Section 7 that "the major source of the observed gain is not the privileged answer but the change in the teacher's token-level behavior" is therefore an interpretation rather than a demonstrated mechanism. Reporting at least a token-level analysis of teacher distributions under the two conditions, or a direct measure of distribution shift, would substantially strengthen the paper's central claim.
minor comments (4)
- [Tables 6–8 and Appendix F] The main training configurations do not state the number of training updates or the checkpoint used for the reported results; Appendix F refers to 100- and 400-update checkpoints for Qwen3-1.7B, but the 4B and 8B runs need the same information for reproducibility.
- [Appendix E] The Qwen3-1.7B setting uses a non-thinking student and a thinking teacher, and the paper's own Target-only control improves over Base in that setting. The 1.7B results therefore do not isolate the worked-solution context; this should be stated more prominently in the main text, not only in the appendix.
- [Section 4.1.1] The statement that OP2SD's "final output is longer" for Qwen3-8B would benefit from explicit mean token counts for both methods, since the budget analysis is otherwise difficult to interpret.
- [Section 6] The limitation paragraph says that the single-run concern is "particularly important" for the fixed, corrupted, and verbose contexts; the same concern should be explicitly applied to the headline OPSD/OP2SD comparison, since that is where the paper's main claim is made.
Circularity Check
No significant circularity: OP2SD is a controlled empirical intervention measured against external benchmarks, with no fitted parameter or self-citation chain forcing the conclusion.
full rationale
This paper is an empirical ablation study rather than a derivation, so the circularity patterns do not apply. The central intervention, OP2SD, is defined by replacing the teacher-only reference context with a worked example from another problem (Eq. 3) while keeping the student rollout, teacher model, and distillation objective (Eq. 4) identical to OPSD. The comparison is not a fitted parameter renamed as a prediction: the OP2SD condition is a controlled change in the teacher prompt, and the outcomes are measured on external benchmarks (AIME 2024, AIME 2025, HMMT 2025) against the base model and the OPSD recipe of Zhao et al. (2026a). The citation to Zhao et al. supplies the training setup and the OPSD baseline, not the paper's conclusion, and the conclusion does not reduce to that citation. The acknowledged limitation that each trained condition is represented by one training run (Section 6) is a statistical robustness concern about seed variance, not a circularity mechanism: the observed gaps could in principle be reversed by training-run noise, but that possibility is explicitly disclosed and does not make the intervention's definition equivalent to its measured outcome. The physics, fixed-example, and corrupted-solution controls are likewise ablations whose results are reported as point estimates with stated uncertainties. No equation in the paper defines an input in terms of the target claim, and no modeled quantity is fitted to the benchmark results and then re-reported as a prediction. The paper therefore contains no self-definitional reduction, no fitted-input-as-prediction step, and no load-bearing self-citation chain; the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption The reference solutions in the OpenThoughts Math dataset are correct and are used verbatim as teacher context.
- domain assumption The boxed-answer extractor and grader used for Avg@12, Pass@12, and Vote@12 correctly determine mathematical correctness.
- domain assumption A single training run per condition is representative of the method's performance.
- domain assumption The evaluation benchmarks (AIME 2024, AIME 2025, HMMT 2025) are uncontaminated and suitable measures of mathematical reasoning.
Cite this review
Pith. "Pith review of Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation." pith.science (2026). https://pith.science/paper/A2CYL6H2
@misc{pith2026260809228,
author = {Pith},
title = {Pith review of: Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation},
year = {2026},
howpublished = {\url{https://pith.science/paper/A2CYL6H2}},
note = {Machine review of arXiv:2608.09228}
}
abstract
On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with $\mathrm{OP}^{2}\mathrm{SD}$ (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, $\mathrm{OP}^{2}\mathrm{SD}$ improves over the base model, remains competitive with OPSD. The success of $\mathrm{OP}^{2}\mathrm{SD}$ implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.
Figures
Reference graph
Works this paper leans on
-
[1]
This intervention removes padding from the boundary between the prompt and response
This shifts the padding to the start of each sequence, restoring adjacency between the unpadded prompt and the sampled response for both policies. This intervention removes padding from the boundary between the prompt and response. Table 9:Qwen3-4B non-thinking results with left-padded loss replay. OP2SD retains a substantial improvement after right paddi...
work page 2025
-
[4]
doi: 10.18653/v1/2023.findings-acl.507
Association for Computational Linguis- tics. doi: 10.18653/v1/2023.findings-acl.507. URL https://aclanthology.org/2023. findings-acl.507/. Jonas H¨ubotter, Frederike L¨ubeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-dist...
arXiv 2023
-
[6]
URL https: //openreview.net/forum?id=VhCJItwQHn. Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras (eds.),Proceedings of the 2016 Conference on Empirical Methods in 11 Natural Language Processing, pp. 1317–1327, Austin, Texas, November
work page 2016
-
[8]
Xiaogeng Liu, Xinyan Wang, Yingzi Ma, Yechao Zhang, and Chaowei Xiao. When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning.arXiv preprint arXiv:2605.21606,
-
[9]
Unifying distillation and privileged information.arXiv preprint arXiv:1511.03643,
David Lopez-Paz, L´eon Bottou, Bernhard Sch¨olkopf, and Vladimir Vapnik. Unifying distillation and privileged information.arXiv preprint arXiv:1511.03643,
-
[10]
Final mass = 1 * (1/2)ˆ3 Final mass = 1 * (1/8) Final mass = 0.125 grams After 30 days, the mass of the radioactive sample will be 0.125 grams. === Unrelated physics Worked Example Solution End === Now solve the target mathematics problem independently. Use the worked example only as a general demonstration of clear, step-by-step problem-solving. Please r...
work page 2024
-
[12]
Zhanming Shen, Jintao Tong, Shaotian Yan, Chen Shen, Hao Chen, Wentao Ye, Xiaomeng Hu, Rui Miao, Haobo Wang, Junbo Zhao, et al. Purified OPSD: On-Policy Self-Distillation Without Losing How to Think.arXiv preprint arXiv:2607.02234,
-
[13]
Self-Distillation Enables Continual Learning.arXiv preprint arXiv:2601.19897,
Idan Shenfeld, Mehul Damani, Jonas H ¨ubotter, and Pulkit Agrawal. Self-Distillation Enables Continual Learning.arXiv preprint arXiv:2601.19897,
Show all 27 references
-
[14]
Learning by distilling context.arXiv preprint arXiv:2209.15189,
Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context.arXiv preprint arXiv:2209.15189,
-
[16]
Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L ´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,
-
[18]
on-policy context distillation for language models.arXiv preprint arXiv:2602.12275,
Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. on-policy context distillation for language models.arXiv preprint arXiv:2602.12275,
-
[19]
ISBN 9781713871088
Curran Associates Inc. ISBN 9781713871088. 12 Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. InForty-third International Conference on Machine Learning,...
-
[21]
We evaluate checkpoint 100 with 12 samples per problem and 4 seeds
This gives 3,200 target exposures, or 2.5 passes over the target set. We evaluate checkpoint 100 with 12 samples per problem and 4 seeds. We repeat the control with Qwen3-4B while disabling thinking for the student, teacher, and evaluation. Table 10 reports the complete result...
2026
-
[22]
The 400- update checkpoint obtains 51.39 Avg@12, a decrease of 3.96 percentage points from the 100-update result
Table 12 reports the four-seed averages for Base and the 100- and 400-update OP2SD conditions. The 400- update checkpoint obtains 51.39 Avg@12, a decrease of 3.96 percentage points from the 100-update result. It nevertheless remains 1.53 points above Base. The change is not a ...
2026
-
[23]
Accuracy metrics are four-seed means in percentages
Table 17:Qwen3-1.7B matched thinking-mode control on AIME 2024.The both-thinking condition does not reproduce the Avg@12 gain observed with the asymmetric training configuration. Accuracy metrics are four-seed means in percentages. Avg@12 is reported with corrected Monte Carlo...
2024
-
[25]
This result does not isolate which property of the physics context causes the decline
In this configuration, an arbitrary worked solution from a different scientific subject is not sufficient to reproduce the benefit of the other-problem mathematical context. This result does not isolate which property of the physics context causes the decline. The mathematics ...
2024
-
[26]
Within each benchmark, the highest Avg@12 point estimate is shown inboldand the second-highest is underlined . Benchmark Condition Avg@12 Pass@12 V ote@12 AIME 2024 Base49.86±0.94 76.67±0.00 70.00±2.72 OPSD55.42±0.82 78.33±0.96 65.00±1.67 OP2SD 55.35±0.83 75.83±0.83 70.00±2.36...
2024
-
[27]
It also matches or exceeds target-solution OPSD on all three benchmarks
Relative to varying-example OP2SD, the corresponding point-estimate differences are +0.21, +2.71, and +0.21 percentage points. It also matches or exceeds target-solution OPSD on all three benchmarks. The result therefore, extends the fixed-context observation beyond the Qwen3-...
2024
-
[2009]
doi: https://doi.org/10.1016/ j.neunet.2009.06.042
ISSN 0893-6080. doi: https://doi.org/10.1016/ j.neunet.2009.06.042. URL https://www.sciencedirect.com/science/article/ pii/S0893608009001130. Advances in Neural Networks Research: IJCNN2009. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chan...
2009 arXiv
-
[2014]
Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes
Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Anna Rogers, Jordan Boyd...
2023
-
[2015]
Semantic soft bootstrapping: Long context reasoning in llms without reinforcement learning.arXiv preprint arXiv:2512.05105,
Purbesh Mitra and Sennur Ulukus. Semantic soft bootstrapping: Long context reasoning in llms without reinforcement learning.arXiv preprint arXiv:2512.05105,
-
[2016]
doi: 10.18653/v1/D16-1139
Association for Computational Linguistics. doi: 10.18653/v1/D16-1139. URL https://aclanthology. org/D16-1139/. Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. DistiLLM: Towards Streamlined Distillation for Large Language Models. InForty-first International Conference ...
-
[2022]
Self-Supervised On-Policy Distillation for Reasoning Language Models.arXiv preprint arXiv:2605.17497,
Zhiquan Tan and Yinrong Hong. Self-Supervised On-Policy Distillation for Reasoning Language Models.arXiv preprint arXiv:2605.17497,
-
[2023]
URL https://doi.org/ 10.48550/arXiv.2308.08998
doi: 10.48550/ARXIV .2308.08998. URL https://doi.org/ 10.48550/arXiv.2308.08998. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.Deep Learning and Representation Learning Workshop in Conjunction with NIPS,
-
[2024]
URLhttps://openreview.net/forum?id=5h0qf7IBZZ. C ¸aglar G¨ulc ¸ehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freita...
-
[2025]
Privileged Information Distillation for Language Models.arXiv preprint arXiv:2602.04942,
Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged Information Distillation for Language Models.arXiv preprint arXiv:2602.04942,
-
[2026]
Rethinking On-Policy Self- Distillation for Thinking Models
Simran Kaur, Narutatsu Ri, Yinghui He, Liam H Fowl, and Sanjeev Arora. Rethinking On-Policy Self- Distillation for Thinking Models. InICML 2026 Workshop on Foundations of Deep Generative Models: Understanding Memorization, Generalization, and Reasoning,
2026
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.