{"id":"8cf47f82-da85-48b5-a19d-7401fe6a2504","arxiv_id":"2608.09228","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Replacing the target reference solution in on-policy self-distillation with a solution to a different problem preserves most of the accuracy gains, showing target-specific privileged information is not necessary.","lead":"A study shows that an AI teacher can improve a student model's math performance even when the sample solution it sees belongs to a different problem, not the one the student is solving. This suggests the value of self-distillation comes partly from the teacher's changed behavior under extra mathematical context, not from knowing the target answer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single training run per condition leaves the central OP2SD-versus-OPSD comparison vulnerable to optimization-seed noise; the four decoding seeds do not capture training-run variance, so the observed gaps could be reversed on replication.","rationale":"The reader's weakest assumption is that each method's single training run is representative, and the paper's own Limitations section explicitly states that the four seeds do not capture training-run variation. This is the most load-bearing concern because the abstract's implication claim ('The success of OP2SD implies ...') depends on the observed OP2SD success being a real effect of the intervention rather than a favorable optimization trajectory. The reported standard errors only cover decoding-seed variability and cannot rule out training-seed effects. The paper's multiple ablation conditions (fixed correct, fixed corrupted, physics, trivial) are each separate training runs and provide some indirect evidence that the effect is not purely a single lucky seed, especially for Qwen3-4B, where several math-context runs all improve over Base. However, the headline OPSD-versus-OP2SD comparison, and especially the 8B results that drive the 'competitive or better' claim, are not replicated. A secondary concern is that the OP2SD teacher prompt differs from the OPSD prompt not only in the worked-solution content but also in instruction wording ('solve independently' vs 'derive the same answer'), which could itself change teacher behavior; yet the fixed correct versus fixed trivial controls hold the OP2SD template constant and show that content matters, so this confound does not overturn the core claim that context-induced behavior is important. The appropriate verdict remains CONDITIONAL: the direction of the finding is plausible, but multiple training seeds are needed before the central claim is treated as settled.","tokens_in":21671,"tokens_out":19070,"duration_ms":201385,"concrete_test":"Retrain the main Qwen3-4B non-thinking OPSD and OP2SD conditions with at least four training seeds (e.g., 42, 43, 44, 45), keeping all hyperparameters and the 100-update checkpoint fixed, and evaluate Avg@12 on AIME 2024, AIME 2025, and HMMT 2025. The central claim is supported if OP2SD improves over Base in every seed on every benchmark and the current +7.56-point OP2SD advantage over OPSD on AIME 2025 remains positive in every seed; if any seed reverses the OP2SD-over-Base improvement or the AIME 2025 gap changes sign, the conclusion that the paired reference is unnecessary must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference is that OP2SD's success shows the paired reference solution is unnecessary. The evidence for this rests on single training runs for each condition: the four decode seeds only quantify sampling noise for a fixed checkpoint, as the authors acknowledge in Section 6. The magnitudes at issue include small differences at 1.7B (e.g., AIME 2025: OPSD 40.35 vs OP2SD 40.69, within 1 MC SE) and large differences at 8B (OP2SD exceeds OPSD by roughly 10 points on all three benchmarks). If training-seed variance is comparable to these gaps, the ordering could flip. The paper's own Appendix F shows the same OP2SD run varies by 3.96 Avg@12 points between the 100-update and 400-update checkpoints, indicating real optimization-trajectory sensitivity. Without replicated training seeds, the claim that OPSD gains do not come from the target reference solution is not yet robustly established; the observed improvements over Base could reflect a favorable training seed and checkpoint rather than the intervention itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether On-Policy Self-Distillation (OPSD) improves reasoning because the teacher has access to the verified solution of the target problem, or because the additional worked-solution context changes the teacher's token-level behavior. To separate these, it introduces OP2SD, which replaces the teacher-only reference solution with a problem-and-solution pair drawn from a different training instance, while keeping the student rollout, teacher model, and distillation objective unchanged. Experiments on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across AIME 2024, AIME 2025, and HMMT 2025 report that OP2SD improves over the base model and is competitive with, or better than, standard OPSD. Additional controls vary the worked example's diversity, correctness, verbosity, mathematical domain, and subject matter. The authors conclude that OPSD's gains do not necessarily require the paired target solution and that context-induced teacher behavior is an important factor.","tokens_in":21819,"tokens_out":7928,"duration_ms":86253,"significance":"If the empirical claims are robust, this is a valuable diagnostic result: it reframes OPSD not as privileged-answer transfer but as a more general context-induced teacher phenomenon, with direct implications for how on-policy distillation methods are designed and interpreted. The paper's strengths are its controlled intervention, the range of models and benchmarks, the thoughtful ablation controls (fixed example, corrupted solution, verbose solution, domain and subject mismatch), the explicit statement of limitations, and the public implementation. The main claims are empirical rather than derived, so the study's value depends on the reliability of the training and evaluation comparisons. The single-training-run design and the multiple simultaneous changes in the intervention currently limit the strength of the central conclusion.","major_comments":[{"comment":"The central comparison between OPSD and OP2SD rests on a single training run per condition. The four decoding seeds used for the reported error bars quantify sampling variation for a fixed checkpoint, not variation across optimization seeds, as the authors acknowledge in Section 6. This is load-bearing: at Qwen3-1.7B several OPSD/OP2SD differences are within the reported Monte Carlo standard error (e.g., AIME 2025: 40.35 versus 40.69), while at Qwen3-8B the 10-plus-point gaps in favor of OP2SD have no training-seed error bars. Appendix F shows that the same OP2SD condition moves 3.96 Avg@12 points between the 100-update and 400-update checkpoints, demonstrating real optimization-trajectory sensitivity. Without repeated training seeds, the observed gaps, especially the 8B results, could be reversed on replication, and the claim that OPSD gains do not come from the target reference solution is not yet robustly established. The limitation is acknowledged but not mitigated; I would require either replicated training runs for the main OPSD/OP2SD comparison or a substantially weakened conclusion.","section":"Section 6, Table 1, Appendix F"},{"comment":"The OP2SD intervention changes more than the target-reference pairing. Compared with the OPSD teacher prompt, OP2SD adds the problem statement of the auxiliary example, inserts an explicit \"not a solution or a hint\" disclaimer, changes the final instruction from \"derive the same answer\" to \"solve the target problem independently,\" and lengthens the prompt. These are separate dimensions of the teacher context, so the experiment does not isolate the removal of target-specific privilege from prompt-format and instruction effects. Figure 1 labels this as \"ONE CHANGE,\" which is misleading. A control that preserves the OPSD prompt template while substituting an unrelated solution, or that varies the disclaimer and instruction wording one at a time, would be needed to attribute the observed outcome specifically to the identity of the reference solution rather than to other context changes. As written, the results support the weaker statement that some other-problem context can work, but not the stronger conclusion that the teacher's context-induced behavior is the factor replacing the privileged solution.","section":"Figure 2 and Section 3.1"},{"comment":"The paper does not directly measure teacher behavior; it infers \"context-induced teacher behavior\" from downstream student accuracy. The teacher's next-token distributions under OPSD, OP2SD, and Base are never compared, so alternative explanations remain plausible: the different prompt format could change teacher output length, formatting, or termination behavior, and the Forward-KL objective could interact with those changes in ways not captured by the accuracy metric. The conclusion in Section 7 that \"the major source of the observed gain is not the privileged answer but the change in the teacher's token-level behavior\" is therefore an interpretation rather than a demonstrated mechanism. Reporting at least a token-level analysis of teacher distributions under the two conditions, or a direct measure of distribution shift, would substantially strengthen the paper's central claim.","section":"Section 3.2 and Section 7"}],"minor_comments":[{"comment":"The main training configurations do not state the number of training updates or the checkpoint used for the reported results; Appendix F refers to 100- and 400-update checkpoints for Qwen3-1.7B, but the 4B and 8B runs need the same information for reproducibility.","section":"Tables 6–8 and Appendix F"},{"comment":"The Qwen3-1.7B setting uses a non-thinking student and a thinking teacher, and the paper's own Target-only control improves over Base in that setting. The 1.7B results therefore do not isolate the worked-solution context; this should be stated more prominently in the main text, not only in the appendix.","section":"Appendix E"},{"comment":"The statement that OP2SD's \"final output is longer\" for Qwen3-8B would benefit from explicit mean token counts for both methods, since the budget analysis is otherwise difficult to interpret.","section":"Section 4.1.1"},{"comment":"The limitation paragraph says that the single-run concern is \"particularly important\" for the fixed, corrupted, and verbose contexts; the same concern should be explicitly applied to the headline OPSD/OP2SD comparison, since that is where the paper's main claim is made.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the OP2SD intervention is a genuinely clean experiment: it swaps the paired reference for a different problem's solution while holding the rollout, teacher, and loss fixed, and it is the first OPSD variant I know of that breaks the pairing entirely rather than filtering or reweighting the target-aligned signal. Second, the central claim is probably right in direction but the magnitudes are not stable: each condition is a single training run, and the four decode seeds only capture sampling noise for a fixed checkpoint. The paper's own Appendix F shows the same OP2SD run drops 3.96 Avg@12 points between the 100- and 400-update checkpoints, so checkpoint and optimization-seed sensitivity are real. At Qwen3-8B, OP2SD beats OPSD by ten points; that gap could easily be a favorable run rather than the intervention.\n\nWhat the paper does well: the control set is unusually thorough. Target-only, answer-only, fixed correct, locally corrupted, verbose, trivial, physics cross-subject, and coarse-domain matching are all tested, and the limitation section explicitly acknowledges the single-run issue. The problem-level analysis in Appendix G is a nice addition—it shows the gains come from increasing success on problems Base already solves occasionally, not from solving new problems. That's an honest, useful observation.\n\nSoft spots: besides the single-run issue, the physics comparison is confounded by prompt wording, format, and dataset differences, though the authors acknowledge this. The 1.7B mode-asymmetry controls are also hard to interpret for the central question, but they treat them as descriptive.\n\nNet: this is a solid diagnostic paper that reframes the privileged-information interpretation of OPSD. The direction of the finding—target-specific privilege is not necessary—is well supported by the fact that OP2SD beats Base in all nine model–benchmark groups; that's hard to explain away by seed noise alone. But the claim that OP2SD is competitive with OPSD, and any ranking between them, needs replicated training runs. I'd send it to peer review: it deserves referee time, but I'd ask the authors to run at least three training seeds per condition, or report loss curves and checkpoint selection criteria, before the magnitudes are taken as settled.","headline":"A cleanly designed intervention that reframes OPSD's privileged-information story, but the single-run-per-condition design leaves the effect sizes too shaky to take the magnitudes on faith.","tokens_in":22385,"tokens_out":2802,"would_cite":true,"duration_ms":27356,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On-policy self-distillation improves language models even when the teacher never sees the target problem's verified solution.","keywords":["on-policy self-distillation","privileged information","knowledge distillation","teacher-student","mathematical reasoning","large language models"],"falsifier":"Run OPSD and OP2SD with, say, ten independent training runs each (changing the data order or initialization) and compare the distribution of Avg@12. If the two distributions overlap substantially, the claim that paired solutions are unnecessary collapses. A complementary test is to log the teacher's next-token distributions on the same student prefixes under both contexts: if they are nearly identical, the context explanation cannot be right.","tokens_in":21446,"feed_emoji":"🎓","tokens_out":7089,"duration_ms":61909,"temperature":0.7,"pith_summary":"This paper asks whether on-policy self-distillation (OPSD) improves a language model because the teacher sees the verified solution to the exact problem the student is solving, or because any well-formed worked-solution context changes the teacher's token-level supervision. To separate the two, the authors introduce OP2SD, which replaces the paired reference with a problem and solution from a different example while keeping the student rollout, teacher, and loss identical. Across Qwen3-1.7B, 4B, and 8B on AIME 2024, AIME 2025, and HMMT 2025, OP2SD improves over the base model and stays competitive with OPSD, sometimes exceeding it. The paper concludes that access to the target's privileged solution is not necessary for the measured gains; the teacher's context-induced behavior is an important factor. The result redirects how self-distillation gains should be explained and engineered.","feed_headline":"Self-distillation gains don't need the target answer","feed_subtitle":"Replacing the paired answer with another problem's solution keeps the teacher's gain on math benchmarks.","key_machinery":"The central object is OP2SD (On-Policy Self-Distillation from Other Problems), an intervention that isolates the teacher-only context. Formally, for target $x_i$, OPSD's teacher conditions on $(x_i, y_i^*)$, while OP2SD's teacher conditions on $(x_i, x_j, y_j^*)$ for $j \\neq i$, with the prompt stating that $(x_j, y_j^*)$ is unrelated. The student sees only $x_i$ in both cases, rolls out $\\hat{y}_i \\sim p_\\theta(\\cdot | x_i)$, and both methods optimize the same clipped Forward-KL token-level distillation loss. This one-change design identifies whether the paired reference is doing causal work or merely shaping the teacher's next-token distribution.","core_discovery":"The central discovery is that the standard interpretation of OPSD as privileged-answer transfer is at least incomplete. The paper constructs a controlled intervention, OP2SD, in which the only change is the teacher-only context: instead of the verified solution to the target problem $A$, the teacher receives the problem and solution of a different example $B$, explicitly marked as unrelated. With student prompt, on-policy rollout, frozen teacher, Forward-KL objective, and optimization unchanged, OP2SD improves Avg@12 over the base model on all nine model–benchmark groups and attains the highest point estimate in eight of nine. For Qwen3-8B it exceeds OPSD by roughly 10 points on each benchmark. The paper also shows the effect is not from arbitrary context: a single fixed worked example suffices, a locally corrupted solution does not hurt, a verbose solution hurts, and cross-subject physics contexts degrade performance below base. This pattern implies the distilled signal is not the answer itself but a teacher behavior elicited by a substantive mathematical worked-solution context.","pith_inferences":["A sharper test of the context-behavior mechanism would compare the teacher's next-token distributions under the two contexts on identical student prefixes, checking whether the OP2SD teacher is already steering toward the correct answer before any privileged information could matter.","The single-fixed-example result suggests a possible 'prompt-style transfer' effect; one could test whether a fixed, well-structured worked example from another reasoning domain (e.g., code or logic) produces a similar gain on math benchmarks.","The physics-context failure may stem from surface-level differences (notation, units, style) rather than subject matter; a controlled extension would rewrite physics solutions in mathematical step-by-step form to isolate content from format.","Because each condition is one training run, the reported OPSD-vs-OP2SD ordering could reverse under training-run variance; multi-seed training runs would tell whether the advantage is stable."],"forward_implications":["Paired verified solutions can be dropped from OPSD-style training; any well-formed worked example from the same subject suffices to retain most of the gain.","Teacher-context design becomes a first-order lever: concise, correct mathematical worked examples help; verbose, trivial, or cross-subject contexts hurt or reverse the gain.","Because a single fixed example works, the data-collection cost for self-distillation could be far lower than curating per-problem reference solutions.","Comparisons between OPSD variants should control for context properties (length, correctness, domain, style) and not only for whether the reference is target-aligned.","The same Forward-KL on-policy objective with a frozen teacher can produce large gains or large degradations depending on what the teacher sees, so context should be reported alongside accuracy numbers."],"supporting_citations":[{"why":"Defines OPSD and provides the base-model comparisons and training setup that the paper dissects.","marker":"Zhao et al., 2026a"},{"why":"Introduces on-policy distillation of language models, the framework OP2SD builds on.","marker":"Agarwal et al., 2024"},{"why":"Supplies the privileged-information paradigm that the standard interpretation of OPSD relies on.","marker":"Vapnik & Vashist, 2009"},{"why":"Establishes context distillation, the alternative explanation the paper's results point toward.","marker":"Snell et al., 2022"}],"fun_headline_variants":["Teacher's context, not answer, drives self-distillation gains","Swap the answer, keep the gain: on-policy distillation insights","Distillation gains come from teacher's context, not reference answer","Self-distillation works even when teacher sees a different problem's solution","On-policy distillation gains don't hinge on the target answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each method is trained once, so the measured gaps between OPSD and OP2SD could be caused by which random training run happened to happen; the four decoding seeds do not capture that training-run variation.","fun_headline_variants_meta":{"raw":{"variants":["Teacher's context, not answer, drives self-distillation gains","Swap the answer, keep the gain: on-policy distillation insights","Distillation gains come from teacher's context, not reference answer","Self-distillation works even when teacher sees a different problem's solution","On-policy distillation gains don't hinge on the target answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1531,"prompt_tokens":940,"completion_tokens":591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":556,"tokens_out":591,"duration_ms":6994,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:08:33.931750+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OPSD and OP2SD with, say, ten independent training runs each (changing the data order or initialization) and compare the distribution of Avg@12. If the two distributions overlap substantially, the claim that paired solutions are unnecessary collapses. A complementary test is to log the teacher's next-token distributions on the same student prefixes under both contexts: if they are nearly identical, the context explanation cannot be right.","supporting_citations":[],"review_version":1}