{"id":"a1f6113d-51bf-427d-baee-3fa2b36df5b4","arxiv_id":"2608.04794","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"When self-distillation with a privileged-information teacher is used alone on hard tasks, the loss decreases but accuracy does not, because the teacher transfers one solution rather than correctness.","lead":"A Microsoft Research team tested self-distillation with privileged answers on hard reasoning, math, coding, and agent tasks. They found the training loss drops while accuracy stays flat or falls, and trace the cause to the teacher copying one reference solution instead of teaching correctness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The negative result is established only for a single-reference-solution PI; the paper never tests a PI that aggregates multiple correct solutions, so the abstract's general 'does not teach anything' overreaches its evidence.","rationale":"The reader's weakest_assumption identified exactly the gap I consider most load-bearing: the default PI is one random correct rollout, and no test uses a PI that aggregates multiple correct solutions. I agree with that assessment. The paper's own Section 8 limitation is narrower than the abstract, but the abstract's universal negative framing is what most readers will take away, and that framing is not supported by the experimental design. The paper has strong independent support for its qualified claim: faithful reproduction of SDPO's easy-setting gains, a consistent non-improvement signature across four domains, two recipes, two model scales, and several robustness splits, plus token-level measurements of the proposed mechanism. My concern does not attack any of that evidence. It attacks only the breadth of the conclusion. A natural fix—conditioning the teacher on multiple correct solutions—is explicitly left open in the Discussion, so the missing experiment is not a minor detail; it is the boundary between 'single-reference PI fails' and 'PI-conditioned dense per-token objectives fail.' Because the reader already issued a CONDITIONAL verdict, my read does not move that verdict; it reinforces it and specifies the experiment that would resolve the condition.","tokens_in":19947,"tokens_out":5195,"duration_ms":69571,"concrete_test":"Run the identical SDPO recipe on DAPO-Math and MMLU-Pro (Qwen3-8B think, same hyperparameters) with PI r replaced by a bag of K=8 correct reference solutions sampled per query, presenting all K in the teacher prompt or taking an equal-weight mixture of the K teacher distributions at each position. Monitor validation accuracy and the PI Bias Score PS* vs PS'. If accuracy improves beyond run-to-run noise, or PS* no longer strongly exceeds PS' while remaining above PS~, the central claim must be narrowed to single-reference PI. If the training signature repeats (loss down, accuracy flat, PS* >> PS'), the concern is resolved and the general conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (1) defines SD for arbitrary privileged information r, and the abstract concludes that as a lone objective SD 'does not teach anything.' But every PI actually tested is either one sampled correct solution (Section 4: 'take one random correct rollout') or a deliberately weaker single hint/skill (Section 6.1). The causal chain in Section 6 is explicitly about a teacher that has 'read one particular reference solution'; the authors never condition the teacher on a set of correct solutions (e.g., K diverse correct rollouts per query, or a teacher distribution marginalized over them), even though the Discussion names exactly this as the open construction that would make the target 'reflect the set of correct continuations.' Consequently, the data support the narrow claim in Section 8 ('in our settings...'), not the broad one in the abstract and title. If a multi-correct-solution PI restored validation accuracy or made the loss track correctness, then the failure would be a property of single-trajectory PI, not of PI-conditioned dense per-token objectives generally. This is the load-bearing scope restriction: the central negative claim is about a family of objectives, but only one member of that family is actually stressed. The positive in-domain cells (DAPO-Math think +1.0, instruct agentic +4.68) and the acknowledged limitation already point in this direction. The concern is not that the reported experiments are wrong; it is that the headline generalization is not entailed by them.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a negative result for self-distillation (SD) used as the sole training objective on difficult tasks. The authors first reproduce SDPO's reported improvement on the easy SciKnowEval setting, then apply the same objective to MMLU-Pro, DAPO-Math, CodeForces, and BFCL multi-turn agentic tool use with Qwen3-8B (and Qwen3-32B for QA), under both SDPO and OPSD training recipes and in both think and instruct modes. Their main empirical finding is a consistent training signature: the per-token SD loss decreases steadily while validation accuracy stays flat or degrades, with the largest degradation in the agentic think setting and some positive exceptions in short-response instruct settings. They then propose a PI Bias Score, computed from forward passes, which shows that a teacher conditioned on one reference solution assigns much higher probability to that solution than to a different correct solution to the same problem. Supporting measurements split the loss by verifier outcome, token type, and on/off-path status, and the paper argues for a causal chain: PI bias creates a loss blind to correctness, which spends most of its budget on low-information tokens, penalizes exploration, and produces a flatter, less decisive student. The paper's stated conclusion is qualified in Section 8 as applying 'in our settings,' but the abstract and conclusion state the broader claim that SD 'does not teach anything' as a lone objective.","tokens_in":20186,"tokens_out":4774,"duration_ms":57430,"significance":"If the qualified finding stands, the paper makes a useful contribution: it provides a concrete, reproducible negative result for a popular training recipe, with error bars, a simple pre-training diagnostic (the PI Bias Score), and a set of token- and distribution-level measurements that give a mechanistic account of why a falling per-token loss can be decoupled from task accuracy. The authors are also commendably careful in some respects: they reproduce the original easy-regime result, run both SDPO and OPSD recipes, vary model scale, reasoning mode, and PI form, and include held-out transfer benchmarks. The connection to the RLSD conditional-mutual-information bound is an asset, because it ties the empirical failure to an existing theoretical decomposition rather than leaving the mechanism as a purely post-hoc narrative. However, the significance is diminished by a scope mismatch: the abstract generalizes from a family of PI-conditioned dense objectives to 'does not teach anything,' while the experiments stress only one member of that family (a single randomly sampled correct reference solution) plus deliberately weaker hints and skills.","major_comments":[{"comment":"The headline claim that SD 'does not teach anything' as a lone objective overreaches the evidence. Equation (1) is defined for arbitrary privileged information, but every PI tested is either one randomly selected correct reference solution (Section 4: 'we sample multiple rollouts per query and take one random correct rollout') or a deliberately weaker hint/skill (Section 6.1). The paper never conditions the teacher on a set of multiple correct solutions, and the Discussion names that construction as the open one that would let the target 'reflect the set of correct continuations.' The data therefore support the Section 8 qualification ('in our settings a PI-conditioned per-token objective alone gives no learning signal aligned with task correctness'), not the abstract's universal negative. Please either add a multi-correct-solution PI condition (K diverse correct rollouts, or a teacher marginalized over them) or restrict the abstract, title, and conclusion to single-trajectory PI. If an aggregated PI restores signal, the failure is a property of this PI form, not of PI-conditioned dense objectives generally.","section":"Abstract; Section 4; Section 8"},{"comment":"The unqualified claim is also in tension with positive cells in the paper's own main table: DAPO-Math think improves by +1.0 and the agentic instruct average improves by +4.68 (Table 1). The text says 'Two domain averages come out positive, neither a gain,' which is internally confusing because a positive average is reported as if it were not positive. At minimum, the abstract and conclusion should state the qualified version (no improvement in general, with typical degradation and specific exceptions in short-response instruct settings), and Section 5.2 should explain plainly why the agentic instruct improvement is not evidence of learning (small 100-task split, short 300-token responses) rather than calling a positive average 'neither a gain.'","section":"Table 1; Section 5.2"},{"comment":"The claim that the student 'flattens rather than sharpens' is supported by rising entropy and a non-contracting variance band, but the settling of the mean teacher–student gap to zero with persistent variance could also arise from distribution shift to a different region of token space rather than purely from cancellation around a fixed mean. Reporting the variance of the per-token gap alongside the entropy trajectory would strengthen the last link of the causal chain; as written, this link rests on two indirect diagnostics rather than a direct measure of the student distribution's sharpness.","section":"Section 6.5; Figure 4f"}],"minor_comments":[{"comment":"The conclusion repeats the unqualified 'It does not' even though Section 8 explicitly limits the finding to 'in our settings.' Please align the abstract, conclusion, and title with the qualified claim.","section":"Section 9; Abstract"},{"comment":"The parenthetical 'for hints and skills, PS* and PS' do not differ in definition' is confusing: the surrounding text then contrasts PS* and PS' for whole-solution PI. Please clarify what target sequence is scored when the PI does not contain a full solution, and whether the reported hint/skill scores are comparable to the whole-solution scores in Figure 3.","section":"Section 6.1"},{"comment":"The text says 'stopwords, uncertainty markers, punctuation and whitespace account for 55.38% of the total per-token loss,' while Figure 5 reports average per-token loss per category. The two presentations are consistent, but the caption should state explicitly that Figure 5 is normalized per token so that readers do not confuse 'highest average per-token loss' with 'largest share of total loss.'","section":"Figure 5; Section 6.3"},{"comment":"The caption says 'Positive Δ in green, negative in red,' but the Section 5.2 text describes some positive cells as 'neither a gain.' Please add a clarifying note in the caption or text about which positive cells the authors regard as meaningful.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands: the paper's central negative claim is about a family of objectives, but the experiments only stress single-trajectory PI. This is fixable either by adding the multi-correct-solution PI condition or by scoping the title/abstract/conclusion, so I do not recommend rejection. I would encourage the editor to ask for the scoping revision; the qualified result is already a useful contribution. There is no circularity concern: the PI Bias Score is computed from forward passes and discriminates targets before the mechanistic story is invoked, and the RLSD bound is independent support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper earns a serious look, but its headline claim is wider than its evidence. The authors show, carefully and consistently, that when the privileged information is one randomly selected correct solution, self-distillation used alone makes the per-token loss fall while validation accuracy stays flat or degrades on four hard domains (QA, math, coding, multi-turn agentic). They reproduce SDPO's easy-domain gains, use both SDPO and OPSD recipes, two model scales, two reasoning modes, and several robustness splits, with error bars. That is real work and a real result.\n\nWhat is genuinely new: the PI Bias Score, a forward-pass-only ratio that separates how much the teacher favors the in-context solution versus another correct solution, is a cheap tool practitioners can use before training. The causal chain (PI bias, loss blind to correctness, uninformative tokens absorbing the loss, exploration penalized, flatter student) is measured link by link, not just asserted. The weak-PI experiments with hints and skills are a nice control: both over-specific and under-specific PI fail, for opposite reasons.\n\nThe soft spots. The abstract says 'does SD teach anything? ... it does not.' That is not entailed by the experiments. Every PI actually tested is a single trajectory: one random correct rollout, or a deliberately weaker hint or skill. The authors never condition the teacher on a set of correct solutions, even though the Discussion names exactly that as the open construction that would encode correctness. The positive in-domain cells (agentic instruct +4.68, DAPO-Math +1.0) and the paper's own limitation statement point the same way: the support is for 'in our settings, no learning signal,' not for 'PI-conditioned dense objectives never teach.' Also minor: OPSD is tested only via the paper's own setup, not reproduced in its original regime; code is said to be in supplementary material but not linked; raw data are not shipped. The PI Bias Score's interpretation leans on the assumption that a different correct solution is the right contrast for 'correctness'—reasonable, but worth stating as an assumption.\n\nThe central conclusion survives in its qualified form. Who this is for: anyone using SDPO, OPSD, or similar self-distillation losses on reasoning tasks, and anyone designing privileged information. It deserves a serious referee; the scope restriction is fixable with a rewritten abstract and, ideally, one experiment with a multi-correct-solution PI. I would accept it for review and push hard on the generalization claim.","headline":"Careful negative result for single-trajectory PI self-distillation, but the abstract's 'does not teach anything' overreaches the evidence.","tokens_in":20758,"tokens_out":2660,"would_cite":true,"duration_ms":28348,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that self-distillation as a lone objective, with no reward term, does not improve task accuracy on difficult reasoning tasks, because a teacher conditioned on one privileged reference solution encodes that trajectory…","keywords":["self-distillation","privileged information","PI bias score","per-token distillation loss","reasoning degradation","large language models","training dynamics","reference solution"],"falsifier":"Run the identical self-distillation recipe with privileged information aggregated over many correct solutions to each query, for example a teacher conditioned on a set or a compressed summary of several correct rollouts, and record whether validation accuracy then tracks the falling loss. If accuracy improves, the paper's PI-bias mechanism would be specific to the single-rollout PI form; if accuracy still stays flat while loss falls, the causal chain would survive the aggregation.","tokens_in":19721,"feed_emoji":"📉","tokens_out":7393,"duration_ms":78065,"temperature":0.7,"pith_summary":"The paper asks whether self-distillation (SD), in which a student is trained to match a teacher that is conditioned on privileged information such as a reference solution, teaches anything when it is the only training signal. The paper reproduces the gains reported for SD on an easy benchmark, then applies the same implementation to difficult question answering, mathematics, coding, and multi-turn agentic tool use, and finds that the per-token loss falls steadily while validation accuracy stays flat or degrades. The paper attributes this to a mechanism beginning with PI bias: a teacher that has seen one reference solution favors that trajectory over other correct solutions, so the dense per-token target is nearly blind to whether a rollout is correct. The conclusion is that as a lone objective, SD optimizes a signal decoupled from task success, and that a falling SD loss should not be read as evidence of learning.","feed_headline":"Self-distillation alone does not improve hard reasoning","feed_subtitle":"Per-token loss falls steadily, but validation accuracy stays flat or drops across math, code, and agentic tool use.","key_machinery":"The load-bearing instrument is the PI Bias Score, $\\text{PS}_t(w)=s^T_t(w)-s^S_t(w)$, the teacher-minus-student average log-probability of appending a target sequence $w$ after a student prefix; it separates a teacher that encodes correctness from one that encodes a single trajectory, and it is cheap enough to run before training. The objective under test is the stop-gradient dense divergence of Eq. (1), in which the student matches a teacher conditioned on privileged information at every token; the paper's token-type attribution, on-path versus off-path KL, and teacher–student gap measurements turn that objective into the causal chain from PI bias to a flattened student. The whole-solution PI is the default target, with hints and skills as weak-PI controls.","core_discovery":"The paper's central claim is that a PI-conditioned per-token objective, used alone, carries no learning signal aligned with task correctness on difficult tasks. Applying the SDPO and OPSD recipes to MMLU-Pro, DAPO-Math, CodeForces, and BFCL with Qwen3-8B (and Qwen3-32B for scale), the paper observes the same signature: loss decreases, validation accuracy does not improve and typically degrades, by up to 3.51 points on average in the agentic domain and 7.0 points on an individual benchmark. The explanation is a measured causal chain: the teacher's PI Bias Score is high (0.52) for the in-context solution and at most 0.02 for a different correct solution, so the teacher transfers a trajectory rather than correctness; the resulting loss overlaps for correct and incorrect rollouts; 55.38% of the loss lands on stopwords, uncertainty markers, punctuation, and whitespace; exploratory off-path tokens in correct rollouts incur about four times the KL of on-path tokens; and the student ends flatter, with rising entropy and a teacher–student gap that closes only through cancellation. The paper concludes that what fails is the target, not the density or the recipe, and that weakening the PI (hints or skills) also fails, so neither over-specific nor under-specific PI encodes correctness.","pith_inferences":["The paper does not test a privileged signal that aggregates multiple correct solutions; if such a target made the teacher's distribution reflect the set of valid continuations, its own causal account predicts the loss would become correctness-aligned, a testable route beyond the paper's findings.","The token-type analysis implies a concrete intervention the paper leaves implicit: reweighting or masking the per-token loss by information content or verifier feedback could shift the gradient from stopwords and uncertainty markers toward answer-determining tokens.","The PI Bias Score could serve as a cheap pre-training diagnostic for other conditioning schemes such as execution feedback, potentially predicting out-of-distribution degradation before compute is spent."],"forward_implications":["A falling self-distillation loss during training is not evidence that the student is learning; validation accuracy is the diagnostic that separates the two cases.","The PI Bias Score can screen a candidate privileged signal before training: a large gap between the in-context solution and a different correct solution predicts trajectory transfer rather than skill transfer, while a uniformly small score predicts that the signal will move the student toward nothing.","Both over-specific and under-specific privileged information fail, so PI is not a free hyperparameter with a benign middle setting; the target itself must encode the set of correct continuations.","Because the failure reproduces under both the SDPO and OPSD recipes and across reasoning modes, model sizes, and domains, fixes need to change the target, not the divergence direction, teacher schedule, or clipping.","On easy tasks with short bounded answers, one reference solution nearly spans the space of correct answers and little exploration exists, which explains why the reported easy-setting gains reproduce and do not extend to harder reasoning."],"supporting_citations":[{"why":"Supplies the self-distillation objective and recipe (SDPO) that the paper reproduces on SciKnowEval and then applies to difficult tasks; its reported easy-setting gains are the claim being tested.","marker":"[Hübotter et al., 2026]"},{"why":"Supplies the OPSD recipe, the second instantiation of Eq. (1), and the observation that stylistic tokens carry high divergence, which the paper unifies into the token-type loss analysis.","marker":"[Zhao et al., 2026]"},{"why":"Supplies the conditional mutual-information leakage decomposition of the loss that the PI Bias Score empirically measures.","marker":"[Yang et al., 2026a]"},{"why":"Documents the reasoning degradation that this paper traces to PI bias through token- and distribution-level measurements.","marker":"[Kim et al., 2026]"},{"why":"Defines GRPO, the verifiable-reward baseline that self-distillation is positioned against and compared with in prior reports.","marker":"[Shao et al., 2024]"},{"why":"Provides MMLU-Pro, the general-QA training and in-domain evaluation benchmark.","marker":"[Wang et al., 2024]"},{"why":"Provides DAPO-Math, the mathematics training data used for the PI-form and response-length experiments.","marker":"[Yu et al., 2025]"},{"why":"Provides BFCL, the multi-turn agentic benchmark where the largest average degradation is observed.","marker":"[Patil et al., 2024]"}],"fun_headline_variants":["Self-distillation alone fails on hard tasks","Teacher bias breaks self-distillation on tough problems","Loss falls, accuracy stalls: self-distillation's limit","Distilling a biased teacher: no learning on hard tasks","PI-conditioned self-distillation: signal decoupled from success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result assumes the privileged information is one reference solution sampled as a single random correct rollout; if a privileged signal that pools many correct solutions restored a correctness-aligned target, the conclusion would apply only to that narrow PI form, not to all PI-conditioned dense per-token objectives.","fun_headline_variants_meta":{"raw":{"variants":["Self-distillation alone fails on hard tasks","Teacher bias breaks self-distillation on tough problems","Loss falls, accuracy stalls: self-distillation's limit","Distilling a biased teacher: no learning on hard tasks","PI-conditioned self-distillation: signal decoupled from success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000498,"raw_usage":{"total_tokens":2544,"prompt_tokens":1157,"completion_tokens":1387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":773,"completion_tokens_details":{"reasoning_tokens":1310}},"tokens_in":773,"tokens_out":1387,"duration_ms":14365,"temperature":1.0,"reasoning_tokens":1310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:24:50.954426+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical self-distillation recipe with privileged information aggregated over many correct solutions to each query, for example a teacher conditioned on a set or a compressed summary of several correct rollouts, and record whether validation accuracy then tracks the falling loss. If accuracy improves, the paper's PI-bias mechanism would be specific to the single-rollout PI form; if accuracy still stays flat while loss falls, the causal chain would survive the aggregation.","supporting_citations":[],"review_version":1}