{"id":"d5e2a973-6e4c-46d8-bc00-1913027d8d65","arxiv_id":"2608.09836","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TIDE, a token-level on-policy distillation method that suppresses student-excess tokens and analytically recovers teacher-preferred deficit tokens, improves math reasoning accuracy while preventing repetitive degenerate generations.","lead":"The paper identifies a failure mode in on-policy distillation where student and teacher agree on repetitive nonsense, then proposes TIDE, which focuses training on mismatched tokens and separately handles tokens the student over-produces and tokens the teacher prefers but the student rarely samples. On math benchmarks with a weak teacher-student pair, TIDE raises accuracy substantially and cuts response length by about a factor of three.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TIDE's deficit branch trusts the teacher's top-K at exactly the high-mismatch states where the paper's own loop-continuation evidence (Sec. C.1) shows the teacher can be corrupted; robustness to teacher hacking is unverified, so the central claim rests on an untested precondition.","rationale":"We read the paper in good faith and checked the mathematical core. Proposition 1's identity (Eq. 2) is correct: with h(a)=2(e^{a/2}-1), E_q[-h(a)∇log q] = -2 Σ sqrt(p/q)∇q = 4∇H²(p,q), using Σ∇q=0. Proposition 2's decomposition d_t = KL(p̄||q^K) - log Q_t is also correct. The empirical ablations are internally consistent and the two teacher-student pairs show the claimed direction-dependent pattern. We therefore did not find a derivation gap or an internal inconsistency.\n\nThe load-bearing weakness is the teacher-reliability precondition for the deficit branch. The paper itself demonstrates that the teacher is hackable: repetitive student prefixes collapse teacher entropy and make the teacher continue loops in 93% of cases. TIDE's deficit branch selects states by d_t, which is large when the student under-covers the teacher's top-K. If the student under-covers a degenerate top-K, TIDE would distill the degeneration. The selection criterion does not explicitly test whether the teacher's top-K is informative; it assumes it is. The strong-mismatch results suggest that, in that run, the teacher at deficit-selected states was informative (otherwise TIDE would lock in loops and shorten length, which it does not), but no direct measurement is provided. This is exactly the reader's weakest_assumption, and it is also the paper's own stated limitation.\n\nWe propose a concrete, low-cost check: log teacher entropy and top-K reference coverage at deficit-selected states during a TIDE run. If those states are informative, the concern is empirically resolved for the tested pair; if not, the central claim is conditional on an unverified assumption. This matches the reader's CONDITIONAL verdict, so we recommend no change.","tokens_in":24354,"tokens_out":14075,"duration_ms":124757,"concrete_test":"During one TIDE training run on Qwen3-8B→Qwen3-1.7B-Base, log the states selected by the deficit gate (m+_t=1) and compute, at each such state: (i) teacher next-token entropy, (ii) whether the teacher's top-5 contains the reference-answer continuation (or the token that leads to a correct rollout), and (iii) repetition count in the preceding context. Compare these distributions against states selected by the excess gate and against the repetitive-prefix states from Section C.1. If deficit-selected states have teacher entropy as low as loop states, or top-5 reference coverage no better than the unselected average, the 'locally reliable teacher' assumption is violated at the very states TIDE relies on; if deficit-selected states show high entropy and above-average reference coverage, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TIDE's central claim—consistent gains over OPD under strong mismatch—depends on the deficit branch (Eq. 4, Algorithm 1) using the teacher's conditional top-K distribution as a training target at student-visited states. The paper's own Section 2.2 (Fig. 4c) and Section C.1 (Fig. 6) show this teacher distribution is not locally reliable everywhere: after sixteen repetitions, student-teacher KL drops 63x and teacher entropy collapses; from naturally occurring loop prefixes, the teacher continues the same loop in 93% of cases. If a TIDE-trained student ever visits such a corrupted state, and the deficit score d_t is large (student mass on the teacher's degenerate top-K is small), the deficit branch would explicitly reinforce the degenerate continuation, locking in degeneration rather than removing it. The d_t gate does not filter by teacher reliability; it only measures student coverage of the teacher's top-K. The strong-mismatch success on Qwen3-8B→Qwen3-1.7B-Base is evidence the teacher is reliable enough at the states actually selected in that configuration, but the paper reports no direct measurement of teacher top-K quality at deficit-selected states. Section 5's stated limitation 'TIDE assumes a locally reliable teacher' concedes exactly this gap. If the assumption fails for another teacher-student pair or another failure mode, the method's advantage could reverse. This is a load-bearing empirical precondition, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a failure mode of on-policy distillation that it calls degenerate agreement: students learn to produce repetitive loops that achieve near-perfect token-level agreement with the teacher while the overall response is globally degenerate. The authors propose a token-level method, TIDE, which separates teacher-student mismatch into student-excess and student-deficit directions. For student-excess tokens, it applies a bounded Hellinger-shaped correction; for student-deficit tokens, it uses an analytic teacher top-K objective that does not require the deficit token to be sampled. The paper proves two propositions: one that the Hellinger transformation is bounded, locally faithful to the log-ratio, and equals the gradient of the squared Hellinger divergence before position selection, and one that the deficit score decomposes into within-top-K KL mismatch plus a coverage gap. Experiments on nine mathematical reasoning benchmarks with two teacher-student pairs report that TIDE consistently outperforms OPD and several baselines, especially under strong mismatch, with large reductions in response length and formatting failures.","tokens_in":24681,"tokens_out":5413,"duration_ms":46447,"significance":"If the empirical claims hold, TIDE is a valuable contribution to on-policy distillation: it provides a clear conceptual decomposition of mismatch into excess and deficit, offers a principled bounded correction for the unstable log-ratio, and addresses the previously neglected coverage problem with analytic top-K guidance. The two propositions are clean, correctly proved, and give the method a sound theoretical core. The paper also ships code and reports detailed ablations that isolate the contributions of selection, excess suppression, and deficit recovery. However, the central empirical claim of consistent gains rests on two load-bearing points that are not fully established: the reliability of the teacher target at the states selected by the deficit branch, and the statistical stability of the reported single-run accuracy numbers.","major_comments":[{"comment":"The deficit branch of TIDE uses the teacher's top-K distribution as the training target at states selected by the deficit score d_t (Eq. 4), yet the paper's own continuation experiments (Sec. C.1, Fig. 6) show that after a repetitive student prefix the Qwen3-8B teacher continues the same loop in 93% of cases, and Fig. 4c shows teacher entropy collapses with repetition. The gate d_t only measures how well the student covers the teacher's top-K, not whether the teacher's top-K is a reliable target at that state, so if the student ever visits a corrupted state with large d_t, the deficit branch would explicitly reinforce the degenerate continuation. The paper states in Section 5 that TIDE assumes a locally reliable teacher, but it provides no direct measurement of teacher top-K quality at the states actually selected by the deficit gate; this is a load-bearing empirical precondition for the central claim of consistent gains under strong mismatch. I request a diagnostic (e.g., teacher top-K accuracy or entropy at deficit-selected states, or an analysis of how often deficit-selected states are within repetitive loops) and, if the precondition fails, an explicit safeguard.","section":"Section 5 / Sec. C.1 / Eq. (4)"},{"comment":"All accuracy and generation numbers are from single runs with no standard errors, confidence intervals, or multiple seeds, although the abstract and Section 4.2 claim that TIDE \"consistently outperforms\" baselines. Some differences are small (e.g., Table 1 weak mismatch: TIDE 46.7 vs FiRe-OPD 46.6 and AOPD 46.2), and without variance estimates the cross-pair and per-benchmark comparisons are difficult to evaluate. Please provide at least three seeds for the main comparison, or if compute constraints are binding, bootstrap confidence intervals over the nine benchmarks and a clear statement of which differences are statistically meaningful.","section":"Tables 1–5, Section 4.2"},{"comment":"Proposition 1's unbiased gradient identity (Eq. 2) is derived for the expectation over a token sampled from the student before any position selection, but the actual excess loss in Algorithm 1 applies a quantile mask m^-_t and token-mean normalization over the batch. The resulting update is therefore not the exact gradient of the squared Hellinger divergence, and the statement in Section 3.1 that Hellinger shaping \"remains the exact gradient of a proper divergence\" is only true for the unselected expectation. Please clarify the scope of Proposition 1 in the text, or, if the divergence interpretation is claimed for the full gated objective, provide the corresponding derivation with the selection mask.","section":"Section 3.1, Eq. (2), Algorithm 1"}],"minor_comments":[{"comment":"The sentence \"Accordingly, we take the following the policy gradient in OPD\" has a grammar error and should read \"Accordingly, we use the following policy gradient in OPD.\"","section":"Section 3.1"},{"comment":"The phrase \"repeatingI hope it is correct149 times\" is missing spaces around the quoted phrase and the number; please fix the formatting.","section":"Section 2.2"},{"comment":"The sentence \"Table 4 and Fig. 5 shows that\" should be \"Table 4 and Fig. 5 show that\" for subject-verb agreement.","section":"Section 4.3"},{"comment":"The column header \"Dist.-4\" is abbreviated, while the text and Appendix C use \"Distinct-4\"; please make the notation consistent throughout the paper.","section":"Table 5"},{"comment":"In the proof of Proposition 2, the display for the decomposition uses an unusual symbol in the KL term; standard notation such as D_KL(\\bar p_t \\parallel q^K_t) would improve readability.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a strong candidate for publication if the authors add a reliability diagnostic for the deficit branch and some form of uncertainty quantification for the main accuracy tables. The paper is well within the journal's scope, and the theoretical propositions are a clear strength. The load-bearing concern is not an internal inconsistency but an empirical precondition that needs direct evidence before the \"consistently outperforms\" claim can be fully trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the TIDE paper. The short version: it names a genuine failure mode in on-policy distillation—student-induced teacher hacking, where repetition drives the teacher into low-entropy agreement with a degenerate loop—and it builds a two-branch correction that works in the tested settings. The core math is simple and correct. The deficit branch's analytic top-K injection is the actual novelty; the Hellinger excess shaping is a bounded transformation with a clean gradient identity, but it is not far from existing bounded advantages. The paper does well to show the failure mode is not a tail event: loop rollouts rise from 17% to 48%, teacher continues the loop in 93% of natural loop prefixes, and matched-token supervision alone barely helps while mismatch selection doubles accuracy. The ablations in Table 4 and Figure 5 are consistent with the story.\n\nThe soft spots are empirical, not logical. The paper does not violate its own math. The teacher-reliability assumption is the real one: the deficit branch takes the teacher's top-K as ground truth at exactly the high-mismatch states that matter most. The paper's own evidence shows the teacher can be corrupted in repetitive states. If a student at such a state tries to exit the loop, the teacher's top-K still says 'continue', the exit token shows up as student-excess and gets suppressed, and the deficit branch reinforces the loop. That would lock in degeneration. Empirically TIDE shortens outputs and raises accuracy, so the assumption holds for the Qwen3 pairs tested; but the paper never measures teacher top-K quality at deficit-selected states, which is the one number that would de-risk the mechanism. The stated limitation in Section 5 is honest, but it is a load-bearing precondition for the 'consistently outperforms' claim.\n\nMinor: single-run tables with no variance estimates; hyperparameters (lambda, keep-rates) were swept on the same nine benchmarks the paper reports; code link without a commit hash. None of these break the argument.\n\nVerdict: worth a serious referee. Send it to review, and make the review ask for (1) multiple seeds or at least paired runs, (2) a direct diagnostic of teacher reliability at deficit-selected states, and (3) a stress test with a deliberately weaker or corrupted teacher. If those come back clean, this is a solid contribution. If not, the method still has value but the advertised scope needs narrowing.","headline":"A real failure mode with a clean two-branch fix; the teacher-reliability caveat is the one thing a referee should push on.","tokens_in":25201,"tokens_out":3298,"would_cite":true,"duration_ms":30796,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that token-level agreement in on-policy distillation can mask degenerate loops, and that routing supervision to severe teacher–student mismatch with separate excess and deficit corrections turns that failure into a…","keywords":["on-policy distillation","token-level mismatch","student-excess tokens","student-deficit tokens","Hellinger divergence","degenerate agreement","teacher top-K recovery","LLM post-training"],"falsifier":"Build a dataset of naturally occurring repetitive student prefixes and check, at loop states, whether the teacher's top-16 support contains any non-repetitive exit token. The paper reports the teacher continues the loop in 93% of greedy continuations; if the same pattern holds for the top-16 candidates — that is, every candidate at loop states is a loop continuation — then TIDE's deficit branch cannot recover exits and its length and accuracy gains should disappear, confirming that the method depends on the teacher remaining locally reliable.","tokens_in":24173,"feed_emoji":"🎯","tokens_out":9774,"duration_ms":83072,"temperature":0.7,"pith_summary":"This paper argues that on-policy distillation (OPD) should stop rewarding token-level agreement between teacher and student, because repetitive loops can make a globally degenerate student response look locally well-aligned: in the paper's diagnostic, repeated prefixes drive teacher–student KL down to 0.0004 nats with 100% top-1 agreement. It identifies two useful mismatch directions: student-excess tokens, which the student overproduces and whose log-ratio corrections grow unbounded, and student-deficit tokens, which the teacher prefers but the student rarely samples. The proposed TIDE method suppresses excess tokens with a bounded Hellinger-shaped correction and restores deficit tokens by analytically distilling the teacher's top-K distribution at mismatched positions, discarding matched tokens from the loss. On math reasoning benchmarks, TIDE is claimed to outperform standard OPD and recent baselines, and under strong teacher–student mismatch to raise Avg@8 from 6.9% to 20.3% while reducing average response length by a factor of 3.6 and sharply cutting formatting failures.","feed_headline":"Distill from mismatch, not token agreement","feed_subtitle":"Bounded excess suppression plus analytic top-K recovery lift Avg@8 from 6.9% to 20.3% and cut length 3.6x.","key_machinery":"The load-bearing mechanism is the pairing of two quantile-gated corrections on student-visited states. The excess branch's Hellinger shaping $h(a)=2(e^{a/2}-1)$ is bounded in $(-2,0)$ for $a<0$, satisfies $h(a)=a+O(a^2)$ near agreement, and yields $E_{v\\sim q}[-h(a(v))\\nabla_\\theta\\log q(v|s)]=4\\nabla_\\theta H^2(p,q)$, so it converts the unstable reverse-KL tail into the exact gradient of a proper divergence. The deficit branch's score $d_t=D_{KL}(\\bar p_t\\Vert q^K_t)+(-\\log Q_t)$ decomposes into probability misallocation inside the teacher's top-$K$ plus the student's total missing mass on that set, which is why the correction is a direct cross-entropy to $\\bar p_t$ rather than a sampled advantage. Matched positions receive zero weight, and the two gates are independent, so a position can be updated by neither, one, or both branches.","core_discovery":"The paper's central claim is that informative supervision in OPD lives in directional teacher–student mismatch rather than agreement. Standard OPD minimizes a trajectory-level reverse KL through sampled token advantages $a_t=\\log[p(o_t|s_t)/q_\\theta(o_t|s_t)]$; this objective is zero-forcing, so it penalizes student-excess tokens harshly while offering almost no pressure on teacher-preferred tokens that the student rarely samples. The paper proposes TIDE, which gates positions by mismatch severity in two independent branches: the excess branch applies $h(a)=2(e^{a/2}-1)$ to negative advantages, a bounded monotone transform that is first-order equal to the log-ratio near agreement and, before selection, equals the gradient of the squared Hellinger divergence ($4\\nabla_\\theta H^2(p,q)$); the deficit branch scores each state by $d_t=\\sum_{v\\in K_t}\\bar p_t(v)\\log[\\bar p_t(v)/q_\\theta(v|s_t)]$ and distills the teacher's renormalized top-$K$ distribution at the worst states, bypassing sampling. The paper reports this combination raises Avg@8 from 6.9% to 20.3% under strong mismatch, reduces the average response length from 22,395 to 7,294 tokens, and cuts responses without a boxed answer from 65.5% to 5.4%.","pith_inferences":["Going beyond the paper, the locally-reliable-teacher assumption could be converted into an adaptive gate: monitor teacher entropy or loop-continuation rate on student prefixes and disable or down-weight the deficit branch when the teacher's top-$K$ itself collapses into repetition.","A testable extension is to use the deficit score as a teacher-health diagnostic: if $d_t$ stays large at loop states but every top-$K$ candidate is a loop token, the bottleneck is teacher-side, and a trustworthy system should stop trusting that teacher's supervision at those states.","The two-branch design is not tied to mathematics: code, dialogue, and multilingual post-training also exhibit repetitive loops, so TIDE's excess suppression and top-$K$ recovery could plausibly transfer wherever a locally reliable teacher is available."],"forward_implications":["OPD pipelines should allocate supervision by the direction and accessibility of disagreement, not by teacher–student overlap.","Bounded Hellinger shaping can replace clipping or power transforms for negative token advantages while keeping the update grounded in a divergence gradient.","Teacher-preferred tokens that are rarely sampled can be recovered analytically from a teacher top-$K$ readout, removing the need to wait for a lucky rollout.","Under strong teacher–student mismatch, concentrating on mismatched tokens with these two corrections yields both higher accuracy and shorter, better-formatted responses than standard OPD.","Matched tokens can be dropped from the distillation loss without sacrificing accuracy, since supervising only the most mismatched 20% of positions more than doubles Avg@8 relative to matched-only supervision."],"supporting_citations":[{"why":"Defines the on-policy distillation objective and sampled-token policy-gradient estimator that TIDE modifies.","marker":"Lu & Lab, 2025"},{"why":"Establishes OPD as a post-training paradigm with dense teacher feedback at student-visited states.","marker":"Agarwal et al., 2024"},{"why":"Documents abrupt length inflation and repetitive generation in OPD, motivating the degenerate-agreement analysis.","marker":"Luo et al., 2026"},{"why":"Provides the bounded power-transformation baseline (PowerOPD) that TIDE's Hellinger shaping is contrasted with for stabilizing sampled-token rewards.","marker":"Zhao et al., 2026a"},{"why":"Supplies the asymmetric token-level baseline AOPD whose handling of positive versus non-positive advantages TIDE extends.","marker":"Jia et al., 2026"},{"why":"Gives the teacher–student top-K overlap criterion used to define weak versus strong mismatch settings.","marker":"Li et al., 2026a"},{"why":"Identifies the low-KL agreement trap on corrupted prefixes, closely related to the student-induced teacher hacking mechanism.","marker":"Xin et al., 2026"},{"why":"Underpins the reverse-KL zero-forcing and exclusive behavior that drives the directional asymmetry between excess and deficit tokens.","marker":"Minka et al., 2005"},{"why":"Provides the detached policy-gradient estimator used by the OPD and TIDE losses.","marker":"Schulman et al., 2017"}],"fun_headline_variants":["Mismatch beats agreement in distillation","TIDE: fix excess, restore deficit tokens","From agreement to mismatch: better LLM distillation","Mismatch-based distillation lifts Avg@8 3x","TIDE triples Avg@8 and cuts length 3.6x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The teacher's next-token opinions stay trustworthy even when the student's own output has become repetitive or degenerate.","fun_headline_variants_meta":{"raw":{"variants":["Mismatch beats agreement in distillation","TIDE: fix excess, restore deficit tokens","From agreement to mismatch: better LLM distillation","Mismatch-based distillation lifts Avg@8 3x","TIDE triples Avg@8 and cuts length 3.6x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000804,"raw_usage":{"total_tokens":3627,"prompt_tokens":1133,"completion_tokens":2494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":749,"completion_tokens_details":{"reasoning_tokens":2416}},"tokens_in":749,"tokens_out":2494,"duration_ms":17335,"temperature":1.0,"reasoning_tokens":2416,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:45:41.017906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a dataset of naturally occurring repetitive student prefixes and check, at loop states, whether the teacher's top-16 support contains any non-repetitive exit token. The paper reports the teacher continues the loop in 93% of greedy continuations; if the same pattern holds for the top-16 candidates — that is, every candidate at loop states is a loop continuation — then TIDE's deficit branch cannot recover exits and its length and accuracy gains should disappear, confirming that the method depends on the teacher remaining locally reliable.","supporting_citations":[{"cited_title":"On-policy distillation of language models: Learning from self- generated mistakes","cited_arxiv_id":null,"evidence_quote":"Establishes OPD as a post-training paradigm with dense teacher feedback at student-visited states."}],"review_version":1}