Pith. sign in

REVIEW 3 major objections 8 minor 14 references

Local teacher–student agreement in on-policy distillation is mostly agreement on failed answers, not safe imitation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 14:34 UTC pith:GWO37TFU

load-bearing objection Useful diagnostic framing for OPD, but the flagship 67% is mostly base rates; the real load-bearing pieces are the teacher-4/4 control and the localization boundary. the 3 major comments →

arxiv 2607.23731 v1 pith:GWO37TFU submitted 2026-07-26 cs.LG

Outcome-Confounded Local Supervision in On-Policy Distillation

Guoqing Ma This is my paper
classification cs.LG
keywords on-policy distillationoutcome-confounded supervisionagreement-on-failureteacher-student divergencemathematical reasoninglocalization limitationprocess supervisionknowledge distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

On-policy distillation trains a student on its own generated trajectories while a larger teacher scores every token the student actually visits. Practitioners often treat local agreement as safe to copy and local disagreement as an error signal. This paper shows both readings are confounded by whether the finished trajectory is correct. Crossing pointwise divergence with final-answer correctness yields four regimes; in math-reasoning runs the largest by far is agreement-on-failure—low divergence on wrong answers, about two-thirds of response-token mass for two model pairs. Even when the teacher can solve the prompt from scratch every time, that mass stays nontrivial. Simple training fixes that imitate, mask, or contrast whole trajectories do not reliably shrink it. The deeper point is a localization limit: a token rule that only sees local divergence and a single trajectory outcome cannot tell where a failing path became unrecoverable, so denser teacher likelihoods alone do not supply process-level credit assignment.

Core claim

Dense local teacher likelihoods on student prefixes are outcome-confounded: the same local agreement or disagreement appears under opposite final outcomes. In the reported setting, agreement-on-failure dominates pooled response-token mass (~68% on full discovery sets; still ~15% on teacher-always-solves prompts), and tokenwise rules of the form weight = f(local divergence, trajectory outcome) cannot localize the first unrecoverable step inside a failed trajectory.

What carries the argument

The outcome-resolved four-regime diagnostic: it bins each response token by pointwise teacher–student divergence (low vs high) and by the verifier’s final-answer label (correct vs wrong), producing safe imitation, productive divergence, harmful divergence, and agreement-on-failure, then measures response-token mass in each cell.

Load-bearing premise

The whole diagnosis hangs on broadcasting a single rule-based final-answer correct/wrong label to every token in the trajectory as if that label were a fair process outcome axis.

What would settle it

Re-run the same eight-seed diagnostic with a trusted process-level or prefix-recoverability label instead of (or in addition to) final-answer correctness; if agreement-on-failure collapses and a tokenwise rule using only local divergence plus that richer label can separate pre-error from post-error tokens on failed traces, the central localization claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Filtering only wrong trajectories or reweighting only by local divergence will mix or discard opposite training cases and should not be expected to fix process errors by itself.
  • Standalone teacher accuracy on a prompt does not guarantee useful local signals once the student has left a recoverable path.
  • OPD evaluations should report the four-regime mass split alongside accuracy, because similar scores can come from expanding safe imitation or merely changing length and failure modes.
  • Closing the gap requires extra positional information: process labels, teacher continuations from student prefixes, or token-level alignment between correct and wrong rollouts of the same prompt.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any dense token objective that pairs a local expert score with only a trajectory-level reward will hit the same indistinguishability wall unless it imports one of the three localization sources the paper names.
  • Early termination, process rewards, and multi-rollout success/failure alignment are not optional tricks but different ways of buying the missing positional bit.
  • If agreement-on-failure remains large under process verifiers, the bottleneck may shift from outcome confounding to teacher–student distribution mismatch on corrupted prefixes rather than label broadcast.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper studies on-policy distillation (OPD) for mathematical reasoning and argues that token-level teacher–student divergence is "outcome-confounded": local agreement need not indicate safety and local disagreement need not indicate error. The authors introduce a four-regime diagnostic crossing a pointwise divergence statistic d_t (the k3 estimator, thresholded at τ) with a rule-based final-answer label z(y). In an eight-seed study (Qwen3-8B student, Qwen3-32B teacher, 1,024 rollouts), agreement-on-failure (low divergence on wrong trajectories) accounts for 67.84% of pooled response-token mass; a Qwen2.5-7B/32B pair gives 67.68%. Threshold sweeps, sequence-level coverage audits, format/truncation checks, and a teacher-4/4 capability control (AoF remains 14.76% at 86.91% student accuracy) are reported. Three matched 150-step training probes (pure OPD, correct-only masking, correct-only + same-prompt contrastive) do not consistently reduce agreement-on-failure. The authors conclude with an indistinguishability argument: any tokenwise rule w_t = f(d_t, z(y)) cannot localize where a failed trajectory became unrecoverable, and they explicitly scope the contribution as diagnostic.

Significance. If the framing is corrected, this is a useful measurement contribution to a fast-moving OPD literature. Strengths: exact pooled numerators/denominators with seed-level bootstrap intervals (Table 3), a genuine external-validity replication on a second model family, a well-designed teacher-4/4 control that cleanly separates standalone teacher competence from teacher usefulness on student prefixes, and honest negative training results rather than an overclaimed method. The indistinguishability boundary for w_t = f(d_t, z(y)) rules is simple but correctly stated and usefully organizes what extra information (process labels, counterfactual continuations, cross-rollout alignment) would be needed. The paper's most interesting empirical finding — that pointwise divergence is nearly statistically independent of trajectory outcome — is real and informative, though the manuscript under-emphasizes it in favor of a mass-dominance framing that largely restates base rates.

major comments (3)
  1. [§4, Fig. 2, Table 3] §4 and Abstract, headline 67.84%: this number is nearly the product of the row and column marginals and therefore does not by itself evidence an interaction between divergence and outcome. From Table 3: P(low divergence) = 129,785.4/140,422.9 = 92.43% and P(wrong token mass) = 103,638.9/140,422.9 = 73.81%, so independence predicts 68.21% agreement-on-failure versus the observed 67.84% — slightly *below* the independence prediction. Cell-wise, correct tokens are 93.8% low-divergence and wrong tokens 91.9%. The teacher-4/4 row shows the same pattern (predicted ≈15.0%, observed 14.76%). The load-bearing content of the diagnostic is thus better stated as 'd_t is essentially uninformative about z(y)' than as 'agreement-on-failure dominates'. Please (i) report the deviation from the independence baseline explicitly (e.g., odds ratio or lift per cell) alongside the raw masses in Table 3 and Fig
  2. [§4, Table 3] Relatedly, the one interaction the paper does report — P(correct|low) = 26.60% vs P(correct|high) = 21.28% — has seed-bootstrap 95% intervals [23.23, 30.90] and [17.60, 25.51] that overlap substantially, so even the modest claimed dependence is not significant at seed level (n=8 jobs). The text says divergence is 'a weak classifier of trajectory success'; the intervals are consistent with no classifier skill at all. Either state this explicitly or add a direct paired test of the difference across seeds.
  3. [§6, Fig. 5] §6, training probes: the probes never test the decision-relevant premise that imitating agreement-on-failure tokens is harmful. All three interventions act on whole trajectories (retain / mask / contrast), so the observed persistence of AoF under the probes is consistent with the indistinguishability argument but also with AoF imitation being benign. In fact, correct-only OPD (which removes all AoF supervision) slightly *underperforms* pure OPD at the final checkpoint (75.20% vs 76.17%), which is weak evidence against the harm premise the diagnosis implicitly rests on. The paper's hedging ('descriptive, not normative'; 'diagnostic rather than a new training method') is appreciated, but §6 should state plainly that no token-targeted intervention was run and that the harm of AoF imitation is assumed, not measured. A minimal token-targeted probe (e.g., masking only low-divergence tokens on
minor comments (8)
  1. [A.3, Table 3] Table 3 reports fractional counts (e.g., 34,520.4 tokens, 46.4 sequences). Presumably these are per-seed averages or normalized quantities; please state the convention, since 'exact logged numerators and denominators' suggests integers.
  2. [§2 / A.1] The bootstrap resamples seed jobs, but it is not stated whether the 32 prompts per seed are drawn disjointly or overlap across seeds. If prompts overlap, prompt-level clustering is unaccounted for; please clarify the sampling design.
  3. [§3, Fig. 3] The default τ = 0.15 is never motivated beyond 'default'. The sweep (Fig. 3) shows the qualitative partition is stable, but a sentence on why 0.15 is the headline value (and that it is a free parameter of the diagnostic) would help. Note the partition is quite sensitive in absolute terms: AoF ranges 68.5–74.8% across τ.
  4. [§5, Fig. 3–4] Fig. 3 and Fig. 4A use three matched seeds while the primary estimate uses eight; the figure captions do note this, but §5's claim that the effect 'is visible within individual wrong trajectories' rests on the three-seed reruns only. Consider running the sequence-level audit on the full eight-seed batch.
  5. [A.5, Fig. 6] Fig. 6 shows a single illustrative trace; given the near-independence finding, one illustrative high-divergence-but-correct trace would make the four-regime picture symmetric.
  6. [Title / headings] Typo/formatting: the title in the PDF body runs words together ('OUTCOME-CONFOUNDEDLOCALSUPERVISION INON-POLICYDISTILLATION'); check section heading spacing throughout.
  7. [§3, Eq. for d_t] The k3 estimator d_t is an unbiased KL estimator only in expectation under student sampling; using its pointwise value for binning is reasonable as a heuristic but the variance of exp(-r_t) at negative r_t deserves one sentence, since heavy right tails in d_t could concentrate the high-divergence mass in a few tokens.
  8. [§8, Table 2] Table 2's self-assessment row ('Ours' with four checkmarks) would read better with the criteria defined in the caption rather than inline, and the 'partial' for KAT's outcome-resolution should be justified in one phrase.

Circularity Check

0 steps flagged

No significant circularity: regimes are defined from observables and then measured; the localization bound is an explicit information-set tautology, not a fitted or self-cited derivation.

full rationale

The paper’s load-bearing chain is measurement plus an information argument, not a first-principles derivation that re-imports its conclusion. The four regimes are defined by crossing the observed pointwise score dt with the trajectory label z(y); the reported masses (e.g., 67.84% agreement-on-failure) are pooled empirical counts over rollouts, not quantities forced by the definition of dt or τ. The training probes (imitate / mask / contrast) are separate interventions that test implications rather than baking the diagnostic into the loss. The claim that any tokenwise rule wt=f(dt,z(y)) cannot localize inside a failed trajectory is presented as an indistinguishability bound on that information set—true by the form of f, and stated as such—not as an empirical “prediction” obtained by fitting or by a self-citation uniqueness theorem. There is no load-bearing self-citation chain, no fitted parameter renamed as a prediction, and no ansatz smuggled in via overlapping-author prior work. Concerns that the flagship mass is close to the product of marginals, or that z(y) is a weak process axis, are about empirical interest and assumption strength, not circular construction. Honest non-finding: score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The load-bearing content is empirical and definitional rather than a long axiom chain. The claim rests on standard OPD reverse-KL-style local scores, a hand-chosen divergence threshold for binning, a rule-based final-answer outcome, and the modeling choice that token mass under student rollouts is the right primary unit. No new physical entities; the four regimes are partitions of observables. Free parameters are diagnostic/training knobs, not constants fitted to force the headline percentage.

free parameters (4)
  • divergence threshold τ = 0.15 (default); sweep {0.05,0.10,0.15,0.20,0.25}
    Default τ=0.15 defines low vs high divergence bins for the primary mass statistic; the partition percentages depend on this cut even though a sweep shows AoF stays large.
  • SPC coefficient β = 0.01
    Hand-set weight on the same-prompt contrastive regularizer in the third training probe; affects whether contrast appears helpful.
  • k3 pointwise divergence estimator d_t = k3 estimator on r_t = log π_θ(y_t|s_t)-log π_T(y_t|s_t)
    d_t=exp(-r_t)-1+r_t is chosen for non-negative binning of sampled-token log-ratios; alternative pointwise scores could shift cell boundaries.
  • probe training hyperparameters = 150 steps; lr=5e-7; β_SPC=0.01
    150 steps, lr 5e-7, 64 prompts×8 responses, temp 1.2, max 8192 tokens shape the negative training result; not exhaustively ablated.
axioms (5)
  • domain assumption Final-answer rule-based correctness z(y)∈{0,1} is a valid trajectory outcome for resolving local OPD signals in math reasoning.
    Invoked throughout §2–§3; every regime column depends on the verifier. Paper notes scorer limitation but treats z(y) as the outcome axis.
  • domain assumption Under student sampling, reverse-KL-style local scores on student prefixes are the relevant dense OPD supervision signal.
    Setting §2 defines r_t and OPD following Agarwal et al.; the diagnostic analyzes this signal class, not all possible teachers.
  • ad hoc to paper Response-token mass is the primary aggregation because the OPD objective applies dense updates over response tokens.
    Stated in §3 Aggregation; sequence-level audits are secondary. Length bias of wrong traces makes this choice consequential.
  • standard math A tokenwise intervention depending only on (d_t, z(y)) must treat equal pairs identically (indistinguishability / localization limitation).
    §6 functional dependence argument; standard equality-of-inputs implication, not a deep theorem, but load-bearing for the negative conclusion about simple fixes.
  • domain assumption Standalone teacher pass@k on a prompt does not imply corrective teacher likelihoods on arbitrary student prefixes.
    Motivates and is supported by the teacher-4/4 control in §5; separates prompt competence from prefix recoverability.
invented entities (2)
  • outcome-confounded local supervision independent evidence
    purpose: Name the mismatch between local teacher–student relations and trajectory-level success in OPD.
    Framing concept introduced in §1; operationalized by the four-regime diagnostic rather than postulated as a hidden mechanism.
  • four-regime outcome-resolved diagnostic (safe imitation / productive divergence / harmful divergence / agreement-on-failure) independent evidence
    purpose: Partition dense OPD token signals by local divergence and final outcome for measurement.
    Descriptive cells defined in §3 and Figure 1; empirically populated. Labels are non-causal by authors’ own caveat.

pith-pipeline@v1.2.0-grok45-kimik3 · 16660 in / 4091 out tokens · 80451 ms · 2026-07-30T14:34:58.199130+00:00 · methodology

0 comments
read the original abstract

On-policy distillation (OPD) trains a student on its own trajectories while a teacher supplies dense token-level likelihoods at student-visited prefixes. These likelihoods are often read locally: agreement appears safe to imitate, whereas disagreement appears to identify an error. We show that both readings are confounded by the outcome of the completed trajectory. We introduce an outcome-resolved diagnostic that crosses pointwise teacher-student divergence with final-answer correctness, separating safe imitation, productive divergence, harmful divergence, and agreement-on-failure. In an eight-seed mathematical-reasoning study with a Qwen3-8B student and Qwen3-32B teacher, agreement-on-failure constitutes 67.84% of pooled response-token mass; with a Qwen2.5-7B/32B pair it remains 67.68%. The result persists across threshold, sequence-level, format, and truncation audits. Even on prompts that the Qwen3 teacher solves in all four independent attempts, student accuracy rises to 86.91% but agreement-on-failure remains 14.76%. We then run three matched training probes that use the available signals to imitate, mask, or contrast whole trajectories; none consistently reduces agreement-on-failure. The result points to a localization limitation: local divergence paired with a trajectory-level outcome does not identify where a failed trajectory became unrecoverable. Addressing this limitation requires additional positional information, such as process labels, teacher continuations from student prefixes, or token-level alignment across rollouts. Our contribution is therefore diagnostic rather than a new training method.

Figures

Figures reproduced from arXiv: 2607.23731 by Guoqing Ma.

Figure 1
Figure 1. Figure 1: Outcome-resolved view of local OPD supervision. A: The teacher scores tokens at student-visited prefixes, while a final-answer verifier labels the same trajectory. B: Teacher–student divergence determines the rows and verifier outcome determines the columns, yielding four distinct regimes. Experimental setting. The primary study uses Qwen3-8B as the student and Qwen3-32B as the teacher on a 5,000-prompt ma… view at source ↗
Figure 2
Figure 2. Figure 2: Outcome-resolved response-token mass at τ = 0.15. Each horizontal bar partitions pooled token mass into the four regimes. Agreement-on-failure dominates the full eight-seed discovery distribution (top). The second row repeats the same diagnostic on a Qwen2.5-7B/32B pair. On the selected Qwen3 teacher-4/4 subset (bottom), student accuracy is much higher, yet all four regimes remain. Cross-row shifts are des… view at source ↗
Figure 3
Figure 3. Figure 3: Threshold robustness in matched three-seed reruns. Varying τ from 0.05 to 0.25 changes the absolute partition as expected, but agreement-on-failure remains large and both high-divergence regimes remain nonzero. Points pool exact token counts over three seeds at each threshold. 5 STRESS TESTS AND TEACHER-CAPABILITY CONTROL Threshold sensitivity [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sequence-level and token-level accounting. A: In the three-seed threshold reruns, most wrong trajectories are themselves dominated by low-divergence tokens. B: In the default eight-seed discovery run, longer wrong responses receive more weight under token-mass accounting than under sequence accounting. The two panels address complementary, not identical, audits. Does the teacher simply not know the answer?… view at source ↗
Figure 5
Figure 5. Figure 5: Two-seed controlled probes at step 150. Within each method, the two labeled bars are the matched seeds. A: SPC has a small final-validation advantage. B: That advantage does not coincide with lower agreement-on-failure. With only two seeds, the figure is diagnostic rather than a method-ranking result. Small performance differences, persistent confounding. Across two seeds, final validation accuracy average… view at source ↗
Figure 6
Figure 6. Figure 6: shows one traced wrong response from the primary Qwen3 discovery run. The student ultimately answers 16 while the gold answer is 14. We re-score the sampled response tokens with the Qwen3-8B student and Qwen3-32B teacher and compute the same pointwise dt diagnostic used in the aggregate analysis. The example is illustrative rather than a causal token annotation: it shows that a failed trajectory can be loc… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 5 linked inside Pith

  1. [1]

    NeurIPS Deep Learning and Representation Learning Workshop , year=

    Distilling the Knowledge in a Neural Network , author=. NeurIPS Deep Learning and Representation Learning Workshop , year=

  2. [2]

    Proceedings of EMNLP , year=

    Sequence-Level Knowledge Distillation , author=. Proceedings of EMNLP , year=

  3. [3]

    Gu, Yuxian and Dong, Li and Wei, Furu and Huang, Minlie , journal=

  4. [4]

    International Conference on Learning Representations , year=

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes , author=. International Conference on Learning Representations , year=

  5. [5]

    arXiv preprint arXiv:2604.00626 , year=

    A Survey of On-Policy Distillation for Large Language Models , author=. arXiv preprint arXiv:2604.00626 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2604.13016 , year=

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe , author=. arXiv preprint arXiv:2604.13016 , year=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:2607.13399 , year=

    Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations , author=. arXiv preprint arXiv:2607.13399 , year=

    Pith/arXiv arXiv

  8. [8]

    Ma, Wenhan and Wei, Jianyu and Zhao, Liang and Zhang, Hailin and Xiao, Bangjun and Li, Lei and Yang, Qibin and Gao, Bofei and Wang, Yudong and Li, Rang and Dong, Jinhao and Sui, Zhifang and Luo, Fuli , journal=

  9. [9]

    arXiv preprint arXiv:2605.12652 , year=

    Multi-Rollout On-Policy Distillation via Peer Successes and Failures , author=. arXiv preprint arXiv:2605.12652 , year=

    Pith/arXiv arXiv

  10. [10]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , journal=

  11. [11]

    arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  12. [12]

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Fan, Tiantian and Liu, Xin and Lin, Haibin and Sheng, Guangming and others , journal=

  13. [13]

    Cui, Ganqu and Yuan, Lifan and Wang, Zefan and Zhang, Yuchen and Chen, Jiacheng and Li, Wendi and He, Bingxiang and Fan, Yuchen and Yu, Tianyu and Xu, Qixin and others , journal=

  14. [14]

    Escaping the

    Xin, Haoran and Zhao, Anhao and Sun, Ying and Li, Jin and Shen, Xiaoyu and Xiong, Hui , journal=. Escaping the