Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Filtering token-level teacher signals that barely depend on the task input improves on-policy distillation across math and vision benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

SA-OPD filters on-policy distillation tokens that are both weakly input-grounded and highly divergent, improving math reasoning and visual understanding benchmarks over vanilla OPD and selective baselines.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection SA-OPD adds a genuinely new selection axis for OPD, and the empirical case mostly supports it, but the core proxy is unvalidated and the headline gains lack error bars. the 4 major comments →

arxiv 2608.03632 v1 pith:DIZJS7MC submitted 2026-08-04 cs.AI

When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation

Yinuo Jiang , Yongjie Ye , Zhou Tao , Xiang Zhuang , Qiang Zhang , Huajun Chen , Tiankai Li This is my paper
classification cs.AI
keywords on-policy distillationtoken-level supervision selectioninput-groundednessspurious signalsknowledge distillationlarge language modelsvision-language modelsreverse-KL objective
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On-policy distillation (OPD) copies a teacher's token-level judgments onto student-generated trajectories, but the paper argues that many of those judgments are not really about the task input: they reflect input-agnostic language priors, formatting conventions, and stereotyped reasoning templates learned during pretraining. It calls these 'spurious signals' and claims they can dominate the gradient while contributing little task-improving direction. The paper proposes SA-OPD, which estimates each token's input-groundedness by comparing the teacher-student divergence under the real prompt with the divergence under a residual no-prompt context, then filters tokens that are both weakly input-grounded and high-impact. Across math reasoning, visual understanding, and visual reasoning benchmarks, SA-OPD beats vanilla OPD and entropy-, divergence-, and learnability-based selective baselines at matched compute. If correct, the paper identifies input-groundedness as a criterion that existing selection methods miss.

Core claim

The paper's central claim is that a teacher's token-level distillation signal decomposes into an input-grounded component and a prior-driven component, and that the prior-driven component can carry large gradient energy while aligning near zero with the task-improving direction. It formalizes harmful tokens as the intersection of low input-groundedness and high optimization impact, and makes this operational with a lightweight proxy: for each token, compute the reverse-KL teacher-student divergence with the original prompt and with the prompt removed, and take the absolute difference as the input-grounding gap. Tokens with a small gap and a large absolute divergence are filtered from the OPD

What carries the argument

The load-bearing object is the Input-Grounding Gap, Delta_IG_t = |A_full_t - A_res_t| (Eq. 14), where A_full_t is the teacher-student divergence at token t under the original prompt and A_res_t is the same divergence with the prompt removed but the same student-generated prefix. The paper uses this gap as an empirical proxy for the conditional mutual information I(X; A_t | Y_<t), i.e. how much the distillation signal depends on the task input. Filtering combines it with the absolute divergence via F(p1,p2)=Bottom_{p1}(Delta_IG) ∩ Top_{p2}(|A_full|), removing only tokens that are both input-blind and high-impact; a dynamic filtered-loss-mass-ratio (FLMR) bound prevents over-aggressive removal

Load-bearing premise

The whole filter rests on treating 'prompt removed' as equivalent to 'teacher's input-agnostic prior': if the gap in divergence between full and empty prompts does not actually isolate task-specific dependence, the method filters by prefix predictability rather than by spuriousness.

What would settle it

Take a fixed batch of student rollouts, estimate the mutual information I(X; A_t | Y_<t) by marginalizing over a varied set of prompts of the same task (or by ablating the input with a control prompt), and compare which tokens the empty-prompt proxy would filter against the tokens with genuinely low input dependence: if the two sets disagree substantially, the proxy is measuring something else.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Input-groundedness becomes a selection axis that complements entropy, teacher confidence, divergence, and learnability in OPD; the paper's ablations show both axes of its filter are needed.
  • In LLM math reasoning, high-impact spurious signals concentrate at the start of training, so early filtering matters most; in VLM tasks they persist throughout training, so filtering helps across the whole run.
  • Entropy-based criteria miss a substantial share of spurious tokens: many filtered tokens are low-entropy yet high-impact and input-blind.
  • The filtered-loss-mass constraint lets practitioners trade spurious-signal suppression against preservation of useful supervision, with a moderate bound (beta = 1.8) giving the best visual-understanding results.
  • The gains transfer across different teacher-student scale pairs and add only 2.6-7.5% training overhead, making the filter a drop-in modification of vanilla OPD.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The empty-prompt residual pass could be reused as a cheap, general diagnostic for spurious token-level supervision in other dense-supervision settings, such as RLVR or preference optimization, where template-driven rewards may contaminate updates.
  • The proxy treats 'empty prompt' as a stand-in for the teacher's input-agnostic prior; a stronger test would replace the empty prompt with a distribution of unrelated prompts and check whether the filtered set tracks the true I(X; A_t | Y_<t), which the paper does not do.
  • Because generic reasoning phrases can be useful scaffolds even when input-blind, filtering them may trade short-term imitation fidelity for better task alignment on longer-horizon or open-ended reasoning; that trade-off is an open extension.
  • The method's benefit appears largest when spurious signals are persistent (VLM tasks), so one testable prediction is that input-groundedness filtering matters more on benchmarks with strong perceptual ambiguity or template-heavy answers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SA-OPD, a token-level filtering method for on-policy distillation. The core idea is that some teacher supervision signals are 'spurious': they have large optimization impact but little dependence on the task input, being driven instead by input-agnostic language priors, formatting conventions, or reasoning templates. To measure input-groundedness, the paper defines an Input-Grounding Gap ΔIG_t = |A_full_t − A_res_t|, where A_full_t is the teacher–student log-ratio under the original prompt and A_res_t is the same log-ratio under an empty (no-prompt) context. Tokens are filtered if they are simultaneously in the bottom p1 fraction of ΔIG_t and the top p2 fraction of |A_full_t| (Eq. 16). The paper reports consistent improvements over Vanilla OPD and selective baselines on math reasoning (LLM) and visual understanding/reasoning (VLM) benchmarks, with ablations supporting the two-axis criterion and a modest computational overhead.

Significance. If the central interpretation is correct, the paper identifies a genuinely underexplored failure mode in OPD—weakly input-grounded high-impact supervision—and proposes a simple, parameter-light filter that improves distillation across LLM and VLM settings. The empirical gains are systematic (+3.5 and +3.1 average points over Vanilla OPD on VLM sets; +1.9 on math reasoning), and the ablation study in Table 4 supports the claim that both axes are necessary. The paper also provides a formal SNR-style framework (Appendix A) and an efficiency analysis showing only 2.6–7.5% training overhead. These are strengths that make the method worth considering. However, the empirical results are all single-seed, and the core input-groundedness proxy is not validated against the mutual-information quantity it purports to approximate; the main conceptual contribution therefore rests on an assumption that is currently unsupported.

major comments (4)
  1. [Section 3.2, Eqs. (11)–(14); Appendix A.1, Eq. (23)] The paper defines input-groundedness as IG_t = I(X; A_t | Y_<t) and then approximates it by ΔIG_t = |A_full_t − A_res_t|, where A_res_t is obtained by conditioning on an empty input. This is not a valid surrogate for the theoretical quantity. The prior component in Appendix A.1, Eq. (23), is A_prior_t = E[A_t | C_t], a conditional expectation that marginalizes over inputs given the response context. Conditioning on X = ∅ is a different operation: for autoregressive instruction-tuned models, an empty prompt is an out-of-distribution context that can produce degenerate, overconfident, or otherwise unrepresentative next-token distributions. A small ΔIG_t may reflect empty-prompt artifacts rather than input-agnostic prior structure, and a large ΔIG_t may reflect prompt-removal distribution shift rather than genuine input grounding. The paper provides no validation that ΔIG_t tracks IG_t or E
  2. [Tables 1–4 and Section 4.3] All reported results are single-seed point estimates with no error bars, confidence intervals, or significance tests. The main claim is that SA-OPD 'consistently outperforms' Vanilla OPD and competitive baselines, but the observed gains are often small (e.g., +0.9, +1.1, +1.2 on Math500/AMC23/MathVision in Tables 1–2) and could be within run-to-run variance. This is especially important because the method has a stochastic filtering component and the training dynamics plot (Figure 3) shows non-stationary losses. Please report the mean and standard deviation over at least 3 seeds for the main tables, and ideally a paired significance test against the strongest baseline. Without this, the central empirical claim is not fully supported.
  3. [Section 4.4, Figure 4(a); Table 6] The dynamic threshold hyperparameter β is selected on CountQA (the paper reports the best β = 1.8 by maximizing the counting score), and CountQA is also one of the headline benchmarks in Table 1. This is test-set tuning: the +7.2-point gain over Vanilla OPD on CountQA is obtained with β chosen on that same benchmark. The same issue may affect the visual-understanding average, since β is only used for visual understanding. Please either fix β a priori, select it on a held-out validation split, or show that the rankings are stable across a range of β values on all benchmarks. As it stands, the CountQA improvement is inflated by fitting the hyperparameter to the test set.
  4. [Eq. (16) vs. Algorithm 2 (Appendix C)] There is an inconsistency in the definition of the high-impact token set. Eq. (16) defines Top_{p2}(|A_full_t|) as the top p2 fraction by absolute value. Algorithm 2, line 5, defines E ← Top_{p2}(A_full_t) ∪ Bottom_{p2}(A_full_t), i.e., the union of the top and bottom p2 fractions of the signed values. These are not equivalent: with p2 = 0.3, Algorithm 2 can select up to 60% of tokens, whereas Eq. (16) selects 30%. This changes the actual filtering behavior and the resulting FLMR. Please align the formal definition and the implementation, and state which one was used in the experiments.
minor comments (5)
  1. [Section 3.1, Eq. (4) and Appendix A.1, Eq. (23)] The main text says A_prior_t is 'predictable from the response prefix alone', but Eq. (23) defines C_t = (Y_<t, Y_t), i.e., it includes the current token Y_t. Please resolve this notational inconsistency; if C_t is meant to include Y_t, the phrase 'response prefix' should be adjusted, and the definition of 'prefix' should be stated explicitly.
  2. [Section 4.3, 'Proxy Replacement'] The ablation 'Proxy Replacement' is described only in one sentence: 'replaces the input-dependent component in the input-groundedness proxy with the corresponding teacher log-probability.' The exact replacement is not specified. Please define what 'input-dependent component' means in the implementation and how teacher log-probability is used, so that the ablation is reproducible.
  3. [Appendix B.2] The paper refers to MathVista as 'MathVistamini' in Appendix B.2 but as 'MathVista' everywhere else. Please clarify whether the main results use the full MathVista or the mini split, and adjust the notation consistently.
  4. [Eq. (18) and Algorithm 2] Eq. (18) states FLMR(F(p1,p2)) ≤ β, while Algorithm 2 uses a target interval [β_min, β_max]. The relationship between β and the interval is not defined in the main text. Please specify how β_min and β_max are set and how the final β value (e.g., 1.8) is used.
  5. [Figure 4(a)] The figure caption and the text are somewhat confusing: the text says the best β is 1.8 with a score increase from 30.8 to 33.6, but the figure appears to show both 'w/o dynamic' (31.8) and several β-dependent values. Please make the reported numbers in the caption and text consistent and clearly label which curve corresponds to which setting.

Circularity Check

2 steps flagged

Partial circularity: the CountQA headline gain is a β-tuned selection result, and the 'spurious signal' label is assigned by the filter's own proxy; the core filter-vs-ablation comparisons remain independently informative.

specific steps
  1. fitted input called prediction [Section 4.4 'Hyper-parameter Analysis' (Figure 4a); Table 1; Table 6]
    "The best result is achieved at β=1.8, where the counting score increases from 30.8 to 33.6."

    The dynamic FLMR bound β is selected by maximizing the CountQA score, and the same CountQA score (33.6) is then reported in Table 1 as SA-OPD's headline result (+7.2 over Vanilla OPD). The CountQA improvement is therefore not an independent prediction of the method; it is the criterion used to pick β. Because CountQA is one of the benchmarks supporting the abstract's claim that SA-OPD 'consistently outperforms' selective methods, that portion of the claimed advantage reduces to test-set fitting rather than to a validated prediction.

  2. self definitional [Section 3.1 'Spurious Signals'; Section 3.2 Eq. (15)-(16); Section 5]
    "We therefore define spurious OPD signals as the intersection of two conditions: low input-groundedness and high update impact. ... Specifically, we define a token-level filtering indicator as TokenFiltered_t = 1[ΔIG_t < τIG] · 1[|A_full_t| > τA]. ... SA-OPD filters only tokens that are both weakly input-grounded and high-impact, thereby suppressing misleading dense supervision."

    The term 'spurious' is operationalized by the same filter that is said to remove it: 'low input-groundedness' in the definition is implemented as low ΔIG and 'high update impact' as high |A_full|, so the statement that SA-OPD removes spurious signals is a restatement of the selection rule rather than a derived property. The formal definition via I(X; A_t | Y_<t) (Eq. 11) and the prior component E[A_t | C_t] (Eq. 23) is never shown to be measured by ΔIG; conditioning on an empty prompt is not the same as marginalizing over inputs. The label 'spurious' is thus assigned by construction. The empirical benchmark gains remain independent evidence, so this is partial, not total, circularity.

full rationale

The paper's derivation chain has two bounded circularities. First, the CountQA number in Table 1 is not an independent prediction: β=1.8 was chosen by maximizing the CountQA score in Section 4.4, and the same 33.6 value is then reported as a headline gain. Second, the causal framing 'SA-OPD suppresses spurious signals' is tautological because 'spurious' is defined as low input-groundedness + high impact, and the filter implements exactly that criterion with ΔIG and |A_full|. The proxy ΔIG is never validated against the formal mutual-information definition or the conditional-expectation prior; this is a correctness/validity gap as well as a definitional labeling issue. However, the central empirical claim has independent content: the ablations (Random Filter, Divergence-only, Input-Groundedness-only, Proxy Replacement) show that the specific two-axis intersection helps on Geo3K and MathVista, and most benchmarks were not used to select the tuned hyperparameter. There is no load-bearing self-citation and no imported uniqueness theorem. The score of 4 reflects partial circularity in one headline benchmark and in the definitional framing, while the core filter-improvement result is not forced by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 1 invented entities

The method rests on a small set of hand-chosen selection ratios (p1, p2, beta) and on the proxy assumption that the no-prompt divergence isolates input-agnostic priors. The paper is honest that the mutual information has no closed form and that the proxy is lightweight, but it never validates the proxy against the quantity it claims to approximate, and beta is tuned on the benchmark with the largest reported gain. No new physical entities are postulated; the only invented entity is the conceptual spurious-signal category, which has no independent evidence outside the paper's own experiments.

free parameters (4)
  • p1 (low input-groundedness ratio) = 0.2 initial, adapted per batch
    Bottom-p1 fraction of Delta_IG tokens selected for filtering (Eq. 16); initial value hand-chosen, adapted via FLMR constraint (Algorithm 2); determines which tokens are removed.
  • p2 (high divergence ratio) = 0.3 initial, adapted per batch
    Top-p2 fraction of |A_full| tokens selected; hand-chosen and adapted; ablations show both p1 and p2 axes matter.
  • beta (FLMR bound) = 1.8 for visual understanding
    Tuned via sensitivity sweep on CountQA (Figure 4a); the headline CountQA result (33.6) is the best point of this sweep, while adjacent values give 32.6 to 33.2. Not used for visual reasoning or math reasoning.
  • beta_min/beta_max target interval = not specified
    Algorithm 2 uses a target FLMR interval [beta_min, beta_max] but the values are never given in the paper, leaving exact reproduction of the adaptive procedure incomplete.
axioms (6)
  • domain assumption At = A_grd_t + A_prior_t with E[A_grd_t | C_t] = 0, where C_t = (Y_<t, Y_t) and A_prior_t = E[A_t | C_t]
    Conceptual decomposition of token divergence into input-grounded and prior components (Eq. 4, Appendix A.1 Eq. 23); used to derive the SNR view and the filtering motivation; not directly testable from data.
  • domain assumption Grounded updates are positively aligned with the task objective: E[<g_grd_t, g*_t>] > 0 (Eq. 8)
    The entire signal-versus-noise argument needs this positivity assumption; stated as an expectation, never justified.
  • domain assumption kappa_t << 1 for template-like tokens (Appendix A.1, after Eq. 37)
    Theorem 1 only bounds alignment by kappa_t; the conclusion that prior-induced gradients have weak alignment requires assuming the score-function residual delta_s_t is small for such tokens. The paper asserts this rather than deriving it.
  • domain assumption Nuisance updates form a martingale-difference sequence with finite second moment (Eq. 49)
    Needed for Theorem 2 (parameter drift grows linearly in steps); the paper explicitly states this as an assumption and notes it only requires no persistent directional signal.
  • ad hoc to paper The no-prompt gap Delta_IG = |A_full - A_res| approximates I(X; A_t | Y_<t) and isolates the prior component A_prior_t
    The proxy (Eqs. 12-14) replaces a conditional expectation over X with conditioning on X = empty; the paper provides no measurement showing the proxy tracks the MI or equals E[A_t | C_t]. This is the load-bearing methodological step.
  • domain assumption Teacher and student next-token distributions conditioned on a bare prefix (no prompt) are well-defined and non-degenerate for instruction-tuned models
    The residual computation evaluates pi(· | empty, y_<t); for instruction-tuned models such conditioning can be out-of-distribution, and the paper does not discuss refusals or degenerate modes.
invented entities (1)
  • Spurious signal category in OPD no independent evidence
    purpose: Names and targets the failure mode SA-OPD filters: tokens with low input-groundedness and high teacher-student divergence
    The category is defined entirely by the paper's own proxy (low Delta_IG plus high |A_full|); no external validation (human annotation, perturbation studies) shows these tokens correspond to input-agnostic language priors rather than merely prefix-predictable content. The token statistics in Table 5 and the visualizations in Appendix E are internal evidence only.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation." pith.science (2026). https://pith.science/paper/DIZJS7MC

@misc{pith2026260803632,
  author       = {Pith},
  title        = {Pith review of: When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DIZJS7MC}},
  note         = {Machine review of arXiv:2608.03632}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable. However, the assumptions overlook a fundamental failure mode of language models: their token-level judgments can be driven by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates rather than task-specific evidence. We refer to such optimization-relevant but weakly input-grounded supervision as spurious signals in OPD, which may produce large gradients while contributing little task-improving direction. To mitigate this issue, we propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision based on input-groundedness and optimization impact. SA-OPD introduces a lightweight input-groundedness proxy estimating whether a token-level distillation signal truly depends on the input. It then filters only tokens that simultaneously exhibit low input-groundedness and extreme distillation divergence, thereby removing high-impact spurious updates and achieving fine-grained OPD optimization. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods. These results establish input-groundedness as a key dimension for OPD supervision selection and offer a simple, effective strategy for mitigating spurious updates.

Figures

Figures reproduced from arXiv: 2608.03632 by Huajun Chen, Qiang Zhang, Tiankai Li, Xiang Zhuang, Yinuo Jiang, YongJie Ye, Zhou Tao.

Figure 1
Figure 1. Figure 1: Illustration of spurious signals in OPD. Panels (a) and (b) present a concrete example: (a) shows the input image [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed SA-OPD framework. Given an on-policy rollout, SA-OPD computes teacher–student [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of SA-OPD’s dynamic threshold and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics across tasks. Hyper-parameter Analysis. For visual reasoning and mathematical reasoning tasks, the FLMR under fixed selec￾tion ratios p1 and p2 is relatively small, around 1pp–2pp at the beginning of training. In contrast, the FLMR in visual un￾derstanding tasks is much higher, indicating that the filtered tokens account for a larger portion of the loss mass. If the filtering is too aggre… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 1 canonical work pages

  1. [3]

    arXiv:2501.12948

    DeepSeek-R1: Incentivizing Rea- soning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. Ding, S.; Wu, S.; Zhao, X.; Zang, Y.; Duan, H.; Dong, X.; Zhang, P.; Cao, Y.; Lin, D.; and Wang, J

    Pith/arXiv arXiv

  2. [4]

    arXiv:2504.07957

    MM-IFEngine: Towards Multimodal Instruction Following. arXiv:2504.07957. Gu, Y.; Dong, L.; Wei, F.; and Huang, M

    Pith/arXiv arXiv

  3. [6]

    arXiv:2604.12002

    Self- Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision. arXiv:2604.12002. He, Z.; Liang, T.; Xu, J.; Liu, Q.; Chen, X.; Wang, Y.; Song, L.;Yu,D.;Liang,Z.;Wang,W.;Zhang,Z.;Wang,R.;Tu,Z.; Mi, H.; and Yu, D

    Pith/arXiv arXiv

  4. [7]

    arXiv:2504.11456

    DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning. arXiv:2504.11456. Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J

    Pith/arXiv arXiv

  5. [9]

    arXiv:2605.03677

    Uni-OPD: Uni- fyingOn-PolicyDistillationwithaDual-PerspectiveRecipe. arXiv:2605.03677. Huang, M.; Lai, H.; Zhang, X.; Wu, W.; Ma, J.; Zhang, L.; and Liu, J

    Pith/arXiv arXiv

  6. [10]

    arXiv:2409.01577

    EvoChart: A Benchmark and a Self- Training Approach Towards Real-World Chart Understand- ing. arXiv:2409.01577. Jin, W.; Min, T.; Yang, Y.; Wei, D.; Zhou, Y.; Kadhe, S. R.; Baracaldo, N.; and Lee, K

    Pith/arXiv arXiv

  7. [11]

    arXiv:2603.07079

    Entropy-Aware On-Policy Distillation of Language Models. arXiv:2603.07079. Lewkowycz, A.; Andreassen, A.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V.; Slone, A.; Anil, C.; Schlag, I.;Gutman-Solo,T.;Wu,Y.;Neyshabur,B.;Gur-Ari,G.;and Misra, V

    Pith/arXiv arXiv

  8. [13]

    arXiv:2310.02255

    MathVista: Evaluating Mathematical Reasoning of Founda- tion Models in Visual Contexts. arXiv:2310.02255. Lu,P.;Gong,R.;Jiang,S.;Qiu,L.;Huang,S.;Liang,X.;and Zhu,S.-C.2021.Inter-GPS:InterpretableGeometryProblem Solving with Formal Language and Symbolic Reasoning. arXiv:2105.04165. Luo,F.;Chuang,Y.-N.;Wang,G.;Xu,Z.;Han,X.;Zhang,T.; and Braverman, V

    Pith/arXiv arXiv 2021

  9. [14]

    arXiv:2604.08527

    Demystifying OPD: Length Infla- tionandStabilizationStrategiesforLargeLanguageModels. arXiv:2604.08527. Qwen Team

    Pith/arXiv arXiv

  10. [15]

    arXiv:2604.04917

    Vero: An Open RL Recipe for General Visual Reasoning. arXiv:2604.04917. Tamarapalli, J. S.; Grover, R.; Pande, N.; and Yerramilli, S

    Pith/arXiv arXiv

  11. [16]

    Wang, H.; Wang, G.; Xiao, H.; Zhou, Y.; Pan, Y.; Wang, J.; Xu, K.; Wen, Y.; Ruan, X.; Chen, X.; and Qi, H

    CountQA: How Well Do MLLMs Count in the Wild? arXiv:2508.06585. Wang, H.; Wang, G.; Xiao, H.; Zhou, Y.; Pan, Y.; Wang, J.; Xu, K.; Wen, Y.; Ruan, X.; Chen, X.; and Qi, H. 2026a. Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents. arXiv:2604.10674. Wang, K.; Pan, J.; Shi, W.; Lu, Z.; Zhan, M.; and Li, H. 2024a. Measuring Multimodal Mat...

    Pith/arXiv arXiv

  12. [17]

    arXiv:2506.01939

    Beyond the 80/20 Rule: High-Entropy Minor- ityTokensDriveEffectiveReinforcementLearningforLLM Reasoning. arXiv:2506.01939. Wang,X.;Pan,J.;Ding,L.;andBiemann,C.2024b. Mitigat- ing Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding. arXiv:2403.18715. Wang, Y.; Lu, S.; Gu, Y.; Wang, P.; Yang, Y.; Yan, Z.; Xie, C.; Wu, J.; an...

    Pith/arXiv arXiv

  13. [18]

    arXiv:2604.14084

    TIP: Token Importance in On-Policy Distillation. arXiv:2604.14084. Yan,J.;Li,Y.;Hu,Z.;Wang,Z.;Cui,G.;Qu,X.;Cheng,Y.; and Zhang, Y

    Pith/arXiv arXiv

  14. [19]

    arXiv:2504.14945

    Learning to Reason under Off-Policy Guidance. arXiv:2504.14945. Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu,B.;Gao,C.;Huang,C.;Lv,C.;Zheng,C.;Liu,D.;Zhou, F.;Huang,F.;Hu,F.;Ge,H.;Wei,H.;Lin,H.;Tang,J.;Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Zhou, J.; Lin, J.; Dang, K.; Bao, K.; Yang, K.; Yu, L.; Deng, L.; Li, M.; Xue, ...

    Pith/arXiv arXiv

  15. [20]

    arXiv:2505.09388

    Qwen3 Technical Report. arXiv:2505.09388. Yang, C.; Qin, C.; Si, Q.; Chen, M.; Gu, N.; Yao, D.; Lin, Z.; Wang, W.; Wang, J.; and Duan, N. 2026a. Self-Distilled RLVR. arXiv:2604.03128. Yang, W.; Liu, W.; Xie, R.; Yang, K.; Yang, S.; and Lin, Y.2026b. LearningbeyondTeacher:GeneralizedOn-Policy Distillation with Reward Extrapolation. arXiv:2602.12125. Yuan, ...

    Pith/arXiv arXiv

  16. [21]

    arXiv:2605.18740

    Vision-OPD: Learning to See Fine De- tails for Multimodal LLMs via On-Policy Self-Distillation. arXiv:2605.18740. Zhang,D.;Yang,Z.;Janghorbani,S.;Han,J.;II,A.R.;Qian, Q.; Lyng, G. D.; Batra, S. S.; and Tillman, R. E. 2026a. Fast and Effective On-policy Distillation from Reasoning Prefixes. arXiv:2602.15260. Zhang, Z.; Jiang, S.; Shen, Y.; Zhang, Y.; Ram, ...

    Pith/arXiv arXiv

  17. [22]

    arXiv:2601.18734

    Self-Distilled Reasoner: On-Policy Self- Distillation for Large Language Models. arXiv:2601.18734. Zheng,B.;Ma,X.;Liang,Y.;Ruan,J.;Fu,X.;Lin,K.;Zhu, B.; Zeng, K.; and Cai, X

    Pith/arXiv arXiv

  18. [23]

    arXiv:2604.10688

    SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adap- tive Weighting. arXiv:2604.10688. Zhu, L.; Guan, Y.; Liang, D.; Ju, J.; Luo, Z.; Qin, B.; Luan, J.; Liu, Y.; and Bai, X

    Pith/arXiv arXiv

  19. [24]

    arXiv:2508.05612

    Shuffle-R1: Efficient RL frameworkforMultimodalLargeLanguageModelsviaData- centric Dynamic Shuffle. arXiv:2508.05612. A Supplementary Theory A.1 Prior-Induced Gradients Have Weak Input-Specific Alignment Recall the sampled teacher–student divergence and its score-function update: At = logπ θ(Yt |X, Y<t)−logπ T(Yt |X, Y<t),(20) gOPD t =−A tst,(21) where st...

    arXiv 2048

  20. [25]

    Visual reasoning includes MathVision, Geo3K, and MathVistamini

    B.2 Evaluation Details ForVLMevaluation,weusesixbenchmarkscoveringvisualunderstandingandvisualreasoning.Visualunderstandingincludes EvoChart, MMIFEval, and CountQA. Visual reasoning includes MathVision, Geo3K, and MathVistamini. For simplicity, we refer to MathVistamini as MathVista throughout the paper. For LLM math reasoning, we evaluate on Math500, AMC...

    work page 2023

  21. [2021]

    arXiv:2103.03874

    Measuring Mathematical Problem Solving With the MATH Dataset. arXiv:2103.03874. Hou,W.;Peng,S.;Wang,W.;Ruan,Z.;Zhang,Y.;Zhou,Z.; Gao,M.;Chen,Y.;Wang,K.;Yang,H.;Zhang,C.;Tian,Z.; Hu, H.; Yang, Y.; Wu, F.; and Fan, H

    Pith/arXiv arXiv

  22. [2022]

    arXiv:2206.14858

    Solving Quantitative Reasoning Problems with Language Models. arXiv:2206.14858. Li, J.; Yin, H.; Xu, H.; Xu, B.; Tan, W.; He, Z.; Ju, J.; Luo, Z.; and Luan, J. 2026a. Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation. arXiv:2602.02994. Li, Y.; Zheng, L.; Yu, Y.; Zhou, W.; Zhong, ...

    Pith/arXiv arXiv

  23. [2024]

    arXiv:2306.13649

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. arXiv:2306.13649. Cheng, D.; Huang, S.; Zhu, X.; Dai, B.; Zhao, X.; Zhang, Z.; and Wei, F

    Pith/arXiv arXiv

  24. [2025]

    arXiv:2505.22617

    The EntropyMechanismofReinforcementLearningforReason- ing Language Models. arXiv:2505.22617. DeepSeek-AI

    Pith/arXiv arXiv

  25. [2026]

    arXiv:2306.08543

    MiniLLM: On-Policy Distillation of Large Language Mod- els. arXiv:2306.08543. Guan,T.;Liu,F.;Wu,X.;Xian,R.;Li,Z.;Liu,X.;Wang,X.; Chen,L.;Huang,F.;Yacoob,Y.;etal.2023.Hallusionbench: an advanced diagnostic suite for entangled language hallu- cinationandvisualillusioninlargevision-languagemodels. arXiv preprint arXiv:2310.14566. He, Y.; Kaur, S.; Bhaskar, A...

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.