Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

Changqing Zhang; Feng Xiong; Leyan Xue; Mingjun Ma

REVIEW 3 major objections 6 minor 17 references

Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing the full teacher distillation target with a Fisher projection of the teacher–student gap onto the student's visual tangent space improves all seven vision-language benchmarks in 8B-to-2B transfer, by 2.77 points on average over…

desk verdict A clean, novel distillation method with a correct algebraic core, but the empirical claim is undercut by a missing scalar-scaled baseline and weak probe ablations; deserves serious review after revision. read the letter →

arxiv 2608.01263 v2 pith:AL274IHD submitted 2026-08-02 cs.LG

Leyan Xue , Feng Xiong , Mingjun Ma , Changqing Zhang This is my paper · ORCID

classification cs.LG

keywords on-policydistillationFisherprojectionvision-languagemodelscapacity-awaretargetreverse-KLvisualtangentspacemultimodalreasoningknowledge

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

On-policy distillation supervises a compact student at prefixes sampled from its own generation, but it still asks the student to match the complete next-token distribution of the teacher. In vision-language reasoning, part of the teacher's correction encodes visual distinctions the student's representation cannot express, so the full target contains unreachable supervision. This paper's target-scaling study supports that diagnosis: moving the target all the way to the teacher makes the prescribed shift harder to fit and degrades downstream reasoning. The paper's proposal, Fisher-Projected On-Policy Distillation (FP-OPD), estimates the student's local visual tangent space with perturbed visual features and projects the teacher-student gap onto that space under the Fisher metric, producing a capacity-aware target. In 8B-to-2B distillation FP-OPD improves all seven benchmarks and raises the average by 2.77 points over the base student and 1.60 points over standard OPD, which the paper takes as evidence that locally realizable teacher corrections are the right target for compact vision-language models.

What carries the argument

The load-bearing object is the Fisher projection of the centered teacher–student log-probability gap onto a finite-difference approximation of the student's local visual tangent space. The student's visual embeddings are perturbed along four fixed binary spatial bands (top, bottom, left, right, each spanning 25% of the grid axis) to give response directions $d_k$, and their span approximates the first-order output changes the student can express through its current visual state. Projection uses the Fisher inner product $\langle a, b\rangle_p = \sum_{v\in V} p(v) a_v b_v$, which is the geometry induced by the OPD loss itself because the local expansion of reverse KL is $\tfrac{1}{2}\langle \delta,\delta\rangle_p + o(\|\delta\|^2)$. A ridge-stabilized solve produces $\hat{g}_R$, and the reachable target $q_R = \mathrm{softmax}(\mathrm{stopgrad}(\ell_\theta) + \hat{g}_R)$ anchors at the detached student distribution; the loss keeps the standard OPD form $\mathrm{KL}(p_\theta \,\|\, q_R)$ with full-vocabulary reverse KL on student-generated trajectories.

What would settle it

Take a small benchmark where the correct answer depends on a fine visual mark at the image center, far from all four boundary bands, and first verify that the small student can learn that correction under standard OPD. If FP-OPD then fails to transfer the same correction — because the center-only change is orthogonal to all four probes and is discarded — while standard OPD succeeds, the four-band tangent space does not contain the student-supported corrections and the central claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that the appropriate distillation target for a compact vision-language student is not the complete teacher distribution but the component of the teacher correction contained in the student's locally reachable output space. Writing the teacher target as $p_T = \mathrm{softmax}(\ell_\theta + g)$ with $g$ the centered teacher–student log-probability gap, FP-OPD replaces $g$ by its Fisher projection $\hat{g}_R$ onto a tangent space estimated from finite-difference responses to four spatial perturbations of the student's visual embeddings, and optimizes full-vocabulary reverse KL against $q_R = \mathrm{softmax}(\mathrm{stopgrad}(\ell_\theta) + \hat{g}_R)$. If the gap lies entirely in the estimated tangent space the target coincides with the teacher, and if it is orthogonal the target produces no update at all. The authors report that this capacity-aware target improves all seven benchmarks in 8B-to-2B distillation and five of seven benchmarks over standard OPD in 32B-to-8B distillation, with gains that persist under greedy decoding, and conclude that locally realizable teacher corrections are more effective supervision for compact vision-language models.

Load-bearing premise

The load-bearing premise is that four fixed spatial bands of visual tokens (top, bottom, left, right, each a quarter of the image grid) span every teacher correction the student could actually learn, so that whatever the Fisher projection discards is genuinely unreachable.

Editorial extensions

If this is right

With the 8B-Instruct teacher, FP-OPD beats standard OPD on all seven benchmarks for the 2B student, raising the average by 1.60 points over OPD and 2.77 points over the base student.
FP-OPD compounds with a stronger teacher: distilling from an 8B Geo3K-GRPO teacher yields a 52.78 average for the 2B student, the best in every column and 0.73 points above standard OPD with the same teacher.
The gains are not sampling artifacts: under greedy decoding the 32B-to-8B student reaches 63.95 on average, 1.12 points above standard OPD, improving six of seven benchmarks.
The projection metric carries most of the benefit: replacing the Fisher metric with Euclidean projection collapses the 8B-to-2B average by 8.79 points, whereas swapping the four spatial probes for random probes costs only 0.37 points.
The student actually absorbs the projected signal during training: median gap energy falls by 43.3% and median target-shift Jensen–Shannon divergence by 35.8% over two epochs, indicating the target shrinks toward the student rather than remaining a fixed unreachable demand.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

My inference: the reachability principle should extend beyond vision — any capacity gap between teacher and student (tool use, long-context reasoning, a second language) creates teacher corrections outside the student's local response space, and projecting the correction under the loss's own metric is a general recipe, not a vision-specific fix.
My inference: the four fixed boundary bands are a hand-chosen, domain-specific proxy for the tangent space, so a data-driven or adaptive perturbation basis is the natural stress test; whether such a basis improves results would reveal whether the specific spatial bands are essential or merely convenient.
My inference: FP-OPD should show its largest edge exactly where the teacher's advantage is spatially localizable (diagram labels, angles, marks near image edges) and its smallest edge where the decisive visual detail is central or texture-based; a per-task breakdown against the image region that carries the evidence would test this.
My inference: if the residual $g_\perp$ is truly unlearnable by the current student, it becomes a natural training signal for a later, larger stage of the same model family, since it is by construction orthogonal to what the smaller student can absorb.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: FP-OPD is a real idea, cleanly formulated, and the math checks out. The paper replaces the full teacher target in on-policy distillation with a target projected onto a student visual tangent space under the Fisher metric. That is new relative to the OPD literature, and the local reverse-KL expansion and the centering identity are correct. The reported gains are consistent across seven benchmarks and two model scales, which is more than most distillation papers show. Credit where it is due: this is a solid, well-motivated technique, not a repackaging of existing tricks.

The soft spots are real but not fatal. The biggest issue, which the stress-test note nails, is the missing scalar-scaled baseline. The paper shows that α=0.5 beats α=1, and FP-OPD always weakens the target by projection. Without comparing against q_α = softmax(ℓθ + αg) with matched norm, we do not know if the specific subspace direction matters or whether any moderate target-strength reduction reproduces the gains. The random-probe ablation strengthens that worry: random probes with Fisher projection reach 50.90 versus 51.27 with structured probes, only 0.37 apart. So the precise 'visual tangent space' may be doing little work beyond providing a low-dimensional, norm-reducing subspace. The Fisher versus Euclidean contrast (8.79 points) shows the metric matters, but that is a separate claim from the subspace direction being right.

The other weaknesses are the usual suspects: hyperparameters (εfd, K, γ, ρ) are tuned on the evaluation benchmarks and reported without error bars or multi-seed statistics, and the unresolved 'Appendix??' is an unacceptable hole in a paper that claims reproducibility. These are fixable but need to be addressed before the central claim is accepted.

Who is this for? Anyone working on distillation for VLMs or on capacity-aware training targets. It is a worthwhile contribution that deserves a serious referee, but the referee should push on the scalar-scaling baseline and the probe-basis sensitivity. I would not desk-reject it, and I would want to see the revision.

Recommendation: engage with it, send it to review, and insist on the missing baseline and proper reproducibility material.

Referee Report

3 major / 6 minor

Summary. The paper proposes Fisher-Projected On-Policy Distillation (FP-OPD), an on-policy distillation variant for vision--language models that replaces the complete teacher distribution p_T with a capacity-aware target q_R. The target is built by writing the OPD teacher target as softmax(ell_theta + g), estimating a student visual tangent space via finite-difference perturbations of visual features, and projecting the centered teacher--student gap g onto that space under the Fisher metric. Experiments on Qwen3-VL 8B-to-2B and 32B-to-8B distillation over seven benchmarks report an average improvement of +1.60 points over standard OPD in the main 8B-to-2B setting, with ablations over the finite-difference step, the number of probes, and the projection geometry.

Significance. The core idea is appealing and clearly presented: instead of uniformly weakening the teacher, keep only the component of the teacher correction that the student's visual representation can locally realize. The algebraic decomposition in Eqs. (6)--(7) and the local reverse-KL/Fisher justification in Eq. (12) are correct, and the experimental results are internally consistent. The paper also deserves credit for a relatively clean ablation chain that isolates the finite-difference step, the number of probes, and the projection geometry. However, the central mechanistic claim--that the specific student-supported visual subspace is what matters--is not yet supported, because no comparison is made against a scalar-scaled target with matched norm, and the random-probe ablation in Table 3 suggests that most of the gain may come from norm reduction rather than subspace direction. If the matched-norm control confirms the subspace story, this would be a solid contribution to distillation for multimodal models; as it stands, the mechanism is underdetermined.

major comments (3)

[Section 4.3, Table 3; Section 3.4, Eqs. (13)--(18); Figure 1(c)] The paper does not compare FP-OPD against a scalar-scaled target of the form q_alpha = softmax(stopgrad(ell_theta) + alpha g) with alpha chosen to match the Fisher norm of the projected correction bhat_g_R. Since the Fisher projection is norm-reducing by the Pythagorean theorem applied to Eq. (18), the reported gains are consistent with simple target weakening. Indeed, the paper's own interpolation experiment in Figure 1(c) shows that alpha = 0.5 is both easier to fit and better on MathVista Geometry than alpha = 1. To establish that the direction of the correction, rather than only its norm, is what matters, the authors should include this matched-norm scalar baseline. The random-probe ablation in Table 3 amplifies the concern: Fisher + random probes reaches 50.90 versus 51.27 for structured visual probes, only 0.37 points apart and 1.23 points above the standard OPD baseline of 49.67, suggesting that the specific visual tangent-space structure is not the dominant factor.
[Section 4.1, Training paragraph] The sentence "Complete training, method, and evaluation configurations are provided in Appendix??" references a non-existent appendix. Without these details, the empirical claims cannot be reproduced or even fully checked. The appendix must be included, or the reference must be removed and the details provided in a supplementary document, before the manuscript is in publishable form.
[Tables 1--3] No error bars, number of seeds, or significance tests are reported. Several headline differences are small (for example, +0.17 on HallusionBench and +0.30 on HallusionBench in Table 1, and -0.53 and -0.21 in the 32B-to-8B setting), and the claims of "consistent gains" and of not sacrificing general performance depend on treating these as meaningful. Please report standard deviations over at least three runs, or otherwise provide an uncertainty estimate, for the main results and ablations.

minor comments (6)

[Abstract] There is a missing space in "proposeFisher-Projected" in the abstract.
[Section 3.1, Eq. (5)] The centering operator display is visually confusing because "a - <a,1>_p 1" is written on the same line as the definition; use a clearer display with braces to indicate that the scalar is subtracted as a vector of ones.
[Section 4.1, Evaluation paragraph] The metric "AVG@8" is used throughout but not defined; state explicitly whether it is the average accuracy over eight sampled rollouts and give the sampling protocol.
[Figure 3] The axes of Figure 3 are not labeled; specify what is on the x-axis (steps) and y-axis for gap energy and target-shift JS.
[Section 4.4, Mechanism Analysis] The phrase "the first and last 32 updates" should specify the step range and whether these are gradient updates or sampled-batch evaluations.
[References, We-Math entry] The We-Math reference contains a duplicated name "MinhuiWu MinhuiWu"; this should be corrected to a single author name.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: FP-OPD's target is constructed from the student's own Fisher geometry and the teacher gap, not from the benchmark scores, so the reported gains are independent empirical evidence rather than a definitional reduction.

full rationale

The derivation chain is self-contained. The teacher target is exactly pT = softmax(ℓθ + g) (Eq. 7); the FP-OPD target is qR = softmax(stopgrad(ℓθ) + bĝR) (Eq. 20); and bĝR is the Fisher-metric least-squares projection of the centered teacher–student gap g onto the finite-difference tangent space bTθ,K (Eqs. 13–18). None of these objects is defined in terms of the seven benchmark averages, so the reported 1.60-point OPD improvement is not forced by construction. The paper's own target-interpolation experiment is an empirical motivation, not an input that the projection is fitted to reproduce. The missing scalar-α control is a genuine scientific gap: FP-OPD always produces a target with Fisher norm no larger than ‖g‖, so gains might be attributable to target weakening rather than to the specific subspace direction. But an alternative explanation is not circularity. Similarly, choosing εfd and K via ablations on the same benchmarks is model selection, not a fitted-input-called-prediction cycle. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled through prior work; the unresolved 'Appendix??' and missing error bars are reporting defects, not circular steps. Overall, the central claim is empirically testable and its inputs do not already contain the conclusion.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on standard Fisher geometry, a first-order visual-only model of student reachability, and an ad hoc choice of four boundary-band probes. No new physical or architectural entities are introduced.

free parameters (4)

finite-difference step epsilon_fd = 0.05
Selected by ablation on the seven benchmarks in Section 4.3; controls the perturbation magnitude in Eq. (9).
number of tangent probes K = 4
Chosen by comparing K=2, 4, and 8 in Section 4.3; K=4 gives the best average on the evaluation benchmarks.
spatial band fraction gamma = 0.25
Sets the width of the four boundary bands used to define the probe fields in Section 4.1; not varied in ablations.
ridge regularization rho = 1e-4
Stabilizes the Gram system in Eq. (16); no sensitivity analysis is reported.

assumptions (4)

standard math The local geometry of KL divergence is captured by the Fisher inner product, so projecting under Eq. (4/12) is the correct geometry for the distillation target.
Invoked in Section 3.4, Eq. (12); standard in information geometry, but assumes higher-order terms in the KL expansion are negligible.
domain assumption The student's locally reachable output space is the range of the Jacobian of student log-probabilities with respect to visual embeddings (Eq. 8).
This first-order, visual-only model of reachability enters at Eq. (8); parameter-space reachability and non-visual sources of correction are not modeled.
ad hoc to paper The span of four boundary-band finite-difference directions approximates the student-supported visual tangent space.
Eqs. (9)-(11) and Section 4.3; no convergence or coverage diagnostic is given, and random probes behave differently.
domain assumption Teacher corrections outside the empirical tangent span are not useful for the compact student.
The residual g⊥ is discarded in Eq. (19); this is the load-bearing premise of the method and is only tested indirectly through downstream benchmarks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models." pith.science (2026). https://pith.science/paper/AL274IHD

@misc{pith2026260801263,
  author       = {Pith},
  title        = {Pith review of: Distill What the Student Can See: Fisher-Projected On-Policy Distillation for Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AL274IHD}},
  note         = {Machine review of arXiv:2608.01263}
}

read the original abstract

On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories. This aligns the distillation states with the student's own generation distribution. However, it still assumes that the complete teacher distribution is an appropriate target across student capacities. In vision--language reasoning, teacher corrections can depend on visual distinctions that a compact student cannot represent. Our target-scaling study shows that, as the target approaches the complete teacher distribution, the student realizes less of the prescribed shift and obtains worse downstream performance. We therefore propose \emph{Fisher-Projected On-Policy Distillation} (FP-OPD), which distills only locally realizable teacher corrections. FP-OPD uses continuous visual perturbations to estimate the student's local visual tangent space and projects the centered teacher--student log-probability gap onto this space under the student's Fisher metric. The resulting capacity-aware target is optimized with full-vocabulary reverse KL on student trajectories, retaining the standard OPD framework. In 8B-to-2B distillation, FP-OPD improves all seven evaluated multimodal benchmarks. It raises the average score by 2.77 points over the pretrained student and by 1.60 points over standard OPD. These results demonstrate that locally realizable teacher corrections provide a more effective target for distilling compact vision--language models.

Figures

Figures reproduced from arXiv: 2608.01263 by the authors.

**Figure 1.** Motivation and empirical evidence for capacity-aware on-policy distillation. (a) Even on a shared student-generated prefix, teacher supervision can reflect visual evidence that is not represented correctly by the student, leading to different token predictions and final answers. (b) The complete teacher target pT can lie outside the student’s local response geometry, whereas Fisher projection produces a reachable ta… view at source ↗

**Figure 2.** Overview of FP-OPD. The student generates an on-policy trajectory, after which the student and teacher distributions are evaluated on the same student-visited prefixes. Continuous visual perturbations produce local output-response directions whose span approximates the studentsupported visual response space. FP-OPD projects the centered teacher–student gap onto this space under the Fisher metric, constructs the rea… view at source ↗

**Figure 3.** Teacher–student gap energy and target-shift divergence during 32B-to-8B FP-OPD training. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 2 canonical work pages

[1]

On-policy distillation of language models: Learning from self- generated mistakes

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pages 21246–21263,

2024
[6]

Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

arXiv
[9]

Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780,

Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780,

arXiv
[11]

Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling

Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. In Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors,International Conference on Learning Representations, volume 2025, pages ...

work page 2025
[12]

Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, et al

URL https://proceedings.iclr.cc/paper_files/paper/2025/ file/a2747a3844ca1e4667fbff3f558eb39b-Paper-Conference.pdf. Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, et al. Nemotron-cascade 2: Post-training llms with cascade rl and multi-domain on-policy distillation.arXiv pre...

arXiv 2025
[13]

Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation.arXiv preprint arXiv:2605.18740,

Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, and Yaojie Lu. Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation.arXiv preprint arXiv:2605.18740,

arXiv
[14]

Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

arXiv
[15]

ISBN 979-8-89176-251-0

Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/ v1/2025.acl-long.1097. URLhttps://aclanthology.org/2025.acl-long.1097/. Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gre- gory D Lyng, Sanjit Singh Batra, and Robert E Tillman. Fast and effective on-policy distillation from reasoning...

work page 2025

Show all 17 references

[16]

Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734,

arXiv
[17]

Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,

Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,

arXiv
[1998]

Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dan...

arXiv
[2015]

Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326,

arXiv
[2018]

Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

arXiv
[2021]

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InInternational Conference on Learning Representati...

2024
[2024]

Gradient descent happens in a tiny subspace.arXiv preprint arXiv:1812.04754,

Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace.arXiv preprint arXiv:1812.04754,

arXiv
[2025]

https://thinkingmachines.ai/blog/on-policy-distillation

doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation. Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. InThe...

doi
[2026]

Deepseek-v4: Towards highly efficient million- token context intelligence.arXiv preprint arXiv:2606.19348,

Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million- token context intelligence.arXiv preprint arXiv:2606.19348,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

On-policy distillation of language models: Learning from self- generated mistakes

Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pages 21246–21263,

2024

[2] [6]

Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016,

arXiv

[3] [9]

Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780,

Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al. Mimo-v2-flash technical report.arXiv preprint arXiv:2601.02780,

arXiv

[4] [11]

Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling

Wenda Xu, Rujun Han, Zifeng Wang, Long Le, Dhruv Madeka, Lei Li, William Wang, Rishabh Agarwal, Chen-Yu Lee, and Tomas Pfister. Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling. In Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors,International Conference on Learning Representations, volume 2025, pages ...

work page 2025

[5] [12]

Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, et al

URL https://proceedings.iclr.cc/paper_files/paper/2025/ file/a2747a3844ca1e4667fbff3f558eb39b-Paper-Conference.pdf. Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, et al. Nemotron-cascade 2: Post-training llms with cascade rl and multi-domain on-policy distillation.arXiv pre...

arXiv 2025

[6] [13]

Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation.arXiv preprint arXiv:2605.18740,

Qianhao Yuan, Jie Lou, Xing Yu, Hongyu Lin, Le Sun, Xianpei Han, and Yaojie Lu. Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation.arXiv preprint arXiv:2605.18740,

arXiv

[7] [14]

Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

arXiv

[8] [15]

ISBN 979-8-89176-251-0

Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/ v1/2025.acl-long.1097. URLhttps://aclanthology.org/2025.acl-long.1097/. Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gre- gory D Lyng, Sanjit Singh Batra, and Robert E Tillman. Fast and effective on-policy distillation from reasoning...

work page 2025