Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

The paper claims that scoring each student-generated token under light- versus heavy-thinking prompts and using the log-likelihood difference as a PPO advantage signal compresses reasoning length while preserving or improving accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:34 UTC pith:QFJVPOIJ

load-bearing objection Paired LT/HT teacher scoring gives a genuinely new distillation signal that compresses outputs across the board, but the claim that A_t encodes token-level reasoning utility is assumed, not demonstrated. the 4 major comments →

arxiv 2607.19046 v1 pith:QFJVPOIJ submitted 2026-07-21 cs.CV

Contrastive On-Policy Distillation

Jiacheng Ruan , Jun Tang , Wenzhen Yuan , Ting Liu , Shuai Bai , Dayiheng Liu , Zhibo Yang , Yuzhuo Fu This is my paper
classification cs.CV
keywords on-policy distillationreasoning compressiontoken-level advantagecontrastive learningchain-of-thought efficiencymultimodal language modelsPPOself-distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to establish that on-policy distillation fails to compress reasoning because matching a single teacher distribution cannot distinguish a token that advances a concise answer from one that merely prolongs the response. COPD replaces that single signal with a paired contrast: a frozen teacher scores the same student-generated token under a light-thinking and a heavy-thinking instruction, and the difference in log-probabilities becomes a token-level advantage for a clipped policy-gradient update. The paper reports that this contrastive signal shortens outputs by roughly half to two-thirds while maintaining or improving accuracy on nine multimodal benchmarks, across several student and teacher sizes, and without any explicit length penalty or answer-level reward. A variant that uses a frozen snapshot of the student itself as the scorer also compresses reasoning, which the authors interpret as re-weighting reasoning modes already present inside the model. A sympathetic reader would care because it proposes a dense, label-free preference signal for making reasoning models cheaper to run.

Core claim

The central claim is that the teacher's log-probability contrast across reasoning-effort instructions is a valid per-token preference signal for efficient reasoning. For each on-policy token y_t, the paper defines ℓ^LT_t = log π_φ(y_t | x^LT, y_<t) and ℓ^HT_t = log π_φ(y_t | x^HT, y_<t), then sets A_t = ℓ^LT_t − ℓ^HT_t, clipped to a bounded range and detached. A positive A_t means the frozen teacher judges the token more compatible with minimal-effort reasoning; a negative value means it favors exhaustive continuation. The student is trained by maximizing a clipped PPO-style objective weighted by A_t, which raises probability of light-favored tokens and lowers heavy-favored ones. The authors

What carries the argument

The mechanism is the paired teacher-scoring contrast: the same frozen teacher evaluates each student-generated token under two instruction contexts, light-thinking and heavy-thinking, at the same input and prefix. The difference A_t = ℓ^LT_t − ℓ^HT_t is used as a detached, clipped token-level advantage inside a clipped policy-gradient objective (Eq. 5). Its role is to convert a single teacher distribution into a state-matched comparison between reasoning modes, so the student learns which tokens are preferred under compact reasoning rather than merely imitating token probabilities.

Load-bearing premise

The load-bearing premise is that A_t = ℓ^LT_t − ℓ^HT_t measures a token's genuine contribution to efficient correct reasoning, rather than the teacher's stylistic response to prompt wording.

What would settle it

A concrete falsifier: run the same training pipeline with the LT and HT instruction templates swapped, or with two arbitrary wordings that are semantically unrelated to reasoning effort; if the response-length reduction persists, the signal is not about reasoning-mode preference. A stronger check is to label each token by whether deleting it preserves the teacher's correct answer and test whether A_t predicts that label.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the contrastive advantage is a valid preference signal, reasoning compression can be trained without length penalties, token budgets, accuracy rewards, or curated concise traces.
  • Because the signal is dense and on-policy, training rollouts themselves get shorter, and the paper reports roughly half the GPU hours of standard OPD baselines.
  • Accuracy per output token (Acc@1K) can be doubled or better on the tested benchmarks, shifting the accuracy–length frontier rather than trading one for the other.
  • The self-distillation variant implies that a model can distill its own reasoning toward lightweight modes without any external teacher, as long as its frozen snapshot exhibits the LT/HT contrast.
  • The framework generalizes in principle to other paired behavioral modes, though the paper only validates reasoning efficiency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the compression signal survives arbitrary instruction-pair swapping, then the mechanism is not about reasoning effort per se but about any stable contrast between two context distributions; that would broaden the method's scope but weaken its semantic interpretation.
  • Editorial inference: a direct token-utility validation — checking whether suppressed tokens can be deleted from the teacher's own complete answers without changing correctness — would settle whether A_t tracks contribution-to-answer or merely stylistic mode preference.
  • Editorial inference: the success of self-distillation suggests the approach may be closer to policy selection (re-allocating probability mass to already-supported concise paths) than to knowledge transfer, connecting it to preference-optimization methods that do not need a stronger model.
  • Editorial inference: the non-monotonic teacher-scaling results imply that the contrast itself, not teacher capability, carries the compression signal; a testable corollary is that even a weaker but similarly-styled teacher should produce comparable compression.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes COPD, a contrastive on-policy distillation method for multimodal LLMs. At each student-generated token, a frozen teacher scores the same token under light-thinking (LT) and heavy-thinking (HT) instruction prefixes; the log-likelihood difference A_t = ℓ_LT − ℓ_HT (Eq. 3) is used as a clipped token-level advantage in a PPO-style objective (Eq. 5). The method is evaluated on nine multimodal benchmarks across four teacher–student configurations, with ablations including a self-distillation variant (COPSD), prompt-template robustness, data/scale generalization, and training efficiency. The reported results show substantial response-length reduction (e.g., 57.0% reduction vs ExOPD in the 8B→2B setting, Table 1) with accuracy maintained or improved, leading to large gains in the Acc@1K metric.

Significance. If validated, COPD would be a significant addition to reasoning-compression research: it derives a dense, token-level training signal without explicit length penalties or answer rewards, works across model scales, and extends to self-distillation. The core idea—contrasting two instruction-conditioned teacher distributions at student-visited states—is simple, elegant, and potentially generalizable beyond LT/HT to other behavioral attributes. The experimental scope is substantial, and the ablations (e.g., Table 6 showing the necessity of the contrast rather than a single LT score) are informative. However, the central assumption that A_t encodes reasoning utility, rather than merely stylistic differences between the two instruction contexts, is not validated. The current evidence is compatible with a style-imitation explanation, so the mechanism claim is not yet established. The paper also lacks code release, single-run results without variance, and full prompt templates.

major comments (4)
  1. [Sec. 2.3, Eq. (3); Sec. 2.4, Eq. (5)] The advantage A_t is defined purely as a log-likelihood ratio between two instruction-conditioned teacher policies. The PPO objective in Eq. (5) includes no task reward, correctness signal, or length penalty; the entire claimed mechanism—compression while maintaining accuracy—rests on the implicit assumption that tokens with higher A_t are more useful for correct concise reasoning. The paper provides no token-level or trajectory-level validation that A_t correlates with answer correctness or with the causal contribution of each token to the final answer. Table 6 shows that the contrast is necessary for compression, but it does not show that the contrast encodes utility; a stylistic difference (e.g., LT prompts yielding more assertive or less self-verifying language) would produce the same effect. To support the mechanism claim, the authors should provide direct evidence—for example, a co
  2. [Sec. 3, Tables 1, 2, 5] All reported results are single-run point estimates without standard deviations, confidence intervals, or significance tests. Some benchmark-level accuracy differences are small relative to likely run-to-run variance (e.g., 52.7 vs 52.3 in Table 1, or 60.4 vs 60.1), while Acc@1K amplifies length effects. The claim of 'consistent' improvements would be considerably stronger with at least three independent seeds per setting and reported variance. This is a load-bearing issue for the central accuracy-maintenance claim, since the method changes both the numerator and denominator of Acc@1K.
  3. [Sec. 3.3.3, Table 4; Sec. 3.1.2] The exact LT and HT instruction templates are not provided in the paper, despite being the entire source of the contrastive signal. Table 4 names 'Simple,' 'Complex,' and 'default' variants but does not include their text. Without the exact prompts, the method is not reproducible, and it is impossible to assess whether the templates themselves introduce confounds. The full templates must be included in an appendix; ideally, code should also be released.
  4. [Sec. 3.3.4, Table 5] The COPSD result—that a frozen student snapshot provides a useful contrastive signal—is interesting and potentially the most generalizable claim. However, the offered explanation (reallocating probability mass to already-existing concise paths) is asserted rather than demonstrated. The same concern as in the first major comment applies: the self-contrast could be selecting style rather than utility. A direct test would be to measure whether the LT/HT contrast of the snapshot correlates with token-level correctness, and whether COPSD-compressed models retain accuracy on held-out tasks requiring genuine multi-step reasoning rather than simple fact recall.
minor comments (5)
  1. [Sec. 3.1.1, footnote 1] The pass-rate filter (retaining only samples with Pass Rate strictly between 0 and 1) is a nontrivial preprocessing choice that could affect both accuracy and length outcomes. Please justify this choice or provide an ablation without the filter.
  2. [Sec. 3.1.3, Table 1] MMGist is authored by the same research group (Yuan et al., 2026) and included as one of the nine evaluation benchmarks. The paper should explicitly disclose this overlap and, ideally, add an externally maintained benchmark to the evaluation set.
  3. [Sec. 2.3] Typo: 'two-side-clip' should be 'two-sided clip'.
  4. [Eq. (6)] The two algebraic forms of Acc@1K should be written more carefully: the first expression as printed is ambiguous about whether the 1/|B| factor applies inside the denominator. Use a single unambiguous definition.
  5. [Figures 1 and 2] The LT/HT prompt fragments shown in the figures are nearly unreadable and are not defined in the captions. Please include the full instruction text in an appendix and reference it in the captions.

Circularity Check

0 steps flagged

No significant circularity: the contrastive advantage is not fitted to the reported outcomes, and the central derivation is self-contained apart from a minor same-group benchmark citation.

full rationale

The paper's core signal, At = log pi_phi(y_t | x_LT, y_<t) - log pi_phi(y_t | x_HT, y_<t) (Eq. 3), is a teacher log-likelihood contrast computed at student-visited prefixes. It is not fitted to the reported accuracy, response length, or Acc@1K values, and no fitted parameter is renamed as a prediction. The PPO-style objective (Eq. 5) uses this detached advantage only as an update direction, and the downstream benchmark results are external empirical outcomes rather than consequences of an equation that equates At with length or accuracy. The paper does not invoke an author-specific uniqueness theorem, nor does it smuggle in the contrastive objective via an unverified citation; the OPD background is cited to standard prior work. The only notable self-citation is MMGist (Yuan et al., 2026), which shares authors with the present paper and is one of nine evaluation benchmarks. It is not load-bearing: the central claims hold across eight other benchmarks and four teacher-student configurations. The ablation in Table 6 provides an independent check that the contrast, rather than the light-thinking likelihood alone, drives the reported effect. The manuscript itself notes in Section 5 that only reasoning efficiency is validated and that transfer to other attributes remains open, which is an honest limitation rather than a circular step. The reviewer's concern that At may encode style rather than token-level utility is a correctness/interpretation risk, not a circularity: the paper does not claim At was derived from task reward, and no equation reduces the reported accuracy gains to the definition of At. Overall, the derivation is self-contained and the empirical claims are not forced by construction. Score 2 reflects only the minor, non-load-bearing same-group benchmark citation.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central result rests on an unvalidated assumption: that the LT/HT teacher-likelihood gap faithfully scores token usefulness. The method also assumes the hand-crafted prompts elicit the intended reasoning modes and that PPO without KL is stable. A_max and the ViRL pass-rate filter are hand-chosen hyperparameters. No new physical or ontological entities are introduced.

free parameters (2)
  • A_max advantage clip bound = 10
    Hand-set clip in Eq. (4) and Sec. 3.1.2 to bound the advantage to [-10,10]; no sensitivity analysis is reported.
  • ViRL pass-rate inclusion interval = (0,1)
    Footnote 1 filters ViRL-39K to samples whose Qwen2.5-VL-32B pass rate is strictly between 0 and 1; this hand-chosen filter shapes the training distribution.
axioms (4)
  • ad hoc to paper Teacher LT/HT log-likelihood difference A_t faithfully scores a token's contribution to concise reasoning
    Central signal; assumed at Eqs. (2)-(3), with no token-level validation against ground-truth token usefulness.
  • domain assumption Hand-written light/heavy instruction prefixes elicit distinct reasoning modes in the frozen teacher
    Used in Sec. 2.2; Table 4 tests wording variants but not whether the teacher's conditional distribution reflects reasoning effort rather than superficial style.
  • domain assumption Clipped PPO update without KL, accuracy reward, or length penalty remains stable and does not degenerate
    Sec. 2.4 and 3.1.2 drop KL/length rewards; stability relies on clipping and a small learning rate, with no formal guarantee.
  • domain assumption ViRL pass-rate-filtered subset and the nine benchmarks are representative for reasoning compression
    Sec. 3.1; no benchmark-contamination or transfer analysis beyond the reported results.

pith-pipeline@v1.3.0-alltime-deepseek · 17612 in / 11997 out tokens · 124613 ms · 2026-08-01T13:34:35.574303+00:00 · methodology

0 comments
read the original abstract

On-policy Distillation (OPD) supervises a student model on trajectories sampled from its own policy by minimizing the divergence between the output distributions of the teacher and student at each token position, thereby providing dense token-level supervision. Although existing OPD methods have demonstrated strong performance in improving the reasoning ability of student models, their objectives fundamentally rely on token-level distribution matching. Consequently, they lack an explicit signal for comparing a token's relative compatibility across reasoning modes and thus do not directly model preferences between these modes. To address this limitation, we propose COPD, a contrastive OPD framework. Specifically, for each token generated by the student model, a frozen teacher model scores the same student state under two contrasting instructions that elicit light and heavy reasoning. The difference between the resulting log probabilities serves as a token-level advantage signal to guide the OPD update. Rather than merely imitating a single teacher distribution, COPD directly encourages the student model to learn more concise and efficient reasoning strategies. We conduct experiments on nine multimodal benchmarks covering both reasoning and understanding tasks. The results show that COPD substantially reduces reasoning length without compromising model performance and consistently improves efficiency across different tasks and model scales. Furthermore, the contrastive formulation can be seamlessly integrated into the On-policy Self-distillation (OPSD) framework, where self-contrastive supervision is constructed without an additional teacher model, thereby enabling the model to distill itself toward lightweight reasoning.

Figures

Figures reproduced from arXiv: 2607.19046 by Dayiheng Liu, Jiacheng Ruan, Jun Tang, Shuai Bai, Ting Liu, Wenzhen Yuan, Yuzhuo Fu, Zhibo Yang.

Figure 1
Figure 1. Figure 1: COPD contrasts OPD-style supervision with a paired Light-thinking and Heavy￾thinking (LT/HT) teacher signal. The same on-policy student token is scored under light- and heavy-thinking prefixes to obtain a dense preference signal. ∗This work was done during an internship with the Qwen Team at Alibaba Group. † Project leader. ‡Corresponding author. 1 arXiv:2607.19046v1 [cs.CV] 21 Jul 2026 [PITH_FULL_IMAGE:f… view at source ↗
Figure 2
Figure 2. Figure 2: Training pipeline of COPD. The student first samples a response to the input. The frozen teacher then scores the same token sequence under light-thinking and heavy-thinking contexts, re￾spectively. The relative likelihood derived from these scores serves as a token-level advantage signal for policy updates. the generated tokens under the same visited prefixes (x, y<t) (Agarwal et al., 2024). This provides … view at source ↗
Figure 3
Figure 3. Figure 3: Response length over training. COPD progressively shortens rollout responses, whereas OPD and ExOPD remain at high re￾sponse lengths. contrast, At = ℓ LT t − ℓ HT t evaluates the same token under both light- and heavy-thinking contexts, thereby transforming the teacher scores into a directional signal that favors lightweight reasoning. This contrastive objective increases the average accuracy to 57.3% whil… view at source ↗
Figure 4
Figure 4. Figure 4: Case study on MathVerse. Compared with the Base model, our method produces a shorter [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation

    cs.CV 2026-07 conditional novelty 6.0

    Counterfactual present/removed teacher views attribute visually supported corrections and reconstruct student-anchored distillation targets that beat source-mixed multimodal OPD.

Reference graph

Works this paper leans on

39 extracted references · 29 linked inside Pith · cited by 1 Pith paper

  1. [1]

    On-policy distillation of language models: Learning from self- generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pp. 21246–21263,

    2024

  2. [6]

    Token- budget-aware llm reasoning

    Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware llm reasoning. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 24842–24855,

    2025

  3. [7]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Pith/arXiv arXiv

  4. [10]

    Towards reasoning in large language models: A survey

    Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. InFindings of the association for computational linguistics: ACL 2023, pp. 1049–1065,

    2023

  5. [11]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. InFindings of the association for computational linguistics: EMNLP 2020, pp. 4163–4174,

    2020

  6. [13]

    Efficient multimodal large language models: A survey.Visual Intelligence, 3(1):27, 2025a

    Yizhang Jin, Jian Li, Tianjun Gu, Yexin Liu, Bo Zhao, Jinxiang Lai, Zhenye Gan, Yabiao Wang, Chengjie Wang, Xin Tan, et al. Efficient multimodal large language models: A survey.Visual Intelligence, 3(1):27, 2025a. 12 Contrastive On-Policy Distillation Zhensheng Jin, Xinze Li, Yifan Ji, Chunyi Peng, Zhenghao Liu, Qi Shi, Yukun Yan, Shuo Wang, Furong Peng, ...

    Pith/arXiv arXiv 2016

  7. [14]

    Leash: Adaptive length penalty and reward shaping for efficient large reasoning model

    Yanhao Li, Lu Ma, Jiaran Zhang, Lexiang Tang, Wentao Zhang, and Guibo Luo. Leash: Adaptive length penalty and reward shaping for efficient large reasoning model. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pp. 2846–2856, 2026a. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaoju...

    Pith/arXiv arXiv 2026

  8. [15]

    Visual-advantage on-policy distillation for vision-language models.arXiv preprint arXiv:2605.21924,

    Ruiqi Liu, Xiaolei Lv, Gengsheng Li, Ximo Zhu, Zhiheng Wang, Zhengbo Zhang, Junkai Chen, Zhiheng Li, Bo Li, Jun Gao, et al. Visual-advantage on-policy distillation for vision-language models.arXiv preprint arXiv:2605.21924,

    Pith/arXiv arXiv

  9. [16]

    Dler: Doing length penalty right-incentivizing more intelligence per token via reinforcement learning.arXiv preprint arXiv:2510.15110,

    Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, et al. Dler: Doing length penalty right-incentivizing more intelligence per token via reinforcement learning.arXiv preprint arXiv:2510.15110,

    arXiv

  10. [17]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101,

    Pith/arXiv arXiv

  11. [18]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InInternational Conference on Learning Representations, volume 2024, pp. 23439–23554,

    2024

  12. [19]

    Self- training elicits concise reasoning in large language models

    13 Contrastive On-Policy Distillation Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, and Se-Young Yun. Self- training elicits concise reasoning in large language models. InFindings of the Association for Computational Linguistics: ACL 2025, pp. 25127–25152,

    2025

  13. [20]

    Concise thoughts: Impact of output length on llm reasoning and cost.arXiv preprint arXiv:2407.19825,

    Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on llm reasoning and cost.arXiv preprint arXiv:2407.19825,

    Pith/arXiv arXiv

  14. [21]

    Kl for a kl: On-policy distil- lation with control variate baseline.arXiv preprint arXiv:2605.07865,

    Minjae Oh, Sangjun Song, Gyubin Choi, Yunho Choi, and Yohan Jo. Kl for a kl: On-policy distil- lation with control variate baseline.arXiv preprint arXiv:2605.07865,

    Pith/arXiv arXiv

  15. [22]

    Privileged information distillation for language models.arXiv preprint arXiv:2602.04942,

    Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models.arXiv preprint arXiv:2602.04942,

    Pith/arXiv arXiv

  16. [23]

    Crisp: Com- pressed reasoning via iterative self-policy distillation.arXiv preprint arXiv:2603.05433,

    Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, and Jiachen Sun. Crisp: Com- pressed reasoning via iterative self-policy distillation.arXiv preprint arXiv:2603.05433,

    Pith/arXiv arXiv

  17. [24]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108,

    Pith/arXiv arXiv 1910

  18. [27]

    A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

    Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

    Pith/arXiv arXiv

  19. [28]

    Patient knowledge distillation for bert model compression

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), pp. 4323–4332,

    2019

  20. [29]

    Measuring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024a

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math-vision dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024a. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenb...

    Pith/arXiv arXiv

  21. [31]

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang

    URLhttps://huggingface.co/datasets/xai-org/ RealworldQA. Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts.arXiv preprint arXiv:2407.04973,

    Pith/arXiv arXiv

  22. [32]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125,

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125,

    Pith/arXiv arXiv

  23. [33]

    Preference- based self-distillation: Beyond kl matching via reward regularization.arXiv preprint arXiv:2605.05040,

    Xin Yu, Liuchen Liao, Yiwen Zhang, Yingchen Yu, Lingzhou Xue, and Qinzhen Guo. Preference- based self-distillation: Beyond kl matching via reward regularization.arXiv preprint arXiv:2605.05040,

    Pith/arXiv arXiv

  24. [34]

    Mmgist: A comprehensive multimodal benchmark for 2027.arXiv preprint arXiv:2606.22437,

    Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong, Chengping Zhao, Ting Liu, and Yuzhuo Fu. Mmgist: A comprehensive multimodal benchmark for 2027.arXiv preprint arXiv:2606.22437,

    Pith/arXiv arXiv 2027

  25. [35]

    Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825,

    Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models.arXiv preprint arXiv:2308.01825,

    Pith/arXiv arXiv

  26. [36]

    On-policy distillation with best-of-n teacher rollout selection.arXiv preprint arXiv:2605.09725, 2026a

    Ke Zhang, Yunjie Tian, Dongdi Zhao, Yijiang Li, Yuanye Liu, Vishal M Patel, and Di Fu. On-policy distillation with best-of-n teacher rollout selection.arXiv preprint arXiv:2605.09725, 2026a. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm ...

    Pith/arXiv arXiv

  27. [37]

    Are full rollouts necessary for on-policy distillation? arXiv preprint arXiv:2605.31490, 2026b

    Yaocheng Zhang, Jiajun Chai, Yuqian Fu, Songjun Tu, Xiaohan Wang, Wei Lin, Guojun Yin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Are full rollouts necessary for on-policy distillation? arXiv preprint arXiv:2605.31490, 2026b. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy s...

    Pith/arXiv arXiv

  28. [38]

    Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277,

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277,

    Pith/arXiv arXiv

  29. [39]

    Least-to-most prompting enables complex reasoning in large language models.arXiv preprint arXiv:2205.10625,

    Denny Zhou, Nathanael Sch ¨arli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models.arXiv preprint arXiv:2205.10625,

    Pith/arXiv arXiv

  30. [2015]

    Uni-opd: Unifying on-policy distillation with a dual-perspective recipe.arXiv preprint arXiv:2605.03677,

    Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, et al. Uni-opd: Unifying on-policy distillation with a dual-perspective recipe.arXiv preprint arXiv:2605.03677,

    Pith/arXiv arXiv

  31. [2017]

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  32. [2019]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  33. [2020]

    Entropy-aware on-policy distillation of language models

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079,

    Pith/arXiv arXiv

  34. [2021]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pp. 32694–32717,

    2024

  35. [2022]

    When more is less: Understanding chain-of-thought length in llms.arXiv preprint arXiv:2502.07266,

    Yuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms.arXiv preprint arXiv:2502.07266,

    Pith/arXiv arXiv

  36. [2023]

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al

    URLhttps://arxiv.org/abs/2308.12966. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025a. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen ...

    Pith/arXiv arXiv 2024

  37. [2024]

    Are we on the right way for evaluating large vision-language models?Advances in Neural Information Processing Systems, 37:27056–27087, 2024a

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?Advances in Neural Information Processing Systems, 37:27056–27087, 2024a. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zha...

    Pith/arXiv arXiv

  38. [2025]

    Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,

    Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao. Revisiting on-policy distillation: Empirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562,

    Pith/arXiv arXiv

  39. [2026]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 8003–8017,

    2023