pith. machine review for the scientific record. sign in

arxiv: 2605.13230 · v1 · submitted 2026-05-13 · 💻 cs.LG · cs.AI

Recognition: 2 theorem links

· Lean Theorem

Teacher-Guided Policy Optimization for LLM Distillation

Xinyu Liu , Kechen Jiao , Chunyang Xiao , Runsong Zhao , Junhao Ruan , Bei Li , Jiahao Liu , Qifan Wang
show 4 more authors
Xin Chen Jingang Wang Tong Xiao Jingbo Zhu
Authors on Pith no claims yet

Pith reviewed 2026-05-14 20:08 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords LLM distillationreverse KLpolicy optimizationon-policy learningteacher guidancereasoning benchmarksRLVR
0
0 comments X

The pith

Teacher-Guided Policy Optimization fixes uninformative feedback in reverse KL by conditioning teacher predictions on student rollouts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies that standard reverse KL in on-policy LLM distillation fails to provide meaningful guidance when student and teacher distributions diverge, leading to uninformative negative feedback. It proposes Teacher-Guided Policy Optimization (TGPO), which conditions teacher predictions on the student's rollout to supply dense directional guidance while remaining on-policy. This approach integrates with existing RLVR frameworks without needing additional annotations. On complex reasoning benchmarks, TGPO outperforms standard baselines and shows robustness to different teacher models. Readers should care because it addresses a practical bottleneck in distilling capable reasoning models from stronger teachers.

Core claim

TGPO is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout to address the inefficiency of standard RKL when distributions diverge significantly.

What carries the argument

Teacher predictions conditioned on the student's rollout, providing dense directional guidance in the on-policy setting.

Load-bearing premise

That conditioning teacher predictions on the student's rollout will reliably produce informative directional guidance even when student and teacher distributions diverge substantially.

What would settle it

Running TGPO and standard RKL on a setup with deliberately large student-teacher divergence and checking if TGPO fails to improve or worsens performance compared to the baseline.

Figures

Figures reproduced from arXiv: 2605.13230 by Bei Li, Chunyang Xiao, Jiahao Liu, Jingang Wang, Jingbo Zhu, Junhao Ruan, Kechen Jiao, Qifan Wang, Runsong Zhao, Tong Xiao, Xin Chen, Xinyu Liu.

Figure 1
Figure 1. Figure 1: RKL vs. TGPO. (a) RKL relies on scalar rewards to penalize deviation. When the policy gap is significant, these penal￾ties fail to provide directional information. (b) TGPO utilizes the teacher’s predicted distribution as guidance, explicitly informing the student what to generate next rather than what not to generate. et al., 2025), resulting in a rigid model with limited explo￾ration capabilities (Chen e… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of RKL distillation dynamics. We distill a Qwen2.5-Math-1.5B student using either an In-Family teacher (Qwen2.5-Math-7B) or a Cross-Family teacher (Qwen3-30B-A3B). While the In-Family setting converges stably, the Cross-Family setting exhibits catastrophic instability, characterized by sharp training score degradation (Left), gradient norm spikes (Middle), and unbounded response length growth (R… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the TGPO. The Policy Model generates a group of rollouts {yi} G i=1 conditioned on input x. At each step, the Teacher Model provides dynamic token-level guidance by predicting the optimal target token y T based on the student’s current prefix. This dense guidance signal (J) complements the outcome-based advantage (A) derived from the Rule-based Verifier to update the policy. with sequence yi (e… view at source ↗
Figure 4
Figure 4. Figure 4: Training Dynamics Analysis. (Left) Training reward. TGPO demonstrates robust growth and convergence compared to RKL. (Middle) Response length. TGPO avoids RKL’s length explosion and aligns with GRPO++’s stability. (Right) Gradient norm. TGPO shows stable optimization compared to the high variance in RKL, KDRL and LUFFY. 5. Experimental Results 5.1. Main Results Overall Performance [PITH_FULL_IMAGE:figures… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation of annealing schedules. The inset details the guidance weight (w) schedule for each setting. Our method yields the best convergence. though both strategies start similarly, the performance gap becomes evident in the later stages. This suggests that a transition to a “pure RL phase” (where wt = 0) before the end of training is essential. By removing the imitation con￾straint at step 200, Ours allow… view at source ↗
Figure 6
Figure 6. Figure 6: Training Dynamics Analysis. (Left) Training reward. TGPO demonstrates robust growth and convergence compared to RKL and KDRL. (Middle) Response length. TGPO avoids RKL’s length explosion and aligns with GRPO++’s stability. (Right) Gradient norm. TGPO shows stable optimization compared to the high variance in RKL, KDRL and LUFFY. B.3. Analysis of Experiments on the 1.5B Model B.3.1. OVERALL PERFORMANCE To f… view at source ↗
read the original abstract

The convergence of reinforcement learning and imitation learning has positioned Reverse KL (RKL) as a promising paradigm for on-policy LLM distillation, aiming to unify exploration with teacher supervision. However, we identify a critical limitation: when the student and teacher distributions diverge significantly, standard RKL often fails to yield meaningful improvement due to uninformative negative feedback. To address this inefficiency, we propose Teacher-Guided Policy Optimization (TGPO), an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout. Because TGPO remains on-policy, the algorithm integrates seamlessly with existing RLVR frameworks without requiring additional data annotation. Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper identifies a limitation in standard Reverse KL (RKL) for on-policy LLM distillation where significant divergence between student and teacher distributions leads to uninformative negative feedback. It proposes Teacher-Guided Policy Optimization (TGPO), which incorporates dense directional guidance by using teacher predictions conditioned on the student's rollout. TGPO is on-policy and compatible with RLVR frameworks. Experiments on complex reasoning benchmarks show that TGPO significantly outperforms standard baselines and is robust to different teachers.

Significance. If the empirical results and the conditioning mechanism hold under scrutiny, TGPO could meaningfully advance on-policy distillation for LLMs by mitigating a documented failure mode of RKL, allowing better unification of exploration and supervision without extra annotation. This would be relevant for reasoning-heavy tasks where distribution shift is routine.

major comments (3)
  1. [Abstract] Abstract: the central claim that TGPO 'significantly outperforms standard baselines' supplies no metrics, benchmark names, run counts, statistical tests, or baseline definitions, so the empirical contribution cannot be evaluated.
  2. [Method] Method description: no derivation, gradient bound, or variance analysis is given showing that teacher predictions conditioned on low-probability student rollouts remain accurate and low-variance; this is the load-bearing assumption needed to fix the RKL divergence problem.
  3. [Experiments] Experiments section: the abstract asserts robustness to different teachers on complex reasoning benchmarks but provides no ablation isolating the conditioning step, no comparison tables, and no details on how divergence was measured or controlled.
minor comments (1)
  1. [Abstract] Abstract: the acronym RLVR is used without expansion on first appearance.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment point by point below. Revisions have been made to the manuscript to improve specificity, clarity, and completeness in the abstract, method discussion, and experiments section.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that TGPO 'significantly outperforms standard baselines' supplies no metrics, benchmark names, run counts, statistical tests, or baseline definitions, so the empirical contribution cannot be evaluated.

    Authors: We agree that the abstract lacked sufficient quantitative detail for independent evaluation. In the revised manuscript, the abstract now specifies the benchmarks (GSM8K, MATH), reports average accuracy improvements over baselines (standard RKL and PPO), notes the use of 5 random seeds, and defines the baselines explicitly. Full statistical details, including standard deviations and t-test results, remain in the experiments section due to length constraints. revision: yes

  2. Referee: [Method] Method description: no derivation, gradient bound, or variance analysis is given showing that teacher predictions conditioned on low-probability student rollouts remain accurate and low-variance; this is the load-bearing assumption needed to fix the RKL divergence problem.

    Authors: This comment correctly identifies a theoretical gap. The manuscript presents TGPO algorithmically and supports the conditioning assumption through empirical results rather than a formal variance bound. We have added an expanded intuitive discussion in Section 3 explaining how conditioning on the student rollout reduces uninformative gradients by aligning the teacher's signal with the actual trajectory. A complete gradient bound or variance analysis is not provided, as it would require substantial new theoretical work; we explicitly flag this as a limitation and future direction in the revised paper. revision: partial

  3. Referee: [Experiments] Experiments section: the abstract asserts robustness to different teachers on complex reasoning benchmarks but provides no ablation isolating the conditioning step, no comparison tables, and no details on how divergence was measured or controlled.

    Authors: We have revised the experiments section to include the requested elements. New comparison tables (Tables 1 and 2) report results on GSM8K and MATH with multiple teachers. An explicit ablation isolating the conditioning mechanism appears in Section 4.3, demonstrating performance degradation when it is removed. Divergence is now quantified using the KL divergence between student and teacher policies, with controlled experiments varying rollout sampling to induce different divergence levels. Robustness results across teacher models are presented with the corresponding metrics. revision: yes

Circularity Check

0 steps flagged

No significant circularity; TGPO framed as extension of RLVR without self-referential reductions

full rationale

The provided abstract and description position TGPO as an on-policy extension that adds conditioned teacher guidance to address RKL limitations in existing RLVR frameworks. No equations, derivations, or fitted parameters are shown that reduce any claimed improvement or prediction to a quantity defined by the method itself. No self-citations, uniqueness theorems, or ansatzes are invoked in a load-bearing way. The central claim rests on empirical outperformance on benchmarks rather than any closed-loop definition or renaming of known results. This is the expected non-finding for a methods paper that does not exhibit the enumerated circular patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The abstract introduces TGPO at a high level without specifying fitted parameters, new entities, or non-standard axioms; it relies on standard on-policy RL assumptions and existing RLVR frameworks.

axioms (1)
  • domain assumption Standard on-policy reinforcement learning assumptions continue to hold when teacher guidance is added via conditioned predictions.
    The method is described as integrating seamlessly with existing RLVR frameworks.

pith-pipeline@v0.9.0 · 5451 in / 1247 out tokens · 67683 ms · 2026-05-14T20:08:08.482193+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

23 extracted references · 22 canonical work pages · 15 internal anchors

  1. [1]

    arXiv preprint arXiv:2504.11468 , year=

    Chen, H., Tu, H., Wang, F., Liu, H., Tang, X., Du, X., Zhou, Y ., and Xie, C. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468,

    work page arXiv

  2. [2]

    SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

    Chu, T., Zhai, Y ., Yang, J., Tong, S., Xie, S., Schuurmans, D., Le, Q. V ., Levine, S., and Ma, Y . Sft memorizes, rl generalizes: A comparative study of foundation model post-training.arXiv preprint arXiv:2501.17161,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Process Reinforcement through Implicit Rewards

    Cui, G., Yuan, L., Wang, Z., Wang, H., Zhang, Y ., Chen, J., Li, W., He, B., Fan, Y ., Yu, T., et al. Process re- inforcement through implicit rewards.arXiv preprint arXiv:2502.01456,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    MiniLLM: On-Policy Distillation of Large Language Models

    Gu, Y ., Dong, L., Wei, F., and Huang, M. Minillm: Knowl- edge distillation of large language models.arXiv preprint arXiv:2306.08543,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Skywork open reasoner 1 technical report.arXiv preprint arXiv:2505.22312,

    He, J., Liu, J., Liu, C. Y ., Yan, R., Wang, C., Cheng, P., Zhang, X., Zhang, F., Xu, J., Shen, W., et al. Sky- work open reasoner 1 technical report.arXiv preprint arXiv:2505.22312,

    work page arXiv

  8. [8]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Distilling the Knowledge in a Neural Network

    Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    OpenAI o1 System Card

    URL https://github. com/huggingface/open-r1. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Solving Quantitative Reasoning Problems with Language Models

    Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V ., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models, 2022.URL https://arxiv. org/abs/2206.14858, 1,

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  12. [12]

    Understanding R1-Zero-Like Training: A Critical Perspective

    Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., Lee, W. S., and Lin, M. Understanding r1-zero-like train- ing: A critical perspective, 2025.URL https://arxiv. org/abs/2503.20783. Lu, K. and Lab, T. M. On-policy distillation.Thinking Machines Lab: Connectionism,

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    20251026

    doi: 10.64434/tml. 20251026. https://thinkingmachines.ai/blog/on-policy- distillation. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A 9 Submission and Formatting Instructions for ICML 2026 graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling,

    work page doi:10.64434/tml 2026

  14. [14]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al

    URL https: //arxiv.org/abs/2512.21852. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    work page arXiv

  15. [15]

    Gemma 2: Improving Open Language Models at a Practical Size

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ram ´e, A., et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Qwen3 Technical Report

    URL https: //arxiv.org/abs/2505.09388. Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language un- derstanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2506.02208 , year =

    Xu, H., Zhu, Q., Deng, H., Li, J., Hou, L., Wang, Y ., Shang, L., Xu, R., and Mi, F. Kdrl: Post-training reasoning llms via unified knowledge distillation and reinforcement learning.arXiv preprint arXiv:2506.02208,

    work page arXiv

  19. [19]

    arXiv preprint arXiv:2504.14945 , year =

    Yan, J., Li, Y ., Hu, Z., Wang, Z., Cui, G., Qu, X., Cheng, Y ., and Zhang, Y . Learning to reason under off-policy guidance, 2025.URL https://arxiv. org/abs/2504.14945,

    work page arXiv 2025

  20. [20]

    Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement

    Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., Lu, K., Xue, M., Lin, R., Liu, T., Ren, X., and Zhang, Z. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., Yue, Y ., Fan, T., Liu, G., Liu, L., Liu, X., et al. Dapo: An open-source llm reinforcement learning system at scale, 2025.URL https://arxiv. org/abs/2503.14476,

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    On-policy rl meets off-policy experts: Harmonizing supervised fine- tuning and reinforcement learning via dynamic weighting, 2026

    Notion Blog. Zhang, W., Xie, Y ., Sun, Y ., Chen, Y ., Wang, G., Li, Y ., Ding, B., and Zhou, J. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting, 2025a. URL https: //arxiv.org/abs/2508.11408. Zhang, Y ., Liu, Y ., Yuan, H., Yuan, Y ., Gu, Q., and Yao, A. C.-C. On the design of kl-r...

    work page arXiv 2026

  23. [23]

    bad” sample yk, rather than reinforcing the majority of “good

    In the context of RL with intrinsic rewards (as discussed in Section 2.2), the RKL term acts as a negative reward. The effective gradient update applied to the student is proportional to: g(y)∝ −∇ θ logπ θ(y|x)·logρ(y).(9) A.2. Instability in the Rejection Regime We now analyze the behavior of this gradient. Note that while language models generate tokens...

    2026