pith. sign in

arxiv: 2605.13230 · v2 · pith:VPZTCO4Jnew · submitted 2026-05-13 · 💻 cs.LG · cs.AI

Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence

Xinyu Liu , Kechen Jiao , Chunyang Xiao , Runsong Zhao , Junhao Ruan , Bei Li , Jiahao Liu , Qifan Wang
show 5 more authors
Xin Chen Jingang Wang Chenglong Wang Tong Xiao JingBo Zhu
This is my paper

Pith reviewed 2026-05-14 20:08 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords LLM distillationreverse KLpolicy optimizationon-policy learningteacher guidancereasoning benchmarksRLVR
0
0 comments X

The pith

Teacher-Guided Policy Optimization fixes uninformative feedback in reverse KL by conditioning teacher predictions on student rollouts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper identifies that standard reverse KL in on-policy LLM distillation fails to provide meaningful guidance when student and teacher distributions diverge, leading to uninformative negative feedback. It proposes Teacher-Guided Policy Optimization (TGPO), which conditions teacher predictions on the student's rollout to supply dense directional guidance while remaining on-policy. This approach integrates with existing RLVR frameworks without needing additional annotations. On complex reasoning benchmarks, TGPO outperforms standard baselines and shows robustness to different teacher models. Readers should care because it addresses a practical bottleneck in distilling capable reasoning models from stronger teachers.

Core claim

TGPO is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout to address the inefficiency of standard RKL when distributions diverge significantly.

What carries the argument

Teacher predictions conditioned on the student's rollout, providing dense directional guidance in the on-policy setting.

Load-bearing premise

That conditioning teacher predictions on the student's rollout will reliably produce informative directional guidance even when student and teacher distributions diverge substantially.

What would settle it

Running TGPO and standard RKL on a setup with deliberately large student-teacher divergence and checking if TGPO fails to improve or worsens performance compared to the baseline.

Figures

Figures reproduced from arXiv: 2605.13230 by Bei Li, Chenglong Wang, Chunyang Xiao, Jiahao Liu, Jingang Wang, Jingbo Zhu, Junhao Ruan, Kechen Jiao, Qifan Wang, Runsong Zhao, Tong Xiao, Xin Chen, Xinyu Liu.

Figure 1
Figure 1. Figure 1: RKL vs. TGPO. (a) RKL relies on scalar rewards to penalize deviation. When the policy gap is significant, these penal￾ties fail to provide directional information. (b) TGPO utilizes the teacher’s predicted distribution as guidance, explicitly informing the student what to generate next rather than what not to generate. et al., 2025), resulting in a rigid model with limited explo￾ration capabilities (Chen e… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of RKL distillation dynamics. We distill a Qwen2.5-Math-1.5B student using either an In-Family teacher (Qwen2.5-Math-7B) or a Cross-Family teacher (Qwen3-30B-A3B). While the In-Family setting converges stably, the Cross-Family setting exhibits catastrophic instability, characterized by sharp training score degradation (Left), gradient norm spikes (Middle), and unbounded response length growth (R… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the TGPO. The Policy Model generates a group of rollouts {yi} G i=1 conditioned on input x. At each step, the Teacher Model provides dynamic token-level guidance by predicting the optimal target token y T based on the student’s current prefix. This dense guidance signal (J) complements the outcome-based advantage (A) derived from the Rule-based Verifier to update the policy. with sequence yi (e… view at source ↗
Figure 4
Figure 4. Figure 4: Training Dynamics Analysis. (Left) Training reward. TGPO demonstrates robust growth and convergence compared to RKL. (Middle) Response length. TGPO avoids RKL’s length explosion and aligns with GRPO++’s stability. (Right) Gradient norm. TGPO shows stable optimization compared to the high variance in RKL, KDRL and LUFFY. 5. Experimental Results 5.1. Main Results Overall Performance [PITH_FULL_IMAGE:figures… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation of annealing schedules. The inset details the guidance weight (w) schedule for each setting. Our method yields the best convergence. though both strategies start similarly, the performance gap becomes evident in the later stages. This suggests that a transition to a “pure RL phase” (where wt = 0) before the end of training is essential. By removing the imitation con￾straint at step 200, Ours allow… view at source ↗
Figure 6
Figure 6. Figure 6: Training Dynamics Analysis. (Left) Training reward. TGPO demonstrates robust growth and convergence compared to RKL and KDRL. (Middle) Response length. TGPO avoids RKL’s length explosion and aligns with GRPO++’s stability. (Right) Gradient norm. TGPO shows stable optimization compared to the high variance in RKL, KDRL and LUFFY. B.3. Analysis of Experiments on the 1.5B Model B.3.1. OVERALL PERFORMANCE To f… view at source ↗
read the original abstract

On-policy distillation (OPD) has become a promising paradigm for reasoning-oriented post-training of large language models (LLMs), especially when combined with reinforcement learning from verifiable rewards (RLVR). Existing OPD methods rely on reverse KL (RKL)-based teacher supervision over trajectories sampled from the student policy. However, we identify a critical limitation: under large teacher--student policy divergence, RL-driven exploration often produces trajectories outside the teacher distribution, resulting in uninformative negative feedback. To address this, we propose Teacher-Guided Policy Optimization (TGPO), an on-policy reasoning distillation method that remains effective under large policy divergence settings. Rather than relying solely on evaluative supervision, TGPO uses teacher to directly guide token level generation conditioning on student-generated contexts; together with RLVR-style trajectory level rewards, TGPO steers exploration toward improved continuations. Experiments on reasoning benchmarks show that TGPO consistently outperforms existing RKL-based OPD methods and remains robust across different teacher models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper identifies a limitation in standard Reverse KL (RKL) for on-policy LLM distillation where significant divergence between student and teacher distributions leads to uninformative negative feedback. It proposes Teacher-Guided Policy Optimization (TGPO), which incorporates dense directional guidance by using teacher predictions conditioned on the student's rollout. TGPO is on-policy and compatible with RLVR frameworks. Experiments on complex reasoning benchmarks show that TGPO significantly outperforms standard baselines and is robust to different teachers.

Significance. If the empirical results and the conditioning mechanism hold under scrutiny, TGPO could meaningfully advance on-policy distillation for LLMs by mitigating a documented failure mode of RKL, allowing better unification of exploration and supervision without extra annotation. This would be relevant for reasoning-heavy tasks where distribution shift is routine.

major comments (3)
  1. [Abstract] Abstract: the central claim that TGPO 'significantly outperforms standard baselines' supplies no metrics, benchmark names, run counts, statistical tests, or baseline definitions, so the empirical contribution cannot be evaluated.
  2. [Method] Method description: no derivation, gradient bound, or variance analysis is given showing that teacher predictions conditioned on low-probability student rollouts remain accurate and low-variance; this is the load-bearing assumption needed to fix the RKL divergence problem.
  3. [Experiments] Experiments section: the abstract asserts robustness to different teachers on complex reasoning benchmarks but provides no ablation isolating the conditioning step, no comparison tables, and no details on how divergence was measured or controlled.
minor comments (1)
  1. [Abstract] Abstract: the acronym RLVR is used without expansion on first appearance.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment point by point below. Revisions have been made to the manuscript to improve specificity, clarity, and completeness in the abstract, method discussion, and experiments section.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that TGPO 'significantly outperforms standard baselines' supplies no metrics, benchmark names, run counts, statistical tests, or baseline definitions, so the empirical contribution cannot be evaluated.

    Authors: We agree that the abstract lacked sufficient quantitative detail for independent evaluation. In the revised manuscript, the abstract now specifies the benchmarks (GSM8K, MATH), reports average accuracy improvements over baselines (standard RKL and PPO), notes the use of 5 random seeds, and defines the baselines explicitly. Full statistical details, including standard deviations and t-test results, remain in the experiments section due to length constraints. revision: yes

  2. Referee: [Method] Method description: no derivation, gradient bound, or variance analysis is given showing that teacher predictions conditioned on low-probability student rollouts remain accurate and low-variance; this is the load-bearing assumption needed to fix the RKL divergence problem.

    Authors: This comment correctly identifies a theoretical gap. The manuscript presents TGPO algorithmically and supports the conditioning assumption through empirical results rather than a formal variance bound. We have added an expanded intuitive discussion in Section 3 explaining how conditioning on the student rollout reduces uninformative gradients by aligning the teacher's signal with the actual trajectory. A complete gradient bound or variance analysis is not provided, as it would require substantial new theoretical work; we explicitly flag this as a limitation and future direction in the revised paper. revision: partial

  3. Referee: [Experiments] Experiments section: the abstract asserts robustness to different teachers on complex reasoning benchmarks but provides no ablation isolating the conditioning step, no comparison tables, and no details on how divergence was measured or controlled.

    Authors: We have revised the experiments section to include the requested elements. New comparison tables (Tables 1 and 2) report results on GSM8K and MATH with multiple teachers. An explicit ablation isolating the conditioning mechanism appears in Section 4.3, demonstrating performance degradation when it is removed. Divergence is now quantified using the KL divergence between student and teacher policies, with controlled experiments varying rollout sampling to induce different divergence levels. Robustness results across teacher models are presented with the corresponding metrics. revision: yes

Circularity Check

0 steps flagged

No significant circularity; TGPO framed as extension of RLVR without self-referential reductions

full rationale

The provided abstract and description position TGPO as an on-policy extension that adds conditioned teacher guidance to address RKL limitations in existing RLVR frameworks. No equations, derivations, or fitted parameters are shown that reduce any claimed improvement or prediction to a quantity defined by the method itself. No self-citations, uniqueness theorems, or ansatzes are invoked in a load-bearing way. The central claim rests on empirical outperformance on benchmarks rather than any closed-loop definition or renaming of known results. This is the expected non-finding for a methods paper that does not exhibit the enumerated circular patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The abstract introduces TGPO at a high level without specifying fitted parameters, new entities, or non-standard axioms; it relies on standard on-policy RL assumptions and existing RLVR frameworks.

axioms (1)
  • domain assumption Standard on-policy reinforcement learning assumptions continue to hold when teacher guidance is added via conditioned predictions.
    The method is described as integrating seamlessly with existing RLVR frameworks.

pith-pipeline@v0.9.0 · 5451 in / 1247 out tokens · 67683 ms · 2026-05-14T20:08:08.482193+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.