The reviewed record of science sign in
Pith

arxiv: 2607.05339 · v1 · pith:CNMHZAEB · submitted 2026-07-06 · cs.LG · cs.AI· stat.ML

TREK: Distill to Explore, Reinforce to Refine

Yuanda Xu , Zhengze Zhou , Kayhan Behdin , Jelena Markovic-Voronov , Hejian Sang , Xiaomin Li , Wenhui Zhu , Xinchen Du
show 5 more authors
Aida Rahmattalabi Ran He Sen Na Zhipeng Wang Alborz Geramifard
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-07 16:14 UTCglm-5.2pith:CNMHZAEBrecord.jsonopen to challenge →

classification cs.LG cs.AIstat.ML
keywords GRPOexploration support expansionforward KLdistillationreinforcement learning from verifiable rewardsreachabilityprompt routinglanguage model reasoning
0
0 comments X

The pith

TREK uses teacher solutions to expand where a student model can explore, then lets reinforcement learning take over

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that when GRPO-style reinforcement learning stalls on hard reasoning prompts, the bottleneck is not reward sparsity but missing exploration coverage: the student never samples reward-bearing trajectories in the first place. TREK addresses this by treating distillation as exploration support expansion rather than imitation. The method identifies prompts where the unaided student has very low pass rate, queries a proposal source (an external teacher, or the same model given extra inference-time context) for verified solutions, keeps only the top-r solutions ranked by a trimmed length-normalized negative log-likelihood metric that approximates how close each trajectory is to the student's current support, applies a short forward-KL update to pull those verified solution modes into the student's sampling distribution, and then returns to ordinary on-policy GRPO. The central mechanism is the separation of prompt-level proposal availability (which prompts are hard enough to warrant teacher help) from trajectory-level reachability (which verified solutions the student can actually internalize). The paper shows consistent gains across math reasoning (AIME 2024/2025) and agentic tasks (ALFWorld, ScienceWorld), with the largest improvements concentrated on the hardest task types where unaided exploration is weakest.

Core claim

The paper's central claim is that forward-KL consolidation of verified, student-proximal teacher trajectories is a more effective support-expansion mechanism than on-policy distillation on the same trajectories, because forward KL directly penalizes the student for assigning low probability to verified proposal modes while OPD-style supervision only reshapes credit on trajectories the student already samples. This is supported by an ablation where replacing forward-KL consolidation with OPD-style supervision weakens performance by 1.1-2.9 points across AIME benchmarks and 3-5 points on agentic tasks. The paper also demonstrates that the proposal source need not be an external teacher: a self

What carries the argument

trimmed length-normalized NLL (dS) as a reachability proxy for ranking verified teacher trajectories by proximity to the student's current policy support

If this is right

  • If TREK's mechanism is correct, any on-policy RL method that depends on the student already sampling useful trajectories (DAPO, GSPO, PPO-based variants) could benefit from the same staged proposal-selection-then-refinement recipe, since the exploration bottleneck is structural rather than GRPO-specific.
  • The self-context variant suggests that inference-time reasoning enhancements (reflection, search, failure lessons) can be folded back into the deployment model's unaided policy through training, rather than requiring permanent test-time overhead.
  • The finding that gains concentrate on hardest task types implies that support-expansion methods should be evaluated on difficulty-stratified benchmarks rather than aggregate scores, which can mask where the mechanism actually operates.
  • The verified-only constraint is conservative; relaxing it to include near-miss teacher trajectories could broaden exploration further if the student has sufficient capacity to internalize them without instability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The trimmed NLL reachability metric is essentially a heuristic proxy for learnability; a more principled measure of whether a trajectory will produce stable gradients after consolidation could improve selection quality, especially for trajectories that are close in surface form but structurally distant in reasoning pattern.
  • The staged schedule (mine, propose, consolidate, refine) resembles curriculum learning but operates on the student's support geometry rather than example difficulty; this suggests a connection to mode-covering vs. mode-seeking divergence choices in distributional alignment, where forward KL is mode-covering and reverse KL is mode-seeking.
  • If the self-context variant's failure-lesson memory is itself learnable online during GRPO, the proposal source could improve in tandem with the student, potentially reducing the gap between self-context and external-teacher variants over training.

Load-bearing premise

The method depends on the trimmed length-normalized NLL being a good proxy for whether a verified teacher trajectory is genuinely internalizable by the student. This metric is sensitive to verbosity and surface form, and the trim parameters are heuristic; if it misranks trajectories, the consolidation phase could inject unstable gradients or waste capacity on trajectories that look close but are not actually learnable.

What would settle it

If forward-KL consolidation of top-r verified trajectories provides no advantage over simply adding more rollout budget to GRPO on hard prompts, or if the reachability ranking by trimmed NLL performs no better than random selection among verified proposals, the core mechanism claim would be undermined.

Figures

Figures reproduced from arXiv: 2607.05339 by Aida Rahmattalabi, Alborz Geramifard, Hejian Sang, Jelena Markovic-Voronov, Kayhan Behdin, Ran He, Sen Na, Wenhui Zhu, Xiaomin Li, Xinchen Du, Yuanda Xu, Zhengze Zhou, Zhipeng Wang.

Figure 1
Figure 1. Figure 1: Summary of main results. (a)–(b): Math reasoning accuracy (avg@16, %) across Qwen3 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of TREK as distillation for exploration support expansion. Standard GRPO stalls [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training success rate over optimization steps on ALFWorld (left) and ScienceWorld (right). [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
read the original abstract

Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student's on-policy support. We propose TREK (Teacher-Routed Exploration via Forward KL), a simple staged procedure that uses distillation not for imitation but for exploration support expansion. A key advantage of TREK is its generality: because it only consumes verified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-$r$ proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student's support, and then returns to standard on-policy GRPO refinement. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the self-context variant reaches 38.5 and 49.6 without an external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 7 minor

Summary. The paper proposes TREK, a staged training procedure that addresses the exploration bottleneck in GRPO on hard prompts. TREK identifies prompts where the student has low unaided pass rate, queries a proposal source (external teacher or same-model with extra context) for verified solutions, selects the top-r student-proximal trajectories by a trimmed length-normalized NLL metric, applies a short forward-KL consolidation phase, and then returns to standard GRPO. Experiments span mathematical reasoning (AIME 2024/2025 across Qwen3-1.7B/8B/14B) and agentic tasks (ALFWorld, ScienceWorld with Qwen2.5-7B-Instruct), with an OPD ablation and a self-context variant that requires no external teacher. The gains are consistent and concentrate on the hardest task types, supporting the central thesis.

Significance. The framing of distillation as exploration support expansion rather than imitation or credit shaping is a useful conceptual contribution. The output-only proposal interface (black-box teacher compatibility) is practically valuable. The self-context variant, which uses failure-lesson memory to generate proposals from the same model, is a notable demonstration that the method does not require an external teacher. The per-task-type ALFWorld breakdown (Table 3) provides falsifiable evidence that gains concentrate where unaided exploration is weakest. The 10-seed aggregation for agentic tasks and the OPD ablation across both domains strengthen the empirical contribution.

major comments (2)
  1. The 2× rollout-budget GRPO baseline is defined in Appendix B (Table 5: '32 unaided samples per prompt; no teacher proposals') but its results are never reported in any table or figure. This is the single most important control for the paper's central thesis. The paper claims in §1 that 'larger rollout groups or sharper relative advantages may still search within a narrow region of the student's current support,' but this claim is asserted rather than tested. If doubling the rollout budget closes most of the gap on AIME or ALFWorld, the problem is insufficient sampling rather than missing support, and the teacher-proposal mechanism would be unnecessary. The reported gains (e.g., +3.4 on AIME 2025 for Qwen3-8B, +7.0 on ALFWorld) cannot be distinguished from 'more samples per prompt would achieve the same' without this baseline. This must be reported, at minimum on AIME 2025 and ALFWorld,to
  2. §2.2, Eq. (5): The trimmed length-normalized NLL dS is the core selection mechanism, and the paper acknowledges it is 'sensitive to verbosity and surface form' (§5). The trim parameters (α=0.10, β=0.02) are heuristic with no sensitivity analysis. While the reader's concern about dS validity is noted, the more pressing issue is that no ablation over r, α, β, or τ_low is reported. Since these are the free parameters governing which proposals are retained, at least a brief sensitivity analysis (e.g., varying r from 1 to 4, or varying τ_low) would strengthen the claim that the method is robust to these choices rather than tuned to the reported benchmarks.
minor comments (7)
  1. Table 1: The OPD (self-context) row for Qwen3-1.7B on AIME 2025 reports 16.0, which is below the direct GRPO baseline of 17.5. This regression is not discussed; a brief note would help.
  2. Table 3, Pick & Place row: TREK shows −3.0 relative to GRPO. While the paper notes 'little remaining headroom,' a regression on any task type deserves explicit discussion of whether this is within noise or a genuine trade-off.
  3. §3.2: The statement 'OPD trails TREK (self-context) by 1.1–1.6 points on AIME 2024' is slightly imprecise; for Qwen3-1.7B the gap is 1.1 (22.4 vs 21.3), but for Qwen3-14B it is 1.6 (50.2 vs 48.6). The range is correct but the phrasing could clarify these are per-scale ranges.
  4. Figure 1 captions: The y-axis labels in (a) and (b) show 'Accuracy (%)' but the values are avg@16, which is a pass-rate-based metric. Clarifying that avg@16 is the average per-problem pass rate would avoid confusion for readers unfamiliar with the convention.
  5. Appendix C: The failure-lesson memory contains 'roughly forty rules' but only seven are shown. Including the full list, perhaps in a supplementary file, would aid reproducibility of the self-context variant.
  6. §3.1: The learning rate for math (1×10⁻⁷, Table 5) is unusually low for GRPO post-training. A brief justification would help readers replicate the setup.
  7. The paper would benefit from a brief note on computational overhead: how many additional teacher queries and forward-KL steps does TREK add per training round compared to vanilla GRPO?

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee correctly identifies two gaps in our experimental evidence: the missing 2× rollout-budget GRPO baseline results and the absence of sensitivity analysis over the selection hyperparameters. We agree with both points and will address them in the revision. Our point-by-point response follows.

read point-by-point responses

  1. Referee: The 2× rollout-budget GRPO baseline is defined in Appendix B (Table 5: '32 unaided samples per prompt; no teacher proposals') but its results are never reported in any table or figure. This is the single most important control for the paper's central thesis. The paper claims in §1 that 'larger rollout groups or sharper relative advantages may still search within a narrow region of the student's current support,' but this claim is asserted rather than tested. If doubling the rollout budget closes most of the gap on AIME or ALFWorld, the problem is insufficient sampling rather than missing support, and the teacher-proposal mechanism would be unnecessary. The reported gains cannot be distinguished from 'more samples per prompt would achieve the same' without this baseline. This must be reported, at minimum on AIME 2025 and ALFWorld.

    Authors: The referee is correct that this baseline is the most important control for our central thesis, and we acknowledge that defining it in Appendix B without reporting results is a significant omission. We will run the 2× rollout-budget GRPO baseline (32 unaided samples per prompt, no teacher proposals, all other hyperparameters unchanged) on both AIME 2025 (Qwen3-8B) and ALFWorld (Qwen2.5-7B-Instruct) and report the results in the revised manuscript. We will also add it on AIME 2024 if compute permits. We agree that without this comparison, the reader cannot distinguish 'missing support' from 'insufficient sampling.' Our conceptual argument for why doubling rollouts should not close the gap is that the student's failure on hard prompts is not primarily a sampling-efficiency problem but a support-coverage problem: if the policy assigns negligible probability to the correct solution mode, drawing twice as many samples from the same narrow distribution does not systematically expand the set of reachable modes. However, we recognize this is an empirical claim that must be tested rather than asserted, and we will present the data honestly regardless of outcome. If the 2× baseline closes a substantial portion of the gap, we will revise our framing accordingly and discuss what the remaining gap (if any) implies about the boundary between sampling insufficiency and support deficiency. revision: yes

  2. Referee: §2.2, Eq. (5): The trimmed length-normalized NLL dS is the core selection mechanism, and the paper acknowledges it is 'sensitive to verbosity and surface form' (§5). The trim parameters (α=0.10, β=0.02) are heuristic with no sensitivity analysis. No ablation over r, α, β, or τ_low is reported. At least a brief sensitivity analysis (e.g., varying r from 1 to 4, or varying τ_low) would strengthen the claim that the method is robust to these choices rather than tuned to the reported benchmarks.

    Authors: We agree that a sensitivity analysis over the selection hyperparameters is needed and absent from the current manuscript. We will add the following experiments in the revision: (1) varying r ∈ {1, 2, 3, 4} on AIME 2025 (Qwen3-8B) and ALFWorld, holding all other parameters fixed; (2) varying τ_low around the current value of 1/8 (e.g., 1/16, 1/8, 1/4) on the same benchmarks; and (3) varying the trim parameters (α, β) over at least two alternative settings (e.g., no trimming, and symmetric trimming at 0.05/0.05) to test whether the specific trim fractions matter. We will report these in a compact table or figure in the appendix. Our expectation is that the method is most sensitive to r (too small reduces multi-mode coverage; too large risks consolidating trajectories far from the student's support) and relatively insensitive to the exact trim fractions, but we will present the data and let the reader judge. We acknowledge that the current lack of this analysis is a legitimate weakness; the trim parameters were chosen on principled grounds (removing boilerplate tokens at the low end and rare-token outliers at the high end) but were not empirically validated against alternatives, and we will correct this. revision: yes

Circularity Check

0 steps flagged

No circularity found

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs by construction. The central claim—that forward-KL consolidation of verified teacher proposals improves GRPO on hard prompts—is tested against external benchmarks (AIME 2024/2025, ALFWorld, ScienceWorld) with external teacher models (DeepSeek-V4) and a same-model self-context variant. The reachability metric dS (Eq. 5) is used only as a ranking/selection criterion for which teacher trajectories to consolidate; it is not fitted to benchmark outcomes and then presented as a prediction. The trim parameters (α=0.10, β=0.02) are stated as heuristic, not fitted. The forward-KL consolidation objective (Eq. 10–11) is a standard teacher-forced NLL on selected trajectories, justified empirically by the OPD ablation (Tables 1–2), not by a self-citation chain. Self-citations to related OPD work by overlapping authors (Xu et al. 2026a,b; Sang et al. 2026a,b) appear only in the related-work section as complementary methods and are not load-bearing for the central derivation. No uniqueness theorem is invoked. The skeptic's concern about the unreported 2× rollout baseline is a missing-control issue (correctness risk), not circularity. The reader's concern about dS as a reachability proxy is a validity concern, not a circularity concern. No step in the derivation chain is equivalent to its inputs by definition or construction.

Axiom & Free-Parameter Ledger

7 free parameters · 3 axioms · 1 invented entities

The free parameters are all standard hyperparameters for a training recipe, not fitted constants masquerading as predictions. The key ad-hoc axiom is the reachability proxy dS, which is load-bearing but acknowledged as imperfect. The invented entity (failure-lesson memory) is a practical artifact rather than a theoretical postulate.

free parameters (7)
  • τ_low (hardness cutoff) = 1/8
    Threshold for routing prompts to proposal learning; chosen by the authors.
  • K (student rollouts for pass rate estimation) = 16
    Number of unaided rollouts per prompt to estimate pS(x).
  • M (proposal queries) = 4
    Maximum number of proposal rollouts per hard prompt.
  • r (top-r selection) = 2
    Number of verifier-passing proposals retained per hard prompt.
  • α (low-end trim fraction) = 0.10
    Fraction of lowest-loss tokens removed before averaging in dS.
  • β (high-end trim fraction) = 0.02
    Fraction of highest-loss tokens removed before averaging in dS.
  • Proposal-learning window length = 1 epoch
    Duration of the forward-KL warm-start phase before returning to GRPO.
axioms (3)
  • domain assumption GRPO can only reinforce modes that the student already samples (§2.1, Eq. 1-2).
    Standard property of on-policy RL methods; the paper's motivation rests on this being the binding constraint on hard prompts.
  • ad hoc to paper Trimmed length-normalized NLL is a valid proxy for trajectory reachability (§2.2, Eq. 5).
    The paper acknowledges this metric is sensitive to verbosity and surface form (§5), but uses it for selection without sensitivity analysis on the trim parameters.
  • ad hoc to paper Verified teacher trajectories that are student-proximal by dS will become sampleable by the student after a short forward-KL phase (§2.3).
    This is the core mechanism assumption—that the forward-KL update on selected proposals actually expands the student's reachable support in a way that survives into GRPO refinement.
invented entities (1)
  • Failure-lesson memory (self-context variant) no independent evidence
    purpose: Natural-language rules prepended to the prompt at proposal generation time to help the same model discover verified solutions.
    The ~40 rules are hand-curated from rejected rollouts. While the concept is grounded in self-reflection literature, the specific rule set is an invented artifact without independent evidence that this particular memory content generalizes beyond the studied benchmarks.

pith-pipeline@v1.1.0-glm · 21425 in / 2959 out tokens · 348631 ms · 2026-07-07T16:14:00.952199+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · 22 internal anchors

  1. [1]

    Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327,

    work page 2016

  2. [2]

    11 Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes.arXiv preprint arXiv:2306.13649,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation

    Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, and Hongxia Yang. Not all disagreement is learnable: Token teachability in on-policy distillation.arXiv preprint arXiv:2605.26844,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence

    Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, and Zhipeng Wang. Paced: Distillation and on-policy self-distillation at the frontier of student competence.arXiv preprint arXiv:2603.11178, 2026a. Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation.arXiv preprint arXiv:2604.14084,...

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    ReAct: Synergizing Reasoning and Acting in Language Models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2023a. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.arXiv ...

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models

    12 Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.arXiv preprint arXiv:2305.10601, 2023b. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arX...

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Technical report for DeepSeek-V4- Pro. Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang,...

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    ScienceWorld: Is your Agent Smarter than a 5th Grader?

    Ruoyao Wang, Peter Jansen, Marc-Alexandre Cˆot´e, and Prithviraj Ammanabrolu. ScienceWorld: Is your agent smarter than a 5th grader?arXiv preprint arXiv:2203.07540, 2022b. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base m...

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Learn hard problems during RL with reference guided fine-tuning.arXiv preprint arXiv:2603.01223,

    Yangzhen Wu, Shanda Li, Zixin Wen, Xin Zhou, Ameet Talwalkar, Yiming Yang, Wenhao Huang, and Tianle Cai. Learn hard problems during RL with reference guided fine-tuning.arXiv preprint arXiv:2603.01223,

    work page arXiv

  12. [12]

    Privileged Information Distillation for Language Models

    Emiliano Penaloza, Dheeraj Vattikonda, Nicolas Gontier, Alexandre Lacoste, Laurent Charlin, and Massimo Caccia. Privileged information distillation for language models.arXiv preprint arXiv:2602.04942,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

    Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and Sanjeev Arora. Self-distillation zero: Self-revision turns binary rewards into dense supervision.arXiv preprint arXiv:2604.12002,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Unifying group-relative and self-distillation policy optimization via sample routing.arXiv preprint arXiv:2604.02288, 2026a

    Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing.arXiv preprint arXiv:2604.02288, 2026a. 13 Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint a...

    work page arXiv

  15. [15]

    Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe

    Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, and Hehe Fan. Uni-OPD: Unifying on-policy distillation with a dual-perspective recipe. arXiv preprint arXiv:2605.03677,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXiv preprint arXiv:2604.13016, 2026b. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context ...

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Black-box on-policy distillation of large language models.arXiv preprint arXiv:2511.10643,

    Tianzhu Ye, Li Dong, Zewen Chi, Xun Wu, Shaohan Huang, and Furu Wei. Black-box on-policy distillation of large language models.arXiv preprint arXiv:2511.10643,

    work page arXiv

  18. [18]

    Self-Refine: Iterative Refinement with Self-Feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback.arXiv preprint arXiv:2303.17651,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    Let's Verify Step by Step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Large Language Models Can Self-Improve

    Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve.arXiv preprint arXiv:2210.11610,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    Explanations from Large Language Models Make Small Reasoners Better

    Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. Explanations from large language models make small reasoners better.arXiv preprint arXiv:2210.06726,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! Outperforming larger language models with less training data and smaller model sizes. InFindings of the Association for Computational Linguistics: ACL 2023,

    work page 2023

  23. [23]

    Knowledge Distillation with Training Wheels

    Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, and Abhinav Sethy. Knowledge distillation with training wheels.arXiv preprint arXiv:2502.17717,

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    URL https://www.preprints.org/ manuscript/202603.1152/v1

    doi: 10.20944/preprints202603.1152.v1. URL https://www.preprints.org/ manuscript/202603.1152/v1. Binqi Shen, Lier Jin, Hanyu Cai, Lan Hu, and Yuting Xin. The efficiency frontier: A unified framework for cost-performance optimization in LLM context management.arXiv preprint arXiv:2605.23071, 2026b. Yuting Xin, Hanyu Cai, Binqi Shen, Lier Jin, and Lan Hu. D...

    work page doi:10.20944/preprints202603.1152.v1

  25. [25]

    DriftGuard: Safety-Aware Multi-Monitor Detection and Selective Adaptation for Evolving Toxicity Moderation

    URL https://arxiv. org/abs/2606.28725. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled RLVR.arXiv preprint arXiv:2604.03128,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    Reinforcement-aware Knowledge Distillation for LLM Reasoning

    Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, and Stefano Soatto. Reinforcement-aware knowledge distillation for LLM reasoning. arXiv preprint arXiv:2602.22495,

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    14 Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu...

    work page internal anchor Pith review Pith/arXiv arXiv

  28. [28]

    Group Sequence Policy Optimization

    URLhttps://arxiv.org/abs/2507.18071. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework.arXiv preprint arXiv:2409.19256,

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    The MATH-500 ablation breakdown is omitted because the saturated regime provides little discriminative signal among the control conditions. Stage Model MATH-500 Direct GRPO Qwen3-1.7B76.2±1.1 Direct GRPO Qwen3-8B88.3±0.9 Direct GRPO Qwen3-14B89.5±0.8 TREK (DeepSeek-V4) Qwen3-1.7B79.6±1.2 TREK (DeepSeek-V4) Qwen3-8B91.5±1.1 TREK (DeepSeek-V4) Qwen3-14B92.7...

    work page 2024