Pith. sign in

REVIEW 3 major objections 7 minor 99 references

Contrastive entropy gating turns privileged self-distillation into a reliable token-level policy gradient for multi-turn LLM agents.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 20:04 UTC pith:CM2VBZ5K

load-bearing objection Competent agentic post-training recipe: entropy-gated InfoNCE over privileged KL fixes a real OPSD failure mode, with broad empirics and standard math; novelty is compositional, and the ΔE label is still correlational. the 3 major comments →

arxiv 2607.28026 v1 pith:CM2VBZ5K submitted 2026-07-30 cs.LG

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

Xingjian Wu , Junlin Liu , Xingchen Liu , Xuhang Zhu , Jianing Wang , Linsen Guo , Xiaoyu Li , Xuezhi Cao
show 1 more author
Xunliang Cai
This is my paper
classification cs.LG
keywords on-policy self-distillationcontrastive learningagentic RLpredictive entropyexposure biasInfoNCEgroup-relative policy optimizationmulti-turn tool use
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When a language model is trained as both student and privileged self-teacher, the teacher often becomes overconfident after tool calls by copying demonstration routes, so uniform distillation produces unstable rewards and collapsed reasoning. This paper claims the fix is to treat original and privileged contexts as two views of the same encoder and to run a group-wise contrastive loss: positions where the teacher’s predictive entropy falls relative to the student are labeled exposure bias and pushed away, while positions of reflective exploration are pulled closer. The resulting objective is already a soft-gated per-token policy gradient and can stand alone or replace the static KL regularizer inside group-relative RL. Across thirteen math, multi-hop, and deep-search benchmarks the method improves accuracy and keeps entropy and tool-use budgets stable, showing that dense self-distillation becomes useful once positive and negative positions are separated by entropy ranking.

Core claim

Reformulating on-policy self-distillation as an InfoNCE contrast whose positive and negative pairs are chosen by ranking the student–teacher entropy gap inside each rollout group yields reliable fine-grained optimization signals; the resulting gradient is a contrastively reweighted per-token policy gradient that both encourages reflective exploration and repels exposure-biased predictions, outperforming pure RLVR and uniform self-distillation on long-horizon agentic tasks.

What carries the argument

CRPO loss: group-wise InfoNCE on negative KL similarity, with positives defined as the bottom-p% entropy-gap positions (reflective exploration) and negatives as the remainder (exposure bias); its gradient supplies a soft gate ci,t of opposite sign on the two sets.

Load-bearing premise

The signed gap between student and teacher predictive entropy, ranked inside a rollout group, faithfully marks which tokens should be imitated versus avoided; if the gap mostly tracks other factors the contrastive labels become wrong.

What would settle it

Replace the entropy-gap judger with random or reverse labeling of the same positive/negative sets and check whether the accuracy and stability gains on GAIA and WebWalkerQA disappear relative to uniform OPSD and to GRPO.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dense self-distillation can be used alone for agentic post-training once positions are contrastively gated, without needing an external stronger teacher.
  • The same CRPO term can replace the frozen-reference KL inside group-relative RL, giving a dynamic position-aware regularizer that still keeps the outcome-level skeleton.
  • Long-horizon tool-use training becomes more sample-efficient because high-uncertainty post-tool positions receive explicit attract/repel signals rather than a single sparse reward.
  • Pass@k sampling improves because the contrast encourages diverse reflective routes instead of route collapse onto privileged demonstrations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any privileged-teacher scheme whose advantage appears mainly after external observations could reuse the same entropy-gap ranking without redesigning the reward model.
  • If the entropy gap proves noisy on non-search tools, replacing it with another cheap uncertainty proxy would keep the contrastive skeleton intact.
  • The soft-gate interpretation suggests CRPO could be viewed as an online, group-normalized form of advantage-weighted regression at the logit level.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Contrastive Reinforced Policy Optimization (CRPO), a reformulation of on-policy self-distillation (OPSD) for multi-turn agentic LLM post-training. It treats student and privileged self-teacher contexts as two views, uses negative KL as similarity, and partitions token positions into positive (reflective exploration) and negative (exposure bias) pairs via the predictive-entropy gap ΔE = E_S − E_T with group-wise Bottom-p% ranking (Eqs. 5–9). An InfoNCE-style objective (Eq. 11) pulls the student toward the teacher at positive positions and pushes away at negative ones; Proposition 1 shows this is a contrastively reweighted per-token policy gradient with soft gates c_{i,t}. A combined variant CRPO* adds a GRPO outcome term. Experiments on three backbone families across 13 math, multi-hop QA, and deep-search benchmarks (Tables 3–4), plus training dynamics, hyperparameter sweeps, negative-pair and rollout-size ablations (Appendix D), report consistent gains over GRPO, ARPO, OPSD, SDPO, and RLSD.

Significance. If the results hold, CRPO is a practically useful contribution to agentic RL post-training: it upgrades dense OPSD supervision without a separate teacher model, addresses a documented failure mode (route convergence / exposure bias after tool calls), and is orthogonal to trajectory-level RL via CRPO*. Strengths include a clean gradient derivation (Appendix C.1), trust-region/EMA teacher analysis (C.3), top-K KL/entropy bounds (C.4), broad multi-backbone evaluation, and targeted ablations (positive-only, p-sweep, G-sensitivity, entropy-collapse statistics in D.1). The work is significant as an empirical methods paper even if the semantic reading of ΔE remains partly correlational.

major comments (3)
  1. [§3.2 Eqs. (5)–(9); Table 2; Appendix D.1] §3.2 Eqs. (5)–(9) and Table 2: the central mechanism assumes that ranking on ΔE = E_S − E_T faithfully labels “reflective exploration” (pull) versus “exposure bias / route convergence” (push). Appendix D.1 shows teacher entropy collapse is concentrated 10–30 tokens after tool calls and that high-ΔE tokens are search/action vocabulary from demonstrations—supportive but correlational. It does not establish that low-ΔE positions are causally the positions whose distillation improves generalization, nor that high-ΔE positions are harmful privilege copying rather than legitimate confidence after useful context. If ΔE mainly tracks phase, length, format, or tool-syntax certainty, the InfoNCE gate (Eq. 11) and c_{i,t} in Prop. 1 could help via selective reweighting without the stated semantics. Please either (i) add a controlled intervention (e.g., shuffle or invert the ΔE ranking; mask only by
  2. [§4.2 Tables 3–4] §4.2 Tables 3–4: vanilla OPSD collapses to near base-model level on several splits (e.g., Qwen2.5-3B Avg. 26.1, GAIA-Avg. 20.4 on Qwen3-8B), so large gaps versus OPSD/SDPO partly reflect a weak self-distillation baseline rather than only the contrastive fix. The more informative comparisons are versus ARPO/GRPO/RLSD, where gains are real but smaller and uneven across tasks. Please report, for the main tables, absolute and relative gains versus the strongest non-CRPO baseline per backbone, and discuss failure cases or tasks where CRPO does not lead (several individual cells in Table 3 are second-best or tied). This keeps the headline claim proportionate to the hardest baselines.
  3. [§3.3 Eq. (11); Proposition 1; §B.2] §3.3 Eq. (11) and Prop. 1: the contrastive denominator pools all positions across the G-rollout group. With default G=8 and long responses, |P∪N| is large and dominated by many weakly related tokens; wall and wP then act as a soft, group-normalized reweighting rather than a classical instance-discrimination InfoNCE. The paper should clarify this interpretation (reweighting vs. true multi-negative contrast) and, if feasible, ablate alternative normalizers (within-trajectory only; detach negatives from the denominator; temperature sensitivity beyond the fixed τ=1.0 in B.2). Without that, the “contrastive learning perspective” risks overstating the link to Chen/He-style InfoNCE while the working ingredient may be signed selective KL.
minor comments (7)
  1. [Figure 1; Appendix D.1] Figure 1 and Appendix D.1: the single-trajectory entropy plot is illustrative; point readers earlier (main text) to the full statistical validation so the intro claim is not left anecdotal.
  2. [§1; Figure 4] Typos and wording: “volunerable”, “augumented”, “encouges”, “heurisitc”, “consative”, “Constrative” in figure caption paths; standardize “OPSD” vs “On-Policy Distillation (OPD)” early on.
  3. [§3.1 Table 1] Table 1 self-teacher prompt: specify how reference trajectories are chosen when multiple successes exist, and what happens when the group has no successful rollout (common early in training).
  4. [§4.1] §4.1 Evaluation: LLM-as-Judge with Qwen2.5-72B for deep-search should report judge prompt, agreement checks, or sensitivity; pass@1 decoding details are given but judge reliability is not.
  5. [§3; Appendix C.3; §B.2] Appendix C.3 introduces trust-region/EMA teacher (α=0.1) used in main experiments; the main methodology should briefly state that the self-teacher is EMA-stabilized so readers are not surprised in B.2.
  6. [Figure 7] Figure 7: mark default (p=30, λ=5) and report variance over seeds if available; single-run sweeps are hard to read for small gaps on HLE.
  7. [§2.2] Related work: RLCSD and HDPO are cited; a short explicit differentiation table (privilege type, positive/negative definition, agentic vs single-turn) would help position CRPO against concurrent contrastive self-distillation work.

Circularity Check

0 steps flagged

No significant circularity: empirical methods paper with algebraic gradient characterization and external benchmark evaluation.

full rationale

CRPO defines an InfoNCE-style contrastive self-distillation loss gated by ranked entropy gaps (Eqs. 5–11), then algebraically rewrites its gradient as a soft-gated per-token policy gradient (Prop. 1 / Appendix C.1). That rewrite is a derivation of properties of the authors’ own objective, not a claim that an independent physical or statistical law is predicted from first principles. Held-out scores on AIME, MATH, GAIA, HLE, WebWalkerQA, xbench, etc. are external to the training loss; hyperparameter sweeps on p and λ are ordinary tuning, not fitted inputs relabeled as predictions. Privileged teacher context built from same-group rollouts is a design choice of OPSD/CRPO, not a tautology that forces the reported benchmark gains. Citations to GRPO, OPSD, InfoNCE, and trust-region RL are standard prior art, not load-bearing self-citation uniqueness theorems. No step reduces a claimed prediction to its inputs by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

Load-bearing content is algorithmic and empirical. The claim rests on standard RL/contrastive math, the domain assumption that privileged OPSD is the right dense signal, and the paper-specific axiom that entropy gaps label exposure bias. Free knobs (p, λ, τ, G, top-K, EMA α) are chosen by sweep/default and affect reported wins. No new physical entities; invented constructs are methodological (judger, positive/negative position sets, CRPO loss).

free parameters (6)
  • positive-pair proportion p = 30%
    Fraction of lowest-ΔE positions treated as positives; main results use p=30%, and Fig. 7 shows large sensitivity (p→100 collapses to OPSD).
  • contrastive weight λ in CRPO* = 5
    Balances GRPO outcome loss vs L_CRPO; default λ=5 from deep-search sweep; λ=0 degenerates to GRPO.
  • contrastive temperature τ = 1.0
    Softmax temperature in InfoNCE; fixed rather than derived.
  • global rollout size G = 8 (deep-search default; 16 listed in some RL configs)
    Group size for ranking ΔE and GRPO-style grouping; performance saturates near G=8 in D.3.
  • top-K vocabulary truncation K = 100
    Approximates KL/entropy over top-K student tokens (C.4); biases estimators but enables computation.
  • EMA teacher coefficient α = 0.1
    Trust-region surrogate mixing for self-teacher parameters in CRPO* experiments.
axioms (5)
  • domain assumption Reverse KL from student to stop-gradient privileged teacher is a valid dense supervision / similarity signal for on-policy self-distillation.
    Inherited from OPSD/SDPO (§3.1–3.2, Eq. 4) and used as sim = −KL throughout.
  • ad hoc to paper Predictive entropy difference ΔE ranks positions into reflective exploration vs exposure bias well enough for Bottom-p% selection.
    Core judger design (§3.2 Eqs. 5–9, Table 2); supported by D.1 statistics but not proven causal.
  • domain assumption Group-relative ranking across G rollouts yields clearer optimization directions than absolute thresholds alone.
    Borrowed from GRPO-style group normalization and stated as design pivot (§1, §3.2).
  • standard math Standard policy-gradient / InfoNCE calculus (score-function identity, log-sum-exp gradients) applies to token-level LLM policies.
    Used in Appendix C.1 proof of Proposition 1.
  • domain assumption Rule-based or LLM-as-judge verifiers plus tool environments provide rewards/feedback adequate to rank methods.
    Evaluation protocol §4.1; deep-search uses Qwen2.5-72B judge and Bing/browser stack.
invented entities (2)
  • Entropy-gap judger (P/N position partition) independent evidence
    purpose: Label tokens as positive (distill) or negative (repel) for contrastive OPSD.
    New control module relative to uniform OPSD; defined by Bottom-p% on ΔE within the rollout group.
  • CRPO / CRPO* objectives independent evidence
    purpose: Standalone contrastive self-distillation loss and optional GRPO + λ L_CRPO hybrid.
    Primary algorithmic contribution (§3.3 Eqs. 11–13); evidence is benchmark performance, not an external physical prediction.

pith-pipeline@v1.2.0-daily-grok45 · 33019 in / 3946 out tokens · 86675 ms · 2026-07-31T20:04:01.436756+00:00 · methodology

0 comments
read the original abstract

Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level supervision, it inherently suffers from exposure bias due to the privileged information of the self-teacher. In multi-turn agentic settings, this leads to reasoning route convergence and the loss of clear optimization directions. To tackle these challenges, we introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates agentic OPSD from a contrastive learning perspective. By leveraging predictive entropy to distinguish between positive positions (reflective exploration) and negative positions (exposure bias), CRPO conducts group-wise contrast to preserve reliable, fine-grained optimization signals. Extensive evaluations across 13 challenging reasoning and deep-search benchmarks demonstrate that CRPO consistently outperforms existing reinforcement learning and self-distillation baselines, significantly enhancing training stability and generalization in long-horizon interactions.

Figures

Figures reproduced from arXiv: 2607.28026 by Jianing Wang, Junlin Liu, Linsen Guo, Xiaoyu Li, Xingchen Liu, Xingjian Wu, Xuezhi Cao, Xuhang Zhu, Xunliang Cai.

Figure 1
Figure 1. Figure 1: Position-level entropy along a student trajectory [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The Reward and KL-Divergence (token-average) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The algorithm pipeline of the proposed Contrastive Reinforced Policy Optimization (CRPO). [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The CRPO pipeline, understanding from a con [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Pass@1, Pass@3, and Pass@5 of CRPO∗ on Qwen3-8B and Qwen3-14B across four deep-search benchmarks (GAIA, HLE, WebWalkerQA, xbench). 4.4 Training Dynamics [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Hyper-parameter sweeps of CRPO∗ on Qwen3- 8B across the four deep-search benchmarks. Left: positive￾pair proportion p; p = 100 degenerates to OPSD. Right: contrastive weight λ; λ= 0 degenerates to GRPO. the contrastive objective, and the regularization weight λ in LCRPO∗ = LGRPO+λLCRPO [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 6
Figure 6. Figure 6: reports four metrics during the post-training. We observe that CRPO and CRPO∗ keep both entropy and KL within a narrow band that closely tracks ARPO and stays well below GRPO, due to their selective distillation at key posi￾tions, while their accuracy curves consistently dominate all baselines training; in contrast, OPSD exhibits a clear entropy collapse between steps 60 and 100, where the student model su… view at source ↗
Figure 9
Figure 9. Figure 9: Word clouds of frequent tokens at (a) high- [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Statistical analysis of teacher entropy collapse after [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: CRPO accuracy (average over four deep-search [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

99 extracted references · 38 linked inside Pith

  1. [2]

    Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597--1607. PmLR

    2020

  2. [3]

    DeepSeek-AI. 2026. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948

    Pith/arXiv arXiv 2026

  3. [4]

    Ding, K. 2026. HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation. arXiv:2603.23871

    arXiv 2026

  4. [5]

    Dong, G.; Bao, L.; Wang, Z.; Zhao, K.; Li, X.; Jin, J.; Yang, J.; Mao, H.; Zhang, F.; Gai, K.; Zhou, G.; Zhu, Y.; Wen, J.-R.; and Dou, Z. 2025 a . Agentic Entropy-Balanced Policy Optimization. arXiv:2510.14545

    arXiv 2025

  5. [7]

    Dong, G.; Chen, Y.; Li, X.; Jin, J.; Qian, H.; Zhu, Y.; Mao, H.; Zhou, G.; Dou, Z.; and Wen, J.-R. 2025 c . Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning. arXiv:2505.16410

    Pith/arXiv arXiv 2025

  6. [8]

    Dong, G.; Mao, H.; Ma, K.; Bao, L.; Chen, Y.; Wang, Z.; Chen, Z.; Du, J.; Wang, H.; Zhang, F.; Zhou, G.; Zhu, Y.; Wen, J.-R.; and Dou, Z. 2025 d . Agentic Reinforced Policy Optimization. arXiv:2507.19849

    Pith/arXiv arXiv 2025

  7. [10]

    Fang, J.; Hong, Z.; Zheng, M.; Song, M.; Li, G.; Jiang, H.; Zhang, D.; Guo, H.; Wang, X.; and Chua, T.-S. 2026. Rubric-based On-policy Distillation. arXiv:2605.07396

    Pith/arXiv arXiv 2026

  8. [11]

    He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9729--9738

    2020

  9. [12]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Vanschoren, J.; and Yeung, S., eds., Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual

    2021

  10. [13]

    D.; Sugawara, S.; and Aizawa, A

    Ho, X.; Nguyen, A. D.; Sugawara, S.; and Aizawa, A. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. In Scott, D.; Bel, N.; and Zong, C., eds., Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020 , 6609--6625. International Comm...

    2020

  11. [14]

    Hou, W.; Peng, S.; Wang, W.; Ruan, Z.; Zhang, Y.; Zhou, Z.; Gao, M.; Chen, Y.; Wang, K.; Yang, H.; Zhang, C.; Tian, Z.; Hu, H.; Yang, Y.; Wu, F.; and Fan, H. 2026. Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe. arXiv:2605.03677

    Pith/arXiv arXiv 2026

  12. [15]

    K.; Guestrin, C.; and Krause, A

    Hübotter, J.; Lübeck, F.; Behric, L.; Baumann, A.; Bagatella, M.; Marta, D.; Hakimi, I.; Shenfeld, I.; Buening, T. K.; Guestrin, C.; and Krause, A. 2026. Reinforcement Learning via Self-Distillation. arXiv:2601.20802

    Pith/arXiv arXiv 2026

  13. [16]

    Jin, B.; Zeng, H.; Yue, Z.; Yoon, J.; Arik, S.; Wang, D.; Zamani, H.; and Han, J. 2025. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv:2503.09516

    Pith/arXiv arXiv 2025

  14. [17]

    Lambert, N.; Morrison, J.; Pyatkin, V.; Huang, S.; Ivison, H.; Brahman, F.; Miranda, L. J. V.; Liu, A.; Dziri, N.; Lyu, S.; Gu, Y.; Malik, S.; Graf, V.; Hwang, J. D.; Yang, J.; Bras, R. L.; Tafjord, O.; Wilhelm, C.; Soldaini, L.; Smith, N. A.; Wang, Y.; Dasigi, P.; and Hajishirzi, H. 2025. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. ar...

    Pith/arXiv arXiv 2025

  15. [18]

    Li, C.; Tang, Z.; Li, Z.; Xue, M.; Bao, K.; Ding, T.; Sun, R.; Wang, B.; Wang, X.; Lin, J.; and Liu, D. 2025 a . CoRT: Code-integrated Reasoning within Thinking. CoRR, abs/2506.09820

    Pith/arXiv arXiv 2025

  16. [19]

    Li, K.; Zhang, Z.; Yin, H.; Zhang, L.; Ou, L.; Wu, J.; Yin, W.; Li, B.; Tao, Z.; Wang, X.; Shen, W.; Zhang, J.; Zhang, D.; Wu, X.; Jiang, Y.; Yan, M.; Xie, P.; Huang, F.; and Zhou, J. 2025 b . WebSailor: Navigating Super-human Reasoning for Web Agent. arXiv:2507.02592

    Pith/arXiv arXiv 2025

  17. [20]

    Li, X.; Dong, G.; Jin, J.; Zhang, Y.; Zhou, Y.; Zhu, Y.; Zhang, P.; and Dou, Z. 2025 c . Search-o1: Agentic search-enhanced large reasoning models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 5420--5438

    2025

  18. [21]

    Li, X.; Jin, J.; Dong, G.; Qian, H.; Wu, Y.; Wen, J.-R.; Zhu, Y.; and Dou, Z. 2026 a . Webthinker: Empowering large reasoning models with deep research capability. Advances in Neural Information Processing Systems, 38: 120091--120131

    2026

  19. [22]

    Li, Y.; Zuo, Y.; He, B.; Zhang, J.; Xiao, C.; Qian, C.; Yu, T.; ang Gao, H.; Yang, W.; Liu, Z.; and Ding, N. 2026 b . Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016

    Pith/arXiv arXiv 2026

  20. [23]

    Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; and Cobbe, K. 2024. Let's Verify Step by Step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  21. [24]

    Lu, Z.; Yao, Z.; Han, Z.; Wang, Z.-H.; Wu, J.; Gu, Q.; Cai, X.; Lu, W.; Xiao, J.; Zhuang, Y.; and Shen, Y. 2026. Self-Distilled Agentic Reinforcement Learning. arXiv:2605.15155

    Pith/arXiv arXiv 2026

  22. [25]

    MacKay, D. J. 1992. A practical Bayesian framework for backpropagation networks. Neural computation, 4(3): 448--472

    1992

  23. [26]

    Mialon, G.; Fourrier, C.; Wolf, T.; LeCun, Y.; and Scialom, T. 2024. GAIA: a benchmark for General AI Assistants. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  24. [27]

    Pan, L.; Tao, S.; Zhai, Y.; Zhang, L.; Liu, Z.; Ding, B.; Liu, A.; and Wen, L. 2026. RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation. arXiv:2606.11709

    Pith/arXiv arXiv 2026

  25. [28]

    B.; Kumar, A.; Zhang, G.; and Levine, S

    Peng, X. B.; Kumar, A.; Zhang, G.; and Levine, S. 2019. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv:1910.00177

    Pith/arXiv arXiv 2019

  26. [29]

    Phan, L.; Gatti, A.; Han, Z.; and Nathaniel Li, e. a. 2025. Humanity's Last Exam. CoRR, abs/2501.14249

    Pith/arXiv arXiv 2025

  27. [30]

    A.; and Lewis, M

    Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N. A.; and Lewis, M. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , 5687--5711. Association for Computational Linguistics

    2023

  28. [31]

    C.; He, Q.; Wang, H.; Chen, X.; Hakkani-Tür, D.; Tur, G.; and Ji, H

    Qian, C.; Acikgoz, E. C.; He, Q.; Wang, H.; Chen, X.; Hakkani-Tür, D.; Tur, G.; and Ji, H. 2025. ToolRL: Reward is All Tool Learning Needs. arXiv:2504.13958

    Pith/arXiv arXiv 2025

  29. [32]

    Qiao, R.; Tan, Q.; Dong, G.; Wu, M.; Sun, C.; Song, X.; GongQue, Z.; Lei, S.; Wei, Z.; Zhang, M.; Qiao, R.; Zhang, Y.; Zong, X.; Xu, Y.; Diao, M.; Bao, Z.; Li, C.; and Zhang, H. 2024. We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? arXiv:2407.01284

    Pith/arXiv arXiv 2024

  30. [33]

    Qwen; :; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin, R.; Li, T.; Xia, T.; Ren, X.; Ren, X.; Fan, Y.; Su, Y.; Zhang, Y.; Wan, Y.; Liu...

    Pith/arXiv arXiv 2024

  31. [34]

    I.; and Abbeel, P

    Schulman, J.; Levine, S.; Moritz, P.; Jordan, M. I.; and Abbeel, P. 2017. Trust Region Policy Optimization. arXiv:1502.05477

    Pith/arXiv arXiv 2017

  32. [35]

    K.; Wu, Y.; and Guo, D

    Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300

    Pith/arXiv arXiv 2024

  33. [36]

    X.; Fang, L.; and Wen, J

    Song, H.; Jiang, J.; Min, Y.; Chen, J.; Chen, Z.; Zhao, W. X.; Fang, L.; and Wen, J. 2025. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. CoRR, abs/2503.05592

    Pith/arXiv arXiv 2025

  34. [37]

    Song, M.; and Zheng, M. 2026. A Survey of On-Policy Distillation for Large Language Models. arXiv:2604.00626

    Pith/arXiv arXiv 2026

  35. [38]

    X.; Liu, Z.; Fang, L.; Wang, Z.; and Wen, J.-R

    Sun, S.; Song, H.; Wang, Y.; Ren, R.; Jiang, J.; Zhang, J.; Bai, F.; Deng, J.; Zhao, W. X.; Liu, Z.; Fang, L.; Wang, Z.; and Wen, J.-R. 2025. SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesis

    2025

  36. [39]

    Team, M. 2025 a . MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585

    Pith/arXiv arXiv 2025

  37. [40]

    Team, Q. 2025 b . Qwen3 Technical Report. CoRR, abs/2505.09388

    Pith/arXiv arXiv 2025

  38. [41]

    Team, Q. 2025 c . Qwq-32b: Embracing the power of reinforcement learning

    2025

  39. [42]

    Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2022. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10: 539--554

    2022

  40. [43]

    Wang, H.; Qian, C.; Zhong, W.; Chen, X.; Qiu, J.; Huang, S.; Jin, B.; Wang, M.; Wong, K.-F.; and Ji, H. 2025. Acting Less is Reasoning More! Teaching Model to Act Efficiently. arXiv:2504.14870

    Pith/arXiv arXiv 2025

  41. [44]

    Wang, H.; Wang, G.; Xiao, H.; Zhou, Y.; Pan, Y.; Wang, J.; Xu, K.; Wen, Y.; Ruan, X.; Chen, X.; and Qi, H. 2026 a . Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents. arXiv:2604.10674

    Pith/arXiv arXiv 2026

  42. [45]

    Wang, J.; Zhang, W.; Shi, W.; Li, Y.; and Cheng, J. 2026 b . TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents. arXiv:2604.24005

    Pith/arXiv arXiv 2026

  43. [46]

    Wu, J.; Yin, W.; Jiang, Y.; Wang, Z.; Xi, Z.; Fang, R.; Zhang, L.; He, Y.; Zhou, D.; Xie, P.; and Huang, F. 2025. WebWalker: Benchmarking LLMs in Web Traversal. CoRR, abs/2501.07572

    Pith/arXiv arXiv 2025

  44. [47]

    Yang, C.; Qin, C.; Si, Q.; Chen, M.; Gu, N.; Yao, D.; Lin, Z.; Wang, W.; Wang, J.; and Duan, N. 2026. Self-Distilled RLVR. arXiv:2604.03128

    Pith/arXiv arXiv 2026

  45. [48]

    Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W.; Salakhutdinov, R.; and Manning, C. D. 2018. H otpot QA : A Dataset for Diverse, Explainable Multi-hop Question Answering. In EMNLP, 2369--2380. Brussels, Belgium: Association for Computational Linguistics

    2018

  46. [49]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; Zuo, X.; Yue, Y.; Dai, W.; Fan, T.; Liu, G.; Liu, L.; Liu, X.; Lin, H.; Lin, Z.; Ma, B.; Sheng, G.; Tong, Y.; Zhang, C.; Zhang, M.; Zhang, W.; Zhu, H.; Zhu, J.; Chen, J.; Chen, J.; Wang, C.; Yu, H.; Song, Y.; Wei, X.; Zhou, H.; Liu, J.; Ma, W.-Y.; Zhang, Y.-Q.; Yan, L.; Qiao, M.; Wu, Y.; and Wang, M. 2025. DAPO: An Op...

    Pith/arXiv arXiv 2025

  47. [50]

    Zhao, S.; Xie, Z.; Liu, M.; Huang, J.; Pang, G.; Chen, F.; and Grover, A. 2026. Self-distilled reasoner: On-policy self-distillation for large language models

    2026

  48. [51]

    Zheng, Y.; Zhang, R.; Zhang, J.; Ye, Y.; Luo, Z.; Feng, Z.; and Ma, Y. 2024. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Bangkok, Thailand: Association for Computational Linguistics

    2024

  49. [52]

    2026 , eprint=

    RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation , author=. 2026 , eprint=

    2026

  50. [53]

    2019 , eprint=

    Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning , author=. 2019 , eprint=

    2019

  51. [54]

    2017 , eprint=

    Trust Region Policy Optimization , author=. 2017 , eprint=

    2017

  52. [55]

    2026 , eprint=

    A Survey of On-Policy Distillation for Large Language Models , author=. 2026 , eprint=

    2026

  53. [56]

    Neural computation , volume=

    A practical Bayesian framework for backpropagation networks , author=. Neural computation , volume=. 1992 , publisher=

    1992

  54. [57]

    2026 , eprint=

    Self-Distilled Agentic Reinforcement Learning , author=. 2026 , eprint=

    2026

  55. [58]

    2026 , eprint=

    Self-Distilled RLVR , author=. 2026 , eprint=

    2026

  56. [59]

    2026 , eprint=

    HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation , author=. 2026 , eprint=

    2026

  57. [60]

    2026 , eprint=

    Reinforcement Learning via Self-Distillation , author=. 2026 , eprint=

    2026

  58. [61]

    2025 , eprint=

    Agentic Reinforced Policy Optimization , author=. 2025 , eprint=

    2025

  59. [62]

    2025 , eprint=

    Agentic Entropy-Balanced Policy Optimization , author=. 2025 , eprint=

    2025

  60. [63]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Search-o1: Agentic search-enhanced large reasoning models , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  61. [64]

    Advances in Neural Information Processing Systems , volume=

    Webthinker: Empowering large reasoning models with deep research capability , author=. Advances in Neural Information Processing Systems , volume=

  62. [65]

    2026 , eprint=

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe , author=. 2026 , eprint=

    2026

  63. [66]

    Self-distilled reasoner: On-policy self-distillation for large language models , author=

  64. [67]

    International conference on machine learning , pages=

    A simple framework for contrastive learning of visual representations , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  65. [68]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Momentum contrast for unsupervised visual representation learning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  66. [69]

    2026 , eprint=

    Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents , author=. 2026 , eprint=

    2026

  67. [70]

    2025 , eprint=

    Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning , author=. 2025 , eprint=

    2025

  68. [71]

    2026 , eprint=

    TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents , author=. 2026 , eprint=

    2026

  69. [72]

    2025 , eprint=

    Acting Less is Reasoning More! Teaching Model to Act Efficiently , author=. 2025 , eprint=

    2025

  70. [73]

    2025 , publisher=

    Qwq-32b: Embracing the power of reinforcement learning , author=. 2025 , publisher=

    2025

  71. [74]

    2024 , eprint=

    We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? , author=. 2024 , eprint=

    2024

  72. [75]

    2024 , eprint=

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models , author=. 2024 , eprint=

    2024

  73. [76]

    2026 , eprint=

    Rubric-based On-policy Distillation , author=. 2026 , eprint=

    2026

  74. [77]

    2026 , eprint=

    Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe , author=. 2026 , eprint=

    2026

  75. [78]

    2025 , eprint=

    MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention , author=. 2025 , eprint=

    2025

  76. [79]

    2025 , eprint=

    Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning , author=. 2025 , eprint=

    2025

  77. [80]

    2025 , eprint=

    ToolRL: Reward is All Tool Learning Needs , author=. 2025 , eprint=

    2025

  78. [81]

    2025 , eprint=

    Tulu 3: Pushing Frontiers in Open Language Model Post-Training , author=. 2025 , eprint=

    2025

  79. [82]

    2026 , eprint=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2026 , eprint=. doi:https://doi.org/10.1038/s41586-025-09422-z , url=

    doi:10.1038/s41586-025-09422-z 2026

  80. [83]

    2025 , eprint=

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale , author=. 2025 , eprint=

    2025

Showing first 80 references.