Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

The paper claims that an annealed blend of on-policy distillation and reinforcement learning, gated by turn-level disagreement-uncertainty weights, lets small language-model agents surpass their teachers on multi-turn interactive tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:52 UTC pith:L7ZR4SLJ

load-bearing objection Nice hybrid method, but single-run validation maxima make the 'consistently outperforms' claim premature; referee-worthy after more seeds and a code release. the 3 major comments →

arxiv 2606.27814 v4 pith:L7ZR4SLJ submitted 2026-06-26 cs.AI

ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks

Qitai Tan , Zefang Zong , Mo Li , Yipeng Shi , Yang Li , Peng Chen This is my paper
classification cs.AI
keywords on-policy distillationreinforcement learninglanguage agentsmulti-turn agentic tasksGRPOannealing scheduleturn-level reweightingteacher-student distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ATOD (Annealed Turn-Aware On-Policy Distillation) claims that the two dominant post-training signals for small language-model agents—dense teacher imitation and reward-driven reinforcement learning—should be combined in one schedule, not treated as alternatives. The paper argues that on-policy distillation gives a fast boot but plateaus at the teacher, while sparse-reward RL climbs slowly but reaches higher; annealing the OPD weight down and the RL weight up exploits both. The turn-level weighting T-DUR then concentrates the teacher signal on high-utility decision turns, preventing supervision from being wasted on routine steps. Across ALFWorld, WebShop, and Search-QA, with 0.6B, 1.7B, and 4B students, the paper reports that ATOD beats every baseline and on average surpasses the teacher models, with the largest gains on the smallest students. If true, this is a practical recipe for deploying capable agents at a fraction of the teacher's inference cost.

Core claim

On its own terms, the paper's central discovery is that a student trained on a single hybrid token-level advantage A_t = κ(s) A_OPD_t + ρ(s) A_GRPO_t can first imitate a teacher quickly and then move beyond it. The annealing schedule reduces κ(s) from 1.0 to 0.1 and raises ρ(s) from 1.0 to 2.0 over 80 training steps, so early updates are mostly teacher-guided and later updates are mostly reward-guided. T-DUR assigns each turn a soft weight w_k = 1 − (1 − d̃_k)(1 − h̃_k), where d̃_k and h̃_k are per-trajectory normalized teacher–student disagreement and student uncertainty; this reweights only the OPD term. The reported outcome is that across three benchmarks and three student sizes ATOD impr

What carries the argument

The load-bearing object is the hybrid advantage A_t = κ(s) Δlog p_t w_{k(t)} + ρ(s) A_GRPO_t, inserted into the standard clipped GRPO surrogate. κ and ρ follow a linear anneal over 80 steps (κ: 1.0→0.1, ρ: 1.0→2.0), which makes the update teacher-dominated early and reward-dominated late. T-DUR is the soft turn-gating function w_k = 1 − (1 − d̃_k)(1 − h̃_k), built from per-trajectory min-max normalized disagreement and entropy proxies computed only from sampled-token log-probabilities; it decides where within a long trajectory the teacher supervision is applied without touching the reward signal.

Load-bearing premise

The load-bearing premise is that the hand-set annealing schedule (κ decaying 1.0→0.1, ρ growing 1.0→2.0 over 80 steps) and T-DUR's per-trajectory normalization are robust across tasks rather than tuned to ALFWorld/WebShop/Search-QA, and that single-run maximum validation numbers are representative enough to support 'consistently outperforms'.

What would settle it

Run the same 150-step protocol with 5–10 random seeds for ATOD, OPD, SOD, and TCOD on the three benchmarks and compare distributions of peak validation success. If ATOD's mean is within one standard deviation of OPD or TCOD on Search-QA, or if a fixed-coefficient OPD/RL mixture beats the annealed schedule on ALFWorld for a 0.6B student, the paper's consistency claim would be contradicted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A small student model can exceed its teacher on interactive benchmarks, not merely converge to it, when imitation is followed by reward-driven refinement.
  • Removing the annealed schedule is the most costly ablation—on ALFWorld the 0.6B student drops from 82.8% to 75.8%—so the transition from teacher guidance to reward optimization is doing the main work.
  • Turn-level reweighting beats both token-level reweighting and uniform turn weights, meaning the turn is a stable, semantically meaningful unit for allocating dense supervision in agentic trajectories.
  • The final κ floor (0.1) keeps a weak teacher anchor that is claimed to reduce severe drift and reward hacking while still allowing beyond-teacher improvement.
  • Training dynamics show ATOD combines the fast bootstrap of OPD with continued late-stage gains, and learns shorter trajectories than GRPO on ALFWorld.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the annealing recipe is not specific to distillation; any pairing of a dense-early signal with a sparse-reward signal could be scheduled the same way, and the paper does not test that generalization.
  • Editorial inference: because the Search-QA margin over the next-best baseline is about 0.2 points and the reported numbers are single-run maximums, repeated-seed variance estimates are the natural next check before treating 'consistently outperforms' as settled.
  • Editorial inference: T-DUR's per-trajectory normalization should make it portable to new environments without per-task scale tuning, but portability is only demonstrated here on three benchmarks.
  • Editorial inference: the beyond-teacher effect appears concentrated on tasks where reward can correct teacher mistakes (ALFWorld, WebShop); on knowledge-heavy Search-QA the student stays within about 1 point of the teacher, so a harder reasoning benchmark would test whether the schedule generalizes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ATOD (Annealed Turn-aware On-policy Distillation), a post-training algorithm for small language-model agents on multi-turn interactive tasks. ATOD combines on-policy distillation (OPD) with GRPO-style RL in a single token-level advantage, annealing the OPD coefficient down and the RL coefficient up over training. It also introduces T-DUR, a turn-level weighting scheme based on student-teacher disagreement and student uncertainty, applied only to the OPD term. The method is evaluated on ALFWorld, WebShop, and Search-QA with Qwen3-0.6B/1.7B/4B students, reporting success rates and trajectory lengths against GRPO, SDAR, OPD, SOD, and TCOD. The central claims are that ATOD consistently outperforms all baselines and surpasses the corresponding teacher models on average, with ablations showing both annealing and T-DUR contribute on ALFWorld.

Significance. If the empirical claims hold, ATOD is a useful and simple contribution: the annealed OPD-RL schedule directly addresses the cold-start/ceiling tradeoff between imitation and reward optimization, and the turn-level T-DUR weighting is a plausible mechanism for focusing dense supervision on high-utility decisions. The paper is clearly written, the algorithm is fully specified in Algorithm 1, and the ALFWorld ablations (Fig. 4) show that both proposed components help. The diagnostic plots in Figs. 5-6 and the appendix case studies are consistent with the intended training dynamics. However, the load-bearing empirical claim of 'consistently outperforms' is currently supported only by single-run maximum-validation numbers with no variance estimates, and there are several internal inconsistencies that call for verification before the results can be accepted.

major comments (3)
  1. [§5.1, Table 1, §5.2 Obs 1] All results are maximum validation success rates from single training runs, with no standard deviations, seeds, or significance tests. On Search-QA, ATOD's margins over the strongest baseline TCOD are only 0.29, 0.19, and 0.73 percentage points for the 0.6B, 1.7B, and 4B students, respectively. These margins are smaller than typical seed-to-seed variation in on-policy RL, and selecting the maximum over the 150-step curve inflates expected performance, possibly differently across methods. Thus the 'consistently outperforms all baselines' claim is not empirically established. Please report multiple seeds with mean±std, or at least a rerun with different seeds for the small-margin cases.
  2. [Table 1, Qwen3-4B block] In the Qwen3-4B student block, the 'GRPO†' row (76.56/18.95/48.19/2.51/82.03/6.21/68.93) is numerically identical to the 'Qwen3-4B GRPO (Teacher)' reference row, even though the teacher for this block is Qwen3-30B-A3B GRPO. This appears to be a copy-paste error or an unintended reuse of the teacher as the GRPO baseline. It directly affects the aggregate margin over GRPO (23.62 points) and the 4B-student comparison. Please correct the table and re-verify the underlying experimental records.
  3. [§5.2 Obs 1] The text states that ATOD 'achieves competitive Search-QA performance within 1% of the teacher.' This is contradicted by Table 1: for the 0.6B, 1.7B, and 4B students, the gaps to the relevant teacher are 5.86, 2.98, and 1.37 percentage points, respectively. The claim is also internally inconsistent with the same paragraph's statement that ATOD surpasses the teacher only on ALFWorld and WebShop. Please correct this factual misstatement.
minor comments (6)
  1. [Abstract] The abstract in the full text reports a 3.03-point improvement over OPD, while §5.2 and Table 1 support 4.16 points (the value in the separate abstract block). This inconsistency should be fixed.
  2. [§4.3, Appendix A.2] The OPD advantage in Eq. (9) contains log πθ(a_t|s_t), which depends on the parameters being optimized. Appendix A.2 states that Δlog p_t is 'treated as detached,' but this stop-gradient is not stated in the main text or Algorithm 1. Please clarify whether the implementation detaches the advantage; otherwise the gradient derivation differs from the implemented objective.
  3. [§4.2, Table 2] The method text uses κ(s) and ρ(s), while the hyperparameter table uses β_kl and β_rl for the same quantities. Unify the notation for reproducibility.
  4. [§5.1, Figure 2a] Figure 2a appears to be a schematic illustration, but the text refers to 'validation dynamics' as if they were measured data. Please label it as illustrative or provide the actual curves.
  5. [§5.3, Figure 4] The ALFWorld ablations are informative, but they are also single-run maximum values. Reporting variance would strengthen the claim that both T-DUR and annealing are beneficial across model sizes.
  6. [§4.2, Annealing robustness] The annealing schedule (κ: 1.0→0.1, ρ: 1.0→2.0, T=80) is fixed across tasks, but no sensitivity analysis is provided. If the schedule is robust, a small perturbation study would support the 'consistently' claim; if it is tuned per task, this should be disclosed.

Circularity Check

0 steps flagged

No circularity: ATOD's central claim is an empirical benchmark result, not a derivation that reduces to its own inputs.

full rationale

ATOD's central claim—that it 'consistently outperforms all baselines and even surpasses the corresponding teacher models'—is an empirical outcome measured on ALFWorld, WebShop, and Search-QA, not a quantity obtained by fitting parameters and then renaming the fit as a prediction. The hybrid advantage in Eq. (8) combines the OPD signal (Eq. 9) and the GRPO advantage (Eq. 2), with annealing coefficients (Eqs. 11–12) and T-DUR weights (Eqs. 13–16) computed from on-the-fly student/teacher log-probabilities and environment rewards. None of these quantities is fitted to the reported success rates, and no equation forces ATOD to outperform OPD, GRPO, or the teacher. Indeed, OPD and GRPO are limiting cases of the hybrid objective (ρ=0 and κ=0 respectively), and the annealed combination plus T-DUR could in principle reduce performance, so the reported superiority is an empirical finding rather than a construction-level identity. The references to prior OPD-family work (SOD, TCOD, token-importance studies) are motivational and baseline-related, not load-bearing self-citations or imported uniqueness theorems; moreover, the cited works are not by the present authors. Appendix A.3's Claim 1 is a standard martingale-difference unbiasedness statement, not a circular derivation. The abstract/full-text discrepancy in the OPD margin (4.16 vs 3.03 points) and the reliance on single-run maximum validation with no error bars are reporting and evidence-quality concerns, not circularity. The paper therefore contains no step in which a predicted result is equivalent by construction to its inputs.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The central method is an empirical combination of known components: OPD and GRPO. The paper introduces no physical or mathematical entities. It introduces design choices (annealing schedule, turn-level soft-OR weighting, normalization threshold) that are free hyperparameters, and relies on standard domain assumptions about the usefulness of teacher log-probabilities and group-relative advantages in long-horizon tasks.

free parameters (7)
  • OPD coefficient κ(s) = 1.0 → 0.1 (linear over T=80)
    Controls how much teacher distillation contributes to the hybrid advantage; hand-set schedule. Ablations show disabling annealing lowers ALFWorld success by 4–7 points.
  • RL coefficient ρ(s) = 1.0 → 2.0 (linear over T=80)
    Controls the reward-driven component weight; chosen by hand as part of the annealed schedule.
  • Annealing window T = 80 training steps
    Determines when the schedule saturates; set relative to the 150-step total training window, not derived.
  • Group size G = 8
    Number of trajectories per prompt used for GRPO group-relative advantage normalization; not swept or justified by variance analysis.
  • Training steps = 150
    All methods are compared at a 150-step window; this may favor methods with fast warm-up and limits insight into long-run convergence.
  • T-DUR normalization denominator threshold = 1e-8
    Avoids division by zero in per-trajectory max-min normalization; arbitrary constant.
  • Student sampling temperature = 1.0
    Rollout exploration temperature used when sampling trajectories; affects the entropy and exploration of the student.
axioms (4)
  • domain assumption Teacher token distributions provide a useful dense supervision signal on student-visited states.
    Standard for on-policy distillation; the paper relies on this for A_OPD (Eq. 9). If teacher log-probs are uninformative or misleading on student rollouts, the distillation term can hurt.
  • domain assumption GRPO's group-relative advantage with group size 8 gives a signal that is low-variance enough to drive reward-based improvement.
    Used for A_GRPO (Eq. 8); the paper does not analyze variance or sensitivity to group size in long-horizon sparse-reward tasks.
  • domain assumption Turn-level disagreement d_k and entropy h_k (Eqs. 13–14) are reliable proxies for distillation utility.
    Central to T-DUR; the design is motivated by prior token-level findings but the extension to turns is assumed without direct evidence that these proxies capture learning value.
  • standard math The reverse-KL OPD objective in Eq. 5 is a valid sampled estimator when using importance ratios.
    Standard importance-sampling estimator; the paper invokes it for the optimization objective, though the practical advantage in Eq. 9 omits the ratio.

pith-pipeline@v1.3.0-alltime-deepseek · 16736 in / 17264 out tokens · 156386 ms · 2026-08-02T09:52:08.652995+00:00 · methodology

0 comments
read the original abstract

Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement. On-policy distillation (OPD) provides dense teacher guidance and typically improves rapidly in the early stage, but its gains saturate once the student approaches the teacher, limiting the final performance ceiling. Reinforcement learning (RL) directly optimizes environment rewards and encourages exploratory improvement toward a higher reward-defined ceiling, but sparse and delayed feedback makes early-stage learning much less efficient than OPD. In this paper, we propose ATOD (Annealed Turn-aware On-policy Distillation), a hybrid online distillation algorithm that explicitly exploits this complementarity. (1) ATOD uses an annealed OPD-RL schedule: OPD dominates early training to approach teacher-level behavior, while RL is gradually strengthened to drive reward-based exploration. (2) ATOD introduces Turn-level Disagreement-Uncertainty Reweighting (T-DUR), which softly amplifies high-utility turns and improves dense supervision in long trajectories. Experiments on ALFWorld, WebShop, and Search-QA show that ATOD consistently outperforms competing post-training baselines: across the three student sizes, ATOD improves average success rate by 4.16 points over OPD and 23.62 points over GRPO, while surpassing the corresponding teacher models by 2.16 points.

Figures

Figures reproduced from arXiv: 2606.27814 by Mo Li, Peng Chen, Qitai Tan, Yang Li, Yipeng Shi, Zefang Zong.

Figure 1
Figure 1. Figure 1: Aggregate Qwen3-1.7B performance across ALFWorld, Search-QA, and WebShop. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Motivation for ATOD. (a) Compared with OPD and GRPO, ATOD preserves OPD’s fast [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of ATOD. (a) Annealed guidance shifts from teacher-guided bootstrapping to reward [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ALFWorld ablation results for Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B students. ATOD [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training dynamics on ALFWorld. (a) Training reward: ATOD maintains higher critic rewards. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Internal diagnostic metrics during ATOD training. (a) OPD/RL signal magnitude: OPD decays [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template used by ATOD for the ALFWorld task environment. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt template used by ATOD for the Search-based QA task environment. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt template used by ATOD for the WebShop task environment. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative ALFWorld trajectory from the ATOD-trained 1.7B student. The example is [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: ALFWorld appendix case study. T-DUR does not simply emphasize long or late turns: it up-weights the initial search plan, the route decision toward cleaning, and the final placement, while assigning near-zero weight to routine environment mechanics such as opening the fridge. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: WebShop appendix case study. Unlike the ALFWorld example, WebShop contains free-text [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: T-DUR diagnostic metrics for the 1.7B student model on the ALFWorld validation set, [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Additional training diagnostic metrics for the 1.7B model on the ALFWorld dataset. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

    cs.LG 2026-07 conditional novelty 6.0

    SAF stabilizes RLVR+on-policy-distillation training by sparsifying, bounding, warming up, and annealing the teacher-token advantage, improving aggregate math/code scores by 0.51–2.70% over fixed-coefficient fusion.

Reference graph

Works this paper leans on

32 extracted references · 16 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023

    2023

  2. [2]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess`ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems, 2023

    2023

  3. [3]

    Alfworld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆot´e, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021

    2021

  4. [4]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. InAdvances in Neural Information Processing Systems, 2022

    2022

  5. [5]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  7. [7]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Nature, 645:633–638, 2025

    2025

  9. [9]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, W...

    Pith/arXiv arXiv 2025

  10. [10]

    Archer: Training language model agents via hierarchical multi-turn rl

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. InProceedings of the 41st International Conference on Machine Learning, pages 62178–62209, 2024

    2024

  11. [11]

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. InInternational Conference on Learning Representations, 2025

    2025

  12. [12]

    Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978, 2025

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Agentic reinforced policy optimization.arXiv preprint arXiv:2507.19849, 2025

    Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. Agentic reinforced policy optimization.arXiv preprint arXiv:2507.19849, 2025. 10

    Pith/arXiv arXiv 2025

  14. [14]

    On-policy distillation of language models: Learning from self-generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. InInternational Conference on Learning Representations, 2024

    2024

  15. [15]

    Minillm: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, 2024

    2024

  16. [16]

    Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079, 2026

    Woogyeol Jin, Taywon Min, Yongjin Yang, Swanand Ravindra Kadhe, Yi Zhou, Dennis Wei, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models.arXiv preprint arXiv:2603.07079, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    On-policy context distillation for language models.arXiv preprint arXiv:2602.12275, 2026

    Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models.arXiv preprint arXiv:2602.12275, 2026

    Pith/arXiv arXiv 2026

  18. [18]

    Stable on-policy distillation through adaptive target reformulation.arXiv preprint arXiv:2601.07155, 2026

    Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation.arXiv preprint arXiv:2601.07155, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation.arXiv preprint arXiv:2602.12125, 2026

    Pith/arXiv arXiv 2026

  20. [20]

    Tcod: Exploring temporal cur- riculum in on-policy distillation for multi-turn autonomous agents.arXiv preprint arXiv:2604.24005, 2026

    Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. Tcod: Exploring temporal cur- riculum in on-policy distillation for multi-turn autonomous agents.arXiv preprint arXiv:2604.24005, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    Sod: Step-wise on-policy distillation for small language model agents.arXiv preprint arXiv:2605.07725, 2026

    Qiyong Zhong, Mao Zheng, Mingyang Song, Xin Lin, Jie Sun, Houcheng Jiang, Xiang Wang, and Junfeng Fang. Sod: Step-wise on-policy distillation for small language model agents.arXiv preprint arXiv:2605.07725, 2026

    Pith/arXiv arXiv 2026

  22. [22]

    Tip: Token importance in on-policy distillation.arXiv preprint arXiv:2604.14084, 2026

    Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, and Alborz Geramifard. Tip: Token importance in on-policy distillation.arXiv preprint arXiv:2604.14084, 2026

    Pith/arXiv arXiv 2026

  23. [23]

    Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXiv preprint arXiv:2601.18734, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents.arXiv preprint arXiv:2604.10674, 2026

    Hao Wang, Guozhi Wang, Han Xiao, Yufeng Zhou, Yue Pan, Jichao Wang, Ke Xu, Yafei Wen, Xiaohu Ruan, Xiaoxin Chen, and Honggang Qi. Skill-sd: Skill-conditioned self-distillation for multi-turn llm agents.arXiv preprint arXiv:2604.10674, 2026

    Pith/arXiv arXiv 2026

  25. [25]

    Self-distilled agentic reinforcement learning.arXiv preprint arXiv:2605.15155, 2026

    Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Self-distilled agentic reinforcement learning.arXiv preprint arXiv:2605.15155, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research.Transact...

    2019

  27. [27]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601–1611, 2017

    2017

  28. [28]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, 2018. 11

    2018

  29. [29]

    Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics, 10:539–554, 2022

    2022

  30. [30]

    Smith, and Mike Lewis

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, 2023. 12 Appendix Contents A Method and Algorithmic Details 14 A.1 Complete Training Algorithm . . . . . . . . . . . . . . . . . . . ...

    2023

  31. [31]

    If you find you lack some knowledge, you MUST call a search engine to get more external information using format:<search>your query</search>

  32. [32]

    I first need to find the lettuce; I will start by checking countertops

    If you have enough knowledge to answer the question confidently, provide your final answer within <answer> </answer>tags, without detailed illustrations. For example, <answer>Beijing</answer>. Figure 8: Prompt template used by ATOD for the Search-based QA task environment. Prompt of ATOD on WebShop You are an expert autonomous agent operating in the WebSh...