Pith. sign in

REVIEW 5 major objections 7 minor 38 references

Turn-level budgeting cuts agent training time by 2.3x

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 23:35 UTC pith:G6HRKA62

load-bearing objection Solid empirical efficiency gains from turn-level budgeting, but the novel loss-rebalancing component is only ablated on one task-student pair. the 5 major comments →

arxiv 2607.05804 v1 pith:G6HRKA62 submitted 2026-07-07 cs.AI cs.CL

TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training

Yuhang Zhou , Kai Zheng , Haoling Li , Dengyun Peng , Can Xu , Jingjing Chen This is my paper
classification cs.AI cs.CL
keywords on-policy distillationlong-horizon agentsturn-level budgetingKL divergencerollout depth adaptationloss allocationmulti-turn interactionlanguage model training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

On-policy distillation (OPD) is a training method where a student language model learns by imitating a stronger teacher on trajectories the student itself generates. This paper argues that when OPD is applied to long-horizon agent tasks — multi-turn interactions with tools and environments — two structural inefficiencies arise. First, rollouts are collected to a fixed maximum depth, wasting compute on late turns where the teacher's correction signal has decayed to noise. Second, the standard loss objective weights tokens uniformly, which means shallow early turns (which have more tokens and higher initial disagreement) consume over 95% of the loss budget, leaving deeper decision turns starved of supervision. The paper proposes TurnOPD, which replaces flat token-level budgeting with turn-level budgeting along two axes: an adaptive controller that truncates rollout depth based on periodic probes measuring where useful signal survives, and a progressive loss controller that gradually shifts KL-divergence weighting from token-proportional to turn-balanced as training proceeds. Across ALFWorld, WebShop, and Multi-Hop Search, TurnOPD achieves up to 2.29x faster training with equal or better accuracy. The broader claim is that the natural unit of supervision for long-horizon agents is not a flat token position but a turn-conditioned decision embedded in an evolving interaction trace.

Core claim

The paper's central diagnostic finding is that reverse-KL supervision in multi-turn agent OPD is both non-uniform and non-identifiable across turns. Raw KL decays with turn depth, but this decay does not necessarily mean the student has learned — it can mean that the student's own generated context has made both models predictable in the same way, compressing genuine disagreement into an unobservable residual. The paper formalizes this as a contamination-compression decomposition: each next-token distribution splits into a context-forced component (shared by student and teacher) and a free component (where real disagreement lives). As rollouts grow longer and student-generated context accumm

What carries the argument

Two controllers: (1) Adaptive rollout-depth budgeting, which periodically runs full-length probe rollouts to estimate a survivor-weighted KL centroid (efficiency signal) and a success-conditioned completion quantile (coverage floor), then sets the training rollout cap to their maximum. (2) Progressive turn-normalized loss budgeting, which linearly interpolates between trajectory-level token weighting and uniform turn-level weighting, shifting the blend coefficient from 0 to 1 over training. The contamination-compression model (Proposition 1) provides the theoretical motivation: observable KL is bounded above by (1-λ) times the latent free-component disagreement, where λ is the fraction of 'd

Load-bearing premise

The paper's theoretical explanation for why deep-turn KL degrades rests on a contamination-compression model that decomposes the next-token distribution into a forced component (shared by student and teacher) and a free component (where real disagreement lives). This decomposition is a theoretical construct: the forced mass λ is never measured or estimated empirically, and the existence of a clean forced/free split is assumed without validation. If the split does not reflecta

What would settle it

Measure or estimate λ(c) directly — for instance by perturbing surface-level context while preserving semantic content — and test whether the corrected free-component KL provides better outcome separation than raw KL at deep turns. If corrected KL still decays or fails to separate outcomes, the contamination-compression narrative is undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The turn-as-supervision-unit principle could extend beyond OPD to any multi-turn training signal — RLHF, rejection sampling fine-tuning, or trajectory-level reward models — wherever loss is aggregated over agent interaction traces.
  • If the contamination-compression mechanism is real, it implies that standard KL-based monitoring of agent training is systematically misleading for deep turns: a model can appear converged at depth while genuine policy disagreement persists, masked by shared context predictability.
  • The adaptive-depth controller's reliance on periodic full-length probes suggests a general design pattern for budgeted training: spend a small fraction of compute on uncensored measurement to inform aggressive truncation of the majority of training steps.
  • The progressive blending schedule (token-level early, turn-level late) instantiates a broader curriculum principle: the optimal granularity of supervision coarsens as the student masters shallow patterns and remaining errors concentrate in harder, deeper decisions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The contamination-compression decomposition predicts that if one could measure or estimate λ(c) — for instance by comparing teacher KL on original vs. paraphrased contexts that preserve semantics but break surface-level forced continuations — one could recover the latent free-component disagreement and build a corrected loss that does not require the progressive blending heuristic.
  • The observation that failed trajectories exhibit higher forced mass (stereotyped loops, template-like patterns) suggests a connection to mode collapse in RL: the compression mechanism may explain why KL-based regularization becomes less effective at preventing degenerate behavior as training progresses, since the regularization signal itself is compressed on exactly the trajectories it should be c
  • If the turn-level budgeting principle generalizes to tasks with much longer horizons (hundreds of turns, as in real computer-use or software engineering), the probe-based approach may face a scalability issue: full-length probes on long-horizon tasks could themselves become expensive, suggesting a need for hierarchical or sampled probing strategies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes TurnOPD, a turn-level budgeting strategy for on-policy distillation (OPD) of long-horizon language agents. The method addresses two diagnosed inefficiencies in vanilla agent OPD: (1) full-horizon rollouts waste compute on deep turns with weak KL supervision (external mismatch), and (2) trajectory-level KL objectives concentrate loss on shallow tokens, leaving deeper decision turns under-trained (internal mismatch). TurnOPD introduces two controllers: an adaptive rollout-depth controller that uses probe-based survivor-weighted KL centroids and success-conditioned coverage quantiles to set rollout length, and a progressive turn-normalized loss controller that linearly blends trajectory-level and turn-level KL weighting over training. Experiments on ALFWorld, WebShop, and Multi-Hop Search with multiple student-teacher pairs show TurnOPD improves the accuracy-time frontier, with up to 2.29x faster training. The paper also provides a contamination-compression model (Proposition 1) as a theoretical explanation for why raw KL degrades at depth.

Significance. The paper makes a solid contribution to the methodology of on-policy distillation for multi-turn agents. The turn-resolved diagnosis (Figures 2-4, Table 1) is well-motivated and provides useful empirical evidence for the two mismatches. The adaptive rollout-depth controller is a practical and well-designed mechanism with clear algorithmic specification (Algorithm 1, Table 7). The contamination-compression bound (Proposition 1, Appendix B) is a clean mathematical result derived from joint convexity of KL, and the existence proof for the optimal rollout horizon (Appendix C) provides formal grounding for the controller. The empirical evaluation spans three benchmarks with proper baselines (vanilla OPD, TCOD-F2B) and includes thorough ablations (Tables 5-6, Figure 8). The coverage-floor sensitivity study (Section 7.3) demonstrating that the controller works without success-conditioned CDFs is a notable strength for generalizability.

major comments (5)
  1. The central claim is that 'turn-level budgeting of both rollout depth and loss allocation is a better unit of supervision than flat token-level optimization.' However, the component ablation (Table 5) is conducted only on ALFWorld-1.7B. On that task, adaptive depth alone does not improve accuracy (82.8 vs 83.0 vanilla), and the accuracy gain comes from the linear blend loss rebalancing (85.1 alone, 86.3 combined). Meanwhile, the '2.29x faster' headline comes entirely from the rollout-truncation mechanism. More critically, the Same-Step results in Table 2 show that combined TurnOPD does not consistently improve per-step quality: it regresses on Multi-Hop Search (47.24 vs 47.82, -0.58) and is only marginally positive on ALFWorld-4B (+0.40). Without ablations on Multi-Hop Search or ALFWorld-4B, we cannot determine whether the loss-rebalancing component (the novel contribution beyond simple,
  2. rollout truncation) generalizes or is task-specific to ALFWorld-1.7B. If the loss rebalancing does not help on Multi-Hop Search, the contribution narrows to 'truncate rollouts to save compute,' which is weaker than the stated central claim. The authors should either (a) run component ablations on at least one additional task to demonstrate generalization of the loss-rebalancing component, or (b) revise the central claim to accurately reflect that the two components may contribute differently across tasks, with the loss-rebalancing benefit being task-specific.
  3. The contamination-compression model (Section 4.3, Eq. 4) decomposes the next-token distribution into a shared forced component p_F and a free component where student and teacher may disagree, with forced mass lambda(c) in [0,1]. This decomposition is a theoretical construct: lambda(c) is never measured or estimated empirically, and the existence of a clean forced/free split is assumed without validation. The entire explanation for why deep-turn KL degrades (and thus the motivation for the progressive loss controller) rests on this unverified decomposition. While the authors note the model is 'one possible mechanism,' the diagnostic narrative in Sections 4.2-4.3 and the design rationale for the loss controller in Section 5.2 lean heavily on it. The paper would be substantially strengthened by even a coarse empirical probe of lambda(c) — for instance, measuring the fraction of next-token概率
  4. mass where both student and teacher assign near-identical high probability to the same token (a proxy for forced mass) — to validate that the decomposition captures a real structural property. Alternatively, the authors should more clearly separate the empirical motivation (Figures 2-4, which are directly measured) from the theoretical narrative, and acknowledge that the loss controller's design is primarily motivated by the directly observed loss-budget concentration (Figure 4, Table 1) rather than by the contamination-compression model.
  5. Table 2 reports that on Multi-Hop Search, TurnOPD's Same-Step accuracy (47.24) is lower than vanilla OPD (47.82), yet the Least-Time accuracy (47.24) is higher than vanilla OPD's Least-Time accuracy (45.77). This is because the Least-Time comparison uses different step counts (TurnOPD reaches 100 steps faster, so vanilla OPD is compared at fewer steps). This comparison protocol is reasonable for the wall-clock efficiency claim, but the paper should more transparently report how many optimizer steps each method completed within the Least-Time window for each task, so readers can assess whether the Least-Time advantage is purely from more optimization steps (enabled by faster per-step compute) rather than from better per-step quality. Adding a column for 'steps completed at cutoff' in Table 2 would clarify this.
minor comments (7)
  1. Section 5.1, Eq. 13: The notation [K_t]_+ is introduced with a footnote, but the footnote text says 'K_t can be slightly negative due to noise.' It would help to clarify what noise source produces negative KL values, since KL is non-negative by definition — this likely refers to estimation noise from finite samples or top-K approximation, which should be stated.
  2. Table 7: The hyperparameter 'distill_topk=50' is listed as K_top in the free_parameters but the role description says 'teacher top-K top support size for the reverse-KL loss.' It would help to clarify whether this means the KL is computed only over the top-50 teacher tokens (a truncated vocabulary), and if so, how this interacts with the contamination-compression model's forced/free decomposition.
  3. Figure 1: The figure is information-dense and somewhat hard to parse. Panels A-D are referenced in the text but not clearly labeled in the figure itself. Adding panel labels (A, B, C, D) directly on the figure would improve readability.
  4. Section 6.1: The paper states 'TurnOPD even exceeds the ALFWorld teacher reference in overall avg@4' for the Qwen3-4B student (91.73 vs teacher 90.75). This is a notable claim that deserves brief discussion — is this within noise (the teacher's 90.75 is a single number without variance), or does it suggest the student benefits from the OPD training beyond the teacher's capability?
  5. Appendix D, Table 7: The 'kl_coef=0, kl_loss_coef=0' entries with 'loss_mode=topk_reverse_kl' could be confusing — if KL coefficients are zero, how is the reverse-KL loss active? This likely means no auxiliary KL penalty/reward term is added, with the main loss being the top-K reverse KL, but the notation is unclear.
  6. The paper uses 'round' and 'turn' somewhat interchangeably in places (e.g., Figure 1 uses 'round' in panel labels while the text uses 'turn'). Standardizing on 'turn' throughout would improve consistency.
  7. References: Several references use 2026 dates (e.g., [8, 11, 12, 24, 33, 34, 36]). If these are genuine publications, no issue; if they are arXiv preprints, the access dates should be verified.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee raises four major points: (1) component ablations are limited to ALFWorld-1.7B, and the Same-Step regression on Multi-Hop Search raises questions about generalization of the loss-rebalancing component; (2) the contamination-compression model (lambda(c)) is never empirically validated; (3) Table 2's Least-Time comparison should report steps completed at cutoff for transparency. We address each below. We agree with points (1) and (3) and will revise accordingly; on point (2) we will both add an empirical probe and restructure the narrative to separate measured diagnostics from the theoretical model.

read point-by-point responses

  1. Referee: Component ablation only on ALFWorld-1.7B; loss-rebalancing may not generalize; Same-Step regression on Multi-Hop Search; central claim may need revision

    Authors: The referee is correct that the component ablation in Table 5 is limited to ALFWorld-1.7B, and that the Same-Step results in Table 2 show the loss-rebalancing component does not uniformly improve per-step quality across tasks. We accept this criticism. We will run component ablations (adaptive depth alone, linear blend alone, full TurnOPD) on at least one additional task — Multi-Hop Search is the most informative choice because it is where the Same-Step regression occurs. This will directly test whether the loss-rebalancing component helps, hurts, or is neutral there. If the result confirms that loss rebalancing is task-specific in its per-step quality effect, we will revise the central claim to state that the two components contribute differently across tasks: adaptive depth provides the efficiency lever universally, while loss rebalancing provides an accuracy lever that is task-dependent in magnitude and sometimes neutral in Same-Step terms. We note that even on Multi-Hop Search, TurnOPD achieves the best Least-Time accuracy (47.24 vs 45.77), so the combined method's wall-clock efficiency advantage holds regardless. But the referee is right that we should not claim uniform per-step quality improvement without evidence, and we will not. revision: yes

  2. Referee: Contamination-compression model: lambda(c) never measured; decomposition unverified; motivation for loss controller rests on unverified construct

    Authors: We agree that the contamination-compression model would be strengthened by an empirical probe of lambda(c), and we will add one. A natural proxy is: for each supervised token context, measure the fraction of next-token probability mass where both student and teacher assign near-identical high probability to the same token (e.g., top-1 agreement with both probabilities above a threshold). This provides a coarse but directly computable estimate of forced mass. We will report this proxy across turn depth on ALFWorld and Multi-Hop Search, showing whether forced mass increases with depth as the model predicts. Additionally, we accept the referee's suggestion to more clearly separate the empirical motivation from the theoretical narrative. The loss controller's design is directly motivated by the measured loss-budget concentration (Figure 4, Table 1) — this is already the primary motivation in Section 5.2, but the narrative in Sections 4.2-4.3 does lean on the contamination-compression model as an explanatory bridge. We will restructure to make clear that the directly observed diagnostics (Figures 2-4, Table 1) are the primary motivation, and that the contamination-compression model is a complementary theoretical explanation rather than a prerequisite for the controller design. If the lambda(c) probe does not support the model, we will say so honestly. revision: yes

  3. Referee: Table 2 Least-Time comparison should report steps completed at cutoff for transparency

    Authors: This is a fair and straightforward request. We will add a column to Table 2 reporting the number of optimizer steps each method completed within the Least-Time window for each task. This will allow readers to see whether the Least-Time advantage comes from more optimization steps (enabled by faster per-step compute from rollout truncation), better per-step quality, or both. We expect the answer will vary by task: on ALFWorld-1.7B, where both speedup and accuracy improve, the advantage is likely a combination; on Multi-Hop Search, where Same-Step quality slightly regresses, the Least-Time advantage is likely driven primarily by completing more steps. Making this transparent is in the interest of honest reporting, and we have no objection to including it. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; derivation is self-contained with standard mathematical proofs and observable-statistic-based controllers.

full rationale

The paper's derivation chain is self-contained. Proposition 1 (the contamination-compression bound, Eq. 6) is proved in Appendix B from the joint convexity of KL divergence via the log-sum inequality — a standard, externally verifiable mathematical result, not a self-citation chain. The two controllers use only observable statistics (per-turn raw KL K_t, survivor counts n_t, success-conditioned completion depths) to compute H_eff and H_cov; no parameter is fitted to a target outcome and then presented as a prediction. The linear blend coefficient α is a simple schedule from 0 to 1 over training progress, not a fitted constant. The existence of H* (Appendix C) is derived from an efficiency-coverage sandwich argument that is self-contained. The λ(c) decomposition (Eq. 4) is explicitly offered as 'one contamination-compression mechanism' and is never measured or fitted; this is a validity concern (the theoretical model is unverified), not a circularity concern, since no claimed prediction reduces to λ(c) by construction. The authors' prior work [38] (OffSeeker) appears only in general references and is not load-bearing for any derivation step. Score 1 reflects the absence of circularity with a minor note that the theoretical model is unmeasured, but this falls under correctness risk rather than circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 1 invented entities

The axiom ledger captures the theoretical scaffolding. The forced/free decomposition is the key invented entity; it is used for explanation but not directly validated. The free parameters are standard hyperparameters for a training controller, not fitted constants masquerading as derived quantities.

free parameters (6)
  • coverage_quantile p = 0.80
    Sets the success-conditioned completion depth floor H_cov. Chosen by the authors; sensitivity analysis in Figure 8 shows it is tunable.
  • ema_alpha = 0.30
    EMA smoothing weight for the rollout-depth controller. Set by hand in Table 7.
  • blend window (s, e) = (0, 1)
    Linear progress window for the loss-blend coefficient alpha. Defaults to full training range.
  • probe_interval = 8
    Steps between full-depth probe rollouts. Set by hand.
  • warmup_steps = 3
    Probe-only warmup before truncation begins. Set by hand.
  • distill_topk K_top = 50
    Teacher top-K support size for the reverse-KL loss. Set by hand.
axioms (4)
  • ad hoc to paper The next-token distribution can be decomposed into a shared forced component p_F and a free component where student and teacher may differ (Eq. 4 in Section 4.3).
    This decomposition is the basis for the contamination-compression model. It is assumed, not derived from a more fundamental principle, and λ(c) is never measured.
  • standard math Joint convexity of KL divergence (log-sum inequality).
    Used in Appendix B to prove Proposition 1. Standard result.
  • domain assumption There exists a signal-exhaustion frontier tau_c after which marginal teacher-correction value stays below the running average (Appendix C).
    Assumed to bound H*_eff from above. Reasonable for the tasks studied but not proven in general.
  • domain assumption The useful unit of supervision in long-horizon agents is a turn-conditioned decision, not a flat token position.
    Stated in Section 4.5 and the conclusion. This is the conceptual premise motivating the entire method.
invented entities (1)
  • Forced component p_F of the next-token distribution no independent evidence
    purpose: Models the portion of the distribution that is context-determined and shared by student and teacher, explaining why raw KL compresses at depth.
    p_F and its mass λ(c) are theoretical constructs introduced to explain observed KL decay. They are never measured or estimated empirically. The paper acknowledges this is a 'first-order approximation' (Eq. 8). No falsifiable prediction about λ(c) is made that could be checked outside the paper.

pith-pipeline@v1.1.0-glm · 25193 in / 3533 out tokens · 257478 ms · 2026-07-08T23:35:12.498682+00:00 · methodology

0 comments
read the original abstract

On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 18 internal anchors

  1. [1]

    On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes, 2024. URLhttps: //arxiv.org/abs/2306.13649

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    Textworld: A learning environment for text-based games

    Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. Textworld: A learning environment for text-based games. In WorkshoponComputerGames, pages 41–75. Springer, 2018

    work page 2018

  3. [3]

    Mind2Web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. InAdvancesinNeuralInformationProcessingSystems(NeurIPS), 2023

    work page 2023

  4. [4]

    A survey of self-evolving agents: On path to artificial super intelligence.arXive-prints, pages arXiv–2507, 2025

    Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: On path to artificial super intelligence.arXive-prints, pages arXiv–2507, 2025

    work page 2025

  5. [5]

    MiniLLM: Knowledge distillation of large language models

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In International Conferenceon Learning Representations(ICLR), 2024

    work page 2024

  6. [6]

    Constructing a multi-hop qa dataset for comprehensiveevaluationofreasoningsteps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensiveevaluationofreasoningsteps. In Proceedingsofthe28thInternationalConferenceonComputational Linguistics, pages 6609–6625, 2020

    work page 2020

  7. [7]

    Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation

    Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Ziyu Ye, Bowei Xia, Tao Sun, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, et al. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation. AdvancesinNeuralInformationProcessingSystems, 38:50859–50906, 2026

    work page 2026

  8. [8]

    Stable On-Policy Distillation through Adaptive Target Reformulation

    Ijun Jang, Jewon Yeom, Juan Yeo, Hyunggu Lim, and Taesup Kim. Stable on-policy distillation through adaptive target reformulation, 2026. URLhttps://arxiv.org/abs/2601.07155

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  9. [9]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations(ICLR), 2024

    work page 2024

  10. [10]

    Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning.arXivpreprintarXiv:2503.09516, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  11. [11]

    Entropy-aware on-policy distillation of language models, 2026

    Woogyeol Jin, Taywon Min, Yongjin Yang, Dennis Wei, Yi Zhou, Swanand Ravindra Kadhe, Nathalie Baracaldo, and Kimin Lee. Entropy-aware on-policy distillation of language models, 2026. URLhttps://arxiv.org/abs/2603. 07079

    work page 2026

  12. [12]

    Scaling reasoning efficiently via relaxed on-policy distillation, 2026

    Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, and Pashmina Cameron. Scaling reasoning efficiently via relaxed on-policy distillation, 2026. URLhttps://arxiv.org/abs/2603.11137

    work page arXiv 2026

  13. [13]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research.Transact...

    work page 2019

  14. [14]

    Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

    Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe.arXivpreprint arXiv:2604.13016, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  15. [15]

    On-policy distillation.Thinking Machines Lab: Con- nectionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.ThinkingMachinesLab: Connectionism, 2025. doi: 10.64434/tml.20251026. https://thinkingmachines.ai/blog/on-policy-distillation

    work page doi:10.64434/tml.20251026 2025

  16. [16]

    Agentmath: Empowering mathematical reasoning for large language models via tool-augmented agent.arXiv preprint arXiv:2512.20745, 2025

    Haipeng Luo, Huawen Feng, Qingfeng Sun, Can Xu, Kai Zheng, Yufei Wang, Tao Yang, Han Hu, and Yansong Tang. Agentmath: Empowering mathematical reasoning for large language models via tool-augmented agent.arXiv preprint arXiv:2512.20745, 2025. 17

    work page arXiv 2025

  17. [17]

    Large Language Model Agent: A Survey on Methodology, Applications and Challenges

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology, applications and challenges.arXiv preprint arXiv:2503.21460, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  18. [18]

    When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. In Proceedings ofthe 61stAnnualMeeting ofthe AssociationforComputationalLinguistics, pages 9802–9822, 2023

    work page 2023

  19. [19]

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprintarXiv:2601.11868, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  20. [20]

    GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Tworek. GDPval: Evaluating AI model performance on real-worl...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  21. [21]

    AndroidWorld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A dynamic benchmarking environment for autonomous agents. In International Conferenceon Learning Re...

    work page 2025

  22. [22]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    ALFWorld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. InInternational Conference on Learning Representations(ICLR), 2021

    work page 2021

  24. [24]

    TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents

    Jiaqi Wang, Wenhao Zhang, Weijie Shi, Yaliang Li, and James Cheng. TCOD: Exploring temporal curriculum in on-policy distillation for multi-turn autonomous agents.arXivpreprintarXiv:2604.24005, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents. arXiv preprintarXiv:2504.12516, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    Agentgym: Evaluating and training large language model-based agents across diverseenvironments

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Xin Guo, Dingwen Yang, Chenyang Liao, Wei He, et al. Agentgym: Evaluating and training large language model-based agents across diverseenvironments. InProceedingsofthe63rdAnnualMeetingoftheAssociationforComputationalLinguistics (Volume1: LongPapers), pages 27914–27961, 2025

    work page 2025

  27. [27]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    TianbaoXie,DanyangZhang,JixuanChen,XiaochuanLi,SihengZhao,RuishengCao,TohJingHua,ZhoujunCheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In AdvancesinNeuralInformationProcessingSystems(Ne...

    work page 2024

  28. [28]

    TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

    FrankF.Xu, YufanSong, BoxuanLi, YuxuanTang, KritanjaliJain, MengxueBao, ZoraZ.Wang, XuhuiZhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Melroy Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. TheAgentCompany: Benchmarking LLM agents on consequential real world tasks.arXiv pre...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [29]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXivpreprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  30. [30]

    Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

    Wenkai Yang, Weijie Liu, Ruobing Xie, Kai Yang, Saiyong Yang, and Yankai Lin. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation, 2026. URLhttps://arxiv.org/abs/2602.12125

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  31. [31]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. InProceedingsofthe 2018 Conferenceon EmpiricalMethods inNaturalLanguageProcessing, pages 2369–2380, 2018. 18

    work page 2018

  32. [32]

    WebShop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    work page 2022

  33. [33]

    Are Full Rollouts Necessary for On-Policy Distillation?

    YaochengZhang,JiajunChai,SongjunTu,YuqianFu,XiaohanWang,WeiLin,GuojunYin,QichaoZhang,Yuanheng Zhu, and Dongbin Zhao. Are full rollouts necessary for on-policy distillation?arXiv preprint arXiv:2605.31490, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  34. [34]

    Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, Huaisong Zhang, Xian Wu, Yi Lu, Minyi Lei, Kai Zou, Huifeng Yin, Ping Nie, Liang Chen, Dongfu Jiang, Wenhu Chen, and Kelsey R. Allen. ClawBench: Can AI agents complete everyday online tasks? arXiv preprintarXiv:2604.08523, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  35. [35]

    Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

    Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models.arXivpreprintarXiv:2601.18734, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  36. [36]

    SOD: Step-wise On-policy Distillation for Small Language Model Agents

    Qiyong Zhong, Mao Zheng, Mingyang Song, Xin Lin, Jie Sun, Houcheng Jiang, Xiang Wang, and Junfeng Fang. SOD: Step-wise on-policy distillation for small language model agents.arXiv preprintarXiv:2605.07725, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  37. [37]

    WebArena: A Realistic Web Environment for Building Autonomous Agents

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. arXiv preprintarXiv:2307.13854, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  38. [38]

    thinking

    Yuhang Zhou, Kai Zheng, Qiguang Chen, Mengkang Hu, Qingfeng Sun, Can Xu, and Jingjing Chen. Offseeker: Online reinforcement learning is not all you need for deep research agents.arXivpreprintarXiv:2601.18467, 2026. 19 A Environments, Tool Configurations, And Task Examples In this work, we use the following three environments and task types: ALFWorld [23],...

    work page arXiv 2026