Pith. sign in

REVIEW 3 major objections 4 minor 42 references

ProGPO: extracting learning signal from all-fail rollout groups using first-visit observation coverage.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:50 UTC pith:RBXLU3BS

load-bearing objection ProGPO is a simple, honest, and potentially useful fix for the all-fail regime in group-based RL; the only real weakness is that the headline 'consistently improves' rests on single training seeds. the 3 major comments →

arxiv 2607.22724 v1 pith:RBXLU3BS submitted 2026-07-22 cs.LG cs.AI

Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks

Kaibing Yang , Guangfeng Cai , Shengtian Yang , Shuo He , Yu Li , Mengyi Liu , Pengwei Chen , Jun Xu
show 1 more author
Lei Feng
This is my paper
classification cs.LG cs.AI
keywords group-based policy optimizationcredit assignmentsparse rewardLLM agentsfirst-visit observation coverageall-fail groupslong-horizon tasksreinforcement learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Group-based policy optimization trains LLM agents by comparing trajectories within a rollout group, but on hard long-horizon tasks many groups contain no successful trajectory. When every reward is zero, the comparison collapses and the policy receives no gradient, a self-reinforcing credit trap that lets repeated low-effect actions persist. ProGPO shows that failed trajectories are not equally uninformative: within an all-fail group, some trajectories visit many distinct observations while others cycle or stall. It uses first-visit observation coverage as a fallback advantage, assigned only when reward variance is zero, and provably reverts exactly to the base estimator when outcome variation exists. On ALFWorld and WebShop with 1.5B and 7B models, this consistently improves over three group-based baselines, with the largest gains on the hardest, all-fail-prone subtasks.

Core claim

The paper's central claim is that the all-fail credit trap—a rollout group in which every trajectory fails, so outcome rewards give zero relative advantage—can be broken without auxiliary models or reward shaping by looking at the behavioral contrast among the failed trajectories. ProGPO scores each trajectory by its first-visit coverage, P = (D-1)/T, the fraction of steps that transition to a previously unseen observation, and uses it only when the reward-based advantage is identically zero. The estimator is a strict extension of the base group-relative estimator: on every group with non-degenerate reward variance, ProGPO reduces exactly to the base advantage, so the difference in gradients

What carries the argument

The first-visit coverage score P(τ) = (D(τ)-1)/T, where D(τ) is the number of distinct observations in the trajectory and T its length, counts each transition to a new observation exactly once and gives zero credit to revisits or no-op actions. This score powers a three-branch conditional advantage estimator: the standard reward-normalized advantage when reward variance exceeds a threshold; a fallback advantage λ(P_i - mean P)/σ_P when all rewards are zero but coverage varies; and zero otherwise. The gating guarantees non-interference (the gradient differs from the base only where the base gradient is identically zero), and the fallback branch is exactly group-normalized REINFORCE with P as

Load-bearing premise

Within an all-fail group, trajectories that visit more distinct observations are closer to eventual success than those that visit fewer, so reinforcing high-coverage failures (and demoting low-coverage ones) improves the policy; this is an empirical property, not a theorem.

What would settle it

Run ProGPO on a variant of ALFWorld where a 'wander' action changes the observation text (or any environment where observations become more diverse without approaching the goal) and compare success rate against the base estimator. If adding the fallback lowers success relative to the baseline, the coverage-progress link fails. A more direct measurement: on collected all-fail groups, compute the correlation between first-visit coverage and ground-truth distance to success; a non-positive correlation across groups would falsify the core assumption.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Rollout groups that previously contributed zero gradient can now contribute weak, behavior-aligned signal, reducing wasted compute on hard long-horizon tasks.
  • Because ProGPO is a strict extension of the base estimator, any group-based policy optimization method can adopt it without changing behavior on informative groups.
  • The success-set containment result means pushing probability toward high-coverage trajectories keeps the policy inside a region that contains all successful behavior, so the fallback cannot contradict the task objective in principle.
  • As the policy improves, the all-fail trigger probability decays at most exponentially in the group size times success probability, so the fallback automatically hands control back to reward when it becomes informative.
  • The measured gains concentrate on the hardest subtasks, suggesting a route to training agents on tasks where outcome-only RL currently stalls.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The coverage signal is environment-dependent: in settings where observation strings change without genuine progress (noise, adversarial detours, hidden state), the fallback could reinforce wandering. A stress test with observation noise injected into otherwise solvable tasks would delimit the method's domain.
  • The conditional-fallback recipe could generalize beyond coverage: any cheap proxy that is a necessary condition for success (e.g., subgoal flags, information gain) could be plugged into the same gate, provided it is only used when reward variance is zero.
  • The exponential disengagement bound predicts that the method's advantage over baselines should be largest early in training and on hard task distributions; a longitudinal study across task difficulty would make this testable.
  • Because coverage is computed from the observation string, the method is naturally applicable to other text-rich agent environments; whether it transfers to visual or latent-state environments is an open question the paper explicitly defers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Progress-conditioned Group Policy Optimization (ProGPO), a fallback advantage estimator for group-based RL of LLM agents. In rollout groups where all trajectories fail, the reward-based advantages of GRPO/GiGPO/HGPO are identically zero; ProGPO replaces them with a normalized first-visit observation-coverage score, and otherwise leaves the base advantage untouched. The authors provide propositions showing that the progress score is a relaxation of success, that the method does not modify updates on reward-informative groups, and that the fallback branch is group-normalized REINFORCE on the progress score. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B-Instruct report consistent improvements over GRPO, GiGPO, and HGPO under matched compute budgets, with negative controls (Shuffle, Random, Neg, Obs-Change), an ablation of conditional gating, and a hyperparameter sweep.

Significance. If the empirical claims hold, this is a valuable contribution: a simple, model-free repair for a genuinely degenerate regime of group-based policy optimization. The paper's strengths are explicit and welcome: no auxiliary critic or reward model, an exact non-interference statement, a transparent per-group characterization of the fallback gradient, matched compute budgets between base and augmented methods, and a battery of negative controls that test the mechanism rather than only reporting aggregate gains. The theoretical core (Propositions 4.1, 4.3, 4.4 and the finite-group bias calculation) is sound and carefully scoped; the authors also honestly flag that within-all-fail-group coverage closeness to success is an empirical property (Remark 4.2) and that observation-level coverage may not transfer to latent-state environments (Section 12). The main weakness is that the central comparative claim — "consistently improves" — rests on a single training seed at the final checkpoints reported in Table 1; this needs to be fixed before the empirical conclusions are supportable.

major comments (3)
  1. [§5.2, Table 1; Appendix 8.8; Appendix 10.4] The claim that ProGPO "consistently improves" group-based baselines is not yet established by the reported evidence. Table 1's caption says results are "averaged over 3 random seeds," but Appendix 8.8 clarifies that this is re-evaluation of one final checkpoint under three inference seeds, and Appendix 10.4 states that controlled comparisons fix the training seed. The only multi-training-seed evidence is a 30-epoch snapshot (Table 10), not the 150-epoch final checkpoints. This matters because Proposition 4.3 guarantees the fallback only modifies all-fail groups, so every claimed gain is a training-dynamics effect whose uncertainty cannot be quantified by inference-seed variance. The near-ceiling 7B HGPO ALFWorld result (95.4→95.3) already shows the effect can vanish, and aggregate gains such as GiGPO 86.7→91.4 or WebShop 65.0→72.4 could be partly seed luck. Please report final-checkpoint
  2. [§5.3, Appendix 10.6, Table 11] The hyperparameter-sensitivity table is internally inconsistent. The text states that the default λ_aux=0.3 achieves 92.2% overall success, with 0.1 at 94.5% and 0.5/0.7 lower. However, the row labeled "1.0" in Table 11 has exactly the same per-category values as the main Table 1 GiGPO+ProGPO row (All 91.4±1.6), while the row labeled "0.3 (default)" reports a different protocol (five training seeds) and a different aggregate (92.2±2.0). Either the row labels are swapped or the evaluation protocols differ in a way that invalidates direct comparison. In addition, the claim that λ_aux is "fixed at 0.3" is qualified in Appendix 8.3 by the deployed λ_eff = λ_aux q_fail, so the sweep in Table 11 is not a sweep of the actually deployed scale unless this is reconciled. Please correct the table and clarify which quantity was swept under which evaluation protocol.
  3. [§4.2, Eq. (7); §8.3, Algorithm 2] There is a mismatch between the analyzed estimator and the deployed implementation. Eq. (7) and Propositions 4.3–4.4 treat λ as a fixed positive constant, while Algorithm 2 and Eq. (30) set λ_eff = λ_aux q_fail(B), where q_fail is the fraction of all-fail groups in the current update. The authors acknowledge this in the audit note and correctly state that a common positive multiplier preserves the groupwise direction of Proposition 4.4, so the non-interference claim is unaffected. Nevertheless, the theoretical claim that the fallback "reverts exactly" to the base estimator and that ProGPO has "fixed λ_aux=0.3 across all environments" is presented in the main text without this qualification. Please state the deployed scaling law in Section 4.2 or explicitly defer it to the implementation appendix, and clarify whether the reported λ_aux sweep controls the effective scale or only its ceilin
minor comments (4)
  1. [Eq. (5)–(7)] The notation σ_R(G_x), σ_P(G_x), ̅R, and ̅P is used in Eq. (7) but the population-variance convention is defined only in Appendix 7.2, Eq. (14). Define it in the main text to make the estimator self-contained.
  2. [§5.3, Appendix 10.3] The text says "More details shown in Appendixx 10.3" — typo "Appendixx." Also, the Edit-D variant is named but not described; please state how edit distance is computed and normalized.
  3. [Table 8, Appendix 10.6] The footnote for Table 11 says entries with a standard deviation are five-seed averages, but the main Table 1 uses three inference seeds from one checkpoint, and Table 8 similarly mixes five-seed and single-seed rows. Please standardize the seed reporting labels so a reader can tell which rows support variance claims.
  4. [References] Reference [31] and [32] are the same paper (SPA-RL); the duplication should be removed.

Circularity Check

0 steps flagged

No significant circularity: the progress signal is computed from rollouts, not fitted to the target metric, and the invariance results are by-construction identities rather than predictions.

full rationale

The derivation chain is self-contained. Eq. (5)-(6) define the first-visit coverage score directly from observation sequences; Eq. (7) defines the ProGPO advantage so that the reward branch is identical to the base estimator when sigma_R >= tau_R. Proposition 4.3 is an immediate consequence of this branch structure, while Proposition 4.4 is an algebraic identity showing the fallback is group-normalized REINFORCE with P as the return. Proposition 4.1(i) is a counting identity, and 4.1(ii) uses the paper's own definition of mx-separation (minimum distinct observations along successful trajectories) to obtain a containment result that the paper explicitly labels a relaxation rather than a causal statement. Remark 4.2 states plainly that whether higher-coverage failed trajectories are closer to success is an empirical property of the environment, and Sections 5.3/Appendix 10 test this against Shuffle, Random, and negated controls. No parameter is fitted to the target metric and then reported as a prediction; no uniqueness theorem or load-bearing self-citation is invoked; the always-on progress variant is ablated and rejected. Concerns about single training-seed evidence are experimental robustness issues, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests mainly on the empirical assumption that coverage ranking within all-fail groups is progress-aligned, plus the textual-observation assumption. The only hand-set parameters that materially affect results are λ_aux and the variance thresholds. No new physical or model-based entity is invented; 'credit trap' is a framing concept, not an entity.

free parameters (4)
  • λ_aux (fallback scale) = 0.3
    Ceiling for the fallback advantage magnitude; fixed across environments, scales, and base algorithms. The sweep in Appendix 10.6 shows 0.1 gives 94.5% vs 92.2% for the default 0.3, so performance is scale-sensitive.
  • τR (reward degeneracy threshold) = 1e-3
    Hand-set numerical threshold used to detect zero reward variance. Below the minimum nonzero binary-reward standard deviation for K=8, so it mainly guards against division by zero.
  • τP (progress variance threshold) = 1e-4
    Hand-set threshold that discards all-fail groups where progress scores also lack within-group variation.
  • K (rollout group size) = 8
    Rollout group size used in all experiments; it enters the variance thresholds and the exponential disengagement bound, but it is inherited from the group-based baselines rather than tuned by the paper.
axioms (4)
  • domain assumption Observation strings fully externalize task-relevant state changes, so exact-string first-visit novelty is a faithful progress signal.
    The progress measure compares raw observation strings (Section 8.2). If consequential state changes are not reflected in the observation text, the coverage score misses them; the paper states this as a limitation in Section 12.
  • domain assumption Within all-fail groups, higher first-visit observation coverage is correlated with being closer to success.
    This is the load-bearing empirical premise behind the fallback. Remark 4.2 and Appendix 7.6 explicitly say it is an empirical property, not a theorem, and the gains depend on it holding in the target environments.
  • ad hoc to paper The task is m_x-separated: completing it requires visiting at least m_x ≥ 2 distinct observations.
    Introduced solely to state the super-level-set bound in Proposition 4.1(ii). It is a definitional assumption about the task rather than a substantive empirical constraint.
  • standard math The K trajectories in a rollout group are sampled independently from the current policy πθ.
    Used in the group-mean baseline expansion (Proposition 4.4), the exponential disengagement bound (Remark 7.1), and the finite-group bias calculation in Appendix 7.3. This is standard for group-based RL.

pith-pipeline@v1.3.0-alltime-deepseek · 29744 in / 11994 out tokens · 130522 ms · 2026-08-01T11:50:39.042027+00:00 · methodology

0 comments
read the original abstract

Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group. However, on difficult long-horizon tasks, this comparison can suffer from a sampling imbalance: repeated or low-effect actions dominate the high-probability region of the policy while useful state-changing actions remain under-sampled. This imbalance produces many all-failed rollout groups, where outcome rewards provide no direction for correcting the policy. Together, these effects can form a self-reinforcing credit trap: failure-dominated sampling yields no outcome-based correction, allowing repeated low-effect actions to persist. To break this loop, we propose Progress-conditioned Group Policy Optimization (ProGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward. Specifically, within such groups, ProGPO assigns higher relative advantages to trajectories or steps that visit more new states since reaching new observations is a prerequisite for task success. Experiments on two challenging agentic benchmarks, ALFWorld and WebShop with Qwen2.5-1.5/7B-Instruct, show that ProGPO consistently improves over group-based baselines, with particularly large gains on hard tasks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 16 linked inside Pith

  1. [1]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, 10 and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248–12267, 2024

    2024

  3. [3]

    Xrpo: Pushing the limits of grpo with targeted exploration and exploitation.arXiv preprint arXiv:2510.06672, 2025

    Udbhav Bamba, Minghao Fang, Yifan Yu, Haizhong Zheng, and Fan Lai. Xrpo: Pushing the limits of grpo with targeted exploration and exploitation.arXiv preprint arXiv:2510.06672, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Progra: Progress-aware reinforcement learning for multi-turn function calling

    Huacan Chai, Zijie Cao, Maolin Ran, Yingxuan Yang, Jianghao Lin, Renjie Ding, Ziyu Wan, Muning Wen, Weiwen Liu, Weinan Zhang, et al. Progra: Progress-aware reinforcement learning for multi-turn function calling

  5. [5]

    Beyond trajectory-level attribution: Graph-based credit assignment for agentic reinforcement learning.arXiv preprint arXiv:2605.26684, 2026

    Xin Cheng, Shuo He, Lang Feng, HaiYang Xu, Ming Yan, Lei Feng, and Bo An. Beyond trajectory-level attribution: Graph-based credit assignment for agentic reinforcement learning.arXiv preprint arXiv:2605.26684, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Proximity-based multi-turn optimization: Practical credit assignment for llm agent training

    Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, and Chang Liu. Proximity-based multi-turn optimization: Practical credit assignment for llm agent training. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), pages 285–307, 2026

    2026

  7. [7]

    Towards efficient online tuning of VLM agents via counterfactual soft reinforcement learning

    Lang Feng, Weihao Tan, Zhiyi Lyu, Longtao Zheng, Haiyang Xu, Ming Yan, Fei Huang, and Bo An. Towards efficient online tuning of VLM agents via counterfactual soft reinforcement learning. InInternational Conference on Machine Learning, 2025

    2025

  8. [8]

    Group-in-group policy optimization for llm agent training

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training. Advances in Neural Information Processing Systems, 38:46375–46408, 2026

    2026

  9. [9]

    Multimodal web navigation with instruction-finetuned foundation models

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetuned foundation models. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=efFmBWioSc

    2024

  10. [10]

    Navigating the digital world as humans do: Universal visual grounding for GUI agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=kxnoqaisCT

    2025

  11. [11]

    Hierarchy-of-groups policy optimization for long-horizon agentic tasks.arXiv preprint arXiv:2602.22817, 2026

    Shuo He, Lang Feng, Qi Wei, Xin Cheng, Lei Feng, and Bo An. Hierarchy-of-groups policy optimization for long-horizon agentic tasks.arXiv preprint arXiv:2602.22817, 2026

    arXiv 2026

  12. [12]

    Buy 4 reinforce samples, get a baseline for free! InICLR 2019 Workshop, 2019

    Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! InICLR 2019 Workshop, 2019

    2019

  13. [13]

    No prompt left behind: Exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping.arXiv preprint arXiv:2509.21880, 2025

    Thanh-Long V Le, Myeongho Jeon, Kim Vu, Viet Lai, and Eunho Yang. No prompt left behind: Exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping.arXiv preprint arXiv:2509.21880, 2025

    arXiv 2025

  14. [14]

    Salt: Step-level advantage assignment for long-horizon agents via trajectory graph

    Jiazheng Li, Yawei Wang, Qiaojing Yan, Yijun Tian, Zhichao Xu, Huan Song, Panpan Xu, and Lin Lee Cheong. Salt: Step-level advantage assignment for long-horizon agents via trajectory graph. InFindings of the Association for Computational Linguistics: EACL 2026, pages 4709–4725, 2026

    2026

  15. [15]

    Embodied agent interface: Benchmarking LLMs for embodied decision making

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking LLMs for embodied decision making. Advances in Neural Information Processing Systems, 37:100428–100534, 2024

    2024

  16. [16]

    Agentic reinforcement learning with implicit step rewards.arXiv preprint arXiv:2509.19199, 2025

    Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, and Jianbin Jiao. Agentic reinforcement learning with implicit step rewards.arXiv preprint arXiv:2509.19199, 2025

    arXiv 2025

  17. [17]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Tspo: Breaking the double homogenization dilemma in multi-turn search policy optimization.arXiv preprint arXiv:2601.22776, 2026

    Shichao Ma, Zhiyuan Ma, Ming Yang, Xiaofan Li, Xing Wu, Jintao Du, Yu Cheng, Weiqiang Wang, Qiliang Liu, Zhengyang Zhou, et al. Tspo: Breaking the double homogenization dilemma in multi-turn search policy optimization.arXiv preprint arXiv:2601.22776, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Ngrpo: Negative-enhanced group relative policy optimization.arXiv preprint arXiv:2509.18851, 2025

    Gongrui Nan, Siye Chen, Jing Huang, Mengyu Lu, Dexun Wang, Chunmei Xie, Weiqi Xiong, Xianzhou Zeng, Qixuan Zhou, Yadong Li, et al. Ngrpo: Negative-enhanced group relative policy optimization.arXiv preprint arXiv:2509.18851, 2025

    arXiv 2025

  20. [20]

    ToolRL: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. ToolRL: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025. 11

    Pith/arXiv arXiv 2025

  21. [21]

    Ride: Rewarding impact-driven exploration for procedurally-generated environments.arXiv preprint arXiv:2002.12292, 2020

    Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally-generated environments.arXiv preprint arXiv:2002.12292, 2020

    Pith/arXiv arXiv 2002

  22. [22]

    Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36:68539–68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36:68539–68551, 2023

    2023

  23. [23]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  24. [24]

    DeepSeekMath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  26. [26]

    ALFWorld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. InInternational Conference on Learning Representations, 2021. URLhttps://openreview.net/forum?id=0IOX0YcCdTn

    2021

  27. [27]

    Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Kimi k1.5: Scaling reinforcement learning with LLMs.arXiv preprint arXiv:2501.12599, 2025

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with LLMs.arXiv preprint arXiv:2501.12599, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024. ISSN 2835-8856

    2024

  30. [30]

    Information gain-based policy optimization: A simple and effective approach for multi-turn llm agents.arXiv preprint arXiv:2510.14967, 2025

    Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying. Information gain-based policy optimization: A simple and effective approach for multi-turn llm agents.arXiv preprint arXiv:2510.14967, 2025

    arXiv 2025

  31. [32]

    Spa-rl: Reinforcing llm agents via stepwise progress attribution.arXiv preprint arXiv:2505.20732, 2025

    Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. Spa-rl: Reinforcing llm agents via stepwise progress attribution.arXiv preprint arXiv:2505.20732, 2025

    Pith/arXiv arXiv 2025

  32. [33]

    Mobile-Agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. Advances in Neural Information Processing Systems, 37:2686–2710, 2024

    2024

  33. [34]

    SWE-RL: Advancing llm reasoning via reinforcement learning on open software evolution.arXiv preprint arXiv:2502.18449, 2025

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. SWE-RL: Advancing llm reasoning via reinforcement learning on open software evolution.arXiv preprint arXiv:2502.18449, 2025

    Pith/arXiv arXiv 2025

  34. [35]

    Agentprm: Process reward models for llm agents via step-wise promise and progress

    Zhiheng Xi, Chenyang Liao, Guanyu Li, Zhihao Zhang, Wenxiang Chen, Binghai Wang, Senjie Jin, Yuhao Zhou, Jian Guan, Wei Wu, et al. Agentprm: Process reward models for llm agents via step-wise promise and progress. InProceedings of the ACM Web Conference 2026, pages 4184–4195, 2026

    2026

  35. [36]

    Watch every step! llm agent learning via iterative step-level process refinement

    Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! llm agent learning via iterative step-level process refinement. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1556–1572, 2024

    2024

  36. [37]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  37. [38]

    WebShop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022. 12

    2022

  38. [39]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Pith/arXiv arXiv 2022

  39. [40]

    DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  40. [41]

    Reinforcement world model learning for llm-based agents.arXiv preprint arXiv:2602.05842, 2026

    Xiao Yu, Baolin Peng, Ruize Xu, Yelong Shen, Pengcheng He, Suman Nath, Nikhil Singh, Jiangfeng Gao, and Zhou Yu. Reinforcement world model learning for llm-based agents.arXiv preprint arXiv:2602.05842, 2026

    arXiv 2026

  41. [42]

    Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning.arXiv preprint arXiv:2510.19807, 2025

    Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, and Jiaya Jia. Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning.arXiv preprint arXiv:2510.19807, 2025

    arXiv 2025

  42. [43]

    put a cool tomato on the countertop

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training language model agents via hierarchical multi-turn rl. InInternational Conference on Machine Learning, pages 62178–62209. PMLR, 2024. 13 7 Proofs of Propositions and Worked Example This appendix is organized to make the theoretical and empirical claims in the main text...

    2024