Pith. sign in

REVIEW 2 major objections 5 minor 45 references

RSPO trains multi-turn LLM agents on true outcome rewards while using dense-process trajectories only for exploration, raising success rates up to 12 percent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 14:42 UTC pith:33D64AP7

load-bearing objection Practical reward-swap loop that uses dense rewards only for exploration then re-labels for outcome-only updates; clean gains on GRPO/PPO/GiGPO, solid ablations, modest novelty. the 2 major comments →

arxiv 2607.04713 v1 pith:33D64AP7 submitted 2026-07-06 cs.LG cs.AI

RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

Qiang Liu , Taian Guo , Ruizhi Qiao , Xing Sun This is my paper
classification cs.LG cs.AI
keywords reward-swap policy optimizationmulti-turn LLM agentssparse outcome rewardsdense process rewardsoff-policy replayALFWorldWebShopGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-turn LLM agents that receive a reward only when a long task ends learn slowly and often never sample the rare successful paths they need. Pure dense intermediate rewards speed learning but can steer the model toward the wrong final objective. RSPO solves both problems by alternating two roles: one agent is briefly trained on dense process rewards so it can explore a wider set of trajectories; those trajectories are then re-labeled with the true sparse outcome reward and mixed, at a small fixed ratio, into the updates of a second agent that is always optimized solely on outcome rewards. The final policy therefore never optimizes a misaligned surrogate, yet still benefits from the diversity that dense rewards make available. On ALFWorld household tasks and WebShop shopping tasks the same plug-in improves GRPO, PPO and GiGPO by several absolute points, with the largest gains on the smaller 1.5 B models.

Core claim

A cyclic reward-swap procedure lets dense process rewards expand the set of trajectories that an outcome-reward policy can learn from, without ever making the final optimization objective depend on those dense rewards. The resulting policy therefore inherits the exploration breadth of process supervision while remaining consistent with the true sparse task reward, producing higher success rates than either pure outcome or pure dense training.

What carries the argument

The reward-swap loop: Agent A is trained for k steps on dense rewards to become Agent B; B rolls out trajectories that are stored in a replay buffer after their rewards are restored to true outcome values; A is then updated off-policy on a mixture of its own on-policy data and high-reward samples drawn from that buffer (α = 1/8), using a generalized clipping center that accounts for the behavior policy of B.

Load-bearing premise

A fixed small fraction of off-policy data plus a simple clip-center shift is enough to keep the distribution of those samples close enough to the current policy that proximal updates stay stable.

What would settle it

Run the identical ALFWorld or WebShop protocol with α increased to 1/2 or higher (or with k large enough that Agent B overfits the dense reward) and check whether success rate falls back to or below the pure-outcome baseline; if it does not, the stability claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Reward-Swap Policy Optimization (RSPO), a cyclic training framework for multi-turn LLM agents under sparse outcome rewards. An agent is first trained for k steps on dense process rewards (Agent B), which then explores and deposits trajectories into a replay buffer; those trajectories are re-labeled with ground-truth outcome rewards and mixed (at fixed ratio α) into off-policy updates of the original agent (Agent A) under a proximal objective (GRPO/PPO/GiGPO) that uses generalized clipping. The final optimization target therefore remains the true sparse reward while the dense-trained explorer expands the set of successful trajectories that can be learned. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B show consistent absolute gains (up to +12 % for 1.5B PPO) over the three baselines, with ablations isolating reward-based sampling, dense-versus-outcome buffer content, dense-reward noise, and the hyperparameters α and k.

Significance. If the reported gains hold under broader evaluation, RSPO supplies a simple, plug-and-play way to exploit imperfect process rewards without permanently biasing the policy toward them—an important practical contribution for long-horizon agent RL where pure outcome rewards are sparse and pure process rewards are misaligned. Strengths include multi-baseline, multi-seed tables, explicit reward-hacking demonstration (Fig. 3), noise-injection and hyper-parameter sensitivity studies (Appendix C–D), and a clear algorithmic description with pseudocode. The method is purely algorithmic and does not rely on circular definitions; the dense reward model is trained once offline and never used for the final objective.

major comments (2)
  1. Section 4.2 and Appendix C: the central stability claim rests on a fixed off-policy fraction α = 1/8 together with generalized clipping (Eq. 2). While the α-sweep correctly shows degradation at 1/4 and 1/2, the paper never measures the actual distribution shift (e.g., KL or TV distance between π_B and π_ heta) nor tests whether the same α remains safe when Agent B is allowed to train longer or when the dense reward model is substantially weaker. A short diagnostic of policy divergence inside each loop would make the load-bearing assumption falsifiable rather than merely empirically tuned.
  2. Table 1 and Section 5.2: absolute gains shrink markedly from 1.5B to 7B (average ~5–6 % → ~2–4 %). The authors attribute this to stronger base exploration, yet supply no quantitative evidence (state-visit diversity, success-trajectory coverage) that the 7B model already saturates the behavioral space that dense rewards open for the 1.5B model. Without such evidence the claim that RSPO “elevates the performance ceiling” remains only partially supported for stronger base models.
minor comments (5)
  1. Figure 1 caption and surrounding text: the phrase “behavioral space guided by outcome rewards” is never formalized; a one-sentence operational definition (e.g., support of the state-action occupancy under pure outcome training) would help readers interpret the schematic.
  2. Section 4.2.3 / Eq. (5)–(7): the dense-reward model is trained with MSE on trajectory sums; it is unclear whether step-level labels are obtained by simple division or by a learned attribution. Clarifying this (or citing the exact SPA-RL procedure) would improve reproducibility.
  3. Table 1: SPEAR reproductions under the authors’ smaller data/epoch budget are markedly worse than the original SPEAR paper; a brief note that the comparison is therefore not apples-to-apples would prevent misinterpretation.
  4. Appendix E reports identical-state counts as a diversity proxy; reporting the complementary metric (unique states visited) would make the exploration claim more transparent.
  5. Minor typos: “GiGPOw/ std” formatting, occasional missing spaces after periods, and inconsistent capitalization of “Agent A/B”.

Circularity Check

0 steps flagged

No significant circularity; RSPO is a self-contained algorithmic framework with empirical validation on external benchmarks.

full rationale

The paper presents an algorithmic training loop (dense-reward exploration by Agent B, reward-swap into an outcome-reward replay buffer, mixed on-/off-policy proximal updates of Agent A) whose objective is explicitly the ground-truth sparse outcome reward (Eqs. 1, 3–4). The dense process reward model is trained once offline via MSE on final returns (Eqs. 5–7) and is used only for trajectory generation, never as the final optimization target; Section 5.4 and Figure 3 explicitly demonstrate reward hacking when dense rewards are used alone, confirming the swap is not definitional. No quantity is fitted to data and then re-presented as a prediction, no uniqueness theorem is imported from the authors, and self-citations are limited to standard baselines (GRPO, PPO, GiGPO) and external dense-reward constructions (SPA-RL). Performance claims rest on independent ALFWorld/WebShop evaluations with multiple seeds and ablations (Tables 1–2, Appendix C–E). The derivation chain therefore contains no reduction of a claimed result to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The central claim rests on standard MDP/RL assumptions plus a handful of hand-chosen hyper-parameters that control the swap cycle; no new physical entities or unproved mathematical axioms are introduced.

free parameters (3)
  • off-policy data ratio α = 1/8
    Fixed at 1/8 throughout main experiments; controls the fraction of replay data mixed into each outcome-reward update.
  • loop steps k = 3
    Number of dense-reward steps and subsequent outcome-reward steps per cycle; set to 3 after a small sweep.
  • train data size split (14 on-policy + 2 off-policy) = 14+2
    Chosen to keep total batch size comparable to baselines while injecting a small amount of diverse trajectories.
axioms (3)
  • domain assumption Multi-turn agent–environment interaction is an MDP with sparse terminal reward.
    Stated in Section 3.1; standard for ALFWorld/WebShop.
  • domain assumption Proximal policy gradient methods (PPO/GRPO) remain stable under a modest amount of off-policy data when generalized clipping is used.
    Invoked in Section 4.2.2 and justified by reference to Queeney et al. (2021).
  • domain assumption A dense process reward model trained by MSE on trajectory returns can serve as a useful (if imperfect) exploration guide.
    Adopted from SPA-RL; validated by the noise-robustness ablation in Appendix D.
invented entities (1)
  • Reward-swap cycle (Agent A ↔ Agent B with outcome re-labeling) no independent evidence
    purpose: To expand the behavioral space of the final outcome-trained policy without ever optimizing the dense surrogate.
    The alternating procedure and the explicit re-labeling step are the paper’s core algorithmic contribution; no independent external evidence is claimed beyond the reported experiments.

pith-pipeline@v1.1.0-grok45 · 19960 in / 2477 out tokens · 23019 ms · 2026-07-11T14:42:39.291048+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning holds significant potential for training large language models (LLMs) to handle multi-turn interactive tasks. However, in long-horizon, multi-turn tasks characterized by sparse outcome rewards, directly training with outcome rewards often results in slow convergence due to the sparsity of signals and the lack of fine-grained feedback. Furthermore, the model may fail to learn successful trajectories that are not sampled during training, thereby limiting its performance. Conversely, while employing customized dense process rewards provides richer signals and accelerates convergence, these surrogate rewards may exhibit potential misalignment with the ground-truth outcome rewards. This inconsistency can bias the training direction and ultimately degrade the model's final performance. In this work, we propose Reward-Swap Policy Optimization (RSPO), a method designed to leverage the rich information from dense process rewards to facilitate training with outcome rewards. By utilizing a reward-swap mechanism, RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards, thereby elevating the performance ceiling of the model. We conduct extensive experiments on two challenging agent benchmarks, WebShop and ALFWorld. By applying our method to various reinforcement learning algorithms, including GRPO, PPO, and GiGPO, we demonstrate that RSPO achieves consistent performance improvements across different baselines and benchmarks.

Figures

Figures reproduced from arXiv: 2607.04713 by Qiang Liu, Ruizhi Qiao, Taian Guo, Xing Sun.

Figure 1
Figure 1. Figure 1: Schematic illustration of the difference between RSPO and baselines. When training with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of RSPO. n and m represent the number of on-policy and off-policy data, respectively. First, Agent A is trained using dense process rewards for k steps to yield Agent B. Subsequently, Agent B interacts with the environment to generate a batch of trajectories, which are stored in a replay buffer with their rewards reverted to the task’s outcome rewards. Next, Agent A is updated using outcome reward… view at source ↗
Figure 3
Figure 3. Figure 3: Experimental results of training exclusively with dense rewards. The blue curve represents [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 30 linked inside Pith

  1. [1]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    2022

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Large language models encode clinical knowledge.Nature, 620(7972):172–180, 2023

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge.Nature, 620(7972):172–180, 2023

    2023

  4. [4]

    Bloomberggpt: A large language model for finance.arXiv preprint arXiv:2303.17564, 2023

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance.arXiv preprint arXiv:2303.17564, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    2022

  6. [6]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Pith/arXiv arXiv 2010

  10. [10]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Spa-rl: Reinforcing llm agents via stepwise progress attribution.arXiv preprint arXiv:2505.20732, 2025

    Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. Spa-rl: Reinforcing llm agents via stepwise progress attribution.arXiv preprint arXiv:2505.20732, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Rlvmr: Reinforce- ment learning with verifiable meta-reasoning rewards for robust long-horizon agents.arXiv preprint arXiv:2507.22844, 2025

    Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, and Xiaolong Li. Rlvmr: Reinforce- ment learning with verifiable meta-reasoning rewards for robust long-horizon agents.arXiv preprint arXiv:2507.22844, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978, 2025

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  15. [15]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu X...

    Pith/arXiv arXiv 2024

  16. [16]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe twelfth International Conference on Learning Representations, 2023

    2023

  17. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 12248–12267, 2024

    2024

  19. [19]

    Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models.arXiv preprint arXiv:2501.03262, 1(3):5, 2025

    Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models.arXiv preprint arXiv:2501.03262, 1(3):5, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models.arXiv preprint arXiv:2310.10505, 2023

    Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models.arXiv preprint arXiv:2310.10505, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945, 2025

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance.arXiv preprint arXiv:2504.14945, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Repo: Replay-enhanced policy optimization.arXiv preprint arXiv:2506.09340, 2025

    Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. Repo: Replay-enhanced policy optimization.arXiv preprint arXiv:2506.09340, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model.arXiv preprint arXiv:2507.06892, 2025

    Jing Liang, Hongyao Tang, Yi Ma, Jinyi Liu, Yan Zheng, Shuyue Hu, Lei Bai, and Jianye Hao. Squeeze the soaked sponge: Efficient off-policy reinforcement finetuning for large language model.arXiv preprint arXiv:2507.06892, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073, 2025

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.arXiv preprint arXiv:2504.20073, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Reinforcing multi-turn reasoning in llm agents via turn-level reward design.arXiv preprint arXiv:2505.11821, 2025

    Quan Wei, Siliang Zeng, Chenliang Li, William Brown, Oana Frunza, Wei Deng, Anderson Schneider, Yuriy Nevmyvaka, Yang Katie Zhao, Alfredo Garcia, and Mingyi Hong. Reinforcing multi-turn reasoning in llm agents via turn-level reward design.arXiv preprint arXiv:2505.11821, 2025

    arXiv 2025

  30. [30]

    Agentic reinforced policy optimization

    Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Agentic entropy-balanced policy optimization.arXiv preprint arXiv:2510.14545, 2025

    Guanting Dong, Licheng Bao, Zhongyuan Wang, Kangzhi Zhao, Xiaoxi Li, Jiajie Jin, Jing- han Yang, Hangyu Mao, Fuzheng Zhang, Kun Gai, et al. Agentic entropy-balanced policy optimization.arXiv preprint arXiv:2510.14545, 2025. 11

    arXiv 2025

  32. [32]

    Learn the ropes, then trust the wins: Self-imitation with progressive exploration for agentic reinforcement learning.arXiv preprint arXiv:2509.22601, 2025

    Yulei Qin, Xiaoyu Tan, Zhengbao He, Gang Li, Haojia Lin, Zongyi Li, Zihan Xu, Yuchen Shi, Siqi Cai, Renting Rui, et al. Learn the ropes, then trust the wins: Self-imitation with progressive exploration for agentic reinforcement learning.arXiv preprint arXiv:2509.22601, 2025

    arXiv 2025

  33. [33]

    Self-imitation learning

    Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. InInterna- tional Conference on Machine Learning, pages 3878–3887. PMLR, 2018

    2018

  34. [34]

    Agent learning via early experience.arXiv preprint arXiv:2510.08558, 2025

    Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, et al. Agent learning via early experience.arXiv preprint arXiv:2510.08558, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Information gain-based policy optimization: A simple and effective approach for multi-turn llm agents.arXiv preprint arXiv:2510.14967, 2025

    Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, and Zhenzhe Ying. Information gain-based policy optimization: A simple and effective approach for multi-turn llm agents.arXiv preprint arXiv:2510.14967, 2025

    arXiv 2025

  36. [36]

    From novice to expert: Llm agent policy optimization via step-wise reinforcement learning.arXiv preprint arXiv:2411.03817, 2024

    Zhirui Deng, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen, Ruibin Xiong, Mang Wang, and Weipeng Chen. From novice to expert: Llm agent policy optimization via step-wise reinforcement learning.arXiv preprint arXiv:2411.03817, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Process reinforcement through implicit rewards

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    Process vs

    Wenlin Zhang, Xiangyang Li, Kuicai Dong, Yichao Wang, Pengyue Jia, Xiaopeng Li, Yingyi Zhang, Derong Xu, Zhaocheng Du, Huifeng Guo, et al. Process vs. outcome reward: Which is better for agentic rag reinforcement learning.arXiv preprint arXiv:2505.14069, 2025

    arXiv 2025

  39. [39]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023

    2023

  40. [40]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  41. [41]

    Generalized proximal policy optimization with sample reuse.Advances in Neural Information Processing Systems, 34: 11909–11919, 2021

    James Queeney, Yannis Paschalidis, and Christos G Cassandras. Generalized proximal policy optimization with sample reuse.Advances in Neural Information Processing Systems, 34: 11909–11919, 2021

    2021

  42. [42]

    High- dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015

    Pith/arXiv arXiv 2015

  43. [43]

    Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  44. [44]

    Soft adaptive policy optimization.arXiv preprint arXiv:2511.20347, 2025

    Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. Soft adaptive policy optimization.arXiv preprint arXiv:2511.20347, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Reward Hacking

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016. 12 A Implementation Details All our experiments are implemented based on VeRL-Agent. [13]. Throughout our experiments, we strictly adhered to the licensing terms for academic use associated with LL...

    Pith/arXiv arXiv 2016