Pith. sign in

REVIEW 3 major objections 5 minor 45 references

Reusing rollout transitions as next-observation prediction on a shared model improves LLM agent RL without extra data or test cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 21:42 UTC pith:QEPYIMW3

load-bearing objection Clean plug-and-play recipe that reuses rollout (s,a,s') for next-obs SL interleaved with GRPO/GiGPO; gains look real, mechanism isolation is the soft spot. the 3 major comments →

arxiv 2607.27973 v1 pith:QEPYIMW3 submitted 2026-07-30 cs.LG cs.AI

TAPO: Transition-Aware Policy Optimization for LLM Agents

Cong Li , Peixi Peng , Yisen Zhao , Xinyu Hu , Shudong Liu , Zhan Su , Zhuojian Li This is my paper
classification cs.LG cs.AI
keywords LLM agentsreinforcement learningtransition supervisionnext-observation predictiongroup-based policy optimizationWebShopALFWorldauxiliary objectives
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard reinforcement learning for language-model agents mostly trains on sparse end-of-task rewards and ignores the dense, action-conditioned feedback the environment already returns after every step. This paper argues that those transitions are a free supervisory signal: by alternating ordinary policy updates with teacher-forced prediction of the next observation given state and action, on the same shared backbone and the same rollouts, the agent becomes more sensitive to action consequences and plans better over long horizons. The method, TAPO, is presented as a lightweight plug-in for existing group-based agent RL algorithms. On WebShop and ALFWorld, across 1.5B and 7B models and two policy optimizers, it consistently raises success rates and scores while adding no expert data, no extra environment samples, and no inference overhead. A sympathetic reader cares because long-horizon agent training is bottlenecked by sparse rewards; turning every interaction into both a policy sample and a dynamics sample is a cheap way to tighten that bottleneck.

Core claim

The paper claims that interleaving standard group-based policy optimization with rollout-derived, action-conditioned next-observation prediction on a shared backbone systematically improves multi-step agent task performance over pure policy optimization, without requiring extra expert data, extra sampling, or any change at test time.

What carries the argument

TAPO: an alternating loop that, every I policy updates, reuses the same (state, action, next-state) triples from RL rollouts to minimize a teacher-forced next-observation loss on the identical model parameters that generate actions.

Load-bearing premise

The performance gains come from the model internalizing genuine action-to-next-observation dynamics, not merely from receiving extra supervised gradient steps on the same trajectories.

What would settle it

Train a control that matches TAPO’s extra gradient steps and token budget but replaces next-observation prediction with a non-causal auxiliary loss (for example, reconstructing the current observation or predicting a shuffled next observation); if task success and next-state perplexity no longer improve together, the causal-transition story fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Existing group-based agent RL recipes (GRPO, GiGPO and similar) can be strengthened by periodic next-observation supervision on their own rollouts.
  • Long-horizon agents can acquire useful predictive knowledge of environment feedback without a separate world-model pre-training stage or expert trajectories.
  • Full-process interleaving beats using transition supervision only as an early warm-up.
  • Gains appear across model scales (1.5B and 7B) and both shopping and embodied household benchmarks.
  • The same recipe leaves a modest out-of-domain capability cost that is already partly present under pure policy RL.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If next-observation supervision is the active ingredient, similar free gains should appear in other interactive LLM settings (tool use, multi-turn dialogue, code agents) wherever the environment returns a textual next state.
  • Matching total gradient steps with a non-causal auxiliary loss would cleanly separate “dynamics learning” from “extra supervised training,” a control the paper leaves open.
  • The mild general-capability tax suggests future schedules may need explicit retention objectives or selective transition sampling rather than uniform interleaving.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TAPO, a post-training recipe for LLM agents that interleaves standard group-based policy optimization (GRPO or GiGPO) with an auxiliary action-conditioned next-observation prediction loss on a shared backbone, using only (s_t, a_t, s_{t+1}) triplets already present in RL rollouts. No extra expert data, environment samples, or inference-time modules are required. On WebShop and ALFWorld, with Qwen2.5-1.5B/7B and both GRPO and GiGPO, TAPO reports consistent gains in score/success rate over pure policy-optimization baselines (Table 1), with supporting analyses of alternation interval I (Table 3), full-process vs early-only transition supervision (Table 4), training dynamics (Fig. 3), next-state perplexity (Fig. 4), and a qualitative WebShop trace (Fig. 5). A limited reference comparison to Early Experience and RWML appears in Table 2.

Significance. If the gains hold under tighter controls, TAPO is a practically useful, low-overhead plug-in for agentic RL: it reuses existing rollouts, needs no separate world-model stage, and improves long-horizon success across two standard benchmarks, two model scales, and two algorithms. Strengths include multi-seed reporting (mean±std over 3 seeds), hyperparameter alignment with the GiGPO setup, an explicit algorithm box (Algorithm 1), prompt templates, PPL definition, and an anonymized code link. The connection to auxiliary-prediction RL and recent theory that goal-conditioned agents encode predictive structure is appropriate. The main scientific value is empirical and engineering rather than a new theoretical guarantee.

major comments (3)
  1. [§4.1–4.2, Eq. (6), Algorithm 1, Tables 3–4, Figs. 3–5] The central mechanistic claim—that gains come from internalized (s,a)→s' dynamics rather than extra supervised updates on rollout text—is not isolated. Algorithm 1 with default I=4 implies roughly one transition-supervision update per four RL updates (~25% more parameter updates at fixed 150 iterations and identical environment samples; Computing Details). Table 4 and Fig. 3 show full-process TS beats early-only and rises faster, and Table 3’s I-sweep is non-monotone, but neither matches total gradient steps/wall-clock of an extended pure-GRPO run nor replaces L_TS (Eq. 6) with a non-causal control (e.g., predict shuffled s_{t+1}, or s_{t+1} from s_t without a_t). Fig. 4 (lower PPL) and Fig. 5 (one CoT) are consistent with the story but do not establish causality. Without at least one matched-budget and one non-causal auxiliary control on WebShop (and ideally ALFWorld), the “transition-a
  2. [§5.3, Table 2] Table 2 presents TAPO (93.6%) against Early Experience (82.8%) and RWML (90.1%) on ALFWorld with Qwen2.5-7B, but footnote 1 correctly notes mismatched objectives, schedules, and protocols. The main text still frames TAPO as “competitive with recent closely related” methods. Either move Table 2 to appendix as purely referential, or run a controlled re-implementation under a shared backbone, rollout budget, and evaluation protocol. As written, the comparison is not load-bearing evidence for superiority and risks overclaim.
  3. [Abstract, §5.2, Table 1] Several Table 1 improvements are modest relative to reported variance (e.g., GiGPO 7B ALFWorld 90.8±1.3 → 93.6±1.4; GiGPO 1.5B ALFWorld 86.7±1.7 → 88.4±3.8). With only three seeds, overlapping intervals weaken the “consistently improves … across … all combinations” claim in the Abstract and §5.2. Please report pairwise significance tests or bootstrap CIs, or soften language where gaps are within noise, and clarify whether evaluation uses the same held-out task splits and decoding settings for all methods.
minor comments (5)
  1. [Appendix B.1, §6] Appendix B.1 (GSM8K capability tax: base 59.1, GRPO 57.9, TAPO-GRPO 56.5) is important for the plug-and-play claim and is only briefly flagged in the Conclusion. A short main-text sentence and, if possible, one more OOD probe would help readers weigh the trade-off.
  2. [§4.1.2] Notation: f_θ(s_t, a_t) for next-observation prediction vs π_θ for the policy on the same θ is clear in prose but could be stated once as a shared LM head with different prompt formats (policy tags vs <prediction> tags).
  3. [Figure 3] Fig. 3 axis labels appear as Unicode glyph runs in the manuscript source; ensure the camera-ready figures render readable “Training Steps” / “Episode Mean Reward” / “Eval Success Rate” labels.
  4. [§2.3] Related work correctly distinguishes decoupled world-model pre-stages [19,20] from interleaved auxiliary TS; a one-sentence contrast with classical auxiliary-task RL (UNREAL, SPR) already cited in the introduction would tighten §2.3.
  5. [Algorithm 1, References] Typos/style: “s(modI)” in Algorithm 1 line 15 should be the iteration counter; “F orty-second” in Ref. [14]; arXiv IDs and concurrent-work dates should be double-checked before camera-ready.

Circularity Check

0 steps flagged

No circularity: TAPO is an empirical training recipe; gains are measured task success, not predictions forced by construction or self-citation.

full rationale

The paper’s load-bearing claim is empirical: interleaving group-based RL (GRPO/GiGPO objectives J_RL from env rewards and advantages) with teacher-forced next-observation NLL L_TS on rollout triples (s_t, a_t, s_{t+1}) improves held-out WebShop/ALFWorld success versus pure RL baselines, without extra expert data or inference cost. L_TS is standard conditional likelihood on observed environment feedback already present in rollouts; advantages come from environment returns; neither quantity is defined in terms of the reported success metric, nor is any free parameter fitted to the evaluation target and then relabeled a prediction. Citation [14] (Richens et al., external) supplies only motivational framing that goal-conditioned agents encode predictive structure—it is not used as a uniqueness theorem that forces TAPO’s design or scores. Related concurrent methods (Early Experience, RWML) are compared as external references with an explicit non-head-to-head caveat, not as self-justifying priors. Hyperparameter I and ablations (Tables 3–4, Figs. 3–5) are sensitivity/mechanism checks, not algebraic identities. Methodological gaps (unmatched gradient-step budgets, missing non-causal auxiliary controls) affect causal attribution of *why* gains occur, not circularity of the derivation. The result does not reduce to its inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

Load-bearing content is almost entirely empirical method design plus standard RL/LM assumptions. No new physical entities. Free knobs are the usual RL hyperparameters plus the alternation interval I. Domain assumptions import group-based agent RL, sparse terminal rewards, and the Richens et al. motivation that general goal-conditioned agents encode predictive environment structure—used as inspiration, not as a proved necessity for TAPO’s gains.

free parameters (3)
  • alternation interval I = 4 (default); swept {2,3,4,5,10,20}
    Controls how often transition supervision runs relative to RL updates; default I=4 chosen via sensitivity table on WebShop 1.5B GRPO; central reported gains use this setting.
  • GiGPO step-advantage weight ω = 1
    Balances trajectory- vs step-level advantages when TAPO is stacked on GiGPO; fixed at 1 without tuning per appendix.
  • RL optimization hyperparameters (lr, KL β, group size, temperatures, γ, mini-batch) = lr 1e-6, β_KL 0.01, group 8, 16 groups, γ=0.95, temp 1.0/0.4, etc.
    Inherited from baseline configs to claim fair comparison; still free choices that shape absolute performance of both baseline and TAPO.
axioms (4)
  • domain assumption Standard policy-gradient / group-relative advantage RL (GRPO/GiGPO objectives, clipping, KL to reference) is a valid way to improve LLM agent policies from sparse episode rewards.
    Entire policy phase (Eqs. 1–5, §3.2) rests on this established agentic-RL practice.
  • domain assumption Teacher-forced next-token likelihood on textual next observations is a meaningful proxy for modeling environmental transitions in text-based simulators.
    Defines L_TS (Eq. 6) and the PPL analysis; assumes WebShop/ALFWorld observation strings are the right prediction target.
  • domain assumption Generalization in multi-step goal-directed tasks benefits from predictive knowledge of environmental consequences (invoking Richens, Everitt & Abel 2025).
    Stated motivation in abstract/intro; TAPO does not prove this theorem, it uses it to justify the auxiliary loss.
  • ad hoc to paper Sharing one backbone for policy generation and transition prediction transfers useful representations into action selection without harming the RL objective enough to erase gains.
    Core design choice of TAPO; supported empirically but not derived; failure would collapse the method to a harmless side task.
invented entities (1)
  • TAPO alternating training loop (policy update then conditional L_TS every I steps on shared θ) no independent evidence
    purpose: Name and operationalize the interleaved post-training recipe claimed to improve agents without extra data.
    Not a physical entity but the paper’s central constructed object; independent evidence is the reported benchmark deltas and ablations, not an external measurement.

pith-pipeline@v1.2.0-daily-grok45 · 21419 in / 3509 out tokens · 80820 ms · 2026-07-31T21:42:18.487924+00:00 · methodology

0 comments
read the original abstract

Recently, Reinforcement Learning (RL) has emerged as a crucial paradigm for the post-training of Large Language Model (LLM) agents. However, existing methods predominantly rely on sparse task rewards for policy optimization, failing to fully exploit another class of inherently dense supervisory signals naturally present during online interaction: environmental feedback following action execution. Recent theoretical studies suggest that generalization in multi-step, goal-oriented tasks hinges on predictive knowledge of environmental consequences. Inspired by this, we propose TAPO: Transition-Aware Policy Optimization for LLM Agents, a unified training framework that alternates between policy optimization and transition supervision. Beyond standard RL updates, TAPO repurposes rollout data to apply action-conditioned next-observation prediction supervision on a shared backbone model. This approach enhances the model's sensitivity to environmental transition dynamics and action consequences while concurrently optimizing the policy. It serves as a computationally lightweight, plug-and-play enhancement module for existing agent RL algorithms, requiring no additional expert data, extra sampling costs, or inference-time overhead. We conduct systematic experiments on WebShop and ALFWorld, integrating foundation models of various scales with different policy optimization algorithms. Empirical results demonstrate that TAPO consistently improves task performance over pure policy optimization baselines.

Figures

Figures reproduced from arXiv: 2607.27973 by Cong Li, Peixi Peng, Shudong Liu, Xinyu Hu, Yisen Zhao, Zhan Su, Zhuojian Li.

Figure 1
Figure 1. Figure 1: Comparison between standard agentic RL and TAPO. Standard RL mainly learns from [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of TAPO. The LLM agent first interacts with the environment based on the task [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation study of training dynamics on Web [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Case Study of TAPO on WebShop. Instead of greedily selecting an action, the agent [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Quantitative evaluation of tran￾sition modeling. We verify the next-state prediction perplexity (PPL) on a held￾out test set. To ascertain whether the gains from TAPO align with an enhanced capacity to model action-conditioned envi￾ronmental feedback, we construct an offline test set of (st, at, st+1) triplets from WebShop and evaluate the per￾plexity (PPL) of conditional next-observation modeling. We pres… view at source ↗
Figure 6
Figure 6. Figure 6: The policy learning prompt template of WebShop agents. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The Transition Supervision prompt template of WebShop agents. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The policy learning prompt template of ALFWorld agents. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The Transition Supervision prompt template of ALFWorld agents. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 33 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Qwen2.5 technical report.ArXiv, abs/2412.15115, 2024

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin,...

    Pith/arXiv arXiv 2024

  4. [4]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Multimodal web navigation with instruction-finetuned foundation models.arXiv preprint arXiv:2305.11854, 2023

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixi- ang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetuned foundation models.arXiv preprint arXiv:2305.11854, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Embodied agent interface: Benchmarking llms for embodied decision making.Advances in Neural Information Processing Systems, 37:100428–100534, 2024

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking llms for embodied decision making.Advances in Neural Information Processing Systems, 37:100428–100534, 2024

    2024

  8. [8]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  9. [9]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  10. [10]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Fei-Fei Li, Lijuan Wang, Yejin Choi, and Manling Li

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica S. Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Fei-Fei Li, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.ArXiv, abs/2504.20073, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978, 2025

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training.arXiv preprint arXiv:2505.10978, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    General agents need world models

    Jonathan Richens, Tom Everitt, and David Abel. General agents need world models. In F orty-second International Conference on Machine Learning, 2025

    2025

  15. [15]

    Reinforcement learning with unsupervised auxiliary tasks.arXiv preprint arXiv:1611.05397, 2016

    Max Jaderberg, V olodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks.arXiv preprint arXiv:1611.05397, 2016. 10

    Pith/arXiv arXiv 2016

  16. [16]

    Deep- mdp: Learning continuous latent space models for representation learning

    Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Marc G Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. InInternational conference on machine learning, pages 2170–2179. PMLR, 2019

    2019

  17. [17]

    Vlms- guided representation distillation for efficient vision-based reinforcement learning

    Haoran Xu, Peixi Peng, Guang Tan, Yiqian Chang, Luntong Li, and Yonghong Tian. Vlms- guided representation distillation for efficient vision-based reinforcement learning. InPro- ceedings of the Computer Vision and Pattern Recognition Conference, pages 29534–29544, 2025

    2025

  18. [18]

    Data-efficient reinforcement learning with self-predictive representations.arXiv preprint arXiv:2007.05929, 2020

    Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations.arXiv preprint arXiv:2007.05929, 2020

    Pith/arXiv arXiv 2007

  19. [19]

    Agent learning via early experience.ArXiv, abs/2510.08558, 2025

    Kai Zhang, Xiang Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, Jian Xie, Yuxuan Sun, Boyu Gou, Qi Qi, Zihang Meng, Jianwei Yang, Ning Zhang, Xian Li, Ashish Shah, Dat Huynh, Hengduo Li, Zi Xian Yang, Sara Cao, Lawrence Jang, Shuyan Zhou, Jiacheng Zhu, Huan Sun, Jason Weston, Yu Su, and Yifan Wu. Agent ...

    Pith/arXiv arXiv 2025

  20. [20]

    Reinforcement world model learning for llm-based agents.arXiv preprint arXiv:2602.05842, 2026

    Xiao Yu, Baolin Peng, Ruize Xu, Yelong Shen, Pengcheng He, Suman Nath, Nikhil Singh, Jiangfeng Gao, and Zhou Yu. Reinforcement world model learning for llm-based agents.arXiv preprint arXiv:2602.05842, 2026

    arXiv 2026

  21. [21]

    Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents.Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  22. [22]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Pith/arXiv arXiv 2010

  23. [23]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges.arXiv preprint arXiv:2401.07339, 2024

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges.arXiv preprint arXiv:2401.07339, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Webdancer: Towards autonomous information seeking agency.arXiv preprint arXiv:2505.22648, 2025

    Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Gang Fu, Yong Jiang, et al. Webdancer: Towards autonomous information seeking agency.arXiv preprint arXiv:2505.22648, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Websailor: Navigating super-human reasoning for web agent.arXiv preprint arXiv:2507.02592, 2025

    Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent.arXiv preprint arXiv:2507.02592, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  29. [29]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe eleventh international conference on learning representations, 2022

    2022

  30. [30]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023. 11

    2023

  31. [31]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    Pith/arXiv arXiv 1909

  32. [32]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  33. [33]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective.arXiv preprint arXiv:2503.20783, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Agentic reinforced policy optimization

    Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603, 2019

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603, 2019

    Pith/arXiv arXiv 1912

  38. [38]

    Mastering atari with discrete world models.arXiv preprint arXiv:2010.02193, 2020

    Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models.arXiv preprint arXiv:2010.02193, 2020

    Pith/arXiv arXiv 2010

  39. [39]

    Mastering diverse domains through world models.arXiv preprint arXiv:2301.04104, 2023

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models.arXiv preprint arXiv:2301.04104, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Reason- ing with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reason- ing with language model is planning with world model. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154–8173, 2023

    2023

  41. [41]

    Web agents with world models: Learning and leveraging environment dynamics in web navigation.arXiv preprint arXiv:2410.13232, 2024

    Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation.arXiv preprint arXiv:2410.13232, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    Is your llm secretly a world model of the internet? model-based planning for web agents.arXiv preprint arXiv:2411.06559, 2024

    Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your llm secretly a world model of the internet? model-based planning for web agents.arXiv preprint arXiv:2411.06559, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Webevolver: Enhancing web agent self-improvement with coevolving world model

    Tianqing Fang, Hongming Zhang, Zhisong Zhang, Kaixin Ma, Wenhao Yu, Haitao Mi, and Dong Yu. Webevolver: Enhancing web agent self-improvement with coevolving world model. arXiv preprint arXiv:2504.21024, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Rlvmr: Reinforce- ment learning with verifiable meta-reasoning rewards for robust long-horizon agents.arXiv preprint arXiv:2507.22844, 2025

    Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, and Xiaolong Li. Rlvmr: Reinforce- ment learning with verifiable meta-reasoning rewards for robust long-horizon agents.arXiv preprint arXiv:2507.22844, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. 12 A Algorithm Algorithm 1TAPO Require:Initial parameterθ Require:EnvironmentE, ta...

    Pith/arXiv arXiv 2021