Pith. sign in

REVIEW 2 major objections 8 minor 110 references

Selective RL that targets only the steps where agents lose the plot beats full-trajectory baselines on long-horizon tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 10:49 UTC pith:XBLQGQCO

load-bearing objection Clean, well-ablated selective RL recipe that actually reduces trajectory neglect and beats GiGPO on the usual agent benches; the single-visit collapse of normalized entropy is real but already owned. the 2 major comments →

arxiv 2607.04963 v1 pith:XBLQGQCO submitted 2026-07-06 cs.AI

STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

Qiuyi Qi , Tian Liang , Mutian Bao , Jinjian Zhang , Dongnan Liu , Wei Zhou , Linjian Mo , Ming Kong
show 3 more authors
Jie Liu Feng Zhang Qiang Zhu
This is my paper
classification cs.AI
keywords LLM agentstrajectory neglectnormalized entropygroup-based RLselective optimizationlong-horizon tasksALFWorldWebShop
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-horizon LLM agents trained with sparse rewards often stop attending to the goal and interaction history at intermediate steps; the paper calls this trajectory neglect and shows it produces low-quality actions. Shannon entropy, the usual uncertainty signal, mixes true state complexity with the agent’s own confidence and therefore mis-labels which steps are the problem. The authors replace it with normalized entropy: within each batch they group steps that share the same anchor state, then measure how far each step’s entropy sits from that state’s mean. Outliers on this scale reliably mark trajectory neglect. STAPO then optimizes only those outliers with a dual signal—a reward that forces the policy to depend on the full trajectory context, plus a penalty that keeps the trajectory-blind policy close to a reference so the agent cannot simply collapse. On ALFWorld, WebShop and multi-turn search QA the method raises success rates over strong group-based RL baselines while cutting the fraction of neglectful steps.

Core claim

Normalized entropy, computed by z-scoring step-level Shannon entropy inside each anchor-state group, isolates the confidence deviations that actually signal trajectory neglect; selectively applying a trajectory-aware reward and a trajectory-independent KL penalty only to those outlier steps yields higher success rates and fewer neglectful actions than optimizing every step indiscriminately.

What carries the argument

Normalized entropy Hn (Eq. 2): for each action, subtract the mean Shannon entropy of all actions that share the same anchor state and divide by that group’s standard deviation; IQR fences on Hn then produce the binary outlier mask that gates STAPO’s selective terms.

Load-bearing premise

Every important state must appear often enough inside a single training batch for the group mean and standard deviation of entropy to be stable; rare states collapse the statistic back to raw Shannon entropy and can mis-flag outliers.

What would settle it

Train the same base models on ALFWorld or WebShop while deliberately forcing every anchor-state group size to 1 (so normalized entropy equals raw Shannon entropy) and check whether the success-rate and outlier-reduction gains over GiGPO disappear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. The paper identifies trajectory neglect—loss of focus on goal and interaction history under sparse delayed rewards—as a driver of low-quality intermediate actions in long-horizon LLM agents. It argues that raw Shannon entropy conflates state complexity with agent confidence, and proposes normalized entropy H_n (Eq. 2): a within-anchor-state z-score of step entropy that is used with IQR fences (Eq. 3) to flag outlier steps. STAPO then applies a selective objective only at those steps: a trajectory-aware reward R_TA (KL between full-prompt and trajectory-blind action distributions, Eq. 4) plus a trajectory-independent penalty P_TI (KL of the masked policy to a reference, Eq. 5), on top of a GiGPO-style two-level advantage. Experiments on ALFWorld, WebShop, and search-augmented QA (Tables 1–2), plus ablations (Table 3, Fig. 4), attribution analysis (Fig. 5), cold-start curves (Fig. 6), and limited VLM pilots, report consistent gains over GiGPO/EMPG and reduced outlier rates.

Significance. If the results hold, STAPO is a practical, modular improvement to group-based RL for agentic post-training: selective step-level supervision without an extra critic, with clear ablations separating localization, masking, and dual-KL terms. Strengths include multi-seed reporting, in- and out-of-domain QA, sensitivity to λ and α/γ, attribution/KL diagnostics (Appendix K), generalization to DAPO-style dynamic sampling (Table 4), and cross-architecture checks (Table 5). The cold-start exploration signal from R_TA (Fig. 6) is a useful empirical observation. The work is incremental relative to GiGPO but addresses a concrete failure mode with a falsifiable localization hypothesis and external success metrics, which is valuable for the agent RL community.

major comments (2)
  1. §3.2 Eqs. (2)–(3) and Limitations: The central premise is that H_n decouples state complexity from confidence and therefore better locates trajectory neglect than Shannon entropy. The paper correctly notes that when an anchor state appears once in the batch, µ/σ are undefined and H_n collapses to raw Shannon entropy. This is load-bearing for the localization claim, yet the manuscript never reports the empirical distribution of |G_S(˜s)| (or the fraction of steps with |G_S|=1) on ALFWorld, WebShop, or QA under the stated N and grouping rules. Without those statistics—and ideally a breakdown of how often outliers are decided under |G_S|≥2 vs. collapse—the advantage of normalized over Shannon entropy (Table 3 top block) cannot be calibrated, and the Limitations paragraph remains unquantified. Please add batch-level group-size histograms or revisit rates and, if collapse is frequent, a mitig
  2. §4.2 Eqs. (4)–(5) and §5.3: R_TA maximizes D_KL[π_θ(·|c) ∥ π_θ(·|(c)\T)] only at outliers. The dual term P_TI is intended to block reward hacking / mode collapse on the masked prompt, and Table 3’s penalty ablations are supportive. Still missing is a direct check that gains in R_TA track increased trajectory use rather than spurious sensitivity (e.g., length or formatting differences between full and masked prompts). Appendix K reports higher avg. KL and attribution after STAPO, which helps, but those are post-hoc aggregates. A short controlled analysis—e.g., correlation of per-step ΔR_TA with attribution change, or a length-matched masking control—would make the causal story for selective optimization tighter and address the second axiom in the method’s justification.
minor comments (8)
  1. Abstract / intro line breaks produce artifacts such as “totra-jectory neglect”; clean hyphenation and wording in the camera-ready text.
  2. Table 3 vs. §5.3: the table says “w/ Single Penalty” / “w/ Inc Penalty Coef” while the prose says “Single Reg” / “Inc Reg Coef”. Align terminology with the global KL term β.
  3. Eq. (1) and Eqs. (4)–(5): clarify whether multi-token KL/entropy is a token-average (as for H) or a sequence-level quantity, and whether stop-gradient is applied to the masked branch when maximizing R_TA.
  4. Figure 1 caption and body: “True/False Trajectory Neglect Signal” is illustrative; state explicitly that the figure is schematic, not measured data, to avoid over-reading.
  5. Appendix prompts (Figs. 16–17): typos “anwsers” / “anwser”; fix for reproducibility.
  6. §5.1: ω=1 “with no further tuning” is fine, but briefly note whether α=β=γ equality was re-tuned per domain (0.01 vs 0.001 on QA) or fixed by scale of rewards.
  7. Table 1: on Qwen2.5-1.5B, STAPO trails GiGPO on Clean/Heat while winning overall; a one-sentence discussion of subtask variance would help readers interpret “SOTA” claims.
  8. Related work §2.3: EPO/EMPG/Cui et al. are cited; a short contrast table (signal type × use as regularizer vs. locator) would sharpen the novelty claim for H_n as a localization gate.

Circularity Check

0 steps flagged

No circularity: normalized entropy, selective dual-KL terms, and external success/attribution metrics are independently defined; results are empirical, not forced by construction.

full rationale

STAPO is an empirical RL method paper. Normalized entropy Hn (Eq. 2) is a within-batch z-score of Shannon entropy over anchor-state groups; the IQR mask Mt (Eq. 3) selects steps for the additive selective objective J_sel = Mt · (α R_TA − γ P_TI) (Eqs. 4–5, 10). R_TA and P_TI are KL divergences between full-prompt and trajectory-masked policies (and reference); they do not appear in the definition of Hn, nor does the external outcome reward R(τ) or step advantage AS enter the localization statistic. Success rates (Tables 1–2), outlier proportions (Fig. 5), and attribution/KL diagnostics (Appendix K) are measured on held-out environment rollouts and feature-ablation scores; none of these quantities is algebraically recovered from Hn or the dual-KL terms. Ablations (Table 3, Fig. 4) and the single-visit collapse noted in Limitations are ordinary sensitivity analyses, not definitional reductions. Hierarchical grouping re-uses the GiGPO-style anchor mechanism as a standard baseline component, not a self-citation uniqueness theorem that forces the SOTA claim. No equation equates a reported success rate or “prediction” to a fitted input by construction. Therefore the derivation chain is self-contained against external benchmarks and exhibits zero circularity under the stated criteria.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 4 invented entities

STAPO’s central performance claim rests on standard RL/group-advantage machinery plus a few paper-specific modeling choices: that state-relative entropy outliers mark neglect, that dual KL terms on masked prompts improve trajectory focus without hacking, and that IQR with fixed λ and equal α/β/γ weights are adequate. Free parameters are the usual RL coefficients plus the selective weights; invented entities are the normalized-entropy gate and the two selective terms.

free parameters (4)
  • IQR coefficient λ = 1.5 (default)
    Controls outlier strictness; fixed at Tukey’s 1.5 in main runs and swept in Figure 4. Directly decides which steps receive selective loss.
  • Selective weights α, γ (and global KL β) = 0.01 / 0.001 (task-dependent)
    Scale R_TA and P_TI relative to the base policy gradient; set equal (0.01 on ALFWorld/WebShop, 0.001 on QA) with a small sensitivity sweep under α=γ.
  • Step-level advantage weight ω = 1
    Mixes episode-level and step-level advantages; fixed at 1 following GiGPO with no tuning.
  • Anchor similarity threshold (QA) = 0.9
    Longest-matching-subsequence threshold for free-text state grouping; default 0.9, ablated 0.5–0.9.
axioms (4)
  • domain assumption Steps that share an anchor state key are exchangeable for mean/std normalization of entropy and returns.
    §3.1–3.2; exact-match or similarity grouping is taken as sufficient for unbiased H_n and A_S.
  • ad hoc to paper Outliers under normalized-entropy IQR are preferentially associated with trajectory neglect and low step advantage.
    Empirical premise of §3.3 / Figure 2 that justifies gating selective optimization only on M_t=1.
  • domain assumption Group-based advantage estimation (episode + anchor-state step advantages) is a valid critic-free baseline for multi-turn agent RL.
    Inherited from GRPO/GiGPO; used in Equations 6–7 without re-derivation.
  • ad hoc to paper KL between full-prompt and trajectory-blind action distributions measures useful trajectory dependence rather than spurious prompt sensitivity.
    Definition of R_TA in Equation 4; validated indirectly via attribution and success gains.
invented entities (4)
  • Normalized entropy H_n no independent evidence
    purpose: Decouple state complexity from agent confidence for outlier localization.
    Defined in Equation 2 as intra-group z-score of Shannon entropy; no independent external theory beyond this paper’s empirics.
  • Trajectory-aware reward R_TA no independent evidence
    purpose: Reward dependence of the action distribution on goal and history.
    Equation 4; KL(π_θ(·|c) || π_θ(·|c\T)).
  • Trajectory-independent penalty P_TI no independent evidence
    purpose: Prevent reward hacking / mode collapse on the masked prompt.
    Equation 5; KL of masked policy to reference on trajectory-blind inputs.
  • STAPO selective objective J_sel no independent evidence
    purpose: Gate α R_TA − γ P_TI by the outlier mask inside a group RL objective.
    Equations 8–10; the hierarchical selective framework as a whole.

pith-pipeline@v1.1.0-grok45 · 28263 in / 3294 out tokens · 30061 ms · 2026-07-11T10:49:58.868308+00:00 · methodology

0 comments
read the original abstract

Reinforcement Learning (RL) is the dominant paradigm for training Large Language Model (LLM) agents on long-horizon tasks. However, sparse and delayed rewards often lead to trajectory neglect, in which agents lose focus on the task goal and interaction history at intermediate steps. Prior work has explored step-level supervision using Shannon-entropy-based uncertainty signals, which conflate inherent state complexity with agent confidence and therefore provide unreliable estimates of decision reliability. To address this issue, we propose normalized entropy, which measures confidence deviations relative to an agent's average behavior under a given state, thereby strengthening the association between low-quality actions and trajectory neglect. Building on this insight, we introduce Selective Trajectory-Aware Policy Optimization (STAPO), a hierarchical group-based RL framework. STAPO leverages normalized entropy to locate outlier steps associated with trajectory neglect and optimizes them via a joint mechanism of trajectory-aware reward and trajectory-independent penalty, enhancing trajectory awareness while preserving training stability. Extensive experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate that STAPO achieves state-of-the-art performance while substantially alleviating trajectory neglect, validating its effectiveness and robustness for agentic tasks.

Figures

Figures reproduced from arXiv: 2607.04963 by Dongnan Liu, Feng Zhang, Jie Liu, Jinjian Zhang, Linjian Mo, Ming Kong, Mutian Bao, Qiang Zhu, Qiuyi Qi, Tian Liang, Wei Zhou.

Figure 1
Figure 1. Figure 1: Illustration of the differences between Shan￾non entropy and normalized entropy in locating trajec￾tory neglect. capable of executing complex, long-horizon tasks, such as embodied control (Shridhar et al., 2021; Li et al., 2024) and web navigation (Furuta et al., 2024; Zheng et al., 2024; Gou et al., 2025). To enable long-horizon reasoning in LLM agents, recent work has increasingly adopted Re￾inforcement … view at source ↗
Figure 2
Figure 2. Figure 2: Empirical analysis on ALFWorld and Web￾Shop. Markers denote the average step-level advantage and attribution score for outliers and inliers across vari￾ous models. hard clustering (exact match or greedy maximum￾similarity assignment), so no step simultaneously belongs to multiple groups. We then utilize the an￾chor state grouping mechanism (Feng et al., 2025) to aggregate actions from the same state across… view at source ↗
Figure 3
Figure 3. Figure 3: The framework of STAPO. It dynamically locates outlier steps utilizing normalized entropy statistics and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study of localization sensitivity. “All” [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Attribution analysis on ALFWorld and Web [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sensitivity analysis of weighting coefficients [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Illustration of task goal neglect in multi-turn interactions. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Illustration of interaction history neglect in multi-turn interactions. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Illustration of normalized entropy avoiding false positives in states with a large action space. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Illustration of normalized entropy reducing false negatives in states with a small action space. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The complete prompt template of ALFWorld agents. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: The trajectory-blind prompt template of ALFWorld agents. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: The complete prompt template of WebShop agents. [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: The trajectory-blind prompt template of WebShop agents. [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: The complete prompt template of Search agents. [PITH_FULL_IMAGE:figures/full_fig_p022_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: The trajectory-blind prompt template of Search agents. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

110 extracted references · 27 linked inside Pith

  1. [1]

    2018 , publisher=

    Reinforcement Learning: An Introduction , author=. 2018 , publisher=

    2018

  2. [2]

    International conference on machine learning , pages=

    Reinforcement learning with deep energy-based policies , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  3. [3]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  4. [4]

    NeurIPS 2024 Workshop on Open-World Agents , year=

    Cradle: Empowering foundation agents towards general computer control , author=. NeurIPS 2024 Workshop on Open-World Agents , year=

    2024

  5. [5]

    Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and others , journal=

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Visual instruction tuning , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    Wang, Junyang and Xu, Haiyang and Jia, Haitao and Zhang, Xi and Yan, Ming and Shen, Weizhou and Zhang, Ji and Huang, Fei and Sang, Jitao , journal=

  8. [8]

    The Dawn of

    Hu, Siyuan and Ouyang, Mingyu and Gao, Difei and Shou, Mike Zheng , journal=. The Dawn of

  9. [9]

    arXiv preprint arXiv:2402.12289 , year=

    Drivevlm: The convergence of autonomous driving and large vision-language models , author=. arXiv preprint arXiv:2402.12289 , year=

    Pith/arXiv arXiv

  10. [10]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: A family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    Pith/arXiv arXiv

  11. [11]

    Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and others , journal=

  12. [12]

    Transactions on Machine Learning Research , issn=

    Voyager: An Open-Ended Embodied Agent with Large Language Models , author=. Transactions on Machine Learning Research , issn=

  13. [13]

    The Twelfth International Conference on Learning Representations , year=

    Multimodal Web Navigation with Instruction-Finetuned Foundation Models , author=. The Twelfth International Conference on Learning Representations , year=

  14. [14]

    arXiv preprint arXiv:2311.15033 , year=

    Agent as Cerebrum, Controller as Cerebellum: Implementing an Embodied LMM-based Agent on Drones , author=. arXiv preprint arXiv:2311.15033 , year=

    Pith/arXiv arXiv

  15. [15]

    2023 , organization=

    Zitkovich, Brianna and Yu, Tianhe and Xu, Sichun and Xu, Peng and Xiao, Ted and Xia, Fei and Wu, Jialin and Wohlhart, Paul and Welker, Stefan and Wahid, Ayzaan and others , booktitle=. 2023 , organization=

    2023

  16. [16]

    arXiv preprint arXiv:2312.13771 , year=

    Appagent: Multimodal agents as smartphone users , author=. arXiv preprint arXiv:2312.13771 , year=

    Pith/arXiv arXiv

  17. [17]

    Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu , booktitle=

  18. [18]

    Hong, Wenyi and Wang, Weihan and Lv, Qingsong and Xu, Jiazheng and Yu, Wenmeng and Ji, Junhui and Wang, Yan and Wang, Zihan and Dong, Yuxiao and Ding, Ming and others , booktitle=

  19. [19]

    Zeng, Aohan and Liu, Mingdao and Lu, Rui and Wang, Bowen and Liu, Xiao and Dong, Yuxiao and Tang, Jie , journal=

  20. [20]

    Advances in Neural Information Processing Systems , volume=

    Fine-tuning large vision-language models as decision-making agents via reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  21. [21]

    Bai, Hao and Zhou, Yifei and Pan, Jiayi and Cemri, Mert and Suhr, Alane and Levine, Sergey and Kumar, Aviral , journal=

  22. [22]

    arXiv preprint arXiv:1910.00177 , year=

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning , author=. arXiv preprint arXiv:1910.00177 , year=

    Pith/arXiv arXiv 1910

  23. [23]

    Advances in Neural Information Processing Systems , volume=

    Android in the Wild: A Large-Scale Dataset for Android Device Control , author=. Advances in Neural Information Processing Systems , volume=

  24. [24]

    Brockman, G , journal=

  25. [25]

    Advances in Neural Information Processing Systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in Neural Information Processing Systems , volume=

  26. [26]

    Advances in Neural Information Processing Systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=

  27. [27]

    International conference on machine learning , pages=

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  28. [28]

    arXiv preprint arXiv:2402.07945 , year=

    Screenagent: A vision language model-driven computer control agent , author=. arXiv preprint arXiv:2402.07945 , year=

    Pith/arXiv arXiv

  29. [29]

    Findings of the Association for Computational Linguistics ACL 2024 , pages=

    You Only Look at Screens: Multimodal Chain-of-Action Agents , author=. Findings of the Association for Computational Linguistics ACL 2024 , pages=

    2024

  30. [30]

    Reinforcing

    Muning Wen and Ziyu Wan and Jun Wang and Weinan Zhang and Ying Wen , booktitle=. Reinforcing. 2024 , url=

    2024

  31. [31]

    International Conference on Machine Learning , pages=

    Grounding large language models in interactive environments with online reinforcement learning , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  32. [32]

    arXiv preprint arXiv:2310.05915 , year=

    Fireact: Toward language agent fine-tuning , author=. arXiv preprint arXiv:2310.05915 , year=

    Pith/arXiv arXiv

  33. [33]

    The Twelfth International Conference on Learning Representations , year=

    True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning , author=. The Twelfth International Conference on Learning Representations , year=

  34. [34]

    Shunyu Yao and Jeffrey Zhao and Dian Yu and Nan Du and Izhak Shafran and Karthik R Narasimhan and Yuan Cao , booktitle=

  35. [35]

    Advances in Neural Information Processing Systems , volume=

    Toolformer: Language Models Can Teach Themselves to Use Tools , author=. Advances in Neural Information Processing Systems , volume=

  36. [36]

    Advances in Neural Information Processing Systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  37. [37]

    Zhang, Kechi and Li, Jia and Li, Ge and Shi, Xianjie and Jin, Zhi , booktitle=

  38. [38]

    Huang and Mustafa Safdari and Yutaka Matsuo and Douglas Eck and Aleksandra Faust , title=

    Izzeddin Gur and Hiroki Furuta and Austin V. Huang and Mustafa Safdari and Yutaka Matsuo and Douglas Eck and Aleksandra Faust , title=. 2024 , url=

    2024

  39. [39]

    Liu, Aixin and Feng, Bei and Xue, Bing and Wang, Bingxuan and Wu, Bochao and Lu, Chengda and Zhao, Chenggang and Deng, Chengqi and Zhang, Chenyu and Ruan, Chong and others , journal=

  40. [40]

    Advances in Neural Information Processing Systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=

  41. [41]

    Advances in Neural Information Processing Systems , volume=

    Unifying count-based exploration and intrinsic motivation , author=. Advances in Neural Information Processing Systems , volume=

  42. [42]

    International conference on machine learning , pages=

    Curiosity-driven exploration by self-supervised prediction , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  43. [43]

    arXiv preprint arXiv:1808.04355 , year=

    Large-scale study of curiosity-driven learning , author=. arXiv preprint arXiv:1808.04355 , year=

    Pith/arXiv arXiv

  44. [44]

    2010 , publisher=

    Modeling purposeful adaptive behavior with the principle of maximum causal entropy , author=. 2010 , publisher=

    2010

  45. [45]

    Tianying Ji and Yongyuan Liang and Yan Zeng and Yu Luo and Guowei Xu and Jiawei Guo and Ruijie Zheng and Furong Huang and Fuchun Sun and Huazhe Xu , booktitle=

  46. [46]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Zhang, Ruoyu and Xu, Runxin and Zhu, Qihao and Ma, Shirong and Wang, Peiyi and Bi, Xiao and others , journal=

  47. [47]

    2025 , url=

    Taiyi Wang and Zhihao Wu and Jianheng Liu and Jianye HAO and Jun Wang and Kun Shao , booktitle=. 2025 , url=

    2025

  48. [48]

    , title =

    Schrader, Max-Philipp B. , title =. GitHub repository , howpublished =. 2018 , publisher =

    2018

  49. [49]

    arXiv preprint arXiv:2503.04697 , year=

    L1: Controlling how long a reasoning model thinks with reinforcement learning , author=. arXiv preprint arXiv:2503.04697 , year=

    Pith/arXiv arXiv

  50. [50]

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Fan, Tiantian and Liu, Gaohong and Liu, Lingjun and Liu, Xin and others , journal=

  51. [51]

    2024 , url=

    OpenAI , title=. 2024 , url=

    2024

  52. [52]

    5 technical report , author=

    Qwen2. 5 technical report , author=. arXiv preprint arXiv:2412.15115 , year=

    Pith/arXiv arXiv

  53. [53]

    Embodied Agent Interface: Benchmarking

    Li, Manling and Zhao, Shiyu and Wang, Qineng and Wang, Kangrui and Zhou, Yu and Srivastava, Sanjana and Gokmen, Cem and Lee, Tony and Li, Erran Li and Zhang, Ruohan and others , journal=. Embodied Agent Interface: Benchmarking

  54. [54]

    Yao, Shunyu and Chen, Howard and Yang, John and Narasimhan, Karthik , journal=

  55. [55]

    Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in

    Ahmadian, Arash and Cremer, Chris and Gall. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year=

  56. [56]

    ICLR 2019 Workshop , year=

    Buy 4 reinforce samples, get a baseline for free! , author=. ICLR 2019 Workshop , year=

    2019

  57. [57]

    arXiv preprint arXiv:2503.20783 , year=

    Understanding R1-Zero-Like Training: A Critical Perspective , author=. arXiv preprint arXiv:2503.20783 , year=

    Pith/arXiv arXiv

  58. [58]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, YK and Wu, Y and others , journal=

  59. [59]

    Chu, Xiangxiang and Huang, Hailang and Zhang, Xiao and Wei, Fei and Wang, Yong , journal=

  60. [60]

    arXiv preprint arXiv:2502.01600 , year=

    Reinforcement Learning for Long-Horizon Interactive LLM Agents , author=. arXiv preprint arXiv:2502.01600 , year=

    Pith/arXiv arXiv

  61. [61]

    Jin, Bowen and Zeng, Hansi and Yue, Zhenrui and Wang, Dong and Zamani, Hamed and Han, Jiawei , journal=

  62. [62]

    arXiv preprint arXiv:1909.08593 , year=

    Fine-tuning language models from human preferences , author=. arXiv preprint arXiv:1909.08593 , year=

    Pith/arXiv arXiv 1909

  63. [63]

    Advances in Neural Information Processing Systems , volume=

    Learning to summarize with human feedback , author=. Advances in Neural Information Processing Systems , volume=

  64. [64]

    Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages=

    Language Understanding for Text-based Games using Deep Reinforcement Learning , author=. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages=

    2015

  65. [65]

    2024 , organization=

    Zhou, Yifei and Zanette, Andrea and Pan, Jiayi and Levine, Sergey and Kumar, Aviral , booktitle=. 2024 , organization=

    2024

  66. [66]

    Putta, Pranav and Mills, Edmund and Garg, Naman and Motwani, Sumeet and Finn, Chelsea and Garg, Divyansh and Rafailov, Rafael , journal=

  67. [67]

    arXiv preprint arXiv:2504.13958 , year=

    Qian, Cheng and Acikgoz, Emre Can and He, Qi and Wang, Hongru and Chen, Xiusi and Hakkani-T. arXiv preprint arXiv:2504.13958 , year=

    Pith/arXiv arXiv

  68. [68]

    Kimi k1.5: Scaling Reinforcement Learning with

    Team, Kimi and Du, Angang and Gao, Bofei and Xing, Bowei and Jiang, Changjiu and Chen, Cheng and Li, Cheng and Xiao, Chenjun and Du, Chenzhuang and Liao, Chonghua and others , journal=. Kimi k1.5: Scaling Reinforcement Learning with

  69. [69]

    Trivedi, Harsh and Khot, Tushar and Hartmann, Mareike and Manku, Ruskin and Dong, Vinty and Li, Edward and Gupta, Shashank and Sabharwal, Ashish and Balasubramanian, Niranjan , booktitle=

  70. [70]

    Nature , volume=

    Mastering the game of go without human knowledge , author=. Nature , volume=. 2017 , publisher=

    2017

  71. [71]

    arXiv preprint arXiv:2010.02903 , year=

    Keep calm and explore: Language models for action generation in text-based games , author=. arXiv preprint arXiv:2010.02903 , year=

    Pith/arXiv arXiv 2010

  72. [72]

    Nature , volume=

    Human-level control through deep reinforcement learning , author=. Nature , volume=. 2015 , publisher=

    2015

  73. [73]

    Lin, Zhihang and Lin, Mingbao and Xie, Yuan and Ji, Rongrong , journal=

  74. [74]

    Mohit Shridhar and Xingdi Yuan and Marc-Alexandre Cote and Yonatan Bisk and Adam Trischler and Matthew Hausknecht , booktitle=

  75. [75]

    Sheng, Guangming and Zhang, Chi and Ye, Zilingfeng and Wu, Xibin and Zhang, Wang and Zhang, Ru and Peng, Yanghua and Lin, Haibin and Wu, Chuan , journal=

  76. [76]

    Qwen2.5-

    Bai, Shuai and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and Song, Sibo and Dang, Kai and Wang, Peng and Wang, Shijie and Tang, Jun and others , journal=. Qwen2.5-

  77. [77]

    Wang, Zihan and Wang, Kangrui and Wang, Qineng and Zhang, Pingyue and Li, Linjie and Yang, Zhengyuan and Yu, Kefan and Nguyen, Minh Nhat and Liu, Licheng and Gottlieb, Eli and others , journal=

  78. [78]

    Sun, Hao and Qiao, Zile and Guo, Jiayan and Fan, Xuanbo and Hou, Yingyan and Jiang, Yong and Xie, Pengjun and Huang, Fei and Zhang, Yan , journal=

  79. [79]

    Navigating the Digital World as Humans Do: Universal Visual Grounding for

    Boyu Gou and Ruohan Wang and Boyuan Zheng and Yanan Xie and Cheng Chang and Yiheng Shu and Huan Sun and Yu Su , booktitle=. Navigating the Digital World as Humans Do: Universal Visual Grounding for

  80. [80]

    Zheng, Boyuan and Gou, Boyu and Kil, Jihyung and Sun, Huan and Su, Yu , journal=

Showing first 80 references.