Pith. sign in

REVIEW 5 major objections 6 minor 44 references

The paper claims that injecting graph centrality into group-based RL — through intrinsic rewards, dynamic discounting, and topology-aware advantage — consistently improves LLM agent success on sparse-reward tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:13 UTC pith:VGWMCXOX

load-bearing objection GEPO integrates graph centrality into LLM agent RL plausibly, but the evidence is thin and the abstract and full text disagree. the 5 major comments →

arxiv 2510.26270 v2 pith:VGWMCXOX submitted 2025-10-30 cs.AI

Graph-Enhanced Policy Optimization in LLM Agent Training

Jiazhen Yuan , Zhike Gong , Jinquan Hang , Zhengbiao Bai , Wei Zhao This is my paper
classification cs.AI
keywords LLM agentsreinforcement learningcredit assignmentstate-transition graphbetweenness centralitysparse rewardsexplorationgraph-enhanced policy optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Group-based reinforcement learning for LLM agents compares whole trajectories and gives every step in a trajectory the same credit, ignoring the fact that some states act as bottlenecks. GEPO builds a state-transition graph from the agent's own sampled trajectories, attaches a betweenness-based criticality score to each state, and converts it into three learning signals: an intrinsic reward for reaching pivotal states, a discount factor that grows when the agent enters a more central state, and an advantage function that weights decisions at critical states more heavily. The paper claims these signals work together: removing any one hurts success rate by one to three percentage points, and removing pairs hurts roughly twice as much as the sum of individual drops. On three text-based agent benchmarks, GEPO reports consistent absolute success-rate gains over strong group-based baselines, from around +1 to +11 percentage points depending on benchmark and model scale, with lower across-seed variance. A sympathetic reader would care because the method offers a parameter-light way to inject structural priors into sparse-reward LLM agent training without handcrafted dense rewards.

Core claim

The central claim is that structural blindness—treating the environment's state space as an undifferentiated set—is a major cause of poor exploration, imprecise credit, and myopic planning in group-based RL for LLM agents, and that it can be fixed by an online state-transition graph. The paper's framework constructs the graph from the agent's own experience, merging text observations by embedding similarity, computes betweenness centrality as a proxy for whether a state lies on many shortest paths, and defines a Task-Conditioned Criticality score that combines this centrality with semantic similarity to the task prompt. That score feeds three mechanisms: a dense intrinsic reward, a dynamic d

What carries the argument

The load-bearing object is a dynamically constructed state-transition graph. Textual observations are embedded and merged into nodes when cosine similarity exceeds a threshold, and transitions between them become edges; node and edge betweenness centrality (how often each lies on shortest paths) is then treated as strategic importance. This graph produces a Task-Conditioned Criticality score, and through it three learning signals: an intrinsic reward that adds centrality to the sparse extrinsic reward, a dynamic discount factor that clips a baseline discount scaled by the tanh of the centrality change, and a graph-enhanced return used for both trajectory-level and state-level advantage. The

Load-bearing premise

The load-bearing premise is that betweenness centrality computed on a graph built from the agent's own trajectory samples marks the states that are genuinely task-critical, so rewarding and discounting toward those states improves learning rather than merely chasing sampling artifacts.

What would settle it

Annotate the true bottleneck states in a text environment (for example, the hallway and doorway states that must be crossed to complete household tasks) and compare them with the highest-centrality nodes in GEPO's graph. If the overlap is no better than chance, or if replacing GEPO's graph-derived intrinsic reward with any dense reward on the same graph reproduces the gains, the central claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • On all three benchmarks and both model scales, GEPO's absolute success rate exceeds the strongest group-based baseline it was compared with, with the largest gain on the strictly ordered procedural-workflow task.
  • Removing any one of the three signals—intrinsic reward, topology-aware advantage aggregation, or dynamic discount—costs 1–3 points; removing pairs costs more than the sum of individual removals, indicating the signals reinforce each other.
  • Betweenness centrality outperforms degree, closeness, and eigenvector centrality as the graph signal, suggesting bottleneck identification, not hub popularity, drives the gain.
  • Across-seed variance is lower for GEPO than for flat group-based methods, implying the graph signal stabilizes training rather than just improving the best run.
  • Increasing the number of rollouts per iteration expands the graph and improves performance up to 16 rollouts, after which gains reverse—so graph scale should be tuned rather than maximized.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the centrality-to-task-bottleneck mapping is correct, the same three signals could be grafted onto other policy-gradient objectives beyond group-based RL, turning any trajectory sampler into a structure-aware learner; this is an extension the paper does not itself claim.
  • A cheaper way to test the mechanism would be to compute centrality only on successful trajectories; if the gains persist, the method is mostly rewarding reachable critical states rather than discovering them through exploration, which would narrow the paper's explanation.
  • The semantic-similarity component of the criticality score suggests the graph could be pruned to task-relevant subgraphs, which might reduce the noise observed when the graph grows too large—an inference, not a paper claim.
  • Because the graph is built only from sampled experience, an agent that explores poorly early on may never visit true bottlenecks; combining GEPO's rewards with an exploration bonus for novel high-degree states could make the structural prior more robust—again, an editorial inference.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Graph-Enhanced Policy Optimization (GEPO), an extension of group-based policy optimization (GRPO/GiGPO) for multi-step LLM agents. GEPO builds an online state-transition graph from sampled trajectories by embedding textual observations with Sentence-BERT and merging states via a cosine threshold δ. Node/edge betweenness centralities are then injected into three learning signals: a structured intrinsic reward (Eq. 4), a dynamic discount factor (Eq. 6), and a trajectory/state-level advantage (Eqs. 8–13). Experiments on ALFWorld, WebShop, and a proprietary Workbench benchmark with Qwen2.5-1.5B/7B and Qwen3 models report consistent success-rate gains, with full-text abstract claims of +4.1% (ALFWorld 7B), +5.3% (WebShop 7B), and +10.9% (Workbench 7B) over GiGPO baselines. Ablations show that each component contributes and that pairwise removals produce drops larger than the sum of individual removals, which the paper interprets as synergy.

Significance. If the centrality signals genuinely identify task-critical bottleneck states, GEPO is a practical and lightweight contribution: it avoids GNNs and pretrained value functions, and the three-signal integration is a plausible way to densify sparse reward in long-horizon LLM-agent tasks. The paper is transparent about computational overhead, includes ablations, and compares centrality choices. However, the current evidence is not sufficient to support the claims as stated. The empirical support rests on three seeds with no significance testing; a proprietary benchmark is used without enough detail for reproduction; and the abstract is inconsistent with the method and with the full-text results. The most load-bearing assumption — that betweenness centrality on the agent's own sampled trajectory graph corresponds to task-criticality — is asserted rather than validated. The idea is promising, but the manuscript needs substantial strengthening before it can be accepted.

major comments (5)
  1. [Abstract vs. §3.2–3.5] The arXiv abstract reports improvements of +1.1%, +3.2%, and +3.8% and describes a 'Task-Conditioned Criticality score' that combines topological betweenness with semantic similarity to the task prompt. The full text reports +4.1%, +5.3%, +10.9% on ALFWorld, WebShop, and Workbench, and Section 3 defines no such score — only node and edge betweenness C_v, C_e. The 'semantic similarity to the task prompt' component never appears in Eqs. 4–13. The two abstracts and the methodology must be reconciled; if the task-conditioned score is part of the method, it must be defined and evaluated, otherwise it should be removed from the abstract.
  2. [§3.2, Eqs. 4, 6, 8, 12] The central claim that empirical betweenness centrality reflects task-critical bottlenecks is not validated. The graph is constructed solely from trajectories sampled by the current policy, with states merged by a cosine threshold δ=0.9. Betweenness on this partial, policy-dependent graph conflates genuine environmental bottlenecks with sampling-coverage artifacts: a rarely visited but task-relevant state will have low centrality, while a frequently revisited but irrelevant state can acquire high centrality. Since C_v scales the intrinsic reward, dynamic discount, and both advantage terms, a systematic mismatch would reinforce coverage artifacts rather than task structure. Provide a ground-truth validation in a known environment (e.g., ALFWorld's underlying room/object graph) and a sensitivity analysis over δ (and, if possible, over the graph-update frequency).
  3. [§4.3, Table 1] Several headline gains are within one standard deviation of the baseline, with only three seeds and no significance test. For example, 7B WebShop success is 80.5±6.7 for GEPO versus 75.2±3.8 for GiGPO w/o std, and 7B ALFWorld is 94.9±3.8 versus 90.8±1.3. With n=3, the statement that GEPO 'consistently improves' is not statistically supported. Report individual seed values, more seeds, or a paired significance test across tasks/seeds. The current Table 1 should be treated as exploratory, not as evidence of consistent gains.
  4. [§4.4, Table 2] The 'super-additive synergy' claim is not supported by the reported statistics. For Workbench 1.5B, the sum of the individual drops (−2.3 and −1.4) is −3.7, while the pairwise removal is −7.2; but this difference-of-differences is computed from means of three seeds with no interaction test and no error propagation. Several single-component ablations overlap with the full model's uncertainty (e.g., 7B WebShop: 77.9±5.2 versus 80.5±6.7). Either provide a formal interaction analysis with appropriate uncertainty estimates, or replace the synergy claim with a more modest statement that the components appear to reinforce each other.
  5. [§4.1.3, Workbench; Appendix E] Workbench is a proprietary benchmark described only informally as a 'simulated business dashboard', and it is the source of the largest reported gain (+10.9% at 7B). This is a serious reproducibility problem: readers cannot assess the task distribution, the evaluation protocol, or whether the gain is due to task-specific properties. In addition, Appendix E reports Qwen3 results without any baseline comparison, so the claim that GEPO generalizes across model families is not demonstrated. Please provide full task definitions and evaluation details for Workbench, and include GiGPO (or GRPO) baselines for the Qwen3 experiments.
minor comments (6)
  1. [Eq. (6) vs. §3.2, Algorithm 1] Eq. (6) uses ΔC_v(t) without normalizing C_v, so the scale of centrality changes depends on graph size and betweenness magnitudes; specify whether C_v is normalized before the tanh. Also, §3.2 says centralities are recomputed when the graph 'doubles' or every K iterations, but Algorithm 1 recomputes them every iteration. Clarify which is actually used.
  2. [Eq. (12)] The '1+C_v(s_t)' scaling assumes C_v is bounded or normalized; with unnormalized betweenness, large centrality values can dominate the local advantage and distort credit assignment. Specify the normalization procedure.
  3. [§3.5] The description of clustering states as those that 'share the identical state observation' is ambiguous. If exact string matching is used, it is inconsistent with the semantic vertex merging in §3.2; if graph-vertex identity is used, say so explicitly.
  4. [§4.5, Table 3] The centrality comparison is limited to ALFWorld and does not consistently separate the metrics: the 7B betweenness result (94.9±3.8) overlaps with eigenvector (92.3±0.9) and degree (91.5±1.8). Support the claim that betweenness is 'most robust' with more seeds or additional environments.
  5. [Figure 3] The figure legend says 'GEPO (200 steps)' and 'GiGPO (150 steps)', but the text compares per-step wall-clock time; explain why the training lengths differ and whether the comparison is apples-to-apples.
  6. [References] Several references are incomplete or duplicated: [1] has no title/authors; [10]/[11], [26]/[27], and [45]/[46] are duplicated entries. Clean up the bibliography.

Circularity Check

0 steps flagged

No significant circularity: graph centrality is an auxiliary reward-shaping signal, and the central claim is evaluated by external task success, not by construction.

full rationale

GEPO's derivation chain is not circular. The graph centrality C_v is computed from the agent's own sampled trajectories (Section 3.2), but it is used only as an auxiliary shaping signal inside the intrinsic reward, dynamic discount, and advantage terms (Eqs. 4-13). No claim is made that the empirical centrality score itself is the benchmark outcome. The central empirical claim—success-rate gains over strong baselines—is evaluated on ALFWorld, WebShop, and Workbench task success (Section 4.3), which is independent of the graph scores used during training. The ablations and centrality-metric comparison (Tables 2 and 3) likewise compare variants against the same external success metric. The paper even explicitly flags the risk that intrinsic rewards could lead to 'merely farming high-centrality regions' (Section 3.3), showing that the authors do not treat centrality reward as identical to the task objective. There are no load-bearing self-citations, no fitted parameter renamed as a prediction, and no uniqueness theorem imported from the authors' prior work. The closest concern—that empirical betweenness is a policy-dependent sampling artifact rather than ground-truth task criticality—is an unvalidated assumption or limitation, not a derivation that reduces to its own inputs. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 1 invented entities

The central claim depends on several hand-set scalar weights (w_node, w_edge, w_gamma, w_struct, lambda, delta, n) plus the domain assumptions that agent-derived betweenness centrality captures task-criticality and that sentence embeddings correctly deduplicate states. The abstract-only Task-Conditioned Criticality construct is not part of the full-text method.

free parameters (7)
  • w_node, w_edge (intrinsic reward weights) = 0.1
    Eq. 4 and Table 6; chosen by hand to balance node/edge centrality rewards; no sensitivity analysis.
  • w_gamma (dynamic discount scaling) = 0.1
    Eq. 6 and Table 6; controls how strongly centrality change alters the discount factor.
  • w_struct (structural advantage weight) = 0.1–0.5
    Eq. 8 and Table 6; balances the return against the trajectory's average state centrality; range given, not per-task values.
  • lambda = 0.5
    Eq. 13 and Table 6; fixed interpolation weight between trajectory-level and state-level advantages.
  • delta (state-merging threshold) = 0.9
    Section 3.2; cosine similarity threshold for merging textual observations; directly determines graph nodes and is not ablated.
  • gamma_base = 0.99
    Eq. 6; baseline discount factor.
  • n (rollouts per group) = 16
    Table 4 and Table 6; number of trajectories per iteration, chosen for near-optimal performance after a hyperparameter scan.
axioms (5)
  • domain assumption Betweenness centrality on the agent-experience graph identifies task-critical bottleneck states.
    Section 3.2 and Section 4.5; the method's core premise is that graph-theoretic betweenness corresponds to states that matter for task completion. This is asserted, not validated against a ground-truth graph.
  • domain assumption Sentence-BERT embeddings with cosine threshold δ=0.9 preserve topological state identity.
    Section 3.2; semantically equivalent states must map to the same node for the graph to be topologically meaningful. No analysis of threshold sensitivity is given.
  • domain assumption The cumulative graph built from on-policy trajectories approximates the true environment state-transition graph.
    Section 3.2 and Section 4.6; the graph is grown only from experiences the current policy collects, so rarely visited but critical states may be missing or have inaccurate centrality.
  • domain assumption Mean-std normalization within groups and state clusters yields unbiased advantage estimates in sparse-reward settings.
    Eqs. 9-11; the z-score normalization assumes the group/cluster return distributions are comparable and that outliers do not dominate.
  • domain assumption Small fixed weights w_node, w_edge, w_gamma, w_struct keep shaped rewards aligned with the true task objective.
    Section 3.3 and Table 6; the authors note that care must be taken so the agent does not farm high-centrality regions, but no formal or empirical guarantee is provided.
invented entities (1)
  • Task-Conditioned Criticality score (abstract only) no independent evidence
    purpose: Abstract claims a state-level score combining topological betweenness with semantic similarity to the task prompt; in the full-text methodology the state score is just node betweenness centrality C_v.
    The abstract introduces this entity as a core mechanism, but Section 3.2-3.5 defines C_v only and never combines it with task-prompt similarity. This is a discrepancy between the abstract and the actual method, not an independently evidenced construct.

pith-pipeline@v1.3.0-alltime-deepseek · 20101 in / 12131 out tokens · 115073 ms · 2026-08-04T07:13:40.578597+00:00 · methodology

0 comments
read the original abstract

Multi-step LLM agents in interactive environments represent a crucial step toward long-horizon decision-making. To train such agents, group-based reinforcement learning is widely adopted, which reinforces trajectories with higher relative performance within the group. However, in most existing methods, every step within a trajectory and every trajectory with the same terminal reward receive identical credit, regardless of their actual contributions. Since different states play different structural roles in an online state-transition graph built from sampled trajectories, their impacts should be differentiated and converted into task-aware credit at both the step and trajectory levels. We therefore present Graph-Enhanced Policy Optimization (GEPO), a framework for dual-level structural credit assignment in multi-step LLM agent training. Specifically, GEPO derives a state-level Task-Conditioned Criticality score that combines topological betweenness on the state-transition graph with semantic similarity to the task prompt. Based on this score, trajectory-level credit is reshaped through a state-adaptive discount, while step-level credit is scaled by the criticality of its successor state. Experimental results show that GEPO outperforms the strongest baselines by 1.1\% in success rate on ALFWorld, 3.2\% on WebShop, and 3.8\% on average across search-augmented QA tasks at the 7B scale. Compared with flat group-based methods, GEPO reduces across-seed variance and concentrates gradient signals on the most critical steps.

Figures

Figures reproduced from arXiv: 2510.26270 by Jiazhen Yuan, Jinquan Hang, Wei Zhao, Zhengbiao Bai, Zhike Gong.

Figure 1
Figure 1. Figure 1: An illustration of how GEPO overcomes structural [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training success rate versus steps for GiGPO (blue/red) and GEPO (green/purple). Solid lines denote the 1.5B model, [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of computational cost per training step [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 41 linked inside Pith

  1. [1]

    [n. d.]. Exploring autonomous agents through the lens of large language models: A review. ([n. d.])

  2. [2]

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting REINFORCE style optimization for learning from Human Feedback in LLMs. (Feb. 2024). arXiv:2402.14740 [cs.LG]

    Pith/arXiv arXiv 2024

  3. [3]

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. 2023. Self-supervised learn- ing from images with a Joint-Embedding Predictive Architecture. (Jan. 2023). arXiv:2301.08243 [cs.CV]

    Pith/arXiv arXiv 2023

  4. [4]

    Elisabetta Bergamini and Henning Meyerhenke. 2015. Fully-dynamic approxi- mation of betweenness centrality. (April 2015). arXiv:1504.07091 [cs.DS]

    Pith/arXiv arXiv 2015

  5. [5]

    Ulrik Brandes. 2001. A faster algorithm for betweenness centrality. https:// snap.stanford.edu/class/cs224w-readings/brandes01centrality.pdf. Accessed: 2025-9-30

    2001

  6. [6]

    Dingyang Chen, Qi Zhang, and Yinglun Zhu. 2024. Efficient sequential decision making with large language models. (June 2024). arXiv:2406.12125 [cs.LG] Graph-Enhanced Policy Optimization in LLM Agent Training Conference’17, July 2017, Washington, DC, USA

    Pith/arXiv arXiv 2024

  7. [7]

    Victor-Alexandru Darvariu, Stephen Hailes, and Mirco Musolesi. 2024. Graph Reinforcement Learning for combinatorial optimization: A survey and unifying perspective. (April 2024). arXiv:2404.06492 [cs.LG]

    Pith/arXiv arXiv 2024

  8. [8]

    Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Fu- ruta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. 2025. Plan- and-Act: Improving planning of agents for long-horizon tasks. (April 2025). arXiv:2503.09572 [cs.CL]

    Pith/arXiv arXiv 2025

  9. [9]

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2025. Group-in-Group Policy Optimization for LLM Agent Training. (Sept. 2025). arXiv:2505.10978 [cs.LG]

    Pith/arXiv arXiv 2025

  10. [11]

    Vincent François-Lavet, Raphael Fonteneau, and Damien Ernst. 2015. How to discount deep reinforcement learning: Towards new dynamic strategies. (Dec. 2015). arXiv:1512.02011 [cs.LG]

    Pith/arXiv arXiv 2015

  11. [12]

    Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. 2023. Multimodal web navigation with instruction-finetuned foundation models. (May 2023). arXiv:2305.11854 [cs.LG]

    Pith/arXiv arXiv 2023

  12. [13]

    Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. 2016. Continuous deep Q-learning with model-based acceleration. (March 2016). arXiv:1603.00748 [cs.LG]

    Pith/arXiv arXiv 2016

  13. [14]

    Simon Hakenes and Tobias Glasmachers. 2025. Deep reinforcement learn- ing based navigation with macro actions and topological maps. (April 2025). arXiv:2504.18300 [cs.LG]

    Pith/arXiv arXiv 2025

  14. [15]

    Rishi Hazra and Luc De Raedt. 2023. Deep Explainable Relational Reinforcement Learning: A neuro-symbolic approach. (April 2023). arXiv:2304.08349 [cs.AI]

    Pith/arXiv arXiv 2023

  15. [16]

    Riashat Islam, Hongyu Zang, Manan Tomar, Aniket Didolkar, Md Mofijul Islam, Samin Yeasar Arnob, Tariq Iqbal, Xin Li, Anirudh Goyal, Nicolas Heess, and Alex Lamb. 2022. Representation learning in deep RL via discrete information bottleneck. (Dec. 2022). arXiv:2212.13835 [cs.LG]

    Pith/arXiv arXiv 2022

  16. [17]

    Mikko Lauri, David Hsu, and Joni Pajarinen. 2022. Partially observable Markov decision processes in robotics: A survey. (Sept. 2022). arXiv:2209.10342 [cs.RO]

    Pith/arXiv arXiv 2022

  17. [18]

    Yufei Lin, Chengwei Ye, Huanzhen Zhang, Kangsheng Wang, Linuo Xu, Shuyan Liu, and Zeyu Zhang. 2025. CCL: Collaborative curriculum learning for sparse- reward multi-agent reinforcement learning via co-evolutionary task evolution. (May 2025). arXiv:2505.07854 [cs.AI]

    Pith/arXiv arXiv 2025

  18. [19]

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, Rongcheng Tu, Xiao Luo, Wei Ju, Zhiping Xiao, Yifan Wang, Meng Xiao, Chenwu Liu, Jingyang Yuan, Shichang Zhang, Yiqiao Jin, Fan Zhang, Xian Wu, Hanqing Zhao, Dacheng Tao, Philip S Yu, and Ming Zhang. 2025. Large Language Model agent: A ...

    Pith/arXiv arXiv 2025

  19. [20]

    Brielen Madureira and David Schlangen. 2020. An overview of natural language state representation for Reinforcement Learning. (July 2020). arXiv:2007.09774 [cs.CL]

    Pith/arXiv arXiv 2020

  20. [21]

    Aleksandra Malysheva, Daniel Kudenko, and Aleksei Shpilman. 2020. MAGNet: Multi-agent graph network for deep multi-agent reinforcement learning. (Dec. 2020). arXiv:2012.09762 [cs.LG]

    Pith/arXiv arXiv 2020

  21. [22]

    Ben McClusky. 2024. Dynamic graph communication for decentralised multi- agent reinforcement learning. (Dec. 2024). arXiv:2501.00165 [cs.MA]

    Pith/arXiv arXiv 2024

  22. [23]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large Language Models: A survey. (Feb. 2024). arXiv:2402.06196 [cs.CL]

    Pith/arXiv arXiv 2024

  23. [24]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human fe...

    Pith/arXiv arXiv 2022

  24. [25]

    Xiaomin Ouyang and Mani Srivastava. 2024. LLMSense: Harnessing LLMs for High-level reasoning over spatiotemporal sensor traces. (March 2024). arXiv:2403.19857 [cs.AI]

    Pith/arXiv arXiv 2024

  25. [27]

    André Quadros, Cassio Silva, and Ronnie Alves. 2025. LLM-driven in- trinsic motivation for sparse reward reinforcement learning. (Aug. 2025). arXiv:2508.18420 [cs.LG]

    Pith/arXiv arXiv 2025

  26. [28]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your language model is secretly a reward model. (May 2023). arXiv:2305.18290 [cs.LG]

    Pith/arXiv arXiv 2023

  27. [29]

    Ramya Ramakrishnan, Ece Kamar, Debadeepta Dey, Julie Shah, and Eric Horvitz. 2018. Discovering blind spots in reinforcement learning. (May 2018). arXiv:1805.08966 [cs.LG]

    Pith/arXiv arXiv 2018

  28. [30]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. (July 2017). arXiv:1707.06347 [cs.LG]

    Pith/arXiv arXiv 2017

  29. [31]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y K Li, Y Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. (Feb. 2024). arXiv:2402.03300 [cs.CL]

    Pith/arXiv arXiv 2024

  30. [32]

    Archit Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar. 2024. A critical evaluation of AI feedback for aligning large language models. (Feb. 2024). arXiv:2402.12366 [cs.LG]

    Pith/arXiv arXiv 2024

  31. [33]

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. (March 2023). arXiv:2303.11366 [cs.AI]

    Pith/arXiv arXiv 2023

  32. [34]

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. ALFWorld: Aligning text and embod- ied environments for interactive learning. (Oct. 2020). arXiv:2010.03768 [cs.CL]

    Pith/arXiv arXiv 2020

  33. [35]

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. (June 2024). arXiv:2406.01014 [cs.CL]

    Pith/arXiv arXiv 2024

  34. [36]

    Mingyang Wang, Zhenshan Bing, Xiangtong Yao, Shuai Wang, Hang Su, Chenguang Yang, Kai Huang, and Alois Knoll. 2023. Meta-reinforcement learning based on Self-Supervised task representation learning. (April 2023). arXiv:2305.00286 [cs.LG]

    Pith/arXiv arXiv 2023

  35. [37]

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. 2025. RAGEN: Understanding self-evolution in LLM agents via multi-turn reinforcement learning. (April 2025). arXiv:2504.2...

    Pith/arXiv arXiv 2025

  36. [38]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. (June 2022). arXiv:2206.07682 [cs.CL]

    Pith/arXiv arXiv 2022

  37. [39]

    Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, Wei He, Yiwen Ding, Guanyu Li, Zehui Chen, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Tao Gui, Zuxuan Wu, Qi Zhang, Xuanjing Huang, and Yu-Gang Jiang. 2025. AgentGym-RL: Training LLM agents for long-horizon decision mak...

    Pith/arXiv arXiv 2025

  38. [40]

    Siheng Xiong, Ali Payani, Yuan Yang, and Faramarz Fekri. 2025. Deliberate reasoning in language models as Structure-aware planning with an Accurate World Model. (Aug. 2025). arXiv:2410.03136 [cs.CL]

    Pith/arXiv arXiv 2025

  39. [41]

    Jiaxi Yang, Mengqi Zhang, Yiqiao Jin, Hao Chen, Qingsong Wen, Lu Lin, Yi He, Weijie Xu, James Evans, and Jindong Wang. 2025. Topological structure learning should be A research priority for LLM-based Multi-Agent Systems. (May 2025). arXiv:2505.22467 [cs.MA]

    arXiv 2025

  40. [42]

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. WebShop: Towards scalable real-world web interaction with grounded language agents. (July 2022). arXiv:2207.01206 [cs.CL]

    Pith/arXiv arXiv 2022

  41. [43]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. ReAct: Synergizing reasoning and acting in language models. (Oct. 2022). arXiv:2210.03629 [cs.CL]

    Pith/arXiv arXiv 2022

  42. [44]

    Wangyang Ying, Haoyue Bai, Kunpeng Liu, and Yanjie Fu. 2024. Topology- aware reinforcement feature space reconstruction for graph data. (Nov. 2024). arXiv:2411.05742 [cs.LG]

    Pith/arXiv arXiv 2024

  43. [46]

    Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. 2018. Relational deep reinforcement learning. (June 2018). arXiv:1806.01830 [cs.LG]

    Pith/arXiv arXiv 2018

  44. [47]

    structural blindness

    Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang, Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Michael Littman, Jun Wang, Shuicheng Yan, Philip Torr, and Lei Bai. 2025. The landscape of agentic reinforcemen...

    Pith/arXiv arXiv 2025