Pith. sign in

REVIEW 3 major objections 5 minor 58 references

A graph of past actions, scored by temporal-difference learning, steers LLM agents' best-of-N search toward proven steps and away from known failures, beating vanilla inference scaling by 20.81% in success rate across four benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:42 UTC pith:5S2XCY5K

load-bearing objection Plausible method, but the headline gain is confounded by within-task warm-up and the theory is circular. the 3 major comments →

arxiv 2607.27415 v1 pith:5S2XCY5K submitted 2026-07-29 cs.AI

Bridging Inference-Time Scaling and Episodic Memory with Action-Centric Graphs

Xu Zheng , Chaohao Lin , Zhuomin Chen , Weijieying Ren , Haifeng Chen , Wei Cheng , Dongsheng Luo This is my paper
classification cs.AI
keywords inference-time scalingLLM agentsepisodic memoryaction-centric graphtemporal difference learningbest-of-N samplingin-context learningdual-stream value estimation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the redundant, stateless search of inference-time scaling for LLM agents can be made to remember. It proposes GAMER, which compresses historical interaction trajectories into a directed graph whose nodes are actions, and learns two value scores per action — one for suggestion and one for avoidance — using temporal-difference updates on past rewards. At inference, the agent is prompted with the best historical trajectory, high-value next actions, and a warning list of actions that historically led to failure, biasing the best-of-N sampling distribution. Across four benchmarks and four LLM backbones, the paper reports a 20.81% success-rate and 6.17% progress-rate improvement over the vanilla baseline, at a modest token overhead and far lower cost than most memory baselines. The core bet is that episodic memory should guide computation by changing which actions the LLM considers, not by merely retrieving text.

Core claim

On its own terms, the paper's discovery is that an action-centric graph with dual-stream TD values can act as a persistent surrogate value function for agent inference scaling. Rather than storing raw trajectories for the LLM to re-read, GAMER merges trajectories into a graph where nodes are unique actions, edges are sequential dependencies, and Q+ and Q− values estimate expected success and risk of failure respectively. During inference, the policy receives the historically best trajectory, the top-K successors by Q+, and the top-K successors by most negative Q−, shifting the LLM's probability mass toward proven actions and away from proven dead ends. The theoretical analysis argues that th

What carries the argument

The central mechanism is the Action-Centric Graph, a directed graph whose nodes are distinct actions and whose edges record which action followed which in historical trajectories. On this graph the paper runs dual-stream temporal-difference learning: Q+ accumulates discounted positive reward as expected success, while Q− accumulates a thresholded negative reward as risk of failure, each updated independently with optimistic max or pessimistic min targets over successors. These two value streams generate the suggested-action list and avoid-action list that are injected into the LLM prompt alongside the historically optimal trajectory. The graph converts past logs into a compact, queryable val

Load-bearing premise

The load-bearing premise is that the first 32 trajectories of each test task are a fair warm-up memory for the remaining 32 of the same task, so the reported gain is episodic-memory transfer rather than in-task self-adaptation; the theoretical guarantee additionally assumes the prompt guidance never shifts probability mass from good actions to bad ones, and the paper's Limitations section concedes that too little warm-up can make the method perform below the vanilla baseline.

What would settle it

Re-run Table 1 with a strict holdout protocol: warm-up trajectories come only from earlier tasks, never from the task being evaluated, and every baseline receives the same warm-up data; if GAMER's success-rate advantage over vanilla narrows to non-significance, the transfer claim fails. A cheaper probe: replace the learned Q+ values with random top-K suggestions from the same graph and check whether performance holds, which would indicate the TD value-learning machinery is not carrying the argument.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • An agent can spend the same best-of-N budget and get a higher expected maximum reward, because the sampling distribution is biased toward historically successful action sequences.
  • Token cost stays close to the vanilla baseline (about 1.41M vs 1.17M average tokens per inference for the two open backends), and memory construction and TD updates cost well under a second and a few seconds per task on CPU, because no extra LLM calls are needed for memory.
  • Across all four LLM backbones, the gains are consistent, with the largest relative success-rate gain on the embodied household benchmark, where the paper reports about a 53% relative improvement over vanilla.
  • The scaling curves keep rising at the steepest slope even at N=64, so the paper argues that further inference compute would widen rather than close the gap.
  • The theoretical guarantee implies that a memory-guided policy needs no more samples than the base policy to reach a target reward with fixed confidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because warm-up trajectories come from the first 32 episodes of the same task, a large part of the measured gain may be in-task self-adaptation rather than transferable episodic memory; a cleaner test holds out entire tasks and uses memory only from previously seen tasks.
  • Editorial inference: graph nodes are literal action strings, so the method may not transfer across tasks whose equivalent actions are phrased differently; an embedding-based action-normalization step could extend it to open-vocabulary domains.
  • Editorial inference: the avoidance mechanism is a soft barrier, and with small warm-up sets it can lock the agent into a narrow suboptimal region, consistent with the paper's own limitation statement; a simple fix worth testing is an exploration bonus for low-count nodes during value updates.
  • Editorial inference: comparing GAMER against a memory prompt that injects only the best trajectory and top-K positive actions, without learned Q-values, would isolate how much of the gain actually comes from temporal-difference value learning versus simple retrieval.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GAMER, a memory mechanism for LLM agents that represents past trajectories as an action-centric graph and learns dual Q+ and Q- values via temporal-difference updates. At inference, it injects top-K suggested actions and warned actions into the LLM prompt, aiming to improve best-of-N inference scaling. The main empirical claim is a 20.81% relative success-rate improvement and a 6.17% progress-rate improvement over a vanilla best-of-N baseline, averaged over four benchmarks and four LLMs. The paper also offers a theoretical argument (Theorems 4.2 and 4.3) that memory-guided sampling first-order stochastically dominates base sampling, with proofs in Appendix B.

Significance. If the empirical result is protocol-clean, the dual-stream action-value memory is a practically useful and token-cheap way to reduce redundant search in long-horizon agentic tasks. The benchmark coverage (AlfWorld, ScienceWorld, PDDL, Tool-Query), the four LLM backbones, and the token/time measurements are strengths. However, the current theoretical section is tautological, and the central experiment appears to rely on same-task warm-up rather than cross-task episodic memory transfer, so the claimed transfer value and the 20.81% headline are not yet established. No code is released, which further limits reproducibility of the graph-construction and prompting details.

major comments (3)
  1. [§5 Setups; §3.3] The evaluation protocol tests same-task warm-up, not cross-task episodic transfer. The text states "the first 32 trajectories serve as warm-up knowledge, while our method is applied to the remaining 32 trajectories," and the prompt template in §3.3 says "This is the best historical trajectory for the current task you had." The paper does not state that the vanilla baseline receives the same 32 warm-up trajectories. If vanilla is a stateless best-of-64, the +20.81% relative success-rate gain measures in-task self-adaptation (32 prior attempts on the exact same task before the measured 32 attempts), not the "structurally similar previously solved task" reuse promised in §1. Add a warm-up control for vanilla and the other baselines, or evaluate GAMER with memory built from a disjoint set of tasks.
  2. [§4, Assumption 4.1; Appendix B.1] The theoretical guarantee is circular. Assumption 4.1 already asserts that memory guidance moves probability mass only from the failure set to the success set with no downward movement; Appendix B.1 then assumes Δp(x)≤0 below ϵ and Δp(x)≥0 above ϵ, which is exactly first-order stochastic dominance. The proof integrates this assumed shift and never uses the definitions of Q+ and Q−, the TD updates, or the thresholded reward; the phrase "ϵ-consistent" is undefined. Theorem 4.3 is a standard consequence of FSD and would hold for any memory policy that dominates the base policy. Replace this section with a non-tautological condition linking the learned value estimates to the sampling shift, or remove the theoretical claim.
  3. [Abstract; §1 Contributions] The headline numbers are inconsistent: the Abstract and Table 1 report +20.81%/+6.17% success/progress over vanilla, while §1 states the improvement is 15.4%/5.47%. Since these numbers are the paper's central quantitative claim, the discrepancy must be resolved and the correct values used consistently.
minor comments (5)
  1. [Table 3] The S/A/T abbreviations are confusing: the caption lists S, A, T for Suggested Action, Action to Avoid, and Trajectory, but the rows use +A, +S, +T and the text discusses the components in a different order. Align the labels and the textual description.
  2. [Figure 4] The two y-axes are not labeled in the figure itself; the caption should identify which axis corresponds to SR and which to PR.
  3. [Figure 4(c)] The term "warm-up episodes" is used without definition; clarify whether these are per-task trajectories and whether the baselines are given the same warm-up data.
  4. [Appendix C.2] The Reflexion implementation note says "for each action, we will use the same LLM to generate feedback," which is an unusual adaptation of Reflexion. Please clarify how this differs from the original episodic-reflection setup and whether it affects the comparison.
  5. [General] No code or data release is indicated. For a method whose value depends on the exact graph construction, TD update schedule, and prompt template, providing the code would materially improve reproducibility.

Circularity Check

1 steps flagged

Theoretical FSD guarantee is Assumption 4.1 restated; empirical central claim has independent content but is subject to a same-task warm-up caveat.

specific steps
  1. self definitional [Section 4 (Assumption 4.1, Theorem 4.2) and Appendix B.1]
    "Assumption 4.1 (Monotonic Probability Re-allocation). ... We assume T reallocates probability mass from the failure set Aavoid (where rewards r < ε) to the success set Asuggest (where rewards r > ε′), such that for any reward threshold x, the cumulative mass moved from [x,∞) to (−∞,x) is zero. [B.1:] According to the Dual-Stream strategy, we define a pivot reward threshold ε. The transformation ensures: • For x < ε (Failure regions), Δp(x) ≤ 0. • For x > ε (Success regions), Δp(x) ≥ 0."

    Assumption 4.1 already states that no probability mass moves from high-reward to low-reward outcomes for any threshold x, which is exactly the definition of first-order stochastic dominance. The proof of Theorem 4.2 then simply assumes the same sign pattern on Δp, integrates to obtain Fmem ≤ Fbase, and concludes Xmem ⪰1 Xbase. It never uses the ε-consistency of Q+/Q− or the TD update rule, so the claimed 'guarantee' is the input assumption restated rather than a derived consequence of the graph/TD mechanism. This makes the theoretical derivation self-definitional.

full rationale

The main empirical claim is not forced by the fitted values: the +20.81%/+6.17% result is a measured outcome on later trajectories, with ablations and per-backbone variation, so the central result has independent content beyond the circular theory. A non-circularity caveat is that §5 uses the first 32 trajectories of the same test task as warm-up and §3.3's prompt injects 'the best historical trajectory for the current task you had'; if vanilla received no comparable within-task warm-up, the headline gain conflates in-task self-adaptation with cross-task episodic memory transfer. That is an evaluation-fairness risk rather than a derivation-level circularity. The clear circular step is the theoretical FSD proof: Assumption 4.1 literally states the conclusion of Theorem 4.2, and Appendix B.1 re-asserts the same sign conditions on Δp without deriving them from Q+/Q− or the TD updates. No load-bearing self-citation was found; SFS and G-Memory citations are external baselines/implementations. Score 6 reflects one definitional 'prediction' in the theory while the empirical result remains partly independent, though warm-up dependent.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The central claim depends on several hand-set hyperparameters (α, γ, TD iterations, K) and on the acceptability threshold ϵ_r that defines the negative stream; ϵ_r is never given a value. The method's values are fitted by TD updates on the same test-task trajectories used for evaluation.

free parameters (6)
  • learning rate α = 0.05
    TD update step size set in Appendix C.3; chosen by hand, not derived.
  • discount factor γ = 0.95
    Determines horizon of value propagation; fixed in Appendix C.3.
  • TD iterations per task = 100
    Q-values updated 100 iterations per task (Appendix C.3); no convergence criterion or sensitivity analysis for this choice.
  • suggestion/avoid count K = 3
    Top-K actions injected into the prompt; ablation shows robustness but the value is chosen by hand.
  • acceptability threshold ϵ_r = not stated
    Eq. (2) defines the negative reward using a 'predefined acceptability threshold,' but the paper never reports its value or selection procedure.
  • warm-up trajectory count = 32
    First half of the N=64 budget is used to build memory; the paper itself notes performance improves with more warm-up, so the headline result depends on this choice.
axioms (5)
  • domain assumption Long-horizon agent tasks are well modeled as POMDPs (Section 2).
    Standard formalization; assumes reward and transition signals are available as specified.
  • ad hoc to paper Assumption 4.1: memory guidance moves probability mass from failure actions to success actions with no downward mass movement.
    This is precisely the FSD conclusion; the theorem does not derive it from Q+/Q− estimates.
  • domain assumption Exact action strings can serve as reusable node identities across trajectories/tasks.
    No embedding or semantic equivalence is used; paraphrased actions become different nodes.
  • domain assumption LLMs respond to injected suggestions and warnings by shifting their sampling distribution as intended.
    Assumed by the prompt-integration design; compliance is not measured directly.
  • domain assumption TD updates converge with fixed α=0.05, γ=0.95, 100 iterations.
    No convergence criterion; the graph is non-stationary as trajectories are added.
invented entities (2)
  • Action-Centric Graph no independent evidence
    purpose: Represent historical trajectories as action nodes and temporal edges for queryable memory.
    Representational construct; no external falsifiable handle beyond the benchmark results.
  • Q+ and Q− dual value streams no independent evidence
    purpose: Estimate positive and negative action values to suggest and avoid actions.
    Internal fitted quantities; their validity is only indirectly measured through downstream task performance.

pith-pipeline@v1.3.0-daily-deepseek · 21073 in / 14487 out tokens · 142506 ms · 2026-08-01T07:42:50.699713+00:00 · methodology

0 comments
read the original abstract

Recent advancements in inference-time scaling have significantly unlocked the complex reasoning capabilities of Large Language Models~(LLMs). However, for agents, these approaches suffer from a critical inefficiency, operating in a stateless manner and engaging in redundant search processes. Existing memory mechanisms largely rely on the reasoning capabilities of LLMs, leading to prohibitive computational costs. In this paper, we propose a novel framework, \textit{GAMER}~(Graph-based Action-centric Memory with Episodic Reasoning), that bridges the gap between inference scaling and episodic memory. Our approach models historical reasoning as a dynamic \textit{Action-Centric Graph}. By decoupling the memory mechanism from LLMs, our method can save token/money usage by providing less memory context than memory mechanism baselines. To extract knowledge from the graph effectively, we use a dual-stream Temporal Difference learning mechanism to estimate the positive~(suggestion) and negative~(avoidance) value of action nodes based on past successes and failures. During the inference phase, this learned value function optimizes decision-making bi-directionally, so that positive values provide action suggestions, while negative values indicate high-risk actions. By performing efficient searches on the graph, our method significantly improves the efficiency of inference scaling. Experiments on multiple benchmarks demonstrate that \textit{GAMER} achieves superior performance by \textbf{20.81\%/6.17\%} for success/progress rate compared to vanilla baselines.

Figures

Figures reproduced from arXiv: 2607.27415 by Chaohao Lin, Dongsheng Luo, Haifeng Chen, Wei Cheng, Weijieying Ren, Xu Zheng, Zhuomin Chen.

Figure 1
Figure 1. Figure 1: The overall pipeline of our framework, which can be divided into three stages, including 1. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall inference scaling results across four benchmarks and four LLMs. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average inference time of a task on the benchmark Science￾World with two open-source LLMs. SFS achieves lower token consumption than the vanilla baseline because, in our implementation, trajectory samples are used to replace the original examples. When the sampled trajectories contain fewer tokens than the original examples, SFS naturally incurs a lower token cost. For AWM, the reduced token usage is prima… view at source ↗
Figure 4
Figure 4. Figure 4: Hyperparameter analysis on AlfWorld with Qwen and Llama. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The average results of four LLM backbones on all Benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 22 linked inside Pith

  1. [1]

    Graph of thoughts: Solving elaborate problems with large language models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. InProceedings of the AAAI conference on artificial intelligence, volume 38, pages 17682–17690, 2024

    2024

  2. [2]

    Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Grounding large language models in interactive environments with online reinforcement learning

    Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th International Conf...

    2023

  4. [4]

    Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems, 37:74325–74362, 2024

    Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents.Advances in neural information processing systems, 37:74325–74362, 2024

    2024

  5. [5]

    Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413, 2025

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Plan-and-act: Improving planning of agents for long-horizon tasks

    Lutfi Eren Erdogan, Hiroki Furuta, Sehoon Kim, Nicholas Lee, Suhong Moon, Gopala Anu- manchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks. InForty-second International Conference on Machine Learning, 2025

    2025

  8. [8]

    Large language models empowered agent-based modeling and simulation: A survey and perspectives.Humanities and Social Sciences Communications, 11(1):1–24, 2024

    Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. Large language models empowered agent-based modeling and simulation: A survey and perspectives.Humanities and Social Sciences Communications, 11(1):1–24, 2024

    2024

  9. [9]

    Enhancing llm reasoning with multi-path collaborative reactive and reflection agents.arXiv preprint arXiv:2501.00430, 2024

    Chengbo He, Bochao Zou, Xin Li, Jiansheng Chen, Junliang Xing, and Huimin Ma. Enhancing llm reasoning with multi-path collaborative reactive and reflection agents.arXiv preprint arXiv:2501.00430, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Chatdb: Augmenting llms with databases as their symbolic memory.arXiv preprint arXiv:2306.03901, 2023

    Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chatdb: Augmenting llms with databases as their symbolic memory.arXiv preprint arXiv:2306.03901, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Embodied agent interface: Benchmarking llms for embodied decision making

    Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking llms for embodied decision making. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, 10 e...

    2024

  12. [12]

    A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges.Vicinagearth, 1(1):9, 2024

    Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges.Vicinagearth, 1(1):9, 2024

    2024

  13. [13]

    I-mcts: Enhancing agentic automl via introspective monte carlo tree search.arXiv preprint arXiv:2502.14693, 2025

    Zujie Liang, Feng Wei, Wujiang Xu, Lin Chen, Yuxi Qian, and Xinhui Wu. I-mcts: Enhancing agentic automl via introspective monte carlo tree search.arXiv preprint arXiv:2502.14693, 2025

    arXiv 2025

  14. [14]

    SFS: Smarter code space search improves LLM inference scaling

    Jonathan Light, Yue Wu, Yiyou Sun, Wenchao Yu, Yanchi Liu, Xujiang Zhao, Ziniu Hu, Haifeng Chen, and Wei Cheng. SFS: Smarter code space search improves LLM inference scaling. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  15. [15]

    Agentlite: A lightweight library for building and advancing task-oriented llm agent system.arXiv preprint arXiv:2402.15538, 2024

    Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, et al. Agentlite: A lightweight library for building and advancing task-oriented llm agent system.arXiv preprint arXiv:2402.15538, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Memochat: Tuning llms to use memos for consistent long-range open-domain conversation

    Junru Lu, Siyu An, Mingbao Lin, Gabriele Pergola, Yulan He, Di Yin, Xing Sun, and Yunsheng Wu. Memochat: Tuning llms to use memos for consistent long-range open-domain conversation. arXiv preprint arXiv:2308.08239, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    A survey on vision- language-action models for embodied ai.arXiv preprint arXiv:2405.14093, 2024

    Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision- language-action models for embodied ai.arXiv preprint arXiv:2405.14093, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023

    2023

  19. [19]

    RET-LLM: Towards a general read-write memory for large language models

    Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schuetze. RET-LLM: Towards a general read-write memory for large language models. InICLR 2024 Workshop: How Far Are We From AGI, 2024

    2024

  20. [20]

    OpenAI. Chatgpt. 2022

    2022

  21. [21]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. https://openai.com/index/ gpt-4o-mini-advancing-\cost-efficient-intelligence/, 2024. Accessed: July 31, 2026

    2024

  22. [22]

    Reasoningbank: Scaling agent self-evolving with reasoning memory.arXiv preprint arXiv:2509.25140, 2025

    Siru Ouyang, Jun Yan, I Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T Le, Samira Daruki, Xiangru Tang, et al. Reasoningbank: Scaling agent self-evolving with reasoning memory.arXiv preprint arXiv:2509.25140, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Memgpt: Towards llms as operating systems.arXiv preprint arXiv:2310.08560, 2023

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G Patil, Ion Stoica, and Joseph E Gonzalez. Memgpt: Towards llms as operating systems.arXiv preprint arXiv:2310.08560, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Bernstein

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. InProceed- ings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY , USA, 2023. Association for Computing Machinery

    2023

  25. [25]

    Mutual reasoning makes smaller LLMs stronger problem-solver

    Zhenting Qi, Mingyuan MA, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller LLMs stronger problem-solver. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  26. [26]

    Branch-solve-merge improves large language model evaluation and generation

    Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-merge improves large language model evaluation and generation. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo...

    2024

  27. [27]

    MemInsight: Autonomous memory augmentation for LLM agents

    Rana Salama, Jason Cai, Michelle Yuan, Anna Currey, Monica Sunkara, Yi Zhang, and Yassine Benajiba. MemInsight: Autonomous memory augmentation for LLM agents. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33136– 33152,...

    2025

  28. [28]

    Meminsight: Autonomous memory augmentation for llm agents.arXiv preprint arXiv:2503.21760, 2025

    Rana Salama, Jason Cai, Michelle Yuan, Anna Currey, Monica Sunkara, Yi Zhang, and Yassine Benajiba. Meminsight: Autonomous memory augmentation for llm agents.arXiv preprint arXiv:2503.21760, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    LLMs can reason faster only if we let them

    Bilgehan Sel, Lifu Huang, Naren Ramakrishnan, Ruoxi Jia, and Ming Jin. LLMs can reason faster only if we let them. InForty-second International Conference on Machine Learning, 2025

    2025

  30. [30]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  31. [31]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Pith/arXiv arXiv 2010

  32. [32]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020

    2020

  33. [33]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  34. [34]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    The 2014 international planning competition: Progress and trends.Ai Magazine, 36(3):90–98, 2015

    Mauro Vallati, Lukas Chrpa, Marek Grze ´s, Thomas Leo McCluskey, Mark Roberts, Scott Sanner, et al. The 2014 international planning competition: Progress and trends.Ai Magazine, 36(3):90–98, 2015

    2014

  38. [38]

    Enhancing large language model with self-controlled memory framework

    Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Enhancing large language model with self-controlled memory framework. arXiv preprint arXiv:2304.13343, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    V oyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. V oyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024

    2024

  40. [40]

    Steca: Step-level trajectory calibration for llm agent learning.arXiv preprint arXiv:2502.14276, 2025

    Hanlin Wang, Jian Wang, Chak Tou Leong, and Wenjie Li. Steca: Step-level trajectory calibration for llm agent learning.arXiv preprint arXiv:2502.14276, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024

    2024

  42. [42]

    Scienceworld: Is your agent smarter than a 5th grader?arXiv preprint arXiv:2203.07540, 2022

    Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader?arXiv preprint arXiv:2203.07540, 2022

    Pith/arXiv arXiv 2022

  43. [43]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InThe Eleventh International Conference on Learning Representations, 2023. 12

    2023

  44. [44]

    Agent workflow memory

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. InForty-second International Conference on Machine Learning, 2025

    2025

  45. [45]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Assoc...

    2022

  46. [46]

    Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  47. [47]

    The rise and potential of large language model based agents: A survey.Science China Information Sciences, 68(2):121101, 2025

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey.Science China Information Sciences, 68(2):121101, 2025

    2025

  48. [48]

    An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080, 2021

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference.arXiv preprint arXiv:2111.02080, 2021

    Pith/arXiv arXiv 2021

  49. [49]

    Self-evaluation guided beam search for reasoning.Advances in Neural Information Processing Systems, 36:41618–41650, 2023

    Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning.Advances in Neural Information Processing Systems, 36:41618–41650, 2023

    2023

  50. [50]

    A-mem: Agentic memory for LLM agents

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for LLM agents. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  51. [51]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen2.5 technical report. arXiv:2412.15115 [cs.CL], 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.Ad- vances in neural information processing systems, 36:11809–11822, 2023

    2023

  53. [53]

    Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search.Advances in Neural Information Processing Systems, 37:64735–64772, 2024

    2024

  54. [54]

    G- memory: Tracing hierarchical memory for multi-agent systems

    Guibin Zhang, Muxin Fu, Kun Wang, Guancheng Wan, Miao Yu, and Shuicheng YAN. G- memory: Tracing hierarchical memory for multi-agent systems. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  55. [55]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges.arXiv preprint arXiv:2401.07339, 2024

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges.arXiv preprint arXiv:2401.07339, 2024

    Pith/arXiv arXiv 2024

  56. [56]

    Expel: Llm agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024

    2024

  57. [57]

    Synapse: Trajectory-as-exemplar prompting with memory for computer control.arXiv preprint arXiv:2306.07863, 2023

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control.arXiv preprint arXiv:2306.07863, 2023

    Pith/arXiv arXiv 2023

  58. [58]

    Memorybank: Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731, 2024. 13 A Related Work Memory Mechanisms in LLM Agents.Memory is a core component that enables LLM-based agents to accumulate exper...

    2024