Pith. sign in

REVIEW 3 major objections 5 minor 38 references

LLM agents should reconstruct retrieved memories for the current state, not replay them verbatim.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 12:37 UTC pith:7FHSLHFS

load-bearing objection Solid agent-memory methods paper: learned state-conditioned rewrite beats verbatim replay on ALFWorld/WebShop, with real mechanism probes—but the “emerges from task reward alone / intrinsic reasoning” claim is oversold relative to the cold-start and baseline controls. the 3 major comments →

arxiv 2607.28272 v1 pith:7FHSLHFS submitted 2026-07-30 cs.AI

MemHarness: Memory Is Reconstructed, Not Replayed

Rong Wu , Daocheng Fu , Licheng Wen , Xuemeng Yang , Shu Zou , Jianbiao Mei , Yuxin Wang , Hairong Zhang
show 5 more authors
Yu Yang Tao Hu Cong Zhang Botian Shi Pinlong Cai
This is my paper
classification cs.AI
keywords LLM agentsexperiential memorymemory reconstructionnegative transferGRPOALFWorldWebShopout-of-distribution generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Memory-augmented language-model agents usually paste past experiences straight into the prompt. That “replay” habit often hurts, because a memory that is topically related can still be wrong for the exact situation the agent is in now. MemHarness argues that agents should do what people do: critique a retrieved memory against the present state and rewrite it into guidance that fits before acting. A single policy learns retrieval, reconstruction, and action end-to-end from task success, without labeled rewrite examples. On household and shopping benchmarks the approach beats pure reinforcement learning and static memory-plus-RL baselines, stays stronger when environments shift, and even leaves the agent smarter when memory is turned off at test time—because reconstruction during training acts as latent guidance.

Core claim

The paper’s central claim is that the gap between abstract stored experience and concrete decision-time state is the main reason memory often causes negative transfer, and that inserting an explicit, state-conditioned critique-and-reconstruction step—learned jointly with action via group-relative policy optimization—closes that gap, prevents harmful replay, and improves both online performance and the agent’s intrinsic reasoning.

What carries the argument

Contextual memory reconstruction: at each step a unified policy, given the task, recent history, and retrieved experiences paired with their source observations, critiques applicability and outputs either rewritten state-specific guidance or an empty fallback before generating the action; the whole retrieve–reconstruct–act pipeline is optimized end-to-end with GRPO from sparse task reward plus light format bonuses.

Load-bearing premise

The gains are attributed to learned reconstruction rather than to the cold-start format training, the way the memory bank is filled and curated, or uneven strength of the static memory-plus-RL baselines.

What would settle it

On ALFWorld and WebShop, hold backbone, history window, and RL recipe fixed and compare full MemHarness to the same policy with reconstruction removed (verbatim replay) and with memory disabled: if reconstruction is the driver, removing it should collapse gains toward raw-memory or pure-RL levels, and OOD success should fall when unadapted memories are injected—as the paper’s own ablations claim (e.g., ALFWorld 85.2% vs 79.6% without reconstruction and 76.4% GRPO-only).

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Verbatim memory injection into RL agents is not a free win; without state-conditioned rewrite it can degrade success relative to pure RL.
  • Training agents to reconstruct memory can raise base policy skill even when external memory is unavailable at inference.
  • OOD layouts and object placements benefit more from adaptive rewrite than from replaying historically successful trajectories.
  • A single shared policy can own critique, rewrite, and action without separate reconstruction modules or human rewrite labels.
  • Memory banks remain inspectable records while their use becomes parameterized and state-sensitive.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same reconstruct-before-act pattern may transfer to tool use and multi-agent settings where retrieved plans or messages routinely mismatch the live world state.
  • If reconstruction is latent curriculum, weaker or noisier memory stores might still help training as long as the policy must judge and rewrite them.
  • Product agents that today stuff long retrieval chunks into context may gain more from short rewritten guidance than from larger retrieval budgets.
  • Measuring reject/adapt rates under controlled state edits could become a standard diagnostic for whether an agent is truly state-comparing memories.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that memory-augmented LLM agents suffer negative transfer when retrieved experiences are replayed verbatim, because historical records are often misaligned with the current state. It proposes MemHarness: a unified policy that, at each step, retrieves experiential memories mi=(ei, osrc_i), critiques and reconstructs them into state-conditioned guidance gt (or falls back to self-reasoning via <EMPTY>), then acts. Reconstruction is treated as a latent intermediate and trained end-to-end with GRPO on sparse task reward plus light format shaping, without supervised rewrite labels. On ALFWorld and WebShop with a Qwen2.5-7B backbone, MemHarness reports 85.2% and 75.6% success, beating pure GRPO (76.4%/66.1%) and several static memory+RL baselines, with supporting ablations (raw memory hurts; w/o reconstruction ≈ replay; generic-LLM rewrite weaker), OOD gains, source-state ablations, counterfactual probes, and training-dynamics analyses of accept/reject behavior.

Significance. If the result holds under matched training recipes, the work offers a clear and useful middle ground between explicit memory banks and fully parametric memory: keep memories inspectable while learning state-conditioned critique and rewrite inside the policy. The empirical package is stronger than typical agent-memory papers—component ablations, OOD split, source-state controls, and counterfactual probes form a coherent chain that reconstruction, not retrieval alone, drives the lift and reduces negative transfer. Residual gains when memory is disabled at inference (Table 2) are potentially important if truly caused by the reconstruction objective acting as latent guidance. Code is promised; the framing is falsifiable and practically relevant for long-horizon agents.

major comments (3)
  1. [§4.3 Table 2; Appendix A.4] §4.2–4.3 and Appendix A.4: The load-bearing claim that reconstruction “emerges naturally through end-to-end training with GRPO” and “fundamentally improv[es] the agent’s intrinsic reasoning” (Abstract; Table 2 w/o-memory 83.0% vs GRPO 76.4% on ALFWorld; 73.6% vs 66.1% on WebShop) is not fully secured by a matched-recipe control. Cold-start uses 400 GPT-5.1 trajectories/summaries per benchmark that already force reconstruct-before-act and memory-augmented multi-turn format, while pure GRPO is not clearly stated to receive an identical protocol SFT without the reconstruction stage. Please either (i) report GRPO (and RL+Raw Memory) from the same cold-start checkpoint with reconstruction disabled, or (ii) temper the intrinsic-reasoning claim to what the current controls support.
  2. [§4.1–4.2; Appendix A.3; Table 1] §4.1–4.2 and Appendix A.3: Attribution of gains specifically to reconstruction versus memory-bank construction is incomplete. MemHarness populates B online from its own trajectories with EvolveR-style utility pruning and environment-specific dedup; several static memory+RL baselines collapse (MemRL 24.0% Avg. SR; Mem0+GRPO 52.0%), which is hard to interpret as a fair “verbatim replay + RL” ceiling. A controlled comparison that freezes the same bank B and retriever for MemHarness vs. verbatim injection (already partly in “RL + Raw Memory” / “w/o reconstruction”) should be stated with identical bank growth, top-k, and curation. Without that, part of the headline margin over memory-augmented RL may reflect bank quality or training coupling rather than reconstruction per se.
  3. [§3.3–3.4; §4.5 Table 4; Fig. 4] §3.3–3.4 Eqs. (2)–(6) and reward design: Reconstruction gt is optimized only via trajectory-level sparse outcome (+ format bonus). The paper shows accept/reject correlates with success (Fig. 4) and that osrc matters (Table 4a), which is good, but does not rule out that the policy learns a shallow format of “emit guidance then act” rewarded by the 0.1 format term and retrieval-frequency shaping (1–5 retrieves/episode) rather than genuine state comparison in all regimes. Strengthening this would help: report success stratified by whether gt is non-empty and content-overlapping with ei, and/or an ablation that removes osrc during training (not only at test time as in Table 4a).
minor comments (5)
  1. [Table 1; Table 2] Table 1: EvolveR is listed under “Memory-Augmented RL-based Methods” at 70.1% ALFWorld Avg. SR and is also the “RL + Raw Memory” row in Table 2; make this identity explicit in the main text to avoid double-counting confusion.
  2. [Figure 2; §3.2] Figure 2 vs. §1 five-stage formulation: critique is “folded into” reconstruction; state this once in the figure caption so the three-stage pipeline and five-stage narrative do not appear inconsistent.
  3. [§3.4; Appendix A.1] Appendix A.1 lists four format criteria while §3.4 describes three; align the reward description.
  4. [Title page; References] Typos/formatting: “Y uxin”, “Y u Yang”, “averbatim”, “toretrieve”; Date “July 31, 2026” and venue strings in references look placeholder-like—clean before camera-ready.
  5. [§4.5–4.6] WebShop rejection rates are very high (~56–72% in Table 4); a short discussion of whether this means memory is mostly filtered noise on WebShop would help readers interpret Fig. 3b.

Circularity Check

0 steps flagged

No derivation circularity: external task success is optimized by GRPO; reconstruction is a latent intermediate, not a quantity defined as its own input.

full rationale

MemHarness is an empirical agent-systems paper. The load-bearing claims are environment success rates on ALFWorld/WebShop (and OOD variants), obtained by rolling out a policy in interactive simulators with sparse outcome reward (10/0) plus a small format bonus. Reconstruction g_t is explicitly treated as an unsupervised latent (Eqs. 2–6) and credited only via end-to-end GRPO on those external returns—not redefined as the evaluation metric. Ablations (w/o reconstruction, generic-LLM reconstruction, w/o memory, source-state and counterfactual probes) compare against held-out environment outcomes rather than tautological rewrites of fitted quantities. Self-citations (e.g., EvolveR) supply baselines and memory-bank curation practice; they do not import a uniqueness theorem that forces the headline result. Online self-distillation of the memory bank and GPT-constructed cold-start affect experimental attribution, not circularity of a derivation chain. No self-definitional loop, fitted-input-as-prediction, or renamed known identity is present. Score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Load-bearing commitments are standard POMDP/agent-RL assumptions, the reconstructive-memory analogy as design inspiration (not a proved cognitive theorem), GRPO as the optimizer, and engineering choices for memory curation and cold-start. No new physical entities; free parameters are training/retrieval hyperparameters and reward shaping weights that affect learned behavior.

free parameters (5)
  • Outcome reward and format bonus weights (R_outcome∈{0,10}, 0.1×R_format) = 10 success / 0 failure; format weight 0.1
    Sparse success reward plus hand-weighted format terms (think/action blocks, 1–5 retrievals/episode, English-only) shape credit assignment for reconstruction vs action.
  • GRPO group size, clip ε, KL β, learning rate = G=8, ε=0.2, β=0.01, lr=1e-6
    Optimizer hyperparameters that determine policy updates; chosen by authors, not derived.
  • Retrieval top-k and history window w = k=3, w=3
    Controls how much memory and context enter reconstruction; fixed design choices.
  • Memory dedup/prune thresholds (θ_sim, θ_prune, min_uses) = θ_sim=0.85 (WebShop on), θ_prune=0.3, min_uses=3
    Curation knobs that change bank contents and thus what reconstruction sees; defaults stated in Appendix A.3.
  • Cold-start dataset size and SFT schedule = 400 examples/benchmark; 2 epochs; lr 1e-5
    200 interaction + 200 summarization examples per benchmark, 2 epochs at 1e-5, built with GPT-5.1—affects initial reconstruct protocol.
axioms (5)
  • domain assumption Interactive decision-making is modeled as a POMDP with partial observability handled by a short history window ht.
    §3.1 problem formulation; standard but untested alternatives (full belief state, longer memory) could change results.
  • domain assumption Human remembering is reconstructive rather than reproductive, justifying an explicit critique/reconstruction stage in agents.
    Introduction cites Loftus & Palmer (1974) as inspiration; used as design analogy, not empirically validated transfer from psychology.
  • ad hoc to paper Task-level sparse reward plus light format shaping is sufficient for a unified policy to learn useful latent reconstruction without supervised rewrite labels.
    Core training claim in §3.3–3.4; depends on cold-start format priming and environment structure.
  • domain assumption Group Relative Policy Optimization (GRPO) with trajectory-level advantages assigned to all tokens fairly credits reconstruction and action tokens.
    §3.4; inherits GRPO credit-assignment assumptions from DeepSeek-style RL.
  • domain assumption Semantic embedding retrieval (BGE-M3) plus utility pruning yields memories whose applicability can be judged from (e, osrc) pairs.
    §3.1 and Appendix A.3; retrieval quality bounds reconstruction upside.
invented entities (2)
  • MemHarness reconstruct–act pipeline (unified policy producing gt then at, with <EMPTY>→pself fallback) independent evidence
    purpose: Operationalize state-conditioned memory use between retrieval and action inside one trainable policy.
    Primary system contribution; defined in §3.2–3.3. Not a physical entity—an architectural construct evaluated empirically.
  • Experiential memory entries mi=(ei, osrc_i) with online self-distillation into bank B independent evidence
    purpose: Provide explicit, attributable experiences plus source states for comparison during reconstruction.
    Extends prior memory-bank designs (e.g., EvolveR-style) with source observation paired for critique; evidence is benchmark performance and ablations.

pith-pipeline@v1.2.0-daily-grok45 · 21969 in / 3930 out tokens · 87522 ms · 2026-07-31T12:37:07.791653+00:00 · methodology

0 comments
read the original abstract

Retrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent's current situation. This ``replay'' paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we propose MemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent's intrinsic reasoning capabilities.

Figures

Figures reproduced from arXiv: 2607.28272 by Botian Shi, Cong Zhang, Daocheng Fu, Hairong Zhang, Jianbiao Mei, Licheng Wen, Pinlong Cai, Rong Wu, Shu Zou, Tao Hu, Xuemeng Yang, Yuxin Wang, Yu Yang.

Figure 1
Figure 1. Figure 1: Memory utilization paradigms. Top: Prior methods directly replay retrieved memories, risking state misalignment. Middle: Human memory reconstructs past experience accord￾ing to the current context. Bottom: Inspired by this process, MemHarness reconstructs retrieved memories into state-aligned guidance. Based on this formulation, we introduce MemHarness, designed to actively harness re￾trieved experiences b… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MemHarness framework. The execution pipeline consists of three stages: (1) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Evolution of memory usage behaviors during RL training. The curves show the average number of memory retrievals, accepted reconstructions, and rejected reconstructions per trajectory. 0 40 80 120 Training Steps 0.0 20.0 40.0 60.0 80.0 100.0 Success Rate (%) ALFWorld Success Rate SR when Accepted SR when Rejected (a) ALFWorld 0 40 80 120 Training Steps 20.0 40.0 60.0 80.0 Success Rate (%) WebShop Success Ra… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of success rates conditioned on memory decisions. “SR when Accepted/Rejected” denotes the success rate of trajectories containing at least one accepted or rejected memory reconstruction. Micro-level Counterfactual Adaptation. We construct 1,000 offline probes by asking a strong LLM to minimally edit each current state so that its retrieved memory becomes inapplicable, and evaluate only th… view at source ↗
Figure 5
Figure 5. Figure 5: Agent system prompts used in ALFWorld. C Prompt Templates This section presents the prompt templates used for memory retrieval, state-conditioned memory reconstruction, agent–environment interaction, trajectory summarization, and fallback reasoning when no adapted memory. Text enclosed in braces denotes a runtime placeholder. Environment feedback Prompt for Agent The agent system prompts specify the enviro… view at source ↗
Figure 6
Figure 6. Figure 6: Agent system prompts used in WebShop, with and without recent interaction history. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt used to reconstruct a retrieved memory principle according to the current environ [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt used for offline trajectory summarization. The model distills raw interaction [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt used to generate counterfactual mismatched observations for ALFWorld. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt used to generate counterfactual mismatched observations for WebShop. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: ALFWorld case demonstrating how MemHarness prevents hallucination by rewriting a [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: WebShop case demonstrating how MemHarness translates an abstract heuristic rule into a [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 16 linked inside Pith

  1. [1]

    Introspective tips: Large language model for in-context decision making.arXiv preprint arXiv:2305.11598, 2023

    Liting Chen, Lu Wang, Hang Dong, Yali Du, Jie Yan, Fangkai Yang, Shuang Li, Pu Zhao, Si Qin, Saravan Rajmohan, et al. Introspective tips: Large language model for in-context decision making.arXiv preprint arXiv:2305.11598, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Re-searcher: Robust agentic search with goal-oriented planning and self-reflection

    Daocheng Fu, Jianbiao Mei, Licheng Wen, Xuemeng Yang, Cheng Yang, Rong Wu, Tao Hu, Siqi Li, Yufan Shen, Xinyu Cai, et al. Re-searcher: Robust agentic search with goal-oriented planning and self-reflection. arXiv preprint arXiv:2509.26048, 2025

    arXiv 2025

  3. [3]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

    2025

  4. [4]

    o2-searcher: A searching-based agent model for open-domain open-ended question answering.arXiv preprint arXiv:2505.16582, 2025

    Jianbiao Mei, Tao Hu, Daocheng Fu, Licheng Wen, Xuemeng Yang, Rong Wu, Pinlong Cai, Xinyu Cai, Xing Gao, Yu Yang, et al. o2-searcher: A searching-based agent model for open-domain open-ended question answering.arXiv preprint arXiv:2505.16582, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    A comprehensive survey of continual learning: Theory, method and application, 2024

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application, 2024

    2024

  6. [6]

    Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828, 2025

    Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Hinrich Schütze, V olker Tresp, and Yunpu Ma. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning.arXiv preprint arXiv:2508.19828, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    A-mem: Agentic memory for llm agents.Advances in Neural Information Processing Systems, 38:17577–17604, 2026

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents.Advances in Neural Information Processing Systems, 38:17577–17604, 2026

    2026

  8. [8]

    Towards self-evolving agents: Enabling autonomy through interactive experience refinement

    Cheng Yang, Xuemeng Yang, Licheng Wen, Daocheng Fu, Jianbiao Mei, Rong Wu, Pinlong Cai, Yufan Shen, Nianchen Deng, Jia Xu, et al. Towards self-evolving agents: Enabling autonomy through interactive experience refinement. InFindings of the Association for Computational Linguistics: ACL 2026, pages 30424–30451, 2026

    2026

  9. [9]

    G-memory: Tracing hierarchical memory for multi-agent systems, 2025

    Guibin Zhang, Muxin Fu, Guancheng Wan, Miao Yu, Kun Wang, and Shuicheng Yan. G-memory: Tracing hierarchical memory for multi-agent systems, 2025

    2025

  10. [10]

    Memgen: Weaving generative latent memory for self-evolving agents.arXiv preprint arXiv:2509.24704, 2025

    Guibin Zhang, Muxin Fu, and Shuicheng Yan. Memgen: Weaving generative latent memory for self-evolving agents.arXiv preprint arXiv:2509.24704, 2025

    arXiv 2025

  11. [11]

    Reconstruction of automobile destruction: An example of the interaction between language and memory.Journal of verbal learning and verbal behavior, 13(5):585–589, 1974

    Elizabeth F Loftus and John C Palmer. Reconstruction of automobile destruction: An example of the interaction between language and memory.Journal of verbal learning and verbal behavior, 13(5):585–589, 1974

    1974

  12. [12]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023

    2023

  13. [13]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  14. [14]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023

    2023

  15. [15]

    Correctad: A self-correcting agentic system to improve end-to-end planning in autonomous driving

    Enhui Ma, Lijun Zhou, Tao Tang, Jiahuan Zhang, Junpeng Jiang, Zhan Zhang, Dong Han, Kun Zhan, Xueyang Zhang, Xianpeng Lang, et al. Correctad: A self-correcting agentic system to improve end-to-end planning in autonomous driving. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 7755–7763, 2026

    2026

  16. [16]

    Agent-flan: Designing data and methods of effective agent tuning for large language models

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. InFindings of the Association for Computational Linguistics: ACL 2024, pages 9354–9366, 2024

    2024

  17. [17]

    Fireact: Toward language agent fine-tuning.arXiv preprint arXiv:2310.05915, 2023

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning.arXiv preprint arXiv:2310.05915, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. 10

    Pith/arXiv arXiv 2017

  19. [19]

    Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational...

    2024

  20. [20]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Group-in-group policy optimization for llm agent training.Advances in Neural Information Processing Systems, 38:46375–46408, 2026

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent training.Advances in Neural Information Processing Systems, 38:46375–46408, 2026

    2026

  22. [22]

    Expel: Llm agents are experiential learners, 2024

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners, 2024

    2024

  23. [23]

    Memp: Exploring agent procedural memory

    Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Memp: Exploring agent procedural memory. InFindings of the Association for Computational Linguistics: ACL 2026, pages 17490–17502, 2026

    2026

  24. [24]

    Mem0: Building production- ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413, 2025

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production- ready ai agents with scalable long-term memory.arXiv preprint arXiv:2504.19413, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Simplemem: Efficient lifelong memory for llm agents.arXiv preprint arXiv:2601.02553, 2026

    Jiaqi Liu, Yaofeng Su, Peng Xia, Siwei Han, Zeyu Zheng, Cihang Xie, Mingyu Ding, and Huaxiu Yao. Simplemem: Efficient lifelong memory for llm agents.arXiv preprint arXiv:2601.02553, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Memrl: Self-evolving agents via runtime reinforcement learning on episodic memory.arXiv preprint arXiv:2601.03192, 2026

    Shengtao Zhang, Jiaqian Wang, Ruiwen Zhou, Junwei Liao, Yuchen Feng, Zhuo Li, Yujie Zheng, Weinan Zhang, Ying Wen, Zhiyu Li, et al. Memrl: Self-evolving agents via runtime reinforcement learning on episodic memory.arXiv preprint arXiv:2601.03192, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Evolver: Self-evolving llm agents through an experience-driven lifecycle

    Rong Wu, Xiaoman Wang, Jianbiao Mei, Pinlong Cai, Daocheng Fu, Cheng Yang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, et al. Evolver: Self-evolving llm agents through an experience-driven lifecycle. arXiv preprint arXiv:2510.16079, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Pith/arXiv arXiv 2010

  29. [29]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems, volume 35, pages 20744–20757. Curran Associates, Inc., 2022

    2022

  30. [30]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    2025

  34. [34]

    Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216, 2024

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi- lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.arXiv preprint arXiv:2402.03216, 2024. 11

    Pith/arXiv arXiv 2024

  35. [35]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Gon- zalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  37. [37]

    Milvus: A purpose-built vector data management system

    Jianguo Wang, Xiaomeng Yi, Rentong Guo, Hai Jin, Peng Xu, Shengjun Li, Xiangyu Wang, Xiangzhou Guo, Chengming Li, Xiaohai Xu, et al. Milvus: A purpose-built vector data management system. InProceedings of the 2021 International Conference on Management of Data, pages 2614–2627, 2021

    2021

  38. [38]

    Buy Now” button. •Original State (s +):... “Price: $19.99

    Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Agentgym: Evolving large language model- based agents across diverse environments, 2024. 12 Appendix Contents...

    2024