Pith. sign in

REVIEW 5 major objections 7 minor 6 cited by

Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework

T0 review · 5 major / 7 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Jointly optimizing retrieval, utilization, and storage of an LLM agent's memory through on-policy interaction outperforms fixed memory baselines on interactive QA tasks.

desk verdict New adaptive memory cycle framework, but the claimed gains are confounded with LLM fine-tuning and need a controlled comparison. read the letter →

arxiv 2508.16629 v1 pith:67AKJ6J2 submitted 2025-08-15 cs.LG cs.AIcs.CLcs.IR

Zeyu Zhang , Quanyu Dai , Rui Li , Xiaohe Bo , Xu Chen , Zhenhua Dong This is my paper
classification cs.LGcs.AIcs.CLcs.IR
keywords LLMagentsmemorycycleon-policyoptimizationretrieval-augmentedgenerationmixture-of-expertsgatedirectpreferencetask-specificreflectionHotpotQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that an LLM agent's memory should be treated as a cycle -- storage, retrieval, and utilization -- whose components are learned from environment feedback rather than hand-designed. It proposes a framework with an MoE gate that re-weights retrieval metrics per state-memory pair, task-specific reflection that tunes what gets stored, and a post-trained aggregation step that merges retrieved memories into the prompt. The authors claim that jointly optimizing these three procedures with on-policy trajectories lets agents answer more questions correctly on HotpotQA and the MemDaily personal-assistant benchmark, while using fewer reasoning steps. The contribution is turning memory design for agents from a labor-intensive manual exercise into a data-driven optimization problem.

What carries the argument

The memory cycle: storage $S(\theta_s;\cdot)$, retrieval $R(\theta_r;\cdot)$, and utilization $U(\theta_u;\cdot)$ form a loop in which the environment state updates as $s_{t+1}\sim p_{\text{env}}(\cdot|s_t,a_t)$. The load-bearing pieces are the MoE gate $g(\theta_r;s_t,m_i)$, which re-weights metric functions (semantic relevance, importance, recency, emotion) for each state-memory pair; the iterative aggregation $p_i^t=\mathrm{LLM}(\theta_u;p_{i-1}^t,\tilde{m}_i^t,s_t)$ with a Bernoulli stop based on estimated information gain; and the task-specific storage prompt $p_{\text{task}}$, updated by reflecting on positive and negative trajectories. On-policy optimization (Algorithm 1) keeps these

What would settle it

Take the framework's retrieval module alone and compare its ranked memory lists against a human-annotated or independently computed gold ranking on HotpotQA queries (e.g., NDCG@10). If training the MoE gate does not improve ranking quality over the fixed weights of Generative Agents on that gold ranking, the retrieval component of the claimed cycle is not doing the work, and the on-policy gains would be attributable to the utilization or storage changes instead.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the memory-cycle effect is exploitable: because stored memories shape actions and actions generate new observations to store, optimizing any one memory stage in isolation is suboptimal. The paper's on-policy model jointly updates the MoE retrieval gate, the LLM-based utilization step (via SFT and DPO), and the storage prompt (via reflection on successful and failed trajectories). Reported results show the on-policy model outperforming all listed baselines in most settings -- for example, with Qwen-2.5 on HotpotQA-medium it reaches 0.4037 exact-match accuracy versus 0.3119 for the default unoptimized model and higher than every other bas

Load-bearing premise

The retrieval gate is trained to imitate its own ranking: the contrastive pairs in the retrieval loss are formed from $M^{t}_{\text{rank}}$, which is the model's current output, under the assumption that self-imitation sharpens retrieval beyond the pre-trained metric weights. If it only reinforces the existing metric weighting, the claimed retrieval gains vanish.

Editorial extensions

If this is right

  • On-policy optimization is the key to the cycle: ablations show that independently optimizing retrieval, utilization, or storage with off-policy data can reduce performance when recombined, because each module's optimum is conditional on the others' parameters.
  • The resulting agents take fewer reasoning steps per question, finding correct answers more directly; per-trajectory wall-clock time drops (from 472.31 seconds for MTMemory to 25.83 seconds for the on-policy model on the hard split).
  • The framework retains some default-parameter functionality before any training, indicating a degree of task generalization beyond task-specific optimization.
  • Pre-trained importance and emotion scoring functions outperform zero-shot and few-shot LLM prompting for ranking memories, while being cheaper and more stable.
  • Off-policy optimization can hurt: trajectories sampled from a reference policy mismatch the optimized policy, so the on-policy stage is needed to recover performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference -- The self-referential ranking signal used to train the retrieval gate is the weakest link; the paper does not compare the learned gate against an external gold ranking. One testable extension: measure NDCG of the learned gate against human or expert relevance judgments; if it is no better than the original metric weights, the retrieval gains may be an artifact of the utilization and st
  • Inference -- The memory-cycle formulation is a general blueprint: the same storage-retrieval-utilization loop with different backbones (vision-language agents, or retrieval-augmented generation pipelines without LLM policy fine-tuning) could be tested without retraining, since the gate and reflection components are parameter-light.
  • Inference -- Because the reported efficiency gain comes from fewer reasoning steps rather than cheaper per-step operations, the advantage may shrink in settings where the number of steps is fixed; that is a boundary condition implicit in the results.
  • Inference -- On-policy optimization may implicitly be doing credit assignment over entire trajectories; a direct comparison to a standard RL update on the same memory parameters would clarify whether the gains come from the cycle formulation itself or from online exploration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes an adaptive, data-driven memory framework for LLM-based agents. The framework models a memory cycle consisting of storage, retrieval, and utilization. Retrieval uses an MoE gate over multiple metric functions; utilization uses an LLM-based iterative aggregation process that is tuned with SFT and DPO; storage uses task-specific reflection prompts. The authors propose off-policy and on-policy optimization strategies and evaluate on HotpotQA (fullwiki mode, three difficulty levels) and MemDaily, reporting that the on-policy optimized model 'outperforms other baselines in most cases' (Section 5.2).

Significance. If the central claim were established, the paper would make a useful contribution: it moves memory design for LLM agents from manually fixed heuristics to an optimizable cycle, and it evaluates the full retrieval-utilization-storage loop rather than a single component. The paper has several strengths: a clearly formulated memory-cycle MDP, a broad comparison across three inference LLMs, pre-trained importance/emotion scorers with their own evaluation, and a released project URL. However, the current evidence does not isolate the contribution of the adaptive memory procedures from ordinary LLM fine-tuning, and some of the optimization signals are self-referential. The significance is therefore conditional on additional controlled experiments.

major comments (5)
  1. [Section 4.2, retrieval contrastive loss] The headline comparison is confounded. Ours-on is obtained by updating the utilization LLM θu with SFT and DPO on task-specific expert outputs, followed by on-policy updates; all baselines (FUMemory, GAMemory, MBMemory, MTMemory, etc.) are used with frozen pretrained weights. The claim that 'our model with on-policy optimization outperforms other baselines in most cases' may therefore reflect task-specific fine-tuning of the inference LLM rather than the adaptive memory cycle. Supporting evidence: Ours-def, the same architecture without optimization, is not best on several rows (e.g., HotpotQA-hard GPT-4o-mini: 0.3274 vs. MTMemory 0.3628), and Ours-off often drops below Ours-def. To support the memory-cycle claim, the authors should fine-tune the baselines under the same SFT/DPO protocol, or provide an on-policy variant with θu frozen so that only retrieval/storage are optimized.
  2. [Section 5.1, Appendix D.1, Table 1] The retrieval gate is trained against the ranking M_t_rank produced by the current retrieval procedure itself. The contrastive pairs are formed by pairing elements of M_t_rank in reverse order; the supervision thus tells the gate to reproduce the order already induced by the pretrained metric functions and the current gate weights, filtered only by whether the trajectory ended with high reward. This is self-imitation and cannot validate that the gate learns a better retrieval policy. No external gold ranking, human evaluation, or independent downstream measure of retrieval quality is provided. The authors should compare the learned gate against fixed metric combinations with identical downstream optimization, or evaluate retrieval against human/LLM-judged relevance.
  3. [Section 4.2, Memory Utilization Optimization] All HotpotQA results are based on only 113, 109, and 107 test questions, respectively, and MemDaily's test size is not reported. Many reported differences are a few questions: e.g., HotpotQA-medium GPT-4o-mini Ours-on 0.4404 vs. 0.4220 is 2/109 questions, and HotpotQA-easy Qwen Ours-on 0.4112 vs. 0.3925 is 2/107. No error bars, confidence intervals, or significance tests are provided. The claim of 'outperforms in most cases' is not statistically supported at these sample sizes. The Limitations section additionally acknowledges possible HotpotQA pre-training leakage; since HotpotQA is the main benchmark, this strengthens the need for a leakage-controlled or additional dataset.
  4. [Section 4.2, Memory Storage Optimization] The DPO objective is not well-specified. The text says that after SFT, the reference model is LLM(θ_SFT_u; ·) and then 're-generate utilization results with ˆp_t^t = LLM(θu; ...)'. If the chosen response ˆp_t^t is generated by the current θu rather than by the expert model E(·), then the DPO 'chosen' examples are self-generated on-policy outputs, and the objective is no longer aligned to expert preferences. Please define exactly how chosen and rejected examples are constructed, and clarify whether the expert outputs E(·) are used as the chosen responses in both the SFT and DPO stages.
  5. [Algorithm 1, line 8] The storage optimization is described as 'ptask ← ptask ∪ LLM({st, mt} ∈ Dpos) ∪ LLM({st, mt} ∈ Dneg)', i.e., a verbal self-reflection that concatenates reflections from both successful and unsuccessful trajectories. This is not an optimization with a well-defined objective, and the mechanism by which negative reflections improve extraction is unclear. The ablation results also show that Ours-S frequently reduces accuracy (e.g., HotpotQA-hard GPT-4o-mini: 0.2920 vs. Ours-def 0.3274). The paper should report the final task prompt, control for prompt-length and hint effects, and show whether the reflection update reliably improves storage across settings.
minor comments (7)
  1. [Appendix A.1] Line 8 contains a typo: 'θr s∗ = θL r' should likely be 'θ∗r = θLr'. Line 6 uses '∪' to update θu; as written this is unclear and probably should update θs with the storage reflection. Please revise the notation.
  2. [Table 5, Appendix B] The sentence 'Finally, we compile a dataset Demo = ...' is duplicated verbatim. Please remove the duplicate.
  3. [Section 5.3, Table 3] The MemDaily evaluation uses a single recall per trajectory and only one memory entity, so it does not exercise retrieval or utilization optimization. The number of test trajectories is not stated. Also, the Ours-on accuracy is reported with three decimals (0.561) while the other entries have four; please make the precision consistent.
  4. [Section 5.6, Figure 4] The efficiency comparison reports only inference time. Since Ours-on includes an SFT/DPO training phase and on-policy interaction, the training cost is not captured. Please state explicitly that training/optimization costs are excluded, or include them in the comparison.
  5. [Section 5.4] Figure 4 is not referenced in the main text and is not described in the hyper-parameter discussion. Please add a reference and a brief description of what is plotted.
  6. [Section 2.3] The average reasoning-step analysis is only reported for one model (Qwen-2.5) and one difficulty level. Given the small test set, it would be helpful to report the distribution of trajectory lengths or a test for difference in means.
  7. [Section 1] The related work on memory mechanisms is brief; MemGPT is cited as 'Memgpt: Towards llms as operating systems' without a year/venue, and the citation format is inconsistent. Please normalize the references.

Circularity Check

1 steps flagged · score 6.0 of 10

Retrieval-gate training is self-imitation: the contrastive labels are the gate's own ranking, so the claimed retrieval improvement is partially circular.

  1. self definitional [Section 3.2 (ranking definition) and Section 4.2 (contrastive loss), Eq. L(θr; Ds)]
    "we calculate the matching scores by f (θr; st, mi) = g(θr; st, mi) · d(st, mi)T . Finally, all the memories mi ∈ M t are ranked according to their matching scores, resulting in a ranked memory list M t rank... we focus on the ranking result M t rank = [ ˜mt 1, ˜mt 2, ...,˜mt t] from their memory retrieval procedures. We pair all the elements ˜mt i ∈ M t rank in reverse order as xi = ( ˜mt i, ˜mt t−i+1) ... we define our loss function as L(θr; Ds) = ... and optimize the parameters with θ∗r = arg minθr L(θr; D) by gradient descent."

    The ranking M t rank is not an external label: §3.2 defines it as the output of ranking all memories by f(θr; st, mi) = g(θr; st, mi)·d(st, mi)^T, the very function whose parameters θr the loss in §4.2 updates. The contrastive pairs (˜m_i, ˜m_{t−i+1}) are taken from that self-produced ranking in reverse order, and the loss asks the new θr to keep the already-top memory above the already-bottom memory. Thus the supervision is the model's own current ordering, filtered only by the binary success/failure of the whole trajectory. In off-policy mode the ranking comes from the reference policy, and in on-policy mode (Algorithm 1, line 3) from the current policy; either way the labels are generated by the same retrieval procedure being trained. A parameter vector that simply preserves the existin

full rationale

The identifiable circularity is in retrieval optimization. The contrastive target is the model's own ranking, so the claim that the MoE gate learns task-appropriate retrieval from data is not fully supported. However, the utilization (SFT/DPO) and storage (reflection) components use external trajectory rewards and expert outputs, and the experimental comparison is not reduced by construction—the superiority claim is confounded by the fact that Ours-on receives SFT/DPO fine-tuning while baselines are frozen, but that is an experimental-design confound rather than a definitional circularity. The paper's self-citations ([7], [23], [27]) are peripheral (survey, dataset, library) and not load-bearing. The pre-trained emotion/importance scorers are trained on LLM-generated synthetic labels (Appendix A), which further weakens the external grounding of the retrieval signal but is not itself a reduction. Overall: one central component is self-referential, so partial circularity, score 6.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The framework rests on five learned parameter sets and five background assumptions. The most consequential is the self-referential retrieval supervision, which means the gate is trained to imitate the ranking it already produces. The pre-trained scorers also inherit biases from LLM-generated synthetic labels. No new physical entities are introduced.

free parameters (6)
  • MoE retrieval gate parameters θr = {W1, W2, b1, b2} = Not reported numerically
    Optimized via the contrastive ranking loss in §4.2 on successful trajectories; weights the metric functions in retrieval.
  • Utilization LLM parameters θu after SFT and DPO = Checkpoint not released as of writing; no numerical value
    Fine-tuned to mimic an expert model E(·) for merging memories (§4.2).
  • Storage prompt θs = p_task = Accumulated reflection text from positive and negative trajectories
    Updated by self-reflection in §3.4/§4.2; task-specific extraction hint.
  • Emotion scorer parameters ϕe = Not reported; trained on LLM-generated synthetic sentences
    Pre-trained in Appendix A.1 on synthetic emotion-labeled data; used as a fixed metric in retrieval.
  • Importance scorer parameters ϕp = Not reported; trained on LLM-generated partially ordered sentences
    Pre-trained in Appendix A.2; used as a fixed metric in retrieval.
  • Hyperparameters (β for DPO, thresholds βr/βs, p-norm order, top-k, batch sizes, learning rates) = Listed in §D.3 (e.g., SFT/DPO lr 0.0001-0.0005, batch sizes 15-32, top-k=10)
    Chosen by hand or via small sensitivity sweeps (§5.6), not derived from theory.
assumptions (5)
  • domain assumption Agent-environment interaction is a Markov Decision Process with state st, action at, transition penv(·|st,at), reward r(st,at).
    Adopted in §3.1 to formalize the memory cycle; standard for RL, but the interactive HotpotQA simulator is an approximation.
  • domain assumption The HotpotQA fullwiki simulator built from a Wikipedia dump faithfully represents the intended interactive QA task.
    Used in §5.1 and Appendix D.1; the searcher returns full documents, which differs from the original HotpotQA paragraph retrieval setting.
  • ad hoc to paper LLM-generated synthetic datasets for emotion and importance scoring provide valid supervision.
    Appendix A.1-A.2: sentences are generated by LLMs and labeled by the same LLM's instruction; no human validation is reported.
  • ad hoc to paper The word-increase rate ci = clip(Δli/Δli−1, 0, 1) approximates information gain for stopping memory merging.
    Introduced in §3.3 as a heuristic stopping rule with no theoretical justification.
  • ad hoc to paper Successful trajectories' retrieval rankings are a valid training target for the retrieval gate.
    The contrastive loss in §4.2 uses the model's own ranking M_t_rank from successful trajectories; quality is never compared with gold rankings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework." pith.science (2026). https://pith.science/paper/67AKJ6J2

@misc{pith2026250816629,
  author       = {Pith},
  title        = {Pith review of: Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/67AKJ6J2}},
  note         = {Machine review of arXiv:2508.16629}
}
read the original abstract

LLM-based agents have been extensively applied across various domains, where memory stands out as one of their most essential capabilities. Previous memory mechanisms of LLM-based agents are manually predefined by human experts, leading to higher labor costs and suboptimal performance. In addition, these methods overlook the memory cycle effect in interactive scenarios, which is critical to optimizing LLM-based agents for specific environments. To address these challenges, in this paper, we propose to optimize LLM-based agents with an adaptive and data-driven memory framework by modeling memory cycles. Specifically, we design an MoE gate function to facilitate memory retrieval, propose a learnable aggregation process to improve memory utilization, and develop task-specific reflection to adapt memory storage. Our memory framework empowers LLM-based agents to learn how to memorize information effectively in specific environments, with both off-policy and on-policy optimization. In order to evaluate the effectiveness of our proposed methods, we conduct comprehensive experiments across multiple aspects. To benefit the research community in this area, we release our project at https://github.com/nuster1128/learn_to_memorize.

Figures

Figures reproduced from arXiv: 2508.16629 by the authors.

Figure 1
Figure 1. (a) In memory retrieval, the optimal weights for different aspects vary across different tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the memory cycle effect and adaptive memory framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Results of average reasoning steps across different baselines. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Results of different hyper-parameters. we craft instructions for LLMs to generate scores within the range of [0.0, 1.0] concerning importance and various emotional aspects of specific messages. Additionally, we employ GPT-4o, GPT-4o-mini, Qwen-2.5, and Llama-3.1 as the…

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Metis: Memory Foundation Model

    cs.CL 2026-07 conditional novelty 6.5 of 10

    Metis equips frozen LLMs with gradient-free native memory states and mid-trained store/use procedures that beat no-context parametric baselines on short memory tasks but still lag full context and degrade over long tr...

  2. Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    HORMA builds a hierarchical memory structure from agent experiences and trains a lightweight RL navigator to retrieve minimal sufficient context, yielding better task performance with at most 22.17% of baseline token ...

  3. Auto-Dreamer: Learning Offline Memory Consolidation for Language Agents

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Auto-Dreamer trains an offline memory consolidator via GRPO on agent performance to abstract cross-session patterns, outperforming baselines by 7 points on ScienceWorld with 12x smaller memory and generalizing to ALFW...

  4. Metis: Memory Foundation Model

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Metis puts a trainable fixed-size memory matrix inside a frozen LLM backbone and learns to remember, update, forget, and reflect across turns without replaying original context.

  5. CoMIC: Collaborative Memory and Insights Circulation for Long-Horizon LLM Agents in Cloud-Edge Systems

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    CoMIC is a parameter-free cloud-edge framework that circulates memory and insights between edge agents and a central critic to improve long-horizon LLM agent performance on symbolic and text tasks.

  6. Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering

    cs.SE 2026-04 accept novelty 5.0 of 10

    LLM agent progress depends on externalizing cognitive functions into memory, skills, protocols, and harness engineering that coordinates them reliably.

Reference graph

Works this paper leans on

31 extracted references · 2 canonical work pages · cited by 5 Pith papers

  1. [1]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024

    2024

  2. [2]

    The rise and potential of large language model based agents: A survey

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025

    2025

  3. [3]

    Large language model based multi-agents: A survey of progress and challenges

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024

    arXiv 2024

  4. [4]

    Large language model agent in financial trading: A survey

    Han Ding, Yinheng Li, Junhao Wang, and Hang Chen. Large language model agent in financial trading: A survey. arXiv preprint arXiv:2408.06361, 2024

    arXiv 2024

  5. [5]

    A survey of large language model empowered agents for recommendation and search: Towards next-generation information retrieval

    Yu Zhang, Shutong Qiao, Jiaqi Zhang, Tzu-Heng Lin, Chen Gao, and Yong Li. A survey of large language model empowered agents for recommendation and search: Towards next-generation information retrieval. arXiv preprint arXiv:2503.05659, 2025

    arXiv 2025

  6. [6]

    Personal llm agents: Insights and survey about the capability, efficiency and security

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024

    arXiv 2024

  7. [7]

    A survey on the memory mechanism of large language model based agents

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.13501, 2024

    arXiv 2024

  8. [8]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2, 2023

    arXiv 2023

Show all 31 references
  1. [9]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  2. [10]

    Memorybank: Enhancing large language models with long-term memory

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731, 2024

    2024

  3. [11]

    Memgpt: Towards llms as operating systems

    Charles Packer, Vivian Fang, Shishir_G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonza- lez. Memgpt: Towards llms as operating systems. 2023

    2023

  4. [12]

    Memory3: Language modeling with explicit memory

    Hongkang Yang, Zehao Lin, Wenjin Wang, Hao Wu, Zhiyu Li, Bo Tang, Wenqiang Wei, Jinbo Wang, Zeyun Tang, Shichao Song, et al. Memory3: Language modeling with explicit memory. arXiv preprint arXiv:2407.01178, 2024

    2024 arXiv

  5. [13]

    Generative agents: Interactive simulacra of human behavior

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed- ings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023

    2023

  6. [14]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  7. [15]

    Reinforcement learning: An introduction, volume 1

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  8. [16]

    Policy gradient meth- ods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. 11

    1999

  9. [17]

    Playing atari with deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  10. [18]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999

    1999

  11. [19]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  12. [20]

    Large language models empowered agent-based modeling and simulation: A survey and perspectives

    Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. Large language models empowered agent-based modeling and simulation: A survey and perspectives. Humanities and Social Sciences Communications, 11(1):1–24, 2024

    2024

  13. [21]

    Understanding the planning of llm agents: A survey

    Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of llm agents: A survey. arXiv preprint arXiv:2402.02716, 2024

    2024 arXiv

  14. [22]

    From isolated conversations to hierarchi- cal schemas: Dynamic tree memory representation for llms

    Alireza Rezazadeh, Zichao Li, Wei Wei, and Yujia Bao. From isolated conversations to hierarchi- cal schemas: Dynamic tree memory representation for llms. arXiv preprint arXiv:2410.14052, 2024

    2024 arXiv

  15. [23]

    Memsim: A bayesian simulator for evaluating memory of llm-based personal assistants

    Zeyu Zhang, Quanyu Dai, Luyu Chen, Zeren Jiang, Rui Li, Jieming Zhu, Xu Chen, Yi Xie, Zhenhua Dong, and Ji-Rong Wen. Memsim: A bayesian simulator for evaluating memory of llm-based personal assistants. arXiv preprint arXiv:2409.20163, 2024

    2024 arXiv

  16. [24]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  17. [25]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  18. [26]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  19. [27]

    Memengine: A unified and modular library for developing advanced memory of llm-based agents

    Zeyu Zhang, Quanyu Dai, Xu Chen, Rui Li, Zhongyang Li, and Zhenhua Dong. Memengine: A unified and modular library for developing advanced memory of llm-based agents. arXiv preprint arXiv:2505.02099, 2025

    2025 arXiv

  20. [28]

    Enhancing large language model with self-controlled memory framework

    Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Enhancing large language model with self-controlled memory framework. arXiv preprint arXiv:2304.13343, 2023

    2023 arXiv

  21. [29]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  22. [30]

    Emotional rag: Enhancing role-playing agents through emotional retrieval

    Le Huang, Hengzhi Lan, Zijun Sun, Chuan Shi, and Ting Bai. Emotional rag: Enhancing role-playing agents through emotional retrieval. arXiv preprint arXiv:2410.23041, 2024. 12 A Pre-trained Metric Functions A.1 Emotion Scoring Function In addition to considering the semantic si...

    2024 arXiv

  23. [31]

    Then, the emotional similarity can be calculated as demo(st, mi) = he(ϕe; st) · he(ϕe; mi)T ||he(ϕe; st)|| · ||he(ϕe; mi)||

    + be 2, where hx is the text embedding of x and ϕe = {W e 1 , We 2 , be 1, be 2} are trainable parameters. Then, the emotional similarity can be calculated as demo(st, mi) = he(ϕe; st) · he(ϕe; mi)T ||he(ϕe; st)|| · ||he(ϕe; mi)|| . To optimize the emotion scoring function, we...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.