pith. sign in

arxiv: 2605.06527 · v1 · submitted 2026-05-07 · 💻 cs.CL

STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?

Hanxiang Chao , Yihan Bai , Rui Sheng , Tianle Li , Yushi Sun This is my paper

Pith reviewed 2026-05-08 10:11 UTC · model grok-4.3

classification 💻 cs.CL
keywords LLM memorybelief updatingimplicit conflictagent benchmarksstate resolutionmemory revision
0
0 comments X

The pith

LLM agents frequently fail to recognize when new evidence makes their stored memories invalid.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper establishes that large language model agents have a significant blind spot in updating their long-term memories when new observations implicitly contradict earlier ones. Current benchmarks focus on static retrieval, but real use requires detecting these conflicts through inference and then revising behavior accordingly. The authors create the STALE benchmark with 400 scenarios across everyday topics to test three specific abilities: spotting outdated beliefs, ignoring queries based on old states, and adapting actions proactively. Evaluation of leading models shows the best one reaches only 55.2 percent accuracy overall, often accepting stale assumptions in user queries. They also introduce a prototype system that consolidates states at write time to address the issue.

Core claim

The paper claims that a critical failure mode exists in LLM agent memory: implicit conflicts where later evidence invalidates prior memories without direct negation, demanding contextual and commonsense reasoning to identify. Using the STALE benchmark of 400 validated scenarios and 1,200 queries, it measures performance on state resolution, premise resistance, and implicit policy adaptation. Results indicate models retrieve updated evidence but do not act on it, with top accuracy at 55.2 percent, and they struggle when changes in one state aspect affect related memories. As an initial fix, the CUPMem prototype strengthens revision through structured consolidation and propagation-aware search

What carries the argument

The STALE benchmark, which includes 400 expert-validated conflict scenarios and a three-dimensional probing framework to test detection and adaptation to implicit memory conflicts.

If this is right

  • Frontier LLMs achieve at most 55.2% accuracy when required to resolve implicit conflicts in memory.
  • Models tend to accept outdated assumptions in user queries instead of rejecting them.
  • Changes in one aspect of user state often fail to invalidate related memories.
  • Explicit state adjudication during memory writing offers a promising path to more robust agent memory.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Agents without this capability may produce inconsistent or erroneous responses in extended interactions.
  • The benchmark could be extended to include multi-turn dialogues to better simulate real agent use.
  • Improving this might require new architectures focused on state tracking rather than pure retrieval.

Load-bearing premise

The expert-validated scenarios in the benchmark truly reflect the implicit conflicts that would occur in actual long-term user-agent interactions and require commonsense to detect.

What would settle it

A controlled test where an LLM is given a sequence of observations creating an implicit conflict and then queried on the updated state; success would be correctly identifying the new state and rejecting stale-based queries across the 1200 evaluations.

Figures

Figures reproduced from arXiv: 2605.06527 by Hanxiang Chao, Rui Sheng, Tianle Li, Yihan Bai, Yushi Sun.

Figure 1
Figure 1. Figure 1: Overview of the implicit conflict setting. User-assistant dialogues are temporally sparse, view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the dataset generation pipeline. All instances are reviewed and edited by view at source ↗
Figure 3
Figure 3. Figure 3: Weighted group ratio curves for Qwen3.5-9B and Qwen3.5-27B. We compare the ratio view at source ↗
Figure 4
Figure 4. Figure 4: Annotation interface used for manual quality control during dataset construction. view at source ↗
Figure 5
Figure 5. Figure 5: Attribute distribution of the final benchmark after generation, verification, filtering, and view at source ↗
Figure 6
Figure 6. Figure 6: Layer-wise attention curves for each correctness split in Qwen3.5-9B and Qwen3.5-27B. view at source ↗
Figure 7
Figure 7. Figure 7: Type-level mean attention curves for Type I and Type II. view at source ↗
Figure 8
Figure 8. Figure 8: Layer-wise ratio Q → Sessionn over Q → Sessiono for each correctness split. Ratios above one indicate stronger relative attention to the updated evidence. Limitations. This analysis is diagnostic rather than causal. Attention scores do not fully determine model predictions, and the sampled groups are limited in size. Nevertheless, the consistent separation from noise baselines and the correctness-condition… view at source ↗
read the original abstract

Large Language Model (LLM) agents are increasingly expected to maintain coherent, long-term personalized memory, yet current benchmarks primarily measure static fact retrieval, overlooking the ability to revise stored beliefs when new evidence emerges. We identify a critical and underexplored failure mode, Implicit Conflict: a later observation invalidates an earlier memory without explicit negation, requiring contextual inference and commonsense reasoning to detect. To rigorously evaluate this capability, we introduce STALE, a benchmark of 400 expert-validated conflict scenarios (1,200 evaluation queries across three probing dimensions) spanning over 100 everyday topics with contexts up to 150K tokens. We propose a three-dimensional probing framework that tests State Resolution (detecting that a prior belief is outdated), Premise Resistance (rejecting queries that falsely presuppose a stale state), and Implicit Policy Adaptation (proactively applying updated states in downstream behavior). A systematic evaluation of frontier LLMs and specialized memory frameworks reveals a pervasive gap between retrieving updated evidence and acting on it, with even the best evaluated model achieving only 55.2% overall accuracy. Models often accept outdated assumptions embedded in a user's query, and they struggle to recognize when a change in one aspect of the user's state should invalidate related memories. To establish an initial baseline for state-aware memory, we further present CUPMem, a prototype that strengthens write-time revision through structured state consolidation and propagation-aware search, suggesting that explicit state adjudication is a promising direction for robust agentic memory.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces the STALE benchmark of 400 expert-validated implicit conflict scenarios (1,200 queries across three dimensions: State Resolution, Premise Resistance, and Implicit Policy Adaptation) to evaluate whether LLM agents can detect when new observations invalidate prior memories without explicit negation. It reports that frontier models and memory frameworks achieve at most 55.2% overall accuracy, highlighting a gap between retrieving updated evidence and acting on it, and presents CUPMem as a prototype using structured state consolidation and propagation-aware search.

Significance. If the benchmark and probing framework validly isolate implicit conflict adjudication, the work would usefully document a pervasive limitation in current LLM agent memory systems for long-term personalized use and provide an initial baseline via CUPMem that could inform more robust state-tracking designs.

major comments (2)
  1. [Evaluation and Probing Framework] The evaluation provides the conflicting observation in context for all three probing dimensions yet includes no control task that directly measures retrieval of the updated state (e.g., a simple factual query 'What is the user's current X?' answered from memory). This leaves open the possibility that low accuracy reflects standard long-context retrieval degradation rather than a specific failure to perform implicit conflict resolution or policy update, weakening the central claim of a 'pervasive gap between retrieving updated evidence and acting on it'.
  2. [Benchmark Construction] The 400 scenarios are described as expert-validated, but the manuscript does not report inter-annotator agreement, the exact validation protocol, or how the scenarios were constructed to ensure they require contextual inference and commonsense reasoning rather than surface-level cues. This detail is load-bearing for interpreting the 55.2% accuracy as evidence of a general capability gap.
minor comments (2)
  1. [Benchmark] The abstract states contexts reach 150K tokens, but the paper would benefit from reporting the token-length distribution across the 400 scenarios and any ablation on context length.
  2. [CUPMem] CUPMem is presented as a prototype; including pseudocode or a high-level architecture diagram would clarify how 'structured state consolidation' and 'propagation-aware search' differ from existing memory frameworks.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below with clarifications and commitments to revisions that strengthen the manuscript without altering its core claims.

read point-by-point responses

  1. Referee: [Evaluation and Probing Framework] The evaluation provides the conflicting observation in context for all three probing dimensions yet includes no control task that directly measures retrieval of the updated state (e.g., a simple factual query 'What is the user's current X?' answered from memory). This leaves open the possibility that low accuracy reflects standard long-context retrieval degradation rather than a specific failure to perform implicit conflict resolution or policy update, weakening the central claim of a 'pervasive gap between retrieving updated evidence and acting on it'.

    Authors: We appreciate the referee's emphasis on isolating retrieval from conflict adjudication. The State Resolution dimension directly queries the model's ability to identify the updated state given the conflicting observation, which tests retrieval and application of new evidence. Nevertheless, to more precisely quantify any gap attributable to implicit conflict rather than general long-context degradation, we will add an explicit control task in the revised manuscript. This control will include non-conflicting scenarios where models retrieve the current state from provided information, enabling direct comparison of baseline retrieval accuracy against the three probing dimensions. This addition will better substantiate our central claim. revision: partial

  2. Referee: [Benchmark Construction] The 400 scenarios are described as expert-validated, but the manuscript does not report inter-annotator agreement, the exact validation protocol, or how the scenarios were constructed to ensure they require contextual inference and commonsense reasoning rather than surface-level cues. This detail is load-bearing for interpreting the 55.2% accuracy as evidence of a general capability gap.

    Authors: We agree that these methodological details are essential for validating the benchmark. The current manuscript does not include them, but the revised version will add a dedicated section (or appendix) describing the scenario construction process, the expert validation protocol, and inter-annotator agreement statistics. We will also explain the design choices used to ensure scenarios require contextual inference and commonsense reasoning, such as avoiding explicit negations and relying on subtle, multi-fact state changes that cannot be resolved via surface cues alone. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark with no derivations, fits, or self-referential claims

full rationale

The paper introduces the STALE benchmark and evaluates frontier LLMs plus a prototype memory framework (CUPMem) on 400 expert-validated scenarios across three probing dimensions. All reported results are direct accuracy measurements on held-out queries; no equations, parameter fits, predictions derived from prior data subsets, or load-bearing self-citations appear in the derivation chain. The central claim of a 'pervasive gap' is presented as an observed empirical outcome rather than a consequence of any self-defined or self-cited premise. The work is therefore self-contained as a measurement study.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is an empirical benchmark and prototype paper. No free parameters, mathematical axioms, or invented physical entities are involved. The central claims rest on the construction and expert validation of the STALE scenarios and the accuracy of the reported model evaluations.

pith-pipeline@v0.9.0 · 5572 in / 1258 out tokens · 87061 ms · 2026-05-08T10:11:29.915723+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references

  1. [1]

    Brachman and Hector J

    Ronald J. Brachman and Hector J. Levesque. Chapter 7 - rules in production systems. InKnowl- edge Representation and Reasoning, The Morgan Kaufmann Series in Artificial Intelligence, pages 117–134. Morgan Kaufmann, San Francisco, 2004

    2004

  2. [2]

    Mem0: Building production-ready ai agents with scalable long-term memory, 2025

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory, 2025

    2025

  3. [3]

    Yiming Du, Wenyu Huang, Danna Zheng, Zhaowei Wang, Sebastien Montella, Mirella Lapata, Kam-Fai Wong, and Jeff Z. Pan. Rethinking memory in llm based agents: Representations, operations, and emerging topics, 2025

    2025

  4. [4]

    Lightmem: Lightweight and efficient memory-augmented generation, 2026

    Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang, Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, and Ningyu Zhang. Lightmem: Lightweight and efficient memory-augmented generation, 2026

    2026

  5. [5]

    Getting sick after seeing a doctor? diagnosing and mitigating knowledge conflicts in event temporal reasoning

    Tianqing Fang, Zhaowei Wang, Wenxuan Zhou, Hongming Zhang, Yangqiu Song, and Muhao Chen. Getting sick after seeing a doctor? diagnosing and mitigating knowledge conflicts in event temporal reasoning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Findings of the Association for Computational Linguistics: NAACL 2024, pages 3846–3868, Mexico City, ...

    2024

  6. [6]

    Retrieval-augmented generation for large language models: A survey, 2024

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024

    2024

  7. [7]

    Gemini 3.1 Flash-lite, February 2026

    Google DeepMind. Gemini 3.1 Flash-lite, February 2026

    2026

  8. [8]

    Gemini 3.1 Pro, February 2026

    Google DeepMind. Gemini 3.1 Pro, February 2026

    2026

  9. [9]

    From RAG to memory: Non-parametric continual learning for large language models

    Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From RAG to memory: Non-parametric continual learning for large language models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Proceedings of the 42nd International Conference on Machine Learning...

    2025

  10. [10]

    Evaluating memory in llm agents via incremental multi-turn interactions, 2026

    Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in llm agents via incremental multi-turn interactions, 2026

    2026

  11. [11]

    Memory in the age of ai agents, 2026

    Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shihan Dou, Zhiheng Xi, Senjie Jin, Jiejun Tan, Yanbin Yin, Jiongnan Liu, Zeyu Zhang, Zhongxiang Sun, Yutao Zhu, Hao Sun, Boci Peng, Zhenrong Cheng, Xuanbo Fan, Jiaxin Guo, Xinlei Yu, Zhenhong Zhou, Zewen Hu, Jiahao Huo, Junhao Wang, Yuwei Niu, Yu Wang, Zhe...

    2026

  12. [12]

    Zhaopei Huang, Qifeng Dai, Guozheng Wu, Xiaopeng Wu, Xubin Li, Tiezheng Ge, Wenxuan Wang, and Qin Jin. Mem-pal: Towards memory-based personalized dialogue assistants for long-term user-agent interaction.Proceedings of the AAAI Conference on Artificial Intelligence, 40(37):31229–31237, Mar. 2026

    2026

  13. [13]

    Licomemory: Lightweight and cognitive agentic memory for efficient long-term reasoning, 2026

    Zhengjun Huang, Zhoujin Tian, Qintian Guo, Fangyuan Zhang, Yingli Zhou, Di Jiang, Zeying Xie, and Xiaofang Zhou. Licomemory: Lightweight and cognitive agentic memory for efficient long-term reasoning, 2026

    2026

  14. [14]

    Taylor, and Dan Roth

    Bowen Jiang, Zhuoqun Hao, Young-Min Cho, Bryan Li, Yuan Yuan, Sihao Chen, Lyle Ungar, Camillo J. Taylor, and Dan Roth. Know me, respond to me: Benchmarking llms for dynamic user profiling and personalized responses at scale, 2025

    2025

  15. [15]

    Personamem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory, 2025

    Bowen Jiang, Yuan Yuan, Maohao Shen, Zhuoqun Hao, Zhangchen Xu, Zichen Chen, Ziyi Liu, Anvesh Rao Vijjini, Jiashu He, Hanchao Yu, Radha Poovendran, Gregory Wornell, Lyle Ungar, Dan Roth, Sihao Chen, and Camillo Jose Taylor. Personamem-v2: Towards personalized intelligence via learning implicit user personas and agentic memory, 2025

    2025

  16. [16]

    Memory-QA: Answering recall questions based on multi- modal memories

    Hongda Jiang, Xinyuan Zhang, Siddhant Garg, Rishab Arora, Shiun-Zu Kuo, Jiayang Xu, Aaron Colak, and Xin Luna Dong. Memory-QA: Answering recall questions based on multi- modal memories. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Proc...

    2025

  17. [17]

    Amemgym: Interactive memory benchmarking for assistants in long-horizon conversations, 2026

    Cheng Jiayang, Dongyu Ru, Lin Qiu, Yiyang Li, Xuezhi Cao, Yangqiu Song, and Xunliang Cai. Amemgym: Interactive memory benchmarking for assistants in long-horizon conversations, 2026

    2026

  18. [18]

    Littman, and Anthony R

    Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains.Artificial Intelligence, 101(1):99–134, 1998

    1998

  19. [19]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY , USA, 2023. Association for Computing Machinery

    2023

  20. [20]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Informa...

    2020

  21. [21]

    Toward multi- session personalized conversation: A large-scale dataset and hierarchical tree framework for implicit reasoning

    Xintong Li, Jalend Bantupalli, Ria Dharmani, Yuwei Zhang, and Jingbo Shang. Toward multi- session personalized conversation: A large-scale dataset and hierarchical tree framework for implicit reasoning. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natu...

    2025

  22. [22]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  23. [23]

    Evaluating very long-term conversational memory of LLM agents

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851–13870, Bangko...

    2024

  24. [24]

    Llama 3.3, 2024

    Meta. Llama 3.3, 2024. 11

    2024

  25. [25]

    MiniMax M2.5, 2026

    MiniMax. MiniMax M2.5, 2026

    2026

  26. [26]

    GPT-4o mini, 2024

    OpenAI. GPT-4o mini, 2024

    2024

  27. [27]

    GPT-5.1 Chat, 2025

    OpenAI. GPT-5.1 Chat, 2025

    2025

  28. [28]

    GPT-5.2, December 2025

    OpenAI. GPT-5.2, December 2025

    2025

  29. [29]

    GPT-5.4, March 2026

    OpenAI. GPT-5.4, March 2026

    2026

  30. [30]

    GPT-5.4 nano, March 2026

    OpenAI. GPT-5.4 nano, March 2026

    2026

  31. [31]

    Patil, Ion Stoica, and Joseph E

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024

    2024

  32. [32]

    Who’s who: Large language models meet knowledge conflicts in practice

    Quang Hieu Pham, Hoang Ngo, Anh Tuan Luu, and Dat Quoc Nguyen. Who’s who: Large language models meet knowledge conflicts in practice. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10142–10151, Miami, Florida, USA, November 2024. Association for Computational Linguistics

    2024

  33. [33]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026

    2026

  34. [34]

    Rabiner and B

    L. Rabiner and B. Juang. An introduction to hidden markov models.IEEE ASSP Magazine, 3(1):4–16, 1986

    1986

  35. [35]

    Zep: A temporal knowledge graph architecture for agent memory, 2025

    Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. Zep: A temporal knowledge graph architecture for agent memory, 2025

    2025

  36. [36]

    Morehopqa: More than multi-hop reasoning, 2024

    Julian Schnitzler, Xanh Ho, Jiahao Huang, Florian Boudin, Saku Sugawara, and Akiko Aizawa. Morehopqa: More than multi-hop reasoning, 2024

    2024

  37. [37]

    Sagi Shaier, Ari Kobren, and Philip V . Ogren. Adaptive question answering: Enhancing language model proficiency for addressing knowledge conflicts with source citations. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17226–17239, Miami, Florida, U...

    2024

  38. [38]

    Long- memeval: Benchmarking chat assistants on long-term interactive memory, 2025

    Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Long- memeval: Benchmarking chat assistants on long-term interactive memory, 2025

    2025

  39. [39]

    Knowme-bench: Benchmarking person understanding for lifelong digital companions, 2026

    Tingyu Wu, Zhisheng Chen, Ziyan Weng, Shuhe Wang, Chenglong Li, Shuo Zhang, Sen Hu, Silin Wu, Qizhen Lan, Huacan Wang, and Ronghao Chen. Knowme-bench: Benchmarking person understanding for lifelong digital companions, 2026

    2026

  40. [40]

    Knowledge conflicts for LLMs: A survey

    Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for LLMs: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8541–8565, Miami, Florida, USA, November 2024. Association for Computationa...

    2024

  41. [41]

    A-mem: Agentic memory for llm agents, 2025

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for llm agents, 2025

    2025

  42. [42]

    Pan, Hinrich Schütze, V olker Tresp, and Yunpu Ma

    Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z. Pan, Hinrich Schütze, V olker Tresp, and Yunpu Ma. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning, 2026

    2026

  43. [43]

    Crag - comprehensive rag benchmark

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, Lingkun Kong, Brian Moran, Jiaqi Wang, Yifan Ethan Xu, An Yan, Chenyu Yang, Eting Yuan, Hanwen Zha, Nan Tang, Lei Chen, Nicolas Scheffer, Yue Liu, Nirav Shah, Rakesh Wanga, Anuj Kumar, Wen-tau Yih, and Xin Luna Dong. Crag...

    2024

  44. [44]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processi...

    2018

  45. [45]

    Memsearcher: Training llms to reason, search and manage memory via end-to-end reinforcement learning, 2025

    Qianhao Yuan, Jie Lou, Zichao Li, Jiawei Chen, Yaojie Lu, Hongyu Lin, Le Sun, Debing Zhang, and Xianpei Han. Memsearcher: Training llms to reason, search and manage memory via end-to-end reinforcement learning, 2025

    2025

  46. [46]

    Assomem: Scalable memory qa with multi-signal associative retrieval, 2025

    Kai Zhang, Xinyuan Zhang, Ejaz Ahmed, Hongda Jiang, Caleb Kumar, Kai Sun, Zhaojiang Lin, Sanat Sharma, Shereen Oraby, Aaron Colak, Ahmed Aly, Anuj Kumar, Xiaozhong Liu, and Xin Luna Dong. Assomem: Scalable memory qa with multi-signal associative retrieval, 2025

    2025

  47. [47]

    Slice of Life

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory.Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19724–19731, Mar. 2024. 13 A Limitations and Future Work Benchmark scope.STALE is a controlled diagnostic setting focused on one-shot implicit state transitio...

    2024

  48. [48]

    Identify a DIFFERENT attribute A such that A -> B holds under common-sense or world knowledge (e.g., health -> routine, employment -> location, physical ability -> transportation)

  49. [49]

    Think up a new realistic value/description of the attribute A that eventually causes the PROPAGATED IMPLICIT CONFLICT

  50. [50]

    M_new":

    Produce a new user statement (M_new) that: - occurs after the value of attribute A has already updated, - explicitly or implicitly reflects the new value of attribute A, - must NOT mention attribute B or the changed value of B in any way, - must NOT explicitly mention any aspects, related objects in M_old, - is grounded in a completely new scenario, - mak...

  51. [52]

    State Conflict: - Assuming M_new is spoken by the SAME user after a certain time gap, does M_new makes the situation in M_old no longer feasible? - M_new must clearly mention a new value/description of the attribute that is strictly incompatible with the one established in M_old

  52. [53]

    don’t",

    Implicit Constraints (Type I Compliance): - NO explicit linguistic negation (phrases like "don’t", "instead of"). - M_new must NOT explicitly mention the name of the underlying attribute. - M_new must NOT explicitly mention the surface text, objects, or scenario of M_old. ### Output Format (JSON) { "plausibility": { "pass": true/false, "reasoning": "brief...

  53. [54]

    Independent Plausibility: - Are both M_old and M_new natural, realistic statements in real life? - They must not sound too absurd in isolation

  54. [55]

    Propagated State Conflict (A -> B Dependency): - M_old relies on a specific value of an attribute (let’s call it Attribute B). - Does M_new introduce a completely different attribute/event (Attribute A)? - Does Attribute A causally or logically propagate to invalidate the value of Attribute B? - Is there a plausible common-sense dependency (A -> B) that m...

  55. [56]

    I don’t",

    Implicit Constraints (Type II Compliance): - NO explicit linguistic negation (phrases like "I don’t", "instead of"). - M_new must NOT explicitly mention or negate Attribute B. - M_new must NOT explicitly mention any surface text, objects, or scenario of M_old. 19 - M_new must NOT explicitly mention the causal dependency chain (A -> B) or directly state th...

  56. [57]

    User is vegetarian

    Logical Contradiction: It CONTRADICTS, NEGATES, or INVALIDATES the Established Fact. ( e.g., Fact: "User is vegetarian", Chat: "I ate a steak today" -> UNSAFE)

  57. [58]

    The user moved recently

    Logical Elaboration/Supplement: It SUPPLEMENTS, ELABORATES ON, or acts as a DIRECT CONTINUATION of the Established Fact. (e.g., Fact: "The user moved recently", Chat: "I’m really enjoying the West Coast weather now" -> UNSAFE, because it acts as a contextual puzzle piece). Be extremely rigorous. Minor, purely coincidental topic overlaps (e.g., both mentio...

  58. [59]

    date_old MUST be in 2027

    2027

  59. [60]

    date_new MUST be later than date_old

  60. [61]

    { time_gap}

    The gap from date_old to date_new should match this annotation when possible: "{ time_gap}"

  61. [62]

    Choose realistic month/day/time values if either fact suggests seasonality, school/ work timing, holidays, weather, routines, recovery periods, deadlines, travel, or other temporal clues

  62. [63]

    If no strong clue exists, choose a normal mid-year daytime date_old and apply the annotated gap. 23

  63. [64]

    {m_old_text}

    Output JSON only. M_old: "{m_old_text}" M_new: "{m_new_text}" Time gap annotation: "{time_gap}" Output format: {{ "reasoning": "brief explanation", "date_old": "YYYY-MM-DD HH:MM", "date_new": "YYYY-MM-DD HH:MM" }} Latest Plausible Query-Time Prompt You are an expert logical timeframe estimator. A user established a New State on a specific date. Sometime a...

  64. [65]

    for 6 weeks

    If M_new has an explicit duration (e.g., "for 6 weeks"), calculate the exact expiration date

  65. [66]

    If M_new is a temporary condition (e.g., an urgent deadline), use common sense to limit the lifespan

  66. [67]

    If M_new is semi-permanent or permanent (e.g., moved to a new city, became a vegetarian, bought a car), output a date far into the future

  67. [68]

    reasoning

    Provide the MAXIMUM plausible timeframe, as long as logically sound. Output Format (JSON): {{ "reasoning": "brief explanation", "latest_plausible_date": "YYYY-MM-DD HH:MM" }} Temporal Validity Audit Prompt Audit this benchmark sample. Sample fields: uid: {uid} M_old: {M_old} M_new: {M_new} explanation: {explanation} time_gap_annotation: {time_gap} Relevan...