Artificial Intelligence · arXiv:2610.18052 v1 · submitted 13 Oct 2026 · 38 references
Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents
The paper says
“Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests.”
more of the abstract
It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://github.com/Nboxff/bemula-benchmark.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Solid benchmark paper for a genuinely underevaluated problem; the central negative finding is robust and important despite the synthetic-data limitation. This line is wrong
3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 38 references say.
3 minor objections
The paper says, at Figure 3
Pith says: minor objection 1 of 3
The labels in panel (c) are very small; consider increasing font size and adding a legend for the dot/diamond symbols to improve readability.
The paper says, at Table 1
Pith says: minor objection 2 of 3
The table caption says 'All scores are percentages' but the column headers do not repeat the '%' unit. Consider adding '%' in the header row for clarity.
show 1 more
The paper says, at Section 3.2.5
Pith says: minor objection 3 of 3
The description of the 'attack' suite states that the gold answer is the marker [NO LEAK] and that the frames are never paraphrased. This is clear, but it would help to explicitly note that the attack frames are fixed and not generated per instance to ensure consistent difficulty.
What would settle it
Independently run the BEMULA evaluation harness on the BEMULA-Lite subset (31 instances, 848 questions) using a different backend LLM (e.g., Claude Sonnet 4) as the judge, and compare the RPR and score against the reported values. If the RPR remains below 2%, the central finding is robust to judge model choice.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 14 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee accept: 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should know two things about this paper. First, it defines and benchmarks a genuine blind spot: LLM agent memory systems are not tested for whether they can honor explicit deletion requests, and the paper shows convincingly that they cannot. Second, the benchmark design is thoughtful—paired control versions isolate forgetting from storage failures, the five test suites prevent degenerate strategies like blanket deletion, and the LLM judge is validated (98% agreement on 100 human-annotated items). That is real work.
What is new: the task definition (memory unlearning in agents), the six-type taxonomy of forget requests, and the paired-control metric that separates never-remembered from genuinely forgotten. None of the existing agent memory benchmarks (LoCoMo, LongMemEval, MemoryAgentBench) test explicit forget requests. The paper ships data and code, and the evaluation harness looks reproducible.
The central result is stark and clean: across six memory systems and three backbone LLMs, no system passes more than 1.5% of forget requests end to end. Most pass none. The information either stays extractable under simple attacks or the system deletes far more than requested. The failure is structural—even replacement requests (F6), where efficacy looks decent on strong backbones, still leak the old value under attack (attack robustness ≤30.5). These results will hold up.
The soft spots are real but the authors own them. The dialogues are synthetic (one pipeline), and the benchmark covers only conversational memory—tool-using agents leave forgotten values in caches, logs, or external services. Both limitations are stated in the conclusion. The LLM judge is validated on 100 items, but model bias cannot be ruled out for the remaining 1,642 items. The per-type and per-length breakdowns are descriptive, not confirmatory, as the authors note (only 13 F4 requests, 3 xlong instances). None of these undermine the headline: memory unlearning is unsolved.
Who this is for: anyone building or evaluating LLM agent memory systems, and anyone working on privacy or safety for deployed agents. The benchmark deserves a serious referee. It is not a method paper—it does not propose a solution—but it establishes a measurement standard that the field needs. I would accept for peer review and recommend a strong revise for clarity on the LLM judge validation protocol and the per-type uncertainty (they already flag the wide intervals, but the main text could state them more explicitly).
The strongest objection
The benchmark design is sound, the paired-control metric is principled, the empirical results are stark and consistent across backbones and systems, and the authors fully acknowledge the limitations of synthetic data and the LLM judge. I find no load-bearing concern that would overturn the central conclusion that memory unlearning is an open challenge.
The paper constructs a careful benchmark with paired controls, multiple test suites, and an LLM judge validated with 98% agreement on 100 items. The main result—that no system passes more than 1.5% of forget requests end-to-end—is robustly supported by the data and bootstrap confidence intervals. The authors explicitly note the limitations of synthetic data and LLM judge bias, so these are acknowledged rather than hidden. I identified no internal inconsistency or missing control that would weaken the central claim.
The referee's report
If the benchmark is valid, this is an important contribution. It formalizes a previously underevaluated capability—honoring explicit forget requests—and provides a controlled, reproducible evaluation framework. The stark negative results across all tested systems and backbones convincingly demonstrate that current agent memory designs cannot reliably delete specific user-requested information without either leaking it or deleting too much. This finding has direct implications for privacy, compliance (e.g., right to be forgotten), and the design of future memory systems. The paired-control methodology is principled and avoids degenerate scoring strategies.
Pith's reading of the paper (a machine's summary, not the authors')
This paper introduces Bemula, the first benchmark dedicated to evaluating whether LLM agents can forget information when explicitly asked by the user. The benchmark comprises 50 multi-session dialogues with 135 forget requests of six distinct types and 1,742 test questions, each paired with a control version that has no forget requests. Evaluating six mainstream memory systems and three backbone LLMs, the paper finds a universal failure: no system passes more than 1.5% of forget requests end to end. Systems either leave the requested information extractable under simple attacks or delete far more than what was requested. The authors argue that memory unlearning remains an open challenge for agent memory design and that existing systems, which optimize for retention, cannot reliably honor deletion requests.
What carries the argument. The central machinery of the benchmark is the paired control version: every dialogue instance has a forget version (with natural-language forget requests) and a control version (where those requests are replaced by neutral filler). All test questions are scored against their counterpart in the control version, ensuring that only systems that actually stored the target can earn credit for forgetting. The five test suites — retention, efficacy, preservation, relearning, and attack robustness — collectively measure whether forgetting is selective, complete, and robust to extraction.
Core claim. The paper establishes that current agent memory systems fundamentally fail at memory unlearning when faced with explicit natural-language forget requests. The key discovery is that no evaluated system can selectively remove a specific piece of information while preserving the rest of memory and resisting adversarial extraction. Systems that appear to forget often over-delete or leave the target fact recoverable through simple attack probes. The paper's paired control design isolates genuine forgetting from a failure to encode, showing that the problem is not storage but deletion.
The premise it rests on. The dialogues are synthetic and generated by a single pipeline, so the benchmark's conclusions depend on how well these scenarios represent the full complexity and unpredictability of real user-agent interactions.
If the paper is right (Pith's inferences, not the authors')
Current memory systems are unsuitable for privacy-sensitive applications where users have the right to delete personal data.
Future agent memory architectures must incorporate explicit, scoped deletion mechanisms, moving beyond designs that only write, retrieve, and compress.
Benchmarks for agent memory should include forgetting evaluations with paired controls and attack probes to capture real-world deletion requirements.
The wide gap between retention (high) and forgetting efficacy (low) indicates a fundamental design trade-off that systems must address.
The benchmark design could be extended to real-user dialogues to validate whether the synthetic findings hold in natural interactions.
Solving memory unlearning may require fundamentally new memory representations that support surgical erasure, not just retrieval-based or note-based systems.
The nearly universal failure suggests that current systems optimize for retention without considering deletion, which may be a harder problem than retention itself.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://github.com/Nboxff/bemula-benchmark.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-14
The authors' main claim, as the classifier picked it out 0.93
“Memory unlearning therefore remains an open challenge for agent memory design.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.96 · states limitation 0.25 · human participants 0.12 · controlled comparison 0.95 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.67 · mentions code or data availability 0.99
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Long-term memory has become a core component of large language model (LLM) agents. | background 1.00 | 0.04 | 0.01 | asserted 1.00 |
| As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. | background 1.00 | 0.06 | 0.01 | asserted 0.98 |
| A memory system should honor such requests. | background 1.00 | 0.04 | 0.03 | asserted 0.96 |
| It should remove exactly what was requested and keep the rest of its memory intact. | background 1.00 | 0.04 | 0.03 | asserted 0.97 |
| However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. | gap 1.00 | 0.11 | 0.04 | asserted 0.98 |
| To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. | contribution 1.00 | 0.93 | 0.04 | asserted 1.00 |
| It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. | method 0.83 | 0.87 | 0.02 | asserted 1.00 |
| Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. | method 0.99 | 0.87 | 0.02 | asserted 1.00 |
| Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. | method 0.96 | 0.85 | 0.03 | asserted 1.00 |
| On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. | method 0.94 | 0.94 | 0.03 | asserted 1.00 |
| The results are negative. | result 0.97 | 0.81 | 0.45 | asserted 0.99 |
| Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. | result 1.00 | 0.95 | 0.59 | asserted 1.00 |
| These systems either leave the requested information extractable under simple attacks or delete far more than requested. | result 0.93 | 0.85 | 0.17 | asserted 1.00 |
| Memory unlearning therefore remains an open challenge for agent memory design. | interpretation 0.99 | 0.28 | 0.85 | asserted 1.00 |
| We release our data and code at https://github.com/Nboxff/bemula-benchmark. | contribution 0.97 | 0.82 | 0.02 | asserted 0.97 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 10,554 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Figures are the authors', reproduced from the paper.
References (38)
- [1] Memory OS of AI Agent , author=. 2025 , eprint= on Pith
- [2] Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory , author=. arXiv preprint arXiv:2504.19413 , year= on PithPith's reviewarXiv
- [3] A-Mem: Agentic memory for llm agents , author=. Advances in Neural Information Processing Systems , year= on Pith
- [4] SimpleMem: Efficient Lifelong Memory for LLM Agents , author=. arXiv preprint arXiv:2601.02553 , year= on PithPith's reviewarXiv
- [5] Evaluating very long-term conversational memory of llm agents , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [6] Memoryarena: Benchmarking agent memory in interdependent multi-session agentic tasks , author=. arXiv preprint arXiv:2602.16313 , year= on PithPith's reviewarXiv
- [7] Longmemeval: Benchmarking chat assistants on long-term interactive memory , author=. arXiv preprint arXiv:2410.10813 , year= on PithPith's reviewarXiv
- [8] Hu, Yuanzhe and Wang, Yu and McAuley, Julian , booktitle =. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions , url = on Pith
- [9] Trust no bot: Discovering personal disclosures in human-llm conversations in the wild , author=. arXiv preprint arXiv:2407.11438 , year= on PithPith's reviewarXiv
- [10] A right to reasonable inferences: re-thinking data protection law in the age of big data and AI , author=. Colum. Bus. L. Rev. , pages=. 2019 , publisher= on Pith
- [11] A survey of machine unlearning , author=. ACM Transactions on Intelligent Systems and Technology , volume=. 2025 , publisher= on Pith
- [12] Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. Advances in neural information processing systems , volume= on Pith
- [13] Longbench: A bilingual, multitask benchmark for long context understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages= on Pith
- [14] RULER: What's the real context size of your long-context language models? , author=. arXiv preprint arXiv:2404.06654 , year= on PithPith's reviewarXiv
- [15] Zhang, Xinrong and Chen, Yingfa and Hu, Shengding and Xu, Zihang and Chen, Junhao and Hao, Moo and Han, Xu and Thai, Zhen and Wang, Shuo and Liu, Zhiyuan and others , booktitle= on Pith
- [16] Membench: Towards more comprehensive evaluation on the memory of llm-based agents , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages= on Pith
- [17] Uddin, Md Nayem and Shubham, Kumar and Blanco, Eduardo and Baral, Chitta and Wang, Gengyu. From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents. Findings of the A ssociation for C omputational L inguistics: ACL 2026. 2026. doi:10.18653/v1/2026.findi... on Pithdoi:10.18653/v1/2026.findings-acl.1337
- [18] Generative agents: Interactive simulacra of human behavior , author=. Proceedings of the 36th annual acm symposium on user interface software and technology , pages= on Pith
- [19] Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume= on Pith
- [20] Jailbroken: How does llm safety training fail? , author=. Advances in neural information processing systems , volume= on Pith
show 18 more
- [21] Memgpt: Towards llms as operating systems , author=. arXiv preprint arXiv:2310.08560 , year= on PithPith's reviewarXiv
- [22] Scaling synthetic data creation with 1,000,000,000 personas , author=. arXiv preprint arXiv:2406.20094 , year= on PithPith's reviewarXiv
- [23] Mirix: Multi-agent memory system for llm-based agents , author=. arXiv preprint arXiv:2507.07957 , year= on PithPith's reviewarXiv
- [24] Zep: a temporal knowledge graph architecture for agent memory , author=. arXiv preprint arXiv:2501.13956 , year= on PithPith's reviewarXiv
- [25] Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning , author=. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [26] A Classification of SQL Injection Attacks and Countermeasures. , author=. ISSSE , year= on Pith
- [27] Yuli Qian and Zechuan Yang and Wenbiao Ding and Hongzhi Li and Yutao Xie , booktitle=. Do. 2026 , url= on Pith
- [28] Pawelczyk, Martin and Neel, Seth and Lakkaraju, Himabindu , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher = on Pith
- [29] Cao, Yinzhi and Yang, Junfeng , title =. Proceedings of the 2015 IEEE Symposium on Security and Privacy , pages =. 2015 , isbn =. doi:10.1109/SP.2015.35 , abstract = on Pithdoi:10.1109/sp.2015.35
- [30] Fang, Jizhan and Deng, Xinle and Xu, Haoming and Jiang, Ziyan and Tang, Yuqi and Xu, Ziwen and Deng, Shumin and Yao, Yunzhi and Wang, Mengru and Qiao, Shuofei and Chen, Huajun and Zhang, Ningyu , booktitle =. LightMem: Lightweight and Efficient Memory-Augmented Generation , url = on Pith
- [31] LangMem SDK for agent long-term memory , author =. 2025 , url = on Pith
- [32] Bjork, Robert A. , year =. Theoretical implications of directed forgetting , booktitle = on Pith
- [33] Levitt, Barbara and March, James G. , title =. Annual Review of Sociology , volume =. 1988 , publisher = on Pith
- [34] Zhang, Zeyu and Dai, Quanyu and Bo, Xiaohe and Ma, Chen and Li, Rui and Chen, Xu and Zhu, Jieming and Dong, Zhenhua and Wen, Ji-Rong , title =. ACM Trans. Inf. Syst. , month = sep, articleno =. 2025 , issue_date =. doi:10.1145/3748302 , abstract = on Pithdoi:10.1145/3748302
- [35] GPT-5.6 Preview (Sol/Terra/Luna) System Card , author=. 2026 , month= on Pith
- [36] DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression , author=. arXiv preprint arXiv:2609.19969 , year= on PitharXiv
- [37] Qwen3 Technical Report , author=. 2025 , eprint= on Pith
- [38] Lin, Ji and Tang, Jiaming and Tang, Haotian and Yang, Shang and Xiao, Guangxuan and Han, Song , title =. GetMobile: Mobile Comp. and Comm. , month = jan, pages =. 2025 , issue_date =. doi:10.1145/3714983.3714987 , abstract = on Pithdoi:10.1145/3714983.3714987
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Forget the Lineage: Forward-Secure, Lineage-Complete Cryptographic Erasure for LLM-Agent Memory cs.CR · Nov 2026
- LEDGER: Auditable Agent Memory Under Change cs.MA · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment