Computation and Language · arXiv:2610.34073 v1 · submitted 25 Oct 2026 · 13 references
Memory Tax: Dissecting the Net Value of Agent Memory
The paper says
“Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear.”
more of the abstract
Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history. They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community. Project Page: https://hexuan21.github.io/MemoryTax/
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Solid empirical study finding that many agent memory systems underperform a simple RAG baseline, introducing a useful "memory tax" framework, but the generality is tempered by retrieval-friendly benchmarks. This line is wrong
3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 13 references say.
3 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 3
The abstract contains a stray rendering artifact "/gl⌢be" before "Project Page". Please remove it.
The paper says, at Appendix A.2 and references
Pith says: minor objection 2 of 3
There are spaces in author names such as "V olcengine" and "V olker Tresp". Please unify to remove the extra space.
show 1 more
The paper says, at Table 1
Pith says: minor objection 3 of 3
The footnote about GPT-5.6-luna scoring lower on PersonaMem-v2 is placed in Appendix A.5 but could be briefly noted in the main text to avoid confusion, as it is an interesting pattern.
What would settle it
Re-run LongMemEval-s (Qwen3.6-27B) for each memory system using its original default top-k (as released, without retrieval scope alignment) and compare the relative performance gap to RAG (1rd). If more than one system that underperforms RAG under RS alignment now outperforms RAG, then the central finding may partly depend on the RS adjustment rather than on inherent memory limitations.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee accept: 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should know this paper for two things: (1) it runs a careful, controlled comparison of a dozen agent memory systems against naive RAG across three benchmarks, and finds that most memory systems fail to beat RAG, often by clear margins; (2) it introduces the conceptual lens of 'memory tax'—the losses that arise from memory processing itself—and dissects that tax into faithfulness, accessibility, and robustness dimensions.
The evaluation is the most systematic I've seen on this question. They align retrieval scope so that every system gets a comparable budget, run ablations on the RAG pipeline to show it isn't a lucky configuration, and trace failures back to specific mechanisms: information lost in construction, evidence buried or fragmented, and added sensitivity to backbone changes. They also acknowledge the obvious caveat—that the benchmarks they use are largely retrieval-friendly, which limits headroom for memory to show benefit—and they show this explicitly by running simple IDF-keyword baselines that already score 50%+ on some benchmarks. Credit where it's due: the authors are open about the scope and don't oversell.
The soft spots are proportionate. The central result—memory tax exists and is often negative—is solid within the tested conditions. The biggest question is how far this generalizes. The three main benchmarks lean toward factual recall and preference questions; tasks requiring abstraction, long-range reasoning, or multi-session integration could shift the balance. The experiential memory and world-model extensions are narrower and less conclusive, so the claim of generalization is more tentative. Also, the memory tax is defined relative to the authors' RAG pipeline; switch to a stronger retriever and the baseline moves.
Who should read it? Anyone working on agent memory, especially those designing new systems. It provides a clean reference point and a diagnostic vocabulary. The paper deserves peer review—it's empirically grounded, reproducible, and addresses an important question without overclaiming.
The strongest objection
The paper's core finding—that many memory systems underperform a carefully constructed naive RAG baseline—is robustly supported by controlled experiments, ablations, and internal consistency checks. The most salient threat to generality (benchmark representativeness) is explicitly acknowledged by the authors (b122, b188). Within the paper's own terms, no critical flaw or unrecognized assumption undermines the central claim. The retrieval scope alignment and adjustments to memory systems are documented and appear fair; if anything, they favor the memory baselines. The analysis of memory tax via faithfulness, accessibility, and robustness is methodical. The extension to experiential and world-model memory widens the claim's scope. No hidden circularity, definitional slip, or unsupported leap was identified.
The reader identified benchmark representativeness as the weakest assumption, which the paper itself highlights as a limitation. My stress-test found no additional load-bearing concern that would alter the verdict. The paper's empirical methodology is thorough: it controls for retrieval scope, unifies embedding/judge backbones, includes extensive ablations (Appendix B), and transparently documents deviations from default configurations that favor memory systems. The internal analysis (faithfulness, accessibility, robustness) dissects the causes of memory tax in a principled way. The central claim—that many memory systems underperform naive RAG on these benchmarks—is convincingly demonstrated. I concur with the reader's ACCEPT verdict and high confidence. The proposed concrete test would further validate the robustness of the RS alignment but is not expected to overturn the result given the paper's own checks.
The referee's report
If the central finding holds, the paper has substantial significance for the agent memory community: it provides a rigorous empirical baseline (naive RAG) that is often missing in prior evaluations, it introduces the actionable concept of memory tax, and it offers practical design directions. The paper is likely to shift evaluation practices toward measuring net value over raw retrieval rather than standalone performance. The thoroughness of the evaluation—multiple benchmarks, backbones, ablations, retrieval scope alignment, and explicit limitation statements—adds to its credibility.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that building complex memory systems for long-horizon LLM agents often yields a negative net value when compared to directly retrieving relevant chunks from the raw interaction history (naive RAG). Across three benchmarks and multiple model backbones, many dedicated memory systems underperform naive RAG, despite higher costs. The authors propose dissecting net value into 'memory benefit' (gains from organizing, distilling) and 'memory tax' (losses from evidence lost, obscured, or made harder to access). They trace tax to three internal causes: faithfulness (information loss during construction), accessibility (harder to retrieve preserved evidence), and robustness (amplified sensitivity to model changes). The paper also shows that current benchmarks are retrieval-friendly, leaving limited headroom for memory to show benefit, which makes the tax more visible.
What carries the argument. The key machinery is the concept of 'memory deviations' from raw history. Every nontrivial memory operation (extraction, summarization, structuring) changes how information is stored and accessed relative to the original text. These deviations are double-edged: they enable benefits like reorganization and preference inference, but also introduce tax through faithfulness loss (evidence lost or distorted during construction), accessibility degradation (evidence preserved but harder to retrieve due to fragmentation, dilution, or rephrasing), and robustness fragility (amplified sensitivity when the construction or query model changes). The framework is reference-relative, with naive RAG serving as the fixed baseline for measuring the deviations.
Core claim. The central discovery is that memory acts as a double-edged deviation from raw history: while intended to improve information organization, memory operations inevitably introduce harmful changes that can remove or obscure evidence. This gives rise to the overlooked 'memory tax'. The paper demonstrates that on current benchmarks, which are retrieval-friendly, the tax often outweighs the benefit, leading to a negative net value of building memory. The authors generalize this benefit-tax view to experiential memory and world-model memory, showing the pattern persists. They further derive simple design implications, such as recovering source history during retrieval or making memory accesses recursive, which reduce the tax.
The premise it rests on. The central findings depend on the assumption that the evaluated benchmarks (static, offline, English-language, retrieval-friendly) are representative of real-world agent memory tasks; the authors explicitly acknowledge this limitation and note that future work should extend to more dynamic, interleaved, and multilingual settings.
If the paper is right (Pith's inferences, not the authors')
Memory tax explains why lightweight or no-memory approaches often match or beat elaborate memory systems on existing benchmarks; the tax can erase the intended benefit.
The analysis directly informs design: making memory deviations recoverable (e.g., joint retrieval from source history) or introducing recursive memory access reduces tax and improves net value.
Future evaluations of agent memory should include naive RAG as a mandatory baseline to expose the added value (or cost) of any memory mechanism.
The benchmark headroom observation suggests that many current benchmarks may be saturating; harder, more realistic scenarios are needed to differentiate memory systems that genuinely help.
The benefit-tax view extends beyond conversational memory to experiential and world-model memory, implying a general principle for any memory operation that transforms raw data.
The work implicitly challenges the current trend toward increasingly complex memory architectures, suggesting that simpler retrieval from raw history is an underappreciated and often superior alternative.
The robustness finding hints that memory systems could be designed to be less brittle by, for example, using model-agnostic representations or ensembling across construction models.
The idea of modeling future demand (treating memory as an implicit recommendation system) is a provocative new lens that could reframe memory construction as a prediction problem rather than a storage problem.
If the benchmark headroom argument generalizes, then many published memory improvements may be artifacts of easy test settings rather than genuine advances, which would have strong implications for the credibility of the current evaluation paradigm.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history. They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community. Project Page: https://hexuan21.github.io/MemoryTax/
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-27
The authors' main claim, as the classifier picked it out 1.00
“Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value.”
How the main claim is stated: asserted 0.77
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.25 · human participants 0.05 · controlled comparison 0.94 · randomized assignment 0.02 · states quantitative result 0.06 · compares with prior results 0.94 · mentions code or data availability 0.98
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. | gap 0.75 | 0.05 | 0.04 | asserted 0.87 |
| Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. | result 1.00 | 0.97 | 0.82 | asserted 0.77 |
| Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. | contribution 0.79 | 0.87 | 0.12 | asserted 0.96 |
| These gains and losses arise because memory process changes how the raw information is organized and accessed. | interpretation 0.80 | 0.22 | 0.04 | asserted 1.00 |
| We view such changes as the double-edged deviations from the raw history. | interpretation 0.78 | 0.19 | 0.05 | asserted 0.99 |
| They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. | interpretation 0.71 | 0.14 | 0.10 | hedged possibility 0.76 |
| Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. | contribution 0.93 | 0.53 | 0.11 | asserted 0.88 |
| We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. | future work 0.65 | 0.11 | 0.18 | hedged possibility 0.88 |
| We will open-source the code and results to the community. | future work 0.58 | 0.34 | 0.02 | asserted 0.90 |
| Project Page: https://hexuan21.github.io/MemoryTax/ | contribution 0.50 | 0.31 | 0.02 | asserted 0.59 |
What kind of paper the classifier reports this is: empirical study 0.95 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 7,482 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (29)
Showing 24 of the paper's 29 figures. Figures are the authors', reproduced from the paper.
References (13)
- [4] The needed evidence may be scattered across several conversations; combine related memories before answering on Pith
- [5] Memgpt: Towards llms as operating systems. Preprint, arXiv:2310.08560. Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. 2025. Zep: A tempo- ral knowledge graph architecture for agent memory. Preprint, arXiv:2501.13956. Mohit Shridhar, Xingdi Y... on PithPith's reviewarXiv
- [6] Appworld: A controllable world of apps and people for benchmarking interactive coding agents. Preprint, arXiv:2407.18901. V olcengine. 2026. OpenViking: An open-source con- text database for ai agents. https://github.com/ volcengine/OpenViking. Feng Wang, Yuqing Li, and Han Xi... on PithPith's reviewarXiv
- [7] Context as memory: Scene-consistent inter- active long video generation with memory retrieval. InProceedings of the SIGGRAPH Asia 2025 Confer- ence Papers, pages 1–11. Yanwei Yue, Boci Peng, Xuanbo Fan, Jiaxin Guo, Qiankun Li, and Yan Zhang. 2026. Mem-t: Densify- ing rewards f... on PitharXiv
- [12] Do not make up information. If the needed information does not appear in your memories, say that the information is not available. [USER] <MEMORY> [[MEMORY BLOCK]] (OPTIONAL USER PROFILE SECTION:) <USER PROFILE> [[USER PROFILE]] Current date: [[QUESTION DATE]] Question: [[QUES... on Pith
- [13] Be concise and specific. For dates use the format "15 July 2023"; for durations answer in years/months/days on Pith
- [14] Time: reason from the timestamps attached to your memories and from the current date given with the question. Resolve relative expressions ("last year", "how long ago") against the current date, not against when the conversation happened on Pith
- [15] Conflicts: if memories disagree because something changed over time, trust the most recent memory, unless the question asks about an earlier point in time on Pith
- [16] The needed evidence may be scattered across several conversations; combine related memories before answering. [USER] <MEMORY> [[MEMORY BLOCK]] (OPTIONAL USER PROFILE SECTION:) <USER PROFILE> [[USER PROFILE]] Current date: [[QUESTION DATE]] Question: [[QUESTION]] Read all sente... on Pith
- [2023] Mind2web: Towards a generalist agent for the web.Preprint, arXiv:2306.06070. Sedigheh Eslami, Maksim Gaiduk, Markus Krimmel, Louis Milliken, Bo Wang, and Denis Bykov. 2026. Diffusion-pretrained dense and contextual embed- dings.Preprint, arXiv:2602.11151. EverMind-AI. 2026. Ev... on PithPith's reviewarXiv
- [2024] Evaluating very long-term conversational memory of llm agents.Preprint, arXiv:2402.17753. Xuanyu Meng, Jiashuo Sun, Jash Rajesh Parekh, and Jiawei Han. 2026. Ensi-rag: Entity-structure- indexed retrieval-augmented generation for long- document question answering.arXiv preprint... on PithPith's reviewarXiv
- [2025] Memory os of ai agent. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25972–25981. Seth Karten, Alex L. Zhang, Kevin Thomas, Sebas- tian Müller, Elie Bakouch, Daniel Auras, Mika Seng- haas, Fares Obeid, Konstantin Dunas, Johanne... on PithPith's reviewarXiv
- [2026] What deserves memory: Adaptive memory dis- tillation for llm agents.Preprint, arXiv:2508.03341. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang on PithPith's reviewarXiv
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment