Pith.

Computation and Language · arXiv:2610.34073 v1 · submitted 25 Oct 2026 · 13 references

Memory Tax: Dissecting the Net Value of Agent Memory

Xuan He, Sining Zhoubian, Ruozhen Yang, Yuhao Wen, Zhineng Wang, Yuyang Hu, Ke Yang, Yuyu Zhang

show 3 more authors

Ping Nie, Jiawei Han, ChengXiang Zhai

The paper says

“Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear.”

more of the abstract

Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history. They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community. Project Page: https://hexuan21.github.io/MemoryTax/

from the abstract

Pith says

Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Solid empirical study finding that many agent memory systems underperform a simple RAG baseline, introducing a useful "memory tax" framework, but the generality is tempered by retrieval-friendly benchmarks. This line is wrong

3 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 13 references say.

3 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 3

The abstract contains a stray rendering artifact "/gl⌢be" before "Project Page". Please remove it.

The paper says, at Appendix A.2 and references

Pith says: minor objection 2 of 3

There are spaces in author names such as "V olcengine" and "V olker Tresp". Please unify to remove the extra space.

show 1 more

The paper says, at Table 1

Pith says: minor objection 3 of 3

The footnote about GPT-5.6-luna scoring lower on PersonaMem-v2 is placed in Appendix A.5 but could be briefly noted in the main text to avoid confusion, as it is an interesting pattern.

What would settle it

Re-run LongMemEval-s (Qwen3.6-27B) for each memory system using its original default top-k (as released, without retrieval scope alignment) and compare the relative performance gap to RAG (1rd). If more than one system that underperforms RAG under RS alignment now outperforms RAG, then the central finding may partly depend on the RS adjustment rather than on inherent memory limitations.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 27 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 32 s · 43,901 tokens read, 3,089 written
  3. Editor wrote the verdict above 15 s · 44,644 tokens read, 1,114 written
  4. Referee accept: 3 minor 15 s · 44,144 tokens read, 1,559 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

You should know this paper for two things: (1) it runs a careful, controlled comparison of a dozen agent memory systems against naive RAG across three benchmarks, and finds that most memory systems fail to beat RAG, often by clear margins; (2) it introduces the conceptual lens of 'memory tax'—the losses that arise from memory processing itself—and dissects that tax into faithfulness, accessibility, and robustness dimensions.

The evaluation is the most systematic I've seen on this question. They align retrieval scope so that every system gets a comparable budget, run ablations on the RAG pipeline to show it isn't a lucky configuration, and trace failures back to specific mechanisms: information lost in construction, evidence buried or fragmented, and added sensitivity to backbone changes. They also acknowledge the obvious caveat—that the benchmarks they use are largely retrieval-friendly, which limits headroom for memory to show benefit—and they show this explicitly by running simple IDF-keyword baselines that already score 50%+ on some benchmarks. Credit where it's due: the authors are open about the scope and don't oversell.

The soft spots are proportionate. The central result—memory tax exists and is often negative—is solid within the tested conditions. The biggest question is how far this generalizes. The three main benchmarks lean toward factual recall and preference questions; tasks requiring abstraction, long-range reasoning, or multi-session integration could shift the balance. The experiential memory and world-model extensions are narrower and less conclusive, so the claim of generalization is more tentative. Also, the memory tax is defined relative to the authors' RAG pipeline; switch to a stronger retriever and the baseline moves.

Who should read it? Anyone working on agent memory, especially those designing new systems. It provides a clean reference point and a diagnostic vocabulary. The paper deserves peer review—it's empirically grounded, reproducible, and addresses an important question without overclaiming.

The strongest objection

The paper's core finding—that many memory systems underperform a carefully constructed naive RAG baseline—is robustly supported by controlled experiments, ablations, and internal consistency checks. The most salient threat to generality (benchmark representativeness) is explicitly acknowledged by the authors (b122, b188). Within the paper's own terms, no critical flaw or unrecognized assumption undermines the central claim. The retrieval scope alignment and adjustments to memory systems are documented and appear fair; if anything, they favor the memory baselines. The analysis of memory tax via faithfulness, accessibility, and robustness is methodical. The extension to experiential and world-model memory widens the claim's scope. No hidden circularity, definitional slip, or unsupported leap was identified.

The reader identified benchmark representativeness as the weakest assumption, which the paper itself highlights as a limitation. My stress-test found no additional load-bearing concern that would alter the verdict. The paper's empirical methodology is thorough: it controls for retrieval scope, unifies embedding/judge backbones, includes extensive ablations (Appendix B), and transparently documents deviations from default configurations that favor memory systems. The internal analysis (faithfulness, accessibility, robustness) dissects the causes of memory tax in a principled way. The central claim—that many memory systems underperform naive RAG on these benchmarks—is convincingly demonstrated. I concur with the reader's ACCEPT verdict and high confidence. The proposed concrete test would further validate the robustness of the RS alignment but is not expected to overturn the result given the paper's own checks.

The referee's report

If the central finding holds, the paper has substantial significance for the agent memory community: it provides a rigorous empirical baseline (naive RAG) that is often missing in prior evaluations, it introduces the actionable concept of memory tax, and it offers practical design directions. The paper is likely to shift evaluation practices toward measuring net value over raw retrieval rather than standalone performance. The thoroughness of the evaluation—multiple benchmarks, backbones, ablations, retrieval scope alignment, and explicit limitation statements—adds to its credibility.

Pith's reading of the paper (a machine's summary, not the authors')

The paper argues that building complex memory systems for long-horizon LLM agents often yields a negative net value when compared to directly retrieving relevant chunks from the raw interaction history (naive RAG). Across three benchmarks and multiple model backbones, many dedicated memory systems underperform naive RAG, despite higher costs. The authors propose dissecting net value into 'memory benefit' (gains from organizing, distilling) and 'memory tax' (losses from evidence lost, obscured, or made harder to access). They trace tax to three internal causes: faithfulness (information loss during construction), accessibility (harder to retrieve preserved evidence), and robustness (amplified sensitivity to model changes). The paper also shows that current benchmarks are retrieval-friendly, leaving limited headroom for memory to show benefit, which makes the tax more visible.

What carries the argument. The key machinery is the concept of 'memory deviations' from raw history. Every nontrivial memory operation (extraction, summarization, structuring) changes how information is stored and accessed relative to the original text. These deviations are double-edged: they enable benefits like reorganization and preference inference, but also introduce tax through faithfulness loss (evidence lost or distorted during construction), accessibility degradation (evidence preserved but harder to retrieve due to fragmentation, dilution, or rephrasing), and robustness fragility (amplified sensitivity when the construction or query model changes). The framework is reference-relative, with naive RAG serving as the fixed baseline for measuring the deviations.

Core claim. The central discovery is that memory acts as a double-edged deviation from raw history: while intended to improve information organization, memory operations inevitably introduce harmful changes that can remove or obscure evidence. This gives rise to the overlooked 'memory tax'. The paper demonstrates that on current benchmarks, which are retrieval-friendly, the tax often outweighs the benefit, leading to a negative net value of building memory. The authors generalize this benefit-tax view to experiential memory and world-model memory, showing the pattern persists. They further derive simple design implications, such as recovering source history during retrieval or making memory accesses recursive, which reduce the tax.

The premise it rests on. The central findings depend on the assumption that the evaluated benchmarks (static, offline, English-language, retrieval-friendly) are representative of real-world agent memory tasks; the authors explicitly acknowledge this limitation and note that future work should extend to more dynamic, interleaved, and multilingual settings.

If the paper is right (Pith's inferences, not the authors')

Memory tax explains why lightweight or no-memory approaches often match or beat elaborate memory systems on existing benchmarks; the tax can erase the intended benefit.

The analysis directly informs design: making memory deviations recoverable (e.g., joint retrieval from source history) or introducing recursive memory access reduces tax and improves net value.

Future evaluations of agent memory should include naive RAG as a mandatory baseline to expose the added value (or cost) of any memory mechanism.

The benchmark headroom observation suggests that many current benchmarks may be saturating; harder, more realistic scenarios are needed to differentiate memory systems that genuinely help.

The benefit-tax view extends beyond conversational memory to experiential and world-model memory, implying a general principle for any memory operation that transforms raw data.

The work implicitly challenges the current trend toward increasingly complex memory architectures, suggesting that simpler retrieval from raw history is an underappreciated and often superior alternative.

The robustness finding hints that memory systems could be designed to be less brittle by, for example, using model-agnostic representations or ensembling across construction models.

The idea of modeling future demand (treating memory as an implicit recommendation system) is a provocative new lens that could reframe memory construction as a prediction problem rather than a storage problem.

If the benchmark headroom argument generalizes, then many published memory improvements may be artifacts of easy test settings rather than genuine advances, which would have strong implications for the credibility of the current evaluation paradigm.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history. They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community. Project Page: https://hexuan21.github.io/MemoryTax/

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-27

The authors' main claim, as the classifier picked it out 1.00

“Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value.”

How the main claim is stated: asserted 0.77

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.25 · human participants 0.05 · controlled comparison 0.94 · randomized assignment 0.02 · states quantitative result 0.06 · compares with prior results 0.94 · mentions code or data availability 0.98

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. gap 0.75 0.05 0.04 asserted 0.87
Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. result 1.00 0.97 0.82 asserted 0.77
Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. contribution 0.79 0.87 0.12 asserted 0.96
These gains and losses arise because memory process changes how the raw information is organized and accessed. interpretation 0.80 0.22 0.04 asserted 1.00
We view such changes as the double-edged deviations from the raw history. interpretation 0.78 0.19 0.05 asserted 0.99
They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. interpretation 0.71 0.14 0.10 hedged possibility 0.76
Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. contribution 0.93 0.53 0.11 asserted 0.88
We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. future work 0.65 0.11 0.18 hedged possibility 0.88
We will open-source the code and results to the community. future work 0.58 0.34 0.02 asserted 0.90
Project Page: https://hexuan21.github.io/MemoryTax/ contribution 0.50 0.31 0.02 asserted 0.59

What kind of paper the classifier reports this is: empirical study 0.95 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 7,482 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (29)
Figure 1 of the paper
Figure 1. Pareto plot of Accuracy vs. Token-cost. Bubble size is the memory store size in tokens.
Figure 3 of the paper
Figure 3. Original performances vs. w/ gold memory units
Figure 4 of the paper
Figure 4. Retrieval rounds replayed for the evidence to be fully retrieved, under raw chunks vs. memory units.
Figure 6 of the paper
Figure 6. Each row fixes the construction model, while
Figure 8 of the paper
Figure 8. Experiential memory results on agent task benchmarks, memory baselines are green when above RAG
Figure 9 of the paper
Figure 9. Example frame sequences from three evalua
Figure 10 of the paper
Figure 10. Per-query correctness overlap between each memory system and RAG (1rd) on LongMemEval-s, with
Figure 11 of the paper
Figure 11. Per-query correctness overlap between each memory system and RAG (1rd) on PersonaMem-v2, with
Figure 12 of the paper
Figure 12. Per-query correctness overlap between each memory system and RAG (1rd) on MINT-Eval, with
Figure 13 of the paper
Figure 13. QA accuracy with the oracle raw context vs. oracle memory units on LongMemEval-s and PersonaMem
Figure 14 of the paper
Figure 14. Query similarity comparison of gold raw turns vs. gold memory units on LongMemEval-s (top) and
Figure 15 of the paper
Figure 15. Cross-backbone robustness of memory sys
Figure 16 of the paper
Figure 16. Simple retrieval baselines across benchmarks with
Figure 17 of the paper
Figure 17. Experiential memory on agentic tasks, task score (bars, left axis) and tokens per task (diamonds, right
Figure 18 of the paper
Figure 18. Examples of memory benefit: (a) focused atomic extraction, (b) multi-hop linking, and (c) implicit
Figure 19 of the paper
Figure 19. Examples of memory benefit-2: (d) temporal anchoring, (e) explicit forgetting, and (f) processing for
Figure 20 of the paper
Figure 20. Faithfulness case studies-1. Memory Tax · Faithfulness Question: How can I gracefully turn down frequent fam￾ily gatherings where the conversation always shifts to my personal life, without damaging relationships or losing precious time for my creative work? Raw dialogue user Please forget the detail about me feeling pressure from family to marry conflicting with my desire to focus on art and career. NO LOSS SIMPLEMEM The user requested to forget the detail about feeling pressure from family to marry conflicting with a desire to focus on art and career. PARTIAL LOSS MAGMA The user…
Figure 22 of the paper
Figure 22. Faithfulness case studies-3. Memory Tax · Faithfulness Question: How many days did it take for my laptop backpack to arrive after I bought it? Raw dialogue user · 2023/01/24 (Tue) 09:37 . . . By the way, I just started using my new laptop backpack and it’s been great. It arrived on 1/20 and has been a lifesaver for carrying my work laptop back and forth. NO LOSS MAGMA The user requests recommendations for a new wireless mouse due to issues with their current one and mentions that their new laptop backpack, which arrived on 1/20, is performing well. PARTIAL LOSS SIMPLEMEM The user…
Figure 24 of the paper
Figure 24. Examples of accessibility degradation: (a) fragmenting, (b) diluting, and (c) rephrasing.
Figure 25 of the paper
Figure 25. Direct examples of the dual nature of memory-1: the same mechanism of a memory system brings
Figure 26 of the paper
Figure 26. Direct examples of the dual nature of memory-2: the same mechanism of a memory system brings
Figure 27 of the paper
Figure 27. Pareto plots on LongMemEval-s with GPT-5.6 Luna (low), accuracy vs. token cost (left) and accuracy vs. the number of LLM calls (right). Bubble size is the memory store size in tokens. LongMemEval-s consists of 500 corpora (for GPT-5.6 Luna, we use a subset of 100 as in Appendix A.1), and report per-corpus averages. E.2 Full Results of PersonaMem-v2
Figure 28 of the paper
Figure 28. Pareto plots on PersonaMem-v2 with Qwen3.6-27B, accuracy vs. token cost (left) and accuracy vs. the number of LLM calls (right). Bubble size is the memory store size in tokens. PersonaMem-v2 consists of 200 corpora (we use a subset of 40 as mentioned in Appendix A.1) and report per-corpus averages.. E.3 Full Results of MINT-Eval
Figure 29 of the paper
Figure 29. Pareto plots on MINT-Eval with Qwen3.5-4B, accuracy vs. token cost (left) and accuracy vs. the number of LLM calls (right). Bubble size is the memory store size in tokens. MINT-Eval consists of 595 corpora (we use a subset of 60 as mentioned in Appendix A.1), and report per-corpus averages. 34

Showing 24 of the paper's 29 figures. Figures are the authors', reproduced from the paper.

References (13)
  1. [4] The needed evidence may be scattered across several conversations; combine related memories before answering on Pith
  2. [5] Memgpt: Towards llms as operating systems. Preprint, arXiv:2310.08560. Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. 2025. Zep: A tempo- ral knowledge graph architecture for agent memory. Preprint, arXiv:2501.13956. Mohit Shridhar, Xingdi Y... on PithPith's reviewarXiv
  3. [6] Appworld: A controllable world of apps and people for benchmarking interactive coding agents. Preprint, arXiv:2407.18901. V olcengine. 2026. OpenViking: An open-source con- text database for ai agents. https://github.com/ volcengine/OpenViking. Feng Wang, Yuqing Li, and Han Xi... on PithPith's reviewarXiv
  4. [7] Context as memory: Scene-consistent inter- active long video generation with memory retrieval. InProceedings of the SIGGRAPH Asia 2025 Confer- ence Papers, pages 1–11. Yanwei Yue, Boci Peng, Xuanbo Fan, Jiaxin Guo, Qiankun Li, and Yan Zhang. 2026. Mem-t: Densify- ing rewards f... on PitharXiv
  5. [12] Do not make up information. If the needed information does not appear in your memories, say that the information is not available. [USER] <MEMORY> [[MEMORY BLOCK]] (OPTIONAL USER PROFILE SECTION:) <USER PROFILE> [[USER PROFILE]] Current date: [[QUESTION DATE]] Question: [[QUES... on Pith
  6. [13] Be concise and specific. For dates use the format "15 July 2023"; for durations answer in years/months/days on Pith
  7. [14] Time: reason from the timestamps attached to your memories and from the current date given with the question. Resolve relative expressions ("last year", "how long ago") against the current date, not against when the conversation happened on Pith
  8. [15] Conflicts: if memories disagree because something changed over time, trust the most recent memory, unless the question asks about an earlier point in time on Pith
  9. [16] The needed evidence may be scattered across several conversations; combine related memories before answering. [USER] <MEMORY> [[MEMORY BLOCK]] (OPTIONAL USER PROFILE SECTION:) <USER PROFILE> [[USER PROFILE]] Current date: [[QUESTION DATE]] Question: [[QUESTION]] Read all sente... on Pith
  10. [2023] Mind2web: Towards a generalist agent for the web.Preprint, arXiv:2306.06070. Sedigheh Eslami, Maksim Gaiduk, Markus Krimmel, Louis Milliken, Bo Wang, and Denis Bykov. 2026. Diffusion-pretrained dense and contextual embed- dings.Preprint, arXiv:2602.11151. EverMind-AI. 2026. Ev... on PithPith's reviewarXiv
  11. [2024] Evaluating very long-term conversational memory of llm agents.Preprint, arXiv:2402.17753. Xuanyu Meng, Jiashuo Sun, Jash Rajesh Parekh, and Jiawei Han. 2026. Ensi-rag: Entity-structure- indexed retrieval-augmented generation for long- document question answering.arXiv preprint... on PithPith's reviewarXiv
  12. [2025] Memory os of ai agent. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25972–25981. Seth Karten, Alex L. Zhang, Kevin Thomas, Sebas- tian Müller, Elie Bakouch, Daniel Auras, Mika Seng- haas, Fares Obeid, Konstantin Dunas, Johanne... on PithPith's reviewarXiv
  13. [2026] What deserves memory: Adaptive memory dis- tillation for llm agents.Preprint, arXiv:2508.03341. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang on PithPith's reviewarXiv
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment