REVIEW 3 major objections 6 minor 36 references
Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems
T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The per-turn serving cost of agentic memory systems is driven by internal pipeline state, not by conversation length or message size.
desk verdict Break-even data and honest reporting make this worth reading, but the 'unpredictable' claim outruns the evidence; the abstract needs softening. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central instrument is the two-term power-law cost model $\log(C+1)=a+p\log(L+1)+q\log(t+1)$ with one exponent for message size and one for conversation depth, validated by leave-one-cell-out cross-validation. The model is designed to separate two cases that a single cumulative-content predictor treats as identical: a long conversation with small messages and a short conversation with large messages. It carries the argument by showing that the baselines' exponents match their known mechanisms while the memory systems' exponents do not generalize to held-out cells; the held-out error, not the in-sample $R^2$, is the diagnostic that exposes internal memory behavior. The break-even length, defined as the turn at which a system's cumulative cost falls below full-history cost and stays below, turns the fitted and measured costs into an actionable deployment quantity.
What would settle it
Run the same three memory systems on a denser grid, say twenty or more (N,L) cells; if a pure length-and-size power-law model then holds leave-one-out error below roughly 7% for Mem0, Hindsight, and Mastra Observational Memory, the claim that internal state drives cost would be refuted. The paper's own baseline error range sets the bar: 2.9–6.5% for the two reference strategies.
Extended reading notes
Core claim
The paper claims that a memory system's serving cost is governed by its internal memory state rather than by the size of the conversation being served. Evidence comes from a separable log-log cost model $\log(C+1)=a+p\log(L+1)+q\log(t+1)$ fitted per system: the full-history baseline fits $p\approx q\approx 1$, the rolling window fits $q\approx 0.1$, and both hold out below 6.5% error, while Mem0, Hindsight, and Mastra Observational Memory hold out with 18–22%, 46–48%, and 41–69% error respectively. The paper reads the high held-out error as a positive finding: internal triggers such as fact extraction, retrieval payload, and consolidation thresholds dominate the bill, so external workload descriptors cannot predict cost. It further claims that break-even against full-history resubmission is workload-dependent, and that accuracy spans 21–54% with no system winning on both cost and accuracy.
Load-bearing premise
The paper's conclusion that memory-system cost is driven by internal state rests on the assumption that the few sampled (conversation-length, message-size) cells are representative enough for leave-one-cell-out error to signal a real cost driver, and on the benchmark controlling each system's internal stages—an assumption that fails for Hindsight, whose ingest stage ran under a fixed configuration the benchmark did not control.
Editorial extensions
If this is right
- When the full transcript is resubmitted every turn, cost scales nearly linearly in both message size and depth ($p\approx q\approx 1$), so as conversations grow long a memory system is eventually cheaper—but within 400 turns Hindsight can still be costlier than the transcript.
- Mastra Observational Memory can break even at turn 0, Mem0 within tens of turns for large messages, and Hindsight sometimes never, so the expected conversation length decides whether a memory system is economical.
- The backbone choice shifts cost as much as the memory system choice: Mem0's 100-turn cost drops from $\$0.059$–$\$0.065$ on gpt-oss-20b to $\$0.019$ on Gemma 4 26B A4B, and raising reasoning effort can lower accuracy (Mem0 drops from 0.322 to 0.214).
- No system wins on both axes; the cheapest per correct answer is Mastra Observational Memory on gpt-oss-20b low in one cell and Mem0 on Gemma 4 26B A4B in another, while Hindsight is the most expensive at the 100-turn reference cell.
- Cost predictions for the three memory systems at unseen conversation lengths should be treated as rough estimates until a mechanistic model of their internal state exists.
Reading between the lines
- A mechanistic cost model that tracks per-stage internal variables—number of extracted facts, retrieval payload size, and consolidation triggers—should be able to predict memory-system serving cost with held-out error closer to the baselines; the paper names this as future work.
- If the internal-state result holds, published memory-system cost comparisons are tied to the backbone they were measured on, so a system that looks cheap on one model may not be cheap on another.
- The break-even framework suggests a practical deployment rule: estimate expected conversation length in turns, compare it with a system's break-even length on the chosen backbone, and keep full-transcript resubmission as the default below that threshold.
- A natural test is to rerun the same protocol on a task-oriented dialogue corpus, such as MultiWOZ 2.2, to see whether the internal-state cost driver appears outside persona-grounded chat.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks the serving cost of three agentic memory systems (Mem0, Hindsight, Mastra Observational Memory) against two reference strategies (a 10-turn rolling window and full-transcript resubmission), across two backbones at two reasoning-effort settings, using synthetic conversations up to 400 turns. Each cost measurement is paired with accuracy on a 665-question LoCoMo subset. The authors fit a separable log-log model log(C+1)=a+p·log(L+1)+q·log(t+1), report leave-one-cell-out MAPE, compute break-even lengths from measured per-turn costs, and assemble a joint cost-per-correct-answer matrix. The main findings are that the cost model generalizes well for the two baselines (LOOCV-MAPE 2.9–6.5%) but poorly for the memory systems (18–69%), that break-even timing is highly workload- and backbone-dependent, and that no single system dominates on both cost and accuracy.
Significance. If the benchmark results hold, the paper provides a genuinely useful controlled measurement of memory-system serving cost: it uses external systems and an external QA corpus, pairs cost and accuracy in matched configurations, caches identical dialogues across systems, uses a cluster bootstrap that respects run-level dependence, and reports failures transparently (Hindsight's uncontrolled ingest, the gpt-oss-20b serving-stack rejection at turn 374, and provider-routing variance). The fitted exponent separation for the reference baselines (p≈q≈1 for full history; q≈0.1 for the rolling window) is a clean, interpretable validation of the measurement methodology. The main weakness is that the central negative claim is stated more strongly than the evidence supports: the paper tests one simple parametric model, not the general proposition that cost cannot be predicted from conversation length and message size. The benchmark is still a solid contribution and a useful target for future, more mechanistic cost models, provided the claims are recalibrated.
major comments (3)
- [Section 4.1, Eq. (1), Table 1, and Abstract] The central claim that a memory system's serving cost 'cannot be predicted from conversation length and message size alone' and is 'driven instead by internal memory behavior' is supported only against the specific separable power-law model in Eq. (1). An 18–69% LOOCV-MAPE shows that this one functional form fails; it does not establish that no (L,t)-only model can predict. In fact, Mastra OM's documented thresholds (observer at 30,000 accumulated message tokens, reflector at 40,000 accumulated observation tokens, Table 5) are deterministic functions of conversation length and message size, so a piecewise or cumulative-token specification would plausibly track its cost. The manuscript's own Section 5.4 concedes the model is 'descriptive, not mechanistic,' which undercuts the causal phrasing in the abstract. I recommend reframing the claim to 'a simple separable power law fails, and cost appears to depend on internal pipeline events such as threshold-triggered consolidation,' and testing at least one alternative (L,t)-only specification, such as a model with cumulative-token terms or threshold indicators.
- [Appendix B and Table 6] With five cells for Mem0 and Hindsight and seven for Mastra OM, leave-one-cell-out MAPE can be dominated by a single atypical corner. The per-setting max APE for Mastra OM reaches 1.918 (Table 6), meaning one held-out cell is off by nearly 2×; averaging this with the other folds produces the reported 0.685. The statement that held-out failures 'cluster at the grid corners' needs a per-fold error table, and the grid should be densified around the 30k/40k token thresholds to test whether a cumulative-token model removes the apparent internal-state dependence. As written, the comparison between baseline and memory-system LOOCV error partly confounds model misspecification with grid sparsity.
- [Section 5.4 and Tables 3, 4, 8] Hindsight's ingest stage ran under a single uncontrolled configuration for all settings, yet Hindsight's dollar costs and break-even turns appear in the headline joint tables and in the cost-per-correct-answer ranking. On the Gemma 4 26B A4B columns, for example, Hindsight's ingest tokens were billed at gpt-oss-20b rates, so its $0.051–$0.052 cost is not commensurable with Mem0 and Mastra OM costs on the same column. This contaminates the 'no system wins on both axes' conclusion and the cross-system break-even comparisons. The paper should either re-run Hindsight with controlled ingest or present Hindsight entries in a clearly separated 'uncontrolled' panel, with explicit statements that its dollar figures are not directly comparable to the other systems'.
minor comments (6)
- [General / Abstract] The abstract has missing spaces in 'by18–69%' and 'by21–54%'; please fix the typesetting.
- [Section 3.3] The paper states confidence intervals for p and q come from a cluster bootstrap, but Table 6 reports only point estimates. Please report the intervals, or state where they are available (e.g., in the supplementary material).
- [Section 3.3, Eq. (1)] The '+1' inside the logarithms is not justified in the text. Since costs can be small at early turns, the addition may matter for the fit; please state the convention and its units.
- [Section 4.2 and Figure 2] Figure 2 extends the full-history curve with a fitted dashed line beyond turn 374, while the text emphasizes that break-even is computed from measured costs with no fitted-model extrapolation. Clarify that the dashed extension is illustrative and does not enter any break-even computation.
- [Section 5.2] The explanation that higher reasoning effort 'uses up the max_tokens budget and leaves less room for the answer' is plausible but not directly supported by reported diagnostics. If kept, report the relevant max_tokens consumption or output-token counts for the affected cells.
- [Section 3.4] Accuracy is judged by a single LLM judge with a single pass at temperature 0. Reporting judge agreement or a small sensitivity analysis would strengthen the accuracy comparisons, especially for the smaller differences between systems.
Circularity Check
No significant circularity: the central cost and break-even claims are computed from measured per-turn costs and a held-out validation diagnostic, not from a fitted parameter that defines the outcome.
full rationale
The load-bearing empirical claims are derived from direct measurement. Per-turn cost C is logged from billed input tokens across the ingest, retrieval, and answer stages of each replayed dialogue; Equation (1) is an ordinary least-squares fit of log(C+1) on log(L+1) and log(t+1). The central negative finding is the leave-one-cell-out MAPE of that fit, which is a genuine held-out diagnostic computed from measured cells: the baselines generalize and the memory systems do not. No fitted coefficient is reused to define break-even; break-even is computed from averaged measured per-turn costs with no model extrapolation, as stated in Section 4.2 and Appendix B. The accuracy figures come from a fixed judge on a LoCoMo subset and are independent of (N,L). There are no self-citations carrying the argument: the cited works are external systems, benchmarks, and statistical references, and the paper does not import any uniqueness or modeling ansatz from the authors' own prior work. The only possible concern is interpretive, not circular: attributing the held-out error to 'internal memory state' is a causal generalization that the evidence does not fully establish (the paper itself calls the model 'descriptive, not mechanistic' in Section 5.4), and the sparse five-to-seven-cell grid weakens the strength of the LOOCV evidence. That is a correctness or evidential limitation, not a reduction of the result to its input by construction. Accordingly, no circular step is identified.
Assumptions & free parameters
free parameters (3)
- p (message-size exponent) =
0.14-0.97 across systems and settings (Table 1)
- q (depth exponent) =
0.07-0.97 across systems and settings (Table 1)
- a (intercept) =
not reported per cell in main text
assumptions (5)
- domain assumption Per-token billing reflects serving cost
- domain assumption Synthetic dialogues are representative for ingest-side behavior
- domain assumption The 4-dialogue LoCoMo subset represents the full corpus
- domain assumption Observed provider routing matches contract pricing
- standard math Standard statistical machinery (OLS, bootstrap, JS divergence, Wilson intervals) is valid here
Cite this review
Pith. "Pith review of Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems." pith.science (2026). https://pith.science/paper/OVATMPM5
@misc{pith2026260811879,
author = {Pith},
title = {Pith review of: Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/OVATMPM5}},
note = {Machine review of arXiv:2608.11879}
}
read the original abstract
Long-running conversational agents increasingly rely on a memory system to avoid resending the whole conversation each turn, yet how much that costs to serve has received little systematic benchmarking. We compare three memory systems (Mem0, Hindsight, and Mastra Observational Memory) against two reference strategies -- a fixed-size rolling window and resubmitting the full transcript -- across two backbones and conversations of up to 400 turns, pairing every cost measurement with answer accuracy on 665 LoCoMo questions. First, a memory system's serving cost cannot be predicted from conversation length and message size alone: a regression that tracks the two reference strategies closely misses the memory systems by 18-69%, their cost driven instead by internal memory behavior. Second, a break-even analysis shows that whether -- and when -- a memory system becomes cheaper to serve than the full transcript is highly sensitive to the system and the backbone, from the first tens of turns for the cheapest to never within 400 turns for the most expensive. Third, no system wins on both axes: accuracy spans 21-54%, and the backbone choice drives cost as much as the memory system does.
Figures
Reference graph
Works this paper leans on
-
[1]
Anthropic. 2024. Prompt Caching. https://platform.claude.com/docs/en/build- with-claude/prompt-caching
work page 2024
-
[2]
Anthropic. 2025. Claude API Pricing. https://platform.claude.com/docs/en/about- claude/pricing
work page 2025
-
[3]
Tyler Barnes. 2026. Observational Memory: 95% on LongMemEval. Mastra Technical Report. https://mastra.ai/research/observational-memory
work page 2026
- [4]
-
[5]
A. Colin Cameron, Jonah B. Gelbach, and Douglas L. Miller. 2008. Bootstrap-Based Improvements for Inference with Clustered Errors.The Review of Economics and Statistics90, 3 (2008), 414–427. doi:10.1162/rest.90.3.414
-
[6]
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav
-
[7]
B. Efron. 1979. Bootstrap Methods: Another Look at the Jackknife.The Annals of Statistics7, 1 (1979), 1–26. doi:10.1214/aos/1176344552
arXiv 1979
-
[8]
Sedigheh Eslami, Maksim Gaiduk, Markus Krimmel, Louis Milliken, Bo Wang, and Denis Bykov. 2026. Diffusion-Pretrained Dense and Contextual Embeddings. arXiv:2602.11151 [cs.LG] https://arxiv.org/abs/2602.11151
arXiv 2026
Show all 36 references
-
[9]
In Gim, Guojun Chen, Seung seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt Cache: Modular Attention Reuse for Low-Latency Inference. arXiv:2311.04934 [cs.CL] https://arxiv.org/abs/2311.04934
2024 arXiv
-
[10]
Google DeepMind. 2026. Gemma 4 Model Card. https://ai.google.dev/gemma/ docs/core/model_card_4
2026
-
[11]
Bowen Jiang, Yuan Yuan, Maohao Shen, Zhuoqun Hao, Zhangchen Xu, Zichen Chen, Ziyi Liu, Anvesh Rao Vijjini, Jiashu He, Hanchao Yu, Radha Poovendran, Gregory Wornell, Lyle Ungar, Dan Roth, Sihao Chen, and Camillo Jose Taylor
-
[12]
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. arXiv preprint arXiv:2310.06839. arXiv:2310.06839 [cs.CL] https://arxiv.org/abs...
2023 arXiv
-
[13]
arXiv:2512.06688 [cs.CL] https://arxiv.org/ abs/2512.06688
PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory. arXiv:2512.06688 [cs.CL] https://arxiv.org/ abs/2512.06688
-
[14]
Chris Latimer, Nicoló Boschi, Andrew Neeser, Chris Bartholomew, Gaurav Srivas- tava, Xuan Wang, and Naren Ramakrishnan. 2025. Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects. arXiv:2512.12818 [cs.CL] https://arxiv.org/abs/2512.12818
2025
-
[15]
Maria Khalusova. 2024. RAG vs. Long-Context Models: Do We Still Need RAG? https://unstructured.io/blog/rag-vs-long-context-models-do-we-still- need-rag
2024
-
[16]
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics 12 (2024), 157–173
2024
-
[17]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing ...
2020
-
[18]
Menéndez, J.A
M.L. Menéndez, J.A. Pardo, L. Pardo, and M.C. Pardo. 1997. The Jensen–Shannon divergence.Journal of the Franklin Institute334, 2 (1997), 307–318. doi:10.1016/ S0016-0032(96)00063-4
1997
-
[19]
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. 2024. Evaluating Very Long-Term Conversational Memory of LLM Agents. arXiv:2402.17753 [cs.CL] https://arxiv.org/abs/2402. 17753
2024 arXiv
-
[20]
OpenAI. 2024. Prompt Caching in the API. https://openai.com/index/api-prompt- caching/
2024
-
[21]
OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Apple- baum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Che...
2025 arXiv
-
[22]
OpenAI. 2025. tiktoken: A Fast BPE Tokeniser for Use with OpenAI’s Models. https://github.com/openai/tiktoken
2025
-
[23]
OpenAI. 2025. OpenAI API Pricing. https://platform.openai.com/docs/pricing
2025
-
[24]
Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. 2025. Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv:2501.13956 [cs.CL] https://arxiv.org/abs/2501.13956
2025 arXiv
-
[25]
Patil, Ion Stoica, and Joseph E
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560 [cs.AI] https://arxiv.org/abs/2310.08560
2024 arXiv
-
[26]
M. Stone. 1974. Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society: Series B (Methodological)36, 2 (1974), 111–
1974
-
[27]
Jeffrey Rengifo and Eduard Martin. 2025. Longer Context≠ Better: Why RAG Still Matters. https://www.elastic.co/search-labs/blog/rag-vs-long-context-model- llm
2025
-
[28]
Edwin B. Wilson. 1927. Probable Inference, the Law of Succession, and Statistical Inference.J. Amer. Statist. Assoc.22, 158 (1927), 209–212. doi:10.1080/01621459. 1927.10502953
1927
-
[29]
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu
-
[30]
Miao Su, Yucan Guo, Zhongni Hou, Long Bai, Zixuan Li, Yufei Zhang, Guo- jun Yin, Wei Lin, Xiaolong Jin, Jiafeng Guo, and Xueqi Cheng. 2026. Beyond Dialogue Time: Temporal Semantic Memory for Personalized LLM Agents. arXiv:2601.07468 [cs.AI] https://arxiv.org/abs/2601.07468
2026
-
[31]
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2: A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines. InProceedings of the 2nd Workshop on Natural Language Processing for Conv...
2020
-
[33]
arXiv:2410.10813 [cs.CL] https://arxiv.org/abs/2410.10813
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. arXiv:2410.10813 [cs.CL] https://arxiv.org/abs/2410.10813
-
[34]
Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang
-
[35]
arXiv:2502.12110 [cs.CL] https://arxiv.org/abs/2502.12110
A-MEM: Agentic Memory for LLM Agents. arXiv:2502.12110 [cs.CL] https://arxiv.org/abs/2502.12110
-
[147]
doi:10.1111/j.2517-6161.1974.tb00994.x
1974
-
[2025]
arXiv:2504.19413 [cs.CL] https://arxiv.org/abs/2504.19413
Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413 [cs.CL] https://arxiv.org/abs/2504.19413
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.