Pith. sign in

REVIEW 2 major objections 5 minor 55 references

Memory leaderboards for scientific agents are uninterpretable unless retrieval budget and modality are reported; on PAIM, Graphiti's lead disappears once its 2.55M-character context per query is capped.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:44 UTC pith:65ATZXBW

load-bearing objection A serious, careful benchmark paper whose central point about reporting budget is right, but the 'Graphiti entirely due to volume' claim overreaches because the ablation changes modality and budget simultaneously. the 2 major comments →

arxiv 2607.16848 v1 pith:65ATZXBW submitted 2026-07-18 cs.LG cs.AI

Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration

Maksim Sheverev , David Finkelstein , Sergey Nikolenko This is my paper
classification cs.LG cs.AI
keywords scientific memorycontext restorationretrieval budgetmemory leaderboardsfull-text question answeringsparse-dense hybrid retrievalLLM-as-a-judgeTheoria
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that ranking LLM-agent memory systems on scientific papers is meaningless unless the evaluation controls how much retrieved context each system feeds the synthesizer and what retrieval modalities it uses. The authors introduce two full-text scientific-memory benchmarks, PAIM and PTr, and evaluate eight memory systems plus a no-retrieval baseline. Their central result is that native leaderboards conflate architecture with context volume: Graphiti wins on PAIM at 2.55M retrieved characters per query but drops to the bottom once the budget is capped. On PTr, adding BM25 lexical retrieval to dense retrieval is the largest intervention measured, and the hybrid variants of Simple RAG, Mem0, and Theoria tie within 0.03 points at a 50K-character budget. The paper proposes evaluating scientific memory as budgeted, modality-aware context restoration rather than as an unconstrained architecture leaderboard.

Core claim

When the retrieval budget—the number of characters of evidence returned per question—is held fixed, the apparent architectural hierarchy of memory systems largely dissolves. Graphiti's native lead on PAIM (8.04 overall) comes from restoring full paper episodes, roughly 2.55M characters per query; with the budget constrained to 30K and the episode channel disabled, its knowledge-graph-only output scores 5.27 and lands at the bottom, while Simple RAG, Theoria, and Mem0 form a tight cluster near 7.2. On PTr, the dominant effect is retrieval modality: fusing BM25 with dense retrieval improves every system tested, and the hybrid variants of Simple RAG, Mem0, and Theoria converge within 0.03 point

What carries the argument

The central instrument is the context-restoration protocol: a fixed synthesizer model combined with a retrieval stage constrained by a character budget B and a retrieval modality M (dense, lexical, or hybrid). The protocol distinguishes uncapped 'native' runs from 'budget-targeted' runs that re-query the same already-ingested store at B = 10K, 30K, and 50K characters, and it adds BM25-over-RRF as a sparse-dense hybrid. This separation is what lets the paper attribute Graphiti's lead to raw context volume rather than to its temporal-graph architecture, and it is also what exposes the modality effect on PTr.

Load-bearing premise

The headline claim that Graphiti's native lead is 'entirely due to context volume' assumes the knowledge-graph-only ablation changes only the retrieval budget, but it also removes the raw-episode-text modality, so the drop from 8.04 to 5.27 conflates a smaller budget with a different representation.

What would settle it

Run Graphiti on PAIM under a strict 30K-character budget while keeping episode retrieval enabled, for example by truncating each matched episode body to 30K characters per query. If the truncated-episode Graphiti scores near its native 8.04 rather than the KG-only 5.27, the volume-only explanation fails. Conversely, run Simple RAG with roughly 2.55M characters per query; if it does not approach Graphiti's native score, context volume alone is not sufficient to explain the lead.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any memory-system leaderboard that omits mean retrieved characters per query and retrieval modality is uninterpretable; reported wins may invert under budget control.
  • At matched retrieval budgets on PAIM, a plain chunk RAG baseline ties the top structured memory systems, so the architectural premium in that corpus is small or absent.
  • Sparse-dense hybrid retrieval is a larger lever than architecture on lexically dense corpora like PTr; hybrid variants of Simple RAG, Mem0, and Theoria effectively tie.
  • LLM-as-a-judge rankings are reliable across frontier models and humans, but score gaps below about one point on a ten-point scale should not be read as real ranking differences.
  • Benchmark rubrics should be audited against full paper text before rankings are reported, since the audit caught defects in 37 of the first-version PAIM questions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the budget-control result generalizes, many published memory-benchmark wins that report only architecture may be artifacts of unconstrained context volume; re-running them under a fixed character budget would settle this.
  • The convergence of Simple RAG+BM25 with Theoria+BM25 on PTr suggests that for one-shot QA over full papers, expensive structured extraction may be unnecessary; the remaining value of systems like Theoria and Mem0 may lie in their agentic update and cold-start endpoints, which this paper explicitly leaves untested.
  • The non-monotonic budget-quality curves observed per question (e.g., PQ44 and PQ52) imply that a per-question optimal retrieval budget may exist; a routing signal that predicts it could realize the 0.4–0.8 point oracle headroom the paper identifies, but difficulty-tier labels alone do not capture it.
  • The judge-resolution result implies that model-selection decisions on this benchmark should not be made on differences under roughly one point; a direct test would be to run additional blinded human side-by-side votes on pairs with Gemini gaps between 0.5 and 1.0.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces two full-text scientific-memory benchmarks (PAIM and PTr), evaluates eight memory/retrieval systems plus a no-retrieval baseline under native and budget-constrained retrieval, and proposes Theoria, a three-layer evidence/community/theory memory system. The central claim is that memory leaderboards are not interpretable without reporting the full evaluation protocol — ingestion granularity, raw-text preservation, retrieval budget, retrieval modality, rubric audit, and judge choice. The evidence includes Graphiti's native lead on PAIM (8.04 with ~2.6M chars/query) that disappears in a budgeted KG-only variant, a PTr result where sparse–dense hybrids converge at the top within 0.03 points, and multi-judge/human calibration supporting an LLM-judge resolution of about one point. The paper releases datasets, harness, raw outputs, and scripts.

Significance. If the claims hold, this is a substantial methodological contribution. The benchmarks are full-text and audited, the evaluation design is unusually careful: a shared synthesizer, paired bootstrap confidence intervals, multi-judge and blinded human calibration, granularity caveats, and an explicit limitations section. The artifact release is a real strength, as is the recommendation that budget and modality be reported alongside scores. The finding that simple chunk RAG is competitive with complex memory systems at matched retrieval budgets, and that BM25 is a larger lever on the PTr corpus, would be important for the memory/agent community. However, the strongest causal attribution — that Graphiti's native PAIM win is 'entirely due to context volume' — rests on an ablation that changes two variables at once; until that is fixed, the paper's headline claim is stronger than the evidence it presents.

major comments (2)
  1. [§6.1–§6.2, Table 4] The attribution that Graphiti's native PAIM lead is 'entirely due to context volume' is not established by the reported ablation. Native Graphiti returns graph plus raw episode bodies (2.55M chars/query); the budgeted runs set include_episodes=false, so they change both the character budget and the output modality (KG-only, no raw text). The drop from 8.04 to 5.27 therefore conflates a budget cut with a representation change. Since the paper itself lists raw-text preservation as a protocol dimension that affects outcomes (abstract, §1), this comparison cannot isolate volume. A control that keeps episodes but truncates their body to B≈30K (or a native no-episode Graphiti) would separate the two; without it, the abstract's 'after controlling for retrieval budget the lead disappears' and §6.2's 'entirely due to raw episode bodies' overstate the evidence.
  2. [Table 4, B≈30K column] The 'fair-comparison anchor' is not a fixed budget in achieved characters. In the B≈30K column, Simple RAG uses 31K chars, Theoria 41K, Mem0 32K, but Graphiti (KG) uses 59K and Cognee 37K (Hindsight is capped at 19K). The narrative treats the 30K anchor as matched across systems, and §6.2 reports '5.27 at 30K' without highlighting that Graphiti's achieved context is roughly 2× that of Simple RAG. The paper's own Recommendation 4 is to report achieved characters rather than nominal budgets; please re-label the anchors by achieved ranges or recalibrate the Graphiti settings, and qualify the 'fair-comparison' language accordingly.
minor comments (5)
  1. [§4, Table 3] The text says 'the audit found that 35 questions in the first version had at least one defect,' but Table 3 reports 'Total questions rewritten 37' and the class counts sum to 42 with multi-class entries. Please reconcile these numbers.
  2. [§6.5, Fig. 6b] The text says 112 blinded human votes, while the inset reports n=111 for the tie-tolerance sweep. Clarify whether one vote is excluded from the sweep and why.
  3. [Table 7, first contrast] The PAIM @30K contrast 'Simple RAG − Theoria' reports n=65, while the other PAIM contrasts report n=66. Explain the missing paired cell.
  4. [Table 4, row labels] In Table 4 the budgeted Graphiti columns use include_episodes=false, but the row is labeled plain 'Graphiti' while the text calls the same configuration 'Graphiti (KG)'. Use a single label throughout to avoid implying the budgeted variant is the same native configuration.
  5. [§6.1, 'Budget-targeted configurations'] The sentence 'changing only retrieval settings' is contradicted by the next paragraph's Graphiti ablation, which also changes the returned representation. Qualify the sentence to exclude the explicit Graphiti raw-text ablation, or move that ablation to a clearly labeled separate track.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are comparative empirical measurements, not derivations that reduce to their inputs.

full rationale

The paper's central claims are empirical benchmark comparisons rather than analytical derivations, and I find no step where a result is equivalent to its inputs by construction. The budget-targeted experiments re-query already-ingested stores at fixed character targets; the scores are measured outputs, not quantities fitted to the target claims. The bootstrap and oracle-router analyses transparently re-aggregate existing judge scores and are explicitly labeled as such ('pure re-aggregation of existing scores'), so they are not disguised predictions. The authors evaluate their own Theoria system, but this is disclosed, Theoria is not the headline winner, and the main leaderboard comparisons involve independently implemented external systems. The closest concern is the Graphiti ablation: the budgeted runs set include_episodes=false, which varies both retrieval budget and retrieval modality, so the claim that Graphiti's native lead is 'entirely due to context volume' is methodologically confounded. However, this is a validity/causal-attribution issue rather than circularity: the paper does not define Graphiti's budgeted score in terms of its native volume, and it discloses the modality change and lists approximate budget control as a limitation. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameters renamed as predictions. The judge-calibration claims are anchored to external human side-by-side votes and second frontier judges. Overall, the derivation chain is self-contained and the empirical claims stand independently of their own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claims are empirical, so the ledger lists the hand-chosen evaluation knobs and background assumptions that the conclusions depend on. No fitted-value circularity; the most consequential assumption is that the fixed synthesizer/judge and audited rubrics give a valid comparison across very different memory representations.

free parameters (4)
  • BM25 RRF fusion constant k = 60
    Used for all sparse-dense hybrids (§6.1); a hand-chosen hyperparameter that affects the magnitude of the BM25 lift.
  • Theoria theory-match cosine threshold = 0.25
    Determines when a new finding matches an existing theory (§5); hand-set and affects Theoria's outputs, though not the central budget claim.
  • Theoria deduplication cosine threshold = 0.80
    Sets when two evidence items are considered duplicates (§5); hand-set.
  • Retrieval budget grid B = 10K/30K/50K chars
    Defines the central budgeted comparison (§6.1); an evaluation design choice rather than a fitted model parameter.
axioms (4)
  • domain assumption The no-retrieval composite score from a frontier LLM judge is a valid measure of answer quality.
    All leaderboard scores are Gemini 3.1 Pro judgments; the paper calibrates against other LLMs and 112 human SxS votes (§6.5), but the large evaluation grids rely on this proxy.
  • domain assumption Holding the synthesis model fixed (gpt-4.1-mini) isolates memory/retrieval differences from answer-generation quality.
    Internal-synthesis systems (Cognee, Prism) use their own models, so synthesis is not actually fixed across all systems (§6.1); interactions between retrieved context and synthesizer are not fully separated (§8.2).
  • domain assumption Audited rubrics are correct representations of what the source papers claim.
    The rubric audit (§4) rewrote 37 questions, but cannot guarantee zero residual errors; all scoring compares answers to these rubrics.
  • domain assumption The out-of-the-box configurations of each memory system are representative of its architecture.
    Several integrations deviate from intended granularity (Hindsight whole-paper items, Mem0 6K chunks, Graphiti episodes) and the paper treats these as evaluation facts (§6.2).

pith-pipeline@v1.3.0-alltime-deepseek · 29645 in / 12654 out tokens · 125746 ms · 2026-08-01T19:44:56.074572+00:00 · methodology

0 comments
read the original abstract

Long-term memory is becoming a core component of LLM agents, but most memory benchmarks evaluate conversations or compact summaries, while research agents need to restore evidence from full scientific papers. We introduce two full-text scientific-memory benchmarks, Public AI Memory (PAIM; 81 papers, 66 questions) and Public Transformers (PTr; 252 papers, 98 questions). We evaluate eight memory/retrieval systems, including our own proposed Theoria, plus a no-retrieval baseline. Our results show that memory leaderboards are not interpretable without the full protocol: ingestion granularity, raw-text preservation, retrieval budget, retrieval modality, rubric audit, and judge choice all affect the outcome. For example, on PAIM Graphiti wins convincingly but uses 2.6M characters of retrieved context per query, and after controlling for retrieval budget the lead disappears. On PTr, for the systems where BM25 retrieval can be added cleanly, the sparse-dense hybrid is the single most significant intervention: hybrid variants of Simple RAG, Mem0, and Theoria tie for the lead within 0.03 points. Multi-judge and human side-by-side calibration show that LLM-as-a-judge rankings are consistent across frontier judges and agree with human evaluation, with an effective resolution of roughly one point on a ten-point scale. We argue that scientific memory should be evaluated as budgeted, modality-aware context restoration rather than as an unconstrained architecture leaderboard, and we release the datasets, harness, raw outputs, judgments, and scripts to reproduce our results and serve as tools for such evaluation. Our code is available at http://gitlab.com/quantellence/research/scientific-recall-bench , and the datasets are available at http://huggingface.co/datasets/quantellence/srb-data .

Figures

Figures reproduced from arXiv: 2607.16848 by David Finkelstein, Maksim Sheverev, Sergey Nikolenko.

Figure 1
Figure 1. Figure 1: The context-restoration protocol. A memory system can differ at every stage: how full papers are split into ingestion units, how those units are represented in the store, how much and what kind of context is returned at retrieval time (the budget 𝐵 and modality 𝑀), and whether the final answer is synthesized internally by the system or externally by a shared model. context restoration has empirical consequ… view at source ↗
Figure 2
Figure 2. Figure 2: The architecture of Theoria. The three-layer evidence + community + theory stack shares the same store with three agent endpoints: GET /theories for cold-start bootstrap, POST /retrieve for community-routed evidence retrieval, and POST /observe for inserting new findings (which can update or contradict existing theories). 6.4× KV compression at 4K, Ring-flash-linear-2.0 advantages pronounced beyond 8K, and… view at source ↗
Figure 3
Figure 3. Figure 3: Native Paim scores by difficulty tier (Gemini judge, audited rubrics). L1 is largely saturated; L2 fans the systems out; on L3 the long-context Direct Read anchor (∼2M chars/query) leads, followed by Hindsight and Simple RAG. These are the “Native” columns of [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Budget sweeps: composite score vs. mean retrieved characters per query (log 𝑥-axis). (a) Paim: Simple RAG, Theoria, and Mem0 form a tight upper cluster that scales monotonically with budget; Hindsight is a single point repeated because its server ignores the budget knob; Cognee and Graphiti’s KG output are flat and low. The dotted line marks Graphiti’s native score with episodes (2.6M chars). (b) PTr (252 … view at source ↗
Figure 5
Figure 5. Figure 5: (a) Paim mean overall by system across three judges on the shared 6-question subset, so absolute levels are directly comparable; the bar heights track each other closely and rank Spearman is 𝜌 ∈ [0.90, 0.97]. (b) PTr corpus+modality progression at a 50K retrieval budget: the 80-paper / 48-question size-matched subset (dense) → PTr dense (252 papers, 98 questions) → PTr hybrid (the same, + BM25). The full-c… view at source ↗
Figure 6
Figure 6. Figure 6: Judge calibration. (a) Per-cell scores, DeepSeek V4 Pro (𝑦) vs. Gemini 3.1 Pro (𝑥) on the PTr 50-question subset (7 systems × 50 questions × 3 budgets); Pearson 𝑟 = 0.93, 𝑛 = 946, best fit 𝑦 = 0.97𝑥 − 0.20. (b) Human SxS vs. Gemini, 112 blinded votes. Stacked bars count votes per |Gemini gap| bucket, split into “Gemini agrees with the human winner”, “Gemini disagrees”, and “human tie”; annotations are agre… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 30 linked inside Pith

  1. [1]

    OpenScholar: Synthesizing scientific literature with retrieval-augmented language models.arXiv, 2024

    AkariAsai,JacquelineHe,RulinShao,WeijiaShi,AmanpreetSingh,JosephChee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Dan Weld, Graham Neubig, Doug Downey, Wen- tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi. OpenScholar: Synthesizi...

    Pith/arXiv arXiv 2024

  2. [2]

    PeerQA:Ascientificquestion answeringdatasetfrompeerreviews

    TimBaumgärtner,TedBriscoe,andIrynaGurevych. PeerQA:Ascientificquestion answeringdatasetfrompeerreviews. InProceedingsofthe2025Conferenceofthe NorthAmericanChapteroftheAssociationforComputationalLinguistics(NAACL),

  3. [3]

    HaluMem: Evaluating hallucinations in memory systems of agents.arXiv preprint arXiv:2511.03506, 2025

    Ding Chen, Simin Niu, Kehang Li, Peng Liu, Xiangping Zheng, Bo Tang, Xinchi Li, Feiyu Xiong, and Zhiyu Li. HaluMem: Evaluating hallucinations in memory systems of agents.arXiv preprint arXiv:2511.03506, 2025. URL https://arxiv.org/abs/2511.03506

    arXiv 2025

  4. [4]

    Mem0: Building production-ready AI agents with scalable long-term memory

    Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. arXiv preprint arXiv:2504.19413, 2025. URLhttps://arxiv.org/abs/2504. 19413

    Pith/arXiv arXiv 2025

  5. [5]

    Cormack, Charles L

    Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 758–759, 2009. doi: 10.1145/1571941.1572114

    arXiv 2009

  6. [6]

    DeepSeek-V3.2: Pushingthefrontierofopenlargelanguagemodels

    DeepSeek-AI. DeepSeek-V3.2: Pushingthefrontierofopenlargelanguagemodels. arXiv preprint arXiv:2512.02556, 2025. URLhttps://arxiv.org/abs/2512. 02556. Introduces DeepSeek Sparse Attention (DSA)

    Pith/arXiv arXiv 2025

  7. [7]

    From local to global: A graph RAG approach to query-focused summarization

    DarrenEdge,HaTrinh,NewmanCheng,JoshuaBradley,AlexChao,ApurvaMody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summarization. 30 arXiv preprint arXiv:2404.16130, 2024. URLhttps://arxiv.org/abs/2404. 16130

    Pith/arXiv arXiv 2024

  8. [8]

    LightMem: Lightweight and efficient memory-augmented generation

    JizhanFang,XinleDeng,HaomingXu,ZiyanJiang,YuqiTang,ZiwenXu,Shumin Deng, Yunzhi Yao, Mengru Wang, Shuofei Qiao, Huajun Chen, and Ningyu Zhang. LightMem: Lightweight and efficient memory-augmented generation. arXiv preprint arXiv:2510.18866, 2025. URLhttps://arxiv.org/abs/2510. 18866

    Pith/arXiv arXiv 2025

  9. [9]

    LightRAG:Simple and fast retrieval-augmented generation.arXiv preprint arXiv:2410.05779, 2024

    ZiruiGuo,LianghaoXia,YanhuaYu,TuAo,andChaoHuang. LightRAG:Simple and fast retrieval-augmented generation.arXiv preprint arXiv:2410.05779, 2024. URLhttps://arxiv.org/abs/2410.05779

    Pith/arXiv arXiv 2024

  10. [10]

    HippoRAG: Neurobiologically inspired long-term memory for large language models

    Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. HippoRAG: Neurobiologically inspired long-term memory for large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. URLhttps://arxiv.org/abs/2405.14831. arXiv:2405.14831

    Pith/arXiv arXiv 2024

  11. [11]

    From RAGtomemory: Non-parametriccontinuallearningforlargelanguagemodels

    Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From RAGtomemory: Non-parametriccontinuallearningforlargelanguagemodels. In Proceedings of the 42nd International Conference on Machine Learning (ICML),

  12. [13]

    arXiv:2502.14802

    URLhttps://arxiv.org/abs/2502.14802. arXiv:2502.14802

    Pith/arXiv arXiv

  13. [14]

    Every tokencounts: Generalizing16Multra-longcontextinlargelanguagemodels.arXiv preprintarXiv:2511.23319,2025.URL https://arxiv.org/abs/2511.23319

    XiangHu,ZhanchaoZhou,RuiqiLiang,ZehuanLi,WeiWu,andJianguoLi. Every tokencounts: Generalizing16Multra-longcontextinlargelanguagemodels.arXiv preprintarXiv:2511.23319,2025.URL https://arxiv.org/abs/2511.23319. HSA-UltraLong

    arXiv 2025

  14. [15]

    EvaluatingmemoryinLLMagentsvia incremental multi-turn interactions.arXiv preprint arXiv:2507.05257, 2025

    YuanzheHu,YuWang,andJulianMcAuley. EvaluatingmemoryinLLMagentsvia incremental multi-turn interactions.arXiv preprint arXiv:2507.05257, 2025. URL https://arxiv.org/abs/2507.05257. Introduces the MemoryAgentBench benchmark

    Pith/arXiv arXiv 2025

  15. [16]

    EverMemOS: A self-organizing memory operating system for structured long-horizon reasoning

    Chuanrui Hu, Xingze Gao, Zuyi Zhou, Dannong Xu, Yi Bai, Xintong Li, Hui Zhang, Tong Li, Chong Zhang, Lidong Bing, and Yafeng Deng. EverMemOS: A self-organizing memory operating system for structured long-horizon reasoning. arXiv preprint arXiv:2601.02163, 2026. URLhttps://arxiv.org/abs/2601. 02163

    arXiv 2026

  16. [18]

    Anatomyofagentic memory: Taxonomy and empirical analysis of evaluation and system limitations

    Dongming Jiang, Yi Li, Songtao Wei, Jinxin Yang, Ayushi Kishore, Alysa Zhao, DingyiKang,XuHu,FengChen,QiannanLi,andBingzheLi. Anatomyofagentic memory: Taxonomy and empirical analysis of evaluation and system limitations. arXiv preprint arXiv:2602.19320, 2026. URLhttps://arxiv.org/abs/2602. 19320

    Pith/arXiv arXiv 2026

  17. [20]

    URLhttps://arxiv.org/abs/2512.13564. 31

    Pith/arXiv arXiv

  18. [22]

    URLhttps://arxiv.org/abs/2601.03236

    Pith/arXiv arXiv

  19. [23]

    Letta: The stateful agents framework with memory, reasoning, and context management

    Letta AI. Letta: The stateful agents framework with memory, reasoning, and context management. GitHub repository, https://github.com/letta-ai/ letta, 2024. Formerly the MemGPT open-source project

    2024

  20. [24]

    MemPalace: An open-source AI memory sys- tem

    Milla Jovovich and Ben Sigman. MemPalace: An open-source AI memory sys- tem. GitHubrepository, https://github.com/milla-jovovich/mempalace, 2026

    2026

  21. [25]

    Kimi Linear: An expressive, efficient attention architecture

    Kimi Team. Kimi Linear: An expressive, efficient attention architecture. arXiv preprint arXiv:2510.26692, 2025. URLhttps://arxiv.org/abs/2510. 26692

    Pith/arXiv arXiv 2025

  22. [26]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, FabioPetroni,andPercyLiang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, FabioPetroni,andPercyLiang. Lostinthemiddle: Howlanguagemodelsuselong contexts.Transactions of the Association for Computational Linguistics, 12:157– 173, 2024. URLhttps://arxiv.org/abs/2307.03172. arXiv:2307.03172. 32

    Pith/arXiv arXiv 2024

  23. [27]

    URLhttps://arxiv.org/abs/2512.12818

    arXiv

  24. [28]

    QASA: Advanced question answering on scientific articles

    Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong- In Lee, and Moontae Lee. QASA: Advanced question answering on scientific articles. InProceedingsofthe40thInternationalConferenceonMachineLearning (ICML),2023. URL https://proceedings.mlr.press/v202/lee23n.html

    2023

  25. [30]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    PatrickLewis,EthanPerez,AleksandraPiktus,FabioPetroni,VladimirKarpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Process- ing Systems (NeurIPS), 2020. URLhttps://arxiv.org/abs/2005.11401. ar...

    Pith/arXiv arXiv 2020

  26. [31]

    Every attention matters: An efficient hybrid architecture for long- contextreasoning.arXivpreprintarXiv:2510.19338,2025

    Ling Team. Every attention matters: An efficient hybrid architecture for long- contextreasoning.arXivpreprintarXiv:2510.19338,2025. URL https://arxiv. org/abs/2510.19338. Ring-flash-linear-2.0

    arXiv 2025

  27. [32]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023. URLhttps://arxiv.org/ abs/2304.03442. arXiv:2304.03442

    Pith/arXiv arXiv 2023

  28. [33]

    Evaluating very long-term conversational memory of LLM agents

    Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. URLhttps://arxiv.org/abs/2402. 17753. arXiv:2402.17753

    Pith/arXiv arXiv 2024

  29. [34]

    SALS: Sparse attention in latent space for KV cache compression.arXiv preprint arXiv:2510.24273, 2025

    Junlin Mu, Hantao Huang, Jihang Zhang, Minghui Yu, Tao Wang, and Yidong Li. SALS: Sparse attention in latent space for KV cache compression.arXiv preprint arXiv:2510.24273, 2025. URLhttps://arxiv.org/abs/2510.24273

    arXiv 2025

  30. [35]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. RAPTOR: Recursive abstractive processing for tree- organized retrieval. InInternational Conference on Learning Representations (ICLR),2024. URL https://arxiv.org/abs/2401.18059. arXiv:2401.18059

    Pith/arXiv arXiv 2024

  31. [36]

    URLhttps://arxiv.org/abs/2508.03341

    Pith/arXiv arXiv

  32. [37]

    MinerU: A one-stop, high-quality open-source pdf extraction tool

    OpenDataLab. MinerU: A one-stop, high-quality open-source pdf extraction tool. GitHub repository,https://github.com/opendatalab/MinerU, 2024

    2024

  33. [38]

    Patil, Ion Stoica, and Joseph E

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. URLhttps://arxiv.org/abs/2310. 08560

    Pith/arXiv arXiv 2023

  34. [39]

    Ross Mitchell

    Mohammad Tavakoli, Alireza Salemi, Carrie Ye, Mohamed Abdalla, Hamed Zamani, and J. Ross Mitchell. Beyond a million tokens: Benchmarking and enhancing long-term memory in LLMs. InInternational Conference on Learning Representations (ICLR), 2026. URLhttps://arxiv.org/abs/2510.27246. arXiv:2510.27246

    arXiv 2026

  35. [40]

    Zep: A temporal knowledge graph architecture for agent memory.arXiv preprint arXiv:2501.13956, 2025

    PrestonRasmussen,PavloPaliychuk,TravisBeauvais,JackRyan,andDanielChalef. Zep: A temporal knowledge graph architecture for agent memory.arXiv preprint arXiv:2501.13956, 2025. URLhttps://arxiv.org/abs/2501.13956

    Pith/arXiv arXiv 2025

  36. [41]

    The probabilistic relevance framework: BM25andbeyond.FoundationsandTrendsinInformationRetrieval,3(4):333–389,

    Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25andbeyond.FoundationsandTrendsinInformationRetrieval,3(4):333–389,

  37. [42]

    MemMachine: A ground-truth-preserving memory system for personalized AI agents.arXiv preprint arXiv:2604.04853, 2026

    Shu Wang, Edwin Yu, Oscar Love, Tom Zhang, Tom Wong, Steve Scargall, and Charles Fan. MemMachine: A ground-truth-preserving memory system for personalized AI agents.arXiv preprint arXiv:2604.04853, 2026. URLhttps: //arxiv.org/abs/2604.04853

    Pith/arXiv arXiv 2026

  38. [43]

    LongMemEval: Benchmarking chat assistants on long-term interactive memory

    Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. LongMemEval: Benchmarking chat assistants on long-term interactive memory. InInternational Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2410.10813. arXiv:2410.10813

    Pith/arXiv arXiv 2025

  39. [44]

    Mem2ActBench: A benchmark for evaluating long-term memory utilization in task-oriented autonomous agents

    Yiting Shen, Kun Li, Wei Zhou, and Songlin Hu. Mem2ActBench: A benchmark for evaluating long-term memory utilization in task-oriented autonomous agents. arXiv preprint arXiv:2601.19935, 2026. URLhttps://arxiv.org/abs/2601. 19935

    arXiv 2026

  40. [45]

    Reflexion: Language agents with verbal reinforce- ment learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforce- ment learning. InAdvances in Neural Information Processing Systems (NeurIPS),

  41. [46]

    A-MEM: Agentic memory for LLM agents

    Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: Agentic memory for LLM agents. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. URLhttps://arxiv.org/abs/2502. 12110. arXiv:2502.12110

    Pith/arXiv arXiv 2025

  42. [47]

    Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L

    Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitivearchitecturesforlanguageagents.TransactionsonMachineLearningRe- search, 2024. URLhttps://arxiv.org/abs/2309.02427. arXiv:2309.02427

    Pith/arXiv arXiv 2024

  43. [49]

    Cognee: Knowledge engine for AI agent memory in 6 lines of code

    Topoteretes. Cognee: Knowledge engine for AI agent memory in 6 lines of code. GitHub repository,https://github.com/topoteretes/cognee, 2024

    2024

  44. [50]

    FromLouvaintoLeiden: Guaranteeing well-connected communities.Scientific Reports, 9(1):5233, 2019

    V.A.Traag,L.Waltman,andN.J.vanEck. FromLouvaintoLeiden: Guaranteeing well-connected communities.Scientific Reports, 9(1):5233, 2019. doi: 10.1038/ s41598-019-41695-z

    2019

  45. [53]

    URL https://arxiv.org/abs/2506.05690.IntroducestheGraphRAG-Bench benchmark

    Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, XiaoHuang,andJinsongSu.WhentousegraphsinRAG:Acomprehensiveanalysis for graph retrieval-augmented generation.arXiv preprint arXiv:2506.05690, 2025. URL https://arxiv.org/abs/2506.05690.IntroducestheGraphRAG-Bench benchmark

    arXiv 2025

  46. [54]

    Learning to forget: Sleep-inspired memory consolidation for resolving proactive interference in large language models, 2026

    Ying Xie. Learning to forget: Sleep-inspired memory consolidation for resolving proactive interference in large language models, 2026. URLhttps://arxiv. org/abs/2603.14517

    arXiv 2026

  47. [56]

    FSA: An alternative efficient implementation of native sparse attention kernel.arXiv preprint arXiv:2508.18224, 2025

    Ran Yan, Youhe Jiang, Zhuoming Chen, Haohui Mai, Beidi Chen, and Binhang Yuan. FSA: An alternative efficient implementation of native sparse attention kernel.arXiv preprint arXiv:2508.18224, 2025. URL https://arxiv.org/ abs/2508.18224

    Pith/arXiv arXiv 2025

  48. [58]

    URLhttps://arxiv.org/abs/2508.19828

    Pith/arXiv arXiv

  49. [59]

    Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar

    Li S. Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar. ResearchQA: Evaluating scholarly question answering at scale across 75 fields with survey- mined questions and rubrics.Transactions of the Association for Computational Linguistics, 2026. URL https://arxiv.org/abs/2509.00496. To appear; arXiv:2509.00496

    arXiv 2026

  50. [60]

    A survey on the memory mechanism of large language model based agents.ACM Transactions on Information Systems, 2025

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents.ACM Transactions on Information Systems, 2025. URLhttps://arxiv.org/abs/2404.13501. arXiv:2404.13501 (2024)

    Pith/arXiv arXiv 2025

  51. [62]

    What Judge score does MAGMA report on LoCoMo, and by what margin does it beat prior systems according to the paper?

    URLhttps://arxiv.org/abs/2510.10114. A Qualitative answer examples This appendix collects additional per-tier examples beyond the cases considered in Section 7 (PQ40, TX17, and the two budget-sensitivity cases). Each is chosen for cross-system disagreement (high standard deviation of judge scores across systems); we show the question, a one-line gold-answ...

    arXiv

  52. [2009]

    doi: 10.1561/1500000019

    doi:10.1561/1500000019

  53. [2023]

    arXiv:2303.11366

    URLhttps://arxiv.org/abs/2303.11366. arXiv:2303.11366. 33

    Pith/arXiv arXiv

  54. [2025]

    arXiv:2502.13668

    URLhttps://arxiv.org/abs/2502.13668. arXiv:2502.13668

    Pith/arXiv arXiv

  55. [2026]

    URLhttps://arxiv.org/abs/2602.16313

    arXiv