Pith.

Artificial Intelligence · arXiv:2610.27524 v1 · submitted 20 Oct 2026 · 50 references

MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories

Guorui Chen, Tao Liu, Zhenzhen Huang, Yidan Zhang, Liyu Jiang, Shuoyuan Wang, Hanchen Sun, Xu He

show 4 more authors

Dong Li, Yihan Hu, Jianye Hao, Lei Feng

The paper says

“Code agents increasingly work on evolving real repositories, and memory systems aim to help them reuse earlier experience. However, such experience is diverse and evolving, so it remains unclear how memory systems organize it and thereby affect code agents.”

more of the abstract

Existing benchmarks cannot answer this reliably because they lack either temporally valid histories or control over prior-task relevance. In this paper, we present MemDev-Bench to evaluate how memory systems affect code agents over diverse repository histories. It pairs each task with relevance-annotated earlier tasks from the same repository to build histories with controlled relevance. We also build an automated pipeline that reliably makes these earlier tasks executable and verifiable, producing 4,869 historical tasks for 555 base tasks. Based on MemDev-Bench, our systematic evaluation of memory systems across diverse development scenarios yields three main findings. (1) Memory mainly improves fine-grained localization and editing, increasing functional correctness. (2) Memory also causes failures on otherwise solvable tasks even with related history, and unrelated history tends to increase such failures. These failures mainly stem from agents reusing memory without reassessing its applicability. (3) Compared with using memory systems, directly providing agents with raw historical trajectories substantially reduces their steps. Further analysis suggests that agents save these steps by reusing process information that compressed memories tend to omit. These results suggest that future memory systems should preserve process information and reassess memory applicability against current task context.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 21 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.

Solid benchmark, careful experiments, and genuinely useful findings—the construct-validity gap on relevance annotations is real but doesn't sink the paper. This line is wrong

3 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 50 references say.

The paper says, at §2.2, §4.1

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The paper constructs history scenarios (Relevant, Noisy, Repo) based on human-annotated relevance labels (Strong/Weak/Unrelated). While inter-annotator agreement is high (Fleiss' κ=0.88), the paper does not validate whether these labels predict how the LLM-based agent actually uses the historical information. This is a concern because the central finding that 'history noise tends to increase regressions' depends on the Noisy scenario genuinely containing distracting information.

more of objection 1

Without validation that Strong Related tasks are more helpful to the agent than Unrelated tasks, the observed increase in Harm under Noisy history could be an artifact of the annotation scheme. The paper's manual failure-mode analysis (Section 4.3) partially addresses this by showing that memory-induced failures follow recognizable patterns, but it does not directly test whether the relevance labels correlate with agent behavior. I recommend either (a) conducting a post-hoc analysis that checks whether agent success rates (or Rescue/Harm) differ between Strong and Weak Related tasks, or (b) explicitly acknowledging this limitation and discussing its potential impact on the findings.

The paper says, at §3, §6

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.95):

“Limitations.Our evaluation is limited to the SWE-bench Verified subset of MemDev-Bench under a single agent–model configuration.”

Pith says: Objection 2 of 3 · major

The evaluation uses only one agent (Claude Code) and one base model (DeepSeek-V4-Flash). The paper acknowledges this as a limitation, but it affects generalizability. The findings about memory gains and regressions may not hold for other agents or models. As a benchmark paper, providing results with at least one alternative agent/model (e.g., a different agent framework or a different base model) would strengthen the claims and demonstrate that the benchmark is not tied to a specific configuration.

The paper says, at §4.4

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The paper reports that FullTraj uses fewer steps and output tokens than memory systems, and attributes this to process information preservation. However, the comparison is confounded by the fact that FullTraj has access to raw trajectories while memory systems use compressed representations.

more of objection 3

The paper does not control for the amount of information provided; memory systems could be tuned to provide more process information. The claim that 'agents save these steps by reusing process information that compressed memories tend to omit' is a plausible interpretation but not directly supported by the data. I recommend either (a) designing a controlled experiment that varies the level of process information in memory systems, or (b) softening the causal claim to a correlation.

7 minor objections

The paper says, at Figure 5

Pith says: minor objection 1 of 7

The labels 'Super' and 'Claude' are not defined in the caption; they should be 'Supermemory' and 'Claude-Mem' for consistency with the text.

The paper says, at Tables 2 and 3

Pith says: minor objection 2 of 7

The table headings use 'Rel.', 'Noi.', 'Repo' without expanding in the caption; consider expanding or defining in the footnotes.

show 5 more

The paper says, at §4.2

Pith says: minor objection 3 of 7

The paper says '68 Improvement Tasks that also fail in all three fresh NoMem runs.' Clarify that these are a subset of Improvement Tasks (not all 133).

The paper says, at Appendix C

Pith says: minor objection 4 of 7

The annotation prompts are very detailed; consider summarizing the key decision rules in the main text for readability.

The paper says, at §3

Pith says: minor objection 5 of 7

The baselines 'NoMem' and 'FullTraj' are introduced in text but not explicitly expanded until later; consider defining them on first use.

The paper says, at §5

Pith says: minor objection 6 of 7

Some references to recent methods (MCMA, MemHarness, etc.) are not explained in the main text; a brief description would help readers unfamiliar with these systems.

The paper says, at §2.1

Pith says: minor objection 7 of 7

The cosine similarity threshold of 0.65 and the min(3,n) file filter are presented without justification; a brief rationale or ablation would strengthen the mining method.

What would settle it

Select 30 base tasks, each with at least one Strong Related and one Unrelated historical task. For each, run the agent with access to only that single historical task (not the full scenario) and measure whether the agent's success rate on the base task increases more with Strong Related than with Unrelated history. If the mean Rescue difference (Strong minus Unrelated) is not positive and statistically significant (p<0.05, paired bootstrap across tasks), then the annotation labels do not predict differential agent benefit, weakening the construct validity of the history scenarios.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 21 Oct 2026

  1. Reader holds up confidence moderate
  2. Skeptic partly agrees: says it holds up only with conditions 1 min 2 s · 38,229 tokens read, 5,051 written Relevance annotations define history scenarios, but the label criteria (human judgment of PR helpfulness) are not validated to predict actual LLM-agent behavior; without this, the key finding that 'noise increases regressions' may be based on labels that don't reflect how the agent actually uses…
  3. Editor wrote the verdict above 21 s · 39,203 tokens read, 943 written
  4. Referee major revision: 3 major objections, 7 minor 44 s · 38,674 tokens read, 3,332 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The paper delivers what it promises: a benchmark that pairs code-agent tasks with temporally valid, relevance-annotated historical tasks, then uses it to measure how memory systems actually behave. The pipeline work is substantial—patch splitting with AST, separate environment prep, the rule–experience–agent hierarchy. 4,869 executable historical tasks is a real engineering achievement. The main findings are concrete and nuanced: memory improves fine-grained editing but also causes regressions, and raw trajectories save steps over compressed memory. These are new and worth reporting.

The soft spot the stress-test note flags is real. The relevance labels (Strong/Weak/Unrelated) are human judgments about whether a historical PR provides “direct and actionable information” for a hypothetical developer. The paper validates inter-annotator agreement (Fleiss' κ=0.88) but never checks whether those labels predict how DeepSeek-V4-Flash with Claude Code actually uses the same history. A task labelled Weak Related might be fully usable by the agent, or a Strong Related task might confuse it. Without that validation, the key claim that “noise increases regressions” rests on an untested mapping between human judgments and agent behavior. The authors’ own limitations section (§6) lists the single agent–model configuration but does not flag this gap. That said, the concern is moderate, not fatal. The regression pattern is consistent across scenarios and systems, and even if the annotation-behavior alignment is imperfect, the benchmark itself remains a useful artifact for future work. The single agent–model issue is already acknowledged.

Who this paper is for: anyone building or evaluating memory systems for code agents, and anyone designing benchmarks for cross-task learning in software engineering. It deserves serious peer review. The editors should send it out, but reviewers should push for an explicit discussion of the construct-validity gap and ideally a small validation experiment showing that the annotation levels do correlate with agent outcomes on a held-out sample.

The strongest objection

The paper's central claims about memory-induced regressions under noisy history depend on the Relevance/Noisy/Repo scenarios being faithful representations of helpful vs. distracting history. These scenarios are constructed from human-annotated relevance labels (Strong/Weak/Unrelated). The annotation criteria (Appendix C) instruct raters to judge whether a historical PR 'provides direct and actionable information' for solving the base task, using the content of both tasks. However, this is a human judgment about information value for a hypothetical expert developer, not a measurement of how an LLM-based agent (DeepSeek-V4-Flash with Claude Code) actually uses that information. The paper validates inter-annotator agreement (Fleiss' κ=0.88) but does not validate that the labels predict agent behavior. For example, a task annotated as Weak Related might be fully usable by the agent because the agent can directly copy code patterns, while a Strong Related task might confuse the agent because its solution uses a different approach. Without such validation, the distinction between Relevant and Noisy scenarios may not correspond to actual helpful vs. harmful history for the agent. This could bias the finding that 'history noise tends to increase regressions' (Section 4.1): if Unrelated tasks in Noisy history are sometimes actually helpful, or if Strong Related tasks are sometimes unhelpful, the measured Harm increase might be attenuated or exaggerated. The paper's own limitation statement (§6) notes 'evaluation is limited to... a single agent–model configuration' but does not flag this construct-validity gap. The benchmark remains a useful artifact; the concern is about the strength of the specific claim about noise-induced regressions.

The reader correctly identifies relevance annotation accuracy as the weakest assumption. I share this view but sharpen it: it is not primarily about rater reliability (which the paper measures well) but about predictive validity — whether the labels capture the actual effect of history on LLM agent behavior. The paper provides strong inter-annotator agreement (κ=0.88) and high retention of LLM labels (87.82%), so random rater error is unlikely to be large. However, the annotation task asks humans to judge 'usefulness for solving the base task' based on textual descriptions and code diffs, which may not align with how an LLM agent actually uses historical trajectories in tool-augmented debugging. For instance, a PR might be judged Strong because it modifies the same function, but the agent may ignore the relevant parts and fixate on irrelevant details. Conversely, an Unrelated PR might coincidentally contain a coding pattern that the agent reuses. The paper's findings about rescue/harm across scenarios are plausible, but the specific claim that 'history noise tends to increase regressions rather than suppress Rescue' depends on the scenarios genuinely differing in how distracting the history is. A concrete validation study — testing whether single-task history with Strong labels actually helps more than with Unrelated labels — would settle this. If the test passes, the claim stands; if not, the scenario-based conclusions need to be softened. I therefore recommend a CONDITIONAL verdict: accept the paper contingent on supplementary validation (or on the authors adding this as an explicit caveat to the scenario analyses). The paper remains a significant contribution regardless, because the benchmark itself and the high-level findings about memory trade-offs (rescue vs. harm at similar rates) do not rely as heavily on perfect scenario fidelity.

The referee's report

The benchmark fills a clear gap: existing benchmarks lack either temporal validity or controlled relevance between historical and target tasks. The construction pipeline for executable historical tasks is a practical contribution. The evaluation rigorously measures both gains (Rescue) and regressions (Harm), providing a nuanced picture that goes beyond net accuracy. The finding that raw trajectories reduce agent steps more than compressed memories is novel and actionable for memory system design. The manual failure-mode analysis gives insight into why regressions occur.

Pith's reading of the paper (a machine's summary, not the authors')

The paper argues that whether memory systems help code agents is not a simple yes or no. It builds MemDev-Bench, a benchmark that pairs each software-engineering task with real, relevance-annotated earlier tasks from the same repository, controlling for temporal order and information leakage. Across three history scenarios (relevant, noisy, repo), memory systems help agents solve tasks they otherwise fail (Rescue 9.9–13.2%) but also cause failures on tasks the agent solves without memory (Harm 4.8–9.9%), with regressions increasing under noisy history. Memory's gains come from finer-grained file localization and editing, not from finding new files. Crucially, directly feeding agents the raw historical trajectories reduces steps by about 34% compared to any compressed memory system, suggesting that compressed memories lose process information that agents can reuse.

What carries the argument. The central object is MemDev-Bench, a benchmark that constructs evaluation instances by pairing a base task (from SWE-bench Verified or Pro) with relevance-annotated historical tasks mined from the same repository via explicit references and code-history linkage, enforcing temporal order. It builds three history scenarios (Relevant, Noisy, Repo) to separate the effects of useful context from noise. The automated construction pipeline uses a rule–experience–agent hierarchy to convert historical PRs into executable, verifiable tasks, and an information-isolation protocol prevents answer leakage from public repositories. This machinery enables controlled measurement of Rescue (gains) and Harm (regressions) by comparing resolve rates with and without memory.

Core claim. The paper discovers that while memory systems provide measurable improvements on tasks that code agents consistently fail, they simultaneously cause regressions on tasks the agent can already solve, and these regressions worsen as history becomes noisier. The benefit of memory concentrates on improving within-file localization and edit accuracy, not on discovering new files. A key finding is that providing the complete, raw solving trajectories (FullTraj) reduces agent steps by roughly 34% compared to any compressed memory system, and this reduction stems from preserving the order and detail of the solving process—information that compressed memory representations tend to omit. The authors conclude that future memory systems should preserve process information and actively reassess whether a past solution applies to the current task.

The premise it rests on. The relevance annotations (Strong/Weak/Unrelated) assigned by an LLM and verified by humans are accurate enough that the history scenarios built from them faithfully represent meaningful memory-usage contexts; any systematic error in these labels could bias all downstream comparisons between memory systems and baselines.

If the paper is right (Pith's inferences, not the authors')

Future memory systems should preserve intermediate reasoning steps, failed attempts, and test outcomes rather than compressing experience into summary notes, because agents reuse this process information to reduce search and editing steps.

Memory systems should include an explicit applicability check before agents use stored solutions, since the paper shows that agents often blindly follow memory without reassessing its fit to the current task, leading to misapplied transfer and compatibility failures.

Benchmarks for memory-augmented agents must control for the relevance and noise level of historical contexts, because unrelated history significantly increases regressions while leaving the benefit of related history largely unchanged.

The information-isolation protocol (blocking external answer sources) is essential for measuring genuine memory effects; without it, agents achieve near-perfect resolve rates through answer retrieval, leaving no headroom for memory evaluation.

The 34% step reduction from raw trajectories suggests that compressed memories destroy a kind of "procedural knowledge" that is valuable for code tasks. A natural extension is to design hybrid systems that retain the full trajectory for fine-grained decisions and the compressed summary for high-level guidance.

The finding that regressions increase with noise implies that a memory system's robustness may depend more on its retrieval filtering and relevance assessment than on its storage capacity. This could be tested by building a memory system that retrieves only after verifying applicability via lightweight checks against the current code state.

The manual and LLM-based relevance annotation process, while carefully validated, introduces a human-judgment element that may not transfer to every repository or task type. An automated method that learns relevance from execution outcomes could make the benchmark scalable and language-agnostic.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Code agents increasingly work on evolving real repositories, and memory systems aim to help them reuse earlier experience. However, such experience is diverse and evolving, so it remains unclear how memory systems organize it and thereby affect code agents. Existing benchmarks cannot answer this reliably because they lack either temporally valid histories or control over prior-task relevance. In this paper, we present MemDev-Bench to evaluate how memory systems affect code agents over diverse repository histories. It pairs each task with relevance-annotated earlier tasks from the same repository to build histories with controlled relevance. We also build an automated pipeline that reliably makes these earlier tasks executable and verifiable, producing 4,869 historical tasks for 555 base tasks. Based on MemDev-Bench, our systematic evaluation of memory systems across diverse development scenarios yields three main findings. (1) Memory mainly improves fine-grained localization and editing, increasing functional correctness. (2) Memory also causes failures on otherwise solvable tasks even with related history, and unrelated history tends to increase such failures. These failures mainly stem from agents reusing memory without reassessing its applicability. (3) Compared with using memory systems, directly providing agents with raw historical trajectories substantially reduces their steps. Further analysis suggests that agents save these steps by reusing process information that compressed memories tend to omit. These results suggest that future memory systems should preserve process information and reassess memory applicability against current task context.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-21

The authors' main claim, as the classifier picked it out 0.97

“These results suggest that future memory systems should preserve process information and reassess memory applicability against current task context.”

How the main claim is stated: hedged possibility 0.99

What the abstract itself states, as probabilities: claims first 0.12 · states limitation 0.24 · human participants 0.05 · controlled comparison 0.89 · randomized assignment 0.02 · states quantitative result 0.96 · compares with prior results 0.90 · mentions code or data availability 0.06

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Code agents increasingly work on evolving real repositories, and memory systems aim to help them reuse earlier experience. background 1.00 0.04 0.02 asserted 1.00
However, such experience is diverse and evolving, so it remains unclear how memory systems organize it and thereby affect code agents. gap 0.96 0.05 0.02 asserted 0.70
Existing benchmarks cannot answer this reliably because they lack either temporally valid histories or control over prior-task relevance. gap 0.99 0.09 0.02 asserted 1.00
In this paper, we present MemDev-Bench to evaluate how memory systems affect code agents over diverse repository histories. contribution 1.00 0.93 0.04 asserted 0.97
It pairs each task with relevance-annotated earlier tasks from the same repository to build histories with controlled relevance. method 0.97 0.94 0.02 asserted 1.00
We also build an automated pipeline that reliably makes these earlier tasks executable and verifiable, producing 4,869 historical tasks for 555 base tasks. method 0.96 0.98 0.02 asserted 1.00
Based on MemDev-Bench, our systematic evaluation of memory systems across diverse development scenarios yields three main findings. result 0.62 0.94 0.09 asserted 0.99
(1) Memory mainly improves fine-grained localization and editing, increasing functional correctness. result 1.00 0.95 0.23 asserted 1.00
(2) Memory also causes failures on otherwise solvable tasks even with related history, and unrelated history tends to increase such failures. result 1.00 0.93 0.17 asserted 1.00
These failures mainly stem from agents reusing memory without reassessing its applicability. interpretation 0.91 0.87 0.11 asserted 0.99
(3) Compared with using memory systems, directly providing agents with raw historical trajectories substantially reduces their steps. result 1.00 0.95 0.19 asserted 1.00
Further analysis suggests that agents save these steps by reusing process information that compressed memories tend to omit. interpretation 0.95 0.88 0.09 hedged possibility 0.97
These results suggest that future memory systems should preserve process information and reassess memory applicability against current task context. future work 0.91 0.60 0.72 hedged possibility 0.99

What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-21: 1 request, 9,310 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (13)
Figure 1 of the paper
Figure 1. Base tasks distribution. Left: SWE-bench Verified (344 tasks, 12 Python repositories). Right: SWE-bench Pro (211 tasks, 10 repositories, four languages). 2 MemDev-Bench: Construction and Statistics MemDev-Bench evaluates how memory systems affect code agents on later tasks after these systems build memory from agent trajectories on earlier tasks in the same repository. Each evaluation case pairs a software engineering task, referred to as the base task, with its historical tasks. We collect base tasks from SWE-bench Verified (Chowdhury et al., 2024) and SWE-bench Pro (Deng et al.,…
Figure 2 of the paper
Figure 2. Benchmark construction and evaluation pipeline. Top: we mine candidate PRs, annotate their relevance, and convert them into executable historical tasks. Bottom: an agent solves historical tasks chronologically to build memory, then solves the base task with the frozen memory. the earlier commits to their PRs. Frequently modified files such as configuration and utility files can create accidental overlap between otherwise unrelated PRs. To reduce such overlap, the mapped PRs pass through two filters. The first filter keeps PRs that share at least min(3, 𝑛) changed files with the base…
Figure 3 of the paper
Figure 3. Empirical cumulative distributions of historical task properties. (a) Time gap to the base task; (b) non-test patch size in characters; (c) number of changed files. 3 Experimental Setup To control evaluation cost, all experiments use the Verified subset of MemDev-Bench, which contains 344 base tasks and 2,987 historical tasks from 12 Python repositories. Agent and base model. We use Claude Code (Anthropic, 2025) as the agent and DeepSeek-V4- Flash (DeepSeek-AI, 2026) as the base model. Automatic context compaction is disabled to avoid affecting the measured effect of memory.…
Figure 4 of the paper
Figure 4. Counts of failure modes. Failure-mode analysis across history scenarios. As shown in
Figure 5 of the paper
Figure 5. FullTraj reductions in steps, output tokens, and action calls. Comparisons with NoMem and five memory systems use pooled means over 183 instances. Super and Claude denote Supermemory and Claude-Mem. code inspection, failed attempts, revisions, and test outcomes, whereas FullTraj retains these events in execution order. The results here show the potential benefit of preserving the historical solving process rather than only compressed summaries. Meanwhile, Section 4.3 shows that historical solutions can mislead the agent when their assumptions do not hold for the current task.…
Figure 6 of the paper
Figure 6. Distribution of fine-grained relation types within each relevance level. Beyond the coarse relevance levels, these fine-grained types capture the specific ways in which historical PRs can help, contextualize, or distract from a base task. Strong Related types characterize directly transferable implementation or design information, Weak Related types capture useful but indirect context, and Unrelated types distinguish different forms of non-useful or misleading overlap
Figure 7 of the paper
Figure 7. Executable historical task construction pipeline. (a) PR changes are split into test patch 𝑇 and solution patch 𝑆, and the task inputs are frozen. (b) The original commit 𝑐 is resolved and validated to construct a shared Base image. (c) The two patched source states independently invoke the shared resolver, and their plans and environment artifacts are frozen as Post A and Post B for offline restoration. (d) Stable outcomes are obtained from three fresh runs on each side. changes introduced by the PR are excluded from the base environment. After validation, the resulting environment…
Figure 8 of the paper
Figure 8. Shared trajectory. An initial change breaks default metadata preservation in the existing methods. The same five tests expose the regression and pass after the repair. G.1 A-Mem: Evolving Note Graph Memory construction. A-Mem (Xu et al., 2025) organizes reusable knowledge from Claude Code as linked notes. In this integration, a session-start hook injects instructions to search memory before each task and save new findings through MCP during code exploration, debugging, or repair. The add_memory_note description covers architectural relations, code locations, errors and solutions,…
Figure 9 of the paper
Figure 9. A-Mem with Claude Code. Claude composes a note from 𝑒3–𝑒5, which A-Mem enriches and links. Retrieval combines semantic previews, selective reading, and optional link expansion. G.2 Mem0: Persistent Repository Memory Memory construction. Mem0 (Chhikara et al., 2025) distills an agent’s interaction history into persistent memory for retrieval across sessions. In Step 1 of
Figure 10 of the paper
Figure 10. Mem0 with Claude Code. Selected interaction evidence is distilled into persistent memories for scoped retrieval. G.3 MemOS: Text Memories in MemCube Memory construction. MemCube is the unified memory abstraction proposed by MemOS and implemented in its open-source framework as a modular container for textual, activation, and parameter memories (Z. Li et al., 2025). It supports the organization and persistence of these memory types
Figure 11 of the paper
Figure 11. MemOS with Claude Code. MemReader extracts textual memories into MemCube for later semantic retrieval. G.4 Supermemory: Documents, Memory Graphs, and Profiles Memory construction. Supermemory stores Claude Code interactions as source documents and distills them into retrievable atomic memories (Daga et al., 2026). In Step 1 of
Figure 12 of the paper
Figure 12. Supermemory with Claude Code. Documents yield linked atomic memories for project profiles and subsequent retrieval. G.5 Claude-Mem: Observations and Progress Summaries Memory construction. Claude-Mem captures coding activity through Claude Code’s lifecycle hooks and compresses it into observations and progress summaries in a background worker (Newman, 2025). In Step 1 of
Figure 13 of the paper
Figure 13. Claude-Mem with Claude Code. Observations and progress summaries support startup context and selective retrieval. MemoraX AI Research · MemDev-Bench 38 / 41

Figures are the authors', reproduced from the paper.

References (50)
  1. [1] Chowdhury, Neil and Aung, James and Shern, Chan Jun and Jaffe, Oliver and Sherburn, Dane and Starace, Giulio and Mays, Evan and Dias, Rachel and Aljubeh, Marwan and Glaese, Mia and Jimenez, Carlos E. and Yang, John and Ho, Leyton and Patwardhan, Tejal and Liu, Kevin and Madry,... on Pith
  2. [2] Deng, Xiang and Da, Jeff and Pan, Edwin and He, Yannis Y. and Ide, Charles and Garg, Kanak and Lauffer, Niklas and Park, Andrew and Rane, Chetan and Sampath, Karmini and Krishnan, Maya and Kundurthy, Srivatsa R. and Hendryx, Sean M. and Wang, Zifan and Zhang, Chen Bo Calvin an... on Pith
  3. [3] Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R. , title =. ICLR , year = on Pith
  4. [4] arXiv preprint arXiv:2606.19348 , year = on PitharXiv
  5. [5] Fleiss, Joseph L. , title =. Psychological Bulletin , volume = on Pith
  6. [6] Xu, Wujiang and Liang, Zujie and Mei, Kai and Gao, Hang and Tan, Juntao and Zhang, Yongfeng , title =. NeurIPS , year = on Pith
  7. [7] 2025 , howpublished = on Pith
  8. [8] 2026 , howpublished = on Pith
  9. [9] Hybrid Episodic Memory , year = on Pith
  10. [10] Newman, Alex , title =. 2025 , howpublished = on Pith
  11. [11] Wen, Deliang and Sun, Ke and Wang, Yu , title =. arXiv preprint arXiv:2603.22306 , year = on PitharXiv
  12. [12] Li, Zhiyu and Xi, Chenyang and Li, Chunyu and Chen, Ding and Chen, Boyu and Song, Shichao and Niu, Simin and Wang, Hanyu and Yang, Jiawei and Tang, Chen and Yu, Qingchen and Zhao, Jihao and Wang, Yezhaohui and Liu, Peng and Lin, Zehao and Wang, Pengyuan and Huo, Jiahao and Che... on PithPith's reviewarXiv
  13. [13] Agent Memory Service , year = on Pith
  14. [14] Fang, Jizhan and Deng, Xinle and Xu, Haoming and Jiang, Ziyan and Tang, Yuqi and Xu, Ziwen and Deng, Shumin and Yao, Yunzhi and Wang, Mengru and Qiao, Shuofei and Chen, Huajun and Zhang, Ningyu , title =. ICLR , year = on Pith
  15. [15] Chhikara, Prateek and Khant, Dev and Aryan, Saket and Singh, Taranjeet and Yadav, Deshraj , title =. ECAI , year = on Pith
  16. [16] Vanilla. 2026 , howpublished = on Pith
  17. [17] Markovic, Vasilije and Obradovic, Lazar and Hajdu, Laszlo and Pavlovic, Jovan , title =. arXiv preprint arXiv:2505.24478 , year = on PithPith's reviewarXiv
  18. [18] Ji, Suozhao and Wu, Baodong and Wang, Zehao and Xia, Lei and Li, Qingping and Wang, Ruisong and Ding, Wenbo and Zhu, Zhenhua and Li, Boxun and Dai, Guohao and Wang, Yu , title =. arXiv preprint arXiv:2606.10677 , year = on PithPith's reviewarXiv
  19. [19] Xu, Buqiang and Chen, Yijun and Fang, Jizhan and Zhong, Ruobin and Yao, Yunzhi and Zhu, Yuqi and Du, Lun and Deng, Shumin , title =. ACL (Short Papers) , year = on Pith
  20. [20] Li, Ruizhe and Zhang, Licheng and Xu, Benfeng and Du, Mingxuan and Fu, Zheren and Chen, Weidong , title =. arXiv preprint arXiv:2608.12888 , year = on PithPith's reviewarXiv
show 30 more
  1. [21] Jovovich, Milla and Sigman, Ben , title =. 2026 , howpublished = on Pith
  2. [22] Daga, Soham and Sreedhar, Sreeram and Shah, Dhravya , title =. 2026 , howpublished = on Pith
  3. [23] Hu, Chuanrui and Gao, Xingze and Zhou, Zuyi and Xu, Dannong and Bai, Yi and Li, Xintong and Zhang, Hui and Li, Tong and Zhang, Chong and Bing, Lidong and Deng, Yafeng , title =. ACL (Long Papers) , year = on Pith
  4. [24] Liu, Jiaqi and Su, Yaofeng and Xia, Peng and Han, Siwei and Zheng, Zeyu and Xie, Cihang and Ding, Mingyu and Yao, Huaxiu , title =. ICML , year = on Pith
  5. [25] Hybrid Memory Search , year = on Pith
  6. [26] Wang, Shu and Yu, Edwin and Love, Oscar and Zhang, Tom and Wong, Tom and Scargall, Steve and Fan, Charles , title =. arXiv preprint arXiv:2604.04853 , year = on PithPith's reviewarXiv
  7. [27] Latimer, Christopher and Boschi, Nicol. ACL (System Demonstrations) , year = on Pith
  8. [28] Liu, Xule and Teng, Hanlin and Li, Chao and Ni, Yanan and Lu, Shuo and Wang, Audrey and Liu, Yijun and Wang, Yunfei and Li, Xiaofeng and Yi, Xian and Li, Yuanfa and Zhao, Kang and Liang, Jian and Chen, Yuxuan and Chen, Jinyuan and Qu, Heng and Shao, Kun and Luan, Jian , title ... on PithPith's reviewarXiv
  9. [29] Dai, Zijie and Deng, Shiyuan and Guan, Sheng and Tian, Yizhou and Yao, Xin and Yan, Xiao and Cheng, James , title =. Findings of ACL , year = on Pith
  10. [30] Liang, Sirui and Cao, Pengfei and Zhao, Jian and Teng, Wenhao and Liao, Xiangwen and Zhao, Jun and Liu, Kang , title =. Findings of ACL , year = on Pith
  11. [31] Wu, Rong and Fu, Daocheng and Wen, Licheng and Yang, Xuemeng and Zou, Shu and Mei, Jianbiao and Wang, Yuxin and Zhang, Hairong and Yang, Yu and Hu, Tao and Zhang, Cong and Shi, Botian and Cai, Pinlong , title =. arXiv preprint arXiv:2607.28272 , year = on PithPith's reviewarXiv
  12. [32] Ye, Bowen and Xu, Yongchao and Li, Zhijian and Yin, Xiang and Ma, Junkai and Li, Wenzhao , title =. arXiv preprint arXiv:2608.01739 , year = on PithPith's reviewarXiv
  13. [33] Guan, Xinyu and Zhao, Qianyang and Deng, Yuming , title =. arXiv preprint arXiv:2606.08151 , year = on PithPith's reviewarXiv
  14. [34] Lin, Zihan and Chen, Zhenyu and Wei, Jiawen and Wang, Xiaohan and Cao, Jie and Chai, Jiajun and Lin, Wei and Yin, Guojun and He, Ran , title =. arXiv preprint arXiv:2608.22339 , year = on PithPith's reviewarXiv
  15. [35] Tang, Bo and Zhang, Yang and Zhuang, Guomian and Wei, Wenqiang and Zheng, Gaoyang and Xie, Lindong and Tan, Yanchao and Xiong, Feiyu and Yang, Qingyu and Chung, Edward and Li, Zhiyu , title =. arXiv preprint arXiv:2607.16621 , year = on PithPith's reviewarXiv
  16. [36] Yu, Zhaochen and Wu, Yingcheng and Yin, Zhenfei and Chen, Kaiyuan and Zhao, Zhe and Wang, Mengdi and Yan, Shuicheng and Yang, Ling , title =. arXiv preprint arXiv:2608.24876 , year = on PithPith's reviewarXiv
  17. [37] Zhang, Linghao and He, Shilin and Zhang, Chaoyun and Kang, Yu and Li, Bowen and Xie, Chengxing and Wang, Junhao and Wang, Maoquan and Huang, Yufan and Fu, Shengyu and Nallipogu, Elsie and Lin, Qingwei and Dang, Yingnong and Rajmohan, Saravan and Zhang, Dongmei , title =. NeurI... on Pith
  18. [38] Kolavi, Adithya S. , title =. 2026 , howpublished = on Pith
  19. [39] Guo, Lianghong and Wang, Yanlin and Li, Caihua and Tao, Wei and Yang, Pengyu and Chen, Jiachi and Song, Haoyu and Tang, Duyu and Zheng, Zibin , title =. Proceedings of the ACM on Software Engineering , volume = on Pith
  20. [40] Guo, Chuanzhe and Wu, Jingjing and He, Sijun and Chen, Yang and Kuang, Zhaoqi and Fan, Shilong and Chen, Bingjin and Bao, Siqi and Liu, Jing and Wu, Hua and Zhu, Qingfu and Che, Wanxiang and Wang, Haifeng , title =. ICML , year = on Pith
  21. [41] Mouxiang Chen and Lei Zhang and Yunlong Feng and Xuwu Wang and Wenting Zhao and Ruisheng Cao and Jiaxi Yang and Jiawei Chen and Mingze Li and Zeyao Ma and Hao Ge and Zongmeng Zhang and Zeyu Cui and Dayiheng Liu and Jingren Zhou and Jianling Sun and Junyang Lin and Binyuan Hui ... on PithPith's reviewarXiv
  22. [42] Jin, Qirui and Tung, Lingching and Li, Kenan and Shi, Qiyang and She, Yushi and Jia, Huanzhong and Zhao, Harrison and Xia, Kejing and Du, Zhenbang and Zhang, Yikai and Pei, Jiaxin and Zhang, Zhenyu and Qi, Zhen and Duan, Yuyan and Lee, Wenke and Jin, Zijian , title =. arXiv pr... on PithPith's reviewarXiv
  23. [43] Joshi, Thomas and Chowdhury, Shayan and Uysal, Fatih , title =. arXiv preprint arXiv:2507.00014 , year = on PithPith's reviewarXiv
  24. [44] Singh, Sarthak , title =. arXiv preprint arXiv:2608.20664 , year = on PithPith's reviewarXiv
  25. [45] Zhu, Jiayuan and Wu, Junde and Hu, Minhao and Zhu, Shengda and Pan, Jiazhen and Shen, Weixiang and Yang, Yijun and Liu, Fenglin and Hao, Jianye and Jin, Yueming and Ho, Qirong and Xu, Min , title =. arXiv preprint arXiv:2602.08316 , year = on PithPith's reviewarXiv
  26. [46] Wang, Bo and Yao, Yuqian and Wang, Enxi and Jin, Luozhijie and Liu, Yang and Suo, Yiran and Cai, Yuxuan and Zhou, Enyu and Gao, Yufei and Guo, Honglin and Huai, Tianyu and Ji, Li and Lei, Zhikai and Li, Bufan and Lin, Lizhi and Liu, Jinxiu and Yang, Jie and Zhou, Jiazheng and ... on PithPith's reviewarXiv
  27. [47] Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik R. and Press, Ofir , title =. NeurIPS , year = on Pith
  28. [48] Wang, Xingyao and Li, Boxuan and Song, Yufan and Xu, Frank F. and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and Song, Yueqi and Li, Bowen and Singh, Jaskirat and Tran, Hoang H. and Li, Fuqiang and Ma, Ren and Zheng, Mingzhang and Qian, Bill and Shao, Yanjun and Muennigh... on Pith
  29. [49] Zhang, Shengtao and Wang, Jiaqian and Zhou, Ruiwen and Liao, Junwei and Feng, Yuchen and Li, Zhuo and Zheng, Yujie and Zhang, Weinan and Wen, Ying and Li, Zhiyu and Xiong, Feiyu and Qi, Yutao and Tang, Bo and Wen, Muning , title =. arXiv preprint arXiv:2601.03192 , year = on PithPith's reviewarXiv
  30. [50] Zhang, Haozhen and Long, Quanyu and Bao, Jianzhu and Feng, Tao and Zhang, Weizhi and Yue, Haodong and Wang, Wenya , title =. arXiv preprint arXiv:2602.02474 , year = on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment