{"id":"a00db90c-1e92-44f9-983f-66487a4317c2","arxiv_id":"2608.12428","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MindMemOS is an agent memory system that organizes information as entity-property-time records, evolves its schema, consolidates memories, and refines skills, reporting top scores on LOCOMO and PersonaMem.","lead":"This paper describes MindMemOS, a memory layer that lets AI agents store and retrieve facts using an entity-property-time structure, then automatically adapts that structure and refines its procedural skills through continued use. It reports top scores on long-term memory benchmarks and a 9.2-point improvement on spreadsheet tasks after skill evolution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LOCOMO SOTA margin rests on cited baselines and a four-type subset; if EverOS numbers used different prompts, judge, or subset, the 0.98-point lead could vanish. A matched-harness rerun would settle it.","rationale":"The reader's weakest_assumption is exactly the fairness of the LOCOMO comparison: imported baselines, a four-type subset, and the 'fully aligned' assertion. My stress-test converges on the same point. The margin is small (0.98 pp), the baselines are not independently rerun, and no error bars are given. This is sufficient to keep the verdict CONDITIONAL: the central claim is plausible and the architecture is coherent, but the headline SOTA statement should not be accepted without a reproducible matched-harness comparison. I considered whether the MindSkillEvolve result (9.2 pp over an initial skill that is worse than no-skill) or the MindMemEvolve test-set gain (3.56 pp on one split) is more load-bearing. Those are weaker concerns: the skill-evolution paper reports standard deviations over three runs and compares against both no-skill and init-skill, and the MindMemEvolve generalization result is presented as modest with an explicit limitation. The LOCOMO comparison, by contrast, is the primary quantitative support for the paper's strongest claim, and its fairness is asserted rather than demonstrated. Therefore the reader's CONDITIONAL verdict is the right level: neither accept as-is nor reject outright. My concern does not move the verdict, so I mark it UNCHANGED and agree with the reader's diagnosis.","tokens_in":19670,"tokens_out":5839,"duration_ms":60097,"concrete_test":"Re-run the LOCOMO evaluation under one harness: obtain the official EverOS codebase and run EverOS and MindMemOS on the same machine with the same model (gpt-4.1-mini for the answer model, same judge prompt, same embedding/reranker), the same memory-system LLM, and the same question subset (report both the full 1,986-question set and the four-type subset with per-category counts). If EverOS's matched-subset overall accuracy equals or exceeds MindSchema's 94.03, the headline SOTA claim is not supported. If EverOS's published numbers are not reproducible, the comparison should be downgraded to an uncontrolled reference rather than a head-to-head result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central headline claim is the LOCOMO comparison in Section 4.1 / Table 1: MindSchema at 94.03% overall vs. EverOS at 93.05%, a 0.98-point margin. The paper states that Mem0, memU, Zep, MemOS, and EverOS baselines are 'cited from the EverOS paper' and that MindMemOS's configuration is 'fully aligned with the EverOS default implementation.' That alignment assertion is the only safeguard against an apples-to-oranges comparison, and the paper does not provide the evidence needed to check it: the exact prompt templates for the baselines, the identity of the 'LLM driving the memory system,' the judge prompt, and the precise composition of the 'four main reasoning types' subset (how many of the 1,986 questions are included, and per-category counts). If EverOS's published numbers were obtained with a different subset definition, a different judge model, or a different answer-model prompting scheme, the 0.98-point margin could vanish or even reverse. The same section also reports no error bars or significance test, so it is unclear whether 94.03 vs. 93.05 is distinguishable from noise. This is a verification gap, not an accusation; the rest of the paper (PersonaMem baselines run locally, dreaming with a within-system before/after, skill evolution with repeated runs) is less vulnerable. But because the abstract's '94.03%' and the 'state-of-the-art' framing rest on this one imported comparison, this is the most load-bearing assumption in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MindMemOS proposes a portable memory operating layer for LLM agents, built on an entity–property–time graph, with four self-adaptation mechanisms: MindMemEvolve (evolutionary schema search), dreaming (offline memory consolidation), feedback (explicit and implicit correction handling), and MindSkillEvolve (trajectory-driven skill refinement). The system is evaluated on LOCOMO, PersonaMem, MemoryAgentBench (FactConsolidation), and SpreadsheetBench, reporting 94.03% on LOCOMO, 70.63% on PersonaMem, dreaming gains up to +0.084 accuracy with ~20% compression, and a 5.9–9.2 percentage-point skill-evolution improvement over no-skill/initial-skill baselines. The paper also includes case studies and full appendix schemas and skill contents. The central claims are empirically evaluated against public benchmarks; the LOCOMO SOTA claim, however, rests on baselines imported from a prior paper.","tokens_in":20054,"tokens_out":4333,"duration_ms":45925,"significance":"If the headline results are robust, MindMemOS is a useful engineering contribution: it unifies schema-guided and schema-free memory, adds a plausible evolutionary schema optimizer, and demonstrates that offline consolidation and skill evolution improve downstream task success. The paper ships a code repository, provides complete schema and skill definitions in appendices, and reports within-system before/after dreaming comparisons, which are not circular. The MindMemEvolve evaluation is held-out with training-set selection, which is methodologically sound. The main significance is bounded by evaluation gaps: the LOCOMO SOTA margin is small and imported, and several tables lack error bars or significance tests. The contributions are credible but not yet fully established as stated.","major_comments":[{"comment":"The headline LOCOMO claim (MindSchema 94.03 vs EverOS 93.05) rests on baselines 'cited from the EverOS paper' and on a subset of 'four main reasoning types' whose exact composition is not reported. The paper states that the configuration is 'fully aligned with the EverOS default implementation,' but it does not provide the evidence needed to verify alignment: the prompt templates used for baselines, the identity of the LLM driving the memory system, the judge prompt, or the per-category question counts in the four-type subset. Since the margin is 0.98 points, the abstract-level state-of-the-art claim depends on this imported comparison. Please either rerun the baselines in the same harness under the authors' control or provide a detailed protocol-matching appendix, and report error bars or a significance test.","section":"Section 4.1, Table 1"},{"comment":"The PersonaMem baselines are run locally, which is better than importing numbers, but the table still lacks error bars or significance tests. The overall gain is 70.63 vs 67.57 (3.06 points), and the per-category column 'Recall Mem.' contains only 17 questions, so a difference of one or two answers can swing category-level accuracy by several points. The paper's own note about this small category undermines confidence that the reported overall advantage is robust. Please report variance (e.g., multiple runs or bootstrap confidence intervals) and a significance test, or moderate the claim of 'best overall accuracy.'","section":"Section 4.1, Table 2"},{"comment":"The evidence for MindMemEvolve, a central contribution, is a single held-out benchmark with a 3.56-point improvement, and the paper itself states that PersonaMem is not a 'sensitivity-demanding' scenario and that the search space explored in 60 steps may be limited. The evolved schema expands to 49 properties, but this is presented only on one dataset with one bootstrap schema. Because the contribution is framed as a general self-adaptation mechanism, a single modest result is insufficient to support that generality. Please add at least one additional scenario or benchmark for MindMemEvolve, or explicitly reframe the contribution as a preliminary demonstration on PersonaMem.","section":"Section 5.3, Figure 14"}],"minor_comments":[{"comment":"The column header 'T emporal' has a spacing typo; it should read 'Temporal.'","section":"Table 1"},{"comment":"The text refers to 'EverOS' but reference [10] lists 'EverMemOS'; please make the naming consistent throughout.","section":"References"},{"comment":"The JSON schema in Appendix A contains '//' comments, which are not valid JSON; either remove them or clearly label the block as a schema illustration rather than a parseable file.","section":"Appendix A"},{"comment":"The feedback case study is forthrightly labeled as a single-instance demonstration, which is good; a brief sentence about generality would help but is not required.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is credible and the engineering is substantial, but the central SOTA claim on LOCOMO depends on a small margin over imported baselines. Please request the authors to provide a matched-harness comparison or a very detailed protocol appendix, and to include variance/significance information for the main benchmark tables. The MindMemEvolve section could also be strengthened by additional scenarios."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious systems paper, not a breakthrough. The actual contribution is an integrated memory stack—entity-property-time modeling, evolutionary schema search (MindMemEvolve), offline consolidation (dreaming), feedback, and trajectory-to-skill evolution—and the authors have been unusually honest about the limits of each piece. The one number I would not relay without checking is the headline LOCOMO 94.03 vs. 93.05: all baselines there are imported from EverOS, and the margin is 0.98 points. That is a verification gap, not an accusation.\n\nWhat's genuinely good: the system is coherent and the evaluation is more diverse than most memory papers. The dreaming experiments are within-system before/after and show consistent gains on MemoryAgentBench plus a 22.5% compression. The MindMemEvolve section uses a clean train/test split, and the paper says outright that the 3.56-point gain is modest and that PersonaMem may not be sensitive enough. The skill evolution on SpreadsheetBench has three runs with error bars, and the case study shows how supervised evolution avoids overfitting to literal-value grading. That is the right kind of evidence.\n\nSoft spots, in proportion: the LOCOMO comparison is the weakest link. The baselines are cited, not reproduced, and the four-type subset is not shown with per-category counts. If EverOS used a different judge or prompt, the margin vanishes. The paper needs to release the harness or rerun baselines. There are also no error bars in Tables 1 and 2; on a 0.98-point margin, that is relevant. PersonaMem's Recall Mem. category is 17 questions, so small-sample noise matters. These are all fixable, and the paper's own hedging suggests the authors know.\n\nWho this is for: systems researchers building agent memory, and anyone who wants a working reference for self-evolving schema and skill loops. I'd rather see it in the literature than not, but it needs a revision that closes the LOCOMO question.\n\nRecommendation: send it to peer review. Ask the authors to share the exact evaluation harness, run the baselines locally, and add confidence intervals to the two main tables.","headline":"A credible integrated memory system, but the headline LOCOMO claim rests on imported baselines and a sub-point margin that needs verification before I'd repeat it.","tokens_in":20630,"tokens_out":2663,"would_cite":true,"duration_ms":25647,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent memory can be made self-evolving by storing facts as entity-property-time records and revising schemas, conflicts, and skills from usage.","keywords":["AI agent memory","entity-property-time structure","memory schema evolution","memory consolidation","skill evolution","LOCOMO benchmark","PersonaMem benchmark","SpreadsheetBench"],"falsifier":"Re-run the four LOCOMO reasoning types with EverOS, Mem0, memU, Zep, and MemOS inside the authors' own evaluation harness, using identical prompt templates and judge model on the same question subset. If 94.03% no longer beats 93.05% by a stable margin, the claimed superiority is an evaluation artifact; if it persists, the comparison stands.","tokens_in":19483,"feed_emoji":"🧠","tokens_out":11306,"duration_ms":99580,"temperature":0.7,"pith_summary":"The paper's central claim is that an AI agent's memory does not have to stay frozen after deployment: if memories are stored in a shared entity–property–time structure, then schemas, stored facts, and even procedural skills can all be revised by autonomous loops driven by task feedback. On the long-term dialogue benchmark LOCOMO, the schema-guided MindSchema configuration reaches 94.03% overall accuracy, ahead of EverOS at 93.05%, and on PersonaMem it reaches 70.63% vs 67.57%. Offline dreaming consolidation raises conflict-resolution accuracy while archiving about a fifth of active memories, and trajectory-driven skill evolution lifts spreadsheet task success from 51.3% to 57.2%. The point of the system is portability: the same memory layer can ingest information with or without a schema and can optimize that schema per scenario without retraining the underlying LLM agent.","feed_headline":"Self-evolving memory layer tops long-term recall at 94.03%","feed_subtitle":"It rewrites its schemas, resolves conflicts offline, and turns execution traces into reusable skills.","key_machinery":"The load-bearing object is the three-dimensional entity–property–time memory structure, where each memory record binds an entity to a property and a temporal reference, with relational edges between entities and timeline edges between successive property versions. This structure is what makes the other mechanisms possible: it gives a stable interface for adding, searching, and mutating memories, and it lets conflicts be represented as competing versions on a timeline rather than as overwrites. The paper's named algorithms are routes through this interface: MindMemEvolve (an LLM-guided evolutionary search over schema definitions, with induced mutation from judge errors), dreaming (offline entity-centered consolidation that archives superseded memories and records supersedes edges), feedback (explicit and implicit correction-to-action conversion with persistence scoping), and MindSkillEvolve (trajectory analysis into versioned skill edits, with a score-supervised variant). The compact search module ties them to queries by hybrid BM25 plus dense retrieval, bidirectional graph traversal, and an agentic controller that decides when enough evidence has been collected.","core_discovery":"MindMemOS claims that memory quality in LLM agents is itself a designable and optimizable layer, not a fixed store. It organizes all memories as nodes in a three-dimensional graph (entity × property × time), so a fact is always bound to what it is about, which attribute it describes, and when it holds; this lets the system keep timelines instead of overwriting history. On top of that representation, four evolution mechanisms operate: MindMemEvolve uses validation-driven evolutionary search to rewrite the schema (expanding 2 properties to 49 in a PersonaMem case, with held-out accuracy rising from 61.07% to 64.63%); dreaming merges redundant and contradictory records offline and archives obsolete ones; feedback converts user corrections into persistent or task-temporary memory edits; and MindSkillEvolve turns execution trajectories into versioned skill updates. The paper reports that this combination achieves the best overall accuracy on LOCOMO (94.03) and PersonaMem (70.63) among the compared systems, improves FactConsolidation accuracy on MemoryAgentBench under two models, and raises SpreadsheetBench success by up to 5.9 points over the no-skill baseline.","pith_inferences":["A testable extension the paper does not run: the same evolutionary loop that rewrites schemas could also tune retrieval depth, consolidation frequency, or feedback sensitivity, since all of them sit behind the same memory interface.","The unevolved initial skill's underperformance suggests a bootstrapping constraint: self-evolution helps only if the starting skill is validated, a design problem the paper documents but does not elevate into a headline result.","Dreaming turns timestamp comparison into persistent supersedes edges; an untested corollary is that the same mechanism could resolve non-temporal conflicts, such as changed preferences across sessions, where the current evidence is limited to one illustrative case.","MindVanilla's 87.60 vs MindSchema's 94.03 on LOCOMO indicates schema guidance is a major source of the gain; a clean ablation would separate modeling from retrieval, which the paper's evaluations do not fully isolate."],"forward_implications":["If the reported evaluations hold, a schema-guided memory layer can outperform hand-tuned memory systems on very long conversational memory without retraining the agent; the 94.03% LOCOMO overall and 82.29% open-domain scores are the concrete manifest.","Dreaming gives a measurable win on selective-forgetting tasks: it raises overall FactConsolidation accuracy (for example, 0.377 to 0.459 with gpt-4o-mini) while moving roughly one fifth of active memories to archive, so consolidation and compression can coexist with better retrieval.","Trajectory-derived skills are sufficient to improve downstream task success even without task scores (55.3% vs 51.3% no-skill on SpreadsheetBench), and adding scores tightens the rules further (57.2%).","Because the same structure supports both schema-free MindVanilla and schema-guided MindSchema, portability and adaptivity are not in tension: one layer can span both modes behind a single API.","Schema evolution generalizes to held-out data in the reported PersonaMem split (61.07% to 64.63%), so the optimized schema is not merely overfitting the training clusters."],"supporting_citations":[{"why":"It is the main comparison point: the paper cites EverOS's LOCOMO numbers from here and says its own configuration is aligned with the EverOS default implementation.","marker":"[10]"},{"why":"It supplies the LOCOMO benchmark, including the 10 long multi-session dialogues, 1,986 annotated questions, and the four reasoning types used in the central evaluation.","marker":"[19]"},{"why":"It supplies the PersonaMem benchmark, with 20 personas and seven query types, used to test personalization and preference tracking.","marker":"[14]"},{"why":"It supplies the MemoryAgentBench FactConsolidation subset and protocol used to evaluate dreaming's conflict resolution.","marker":"[11]"},{"why":"It supplies SpreadsheetBench's verified 400-task subset, used as the testbed for MindSkillEvolve.","marker":"[18]"},{"why":"It supplies the initial spreadsheet skill that MindSkillEvolve starts from, plus the observation that an unevolved skill can underperform no-skill execution.","marker":"[26]"},{"why":"It is a baseline memory system whose reported numbers on LOCOMO and PersonaMem MindMemOS is compared against.","marker":"[5]"},{"why":"It is a baseline memory system (MemOS) whose reported numbers on LOCOMO and PersonaMem MindMemOS is compared against.","marker":"[17]"}],"fun_headline_variants":["Self-evolving memory: agents rewrite their own recall schemas","MindMemOS: memory layer evolves itself for AI agents","AI agents get self-updating memory: 94% on recall benchmark","Memory that evolves: LLM agents self-optimize recall","MindMemOS: self-evolving memory boosts AI agent recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The LOCOMO superiority claim depends on the baseline numbers taken from the EverOS paper being produced by the same protocol, prompt templates, judge model, answer model, and question subset; if they are not, the 0.98-point lead could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Self-evolving memory: agents rewrite their own recall schemas","MindMemOS: memory layer evolves itself for AI agents","AI agents get self-updating memory: 94% on recall benchmark","Memory that evolves: LLM agents self-optimize recall","MindMemOS: self-evolving memory boosts AI agent recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":3075,"prompt_tokens":997,"completion_tokens":2078,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1993}},"tokens_in":613,"tokens_out":2078,"duration_ms":15209,"temperature":1.0,"reasoning_tokens":1993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:21:30.680786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four LOCOMO reasoning types with EverOS, Mem0, memU, Zep, and MemOS inside the authors' own evaluation harness, using identical prompt templates and judge model on the same question subset. If 94.03% no longer beats 93.05% by a stable margin, the claimed superiority is an evaluation artifact; if it persists, the comparison stands.","supporting_citations":[{"cited_title":"Taylor, and Dan Roth","cited_arxiv_id":null,"evidence_quote":"It supplies the PersonaMem benchmark, with 20 personas and seven query types, used to test personalization and preference tracking."},{"cited_title":"Evaluating Memory in LLM Agents via In- cremental Multi-Turn Interactions","cited_arxiv_id":null,"evidence_quote":"It supplies the MemoryAgentBench FactConsolidation subset and protocol used to evaluate dreaming's conflict resolution."}],"review_version":1}