{"id":"25e6ff17-4d7c-48ec-a2ab-aa20be98860e","arxiv_id":"2504.19413","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Mem0 improves long-term LLM conversational performance by up to 26% on LLM-as-Judge while cutting p95 latency 91% and token costs over 90% versus full-context baselines.","lead":"Mem0 introduces a memory architecture for LLMs that extracts, consolidates, and retrieves key conversation details dynamically, with a graph-based variant for relations. This matters for building AI agents that stay consistent across long sessions without high compute costs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Dynamic extraction may omit facts critical for multi-hop/temporal reasoning on LOCOMO","rationale":"The reader's weakest assumption is exactly the load-bearing point. Full-text access does not add the missing extraction audit or error breakdown, so the empirical superiority remains conditional on unverified preservation of context. Efficiency numbers are less contested because they follow directly from shorter retrieved context.","tokens_in":1843,"tokens_out":325,"duration_ms":15398,"concrete_test":"Sample 100 LOCOMO conversations; for each multi-hop or temporal question, extract the minimal set of facts required to answer it, then check whether Mem0's stored memories contain all of them (exact or paraphrased). Report the fraction of questions where at least one required fact is missing; if >15% of such questions lose a fact, re-run the LLM-as-Judge evaluation on only the subset that lost no facts.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline accuracy claim (26% relative LLM-as-Judge gain, graph variant +2%) rests on the extraction/consolidation step faithfully preserving every fact needed for the four question categories. If the LLM-based extractor drops or mis-summarizes a temporal anchor or cross-turn entity, downstream retrieval cannot recover it. The paper reports aggregate scores but supplies no per-question error analysis, no extraction-precision audit against gold facts, and no ablation that isolates extraction failures from retrieval or graph-construction failures. This makes it impossible to tell whether the observed gains come from better memory or from the baselines simply being weaker at long context.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Mem0, a scalable memory-centric architecture for LLMs that dynamically extracts, consolidates, and retrieves salient information from multi-session conversations, along with a graph-based variant for capturing relational structures. It evaluates both variants on the LOCOMO benchmark against six categories of baselines (memory-augmented systems, RAG variants, full-context, open-source, proprietary, and dedicated platforms), claiming consistent outperformance across single-hop, temporal, multi-hop, and open-domain questions, including a 26% relative gain in LLM-as-Judge over OpenAI, ~2% additional gain from the graph variant, 91% lower p95 latency, and >90% token cost savings versus full-context.","tokens_in":1933,"tokens_out":536,"duration_ms":33616,"significance":"If the results hold after addressing the gaps below, this would represent a practical contribution to production-ready long-term memory for AI agents, with notable efficiency advantages over full-context baselines that could enable scalable deployment. The breadth of baseline comparisons across question categories is a strength, though the absence of targeted ablations and error analysis limits the ability to attribute gains specifically to the proposed extraction and graph mechanisms.","major_comments":[{"comment":"Experimental evaluation (Section 4 / LOCOMO results): Aggregate scores are reported for the four question categories and LLM-as-Judge metric, but no per-question error analysis, extraction-precision audit against gold facts, or ablation isolating dynamic extraction/consolidation failures from retrieval/graph issues is provided. This is load-bearing for the central claim, as omissions in temporal anchors or cross-turn entities could explain gains without the memory mechanism itself being superior.","section":"Experimental evaluation (Section 4)"},{"comment":"Methodology and implementation details: The manuscript does not specify data splits for LOCOMO, exact extraction prompts/models, graph construction algorithm, or precise configurations for all six baseline categories (e.g., chunk sizes and k for RAG). Without these, the 26% relative improvement and efficiency metrics cannot be independently verified or reproduced.","section":"Methodology"}],"minor_comments":[{"comment":"The abstract states 'around 2% higher overall score' for the graph variant; the main text should report the exact metric, absolute values, and statistical significance for this comparison.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The work reads primarily as a systems/engineering contribution focused on practical implementation and benchmarking rather than novel algorithmic theory; confirm fit with the target journal's scope before proceeding."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback and for recognizing the practical contributions of Mem0 to scalable long-term memory for AI agents. The comments highlight important areas for improving the strength of our claims and reproducibility. We address each major comment below and have revised the manuscript to incorporate additional analysis and details where feasible.","responses":[{"response":"We agree that aggregate metrics alone make it harder to isolate the contributions of dynamic extraction, consolidation, and graph-based retrieval. In the revised manuscript we will add a dedicated error analysis subsection in Section 4 that provides per-category breakdowns (single-hop, temporal, multi-hop, open-domain) with representative success and failure examples, focusing on cases involving temporal anchors and cross-turn entities. We will also include targeted ablations: (i) Mem0 without dynamic extraction/consolidation, (ii) base Mem0 versus graph variant, and (iii) retrieval-only versus full memory pipeline. These will help attribute gains more precisely to the proposed mechanisms. A full extraction-precision audit against gold facts is not possible because LOCOMO does not provide such annotations; we will instead report precision estimates from manual inspection of a sampled subset of extracted memories and note this as a limitation.","revision_made":"partial","referee_comment":"[Experimental evaluation (Section 4)] Experimental evaluation (Section 4 / LOCOMO results): Aggregate scores are reported for the four question categories and LLM-as-Judge metric, but no per-question error analysis, extraction-precision audit against gold facts, or ablation isolating dynamic extraction/consolidation failures from retrieval/graph issues is provided. This is load-bearing for the central claim, as omissions in temporal anchors or cross-turn entities could explain gains without the memory mechanism itself being superior."},{"response":"We acknowledge that the original manuscript omitted several implementation details necessary for full reproducibility. The revised version will expand the Experimental Setup section with: (1) LOCOMO data usage and any train/test splits applied; (2) the exact extraction and consolidation prompts together with the underlying models (gpt-4o for extraction, gpt-4o-mini for retrieval); (3) the graph construction algorithm, which uses LLM-based entity-relation extraction followed by incremental graph updates; and (4) complete baseline configurations, including chunk sizes (256/512/1024 tokens) and k values (3/5/10) for all RAG variants, as well as the exact settings for the other five baseline categories. These additions will allow independent verification of the reported accuracy gains, 91% p95 latency reduction, and >90% token cost savings.","revision_made":"yes","referee_comment":"[Methodology] Methodology and implementation details: The manuscript does not specify data splits for LOCOMO, exact extraction prompts/models, graph construction algorithm, or precise configurations for all six baseline categories (e.g., chunk sizes and k for RAG). Without these, the 26% relative improvement and efficiency metrics cannot be independently verified or reproduced."}],"tokens_in":1534,"tokens_out":632,"duration_ms":22969,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The key takeaway is that Mem0 provides a scalable way to give LLM agents long-term memory by extracting and consolidating information from dialogues, plus a graph-based version, and it delivers measurable efficiency improvements alongside accuracy gains on the LOCOMO benchmark.","headline":"Mem0 is a practical memory architecture for long-term LLM agents that improves efficiency on LOCOMO but needs better validation of its extraction step.","tokens_in":2425,"tokens_out":120,"would_cite":false,"duration_ms":30665,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Mem0 dynamically extracts and consolidates key facts from conversations to give LLMs reliable long-term memory without processing full histories.","keywords":["long-term memory","LLM agents","memory architecture","conversational AI","graph memory","RAG","scalability","LOC OMO benchmark"],"falsifier":"A new evaluation set of long multi-session dialogues containing explicit temporal chains and multi-hop dependencies where full-context processing scores measurably higher than Mem0 on accuracy metrics.","tokens_in":2732,"feed_emoji":"🧠","tokens_out":670,"duration_ms":27229,"temperature":0.7,"pith_summary":"The paper introduces Mem0 as a memory architecture that pulls salient details from ongoing dialogues, stores them efficiently, and retrieves them as needed for consistent answers across sessions. A sympathetic reader would care because current LLMs struggle with extended interactions, either forgetting earlier context or incurring high costs from retaining everything. The authors evaluate it on the LOCOMO benchmark against six categories of baselines including full-context processing, RAG variants, and other memory systems. Results show higher accuracy on single-hop, temporal, multi-hop, and open-domain questions plus major reductions in latency and token use. A graph-based extension adds relational structure among stored facts for further gains.","feed_headline":"Mem0 raises LLM memory accuracy 26% while cutting latency 91%","feed_subtitle":"Dynamic fact extraction and optional graph relations let agents stay coherent across sessions at far lower cost than full history.","key_machinery":"Mem0's dynamic extraction, consolidation, and retrieval pipeline for salient conversational information, together with its optional graph-based memory representation for relational structures.","core_discovery":"Mem0 is a scalable memory-centric architecture that dynamically extracts, consolidates, and retrieves salient information from ongoing conversations. An enhanced variant uses graph-based representations to capture complex relational structures among conversational elements. On the LOCOMO benchmark it outperforms established memory systems, RAG setups, full-context processing, open-source solutions, proprietary systems, and dedicated memory platforms across single-hop, temporal, multi-hop, and open-domain questions. Mem0 achieves 26% relative improvement in the LLM-as-a-Judge metric over OpenAI, the graph version scores about 2% higher overall, and both deliver 91% lower p95 latency with more","pith_inferences":["If extraction remains reliable at scale, the approach could support agents that maintain coherence across weeks of interaction rather than single sessions.","The relational graph may prove especially useful for tasks that track how facts evolve or connect over time, suggesting targeted tests on longer dependency chains.","Combining this memory layer with other agent components such as planning or tool use could further improve production deployment without proportional cost increases.","The efficiency gains open the possibility of running multiple parallel agents on the same hardware while each retains its own long-term context."],"forward_implications":["Outperforms all tested baselines on single-hop, temporal, multi-hop, and open-domain questions.","Delivers 26% relative gain in LLM-as-a-Judge score over OpenAI memory.","Graph memory variant adds roughly 2% overall score improvement over the base Mem0.","Reduces p95 latency by 91% and token cost by more than 90% versus full-context processing."],"fun_headline_variants":["Mem0 delivers 26% better LLM memory accuracy at 91% lower latency","Mem0 graph memory scores 2% higher than base configuration","Mem0 reduces p95 latency by 91% and token costs over 90%","Scalable Mem0 memory outperforms baselines across dialogue question types"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Extracting and consolidating only the most salient facts from conversations preserves every piece of context required for correct answers to complex multi-hop and temporal questions.","fun_headline_variants_meta":{"raw":{"variants":["Mem0 delivers 26% better LLM memory accuracy at 91% lower latency","Mem0 graph memory scores 2% higher than base configuration","Mem0 reduces p95 latency by 91% and token costs over 90%","Scalable Mem0 memory outperforms baselines across dialogue question types"]},"model":"grok-4.3","cost_usd":0.008295,"raw_usage":{"total_tokens":3750,"prompt_tokens":810,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":82953000,"prompt_tokens_details":{"text_tokens":810,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2864,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":810,"tokens_out":76,"duration_ms":14901,"temperature":1.0,"reasoning_tokens":2864,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-10T23:07:10.637047+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new evaluation set of long multi-session dialogues containing explicit temporal chains and multi-hop dependencies where full-context processing scores measurably higher than Mem0 on accuracy metrics.","supporting_citations":[],"review_version":1}