{"id":"1a99bb26-5cc3-4581-a8c5-42f30bde97c2","arxiv_id":"2505.02099","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MemEngine is a modular library that standardizes and reimplements nine memory models from LLM-agent research under a unified function-operation-model hierarchy.","lead":"This paper presents MemEngine, a Python library that unifies memory implementations for LLM-based agents under a three-level modular framework. It reimplements nine published memory models and offers plug-in use across agent frameworks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'first library' claim rests on the nine reimplementations being faithful, yet Section 3.2's unspecified 'reasonable adaptations' and the absence of tests or comparisons against original implementations leave that assumption unverified.","rationale":"The reader's weakest assumption is exactly the fidelity of reimplementations, and I agree. The stress-test pass did not find a separate, more fundamental flaw: no experimental claims are made, and a systems paper can legitimately be evaluated by the artifact. However, the artifact's centrality makes the missing verification the load-bearing concern. The concrete test above would settle it by checking whether the code preserves the defining mechanisms of at least the two most complex models. If it fails, the verdict should remain CONDITIONAL or drop; if it passes, the central claim is supported. I do not recommend REJECT because the repo and docs are linked and the architecture may well be sound; I also do not recommend ACCEPT because the paper alone does not demonstrate the claim. Therefore verdict_should_be is UNCHANGED, meaning the reader's CONDITIONAL remains appropriate.","tokens_in":6300,"tokens_out":3940,"duration_ms":55074,"concrete_test":"Clone https://github.com/nuster1128/MemEngine and run a targeted fidelity probe: instantiate GAMemory and MBMemory with a fixed LLM and embeddings stub, feed a scripted 20-observation history, and inspect the recall-ranked output and the manage/optimize traces. Compare these against the algorithms in the cited papers (Park et al. 2023; Zhong et al. 2024). Concretely, assert that GAMemory's retrieval score combines recency, importance, and relevance and that reflection is scheduled, and that MBMemory performs dynamic summarization and applies its forgetting trigger based on time. If either model reduces to naive retrieval or concatenation, the implementation claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 1) is that MemEngine 'implements a wide variety of memory models from research works' and is 'the first library' to do so under a unified modular framework. This requires that FUMemory, LTMemory, STMemory, GAMemory, MBMemory, SCMemory, MGMemory, RFMemory, and MTMemory are genuine implementations of the cited models, not loose analogues. Section 3.2 says only that 'we make some reasonable adaptations in their implementations' and defers all details to documentation and source code. No unit tests, no side-by-side comparison with the original papers or code, and no demonstration that core mechanisms survive are included. For example, GAMemory is defined in the original work by weighted retrieval combining recency, importance, and relevance plus self-reflection; MBMemory by dynamic summarization with an explicit forgetting trigger; MGMemory by OS-style paging between memory tiers. If any of these mechanisms is replaced by a generic recall/store wrapper, then the advertised 'abundant memory implementation' is not what is claimed. This is not a disagreement with consensus; it is an evidentiary gap over the artifact itself. Because the paper makes no quantitative claims, the only thing that can carry the central claim is the code, and the paper gives the reader no way to confirm the code does what the text says.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MemEngine, a Python library that aims to provide a unified, modular framework for memory components in LLM-based agents. The library organizes memory into three levels (functions, operations, and models) and claims to implement nine memory models from prior research, including Generative Agents, MemoryBank, SCM, MemGPT, Reflexion, and MemTree. The paper describes the architecture, compares MemEngine with existing agent-memory libraries, and illustrates usage modes (default, configurable, automatic) and deployment options (local and remote). No experimental evaluation, unit tests, or comparisons with original implementations are included. The central claim, stated in Section 1, is that MemEngine is the first library to implement a wide variety of research memory models under a unified and modular framework.","tokens_in":6638,"tokens_out":3675,"duration_ms":48259,"significance":"If the implementation is faithful to the cited memory models, MemEngine could be a useful resource for the LLM-agent community, lowering the cost of experimenting with different memory mechanisms. The paper's strengths are its concrete artifact (public repository and documentation), its modular three-level design, and its comparison table covering both agent libraries and standalone memory libraries. These strengths do not, however, substitute for validation: the central claim that MemEngine 'implements' the listed research models is only as strong as the fidelity of the reimplementations, and the manuscript provides no evidence of that fidelity. The modular abstraction itself is sensible, and the paper is clearly written, but the absence of tests or behavioral comparisons leaves the core claim unverified.","major_comments":[{"comment":"The paper's central claim is that MemEngine implements a wide variety of memory models from research works, but the only statement about implementation fidelity is that 'we make some reasonable adaptations in their implementations' (Section 3.2), with details deferred to documentation and source code. The nine models include nontrivial mechanisms from the original papers, such as GAMemory's weighted retrieval with self-reflection, MBMemory's dynamic summarization and forgetting trigger, and MGMemory's operating-system-style paging between memory tiers. Without unit tests, behavioral comparisons against the original implementations, or a precise specification of the adaptations, the reader cannot verify that MemEngine actually reproduces these mechanisms rather than providing generic store-and-recall wrappers. This is a load-bearing gap for the central claim and should be addressed before publication, for example by adding a validation section or an appendix that documents each adaptation and shows that the core behavior of each model is preserved.","section":"Section 3.2"},{"comment":"The claim that MemEngine is 'the first library that implements a wide variety of memory models from research works under a unified and modular framework' is a strong negative claim that is not established by the comparison in Section 2 and Table 1. Table 1 uses subjective categories such as 'Comprehensive Default Models' and 'Advanced Model Customization' without definitions or external evidence, and it does not rule out other libraries (e.g., later versions of Zep or MemoryScope) from satisfying these criteria. The authors should either temper the 'first' claim, define the criteria precisely, or provide reproducible evidence that competing libraries lack the stated capabilities.","section":"Section 1 and Section 2"},{"comment":"The 'automatic mode' is described as automatically selecting memory models, hyper-parameters, and prompts based on 'a specific task's criteria,' but no algorithm, search procedure, or evaluation metric is specified anywhere in the paper. Since automatic selection is one of the library's advertised user-facing features, and since no experiments demonstrate that the automatic selector produces sensible choices, this part of the 'user-friendly' claim is unsupported. The authors should either describe the selection mechanism or remove this claim from the list of supported features.","section":"Section 4.1"}],"minor_comments":[{"comment":"MemoryScope and Zep appear in Table 1 but are not introduced in the text; the list of independent memory libraries in Section 2 mentions only Memary, Cognee, Mem0, and Agentmemory. Please add a sentence describing MemoryScope and Zep for consistency.","section":"Section 2"},{"comment":"FUMemory, LTMemory, and STMemory are listed among the implemented memory models, but unlike the other six models, they are not associated with any cited research work. If they are baselines rather than research models, this should be stated explicitly.","section":"Section 3.2"},{"comment":"The phrase 'statistic manners' should be 'static manners'.","section":"Section 3.5"},{"comment":"The tool name 'vllm' should be capitalized as 'vLLM'.","section":"Section 4.1"},{"comment":"The qualifier 'first' appears twice without a scope or a reference to a systematic search; consider adding 'to our knowledge' and citing the sources for the comparison.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"This is a short system-demo paper, and its contribution is an artifact rather than a scientific result. The main concern is evidentiary: the library's central promise of faithful reimplementations of nine research models is not supported by any tests or comparisons. This is fixable within the paper's scope if the authors add a validation subsection or, failing that, narrow the claims. I recommend major revision rather than rejection because the modular design is plausible and the community would benefit from a verified tool."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is a short systems paper for MemEngine, a Python library that unifies memory implementations for LLM agents under a three-level abstraction: functions, operations, and models. It ships nine memory models from the literature (Generative Agents, MemoryBank, MemGPT, Reflexion, etc.), plus configuration, utilities, local/remote deployment, and AutoGPT compatibility. The paper is honest about being an engineering contribution: no quantitative claims, no experiments, just the artifact and its design. The comparison table against existing agent libraries and independent memory libraries is genuinely useful for practitioners trying to choose a memory layer.\n\nThe three-level abstraction is the main intellectual contribution, and it is well explained. Lower-level functions (encoder, retrieval, reflector, summarizer, judge, LLM interface) compose into operations (store, recall, manage, optimize), which compose into complete models. That is a clean decomposition, and it makes the library's extensibility plausible. The documentation and repo are linked, which is more than many systems papers do.\n\nSoft spots, in proportion. The central claim — 'first library that implements a wide variety of memory models under a unified framework' — rests on the nine reimplementations being faithful. Section 3.2 says only 'reasonable adaptations' and defers to docs. No unit tests, no side-by-side comparisons with original code or papers, no demonstration that core mechanisms survive. If GAMemory's weighted retrieval with recency/importance/relevance plus self-reflection is actually implemented, fine; but if it has been replaced by a generic store/recall wrapper, the claim weakens. This is the evidentiary gap. It is not a fatal flaw for a systems paper — the code is the evidence — but the paper alone does not establish fidelity. Minor: 'first library' is a hard claim to prove, and the self-citations in the intro ([11], [13]) are not load-bearing, so no circularity concern.\n\nThe paper is for practitioners building LLM agents who want pluggable memory without reimplementing ten models. A serious referee should check the repo, probe one or two model implementations against the original papers, and ask the authors to add a small verification table (which original component maps to which operation/function).\n\nMy recommendation: send it to peer review. It is a legitimate engineering contribution with a public artifact, and the design is clean. I would accept with minor revisions conditional on the authors either providing a reproducibility appendix or explicitly narrowing the fidelity claim. If they cannot show that the models actually match their sources, the 'abundant memory implementation' bullet should be softened.","headline":"A tidy, honest systems paper for a memory library; the real gap is verification that the nine reimplemented models actually behave like the originals.","tokens_in":7058,"tokens_out":1357,"would_cite":false,"duration_ms":19920,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces MemEngine, a library that reimplements nine published memory models for LLM-based agents under one modular, pluggable framework.","keywords":["LLM-based agents","agent memory","memory models","modular library","unified framework","memory operations","memory functions","MemEngine"],"falsifier":"Run the original implementation of any one of the nine models and MemEngine's reimplementation on identical task traces and check whether stored, recalled, and optimized memory contents match; any divergence beyond documented configuration differences would show that the 'reasonable adaptations' are not faithful.","tokens_in":6080,"feed_emoji":"🧠","tokens_out":4667,"duration_ms":50931,"temperature":0.7,"pith_summary":"The paper introduces MemEngine, a library that reimplements nine published memory models for large-language-model agents inside one framework, and claims this is the first such unified implementation. The point is to let developers switch memory models, tune their prompts and hyper-parameters, and build new models from shared building blocks without rewriting agent code. A sympathetic reader would treat the contribution as an engineering standardization: instead of each research model living in its own non-pluggable pipeline, memory becomes a modular component with common interfaces.","feed_headline":"One library, nine memory models for LLM agents","feed_subtitle":"MemEngine unifies research memory models behind common interfaces so developers can swap, tune, and build new ones.","key_machinery":"The central object is the three-level hierarchy of memory functions, memory operations, and memory models. Each level is modular and reuses lower levels: functions are the atoms, operations are the reactions, and models are the molecules. A configuration module lets users adjust hyper-parameters and prompts at any level, and a utility module handles storage, visualization, remote clients, and automatic model selection. This hierarchy is what turns memory from a fixed component into a pluggable, customizable one.","core_discovery":"MemEngine organizes memory into three hierarchical levels. The lowest level, memory functions, provides primitive abilities such as encoding, retrieval, reflection, summarization, triggering, utilization, forgetting, truncation, judging, and LLM calls. The middle level, memory operations, combines functions into pipelines such as store, recall, manage, and optimize. The top level, memory models, assembles operations to reproduce the behavior of nine research models: full-context memory, long-term and short-term memory, generative-agents-style weighted retrieval with self-reflection, memory-bank-style summarization with forgetting, self-controlled recall, operating-system-style hierarchical memory, trajectory-optimization memory, and tree-structured memory. The paper states that all models expose the same interfaces and can be switched, configured, and deployed locally or remotely.","pith_inferences":["If the reimplementations are faithful, MemEngine effectively turns memory-model selection into an empirical hyper-parameter choice, which could accelerate ablation studies across a large class of agent memory designs.","Because the paper reports no evaluation of the reimplementations, the library's scientific value depends on a validation suite comparing each model to its original; that suite is a natural immediate addition.","The function-level modularity points toward multimodal memory: new encoders, retrievers, and storage backends could be added without changing the operation and model layers.","Standardizing nine models under one interface may reveal which memory operations contribute most to agent performance, since all other components can be held fixed."],"forward_implications":["Researchers can swap one memory model for another in an agent with minimal code changes, making model comparisons a configuration change.","New memory models can be assembled by reusing existing operations and functions, reducing duplicated implementation of retrieval, summarization, and reflection.","The same memory service can be deployed locally or on a server and accessed remotely, allowing lightweight devices to offload memory.","The library's compatibility layer means agents built in other frameworks can adopt MemEngine's memory without rewriting their own memory logic."],"supporting_citations":[{"why":"Supplies the generative-agents memory model with weighted retrieval and self-reflection that GAMemory reimplements.","marker":"[7]"},{"why":"Supplies the long-term memory model with dynamic summarization and forgetting that MBMemory reimplements.","marker":"[14]"},{"why":"Supplies the hierarchical operating-system-style memory model that MGMemory reimplements.","marker":"[6]"},{"why":"Supplies the trajectory-optimization memory model that RFMemory reimplements.","marker":"[9]"},{"why":"Supplies the tree-structured semantic memory model that MTMemory reimplements.","marker":"[8]"},{"why":"Supplies the self-controlled retrieval model that SCMemory reimplements.","marker":"[10]"},{"why":"Provides the text-embedding model used by the encoder memory function for retrieval.","marker":"[12]"},{"why":"Establishes the survey of memory mechanisms that motivates the library's component taxonomy.","marker":"[13]"}],"fun_headline_variants":["MemEngine unifies nine memory models for LLM agents","One library, nine agent memory models","MemEngine: modular memory for LLM agents","Hierarchical memory library for LLM agents","Swap, tune, and build agent memory with MemEngine"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the nine reimplementations, which the paper says involve 'reasonable adaptations', genuinely reproduce the original models' behavior, even though no evaluation, unit tests, or comparisons are reported.","fun_headline_variants_meta":{"raw":{"variants":["MemEngine unifies nine memory models for LLM agents","One library, nine agent memory models","MemEngine: modular memory for LLM agents","Hierarchical memory library for LLM agents","Swap, tune, and build agent memory with MemEngine"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2529,"prompt_tokens":830,"completion_tokens":1699,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":1627}},"tokens_in":446,"tokens_out":1699,"duration_ms":12661,"temperature":1.0,"reasoning_tokens":1627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T01:01:05.883507+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the original implementation of any one of the nine models and MemEngine's reimplementation on identical task traces and check whether stored, recalled, and optimized memory contents match; any divergence beyond documented configuration differences would show that the 'reasonable adaptations' are not faithful.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the trajectory-optimization memory model that RFMemory reimplements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the generative-agents memory model with weighted retrieval and self-reflection that GAMemory reimplements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the long-term memory model with dynamic summarization and forgetting that MBMemory reimplements."}],"review_version":1}