{"id":"a0b02e59-3961-442d-b32a-9b0cafe88194","arxiv_id":"2506.07398","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"G-Memory stores past multi-agent teamwork in a three-tier graph and retrieves it to boost performance on five benchmarks.","lead":"This paper introduces G-Memory, a memory system that lets teams of AI agents learn from past interactions by storing them in a three-layer graph of insights, queries, and dialogue. It reports performance boosts of up to 20.89% on embodied tasks and 10.12% on knowledge questions across five benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains lack error bars and the hyperparameters k/hop are tuned on the same benchmarks whose headline numbers are reported; without multi-seed evaluation or a held-out split, the 20.89% claim cannot be distinguished from noise.","rationale":"The paper has real strengths: a clear architecture, a plausible mechanism, extensive coverage (5 benchmarks × 3 backbones × 3 frameworks), ablations, cost analysis, and released code. I read it in good faith, and I do not think the work is unsound in its design. The load-bearing weakness is at the evaluation layer. The reader's conditional verdict identified a possible train/test leak; I partially agree, but I think the more decisive issue is statistical grounding: single point estimates without seeds or intervals, combined with hyperparameter selection on the same test benchmarks, make the headline gains unverifiable. The memory-accumulation protocol (Figures 6) compounds this by averaging cold-start and warmed-up trials, so the 'generalizable' part of the central claim is not yet demonstrated. These concerns are checkable and the released code makes the test feasible. They do not overturn the reader's CONDITIONAL verdict; they substantiate it.","tokens_in":25305,"tokens_out":7865,"duration_ms":107855,"concrete_test":"Using the released code, re-run the headline conditions (MacNet+ALFWorld+Qwen-2.5-14b with the claimed 20.89% gain, and AutoGen+HotpotQA with gpt-4o-mini) for at least 5 seeds and report mean ± 95% CI for G-Memory and the best baseline. Also split each benchmark into a tuning subset (20%) and a held-out evaluation subset; select k, hop, and M only on the tuning subset, then report held-out performance. If the CI for the G-Memory minus best-baseline difference includes zero, or the held-out gain is substantially lower than the within-distribution gain, the current headline should be reframed as a promising but unverified test-time adaptation result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"G-Memory's central contribution is an empirical performance claim: up to 20.89% better embodied action and 10.12% better knowledge QA across five benchmarks, three backbones, and three frameworks. For that claim to hold, the reported point estimates must be reproducible and not a by-product of selecting configuration on the test set. The paper reports only single point estimates in Tables 1-3; it never states the number of runs, random seeds, temperature, or confidence intervals. This matters because both the LLM decodes and MacNet's random graph topology are stochastic, so single-run differences of several points are plausible noise. Section 5.4 further selects the hop count and retrieval size k (Equations 4-5) by inspecting performance on the same five benchmarks that are then reported in the main tables, i.e., the reported numbers are partly optimized on the test benchmarks. Finally, memory is accumulated during evaluation (Figures 6), so the averages mix cold-start and memory-warmed trials; the abstract's wording 'generalizable insights' is not separately supported by a held-out evaluation. The architecture itself is coherent and the ablations are informative, but the empirical basis for the headline gain is weaker than the prose suggests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces G-Memory, a hierarchical three-tier graph memory system for LLM-based multi-agent systems (MAS), comprising an insight graph, a query graph, and an interaction graph. The method performs coarse-grained retrieval over the query graph, bi-directional traversal to retrieve high-level insights and condensed interaction trajectories, and then updates all three levels after task execution. The authors claim that G-Memory is plug-and-play and improves embodied action success by up to 20.89% and knowledge QA accuracy by up to 10.12% across five benchmarks, three LLM backbones, and three MAS frameworks. The evaluation is currently reported as single point estimates without error bars, with hyperparameters selected by inspecting the same benchmarks whose results are reported, and without an explicit train/test split in the memory-accumulation protocol.","tokens_in":25531,"tokens_out":4779,"duration_ms":56592,"significance":"If the empirical claims withstand scrutiny, G-Memory addresses a real and currently underserved need: most existing MAS memory mechanisms are limited to inside-trial memory or coarse final artifacts, whereas G-Memory offers role-specific, multi-granularity memory with a principled graph-based organization. The architecture is clear and coherent, the ablations in Figure 4c show that both the insight and interaction components contribute, and the paper covers a wide range of benchmarks, backbones, and frameworks. The public code availability is a further strength. However, the central contribution is empirical, and the current evidence is not yet sufficient to support the headline numbers: there are no variance estimates, hyperparameters are tuned on the same test benchmarks, and the evaluation does not clarify whether memory is built from queries disjoint from those used for final evaluation. The use of LLMs to generate insights is not circular because performance is measured on external benchmarks, but the statistical and protocol issues are load-bearing for the paper's main claim.","major_comments":[{"comment":"The hyperparameters k and hop are selected by observing performance on the same five benchmarks whose results are then reported as the headline numbers. Section 5.4 states that 'we employ 1-hop expansion and k∈{1,2} throughout the experiments' after Figure 4 shows performance on ALFWorld, PDDL, and FEVER for different hop counts and k values. This is a form of test-set tuning: the reported 20.89% and 10.12% gains reflect configurations chosen on the benchmark test sets. Please provide an independent validation split for hyperparameter selection, report results for all configurations, or otherwise justify that the choice does not inflate the reported gains.","section":"§5.1, §5.4, Tables 1–3"},{"comment":"The experiments are reported as single point estimates with no number of runs, random seeds, sampling temperatures, or confidence intervals. Both LLM decoding and MacNet's random graph topology are stochastic, so differences of several percentage points between rows in Tables 1–3 may be consistent with sampling noise. For example, Table 3 shows Voyager on DyLAN/HotpotQA at 30.71 versus no-memory at 30.61, a 0.10-point difference, while other comparisons range from a few points to more than twenty points. Without multiple seeds and variance estimates, the reader cannot determine which improvements are meaningful. Please report means and standard errors over repeated runs, or an equivalent significance analysis.","section":"§5.1, §A, Tables 1–3"},{"comment":"The evaluation protocol never specifies a train/test split. Figures 6a–c show success curves over trials in which memory is accumulated during the evaluation run, and Section A does not state whether the queries used to build memory are disjoint from the queries whose success is reported. If memory is built from the same distribution being tested, the observed gains could reflect within-distribution memorization rather than cross-trial generalization, which is what the abstract's 'generalizable insights' claims. Please clarify the split between memory-building queries and evaluation queries, or add a held-out evaluation where memory is constructed only on a training set.","section":"§5, §B.1, Figures 6a–c"}],"minor_comments":[{"comment":"The heading 'Inisght Summarization Function' contains a typo; it should be 'Insight Summarization Function'.","section":"Appendix C"},{"comment":"The set QR is used in Eq. (9) but is not explicitly named when the top-M relevant queries are introduced in Eq. (7); please define QR there or restate it in the text around Eq. (9).","section":"Eq. (9)"},{"comment":"The parameter M in Eq. (7) is listed as being chosen from {2,3,4,5} in Section 5.1, but Section 5.4 reports sensitivity analysis only for hop and k, not for M. Please state how M was set and whether the results are robust to that choice.","section":"§5.4"},{"comment":"The text says 'the optimal k is among {1,2}' and later 'we employ 1-hop expansion and k∈{1,2} throughout the experiments,' which is ambiguous about whether k=1 or k=2 is selected per benchmark, per framework, or fixed globally. Please state the exact setting for each reported cell.","section":"§5.4"},{"comment":"The phrase 'we are alarmed to discover' is editorializing and out of place in a formal research paper; please rephrase.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The architecture is well-motivated and the paper is likely to be a useful contribution to the MAS memory literature once the evaluation protocol is strengthened. The main risk is that the headline improvements may be partly an artifact of test-set hyperparameter selection and single-run reporting. I would ask the authors for multi-seed results, a clear held-out split, and an independent validation procedure before recommending acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on arXiv:2506.07398. The core idea is a three-tier graph memory—insight, query, interaction—for LLM-based multi-agent systems. That's genuinely new for MAS: prior memory designs like MetaGPT-M, ChatDev-M, and MacNet-M only store coarse artifacts like final solutions. The paper also does a good job identifying the bottleneck: MAS memory is oversimplified and doesn't capture collaboration trajectories. The architecture is coherent, and the ablations (insights vs. interactions) are informative. I also appreciate the token-cost analysis, which shows the memory doesn't blow up the budget.\n\nThe evaluation is the weak spot. The headline claims—up to 20.89% on embodied action and 10.12% on knowledge QA—are based on single point estimates. There are no error bars, no significance tests, no number of runs. Both LLM decodes and MacNet's random graph topology are stochastic, so a few percentage points could easily be noise. Section 5.4 selects the hop count and retrieval size k on the same five benchmarks that are later reported in the main tables. That's test-set tuning, even if coarse. And the protocol never states whether the memory accumulated during evaluation crosses a train/test boundary; the learning curves in Figure 6 suggest memory is built from the same distribution being tested. So the 'generalizable insights' claim isn't supported by a held-out evaluation.\n\nThis doesn't mean the paper is wrong. The improvements are large and consistent in direction across many settings, and the architecture is plausible. It means the empirical case needs more rigor before I'd fully trust the numbers. The authors need multi-seed runs, confidence intervals, and an explicit data split for memory accumulation.\n\nWho's this for? Anyone working on LLM agent memory or self-evolving MAS. The three-tier structure is a reusable idea, and the code is available. I'd send it to peer review, but for a major revision that addresses the statistical rigor. Worth bringing to our reading group, too.","headline":"A well-motivated hierarchical memory architecture for MAS with a broad evaluation, but the empirical claims are undermined by missing error bars and test-set hyperparameter selection.","tokens_in":26075,"tokens_out":3604,"would_cite":true,"duration_ms":41492,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-tier graph memory lets LLM-based multi-agent teams learn from past collaborations, lifting success by up to 20.89% on embodied tasks and 10.12% on knowledge QA without touching the base framework.","keywords":["multi-agent systems","LLM agents","hierarchical memory","graph-based memory","self-evolving agents","retrieval-augmented generation","embodied action","knowledge question answering"],"falsifier":"Run the identical comparison with a held-out test set: build all G-Memory graphs using only queries from a training split, then evaluate on a disjoint split (or on a second distribution entirely). If the improvements over the no-memory baseline shrink to near zero or reverse, the central claim of cross-trial generalization fails. A cheaper check is to clear memory between trials and confirm success rates fall back to baseline.","tokens_in":25107,"feed_emoji":"🧠","tokens_out":8549,"duration_ms":89101,"temperature":0.7,"pith_summary":"The paper argues that LLM-powered multi-agent systems fail to improve across tasks because their memory mechanisms are too simple: they either keep context only inside one trial or pass along only final answers, discarding the inter-agent dialogue that carries the reasoning. To fix this, it proposes G-Memory, a plug-and-play module that organizes a team's history into a three-tier graph hierarchy—high-level insights, a query graph linking related past tasks, and fine-grained interaction logs—and retrieves from all three levels before a new query. This is why it would matter: if the claim holds, any existing multi-agent system can gain self-evolution and better collaboration simply by adding a memory layer, with no changes to its orchestration. The paper's evidence is that G-Memory improves success rates on embodied action and accuracy on knowledge QA by up to 20.89% and 10.12% respectively across five benchmarks, three LLM backbones, and three multi-agent frameworks.","feed_headline":"Three-tier memory lifts multi-agent scores by up to 21%","feed_subtitle":"A plug-in memory of insights, queries, and interaction logs improves agent teams without altering the base system.","key_machinery":"The load-bearing mechanism is the three-tier graph hierarchy with bi-directional traversal. Coarse similarity search over the query graph returns candidate past tasks; a one-hop expansion widens the candidate set; an upward traversal from query to insight graph collects high-level lessons, and a downward traversal uses an LLM-facilitated sparsifier to condense each relevant interaction graph into the dialogue segments that matter for the current task. An operator then personalizes these memories for each agent role, and after execution the hierarchy is updated: new interaction graphs are stored, new query nodes are linked to the relevant past queries and to the queries supporting the used insights, and new or merged insights are distilled from contrasting successful and failed trajectories. The hierarchy is what lets the system trade off abstraction and detail: general lessons guide strategy, while condensed trajectories provide concrete corrective steps.","core_discovery":"The central claim is that the bottleneck in multi-agent self-evolution is memory, not orchestration. G-Memory counters the simplistic designs it surveys with a three-tier graph: the insight graph holds distilled, generalizable lessons together with the queries that support them; the query graph stores past queries, their success or failure status, and a semantic network connecting them; the interaction graph records every agent utterance and the influence edges that trace how one utterance inspired another. For a new query, G-Memory retrieves similar past queries, expands to their one-hop neighbors, then traverses upward to pull relevant insights and downward to condense the most useful interaction histories into core subgraphs. Each agent receives memory filtered to its role before the team acts, and after execution the query and insight graphs are extended with the new experience. The paper reports that this cycle consistently improves performance across three task domains, with the largest gains on embodied action tasks.","pith_inferences":["A decisive untested question is generalization beyond the evaluation distribution: the paper does not report a train/test split, and its success curves show memory accumulating during the test run; a held-out evaluation would separate cross-trial learning from memorization.","The insight graph's cross-task edges hint at a transfer extension the paper does not run: checking whether lessons learned in one domain (such as verifying web-search facts) improve performance in another (such as embodied object placement).","The query graph could support richer retrieval keys than semantic similarity, such as failure type or agent role; the paper only explores embedding similarity plus one-hop expansion.","Because insight distillation and trajectory sparsification are done by LLMs, the quality of the memory likely depends on the backbone's summarization reliability; the paper does not isolate this effect."],"forward_implications":["G-Memory can be retrofitted onto any MAS that accepts extra context: the reported gains are obtained without modifying the frameworks themselves.","Memory should be role-specific: giving every agent the same retrieved history underperforms, suggesting that agents with different functions need different cues.","High-level insights and fine-grained interaction traces are complementary; removing either one costs roughly 3 to 5 percentage points on the tested frameworks.","Teams improve with exposure: on embodied benchmarks, G-Memory reaches higher success ceilings and succeeds with fewer trials as memory accumulates.","The performance gain is not bought by excessive prompting: token consumption stays in line with cheaper baselines."],"supporting_citations":[{"why":"Provides the organizational-memory theory (schemas before details) that motivates the three-tier hierarchy and its retrieval order.","marker":"[1]"},{"why":"One of the three multi-agent frameworks used in the main evaluation; G-Memory is plugged into it without modification.","marker":"[13]"},{"why":"A second multi-agent framework in the experiments, used with three LLM backbones to show the module's generality.","marker":"[72]"},{"why":"A third framework and also a baseline whose memory stores only round-final answers, illustrating the limitation G-Memory addresses.","marker":"[47]"},{"why":"Supplies a single-agent cross-trial memory baseline that the paper adapts to agent-specific retrieval for comparison.","marker":"[16]"},{"why":"Supplies a retrieval-based single-agent memory baseline used as a point of comparison.","marker":"[36]"},{"why":"Supplies the reflective-memory baseline combining raw observations with high-level reflections.","marker":"[19]"},{"why":"Supplies the MetaGPT-style inside-trial memory baseline that stores collaboration artifacts only within a task.","marker":"[21]"},{"why":"Supplies the ChatDev-style cross-trial baseline that compresses prior experience into final solutions.","marker":"[46]"},{"why":"The MiniLM embedding model used for coarse-grained query retrieval in the memory lookup.","marker":"[81]"}],"fun_headline_variants":["Plug-in memory boosts multi-agent success by 21%","G-Memory: hierarchy that helps agent teams learn","Three-tier memory elevates agent teamwork up to 21%","Memory not orchestration: G-Memory lifts agents","Agent teams evolve faster with hierarchical memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume that the memory accumulated during earlier evaluation trials does not leak information about later test queries; the experimental setup does not state a train/test split, so the improvements could reflect within-distribution memorization rather than cross-trial generalization.","fun_headline_variants_meta":{"raw":{"variants":["Plug-in memory boosts multi-agent success by 21%","G-Memory: hierarchy that helps agent teams learn","Three-tier memory elevates agent teamwork up to 21%","Memory not orchestration: G-Memory lifts agents","Agent teams evolve faster with hierarchical memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1312,"prompt_tokens":1002,"completion_tokens":310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":234}},"tokens_in":618,"tokens_out":310,"duration_ms":3686,"temperature":1.0,"reasoning_tokens":234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:34:33.219879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical comparison with a held-out test set: build all G-Memory graphs using only queries from a training split, then evaluate on a disjoint split (or on a second distribution entirely). If the improvements over the no-memory baseline shrink to near zero or reverse, the central claim of cross-trial generalization fails. A cheaper check is to clear memory between trials and confirm success rates fall back to baseline.","supporting_citations":[{"cited_title":"Autogen: Enabling next-gen llm applications via multi-agent conversation framework, August 01, 2023 2023","cited_arxiv_id":null,"evidence_quote":"One of the three multi-agent frameworks used in the main evaluation; G-Memory is plugged into it without modification."},{"cited_title":"O’Brien, Carrie J","cited_arxiv_id":null,"evidence_quote":"Supplies the reflective-memory baseline combining raw observations with high-level reflections."},{"cited_title":"Metagpt: Meta programming for multi-agent collaborative framework, August 01, 2023 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the MetaGPT-style inside-trial memory baseline that stores collaboration artifacts only within a task."},{"cited_title":"Communicative agents for software development, July 01, 2023 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the ChatDev-style cross-trial baseline that compresses prior experience into final solutions."},{"cited_title":"You are a smart agent designed to solve problems","cited_arxiv_id":null,"evidence_quote":"The MiniLM embedding model used for coarse-grained query retrieval in the memory lookup."}],"review_version":1}