{"id":"2cca44ab-35b4-406f-a6b8-7020246c01b9","arxiv_id":"2412.18431","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GeAR augments any standard retriever with entity-linked graph expansion and a memory-based agent loop, setting new state-of-the-art retrieval scores on MuSiQue, HotpotQA, and 2WikiMultihopQA.","lead":"GeAR is a retrieval system that first looks up passages, then expands them by hopping between related facts, and lets an AI agent repeat this until a multi-hop question can be answered. The authors report higher retrieval and answer accuracy than previous graph-based retrieval systems on three benchmarks, using fewer AI tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ungrounded proximal triples may let the LLM's parametric memory supply missing hops; with no grounding check reported, the SOTA gain may not come from graph expansion.","rationale":"The reader's conditional verdict flags missing code, missing significance tests, dev-set hyperparameter tuning, and absent baselines such as TRACE and GraphReader; these are valid but secondary. The load-bearing issue is different: the method's retrieval signal is produced by an LLM that is not constrained to the retrieved evidence, and the paper's error analysis only counts hallucinations when they cause failures. Correct hallucinations are invisible in the reported R@k numbers but can inflate them. This directly attacks the central claim that a graph-enhanced agent outperforms previous graph+LLM retrievers: if the LLM is the one supplying the multi-hop fact, the graph and agent become post hoc passage finders rather than the retrieval mechanism. The proposed grounding test is feasible and decisive: filter proximal triples for entailment and see whether the margin survives. I therefore keep the reader's conditional verdict but make the condition explicit: acceptance should require this grounding check or an equivalent analysis. Agreement is partial because the reader identified reader hallucinations as a failure source but not their potential role in successful retrievals.","tokens_in":22426,"tokens_out":8939,"duration_ms":90398,"concrete_test":"Sample 50 MuSiQue test questions. For every proximal triple emitted by the read step at iteration 1 (and every gist-memory triple at later iterations), manually or with an NLI model label whether the triple is entailed by the base-retrieved passages C'_q that the LLM was given. Then rerun GEAR and Hybrid+SyncGE after filtering out all non-entailed triples and recompute R@15. If R@15 drops by more than roughly 5 points on MuSiQue, the reported gains depend on ungrounded (likely parametric) triples; if R@15 is stable, the concern is refuted and the graph-expansion attribution stands.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"GEAR's critical path is the LLM 'read' step (Eq. 1 and Eq. 4, prompts in Appendix K.2): it converts base-retrieved passages C'_q into proximal triples that seed graph expansion and, via gist memory, drive final passageLink retrieval (Eqs. 8-9). Nothing verifies that a proximal triple is entailed by, or even appears in, the passages the LLM was given. Since the benchmarks are Wikipedia-derived and the LLM is run at temperature 0, correct triples can be supplied from parametric memory rather than from the retrieved context. The paper's own negative-instance analysis (Appendix I.2) reports 7/20 failures from reader hallucinations, proving ungrounded triples occur; but it only examines failures, so it cannot measure how often correct-but-ungrounded triples produce successful retrievals. In Table 13, GEAR's memory contains a precise population figure (Venice, 2018, 260,897) that is not shown to be in the retrieved passages; if such triples are common, then the recall gains attributed to SyncGE/graph expansion and to the agent loop actually come from the LLM doing closed-book QA and passageLink merely finding a passage that matches the hallucinated triple. This would not make the system useless, but it would invalidate the claim that graph expansion is what achieves SOTA retrieval, and it would help explain the large margins over HippoRAG w/IRCoT, whose PPR is grounded in the offline triple index rather than in LLM-fabricated query-time triples.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GeAR, a retrieval framework for multi-hop question answering. GeAR has two components: (i) SyncGE, a graph-expansion layer that wraps any base retriever by having an LLM read the base-retrieved passages and emit 'proximal triples', linking those triples to an offline triple index, and then expanding the subgraph via a diverse triple beam search over shared-entity neighborhoods with a dense-embedding cosine scoring function; and (ii) a multi-step agent that maintains a 'gist memory' of proximal triples, uses an LLM reasoner to decide whether enough evidence has been accumulated, rewrites the query when needed, and finally fuses passageLink results with the per-iteration retrieved lists via Reciprocal Rank Fusion. The system is evaluated on MuSiQue, 2WikiMultihopQA, and HotpotQA in terms of retrieval Recall@5/10/15 and end-to-end QA EM/F1, with additional ablations on diversity, beam length, open-weight LLMs, triple-density robustness, and a manual error analysis.","tokens_in":22710,"tokens_out":7657,"duration_ms":68207,"significance":"If the reported results are sound, this is a practically valuable systems contribution: it shows that a retriever-agnostic graph-expansion mechanism combined with a lightweight agent loop can outperform existing graph+LLM retrieval baselines such as HippoRAG w/IRCoT on standard multi-hop benchmarks, often with fewer LLM tokens and iterations. The paper is careful in several respects: all LLM-based baselines are reproduced with the same model (GPT-4o mini) at temperature 0, the triple-extraction prompt is compared against HippoRAG's sequential extraction, hyperparameters are selected on a non-overlapping MuSiQue dev sample, and appendices provide additional experiments with open-weight 7-8B models, diversity ablations, and robustness checks across triple densities. The main risk is attribution: because the read step's proximal triples are not verified to be grounded in the retrieved passages, some of the observed gains could originate from the LLM's parametric memory rather than from the graph-expansion mechanism itself.","major_comments":[{"comment":"The central attribution of GeAR's gains to SyncGE/graph expansion depends on the proximal triples T'_q being grounded in the base-retrieved passages C'_q, but the paper never verifies this. The read step in Eq. (1) asks the LLM to output triples that help answer the question, and the prompt in Appendix K.2 asks for facts 'based on these documents', yet nothing checks whether each proximal triple is mentioned in, or entailed by, C'_q. The paper's own negative-instance analysis (Appendix I.2, Table 14) reports 7/20 failures from reader hallucinations, and Table 13 shows a successful case containing the proximal triple (Venice, population in 2018, 260 897) with no evidence that this fact appeared in the base-retrieved passages. If correct-but-ungrounded triples are common in successful queries, then the recall improvements attributed to graph expansion could instead come from the LLM's closed-book knowledge plus passageLink retrieving passages that match the hallucinated triple. I therefore ask for a quantitative grounding analysis (e.g., estimating how often T'_q triples are entailed by or string-overlap with C'_q), and, if possible, an ablation that forces proximal triples to be drawn from the passage-derived triple index before graph expansion.","section":"§4.1, Eq. (1); Appendix I.2, Table 14; Table 13"},{"comment":"The headline comparisons are reported without confidence intervals, significance tests, or indication of whether the numbers come from a single run. The test sets are small (500 questions each for MuSiQue and 2Wiki, 1,000 for HotpotQA), and on HotpotQA the R@15 margin between GeAR and HippoRAG w/IRCoT is only 1.4 points (97.3 vs 95.9), which could easily fall within sampling noise. Please report bootstrap confidence intervals or a paired significance test for at least the main retrieval comparisons in Table 2, and state explicitly whether the reported values are single runs.","section":"§6, Table 2"}],"minor_comments":[{"comment":"The 2Wiki values for IRCoT (ColBERTv2) are non-monotonic: R@10 is 86.6 while R@15 is 69.7, which is impossible for recall if the retrieval lists are nested; this is presumably a typo and should be corrected.","section":"Table 2, IRCoT (ColBERTv2) row"},{"comment":"The caption says graph expansion enables retrieval 'without using an LLM', but the SyncGE pipeline in §4.1 uses an LLM in the read step (Eq. (1)) to produce proximal triples; only the diverse triple beam search itself is LLM-free.","section":"Table 1 caption"},{"comment":"There are small typos: 'knowlege triples' in the Reader prompt should be 'knowledge triples', and Appendix F contains 'beam sffearch length' instead of 'beam search length'.","section":"Appendix K.2 and Appendix F"},{"comment":"The notation T is used both for the set of triples and for its cardinality in T = {t_1, ..., t_T}; consider writing |T| for the size to avoid ambiguity.","section":"Section 3"},{"comment":"The population value is formatted as '260 897' with a space; using '260,897' or a similar standard separator would improve readability.","section":"Table 13"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical systems paper whose main risk is the attribution of the reported gains to the graph-expansion mechanism rather than to the LLM's parametric memory. I believe this is fixable with a grounding analysis and an explicit discussion of the closed-book component, so I would not reject. The missing confidence intervals are also fixable and matter for the smaller reported margins. I would encourage the editor to ask for these additions rather than treating the current version as final."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. The stress-test note is on target: the paper's headline claim—that graph expansion is what gets SOTA multi-hop retrieval—is not actually supported, because the LLM 'read' step can inject facts from parametric memory, and nothing checks that the proximal triples are entailed by the retrieved passages. The Venice population example in Table 13 is exactly the kind of fact that GPT-4o-mini would know regardless of retrieval. So the credit for the gains may belong to the LLM's closed-book knowledge, with the graph expansion and passageLink just matching the hallucinated triples to the right passages. That said, this is a serious engineering effort, not a toy.\n\nWhat is genuinely new: the diverse triple beam search (Algorithm 1) is a real algorithmic contribution, and the gist-memory agent loop is cleanly described. The ablations—NaiveGE vs SyncGE, diversity on/off, open-weight models—are useful, and the robustness checks across triple density are thoughtful. The paper is honest about its limitations and includes a candid error analysis.\n\nThe soft spots beyond the grounding issue: no code release, no confidence intervals or significance tests, test sets of only 500–1,000 questions, and hyperparameters tuned on a MuSiQue dev sample. TRACE and GraphReader, the closest graph-agent systems, are missing from the baseline table. These are all fixable, but they temper the 'state-of-the-art' framing.\n\nWho is this for? Researchers working on multi-hop retrieval and RAG, especially those building agentic graph systems. The paper is worth reading for the diverse beam search idea, but I would not rely on the headline numbers until the grounding issue is checked. The ablation story does not separate the LLM's memory from the graph expansion, so the central attribution claim is currently under-supported.\n\nRecommendation: yes, send it to peer review—it deserves referee time—but the review should focus on the grounding question and ask for an analysis that separates closed-book LLM knowledge from graph-based multi-hop reasoning. If that cannot be addressed, the paper should be reframed as a hybrid system with a more modest claim.","headline":"Strong engineering with a real algorithmic novelty, but the gains may partly come from the LLM's parametric memory via ungrounded 'proximal triples,' so the graph-expansion claim needs scrutiny.","tokens_in":23379,"tokens_out":2676,"would_cite":false,"duration_ms":25367,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GEAR claims that adding triple-graph expansion and a gist-memory agent loop to any base retriever yields state-of-the-art multi-hop retrieval: R@15 of 71.5 on MuSiQue, 95.9 on 2Wiki, 97.3 on HotpotQA, with fewer LLM tokens and iterations.","keywords":["multi-hop question answering","retrieval-augmented generation","graph-based retrieval","diverse triple beam search","gist memory","Reciprocal Rank Fusion","recall@k"],"falsifier":"Take the set of MuSiQue questions where the gold passages are reachable only through triple chains whose cosine similarity to the question is lower than that of several distractor chains; if SyncGE on this subset shows no meaningful recall gain over the base retriever, the embedding-based beam score is not doing the work the paper attributes to it.","tokens_in":22178,"feed_emoji":"🔎","tokens_out":8990,"duration_ms":70319,"temperature":0.7,"pith_summary":"GEAR is a retrieval system for multi-hop question answering that claims to make any ordinary retriever—sparse, dense, or hybrid—capable of following reasoning chains across passages. It does this in two stages: a graph-expansion step (SyncGE) that uses an LLM only to extract the first set of proximal triples from base-retrieved passages and then explores the triple graph with a diverse beam search scored by cosine similarity, and a multi-step agent that accumulates a gist memory of triples, decides when enough evidence is gathered, and rewrites the query. On MuSiQue, 2WikiMultihopQA, and HotpotQA, GEAR reports state-of-the-art Recall@5/10/15, including a more than 10-point gain over the strongest graph+LLM baseline on MuSiQue at R@15, while consuming fewer tokens and iterations. If correct, this matters because it shifts the expensive part of graph-based retrieval from iterative LLM graph-traversal to a cheap, retriever-agnostic expansion that any RAG pipeline could bolt on.","feed_headline":"Graph expansion plus gist memory tops multi-hop retrieval","feed_subtitle":"GEAR reaches R@15 of 71.5 on MuSiQue, beating graph+LLM baselines with fewer tokens.","key_machinery":"The load-bearing object is the triple-indexed passage graph: every extracted triple is uniquely linked to one source passage, so triples can be traversed and then mapped back to passages. On top of this graph sits Diverse Triple Beam Search (Algorithm 1), which maintains a beam of triple chains, expands each chain only through triples sharing a head or tail entity with the last triple, scores a candidate chain by cosine similarity between the query embedding and the concatenated triple sequence, and applies a position-based diversity weight (multiplying by $e^{-\\min(n,\\gamma)/\\gamma}$) so beams do not collapse onto near-identical chains. The SyncGE module combines this expansion with the LLM's proximal-triple extraction and Reciprocal Rank Fusion of expanded and base passages; the GEAR agent adds gist memory $G^{(n)}$, a concatenated array of proximal triples across iterations, plus an LLM reasoner and query rewriter. The design claim is that the LLM is needed only to locate starting nodes and to summarize evidence, while graph traversal itself is carried by a small semantic model.","core_discovery":"The central claim, stated on the paper's own terms, is that a retriever-agnostic graph expansion synchronized with an LLM's one-shot read of initially retrieved passages, followed by a multi-step agent with gist memory, is enough to achieve state-of-the-art multi-hop retrieval. In SyncGE, the LLM reads the base-retrieved passages and outputs proximal triples; each is linked to its nearest triple in a prebuilt triple index, and these seed a Diverse Triple Beam Search that walks shared-entity neighborhoods, scoring each candidate chain by cosine similarity between the query and the concatenated triple sequence, with a diversity penalty to keep beams distinct. The flattened beam outputs map back to source passages and are fused with the original retrieval by Reciprocal Rank Fusion. The agent component repeats this with a rewritten query, storing proximal triples in a gist memory that conditions subsequent reads, and terminates when the LLM reasoner judges the memory sufficient. The paper's evidence is recall and QA tables showing GEAR ahead of HippoRAG w/ IRCoT and IRCoT variants on all three datasets, with the largest margin on MuSiQue.","pith_inferences":["The same 'locate then semantically expand' split could apply to other aligned index structures, such as passage-to-entity or passage-to-document graphs, where an LLM seeds nodes and a lightweight scorer walks the edges.","Because the beam-scoring function is pluggable, a natural test is whether a cross-encoder or entailment model replacing cosine similarity widens the margin further on MuSiQue-style 3- and 4-hop questions.","The diversity penalty acts as a recall-oriented regularizer; tuning it per hop-depth or per query might let the system trade precision and recall more deliberately than the fixed $\\gamma = 2b$ setting.","If the gist-memory loop generalizes beyond QA, it suggests a template for any iterative evidence-gathering task: compress retrieved evidence to triples, store them as memory, and let a cheap retriever, not the LLM, do the graph walking."],"forward_implications":["Any base retriever (BM25, SBERT, or hybrid) can be upgraded to a multi-hop retriever by adding SyncGE; the paper shows Hybrid + SyncGE beating HippoRAG in single-step retrieval on MuSiQue and HotpotQA.","GEAR reaches its best recall on MuSiQue with about one to two iterations, whereas IRCoT-style methods need more, so the marginal cost of the extra hops is small.","Recall gains translate into end-to-end QA gains: GEAR reports the highest EM/F1 on all three datasets, with the relative EM/F1 improvement over HippoRAG w/ IRCoT on MuSiQue being larger than the recall improvement.","The method works without closed-source LLMs: with Llama-3.1-8B and Qwen-2.5-8B, GEAR still matches or exceeds HippoRAG w/ IRCoT, including a HotpotQA R@15 of 98.1 for Llama-3.1-8B.","Diverse beam search contributes consistently; removing diversity lowers R@15 on every dataset in Table 4."],"supporting_citations":[{"why":"Defines the HippoRAG baseline and the passage–triple alignment and triple-extraction methodology that GEAR builds on and compares against.","marker":"Gutierrez et al., 2024"},{"why":"IRCoT is the iterative retrieval-plus-chain-of-thought baseline, and its released MuSiQue/2Wiki data and evaluation setting are used.","marker":"Trivedi et al., 2023"},{"why":"Provides the reasoning-triple-chain idea that Diverse Triple Beam Search adapts, and is a TRACE baseline.","marker":"Fang et al., 2024"},{"why":"GraphReader is the LLM agent graph-navigation baseline whose design contrasts with GEAR's cheap graph walk.","marker":"Li et al., 2024"},{"why":"Reciprocal Rank Fusion is the mechanism that merges expanded and base-passage rankings in SyncGE and the final GEAR output.","marker":"Cormack et al., 2009"},{"why":"Motivates the diversity weighting in beam search by showing that standard beam search produces low-diversity sequences.","marker":"Ippolito et al., 2019"},{"why":"Introduces the diverse beam search idea that the diversity penalty in Algorithm 1 is adapted from.","marker":"Vijayakumar et al., 2018"},{"why":"MuSiQue dataset and its hop-type categorisation, used for the main evaluation and the per-hop analysis.","marker":"Trivedi et al., 2022"},{"why":"HotpotQA dataset used for evaluation.","marker":"Yang et al., 2018"},{"why":"2WikiMultihopQA dataset used for evaluation.","marker":"Ho et al., 2020"}],"fun_headline_variants":["Graph expansion and agent memory lift multi-hop retrieval to SOTA","GeAR: graph-enhanced agent for efficient multi-hop retrieval","Agent-driven graph expansion beats baselines on multi-hop QA","GeAR's graph expansion and gist memory top multi-hop retrieval","Graph-enhanced agent achieves SOTA multi-hop QA with fewer tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole expansion rests on the assumption that cosine similarity between the query embedding and a concatenated triple chain ranks the genuinely useful reasoning paths above distractor paths, and that the LLM's seed triples are correct enough that the right paths are reachable from them.","fun_headline_variants_meta":{"raw":{"variants":["Graph expansion and agent memory lift multi-hop retrieval to SOTA","GeAR: graph-enhanced agent for efficient multi-hop retrieval","Agent-driven graph expansion beats baselines on multi-hop QA","GeAR's graph expansion and gist memory top multi-hop retrieval","Graph-enhanced agent achieves SOTA multi-hop QA with fewer tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000638,"raw_usage":{"total_tokens":2926,"prompt_tokens":915,"completion_tokens":2011,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1926}},"tokens_in":531,"tokens_out":2011,"duration_ms":13613,"temperature":1.0,"reasoning_tokens":1926,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:42:24.236471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the set of MuSiQue questions where the gold passages are reachable only through triple chains whose cosine similarity to the question is lower than that of several distractor chains; if SyncGE on this subset shows no meaningful recall gain over the base retriever, the embedding-based beam score is not doing the work the paper attributes to it.","supporting_citations":[],"review_version":1}