{"id":"72adcda4-6007-4667-98ba-7aca1b029d57","arxiv_id":"2507.17399","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An online pseudo-alignment between retrieved passages and Wikidata triples lets a graph-based RAG pipeline run over millions of documents without offline triple extraction, at the cost of frequent topic misalignment.","lead":"Researchers adapted their graph-based retrieval system GeAR to a web-scale RAG challenge with millions of documents, replacing costly offline triple extraction with online matching of retrieved passages against Wikidata triples. The system scored 0.876 correctness and 0.529 faithfulness on the SIGIR 2025 LiveRAG Challenge, but the paper provides no baseline comparison or error analysis beyond two failure examples.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline effectiveness claim is not established because the Wikidata pseudo-alignment step (Eq. 3) is never ablated; the case study shows it can add off-topic passages, so the reported scores may come from baseline retrieval alone.","rationale":"The paper is a candid system description: it discloses its failure cases, explains design choices, and does not overclaim the scale of its evaluation. The reader's conditional verdict already captures the main weakness, and our stress test agrees that the key assumption—proximal triples can stand in for Wikidata triples via sparse linking—is under-supported. We go slightly further by emphasizing that the paper reports no ablation isolating the contribution of Eq. (3)–(4). Because the method is only activated after the first iteration, and because the case study demonstrates concrete topic drift, the reported end-to-end scores could in principle be produced by the baseline hybrid retriever and the reader alone. The missing ablation is thus directly load-bearing for the 'simple yet effective' claim. The proposed concrete test—comparing against a graph-free variant of the same pipeline—would settle whether the graph expansion adds value. A complementary annotation of link precision on second-hop questions would diagnose the failure mode, but the ablation is the decisive check for the effectiveness claim. No ad hominem is intended; the issue is an evidence gap, not a question of author integrity. If the ablation shows a clear gain, the conditional acceptance is appropriate; if it shows no gain, the central claim should be rejected as stated.","tokens_in":9488,"tokens_out":5381,"duration_ms":58720,"concrete_test":"Run the full submitted pipeline and a graph-free variant (delete Eq. (3)–(4), using only C′_q(n) from the hybrid retriever, with identical reader, rewrite, filter, and answer prompts) on the same DataMorgana dev sample. Compare correctness and faithfulness; if the graph-free variant matches or exceeds 0.875714/0.529335, the Wikidata pseudo-alignment is not the source of the reported performance and the central effectiveness claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that replacing offline LLM triple extraction with online sparse alignment to Wikidata is 'simple yet effective' (Section 1)—rests on the assumption that Eq. (3) links proximal FineWeb triples T′_q(n) to Wikidata triples T_q(n) that support the same reasoning chain. Section 6's own case study shows two failures: 'pacific geoducks' maps to 'pacific oyster' and 'hot tub heater' maps to geography/biology items. Because Section 5 reports only end-to-end correctness/faithfulness (0.875714/0.529335) with no comparison to a graph-free baseline and no ablation of Eq. (3)–(4), it is unknown whether graph expansion improves, degrades, or is simply unused: the authors state that Wikidata triples are not used in the first iteration, so only second-iteration multi-hop questions are affected. If Eq. (3) link precision is low (as the two examples suggest), the 'expanded' passages are noise, and the reported scores could be entirely attributable to the hybrid retriever plus Falcon reader. Thus the load-bearing condition is not just that linking is imperfect, but that the paper never demonstrates the linked triples add retrieval value; the case study provides evidence they may subtract it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes the authors' submission of a GraphRAG system, 'Graph-Enhanced RAG', to the SIGIR 2025 LiveRAG Challenge, which uses the FineWeb-10BT corpus containing millions of passages. The authors adapt their prior GeAR system by replacing the offline LLM-based triple extraction step with an online pseudo-alignment procedure: proximal triples read from baseline-retrieved chunks are linked to Wikidata triples via sparse retrieval (Eq. 3), and those triples are used to retrieve additional passages at later reasoning steps (Eq. 4). The system also includes query rewriting, passage filtering, and a final reader. The paper reports preliminary challenge scores of correctness 0.875714 and faithfulness 0.529335, and a case study showing two instances where the Wikidata linking step shifts topic. The central claim is that this online alignment approach is 'simple yet effective' and that it identifies limitations of current graph-text alignment methods.","tokens_in":9747,"tokens_out":3502,"duration_ms":41910,"significance":"If the effectiveness claim were established, the paper would make a useful contribution: it shows a way to run graph-based RAG at the scale of millions of documents without the prohibitive cost of offline LLM-based triple extraction, and it makes concrete falsifiable predictions about the need for shared semantic-space models for graph and text. The paper is honest about its limitations, and the two worked failure cases in Table 2 are valuable negative evidence for the research community. However, the central claim is not supported by the reported experiments: there is no baseline, no ablation of the new alignment step, no error bars, and no per-category breakdown. The case study itself suggests that the linking step can introduce off-topic passages, so the reported scores may not reflect any benefit from the graph component. The contribution is therefore currently an interesting system description with an unsupported effectiveness claim.","major_comments":[{"comment":"The paper's only quantitative evidence is the pair of challenge scores reported in Section 1 (correctness 0.875714, faithfulness 0.529335), with no comparison to a graph-free baseline, no ablation of Eqs. (3)-(4), and no error bars or multiple runs. Since the Wikidata-based graph expansion is the only new component and Section 5 states that it is skipped during the first iteration, the reader cannot determine whether graph expansion improves, degrades, or is simply unused in the final scores. The 'simple yet effective' claim in Sections 1 and 7 is therefore not established; the authors should report scores for the hybrid retriever alone, for the full system, and ideally for a control condition in which the graph-expanded passages are replaced by randomly retrieved passages.","section":"Section 5 and Section 6"},{"comment":"The case study in Table 2 directly undermines the load-bearing assumption that proximal FineWeb triples can be reliably linked to Wikidata triples via sparse retrieval. In both examples, the linked Wikidata triples shift topic (from 'pacific geoducks' to 'pacific oyster', and from a hot tub heater to geography/biology items), and the manuscript itself states that this 'challenges a key assumption of the original GeAR system'. Because Eq. (4) uses passages retrieved from these linked triples, such misalignments can inject noise into the retrieval context. Without quantifying the precision of the Eq. (3) linking step on a sample and without showing that passages introduced through Eq. (4) improve or even affect the final answer, the reported scores may be entirely attributable to the hybrid retriever and the Falcon reader rather than to the graph component.","section":"Section 6 and Eq. (3)"},{"comment":"The evaluation is conducted on a development set constructed with DataMorgana, and the paper states that hyperparameters were chosen based on this set, but it does not report results on a held-out test set or break down results by question category. This matters because the DataMorgana taxonomy in Table 1 assigns only 30% of questions to the multi-hop categories (path-following and path-finding), and Section 5 explicitly disables graph expansion for the first iteration. Thus the aggregate scores may be dominated by single-hop questions that do not exercise the proposed graph alignment at all. The authors should report correctness and faithfulness separately for multi-hop and single-hop subsets, and should specify which hyperparameter choices were tuned on the development set.","section":"Section 5.1"}],"minor_comments":[{"comment":"The abstract and introduction refer to 'Falcon-3B-Instruct' as the knowledge synchronizer, while Section 3 and the prompts in Appendix A refer to 'Falcon3-10B-Instruct' as the reader; please clarify which model is used for reading, rewriting, and answering, since the two names denote different model sizes.","section":"Section 1 vs. Section 3"},{"comment":"The notation in Eq. (3) is inconsistent: the left-hand side is written Tq( j) with index j, while the text and surrounding equations use n for the current step; also, the function h1_sparse was previously defined for passages, and its application to triples should be specified more precisely.","section":"Eq. (3)"},{"comment":"The table refers to green and red keywords to indicate topics, but this is not accessible in monochrome print or for color-blind readers; please use explicit labels or markers in addition to color.","section":"Table 2"},{"comment":"There are several typos in the prompt listing, for example 'which we be think are relevant' should be 'which we think are relevant'.","section":"Appendix A.2"},{"comment":"The phrase 'topic shifts from toiletry machinery' is misleading, since a hot tub heater is not toiletry; consider rephrasing to 'domestic hot-water equipment' or similar.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest system description with a useful negative case study, but it currently lacks the experimental evidence needed to support its central effectiveness claim. The authors should be encouraged to add an ablation of the graph-expansion component and a baseline comparison; with those additions the paper could become a solid contribution. This is a borderline case between major revision and rejection, and I lean toward major revision because the missing analysis is well within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an honest challenge-paper about adapting GeAR to millions of documents by replacing offline LLM triple extraction with online sparse alignment to Wikidata. The new piece is Eq. (3): proximal triples read from retrieved chunks are linked to Wikidata triples via sparse retrieval, and those linked triples drive graph expansion. That is a legitimate cost-saving mechanism, and the paper deserves credit for reporting a case study (Table 2) that shows the linking frequently drifts off-topic, and for explicitly saying this challenges GeAR's key proxy assumption.\n\nThe soft spot is the one you'd expect: the 'simple yet effective' claim is not backed by evidence. The paper reports only two end-to-end scores (correctness 0.876, faithfulness 0.529) with no baseline, no ablation, no error bars, and no code. Since Wikidata triples are not used in the first iteration, graph expansion only affects second-iteration multi-hop questions. The two case-study examples suggest link precision is low enough that expanded passages may be noise; the reported scores could be entirely due to the hybrid retriever plus Falcon reader. So the stress-test note holds: the paper never demonstrates the graph component adds retrieval value, and its own evidence suggests it may subtract value. That is load-bearing for the effectiveness claim, though not fatal for a challenge report.\n\nMinor: hyperparameters (max_steps_n, aliases_per_entity) were tuned on a DataMorgana dev set, and there is no breakdown of which question types benefit. I'd want to see that in any revision.\n\nWho this is for: people working on GraphRAG scaling, RAG at web scale, and challenge participants. It deserves a serious referee because the mechanism is real and the writing is candid; the referee should ask for an ablation of Eqs. (3)-(4), a graph-free baseline, and confidence intervals. My own verdict is skeptical on the headline claim, but this is a useful system description and a good discussion piece. Send it to review with a request for revision.","headline":"A candid challenge-paper that proposes a neat way to skip offline triple extraction, but never shows the graph part actually helps; the case study is informative enough to warrant engagement.","tokens_in":10271,"tokens_out":2690,"would_cite":true,"duration_ms":28801,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that graph-based RAG can be extended to millions of documents by replacing offline LLM triple extraction with an online, sparse-retrieval alignment to Wikidata triples, reporting correctness 0.875714 and faithfulness…","keywords":["GraphRAG","retrieval-augmented generation","multi-hop question answering","Wikidata","knowledge graph alignment","sparse retrieval","LiveRAG","large language models"],"falsifier":"A reader could test the claim by running the full system on multi-hop questions with graph expansion from Wikidata enabled versus disabled; if the expanded retrieval does not improve answer correctness over the baseline hybrid retrieval alone, the central claim that online-linked Wikidata triples carry the multi-hop signal is falsified. A cheaper check is to inspect the linked triples for a sample of questions and count how often the topic shifts as in Table 2.","tokens_in":9307,"feed_emoji":"🔗","tokens_out":6223,"duration_ms":55493,"temperature":0.7,"pith_summary":"Graph-based retrieval-augmented generation usually pays for multi-hop reasoning by running an LLM over every passage to extract triples, a cost that becomes prohibitive on millions of documents. This paper tries to show that the triple extraction step can be skipped entirely: the triples are read only from the few passages retrieved for a question, then linked on the fly to the nearest Wikidata triples using sparse retrieval, and those linked triples drive graph expansion to pull in more distant evidence. The authors report that this online approach, applied to the SIGIR 2025 LiveRAG corpus, achieves a correctness of 0.875714 and a faithfulness of 0.529335. If the approach holds up, it would remove the main cost barrier to applying GraphRAG at web scale.","feed_headline":"GraphRAG scales to millions of documents via online Wikidata alignment","feed_subtitle":"The online alignment to Wikidata achieves correctness 0.876 and faithfulness 0.529 on LiveRAG.","key_machinery":"The load-bearing mechanism is the online triple-alignment step, Equation 3: each triple read from the retrieved chunks ($T'_{q(n)}$) is linked to the most similar triple in Wikidata by sparse retrieval ($h^1_{sparse}$). Linked triples are expanded into reasoning chains via GeAR's diverse triple beam search, and each triple in the beams is then mapped back to corpus chunks using another sparse-retrieval step. The candidate passages from graph expansion are fused with the baseline retrieval by Reciprocal Rank Fusion, Equation 4, and a filtering prompt removes irrelevant passages before the reader answers.","core_discovery":"The central claim is that proximal triples—atomic facts read from a small set of retrieved chunks—can be linked, through a simple sparse-retrieval step, to triples in an external knowledge graph such as Wikidata, and that these linked triples can then drive graph expansion to retrieve passages that support multi-hop reasoning. This makes triple extraction online and corpus-agnostic: the knowledge graph is not built from the corpus but borrowed. The paper reports that the resulting system, a modified GeAR, achieves correctness 0.875714 and faithfulness 0.529335 on the LiveRAG benchmark. The authors also present a case study showing that the sparse linking often misaligns topics—for instance, 'pacific geoducks' links to 'Pacific oyster'—which they take as evidence that the proxy assumption behind their approach is fragile and that better semantic models operating in a shared space for text and graph data are needed.","pith_inferences":["The same online alignment could be applied to any external knowledge graph and any dense/sparse passage index, making the approach a generic retrofit for existing GraphRAG pipelines that currently rely on offline extraction.","A natural follow-up would be replacing the sparse linker with a cross-encoder or an embedding model trained on (text, triple) pairs; if correctness rises substantially, the sparse linker, not the idea of borrowing Wikidata triples, is the bottleneck.","If the proxy assumption fails in general, a middle path would build a small task-specific knowledge graph from only the retrieved passages using an LLM at query time, preserving the scaling advantage while avoiding topic drift.","The gap between correctness and faithfulness suggests some final answers are right but not supported by the retrieved evidence; tracing whether the unsupported answers come from baseline or graph-expanded passages would isolate where the alignment noise hurts."],"forward_implications":["GraphRAG becomes applicable to corpora of millions of passages without the need to run LLM-based triple extraction over every passage, removing the main scaling bottleneck.","The reported scores (correctness 0.875714, faithfulness 0.529335) define a concrete performance reference for graph-enhanced retrieval on the LiveRAG benchmark.","Skipping graph expansion for the first retrieval step and applying it only to later, multi-hop iterations reduces compute while keeping graph benefits where they matter.","The demonstrated misalignment implies that improving the triple-linking model, not the graph expansion itself, is the next lever for better performance."],"supporting_citations":[{"why":"Supplies the GeAR backbone: multi-step agentic retrieval, triple beam search, and the read/rewrite prompts that the paper adapts for the challenge.","marker":"[7]"},{"why":"Provides DataMorgana, the question-generation methodology used to build the development set and to measure correctness and faithfulness.","marker":"[3]"},{"why":"Supplies Reciprocal Rank Fusion, used to combine dense and sparse retrieval and to fuse graph-expanded passages with baseline retrieval.","marker":"[1]"},{"why":"Represents the GraphRAG line of work that motivates using an external knowledge graph for multi-hop reasoning; the paper adapts this idea to online triple linking.","marker":"[4]"},{"why":"Provides the retrieval-augmented generation setting in which the reader model Falcon3-10B-Instruct is prompted with retrieved passages to answer questions.","marker":"[5]"}],"fun_headline_variants":["GeAR scales GraphRAG to millions via online Wikidata linking","Online Wikidata alignment pushes GraphRAG to millions of docs","Borrowing Wikidata triples lets GraphRAG handle millions","LiveRAG benchmark: GeAR hits 0.876 with Wikidata"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a sparse-retrieval match between a triple read from a retrieved chunk and a triple in Wikidata is semantically faithful enough that expanding the graph from the linked triples brings in useful evidence rather than noise; the paper's own case study shows two examples where the match shifts topic.","fun_headline_variants_meta":{"raw":{"variants":["GeAR scales GraphRAG to millions via online Wikidata linking","Online Wikidata alignment pushes GraphRAG to millions of docs","Borrowing Wikidata triples lets GraphRAG handle millions","LiveRAG benchmark: GeAR hits 0.876 with Wikidata"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000845,"raw_usage":{"total_tokens":3619,"prompt_tokens":825,"completion_tokens":2794,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":2722}},"tokens_in":441,"tokens_out":2794,"duration_ms":22993,"temperature":1.0,"reasoning_tokens":2722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:47:40.487763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test the claim by running the full system on multi-hop questions with graph expansion from Wikidata enabled versus disabled; if the expanded retrieval does not improve answer correctness over the baseline hybrid retrieval alone, the central claim that online-linked Wikidata triples carry the multi-hop signal is falsified. A cheaper check is to inspect the linked triples for a sample of questions and count how often the topic shifts as in Table 2.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the GraphRAG line of work that motivates using an external knowledge graph for multi-hop reasoning; the paper adapts this idea to online triple linking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the retrieval-augmented generation setting in which the reader model Falcon3-10B-Instruct is prompted with retrieved passages to answer questions."}],"review_version":1}