{"id":"363eb2ce-b8d7-42e9-bde1-f24d101cf1b5","arxiv_id":"2506.22518","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ReG refines weak graph-retriever supervision with LLM-selected reasoning chains and reorganizes retrieved triples into coherent evidence chains, improving KGQA accuracy, data efficiency, and reasoning token efficiency.","lead":"ReG is a new pipeline for graph-based retrieval-augmented generation that uses large language model feedback to clean up the noisy training signals used to train graph retrievers, and then reorganizes retrieved facts into ordered evidence chains before asking the LLM to answer. On multi-hop knowledge graph question answering benchmarks it reports consistent gains across LLM backbones, better data efficiency, and lower reasoning token costs for reasoning models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central refinement loop depends on P containing every oracle triple, yet Eq. 33/34 label every triple outside bG+ as negative; missing coverage therefore not only caps recall but actively suppresses correct evidence, and the paper never audits P's oracle coverage.","rationale":"ReG is a plausible and well-engineered pipeline: LLM-based candidate selection, retriever retraining, and structure-aware reorganization are all coherent, and the reported gains over SubgraphRAG are directionally consistent. The reader's conditional verdict is appropriate. The coverage assumption is the deepest point because the method's entire supervision signal is restricted to P. I agree with the reader's choice, and would sharpen it: missing candidates are not inert, they become negative labels in Eq. 33/34, so a coverage gap actively corrupts the retriever. This is a correctness risk that can be tested directly. I do not see a basis for rejection: the core mechanism is sound under coverage, and the experiments are extensive. The lack of error bars and code, the 5%-data overclaim, and the disconnected theory are secondary; they support CONDITIONAL but not REJECT. No internal contradiction in the main pipeline was found other than the typo 'P := Psp union Pa union Pa' in Sec. 4.1.","tokens_in":912,"tokens_out":2431,"duration_ms":93587,"concrete_test":"Run an oracle-coverage audit on at least 200 randomly sampled queries from WebQSP-Sub, CWQ-Sub, and GrailQA: construct P exactly as in Sec. 4.1 (including the structural merging), give GPT-4o the query plus only P's textualized candidates, and record answer accuracy. Then give GPT-4o an expanded pool P_exp containing P plus all triples within 2 hops of query/answer entities (or, where available, human-annotated support triples). If accuracy on P_exp exceeds accuracy on P by more than a small margin, P fails to cover essential evidence; additionally train two retrievers, one on bG+ and one on bG+ union (oracle triples found in P_exp minus P), and check whether adding those triples improves downstream QA. If it does, Eq. 33/34's negative labeling of out-of-pool oracle triples is the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing condition for ReG is that the candidate path pool P (Sec. 4.1) contains the oracle subgraph bG*, i.e., every triple needed to answer the query. The paper does not measure this. It justifies P by citing Li et al. (2024a) for high-recall shortest-path coverage and adds one-hop query/answer neighborhoods, then further compresses P by roughly 95% (Table 1) via structural merging. None of these steps is evaluated for oracle recall on WebQSP, CWQ, or GrailQA. The consequence is stronger than missing evidence: because the retriever is trained by Eq. 33/34 with positive labels only from bG+ subset of P, every oracle triple outside P is explicitly trained as a negative. Thus a non-shortest correct reasoning path, or a constraint triple more than one hop from both query and answer entities, is not merely invisible to the refiner; the trained retriever is biased against it. A concrete instance would be a query where a spurious short q-a path exists (so Psp contains only the spurious path) while the true evidence is a longer path, or where a numerical constraint is attached to an intermediate entity at distance at least 2 from both q and a. If such cases are nontrivial, the SOTA and 5%-data claims overstate what the method can achieve, independent of LLM refinement quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReG, a graph-based retrieval-augmented generation framework for KGQA. ReG first constructs a candidate path pool from query-answer shortest paths plus one-hop query- and answer-centric neighborhoods, then uses an LLM to select high-quality reasoning chains from that pool; the selected chains serve as refined supervision for training a retriever at triple, entity, or path granularity. At inference time, ReG reorganizes the retrieved triples into BFS-expanded evidence chains before presenting them to the reasoning LLM. The authors formulate graph-based RAG as black-box combinatorial optimization, prove a query-complexity lower bound, and report experiments on WebQSP, CWQ, and GrailQA across multiple LLMs, claiming state-of-the-art accuracy, data efficiency at 5% training data, OOD transferability, and reduced reasoning token cost with large reasoning models.","tokens_in":24571,"tokens_out":6297,"duration_ms":65463,"significance":"If the empirical claims hold, ReG is a useful contribution to graph-based RAG: it directly addresses the weak-supervision problem for graph retrievers, shows that LLM-refined signals transfer across different refinement and reasoning LLMs, and demonstrates efficiency gains with reasoning-focused models. The paper's strengths are its clear problem decomposition, the coverage of three retrieval granularities, the inclusion of OOD and LRM evaluations, and an ablation that separates the supervision-refinement and reorganization components. However, the central coverage assumption of the candidate pool is never audited, and the state-of-the-art claim is overstated relative to the reported tables. These issues are load-bearing for the main conclusions and need to be resolved before the contribution can be fully assessed.","major_comments":[{"comment":"The method's core assumption that the candidate pool P := Psp ∪ Pq ∪ Pa covers the oracle subgraph bG* is never audited. Because Eq. (33) and (34) train every triple outside bG+ as a negative, an oracle triple missing from P is not merely unretrievable; the retriever is explicitly trained to suppress it. Table 1 reports that the complexity-control step compresses the pool to about 5% of its original size, yet no experiment checks whether this compression preserves the evidence needed for test queries. Please add an oracle-coverage audit (e.g., the fraction of test questions whose gold evidence triples from the dataset logical forms are contained in P and in the compressed pool) and ablate the pool construction (e.g., adding two-hop neighborhoods) so that the coverage assumption is empirically grounded.","section":"Sec. 4.1; Eq. (33)-(34); Table 1"},{"comment":"The statement in Section 5.2 that 'all 12 metrics outperforming existing baselines' is not supported by a single consistent configuration. In Table 2, ReG@Triple (GPT-4o) obtains Micro-F1 57.88 on WebQSP-Sub, which is lower than SubgraphRAG (GPT-4o) at 58.91, and in Table 3 ReG@Triple (Llama3.1-8B) obtains Macro-F1 69.91 on WebQSP, which is lower than SubgraphRAG (Llama3.1-8B) at 70.57. If the claim refers to the best ReG variant for each metric, this must be stated explicitly and the per-variant results reported; as written, the claim overstates the evidence.","section":"Sec. 5.2; Tables 2 and 3"},{"comment":"Figure 4a is used to support the abstract-level claim that ReG 'matches the state-of-the-art performance with 5% training data,' but the figure reports no error bars or significance tests, and Section 5.1 fixes a single seed. Please report the underlying numbers and variance over at least three seeds for the 5% and 80% training ratios, and specify the number of LLM refinement calls and the token cost so the reader can assess the claimed computational savings.","section":"Sec. 5.2; Figure 4"}],"minor_comments":[{"comment":"Equation (2) uses 't ∈ (bG ∩ bG*)' where 'τ' is meant; please fix the variable.","section":"Def. 3.1, Eq. (2)"},{"comment":"The final candidate pool is written as 'P := Psp ∪ Pa ∪ Pa'; this should read P := Psp ∪ Pq ∪ Pa.","section":"Sec. 4.1"},{"comment":"Algorithm 1 defines bGsrc as triples with h ∈ Eq, whereas Section 4.2 defines it as h ∈ Eq ∨ t ∈ Eq; the algorithm and the text should match.","section":"Appendix D.4.1, Algorithm 1"},{"comment":"Figure 1 contains truncated labels (e.g., 'viewf', 'Optical') that make the pipeline difficult to follow; a higher-resolution figure is needed.","section":"Figure 1"},{"comment":"Table 6 should define the intersection and union operations over bG+ (paths versus triples) and report how the union handles conflicting selections.","section":"Table 6"},{"comment":"The 'w/o S(I) & S(II)' row is left empty for the Path level; please explain why this variant is not applicable.","section":"Table 4"},{"comment":"The answer-merging step says one representative answer a* is selected because answers share the same reasoning path, but the selection criterion is unspecified; please state it and discuss cases where distinct answers require distinct reasoning paths.","section":"Appendix D.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits a CL/NLP venue and the core idea is worth pursuing. The main risk is the unverified candidate-pool coverage assumption, which interacts with the negative-labeling objective in a way that could invalidate the empirical conclusions if coverage is low on realistic queries. The SOTA and 5%-data claims also need de-overstating. These are fixable within the paper's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's my read of ReG. The paper is a genuinely useful contribution to graph-based RAG. The core idea—use LLM feedback to clean weak shortest-path supervision for retriever training, then reorganize retrieved triples into coherent chains at inference—is simple and seems to work. The experiments are extensive: three KGQA datasets, three retrieval granularities, five LLM reasoners, plus ablations. The data-efficiency result (ReG@Triple at 5% training beats SubgraphRAG at 80%) is striking if it holds, and the token-cost reduction for LRMs (up to 30%) is a nice bonus.\n\nWhat's new here is the combination. Prior text-RAG used LLM feedback for retrieval (RePlug, RAG-DDR), and graph-RAG used noisy shortest-path supervision (RoG, SubgraphRAG). ReG is the first to bring LLM-guided supervision into graph retrieval, and the BFS-based chain reorganization is a sensible way to address the presentation gap. The ablations show both components matter. The transferability and OOD experiments give independent grounding; the central claim is not circular.\n\nNow the soft spots, in order of severity.\n\nFirst, the abstract overclaims. “Match SOTA with 5% training data” is supported only by a comparison against SubgraphRAG at 80% training—not against all baselines. The main SOTA claim (all 12 metrics in Tables 2-3) is plausible, but the 5% claim needs softening.\n\nSecond, no error bars or multiple seeds anywhere. For an empirical paper with claims of “significant and consistent” improvements, that's a real gap. The numbers might be stable, but I can't tell.\n\nThird—and this is the one I'd push hardest on in review—the paper never audits the coverage of the candidate path pool P. The training labels come from bG+ ⊆ P; any oracle triple outside P is explicitly trained as a negative (Eq. 33-34). So if a correct reasoning path isn't a q-a shortest path or within one hop of a query/answer entity, the retriever is actively biased against it. The paper justifies P by citing shortest-path coverage from prior work but doesn't measure oracle recall on WebQSP, CWQ, or GrailQA, and the compression step cuts the pool by ~95%. This is a load-bearing assumption, and it's unverified. The method may still work—the empirical results suggest it does—but the authors need to measure coverage directly.\n\nThe theory section is weak but not load-bearing: Proposition 3.1 is a lower bound for iterative black-box search, yet ReG does one-shot refinement, so the connection is loose. The “Theoretical Discussion” in Sec 4.1 asserts approximation quality without proof. Minor.\n\nThe LLM refiner uses the gold answer during training, which creates a train/inference mismatch. The paper doesn't analyze it. Moderate.\n\nNo code released; that hurts reproducibility.\n\nBottom line: this is a solid paper that deserves a serious referee. The main risk is the coverage assumption. I'd send it to review, asking for a coverage audit, variance reporting, and a corrected abstract. Worth a reading group slot.","headline":"ReG is a credible graph-RAG method with strong experiments, but the abstract overreaches and the candidate-pool coverage assumption needs a direct audit before the claims are taken as ground truth.","tokens_in":25226,"tokens_out":4184,"would_cite":true,"duration_ms":42087,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReG, a new method for graph-based retrieval-augmented generation, aligns weak retrievers with LLM reasoning by having the LLM filter candidate reasoning paths and reorganizing retrieved evidence into coherent chains, achieving…","keywords":["graph-based retrieval-augmented generation","knowledge graph question answering","weak supervision refinement","LLM feedback","evidence chain reorganization","retriever alignment","data efficiency","zero-shot generalization"],"falsifier":"Build a set of KGQA questions whose correct answer requires a reasoning chain that is neither a shortest path between the query entity and the answer entity nor within one hop of either, and verify by direct inspection that these chains are absent from the candidate pool P. If ReG's retriever, trained on LLM-refined supervision from that pool, systematically fails these questions while a retriever given the full KG succeeds, the coverage assumption is violated and the method's guarantee collapses.","tokens_in":24035,"feed_emoji":"🕸️","tokens_out":5249,"duration_ms":48618,"temperature":0.7,"pith_summary":"The paper argues that the weakest link in graph-based retrieval-augmented generation is not the retriever's architecture but the quality of the supervision it is trained on and the way retrieved knowledge is presented to the LLM. To fix this, ReG uses the LLM itself as a filter: from a candidate pool built from shortest paths and one-hop neighborhoods, the LLM selects the reasoning chains that are actually needed to answer a question, and those chains become the training signal for the retriever. At inference time, retrieved triples are reorganized into logically connected evidence chains via breadth-first expansion, matching the way LLMs prefer to read. The paper claims this yields state-of-the-art results on three knowledge-graph QA benchmarks, matches the best baselines with only 5% of the training data, and cuts reasoning-token use by up to 30% when paired with large reasoning models. If true, it means weak retrievers can be made strong simply by giving the LLM a voice in what counts as evidence.","feed_headline":"LLM-refined evidence sets new KGQA state of the art","feed_subtitle":"Aligns weak retrievers with LLM reasoning: SOTA with 5% training data, 30% fewer reasoning tokens.","key_machinery":"The candidate path pool P := Psp ∪ Pq ∪ Pa is the central object: it defines the universe of evidence the LLM is allowed to judge, so everything downstream inherits its coverage. The LLM-guided refinement step uses in-context learning with explanation-based demonstrations to select a subset bP+ of logically coherent paths, whose triples become the training labels bG+. At inference, a structure-aware reorganization module performs BFS-guided chain expansion over the retrieved triples—starting from query-anchored triples and extending only through entity-matched links—plus structural merging for multi-answer and multi-entity paths, producing the ordered evidence chains fed to the LLM. The compression step keeps the candidate pool at about 5% of its raw size, making the LLM calls tractable.","core_discovery":"ReG treats graph-based RAG as a black-box combinatorial search for a minimal sufficient subgraph, shows that directly solving this search is intractable, and instead obtains high-quality supervision by prompting an LLM to pick plausible reasoning paths from a candidate pool P = Psp ∪ Pq ∪ Pa (shortest paths, query-centric and answer-centric one-hop neighborhoods). The LLM-selected paths form the refined supervision signal bG+ that trains the retriever. A second component, structure-aware reorganization, turns the retrieved subgraph into ordered evidence chains by BFS-guided expansion from query-anchored triples, with merging rules for redundant multi-answer and multi-entity paths. The paper's central claim is that this two-step alignment—LLM-refined supervision during training and structure-aware presentation at inference—is what lets weak retrievers match or beat much stronger baselines across WebQSP, CWQ, and GrailQA.","pith_inferences":["The coverage assumption deserves direct measurement: one could take a sample of questions, annotate the oracle subgraphs, and compute what fraction of oracle triples falls inside P. The paper cites prior work for shortest-path recall but does not measure this on the benchmarks it uses.","The two components are logically separable: LLM-refined supervision improves the retriever's precision, while structure-aware reorganization improves the reasoner's use of correct retrievals. A natural test is applying each component independently to other graph-RAG pipelines to see which contributes more in different regimes.","The token-cost result suggests a cheap probe for reasoning quality: organize the same retrieved triples in random order versus evidence-chain order and measure both accuracy and reasoning-token count on the LRMs; the paper's claim predicts a measurable drop in tokens with no loss in accuracy for the reorganized form."],"forward_implications":["On WebQSP-sub and CWQ-sub, ReG with GPT-4o-mini or GPT-4o outperforms all baselines on all four reported metrics (Macro-F1, Micro-F1, Hit, Hit@1).","Retraining with only 5% of the training data yields better Macro-F1 and Hit than the SubgraphRAG baseline trained with 80%, showing supervision quality dominates supervision quantity.","With QwQ-32B and DeepSeek-R1 reasoning models, ReG reduces average reasoning tokens by up to 30% while improving several accuracy metrics, indicating that cleaner, better-organized evidence curbs overthinking.","LLM-refined supervision transfers across backbone LLMs and improves zero-shot OOD generalization on GrailQA's unseen schemas."],"supporting_citations":[{"why":"SubgraphRAG baseline; source of the shortest-path high-recall coverage assumption and the Directional Distance Encoding used in ReG@Triple.","marker":"Li et al., 2024a"},{"why":"RoG baseline; provides the path-level retrieval setting and the planning-module training approach that ReG@Path builds on.","marker":"Luo et al., 2023"},{"why":"Introduces the query-answer shortest-path weak supervision that ReG refines; the baseline whose noisy signals motivate the work.","marker":"Zhang et al., 2022"},{"why":"GNN-RAG baseline; representative of entity-level graph retrieval with unstructured output that ReG's reorganization targets.","marker":"Mavromatis & Karypis, 2024"},{"why":"GrailQA dataset with its zero-shot generalization split used to evaluate OOD transfer.","marker":"Gu et al., 2021"},{"why":"CWQ dataset, the multi-hop benchmark where ReG shows the largest gains.","marker":"Talmor & Berant, 2018"},{"why":"WebQSP dataset, a primary evaluation benchmark.","marker":"Yih et al., 2016"},{"why":"In-context learning, the mechanism used in the LLM refinement and downstream QA prompts.","marker":"Brown et al., 2020"}],"fun_headline_variants":["LLM feedback aligns weak retrievers for graph RAG","Refined supervision turns weak graph retrievers into strong ones","Structure-aware evidence chains improve graph RAG","Weak-to-strong graph RAG via LLM-guided supervision","ReG: Align weak retrievers with LLM reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The candidate pool built from shortest paths plus one-hop neighborhoods around query and answer entities must contain every piece of evidence needed to answer the question; if an essential reasoning step lies outside this pool, the LLM refiner can never select it and the retriever will be blind to it.","fun_headline_variants_meta":{"raw":{"variants":["LLM feedback aligns weak retrievers for graph RAG","Refined supervision turns weak graph retrievers into strong ones","Structure-aware evidence chains improve graph RAG","Weak-to-strong graph RAG via LLM-guided supervision","ReG: Align weak retrievers with LLM reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3365,"prompt_tokens":995,"completion_tokens":2370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2290}},"tokens_in":611,"tokens_out":2370,"duration_ms":17721,"temperature":1.0,"reasoning_tokens":2290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:24:18.100108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a set of KGQA questions whose correct answer requires a reasoning chain that is neither a shortest path between the query entity and the answer entity nor within one hop of either, and verify by direct inspection that these chains are absent from the candidate pool P. If ReG's retriever, trained on LLM-refined supervision from that pool, systematically fails these questions while a retriever given the full KG succeeds, the coverage assumption is violated and the method's guarantee collapses.","supporting_citations":[{"cited_title":"The value of semantic parse labeling for knowledge base question answering","cited_arxiv_id":null,"evidence_quote":"WebQSP dataset, a primary evaluation benchmark."}],"review_version":1}