{"id":"53bfd8a9-b30e-473d-a76c-f32ab70e80de","arxiv_id":"2507.04127","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"BYOKG-RAG combines LLM-generated entities, paths, queries, and candidate answers with multiple graph retrieval tools to answer questions over custom knowledge graphs without training data.","lead":"BYOKG-RAG is a new system for answering questions over custom knowledge graphs by combining several graph retrieval tricks with a large language model. The paper reports gains on five benchmarks, but the evaluation may be inflated because the model's own guesses are fed back into the retrieved context.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Draft-answer feedback loop is a load-bearing leakage channel: the LLM's own guess is linked, path-retrieved, and unioned into MedQA H@2, so the reported 4.5 pp gain may not be attributable to graph retrieval.","rationale":"The reader's weakest_assumption is exactly the channel I would stress, so my agreement is 'agree'. The channel is load-bearing because the abstract's headline number is computed from Table 1, where BYOKG-RAG uses the draft answer and the baselines do not. The MedQA H@2 definition makes the issue explicit: the metric keeps the model's original prediction regardless of retrieval, and the draft answer is also used as a retrieval anchor, so the final answer can simply confirm the model's own guess. The framework's design is otherwise reasonable: multi-strategy retrieval and iterative refinement are sensible engineering contributions, and the open-sourced implementation is a point in its favor. I considered the uncharacterized in-house subsets (WebQSP-IH, CWQ-IH) and filtered MedQA/Text2cypher as a competing concern about external validity, but the draft-answer loop is more fundamental because it threatens the internal validity of the causal attribution to graph retrieval. The proposed ablation is cheap and would settle the question: if the no-draft version retains the 4.5 pp gain, the reader's objection would be resolved and the central claim could stand; if it does not, the headline claim should be re-scoped to 'LLM draft answers plus graph tools' rather than 'graph retrieval tools'. This confirms the reader's REJECT rather than moving the verdict.","tokens_in":18944,"tokens_out":9731,"duration_ms":98981,"concrete_test":"Run the Claude-Sonnet-3.5 configuration of Table 1 with a 'no-draft' ablation: remove the Draft Answer Generation task from KG-Linker, do not link or retrieve to draft entities in Sections 3.2-3.3, and exclude them from agentic and triplet retrieval (Algorithm 2, line 9); in parallel, report MedQA H@2 without unioning the original prediction. If the average gap over the strongest baseline drops from 4.5 pp to near zero, or if no-augmentation MedQA H@2 equals Vanilla LLM's 57.9%, then the reported gain is attributable to the draft-answer channel rather than to graph retrieval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the draft-answer feedback loop. In KG-Linker (Section 3.1, Eq. 4), the LLM produces a draft answer from parametric memory before retrieval; on the first iteration the graph context is empty (Algorithm 2, line 4). That draft answer is entity-linked (Section 3.2), used as a target in Shortest-Paths (Section 3.3, Eq. 6), and included in the entity set for agentic and triplet retrieval (Algorithm 2, line 9). The retrieved context C (Eq. 9) therefore contains paths and triplets anchored at the LLM's own guess, and the final answer (Algorithm 2, line 12) can recover that guess from context. For MedQA, Section 4.1 additionally unions the LLM's original prediction into H@2, so even a method that ignores the KG entirely scores at the Vanilla-LLM level. The baselines in Table 1 (Agentic Traversal, Text-based Retrieval, Graph Reranker, LLM+graph-query) do not have this draft-answer channel. Consequently, the reported 4.5 pp average gain over the strongest baseline may express parametric recall rather than multi-strategy graph retrieval. The paper's ablations do not close this gap: Table 3 compares 'Entity Linking only' against full KG-Linker, which bundles the draft answer with paths and queries, and Table 7 tests retrieval refinement, not removal of the draft channel.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BYOKG-RAG, a zero-shot KGQA framework that combines an LLM-based KG-Linker with specialized graph retrieval tools. The LLM generates question entities, draft answer entities, relation paths, and OpenCypher queries; a graph toolkit then performs entity linking, path retrieval, graph query execution, and triplet retrieval, with an iterative refinement loop before final answer generation. Experiments on WebQSP(-IH), CWQ(-IH), CronQuestions, MedQA, and Northwind report that BYOKG-RAG outperforms the second-best graph retrieval baseline by 4.5 percentage points on average and generalizes to custom KGs, while using fewer LLM calls than several competing KGQA agents. The authors also provide an open-source implementation.","tokens_in":19293,"tokens_out":4605,"duration_ms":50904,"significance":"The paper addresses a practically important problem: bringing zero-shot KGQA to custom knowledge graphs without task-specific training data. The framework is well engineered, open-sourced, and evaluated across multiple backbone LLMs and several benchmark types, with consistent qualitative improvements and moderate context sizes. The multi-strategy retrieval design and the use of specialized graph tools are sensible and could be a valuable contribution to the Graph RAG literature if the reported gains are attributable to graph retrieval. However, the current experimental protocol contains a draft-answer feedback loop that may inflate the headline numbers, and several core evaluations rely on in-house or filtered subsets. The central claim that multi-strategy graph retrieval is responsible for the 4.5-point gain is therefore not yet established by the evidence presented.","major_comments":[{"comment":"The draft-answer feedback loop is a load-bearing confound. In §3.1, Eq. (4), the LLM generates a draft answer from its parametric knowledge before any graph context is available (Algorithm 2, line 4). In §3.2, those draft answers are entity-linked together with question entities, and in §3.3, Eq. (6), they are used as endpoints for shortest-path retrieval. Algorithm 2, line 9, also includes them in the entity set for triplet retrieval, and line 10 unions the resulting paths and triplets into the final context C. The final answer generation (Algorithm 2, line 12) therefore sees graph context that is anchored at the LLM's own guess, so it can recover that guess even when the graph provides no additional evidence. The baselines in Table 1 (Vanilla LLM, LLM+graph-query, Text-based Retrieval, Graph Reranker, Agentic Traversal) do not have this draft-answer channel, making the comparison asymmetric. The ablations in Table 3 compare \"Entity Linking only\" against full KG-Linker, but the latter bundles draft answers together with paths and queries, so they do not isolate the effect of the draft channel. Please rerun the evaluation with the draft-answer component removed from entity linking, shortest-path retrieval, and triplet retrieval, or otherwise show that the reported gains persist without this channel.","section":"§3.1–3.6, Eqs. (4), (6), (9), Algorithm 2"},{"comment":"The MedQA Hit@2 metric explicitly includes the LLM's original prediction. Section 4.1 states: \"we report Hit@2 (H@2) by augmenting the LLM generation based on the retrieved KG context with its original prediction.\" This means that even a retrieval method that contributes nothing can achieve at least the Vanilla LLM score, which is 57.9 for Claude-Sonnet-3.5 in Table 1. The reported BYOKG-RAG score of 65.0 on MedQA is therefore not a clean measure of the value added by graph retrieval, especially because the draft-answer feedback loop already places the original prediction into the retrieved context. Please report Hit@1 or Hit without augmentation for all methods on MedQA, and provide a breakdown of how often the correct answer comes from the original prediction versus from graph-derived context.","section":"§4.1, Metrics; Table 1, MedQA row"},{"comment":"The main results rely heavily on in-house and filtered subsets whose representativeness is not established. WebQSP-IH and CWQ-IH are described only as \"in-house subsets, containing 500 questions each from the original set,\" with no description of how the questions were sampled or whether the subsets preserve the difficulty distribution of the full benchmarks. MedQA is filtered to 227 questions whose answer candidates appear in the KG via entity matching, and Text2cypher is filtered to questions whose ground-truth answers can be retrieved; the effect of these filters on the reported accuracy is not analyzed. Since the abstract claims improvements across \"five benchmarks\" and \"better generalization to custom KGs,\" the paper should either report full-benchmark numbers for WebQSP and CWQ, or justify that the in-house subsets and filtering criteria do not bias the comparison.","section":"§4.1, Dataset descriptions; Table 6"}],"minor_comments":[{"comment":"Equation (10) writes the text-based retrieval score as Embed(q,h)+Embed(q,r)+Embed(q,t); the notation should clarify that these are cosine-similarity scores from a pretrained embedding model and that the sum is over the decomposed triplet components.","section":"§3.5, Eq. (10)"},{"comment":"The sentence \"BYOKG-RAG generates an executable cypehr query\" contains a typo: \"cypehr\" should be \"Cypher.\"","section":"Table 9, Northwind row"},{"comment":"The transition from Eq. (12) to Eq. (13) contains a typo: \"Next, we collect triplets of which relations are present in Rq t9 T(L)r\" should read something like \"Next, we collect triplets whose relations are present in Rq.\"","section":"Appendix A.2"},{"comment":"The phrase \"outperforming the strongest baseline across benchmarks ... by 4.5% points\" is ambiguous because the strongest baseline differs per benchmark; please state whether the average is computed over the per-benchmark best baseline and report the standard deviation or per-benchmark deltas.","section":"§5.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The draft-answer leakage is the central issue: unless the authors can demonstrate that the reported gains survive removal of the draft-answer channel from retrieval, the paper's main attribution claim is unsupported. I do not recommend rejection at this stage because the issue is testable and fixable through additional ablations and metric changes, and the framework itself may be a useful contribution. However, if the revised experiments still show no significant advantage over baselines that receive the same draft-answer context, acceptance would not be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the framework is a sensible engineering contribution: a zero-shot KGQA system that combines entity linking, path retrieval, executable query generation, and triplet retrieval in an iterative loop, with code released. Second, the evaluation does not support the headline claim that the 4.5-point gain comes from graph retrieval. The draft-answer feedback loop is a leaky channel. The LLM produces draft answers from parametric memory before seeing any graph context; those drafts are entity-linked, used as anchors for shortest-path retrieval and triplet retrieval, and the final answer then sees 'evidence' that points at its own guess. For MedQA, the H@2 metric explicitly unions the original prediction, so the method gets credit for its parametric guess even when the graph contributes nothing. The ablations don't isolate this: Table 3 compares entity-linking-only against full KG-Linker, which bundles drafts with paths and queries, and Table 7 tests refinement, not removal of the draft channel. If you remove the draft answer from the retrieval anchors, you may well lose most of the reported edge over Agentic Traversal.\n\nThe paper does several things well. It benchmarks across five KGs, includes comparisons to fine-tuned retrievers, reports latency, and the toolkit is open-sourced. The multi-strategy idea itself is worth building on. But the in-house subsets (WebQSP-IH, CWQ-IH) are only described as 500-question samples with no sampling protocol, and MedQA and Northwind are filtered by answer presence or executable query availability. Those are secondary concerns alongside the leakage issue.\n\nMy read: the authors have constructed a real system and mostly tell you what it does, but they over-attribute the gains to graph retrieval. A revision that (a) reruns MedQA without the H@2 augmentation, (b) reports a variant where the draft answer is excluded from retrieval anchors, and (c) gives full dataset statistics and sampling details would materially strengthen the paper. As it stands, the central claim is not established.\n\nWho gets value from this? Practitioners building zero-shot KGQA over custom KGs will find the framework and engineering details useful. It also serves as a warning case for evaluation design in RAG. I'd bring it to a reading group. It deserves a serious referee — the engineering is sound and the issue is fixable — but I wouldn't accept it in its current form without the re-evaluation.","headline":"A real system and a real evaluation leak: the draft-answer feedback loop and MedQA H@2 protocol make the 4.5-point gain uninterpretable as evidence for graph retrieval.","tokens_in":19835,"tokens_out":4045,"would_cite":true,"duration_ms":42896,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BYOKG-RAG claims that multi-strategy graph retrieval—LLM-generated entities, paths, queries, and draft answers handed to specialized graph tools—improves KGQA by 4.5 percentage points over the strongest baseline and generalizes to custom…","keywords":["knowledge graph question answering","retrieval-augmented generation","graph retrieval","entity linking","OpenCypher query generation","agentic traversal","zero-shot KGQA","iterative refinement"],"falsifier":"Run BYOKG-RAG with the draft-answer pathway ablated: do not entity-link draft answers, do not use them as shortest-path anchors, and score MedQA without augmenting H@2 with the original prediction. If the 4.5-point lead over the second-best method and the MedQA gain shrink or vanish, the claim that multi-strategy graph retrieval drives the improvement would be falsified.","tokens_in":18736,"feed_emoji":"🧠","tokens_out":8773,"duration_ms":83207,"temperature":0.7,"pith_summary":"BYOKG-RAG claims that question answering over 'bring-your-own' knowledge graphs should not ask the LLM to traverse the graph; instead, the LLM should propose graph artifacts and specialized graph tools should do the linking and retrieval. The paper shows that this division of labor—entity linking, path retrieval, query execution, and triplet retrieval run in parallel and then iteratively refined—beats the second-best graph retrieval method by 4.5 percentage points across five benchmarks, with no task-specific training data. This matters because existing approaches force a choice between fine-tuned retrievers that do not transfer to new graphs and agentic traversal that is brittle to initialization errors; BYOKG-RAG claims a middle path that is both zero-shot and general across general, temporal, medical, and enterprise graphs.","feed_headline":"Multi-strategy graph retrieval beats KGQA agents by 4.5 points","feed_subtitle":"LLM drafts plus specialized graph tools answer questions over custom knowledge graphs with no training data.","key_machinery":"The load-bearing machinery is the KG-Linker prompt and the graph toolkit built around it. KG-Linker is a single LLM call that produces entities, paths, an OpenCypher query, and draft answers in tagged sections; the toolkit consumes those artifacts through four retrieval routes—entity linking (string plus bge-m3 embedding similarity, top-m=3), path retrieval (follow-paths BFS plus Dijkstra shortest paths between entities and draft answers), graph query execution, and triplet retrieval (agentic or scoring-based). The refinement loop collects the verbalized union of these retrievals as context, feeds it back into KG-Linker, and repeats until no new entities are produced, which is what lets the framework correct early linking errors.","core_discovery":"The paper's central claim is that KGQA over custom graphs should treat the LLM as a generator of graph anchors rather than as a graph traverser. Given a question and schema, the KG-Linker LLM emits four artifacts: topic entities, candidate answer entities, plausible relation paths, and an OpenCypher query. A graph toolkit then performs fuzzy-string and embedding-based entity linking, follows and validates the generated paths, executes the query, and retrieves triplets either by agentic traversal or by scoring; the union of these results is verbalized, fed back into KG-Linker for up to two refinement iterations, and finally given to the LLM for answer generation. The paper reports that this framework raises Hit rates to 86.6 on WebQSP-IH, 73.6 on CWQ-IH, and 65.5 on CronQuestions, lifts MedQA H@2 to 65.0, and reaches 64.9 on the Northwind Cypher benchmark, beating the second-best graph retrieval method by 4.5 percentage points on average and matching or exceeding state-of-the-art KG agents with fewer LLM calls.","pith_inferences":["Ablating the draft-answer channel would test self-confirmation: because draft answers are entity-linked and used as shortest-path anchors, and MedQA's H@2 augments the model's own prediction, some of the measured gain may come from answer leakage rather than graph evidence.","The artifact-driven design should transfer to hybrid sources: the paper lists text databases as future work, and the same KG-Linker prompt could emit entities and queries that link into a text corpus or a combined text+KG index.","Context length is the main scaling constraint: the paper's own limitation notes that the union of retrieval results can grow long, so token-budgeted pruning is a natural next experiment to keep the method usable for small-context models.","Entity-linking hyperparameters are a sensitivity knob: the default top-m=3 with the union of string and embedding matching could be tuned per graph, and downstream accuracy's dependence on linking recall and precision is not isolated in the paper."],"forward_implications":["Zero-shot KGQA becomes competitive with trained retrievers: BYOKG-RAG reaches 70.5% Recall@10 on CWQ against 54.5–64.1% for RoG, SubgraphRAG, and GNN-RAG, using no training data.","No single retrieval strategy dominates: the best component varies by graph—agentic traversal on Freebase multi-hop, text-based retrieval on temporal questions, graph query execution on enterprise Cypher graphs—so combining them is what carries the accuracy gain.","LLM call efficiency improves over agent baselines: the agentic variant uses 4.5 calls on WebQSP and 6.3 on CWQ versus 11.2 and 14.3 for ToG, and the scoring variant uses 2–3 calls.","Iterative refinement with self-termination contributes on top of each base retriever: on CWQ-IH it lifts Graph Reranker from 63.0 to 68.8, and on Northwind it lifts Graph-Query from 55.3 to 64.9."],"supporting_citations":[{"why":"Provides StructGPT, the agentic traversal baseline that motivates the agentic triplet retrieval component and anchors the zero-shot comparison.","marker":"Jiang et al., 2023a"},{"why":"Provides Think-on-Graph, the strong zero/few-shot KG agent whose LLM-call counts BYOKG-RAG compares against.","marker":"Sun et al., 2024"},{"why":"Provides FiDeLiS, a state-of-the-art KGQA baseline used in the WebQSP/CWQ comparison.","marker":"Sui et al., 2024"},{"why":"Provides GNN-RAG, a supervised graph neural retriever that BYOKG-RAG matches without training data in the Recall@10 comparison.","marker":"Mavromatis and Karypis, 2024"},{"why":"Provides RoG, a supervised reasoning-on-graph retriever baseline in the retrieval-effectiveness comparison.","marker":"Luo et al., 2024a"},{"why":"Supplies both the SubgraphRAG baseline and the efficient embedding decomposition used for text-based triplet retrieval.","marker":"Li et al., 2024b"},{"why":"Supplies bge-m3, the embedding model used for entity linking and question-triplet similarity scoring.","marker":"Chen et al., 2024"},{"why":"Defines WebQSP and the staged query-graph entity-linking paradigm that grounds the main Freebase benchmark.","marker":"Yih et al., 2015"},{"why":"Establishes the MedQA-plus-DrugBank setup and the KGQA evaluation protocol that the paper adapts.","marker":"Yasunaga et al., 2021"}],"fun_headline_variants":["LLM + graph tools beat KG agents by 4.5 points","BYOKG-RAG: custom KG QA without training data","Let LLMs draft, tools retrieve: 4.5pt KGQA win","Graph retrieval toolkit beats agents on your own KG","KGQA: LLM plans, graph tools execute, +4.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported gains come from graph evidence rather than from the LLM confirming its own draft answers, since draft answers are entity-linked and used as shortest-path anchors, and MedQA's H@2 metric explicitly augments the final generation with the model's original prediction.","fun_headline_variants_meta":{"raw":{"variants":["LLM + graph tools beat KG agents by 4.5 points","BYOKG-RAG: custom KG QA without training data","Let LLMs draft, tools retrieve: 4.5pt KGQA win","Graph retrieval toolkit beats agents on your own KG","KGQA: LLM plans, graph tools execute, +4.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1556,"prompt_tokens":1011,"completion_tokens":545,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":452}},"tokens_in":627,"tokens_out":545,"duration_ms":5283,"temperature":1.0,"reasoning_tokens":452,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:54:07.162248+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BYOKG-RAG with the draft-answer pathway ablated: do not entity-link draft answers, do not use them as shortest-path anchors, and score MedQA without augmenting H@2 with the original prediction. If the 4.5-point lead over the second-best method and the MedQA gain shrink or vanish, the claim that multi-strategy graph retrieval drives the improvement would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the MedQA-plus-DrugBank setup and the KGQA evaluation protocol that the paper adapts."}],"review_version":1}