Pith. sign in

REVIEW 3 major objections 3 minor 13 references

KERAG argues that KGQA should retrieve a broad filtered neighborhood around the topic entity rather than exact semantic-parsing paths, and reports 7-21% truthfulness gains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

KERAG improves knowledge-graph question answering by retrieving broad entity neighborhoods instead of exact query paths and using a fine-tuned chain-of-thought summarizer.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid, transparent KGQA pipeline, but the headline '7% over SOTA' compares against KDD Cup winners with their web modules disabled; treat the claim with caution. the 3 major comments →

arxiv 2509.04716 v1 pith:WZSP72G7 submitted 2025-09-05 cs.CL cs.AIcs.IR

KERAG: Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering

Yushi Sun , Kai Sun , Yifan Ethan Xu , Xiao Yang , Xin Luna Dong , Nan Tang , Lei Chen This is my paper
classification cs.CL cs.AIcs.IR
keywords knowledge graph question answeringretrieval-augmented generationentity-level retrievalretrieval-filter-summarizationchain-of-thought summarizationsemantic parsingtruthfulness evaluationmulti-hop reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KERAG's central claim is that knowledge-graph question answering does not need to know in advance which exact triples answer a question. Instead of writing a SPARQL query or following a handful of reasoning paths, the system retrieves a wide neighborhood around the topic entity, filters out clearly irrelevant relations by consulting the graph's schema, and lets a fine-tuned Chain-of-Thought summarizer reason over what remains. The paper reports that this retrieve-broad-then-filter approach raises retrieval recall well above path-based methods and boosts truthfulness—accuracy minus hallucination—by about 7% over state-of-the-art baselines and by 10-21% over tool-calling GPT-4o on two benchmarks. If right, it implies that LLM reasoning power can substitute for brittle semantic parsing at the retrieval step, making KGQA more robust to schema gaps and ambiguous phrasing.

Core claim

On the paper's own terms, the discovery is that entity-level retrieval outperforms triple-level semantic parsing for KGQA: retrieving all potentially relevant facts in a filtered, multi-hop neighborhood lifts recall and accuracy without inflating hallucination. The planner outputs a quadruplet—domain, topic entity, relations to keep, hop count—instead of a logical form; the filter removes only relations that are absolutely irrelevant; and the summarizer, fine-tuned on Chain-of-Thought reasoning, produces the answer. Supporting measurements include retrieval recall of 0.952 on the Head2Tail benchmark versus 0.844 for a path-based baseline, accuracy of 0.908 against 0.895 for a structured-data

What carries the argument

The load-bearing machinery is the retrieval plan (D, E, R-bar, h): the planner identifies the domain and topic entity, inspects the KG schema one hop at a time, filters out relations judged absolutely irrelevant, and stops when the LLM says enough information is available. That plan is translated into SPARQL or API calls, and the retrieved triples are handed to a fine-tuned Chain-of-Thought summarizer. The filter operates at the predicate and schema level, so the summarizer receives a broad but not flooded context.

Load-bearing premise

The reported quality gains hold only if the automated critic used to score answers is itself unbiased and if all facts needed to answer a question lie inside the filtered neighborhood the planner chooses to retrieve.

What would settle it

Run the same Head2Tail and CRAG comparisons with a second, independently designed judge (or a human-annotated sample) over the same system outputs; if the relative ordering of KERAG versus GPT-4o(tool) or ToG shrinks or flips, the reported truthfulness gains are an artifact of critic bias. Separately, craft a question whose ground-truth answer depends on a fact beyond the planner's chosen hop boundary and check whether retrieval recall drops.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, KERAG shows a KGQA system can reach high accuracy on simple questions and competitive numbers on complex ones without generating a single logical form.
  • The ablation numbers imply the fine-tuned CoT summarizer, not the retriever, is the dominant factor: plain CoT reasoning alone refrains from hallucinating for 18% of questions, and fine-tuning adds 10% accuracy on CRAG.
  • The pipeline's reliance on schema-level filtering means it can be dropped onto any KG with an accessible schema, whether exposed through SPARQL or through API calls.
  • Multi-hop retrieval is necessary for coverage: ablation without multi-hop raises the miss rate by 25% on CRAG; filtering is needed to keep hallucination from climbing with the extra context.
  • Entity-level retrieval gives recall 0.952 on Head2Tail versus 0.844 for the path-based baseline, suggesting that missed content—not wrong content—is the failure mode that broad retrieval fixes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave implicit: the same broad-retrieval-then-filter recipe should transfer to less structured corpora—tables, documents, API results—where an exact logical form is even harder to write; the cost is paying retrieval and filtering for a larger neighborhood.
  • A reader could test whether the reported ordering is stable by re-scoring the same outputs with a different judge or a human sample; the paper's validated-critic sample checks reliability but not bias between systems.
  • Because the error analysis attributes most misses to entity linking and summarization rather than retrieval, the next cheap gain is likely a stronger entity linker or a stronger summarizer, not wider retrieval.
  • If the planner were extended to multiple topic entities (the paper sketches this for comparison questions), the same planning loop should answer questions that span several entities, at the price of more hops and more filtering decisions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces KERAG, a knowledge-graph retrieval-augmented generation pipeline that addresses KGQA by retrieving a broad entity neighborhood rather than exact semantic-parsing paths, then filtering irrelevant predicates and summarizing the remaining content with a fine-tuned Chain-of-Thought model. The pipeline is evaluated on the CRAG KG subset, the Head2Tail open-domain benchmark, and four additional SPARQL-based datasets (QALD-10-en, WebQSP, AdvHotpotQA, CWQ). The central claim is that KERAG 'surpasses state-of-the-art solutions by about 7% in quality' and outperforms GPT-4o (Tool) by 10–21%, with ablations showing the contributions of multi-hop retrieval, filtering, CoT prompting, and fine-tuning. The paper also reports robustness across head/torso/tail splits, multi-entity generalization, backbone robustness, latency, and detailed prompts in appendices.

Significance. If the headline claims hold, the contribution is meaningful: KERAG provides a practical alternative to semantic-parsing KGQA by moving the reasoning burden from path/query generation to retrieval-filter-summarization, and the fine-tuning data generation scheme (using the model's own correct CoT outputs) is a useful technique for improving summarization. The paper is unusually transparent in several respects: it ships full prompts, reports an auto-evaluation reliability table, includes ablation and latency analyses, and states its limitations (error propagation, entity linking) explicitly. The core idea is not circular: fine-tuning uses training-set ground truths, and the retrieval-recall claims are measured on the test set. However, the headline quantitative superiority is currently weakened by (i) degraded SOTA baselines on CRAG and (ii) the absence of uncertainty quantification around single-run point estimates, as detailed below.

major comments (3)
  1. [Appendix C and Table 1] The abstract's central claim 'surpasses state-of-the-art solutions by about 7%' rests on Table 1, where the two KDD Cup 2024 winners, apex and db3, are run with '<EMPTY>' fed into their web-content modules. As stated in Appendix C, db3 'jointly considers the inputs from KG and web content' and apex is a 'router-based adaptive RAG pipeline'; feeding empty web content changes these systems. The reported truthfulness scores (apex 0.458, db3 0.337 vs. KERAG 0.529) are therefore not the scores of the actual CRAG SOTA systems, and the 7.1% improvement over apex is an improvement over a KG-only variant. The claim should be re-scoped or the comparison re-run with the original systems (or their published KG-split scores). This is load-bearing because it is the basis for the paper's main quantitative claim.
  2. [Section 5.3, Appendix D, Appendix A] All headline results are single-run point estimates obtained with Llama-3.1-70B as an LLM judge. The paper reports aggregate reliability of the critic (98.4% in Table 11) but no confidence intervals, no per-system bias check, and no human agreement for the specific KERAG outputs. On Head2Tail, the test set is 375 questions per split (margin of error <5%, as stated in Appendix A), yet Table 2 presents differences of 7–10 truthfulness points over StructGPT, ToG, and GPT-4o (tool) as decisive. These differences are within the plausible sampling error of the test set. Please report bootstrap confidence intervals for the key comparisons and, if possible, a small human-annotated validation of the critic's judgments across systems to rule out style bias.
  3. [Section 5.4] The paper uses retrieval recall to validate the core mechanism: 'the retrieval recall of our approach is 0.952, while the recall of ToG is 0.844.' However, retrieval recall is never defined, and the method by which ToG's recall is computed is not given. Since recall is a central explanatory quantity for the performance difference, specify the denominator (e.g., number of answer entities/predicates in the gold subgraph), how it is computed for KERAG, and how it is computed for ToG; otherwise this comparison cannot be verified.
minor comments (3)
  1. [Throughout] There are numerous typos and formatting issues: 'DBPeida' should be 'DBpedia' (Section 5.2); 'amd' in the planning prompt (Appendix I.1); 'grount truth' in the auto-evaluation prompt (Appendix I.5); 'Trustfulness' in Figure 1(c) and Figure 5; missing spaces in table entries (e.g., 'KERAG0.732' in Tables 1 and 6).
  2. [Abstract and Section 5.3] The term 'quality' is used loosely: in the abstract it refers to the 7% truthfulness gain, while later 'quality' sometimes means accuracy. Please use consistent terminology and state explicitly which metric is being discussed in each claim.
  3. [Table 3] The prior SOTA and ToG/ToG-2 numbers are 'directly taken' from previous papers. Please confirm that the evaluation metrics, data splits, and answer formats are identical; otherwise small differences (e.g., 0.558 vs. 0.541 on QALD-10-en) may not be comparable.

Circularity Check

0 steps flagged

No significant circularity: KERAG's reported gains are measured against external benchmarks and ground-truth labels; the db3/apex disabled-web concern is a fairness issue, not a circularity.

full rationale

KERAG is an empirical systems paper, not a formal derivation, and its central claims are benchmark measurements rather than consequences of the paper's own definitions. The 7% and 10-21% gains are computed by running KERAG and external baselines on CRAG and Head2Tail and scoring with the CRAG-style critic template (Section 5.3, Appendix I.5); the truthfulness score T = A - H is a stated evaluation metric, not a fitted quantity. Fine-tuning (Section 4.3) uses benchmark ground-truth labels to filter LLM-generated CoT chains, and the ablations (Table 4) independently quantify the contribution of each component, so the main result does not reduce to the pipeline's own outputs. The paper cites several works by its own authors (ToG, Head2Tail, CRAG, taxonomies survey), but these are used as benchmark references or comparison systems, not as load-bearing justification for the claimed superiority. The one substantive reviewer concern—that KDD Cup baselines db3 and apex are run with '<EMPTY>' in their web modules (Appendix C)—is a baseline-validity / experimental-fairness issue; it does not make the KERAG results definitionally equivalent to its inputs, does not involve a fitted parameter renamed as a prediction, and does not invoke a uniqueness theorem from prior work. No self-definitional equation, no ansatz smuggled via citation, and no renaming of a known result as a new derivation were found. Thus the paper is not circular, and the score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on benchmark correctness, judge reliability, neighborhood sufficiency, transfer of fine-tuning, and several hand-set hyperparameters. No new physical or formal entity is introduced, so the invented-entities ledger is empty.

free parameters (4)
  • Top-30 predicate retention threshold (DPR) = 30
    Appendix E: on Head2Tail and related SPARQL datasets, the top 30 predicates by DPR similarity are kept before summarization. This hand-set threshold trades recall against noise.
  • Maximum candidate (entity, predicate) pairs = 3
    Appendix E: a 3-gram similarity method identifies up to three candidate pairs per Head2Tail query before main entity extraction.
  • CRAG maximum hops = 3
    Section 5.5 footnote: the CRAG API schema limits n to 3. This is an implementation constraint, but it bounds the neighborhood that the central claim depends on.
  • Summarizer fine-tuning schedule = 20 epochs, lr=5e-5
    Appendix E: the CRAG summarizer is fine-tuned for 20 epochs with LoRA/FSDP. These hyperparameters are chosen by hand and the paper does not report sensitivity to them.
axioms (4)
  • domain assumption Benchmark ground-truth answers are correct and complete.
    The auto-evaluation prompt instructs the critic to 'take it as granted that the Ground Truth is always correct' (Appendix I.5). If any gold label is wrong, accuracy and truthfulness scores are affected.
  • domain assumption The Llama-3.1-70B critic preserves ordinal comparisons across systems.
    All headline truthfulness numbers are produced by this critic (Section 5.3). Appendix D reports high overall reliability on a sample, but does not establish bias-free behavior on KERAG's concise answer format or per-system agreement.
  • domain assumption Answer-relevant facts lie inside the topic entity's filtered multi-hop neighborhood.
    Planning assumes that retrieving around one or a few topic entities and filtering by ontology retains all necessary facts (Section 4.2). The paper reports high recall on CRAG (0.967) and Head2Tail (0.952), but this is an empirical property of the benchmarks, not a guaranteed property of KGs.
  • domain assumption Fine-tuning on the CRAG validation set transfers to the private test set.
    Section 5.1 uses the public validation set for training and the private test set for evaluation. The paper does not analyze distribution shift between the two splits.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of KERAG: Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering." pith.science (2026). https://pith.science/paper/WZSP72G7

@misc{pith2026250904716,
  author       = {Pith},
  title        = {Pith review of: KERAG: Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZSP72G7}},
  note         = {Machine review of arXiv:2509.04716}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Retrieval-Augmented Generation (RAG) mitigates hallucination in Large Language Models (LLMs) by incorporating external data, with Knowledge Graphs (KGs) offering crucial information for question answering. Traditional Knowledge Graph Question Answering (KGQA) methods rely on semantic parsing, which typically retrieves knowledge strictly necessary for answer generation, thus often suffer from low coverage due to rigid schema requirements and semantic ambiguity. We present KERAG, a novel KG-based RAG pipeline that enhances QA coverage by retrieving a broader subgraph likely to contain relevant information. Our retrieval-filtering-summarization approach, combined with fine-tuned LLMs for Chain-of-Thought reasoning on knowledge sub-graphs, reduces noises and improves QA for both simple and complex questions. Experiments demonstrate that KERAG surpasses state-of-the-art solutions by about 7% in quality and exceeds GPT-4o (Tool) by 10-21%.

Figures

Figures reproduced from arXiv: 2509.04716 by Kai Sun, Lei Chen, Nan Tang, Xiao Yang, Xin Luna Dong, Yifan Ethan Xu, Yushi Sun.

Figure 1
Figure 1. Figure 1: (a) An exemplar query. (b) On the left, we [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overall pipeline of our KERAG approach. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An exemplar query illustrating the planning. • Step 1: We start with prompt to decide the topic entity E and the domain D from the in￾put question Q. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Generation of the fine-tuning data. and reason about the data in each hop of neighbor￾hood). On the CRAG dataset, multi-hop retrieval improved truthfulness by 5.5% and filtering im￾proved by 3.9% ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison between one-hop, two-hop, and [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 10 canonical work pages · 1 internal anchor

  1. [1]

    finance",

    Determine the domain the query is about. The domain should be one of the following: "finance", "sports", "mu- sic", "movie", "encyclopedia". If none of the domain ap- plies, use "other". Use "domain" as the key in the result json

    work page

  2. [2]

    domain":

    Extract structured information from the query. Include different keys into the result json depending on the domains, amd put them DIRECTLY in the result json. Here are the rules: For ‘encyclopedia‘ and ‘other‘ queries, these are possible keys: - ‘main_entity‘: extract the main entity of the query. For ‘finance‘ queries, these are possible keys: - ‘market_...

    work page 2022

  3. [4]

    InThe Thirteenth In- ternational Conference on Learning Representations

    Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. InThe Thirteenth In- ternational Conference on Learning Representations. Costas Mavromatis and George Karypis. 2022. Rearev: Adaptive reasoning for question answering over knowledge graphs. InFindings of the Association for Computatio...

    work page 2022

  4. [5]

    Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih

    Towards trustworthy retrieval augmented gen- eration for large language models: A survey.arXiv preprint arXiv:2502.06872. Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022. Unik-qa: Unified representations of structured and unstructured knowledge for open-d...

    Pith/arXiv arXiv 2022

  5. [8]

    InThe Eleventh International Conference on Learning Representations

    Decaf: Joint decoding of answers and log- ical forms for question answering over knowledge bases. InThe Eleventh International Conference on Learning Representations. Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. InProceedings o...

    work page 2022

  6. [9]

    I don’t know

    Pytorch fsdp: Experiences on scaling fully sharded data parallel.Proceedings of the VLDB En- dowment, 16(12):3848–3860. A Dataset Details The statistics of training and testing sets in the CRAG KG subset is presented in Table 9. Table 9: Training and testing sets statistics of CRAG. Training Testing head 219 187 torso 197 203 tail 188 188 overall 604 578 ...

    work page arXiv 2025

  7. [12]

    score" should be

    Please select the main entity from the QUESTION; 2. Select the proper predicate from the given PREDICATE LIST; 3. Based on the main entity, predicate selected, write a SPARQL query to answer the QUESTION. Enclose the main entity selected with <>; Enclose the predicate selected with []. Only respond the main entity and predicate!!!" QUESTION: <Question> PR...

    work page 1962

  8. [13]

    Output: {

    2023-01-13, 2. 2023-03-25, 3. 2023-10-21." Output: {"score": 0, "explanation": "The prediction contains an item, 2023-10-21, which is not in ground truth."} Question: "who are current band members of maroon 5?" Ground Truth: "adam Levine, jesse carmichael, james valentine, matt flynn, pj morton, sam farrar" Prediction: "they are jesse carmichael, matt fly...

    work page 2023

  9. [2018]

    InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2185–2194

    Knowledge base question answering via en- coding of complex query graphs. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 2185–2194. Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Cehao Yang, Jiaxin Mao, and Jian Guo

    work page 2018

  10. [2020]

    InProceedings of the 58th annual meeting of the as- sociation for computational linguistics, pages 4498– 4507

    Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. InProceedings of the 58th annual meeting of the as- sociation for computational linguistics, pages 4498– 4507. Priyanka Sen, Armin Oliya, and Amir Saffari. 2021. Expanding end-to-end question answering on differ- entiable knowledge graphs with intersection. InPro...

    work page arXiv 2021

  11. [2022]

    InProceedings of the 29th International Con- ference on Computational Linguistics, pages 1687– 1696

    Logical form generation via multi-task learn- ing for complex question answering over knowledge bases. InProceedings of the 29th International Con- ference on Computational Linguistics, pages 1687– 1696. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2...

    Pith/arXiv arXiv 2024

  12. [2023]

    InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 5778–5791

    Fine-tuned llms know more, hallucinate less with few-shot sequence-to-sequence semantic pars- ing over wikidata. InProceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 5778–5791. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang,...

    Pith/arXiv arXiv 2023

  13. [2025]

    PIPE: Physics-Informed Position Encoding for Alignment of Satellite Images and Time Series

    Pipe: Physics-informed position encoding for alignment of satellite images and time series.arXiv preprint arXiv:2506.14786. Haobo Li, Zhaowei Wang, Jiachen Wang, YueYa Wang, Alexis Kai Hon Lau, and Huamin Qu. 2024a. Cllmate: A multimodal benchmark for weather and climate events forecasting.arXiv preprint arXiv:2409.19058. Tianle Li, Yushi Sun, Shang-ling ...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.