REVIEW 3 major objections 4 minor 21 references
Addressing accuracy and hallucination of LLMs in Alzheimer's disease research through knowledge graphs
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Graph-based retrieval-augmented generation improves the quality of GPT-4o's answers to Alzheimer's disease research questions, but current systems cannot provide detailed source references.
desk verdict Useful applied GraphRAG comparison and a genuinely useful traceability taxonomy; the headline win rate is confounded by a closed corpus and missing statistical testing, but the paper deserves a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a graph-based retrieval-augmented generation pipeline. Instead of retrieving raw text chunks, the system indexes documents into a knowledge graph of entities and relations, partitions the graph into hierarchical communities, and generates summary reports for each community. At query time, a global search visits these community reports, produces intermediate answers, and synthesizes a final answer; a local search retrieves entities, relations, and subgraphs around query keywords. This graph-plus-community structure is what the paper credits for more comprehensive, diverse, and empowering answers, and it is also what makes fine-grained source tracing hard, because each communi
What would settle it
Run the same 70 questions through both systems but with a knowledge base built from 50 Alzheimer's papers written by an independent group, and check whether the graph system still wins more than half of the LLM-as-judge comparisons against plain GPT-4o; if it does not, the reported advantage is an artifact of the closed-loop evaluation.
Extended reading notes
Core claim
The paper's central claim is that for knowledge-intensive Alzheimer's disease questions, a GraphRAG system that retrieves from a knowledge graph built from 50 relevant papers produces answers judged better than those of a standard GPT-4o chatbot on comprehensiveness, diversity, and empowerment, with a win rate above 50 percent under an LLM-as-judge protocol. The same claim does not hold for every graph system: the second system studied trails plain GPT-4o on those metrics and only wins on directness. The paper further claims that current GraphRAG systems cannot provide detailed references: global graph search can point to community clusters, and local search to entities or relations, but nei
Load-bearing premise
The evaluation assumes the 50-paper corpus and the 70 expert questions are a neutral test bed, but the papers are coauthored by the study's senior authors and the questions were curated by the same research group, so plain GPT-4o is tested without access to the very material the questions ask about.
Editorial extensions
If this is right
- If the measured win rate is right, adding a graph knowledge base is a practical way to make biomedical chatbots give more complete and empowering answers than the same LLM alone.
- The subtype analysis implies graph retrieval helps most on questions about specific study results, which plain LLMs often have not memorized.
- The scaling cost estimate implies that applying global-search GraphRAG to large literature collections is expensive, so community-level iteration needs a cheaper alternative.
- The traceability finding implies that even when GraphRAG answers are judged better, scientists cannot verify them against exact sources without additional changes.
- The interface with a prebuilt Alzheimer's database lets other researchers test whether the comparison holds on their own questions.
Reading between the lines
- Because the 50 papers are coauthored by the senior authors and the 70 questions were written by the same group, the graph system's advantage over plain GPT-4o likely overstates how much it would help on genuinely external or closed-book questions; an independent corpus and question set would be the direct test.
- LightRAG's failure on comprehensiveness and diversity may be a retrieval-tuning problem rather than a graph-structure problem; improving keyword extraction or combining local and global signals could close the gap.
- The traceability taxonomy suggests a hybrid design that returns both a graph summary and the specific passages it drew from could preserve answer quality and restore verifiability, which is the natural next benchmark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates two graph-based retrieval-augmented generation (GraphRAG) systems, Microsoft GraphRAG and LightRAG, against a standard GPT-4o chatbot for answering 70 Alzheimer's disease (AD) questions. The knowledge base is a set of 50 AD papers coauthored by the senior authors. A Claude 3.5 LLM judge compares answer pairs on comprehensiveness, diversity, empowerment, and directness, and the paper reports that Microsoft GraphRAG exceeds 50% win rates on the first three metrics, while LightRAG generally underperforms GPT-4o except on directness. The paper also introduces a qualitative four-level traceability taxonomy and describes a kotaemon-based interface. It concludes that GraphRAG improves answer quality but remains limited in traceability.
Significance. If the evaluations were methodologically sound, this would be a useful domain-specific assessment of GraphRAG for biomedical question answering, and the interface plus the 50-paper/70-question resource would be valuable to the AD research community. The qualitative traceability taxonomy is a reasonable framework for comparing RAG variants. However, the central quantitative claim—that GraphRAG 'consistently generates more comprehensive, diverse, and empowering responses'—is not supported by the evidence as presented because of information-access asymmetry, the absence of any accuracy or hallucination measurement, and the lack of statistical inference. The paper also promises but does not deliver a quantitative traceability evaluation. These issues are load-bearing for the paper's two main claims, so the manuscript needs substantial revision.
major comments (3)
- [Section 4.1, Data and Evaluation procedure; Section 4.2 headline claim] The evaluation is confounded by information-access asymmetry. The 50-paper knowledge base is coauthored by the senior authors (Section 1) and the 70 questions were curated by researchers in the same group (Section 4.1). Microsoft GraphRAG is given these papers as its context, while the GPT-4o baseline has no access to them. If the questions were constructed with these papers' findings in mind, the measured win rate reflects privileged access to the answer sources, not a general GraphRAG advantage. This is especially visible in the subtype analysis (Section 4.2, Figure 5), where the largest advantage is on 'Results' questions—facts that may exist only in the supplied papers. A matched baseline with document access (e.g., GPT-4o prompted with the same 50 papers, or a standard RAG system with the same corpus) and ideally an independently curated question set is required before the >50% win-
- [Section 4.1, Evaluation procedure; Section 1 contributions] The paper's title and introduction promise to address accuracy and hallucination, and the contribution list states 'five key metrics: accuracy, comprehensiveness, diversity, empowerment, and directness.' However, Section 4.1 explicitly says the first four aspects 'do not assess the accuracy of the answers' and no human expert accuracy scoring or hallucination check is reported. In addition, the LLM-as-judge comparison is a single run with no confidence intervals, significance tests, or inter-judge agreement. The statement in Section 4.2 that 'GraphRAG consistently generates more comprehensive, diverse, and empowering responses... as evidenced by its win rate exceeding 50%' is therefore not statistically supported, and the paper's accuracy/hallucination claim is not evaluated at all. The authors should add a direct accuracy/reference-factuality assessment (for example, human expert scorin
- [Section 5] The section is titled 'Analyzing the traceability of RAG and GraphRAG' and the introduction to Section 5 states the paper 'takes the first step in addressing this gap by evaluating traceability from both qualitative and quantitative perspectives.' However, the section delivers only a qualitative taxonomy (non-traceable, cluster-level, multi-paragraph, single-paragraph) and a discussion of trade-offs. No quantitative traceability evaluation is presented, so the conclusion that 'the current GraphRAG systems are still limited in providing detailed references' is not substantiated with numbers. A quantitative measure—for instance, the fraction of answers whose claims map to correct paragraphs, or the correctness and granularity of returned citations—is needed to support the second main claim of the paper.
minor comments (4)
- [Figures 3 and 4] The figure numbering and captions are inconsistent. Section 4.1 refers to the evaluation prompt as Figure 3, but Section 4.2 says the evaluation results are 'presented in Figure 3' when the results actually appear in Figure 4. Additionally, the caption for Figure 4 appears twice (once for the comparison bar chart and again for the evaluation prompts), and the bar chart's x-axis labels are unreadably small.
- [Throughout] Several typos and formatting errors should be corrected: 'parital' (Figure 2 caption), 'comthan' (Section 4.2), 'a.' in Section 7, 'the 70 questions' in Section 6 missing a closing parenthesis, and inconsistent reference formatting (e.g., ref. 19 mixes ACL and journal styles).
- [Supplement Tables 1–4] Supplement Table 1 has a header 'Index Question' for a paper list, and Supplement Table 2 repeats 'The detailed papers used in database.' Please align the headers with the table contents.
- [Section 4.1, Models] The paper states GPT-4o was used as the base LLM for both indexing and answering 'to ensure a fair evaluation.' It would be helpful to also report the exact GPT-4o version/date and the Claude 3.5 sonnet version, since LLM-as-judge results can be sensitive to model version.
Circularity Check
GraphRAG's >50% win rate is forced by giving GraphRAG the answer corpus while denying the GPT-4o baseline access to it.
-
self definitional
[Section 1 (Data) and Section 4.2 (Evaluation results and subtype analysis)]
"we constructed a knowledge base comprising 50 research papers on Alzheimer’s disease that are coauthored by senior authors in our group. We then collected 70 questions, curated by professional researchers in the field ... GraphRAG consistently generates more comprehensive, diverse, and empowering responses compared to standard LLM chatbots, as evidenced by its win rate exceeding 50% ... for GraphRAG, since we explicitly include these results in the graph database, the system can answer such questions much better than standard LLMs."
The evaluation lacks any document-access baseline. GraphRAG is given the 50 coauthored papers as its indexed context; GPT-4o is not. The 70 questions were curated by researchers in the same group, and the paper's own subtype analysis shows the largest advantage is on Results questions, which 'ask about specific results that exist in a particular paper' and are explicitly included in the graph database. The reported >50% win rate therefore measures the access asymmetry itself: the system handed the answer corpus beats the system not given it. This is forced by construction, not evidence about graph-based retrieval. A matched RAG baseline with the same 50 papers would be needed to separate graph organization from mere document access.
full rationale
This paper contains no mathematical derivation or fitted parameter, so the usual fitted-value circularity does not apply. The central empirical claim, however, is circular in an evaluation-design sense: the test corpus (50 papers coauthored by senior authors) and the test questions (70 curated by the same research community) are both internal, and only GraphRAG is given the corpus as context. The paper's own explanation of the Results-subtype advantage concedes that the gain comes from explicitly including the results in the graph database. The >50% win-rate headline therefore reduces to the construction of the comparison, not to a property of GraphRAG. This is not a hidden flaw—the authors disclose the corpus—but it does make the headline claim a by-construction artifact rather than an independent benchmark result. Secondary issues (LightRAG underperforming, no significance tests, traceability only qualitatively assessed) do not change that central reduction. Score 6: one or more predicted advantages reduce by construction, while the paper still contains independent descriptive content about the systems' traceability behavior.
Assumptions & free parameters
free parameters (2)
- community level (CL)
- LightRAG retrieval mode
assumptions (4)
- domain assumption Claude 3.5 Sonnet LLM-as-judge provides valid and reliable scoring of the four quality metrics
- domain assumption The 70 questions curated by the authors' research group are representative of real AD researcher queries
- domain assumption The four metrics (comprehensiveness, diversity, empowerment, directness) are the right operationalization of answer quality
- domain assumption GPT-4o as the base LLM for indexing, retrieval, and answering isolates the effect of the retrieval architecture
Cite this review
Pith. "Pith review of Addressing accuracy and hallucination of LLMs in Alzheimer's disease research through knowledge graphs." pith.science (2026). https://pith.science/paper/6DLK7HGS
@misc{pith2026250821238,
author = {Pith},
title = {Pith review of: Addressing accuracy and hallucination of LLMs in Alzheimer's disease research through knowledge graphs},
year = {2026},
howpublished = {\url{https://pith.science/paper/6DLK7HGS}},
note = {Machine review of arXiv:2508.21238}
}
read the original abstract
In the past two years, large language model (LLM)-based chatbots, such as ChatGPT, have revolutionized various domains by enabling diverse task completion and question-answering capabilities. However, their application in scientific research remains constrained by challenges such as hallucinations, limited domain-specific knowledge, and lack of explainability or traceability for the response. Graph-based Retrieval-Augmented Generation (GraphRAG) has emerged as a promising approach to improving chatbot reliability by integrating domain-specific contextual information before response generation, addressing some limitations of standard LLMs. Despite its potential, there are only limited studies that evaluate GraphRAG on specific domains that require intensive knowledge, like Alzheimer's disease or other biomedical domains. In this paper, we assess the quality and traceability of two popular GraphRAG systems. We compile a database of 50 papers and 70 expert questions related to Alzheimer's disease, construct a GraphRAG knowledge base, and employ GPT-4o as the LLM for answering queries. We then compare the quality of responses generated by GraphRAG with those from a standard GPT-4o model. Additionally, we discuss and evaluate the traceability of several Retrieval-Augmented Generation (RAG) and GraphRAG systems. Finally, we provide an easy-to-use interface with a pre-built Alzheimer's disease database for researchers to test the performance of both standard RAG and GraphRAG.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
The Claude 3 Model Family: Opus, Sonnet, Haiku
-
[3]
Bedi, S. et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA 333, 319–328 (2025)
2025
-
[4]
D., Burkholder, E., Wieman, C., Salehi, S
Wang, K. D., Burkholder, E., Wieman, C., Salehi, S. & Haber, N. Examining the potential and pitfalls of ChatGPT in science and engineering problem-solving. Front Educ (Lausanne) 8, 1330486 (2023)
work page 2023
-
[5]
HuangLei et al. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Trans Inf Syst 43, 1–55 (2025)
work page 2025
-
[6]
Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. doi:10.5555/3495724.3496517
-
[7]
Guo, Z., Xia, L., Yu, Y., Ao, T. & Huang, C. LightRAG: Simple and Fast Retrieval-Augmented Generation. (2024)
work page 2024
-
[8]
Edge, D. et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. (2024)
work page 2024
Show all 21 references
-
[9]
Peng, B. et al. Graph Retrieval-Augmented Generation: A Survey. Journal of the ACM 37, (2024)
2024
-
[10]
Wu, J. et al. Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation. (2024)
2024
-
[11]
Wang, X. et al. Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models. 8299–8307 (2024) doi:10.24963/ijcai.2024/917
2024 doi
-
[12]
& Tian, Y
Chen, S., Wong, S., Chen, L. & Tian, Y. Extending Context Window of Large Language Models via Positional Interpolation. (2023)
2023
-
[13]
Ding, Y. et al. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens. Proc Mach Learn Res 235, 11091–11104 (2024)
2024
-
[14]
Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts. Trans Assoc Comput Linguist 12, 157–173 (2023)
2023
-
[15]
Gao, Y. et al. Retrieval-Augmented Generation for Large Language Models: A Survey. (2023)
2023
-
[16]
J., Shu, Y., Gu, Y., Yasunaga, M
Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M. & Su, Y. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. (2024)
2024
-
[17]
Li, M., Miao, S. & Li, P. Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation. (2024)
2024
-
[18]
Lála, J. et al. PaperQA: Retrieval-Augmented Generative Agent for Scientific Research. (2023)
2023
-
[19]
Wang, J. et al. Is ChatGPT a Good NLG Evaluator? A Preliminary Study. NewSumm 2023 - Proceedings of the 4th New Frontiers in Summarization Workshop, Proceedings of EMNLP Workshop 1–11 (2023) doi:10.18653/V1/2023.NEWSUM-1.1
2023 doi
-
[20]
https://github.com/Cinnamon/kotaemon
Cinnamon/kotaemon: An open-source RAG-based tool for chatting with your documents. https://github.com/Cinnamon/kotaemon
-
[21]
A. K. Lahiri and Q. V. Hu, AlzheimerRAG: Multimodal Retrieval Augmented Generation for Clinical Use Cases, arXiv preprint arXiv:2412.16701, 2025. Available: https://arxiv.org/abs/2412.16701v2. 21 Supplement Table 1: The detailed 50 papers included in the external database. Ind...
2025 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.