REVIEW 2 major objections 4 minor 1 cited by
KERAP: A Knowledge-Enhanced Reasoning Approach for Accurate Zero-shot Diagnosis Prediction Using Multi-agent LLMs
T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper proposes KERAP, a multi-agent LLM framework that uses a biomedical knowledge graph to make zero-shot next-visit diagnosis predictions, and reports accuracy close to supervised models that need hundreds of thousands of labeled…
desk verdict KERAP is a well-ablated LLM+KG pipeline with a plausible zero-shot story, but the task definition never rules out target-disease leakage and the KG split produces a clinically wrong exclusion; the headline numbers need that check before they can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing ingredient is the knowledge graph itself, iBKH, with 2.38 million entities and 48 million relation triples, plus the way its triples are categorized. The linkage agent uses pretrained biomedical embeddings and an LLM to map a disease name from an EHR to a KG entity; the retrieval agent then splits that entity's relations into positive knowledge (inclusion criteria such as 'curcumin relates to cognitive dysfunction') and negative knowledge (exclusion criteria such as 'pravastatin does not treat cognitive dysfunction'); the prediction agent consumes these in two conversational stages and issues YES or NO. The two-stage design is what turns the KG from a text inoculation into a reasoning scaffold: Stage I forms a hypothesis from inclusion criteria, Stage II rechecks it against exclusion criteria.
What would settle it
A decisive check is to rerun KERAP on the same datasets after removing every visit whose current record already contains the target disease; if accuracy and F1 fall toward the prompting baselines, the advantage largely comes from code presence rather than knowledge-guided temporal reasoning. A second check is to have clinicians annotate a random sample of the retrieved positive and negative triples: if a substantial share of exclusion relations (for example, hypertension marked unrelated to CKD) are wrong, the guidance itself carries errors.
Extended reading notes
Core claim
The central claim is that the combination of knowledge-graph retrieval and staged multi-agent reasoning removes the two failure modes of direct LLM diagnosis prediction: hallucinated or unsupported reasoning, and a bias toward labeling almost everyone high-risk. The discovery is not a new medical fact but a pipeline finding: separating the linking, retrieval, and prediction tasks into specialized agents, and separating KG relations into positive and negative evidence, lets a general-purpose LLM match supervised classifiers in zero-shot conditions. The paper demonstrates this on four conditions (post-stroke cognitive impairment, pneumonia, chronic kidney disease, and congestive heart failure), where KERAP reports the highest accuracy and F1 among the compared zero-shot methods on all four datasets.
Load-bearing premise
The load-bearing premise is that a positive prediction is not trivially available from the target disease already being in the current visit, and that the knowledge graph's automatic relation split into inclusion and exclusion evidence is medically correct.
Editorial extensions
If this is right
- Diagnosis prediction with LLMs no longer needs a labeled training set for every new disease; the same three-agent pipeline can be pointed at a new target disease as long as the knowledge graph contains it.
- Adding a negative-evidence stage to a reasoning pipeline can correct the all-positive bias, since the Stage II check is aimed precisely at cases the model initially over-predicts.
- Performance comparisons should start treating zero-shot pipelines as comparable to supervised classifiers, since the paper reports a gap of roughly 2% in average accuracy against logistic regression and random forest trained on 500-1000 samples.
- The design separates entity linking from evidence retrieval from prediction, so each agent can be upgraded or audited independently without re-engineering the whole system.
Reading between the lines
- The paper never states that visits where the target disease already appears in the current record were excluded, so a conservative re-test that drops such visits would show how much of the reported accuracy reflects genuine temporal generalization rather than code-presence.
- Because the KG relations are classified automatically, clinically wrong labels can enter the guidance; Case B lists hypertension as unrelated to CKD, which standard medicine contradicts. Curating or validating relation polarity before retrieval could change predictions and is a cheap external check.
- The same positive/negative two-stage structure could transfer to other high-stakes prediction tasks with structured prior knowledge, such as adverse drug events or hospital readmission, but the paper does not test that.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes KERAP, a zero-shot diagnosis prediction pipeline in which a linkage agent maps EHR concepts to entities in the iBKH knowledge graph, a retrieval agent extracts positive and negative relation knowledge, and a prediction agent performs two-stage reasoning with GPT-4o-mini. The authors evaluate on four EHR datasets (PSCI, PNA, CKD, CHF) against direct, step-by-step, KG-augmented, iterative, and pre-trained medical-LLM baselines. KERAP reports higher accuracy and F1 than all baselines on all four datasets, with ablations separating the KG and multi-stage contributions, an LLM-choice study, a cost analysis, supervised-learning comparisons, and case studies. The public repository is referenced for code, prompts, and retrieved knowledge.
Significance. If the reported results hold, the contribution is substantial: a zero-shot pipeline whose performance approaches supervised logistic regression and random forest trained on 500-1000 labeled samples, with interpretable KG-grounded reasoning, could be practically useful in low-label medical settings. The paper deserves credit for the clean architecture, the five-run averages with standard deviations in Table 1, and the explicit separation of KG and multi-stage contributions through the Iterative Prompting and KG-Augmented Prompting baselines. The central caveat is that the label-construction protocol must rule out target-disease code leakage, and the KG's positive/negative relation splits must be clinically trustworthy before the headline claim can be accepted.
major comments (2)
- [Method, Task Definition] The task definition states only that the model predicts whether disease d will be diagnosed at the next visit given the current record r_i, and it does not state whether visits in which d already appears in r_i are excluded. For chronic conditions such as CKD, CHF, and PSCI, a diagnosis code once recorded often persists in subsequent visits, so code presence can almost determine the next-visit label. Case A's prompt even instructs the model to treat existing cerebrovascular diagnoses as risk factors, which would mechanically amplify such leakage. The public repository may clarify the preprocessing, but as written the paper does not rule out this failure mode. The authors should state the exclusion rule, re-run the experiments after removing target-disease codes from current-visit inputs or excluding prevalent cases, and report performance on the clean subset.
- [Case Studies, Case B] The Stage II negative knowledge for CKD in Case B lists 'hypertension' as an unrelated factor, despite hypertension being a leading and well-established risk factor for CKD. This indicates that the automatic categorization of iBKH relations into RTx,+ and RTx,− can produce clinically invalid exclusion knowledge, undermining the claim that retrieved knowledge is 'clinically validated.' The example still ends with a correct prediction, so this is not by itself a demonstration of failure, but it is evidence that the relation categorization can inject errors that the framework presents as authoritative. The authors should audit the positive/negative splits for all four target diseases, quantify the frequency of clinically incorrect relations, and test the prediction agent's sensitivity to such errors.
minor comments (4)
- [Figures 2 and 3] The bar charts show no error bars even though Table 1 reports standard deviations over five runs; add error bars or state explicitly that they are omitted for visual clarity.
- [Comparison with Supervised Learning Methods] The text reports KERAP as 73.20% ACC and 72.88% F1, while Table 2's averages are 73.41% and 72.24%; reconcile the numbers and clarify whether Figure 4 uses the same evaluation set as Table 1.
- [KG Data Source] The claim that iBKH is 'clinically validated' is stated without a citation or validation metric; please substantiate this claim or soften the wording.
- [Experiments] PSCI is used both for the post-stroke cognitive impairment disease and for the dataset name; define the dataset name once to avoid ambiguity for readers.
Circularity Check
No circular derivation: KERAP's predictions are generated from an external KG, a frozen LLM, and patient data with no fitting to test labels; self-citations are not load-bearing.
full rationale
KERAP's derivation chain does not reduce to its inputs. In the Method section, the linkage agent maps a disease name to an iBKH entity via SAPBERT embeddings and an LLM (Eq. 1); the retrieval agent summarizes external KG triples into positive and negative knowledge (Eq. 2); the prediction agent is a prompted, frozen GPT-4o-mini that outputs YES or NO from the patient record plus retrieved knowledge (Eq. 3). No stage fits any parameter to the target labels, and the KG and LLM are external to the paper's experiments. The reported zero-shot numbers are therefore genuinely out-of-sample with respect to training, not renamed fitted outputs. The self-citations are minor: ref. 16 (PromptLink) is cited only as inspiration for the linkage agent, not as a theorem forcing the result, and refs. 29-30 provide dataset settings. One experimental-validity concern, flagged as missing support rather than circularity, is that the Task Definition does not explicitly exclude visits where the target disease code already appears in the current record ri; if such visits were kept, code presence could leak into the next-visit label, especially for chronic diseases like CKD and CHF. This would threaten the empirical zero-shot claim but is not a circular derivation because the input and label are not defined as identical, and no fitted parameter is renamed a prediction.
Assumptions & free parameters
free parameters (1)
- LC (linkage candidates) =
10
assumptions (4)
- domain assumption iBKH knowledge graph relations are clinically valid and useful as inclusion/exclusion criteria
- domain assumption GPT-4o-mini reliably executes linkage, retrieval summarization, and prediction without task-specific tuning
- ad hoc to paper Next-visit diagnosis labels are constructed so the target disease is not already present in the current visit record
- domain assumption The selected EHR attributes are sufficient to represent the clinical state relevant to the target disease
Cite this review
Pith. "Pith review of KERAP: A Knowledge-Enhanced Reasoning Approach for Accurate Zero-shot Diagnosis Prediction Using Multi-agent LLMs." pith.science (2026). https://pith.science/paper/5CHDW3SC
@misc{pith2026250702773,
author = {Pith},
title = {Pith review of: KERAP: A Knowledge-Enhanced Reasoning Approach for Accurate Zero-shot Diagnosis Prediction Using Multi-agent LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/5CHDW3SC}},
note = {Machine review of arXiv:2507.02773}
}
read the original abstract
Medical diagnosis prediction plays a critical role in disease detection and personalized healthcare. While machine learning (ML) models have been widely adopted for this task, their reliance on supervised training limits their ability to generalize to unseen cases, particularly given the high cost of acquiring large, labeled datasets. Large language models (LLMs) have shown promise in leveraging language abilities and biomedical knowledge for diagnosis prediction. However, they often suffer from hallucinations, lack structured medical reasoning, and produce useless outputs. To address these challenges, we propose KERAP, a knowledge graph (KG)-enhanced reasoning approach that improves LLM-based diagnosis prediction through a multi-agent architecture. Our framework consists of a linkage agent for attribute mapping, a retrieval agent for structured knowledge extraction, and a prediction agent that iteratively refines diagnosis predictions. Experimental results demonstrate that KERAP enhances diagnostic reliability efficiently, offering a scalable and interpretable solution for zero-shot medical diagnosis prediction.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 1 Pith paper
-
HypKG: Hypergraph-based Knowledge Graph Contextualization for Precision Healthcare
HypKG integrates EHR patient context with a biomedical knowledge graph via LLM-based entity linking and a hypergraph transformer, reporting improved performance on phenotyping and post-stroke cognitive impairment prediction.
Reference graph
Works this paper leans on
-
[1]
Evaluation of artificial intelligence techniques in disease diagnosis and prediction
Ghaffar Nia N, Kaplanoglu E, Nasab A. Evaluation of artificial intelligence techniques in disease diagnosis and prediction. Discover Artificial Intelligence. 2023;3(1):5
work page 2023
-
[2]
Zhang Z, Cui H, Xu R, Xie Y , Ho JC, Yang C. TACCO: Task-guided Co-clustering of Clinical Concepts and Patient Visits for Disease Subtyping based on EHR Data. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; 2024. p. 6324-34
work page 2024
-
[3]
Machine learning model to predict mental health crises from electronic health records
Garriga R, Mas J, Abraha S, Nolan J, Harrison O, Tadros G, et al. Machine learning model to predict mental health crises from electronic health records. Nature medicine. 2022;28(6):1240-8
work page 2022
-
[4]
Kumar Y , Koul A, Singla R, Ijaz MF. Artificial intelligence in disease diagnosis: a systematic literature review, synthesizing framework and future research agenda. Journal of ambient intelligence and humanized computing. 2023;14(7):8459-86
work page 2023
-
[5]
Large language models encode clinical knowledge
Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-80
work page 2023
-
[6]
Bhasuran B, Jin Q, Xie Y , Yang C, Hanna K, Costa J, et al. Preliminary analysis of the impact of lab results on large language model generated differential diagnoses. npj Digital Medicine. 2025;8(1):166
work page 2025
-
[7]
Large language models for disease diagnosis: A scoping review
Zhou S, Xu Z, Zhang M, Xu C, Guo Y , Zhan Z, et al. Large language models for disease diagnosis: A scoping review. arXiv preprint arXiv:240900097. 2024
work page 2024
-
[8]
Cui H, Shen Z, Zhang J, Shao H, Qin L, Ho JC, et al. Llms-based few-shot disease predictions using ehr: A novel approach combining predictive agent reasoning and critical agent instruction. arXiv:240315464. 2024
work page 2024
Show all 31 references
-
[9]
Detecting hallucinations in large language models using semantic entropy
Farquhar S, Kossen J, Kuhn L, Gal Y . Detecting hallucinations in large language models using semantic entropy. Nature. 2024;630(8017):625-30
2024
-
[10]
Simulated misuse of large language models and clinical credit systems
Anibal JT, Huth HB, Gunkel J, Gregurick SK, Wood BJ. Simulated misuse of large language models and clinical credit systems. NPJ Digital Medicine. 2024;7(1):317
2024
-
[11]
Unifying large language models and knowledge graphs: A roadmap
Pan S, Luo L, Wang Y , Chen C, Wang J, Wu X. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering. 2024;36(7):3580-99
2024
-
[12]
Large Language Models and Knowledge Graphs: Opportunities and Challenges
Pan J, Razniewski S, Kalo JC, Singhania S, Chen J, Dietze S, et al. Large Language Models and Knowledge Graphs: Opportunities and Challenges. Transactions on Graph Data and Knowledge. 2023
2023
-
[13]
Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study
Gao Y , Li R, Croxford E, Caskey J, Patterson BW, Churpek M, et al. Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study. JMIR AI. 2025;4:e58670
2025
-
[14]
medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRs
Jia M, Duan J, Song Y , Wang J. medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRs. In: 31st International Conference on Computational Linguistics; 2025. p. 9278-98
2025
-
[15]
Guiding clinical reasoning with large language models via knowledge seeds
Wu J, Wu X, Yang J. Guiding clinical reasoning with large language models via knowledge seeds. In: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence; 2024. p. 7491-9
2024
-
[16]
PromptLink: Leveraging Large Language Models for Cross-Source Biomedical Concept Linking
Xie Y , Lu J, Ho J, Nahab F, Hu X, Yang C. PromptLink: Leveraging Large Language Models for Cross-Source Biomedical Concept Linking. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval; 2024. p. 2589-93
2024
-
[17]
Chain-of-thought prompting elicits reasoning in large language models
Wei J, Wang X, Schuurmans D, Bosma M, Xia F, Chi E, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems. 2022;35:24824-37
2022
-
[18]
Self-reflection in llm agents: Effects on problem-solving performance
Renze M, Guven E. Self-reflection in llm agents: Effects on problem-solving performance. 2nd International Conference on Foundation and Large Language Models. 2024
2024
-
[19]
Evaluating the Diagnostic Performance of Large Language Models on Complex Multimodal Medical Cases
Chiu WHK, Ko WSK, Cho WCS, Hui SYJ, Chan WCL, Kuo MD. Evaluating the Diagnostic Performance of Large Language Models on Complex Multimodal Medical Cases. Journal of Medical Internet Research. 2024;26:e53724
2024
-
[20]
Structured clinical reasoning prompt en- hances LLM’s diagnostic capabilities in diagnosis please quiz cases
Sonoda Y , Kurokawa R, Hagiwara A, Asari Y , Fukushima T, Abe O. Structured clinical reasoning prompt en- hances LLM’s diagnostic capabilities in diagnosis please quiz cases. Japanese Journal of Radiology. 2024:1-7
2024
-
[21]
Huatuogpt-ii, one-stage training for medical adaption of llms
Chen J, Wang X, Ji K, Gao A, Jiang F, Chen S, et al. Huatuogpt-ii, one-stage training for medical adaption of llms. arXiv preprint arXiv:231109774. 2023
2023
-
[22]
RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records
Xu R, Shi W, Yu Y , Zhuang Y , Jin B, Wang MD, et al. RAM-EHR: Retrieval Augmentation Meets Clinical Predictions on Electronic Health Records. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 2: Short Papers); 2024. p. 754-65
2024
-
[23]
RetCare: Towards Interpretable Clinical Decision Making through LLM-Driven Medical Knowledge Retrieval
Wang Z, Zhu Y , Gao J, Zheng X, Zeng Y , He Y , et al. RetCare: Towards Interpretable Clinical Decision Making through LLM-Driven Medical Knowledge Retrieval. In: Artificial Intelligence and Data Science for Healthcare: Bridging Data-Centric AI and People-Centric Healthcare; 2024
2024
-
[24]
Health-LLM: Personalized retrieval-augmented disease prediction system
Jin M, Yu Q, Shu D, Zhang C, Fan L, Hua W, et al. Health-LLM: Personalized retrieval-augmented disease prediction system. arXiv preprint arXiv:240200746. 2024
2024
-
[25]
Emerge: Integrating rag for improved multimodal ehr predictive modeling
Zhu Y , Ren C, Wang Z, Zheng X, Xie S, Feng J, et al. Emerge: Integrating rag for improved multimodal ehr predictive modeling. arXiv preprint arXiv:240600036. 2024
2024
-
[26]
Cpllm: Clinical prediction with large language models
Ben Shoham O, Rappoport N. Cpllm: Clinical prediction with large language models. PLOS Digital Health. 2024;3(12):e0000680
2024
-
[27]
Self-alignment pretraining for biomedical entity representa- tions
Liu F, Shareghi E, Meng Z, Basaldella M, Collier N. Self-alignment pretraining for biomedical entity representa- tions. arXiv preprint arXiv:201011784. 2020
2020
-
[28]
MIMIC-III, a freely accessible critical care database
Johnson AE, Pollard TJ, Shen L, Lehman LwH, Feng M, Ghassemi M, et al. MIMIC-III, a freely accessible critical care database. Scientific data. 2016;3(1):1-9
2016
-
[29]
Hypergraph transformers for ehr-based clinical predictions
Xu R, Ali MK, Ho JC, Yang C. Hypergraph transformers for ehr-based clinical predictions. AMIA Summits on Translational Science Proceedings. 2023;2023:582
2023
-
[30]
Predicting Post-Stroke Cognitive Impairment (PSCI) Using Multiple Machine Learning Approaches
Xie Y , Nahab F, Ge Y , Wu Y , Saurman J, Yang C, et al. Predicting Post-Stroke Cognitive Impairment (PSCI) Using Multiple Machine Learning Approaches. Stroke. 2025;56(Suppl 1):AWP175-5
2025
-
[31]
Biomedical discovery through the integrative biomedical knowledge hub (iBKH)
Su C, Hou Y , Zhou M, Rajendran S, Maasch JR, Abedi Z, et al. Biomedical discovery through the integrative biomedical knowledge hub (iBKH). Iscience. 2023;26(4)
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.