Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MedRAG claims that adding a four-tier diagnostic knowledge graph to retrieval-augmented generation yields more accurate and specific medical diagnoses, especially for diseases with similar manifestations.

desk verdict A coherent KG-augmented RAG system for clinical diagnosis, but the headline accuracy claim is not yet secure because the knowledge graph may be built over the test set. read the letter →

arxiv 2502.04413 v2 pith:X2PL4WUU submitted 2025-02-06 cs.CL cs.AIcs.IR

Xuejiao Zhao , Siyan Liu , Su-Yin Yang , Chunyan Miao This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords retrieval-augmentedgenerationknowledgegraphhealthcarecopilotdifferentialdiagnosiselectronichealthrecordslargelanguagemodelsdiagnosticreasoningchronicpain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's thesis is that retrieval-augmented generation gives vague or wrong diagnoses precisely when two diseases look alike, because retrieved records say what each disease is without saying how to tell it apart from its look-alikes. MedRAG's fix is a four-tier diagnostic knowledge graph, built from the EHR database and augmented by an LLM, whose bottom layer stores the critical diagnostic differences between similar diseases. At query time the patient's manifestations are matched into this graph, the relevant differences are extracted, and an LLM reasons over the retrieved records plus those differences to produce a diagnosis, treatment plan, and follow-up questions. The authors report that this pipeline beats six existing RAG baselines on diagnostic accuracy and specificity on both the public DDXPlus dataset and a private chronic-pain EHR dataset, with the largest gains at the finest granularity, where similar diseases must be told apart.

What carries the argument

The load-bearing object is the four-tier diagnostic knowledge graph $\mathcal{G}$, a named hierarchy of the form $E_{L1} \xleftarrow{r_s} E_{L2} \xleftarrow{r_s} E_{L3} \xrightarrow{r_m} E_{L4}$: broad categories, subcategories, disease names, and manifestation features, where $r_s$ is an 'is_a' relation and $r_m$ is a 'has_manifestation_of' relation. The design's key move is splitting the bottom layer into $E_{L4d}$, features decomposed from actual EHR histories, and $E_{L4a}$, distinguishing features an LLM writes for each disease from a targeted prompt, so that look-alike diseases carry explicit contrasts such as sitting alleviating lumbar canal stenosis but worsening sciatica. Two mechanisms ride on this graph: upward traversal with shortest-path voting selects the patient's subcategory, and a downward traversal then collects the diagnostic differences $K$; a discriminability score defined as the reciprocal of degree centrality ranks features so the system can ask follow-up questions about the most informative missing details. The reasoning step concatenates $K$ with the retrieved EHRs in the generator prompt, which the ablation attributes the accuracy gain to.

What would settle it

Have clinicians audit a random sample of the LLM-generated diagnostic differences in $E_{L4a}$ (Section 4.1.2) against standard medical references and tally how many are correct, incomplete, or wrong; a substantial error rate would mean the reasoning step can confidently reinforce an incorrect diagnosis even when retrieval is good. A sharper control is to swap $E_{L4a}$ for distinguishing features taken from an independent, verified medical source and re-run the CPDD and DDXPlus evaluations: if fine-grained accuracy collapses, the gain comes from the generated contrasts themselves, not from the graph's structure.

Watch

Extended reading notes

Core claim

MedRAG's central claim is that the reason RAG misdiagnoses is not missing documents but missing discrimination. The paper therefore builds a knowledge graph whose spine is a four-level hierarchy — broad categories, subcategories, disease names, and manifestation features — with bottom-level features of two kinds: features decomposed from the EHR database ($E_{L4d}$) and disease-specific diagnostic differences written by an LLM ($E_{L4a}$). A search module decomposes the new patient's manifestations into features, matches them against the graph, votes upward to the most plausible subcategory, and pulls out the diagnostic-differences knowledge $K$ for the diseases under it. The generator LLM then receives the patient description, the top-$k$ similar EHRs, and $K$ together, so a disease like sciatica is contrasted with lumbar canal stenosis by an explicit distinguishing fact — for example, opposite responses to sitting. The authors claim that this KG-elicited reasoning yields the best accuracy and specificity among the compared RAG models, lifts fine-grained ($L_3$) accuracy by 11.32 points on CPDD over the second-best method, and generalizes across open- and closed-source backbone LLMs.

Load-bearing premise

The load-bearing premise is that the distinguishing features an LLM writes into the bottom layer of the knowledge graph are medically accurate and complete: the same family of models that makes the final diagnosis supplies the contrasts it reasons over, and the paper reports no clinician audit or external medical verification of those generated features.

Editorial extensions

If this is right

  • If the paper's account is right, medical RAG systems should treat diagnostic differences as first-class knowledge: two diseases that share manifestations need an explicit contrast to be told apart, not merely more retrieved documents.
  • The diagnostic-differences module plugs into any backbone LLM: the paper reports fine-grained ($L_3$) accuracy gains for every tested open- and closed-source model, with the open-source Mixtral-8x7B rising from 22.34% to 63.46% when the KG-elicited reasoning is added.
  • A copilot built this way can drive the consultation instead of waiting for complete input: in the masking experiment, restoring the most discriminative missing features raised $L_3$ accuracy from 52.83% to 66.04%.
  • Because graph construction is automated from an EHR database plus an LLM, with ICD-11 used only as structural inspiration, the pipeline transfers across medical specialties and local databases without an external ontology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the LLM-written differences in $E_{L4a}$ were audited by clinicians or checked against an independent medical source, the fine-grained gains would be on firmer ground; the paper's loop, in which the same model family writes and then reads the contrasts, is exactly the part an external verification would pressure-test.
  • The same four-tier pipeline could be dropped into other specialties where look-alike cases dominate the failure mode, such as dermatology rashes or psychiatric differentials, since the paper's construction recipe needs only an EHR collection and an LLM.
  • A decisive follow-up ablation would replace $E_{L4a}$ with distinguishing features from an independent structured source and re-measure $L_3$ accuracy; the paper's own ablation varies the retriever and the KG on and off but never varies where the diagnostic differences come from.
  • The paper itself reports one self-acknowledged dip: adding KG knowledge lowered $L_1$ accuracy for small closed-source models (GPT-3.5 and GPT-4o-mini), which the authors attribute to knowledge conflicts at coarse granularity; a practical deployment would need to decide when the graph helps and when it confuses a weak backbone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. MedRAG proposes a retrieval-augmented generation framework for medical diagnosis that augments a standard RAG pipeline with a four-tier hierarchical diagnostic knowledge graph. The KG is built from an EHR database by disease clustering, hierarchical aggregation, and LLM-based augmentation of disease-specific diagnostic differences; at inference, patient manifestations are decomposed into features, matched to KG nodes, and used to select a disease subcategory whose diagnostic-differences subgraph is injected into the LLM prompt together with retrieved similar EHRs. The system also generates proactive follow-up questions based on a discriminability score derived from degree centrality. The paper evaluates MedRAG on the public DDXPlus dataset and a private chronic pain dataset (CPDD), comparing against six RAG baselines on accuracy at three specificity levels (L1-L3), report-generation metrics, and ablations across several LLM backbones.

Significance. If the empirical claims hold, MedRAG is a practically useful and well-motivated contribution to medical RAG: it addresses the clinically important problem of distinguishing diseases with similar manifestations, uses a structured KG to elicit reasoning rather than merely retrieving text, and demonstrates compatibility with multiple open- and closed-source LLMs. The paper also ships useful evaluation components: a balanced DDXPlus sub-sample, a real-world private CPDD dataset, ablations of retriever and KG modules, and an appendix with prompts and dataset preprocessing details. The central empirical superiority claim is, however, not yet secure. The main risks are potential transductive leakage from using the full EHR database for KG construction, absence of expert or external verification of LLM-generated diagnostic differences, a partially circular evaluation of the proactive questioning mechanism, and lack of statistical reliability evidence on a small test set.

major comments (4)
  1. [Section 4.1, Definition 3.1, and Appendix B.1] The paper never states that the knowledge graph is constructed from the training split only. Definition 3.1 and Eqs. (3)-(7) define G over the full EHR database D with no train/test restriction, and Appendix B.1 only says the CPDD training set is the set 'to be retrieved'; it does not restrict disease clustering, clinical feature decomposition into E_L4d, hierarchical aggregation, or LLM augmentation. If D includes test patients, then E_L4d nodes, the graph topology used by upward traversal and Eq. (14), and the discriminability scores in Eq. (15) are all informed by test records and their ground-truth disease names, which can inflate the Table 1 and Table 3 results relative to baselines without a KG. Please either state explicitly and implement a train-only KG construction protocol, or report results under both full-database and train-only KG construction so the reader can assess the magnitude of any leakage.
  2. [Section 4.1.2, Eq. (5)] The augmented diagnostic differences E_L4a are generated by LLM M_a, and the same LLM families (Llama, Mixtral, GPT) are used as the final generator M_g, but no clinician audit or external medical source verification is reported. In a diagnostic task, incorrect LLM-generated 'key differences' can confidently reinforce an incorrect diagnosis even when retrieval is good, so the quality of E_L4a is load-bearing for the central accuracy claim. Please add a clinician audit of a sample of augmented triples for both datasets, report error rates, and verify that the main conclusions in Table 1 and Table 2 are unchanged when erroneous or unsupported triples are removed.
  3. [Section 6.3, Appendix B.2, Eq. (15)] The proactive questioning evaluation is circular in a specific way: the manifestations removed from the patient record are those with the highest discriminability score sigma computed on the same KG used to select the follow-up questions. The experiment therefore measures whether the mechanism can recover the features it was optimized to ask about, rather than whether the questions are clinically relevant or useful when information is missing for reasons independent of the scoring function. Please add an evaluation with independently chosen missing features (e.g., randomly masked subsets or features that clinicians mark as missing), and report agreement between MedRAG's questions and expert-elicited questions.
  4. [Section 6.1, Table 1] No variance estimates, confidence intervals, or significance tests are reported, and the CPDD test set under the 9:1 split contains only about 55 patients. The 11.32-percentage-point L3 improvement on CPDD could represent a small number of additional correct cases, and the DDXPlus improvements are likewise reported as point estimates. Please report bootstrap confidence intervals or per-patient matched comparisons, and provide complete baseline hyperparameters (retrieval k, embedding model, similarity threshold, prompt settings) so the comparisons are reproducible.
minor comments (6)
  1. [Table A1] The header says 'CCPD' but the dataset is CPDD; please correct the typo.
  2. [Appendix B.1] The sentence 'DDXPlus We directly use the training set and test set in a split dataset in the ratio of 8:1:1(validation set)' is ambiguous; please clarify which ratio applies to train/validation/test and how the 30-sample test subset is selected.
  3. [Section 6.2, Table 2] L1 accuracy drops for GPT-3.5 and GPT-4o-mini when KG-elicited reasoning is added (83.02 to 70.56 and 88.68 to 85.85); the text offers a plausible explanation, but no controlled experiment isolates whether the drop comes from KG noise, prompt sensitivity, or retriever changes.
  4. [Section 4.2.4, Eq. (15)] The definition of n in the discriminability score is imprecise: it should state whether n is the total number of feature nodes in E_L4d, in E_L4, or in G, and whether degree centrality is computed on the full graph or only on the E_L4d nodes, so the score is reproducible.
  5. [Appendix F, Table A2] The text says the ablation examined 'patient clinical feature matching,' but Table A2 has no row isolating that component; please add the corresponding ablation or revise the text to match the table.
  6. [References] Reference [55] lacks publication year and venue; please complete the citation.

Circularity Check

1 steps flagged · score 4.0 of 10

Main diagnostic-accuracy claim is not circular; the proactive-questioning evaluation is partially circular because the same discriminability score defines both the proposed questions and the 'key' manifestations removed in testing.

  1. fitted input called prediction [Section 4.2.4 (Eqs. 15-16), Appendix B.2 (Eqs. 18-19), Table 3 / Section 6.3]
    "e^del_{L4d_s} = Top-r{e_{L4d_s}, σ(e_{L4d_s})} ... We begin by selecting all matching manifestation nodes E_{L4d_s} and ranking them according to their discriminability scores. A proportion r of the nodes with the highest discriminability scores is then removed... After removing, we match the removed nodes E^del_{L4d_s} with each f_i, if the similarity score, the corresponding sentence f_i is also removed."

    The same discriminability score σ (Eq. 15) drives both sides of the RQ3 evaluation: Eq. 16 selects the highest-σ features to propose as follow-up questions, while Eq. 18 removes the top-r highest-σ features to simulate missing key manifestations. Thus the features restored during simulated questioning are exactly the features the mechanism is designed to ask about, so Table 3's improvement shows that adding back the mechanism's own top-ranked features helps, not that the questions are independently or clinically validated. The LLM must still ask and interpret answers, so the improvement is not a pure identity, but the evaluation cannot falsify the σ-ranking assumption and is circular as a validation of question quality.

full rationale

MedRAG's central claim—higher diagnostic accuracy and specificity than state-of-the-art RAG baselines on DDXPlus and CPDD—is an empirical claim evaluated on test samples and compared against six baselines. No equation in the construction chain (Eqs. 3-17) makes the final diagnosis equal to a fitted parameter or to the retrieved documents by definition; the KG-elicited reasoning adds a genuine external knowledge channel, and the Table 1 results are not forced by construction. The self-citations to prior KG work ([70]-[72]) are related-work citations, not load-bearing uniqueness arguments. I therefore find no definitional circularity in the main accuracy claim. One supporting evaluation is partially circular: the proactive diagnostic questioning mechanism selects features by discriminability score σ (Eq. 16), and the evaluation in Appendix B.2 removes the top-r features using the same σ (Eq. 18). Table 3 therefore measures the value of asking about features the mechanism itself has ranked as important, which is a self-confirming test rather than an independent validation of question quality. This does not invalidate the accuracy results but should lower confidence in the questioning-effectiveness claim. A separate, non-circularity risk is that KG construction is defined over an unspecified EHR database D (Definition 3.1), and the paper never explicitly states the KG is built only from the training split; if D included test records, the KG would encode test manifestations and labels. That would be data leakage rather than a circular derivation, and the paper's use of the training set 'to be retrieved' suggests the intended reading is train-only, but the omission should be fixed.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim depends on a modest set of hand-set matching and retrieval parameters whose values are not reported, on the assumption that embedding similarity reflects clinical similarity, on the accuracy of EHR labels, and on the correctness of LLM-generated disease differences. No new medical entities are introduced. The axioms above capture the load-bearing background assumptions.

free parameters (4)
  • matching threshold t_matching = not reported
    Filters feature-to-KG-node matches in Eq. 10; changing it changes which diseases are considered.
  • top-m closest feature nodes = not reported
    Eq. 11: number of KG nodes kept per patient feature; affects vote distribution in upward traversal.
  • top-k retrieved EHR documents = not reported
    Section 4.3: number of retrieved records passed to the LLM; affects prompt length and evidence mix.
  • number of follow-up features selected = not reported
    Eq. 16: how many high-discriminability features are proposed as questions; controls consultation length.
assumptions (6)
  • domain assumption Cosine similarity in the embedding space is a valid proxy for clinical similarity between patient features and KG nodes.
    Invoked in Eq. 9-11 for feature matching and in FAISS retrieval; no clinical validation is provided.
  • domain assumption Diagnoses and manifestations in the EHR database are complete and accurate enough to serve as ground truth.
    Section 4.1.1 extracts disease clusters and features from D; noisy or biased labels propagate into the KG.
  • ad hoc to paper LLM-generated disease-specific features in E_L4a are medically correct without expert audit.
    Eq. 5 uses M_a to write key diagnostic differences; no clinician verification or error analysis is described.
  • domain assumption The four-tier hierarchy produced by LLM topic aggregation and hierarchical clustering reflects clinically meaningful groupings.
    Section 4.1.1; upward traversal in Eq. 12-13 is only as good as this grouping.
  • ad hoc to paper Inverse degree centrality measures how useful a manifestation is for telling similar diseases apart.
    Eq. 15-16 drive the proactive questioning mechanism; the metric is not compared with clinical informativeness.
  • domain assumption The balanced DDXPlus sub-dataset with 30 test samples per disease is representative of realistic diagnostic difficulty.
    Appendix B.1 fixes prevalence artificially; results may not generalize to natural disease distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot." pith.science (2026). https://pith.science/paper/X2PL4WUU

@misc{pith2026250204413,
  author       = {Pith},
  title        = {Pith review of: MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X2PL4WUU}},
  note         = {Machine review of arXiv:2502.04413}
}
read the original abstract

Retrieval-augmented generation (RAG) is a well-suited technique for retrieving privacy-sensitive Electronic Health Records (EHR). It can serve as a key module of the healthcare copilot, helping reduce misdiagnosis for healthcare practitioners and patients. However, the diagnostic accuracy and specificity of existing heuristic-based RAG models used in the medical domain are inadequate, particularly for diseases with similar manifestations. This paper proposes MedRAG, a RAG model enhanced by knowledge graph (KG)-elicited reasoning for the medical domain that retrieves diagnosis and treatment recommendations based on manifestations. MedRAG systematically constructs a comprehensive four-tier hierarchical diagnostic KG encompassing critical diagnostic differences of various diseases. These differences are dynamically integrated with similar EHRs retrieved from an EHR database, and reasoned within a large language model. This process enables more accurate and specific decision support, while also proactively providing follow-up questions to enhance personalized medical decision-making. MedRAG is evaluated on both a public dataset DDXPlus and a private chronic pain diagnostic dataset (CPDD) collected from Tan Tock Seng Hospital, and its performance is compared against various existing RAG methods. Experimental results show that, leveraging the information integration and relational abilities of the KG, our MedRAG provides more specific diagnostic insights and outperforms state-of-the-art models in reducing misdiagnosis rates. Our code will be available at https://github.com/SNOWTEAM2023/MedRAG

Figures

Figures reproduced from arXiv: 2502.04413 by the authors.

Figure 1
Figure 1. (a) The existing RAG and LLMs rely on heuristic-based approaches, leading to incorrect or vague outputs, particularly [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of MedRAG. MedRAG first extracts patient (red node) manifestations from structured or [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation result on Llama-3.1-Instruct 8B backbone [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DentalBench: Benchmarking and Advancing LLMs Capability for Bilingual Dentistry Understanding

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A new bilingual dental QA benchmark and corpus reveals large performance gaps in LLMs for dentistry, and shows that domain adaptation with the corpus improves accuracy.

  2. CLARIFY: A Specialist-Generalist Framework for Accurate and Lightweight Dermatological Visual Question Answering

    cs.CV 2025-08 reject novelty 4.0 of 10

    A specialist classifier feeding a pruned VLM with knowledge-graph grounding reports 82.1% diagnostic accuracy on a 39-image dermatology test set, about 18 percentage points above a fine-tuned VLM baseline.

Reference graph

Works this paper leans on

82 extracted references · 39 canonical work pages · cited by 2 Pith papers

  1. [1]

    Doctor Co-Pilot

    2024. Doctor Co-Pilot. https://demos.amotion.ai/3. Accessed: 2024-10-11

    2024

  2. [2]

    Microsoft Copilot in Healthcare

    2024. Microsoft Copilot in Healthcare. https://www.avanade.com/en/services/ artificial-intelligence/ai-copilot-hub/health-ai-copilot. Accessed: 2024-10-11

    2024

  3. [3]

    Durga Prasad Amballa. 2023. AI-Powered Copilot for Healthcare Sales Agents: Enhancing Customer Engagement and Test Recommendations. Journal of Scien- tific and Engineering Research 10, 10 (2023), 164–167

    work page 2023

  4. [4]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi

  5. [5]

    Payal Chandak, Kexin Huang, and Marinka Zitnik. 2023. Building a knowledge graph to enable precision medicine. Scientific Data 10, 1 (2023), 67

    work page 2023

  6. [6]

    Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023. LLM-empowered chatbots for psychiatrist and patient simulation: application and evaluation. arXiv preprint arXiv:2305.13614 (2023)

    arXiv 2023

  7. [7]

    Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2021. UnitedQA: A hybrid approach for open domain question answering. arXiv preprint arXiv:2101.00178 (2021)

    arXiv 2021

  8. [8]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 719–729

    work page 2024

Show all 82 references
  1. [9]

    Ram A Dixit, Christian L Boxley, Sunil Samuel, Vishnu Mohan, Raj M Ratwani, and Jeffrey A Gold. 2023. Electronic health record use issues and diagnostic error: MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot a sco...

    2023

  2. [10]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The faiss library. arXiv preprint arXiv:2401.08281 (2024)

    2024 arXiv

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  4. [12]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130 (2024)

    2024 arXiv

  5. [13]

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining ...

    2024

  6. [14]

    Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn

  7. [15]

    Yanjun Gao, Ruizhe Li, John Caskey, Dmitriy Dligach, Timothy Miller, Matthew M Churpek, and Majid Afshar. 2023. Leveraging a medical knowledge graph into large language models for diagnosis prediction. arXiv preprint arXiv:2308.14321 (2023)

    2023 arXiv

  8. [16]

    Yifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen, Yongquan He, and Dongsheng Li. 2024. Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models. arXiv preprint arXiv:2402.16568 (2024)

    2024 arXiv

  9. [17]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang

  10. [18]

    Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023. MedAlpaca–An Open-Source Collection of Medical Conversational AI Models and Training Data. arXiv preprint arXiv:2304.08247 (2023)

    2023 arXiv

  11. [19]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bo- janowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense in- formation retrieval with contrastive learning. arXiv preprint arXiv:2112.09118 (2021)

    2021 arXiv

  12. [20]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al . 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088 (2024)

    2024 arXiv

  13. [21]

    Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yaliang Li, and Ji-Rong Wen. 2023. Rea- soninglm: Enabling structural subgraph reasoning in pre-trained language mod- els for question answering over knowledge graph.arXiv preprint arXiv:2401.00158 (2023)

    2023 arXiv

  14. [22]

    Xinke Jiang, Yue Fang, Rihong Qiu, Haoyu Zhang, Yongxin Xu, Hao Chen, Wentao Zhang, Ruizhe Zhang, Yuchen Fang, Xu Chu, et al . 2024. TC-RAG: Turing-Complete RAG’s Case study on Medical LLM Systems. arXiv preprint arXiv:2408.09199 (2024)

    2024 arXiv

  15. [23]

    Xinke Jiang, Ruizhe Zhang, Yongxin Xu, Rihong Qiu, Yue Fang, Zhiyuan Wang, Jinyi Tang, Hongxin Ding, Xu Chu, Junfeng Zhao, et al. 2024. HyKGE: A Hypoth- esis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responses. arXiv preprint arXiv:2312.15883 (2024)

    2024 arXiv

  16. [24]

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983 (2023)

    2023 arXiv

  17. [25]

    Wall Street Journal. 2023. OpenAI Expands Healthcare Push With Color Health’s Cancer Copilot. The Wall Street Journal (2023). https://www.wsj.com/articles/openai-expands-healthcare-push-with-color- healths-cancer-copilot-86594ff1 Accessed: 2024-09-18

    2023

  18. [26]

    Minki Kang, Jin Myung Kwak, Jinheon Baek, and Sung Ju Hwang. 2023. Knowl- edge graph-augmented language models for knowledge-grounded dialogue gen- eration. arXiv preprint arXiv:2305.18846 (2023)

    2023 arXiv

  19. [27]

    Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172 (2019)

    2019 arXiv

  20. [28]

    Ching Hung Lee, Zehao Zhang, and Xuejiao Zhao. 2021. A survey of smart healthcare for the elderly based on user requirements and supply accessibility. In 5th International Conference on Crowd Science and Engineering . 108–112

    2021

  21. [29]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...

    2020

  22. [30]

    Hongwei Li, Sirui Li, Jiamou Sun, Zhenchang Xing, Xin Peng, Mingwei Liu, and Xuejiao Zhao. 2018. Improving api caveats accessibility by mining api caveats knowledge graph. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 183–193

    2018

  23. [31]

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang

  24. [32]

    Zhiang Li and Tong Ruan. 2024. Knowledge-routed Automatic Diagnosis with Heterogeneous Patient-oriented Graph. IEEE Access (2024)

    2024

  25. [33]

    Ye Liu, Yao Wan, Lifang He, Hao Peng, and S Yu Philip. 2021. Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning. In Proceedings of the AAAI conference on artificial intelligence , Vol. 35. 6418–6425

    2021

  26. [34]

    Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061 (2023)

    2023 arXiv

  27. [35]

    Cureus 15, 6 (2023)

    Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus 15, 6 (2023)

    2023

  28. [36]

    David E Newman-Toker, Najlla Nassery, Adam C Schaffer, Chihwen Winnie Yu- Moe, Gwendolyn D Clemens, Zheyu Wang, Yuxin Zhu, Ali S Saber Tehrani, Mehdi Fanai, Ahmed Hassoon, et al. 2024. Burden of serious harms from diagnostic error in the USA. BMJ Quality & Safety 33, 2 (2024), 109–120

    2024

  29. [37]

    John J Norcini, Linda L Blank, Gerald K Arnold, and Harry R Kimball. 1995. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Annals of internal medicine 123, 10 (1995), 795–799

    1995

  30. [38]

    OpenAI. 2023. Color Health’s Cancer Copilot. https://openai.com/index/color- health/ Accessed: 2024-09-18

    2023

  31. [39]

    Kaixin Ma, Hao Cheng, Yu Zhang, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao

  32. [40]

    arXiv preprint arXiv:2305.03130 (2023)

    Chain-of-skills: A configurable model for open-domain question answering. arXiv preprint arXiv:2305.03130 (2023)

    2023 arXiv

  33. [41]

    World Health Organization et al. 1992. ICD-11. (No Title) (1992)

    1992

  34. [42]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics 11 (2023), 1316–1331

    2023

  35. [43]

    Haocong Rao, Minlin Zeng, Xuejiao Zhao, and Chunyan Miao. 2024. A Survey of Artificial Intelligence in Gait-Based Neurodegenerative Disease Diagnosis. arXiv preprint arXiv:2405.13082 (2024)

    2024 arXiv

  36. [44]

    OpenAI. 2024. ChatGPT. https://openai.com/index/chatgpt/ Accessed: 2024-10- 07

    2024

  37. [45]

    OpenAI. 2024. GPT-4o. https://openai.com/index/hello-gpt-4o/ Accessed: 2024-10-07

    2024

  38. [46]

    Priyanka Sen, Sandeep Mavadia, and Amir Saffari. 2023. Knowledge graph- augmented language models for complex question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE). 1–8

    2023

  39. [47]

    Xiaoming Shi, Jie Xu, Jinru Ding, Jiali Pang, Sichen Liu, Shuqing Luo, Xingwei Peng, Lu Lu, Haihong Yang, Mingtao Hu, et al. 2023. Llm-mini-cex: Automatic evaluation of large language model for diagnostic conversation. arXiv preprint arXiv:2308.07635 (2023)

    2023 arXiv

  40. [48]

    Karthik Soman, Peter W Rose, John H Morris, Rabia E Akbas, Brett Smith, Braian Peetoom, Catalina Villouta-Reyes, Gabriel Cerono, Yongmei Shi, Angela Rizk- Jackson, et al. 2023. Biomedical knowledge graph-enhanced prompt generation for large language models. arXiv preprint arXi...

    2023 arXiv

  41. [49]

    Zhiyao Ren, Yibing Zhan, Baosheng Yu, Liang Ding, and Dacheng Tao. 2024. Healthcare copilot: Eliciting the power of general llms for medical consultation. arXiv preprint arXiv:2402.13408 (2024)

    2024 arXiv

  42. [50]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. 2024. Raptor: Recursive abstractive processing for tree-organized retrieval. arXiv preprint arXiv:2401.18059 (2024)

    2024 arXiv

  43. [51]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal

  44. [52]

    Yi-Lin Tuan, Yun-Nung Chen, and Hung-yi Lee. 2019. Dykgchat: Benchmarking dialogue generation grounding on dynamic knowledge graphs. arXiv preprint arXiv:1910.00610 (2019)

    2019 arXiv

  45. [53]

    Deeksha Varshney, Aizan Zafar, Niranshu Kumar Behera, and Asif Ekbal. 2023. Knowledge graph assisted end-to-end medical dialog generation. Artificial Intelligence in Medicine 139 (2023), 102535

    2023

  46. [54]

    Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. arXiv preprint arXiv:2403.10081 (2024)

    2024 arXiv

  47. [55]

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let me speak freely? a study on the impact of for- mat restrictions on performance of large language models. arXiv preprint arXiv:2408.02442 (2024)

    2024 arXiv

  48. [56]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models. Advances in neural information processing systems Zhao et al. 35 (2022), 24824–24837

    2022

  49. [57]

    arXiv preprint arXiv:2212.10509 (2022)

    Interleaving retrieval with chain-of-thought reasoning for knowledge- intensive multi-step questions. arXiv preprint arXiv:2212.10509 (2022)

    2022 arXiv

  50. [58]

    Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yan- feng Wang. 2024. PMC-LLaMA: toward building open-source language models for medicine. Journal of the American Medical Informatics Association (2024), ocae045

    2024

  51. [59]

    Junde Wu, Jiayuan Zhu, and Yunli Qi. 2024. Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation. arXiv preprint arXiv:2408.04187 (2024)

    2024 arXiv

  52. [60]

    Jinge Wang, Zien Cheng, Qiuming Yao, Li Liu, Dong Xu, and Gangqing Hu

  53. [61]

    Yunfei Xie, Juncheng Wu, Haoqin Tu, Siwei Yang, Bingchen Zhao, Yongshuo Zong, Qiao Jin, Cihang Xie, and Yuyin Zhou. 2024. A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor? arXiv preprint arXiv:2409.15277 (2024)

    2024 arXiv

  54. [62]

    Zixiang Wang, Yinghao Zhu, Junyi Gao, Xiaochen Zheng, Yuhui Zeng, Yifan He, Bowen Jiang, Wen Tang, Ewen M Harrison, Chengwei Pan, et al. [n. d.]. RetCare: Towards Interpretable Clinical Decision Making through LLM-Driven Medical Knowledge Retrieval. In Artificial Intelligence ...

  55. [63]

    Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2023. Making retrieval-augmented language models robust to irrelevant context.arXiv preprint arXiv:2310.01558 (2023)

    2023 arXiv

  56. [64]

    Sidong Wei, Xuejiao Zhao, and Chunyan Miao. 2018. A comprehensive explo- ration to the machine learning techniques for diabetes identification. In 2018 IEEE 4th World Forum on Internet of Things (WF-IoT) . IEEE, 291–295

    2018

  57. [65]

    Cyril Zakka, Joseph Cho, Gracia Fahed, Rohan Shad, Michael Moor, Robyn Fong, Dhamanpreet Kaur, Vishnu Ravi, Oliver Aalami, Roxana Daneshjou, et al. 2024. Almanac Copilot: Towards Autonomous Electronic Health Record Navigation. arXiv preprint arXiv:2405.07896 (2024)

    2024 arXiv

  58. [66]

    Charlotte Zelin, Wendy K Chung, Mederic Jeanne, Gongbo Zhang, and Chunhua Weng. 2024. Rare disease diagnosis using knowledge guided retrieval augmenta- tion for ChatGPT. Journal of Biomedical Informatics 157 (2024), 104702

    2024

  59. [67]

    Sixing Wu, Ying Li, Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2020. Diverse and informative dialogue generation with context-specific commonsense knowl- edge awareness. In Proceedings of the 58th annual meeting of the association for computational linguistics. 5811–5820

    2020

  60. [68]

    Kai Zhang, Yangyang Kang, Fubang Zhao, and Xiaozhong Liu. 2024. LLM- based Medical Assistant Personalization with Short-and Long-Term Memory Coordination. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Huma...

    2024

  61. [69]

    Ziqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers, Shao Zhang, Stephen Intille, Nawar Shara, Guodong Gordon Gao, and Dakuo Wang. 2024. Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults. Proceedings of the ACM on Interactive,...

    2024

  62. [70]

    Xuejiao Zhao. 2021. Explainable Q&A system based on domain-specific knowl- edge graph. (2021)

    2021

  63. [71]

    Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yi- chong Xu, Xiang Ren, Yiming Yang, and Michael Zeng. 2021. Kg-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering. arXiv preprint arXiv:2110.04330 (2021)

    2021 arXiv

  64. [72]

    Xuejiao Zhao, Zhenchang Xing, Muhammad Ashad Kabir, Naoya Sawada, Jing Li, and Shang-Wei Lin. 2017. Hdskg: Harvesting domain specific knowledge graph from content of webpages. In 2017 ieee 24th international conference on software analysis, evolution and reengineering (saner) ...

    2017

  65. [73]

    Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. 2023. A compre- hensive survey on automatic knowledge graph construction. Comput. Surveys 56, 4 (2023), 1–62

    2023

  66. [74]

    Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2019. Grounded conversation generation as guided traverses in commonsense knowledge graphs. arXiv preprint arXiv:1911.02707 (2019)

    2019 arXiv

  67. [76]

    Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. 2024. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473 (2024)

    2024 arXiv

  68. [78]

    Xuejiao Zhao, Huanhuan Chen, Zhenchang Xing, and Chunyan Miao. 2021. Brain- inspired search engine assistant based on knowledge graph. IEEE Transactions on Neural Networks and Learning Systems 34, 8 (2021), 4386–4400

    2021

  69. [81]

    Hao Zhou, Minlie Huang, Yong Liu, Wei Chen, and Xiaoyan Zhu. 2021. EARL: informative knowledge-grounded conversation generation with entity-agnostic representation learning. In Proceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing . 2383–2395...

    2021

  70. [82]

    has_manifestation_of

    [37], and assessed by GPT-4o [ 47], with validation by doctors. The results were: SRRAG 277, MedRAG 290 out of 360. F Ablation Study on KG Components In order to evaluate how different components in diagnostic dif- ferences KG, we conducted extra ablation study focusing on key...

  71. [2020]

    In International confer- ence on machine learning

    Retrieval augmented language model pre-training. In International confer- ence on machine learning . PMLR, 3929–3938

  72. [2022]

    Advances in neural information processing systems 35 (2022), 31306–31318

    Ddxplus: A new dataset for automatic medical diagnosis. Advances in neural information processing systems 35 (2022), 31306–31318

    2022

  73. [2023]

    arXiv preprint arXiv:2310.11511 (2023)

    Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511 (2023)

    2023 arXiv

  74. [2024]

    Quantitative Biology (2024)

    Bioinformatics and biomedical informatics with ChatGPT: Year one review. Quantitative Biology (2024)

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.