Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Combining FHIR patient data with retrieval-augmented generation lets a local language model produce personalized, guideline-grounded recommendations that outperform non-RAG medical LLMs across four guideline areas.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

FHIR-RAG-MEDS integrates HL7 FHIR patient summaries into a RAG system and reports improved guideline-based recommendation quality over bare medical LLMs across four clinical domains.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection An honest, clearly described FHIR+RAG decision-support prototype whose headline claim about FHIR's contribution is undercut by a missing no-FHIR control and self-authored human evaluation. the 5 major comments →

arxiv 2509.07706 v2 pith:BLWYVJX2 submitted 2025-09-09 cs.AI

FHIR-RAG-MEDS: Integrating HL7 FHIR with Retrieval-Augmented Large Language Models for Enhanced Medical Decision Support

Yildiray Kabak , Gokce B. Laleci Erturkmen , Mert Gencturk , Tuncay Namli , A. Anil Sinaci , Ruben Alcantud Corcoles , Cristina Gomez Ballesteros , Pedro Abizanda
show 1 more author
Asuman Dogac
This is my paper
classification cs.AI
keywords HL7 FHIRRetrieval-Augmented GenerationClinical decision supportClinical guidelinesLarge language modelsSMART on FHIRMedical summarizationMultimorbidity guidelines
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that the main weakness of medical large language models—recommendations that are generic, stale, or ungrounded—can be fixed by plugging in two existing standards: HL7 FHIR (the standard format for electronic health records) and retrieval-augmented generation (RAG), which looks up relevant guideline passages and feeds them to the model before it answers. The FHIR-RAG-MEDS system converts a patient's FHIR record into a short medical summary, uses that summary plus the clinician's question to retrieve the four most relevant chunks of a curated multi-morbidity guideline, and has a locally running 8-billion-parameter model write the response. Across the four guideline areas tested—dementia, COPD, hypertension, and sarcopenia—this grounded pipeline outperformed four medical LLMs without retrieval on text-similarity metrics, automated judge ratings, and physician scoring; the authors flag factual faithfulness on dementia questions as the weakest area. If the claim is right, guideline-based decision support can be made patient-specific and kept current by updating a document store rather than retraining a model.

Core claim

The central claim is that static medical LLMs fail on clinical questions not because they lack knowledge, but because they lack the patient's context and a grounded source to consult. FHIR-RAG-MEDS supplies both: it pulls the patient's conditions, medications, and observations from a FHIR server through SMART on FHIR, compresses them into a plain-language summary with a small open model, retrieves the closest guideline passages from a vector store, and only then asks the model to answer. Based on roughly 70 physician-written questions and answers, the system scored higher than the four non-RAG comparison models on semantic and lexical alignment metrics and on an LLM-based judge across dement

What carries the argument

The load-bearing mechanism is a five-step pipeline: authenticate to a FHIR server via SMART on FHIR; fetch the patient's recent condition, medication, and observation resources as a FHIR bundle; prompt an 8-billion-parameter open model to convert that JSON into a short medical summary; embed the summary plus the clinician's question and retrieve the four nearest guideline chunks from a cosine-similarity vector store; then generate the final recommendation from the retrieved chunks plus the summary. The retrieval chunks are what ground the answer in evidence, and the FHIR summary is what personalizes it.

Load-bearing premise

The load-bearing premise is that the prompt-based conversion of a patient's FHIR JSON into a short text summary (Section 3.1.3) keeps every clinically relevant fact, and the paper reports only informal testing of that conversion, not a quantitative fidelity check.

What would settle it

Run the same roughly 70 guideline questions with the same patient bundles twice—once with the summary pipeline and once with the full FHIR JSON given directly to the generator—and have physicians blind-rate the pairs; if the summary version is not consistently better, or if an entity-level check shows the summaries drop medications or conditions, the personalization claim collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Clinicians can type patient-specific questions and get answers drawn from vetted guideline text rather than the model's memory.
  • The same application can plug into any SMART-on-FHIR-compliant hospital system without custom interfaces.
  • Guideline updates take effect by re-ingesting documents into the vector store, not by retraining or fine-tuning the model.
  • The evaluation template—similarity metrics, automated judge, and a small physician panel—can be reused to benchmark other guideline-query systems.
  • Keeping the generator on a local server lets health data stay inside the institution's network.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our reading: the experiments use the CAREPATH consensus guideline and curated FHIR bundles, not live records, so the real-world gain will depend on how well the FHIR-to-summary step survives messy, incomplete EHR data.
  • A testable extension: compare answers generated from raw FHIR JSON against answers from the prompted summary for the same patient; if a physician panel finds no consistent difference, the summarizer is not the bottleneck.
  • The same architecture is domain-agnostic: replacing the guideline corpus and FHIR server should let it cover other specialties, with chunk size and prompts as the main tuning knobs.
  • If the strong automated-to-human correlation replicates, LLM-based judges could be used to screen candidate prompts and retrievers cheaply, with physician review reserved for final sign-off.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes FHIR-RAG-MEDS, a retrieval-augmented generation system that combines HL7 FHIR patient data with a vector store of cleaned clinical guidelines to produce personalized, evidence-based recommendations. The system is implemented with Llama 3.1 8B, SMART on FHIR for data access, LangChain/Chroma for chunking and retrieval, and is evaluated on four CAREPATH guideline areas: dementia, COPD, hypertension, and sarcopenia. The evaluation compares FHIR-RAG-MEDS against Meditron, OpenBioLLM, BioMistral, and vanilla Llama 3.1 8B using BERTScore, ROUGE-L, METEOR, Prometheus 2, and RAGAS, plus a physician rating study. The paper reports that FHIR-RAG-MEDS outperforms the non-RAG baselines across most metrics, with the exception of a low RAGAS faithfulness score for dementia (0.481).

Significance. If the claims were fully supported, the paper would provide a useful practical architecture for standards-based, patient-specific clinical decision support using locally hosted open-source LLMs. The system design is sensible, the guideline coverage is broad, and the evaluation uses several complementary metrics. The authors are also transparent in reporting a weak faithfulness score for dementia instead of cherry-picking favorable results. However, the central contribution—the benefit of FHIR integration—remains untested because the paper lacks a RAG-without-FHIR control. In addition, the ground truth and human scoring are entangled with the authorship of the paper, weakening the validity of the evaluation. The significance is therefore conditional: the architecture is promising, but the evidence presented does not yet establish that FHIR integration, rather than RAG in general, is responsible for the observed improvements.

major comments (5)
  1. [§4, Tables 3, 5, 6, 7] The central claim that FHIR integration drives the improvement is not supported by the reported comparisons. All baselines (BioMistral, Llama 3.1 8B, Meditron, OpenBioLLM) are non-RAG LLMs; there is no RAG control that retrieves guideline chunks but omits the FHIR patient summary. The observed gains could therefore be due entirely to standard RAG grounding in the guideline text, independent of patient-specific data. A comparison against FHIR-RAG-MEDS with the FHIR summary removed (or against a RAG pipeline using non-FHIR patient context) is necessary to isolate the contribution of HL7 FHIR.
  2. [§3.2, §4.5, §6.1, §6.2] The reference answers and the human evaluation are not independent of the authors. §6.1 states that the evaluating medical doctor participants 'are also the co-authors of this submission,' yet §4.5 describes the physicians as 'three independent physicians.' This circularity affects both the ground truth used for automated metrics and the claimed 'rigorous human assessment.' The authors should either obtain evaluation from external clinicians or explicitly reframe the human study as a pilot with internal reviewers, reporting the conflict of interest and tempering the claims accordingly.
  3. [§3.1.3] The conversion of FHIR JSON to a textual medical summary using Llama 3.1 8B is a critical transformation: the summary is the only patient-specific representation fed into retrieval and generation. The paper justifies this choice with 'testing with an example FHIR bundle set' and provides no quantitative fidelity check. If the summary drops, alters, or hallucinates conditions, medications, or observations, the system will retrieve on a corrupted patient representation. The authors should report precision/recall or an expert-verified agreement rate for the summarization, or at minimum provide a detailed error analysis on a held-out set of FHIR bundles.
  4. [§4, Tables 3–7] The paper repeatedly uses the word 'significant' (e.g., 'significant improvements,' 'significantly higher') but reports only point estimates without confidence intervals, standard deviations, or hypothesis tests. The number of questions per guideline is not given, and with approximately 70 total questions split across four guidelines, the reported score gaps may not be statistically reliable. The authors should report per-guideline N, variance, and appropriate statistical comparisons (e.g., paired bootstrapping or permutation tests) to support the strength of the claims.
  5. [§4.1, Table 4] The dementia faithfulness score of 0.481 is below 0.5, and the text in §4.1 first calls the RAGAS performance 'notably strong' before conceding that faithfulness 'could improve.' For a clinical decision support system, a faithfulness score below 0.5 is a material limitation that should be prominently discussed, especially since the paper argues that RAG reduces hallucination. The authors should analyze why faithfulness is low for dementia and explain what this implies for clinical deployment, rather than presenting the overall result as uniformly strong.
minor comments (6)
  1. [§3, first paragraph] Typo: 'we have chose n' should read 'we have chosen.'
  2. [Table 3 header] The column header 'A VERAGE SCORE' contains an unintended space; also 'LLM NAME' formatting is inconsistent.
  3. [§3.1.3] Grammar: 'with the a prompt template' should be 'with a prompt template.'
  4. [References] Reference [27], LangChain, is dated '2004'; this is almost certainly wrong and should be corrected.
  5. [§4.5 / §6.3] The paper refers to supplementary material containing the full list of questions and physician responses, but no supplementary file is included with this manuscript. Please clarify availability.
  6. [§6.1] The statement 'This study did not involve real patient data' is useful but should be clarified: the FHIR server and example bundles appear to contain synthetic or simulated data, and this should be stated explicitly in the methods.

Circularity Check

1 steps flagged

Evaluation loop is closed: ground-truth answers and human expert judges are authored by the same physician co-authors who produced the underlying CAREPATH guideline.

specific steps
  1. self citation load bearing [Section 3.2, Section 4.5, Section 6.1]
    "we tasked our evaluation panel of physicians to generate questions and answers related to their daily clinical practices. ... In total, we compiled approximately 70 questions and answers, regarded as ground truths. ... we conducted a human evaluation study involving three independent physicians, who are geriatricians and experts in multimorbidity in older adults with dementia. ... Informed consent was obtained from the evaluating medical doctor participants, who are also the co-authors of this submission, involved in the study."

    The ground-truth reference answers used for automated scoring were written by the same physician group that authored the CAREPATH consolidated guideline and this article; the 'independent physicians' who provide the human-expert validation are, per Section 6.1, also co-authors. The evaluation therefore compares FHIR-RAG-MEDS outputs against a gold standard and expert judgments that originate from the same authors who supplied the system's guideline knowledge base. High metric scores and high human scores are not an independent confirmation of clinical quality; they can reflect the authors' prior commitment to their own guideline and system. This is a load-bearing self-referential validation loop rather than a derivation, so it is scored as partial circularity.

full rationale

The paper is an empirical system paper, not a mathematical derivation: there are no fitted parameters and no equation-level reduction of a prediction to an input. The head-to-head comparisons against public LLMs (Meditron, BioMistral, OpenBioLLM, Llama 3.1) provide some independent reference points, and RAGAS/Prometheus are external evaluation tools. However, the evaluation's gold standard is not external: the ~70 'ground truth' question-answer pairs were written by the physician-panel co-authors (Section 3.2), and the 'three independent physicians' of Section 4.5 are the same co-authors per Section 6.1. Since the RAG system retrieves from the CAREPATH consolidated guideline produced by these same clinical partners, the scoring loop is closed: the reference answers, the retrieved knowledge, and the human judges all come from one authorial group. This is a load-bearing self-validation, not a mathematical circularity, so it is scored at 4 rather than higher. Separately, the paper does not include a RAG-without-FHIR condition, so the unique contribution of the FHIR patient summary over ordinary RAG is untested; that is an experimental-design gap, not a circularity, and is not counted in the score.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's central claims rest on hand-chosen RAG parameters (chunk size, overlap, k, embedding model, generation settings) and on several unvalidated domain assumptions, chiefly the fidelity of the FHIR-to-text summarization and the correctness of the guideline. No new physical or conceptual entities are introduced.

free parameters (5)
  • chunk_size = 1200 units
    Chosen by hand for splitting guideline text; affects retrieval granularity and the quality of generated answers.
  • chunk_overlap = 100 units
    Chosen alongside chunk size; influences context continuity across chunks.
  • k = 4
    Number of retrieved guideline chunks per query; set to 4 without reported tuning or sensitivity analysis.
  • embedding_model = mxbai-embed-large
    Selected from Ollama; the embedding space determines retriever relevance.
  • LLM generation parameters (temperature, sampling)
    Not reported; default Ollama settings were likely used. These affect output variability and reproducibility.
axioms (4)
  • domain assumption The CAREPATH consolidated guideline accurately represents evidence-based recommendations for dementia, COPD, hypertension, and sarcopenia.
    The system's recommendations are grounded entirely in this guideline. The guideline was built by the project's Clinical Reference Group and is cited as external, but its clinical correctness is not independently validated in this paper.
  • domain assumption The Llama 3.1 prompt-based conversion of FHIR JSON into a text summary preserves clinically relevant facts.
    Section 3.1.3 states that testing with an example FHIR bundle set demonstrated good performance, but no quantitative fidelity check is provided. If the summary omits or alters conditions, medications, or observations, the personalized recommendation is compromised.
  • domain assumption Cosine similarity over mxbai-embed-large embeddings retrieves the guideline chunks that are semantically most relevant to the patient summary and query.
    The RAG retriever is assumed to find the right evidence. RAGAS context precision and recall scores partially support this, but those scores are themselves LLM-based estimates.
  • domain assumption The test data used in the case study is representative of real FHIR patient records.
    The paper states that the study did not involve real patient data (Section 6.1). The FHIR bundles used appear to be synthetic, so real-world data variability, missingness, and coding errors are not tested.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of FHIR-RAG-MEDS: Integrating HL7 FHIR with Retrieval-Augmented Large Language Models for Enhanced Medical Decision Support." pith.science (2026). https://pith.science/paper/BLWYVJX2

@misc{pith2026250907706,
  author       = {Pith},
  title        = {Pith review of: FHIR-RAG-MEDS: Integrating HL7 FHIR with Retrieval-Augmented Large Language Models for Enhanced Medical Decision Support},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLWYVJX2}},
  note         = {Machine review of arXiv:2509.07706}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In this study, we propose FHIR-RAG-MEDS system that aims to integrate Health Level 7 Fast Healthcare Interoperability Resources (HL7 FHIR) with a Retrieval-Augmented Generation (RAG)-based system to improve personalized medical decision support on evidence-based clinical guidelines, emphasizing the need for research in practical applications. In the evolving landscape of medical decision support systems, integrating advanced technologies such as RAG and HL7 FHIR can significantly enhance clinical decision-making processes. Despite the potential of these technologies, there is limited research on their integration in practical applications.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Medi-Gemma: A Hybrid Clinical Decision Support System Integrating Deterministic EMR Analytics and Retrieval-Augmented Generation

    cs.AI 2026-07 conditional novelty 4.5

    A modular CDSS architecture separates deterministic EMR analytics from generative reasoning and injects latest patient ground-truth into LLM prompts for safer wound-care support.

Reference graph

Works this paper leans on

56 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    This can lead to the generation of incorrect or misleading information [7], especially in rapidly changing medical fields

    Accuracy and Trustworthiness: Medical LLMs rely solely on their training data, which may become outdated or inaccurate as medical knowledge evolves [6]. This can lead to the generation of incorrect or misleading information [7], especially in rapidly changing medical fields. RAG applications, on the other hand, are able to retrieve answers from validated ...

    work page

  2. [2]

    The 4 lack of transparency can be a significant issue in clinical environments [10]

    Evidence-Based Responses: Medical LLMs often generate responses without directly citing their sources, making it harder for clinicians to trust the recommendations without external verification. The 4 lack of transparency can be a significant issue in clinical environments [10]. RAG systems combine Generative Artificial Intelligence ( AI) with retrieved e...

    work page

  3. [3]

    RAG systems provide greater flexibility by combining generative capabilities with retrieval from diverse and specialized datasets

    Scalability and Flexibility: Medical LLMs require extensive training data for each medical field or domain, which may not always be feasible and can result in inefficiencies when handling cross-domain knowledge. RAG systems provide greater flexibility by combining generative capabilities with retrieval from diverse and specialized datasets. This allows th...

    work page

  4. [4]

    Reduced Computational Costs: Medical LLMs require heavy computational resources to generate responses, particularly as model sizes increase, leading to slower response times and higher infrastructure costs [12]. RAG systems reduce computational burdens by retrieving relevant information from external databases before generating responses, which can be mor...

    work page

  5. [5]

    You should refer to the score rubric

    work page

  6. [6]

    This ensures that sensitive medical information remains protected while maintaining seamless usability

    Authentication and Authorization : The system employs OAuth 2.0 protocols, as defined by SMART on FHIR, to securely authenticate users and authorize access to patient data. This ensures that sensitive medical information remains protected while maintaining seamless usability

    work page

  7. [7]

    All recent condition, medication, and observation resources are retrieved individually and then combined into a FHIR bundle for interpretation by the FHIR-RAG-MEDS system

    Data Retrieval and Management : The integration enables real -time retrieval of patient information, such as demographics, medications, conditions, and observations, from any compliant FHIR server. All recent condition, medication, and observation resources are retrieved individually and then combined into a FHIR bundle for interpretation by the FHIR-RAG-...

    work page

  8. [8]

    The system’s query endpoints provide evidence-based clinical suggestions tailored to the patient’s specific medical context

    Interoperable Query Processing: Once patient data is retrieved and summarized, it is sent to the FHIR-RAG-MEDS system for interpretation and recommendation generation as explained in the next section. The system’s query endpoints provide evidence-based clinical suggestions tailored to the patient’s specific medical context

    work page

  9. [9]

    I have a patient with diabetes and hypertension. What should be the initial drug therapy?

    Extensibility and Compatibility: The SMART on FHIR integration is designed to be extensible, allowing it to support additional use cases, such as integration with third -party applications, EHR systems, and telehealth platforms. By adhering to open standards, it ensures compatibility acr oss diverse healthcare ecosystems. 3.1.3 Processing FHIR Bundles Whe...

    work page

  10. [10]

    Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation,

    J. Z. Y . Q. Junde Wu, “Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation,” ArXiv, 2024, doi: https://doi.org/10.48550/arXiv.2408.04187

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.04187 2024

  11. [11]

    Write detailed feedback that assesses the quality of the response strictly based on the given score rubric, not evaluating in general

    work page

  12. [12]

    After writing a feedback, write a score that is an integer between 1 and

    work page

  13. [13]

    Large Language Models in Healthcare and Medical Domain: A Review,

    Z. Al Nazi and W. Peng, “Large Language Models in Healthcare and Medical Domain: A Review,” Informatics, vol. 11, no. 3, p. 57, Aug. 2024, doi: 10.3390/informatics11030057

    doi:10.3390/informatics11030057 2024

  14. [14]

    Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}

    The output format should look as follows: "Feedback: {{write a feedback for criteria}} [RESULT] {{an integer number between 1 and 5}}"

    work page

  15. [15]

    Be sure to include [RESULT] in your output

    Please do not generate any other opening, closing, and explanations. Be sure to include [RESULT] in your output. ###The instruction to evaluate: {instruction} ###Response to evaluate: {response} ###Reference Answer (Score 5): {reference_answer} ###Score Rubrics: [Is the response correct, accurate, and factual based on the reference answer?] Score 1: The r...

    work page

  16. [16]

    However, the context utilization score of 1.0 highlights its capability to rank relevant information effectively

    These metrics reveal that the FHIR -RAG-MEDS system excels in answer similarity and context precision but could improve in faithfulness, as it shows some gaps in factual correctness with a score of 0.481. However, the context utilization score of 1.0 highlights its capability to rank relevant information effectively. Table 4 – The RAGAS metrics of FHIR-RA...

    work page

  17. [17]

    lacking specific actionable insights,

    The faithfulness score of 0.725 is considerably higher than in the dementia guideline, suggesting better factual accuracy in the COPD domain. The context recall is slightly lower (0.792), but the answer correctness and similarity are strong at 0.908 and 0.930, respectively . This indicates that the responses generated are both semantically and factually a...

    work page 2020

  18. [18]

    MEDITRON-70B: Scaling Medical Pretraining for Large Language Models,

    A. H. C. et. al. Zeming Chen, “MEDITRON-70B: Scaling Medical Pretraining for Large Language Models,” ArXiv, vol. abs/2311.16079, 2023, Accessed: Oct. 08, 2024. [Online]. Available: 10.48550/arXiv.2311.16079

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.16079 2023

  19. [19]

    BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains,

    A. B. E. M. P.-A. G. M. R. R. D. Yanis Labrak, “BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains,” in ACL 2024 - Proceedings of the 62st Annual Meeting of the Association for Computational Linguistics, 2024. doi: https://doi.org/10.48550/arXiv.2402.10373

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.10373 2024

  20. [20]

    Open Source Biomedical Large Language Model,

    “Open Source Biomedical Large Language Model,” 2024. Accessed: Oct. 09, 2024. [Online]. Available: https://huggingface.co/aaditya/Llama3-OpenBioLLM-70B

    work page 2024

  21. [21]

    Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks,

    E. P. A. P. F. P. V . K. N. G. H. K. M. L. W. Y . T. R. S. R. D. K. Patrick Lewis, “Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks,” ArXiv, 2021, doi: https://doi.org/10.48550/arXiv.2005.11401

    Pith/arXiv arXiv doi:10.48550/arxiv.2005.11401 2021

  22. [22]

    Integrating Retrieval-Augmented Generation with Large Language Models in Nephrology: 28 Advancing Practical Applications,

    J. Miao, C. Thongprayoon, S. Suppadungsuk, O. A. Garcia Valencia, and W. Cheungpasitporn, “Integrating Retrieval-Augmented Generation with Large Language Models in Nephrology: 28 Advancing Practical Applications,” Medicina (B Aires), vol. 60, no. 3, p. 445, Mar. 2024, doi: 10.3390/medicina60030445

    work page doi:10.3390/medicina60030445 2024

  23. [23]

    mxbai- embed-large

    which aims to deliver a patient -centered integrated care platform to meet the needs of older patients with multimorbidity, including mild cognitive impairment (MCI) or mild dementia (MD) , based on the recommendations of evidence -based guidelines. In the CAREPATH project, a Clinical Reference Group (CRG), formed by the project’s clinical partners, has a...

    work page

  24. [24]

    Opportunities and challenges for ChatGPT and large language models in biomedicine and health,

    S. Tian et al., “Opportunities and challenges for ChatGPT and large language models in biomedicine and health,” Brief Bioinform, vol. 25, no. 1, Nov. 2023, doi: 10.1093/bib/bbad493

    work page doi:10.1093/bib/bbad493 2023

  25. [25]

    Survey of Hallucination in Natural Language Generation,

    Z. Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Comput Surv, vol. 55, no. 12, pp. 1–38, Dec. 2023, doi: 10.1145/3571730

    doi:10.1145/3571730 2023

  26. [26]

    Retrieval-Augmented Generation for Large Language Models: A Survey,

    Y . X. X. G. K. J. J. P. Y . B. Y . D. J. S. M. W. H. W. Yunfan Gao, “Retrieval-Augmented Generation for Large Language Models: A Survey,” ArXiv, 2024, doi: https://doi.org/10.48550/arXiv.2312.10997

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.10997 2024

  27. [27]

    Retrieval-Augmented Generation for AI-Generated Content: A Survey,

    H. Z. Q. Y . Z. W. Y . G. F. F. L. Y . W. Z. J. J. B. C. Penghao Zhao, “Retrieval-Augmented Generation for AI-Generated Content: A Survey,” ArXiv, 2024, doi: https://doi.org/10.48550/arXiv.2402.19473

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.19473 2024

  28. [28]

    Almanac — Retrieval-Augmented Language Models for Clinical Medicine,

    C. Zakka et al., “Almanac — Retrieval-Augmented Language Models for Clinical Medicine,” NEJM AI, vol. 1, no. 2, Jan. 2024, doi: 10.1056/AIoa2300068

    doi:10.1056/aioa2300068 2024

  29. [29]

    Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report,

    L. J. K. E. H. R. A. N. L. A. T. H. S. C. R. S. J. Y . M. T. J. C. L. O. D. S. W. T. YuHe Ke, “Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report,” ArXiv, 2024, doi: https://doi.org/10.48550/arXiv.2402.01733

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.01733 2024

  30. [30]

    ROUGE: A Package for Automatic Evaluation of summaries,

    Chin-Yew Lin, “ROUGE: A Package for Automatic Evaluation of summaries,” in In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), 2004

    work page 2004

  31. [31]

    LLM-as-a- Judge

    and BERTScore [32]. In text comparisons, BERTScore is used to evaluate semantic accuracy, while ROUGE/METEOR are used to assess word-level matching. Specifically: • BERTScore can be the primary metric because, in healthcare, capturing the correct meaning of an answer is more important than exact word matches. • ROUGE can be used to ensure that the model c...

    work page

  32. [32]

    A scoping review of using Large Language Models (LLMs) to investigate Electronic Health Records (EHRs),

    J. Z. Z. G. W. H. L. F. H. Y . L. H. Y . Z. T. L. A. L. H. S. M. Lingyao Li, “A scoping review of using Large Language Models (LLMs) to investigate Electronic Health Records (EHRs),” ArXiv, 2024, doi: https://doi.org/10.48550/arXiv.2405.03066

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.03066 2024

  33. [33]

    LLM on FHIR -- Demystifying Health Records,

    A. R. P. Z. V . R. A. Z. A. F. O. A. Paul Schmiedmayer, “LLM on FHIR -- Demystifying Health Records,” ArXiv, 2024, doi: https://doi.org/10.48550/arXiv.2402.01711

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.01711 2024

  34. [34]

    FHIR-GPT Enhances Health Interoperability with Large Language Models,

    Y . Li, H. Wang, H. Z. Yerebakan, Y . Shinagawa, and Y . Luo, “FHIR-GPT Enhances Health Interoperability with Large Language Models,” NEJM AI, vol. 1, no. 8, Jul. 2024, doi: 10.1056/AIcs2300301

    doi:10.1056/aics2300301 2024

  35. [35]

    Speaking the Same Language: Leveraging LLMs in Standardizing Clinical Data for AI,

    A. Sett, S. Hashemifar, M. Yadav, Y . Pandit, and M. Hejrati, “Speaking the Same Language: Leveraging LLMs in Standardizing Clinical Data for AI,” Aug. 2024

    work page 2024

  36. [36]

    Applying generative AI with retrieval augmented generation to summarize and extract key clinical information from electronic health records,

    M. Alkhalaf, P. Yu, M. Yin, and C. Deng, “Applying generative AI with retrieval augmented generation to summarize and extract key clinical information from electronic health records,” J Biomed Inform, vol. 156, p. 104662, Aug. 2024, doi: 10.1016/j.jbi.2024.104662

    work page arXiv 2024

  37. [37]

    Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework,

    S. Kresevic, M. Giuffrè, M. Ajcevic, A. Accardo, L. S. Crocè, and D. L. Shung, “Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework,” NPJ Digit Med, vol. 7, no. 1, p. 102, Apr. 2024, doi: 10.1038/s41746-024-01091-y. 29

    work page doi:10.1038/s41746-024-01091-y 2024

  38. [38]

    Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening,

    O. Unlu et al., “Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening,” NEJM AI, vol. 1, no. 7, Jun. 2024, doi: 10.1056/AIoa2400181

    work page doi:10.1056/aioa2400181 2024

  39. [39]

    Benchmarking Retrieval-Augmented Generation for Medicine,

    G. Xiong, Q. Jin, Z. Lu, and A. Zhang, “Benchmarking Retrieval-Augmented Generation for Medicine,” Feb. 2024

    work page 2024

  40. [40]

    CAREPATH Project Website

    “CAREPATH Project Website.” Accessed: Oct. 09, 2024. [Online]. Available: https://www.carepath.care/

    work page 2024

  41. [41]

    CAREPATH: developing digital integrated care solutions for multimorbid patients with dementia,

    O. Pournik et al., “CAREPATH: developing digital integrated care solutions for multimorbid patients with dementia,” in Advances in Informatics, Management and Technology in Healthcare, IOS Press, 2022, pp. 487-490. doi: 10.3233/SHTI220771

    work page doi:10.3233/shti220771 2022

  42. [42]

    Protocol for Creating a Single, Holistic and Digitally Implementable Consensus Clinical Guideline for Multiple Multi-morbid Conditions,

    T. D. Robbins et al., “Protocol for Creating a Single, Holistic and Digitally Implementable Consensus Clinical Guideline for Multiple Multi-morbid Conditions,” in Proceedings of the 10th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, New York, NY , USA: ACM, Aug. 2022, pp. 1–6. do...

    work page arXiv 2022

  43. [43]

    Transforming evidence-based clinical guidelines into implementable clinical decision support services: the CAREPATH study for multimorbidity management,

    M. Gencturk et al., “Transforming evidence-based clinical guidelines into implementable clinical decision support services: the CAREPATH study for multimorbidity management,” Front Med (Lausanne), vol. 11, May 2024. doi: 10.3389/fmed.2024.1386689

    work page arXiv 2024

  44. [44]

    SMART on FHIR: a standards-based, interoperable apps platform for electronic health records,

    J. C. Mandel et al., “SMART on FHIR: a standards-based, interoperable apps platform for electronic health records,” Journal of the American Medical Informatics Association, vol. 23, 2016, pp. 899-908, doi: 10.1093/jamia/ocv189

    doi:10.1093/jamia/ocv189 2016

  45. [45]

    LangChain Open Source Library,

    “LangChain Open Source Library,” 2004. Accessed: Oct. 09, 2024. [Online]. Available: https://www.langchain.com

    work page 2004

  46. [46]

    Chroma Open Source AI Application Database,

    “Chroma Open Source AI Application Database,” 2024. Accessed: Oct. 09, 2024. [Online]. Available: https://www.trychroma.com

    work page 2024

  47. [47]

    Llama 3.1 8B LLM,

    “Llama 3.1 8B LLM,” 2024. Accessed: Oct. 09, 2024. [Online]. Available: https://ai.meta.com/blog/meta-llama-3-1/

    work page 2024

  48. [48]

    METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments,

    A. L. Satanjeev Banerjee, “METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005

    work page 2005

  49. [49]

    BERTScore: Evaluating Text Generation with BERT,

    V . K. F. W. K. Q. W. Y . A. Tianyi Zhang, “BERTScore: Evaluating Text Generation with BERT,” in International Conference on Learning Representations, 2020. doi: https://doi.org/10.48550/arXiv.1904.09675

    Pith/arXiv arXiv doi:10.48550/arxiv.1904.09675 2020

  50. [50]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, Morristown, NJ, USA: Association for Computational Linguistics, 2001, p. 311. doi: 10.3115/1073083.1073135

    arXiv 2001

  51. [51]

    Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)

    Z. Yan, “Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge).” Accessed: Oct. 09, 2024. [Online]. Available: https://eugeneyan.com/writing/llm-evaluators/ 30

    work page 2024

  52. [52]

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models,

    J. S. S. L. B. Y . L. J. S. S. W. G. N. M. L. K. L. M. S. Seungone Kim, “Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models,” ArXiv, 2024, doi: https://doi.org/10.48550/arXiv.2405.01535

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.01535 2024

  53. [53]

    RAGAS Evaluation Framework,

    “RAGAS Evaluation Framework,” 2024. Accessed: Oct. 09, 2024. [Online]. Available: https://docs.ragas.io/en/stable/

    work page 2024

  54. [54]

    Artificial intelligence‐ based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance,

    G. Mahadevaiah, P. RV , I. Bermejo, D. Jaffray, A. Dekker, and L. Wee, “Artificial intelligence‐ based clinical decision support in modern medical physics: Selection, acceptance, commissioning, and quality assurance,” Med Phys, vol. 47, no. 5, May 2020, doi: 10.1002/mp.13562

    work page doi:10.1002/mp.13562 2020

  55. [55]

    Applying artificial intelligence to clinical decision support in mental health: What have we learned?,

    G. Golden et al., “Applying artificial intelligence to clinical decision support in mental health: What have we learned?,” Health Policy Technol, vol. 13, no. 2, p. 100844, Jun. 2024, doi: 10.1016/j.hlpt.2024.100844

    work page arXiv 2024

  56. [56]

    Understanding the effects of rlhf on llm generalisation and diversity

    R. Kirk et al., “Understanding the effects of rlhf on llm generalisation and diversity”. arXiv preprint arXiv:2310.06452. 2023 Oct 10. List of Abbreviations: AI: Artificial Intelligence BLEU: Bilingual Evaluation Understudy CDSS: Clinical Decision Support Systems COPD: Chronic Obstructive Pulmonary Disease CRG: Clinical Reference Group EC: European Commis...

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.