REVIEW 4 major objections 6 minor 3 cited by
CLI-RAG: A Retrieval-Augmented Framework for Clinically Structured and Context Aware Text Generation with LLMs
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A clinically structured retrieval-augmented generation framework claims that LLM-written progress notes can be more temporally coherent than the human notes they replace, reporting a 7-point alignment advantage over clinician-authored…
desk verdict A plausible RAG pipeline for progress-note generation, but the headline temporal-coherence claim is contradicted by the paper's own table and rests on a circular embedding-based metric. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the pair of retrieval stages. Global retrieval answers task-driven clinical questions across all note types; local retrieval then drills into specific note types with tailored questions. Retrieved chunks are deduplicated, reranked by a hybrid of BM25 and cosine similarity, annotated with metadata (note type, section, chunk order, visit date), and assembled into prompts that include a summary of the prior visit when one exists. The same embedding model used for retrieval (all-mpnet-base-v2) also computes the semantic and temporal similarity scores.
What would settle it
Regenerate the notes after swapping the retrieval embedding model (e.g., to a clinical BioBERT or a GPT-based embedder) and evaluate the temporal alignment with a third, independent embedder. If the 87.7% vs 80.7% margin does not survive, the claim depends on the metric's self-selection rather than on genuine temporal reasoning.
Extended reading notes
Core claim
The central claim is that a retrieval pipeline that is 'clinically informed'—hierarchically chunked, query-driven at both note-type and section level, and conditioned on prior-visit summaries—can generate structured progress notes that are semantically faithful and temporally more coherent than human-authored notes. The headline result is an average temporal alignment score of 87.7 percent from generated notes versus 80.7 percent from the gold clinician notes, computed as the mean cosine similarity between embeddings of adjacent notes in a patient trajectory.
Load-bearing premise
The entire headline result rests on the assumption that temporal coherence is correctly captured by cosine similarity between adjacent note embeddings, where the same embedding model that picks the content also scores the output; higher similarity could reflect copy-heavy or repetitive generation rather than better clinical storytelling.
Editorial extensions
If this is right
- Progress notes that are absent from a patient record (only 8.56% of MIMIC-III visits have them) can be back-filled from contemporaneous notes of other types.
- The framework transfers across LLMs: LLaMA-3 70B and Mistral-7B both produce full SOAP coverage and high alignment, so structure and retrieval do the work, not scale alone.
- Because outputs are traceable to source chunks, clinicians could audit generated notes against evidence.
- The same pipeline can support synthetic longitudinal EHR generation and documentation-drafting assistance.
Reading between the lines
- Because the temporal alignment score is computed with the same embedding model that selected the retrieved chunks, part of the 87.7% advantage may reflect the generator repeating retrieved language rather than true clinical continuity; an independent human or model-based review would be needed to separate those.
- If the alignment metric were swapped for a different embedder or for a repetition-aware measure (e.g., n-gram overlap between adjacent notes), the reported margin over clinician notes could shrink; that would be a direct test of the claim.
- The framework suggests a general design rule for structured summarization: make the chunking follow the document's own sections and let retrieval ask task-specific questions, rather than splitting text into fixed windows.
- The paper's filtering of visits to 56 patients with 10–57 visits means the temporal-coherence result applies to patients with dense admission histories; sparse trajectories remain untested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CLI-RAG, a retrieval-augmented framework for generating SOAP-format progress notes from heterogeneous electronic health record notes in MIMIC-III. The method combines hierarchical chunking that respects clinical document structure, a dual-stage retrieval mechanism (global question-driven retrieval across all note types, followed by note-type-specific local retrieval), and longitudinal prompting with summaries of prior visits. The system is evaluated with LLaMA-3 70B and Mistral-7B on 1,108 hospital visits across lexical, semantic, structural, length, temporal, and preference-based metrics. The headline claim is that generated notes preserve temporal and semantic alignment across visits, achieving an average alignment score of 87.7%, surpassing the 80.7% baseline from real clinician-authored notes.
Significance. If the central empirical claim were valid, the paper would be a useful contribution to clinical NLP: it addresses a real problem, uses a standard public corpus, makes retrieval evidence traceable to source chunks, and evaluates on multiple dimensions with two open-source LLMs. The hierarchical chunking and dual-stage retrieval design are clearly described and are plausible engineering contributions. However, the headline claim is not established by the evidence in the manuscript. The temporal consistency metric is unvalidated and is computed with the same embedding model used for retrieval, the gold baseline is inconsistent across models, and the preference voting lacks critical details. The authors themselves acknowledge that human-in-the-loop evaluation is essential. As presented, the results may reflect artifacts of the evaluation design rather than clinically meaningful improvements, so the significance of the contribution is currently limited.
major comments (4)
- [§3.2, Table 1] Temporal Consistency (Gold) is reported as 0.807 for LLaMA-3 but 0.9126 for Mistral-7B, even though the gold notes are the same fixed MIMIC-III corpus. This internal inconsistency makes the abstract's comparison '87.7% vs 80.7%' ill-defined. Only LLaMA-3's generated value (0.877) exceeds its gold baseline; Mistral-7B's generated value (0.8248) is below its gold baseline (0.9126). The claim that generated notes 'preserve temporal alignment' and surpass clinician-authored notes is therefore supported only by a selective reading of one model's results.
- [§3.2, Temporal Coherence; §2.1] The temporal consistency metric averages cosine similarity between adjacent note embeddings using all-mpnet-base-v2, the same sentence transformer used in §2.1 to embed and retrieve the chunks that are inserted into the prompt. This is a circular evaluation for the generated notes, since the generated text is assembled from chunks selected by that embedding model. Additionally, high cosine similarity can reflect repetitive or copy-heavy text rather than clinically meaningful temporal coherence, and the paper provides no human or clinical validation that this metric tracks clinician judgment. Section 5 explicitly states that 'human-in-the-loop evaluation will be essential,' which undermines the confidence in the central claim.
- [§2.2-§2.3, Table 1] The pipeline contains numerous hand-crafted free parameters: the global clinical question set, the per-note-type local question sets, the symptom keyword heuristics, the BM25/cosine hybrid weight, the deduplication cosine threshold, and the chunk size/overlap settings. No ablation or sensitivity analysis is reported for any of these choices. As a result, the observed differences cannot be attributed to the 'clinically informed' retrieval design rather than to specific choices of these parameters, and the generalizability of the method is unassessed.
- [§3.2, LLM Preference Voting] The preference voting is reported only as aggregate percentages (96% for LLaMA-3, 93.5% for Mistral-7B), without specifying how many notes were rated, how cases were sampled, what prompt was used, or whether the judge was the same Mistral-7B model that generated one of the outputs. This matters because Mistral-7B generated notes have lower temporal consistency (0.8248) than gold notes (0.9126), yet the voting claims generated notes were preferred over clinician notes in 93.5% of cases. The tension between the preference result and the temporal metric is unresolved, and self-preference or formatting bias is not ruled out.
minor comments (6)
- [Table 1] The table contains a typo, 'LLM V oting Preference,' and the 'Length Ratio' is listed as N/A for Mistral-7B without explanation.
- [Abstract] The phrase 'average alignment score of 87.7%' is not explicitly defined; it corresponds to LLaMA-3's temporal consistency only, not an average across the two models or across all alignment metrics.
- [§3.3] The statement that 'both models outperformed real notes in longitudinal coherence' is contradicted by Table 1, where Mistral-7B generated temporal consistency (0.8248) is below its gold baseline (0.9126).
- [§2.3] There is a typo, 'sytem,' and the reference to 'Appendix 2' should likely be to Figure 2.
- [§3.2, Semantic Alignment] Semantic similarity is computed with the same embedding model used for retrieval; the paper should report a reference baseline (e.g., gold-vs-gold similarity) so readers can calibrate the generated-vs-gold values.
- [§3.1-3.2] No confidence intervals, standard errors, or statistical significance tests are reported for the metric differences, so it is unclear whether values such as 0.877 versus 0.807 reflect meaningful differences or noise.
Circularity Check
No construction-level circularity: the temporal-coherence headline is a measured outcome, not a fitted or definitional identity; the same-embedding metric and inconsistent gold baseline are evaluation validity risks, not circular derivation steps.
full rationale
CLI-RAG's derivation chain is empirical rather than definitional. The system embeds chunks with all-mpnet-base-v2 for retrieval (§2.1), assembles prompts from retrieved chunks, and generates SOAP notes with an LLM. Temporal coherence is then measured as the average cosine similarity between adjacent generated notes using the same embedding model (§3.2). Although this creates a shared-representation concern, the measurement is a post-hoc evaluation of generated text, not an identity with the retrieval scores. No parameter is fitted to the temporal metric, no optimization loop maximizes it, and the embedding model is an external, off-the-shelf component. The comparison against real clinician-authored notes uses the same metric for both sides, so the central claim does not reduce by construction. The manuscript even concedes that human-in-the-loop evaluation is needed (§5), indicating that the metric is treated as a proxy rather than as the definition of the claimed property. The internal inconsistency in Table 1 (gold temporal consistency is 0.807 for LLaMA-3 but 0.9126 for Mistral-7B, despite the same gold corpus) and the use of Mistral-7B as both generator and preference judge are serious correctness and validity concerns, but they are not circular steps in the derivation chain. Therefore the appropriate circularity score is low.
Assumptions & free parameters
free parameters (6)
- Global clinical question set =
Not specified (hand-crafted)
- Local per-note-type question set =
Not specified (hand-crafted)
- Symptom keyword heuristics =
Not specified (hand-crafted)
- Hybrid scoring BM25/cosine weight =
Not specified
- Deduplication cosine threshold =
Not specified
- Chunk size threshold and overlap =
Not specified
assumptions (4)
- domain assumption All information needed for a progress note is contained in the 15 processed note types.
- domain assumption all-mpnet-base-v2 embeddings faithfully represent clinical semantic similarity.
- domain assumption Higher cosine similarity between adjacent generated notes indicates better temporal coherence.
- ad hoc to paper The hand-crafted question and keyword sets are sufficient to surface task-relevant content.
Cite this review
Pith. "Pith review of CLI-RAG: A Retrieval-Augmented Framework for Clinically Structured and Context Aware Text Generation with LLMs." pith.science (2026). https://pith.science/paper/BH3C2JPL
@misc{pith2026250706715,
author = {Pith},
title = {Pith review of: CLI-RAG: A Retrieval-Augmented Framework for Clinically Structured and Context Aware Text Generation with LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/BH3C2JPL}},
note = {Machine review of arXiv:2507.06715}
}
read the original abstract
Large language models (LLMs), including zero-shot and few-shot paradigms, have shown promising capabilities in clinical text generation. However, real-world applications face two key challenges: (1) patient data is highly unstructured, heterogeneous, and scattered across multiple note types and (2) clinical notes are often long and semantically dense, making naive prompting infeasible due to context length constraints and the risk of omitting clinically relevant information. We introduce CLI-RAG (Clinically Informed Retrieval-Augmented Generation), a domain-specific framework for structured and clinically grounded text generation using LLMs. It incorporates a novel hierarchical chunking strategy that respects clinical document structure and introduces a task-specific dual-stage retrieval mechanism. The global stage identifies relevant note types using evidence-based queries, while the local stage extracts high-value content within those notes creating relevance at both document and section levels. We apply the system to generate structured progress notes for individual hospital visits using 15 clinical note types from the MIMIC-III dataset. Experiments show that it preserves temporal and semantic alignment across visits, achieving an average alignment score of 87.7%, surpassing the 80.7% baseline from real clinician-authored notes. The generated outputs also demonstrate high consistency across LLMs, reinforcing deterministic behavior essential for reproducibility, reliability, and clinical trust.
Figures
Forward citations
Cited by 3 Pith papers
-
Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studies
Synthetic clinical communication generated by LLMs can train clinical NLP models in thirteen case studies, but only one is tested on real patient text, leaving transfer to authentic communication unproven.
-
When Reasoning Hurts: Source-Aware Evaluation of Frontier LLMs for Clinical SOAP Note Generation
Non-reasoning GPT-5.4 outperforms its reasoning version on SOAP note quality across datasets, with RAG giving smaller model-dependent gains.
-
DENSE: Longitudinal Progress Note Generation with Temporal Modeling of Heterogeneous Clinical Notes Across Hospital Visits
DENSE synthesizes progress notes across hospital visits using retrieval over heterogeneous clinical notes, claiming temporal continuity that even exceeds gold-standard notes.
Reference graph
Works this paper leans on
-
[1]
Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.130 Large language models are few-shot clinical information extractors . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998--2022, Abu Dhabi, United Arab Emirates. Association for ...
-
[2]
Nate C Apathy, Allison J Hare, Sarah Fendrich, and Dori A Cross. 2022. Early changes in billing and notes after evaluation and management guideline change. Annals of internal medicine, 175(4):499--504
work page 2022
-
[3]
Anjanava Biswas and Wrick Talukdar. 2024. https://doi.org/10.38124/ijisrt/ijisrt24may1483 Intelligent clinical documentation: Harnessing generative ai for patient-centric clinical note generation . International Journal of Innovative Science and Research Technology (IJISRT), page 994–1008
-
[4]
Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2019. http://arxiv.org/abs/1907.05545 The dynamic embedded topic model
arXiv 2019
-
[5]
Simon Ellershaw, Christopher Tomlinson, Oliver E Burton, Thomas Frost, John Gerrard Hanrahan, Danyal Zaman Khan, Hugo Layard Horsfall, Mollie Little, Evaleen Malgapo, Joachim Starup-Hansen, Jack Ross, Martinique Vella-Baldacchino, Kawsar Noor, Anoop D. Shah, and Richard Dobson. 2024. https://openreview.net/forum?id=1kDJJPppRG Automated generation of hospi...
work page 2024
-
[6]
Changchang Fang, Yuting Wu, Wanying Fu, Jitao Ling, Yue Wang, Xiaolin Liu, Yuan Jiang, Yifan Wu, Yixuan Chen, Jing Zhou, Zhichen Zhu, Zhiwei Yan, Peng Yu, and Xiao Liu. 2023. https://doi.org/10.1371/journal.pdig.0000397 How does chatgpt-4 preform on non-english national medical licensing examination? an evaluation in chinese language . PLOS Digital Health...
-
[7]
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org
work page 2020
-
[8]
Masaru Isonuma, Junichiro Mori, Danushka Bollegala, and Ichiro Sakata. 2020. https://doi.org/10.18653/v1/2020.acl-main.73 T ree- S tructured N eural T opic M odel . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 800--806, Online. Association for Computational Linguistics
Show all 31 references
-
[9]
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. http://arxiv.org/abs/2208.03299 Atlas: Few-shot learning with retrieval augmented language models
2022 arXiv
-
[10]
Raghav Jain, Anubhav Jangra, Sriparna Saha, and Adam Jatowt. 2022. http://arxiv.org/abs/2212.01669 A survey on medical document summarization
2022 arXiv
-
[11]
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-Wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. 2016. https://doi.org/10.1038/sdata.2016.35 Mimic-iii, a freely accessible critical care database . Scient...
2016 doi
-
[12]
Thomson Kuhn, Peter Basch, Michael Barr, Thomas Yackel, and Medical Informatics Committee of the American College of Physicians*. 2015. Clinical documentation in the 21st century: executive summary of a policy position paper from the american college of physicians. Annals of i...
2015
-
[13]
u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proc...
2020
-
[14]
Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics
2004
-
[15]
Qiuhao Lu, Dejing Dou, and Thien Nguyen. 2022. https://doi.org/10.18653/v1/2022.findings-emnlp.398 C linical T 5: A generative language model for clinical text . In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5436--5443, Abu Dhabi, United Arab ...
2022 doi
-
[16]
Arie Markel. 2010. Copy and paste of electronic health records: a modern medical illness. The American journal of medicine, 123(5):e9
2010
-
[17]
S. M. Meystre, G. K. Savova, K. C. Kipper-Schuler, and J. F. Hurdle. 2008. https://pubmed.ncbi.nlm.nih.gov/18660887/ Extracting information from textual documents in the electronic health record: a review of recent research . Yearbook of Medical Informatics, pages 128--144. Sy...
2008
- [18]
-
[19]
Bethany Percha. 2021. https://doi.org/https://doi.org/10.1146/annurev-biodatasci-030421-030931 Modern clinical text mining: A guide and review . Annual Review of Biomedical Data Science, 4(Volume 4, 2021):165--187
2021 doi
-
[20]
Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...
2019 doi
-
[21]
Adam Rule, Steven Bedrick, Michael F Chiang, and Michelle R Hribar. 2021. Length and redundancy of outpatient progress notes across a decade at an academic medical center. JAMA Network Open, 4(7):e2115334--e2115334
2021
-
[22]
Seiji Shimizu, Lis Pereira, Shuntaro Yada, and Eiji Aramaki. 2024. https://aclanthology.org/2024.lrec-main.1171/ QA -based event start-points ordering for clinical temporal relation annotation . In Proceedings of the 2024 Joint International Conference on Computational Linguis...
2024
-
[23]
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansf...
2023
-
[24]
Jiwoong Sohn, Yein Park, Chanwoong Yoon, Sihyeon Park, Hyeon Hwang, Mujeen Sung, Hyunjae Kim, and Jaewoo Kang. 2025. https://doi.org/10.18653/v1/2025.naacl-long.635 Rationale-guided retrieval augmented generation for medical question answering . In Proceedings of the 2025 Conf...
2025 doi
-
[25]
Sarvesh Soni and Dina Demner-Fushman. 2024. http://arxiv.org/abs/2410.12845 Toward relieving clinician burden by automatically generating progress notes using interim hospital data
2024 arXiv
-
[26]
Xiangru Tang, Andrew Tran, Jeffrey Tan, and Mark Gerstein. 2023. https://doi.org/10.18653/v1/2023.clinicalnlp-1.58 G erstein L ab at MEDIQA -chat 2023: Clinical note summarization from doctor-patient conversations through fine-tuning and in-context learning . In Proceedings of...
2023 doi
-
[27]
Langlotz, Jason Hom, Sergios Gatidis, John Pauly, and Akshay S
Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Sergios...
2024 doi
-
[28]
Michael D Wang, Raman Khanna, and Nader Najafi. 2017. Characterizing the source of text in electronic health record progress notes. JAMA internal medicine, 177(8):1212--1213
2017
-
[29]
Sheng Wang, Zihao Zhao, Xi Ouyang, Tianming Liu, Qian Wang, and Dinggang Shen. 2024. Interactive computer-aided diagnosis on medical image using large language models. Communications Engineering, 3(1):133
2024
-
[30]
Williams, Jaskaran Bains, Tianyu Tang, Kishan Patel, Alexa N
Christopher Y.K. Williams, Jaskaran Bains, Tianyu Tang, Kishan Patel, Alexa N. Lucas, Fiona Chen, Brenda Y. Miao, Atul J. Butte, and Aaron E. Kornblith. 2024. https://doi.org/10.1101/2024.04.03.24305088 Evaluating large language models for drafting emergency department dischar...
2024 doi
-
[31]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. https://arxiv.org/abs/2407.10671 Qwen2 technical report . arXiv preprint arXiv:2407.10671
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.