Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-source LLMs still cannot reliably summarise long patient trajectories or reason across time; retrieval augmentation helps only partly.

desk verdict Useful evaluation of open LLMs on long clinical sequences, but the headline temporal-reasoning claim isn't actually measured; needs a direct temporal check and statistical rigor. read the letter →

arxiv 2501.18724 v3 pith:CYBW3MU4 submitted 2025-01-30 cs.CL

Maya Kruse , Shiyue Hu , Nicholas Derby , Yifu Wu , Samantha Stonbraker , Bingsheng Yao , Dakuo Wang , Elizabeth Goldberg
show 1 more author
Yanjun Gao
This is my paper · ORCID
classification cs.CL
keywords clinicaltextsummarizationtemporalreasoningelectronichealthrecordslargelanguagemodelsretrieval-augmentedgenerationchain-of-thoughtpromptingdischargediagnosisprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish where current open-source large language models stand on longitudinal clinical reasoning: given multi-day, multi-modal electronic health records, can they write discharge summaries, generate daily Assessment and Plan notes, and predict future diagnoses? Across MIMIC-III and EHRShot, the answer is no, not reliably. Longer context windows help models integrate more input but do not consistently improve clinical reasoning, and retrieval-augmented generation reduces some hallucinations without fixing temporal progression or rare-disease predictions. A clinician's qualitative review confirms the automated scores: summaries oversimplify, retain outdated diagnoses, and can drift from the record. The intended contribution is an evaluation design that reframes clinical summarization as a long-context, temporally ordered task rather than a single-snapshot benchmark.

What carries the argument

The load-bearing mechanism is the task itself: each record is verbalized into natural language with relative timestamps, grouped by time, and either fed whole (direct generation), split and retrieved (RAG), or first compressed into a temporally ordered 'key events' list by chain-of-thought prompting before summarization. The comparison manipulates context availability: no prior context, single-day context, or multi-day context for progress notes, and 24- vs 48-hour windows with notes, tabular data, or both for discharge summaries. These manipulations are what let the paper attribute differences in performance to temporal reasoning rather than to dataset or modality alone.

What would settle it

Give a long-context LLM a full multi-day admission instead of the last 24 or 48 hours and compare its discharge summary against the same model restricted to the short window; if full-input summaries reach clinician-level completeness and accuracy, the paper's claim that models cannot reason over long longitudinal trajectories would be undermined. A simpler version: find one 72-hour-plus admission where an event on day 1 determines the discharge diagnosis, and check whether the model using only the final window misses it while the full-input model catches it.

Watch

Extended reading notes

Core claim

The central finding is that the bottleneck is temporal reasoning, not context length. On discharge summarization, feeding the model only the last 24 or 48 hours produced modest ROUGE-L scores, with Qwen and RAG reaching the highest value; for daily Assessment and Plan generation, adding one prior day helped, but adding all prior days did not consistently beat a single prior day; and on six EHRShot diagnosis-prediction tasks, models defaulted to the majority class, giving high accuracy but F1 scores near zero for rare diseases such as celiac disease and lupus. RAG improved hallucination in expert review but did not close the gap, and a chain-of-thought event-extraction step did not beat direct generation. The paper argues that these results establish a need for evaluations and models that are temporally grounded, rather than simply long-context or retrieval-heavy.

Load-bearing premise

The discharge-summarization task assumes that the last 24 or 48 hours of a stay contain all the information needed to write the full discharge summary, so clinically important events from earlier in a long admission are never shown to the model.

Editorial extensions

If this is right

  • Evaluation of clinical summarization should be redesigned so that correct answers require information from several days earlier, rather than a short pre-discharge window.
  • Retrieval augmentation is a partial mitigation for hallucination but not a substitute for temporal grounding; models need training or prompting that explicitly links events across timestamps.
  • Rare-disease prediction will need methods beyond majority-class calibration, because raw accuracy is inflated by class imbalance.
  • Chain-of-thought event extraction offers interpretability but, as currently pipelined, does not improve generation quality; any future use should include adaptation or integration with instruction tuning.
  • Adding prior context improves progress-note generation up to a point; single-day context can outperform multi-day context in later hospital days, implying diminishing returns from longer history.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the bottleneck is temporal reasoning rather than context length, then a fair test is to construct patient trajectories where the key diagnosis is only inferable from a critical event many days before the summary window; the paper's short-window design leaves that test open.
  • Editorial inference: the near-zero F1 for rare diseases suggests that future work should report precision and recall rather than accuracy, and could explore counterfactual data augmentation or retrieval of similar rare cases.
  • Editorial inference: the finding that shuffled tabular data sometimes outperforms chronological tabular data in the 24-hour window hints that near-discharge data is nearly order-independent; outside that window, temporal order may matter more, which is a testable extension.
  • Editorial inference: since RAG improved hallucination in expert review but not automated metrics, evaluation of longitudinal summarization may need richer clinician-annotated criteria, such as retention of outdated diagnoses, rather than ROUGE and BERTScore alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates five open-source LLMs (Mistral-7B, Llama3-8B, Qwen2.5-7B, DeepSeek-R1-Distill-Qwen-32B, Llama2-13B), their RAG variants, and a chain-of-thought event-extraction pipeline on two public EHR datasets (MIMIC-III and EHRShot), across three tasks: discharge summarization, assessment-and-plan (A&P) generation, and diagnosis prediction. It re-engineers these tasks into long-context, multi-modal settings with temporally ordered inputs, and reports ROUGE-L, SapBERT-BERTScore, and macro F1/accuracy, plus a small expert review. The main conclusions are that long context windows improve input integration but do not consistently enhance clinical reasoning, that models struggle with temporal progression and rare disease prediction, and that RAG yields only partial, inconsistent improvements.

Significance. If the findings hold, the paper provides a useful, reproducible evaluation of open-source LLMs on longitudinal clinical summarization and prediction, and it highlights a real gap: raw context length and retrieval do not by themselves solve temporal reasoning or rare-disease prediction. The design is methodical, uses public datasets, evaluates zero-shot on held-out patients, and compares multiple models, prompting strategies, and input modalities. The authors also provide a transparent account of preprocessing and hyperparameter selection. However, the paper's distinctive claim about temporal reasoning is not directly supported by the reported metrics, because none of ROUGE-L, BERTScore, or macro F1 isolates temporal ordering correctness. The manuscript is therefore a solid empirical study with an overclaimed interpretive layer, and it is best published after either adding direct temporal-consistency evaluation or rephrasing the conclusions to match what the metrics can actually show.

major comments (4)
  1. [Section 4.2 and Section 5 (Tables 2, 4, 5, 6)] The headline conclusion that models "struggle with ... temporal reasoning" is not directly measured. ROUGE-L, SapBERT-BERTScore, and macro F1/accuracy reward lexical or semantic overlap with a gold reference or binary classification accuracy; they do not score whether generated events appear in the correct temporal order or whether causal links across time are preserved. The only temporal manipulation, SHUFFLED TABULAR in §3.1 and Figure 5, is applied to the final 24/48 hours, which the authors themselves describe as a clinically stable window, so it tests order sensitivity in a snapshot rather than reasoning across a trajectory. The extracted event lists in the CoT approach (Table 5) are never checked for temporal correctness. Please add a direct temporal-consistency evaluation (for example, event-order accuracy, timestamp-violation rate, or human judgments of temporal coherence) or restrict the conclusions to summarization and prediction deficits without the temporal-reasoning framing.
  2. [Section 3.1 (Discharge Summarization)] The discharge summarization task limits input to the last 24 or 48 hours of hospitalization, yet the abstract and introduction claim the paper evaluates "full patient trajectories" and "long-context" clinical data. With this window, any clinically important events that occurred earlier in the admission are never seen by the model, so the task measures short-window summarization rather than reasoning over the complete longitudinal course. This is a load-bearing mismatch between the task design and the claimed scope. Please either extend the input to the full stay (with appropriate length controls or retrieval over the full stay) or explicitly describe the discharge task as a short-window summarization task and adjust the abstract and conclusion accordingly.
  3. [Section 4.1 and Appendix A.4] RAG hyperparameters (chunk size, top-k, overlap) were selected on a held-out set of n=5, and the CoT temporal-context settings were selected on n=20 development patients, but no significance testing or confidence intervals are reported anywhere in the paper. Many comparative claims in Section 5 (for example, "RAG generally improves performance" and "long context windows improve input integration") are based on differences that fall within one standard deviation of the reported means, such as Mistral's discharge-summarization RAG vs. direct-generation scores (15.04 vs. 16.28 with σ≈2–13). Without significance tests or an analysis of how the small tuning sets affect generalization, these comparative conclusions are not established. Please add appropriate statistical tests or confidence intervals and discuss the sensitivity of the conclusions to the tuning procedure.
  4. [Table 6 (EHRShot diagnosis prediction)] The near-zero F1 scores for Celiac and Lupus with accuracies around 95% are exactly what one expects from majority-class prediction given negative prevalence of 96.54% and 95.56%. These results demonstrate class-imbalance failure, not specifically a temporal-reasoning failure. The text in Section 5 partially acknowledges this, but the discussion should report balanced metrics such as AUROC or AUPRC, and should avoid presenting these results as evidence about temporal reasoning. Without such metrics, the reader cannot distinguish a model that ignores all temporal information from one that reasons well but is not calibrated for rare outcomes.
minor comments (5)
  1. [Section 4.2] The word "summarizatoin" is a typo and should read "summarization."
  2. [Appendix A.9, Table 18] In the Qwen row, the Single-Day ROUGE-L value is reported as 77.50 ± 5.37, which appears inconsistent with the value 20.74 reported in Table 4 for the same condition; this is likely a copy-and-paste error and should be corrected.
  3. [Figure 5 caption] The caption states "Average f1 results across modalities and time windows," but the surrounding text discusses ROUGE-L and BERTScore; please clarify which metric (or average of metrics) is plotted.
  4. [Section 3.1] The phrase "shuffled the tabular data by their timestamps" is ambiguous; it should read "shuffled the tabular data with respect to their timestamps" or "shuffled the timestamp order of the tabular data."
  5. [Section 6 and Limitations] The expert error analysis is based on 10 pairs of outputs from a single physician using an instrument that the authors explicitly state was not validated. This is an acknowledged limitation, but the claim in Section 6 that "RAG performs slightly better, with fewer hallucinations" should be presented more cautiously, for example as an exploratory observation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical evaluations on held-out data, and the self-citations found are method choices that are not load-bearing.

full rationale

The paper makes no formal derivation claims; its results are zero-shot empirical evaluations on held-out patients, with hyperparameter and prompt optimization performed on small separate development sets (Sections 4.1 and Appendices A.4, A.5), so no reported number is forced by fitted parameters or renamed as a prediction. The central limitation claim that models struggle with temporal reasoning is an interpretation of low ROUGE-L, BERTScore, and F1 values rather than a quantity defined in terms of those scores; the absence of a direct temporal-ordering metric is a measurement-validity concern, not a circularity. Self-citations (e.g., Gao et al. 2023a, 2022; Myers et al. 2024; Gao et al. 2024) support method choices such as BGE embeddings and table-to-text verbalization, but these choices are not the paper's conclusions, are not invoked to forbid alternatives, and do not assert any uniqueness theorem. The discharge-summarization restriction to the last 24 or 48 hours is an explicitly disclosed task-design choice ('We limit input to the last 24 hours to prevent overloading the LLM while prioritizing the most relevant information for discharge'), not a hidden definitional equivalence. No equation or fitted value is equivalent by construction to the outputs claimed as findings, and no load-bearing argument reduces to the authors' prior work. Accordingly, no circular steps are identified.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims do not rest on new physical or mathematical entities. The load-bearing choices are empirical: what data is included (24/48h windows, abnormal-only labs), how structured data is verbalized, which hyperparameters and CoT contexts are selected, and which metrics count as evidence. These choices are documented but mostly not justified with ablations or significance tests.

free parameters (6)
  • RAG chunk size = selected from {250, 500, 750}
    Tuned per task/model on a held-out set of n=5 (Appendix A.4, Table 10).
  • RAG top-k = selected from {10, 20, 50}
    Tuned along with chunk size and overlap on the same small held-out set.
  • RAG chunk overlap = selected from {50, 100, 200}
    Tuned as part of the RAG hyperparameter search.
  • CoT temporal context = FORWARD all prior days and BACKWARD one following day
    Selected by evaluating seven configurations on a development batch of 20 patients (Section 4.1).
  • Discharge input window = 24 or 48 hours
    Chosen by hand to limit input length; may exclude earlier relevant events (Section 3.1).
  • Prompt and query variant per model = per-model prompts in Tables 13-17
    Prompt and RAG query templates were optimized on a small sample set (Appendix A.5).
assumptions (5)
  • domain assumption ICU stays longer than 72 hours provide sufficient context for evaluating long-document summarization and temporal reasoning.
    Cohort selection in Section 3.1; if shorter stays contain different temporal patterns, conclusions may not generalize.
  • domain assumption Converting structured EHR values into natural-language templates preserves the temporal and clinical semantics needed for reasoning.
    Section 3.3; if the verbalization loses dose-response timing or units, the model reasons over a degraded representation.
  • domain assumption ROUGE-L and SapBERT-based BERTScore are adequate proxies for the clinical quality of summaries.
    Section 4.2; the only human check covers 10 Qwen discharge outputs by one physician, so automated metrics carry most of the evidential weight.
  • ad hoc to paper Hyperparameters and CoT context settings selected on n=5 or n=20 development cases generalize to the reported test cases.
    Appendix A.4 and Section 4.1; very small selection sets risk overfitting the evaluation protocol to noise.
  • domain assumption The EHRShot binary diagnosis tasks are meaningful tests of temporal reasoning rather than pattern-matching on feature presence.
    Section 3.2; no ablation distinguishes temporal ordering from simple feature occurrence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction." pith.science (2026). https://pith.science/paper/CYBW3MU4

@misc{pith2026250118724,
  author       = {Pith},
  title        = {Pith review of: Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYBW3MU4}},
  note         = {Machine review of arXiv:2501.18724}
}
read the original abstract

Recent advances in large language models (LLMs) have shown potential in clinical text summarization, but their ability to handle long patient trajectories with multi-modal data spread across time remains underexplored. This study systematically evaluates several state-of-the-art open-source LLMs, their Retrieval Augmented Generation (RAG) variants and chain-of-thought (CoT) prompting on long-context clinical summarization and prediction. We examine their ability to synthesize structured and unstructured Electronic Health Records (EHR) data while reasoning over temporal coherence, by re-engineering existing tasks, including discharge summarization and diagnosis prediction from two publicly available EHR datasets. Our results indicate that long context windows improve input integration but do not consistently enhance clinical reasoning, and LLMs are still struggling with temporal progression and rare disease prediction. While RAG shows improvements in hallucination in some cases, it does not fully address these limitations. Our work fills the gap in long clinical text summarization, establishing a foundation for evaluating LLMs with multi-modal data and temporal reasoning.

Figures

Figures reproduced from arXiv: 2501.18724 by the authors.

Figure 1
Figure 1. An illustration of the longitudinal patient trajectory summarization process from multi-modal EHRs. Key causal relationships among medical observations and interventions are highlighted, leading to the final physician diagnosis. errors (Dymek et al., 2021; Singh et al., 2017). Au￾tomated summarization can help improve care con￾tinuity and decision-making (Dymek et al., 2021; Adams et al., 2021; Gao et al., 2023a; La… view at source ↗
Figure 2
Figure 2. Illustration of the A&P generation workflow (top) and a comparison of input data formulations (bottom). Assessment and Plan (A&P) Generation Daily progress notes are the documents where physicians record diagnoses, treatment, and clinical status, providing key insights into a patient’s condition throughout their hospital stay. A progress note typically consists of four sections: Subjective, Ob￾jective, Assessment an… view at source ↗
Figure 3
Figure 3. The token length of input data used for A&P generation across days and methods (left), as well as that of input data used for Discharge Summarization across time windows and modalities(right). in summarizing longitudinal data, the Assessment and Plan sections provide a great testbed, requiring integration of information from multiple days, cap￾turing the evolution of the patient’s condition, and synthesizing key cli… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Chain-of-Thought prompting template and example output used for ICU daily event extraction from structured EHR data for A&P generation. Experiment settings. For all LLMs, we run their 8-bit quantized version. We set the output token length as 1,000, but almost all task…
Figure 5
Figure 5. Figure 5: Average f1 results across modalities and time win￾dows (on direct generation) 6 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Correlation of token length and average performance across all tasks and metrics Model Metric CURRENT DAY FORWARD+N BACKWARD-1 Mistral ROUGE-L 19.85±10.87 20.82±12.02 20.10±11.15 BERTScore 68.63±11.56 69.69±10.82 69.35±9.67 Llama3 ROUGE-L 31.56±16.56 31.34±16.93 28.40±…
Figure 7
Figure 7. Figure 7: Expert review scores for Qwen direct generation and RAG on Discharge Summarization Discharge Summarization [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: reports detailed statistics of EHRShot co￾hort, regarding the input token length and num￾ber of distinct clinical measurements. The prompt [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Chain-of-Thought prompting template and example output used for discharge summary event extraction based on the final 48 hours of hospitalization. no prior-day context (Method -1) to single-day (Method 1) and multi-day context (Method 2)—af￾fect the alignment between g…
Figure 10
Figure 10. Figure 10: Impact of prior-day context on F1 score across consecutive days enhances the accuracy and stability of generated patient notes, reducing hallucinated content while preserving clinically relevant information. A.10 Break-down results behind [PITH_FULL_IMAGE:figures/ful…
Figure 11
Figure 11. Figure 11: Comparison of generated patient notes across different methods of incorporating prior-day context. Each subplot represents a different patient admission, with bars indicating the composition of generated notes across Days 2, 3, and 4. The stacked bars show the proport…
Figure 12
Figure 12. Figure 12: Metric breakdown across modalities for the 24 hour time window (on direct generation) 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Series

    cs.CL 2026-07 conditional novelty 6.0 of 10

    CLIR-Bench shows generalist and time-series LLMs struggle to ground clinical answers in sparse irregular ICU evidence, with top accuracy near 50% and weak causal evidence use.

Reference graph

Works this paper leans on

53 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Griffin Adams, Emily Alsentzer, Mert Ketenci, Jason Zucker, and No \'e mie Elhadad. 2021. What’s in a summary? laying the groundwork for advances in hospital-course summarization. In Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting, volume 2021, page 4794. NIH Public Access

    work page 2021

  4. [4]

    Griffin Adams, Jason Zucker, and No \'e mie Elhadad. 2024. Speer: Sentence-level planning of long clinical summaries via embedded entity retrieval. arXiv preprint arXiv:2401.02369

    arXiv 2024

  5. [5]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  6. [6]

    Fadi Aljamaan, Mohamad-Hani Temsah, Ibraheem Altamimi, Ayman Al-Eyadhy, Amr Jamal, Khalid Alhasan, Tamer A Mesallam, Mohamed Farahat, Khalid H Malki, et al. 2024. Reference hallucination score for medical artificial intelligence chatbots: development and usability study. JMIR Medical Informatics, 12(1):e54345

    work page 2024

  7. [7]

    Kenichiro Ando, Takashi Okumura, Mamoru Komachi, Hiromasa Horiguchi, and Yuji Matsumoto. 2022. Is artificial intelligence capable of generating hospital discharge summaries from inpatient records? PLOS Digital Health, 1(12):e0000158

    work page 2022

  8. [8]

    Jeffery L Belden, Richelle J Koopman, Sonal J Patil, Nathan J Lowrance, Gregory F Petroski, and Jamie B Smith. 2017. Dynamic electronic health record note prototype: seeing more by showing less. The Journal of the American Board of Family Medicine, 30(6):691--700

    work page 2017

Show all 53 references
  1. [9]

    Asma Ben Abacha, Wen-wai Yim, George Michalopoulos, and Thomas Lin. 2023. https://doi.org/10.18653/v1/2023.findings-acl.161 An investigation of evaluation methods in automatic medical note generation . In Findings of the Association for Computational Linguistics: ACL 2023, pag...

    2023 doi

  2. [10]

    Harrison Chase. 2022. https://github.com/langchain-ai/langchain Langchain . Accessed: 2025-02-14

    2022

  3. [11]

    Croxford, Y

    E. Croxford, Y. Gao, B. Patterson, D. To, S. Tesch, D. Dligach, A. Mayampurath, M. M. Churpek, and M. Afshar. 2024. https://doi.org/10.1101/2024.03.20.24304620 Development of a human evaluation framework and correlation with automated metrics for natural language generation of...

    2024 doi

  4. [12]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  5. [13]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazar \'e , Maria Lomeli, Lucas Hosseini, and Herv \'e J \'e gou. 2024. The faiss library. arXiv preprint arXiv:2401.08281

    2024 arXiv

  6. [14]

    Christine Dymek, Bryan Kim, Genevieve B Melton, Thomas H Payne, Hardeep Singh, and Chun-Ju Hsiao. 2021. Building the evidence-base to reduce electronic health record--related clinician burden. Journal of the American Medical Informatics Association, 28(5):1057--1061

    2021

  7. [15]

    Yanjun Gao, Dmitriy Dligach, Timothy Miller, Matthew M Churpek, and Majid Afshar. 2023 a . Overview of the problem list summarization (probsum) 2023 shared task on summarizing patients’ active diagnoses and problems from electronic health record progress notes. In Proceedings ...

    2023

  8. [16]

    Yanjun Gao, Timothy Miller, Dongfang Xu, Dmitriy Dligach, Matthew M Churpek, and Majid Afshar. 2022. Summarizing patients’ problems from hospital progress notes using pre-trained sequence-to-sequence models. In Proceedings of COLING. International Conference on Computational L...

    2022

  9. [17]

    Yanjun Gao, Skatje Myers, Shan Chen, Dmitriy Dligach, Timothy A Miller, Danielle Bitterman, Matthew Churpek, and Majid Afshar. 2024. When raw data prevails: Are large language model embeddings effective in numerical data representation for medical machine learning applications...

    2024 arXiv

  10. [18]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023 b . Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  11. [19]

    Patricia Garcia, Stephen P Ma, Shreya Shah, Margaret Smith, Yejin Jeong, Anna Devon-Sand, Ming Tai-Seale, Kevin Takazawa, Danyelle Clutter, Kyle Vogt, et al. 2024. Artificial intelligence--generated draft replies to patient inbox messages. JAMA Network Open, 7(3):e243201--e243201

    2024

  12. [20]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  13. [21]

    Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. 2020. Mimic-iv. PhysioNet. Available online at: https://physionet. org/content/mimiciv/1.0/(accessed August 23, 2021), pages 49--55

    2020

  14. [22]

    Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Leandra A Barnes, Hong-Yu Zhou, Zhuo Ran Cai, Eliezer M Van Allen, David Kim, et al. 2025. An evaluation framework for clinical use of large language models in patient interaction tasks....

    2025

  15. [23]

    Archana Laxmisan, Allison B McCoy, Adam Wright, and Dean F Sittig. 2012. Clinical summarization capabilities of commercially-available and internally-developed electronic health records. Applied clinical informatics, 3(01):80--93

    2012

  16. [24]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  17. [25]

    Rui Li, Fenglong Ma, and Jing Gao. 2022. Integrating multimodal electronic health records for diagnosis prediction. In AMIA Annual Symposium Proceedings, volume 2021, page 726

    2022

  18. [26]

    Jennifer Liang, Ching-Huei Tsou, and Ananya Poddar. 2019. https://doi.org/10.18653/v1/W19-1906 A novel system for extractive clinical note summarization using EHR data . In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 46--54, Minneapolis, Minneso...

    2019 doi

  19. [27]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  20. [28]

    Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2020. Self-alignment pretraining for biomedical entity representations. arXiv preprint arXiv:2010.11784

    2020 arXiv

  21. [29]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  22. [30]

    Ivan Lopez, Akshay Swaminathan, Karthik Vedula, Sanjana Narayanan, Fateme Nateghi Haredasht, Stephen P Ma, April S Liang, Steven Tate, Manoj Maddali, Robert Joseph Gallo, et al. 2025. Clinical entity augmented retrieval for clinical information extraction. npj Digital Medicine...

    2025

  23. [31]

    discharge me!

    Mengxian Lyu, Cheng Peng, Daniel Paredes, Ziyi Chen, Aokun Chen, Jiang Bian, and Yonghui Wu. 2024. Uf-hobi at “discharge me!”: A hybrid solution for discharge summary generation through prompt-based tuning of gatortrongpt models. In Proceedings of the 23rd Workshop on Biomedic...

    2024

  24. [32]

    Farida Mohsen, Hazrat Ali, Nady El Hajj, and Zubair Shah. 2022. Artificial intelligence-based methods for fusion of electronic health records and imaging data. Scientific Reports, 12(1):17981

    2022

  25. [33]

    Skatje Myers, Timothy A Miller, Yanjun Gao, Matthew M Churpek, Anoop Mayampurath, Dmitriy Dligach, and Majid Afshar. 2024. Lessons learned on information retrieval in electronic health records: a comparison of embedding models and pooling strategies. Journal of the American Me...

    2024

  26. [34]

    Rimma Pivovarov and No \'e mie Elhadad. 2015. Automated methods for the summarization of electronic health records. Journal of the American Medical Informatics Association, 22(5):938--947

    2015

  27. [35]

    Christina Silcox, Eyal Zimlichmann, Katie Huber, Neil Rowen, Robert Saunders, Mark McClellan, Charles N Kahn III, Claudia A Salzberg, and David W Bates. 2024. The potential for artificial intelligence to transform healthcare: perspectives from international health leaders. NPJ...

    2024

  28. [36]

    Hardeep Singh, Gordon D Schiff, Mark L Graber, Igho Onakpoya, and Matthew J Thompson. 2017. The global burden of diagnostic errors in primary care. BMJ quality & safety, 26(6):484--494

    2017

  29. [37]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172--180

    2023

  30. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [39]

    Robert M Wachter and Erik Brynjolfsson. 2024. Will generative artificial intelligence deliver on its promise in health care? Jama, 331(1):65--69

    2024

  32. [40]

    Sijia Wang, Mo Yu, and Lifu Huang. 2023. The art of prompting: Event detection based on type specific prompts. In The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  33. [41]

    Lawrence L Weed et al. 1968. Medical records that guide and teach. N Engl J Med, 278(11):593--600

    1968

  34. [42]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  35. [43]

    Christopher YK Williams, Brenda Y Miao, Aaron E Kornblith, and Atul J Butte. 2024. Evaluating the use of large language models to provide clinical recommendations in the emergency department. Nature Communications, 15(1):8236

    2024

  36. [44]

    Michael Wornow, Rahul Thapa, Ethan Steinberg, Jason Fries, and Nigam Shah. 2023. https://arxiv.org/abs/2307.02028 Ehrshot: An ehr benchmark for few-shot evaluation of foundation models

    2023 arXiv

  37. [45]

    Adam Wright, Dean F Sittig, Julie McGowan, Joan S Ash, and Lawrence L Weed. 2014. Bringing science to medicine: an interview with larry weed, inventor of the problem-oriented medical record. Journal of the American Medical Informatics Association, 21(6):964--968

    2014

  38. [46]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. https://arxiv.org/abs/2309.07597 C-pack: Packaged resources to advance general chinese embedding . Preprint, arXiv:2309.07597

    2023 arXiv

  39. [47]

    Collins, Ankit Modi, Robert Lloyd, Benjamin Hopkins, Curtis Langlotz, and Jean-Benoit Delbrouck

    Justin Xu, Zhihong Chen, Andrew Johnston, Louis Blankemeier, Maya Varma, Jason Hom, William J. Collins, Ankit Modi, Robert Lloyd, Benjamin Hopkins, Curtis Langlotz, and Jean-Benoit Delbrouck. 2024 a . https://doi.org/10.18653/v1/2024.bionlp-1.7 Overview of the first shared tas...

    2024 doi

  40. [48]

    discharge me!

    Justin Xu, Zhihong Chen, Andrew Johnston, Louis Blankemeier, Maya Varma, Jason Hom, William J Collins, Ankit Modi, Robert Lloyd, Benjamin Hopkins, et al. 2024 b . Overview of the first shared task on clinical text generation: Rrg24 and “discharge me!”. In Proceedings of the 23...

    2024

  41. [49]

    Retrieval meets long context large language models

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference on Learning Representations

  42. [50]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  43. [51]

    Bowen Yu, Cheng Fu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023. Unified language representation for question answering over text, tables, and images. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4756--4765

    2023

  44. [52]

    Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. 2023. https://doi.org/10.18653/v1/2023.bionlp-1.7 Zero-shot temporal relation extraction with C hat GPT . In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 92--102, Toronto, Canada. A...

    2023 doi

  45. [53]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.