MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

Gregory D. Lyng; Robert E. Tillman; Sanjit Singh Batra; Zhichao Yang

arxiv: 2605.20197 · v1 · pith:YXM2IVOCnew · submitted 2026-04-05 · 💻 cs.CL

MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction

Zhichao Yang , Gregory D. Lyng , Sanjit Singh Batra , Robert E. Tillman This is my paper

Pith reviewed 2026-05-21 10:25 UTC · model grok-4.3

classification 💻 cs.CL

keywords medical concept extractionimplicit reasoninglarge language modelsbenchmarkelectronic health recordsevidence groundingdischarge summaries

0 comments

The pith

Large language models achieve only modest accuracy when extracting medical concepts implied rather than explicitly stated in patient notes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MedicalBench tests whether large language models can identify medically relevant concepts from electronic health records when those concepts must be inferred from the surrounding text instead of being directly named. The benchmark frames the task as verifying note-concept pairs while also requiring the model to point to the exact sentences that support its decision. The dataset is drawn from real discharge summaries and deliberately mixes cases that demand implicit reasoning, near-miss negatives that share similar wording, and situations where models disagree with human experts. Benchmark results show that even leading models perform modestly on these tasks. The same results stay roughly constant no matter how long the note is, which suggests the benchmark measures genuine reasoning difficulty rather than the ability to handle longer text.

Core claim

MedicalBench formulates medical concept extraction as a verification task over note-concept pairs together with sentence-level evidence identification; the dataset is constructed from MIMIC-IV discharge summaries through an LLM triage pipeline, medical annotation, and expert review that includes implicit positives, semantically confusable negatives, and LLM-expert disagreement cases; evaluations of state-of-the-art models reveal modest performance that remains largely unchanged across varying note lengths.

What carries the argument

Verification of medical note-concept pairs coupled with sentence-level evidence identification, curated via multi-stage LLM triage plus expert review to capture implicit positives and confusable negatives.

If this is right

Models that improve on MedicalBench could support more reliable downstream applications that depend on understanding unstated medical information in records.
The finding that performance is invariant to note length directs attention toward improving conceptual reasoning rather than extending context windows.
The benchmark supplies a concrete testbed for developing medical language models that both extract concepts and provide traceable sentence evidence.
Complementary use of the two tasks (concept verification and evidence retrieval) can measure both correctness and interpretability of model outputs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same curation approach could be applied to create benchmarks for implicit reasoning in other specialized domains such as legal or scientific documents.
Fine-tuning strategies that first master explicit concept extraction might then be tested for transfer to the implicit cases in this dataset.
Integration with structured medical knowledge sources could be evaluated by measuring whether such knowledge reduces errors on the semantically confusable negative cases.

Load-bearing premise

The multi-stage LLM triage pipeline followed by medical annotation and expert review produces a dataset that accurately represents real medical reasoning challenges including implicit concepts.

What would settle it

New expert review of the same notes that produces substantially different implicit-concept labels, or a model whose accuracy rises sharply once note length is controlled in a fresh test set, would indicate that the benchmark does not isolate the intended reasoning difficulty.

Figures

Figures reproduced from arXiv: 2605.20197 by Gregory D. Lyng, Robert E. Tillman, Sanjit Singh Batra, Zhichao Yang.

**Figure 2.** Figure 2: Performance of LLMs on the medical concept extraction task (y-axis) and evidence re [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Precision–recall comparison of LLMs and a pretrained language model (PLM-ICD), which [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 4.** Figure 4: GPT-5 performance across different note lengths [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

read the original abstract

Medical concept extraction from electronic health records underpins many downstream applications, yet remains challenging because medically meaningful concepts are frequently implied rather than explicitly stated in medical narratives. Existing benchmarks with human-annotated evidence spans underscore the importance of grounding extracted concepts in medical text. However, they predominantly focus on explicitly stated concepts instead of implicit concepts. We present MedicalBench, a benchmark for medical concept extraction with evidence grounding that evaluates implicit medical reasoning. MedicalBench formulates medical concept extraction as a verification task over medical note-concept pairs, coupled with sentence-level evidence identification. Built from MIMIC-IV discharge summaries and human-verified ICD-10 codes, the dataset is curated through a multi-stage large language model (LLM) triage pipeline followed by medical annotation and expert review. It deliberately includes implicit positives, semantically confusable negatives, and cases where LLM judgments disagree with medical expert assessments. We define two complementary evaluation tasks: (1) medical concept extraction and (2) sentence-level evidence retrieval, enabling assessment of both correctness and interpretability. Benchmarking state-of-the-art LLMs reveals that performance remains modest, highlighting the difficulty of extracting implicitly expressed concepts. We further show that performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders. MedicalBench provides the first systematic benchmark for implicit, evidence-grounded medical concept extraction, offering a foundation for developing medical language models that can both identify medically relevant concepts and justify their predictions in a transparent and medically faithful manner.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

MedicalBench introduces a dataset targeting implicit medical concepts with evidence grounding, but the curation lacks reported validation metrics that would confirm it truly isolates reasoning difficulty.

read the letter

This paper creates MedicalBench, a benchmark for medical concept extraction that focuses on implicit rather than explicit concepts in clinical notes, paired with sentence-level evidence retrieval. It draws from MIMIC-IV discharge summaries linked to ICD-10 codes and uses a multi-stage LLM triage followed by medical annotation and expert review to include implicit positives, semantically close negatives, and LLM-expert disagreement cases. The two tasks let models be scored on both correctness and interpretability. Benchmarking current LLMs shows modest results overall, and performance stays roughly the same across different note lengths, which the authors interpret as evidence that the difficulty comes from reasoning rather than superficial factors like text volume. That length-invariance check is a useful control and gives the work some practical value for people building clinical evaluation sets. The main weakness is that the methods description gives no numbers on inter-annotator agreement, no count of how often experts overruled the initial LLM triage, and no explicit operational definition of what counts as implicit. Without those details the claim that the benchmark cleanly separates reasoning difficulty from selection artifacts remains hard to verify. The risk of circularity in an LLM-heavy curation pipeline is real and should have been quantified. This is for researchers working on medical NLP benchmarks and clinical AI evaluation who need test sets that go beyond explicit mentions. A reader looking for concrete data on where current models fall short on implicit reasoning would find it worth reading. It deserves peer review because the gap it targets is genuine and the basic construction approach is sound, even though the validation reporting needs to be tightened before the dataset can be used with full confidence.

Referee Report

2 major / 2 minor

Summary. The paper introduces MedicalBench, a benchmark dataset and evaluation framework for medical concept extraction from EHR notes that targets implicitly expressed rather than explicitly stated concepts. Constructed from MIMIC-IV discharge summaries via a multi-stage LLM triage pipeline followed by medical annotation and expert review, the dataset deliberately incorporates implicit positives, semantically confusable negatives, and LLM-expert disagreement cases. It defines two tasks—medical concept verification over note-concept pairs and sentence-level evidence retrieval—and reports that state-of-the-art LLMs achieve only modest performance that remains largely invariant to note length, which the authors interpret as evidence that the benchmark isolates reasoning difficulty rather than superficial confounders such as length.

Significance. If the curation process is shown to reliably isolate implicit reasoning challenges, MedicalBench would fill an important gap in medical NLP by providing the first systematic, evidence-grounded benchmark focused on implicit concept extraction. The length-invariance result, if substantiated, would strengthen claims that performance gaps reflect genuine medical reasoning limitations rather than dataset artifacts, potentially guiding development of more interpretable and medically faithful LLMs.

major comments (2)

[Abstract / Dataset Construction] Abstract and Dataset Construction section: the multi-stage LLM triage followed by human annotation and expert review is described at a high level, but no quantitative validation is reported (e.g., inter-annotator agreement scores, fraction of initial LLM labels overturned by experts, or operational criteria used to classify a concept as 'implicit' versus explicit). Without these metrics it is difficult to verify that the final dataset contains a high proportion of truly implicit cases rather than LLM-biased selections or explicitly stated concepts.
[Results] Results section on length invariance: the claim that performance is 'largely invariant to note length' is used to argue that MedicalBench isolates reasoning difficulty. However, without details on how note length was measured, the distribution of lengths across implicit vs. explicit subsets, or statistical tests controlling for other confounders (e.g., concept rarity or note complexity), the isolation from superficial factors remains incompletely supported.

minor comments (2)

The abstract states that the dataset 'deliberately includes' LLM-expert disagreement cases; a table or figure quantifying the size of each subset (implicit positives, confusable negatives, disagreement cases) would improve clarity and allow readers to assess balance.
Consider adding a brief comparison table against existing medical concept extraction benchmarks (e.g., those focused on explicit spans) to highlight the unique contribution of the implicit focus.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed and constructive feedback on our manuscript. We address each of the major comments below and outline the revisions we plan to make to improve the paper's clarity and rigor.

read point-by-point responses

Referee: [Abstract / Dataset Construction] Abstract and Dataset Construction section: the multi-stage LLM triage followed by human annotation and expert review is described at a high level, but no quantitative validation is reported (e.g., inter-annotator agreement scores, fraction of initial LLM labels overturned by experts, or operational criteria used to classify a concept as 'implicit' versus explicit). Without these metrics it is difficult to verify that the final dataset contains a high proportion of truly implicit cases rather than LLM-biased selections or explicitly stated concepts.

Authors: We agree with the referee that quantitative validation metrics are essential for establishing the reliability of the dataset curation process. The current version of the manuscript describes the multi-stage pipeline at a high level without providing specific numbers. In the revised manuscript, we will add the following: (1) inter-annotator agreement scores (e.g., Cohen's kappa) among the medical annotators, (2) the fraction of initial LLM labels that were overturned by the expert review, and (3) detailed operational criteria for classifying concepts as implicit, such as the absence of direct lexical matches to the concept in the note text. These additions will allow readers to better assess the proportion of truly implicit cases and the robustness of the human verification step. revision: yes
Referee: [Results] Results section on length invariance: the claim that performance is 'largely invariant to note length' is used to argue that MedicalBench isolates reasoning difficulty. However, without details on how note length was measured, the distribution of lengths across implicit vs. explicit subsets, or statistical tests controlling for other confounders (e.g., concept rarity or note complexity), the isolation from superficial factors remains incompletely supported.

Authors: We appreciate the referee's call for more rigorous analysis of the length invariance result. In the original manuscript, we reported that performance is largely invariant to note length but did not provide full methodological details. In the revision, we will clarify that note length was measured as the number of tokens in the discharge summary. We will also include histograms or summary statistics showing the length distributions for the implicit and explicit subsets. Furthermore, we will conduct and report statistical tests, such as linear regression or correlation analyses, that control for additional confounders including concept rarity (measured by frequency in the corpus) and note complexity (e.g., number of unique medical terms). This will provide stronger evidence that the benchmark's difficulty stems from implicit reasoning rather than superficial factors like length. revision: yes

Circularity Check

0 steps flagged

No significant circularity in dataset curation and empirical benchmarking

full rationale

The paper's contribution centers on constructing MedicalBench via a multi-stage LLM triage pipeline followed by human medical annotation and expert review from MIMIC-IV discharge summaries, then empirically benchmarking LLMs on implicit concept extraction and evidence retrieval tasks. No mathematical derivations, equations, or first-principles results are presented that reduce to inputs by construction. The reported length invariance is an empirical observation from evaluation, not a fitted parameter renamed as prediction or a self-definitional loop. Central claims rest on the deliberate inclusion of implicit positives and expert-disagreement cases, which is a design and curation choice externally verifiable against the released dataset rather than a self-citation chain or ansatz smuggled in. This is a standard self-contained benchmark paper with low circularity burden.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central contribution rests on assumptions about data quality and annotation fidelity rather than mathematical axioms or fitted parameters; no free parameters or invented entities are introduced.

axioms (2)

domain assumption MIMIC-IV discharge summaries contain a representative distribution of implicit and explicit medical concepts suitable for benchmarking LLM reasoning.
The dataset curation begins from MIMIC-IV as stated in the abstract.
domain assumption Human medical annotation and expert review provide reliable ground truth for implicit concept presence and evidence spans.
The multi-stage pipeline relies on this for curating positives, negatives, and disagreement cases.

pith-pipeline@v0.9.0 · 5806 in / 1351 out tokens · 79914 ms · 2026-05-21T10:25:35.610531+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

MedicalBench formulates medical concept extraction as a verification task over medical note–concept pairs, coupled with sentence-level evidence identification... curated through a multi-stage large language model (LLM) triage pipeline followed by medical annotation and expert review.
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

performance is largely invariant to note length, indicating that MedicalBench isolates reasoning difficulty rather than superficial confounders.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages · 1 internal anchor

[1]

URLhttps://ai.nejm.org/doi/ full/10.1056/AIdbp2401267

doi: 10.1056/AIdbp2401267. URLhttps://ai.nejm.org/doi/ full/10.1056/AIdbp2401267. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilin- gual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Viv...

work page doi:10.1056/aidbp2401267
[2]

LongBench:

Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172/. Katharina Beckh, Elisa Studeny, Sujan Sai Gannamaneni, Dario Antweiler, and Stefan Rueping. The anatomy of evidence: An investigation into explainable ICD coding. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Ta...

work page doi:10.18653/v1/2024.acl-long.172 2024
[3]

Fact, fetch, and reason: A unified evaluation of retrieval- augmented generation

Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025. findings-acl.864. URLhttps://aclanthology.org/2025.findings-acl.864/. Hua Cheng, Rana Jafari, April Russell, Russell Klopfer, Edmond Lu, Benjamin Striner, and Matthew Gormley. MDACE: MIMIC documents annotated with code evidence. In Anna Rogers, Jordan Boyd-Graber, an...

work page doi:10.18653/v1/2025 2025
[4]

doi: 10.18653/v1/2023.acl-long.416

Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.416. URL https://aclanthology.org/2023.acl-long.416/. Avisha Das, Ish A Talati, Juan Manuel Zambrano Chaves, Daniel Rubin, and Imon Banerjee. Weakly supervised language models for automated extraction of critical findings from radiology reports. npj Digital Medicine, 8(1):257,

work page doi:10.18653/v1/2023.acl-long.416 2023
[5]

En- tity anchored icd coding.arXiv preprint arXiv:2208.07444,

Jay DeYoung, Han-Chin Shing, Luyang Kong, Christopher Winestock, and Chaitanya Shivade. En- tity anchored icd coding.arXiv preprint arXiv:2208.07444,

work page arXiv
[6]

10 Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan ¨O

URL https://api.semanticscholar.org/CorpusID:232185406. 10 Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan ¨O. Arik. Long-context llms meet rag: Over- coming challenges for long inputs in rag.ArXiv, abs/2410.05983,

work page arXiv
[7]

Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark

URLhttps: //api.semanticscholar.org/CorpusID:273229050. Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv-note: Deidentified free-text clinical notes (version 2.2).PhysioNet, 2023a. doi: 10.13026/1n74-ne17. Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv (versi...

work page doi:10.13026/1n74-ne17
[8]

Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al

doi: 10.1038/s41597-022-01899-x. Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset.Scientific data, 10(1):1, 2023b. Mohamed Yassine Landolsi, Lobna Hlaoua, and Lotfi Ben Romdhane. Information...

work page doi:10.1038/s41597-022-01899-x
[9]

Lost in the Middle: How Language Models Use Long Contexts

doi: 10.1162/tacl a 00638. URL https://aclanthology.org/2024.tacl-1.9/. Ahmed Mattar, David Carlston, Glen Sariol, T. Yu, Ahmad Almustafa, Genevieve B. Melton, and Adil Ahmed. The prevalence of obesity documentation in primary care electronic medical records. Applied Clinical Informatics, 08:67 – 79,

work page internal anchor Pith review doi:10.1162/tacl 2024
[10]

Explainable prediction of medical codes from clinical text

James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. Explainable prediction of medical codes from clinical text. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.),Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)...

work page 2018
[11]

doi: 10.18653/v1/N18-1100

Association for Computational Linguistics. doi: 10.18653/v1/N18-1100. URLhttps://aclanthology.org/N18-1100/. Sujan Perera, Pablo Mendes, Amit Sheth, Krishnaprasad Thirunarayan, Adarsh Alex, Christopher Heid, and Greg Mott. Implicit entity recognition in clinical documents. In Martha Palmer, Gemma Boleda, and Paolo Rosso (eds.),Proceedings of the Fourth Jo...

work page doi:10.18653/v1/n18-1100
[12]

doi: 10.18653/v1/S15-1028

Association for Computational Linguistics. doi: 10.18653/v1/S15-1028. URLhttps://aclanthology. org/S15-1028/. PhysioToolkit PhysioBank. Physionet: components of a new research resource for complex physio- logic signals.Circulation, 101(23):e215–e220,

work page doi:10.18653/v1/s15-1028
[13]

Integrating physiological time series and clinical notes with deep learning for improved icu mortality prediction.arXiv preprint arXiv:2003.11059,

Satya Narayan Shukla and Benjamin M Marlin. Integrating physiological time series and clinical notes with deep learning for improved icu mortality prediction.arXiv preprint arXiv:2003.11059,

work page arXiv 2003
[14]

doi: 10.1093/jamia/ ocy068

ISSN 1527-974X. doi: 10.1093/jamia/ ocy068. URLhttps://doi.org/10.1093/jamia/ocy068. A APPENDIX A.1 PROMPT TEMPLATE Below is the template to extract medical concept and locate evidence. Unless otherwise specified, it is the default prompt for all LLMs in our experiments. 12 Prompt to extract concept and locate evidence You are an expert in annotating clin...

work page doi:10.1093/jamia/

[1] [1]

URLhttps://ai.nejm.org/doi/ full/10.1056/AIdbp2401267

doi: 10.1056/AIdbp2401267. URLhttps://ai.nejm.org/doi/ full/10.1056/AIdbp2401267. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilin- gual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Viv...

work page doi:10.1056/aidbp2401267

[2] [2]

LongBench:

Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172/. Katharina Beckh, Elisa Studeny, Sujan Sai Gannamaneni, Dario Antweiler, and Stefan Rueping. The anatomy of evidence: An investigation into explainable ICD coding. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Ta...

work page doi:10.18653/v1/2024.acl-long.172 2024

[3] [3]

Fact, fetch, and reason: A unified evaluation of retrieval- augmented generation

Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025. findings-acl.864. URLhttps://aclanthology.org/2025.findings-acl.864/. Hua Cheng, Rana Jafari, April Russell, Russell Klopfer, Edmond Lu, Benjamin Striner, and Matthew Gormley. MDACE: MIMIC documents annotated with code evidence. In Anna Rogers, Jordan Boyd-Graber, an...

work page doi:10.18653/v1/2025 2025

[4] [4]

doi: 10.18653/v1/2023.acl-long.416

Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.416. URL https://aclanthology.org/2023.acl-long.416/. Avisha Das, Ish A Talati, Juan Manuel Zambrano Chaves, Daniel Rubin, and Imon Banerjee. Weakly supervised language models for automated extraction of critical findings from radiology reports. npj Digital Medicine, 8(1):257,

work page doi:10.18653/v1/2023.acl-long.416 2023

[5] [5]

En- tity anchored icd coding.arXiv preprint arXiv:2208.07444,

Jay DeYoung, Han-Chin Shing, Luyang Kong, Christopher Winestock, and Chaitanya Shivade. En- tity anchored icd coding.arXiv preprint arXiv:2208.07444,

work page arXiv

[6] [6]

10 Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan ¨O

URL https://api.semanticscholar.org/CorpusID:232185406. 10 Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan ¨O. Arik. Long-context llms meet rag: Over- coming challenges for long inputs in rag.ArXiv, abs/2410.05983,

work page arXiv

[7] [7]

Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark

URLhttps: //api.semanticscholar.org/CorpusID:273229050. Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv-note: Deidentified free-text clinical notes (version 2.2).PhysioNet, 2023a. doi: 10.13026/1n74-ne17. Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv (versi...

work page doi:10.13026/1n74-ne17

[8] [8]

Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al

doi: 10.1038/s41597-022-01899-x. Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset.Scientific data, 10(1):1, 2023b. Mohamed Yassine Landolsi, Lobna Hlaoua, and Lotfi Ben Romdhane. Information...

work page doi:10.1038/s41597-022-01899-x

[9] [9]

Lost in the Middle: How Language Models Use Long Contexts

doi: 10.1162/tacl a 00638. URL https://aclanthology.org/2024.tacl-1.9/. Ahmed Mattar, David Carlston, Glen Sariol, T. Yu, Ahmad Almustafa, Genevieve B. Melton, and Adil Ahmed. The prevalence of obesity documentation in primary care electronic medical records. Applied Clinical Informatics, 08:67 – 79,

work page internal anchor Pith review doi:10.1162/tacl 2024

[10] [10]

Explainable prediction of medical codes from clinical text

James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. Explainable prediction of medical codes from clinical text. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.),Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)...

work page 2018

[11] [11]

doi: 10.18653/v1/N18-1100

Association for Computational Linguistics. doi: 10.18653/v1/N18-1100. URLhttps://aclanthology.org/N18-1100/. Sujan Perera, Pablo Mendes, Amit Sheth, Krishnaprasad Thirunarayan, Adarsh Alex, Christopher Heid, and Greg Mott. Implicit entity recognition in clinical documents. In Martha Palmer, Gemma Boleda, and Paolo Rosso (eds.),Proceedings of the Fourth Jo...

work page doi:10.18653/v1/n18-1100

[12] [12]

doi: 10.18653/v1/S15-1028

Association for Computational Linguistics. doi: 10.18653/v1/S15-1028. URLhttps://aclanthology. org/S15-1028/. PhysioToolkit PhysioBank. Physionet: components of a new research resource for complex physio- logic signals.Circulation, 101(23):e215–e220,

work page doi:10.18653/v1/s15-1028

[13] [13]

Integrating physiological time series and clinical notes with deep learning for improved icu mortality prediction.arXiv preprint arXiv:2003.11059,

Satya Narayan Shukla and Benjamin M Marlin. Integrating physiological time series and clinical notes with deep learning for improved icu mortality prediction.arXiv preprint arXiv:2003.11059,

work page arXiv 2003

[14] [14]

doi: 10.1093/jamia/ ocy068

ISSN 1527-974X. doi: 10.1093/jamia/ ocy068. URLhttps://doi.org/10.1093/jamia/ocy068. A APPENDIX A.1 PROMPT TEMPLATE Below is the template to extract medical concept and locate evidence. Unless otherwise specified, it is the default prompt for all LLMs in our experiments. 12 Prompt to extract concept and locate evidence You are an expert in annotating clin...

work page doi:10.1093/jamia/