Pith.

Computation and Language · arXiv:2611.17053 v1 · submitted 12 Nov 2026 · 28 references

Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks

Ashvin Gupta, Brendan Delaney, Alessandra Russo

The paper says

“Structured Electronic Health Records (EHR) models typically treat each clinical concept as a dataset-specific token, discarding the natural-language semantics associated with the event.”

more of the abstract

As a result, relationships between concepts must largely be learned from EHR co-occurrence, without directly exploiting any prior semantic knowledge. We investigate how pretrained causal language models can be adapted for longitudinal EHR prediction by developing tEHR. Here, we represent events with their natural language description and use this text stream to apply EHR-adaptive continued pretraining, followed by task-specific discriminative fine-tuning using LoRA, utilising the LLM as a longitudinal patient-encoder. We evaluate this approach across four prediction tasks on the MIMIC-IV dataset and pancreatic cancer prediction on the CPRD dataset at 1 to 12 month horizons, with tEHR using the Mistral-7B-v0.3 backbone achieving the strongest overall performance. Ablations show that performance declines when clinical descriptions are removed or misaligned, when numerical values are omitted, and when either stage of adaptation is removed, while explicit elapsed-time descriptions provide little additional benefit. Analysis of the learned patient representations further reveals clinically coherent structure associated with distinct patterns of disease risk and clinical history.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Solid empirical paper on using LLMs for EHR prediction; the central claim about lexical grounding holds up, but the stress-test concern about the scrambled-concepts ablation is real and not fully addressed. This line is wrong

the referee pass did not run

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 28 references say.

What would settle it

Replace each clinical code's description with a description of a semantically unrelated concept from a non-medical domain (e.g., legal terms, financial instruments) that preserves the same part-of-speech distribution and sentence structure. If performance drops to near the 'hashed concepts' level, the original ablation's interpretation is supported. If performance remains closer to the full tEHR level, then the model is leveraging general linguistic patterns rather than specific clinical semantics.

from the skeptic's reading

How the review went

3 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 7 s · 22,519 tokens read, 641 written
  3. Editor wrote the verdict above 8 s · 23,353 tokens read, 649 written

Not run on this paper: the referee, the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Hey,

This is a well-executed empirical paper. The core idea is straightforward: convert structured EHR events into natural-language descriptions and adapt a causal LLM through continued pretraining + discriminative fine-tuning. The main contribution is the systematic ablation study that isolates the contribution of pretrained language semantics. The paper does this cleanly, with multiple baselines, three seeds, and honest reporting of limitations.

What's new: The two-stage framework (continued pretraining then LoRA fine-tuning) applied to textualised EHR is not entirely novel—there are prior works doing pieces of this—but the thorough ablations are the value add. The finding that lexical grounding matters more than elapsed-time descriptions, and that the atomic-code variant underperforms despite same backbone, is genuinely informative for the field.

The paper is sound. The experiments are well-designed, the results are clear, and the authors acknowledge the main limitations: CPRD evaluation is only one task, the ablation suite is only on MIMIC-IV, computational cost is high, and the CLMBR-T comparison is incomplete due to vocabulary mapping issues. They also note that clustering does not establish causality.

Soft spots: The stress-test concern about the scrambled-concepts ablation is real. The authors argue that scrambling 'breaks the correspondence between each concept and its true clinical meaning' while preserving 'plausible medical language and broad event type.' But scrambled descriptions still form coherent English, and the LLM could exploit general linguistic patterns (syntax, distributional cues) that correlate with outcomes. The ablation does not fully isolate lexical semantics from these other signals. A stronger control would be to replace descriptions with semantically unrelated but syntactically similar text from a different domain (e.g., legal or financial). The paper does not address this confound. That said, the atomic-code variant (which removes all lexical semantics) also shows a large drop, which partially mitigates the concern—but the scrambled condition is still ambiguous.

Another minor point: the paper claims tEHR 'avoids dataset-specific preprocessing,' but the textualisation still requires mapping codes to descriptions via official coding-system metadata, which is itself a form of preprocessing. This is a minor overstatement.

Who is this for: Researchers working on clinical AI, especially those building EHR foundation models. The paper provides practical guidance on what matters (lexical grounding, continued pretraining) and what doesn't (elapsed-time descriptions). It deserves a serious referee.

Recommendation: Accept with minor revisions. The scrambled-concepts ablation should be discussed more carefully, and the authors should either add a stronger control or explicitly acknowledge the confound. The paper is otherwise solid.

Best, [Your name]

The strongest objection

The paper's central claim is that pretrained language semantics are a key contributor to tEHR's performance. The key ablation is 'scrambled concepts' (Section 3.3), where concept descriptions are deterministically permuted within event categories for each patient. The authors argue this 'breaks the correspondence between each concept and its true clinical meaning' while preserving 'plausible medical language and broad event type'. However, this ablation may not fully isolate lexical semantics from other cues. The scrambled descriptions still form coherent English text, and the LLM's pretrained knowledge of syntax, grammar, and general medical terminology (e.g., 'pancreatitis' vs 'diabetes') could still provide signal. The model might exploit distributional patterns in the scrambled text that correlate with outcomes, even if individual concept-description mappings are wrong. The ablation does not control for the possibility that the LLM is using general linguistic patterns rather than specific semantic knowledge of each code. A stronger test would be to compare against a condition where descriptions are replaced with semantically unrelated but syntactically similar text from a different domain (e.g., legal or financial descriptions), which would preserve linguistic structure while removing clinical semantics. The paper acknowledges limitations but does not address this specific confound.

The reader's weakest_assumption focuses on whether the natural language descriptions are a faithful and sufficient representation, and notes that the paper does not compare against using code hierarchies. My concern is related but distinct: it targets the internal validity of the key ablation that is used to support the central claim about pretrained language semantics. The 'scrambled concepts' ablation is the primary evidence that lexical grounding matters, but it may not fully isolate the contribution of pretrained semantics from other linguistic features. This is a load-bearing concern because the paper's main contribution is demonstrating the importance of pretrained language semantics. However, the paper does provide other supporting evidence (e.g., atomic codes ablation, hashed concepts, alternative semantic injection methods) that collectively strengthen the claim. The concern is real but not fatal; the paper's overall conclusion is still well-supported. Therefore, I recommend UNCHANGED verdict. I partially agree with the reader because my concern is about the interpretation of a specific ablation rather than the overall assumption about description faithfulness.

Pith's reading of the paper (a machine's summary, not the authors')

Electronic health records store patient information using medical codes like ICD-10, which are essentially arbitrary identifiers. Traditional machine learning models must learn the relationships between these codes from scratch, requiring large amounts of data and making it difficult to transfer models between hospitals that use different coding systems. tEHR takes a different approach: it converts each medical code into its plain English description (e.g., 'type 2 diabetes' instead of 'E11.9') and feeds this text stream to a large language model. The model is first adapted to the general structure of patient records through continued pretraining, where it learns to predict the next clinical event in a sequence. Then, for a specific prediction task like mortality or cancer diagnosis, the model is fine-tuned using a special [PREDICT] token and a linear classifier. The paper evaluates tEHR on four tasks from the MIMIC-IV intensive care dataset and pancreatic cancer prediction from the CPRD primary care dataset. tEHR with the Mistral-7B model achieves the best overall performance, outperforming traditional code-based models and other LLM-based approaches. Through careful ablation experiments, the authors show that the natural language descriptions are crucial: replacing them with meaningless identifiers or scrambled descriptions significantly hurts performance. Including numerical values like lab results also helps, but explicitly describing the time between events provides little additional benefit. The learned patient representations reveal clinically meaningful clusters, such as a group of patients enriched for pancreatic cancer with associated symptoms and biomarkers.

Core claim. tEHR, a framework that converts structured EHR events into natural language descriptions and adapts a pretrained LLM through continued pretraining and discriminative fine-tuning, achieves strong predictive performance across multiple clinical prediction tasks and datasets, with the pretrained language semantics being a key contributor to this performance (Table 1, Figure 2, and ablation studies in Figure 3).

The premise it rests on. The paper assumes that the natural language descriptions of clinical codes (from official coding system metadata) are a faithful and sufficient representation of the clinical concepts for the purpose of prediction (Section 2.2: 'For each event e_i, the symbolic concept c_i is replaced by its human-readable description using official coding-system translation metadata'). If these descriptions are ambiguous, incomplete, or lose important information present in the code structure (e.g., hierarchical relationships), the model's performance could be affected. The paper partially addresses this through the 'scrambled concepts' ablation, but does not compare against using code hierarchies or other structured representations.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Structured Electronic Health Records (EHR) models typically treat each clinical concept as a dataset-specific token, discarding the natural-language semantics associated with the event. As a result, relationships between concepts must largely be learned from EHR co-occurrence, without directly exploiting any prior semantic knowledge. We investigate how pretrained causal language models can be adapted for longitudinal EHR prediction by developing tEHR. Here, we represent events with their natural language description and use this text stream to apply EHR-adaptive continued pretraining, followed by task-specific discriminative fine-tuning using LoRA, utilising the LLM as a longitudinal patient-encoder. We evaluate this approach across four prediction tasks on the MIMIC-IV dataset and pancreatic cancer prediction on the CPRD dataset at 1 to 12 month horizons, with tEHR using the Mistral-7B-v0.3 backbone achieving the strongest overall performance. Ablations show that performance declines when clinical descriptions are removed or misaligned, when numerical values are omitted, and when either stage of adaptation is removed, while explicit elapsed-time descriptions provide little additional benefit. Analysis of the learned patient representations further reveals clinically coherent structure associated with distinct patterns of disease risk and clinical history.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-14

The authors' main claim, as the classifier picked it out 0.99

“We evaluate this approach across four prediction tasks on the MIMIC-IV dataset and pancreatic cancer prediction on the CPRD dataset at 1 to 12 month horizons, with tEHR using the Mistral-7B-v0.3 backbone achieving the strongest overall performance.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.87 · human participants 0.78 · controlled comparison 0.44 · randomized assignment 0.01 · states quantitative result 0.38 · compares with prior results 0.46 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Structured Electronic Health Records (EHR) models typically treat each clinical concept as a dataset-specific token, discarding the natural-language semantics associated with the event. background 0.94 0.06 0.03 asserted 1.00
As a result, relationships between concepts must largely be learned from EHR co-occurrence, without directly exploiting any prior semantic knowledge. background 0.58 0.06 0.03 asserted 0.98
We investigate how pretrained causal language models can be adapted for longitudinal EHR prediction by developing tEHR. contribution 0.99 0.91 0.06 asserted 0.85
Here, we represent events with their natural language description and use this text stream to apply EHR-adaptive continued pretraining, followed by task-specific discriminative fine-tuning using LoRA, utilising the LLM as a longitudinal patient-encoder. method 0.99 0.95 0.06 asserted 0.98
We evaluate this approach across four prediction tasks on the MIMIC-IV dataset and pancreatic cancer prediction on the CPRD dataset at 1 to 12 month horizons, with tEHR using the Mistral-7B-v0.3 backbone achieving the strongest overall performance. result 0.93 0.97 0.61 asserted 1.00
Ablations show that performance declines when clinical descriptions are removed or misaligned, when numerical values are omitted, and when either stage of adaptation is removed, while explicit elapsed-time descriptions provide little additional benefit. result 1.00 0.97 0.16 asserted 1.00
Analysis of the learned patient representations further reveals clinically coherent structure associated with distinct patterns of disease risk and clinical history. interpretation 0.59 0.95 0.14 asserted 0.99

What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.66

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-14: 1 request, 5,736 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. Structured EHRs are converted into the MEDS event-based representation (Bert Arnrich et al., 2024) and textualised using natural-language descriptions of clinical events, numerical values, and elapsed time between events. A pretrained large language model (LLM) is adapted to unlabelled patient trajectories using next-token prediction, with low-rank adaptation (LoRA) while keeping the backbone frozen. For downstream prediction tasks, patient trajectories are truncated at the prediction time and appended with a [PREDICT] token. The adapted LLM produces a contextual representation of…
Figure 2 of the paper
Figure 2. AUROC for pancreatic cancer prediction at horizons of 1, 3, 6 and 12 months preced￾ing diagnosis for tEHR, XGBoost, Trans￾former Encoder, BioClinical-ModernBERT and Zero-shot LLM. UMAP projection followed by HDBSCAN cluster￾ing of the [PREDICT] patient embeddings (
Figure 3 of the paper
Figure 3. Change in AUROC relative to the full tEHR-Mistral-7B model across four downstream tasks on MIMIC-IV for the ablation experiments
Figure 4 of the paper
Figure 4. Two-dimensional UMAP visualisation of pancreatic-cancer patient embeddings. Colour denotes case-control status and opacity denotes predicted cancer probabil￾ity. 5. Discussion We present tEHR, a framework for adapting pre￾trained autoregressive LLMs to EHR prediction tasks. tEHR can in principle be applied to different autore￾gressive LLMs and relies on clinical events being repre￾sented with their natural-language descriptions rather than as symbolic tokens, allowing the model to reuse semantic information learnt during large-scale pre￾training. tEHR requires simple data…
Figure 5 of the paper
Figure 5. Clinical phenotypes of interest and their fold enrichment for each cluster. Cluster 0: baseline, cluster 1: pancreatic malignancy, cluster 2: inflammatory phenotype, cluster 3: gynaecological and urological conditions
Figure 6 of the paper
Figure 6. Lab value distribution by cluster. Red dotted lines indicate clinically healthy range.
Figure 7 of the paper
Figure 7. AUPRC for pancreatic cancer prediction at horizons of 1, 3, 6 and 12 months preceding diagnosis for tEHR, XGBoost, Transformer Encoder and BioClinical ModernBERT and zero-shot LLM. 21

Figures are the authors', reproduced from the paper.

References (28)
  1. [1] Placido, Davide and Yuan, Bo and Hjaltelin, Jessica X. and Zheng, Chunlei and Haue, Amalie D. and Chmura, Piotr J. and Yuan, Chen and Kim, Jihye and Umeton, Renato and Antell, Gregory and Chowdhury, Alexander and Franz, Alexandra and Brais, Lauren and Andrews, Elizabeth and Ma... on Pith
  2. [2] Makarov, Nikolai and Lipkovich, Mikhail , month =. 2025 , journal =. doi:10.1140/epjs/s11734-024-01447-1 , issn = on Pithdoi:10.1140/epjs/s11734-024-01447-1
  3. [3] Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina , pages =. 2019 , booktitle = on Pith
  4. [4] DeLong, Elizabeth R and DeLong, David M and Clarke-Pearson, Daniel L , number =. 1988 , journal = on Pith
  5. [5] Herrett, Emily and Gallagher, Arlene M and Bhaskaran, Krishnan and Forbes, Harriet and Mathur, Rohini and van Staa, Tjeerd and Smeeth, Liam , number =. 2015 , journal =. doi:10.1093/ije/dyv098 , issn = on Pithdoi:10.1093/ije/dyv098
  6. [6] Wolf, Achim and Dedman, Daniel and Campbell, Jennifer and Booth, Helen and Lunn, Darren and Chapman, Jennifer and Myles, Puja , number =. 2019 , journal =. doi:10.1093/ije/dyz034 , issn = on Pithdoi:10.1093/ije/dyz034
  7. [7] Meng, Qingcai and Shi, Si and Liang, Chen and Xiang, Jinfeng and Liang, Dingkong and Zhang, Bo and Qin, Yi and Ji, Shunrong and Xu, Wenyan and Xu, Jin and Ni, Quanxing and Yu, Xianjun , number =. 2017 , journal =. doi:10.7150/jca.18901 , issn = on Pithdoi:10.7150/jca.18901
  8. [8] Xing, Haibo and Wang, Jing and Wang, Yanling and Tong, Mengting and Hu, Hong and Huang, Changxin and Li, Da , pages =. 2018 , journal =. doi:10.1155/2018/8704751 , issn = on Pithdoi:10.1155/2018/8704751
  9. [9] Park, Jiheum and Patterson, Jason and Acitores Cortina, Jose M. and Gu, Tian and Hur, Chin and Tatonetti, Nicholas , number =. 2025 , journal =. doi:10.1038/s41746-025-01869-8 , issn = on Pithdoi:10.1038/s41746-025-01869-8
  10. [10] Newsome, Philip N and Cramb, Rob and Davison, Suzanne M and Dillon, John F and Foulerton, Mark and Godfrey, Edmund M and Hall, Richard and Harrower, Ulrike and Hudson, Mark and Langford, Andrew and Mackie, Anne and Mitchell-Thain, Robert and Sennett, Karen and Sheron, Nicholas... on Pithdoi:10.1136/gutjnl-2017-314924
  11. [11] Montgomery, Maria Elkjær and Igel, Christian and Odgaard, Mikkel Fruelund and Sillesen, Martin and Nielsen, Mads , url =. 2026 , booktitle = on Pith
  12. [12] Hegselmann, Stefan and von Arnim, Georg and Rheude, Tillmann and Kronenberg, Noel and Sontag, David and Hindricks, Gerhard and Eils, Roland and Wild, Benjamin , number =. 2026 , journal =. doi:10.1038/s41746-026-02915-9 , issn = on Pithdoi:10.1038/s41746-026-02915-9
  13. [13] Shmatko, Artem and Jung, Alexander Wolfgang and Gaurav, Kumar and Brunak, Søren and Mortensen, Laust Hvas and Birney, Ewan and Fitzgerald, Tom and Gerstung, Moritz , month =. 2025 , journal =. doi:10.1038/s41586-025-09529-3 , issn = on Pithdoi:10.1038/s41586-025-09529-3
  14. [14] Schulman, John and Lab, Thinking Machines , doi =. 2025 , journal = on Pith
  15. [15] Rasmy, Laila and Xiang, Yang and Xie, Ziqian and Tao, Cui and Zhi, Degui , number =. 2021 , journal =. doi:10.1038/s41746-021-00455-y , issn = on Pithdoi:10.1038/s41746-021-00455-y
  16. [16] Johnson, Alistair and Bulgarelli, Lucas and Pollard, Tom and Gow, Brian and Moody, Benjamin and Horng, Steven and Celi, Leo Anthony and Mark, Roger , month =. 2024 , journal = on Pith
  17. [17] 2019 , author =. doi:10.1038/s41597-019-0103-9 , arxivId = on Pithdoi:10.1038/s41597-019-0103-9
  18. [18] Tao, Lan Valerie and Karasinska, Joanna M. and Souza, Vanessa G. P. and Loree, Jonathan M. and Johnson, James D. and Renouf, Daniel J. and Schaeffer, David F. , month =. 2026 , journal =. doi:10.3389/fgstr.2025.1645459 , issn = on Pithdoi:10.3389/fgstr.2025.1645459
  19. [19] McGuigan, Andrew and Kelly, Paul and Turkington, Richard C and Jones, Claire and Coleman, Helen G and McCain, R Stephen , number =. 2018 , journal =. doi:10.3748/wjg.v24.i43.4846 , issn = on Pithdoi:10.3748/wjg.v24.i43.4846
  20. [20] Garrido, Quentin and Balestriero, Randall and Najman, Laurent and Lecun, Yann , editor =. 2023 , booktitle = on Pith
show 8 more
  1. [21] Stearns, M Q and Price, C and Spackman, K A and Wang, A Y , pages =. 2001 , journal = on Pith
  2. [22] Schmidt-Hansen, Mia and Berendse, Sabine and Hamilton, William , number =. 2016 , journal =. doi:10.1097/MPA.0000000000000527 , issn = on Pithdoi:10.1097/mpa.0000000000000527
  3. [23] Huang, Da Wei and Sherman, Brad T and Lempicki, Richard A , number =. 2009 , journal =. doi:10.1038/nprot.2008.211 , issn = on Pithdoi:10.1038/nprot.2008.211
  4. [24] Chisholm, J , number =. 1990 , journal =. doi:10.1136/bmj.300.6732.1092 , issn = on Pithdoi:10.1136/bmj.300.6732.1092
  5. [25] Wang, Lan and Yin, Yonghua and Glampson, Ben and Peach, Robert and Barahona, Mauricio and Delaney, Brendan C. and Mayer, Erik K. , month =. 2024 , journal =. doi:10.1016/j.ebiom.2024.105442 , issn = on Pithdoi:10.1016/j.ebiom.2024.105442
  6. [26] Draxler, Felix and Meng, Yang and Nelson, Kai and Laskowski, Lukas and Yang, Yibo and Karaletsos, Theofanis and Mandt, Stephan , url =. 2025 , booktitle = on Pith
  7. [27] Stewart, Elizabeth A , number =. 2001 , journal =. doi:10.1016/S0140-6736(00)03622-9 , issn = on Pithdoi:10.1016/s0140-6736(00)03622-9
  8. [28] Renc, Pawel and Jia, Yugang and Samir, Anthony E. and Was, Jaroslaw and Li, Quanzheng and Bates, David W. and Sitek, Arkadiusz , number =. 2024 , journal =. doi:10.1038/s41746-024-01235-0 , issn = on Pithdoi:10.1038/s41746-024-01235-0

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment