Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Pre-training a clinical time-series encoder against LLM-encoded event summaries improves ICU outcome predictions and transfers to new patient cohorts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A pre-training method that aligns ICU time-series windows with LLM-encoded event summaries via a regularised InfoNCE loss improves downstream predictions and cross-dataset transfer.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection Solid empirical pretraining recipe with a confounded mechanism: event embeddings contain the same window's TS statistics, so the 'event-guided' story is unproven — but the paper is worth a serious referee and the flaw is fixable. the 3 major comments →

arxiv 2607.15447 v1 pith:SNEGHXLV submitted 2026-07-16 cs.LG

LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models

Jingteng Li , Alexander Capstick , Louise Rigny , Iona Biggart , Neil J Sebire , Payam Barnaghi This is my paper
classification cs.LG
keywords Electronic health recordsClinical predictive modelingContrastive learningRepresentation learningLarge Language ModelTime seriesICU foundation modelPre-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that routinely recorded ICU events—prescriptions, infusions, ventilator settings, line insertions—and vital-sign time series share a temporal structure that existing clinical models leave unexploited. It presents LLM4EHR, a pre-training method that aligns each window's time-series embedding with the embedding of a language-model-encoded plain-text summary of that window's clinical events, using a regularised InfoNCE objective. The claim is that this temporal alignment produces time-series representations that improve downstream ICU predictions—mortality, phenotyping, decompensation, and remaining length of stay—and that these representations adapt to new patient cohorts with few labels. If correct, the method converts unstructured event logs into free supervision for clinical time-series foundation models.

Core claim

On its own terms, the paper establishes that pre-training a transformer time-series encoder with a temporal, semantic-regularised contrastive objective against LLM-embedded EHR event summaries yields time-series embeddings that outperform specialised supervised models and existing contrastive pre-training baselines across four ICU prediction tasks, and that these embeddings transfer to a second public ICU cohort via k-shot adaptation. The authors show the largest gains come from combining the temporal alignment loss with an autoregressive reconstruction auxiliary objective, and that the semantic regulariser gives marginal but consistent improvements. They further show that long-context, doma

What carries the argument

The load-bearing mechanism is a regularised InfoNCE contrastive loss applied at each time step: a frozen, domain-adapted language model encodes a fixed-schema plain-language summary of all clinical events in a 5-hour window (z_t), a trainable transformer encoder produces the time-series embedding (v_t), and the loss pulls v_t toward z_t while an omega-regulariser weights pairwise attraction among time points by the semantic similarity of their event summaries. This transfers event semantics into the TS encoder and softens the class-collision problem of strict one-to-one contrastive sampling; an auxiliary autoregressive reconstruction of the input TS stabilises pre-training.

Load-bearing premise

The 'event' side of the contrastive loss already contains a plain-text summary of the same window's time-series values, so the alignment may be teaching the TS encoder to match a textual echo of itself; without an ablation removing that text, the claim that event semantics, rather than the appended TS summary, drive the gains is not established.

What would settle it

Pre-train LLM4EHR twice on the same ICU data: once with the plain-text TS summary removed from the event-side text (events only) and once with event summaries shuffled across time windows while keeping the text intact. If removing the TS summary collapses the downstream gains, or if shuffling the event order does not hurt, then temporal event alignment is not the active mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Routinely recorded event logs can serve as free, task-agnostic supervision for clinical time-series models, reducing reliance on expert labels.
  • Event-aligned representations improve rare-phenotype classification, which is typically the hardest setting for supervised models.
  • k-shot adaptation to a new ICU cohort suggests a single pre-trained encoder can be deployed across sites with a handful of locally labelled patients.
  • The framework is language-model-agnostic; stronger long-context clinical LLMs would likely further improve the learned time-series embeddings.
  • Because only the time-series encoder is used at inference, the pre-training adds no extra cost or data requirement at prediction time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The event-side summary appends a plain-text summary of the same window's TS values; without an ablation removing that text, it is unclear whether the gains come from event semantics or from a textual re-encoding of the TS itself. This is the paper's unstated circular step and should be tested.
  • If the temporal alignment is the true carrier, bin width (5 hours) is pivotal; sweeping bin sizes would reveal the temporal resolution at which event context helps and would link to event sparsity.
  • The omega regulariser is a soft, LLM-derived similarity; comparing it against random or TS-derived soft assignments would isolate whether semantic similarity from the LLM, rather than any smooth weighting, drives the regularisation effect.
  • The same event/TS alignment recipe could extend beyond the ICU to outpatient records, and the paper's note about future natural-language inference suggests the event embeddings themselves may carry reusable clinical structure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LLM4EHR, a multimodal pre-training framework for ICU electronic health records. EHR events in each 5-hour window are rendered as natural-language summaries and embedded by a frozen, domain-adapted LLM; a trainable transformer encoder produces embeddings for the EHR time series (TS) in the same window. The two are aligned with a regularised InfoNCE objective that also weights TS-TS similarities by semantic similarities of the event embeddings, plus an autoregressive reconstruction loss. Downstream evaluation covers mortality, 25-way phenotyping, rolling decompensation, and remaining length-of-stay on MIMIC-IV, with transfer to Physionet2012 via full-shot and k-shot adaptation. The paper reports consistent gains over supervised and self-supervised baselines, supported by ablations of the objective, the choice of LLM, and the contrastive temperature.

Significance. If the central claim holds — that temporally aligning TS embeddings with LLM-encoded EHR event summaries transfers event semantics into a reusable TS encoder — the method would provide a practical recipe for using event streams as free supervision in clinical foundation models. The evaluation is unusually thorough for the area: 10 seeds, paired t-tests, task-by-task hyperparameter search, an ablation study, honest reporting of poor LoS R2 and low-k behavior, and a public code repository. The transfer experiments to an external cohort are a real strength. However, the main empirical gains are not currently diagnostic of the proposed mechanism, because the 'event' embedding already contains a statistical summary of the very TS window being aligned. The paper needs one or two targeted ablations to establish that event semantics — rather than the embedded TS statistics — drive the improvements.

major comments (3)
  1. [§4.1, Table 1; §5.6] The central mechanistic claim is not established because z_t is not a pure event embedding. Section 4.1 states: 'Finally, we append a plain text summary of TS variables to the end of EHR event summary at t', and Table 1 confirms that the summary contains mean/min/max/std/count for the same window's TS variables. The contrastive loss in Eq. (1) therefore aligns v_t with a representation that already contains information about the TS window itself. Since the projection layer on z_t is trainable, the objective can be minimized by making v_t imitate those TS statistics, and the observed downstream gains may reflect this shortcut rather than alignment to event semantics. The comparison to King et al. is confounded for the same reason: their clinical notes do not contain the target window's TS statistics. The paper needs an ablation that removes the TS summary block from the EHR summary, and i
  2. [§5.6, Table 8] The ablation study is informative but does not fully support the paper's attribution of gains to the semantic alignment objective. Table 8 shows that L_recon alone already achieves strong results, L_NCE alone is much weaker, and the largest improvement comes from L_recon + L_NCE; L_omega adds only marginal, albeit consistent, gains. The paper marks significance only between the best and second-best configurations. The reader cannot tell whether L_recon + L_NCE is significantly better than L_recon alone, which is the comparison that would establish that temporal alignment (rather than reconstruction or the TS-summary shortcut) is responsible for the improvement. Please report paired significance tests for all adjacent ablation pairs, not only best vs. second-best, and ideally include the no-TS-summary condition from the previous comment in this table.
  3. [§4.2, Eq. (3)–(4)] The class-collision motivation for L_omega is only partially realized. The loss weights each pair (t, t*), but the denominator in Eq. (3) still contains all other timestamps in T\{t}, including other timestamps t' that are semantically similar to t and should not be treated as negatives. The soft assignment omega mitigates the problem by up-weighting similar pairs, but it does not remove the collision; similar instances still appear as negatives for each other in the denominator. This is not fatal to the method, but the text should state this limitation or modify the denominator to exclude semantically similar negatives. A more careful formulation would also make the regulariser symmetric, matching the claim that omega is symmetric.
minor comments (5)
  1. [§4.1] The notation k_{1:T} = {k_1^{n_1}, ..., k_T^{n_T}} is confusing because n_t is not defined formally and the superscript resembles an exponent. Please clarify that n_t denotes the (variable) number of events at time t.
  2. [§4.2, Eq. (3)] The temperature symbol tau is used in both L_NCE and L_omega, but the text later says tau is kept at 0.14 for L_omega while tuned for L_NCE. Please define tau_omega explicitly and state the value used in the main results.
  3. [§5.2, Table 3] The hyperparameter search spaces are given, but the final selected hyperparameters for each baseline and for LLM4EHR are not reported. Providing a table of final hyperparameters would improve reproducibility.
  4. [§5.4, Table 6] For the LoS task, the paper reports MAE on uncensored predictions and C-index on all predictions, but it is not stated whether the evaluated model outputs are the predicted median or mean of the log-normal distribution. This matters for interpreting MAE and for reproducing the results.
  5. [§1 and Figure 1c] Figure 1c is referenced in the introduction but not described in the caption in enough detail. Please add a sentence explaining what the soft assignment omega looks like and how it modifies the contrastive objective.

Circularity Check

1 steps flagged

Event embeddings include the same window's TS summary, so the 'event' target is partly self-referential; the event-semantics claim is not isolated by the ablations.

specific steps
  1. self definitional [Section 4.1 (LLM4EHR structure), Table 1]
    "EHR TS variables within each time step are aggregated with windowed features (mean, min, max, std and count). ... Finally, we append a plain text summary of TS variables to the end of EHR event summary at t to provide an additional layer of mutual information between the two modalities."

    The positive pair in Eq. (1) is (v_t, z_t), where v_t is the TS encoder's embedding of the 5-hour TS window and z_t is the LLM embedding of the EHR summary for that same window. Because the EHR summary includes a text block of the same TS window's mean/min/max/std/count (Table 1), z_t already contains information computed from v_t's input. The InfoNCE loss can therefore be minimized by making v_t imitate these summary statistics, without transferring any event-specific semantics. The paper's central claim that 'event semantics' guide the TS encoder is thus not established by the objective; an ablation that removes the TS text block from the event summary is absent, and Table 8 varies only the loss, holding the summary content fixed.

full rationale

The downstream evaluations (Tables 4, 6, 7) use held-out labels and are independent of the pre-training objective, so the empirical performance gains are not label-fitted and there is no fitted-input-called-prediction circularity. There are no load-bearing self-citations or imported uniqueness theorems; citations to the authors' prior work are not central. The circularity is confined to the construction of the event embedding: since the EHR summary includes a plain-text version of the same TS summary statistics that the TS encoder sees, the cross-modal alignment is partly self-referential. Consequently, the specific mechanistic claim that event semantics are transferred to the TS encoder is underdetermined, and the method is better described as aligning TS embeddings with an LLM-encoded mixture of events and TS summaries. This is a partial, not total, circularity: the downstream benefits are still empirically measured, so the central result retains independent content.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical entities. Free parameters are standard hyperparameters of contrastive pre-training. The key untested assumption is that the TS summary text appended to the event summary does not turn the alignment into self-distillation.

free parameters (4)
  • Bin width = 5 hours
    Hourly alignment fails due to sparse events; 5-hour bins were chosen by hand to balance emptiness and temporal consistency (Section 4.1).
  • Contrastive temperature τ for L_NCE = 0.02
    Selected based on downstream validation performance across tasks (Table 11).
  • Contrastive temperature for L_ω = 0.14
    Kept fixed while tuning τ for L_NCE; chosen to avoid pre-training instability (Section 5.6).
  • Projection layer momentum m = 0.9
    Used to stabilise z_t during pre-training (Section 4.2).
axioms (4)
  • domain assumption EHR events and EHR TS are temporally aligned and jointly informative about patient state.
    Core premise of the alignment objective (Section 3.2).
  • domain assumption Frozen pre-trained LLM embeddings of event summaries provide semantic structure that can guide TS representations.
    Assumes domain-adapted LLM (BioClinical ModernBERT) captures clinically meaningful similarities (Section 4.2).
  • ad hoc to paper Appending a summary of TS statistics to the event summary improves mutual information without introducing leakage that invalidates the method.
    Design choice in Section 4.1; not independently justified and creates self-reference.
  • standard math Standard InfoNCE assumptions (positive pairs defined by alignment, negatives uniformly sampled) apply.
    Basis of L_NCE and L_ω (Equations 1-4).

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models." pith.science (2026). https://pith.science/paper/SNEGHXLV

@misc{pith2026260715447,
  author       = {Pith},
  title        = {Pith review of: LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SNEGHXLV}},
  note         = {Machine review of arXiv:2607.15447}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent research in clinical machine learning, focusing on outcome predictions in intensive care unit (ICU), has shifted from bespoke supervised models to foundation models, utilising modern representation learning methods. Here, foundation models are pre-trained on mixtures of complex clinical data modalities, useful for various downstream tasks. Existing works often utilise Electronic Health Records (EHR) to provide rich and diverse patient observations to train clinical foundation models. However, existing methods do not sufficiently explore the shared temporal structures between clinical events and time series (TS) observations recorded in EHRs. This limitation potentially leads to less robust and adaptive clinical foundation models, resulting in reduced performance on downstream tasks. To fully exploit this temporal structure, we propose LLM4EHR, a new clinical foundation model trained on ICU EHR data. Combining domain adapted large language models with a transformer TS encoder, we pre-trained LLM4EHR by temporally aligning the EHR events and TS. For this, we propose a regularised contrastive objective to learn robust EHR TS representations conditioned on EHR event embeddings produced by the domain adapted LLM. Supported by an ablation study, we find that learnt EHR TS embeddings from LLM4EHR improve performance on various downstream clinical tasks with competitive performance. Further, we empirically demonstrate that LLM4EHR learns transferable clinical TS embeddings that can be deployed to new cohorts via k-shot adaptation. These findings provide a step towards building more generalisable and performant clinical foundation models.

Figures

Figures reproduced from arXiv: 2607.15447 by Alexander Capstick, Iona Biggart, Jingteng Li, Louise Rigny, Neil J Sebire, Payam Barnaghi.

Figure 1
Figure 1. Figure 1: a) An example of EHR TS, b) an example of EHR events and c) the difference between our regularised contrastive objective and the InfoNCE [21, 22] objective, during LLM4EHR pre-training. EHR events embeddings produced by a domain adapted LLM areused to guide the EHR TS embeddings via our proposed objective. However, many current methods fail to incorporate mul￾tiple temporal EHR data modalities during pre-t… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of LLM4EHR, EHR entries and clinical TS are centred on ICU admissions, where EHR operations are coloured in beige and TS operations are coloured in red, pre-trained LLM layers (besides the embedding weights for new tokens) are frozen during pre-training. can lead to fine-grained temporal dynamics being lost or obscured during modelling. Self-supervised representation learning for TS learns transfe… view at source ↗
Figure 3
Figure 3. Figure 3: Calculating the semantic alignment loss for a given episode; the figures only include the lower triangular and non￾diagonal part of the soft assignment 𝜔 since 𝜔 is symmetric. Finally, we append a plain text summary of TS variables to the end of EHR event summary at 𝑡 to provide an additional layer of mutual information between the two modalities. 4.2. Semantic alignment The attraction between elements of … view at source ↗
Figure 4
Figure 4. Figure 4: Box plot for mean AUROC and AUPRC across all 25 phenotypes [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: AUROC (upper) and AUPRC (lower) for k-shot mortality prediction on Physionet2012 with mean and 95% confidence over 1000 bootstrap samples. moderate and high-k settings. The model by King et al., similarly pre-trained via cross-modal alignment on MIMIC￾IV, experienced a larger relative performance drop than LLM4EHR (−3.4% in AUROC) when adapted to Phys￾ionet2012, indicating that our temporal alignment objec… view at source ↗
Figure 6
Figure 6. Figure 6: Flowchart for curating the MIMIC-IV ICU stay patient cohort, short stays < 48 hours are removed to allow adequate patient observations before prediction. and Chen and Guestrin [49] respectively. For remaining LoS prediction, LLM4EHR and other deep learning baselines are fitted with a accelerated failure prediction head that predicts log-scale mean 𝜇 and standard deviation log 𝜎 of the expected time to disc… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 3 canonical work pages

  1. [1]

    Ghassemi, T

    M. Ghassemi, T. Naumann, F. Doshi-Velez, N. Brimmer, R. Joshi, A. Rumshisky, P. Szolovits, Unfolding physiological state, in: Proceedings of the 20th ACM SIGKDD international conference on Knowledgediscoveryanddatamining,ACM,2014,pp.75–84.URL: https://dl.acm.org/doi/10.1145/2623330.2623742 . doi:10.1145/2623 330.2623742

    arXiv 2014

  2. [2]

    K. L. C. Barajas, R. Akella, Dynamically modeling patient’s health state from electronic medical records, in: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery andDataMining,volume2015-August,ACM,2015,pp.69–78.URL: https://dl.acm.org/doi/10.1145/2783258.2783289 . doi:10.1145/2783 258.2783289

    arXiv 2015

  3. [3]

    C.A.Alvarez,C.A.Clark,S.Zhang,E.A.Halm,J.J.Shannon,C.E. Girod,L.Cooper,R.Amarasingham, Predictingoutofintensivecare unit cardiopulmonary arrest or death using electronic medical record data, BMC Medical Informatics and Decision Making 13 (2013) 28

    2013

  4. [4]

    Cheng, F

    Y. Cheng, F. Wang, P. Zhang, J. Hu, Risk prediction with electronic health records: A deep learning approach, in: Proceedings of the 2016 SIAM International Conference on Data Mining, Society for IndustrialandAppliedMathematics,2016,pp.432–440.URL: https: //epubs.siam.org/doi/10.1137/1.9781611974348.49 . doi:10.1137/1.97 81611974348.49

    work page doi:10.1137/1.9781611974348.49 2016

  5. [5]

    D. J. Albers, N. Elhadad, E. Tabak, A. Perotte, G. Hripcsak, Dy- namical phenotyping: Using temporal analysis of clinically collected physiologicdatatostratifypopulations, PLoSONE9(2014)e96443

    2014

  6. [6]

    C. Liu, F. Wang, J. Hu, H. Xiong, Temporal phenotyping from longitudinal electronic health records: A graph based framework, in: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, volume 2015-August, 2015. doi:10.1145/2783258.2783352

    arXiv 2015

  7. [7]

    : Preprint submitted to Elsevier Page 12 of 16

    H.Harutyunyan,H.Khachatrian,D.C.Kale,G.V.Steeg,A.Galstyan, Multitask learning and benchmarking with clinical time series data, Scientific Data 6 (2019) 96. : Preprint submitted to Elsevier Page 12 of 16

    2019

  8. [8]

    S.Sheikhalishahi,V.Balaraman,V.Osmani, Benchmarkingmachine learningmodelsonmulti-centreeicucriticalcaredataset, PLOSONE 15 (2020) e0235424

    2020

  9. [9]

    A.Radford,J.Wu,R.Child,D.Luan,D.Amodei,I.Sutskever,etal., Language models are unsupervised multitask learners, OpenAI blog 1 (2019) 9

    2019

  10. [10]

    Wornow, Y

    M. Wornow, Y. Xu, R. Thapa, B. Patel, E. Steinberg, S. Fleming, M. A. Pfeffer, J. Fries, N. H. Shah, The shaky foundations of large languagemodelsandfoundationmodelsforelectronichealthrecords, npj Digital Medicine 6 (2023) 135

    2023

  11. [11]

    Y. Li, S. Rao, J. R. A. Solares, A. Hassaine, R. Ramakrishnan, D. Canoy, Y. Zhu, K. Rahimi, G. Salimi-Khorshidi, Behrt: Trans- former for electronic health records, Scientific Reports 10 (2020) 7155

    2020

  12. [12]

    Rasmy, Y

    L. Rasmy, Y. Xiang, Z. Xie, C. Tao, D. Zhi, Med-bert: pretrained contextualizedembeddingsonlarge-scalestructuredelectronichealth records for disease prediction, npj Digital Medicine 4 (2021) 86

    2021

  13. [13]

    P. Renc, Y. Jia, A. E. Samir, J. Was, Q. Li, D. W. Bates, A. Sitek, Zero shot health trajectory prediction using transformer, npj Digital Medicine 7 (2024) 256

    2024

  14. [14]

    Kraljevic, D

    Z. Kraljevic, D. Bean, A. Shek, R. Bendayan, H. Hemingway, J. A. Yeung,A.Deng,A.Balston,J.Ross,E.Idowu,J.T.Teo,R.J.B.Dob- son, Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study, The Lancet Digital Health 6 (2024) e281–e290

    2024

  15. [15]

    Chang, W.-Y

    C. Chang, W.-Y. Wang, W.-C. Peng, T.-F. Chen, Llm4ts: Aligning pre-trainedllmsasdata-efficienttime-seriesforecasters, ACMTrans. Intell. Syst. Technol. 16 (2025)

    2025

  16. [16]

    Y. Liu, G. Qin, X. Huang, J. Wang, M. Long, Autotimes: Au- toregressive time series forecasters via large language models, in: A.Globerson,L.Mackey,D.Belgrave,A.Fan,U.Paquet,J.Tomczak, C. Zhang (Eds.), Advances in Neural Information Processing Sys- tems,volume37,CurranAssociates,Inc.,2024,pp.122154–122184. URL: https://proceedings.neurips.cc/paper_files/p...

    2024

  17. [17]

    R.King,T.Yang,B.J.Mortazavi, Multimodalpretrainingofmedical time series and notes, in: Machine Learning for Health (ML4H), PMLR, 2023, pp. 244–255

    2023

  18. [18]

    N. Chan, F. Parker, W. Bennett, T. Wu, M. Y. Jia, J. Fackler, K.Ghobadi,Medtsllm:Leveragingllmsformultimodalmedicaltime series analysis, 2024. URL:https://arxiv.org/abs/2408.07773 . arXiv:2408.07773

    Pith/arXiv arXiv 2024

  19. [19]

    Y. An, H. Zhang, Y. Sheng, J. Wang, X. Chen, Main: Multimodal attention-based fusion networks for diagnosis prediction, in: 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2021, pp. 809–816. doi:10.1109/BIBM52615.2021.9669634

    arXiv 2021

  20. [20]

    Kline, H

    A. Kline, H. Wang, Y. Li, S. Dennis, M. Hutch, Z. Xu, F. Wang, F. Cheng, Y. Luo, Multimodal machine learning in precision health: A scoping review, npj Digital Medicine 5 (2022) 171

    2022

  21. [21]

    van den Oord, Y

    A. van den Oord, Y. Li, O. Vinyals, Representation learning with contrastive predictive coding, ArXiv abs/1807.03748 (2018)

    Pith/arXiv arXiv 2018

  22. [22]

    K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum contrast for unsupervised visual representation learning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738

    2020

  23. [23]

    T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, in: Proceedings of the 37th International Conference on Machine Learning, ICML’20, JMLR.org, 2020

    2020

  24. [24]

    Zheng, F

    M. Zheng, F. Wang, S. You, C. Qian, C. Zhang, X. Wang, C. Xu, Weakly supervised contrastive learning, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 10042–10051

    2021

  25. [25]

    S. Wu, H. Li, X. Liu, D. Tian, H. Su, Class-aware patch based contrastive learning for medical image segmentation, in: 2024 In- ternational Joint Conference on Neural Networks (IJCNN), 2024, pp. 1–9. doi:10.1109/IJCNN60899.2024.10650546

    arXiv 2024

  26. [26]

    Dobson,Medgpt:Medicalconceptpredictionfromclinicalnarratives, ArXiv abs/2107.03134 (2021)

    Z.Kraljevic,A.Shek,D.M.Bean,R.Bendayan,J.T.H.Teo,R.J.B. Dobson,Medgpt:Medicalconceptpredictionfromclinicalnarratives, ArXiv abs/2107.03134 (2021)

    Pith/arXiv arXiv 2021

  27. [27]

    Cascella, J

    M. Cascella, J. Montomoli, V. Bellini, E. Bignami, Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios, Journal of medical systems 47 (2023) 33

    2023

  28. [28]

    McDermott, Publicly available clinical BERT embeddings, in: A

    E.Alsentzer,J.Murphy,W.Boag,W.-H.Weng,D.Jindi,T.Naumann, M. McDermott, Publicly available clinical BERT embeddings, in: A. Rumshisky, K. Roberts, S. Bethard, T. Naumann (Eds.), Proceed- ings of the 2nd Clinical Natural Language Processing Workshop, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 72–78. URL:https://aclanth...

    doi:10.18653/v1/w19-1909 2019

  29. [29]

    C. Pang, X. Jiang, K. S. Kalluri, M. Spotnitz, R. Chen, A. Perotte, K. Natarajan, Cehr-bert: Incorporating temporal information from structuredehrdatatoimprovepredictiontasks, in:MachineLearning for Health, PMLR, 2021, pp. 239–260

    2021

  30. [30]

    J. Dong, H. Wu, H. Zhang, L. Zhang, J. Wang, M. Long, Simmtm: a simple pre-training framework for masked time-series modeling, in: Proceedings of the 37th International Conference on Neural Infor- mation Processing Systems, NIPS ’23, Curran Associates Inc., Red Hook, NY, USA, 2023

    2023

  31. [31]

    Y. Nie, N. H. Nguyen, P. Sinthong, J. Kalagnanam, A time series is worth 64 words: Long-term forecasting with transformers, 2023. URL: https://arxiv.org/abs/2211.14730. arXiv:2211.14730

    Pith/arXiv arXiv 2023

  32. [32]

    Ekambaram, A

    V. Ekambaram, A. Jati, N. Nguyen, P. Sinthong, J. Kalagnanam, Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting, in: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, ACM, 2023, p. 459–469. URL: http://dx.doi.org/10.1145/3580305.3599533 . doi:10.1145/3580305.3599533

    arXiv 2023

  33. [33]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,Ł.Kaiser,I.Polosukhin, Attentionisallyouneed, Advances in neural information processing systems 30 (2017)

    2017

  34. [34]

    Jeong, N

    H. Jeong, N. Oufattole, M. Mcdermott, A. Balagopalan, B. Jangeesingh, M. Ghassemi, C. Stultz, Event-based contrastive learning for medical time series, 2024. URL: https://arxiv.org/abs/2312.10308. arXiv:2312.10308

    Pith/arXiv arXiv 2024

  35. [35]

    Kiyasseh, T

    D. Kiyasseh, T. Zhu, D. Clifton, Crocs: Clustering and retrieval of cardiac signals based on patient disease class, sex, and age, in: M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems, vol- ume 34, Curran Associates, Inc., 2021, pp. 15557–15569. URL: https://proceedings.neurips.cc/pap...

    2021

  36. [36]

    Le Gall, P

    J.-R. Le Gall, P. Loirat, A. Alperovitch, P. Glaser, C. Granthil, D. Mathieu, P. Mercier, R. Thomas, D. Villers, A simplified acute physiology score for icu patients, Critical care medicine 12 (1984) 975–977

    1984

  37. [37]

    A. E. Jones, S. Trzeciak, J. A. Kline, The sequential organ failure assessmentscoreforpredictingoutcomeinpatientswithseveresepsis and evidence of hypoperfusion at the time of emergency department presentation*, Critical Care Medicine 37 (2009)

    2009

  38. [38]

    Steinberg, M

    E. Steinberg, M. Wornow, S. Bedi, J. A. Fries, M. McDermott, N. H. Shah, meds_reader: A fast and efficient ehr processing library, arXiv preprint arXiv:2409.09095 (2024)

    Pith/arXiv arXiv 2024

  39. [39]

    D. F. Stein, M. J. Carter, J. Booth, M. J. Peters, S. Ray, N. J. Sebire, P. Barnaghi, M. Cortina-Borja, Prediction cardiovascular deterioration in a paediatric intensive care unit (picews): a machine learning modelling study of routinely collected health-care data, Eclinicalmedicine 85 (2025)

    2025

  40. [40]

    F.Wang,F.Wu,Y.Tang,L.Yu,CTPD:Cross-modaltemporalpattern discoveryforenhancedmultimodalelectronichealthrecordsanalysis, in:W.Che,J.Nabende,E.Shutova,M.T.Pilehvar(Eds.),Findingsof the Association for Computational Linguistics: ACL 2025, Associa- tionforComputationalLinguistics,Vienna,Austria,2025,pp.6783–

    2025

  41. [41]

    Z. Wu, A. Dadu, M. Nalls, F. Faghri, J. Sun, Instruction tuning large language models to understand electronic health records, in: A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tom- czak, C. Zhang (Eds.), Advances in Neural Information Processing Systems,volume37,CurranAssociates,Inc.,2024,pp.54772–54786. URL: https://proceedings.neurips.cc/...

    work page doi:10.52202/079017-1737 2024

  42. [42]

    Mahajan, M

    Y. Mahajan, M. Freestone, N. Bansal, S. Aakur, S. K. K. Santu, Revisitingwordembeddingsinthellmera,2025.URL: https://arxi v.org/abs/2402.11094. arXiv:2402.11094

    Pith/arXiv arXiv 2025

  43. [43]

    URL:https://arxiv.org/abs/2506.10896

    T.Sounack,J.Davis,B.Durieux,A.Chaffin,T.J.Pollard,E.Lehman, A.E.W.Johnson,M.McDermott,T.Naumann,C.Lindvall,Bioclin- ical modernbert: A state-of-the-art long-context encoder for biomed- ical and clinical nlp, 2025. URL:https://arxiv.org/abs/2506.10896. arXiv:2506.10896

    Pith/arXiv arXiv 2025

  44. [44]

    Warner, A

    B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S.Taghadouini,A.Gallagher,R.Biswas,F.Ladhak,T.Aarsen,etal., Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, in: Proceedings of the 63rd Annual Meeting of the Association for ComputationalLinguistics(Volume1:Long...

    2025

  45. [45]

    D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014)

    Pith/arXiv arXiv 2014

  46. [46]

    A. E. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, S. Hao, B. Moody, B. Gow, et al., Mimic-iv, a freely accessible electronic health record dataset, Scientific data 10 (2023) 1

    2023

  47. [47]

    Silva, G

    I. Silva, G. Moody, D. J. Scott, L. A. Celi, R. G. Mark, Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012, in: 2012 computing in cardiology, IEEE, 2012, pp. 245–248

    2012

  48. [48]

    Z. C. Lipton, D. C. Kale, C. Elkan, R. Wetzel, Learning to diagnose withlstmrecurrentneuralnetworks,arXivpreprintarXiv:1511.03677 (2015)

    Pith/arXiv arXiv 2015

  49. [49]

    T. Chen, C. Guestrin, Xgboost: A scalable tree boosting system, in: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794

    2016

  50. [50]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T.Killeen,Z.Lin,N.Gimelshein,L.Antiga,A.Desmaison,A.Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style,high-performancedeeplearninglibrary, CoRRabs/1912.01703 (2019)

    Pith/arXiv arXiv 1912

  51. [51]

    E.Martinsson,WTTE-RNN:WeibullTimeToEventRecurrentNeu- ral Network, Master’s thesis, Chalmers University Of Technology, 2016

    2016

  52. [52]

    Z.Che,S.Purushotham,K.Cho,D.Sontag,Y.Liu, Recurrentneural networks for multivariate time series with missing values, Scientific reports 8 (2018) 6085

    2018

  53. [53]

    H.Song,D.Rajan,J.J.Thiagarajan,A.Spanias, Attendanddiagnose: clinical time series analysis using attention models, in: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Con- ference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’1...

    2018

  54. [54]

    A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, R. G. Mark, Mimic-iii, a freely accessible critical care database, Scientific Data 3 (2016) 160035

    2016

  55. [55]

    Lambden, P

    S. Lambden, P. F. Laterre, M. M. Levy, B. Francois, The sofa score—development,utilityandchallengesofaccurateassessmentin clinical trials, Critical Care 23 (2019) 374

    2019

  56. [56]

    Hempel, S

    L. Hempel, S. Sadeghi, T. Kirsten, Prediction of intensive care unit length of stay in the mimic-iv dataset, Applied Sciences 13 (2023)

    2023

  57. [57]

    Y. Li, R. M. Wehbe, F. S. Ahmad, H. Wang, Y. Luo, Clinical- longformer and clinical-bigbird: Transformers for long clinical se- quences, CoRR abs/2201.11838 (2022)

    Pith/arXiv arXiv 2022

  58. [58]

    Gururangan, A

    S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, N. A. Smith, Don’t stop pretraining: Adapt language models to domains and tasks, in: Proceedings of ACL, 2020

    2020

  59. [59]

    Beltagy, M

    I. Beltagy, M. E. Peters, A. Cohan, Longformer: The long-document transformer, CoRR abs/2004.05150 (2020)

    Pith/arXiv arXiv 2004

  60. [60]

    Zettlemoyer, V

    Y.Liu,M.Ott,N.Goyal,J.Du,M.Joshi,D.Chen,O.Levy,M.Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019)

    Pith/arXiv arXiv 1907

  61. [61]

    Davidson-Pilon, lifelines: survival analysis in python, Journal of Open Source Software 4 (2019) 1317

    C. Davidson-Pilon, lifelines: survival analysis in python, Journal of Open Source Software 4 (2019) 1317

    2019

  62. [62]

    : Preprint submitted to Elsevier Page 14 of 16 A

    A.E.W.Johnson,D.J.Stone,L.A.Celi,T.J.Pollard,Themimiccode repository: enabling reproducibility in critical care research, Journal of the American Medical Informatics Association 25 (2018) 32–39. : Preprint submitted to Elsevier Page 14 of 16 A. Appendix A.1. Curating the datasets Here we describe the process for curating the MIMIC- IV dataset, correspondi...

    2018

  63. [64]

    Specifically, we computed SOFA score hourly using the worst measurements over the past 24 hour window

    used in-unit mortality as a proxy label, we instead use changes in SOFA score, which more directly reflect acute organ dysfunction. Specifically, we computed SOFA score hourly using the worst measurements over the past 24 hour window. Missing components are treated as 0 (no related risk) if no previous observations and front filled (no change in risk) if ...

  64. [6799]

    doi:10.18653/v1/2025.findings-acl.352

    URL: https://aclanthology.org/2025.findings- acl.352/ . doi:10.18653/v1/2025.findings-acl.352. : Preprint submitted to Elsevier Page 13 of 16

    work page doi:10.18653/v1/2025.findings-acl.352 2025

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.