Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

The paper argues that a single GPT-style EHR foundation model, built on artificial time tokens and two auxiliary time objectives, can simultaneously generate patient embeddings, forecast future events without retraining, and produce realist

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 10:47 UTC pith:JCMNO5TL

load-bearing objection Solid multi-task EHR model with real utility, but the zero-shot evaluation has a censoring flaw that needs fixing before I'd trust Table 4. the 3 major comments →

arxiv 2509.03643 v2 pith:JCMNO5TL submitted 2025-09-03 cs.LG cs.AI

CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records

Chao Pang , Jiheum Park , Xinzhuo Jiang , Nishanth Parameshwar Pavinkurve , Krishna S. Kalluri , Shalmali Joshi , No\'emie Elhadad , Karthik Natarajan This is my paper
classification cs.LG cs.AI
keywords EHR foundation modelzero-shot predictionsynthetic EHR datatime tokenspatient representationOMOP common data modeltransformer decodertemporal modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that one pretrained model, CEHR-XGPT, can do three jobs that have usually been separate: produce patient embeddings, forecast future events without task-specific training, and generate realistic synthetic patient timelines. It argues this unification is possible because a GPT-style decoder, prompted with token sequences whose inter-event gaps are marked by artificial time tokens, learns the full temporal structure of a patient's record. If true, a single model could replace task-specific pipelines in low-resource and zero-label settings. The paper supports the claim with evaluations on multiple prediction tasks, an external benchmark, synthetic-data utility tests, and privacy attacks, and reports that a variant without its time-token objectives performs worse on zero-shot and synthetic-data tasks.

Core claim

On the paper's own terms, CEHR-XGPT is the first EHR foundation model to unify feature representation, zero-shot prediction, and synthetic data generation in one architecture. Patient histories are tokenized with explicit artificial time tokens marking gaps between visits, and two auxiliary objectives, Time Decomposition and Time-to-Event, shape those time-token embeddings. The paper reports strong zero-shot AUROC on a 30-day readmission task, a 1-year CABG surgery task, and a lifetime type-2-diabetes-to-heart-failure task; synthetic timelines that reproduce concept prevalence and treatment-pathway prevalence closely enough for real-world model development and augmentation; and external-data

What carries the argument

The central mechanism is the artificial time token (ATT): a vocabulary symbol inserted between visits that names the gap in days, such as D10 for a ten-day gap, with a special [LT] token for gaps beyond 1080 days. Two auxiliary objectives operate directly on ATT embeddings: Time Decomposition splits each token into year, month, and day classification heads, and Time-to-Event fits a Gamma distribution to the actual interval length. The GPT-style decoder drops positional embeddings entirely and relies on these time tokens to carry order and spacing, so the same representation supports embedding extraction, prompt-based trajectory generation, and synthetic sequence sampling.

Load-bearing premise

The zero-shot evaluation assumes that a simulated trajectory ending early with the end-of-sequence marker tells us nothing about the patient, so it discards those runs and recomputes outcome rates from the rest; if the model ends early because the patient has no more recorded events, the predicted outcome rates will be biased upward.

What would settle it

Recompute zero-shot outcome probabilities on the same test cases two ways: once as reported, discarding trajectories that hit the end-of-sequence token inside the prediction window, and once counting an end-of-sequence-terminated trajectory as no outcome within the window. If the alternative estimates shift materially, or if early-termination rates correlate with baseline risk, the zero-shot AUROC claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • New prediction tasks can be evaluated with no task-specific labels or fine-tuning: the model is prompted with outcome concepts and a prediction window, and outcome probability is estimated from generated trajectories.
  • Synthetic timelines can be converted back into a standard common data model, so researchers can define cohorts with SQL after generation.
  • Models trained on synthetic data alone were comparable to real-data training for two of four evaluated tasks, and mixing synthetic data into real training improved all four tasks.
  • Long-range treatment-pathway prevalence was reproduced more closely with the time-objective model than without, showing temporal coherence in long synthetic timelines.
  • On an external dataset, vocabulary expansion plus fine-tuning let the model match the best reported patient-outcome models and exceed them on new-diagnosis tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether ATT embeddings can be read directly as interval estimates; a simple linear probe on those embeddings could turn the model into a continuous time-interval regressor.
  • If the zero-shot censoring concern is settled favorably, the same sampling-based evaluation could be applied to any patient subpopulation by changing the prompt, making the model a queryable risk engine without new training.
  • The evidence that a strong full-trajectory generator also transfers across hospital systems suggests generation quality could serve as a cheap pre-screen for foundation-model quality on new data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents CEHR-XGPT, a GPT-2-style autoregressive model for structured EHR data, and claims it is the first multi-task foundation model to unify three capabilities: patient feature representation, zero-shot future-event prediction, and synthetic data generation. The model represents each patient as a chronological sequence of visits and medical concepts, with artificial time tokens (ATTs) encoding gaps between visits, plus two auxiliary objectives: time decomposition (TD) and time-to-event (TTE) prediction. Evaluation is performed on six in-distribution clinical prediction tasks, three zero-shot prediction tasks, four synthetic-data utility tasks, and the external ehrshot benchmark, with an ablation variant (GPT) that removes the TD and TTE objectives. The authors report strong fine-tuning results, competitive representation performance that is generally below MOTOR under linear probing, zero-shot AUROCs between roughly 67% and 91%, and synthetic data that improves downstream models when used as augmentation.

Significance. If the central claims hold, the paper makes a useful contribution to EHR foundation models: a single pretrained model that can produce embeddings, forecast outcomes without task-specific training, and generate synthetic timelines, while transferring to an external OMOP database through vocabulary expansion. Strengths of the study include the breadth of evaluation (six in-distribution tasks, external ehrshot tasks, treatment-pathway replication, four privacy attack types), a clearly described ablation isolating the proposed objectives, and a public code repository with replication instructions. The fine-tuning results and ehrshot transfer are credible and add independent grounding. The main weakness is the zero-shot evaluation's treatment of [END]-terminated simulations, which can bias the reported AUROC/AUPRC and directly affects one of the three claimed capabilities; the representation claim is also weaker than the abstract suggests because MOTOR wins all linear-probing comparisons.

major comments (3)
  1. [3.7.2] The zero-shot estimator discards and replaces any simulation that hits the [END] token inside the prediction window, then computes outcome probability as the fraction of kept runs in which the outcome occurs. This is valid only if [END] is uninformative administrative censoring. In this model, [END] is a generated token that likely represents the model's prediction that the patient's recorded timeline has no further events; if so, discarding those runs conditions on continued generation, which is correlated with outcome risk, and can inflate the AUROC/AUPRC values in Table 4. The paper does not report the frequency of [END] terminations nor a sensitivity analysis (e.g., counting [END] as 'no outcome' or using a survival-style estimator). Because zero-shot prediction is one of the three central capabilities, Table 4 cannot currently be interpreted as strong evidence for that capability.
  2. [4.1, Table 3] MOTOR achieves the highest AUROC and AUPRC for all six tasks in the linear-probing setting, and CEHR-XGPT-L is not consistently better than GPT-L. The abstract's claim of 'strong performance across all three tasks' is therefore not uniformly supported by the feature-representation leg: only fine-tuned CEHR-XGPT-T is competitive or superior. The authors acknowledge this in Section 5, but the central claim should be reworded to distinguish representation quality (competitive, not SOTA) from fine-tuning performance, and the paper should either present additional representation evidence (e.g., clustering, non-linear probes, or propensity-score tasks) or temper the claim.
  3. [3.6 and 3.7.3] Two of the six evaluation cohorts—Discharge home death and T2DM HF—are excluded from the synthetic-data evaluation because the synthetic dataset omits death and measurement domains. Table 5 therefore covers only four cohorts, all of which are procedure- or hospitalization-focused; long-term mortality and diabetes-related trajectory fidelity are untested. This narrows the synthetic-data claim. The authors should either extend the synthetic generation to include death and measurement domains or explicitly limit the synthetic-data claim to the four evaluated cohorts and discuss the missing-domain gap as a limitation of the current model version.
minor comments (6)
  1. [3.3] Typo: 'limiation' should be 'limitation'. Also, the notation [e_year_ij, e_month_ij, e_day_ij] = split(e_att_ij) is not formally defined; please clarify how an embedding is split into sub-embeddings.
  2. [3.7.4] The text says 'similar to the procedure described in Section 4.1'; the evaluation procedure is described in Section 3.7.1, not Section 4.1.
  3. [Table 3 / Table 2] The cohort named 'Discharge home death' in Table 2 appears as 'Discharge mortality' in Table 3 and Supplementary Figure 6. Please unify the naming for consistency.
  4. [Table 6] The 'Context' column is not identified in the caption. It should be labeled as 'Context Clues (Wornow et al.)' and state whether AUPRC is unavailable for that model, since only AUROC is reported.
  5. [3.5] The hand-constructed example with matrices W1, W2 and bias vectors is used to claim that time tokens are superior to time embeddings. The demonstration is for a single input [0,6,1], and the statement that the weights 'should work with all the other inputs' is asserted rather than proved. Please either provide a proof or clearly label this as an illustrative construction. Also, the sentence 'We define matrices, W, and bias vectors' is ungrammatical.
  6. [3.7.2 / 9.4] The zero-shot generation uses 50 simulated trajectories but the sampling hyperparameters (temperature, top_p, top_k) are not reported for this inference setting. Please include them for reproducibility.

Circularity Check

0 steps flagged

No significant circularity: central claims are externally benchmarked (ehrshot, held-out real test sets, MOTOR comparison); self-citations are lineage/support, not load-bearing reductions.

full rationale

CEHR-XGPT's three capability claims are each tested against quantities not derived from the model's own fitted values: (i) feature representation is compared with MOTOR and a GPT-2 ablation on held-out CUIMC cohorts and on the external ehrshot benchmark; (ii) zero-shot prediction is scored by AUROC/AUPRC against ground-truth outcomes from reserved patients, with predicted probabilities coming from 50 generative rollouts rather than from any parameter fitted to the test labels; (iii) synthetic data utility is measured by training logistic regressions on generated OMOP rows and testing on real data, while privacy is scored with external attack thresholds. The TD/TTE objectives are self-supervised auxiliary losses whose targets (year/month/day components and time intervals) are derived from the ATT tokens' own definitions; they shape the embedding space but are not used as the outcome in any downstream evaluation, so no prediction reduces to a fitted input. The only self-citations (CEHR-GPT [35], CEHR-BERT [11]) supply the baseline architecture, training configuration, and a supporting ablation for time tokens; the paper adds its own toy construction and simulation, and the central comparisons to MOTOR and ehrshot do not depend on those citations being accepted. The zero-shot protocol's discarding of [END]-terminated rollouts is a potential validity threat (informative censoring), but it is a bias in estimating outcome probabilities, not a circular reduction of the prediction to its input. Therefore no circular step is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central design depends on hand-picked temporal representation choices (1080-day cutoff, calendar decomposition, Gamma family) and on the unstated mixture-of-experts generation grid. The model's claims rest on domain assumptions about OMOP completeness and the validity of the zero-shot censoring rule. No new physical or theoretical entities are introduced; ATT tokens are a representational device.

free parameters (4)
  • 1080-day cutoff for [LT] token = 1080 days
    Hand-picked threshold; all inter-visit gaps above it collapse to a single token, losing temporal resolution for long gaps (Section 3.2).
  • Time decomposition granularity = year/month/day
    Hand-chosen calendar decomposition used in the TD loss (Section 3.3); imposes an inductive bias on ATT embeddings.
  • Gamma distribution family for TTE = Gamma(alpha, beta) with learned alpha, beta
    Assumed parametric form for time-to-event loss (Section 3.3); the feed-forward parameters are learned but the family is fixed by the authors.
  • Mixture-of-experts generation settings = unreported values
    Synthetic data pooled from different sample sizes, checkpoints, temperatures, top_p, top_k, and repetition penalties (Section 3.7.3); the specific grid and selection rule are not provided, so the choice is effectively hand-tuned.
axioms (5)
  • domain assumption Transformers without positional encodings learn positional information implicitly (Haviv et al., 2022)
    Basis for removing positional embeddings from GPT-2 (Section 3.3); if false, the model loses ordering information.
  • domain assumption OMOP concepts for conditions, procedures, medications, visit types, and demographics are sufficient to represent patient trajectories for all three claimed capabilities
    Training data excludes measurements, observations, and death (Section 3.1); the paper itself excludes cohorts requiring those domains from synthetic evaluation (Section 3.6).
  • domain assumption Next-token prediction with TD/TTE auxiliary losses is an effective pretraining objective for EHR sequence modeling
    Central training design (Sections 3.3 and 3.4); the paper provides only an internal ablation, not independent theoretical support.
  • ad hoc to paper Discarding [END]-terminated simulations in zero-shot evaluation is unbiased (censoring is uninformative)
    Zero-shot probability estimation replaces censored runs (Section 3.7.2); early [END] may indicate no future events, which would bias estimates.
  • ad hoc to paper The hand-constructed example and simulation in Sections 3.5 and 9.1 demonstrate general superiority of time tokens over time embeddings
    The toy uses a modified ReLU and a purpose-built rule set; it is illustrative, not a proof of general superiority.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 20987 in / 18723 out tokens · 177483 ms · 2026-08-05T10:47:18.111093+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records." pith.science (2026). https://pith.science/paper/JCMNO5TL

@misc{pith2026250903643,
  author       = {Pith},
  title        = {Pith review of: CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JCMNO5TL}},
  note         = {Machine review of arXiv:2509.03643}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Electronic Health Records (EHRs) provide a rich, longitudinal view of patient health and hold significant potential for advancing clinical decision support, risk prediction, and data-driven healthcare research. However, most artificial intelligence (AI) models for EHRs are designed for narrow, single-purpose tasks, limiting their generalizability and utility in real-world settings. Here, we present CEHR-XGPT, a general-purpose foundation model for EHR data that unifies three essential capabilities - feature representation, zero-shot prediction, and synthetic data generation - within a single architecture. To support temporal reasoning over clinical sequences, CEHR-XGPT incorporates a novel time-token-based learning framework that explicitly encodes patients' dynamic timelines into the model structure. CEHR-XGPT demonstrates strong performance across all three tasks and generalizes effectively to external datasets through vocabulary expansion and fine-tuning. Its versatility enables rapid model development, cohort discovery, and patient outcome forecasting without the need for task-specific retraining.

Figures

Figures reproduced from arXiv: 2509.03643 by Chao Pang, Jiheum Park, Karthik Natarajan, Krishna S. Kalluri, Nishanth Parameshwar Pavinkurve, No\'emie Elhadad, Shalmali Joshi, Xinzhuo Jiang.

Figure 2
Figure 2. Figure 2: The distribution of time tokens between visits is not uniform but instead biased toward specific intervals, such as 7 and 14 days. The TD learning objective introduces a structural con￾straint on the ATTs to address the skewness in the distribu￾tion of time intervals, as depicted in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: CEHR-XGPT is built on the GPT-2 architecture with a specialized patient representation designed to capture the full longitudinal trajectory of each patient. Two additional learning objectives were applied to artificial time token embeddings to improve the modeling of temporal information, namely, time decomposition and time-to-event prediction. CEHR-XGPT supports three key capabilities: patient-level featu… view at source ↗
Figure 3
Figure 3. Figure 3: Treatment pathway study cohort criteria: Patients were required to have at least 365 days of observation prior to their first drug exposure for the disease of interest. Following this initial exposure, patients had to maintain continuous treatment, defined as having at least one drug exposure every 120 days over a 3-year period. and used it to train a linear classifier. The key distinction between the two … view at source ↗
Figure 6
Figure 6. Figure 6: We reported ROC-AUC and PR-AUC across all [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 4
Figure 4. Figure 4: Receiver Operating Characteristic (ROC) curves comparing the performance across six clinical prediction tasks. Models are distinguished by unique color and line style combinations. Performance metrics show Area Under the Curve (AUC) values with 95% confidence intervals calculated using bootstrap sampling. Models are ranked in descending order of AUC performance within each sub￾plot, with annotations positi… view at source ↗
Figure 5
Figure 5. Figure 5: , the model employing time tokens exhibited rapid convergence, achieving perfect accuracy (1.0). In contrast, the accuracy of the model using the summation strategy improved gradually over time and failed to converge even after 20,000 steps. While Modelsum could potentially resolve the constructed function in the simulation by either extending its training duration or increasing its model size, it is clear… view at source ↗
Figure 6
Figure 6. Figure 6: presents the Precision-Recall curves for all evaluated models across six clinical tasks, as defined in [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The concept prevalence comparison between the real OMOP and the synthetic datasets stratified by domains (condition, drug, procedure, and visit) and populations (full, female, and hospitalization cohorts. Each dot represents an OMOP concept. As shown in [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models

    cs.CL 2026-06 unverdicted novelty 6.0

    PORTER is a language-grounded EHR foundation model that uses text descriptions for events and a numeric pathway, matching fixed-vocabulary performance on 74 tasks while recovering 97.1% AUROC on unseen vocabularies an...

  2. Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction

    cs.LG 2026-03 unverdicted novelty 6.0

    RAVEN pretrains on over one million EHR sequences via recurrence-aware next-visit event prediction, enabling zero-shot disease incidence forecasting that rivals fine-tuned models and generalizes across cohorts.

  3. How Should Transformers Encode Numeric Values in Electronic Health Records?

    cs.LG 2026-07 unverdicted novelty 5.0

    Hybrid token-based binning for numeric values in EHR transformers is more robust than explicit interaction modeling, with optimal bin count following an empirically derived power-law in dataset size.

Reference graph

Works this paper leans on

50 extracted references · 41 canonical work pages · cited by 3 Pith papers · 4 internal anchors

  1. [1]

    & Albers, D

    Hripcsak, G. & Albers, D. J. Next-generation phe- notyping of electronic health records.Journal of the American Medical Informatics Association : JAMIA 20, 117–121 (2013). URLhttps://pubmed.ncbi.nlm. nih.gov/22955496/

    work page arXiv 2013

  2. [2]

    Hripcsak, G.et al.Observational health data sciences and informatics (ohdsi): Opportunities for observational researchers. vol. 216, 574–578 (IOS Press, 2015)

    work page 2015

  3. [3]

    URLhttps://www

    Hripcsak, G.et al.Characterizing treatment pathways at scale using the ohdsi network.Proceedings of the National Academy of Sciences of the United States of 12 America113, 7329–7336 (2016). URLhttps://www. pnas.org/doi/abs/10.1073/pnas.1510502113

    work page doi:10.1073/pnas.1510502113 2016

  4. [4]

    URLhttps://arxiv.org/ pdf/2108.07258

    Bommasani, R.et al.On the opportunities and risks of foundation models (2021). URLhttps://arxiv.org/ pdf/2108.07258

    Pith/arXiv arXiv 2021

  5. [5]

    URLhttps://arxiv.org/abs/2412.16178v1

    Wornow, M.et al.Context clues: Evaluating long con- text models for clinical prediction tasks on ehrs (2024). URLhttps://arxiv.org/abs/2412.16178v1

    Pith/arXiv arXiv 2024

  6. [6]

    & Sun, J

    Theodorou, B., Xiao, C. & Sun, J. Synthesize high-dimensional longitudinal electronic health records via hierarchical autoregressive language model.Nature Communications 2023 14:114, 1–13 (2023). URLhttps://www.nature.com/articles/ s41467-023-41093-0

    work page 2023

  7. [7]

    Steinberg, E.et al.Language models are an effective representation learning technique for electronic health record data (2020)

    work page 2020

  8. [8]

    A., Xu, Y., Shah, N

    Steinberg, E., Fries, J. A., Xu, Y., Shah, N. & Health- care, S. Motor: A time-to-event foundation model for structured medical records (2023). URLhttps: //arxiv.org/abs/2301.03150v4

    Pith/arXiv arXiv 2023

  9. [9]

    URLhttps://www.nature.com/articles/ s41746-023-00879-8

    Wornow, M.et al.The shaky foundations of large language models and foundation models for electronic health records.npj Digital Medicine 2023 6:16, 1– 10 (2023). URLhttps://www.nature.com/articles/ s41746-023-00879-8

    work page 2023

  10. [10]

    & Zhi, D

    Rasmy, L., Xiang, Y., Xie, Z., Tao, C. & Zhi, D. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for dis- ease prediction.npj Digital Medicine 2021 4:14, 1– 13 (2021). URLhttps://www.nature.com/articles/ s41746-021-00455-y

    work page 2021

  11. [11]

    In Roy, S.et al.(eds.)Proceedings of Machine Learning for Health, vol

    Pang, C.et al.Cehr-bert: Incorporating temporal in- formation from structured ehr data to improve pre- diction tasks. In Roy, S.et al.(eds.)Proceedings of Machine Learning for Health, vol. 158 ofProceed- ings of Machine Learning Research, 239–260 (PMLR, 2021). URLhttps://proceedings.mlr.press/v158/ pang21a.html

    work page 2021

  12. [12]

    URLhttps://www.nature.com/articles/ s41746-024-01235-0

    Renc, P.et al.Zero shot health trajectory predic- tion using transformer.npj Digital Medicine7, 1– 10 (2024). URLhttps://www.nature.com/articles/ s41746-024-01235-0

    work page 2024

  13. [13]

    & Beheshti, R

    Poulain, R., Gupta, M. & Beheshti, R. Few- shot learning with semi-supervised transform- ers for electronic health records.Proceedings of machine learning research182, 853 (2022). URLhttps://pmc.ncbi.nlm.nih.gov/articles/ PMC10399128/http://www.pubmedcentral.nih.gov/ articlerender.fcgi?artid=PMC10399128

    work page 2022

  14. [14]

    URLhttps://arxiv.org/ abs/2405.14567v3

    Fallahpour, A.et al.Ehrmamba: Towards general- izable and scalable foundation models for electronic health records.Proceedings of Machine Learning Re- search259, 1–14 (2024). URLhttps://arxiv.org/ abs/2405.14567v3

    Pith/arXiv arXiv 2024

  15. [15]

    McDermott, M. B. A., Nestor, B., Argaw, P. & Ko- hane, I. Event stream gpt: A data pre-processing and modeling library for generative, pre-trained transform- ers over continuous-time sequences of complex events (2023). URLhttps://arxiv.org/abs/2306.11547v2

    Pith/arXiv arXiv 2023

  16. [16]

    URL https://pubmed.ncbi.nlm.nih.gov/35737602/

    Rao, S.et al.Targeted-behrt: Deep learning for ob- servational causal inference on longitudinal electronic health records.IEEE Transactions on Neural Networks and Learning Systems35, 5027–5038 (2024). URL https://pubmed.ncbi.nlm.nih.gov/35737602/

    work page arXiv 2024

  17. [17]

    Yang, Z., Mitra, A., Liu, W., Berlowitz, D. & Yu, H. Transformehr: transformer-based encoder-decoder gen- erative model to enhance prediction of disease outcomes using electronic health records.Nature Communica- tions14, 1–10 (2023). URLhttps://www.nature.com/ articles/s41467-023-43715-z

    work page 2023

  18. [18]

    URLhttps://www.nature.com/articles/ s41598-020-62922-y

    Li, Y.et al.Behrt: Transformer for electronic health records.Scientific Reports 2020 10:110, 1– 12 (2020). URLhttps://www.nature.com/articles/ s41598-020-62922-y

    work page 2020

  19. [19]

    IEEE journal of biomedical and health informatics27, 1106 (2023)

    Li, Y.et al.Hi-behrt: Hierarchical transformer-based model for accurate prediction of clinical events us- ing multimodal longitudinal electronic health records. IEEE journal of biomedical and health informatics27, 1106 (2023). URLhttps://pmc.ncbi.nlm.nih.gov/ articles/PMC7615082/

    work page 2023

  20. [20]

    URL https://arxiv.org/pdf/2404.15201

    Odgaard, M.et al.Core-behrt: A carefully opti- mized and rigorously evaluated behrt.Proceedings of Machine Learning Research252, 1–33 (2024). URL https://arxiv.org/pdf/2404.15201

    Pith/arXiv arXiv 2024

  21. [21]

    GenHPF: General Healthcare Predictive Framework with Multi-task Multi-source Learning

    Hur, K.et al.Genhpf: General healthcare predictive framework with multi-task multi- source learning.IEEE Journal of Biomedi- cal and Health Informatics28, 502–513 (2023). URLhttp://arxiv.org/abs/2207.09858http: //dx.doi.org/10.1109/JBHI.2023.3327951

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  22. [22]

    URLhttps://arxiv.org/abs/2212

    Kraljevic, Z.et al.Foresight – generative pretrained transformer (gpt) for modelling of patient timelines us- ing ehrs (2022). URLhttps://arxiv.org/abs/2212. 08072v2

    work page 2022

  23. [23]

    In Doshi- Velez, F.et al.(eds.)Proceedings of the 2nd Machine Learning for Healthcare Conference, vol

    Choi, E.et al.Generating multi-label discrete patient records using generative adversarial networks. In Doshi- Velez, F.et al.(eds.)Proceedings of the 2nd Machine Learning for Healthcare Conference, vol. 68 ofProceed- ings of Machine Learning Research, 286–305 (PMLR, 2017). URLhttps://proceedings.mlr.press/v68/ choi17a.html. 13

    work page 2017

  24. [24]

    Li, Y., Bengio, S. & Du, N. Time-dependent rep- resentation for neural event sequence prediction.6th International Conference on Learning Representations, ICLR 2018 - Workshop Track Proceedings(2017). URL https://arxiv.org/abs/1708.00065v4

    Pith/arXiv arXiv 2018

  25. [25]

    CONAN: Complementary Pattern Augmentation for Rare Disease Detection

    Cui, L.et al.Conan: Complementary pattern augmen- tation for rare disease detection.AAAI 2020 - 34th AAAI Conference on Artificial Intelligence614–621 (2019). URLhttps://arxiv.org/abs/1911.13232v1

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  26. [26]

    K., Lin, C

    Baowaly, M. K., Lin, C. C., Liu, C. L. & Chen, K. T. Synthesizing electronic health records using improved generative adversarial networks.Journal of the Amer- ican Medical Informatics Association : JAMIA26, 228 (2019). URL/pmc/articles/PMC7647178//pmc/ articles/PMC7647178/?report=abstracthttps:// www.ncbi.nlm.nih.gov/pmc/articles/PMC7647178/

    work page 2019

  27. [27]

    Generating Electronic Health Records with Multiple Data Types and Constraints

    Yan, C., Zhang, Z., Nyemba, S. & Malin, B. A. Gener- ating electronic health records with multiple data types and constraints.AMIA ... Annual Symposium proceed- ings. AMIA Symposium2020, 1335–1344 (2020). URL https://arxiv.org/abs/2003.07904v2

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  28. [28]

    URLhttps://pubmed.ncbi

    Lee, D.et al.Generating sequential electronic health records using dual adversarial autoencoder.Journal of the American Medical Informatics Association : JAMIA 27, 1411–1419 (2020). URLhttps://pubmed.ncbi. nlm.nih.gov/32989459/

    work page arXiv 2020

  29. [29]

    CorGAN: Correlation-Capturing Convolutional Generative Adversarial Networks for Generating Synthetic Healthcare Records

    Torfi, A. & Fox, E. A. Corgan: Correlation-capturing convolutional generative adversarial networks for gen- erating synthetic healthcare records.Proceedings of the 33rd International Florida Artificial Intelligence Research Society Conference, FLAIRS 2020335–340 (2020). URLhttps://arxiv.org/abs/2001.09346v2

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  30. [30]

    URLhttps://link.springer.com/chapter/ 10.1007/978-3-030-59137-3_4

    Rashidian, S.et al.Smooth-gan: Towards sharp and smooth synthetic ehr data generation.Lec- ture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lec- ture Notes in Bioinformatics)12299 LNAI, 37–48 (2020). URLhttps://link.springer.com/chapter/ 10.1007/978-3-030-59137-3_4

    work page doi:10.1007/978-3-030-59137-3_4 2020

  31. [31]

    Biswal, S.et al.Eva: Generating longitudinal elec- tronic health records using conditional variational au- toencoders.Proceedings of Machine Learning Research 149, 1–22 (2021)

    work page 2021

  32. [32]

    URLhttps://link.springer.com/ chapter/10.1007/978-3-030-93663-1_12

    Sun, S.et al.Generating longitudinal synthetic ehr data with recurrent autoencoders and generative adversar- ial networks.Lecture Notes in Computer Science (in- cluding subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)12921 LNCS, 153–165 (2021). URLhttps://link.springer.com/ chapter/10.1007/978-3-030-93663-1_12

    doi:10.1007/978-3-030-93663-1_12 2021

  33. [33]

    A., Sun, J

    Zhang, Z., Yan, C., Lasko, T. A., Sun, J. & Malin, B. A. Synteg: a framework for temporal structured electronic health data simulation.Journal of the American Med- ical Informatics Association28, 596–604 (2021). URL https://dx.doi.org/10.1093/jamia/ocaa262

    work page doi:10.1093/jamia/ocaa262 2021

  34. [34]

    URLhttps://www.nature.com/articles/ s41746-023-00888-7

    Yoon, J.et al.Ehr-safe: generating high-fidelity and privacy-preserving synthetic electronic health records.npj Digital Medicine 2023 6:16, 1–11 (2023). URLhttps://www.nature.com/articles/ s41746-023-00888-7

    work page 2023

  35. [35]

    URLhttps://arxiv.org/abs/2402

    Pang, C.et al.Cehr-gpt: Generating electronic health records with chronological patient timelines.arXiv preprint(2024). URLhttps://arxiv.org/abs/2402. 04400v2

    work page 2024

  36. [36]

    InICLR 2024 Workshop on Learning from Time Series For Health(2024)

    Arnrich, B.et al.Medical event data standard (MEDS): Facilitating machine learning for health. InICLR 2024 Workshop on Learning from Time Series For Health(2024). URLhttps://openreview.net/forum? id=IsHy2ebjIG

    work page 2024

  37. [37]

    R., Openai, K

    Openai, A. R., Openai, K. N., Openai, T. S. & Ope- nai, I. S. Improving language understanding by gener- ative pre-training URLhttps://gluebenchmark.com/ leaderboard

    work page

  38. [38]

    & Levy, O

    Haviv, A., Ram, O., Press, O., Izsak, P. & Levy, O. Transformer language models without positional en- codings still learn positional information.Findings of the Association for Computational Linguistics: EMNLP 20221382–1390 (2022). URLhttps://arxiv.org/ abs/2203.16634v2

    Pith/arXiv arXiv 2022

  39. [39]

    URLhttps://books.google.com/books?id= JxpnzQEACAAJ

    OHDSI.The Book of OHDSI: Observational Health Data Sciences and Informatics(OHDSI, 2019). URLhttps://books.google.com/books?id= JxpnzQEACAAJ

    work page 2019

  40. [40]

    URL/doi/pdf/10

    Hripcsak, G.et al.Characterizing treatment pathways at scale using the ohdsi network.Proceedings of the National Academy of Sciences of the United States of America113, 7329–7336 (2016). URL/doi/pdf/10. 1073/pnas.1510502113?download=true

    work page 2016

  41. [41]

    URLhttps://doi

    Yan, C.et al.A multifaceted benchmarking of syn- thetic electronic health record generation models.Na- ture Communications13(2022). URLhttps://doi. org/10.1038%2Fs41467-022-35295-1

    work page 2022

  42. [42]

    & Malin, B

    Zhang, Z., Yan, C. & Malin, B. A. Membership in- ference attacks against synthetic health data.Jour- nal of biomedical informatics125(2022). URLhttps: //pubmed.ncbi.nlm.nih.gov/34920126/

    work page arXiv 2022

  43. [43]

    & Bass, J

    El Emam, K., Mosquera, L. & Bass, J. Evaluating iden- tity disclosure risk in fully synthetic health data: model development and validation.Journal of medical Internet research22, e23139 (2020). 14

    work page 2020

  44. [44]

    & Chaudhuri, K

    Wang, Y., Jha, S. & Chaudhuri, K. Analyzing the ro- bustness of nearest neighbors to adversarial examples 5133–5142 (2018). URLhttps://proceedings.mlr. press/v80/wang18c.html

    work page 2018

  45. [45]

    W., Lee, K

    Devlin, J., Chang, M. W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. InNAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference (2019). 15 9 Supplementary Materials 9.1 Simulatio...

    work page 2019

  46. [46]

    If (t 2 −t 1) mod 4 = 0 andx 1 = 1, theny=¬x 2

    work page

  47. [47]

    Else if (t 2 −t 1) mod 3 = 0 andx 1 = 0, theny=x 2

    work page

  48. [48]

    Else ift 2 −t 1 ≤7, theny= XOR(x 1, x2)

    work page

  49. [49]

    Else if 7< t 2 −t 1 ≤14, theny= AND(x 1, x2)

    work page

  50. [50]

    30 _ d a y _ r e a d m i s s i o n _ p r e d i c t i o n

    Else ift 2 −t 1 >14, theny= OR(x 1, x2). We generated 1,000 simulated data points using uniform sampling and constructed the corresponding outputy. For illustration, we used the Transformer encoder architecture to develop two models: one employing the summation strategy (M odelsum) and the other incorporating a time token (M odel timetoken) betweenx 1 and...

    work page