Pith. sign in

REVIEW 3 major objections 6 minor 46 references

Towards modeling evolving longitudinal health trajectories with a transformer-based deep learning model

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a causally masked transformer trained to predict the same future diagnoses at every past health-code event yields evolving risk trajectories that match a bidirectional transformer and classical baselines on…

desk verdict Honest, novel training trick for evolving EHR trajectories, but the t2f generalization claim is not supported by the training setup. read the letter →

arxiv 2412.08873 v1 pith:ZOF7WEQD submitted 2024-12-12 cs.LG cs.AI

Hans Moen , Vishnu Raj , Andrius Vabalas , Markus Perola , Samuel Kaski , Andrea Ganna , Pekka Marttinen This is my paper · ORCID
classification cs.LGcs.AI
keywords EHRlongitudinalhealthtrajectoriesdiseasepredictiontransformerscausalattentionmasktrajectoryanalysisforecastintervalriskevolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a transformer can be trained to give a running, per-event prediction of whether a person will develop any of 21 health outcomes in a fixed five-year window, instead of a single prediction at the end of the person's history. The method, called Evolve, applies a causal attention mask and asks the model, at every historical code, to predict the same label set for the forecast interval; the model is conditioned on how many years remain until that interval through a time-to-forecast input. On a nationwide register cohort, Evolve's discrimination is on par with a bidirectional BERT-style transformer and with logistic regression and XGBoost, so the extra trajectory information does not come at a large accuracy cost. The authors then show two uses of the per-time predictions: finding which diagnoses and drug codes most often trigger upward jumps in predicted risk, and measuring how a person's nearest neighbours in the embedding space change from year to year, with shifts visible around childbirth and bereavement. The payoff, if the approach holds, is continuous monitoring of health trajectories and earlier detection of events that precede disease onset.

What carries the argument

The load-bearing mechanism is the combination of a causal attention mask with a per-position training objective. At every input position $t$, the model must predict the same label vector $y_n$ for the forecast interval, so the total loss is the average over positions of the binary cross-entropy between $\hat{y}_n(t)$ and $y_n$; this forces each position to act as a standalone predictor using only the history up to that point. The other essential input is the time-to-forecast embedding, $t2f$, which encodes the number of years from each code to the start of the forecast interval and is summed with code, age, and position embeddings, letting the model condition each per-position prediction on how far the forecast window lies ahead. A sigmoid decision layer on top of the final hidden state at each position converts the representation into evolving class probabilities, and position-weighted mean pooling of the per-position embeddings yields age-wise embeddings for neighbourhood analysis.

What would settle it

Train the same Evolve architecture on a second, differently positioned forecast interval (e.g., 2010–2014) and check whether the model, with only $t2f$ changed at inference, produces sensible per-position risk trajectories; if the predictions degrade or the embedding must be retrained for each interval, the claimed generality of the evolving-prediction mechanism is refuted.

Watch

Extended reading notes

Core claim

The central claim is that a unidirectional transformer trained with a causal attention mask and a position-wise multi-label objective produces evolving predictions, $\hat{y}_n(1:T_n)$, where each $\hat{y}_n(t)$ depends only on the person's health history up to time $t$, and that these predictions perform comparably to a bidirectional CLS transformer while enabling trajectory analysis. For each individual, the model outputs sigmoid probabilities for 19 diagnoses plus death and a 'none' class at every code position; as more of the person's history is observed, these probabilities are intended to track the changing risk of outcomes in the 2016–2020 forecast interval. The paper demonstrates that the largest upward jumps in these sigmoid probabilities are frequently caused by clinically sensible codes (for example, selective $\beta$-blockers before coronary heart disease and SSRIs before depression), and that the rate at which a person's $k$-nearest neighbours in the embedding space change from one age to the next rises around major life events such as childbirth and the death of a child. The paper positions this as an alternative to change-point detection: instead of looking for discontinuities, it models continuously evolving changes in predictions and in the latent neighborhood.

Load-bearing premise

The approach assumes the model has learned a general notion of time remaining until the forecast window rather than simply the distance to the year 2016, because it is trained on only one fixed forecast interval; if that is false, changing the time-to-forecast input at inference would not produce meaningful predictions for other intervals, and the method's main advantage over a fixed-window classifier disappears.

Editorial extensions

If this is right

  • If the central claim holds, a single trained model can display a person's predicted risk for each of 21 outcomes at every point in their recorded history, without retraining for different cutoff times.
  • Because Evolve matches a bidirectional transformer and classical baselines on AUROC, AUPRC, and Recall@4, the causal mask and position-wise loss do not force a meaningful accuracy trade-off for this task.
  • The sigmoid-jump analysis can surface early predictors: codes such as beta-blockers before coronary heart disease and SSRIs before depression appear years before the forecast interval, supporting the idea of early-detection screening.
  • The embedding neighbourhood-change measure rises around life events like childbirth and bereavement, so the model can be used retrospectively to locate when a person's health trajectory shifted.
  • The time-to-forecast conditioning, if it generalizes, would let the same model be queried for other forecast horizons by changing $t2f$ at inference time, as the paper hypothesizes in its discussion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the $t2f$ embedding transfers to a second forecast interval, so a direct experiment would be to train Evolve on 2010–2014 and check whether changing $t2f$ at inference still yields sensible per-position trajectories.
  • The per-position sigmoid jumps could be turned into an individual-level early-warning score by comparing each person's jump magnitudes against their own baseline trajectory; the paper only aggregates jumps across the validation set.
  • The neighbourhood-change measure $r_n(a,k)$ is presented as a population-level signal, and a natural extension is to test whether large deviations from a person's own average rate of change precede a diagnosis, giving a personalized alarm.
  • Because the data are coded Finnish register entries, the trajectory properties may be tied to that specific code vocabulary and healthcare system; transfer to other coding systems is an open question the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript introduces Evolve, a decoder-only transformer trained with causal masking on longitudinal Finnish register data, producing multi-label disease-onset predictions at every code position. The model is conditioned on code, age, position, and 'years to forecast interval' (t2f). On a 20% held-out test of about 1.04 million individuals, Evolve's final-position AUROC, AUPRC, and Recall@4 are statistically indistinguishable from the bidirectional CLS transformer and slightly below XGBoost. The remaining sections present qualitative trajectory analyses: sigmoid jumps, embedding-neighborhood changes around bereavement and childbirth, and nearest-neighbor similarity trajectories.

Significance. If the trajectory claims were quantitatively supported, this would be a useful tool for monitoring evolving health trajectories in national register data. The paper deserves credit for a large real-world evaluation with bootstrapped confidence intervals, a reasonable head-to-head comparison, and careful separation of historical and forecast intervals with a buffer. However, the distinguishing contribution is currently supported mainly by selected visualizations and descriptive statistics, and the one general claim about changing the forecast interval is not supported by the training setup.

major comments (3)
  1. [Section 3.1 and Discussion] The Discussion claims that 'the trained model should allow exploration of health trajectories relative to other forecast intervals simply by changing t2f at inference time.' This is not supported by the training setup. Because there is a single forecast interval (2016-2020) for every individual, each code at year y and age a satisfies t2f = 2016 - y = (2016 - birth_year) - a. Thus, within each individual, t2f is perfectly collinear with age, and across individuals it is an encoding of birth cohort. The model never sees the same code history paired with two different t2f values, so the t2f embedding cannot have learned a transferable 'distance to forecast' semantic; changing t2f at inference is an out-of-distribution intervention equivalent to holding age fixed and changing an unobserved calendar or birth-cohort variable. The fixed-interval results in Table 2 are unaffected, but the abstract's 'conditioned on the time of the forecast period' and the transferability claim need either retraction or a new experiment with multiple forecast intervals or varied t2f offsets.
  2. [Section 5.2, Figures 4-7, Table 3] The paper's central contribution, 'promising trajectory modeling properties,' is not quantitatively evaluated. Section 5.2's event analysis uses the model's own sigmoid jumps to define important codes; Table 3 reports descriptive frequencies with no comparison to a null model (for example, codes expected by base rate or random jump positions), so the listed associations cannot be distinguished from artifacts of the jump-extraction procedure. Figure 4 shows mean neighborhood-change curves for groups of 18-27 mothers without confidence intervals or a statistical test for the target-control difference, making the claim that bereavement 'seem[s] to have an impact' unverifiable. Figures 5-7 are selected individual examples. I request at least one quantitative validation of the trajectory output, such as a frequency-matched null or permutation test for Table 3, bootstrap confidence intervals and a difference test for Figure 4, or an early-warning metric with time-to-detection compared against a non-evolving baseline.
  3. [Section 5.1 and Section 3.1] Table 2 evaluates Evolve only at the final position t = T_n. Since the model's defining property is a prediction at every code position, the paper should report the quality of intermediate predictions, for example AUROC(t) or AUPRC(t) as a function of age, or time until the prediction first exceeds a high-precision threshold. Without this, it is unknown whether the intermediate sigmoid values used in Figures 1, 5, 6, and 7 carry meaningful information beyond a smoothed version of the final prediction. This is a load-bearing gap because the trajectory analyses are predicated on those intermediate outputs.
minor comments (6)
  1. [Abstract] The phrase 'our approach enable the model' should be 'our approach enables the model.'
  2. [Equation (5) and Figure 4] Please define the reference population over which the k-nearest neighbors are computed; with 18-27 target individuals and k=1000, it is unclear whether neighbors are drawn from the whole test set, the validation set, or the matched control group.
  3. [Section 3.3] The description of position-weighted mean pooling should clarify how age embeddings are handled for ages with no recorded codes and whether the same reference individuals are used at every age when computing neighborhood changes.
  4. [Section 3.2] For the CLS baseline, please clarify whether it receives the same input features, including t2f, and how its position and age embeddings are constructed; this matters for the comparability of the Table 2 results.
  5. [Table 3 and Table 5] Several class and code labels are abbreviated or inconsistently formatted (for example, 'CHD Heart failure, not strict' and 'Noninflammatory disorders of female genital tract'); mapping these to the class definitions in Appendix B would improve readability.
  6. [Appendix A] The statement that class weighting makes the latent embedding space more discriminative is made in the main text's Discussion but the supporting Figure 8 is only in the appendix; consider adding a brief pointer in Section 5 or 6.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central prediction result is an honest held-out comparison against external baselines, and the trajectory analyses are descriptive uses of the model's own outputs rather than derivations that reduce to their inputs.

full rationale

The paper's primary quantitative claim—that Evolve performs comparably to LogReg, XGBoost, and the CLS transformer on held-out test data (Table 2)—is an external evaluation: the Evolve model is trained on the training split and scored on the test split with standard metrics (AUROC, AUPRC, Recall@4), and none of the baseline models is fitted to the Evolve outputs. The per-position training objective (Eq. 2-3) is a design choice, not a fitted parameter whose value is later relabeled as a prediction. The trajectory analyses in Sections 5.2 and 5.3 are explicitly model introspection: they report the model's own sigmoid jumps, embedding-neighborhood changes, and age-wise similarities. Interpreting these as medically meaningful is a hypothesis, but the paper does not claim these interpretations are independently validated predictions, so there is no reduction-by-construction. The Discussion's claim that changing t2f at inference time 'should allow exploration of health trajectories relative to other forecast intervals' is a speculation about extrapolation, not a derived result; the acknowledged single forecast interval means this generalization is untested, but an unsupported extrapolation is a validity risk, not a circularity. The only self-citations are data-resource and related-work references (e.g., FinRegistry [38], Vabalas et al. [16]) that are not load-bearing for the method's derivation. Accordingly no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 10 free parameters · 7 assumptions · 0 invented entities

The model rests on standard deep-learning machinery and a set of domain assumptions about register data quality and label validity. The main free parameters are hyperparameters chosen by grid search or set directly; the analysis-specific parameters (k values, jump thresholds) are also chosen by hand. No new physical entities are introduced.

free parameters (10)
  • learning rate = 1e-3
    Chosen via basic grid search (Appendix A).
  • dropout = 0.1
    Chosen via basic grid search (Appendix A).
  • batch size = 160
    Largest batch that fits in 40GB A100 memory (Appendix A).
  • embedding size = 384
    Set by authors without reported tuning (Appendix A).
  • num layers = 8
    Set by authors (Appendix A).
  • num attention heads = 8
    Set by authors (Appendix A).
  • max sequence length = 400
    Used to truncate patient histories; impact on trajectory coverage not analyzed (Appendix A).
  • sigmoid jump threshold = class-specific mean jump from validation set
    Threshold for detecting important events in Section 5.2, fitted to validation data.
  • neighborhood size k = 1000
    Used in neighborhood-change analysis (Section 5.2).
  • top-k reference individuals = 100
    Used in nearest-neighbors analysis (Section 5.3).
assumptions (7)
  • standard math Transformer self-attention and backpropagation provide a valid learning framework.
    Underpins the entire architecture, Section 3.1.
  • domain assumption Finnish register codes (diagnoses, procedures, drug purchases) meaningfully represent health status.
    Needed for the model to learn trajectories; Section 4.
  • domain assumption First-time diagnosis labels in the forecast interval are accurate ground truth and the 3-month buffer prevents leakage.
    Defines the prediction task; Section 4.
  • domain assumption The 19 curated clinical endpoints adequately represent major disease burden.
    Determines label set; Section 4.
  • domain assumption Propensity score matching (age, sex of child, parental age, number of children) makes controls comparable to the target group.
    Section 5.2; matching criteria are limited and may not capture all confounders.
  • domain assumption Cosine similarity in the embedding space reflects clinical similarity between individuals.
    Used throughout Sections 5.2 and 5.3; not independently validated.
  • ad hoc to paper Changing t2f at inference generalizes to different forecast intervals.
    Stated in Discussion, never tested; all training uses the same 2016-2020 forecast interval.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards modeling evolving longitudinal health trajectories with a transformer-based deep learning model." pith.science (2026). https://pith.science/paper/ZOF7WEQD

@misc{pith2026241208873,
  author       = {Pith},
  title        = {Pith review of: Towards modeling evolving longitudinal health trajectories with a transformer-based deep learning model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZOF7WEQD}},
  note         = {Machine review of arXiv:2412.08873}
}
read the original abstract

Health registers contain rich information about individuals' health histories. Here our interest lies in understanding how individuals' health trajectories evolve in a nationwide longitudinal dataset with coded features, such as clinical codes, procedures, and drug purchases. We introduce a straightforward approach for training a Transformer-based deep learning model in a way that lets us analyze how individuals' trajectories change over time. This is achieved by modifying the training objective and by applying a causal attention mask. We focus here on a general task of predicting the onset of a range of common diseases in a given future forecast interval. However, instead of providing a single prediction about diagnoses that could occur in this forecast interval, our approach enable the model to provide continuous predictions at every time point up until, and conditioned on, the time of the forecast period. We find that this model performs comparably to other models, including a bi-directional transformer model, in terms of basic prediction performance while at the same time offering promising trajectory modeling properties. We explore a couple of ways to use this model for analyzing health trajectories and aiding in early detection of events that forecast possible later disease onsets. We hypothesize that this method may be helpful in continuous monitoring of peoples' health trajectories and enabling interventions in ongoing health trajectories, as well as being useful in retrospective analyses.

Figures

Figures reproduced from arXiv: 2412.08873 by the authors.

Figure 1
Figure 1. Figure shows an example of two artificial persons’ health progressions. The plots shows how the model’s [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The figure illustrates the model and its inputs: codes, ages, positional information (pos), and years to forecast [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The dataset was split into historical and forecast intervals and further split into train, validation, and test sets. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Average neighborhood change from one year to the next. The included individuals are mothers who did (red, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The figure shows the evolving age-wise similarities between the target individuals and the top [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The figure shows the evolving health trajectory of a person who will end up with knee arthritis and atrial [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The figure shows the evolving health trajectory of a person who will get multiple diseases and die in the [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: This illustrates the differences between not using class weighting (top) and using class weighting (bottom) [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 38 canonical work pages

  1. [1]

    Deep representation learning of patient data from electronic health records (ehr): A systematic review

    Yuqi Si, Jingcheng Du, Zhao Li, Xiaoqian Jiang, Timothy Miller, Fei Wang, W Jim Zheng, and Kirk Roberts. Deep representation learning of patient data from electronic health records (ehr): A systematic review. Journal of biomedical informatics, 115:103671, 2021

    work page 2021

  2. [2]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  3. [3]

    Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

    Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4(1):86, 2021

    work page 2021

  4. [4]

    Interpreting deep embeddings for disease progression clustering

    Anna Munoz-Farre, Antonios Poulakakis-Daktylidis, Dilini Mahesha Kothalawala, and Andrea Rodriguez- Martinez. Interpreting deep embeddings for disease progression clustering. arXiv preprint arXiv:2307.06060, 2023

    work page Pith review arXiv 2023

  5. [5]

    Generic medical concept embedding and time decay for diverse patient outcome prediction tasks

    Yupeng Li, Wei Dong, Boshu Ru, Adam Black, Xinyuan Zhang, and Yuanfang Guan. Generic medical concept embedding and time decay for diverse patient outcome prediction tasks. Iscience, 25(9), 2022

    work page 2022

  6. [6]

    Carson Lam, Rahul Thapa, Jenish Maharjan, Keyvan Rahmani, Chak Foon Tso, Navan Preet Singh, Satish Casie Chetty, and Qingqing Mao. Multitask learning with recurrent neural networks for acute respiratory distress syndrome prediction using only electronic health record data: model development and validation study. JMIR Medical Informatics, 10(6):e36202, 2022

    work page 2022

  7. [7]

    Transformers for cardiac patient mortality risk prediction from heterogeneous electronic health records

    Emmi Antikainen, Joonas Linnosmaa, Adil Umer, Niku Oksala, Markku Eskola, Mark van Gils, Jussi Hernesniemi, and Moncef Gabbouj. Transformers for cardiac patient mortality risk prediction from heterogeneous electronic health records. Scientific Reports, 13(1):3517, 2023

    work page 2023

  8. [8]

    An optimized stacked support vector machines based expert system for the effective prediction of heart failure

    Liaqat Ali, Awais Niamat, Javed Ali Khan, Noorbakhsh Amiri Golilarz, Xiong Xingzhong, Adeeb Noor, Redhwan Nour, and Syed Ahmad Chan Bukhari. An optimized stacked support vector machines based expert system for the effective prediction of heart failure. IEEE Access, 7:54007–54014, 2019

    work page 2019

Show all 46 references
  1. [9]

    Endpoint prediction of heart failure using electronic health records

    Jiebin Chu, Wei Dong, and Zhengxing Huang. Endpoint prediction of heart failure using electronic health records. Journal of Biomedical Informatics, 109:103518, 2020

    2020

  2. [10]

    Xgboost model for chronic kidney disease diagnosis

    Adeola Ogunleye and Qing-Guo Wang. Xgboost model for chronic kidney disease diagnosis. IEEE/ACM transactions on computational biology and bioinformatics, 17(6):2131–2140, 2019

    2019

  3. [11]

    Detection of the chronic kidney disease using xgboost classifier and explaining the influence of the attributes on the model using shap

    Md Johir Raihan, Md Al-Masrur Khan, Seong-Hoon Kee, and Abdullah-Al Nahid. Detection of the chronic kidney disease using xgboost classifier and explaining the influence of the attributes on the model using shap. Scientific Reports, 13(1):6263, 2023

    2023

  4. [12]

    Xgboost-shap-based interpretable diagnostic framework for alzheimer’s disease

    Fuliang Yi, Hui Yang, Durong Chen, Yao Qin, Hongjuan Han, Jing Cui, Wenlin Bai, Yifei Ma, Rong Zhang, and Hongmei Yu. Xgboost-shap-based interpretable diagnostic framework for alzheimer’s disease. BMC medical informatics and decision making, 23(1):137, 2023

    2023

  5. [13]

    Interpretable classifiers for prediction of disability trajectories using a nationwide longitudinal database

    Yafei Wu, Chaoyi Xiang, Maoni Jia, and Ya Fang. Interpretable classifiers for prediction of disability trajectories using a nationwide longitudinal database. BMC geriatrics, 22(1):627, 2022

    2022

  6. [14]

    Early prediction of heart disease via lstm-xgboost

    Xiaodong Zang, Jin Du, and Yuansheng Song. Early prediction of heart disease via lstm-xgboost. In Proceedings of the 2023 9th international conference on computing and artificial intelligence, pages 631–637, 2023

    2023

  7. [15]

    Predicting cardiovascular health trajectories in time-series electronic health records with lstm models

    Aixia Guo, Rahmatollah Beheshti, Yosef M Khan, James R Langabeer, and Randi E Foraker. Predicting cardiovascular health trajectories in time-series electronic health records with lstm models. BMC medical informatics and decision making, 21:1–10, 2021

    2021

  8. [16]

    Rodos- thenous, Aoxing Liu, Sara Hägg, Markus Perola, and Andrea Ganna

    Andrius Vabalas, Tuomo Hartonen, Pekka Vartiainen, Sakari Jukarainen, Essi Viippola, Rodosthenis S. Rodos- thenous, Aoxing Liu, Sara Hägg, Markus Perola, and Andrea Ganna. Deep learning-based prediction of one-year mortality in the entire finnish population is an accurate but ...

    2023 doi

  9. [17]

    Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies

    Feng Xie, Han Yuan, Yilin Ning, Marcus Eng Hock Ong, Mengling Feng, Wynne Hsu, Bibhas Chakraborty, and Nan Liu. Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies. Journal of biomedical informatics, ...

    2022

  10. [18]

    MuST: Multimodal spatiotemporal graph-transformer for hospital readmission prediction

    Yan Miao and Lequan Yu. MuST: Multimodal spatiotemporal graph-transformer for hospital readmission prediction. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 276–285. Springer, 2023

    2023

  11. [19]

    Predicting unplanned readmissions in the intensive care unit: a multimodality evaluation

    Eitam Sheetrit, Menachem Brief, and Oren Elisha. Predicting unplanned readmissions in the intensive care unit: a multimodality evaluation. Scientific Reports, 13(1):15426, 2023

    2023

  12. [20]

    Patient phenotyping for atopic dermatitis with transformers and machine learning: Algorithm development and validation study

    Andrew Wang, Rachel Fulton, Sy Hwang, David J Margolis, Danielle Mowery, et al. Patient phenotyping for atopic dermatitis with transformers and machine learning: Algorithm development and validation study. JMIR Formative Research, 8(1):e52200, 2024

    2024

  13. [21]

    BEHRT: transformer for electronic health records

    Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi. BEHRT: transformer for electronic health records. Scientific reports, 10(1):7155, 2020

    2020

  14. [22]

    Hi-BEHRT: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records

    Yikuan Li, Mohammad Mamouei, Gholamreza Salimi-Khorshidi, Shishir Rao, Abdelaali Hassaine, Dexter Canoy, Thomas Lukasiewicz, and Kazem Rahimi. Hi-BEHRT: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic hea...

    2022

  15. [23]

    Learning the graphical structure of electronic health records with graph convolutional transformer

    Edward Choi, Zhen Xu, Yujia Li, Michael Dusenberry, Gerardo Flores, Emily Xue, and Andrew Dai. Learning the graphical structure of electronic health records with graph convolutional transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages...

    2020

  16. [24]

    Self- supervised forecasting in electronic health records with attention-free models

    Yogesh Kumar, Alexander Ilin, Henri Salo, Sangita Kulathinal, Maarit K Leinonen, and Pekka Marttinen. Self- supervised forecasting in electronic health records with attention-free models. IEEE Transactions on Artificial Intelligence, 2024

    2024

  17. [25]

    Clinical knowledge extraction via sparse embedding regression (KESER) with multi-center large scale electronic health record data

    Chuan Hong, Everett Rush, Molei Liu, Doudou Zhou, Jiehuan Sun, Aaron Sonabend, Victor M Castro, Petra Schubert, Vidul A Panickan, Tianrun Cai, et al. Clinical knowledge extraction via sparse embedding regression (KESER) with multi-center large scale electronic health record da...

    2021

  18. [26]

    Exploiting hierarchy in medical concept embedding

    Anthony Finch, Alexander Crowell, Mamta Bhatia, Pooja Parameshwarappa, Yung-Chieh Chang, Jose Martinez, and Michael Horberg. Exploiting hierarchy in medical concept embedding. JAMIA open, 4(1):ooab022, 2021

    2021

  19. [27]

    MedGraph: Structural and temporal representation learning of electronic medical records

    Bhagya Hettige, Yuan-Fang Li, Weiqing Wang, Suong Le, and Wray Buntine. MedGraph: Structural and temporal representation learning of electronic medical records. arXiv preprint arXiv:1912.03703, 2019

    1912 arXiv

  20. [28]

    Temporal phenotyping using deep predictive clustering of disease progression

    Changhee Lee and Mihaela Van Der Schaar. Temporal phenotyping using deep predictive clustering of disease progression. In International conference on machine learning, pages 5767–5777. PMLR, 2020

    2020

  21. [29]

    Interpreting differentiable latent states for healthcare time-series data

    Yu Chen, Nivedita Bijlani, Samaneh Kouchaki, and Payam Barnaghi. Interpreting differentiable latent states for healthcare time-series data. arXiv preprint arXiv:2311.17560, 2023

    2023 arXiv

  22. [30]

    Multiple change-point detection for poisson processes

    Charlotte Dion-Blanc, E Lebarbier, and Stéphane Robin. Multiple change-point detection for poisson processes. arXiv preprint arXiv:2302.09103, 2023

    2023 arXiv

  23. [31]

    Real-time change-point detection: A deep neural network- based adaptive approach for detecting changes in multivariate time series data

    Muktesh Gupta, Rajesh Wadhvani, and Akhtar Rasool. Real-time change-point detection: A deep neural network- based adaptive approach for detecting changes in multivariate time series data. Expert Systems with Applications, 209:118260, 2022

    2022

  24. [32]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186, 2019

    2019

  25. [33]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  26. [34]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  27. [35]

    John Wiley & Sons, 2013

    David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant.Applied logistic regression. John Wiley & Sons, 2013

    2013

  28. [36]

    XGBoost: A scalable tree boosting system

    Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016. 12 Modeling evolving health trajectories

    2016

  29. [37]

    SGPT: GPT sentence embeddings for semantic search

    Niklas Muennighoff. SGPT: GPT sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904, 2022

    2022 arXiv

  30. [38]

    Data resource profile: Nationwide registry data for high-throughput epidemiology and machine learning (FinRegistry)

    Essi Viippola, Sara Kuitunen, Rodosthenis S Rodosthenous, Andrius Vabalas, Tuomo Hartonen, Pekka Vartiainen, Joanne Demmler, Anna-Leena Vuorinen, Aoxing Liu, Aki S Havulinna, et al. Data resource profile: Nationwide registry data for high-throughput epidemiology and machine le...

    2023

  31. [39]

    Andrej Karpathy. nanoGPT. https://github.com/karpathy/nanoGPT, 2023. Accessed: 2023-04-20

    2023

  32. [40]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  33. [41]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fe...

    2020

  34. [42]

    Scikit-learn: Machine learning in Python

    Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12:2825–2830, 2011

    2011

  35. [43]

    Biological age predictors

    Juulia Jylhävä, Nancy L Pedersen, and Sara Hägg. Biological age predictors. EBioMedicine, 21:29–36, 2017

    2017

  36. [44]

    A unified approach to interpreting model predictions

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017

    2017

  37. [45]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319–3328, 2017

    2017

  38. [46]

    Transformers in time series: a survey

    Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: a survey. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6778–6786, 2023. Appendix A Model training detai...

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.