{"id":"0e4879b3-fbf6-4a88-a732-a50b7c93977f","arxiv_id":"2507.00574","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A decoder-only Transformer pretrained with next-visit multi-label prediction on EHRs achieves zero-shot dementia and knee OA forecasting comparable to a fully fine-tuned BERT baseline.","lead":"A generative AI model trained on 1.29 million patients' electronic health records predicts future dementia and knee osteoarthritis diagnoses without any task-specific fine-tuning, matching a fine-tuned BERT model on several metrics. The paper also warns that repeated diagnosis codes can inflate apparent model performance when new disease onsets are not separated from recurring ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies how the 2/5-year horizon is encoded at zero-shot inference, and the only horizon-control mechanism—<sep> positional conditioning on next-visit time—has no demonstrated extrapolation beyond typical inter-visit intervals; Table 1 may not measure long-term zero-shot…","rationale":"The reader identified the same load-bearing assumption: the model conditions zero-shot forecasts on the time of the next visit through the sep positional embedding, but extrapolation to 2- and 5-year horizons is unvalidated. I agree this is the most serious issue. The baseline comparison is also under-specified, but even a strong, well-described baseline would not rescue the central claim if the horizon-conditioning mechanism is not demonstrated. Appendix B.2 explicitly defines the predictive distribution as conditioned on t_i+1 via the sep position, and Appendix A advertises 'flexible zero-shot forecasting across various future horizons without retraining,' yet §3.2, which contains the headline numbers, does not state what value t_i+1 takes during zero-shot inference. This is not a stylistic omission: it determines whether the method is doing what the claim says. If the authors used the actual next-visit time, the model has access to information unavailable at prediction time and the reported numbers are not long-term zero-shot forecasts. If they used T+H, the extrapolation is an empirical assumption that needs support. The suggested control cleanly separates these cases: if fixing the position to a short interval does not change rankings, the model's output is essentially position-invariant and the headline comparison is an artifact of the label window rather than the method. I therefore keep the reader's CONDITIONAL verdict and would require the inference protocol plus the control before accepting the central claim.","tokens_in":11611,"tokens_out":8189,"duration_ms":101683,"concrete_test":"Require the authors to disclose the exact inference-time sep-position rule used for Table 1, then run a control: keep all inputs, label definitions, and patient windows identical, but set the sep position to a fixed short interval (e.g., 30 days) for every patient instead of the reported 2- or 5-year horizon. If AUROC/AUPRC are statistically indistinguishable from the reported runs, the time-conditioning mechanism is inert and the zero-shot long-horizon claim is not established; if the short-interval run degrades substantially, the mechanism is doing horizon-specific work. Additionally, report the training distribution of inter-visit intervals and the fraction of next-visit gaps at or above 730 and 1825 days to show whether the required positions are in-distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the model forecasts 2- and 5-year incidence zero-shot. The only mechanism connecting the pretraining objective to those horizons is the positional embedding of <sep>: §2.1 assigns the <sep> token at visit i the position of visit i+1, so the model sees the inter-visit interval through RoPE, and §B.2 frames the target as P(k|H_ti, t_i+1) conditioned on that position. What §3.2 never states is how t_i+1 is set for the 2/5-year evaluations. If t_i+1 is the patient's actual next recorded visit, the model receives future visit timing unavailable at prediction time, and the '2/5-year' label is attached only to the label window, not to the model input. If t_i+1 is set to the prediction time plus 730 or 1825 days, the model is asked to generate a visit after a gap far beyond most training intervals: with a median of 21 visits over 10 years, typical next-visit gaps are on the order of months, while 2- and 5-year gaps are extreme tails of the observed interval distribution. RoPE has no guaranteed extrapolation to unseen relative position magnitudes, and the paper reports no calibration curve, no horizon ablation, and no analysis of whether performance depends on the encoded interval. Without this, Table 1 cannot distinguish genuine long-horizon forecasting from a model that has learned generic risk correlates and effectively ignores the sep-position conditioning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GPT-EHR, a decoder-only Transformer pretrained on a large institutional EHR dataset (NYU Langone, ~1.29M patients) using next-visit multi-label prediction with a power-decay regularization that down-weights repeated clinical tokens. The authors claim that this generative pretraining enables zero-shot forecasting of dementia and knee osteoarthritis incidence within 2- and 5-year horizons, matching or exceeding a fully fine-tuned masked-pretrained BERT baseline. They also introduce an 'on-time rate' metric to distinguish prediction of new onsets from repetition of known chronic conditions, and they report a comparison against a zero-shot LLaMA-3.3-70B model in an appendix. The central empirical claim rests on Table 1, which compares AUROC and AUPRC for the two conditions and horizons.","tokens_in":12024,"tokens_out":4294,"duration_ms":49173,"significance":"If the central claim holds, the paper demonstrates a valuable result: a generative pretraining objective for structured EHR data can produce clinically useful risk forecasts without task-specific fine-tuning, potentially lowering the cost of deploying predictive models across many conditions. The paper also contributes a practical warning about evaluation pitfalls in EHR foundation models (repeated events inflating performance) and proposes a concrete regularization and metric to address it. The scale of pretraining (1.6B parameters on 1.29M patients, 10 years of data) and the inclusion of a zero-shot LLM baseline are strengths. However, the significance is currently tempered by missing evaluation details that are load-bearing for the main claim, as detailed below.","major_comments":[{"comment":"The inference-time mechanism for the 2-year and 5-year prediction horizons is never specified. In pretraining, the model conditions on the time of the next visit through the positional embedding of the <sep> token (RoPE). At zero-shot evaluation, it is not stated whether the <sep> embedding is set to the patient's actual next recorded visit time (which would be future information unavailable at prediction time), or to prediction time plus 730/1825 days. If the former, the model is receiving oracle visit timing and the '2/5-year' label applies only to the label window, not the model input. If the latter, the model is asked to extrapolate RoPE to relative gaps far beyond typical training inter-visit intervals (median 21 visits over 10 years implies gaps on the order of months). The paper reports no ablation, calibration curve, or analysis showing that predictions actually vary with the encoded horizon. Without this, Table 1 cannot be interpreted as evidence of genuine long-horizon zero-shot forecasting. This is a central, load-bearing point for the paper's main claim.","section":"§2.1, §3.2, Appendix B.2"},{"comment":"The fine-tuned BERT baseline is never described. The paper does not state the BERT architecture (e.g., number of layers/parameters), its pretraining data and masking strategy, the tokenization and visit-boundary representation used for fine-tuning, the fine-tuning procedure (including task-specific heads, loss, hyperparameters, and validation-based threshold selection), or whether it uses the same exclusion criteria as the GPT model. Since the central claim is that zero-shot GPT-EHR 'matches, and in some cases exceeds' a fully fine-tuned masked pretrained Transformer, the baseline must be specified in enough detail to assess whether the comparison is fair and reproducible. As written, the comparison is not verifiable.","section":"§3.2, Table 1"},{"comment":"The reported metrics do not fully support the claim of matching or exceeding the fine-tuned baseline. For knee OA, the BERT baseline has higher AUROC and AUPRC at both horizons. For dementia, GPT-EHR has higher AUROC but substantially lower AUPRC at both horizons (e.g., 5-year AUPRC 0.106 vs. 0.146, with non-overlapping confidence intervals). Given the low prevalence of these conditions, AUPRC is the more informative metric for clinical risk prediction, and the text 'maintained superior predictive performance' (for dementia) and 'matches, and in some cases exceeds' (Conclusion) overstates the evidence. The paper should either report a balanced interpretation or adjust the claim; at minimum, the discrepancy between AUROC and AUPRC should be explicitly discussed.","section":"Table 1, §4 Conclusion"},{"comment":"The contribution of the repeat-token regularization to the downstream zero-shot results is not demonstrated. The regularization is evaluated only on the pretraining next-visit prediction task (on-time rate for dementia and pancreatic cancer), and the downstream experiments use only δ=0.5 with no comparison to δ=1 (no regularization) or other values. The central 'key pitfall' claim about repeated events inflating performance is supported for pretraining metrics, but the fact that the proposed model's zero-shot advantage (or lack thereof) depends on the regularization is not established. Additionally, the definition of the on-time rate and the precision/recall calculation appears inconsistent: true positives are defined at the trajectory level ('a patient is considered a true positive if they eventually develop the condition and the model predicts it at any point across the rolling evaluation windows'), yet the text also states 'we only count the prediction as correct if it appears in the next visit.' These two definitions cannot both hold; the metric needs to be clarified.","section":"§3.1, Figure 2, §4"}],"minor_comments":[{"comment":"The hyperparameter table lists 'temporal decay 0.5' but the main text refers to 'decay factor δ'; please unify the nomenclature to avoid confusion.","section":"Appendix B.4, Table 3"},{"comment":"Several medication names contain typographical errors, e.g., 'RIV ASTIGMINE' should be 'RIVASTIGMINE'. These should be corrected for reproducibility since the codes are used for label construction.","section":"Appendix B.3, Table 2"},{"comment":"The LLaMA comparison reports only precision and recall for dementia at 5 years, with no AUROC/AUPRC, no number of test patients, and no threshold details; a fuller description would strengthen the comparison.","section":"Appendix C.2"},{"comment":"The exclusion criterion 'the disease onset occurred within one year following the prediction time point' is described, but it is not stated whether this exclusion is applied to both GPT and BERT evaluations identically; please clarify.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style submission, and the core idea is interesting, but the missing specification of the horizon-encoding mechanism at inference and the undefined BERT baseline are not merely presentation issues; they prevent a reader from interpreting Table 1 as evidence for the central claim. These are fixable with additional details and targeted experiments (e.g., an ablation varying the encoded horizon, and a description of the baseline). The AUPRC/AUROC discrepancy also needs a careful rewrite of the conclusion. I do not see grounds for rejection, but the revision must address the load-bearing gaps. The self-citation to Zhu et al. (2024) is a medRxiv preprint from the same group; it is relevant background but not essential to the paper's claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The repeat-token regularization and the on-time rate are the genuinely useful parts of this paper; the long-horizon zero-shot claim is the shaky part, because the paper never says how the 2- and 5-year horizons are encoded at inference.\n\nThe setup is sensible: next-visit multi-label prediction with a decoder-only Transformer, RoPE positions shared within a visit, and the <sep> token positioned at the time of the next visit. The authors identify a real pitfall—repeated codes inflate apparent next-visit performance—and they propose a weighted loss that down-weights tokens already in the patient's history. The on-time rate is a useful diagnostic, and Figure 2 shows an honest trade-off between repetition-based precision/recall and new-onset detection. That part is solid and worth emulating.\n\nThe soft spot is exactly where the stress-test note lands. The pretraining objective conditions on t_{i+1} through the <sep> positional embedding, but the zero-shot section never states how the model is run for a 2- or 5-year forecast. If the <sep> position is the patient's actual next visit, then the model is not being asked about a 2-year horizon at all, and the '2/5-year' label only describes the outcome. If the <sep> position is set to +730 or +1825 days, the model is being asked to extrapolate RoPE far outside the observed inter-visit interval distribution, and no calibration or horizon ablation is provided. Either way, Table 1 doesn't establish what it claims. This needs a clear description of the inference procedure and some evidence that the conditioning actually works at those horizons.\n\nThe fine-tuned BERT baseline is also underdescribed: no architecture, pretraining data, or fine-tuning details are given, so the comparison in Table 1 isn't reproducible. And the stated conclusion is a bit generous—the zero-shot model actually has lower AUPRC in all four tasks and lower AUROC for knee OA; it only beats BERT on dementia AUROC. That's still a decent zero-shot result, but 'matches, and in some cases exceeds' oversells it.\n\nWho gets value: anyone working on EHR foundation models, especially the evaluation pitfall. It deserves a serious referee, but the revision needs to specify the inference-time horizon mechanism and the baseline before the central claim can be trusted.","headline":"Useful evaluation commentary on repeated tokens, but the zero-shot long-horizon claim hinges on an unstated inference-time mechanism.","tokens_in":12466,"tokens_out":3123,"would_cite":false,"duration_ms":34667,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A decoder-only Transformer pretrained on next-visit prediction can forecast dementia and knee osteoarthritis 2 and 5 years out without fine-tuning.","keywords":["electronic health records","foundation models","zero-shot prediction","next-visit prediction","generative pretraining","dementia forecasting","knee osteoarthritis","repeated event regularization"],"falsifier":"Hold a fixed test patient's history constant and vary only the separator positional offset across horizons from one month to five years; a true long-horizon forecast should produce smoothly increasing predicted incidence of the new-onset code. A flat, non-monotonic, or discontinuous curve would show that the positional mechanism does not extrapolate, undermining the zero-shot long-horizon scores.","tokens_in":11476,"feed_emoji":"🩺","tokens_out":6459,"duration_ms":65988,"temperature":0.7,"pith_summary":"This paper argues that a decoder-only Transformer trained on one objective—predict every clinical event in the next visit from the full patient history plus the time elapsed until that visit—can carry out useful long-horizon disease forecasting with no task-specific fine-tuning. The authors pretrain a roughly 1.6-billion-parameter model on ten years of records from about 1.29 million patients at a large health system, then test it zero-shot on first incidence of dementia and knee osteoarthritis within 2 and 5 years. On dementia their model's AUROC exceeds that of a fully fine-tuned masked-pretrained BERT baseline; on knee OA it is close behind despite seeing no task labels. The paper also identifies an evaluation pitfall: repeated chronic-condition codes can inflate apparent performance, and it introduces a decay-based regularization plus an 'on-time rate' metric to separate new onsets from repeated events.","feed_headline":"Pretrained once, it forecasts dementia and knee OA years ahead","feed_subtitle":"Trained only to predict the next visit's events, the model matches fine-tuned baselines at 2- and 5-year horizons.","key_machinery":"The central object is a decoder-only GPT-2-style Transformer modified for visit-structured clinical sequences. Tokens within the same visit share a rotary positional embedding corresponding to elapsed time since the first visit, and the separator token ending visit i receives the positional embedding of visit i+1, so the representation the model conditions on explicitly encodes the inter-visit interval. Attention is causal across visits but full within a visit, reflecting the lack of fine-grained temporal order inside a single encounter. The training objective is weighted binary cross-entropy over the multi-hot token set of the next visit; the weight for each positive token decays as a power of its frequency in the patient's history, so repeated codes are down-weighted and new events are emphasized. This design is what allows zero-shot inference: the same separator representation plus a chosen horizon stands in for the next-visit time.","core_discovery":"The central claim is that next-visit multi-label prediction is a sufficient pretraining task for clinically useful zero-shot forecasting from structured electronic health records. Because every token in the next visit is predicted jointly, the model learns relationships across diagnoses, medications, and labs without task-specific labels; because the separator token carries the positional embedding of the next visit's timestamp, the model can in principle be asked to forecast over a chosen horizon. The pretraining regularization down-weights tokens that have appeared many times in the patient's history, shifting the model toward new onsets. Tested on dementia and knee osteoarthritis, the zero-shot model reaches AUROC 0.814 at two years and 0.781 at five years for dementia, beating fine-tuned BERT (0.731 and 0.721); for knee OA it reaches 0.724 and 0.693 versus 0.743 and 0.717. The authors read this as showing that the generative objective captures complex clinical dependencies without task-specific fine-tuning.","pith_inferences":["If the horizon mechanism extrapolates, this could collapse disease-specific model building into one pretraining run; any outcome with a code set becomes a zero-shot query.","A direct test of the mechanism would be to compare model probabilities for the same patient at interpolated horizons against actual two- and five-year outcomes, checking for monotonic increase with the horizon.","The on-time rate metric could generalize beyond this paper as a standard reporting practice, since it penalizes models that inflate scores by repeating known chronic codes."],"forward_implications":["Zero-shot forecasting works without task-specific fine-tuning for the two tested conditions, so the same checkpoint can be pointed at a new disease by supplying its code set.","Repeated-event regularization trades off overall precision and recall against earlier detection, so downstream users can choose an operating point by setting the decay factor.","The on-time rate gives a way to measure whether a model is forecasting new onsets or merely repeating chronic diagnoses.","Because the model conditions on the time of the next visit, the same pretrained weights can address multiple prediction horizons without retraining."],"supporting_citations":[{"why":"Supplies the next-visit multi-label prediction paradigm that the pretraining objective builds on.","marker":"[Steinberg et al., 2021]"},{"why":"Supplies the GPT-2 decoder-only Transformer architecture and generative pretraining approach.","marker":"[Radford et al., 2018]"},{"why":"Supplies rotary positional embeddings, repurposed so the separator token encodes the inter-visit time interval.","marker":"[Su et al., 2024]"},{"why":"Defines the masked-pretraining BERT model that is fine-tuned and used as the comparison baseline.","marker":"[Devlin et al., 2019]"},{"why":"First demonstrated performance degradation on new concepts and motivates the repeated-event regularization.","marker":"[Kraljevic et al., 2022]"},{"why":"Prior zero-shot health trajectory transformer using next-token prediction that this work contrasts with.","marker":"[Renc et al., 2024]"}],"fun_headline_variants":["Next-visit pretraining forecasts dementia and knee OA zero-shot","EHR foundation model: predict next visit, forecast years ahead","Zero-shot disease forecasting from next-visit EHR pretraining","Predict next visit, forecast dementia and knee OA without fine-tuning","A generative EHR model that rivals fine-tuning out of the box"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The zero-shot forecasts depend on the separator token's positional embedding, learned from real gaps between visits, to stand in for 2- and 5-year horizons at inference; if that embedding does not generalize to those longer gaps, the zero-shot scores do not measure long-horizon forecasting.","fun_headline_variants_meta":{"raw":{"variants":["Next-visit pretraining forecasts dementia and knee OA zero-shot","EHR foundation model: predict next visit, forecast years ahead","Zero-shot disease forecasting from next-visit EHR pretraining","Predict next visit, forecast dementia and knee OA without fine-tuning","A generative EHR model that rivals fine-tuning out of the box"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2475,"prompt_tokens":902,"completion_tokens":1573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1488}},"tokens_in":518,"tokens_out":1573,"duration_ms":14553,"temperature":1.0,"reasoning_tokens":1488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:11:45.234970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold a fixed test patient's history constant and vary only the separator positional offset across horizons from one month to five years; a true long-horizon forecast should produce smoothly increasing predicted incidence of the new-onset code. A flat, non-monotonic, or discontinuous curve would show that the positional mechanism does not extrapolate, undermining the zero-shot long-horizon scores.","supporting_citations":[{"cited_title":"A., Corbin, C","cited_arxiv_id":null,"evidence_quote":"Supplies the next-visit multi-label prediction paradigm that the pretraining objective builds on."},{"cited_title":"Improving language understanding by generative pre-training","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-2 decoder-only Transformer architecture and generative pretraining approach."},{"cited_title":"E., Was, J., Li, Q., Bates, D","cited_arxiv_id":null,"evidence_quote":"Prior zero-shot health trajectory transformer using next-token prediction that this work contrasts with."}],"review_version":1}