{"id":"0cc6a488-a90f-4bab-9ffe-c8cd7fa0372d","arxiv_id":"2502.06124","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A generative transformer trained on tokenized patient histories can be repurposed as a dynamic early-warning system, outperforming standard clinical scores and tabular baselines on MIMIC-IV emergency department predictions.","lead":"This paper presents ARES, a system that turns a generative AI model of patient medical timelines into dynamic, personalized risk estimates for outcomes like death, ICU admission, and long hospital stays. It reports higher prediction accuracy than several standard clinical scores and machine-learning baselines on a large emergency department dataset, and it is worth reading because it shows how generative foundation models could replace fixed-threshold early warning scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table S1 contradicts the paper's universal superiority claim: for Unknown-race ICU admission, MEDS-Tab has AUROC 0.953 vs ETHOS 0.941, so 'higher AUC in every case' is false as stated.","rationale":"The paper's contribution is framed as a universal zero-shot advantage over a strong tabular baseline. The abstract says ETHOS 'outperformed benchmark models... achieving superior AUC scores,' and the Results section explicitly says 'higher AUC values in every case.' This universal claim is the version that would justify 'foundation model' status. The paper's own Table S1 contains a direct counterexample: for ICU Admission among patients of Unknown race, MEDS-Tab's AUROC is 0.953 (0.942–0.964) versus ETHOS's 0.941 (0.928–0.953). Since the point estimate is lower for ETHOS, the literal 'every case' statement is false; the CIs overlap, so the appropriate claim is 'not significantly worse' in that subgroup, which is weaker. Figure 4's caption repeats the universal claim, compounding the issue. This is not a stylistic quibble: the abstract and Results rest on the claim to motivate the model's generalizability. The reader's flagged MC-sampling gap (Appendix A assumes exact sampling, while inference uses top-p nucleus sampling) is real but secondary: the paper presents calibration curves that empirically test the risk estimates, so the top-p concern is partially addressed; the Table S1 contradiction is not addressed anywhere. I therefore recommend keeping the CONDITIONAL verdict: the core methodology and code release are valuable, but the universal superiority claim must be corrected and significance testing added. The concrete check is to recompute the contested row from released code and, if confirmed, revise the claim. This is a good-faith finding; the paper's own Discussion already acknowledges limited benchmarking and uncertain clinical impact, which supports a nuanced interpretation.","tokens_in":23274,"tokens_out":6785,"duration_ms":61260,"concrete_test":"Recompute the Unknown-race ICU Admission AUCs from Table S1 using the released code and identical test split, with bootstrapped 95% CIs, and compare point estimates. If MEDS-Tab's point estimate remains higher than ETHOS's, amend the abstract, Figure 4 caption, and Results text to replace 'every case' with 'overall and in most subgroups,' and add significance tests for subgroup differences. As a verification, also scan all rows of Table S1 for any other subgroup/task where MEDS-Tab's point estimate exceeds ETHOS's.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'ARES consistently outperformed MEDS-Tab across both individual and composite endpoints, achieving higher AUC values in every case'—is directly contradicted by the paper's own Table S1. For ICU Admission in the Unknown-race subgroup, MEDS-Tab attains AUROC 0.953 (95% CI 0.942–0.964) while ETHOS attains 0.941 (95% CI 0.928–0.953). The ETHOS point estimate is lower, so the literal 'every case' claim fails; because Unknown race is 42.3% of the test set (Table S2), this is not a negligible subgroup. Figure 4's caption ('ETHOS consistently outperforms MEDS-Tab across all demographic subgroups and tasks') is also false for this row. The confidence intervals overlap, so the accurate statement would be that ETHOS is not significantly worse in this subgroup, which is materially weaker than 'higher AUC in every case.' This internal inconsistency is the most load-bearing issue because the abstract, Results, and Figure 4 all rest on universal superiority. A secondary concern is the Appendix A unbiasedness proof, which assumes exact sampling from p(x) while inference uses top-p nucleus sampling (p=0.9); however, the reported calibration curves partially address that concern, whereas no analysis in the paper addresses the Table S1 contradiction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ARES, an adaptive risk estimation system built on the previously published ETHOS generative transformer that operates on tokenized Patient Health Timelines (PHTs) from electronic health records. ARES estimates event probabilities by Monte Carlo sampling of future PHTs (N=100 per patient) and counting outcome tokens, producing dynamic risk scores for hospital mortality, ICU admission, prolonged stay, and a composite endpoint, alongside an explainability module. The authors evaluate on MIMIC-IV v2.2 with its ED extension, benchmark against traditional early warning scores and machine learning baselines including MEDS-Tab, and report AUC, calibration, and subgroup-stratified results. The paper also includes ED benchmark tasks (hospitalization at triage, critical outcome within 12 hours, and 72-hour ED re-presentation), discusses limitations, and releases source code.","tokens_in":23573,"tokens_out":4347,"duration_ms":36146,"significance":"If the central claims held, this would be a valuable demonstration that a single zero-shot generative model can produce dynamic, well-calibrated risk estimates for multiple clinical endpoints without task-specific retraining, with the additional benefit of personalized explainability. The paper's strengths include a large-scale tokenized corpus (over 360 million tokens), public code release, calibration reporting with Brier scores, and a candid discussion of limitations. The overall AUC results for the primary endpoints are strong and internally coherent for the unstratified comparisons. However, the paper's headline claim of universal superiority is contradicted by its own supplementary table in one non-negligible demographic subgroup, and the Monte Carlo justification does not fully cover the approximate sampling scheme used at inference. These issues are correctable but require a substantive revision of the paper's claims and presentation.","major_comments":[{"comment":"The text states that ARES 'consistently outperformed MEDS-Tab across both individual and composite endpoints, achieving higher AUC values in every case,' and Figure 4's caption says that 'ETHOS consistently outperforms MEDS-Tab across all demographic subgroups and tasks.' Table S1 contradicts this: for ICU Admission in the Unknown race subgroup, MEDS-Tab achieves AUROC 0.953 (95% CI 0.942–0.964) while ETHOS achieves 0.941 (95% CI 0.928–0.953). Because the Unknown race group constitutes 42.3% of the test set (Table S2), this is not a negligible subgroup. The confidence intervals overlap, so the accurate statement is that ETHOS is not significantly worse in this subgroup, which is materially weaker than 'higher AUC in every case.' The abstract, Results, and Figure 4 caption should be revised to report the direction and statistical significance of each subgroup comparison rather than claiming uniform superiority.","section":"§3, Table S1, Figure 4 caption"},{"comment":"Appendix A proves that the Monte Carlo ratio M/N is an unbiased estimator of Pr(A) under the assumption that the fPHTs are drawn exactly from the model distribution p(x). In practice, inference uses nucleus sampling with top-p = 0.9 (Appendix B) and only N = 100 samples, so the proof does not apply to the actual sampling distribution. The paper should either justify that the approximate sampling distribution is sufficiently close to p(x) for the ratio to be a valid risk estimate, or characterize the potential bias, particularly for rare events such as hospital mortality (1.85% prevalence). The calibration curves in Figure S4 provide empirical evidence but do not replace a formal justification for the central probability definition.","section":"§2.1 and Appendix A"}],"minor_comments":[{"comment":"The abstract and Section 3 report 285,622 PHTs, while Table S6 reports a total of 285,620 timelines; similarly, the full text states 299,721 unique patients while Table S2 reports 299,712. These numerical discrepancies should be reconciled.","section":"Abstract and Table S6"},{"comment":"Section 3 lists task prevalences as 1.85%, 15.44%, 9.01%, and 20.39%, while Table S1 lists the hospital mortality prevalence as 1.95% and the composite prevalence as 20.41%. Please correct the inconsistent numbers.","section":"§3 and Table S1"},{"comment":"Section 3 states that Brier scores were in the range 0.01–0.14, but Figure S4 shows a maximum Brier score of 0.094 (for hospitalization at triage). The stated range should be aligned with the figure.","section":"§3 and Figure S4"},{"comment":"The caption mentions a 'percentage of discarded ambiguous inference repetitions (0.2–0.3%)' without defining what makes a repetition ambiguous; please provide a concrete definition or remove the phrase.","section":"Figure S1 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's central quantitative results are strong, but the universal-superiority claim is contradicted by the authors' own Table S1, and this overstatement appears in the abstract, Results, and a figure caption. Before resubmission, the authors should also double-check all numerical consistencies between the main text and the supplement, as several discrepancies (patient counts, timeline counts, prevalences, Brier score ranges) suggest the supplementary tables were not fully proofread. These issues are fixable within the manuscript's scope, so I do not recommend rejection, but the revision must be substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate extension of the authors' earlier ETHOS work, and the core results are probably real, but the paper overclaims in a way that is directly checkable. The abstract, Results, and Figure 4 all say ETHOS/ARES achieves \"higher AUC in every case\" versus MEDS-Tab. Table S1 shows otherwise: for ICU admission in the Unknown race subgroup, MEDS-Tab gets AUROC 0.953 (0.942–0.964) and ETHOS gets 0.941 (0.928–0.953). Unknown race is 42.3% of the test set, so this is not a rounding error. The honest claim would be that ETHOS is not significantly worse there, which is materially weaker. I agree with the stress-test note: this is the biggest flaw and it is fixable.\n\nWhat is actually new: dynamic risk trajectories across a patient's stay, adaptive deactivation/reclassification of risk components (e.g., switching from >10-day to >15-day stay risk), and a personalized explainability module that attributes risk changes to specific tokens. The generative core was in the prior ETHOS paper, so novelty is moderate but real. The evaluation on MIMIC-IV-ED is thorough: many baselines, code released, calibration curves and Brier scores reported. The Monte Carlo risk estimation is transparent and the calibration looks genuinely good, especially for mortality.\n\nSoft spots beyond the overclaim: Appendix A proves unbiasedness only under exact sampling from p(x), but inference uses top-p nucleus sampling (p=0.9). The calibration curves partially mitigate this, but the proof as written does not cover the actual procedure—worth a caveat, not a fatal flaw. Also, the four core ARES endpoints are benchmarked against only MEDS-Tab for the main comparison; more ML baselines there would strengthen the paper. The prolonged-stay outcome is defined as \">90th percentile\" in the text but as \">10 days\" in Table S1 and Figure 1. These are probably the same thing but the paper should say so. The limitations section is candid about single-center data and lack of clinical validation, which makes the inflated superiority claim more frustrating.\n\nVerdict: this paper deserves serious peer review. The method is substantial, the code is out, and the dynamic risk framework is worth engaging with. But the authors need to correct the \"every case\" claim, clarify the outcome definition, and soften the sampling guarantee. I would not desk reject it—I would send it to review with the expectation of revision.\n\nWho gets value: researchers working on generative EHR models, early warning systems, and zero-shot clinical prediction. I would bring it to a reading group and would cite it if I worked in this area.","headline":"A useful dynamic risk extension of the authors' own ETHOS generative model, with strong AUCs on MIMIC-IV, but the universal superiority claim over MEDS-Tab is contradicted by their own Table S1.","tokens_in":24165,"tokens_out":1573,"would_cite":true,"duration_ms":16127,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single generative transformer, pretrained on tokenized electronic health records, can predict hospital mortality, ICU admission, prolonged stay, and composite risk with higher AUC than a strong tabular baseline…","keywords":["Generative AI","Zero-shot inference","Patient health timelines","Risk prediction","Electronic health records","Foundation model","Early warning systems","MIMIC-IV"],"falsifier":"Take the same test patients and re-estimate mortality risk with a much larger number of simulations (say 10,000 instead of 100) and recompute the calibration curve and AUC for hospital mortality; if the estimates or the reported 0.940 AUC change materially, the finite-sample Monte Carlo approximation is not reliable. A complementary check is to run ARES on a second hospital system's records without retraining and see whether the claimed subgroup-robust AUC advantages and calibration persist.","tokens_in":23059,"feed_emoji":"🏥","tokens_out":5481,"duration_ms":46302,"temperature":0.7,"pith_summary":"This paper claims that one generative foundation model over tokenized electronic health records can replace multiple task-specific early warning systems. The Adaptive Risk Estimation System (ARES) uses ETHOS, a transformer pretrained to predict the next clinical event in a patient's timeline, to simulate many plausible futures and count how often a target event—death, ICU admission, or a stay longer than ten days—occurs. On the MIMIC-IV dataset, ARES reports higher AUC than the MEDS-Tab baseline for every endpoint, including composite risks, and maintains that edge across demographic subgroups. The appeal is that a single model yields dynamic, personalized, explainable probabilities that update as new data arrive, without retraining for each task.","feed_headline":"One zero-shot model predicts death, ICU, and long stays","feed_subtitle":"ARES turns 100 simulated patient timelines into dynamic, explainable risk scores that beat tabular baselines on MIMIC-IV.","key_machinery":"The central machinery is the Patient Health Timeline (PHT), a chronological token sequence of every coded clinical event (diagnoses, medications, labs, procedures, vitals, demographics) with time-interval tokens between events. ETHOS is a decoder-only transformer trained with next-token cross-entropy on 360 million tokens drawn from 285,622 PHTs. At inference it repeatedly completes a patient's PHT into N=100 simulated futures using top-p nucleus sampling (p=0.9); the estimated probability of an event is M/N, the fraction of futures containing that event's token. This ratio is justified in Appendix A as an unbiased Monte Carlo estimator when sampling is exact, and the method's dynamic and explainable behavior comes from re-running simulations from any earlier token and inspecting which tokens change the counts.","core_discovery":"The discovery claimed is that an autoregressive transformer trained only to predict the next token of patient health timelines generalizes zero-shot to clinical risk estimation: given a patient's tokenized history, ETHOS generates N=100 future timelines (fPHTs) under nucleus sampling, and the fraction containing an outcome token is the risk probability. ARES applies this to hospital mortality (AUC 0.940), ICU admission (0.932), prolonged stay (0.853), and a composite outcome (0.906) at hospital admission, as well as to the emergency-department benchmarks of hospitalization at triage (0.946), critical outcome within 12 hours (0.945), and 72-hour ED re-presentation (0.745). The paper further claims these risks are well calibrated, robust across race and gender subgroups, and decomposable through an explainability module that highlights which events raise or lower risk.","pith_inferences":["Editorial inference: the 100-timeline Monte Carlo count has a standard deviation of sqrt(p(1-p)/100) around any true risk p, so for rare endpoints like hospital mortality (1.9% prevalence) the finite-sample wobble is comparable to the calibration error the paper reports; production use may need larger N.","Editorial inference: the reported advantage of ETHOS over MEDS-Tab on every demographic subgroup suggests generative pretraining captures patterns that tabular aggregation loses, but whether that advantage survives transfer to other hospital systems is untested and would require external validation.","Editorial inference: the explainability traces (e.g., intubation immediately raising ICU risk) may be more useful for retrospective understanding than for real-time alerts, since some procedures occur too late to guide intervention; prospective pilots would clarify this."],"forward_implications":["One zero-shot model can serve multiple endpoints at once, so hospitals would not need separate trained models for mortality, ICU admission, or length-of-stay prediction.","Risk estimates automatically update as new events appear, since each new token extends the patient's timeline before the next round of simulations.","Because all simulated futures are generated once, new endpoints can be scored by re-counting tokens in already generated timelines, without retraining.","If the reported AUC and calibration hold outside MIMIC-IV, ARES could be deployed as a dynamic early warning system whose five risk levels (0-20%, 20-40%, and so on) are actionable for clinicians."],"supporting_citations":[{"why":"Supplies the ETHOS model, its tokenization scheme, and the zero-shot generative inference that ARES is built on.","marker":"[15]"},{"why":"Provides the MIMIC-IV v2.2 and ED datasets used for training, testing, and all benchmark comparisons.","marker":"[25, 26]"},{"why":"Defines the three emergency-department benchmark tasks and the classical and ML baseline methods ARES is compared with.","marker":"[29]"},{"why":"The tabular baseline (MEDS-Tab) that ARES must beat on the four clinical-risk endpoints.","marker":"[37]"},{"why":"The conversion pipeline that turns raw EHR tables into the MEDS intermediate format from which PHTs are tokenized.","marker":"[27]"},{"why":"The MEDS data standard that makes the tokenization pipeline reproducible and compatible across institutions.","marker":"[28]"},{"why":"XGBoost is the model class used inside the MEDS-Tab baseline for each prediction task.","marker":"[38]"}],"fun_headline_variants":["Zero-shot AI predicts death, ICU, and long stays","One model, zero shots: dynamic risk scores for every patient","EHR foundation model yields explainable zero-shot risk estimates","Adaptive risk AI: zero-shot predictions with explanations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that 100 randomly generated future timelines from the model faithfully represent what actually happens to patients; if the generated futures are skewed—especially for rare events like in-hospital death—then the reported risk probabilities and their edge over other methods lose their meaning.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot AI predicts death, ICU, and long stays","One model, zero shots: dynamic risk scores for every patient","EHR foundation model yields explainable zero-shot risk estimates","Adaptive risk AI: zero-shot predictions with explanations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000957,"raw_usage":{"total_tokens":4120,"prompt_tokens":1030,"completion_tokens":3090,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":3023}},"tokens_in":646,"tokens_out":3090,"duration_ms":18666,"temperature":1.0,"reasoning_tokens":3023,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:41:47.951996+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same test patients and re-estimate mortality risk with a much larger number of simulations (say 10,000 instead of 100) and recompute the calibration curve and AUC for hospital mortality; if the estimates or the reported 0.940 AUC change materially, the finite-sample Monte Carlo approximation is not reliable. A complementary check is to run ARES on a second hospital system's records without retraining and see whether the claimed subgroup-robust AUC advantages and calibration persist.","supporting_citations":[{"cited_title":"MEDS-Tab: Automated tabularization and baseline methods for MEDS datasets","cited_arxiv_id":null,"evidence_quote":"The tabular baseline (MEDS-Tab) that ARES must beat on the four clinical-risk endpoints."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The conversion pipeline that turns raw EHR tables into the MEDS intermediate format from which PHTs are tokenized."},{"cited_title":"Medical Event Data Standard (MEDS): Facilitating Machine Learning for Health","cited_arxiv_id":null,"evidence_quote":"The MEDS data standard that makes the tokenization pipeline reproducible and compatible across institutions."}],"review_version":1}