{"id":"63cfc1a6-0e54-42f4-baf6-4bb89f2085a8","arxiv_id":"1908.04759","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A GRU-based survival model predicts sepsis onset up to 12 hours ahead with AUC 0.90/0.87 on internal and external cohorts, and an ATEE analysis ranks Sepsis-3 over CDC criteria for early antibiotic policies.","lead":"DeepAISE, a recurrent neural network survival model, predicts sepsis onset in ICU patients four hours ahead with AUC 0.90 on an internal cohort and 0.87 on an external cohort, while reporting interpretable risk factor scores. The paper also compares two sepsis definitions using counterfactual policy evaluation, estimating that antibiotics six hours before Sepsis-3 onset could improve survival by about 8%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The key unresolved risk is label-feature overlap: tsepsis-3 is defined from SOFA components and antibiotic/culture timestamps that also appear among the 65 model inputs, so without a strict left-censoring audit the reported AUC 0.90 may reflect leakage; the delta-SOFA control does not rule this out.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the sepsis-3 label is built from SOFA components and suspicion-of-infection timestamps that are also candidate model inputs, and the paper's control does not adequately rule out that the model is reproducing the label rather than predicting it early. I agree that this is the most important threat to the central claim, and I do not see evidence in the manuscript that the temporal boundary between input features and the future label was audited. The concern is not that feature-label overlap is automatically fatal: predicting sepsis from SOFA components is the intended task, and a model may legitimately learn that current abnormal components precede a future two-point increase. The problem is that the paper does not establish that features are left-censored at the prediction horizon, and the delta-SOFA control is too weak to do so. I considered alternative concerns, including the overstated 'external validation' on MIMIC-III (models were retrained rather than transferred) and the fragility of the ATEE causal estimates, but those are secondary to the label-contamination question because they affect generalizability and clinical effect size without invalidating the core prediction claim in the same way. Given that the paper is a solid applied contribution with a plausible architecture and substantial evaluation, the appropriate verdict remains conditional: the AUC and FAR claims should be accepted only after the left-censoring audit and the suggested re-run confirm that no future information enters the hourly feature windows.","tokens_in":17303,"tokens_out":7664,"duration_ms":88831,"concrete_test":"Rebuild the Emory testing-set feature matrices for every hourly prediction window with strict past-only semantics: each input feature at hour t is the last measurement timestamped at or before t, antibiotic and culture indicators remain 0 until their actual order time, and no SOFA component value from after t is carried backward. Retrain DeepAISE with the same architecture and hyperparameters on this truncated feature set and recompute the 4-hour ahead AUC on the same test patients. If the AUC remains at 0.90 within the DeLong confidence bounds, the leakage hypothesis is rejected; if it drops materially toward the delta-SOFA control level, the reported AUC is inflated by outcome contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DeepAISE predicts tsepsis-3 four hours ahead with AUC 0.90/0.87 and low false-alarm rates. But tsepsis-3 is constructed from a two-point SOFA increase within six hours and a suspicion-of-infection time derived from antibiotic and culture order timestamps, and the model's 65 input features are described only as 'commonly available in the EHR,' with details deferred to Appendix C. The paper never demonstrates that each hourly feature vector is strictly left-censored at the prediction time. If SOFA components, antibiotic/culture timestamps, or last-observation-carried-forward values from after the prediction boundary enter the input, the GRU can learn the label trajectory directly rather than performing early prediction. The reported delta-SOFA FFNN control (AUC 0.54) is not a sufficient guard: it inputs only a summary change score, not the full temporal sequence of SOFA components or the treatment timestamps that define suspicion of infection, so it cannot establish whether DeepAISE is exploiting the label's own building blocks. This concern is load-bearing because the headline accuracy, false-alarm rate, and the ATEE-based recommendation of a 6-hour-earlier antibiotic policy all depend on the predictions being genuinely early and uncontaminated by the outcome definition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DeepAISE, an end-to-end recurrent neural survival model for early prediction of sepsis onset in ICU patients. The model combines a two-layer GRU with a Weibull-Cox proportional hazards layer and is trained on hourly EHR data to predict tsepsis-3 (the Sepsis-3 onset time) at horizons of 2–12 hours. The authors report AUCs of 0.90 (Emory internal test) and 0.87 (MIMIC-III) at a fixed sensitivity of 0.85, with corresponding false-alarm rates of 0.20 and 0.26, outperforming logistic regression, Weibull-Cox, and feedforward baselines. They also introduce an ATEE (attributable treatment effect) framework to compare sepsis labeling criteria and claim that treating patients with antibiotics six hours before tsepsis-3 would yield an 8.2% [7.4, 9.9] improvement in survival. The paper additionally describes an interpretability mechanism (saliency-based relevance scores), a low-dimensional patient trajectory visualization, and a deployed tele-ICU decision-support workflow.","tokens_in":17655,"tokens_out":2746,"duration_ms":29003,"significance":"If the central claims survive scrutiny, this is a substantial contribution to clinical machine learning. Strengths include: (1) evaluation in a sequential-prediction design with real-world prevalence, rather than case-control enrichment, which is methodologically more realistic; (2) comparison against multiple baselines with DeLong significance testing; (3) a concrete deployment narrative including a web-based demo; (4) a concrete attempt at clinically meaningful interpretability via per-prediction feature relevance; and (5) a novel use of counterfactual treatment-effect estimation to compare sepsis labeling criteria. The paper also leverages a large internal cohort (25,820 admissions) and a public external dataset. However, the generalizability claim is weakened because the MIMIC-III model was retrained on MIMIC-III rather than evaluated as a true external transfer, and the headline accuracy depends on the input features having no temporal leakage with respect to the Sepsis-3 label definition. The ATEE-derived clinical recommendation also requires the intervention to be well-defined when the treatment time is itself part of the label definition.","major_comments":[{"comment":"The MIMIC-III result is not a true external validation. The manuscript states: 'The DeepAISE model was trained and evaluated on both the Emory cohort and the MIMIC-III external validation cohort separately.' Since the model was retrained on MIMIC, the MIMIC AUC of 0.87 demonstrates reproducibility across institutions, but not external validity in the usual sense of applying a fixed model to a new population. The abstract and Results claim 'external validation' is therefore overstated. Please either report the AUC of the Emory-trained model applied directly to MIMIC without retraining, or rephrase the claim as 'reproduced in a second cohort.'","section":"Materials and Methods, 'Study design'; Table 3"},{"comment":"The central early-prediction claim is vulnerable to label-feature overlap. tsepsis-3 is defined by a two-point SOFA increase within a 6-hour window plus tsuspicion, which is derived from antibiotic and blood-culture order timestamps. The model inputs are 65 EHR features whose full list is deferred to Appendix C, and the main text does not demonstrate that every hourly feature vector is strictly left-censored at the time of prediction. If SOFA components (PaO2, GCS, MAP, vasopressors, creatinine, bilirubin, platelets), antibiotic/culture timestamps, or last-observation-carried-forward values after the prediction boundary enter the input, a GRU can learn the Sepsis-3 rule itself. The delta-SOFA FFNN control (AUC 0.54) does not resolve this because it feeds only a summary score, not the full temporal feature sequence the GRU sees. Please provide a feature-by-feature temporal audit, explicitly listing each of the 65 features, how it is computed at prediction time, and how any future values are excluded; and add a control experiment that feeds the actual SOFA components and antibiotic/culture times to the same GRU-FFNN-WCPH architecture to show that the model outperforms this direct label-reconstruction oracle.","section":"Results, 'DeepAISE prediction performance for sepsis onset'; Appendix C; Table 1"},{"comment":"The ATEE counterfactual 'antibiotics six hours before tsepsis-3' is not well-defined because tsepsis-3 depends on tsuspicion, which is defined from antibiotic and culture order times. An intervention that administers antibiotics earlier will itself alter tsuspicion and therefore tsepsis-3, changing the label used to anchor the counterfactual. This creates a potential circularity in the causal estimand. The 8.2% [7.4, 9.9] improvement claim therefore requires either a sensitivity analysis under an alternative label definition that fixes the infection-suspicion time independently of treatment timing, or a clear causal model of how the intervention affects the label. At minimum, please discuss this endogeneity and present an analysis with tsepsis-CDC or a SOFA-only onset definition as a robustness check.","section":"Methods, 'Attributable treatment effect estimates (ATEE)'; Table 2"}],"minor_comments":[{"comment":"There is a typo: 'negative relavance score' should be 'negative relevance score'.","section":"Figure 4B caption"},{"comment":"The sentence after Eq. (1) reads 'on the right-hand-side of the question'; this should be 'of the equation'.","section":"Methods, Eq. (1)"},{"comment":"The definition of tsepsis-3 says 'The earlier of the tSOFA or tsuspicion was assigned to tsepsis,' but the proximity condition 'tSOFA + 24 hours > tsuspicion > tSOFA - 12 hours' is asymmetric and does not uniquely determine which is 'earlier' in all cases. Please clarify the exact assignment rule, including whether the earlier time point is used or whether tSOFA is the anchor.","section":"Table 1"},{"comment":"The manuscript reports AUC on the Emory year-based holdout (0.88) but does not state whether this holdout overlaps with the main Emory testing set. Please clarify the relationship between the year-based split and the 70/10/20 split.","section":"Results, 'DeepAISE prediction performance for sepsis onset'"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: DeepAISE is a genuine step forward for applied sepsis prediction. The GRU-Weibull-Cox architecture plus the ATEE comparison of Sepsis-3 vs. CDC labels is a real contribution, and the reported AUCs are strong. But the paper overstates external validation and leaves a load-bearing leakage question open.\n\nThe best parts: the architecture learns temporal representations end-to-end and the authors make a serious effort at interpretability, with saliency maps, local feature replacement, and a workflow UI. The delta-SOFA control (0.54 AUC) shows the model is not just a SOFA recapitulation, which is more than most sepsis ML papers do. The ATEE analysis is an interesting way to choose a label definition against a treatment policy.\n\nNow the soft spots. The MIMIC section is described as external validation, and the abstract says external cohort, but the Methods explicitly say the model was trained and evaluated on both the Emory cohort and the MIMIC-III cohort separately. So MIMIC is a second-site retrain, not a test of the same model. That matters when you claim generalizability.\n\nThe bigger issue is label-feature overlap. tsepsis-3 depends on a two-point SOFA increase plus antibiotic/culture timestamps. If the 65 features include SOFA components or those treatment timestamps, and if last-observation-carried-forward does not strictly censor future values, the GRU could be learning the Sepsis-3 rule itself. The paper does not show an audit of feature processing at each hourly boundary. The delta-SOFA FFNN control uses only summary change scores, so it does not settle this. The question is not whether the model is cheating in the usual sense; it is whether early prediction is genuinely early or just spotting label components already present. The authors may be right that the model is legitimate, but the control is not strong enough to prove it.\n\nOne more minor concern: the ATEE 8.2% improvement is a model-based counterfactual from observational data, not a trial result. Worth saying once, but not a flaw if labeled as an estimate.\n\nBottom line: these are fixable concerns, not fatal ones. The paper deserves serious peer review, but reviewers should push for the full feature list, exact temporal handling, and a same-model external test. If the authors can show strict left-censoring and a model that does not see its own label components, the AUCs and the ATEE are a real contribution. I would bring it to our reading group and would cite it, but with the second-site retrain caveat attached.","headline":"Solid applied sepsis-prediction paper with strong AUCs and a real ATEE contribution, but the MIMIC 'external validation' is a second-site retrain and the label-feature overlap needs a stricter audit before the effect sizes are taken at face value.","tokens_in":18185,"tokens_out":3600,"would_cite":true,"duration_ms":38643,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeepAISE predicts sepsis onset four hours ahead with AUC 0.90, and its counterfactual analysis suggests that giving antibiotics six hours earlier could improve survival by about 8%.","keywords":["sepsis prediction","recurrent neural network","survival analysis","Weibull-Cox proportional hazards","clinical decision support","counterfactual treatment effects","interpretable machine learning"],"falsifier":"Retrain DeepAISE on the same internal cohort after removing the exact variables used in the Sepsis-3 label (PaO2, GCS, MAP, vasopressors, creatinine, bilirubin, platelets, and antibiotic and culture timestamps) and after strictly lagging all inputs so that no future value enters an hourly window. If the four-hour AUC stays near 0.90, the early-prediction claim holds; if it falls substantially, the original score was largely recognizing the Sepsis-3 rule rather than anticipating it.","tokens_in":17128,"feed_emoji":"🩺","tokens_out":6786,"duration_ms":71058,"temperature":0.7,"pith_summary":"The paper sets out to show that sepsis onset in ICU patients can be predicted hours before it is clinically recognized, with enough accuracy and low enough false-alarm rates to be useful at the bedside, while still explaining each prediction in terms of the patient's own vital signs and labs. To do this, it trains a recurrent neural network on hourly clinical time series, feeds the learned trajectory representation into a survival model, and evaluates the system on an internal cohort and a separate external critical-care database. The stated result is an area under the ROC curve of 0.90 internally and 0.87 externally for predicting Sepsis-3 onset four hours ahead, with false-alarm rates of 0.20 and 0.26. The same counterfactual analysis is used to argue that administering antibiotics six hours before the Sepsis-3 onset time would have improved survival over the actual clinical policy by about 8.2%.","feed_headline":"Sepsis risk model hits 0.90 AUC four hours before onset","feed_subtitle":"Hourly risk scores with patient-specific explanations point to an 8% survival gain from earlier antibiotics.","key_machinery":"The model is a two-layer stacked Gated Recurrent Unit (GRU) followed by a fully connected layer whose output feeds a parametric Weibull-Cox proportional-hazards survival layer; the whole stack is trained end-to-end by minimizing the negative log-likelihood of time-to-sepsis with right-censored events. The Weibull-Cox layer defines the hazard as a baseline hazard times an exponentiated linear combination of learned trajectory features, allowing hourly survival probabilities and hence sepsis risk scores. Interpretability comes from relevance scores computed as the gradient of the risk score with respect to each input multiplied by the input value, then z-scored and thresholded to list the top contributing factors. For the label-selection question, the machinery is a generalized-propensity-score network plus an averaged dose-response network that estimates expected survival under counterfactual antibiotic timing intervals, with isotonic regression for calibration.","core_discovery":"The central claim is that a time-to-event model built on recurrent neural representations can learn the temporal patterns and higher-order interactions among routine ICU measurements that precede sepsis, and that these learned representations outperform logistic regression, a plain Weibull-Cox model, and a feedforward neural network at every prediction horizon tested. The paper further claims that the choice of sepsis label matters: using counterfactual policy evaluation, the Sepsis-3 definition supports a clinically meaningful early-antibiotic policy, with an expected 8.2% improvement in survival when antibiotics are given six hours before onset, while an alternative CDC-style label does not support such a policy. DeepAISE also produces a local explanation for each hourly score by computing input-gradient relevance scores, and the paper argues that these local attributions reflect the actual drivers of individual predictions better than a global list of important features.","pith_inferences":["A reader may infer that by monitoring which labs and vitals dominate local relevance scores, hospitals could schedule laboratory draws to refresh precisely those measurements, potentially improving model sensitivity without adding all measurements.","The paper's ATEE estimate assumes that the counterfactual antibiotics policy has no net harm from unnecessary treatment; because the false-alarm rate is around 20 to 26 percent, a reader should weigh possible antibiotic overuse and resistance before translating the 8.2% into expected population benefit.","The label-leakage concern is not fully settled by the delta-SOFA control: a more decisive test would retrain the model with all SOFA component variables and antibiotic and culture timestamps removed, and compare the AUC.","The same procedure of choosing a label definition by counterfactual policy value could be applied to other consensus-defined syndromes, such as acute kidney injury or acute respiratory distress syndrome, where onset-time definitions are similarly debated."],"forward_implications":["Hourly risk scores can be generated from routine EHR data starting four hours after ICU admission and remain discriminative out to a 12-hour horizon, so the method is deployable without high-resolution bedside monitors.","If the 8.2% attributable treatment effect estimate holds, a policy of acting on alerts six hours before Sepsis-3 onset could prevent a meaningful fraction of sepsis-related deaths, at the cost of treating patients whose alarms are false.","Because local relevance scores identify the physiology driving each alert, clinicians can use the explanation to decide whether an alert reflects infection, postoperative inflammation, or another deterioration.","The same GRU-survival architecture and label-selection procedure transfers to the external validation cohort with lower but still strong AUC, suggesting that the learned temporal features are not hospital-specific.","The separation between global and local feature importance implies that a one-size-fits-all explanation of a sepsis model is inadequate, so deployment tools should surface patient-specific contributors."],"supporting_citations":[{"why":"Defines the Sepsis-3 onset time, the two-point SOFA rise plus suspicion of infection, that the model is trained to predict.","marker":"[1]"},{"why":"Prior interpretable Weibull-Cox sepsis predictor with 0.85 AUC that DeepAISE extends with GRU-learned temporal features.","marker":"[16]"},{"why":"Supplies the attributable treatment effect framework used to compare sepsis labeling criteria under counterfactual antibiotic policies.","marker":"[27]"},{"why":"Provides the external validation cohort on which the model is retrained and tested.","marker":"[28]"},{"why":"Assesses the clinical criteria for Sepsis-3 and informs the threshold choices used to construct the sepsis onset labels.","marker":"[29]"},{"why":"Introduces the gated recurrent unit architecture that carries the temporal modeling of each patient's clinical trajectory.","marker":"[31]"},{"why":"Propensity score foundation for the generalized propensity score network used in the ATEE analysis.","marker":"[40]"},{"why":"Extends propensity score methods to multi-level treatments, supporting the counterfactual antibiotic timing comparisons.","marker":"[41]"},{"why":"Cox proportional hazards model, the parametric counterpart that the Weibull-Cox survival layer adapts.","marker":"[44]"}],"fun_headline_variants":["Sepsis AI hits 0.90 AUC, cuts false alarms","Recurrent survival model predicts sepsis hours early","Policy eval: Sepsis-3 label enables early antibiotics","DeepAISE: hourly sepsis risk with interpretable insights","Time-to-event model beats baselines on sepsis detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model's reported accuracy rests on the assumption that the SOFA components, antibiotic timestamps, and culture timestamps in its input stream do not already contain the information that defines the sepsis onset label; if future values leak into the prediction window, the AUCs overstate true early prediction.","fun_headline_variants_meta":{"raw":{"variants":["Sepsis AI hits 0.90 AUC, cuts false alarms","Recurrent survival model predicts sepsis hours early","Policy eval: Sepsis-3 label enables early antibiotics","DeepAISE: hourly sepsis risk with interpretable insights","Time-to-event model beats baselines on sepsis detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":3065,"prompt_tokens":1015,"completion_tokens":2050,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":1971}},"tokens_in":631,"tokens_out":2050,"duration_ms":16488,"temperature":1.0,"reasoning_tokens":1971,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:01:43.730996+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain DeepAISE on the same internal cohort after removing the exact variables used in the Sepsis-3 label (PaO2, GCS, MAP, vasopressors, creatinine, bilirubin, platelets, and antibiotic and culture timestamps) and after strictly lagging all inputs so that no future value enters an hourly window. If the four-hour AUC stays near 0.90, the early-prediction claim holds; if it falls substantially, the original score was largely recognizing the Sepsis-3 rule rather than anticipating it.","supporting_citations":[],"review_version":1}