{"id":"eb672b66-cca0-485b-95bb-b6ed7a043f7f","arxiv_id":"2501.00190","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SepsisCalc adds machine-estimated clinical calculator scores as confidence-gated nodes in temporal graphs, improving early sepsis prediction AUC over baselines on MIMIC-III, AmsterdamUMCdb, and OSUWMC.","lead":"SepsisCalc is a machine-learning system that predicts sepsis by turning electronic health records into dynamic graphs and adding estimated clinical calculator scores, such as SOFA, as explicit nodes. The paper reports that this improves early sepsis prediction accuracy across three hospital datasets and includes a deployed interface for clinicians.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Confidence-gating module is trained only on near-complete calculator observations and may not transfer to the high-missingness deployment regime; no calibration check is reported.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the confidence-gating module is trained in a nearly fully observed regime and assumed to transfer to high missingness. I agree with this diagnosis and believe it is the single most serious threat to the central claim, because the dynamic addition of accurately estimated calculators is the paper's main novelty and the source of the reported gains in Table 3. The paper contains no experiment that validates the gating decision under realistic missingness, and the training objective in Equation 10 makes the problem more acute than the reader stated: the supervision signal y_c is not merely rare, it is also binary exactness for integer calculators, so the confidence score is trained to predict a much stricter property than 'useful for downstream prediction.' I do not see a fatal internal inconsistency in the reported numbers, and the three-dataset evaluation with ablations is a real strength. The concern is addressable with a calibration and oracle-gating experiment, so the appropriate verdict remains CONDITIONAL rather than REJECT. Since the reader already reached CONDITIONAL on the same grounds, no verdict change is needed.","tokens_in":20318,"tokens_out":3879,"duration_ms":46231,"concrete_test":"On a held-out fold of MIMIC-III with complete calculator observations, run the trained SepsisCalc, then randomly mask component variables to match the per-variable 4-hour missing rates in Table 8. For each collection, record the true calculator estimation error |e - ehat|, the model's confidence p_c, and whether the calculator node enters the graph. Compute the ROC AUC of p_c for detecting 'error <= 1' (clinically acceptable for SOFA) and the expected calibration error (ECE) of p_c. If AUC < 0.7 or ECE > 0.2, the gating does not transfer. Additionally, replace the learned filter with an oracle that adds the node iff true error <= 1 and compare downstream sepsis AUC; a large gap between oracle and learned SepsisCalc shows the confidence module is the bottleneck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dynamic graph construction's central claim is that only 'accurately estimated' calculators are added, decided by the confidence score p_c in Equation 9. However, the supervision for this confidence module, Equation 10, uses M_{t,i}, which is 1 only when all calculator component variables are observed and the ground-truth calculator score is computable. Table 8 shows that for SOFA, this condition holds in only about 5-6% of collections across the three datasets. The confidence classifier is therefore trained almost entirely on the fully observed regime, while deployment is dominated by the 94-95% missingness regime where the estimated calculators are most needed. Because the input representation h_L_t in Equation 7 is computed from a temporal graph that lacks many variable nodes and edges in deployment, the feature distribution at the gating head is systematically different from training. If p_c is miscalibrated under informative missingness, the model either injects inaccurate calculator nodes into the dynamic graph or suppresses useful ones, degrading to SepsisCalc-d in the latter case. Moreover, the label in Equation 10 defines y_c = I[(e_c - ehat_c)^2 < 0.01]; for integer-valued calculators such as SOFA, this is equivalent to requiring exact integer match, so the learned confidence reflects exactness rather than clinically meaningful accuracy (e.g., SOFA error <= 1). The paper reports no calibration curve, no gating statistics under realistic missingness, and no comparison to an oracle gating rule, so the core mechanism's transferability is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes SepsisCalc, an early sepsis prediction model that converts EHRs into temporal heterogeneous graphs, estimates clinical calculators (SOFA, qSOFA, NEWS, and related organ-specific scores) from the observed graph, and dynamically adds only those estimated calculator nodes whose confidence score p_c exceeds 0.5 before a second round of temporal heterogeneous message passing. The model is trained with a weighted loss combining sepsis risk, organ dysfunction risk, calculator score estimation, and confidence gating. Across MIMIC-III, AmsterdamUMCdb, and OSUWMC, SepsisCalc reports AUC 0.839/0.848/0.918 versus best baseline AUCs 0.822/0.828/0.894, with ablations showing contributions from event interactions, dynamic calculator nodes, and the organ-dysfunction auxiliary task. The paper also describes a deployed Epic EHR interface for organ dysfunction and sepsis risk review.","tokens_in":20609,"tokens_out":6526,"duration_ms":65723,"significance":"The contribution is potentially valuable: injecting clinically validated calculators as explicit graph nodes is a plausible way to combine clinical workflow knowledge with learned representations, and the multi-dataset evaluation with ablations, code release, and mask-based calculator estimation experiments is a strength. If the confidence-gating mechanism transfers to real high-missingness settings, the reported gains are meaningful and the system is clinically relevant. However, the central transfer assumption is not yet validated, and the organ-dysfunction label definition is missing; these are load-bearing gaps that can be addressed with additional experiments.","major_comments":[{"comment":"Eq. (10) trains the confidence head only on collections with M_{t,i}=1, i.e., where the ground-truth calculator is computable because all component variables are observed. Table 8 shows that for SOFA this holds in only about 5–6% of collections on all three datasets, so the confidence classifier is trained almost entirely in the near-complete-observation regime, while deployment is dominated by the 94–95% missingness regime. The input representation h_L_t in Eq. (7) has systematically fewer variable nodes and edges when components are missing, so the feature distribution at the gating head differs between training and deployment. Since p_c determines which calculator nodes enter the dynamic graph, an unvalidated transfer assumption is load-bearing for the paper's central claim. Please report calibration curves for p_c on observed versus missing collections, the distribution and inclusion rate of calculator nodes under natural missingness, and a version of the gating head trained with masking augmentation or a missingness-robust representation.","section":"§3.4.2, Eq. (10), Table 8"},{"comment":"The mask-observation evaluation randomly masks 70% of calculator component variables. This is MCAR and does not reproduce the empirical missingness pattern in Table 8, which ranges from 23% (urine output) to 99% (Bands, C-reactive protein) and is likely informative because test ordering depends on acuity. The claim that SepsisCalc naturally handles missing values in deployment therefore rests on an untested equivalence between random masking and real missingness. Please evaluate calculator estimation and gating under an empirical mask distribution matched to per-variable missing rates, and, if possible, use naturally missing collections as a held-out test set.","section":"§5.2.2, Fig. 7, Table 8"},{"comment":"The binary confidence label y_c = I[(e_c - ehat_c)^2 < 0.01] is an exact-match criterion for integer-valued calculators such as SOFA: an estimated score that differs by 1 point is labeled inaccurate even though one-point SOFA error is clinically tolerable. The fixed inclusion threshold 0.5 in Eq. (9) is also presented without sensitivity analysis. Please add a sweep over the threshold and consider a tolerance-aware label (e.g., absolute error <= 1) so that 'accurately estimated' matches clinical accuracy rather than exact equality.","section":"Eq. (10), §3.4.2"},{"comment":"The organ dysfunction auxiliary task is part of the method and of the SepsisCalc-o ablation, but the paper never defines how the ground-truth organ dysfunction labels y_o,i_t are generated from the EHR data (e.g., which SOFA component threshold or charting source is used). Without this definition, Table 9 and the multi-task loss are not reproducible. Please state the label construction explicitly.","section":"§3.5, Eq. (12), Table 9"}],"minor_comments":[{"comment":"The '10-fold cross-validation' description says 7 sets are used for training, 1 for validation, and 2 for testing; this is a 70/10/20 split repeated 10 times rather than standard 10-fold CV. Please clarify the protocol and what the standard deviations in Table 2 represent.","section":"§4.3 / A.4.4"},{"comment":"The sepsis loss as printed is '-y_t log(p_t) - (1-y_t) log p_t'; the second term should be log(1-p_t).","section":"Eq. (12)"},{"comment":"Typo: 'Constrution' should be 'Construction'; also several affiliation and author strings misspell 'Northeastern' as 'Northestern'.","section":"§3.4.2 heading"},{"comment":"Algorithm 1 is described as returning h^L_t and h^{L,i}_t, but the pseudocode does not show how the organ node features are separated from the collection node features; a sentence or an explicit line in the pseudocode would improve reproducibility.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the stress-test concern in the reader's report lands. The main risk is the confidence-gating transfer from 5–6% observed collections to the 94–95% missingness regime; no calibration or gating analysis is currently reported. This is fixable through additional experiments, so I recommend major revision rather than rejection. I also flag that '10-fold cross-validation' appears to be a repeated 70/10/20 split; the authors should clarify before any resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution, not a repackaging. The idea of estimating clinical calculators and adding them as confidence-gated nodes in a temporal heterogeneous graph for sepsis prediction is new as far as I can tell, and the evaluation is more thorough than most ML-for-health preprints: three datasets, 10-fold CV, ablations that isolate the calculator nodes, the dynamic gating, the event-interaction edges, and the organ-dysfunction multitask head. The AUC gains over strong GNN/RNN baselines are modest (about 0.02) but consistent across all three datasets, and the organ-level outputs align with how clinicians actually reason about sepsis. I believe the headline result.\n\nThe soft spots are real but not fatal. The biggest is the one the stress-test flagged: the confidence module that decides which estimated calculators enter the graph is trained only on collections where all calculator component variables are observed—under 6% of SOFA collections. In deployment, the module sees graphs with most variables missing, so the feature distribution at the gating head is different from training. The paper shows the estimator itself works under 70% random masking, but it never shows the confidence scores are calibrated in that regime, nor compares against an oracle gating rule. If the gating is miscalibrated, the model either injects noisy calculator nodes or degenerates toward the ablation without dynamic construction. The fact that the full model beats SepsisCalc-d suggests the calculator features help even with imperfect gating, but the paper's stronger claim—that the confidence filter is what makes it work—is not yet backed by evidence.\n\nMinor issues: the 'one hour earlier' alert-time claim has no error bars; the OSUWMC data are proprietary; and the GitHub link doesn't point to a tagged release with a commit hash, so the experiments aren't reproducible as-is. The nRMSE table in the appendix is a nice check that the estimator learns the calculator arithmetic, though it would be stronger with error bars.\n\nOverall: this deserves serious peer review. The gap between training and deployment needs to be closed—a calibration curve under realistic missingness plus an oracle-gating comparison would do it—but the core architecture is sound and the clinical motivation is genuine. I'd send it to reviewers with a request to focus on the confidence-transfer question, and I'd cite it in related work.","headline":"Solid empirical paper with a genuinely new architecture, but the confidence-gating mechanism is trained in a different missingness regime than the one it operates in, so the core claim needs one more experiment.","tokens_in":21156,"tokens_out":2603,"would_cite":true,"duration_ms":26653,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SepsisCalc claims that dynamically estimating clinical calculators, adding only the confident estimates to a temporal graph, and jointly predicting organ dysfunction and sepsis risk improves early sepsis prediction over existing baselines.","keywords":["early sepsis prediction","clinical calculators","SOFA score","temporal heterogeneous graph","graph neural network","electronic health records","missing data","organ dysfunction prediction"],"falsifier":"Take a cohort where true SOFA can be recovered for many time points despite the usual high missingness, for example by chart review or by collecting the missing components on a random sample. Compare the squared error of SepsisCalc's estimated SOFA scores for the collections the confidence gate admits (confidence at least 0.5) with the error for collections it rejects. If the admitted estimates are not systematically more accurate, or if the confidence score does not rank errors at realistic missing rates, the gate that the ablation credits for the gains is not doing the work attributed to it and the reported improvement would not be expected to transfer to deployment.","tokens_in":1700,"feed_emoji":"🩺","tokens_out":1573,"duration_ms":153885,"temperature":0.7,"pith_summary":"The paper tries to establish that an early sepsis predictor should mirror the clinician's workflow rather than output one opaque risk score. SepsisCalc first builds a temporal heterogeneous graph of a patient's observed variables, then estimates the standard clinical calculators clinicians rely on (SOFA, qSOFA, SIRS, NEWS, MEWS, APACHE II), and dynamically adds only the estimates it is confident about as new graph nodes. From that augmented graph it predicts both the six organ-dysfunction risks and the overall sepsis risk. In experiments on two public ICU datasets and one proprietary hospital cohort, SepsisCalc reports AUCs of 0.839, 0.848 and 0.918 against best-baseline AUCs of 0.822, 0.828 and 0.894, and the ablations attribute part of the gain to the confidence-gated calculator nodes rather than to imputation. If the claim is right, sepsis warning systems could give clinicians organ-level evidence alongside the risk score, which is the information needed for timely intervention.","feed_headline":"Clinical calculators boost AI early sepsis prediction in ICU data","feed_subtitle":"SepsisCalc embeds organ-failure scores like SOFA as graph nodes and beats baseline sepsis predictors on three ICU cohorts.","key_machinery":"The load-bearing mechanism is the dynamic temporal heterogeneous graph with four node types: collection nodes, observed clinical variables, organ nodes, and calculator nodes; and three edge types: directed edges linking successive observations of the same variable (with the elapsed time as an edge attribute), undirected edges connecting variables observed at the same timestamp, and directed clinical-event-interaction edges (for example, vasopressor use linked to low blood pressure). The calculator estimator reads the collection-level embedding and produces both an estimated calculator score and a confidence value $p^c_t$, and the confidence gate discards estimates with confidence below 0.5 before adding a calculator node. New edges are generated between the added calculator and its component variables. A temporal heterogeneous message-passing network with multi-head attention propagates information through this graph, and the total loss combines sepsis prediction, organ-dysfunction prediction, calculator estimation, and confidence-gating objectives.","core_discovery":"SepsisCalc claims that a dynamic temporal heterogeneous graph can carry the information that matters for sepsis prediction while staying faithful to how sepsis is actually diagnosed. The model estimates clinical calculators from the current collection-level representation, assigns each estimate a confidence value, and inserts the calculator as a node only when the confidence is at least 0.5. It then re-reads the augmented graph with a temporal heterogeneous message-passing network and jointly outputs a sepsis risk score and six organ-dysfunction risk scores. The paper's experiments show SepsisCalc outperforming the strongest baselines by roughly 0.017 to 0.024 area under the ROC curve across the three ICU cohorts, and the ablations show that removing the dynamic calculator construction or replacing it with imputation lowers performance. The intended payoff is a model whose predictions are tied to the same organ-failure assessments clinicians already use.","pith_inferences":["Beyond the reported experiments, the central risk to the claim is deployment shift: the confidence gate is trained on the rare fully observed collections, and its behavior when the graph is fed mostly missing SOFA components is not measured; a direct calibration test under realistic missingness would settle whether the reported gains survive.","Read as a missing-data strategy, SepsisCalc is a task-aware alternative to imputation: it reconstructs the clinical summary clinicians would compute rather than the underlying lab values, which may be a more stable target when labs are missing not at random.","The same dynamic-node recipe extends naturally beyond sepsis calculators to other composite severity scores, since the estimator learns their computation rules from observed components and then generalizes to partial observations."],"forward_implications":["If the claim holds, an early-warning system can report a sepsis risk score and the six organ-dysfunction risks from the same model, so clinicians see which organ signals are driving the risk.","The graph representation removes the need to impute missing lab values: only observed variables and confidently estimated calculators enter the model, which should reduce imputation bias in high-missing-rate settings.","The confidence gate formalizes when a calculator like SOFA is trustworthy: when its components are too sparse, the model withholds the score instead of guessing it.","Because calculator definitions enter only through component variables and supervision, adding new calculators or organ scores would not require redesigning the architecture.","Deployed in an EHR system, the model's organ-specific outputs can be shown alongside the estimated SOFA score, letting clinicians prepare interventions for the organ that is failing."],"supporting_citations":[{"why":"Supplies the SOFA score, the central multi-organ clinical calculator that SepsisCalc estimates and dynamically adds to the graph.","marker":"[43]"},{"why":"Defines the Sepsis-3 criteria and suspected-infection screening used to label case and control patients, and supplies the qSOFA comparator.","marker":"[36]"},{"why":"Supplies the MIMIC-III public ICU dataset used to train and evaluate SepsisCalc.","marker":"[17]"},{"why":"Supplies the AmsterdamUMCdb public ICU dataset used to train and evaluate SepsisCalc.","marker":"[40]"},{"why":"Supplies the 3D-MICE imputation method that SepsisCalc is designed to outperform and that the ablation version SepsisCalc-imp uses.","marker":"[27]"},{"why":"Motivates the clinical-event-interaction edges (vasopressors, IV fluid, antibiotics, mechanical ventilation) in the temporal graph construction.","marker":"[22]"},{"why":"Supplies the DFSP baseline, the strongest deep-learning comparator in Table 2 that SepsisCalc is measured against.","marker":"[8]"}],"fun_headline_variants":["SepsisCalc embeds SOFA scores in graphs for better ICU prediction","Dynamic graph adds clinical calculators to sepsis AI","Clinical calculators as graph nodes boost sepsis prediction","Integrating organ-failure scores into sepsis prediction graphs"],"cache_read_input_tokens":23168,"weakest_assumption_plain":"The confidence classifier that decides which estimated calculators enter the graph is trained only on the rare collections where every calculator component is observed (fewer than 6% of SOFA collections), and the paper assumes this low-missingness training transfers to deployment, where those components are missing about 94% of the time.","fun_headline_variants_meta":{"raw":{"variants":["SepsisCalc embeds SOFA scores in graphs for better ICU prediction","Dynamic graph adds clinical calculators to sepsis AI","Clinical calculators as graph nodes boost sepsis prediction","Integrating organ-failure scores into sepsis prediction graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000459,"raw_usage":{"total_tokens":2320,"prompt_tokens":982,"completion_tokens":1338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1287}},"tokens_in":598,"tokens_out":1338,"duration_ms":10011,"temperature":1.0,"reasoning_tokens":1287,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:57:20.687493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a cohort where true SOFA can be recovered for many time points despite the usual high missingness, for example by chart review or by collecting the missing components on a random sample. Compare the squared error of SepsisCalc's estimated SOFA scores for the collections the confidence gate admits (confidence at least 0.5) with the error for collections it rejects. If the admitted estimates are not systematically more accurate, or if the confidence score does not rank errors at realistic missing rates, the gate that the ablation credits for the gains is not doing the work attributed to it and the reported improvement would not be expected to transfer to deployment.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SOFA score, the central multi-organ clinical calculator that SepsisCalc estimates and dynamically adds to the graph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Sepsis-3 criteria and suspected-infection screening used to label case and control patients, and supplies the qSOFA comparator."},{"cited_title":"Johnson, Tom J","cited_arxiv_id":null,"evidence_quote":"Supplies the MIMIC-III public ICU dataset used to train and evaluate SepsisCalc."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AmsterdamUMCdb public ICU dataset used to train and evaluate SepsisCalc."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 3D-MICE imputation method that SepsisCalc is designed to outperform and that the ablation version SepsisCalc-imp uses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the clinical-event-interaction edges (vasopressors, IV fluid, antibiotics, mechanical ventilation) in the temporal graph construction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DFSP baseline, the strongest deep-learning comparator in Table 2 that SepsisCalc is measured against."}],"review_version":1}