{"id":"e594c207-de96-4b0c-9d21-440d04113822","arxiv_id":"2412.03737","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"The paper trains six ML models on MIMIC-III to predict AKI in septic patients and reports logistic regression with 23 features reaching AUC 0.887, but circular predictors and leaky preprocessing invalidate the claim as stated.","lead":"Septic ICU patients' risk of acute kidney injury is predicted with a 23-feature logistic regression model trained on MIMIC-III data, reporting an AUC of 0.887. The result matters only if the model can separate prediction from the creatinine and urine-output measurements that define the disease, which the paper does not establish.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.887 AUC is likely inflated by label leakage: urine output, creatinine, eGFR, and BUN from the first 24 hours are the same measurements used to define AKI, and the paper never states the AKI definition or outcome timing.","rationale":"The paper is well-structured and reports several evaluation metrics (AUC, accuracy, F1, Brier, calibration curves, Shapley), which is more than many clinical machine-learning papers provide. However, these strengths do not address the core identification problem. The abstract promises to 'predict AKI in septic patients,' but the manuscript never states how AKI was defined, when the outcome window begins, or whether patients with AKI at admission were excluded. The features with the largest univariate separation in Table 2 and the largest Shapley contributions in Section 3.3 are exactly the laboratory values and urine output used in KDIGO criteria. Because eGFR is computed from creatinine, including min/max creatinine and min eGFR double-counts the same renal signal. This makes the high AUC and Brier score uninterpretable as prospective predictive performance. The same issue also undermines the literature comparison in Section 4.1, since the comparator may use different outcome definitions and windows. The concrete test above would settle the question by isolating the contribution of outcome-defining features. If the AUC survives their removal, the model may still have value; if not, the central claim is not supported. The reader's weakest_assumption coincides with this concern, so agreement is 'agree' and the verdict remains REJECT.","tokens_in":11593,"tokens_out":4082,"duration_ms":38768,"concrete_test":"Re-run the cohort with an explicit AKI definition anchored to ICU admission, e.g., KDIGO: serum creatinine rise >= 0.3 mg/dL within 48 hours or >= 1.5x baseline within 7 days, and/or urine output < 0.5 mL/kg/h for 6 hours, excluding patients who already meet criteria in the first 6 hours. Retrain the logistic regression after removing urine output, min/max creatinine, min eGFR, and BUN from the feature set. If the test AUC drops from 0.887 to below about 0.80, or the 95% CI no longer excludes 0.75, the headline performance is largely attributable to outcome-defining features rather than independent prediction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim (Abstract, Sections 3.2 and 4.1): logistic regression predicts sepsis-associated AKI with AUC 0.887, outperforming five baselines and prior literature. For this to be a genuine prediction, the 23 features must be available before the outcome and must not encode the outcome definition. The paper violates this condition. Section 1 states that AKI diagnosis is 'based on increased creatinine levels and a decrease in urine output.' Section 2.3 extracts first-24-hour features that include urine output, minimum and maximum creatinine, minimum and maximum BUN, and minimum eGFR; Table 1 feeds these into the model. The AKI label itself is never defined: no KDIGO stage, no index time, no 48-hour or 7-day window, no baseline creatinine rule. Under any standard KDIGO definition, low urine output or a creatinine rise in the same first 24 hours already constitutes AKI. Thus min/max creatinine and urine output are not predictors of a future event; they are contemporaneous measurements of the event. eGFR is a direct transform of creatinine, and BUN is a renal-function marker. The strong feature importances reported in Section 3.3 (urine output, bilirubin, BUN) and the high AUC are therefore expected even if the model has no prospective predictive content. Secondary design choices (correlation-based feature selection on the full cohort in Section 2.4; SMOTE before the split in Section 2.6) worsen the optimism, but the outcome-definition overlap is the load-bearing flaw.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a logistic regression model to predict acute kidney injury (AKI) in septic ICU patients using MIMIC-III data, reporting a test-set AUC of 0.887 (95% CI [0.861–0.915]), accuracy 0.817, F1 0.866, and Brier score 0.13, which it claims outperforms five baseline machine learning models and the best existing literature while using only 23 features. The features are derived from the first 24 hours after ICU admission, selected by correlation with the AKI outcome, and the data imbalance is addressed with SMOTE. The central claim is that this model provides an effective, interpretable tool for early AKI prediction in septic patients.","tokens_in":11906,"tokens_out":4169,"duration_ms":40121,"significance":"If the reported results were valid, the contribution would be a modest but practically useful model: a small, interpretable feature set with a transparent logistic-regression baseline, calibration assessment via Brier scores, and comparison against several common ML algorithms. The authors deserve credit for including calibration curves, confidence intervals, and Shapley-based interpretability, which are good practices in clinical prediction modeling. However, because the AKI outcome is never defined and the modeling pipeline leaks outcome information into feature selection and the training process, the reported performance is not credible, and the clinical significance cannot be assessed without a full reanalysis.","major_comments":[{"comment":"The AKI outcome is never defined. The paper states in Section 1 that AKI is diagnosed 'based on increased creatinine levels and a decrease in urine output,' yet the predictor set includes urine output, minimum and maximum creatinine, minimum eGFR, and BUN from the first 24 hours. Under standard KDIGO criteria, these are the same measurements used to determine the outcome, so the model is partly predicting the outcome from its own definition. The manuscript must specify the AKI definition, the observation window, the baseline-creatinine rule, and demonstrate that predictors are measured before the outcome; otherwise, the headline AUC of 0.887 may be a tautology rather than a prospective prediction.","section":"Sections 1, 2.3, and Table 1"},{"comment":"SMOTE is applied before the train/test/validation split. The text says 'SMOTE was added to every feature that has been selected to balance the data' and then 'The data was then split into a training, testing, and validation cohort.' Generating synthetic minority-class examples from the entire dataset means the test set is not independent of the training data, so the reported test-set AUC and all comparison metrics in Tables 4 and 5 are optimistically biased. SMOTE must be applied only within the training fold after the data split.","section":"Section 2.6"},{"comment":"Correlation-based feature selection is performed on the full 3301-patient cohort using the AKI outcome before any data split. This leaks test-set information into the choice of the 23 features, so the test-set evaluation is not an unbiased estimate of generalizable performance. The feature-selection procedure should be conducted inside the training set only, and the selected features should then be frozen before evaluating on the held-out test set.","section":"Section 2.4"},{"comment":"The reported split proportions and sample sizes are internally inconsistent. The abstract and Section 3.1 state the data was divided into 40% training, 10% testing, and 50% validation, but the reported counts of 1980, 661, and 660 correspond to approximately 60%, 20%, and 20% of the 3301 patients. This error affects the interpretation of the headline test-set results, including the confidence intervals, and must be corrected and the analysis rerun with a properly sized held-out test set.","section":"Abstract and Section 3.1"}],"minor_comments":[{"comment":"The heading 'Data Souce' should be 'Data Source.'","section":"Section 2.1"},{"comment":"The p-value entry '¡0.0001' should be '<0.0001'.","section":"Table 2"},{"comment":"The text mentions 'minimum eGFR, maximum eGFR' among the top features, but Table 1 lists only 'Minimum Estimated Glomerular Filtration Rate (eGFR)'; please clarify which eGFR features were included.","section":"Section 2.4 and Table 1"},{"comment":"The abstract says '13 fewer variables' and an '8.57% improvement,' while Section 4.1 says '13 less variables' and an '8.56% improvement'; these numbers should be unified.","section":"Abstract and Section 4.1"},{"comment":"The Figure 5 caption describes 'predictors' in a way that reads like a feature list, but the figure appears to be a Shapley importance plot; please revise the caption to describe what is shown.","section":"Figure 5"},{"comment":"The paper claims compliance with TRIPOD guidelines, but it does not report essential TRIPOD items such as the outcome definition, how missing data were handled across imputations, or how hyperparameters were selected; either provide a completed TRIPOD checklist or temper the claim.","section":"Sections 1 and 5"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early draft with multiple unresolved internal inconsistencies (split sizes and percentages, feature lists, metric values) and, more importantly, a modeling pipeline that leaks outcome information through the undefined AKI label, pre-split SMOTE, and full-cohort feature selection. The central claim cannot be salvaged by local edits; the experiment must be redesigned with a clearly defined, temporally anchored outcome and a leakage-free validation protocol. The novelty beyond existing MIMIC-based AKI studies is also incremental, and the literature comparison relies on a single reference for the 'best existing model,' so the 8.57% improvement claim should be verified against a broader set of recent studies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the reported AUC of 0.887 should not be read as a prediction result. The model's most important features—minimum and maximum creatinine, minimum eGFR, BUN, and urine output—are the same measurements that define AKI. The paper never states the outcome definition, the index time, or the observation window, so the model is rediagnosing AKI from contemporaneous labs. The high AUC is the expected consequence, not evidence of predictive skill.\n\nWhat is new: the specific 23-feature logistic model and its SHAP ranking are not in the prior work they cite. The writing is clear, and the feature lists, hyperparameters, and cohort tables are transparent enough that the flaws are easy to identify. That transparency is a real virtue.\n\nThe problems are not minor. Section 1 states that AKI diagnosis is based on increased creatinine and decreased urine output; Section 2.3 extracts first-24-hour values of exactly those variables as predictors. Under standard KDIGO criteria, these are the diagnostic criteria themselves. The AKI label is never defined: no stage, no baseline rule, no time window. This is label leakage through feature construction. On top of that, SMOTE is applied before the data split, and feature selection with the outcome is done on the full cohort—both inflate the test metrics. The split itself is odd (40% train, 10% test, 50% validation), with the smallest set carrying the headline result. The cross-study AUC comparison with 8.57% improvement is invalid because outcome definitions and cohorts differ, and no code or external validation is provided.\n\nWho this is for: someone studying how leakage can corrupt clinical prediction models. It's a useful example for a methods seminar. A corrected version with a defined outcome, a prospective feature window, and leakage-free preprocessing could be a legitimate contribution. As it stands, I wouldn't cite it, but I'd still send it to peer review—the flaw is exactly the kind of thing reviewers should catch and document. The central claim fails, but the paper is worth a serious referee's time to say so.","headline":"The 0.887 AUC is not a prediction—the model's top features are the same measurements used to define AKI.","tokens_in":12485,"tokens_out":4039,"would_cite":false,"duration_ms":38586,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A logistic regression model with 23 features predicts acute kidney injury in septic patients with an AUC of 0.887 and outperforms five machine-learning baselines.","keywords":["sepsis","acute kidney injury","AKI prediction","logistic regression","MIMIC-III","machine learning","Shapley analysis","SMOTE"],"falsifier":"Look at the cohort-extraction code: if the AKI outcome is defined by the standard KDIGO criteria using creatinine and urine output over the same first 24 hours that supply the predictor values, then retrain the model with the outcome drawn from measurements taken 48 to 72 hours later; if the AUC falls materially below 0.887, the reported performance was inflated by label leakage.","tokens_in":11371,"feed_emoji":"🩺","tokens_out":5539,"duration_ms":54085,"temperature":0.7,"pith_summary":"The paper tries to establish that a simple logistic regression, trained on 23 clinical features from the first 24 hours of an ICU stay, can predict acute kidney injury (AKI) in septic patients as well as or better than heavier machine-learning models. If true, this matters because a small, interpretable feature set is easier for clinicians to act on than a black box with dozens of variables. The authors report an area under the curve (AUC) of 0.887 with a 95% confidence interval of [0.861, 0.915], accuracy of 0.817, F1 score of 0.866, recall of 0.827, and a Brier score of 0.13 on a held-out test set, beating five baselines and improving on the best prior model in the literature by 8.57% while using 13 fewer variables.","feed_headline":"Logistic regression predicts sepsis-related AKI with AUC 0.887","feed_subtitle":"A 23-feature model beats five machine-learning baselines and the best published sepsis-AKI model.","key_machinery":"The machine is a logistic regression classifier fed by 23 features whose absolute Pearson correlation with the AKI label falls between 0.1 and 1; Synthetic Minority Over-Sampling Technique (SMOTE) balances the minority non-AKI class; and Shapley values attribute each feature's contribution to the prediction. Logistic regression carries the argument because it outputs directly interpretable, calibrated probabilities, and the paper uses its low Brier score and calibration curve to claim that the risk estimates are reliable enough for clinical use.","core_discovery":"The central claim is that a logistic regression model with 23 features, selected by correlation magnitude from an initial set of 50, outperforms XGBoost, K-Nearest Neighbors, Support Vector Machines, Random Forest, and LightGBM for predicting AKI in septic patients, and also beats the best published model. The authors report test-set AUC 0.887 (95% CI [0.861, 0.915]), accuracy 0.817, F1 0.866, recall 0.827, and a Brier score of 0.13, with the top Shapley drivers being urine output, maximum and minimum bilirubin, weight, maximum blood urea nitrogen, and minimum estimated glomerular filtration rate. They interpret the result as evidence that a parsimonious, interpretable model can match or exceed complex algorithms for this clinical prediction task.","pith_inferences":["Editorial inference: the paper never defines how AKI was labeled or when the outcome window ends, and several predictor features (urine output, creatinine, blood urea nitrogen, and estimated glomerular filtration rate) are the same measurements normally used to define AKI; the headline AUC may therefore be inflated by label leakage.","Editorial inference: a fair clinical test would retrain the model without renal-function features and check whether the remaining 16 features still predict AKI; if they do, the model has genuine predictive content beyond restating the diagnostic criteria.","Editorial inference: the model was developed on a single hospital system's data, so its transferability is untested; applying the same 23-feature logistic regression to MIMIC-IV or another critical-care database would show whether the reported performance gap persists."],"forward_implications":["If the AUC of 0.887 generalizes, clinicians in the ICU could screen septic patients with 23 readily available measurements and flag high-risk patients for earlier intervention.","The small feature set lowers data-collection burden and overfitting risk, which matters when electronic health records are sparse or missing.","The result implies that complex gradient-boosting or kernel-based models may not be necessary for this task; a calibrated linear model can suffice when paired with well-chosen features.","If the comparison with the best existing literature holds, the field gains a simpler reference model that future AKI prediction studies need to beat."],"supporting_citations":[{"why":"Supplies the best existing literature model (XGBoost with 36 variables) that the paper compares against for its 8.57% AUC improvement claim.","marker":"[5]"},{"why":"The MIMIC-III database is the sole source of patient data and all 23 predictor features.","marker":"[12]"},{"why":"Provides the SMOTE oversampling method used to balance the AKI and non-AKI classes before training.","marker":"[21]"},{"why":"Supplies the Shapley analysis method used to identify the most influential features after model training.","marker":"[31]"},{"why":"Supports the clinical relevance of urine output as a key indicator of acute kidney injury.","marker":"[32]"},{"why":"Supports the claim that eGFR decreases after AKI, used to justify including minimum eGFR as a predictor.","marker":"[14]"}],"fun_headline_variants":["Logistic regression tops 5 ML models for sepsis AKI","AUC 0.887: Simple model wins sepsis kidney injury race","23 features beat XGBoost, SVM, and more for AKI in sepsis","Interpretable model outshines complex ones for sepsis AKI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 23 features are known before AKI occurs and do not encode the diagnosis itself; if AKI was labeled from the same creatinine, urine output, blood urea nitrogen, and eGFR values that serve as predictors, then the high AUC mostly restates the label rather than predicting it.","fun_headline_variants_meta":{"raw":{"variants":["Logistic regression tops 5 ML models for sepsis AKI","AUC 0.887: Simple model wins sepsis kidney injury race","23 features beat XGBoost, SVM, and more for AKI in sepsis","Interpretable model outshines complex ones for sepsis AKI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000437,"raw_usage":{"total_tokens":2313,"prompt_tokens":1127,"completion_tokens":1186,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":743,"completion_tokens_details":{"reasoning_tokens":1120}},"tokens_in":743,"tokens_out":1186,"duration_ms":9367,"temperature":1.0,"reasoning_tokens":1120,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:07:55.772370+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look at the cohort-extraction code: if the AKI outcome is defined by the standard KDIGO criteria using creatinine and urine output over the same first 24 hours that supply the predictor values, then retrain the model with the outcome drawn from measurements taken 48 to 72 hours later; if the AUC falls materially below 0.887, the reported performance was inflated by label leakage.","supporting_citations":[{"cited_title":"Effect of a Process Mining based Pre-processing Step in Prediction of the Critical Health Outcomes","cited_arxiv_id":"2407.02821","evidence_quote":"Provides the SMOTE oversampling method used to balance the AKI and non-AKI classes before training."}],"review_version":1}