{"id":"fb9e69ad-184b-4590-80a4-e67feff0811a","arxiv_id":"2411.10754","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Using SHAP-based feature selection from six ML models plus Cox regression on MIMIC-IV, the paper reports modest C-index and Brier score gains over the KFRE-8 baseline for CKD progression, though selection leakage and temporal ambiguity weaken the claim.","lead":"This paper combines machine learning feature selection with Cox survival models to predict chronic kidney disease progression in intensive care patients. It reports small improvements over a standard clinical risk score, but the evaluation has methodological gaps that make the gains uncertain.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No index date is defined; predictors like 'creatinine last' can be measured after the CKD progression event, so the reported C-index/Brier gains are consistent with lookahead leakage.","rationale":"The paper's contribution is an empirical claim of improved risk stratification, supported by C-index and Brier-score comparisons. That claim is only valid if covariates are fixed at a baseline time before the outcome. Section 3.1 gives the outcome definition but no index date, and the feature set is described without any temporal restriction. The SHAP-selected features themselves contain 'last' and 'first' lab values, which in an ICU database like MIMIC-IV are naturally time-stamped relative to admission and are vulnerable to being measured after the progression event. Because the Cox model in Appendix F assumes time-invariant baseline covariates, any post-event measurement is a direct form of label leakage. The magnitude of the reported gains is small (C-index 0.8900 vs 0.8820; 5-year Brier 0.0801 vs 0.1120), and even a small leakage component could account for the entire difference. The reader's CONDITIONAL verdict was generous, treating the temporal issue as an addressable missing detail; however, as submitted, the results are uninterpretable and the central claim is not established. The authors do note the need for external validation in Section 5, but they do not acknowledge the index-date problem, which is more fundamental. Because the code is available, a targeted re-analysis can settle the issue, but the current manuscript does not support acceptance even conditionally without that re-analysis.","tokens_in":17998,"tokens_out":4771,"duration_ms":49625,"concrete_test":"Rerun the analysis with a defined index date: for patients whose CKD diagnosis appears during the index admission, use the admission date as baseline and require all labs/diagnoses to have charttime < admission time; for patients whose CKD ICD code is recorded after admission, use the date of that first code as baseline and truncate features to before it. Keep the same five-fold splits, but perform SHAP-based feature selection inside each training fold only. Compare the LR- and XGBoost-augmented Cox C-index and 5-year Brier score against the KFRE-8 baseline. If the average C-index gap drops below ~0.005 or the Brier gap reverses, the paper's central improvement is a temporal-leakage artifact.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim — that ML-derived predictors integrated into a Cox model improve CKD progression prediction — depends on all 1,373 features being measured before the progression event. Section 3.1 defines progression as 'diagnosed with a more advanced CKD stage at any time after an earlier diagnosis of a less severe stage,' but never specifies an index date or a feature-measurement window. The extracted features are simply 'demographic, diagnostic, and lab recording features' (Table 2), and the SHAP-selected sets include time-indexed variables such as Creatininelast, UreaNitrogenlast, Protein/CreatinineRatiolast, and CholesterolTotalfirst (Tables 9–14). In MIMIC-IV, lab events carry chart times; without truncation at baseline, 'last' can refer to a value drawn after the outcome, and 'first' can be a post-baseline value if the first CKD code is assigned late. The Cox model in Appendix F requires covariates measured at time zero; a model using future creatinine or dialysis status (Renal dialysis status HR 1.77–2.11, p<1e-8) will inflate C-index and lower Brier score even if it has no real predictive content. A secondary issue is that Section 3.2.2 describes a single 'final feature set' selected by SHAP, suggesting selection on the full dataset before the five-fold Cox evaluation, which would add optimistic bias even under correct temporal handling. Both issues are checkable because code is available, but as written the reported 0.008 C-index gain and 0.032 Brier-score gain are not interpretable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline in which binary machine-learning classifiers are trained on a CKD-progression outcome, SHAP values from these classifiers are used to select the top 40 features per model, the selected features are unioned with the KFRE-8 features, and Cox proportional hazards models are fitted on the resulting feature set to predict CKD stage progression in the MIMIC-IV cohort. The authors report that the augmented Cox models outperform a KFRE-8-only baseline, with the LR-augmented model reaching an average C-index of 0.8900 versus 0.8820 for baseline and the XGBoost-augmented model achieving a 5-year Brier score of 0.0801 versus 0.1120 for baseline.","tokens_in":18287,"tokens_out":3622,"duration_ms":36286,"significance":"The application area is clinically relevant, and the idea of combining ML-based feature screening with classical survival models is reasonable and builds on prior work. If the reported improvements were valid, the identified features would merit further investigation. The manuscript also ships code and uses a public dataset, which supports reproducibility. However, the current evaluation is compromised by temporal leakage and selection leakage, so the numerical claims are not supported as presented. The limitations section itself acknowledges the need for external validation and warns that features may proxy unmodeled processes, consistent with the concerns raised below.","major_comments":[{"comment":"No index date is defined for the survival analysis. Progression is defined as an advanced CKD stage diagnosed after an earlier less-severe stage, but no baseline time is specified at which all predictor features must have been measured. The selected feature sets include time-indexed variables such as Creatininelast, UreaNitrogenlast, Protein/CreatinineRatiolast, CholesterolTotalfirst, and Renal dialysis status. In a Cox model (Appendix F) covariates must be measured at time zero; if any of these values were recorded after the progression event, the model uses future information, which inflates the C-index and lowers the Brier score. The authors should define a baseline (e.g., the first CKD diagnosis) and truncate all measurements at that date, then re-run the analysis.","section":"§3.1, Tables 9–14"},{"comment":"Feature selection is performed on the full dataset before the Cox cross-validation. Section 3.2.2 describes a single 'final feature set' selected by SHAP, and Section 3.3 then applies five-fold cross-validation to the Cox model. This means the test-fold outcomes can influence which features are selected, giving optimistically biased performance estimates. The SHAP selection, the Bayesian hyperparameter optimization of the classifiers, and the choice of the Cox penalizer (0.0007) must be nested inside the cross-validation loop to obtain unbiased estimates.","section":"§3.2.2, §3.3"},{"comment":"The feature-selection classifiers are trained on the same binary progression outcome and the same cohort that is later used to evaluate the Cox models. This creates circularity: the selected predictors are chosen by models that already have access to the outcome, and the Cox improvement is therefore partly by construction. The authors should demonstrate added predictive value through a properly nested internal validation, or report external validation, which they already identify as necessary in Section 5.","section":"§3.2.1, §3.3"},{"comment":"The reported improvements are small and are presented without any measure of uncertainty or significance testing. The average C-index difference between the LR-augmented and baseline models is 0.008, and Table 6 reports standard deviations of 0.0092 and 0.0082, respectively; similarly, the best 5-year Brier score gain is 0.032. Without confidence intervals, bootstrap estimates, or paired significance tests, the observed differences cannot be distinguished from noise. The authors should provide such tests, especially because six augmented models are compared against one baseline.","section":"Tables 6, 7"}],"minor_comments":[{"comment":"The text contains a typo: 'descibed' should be 'described', and the MIMIC-IV size is written as '364,627 thousand medical records', which is unclear; likely '364,627' patients is intended.","section":"§3.1"},{"comment":"The word 'coaliton' should be 'coalition'.","section":"Appendix E"},{"comment":"The table headers read 'F eatures' with an extra space; this should be 'Features'.","section":"Tables 9–14"},{"comment":"Figure 13 is captioned 'FCNN-augmented Cox model' and Figure 14 'ResNet-augmented Cox model', but these duplicates appear after Figures 10 and 11 already show FCNN and ResNet. The captions or figure numbering appear to be mislabeled and should be corrected.","section":"Appendix J, Figures 13–14"},{"comment":"The median follow-up is 111.5 days, yet Brier scores are reported at 5 years. The authors should clarify how 5-year predictions are obtained from a cohort with such short observed follow-up and whether censoring is properly accounted for in the Brier score.","section":"§3.1, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The referee report largely agrees with the reader's and skeptic's assessments: the temporal and selection leakage concerns are central and are not addressed in the manuscript. The paper is a workshop-track submission; the scope of the required reanalysis (defining baseline, truncating features, nesting feature selection in CV, and adding significance tests) is substantial but feasible, so major_revision rather than reject is appropriate. The authors should be encouraged to also correct the figure/caption inconsistencies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a straightforward extension of Liu et al. (2023) to CKD using MIMIC-IV, with more ML feature selectors and the same SHAP-to-Cox pipeline. Second, the central numerical claim — that ML-augmented Cox models beat the KFRE-8 baseline — is not trustworthy as reported, because the pipeline selects features on the full data before cross-validating the Cox model, and no index date is defined, so 'last' lab values may be measured after the progression event.\n\nWhat the paper does well: it is clearly written, the method section actually explains the models and evaluation, the code is available, and the authors are honest that they are extending existing work rather than claiming a de novo method. The feature sets with hazard ratios and confidence intervals (appendix H) are a useful reference for renal markers. The SHAP results are consistent with known biology, which suggests the pipeline is working mechanically.\n\nThe soft spots are serious. Section 3.2.2 says the final feature set is the union of top-40 SHAP features from each classifier, with no mention of nested cross-validation. The Cox evaluation in 3.3 uses five-fold CV, but the features were already selected on all folds; that is selection leakage and it inflates C-index. More importantly, Section 3.1 defines progression only as a later diagnosis of a more severe stage, without an index date or a feature-measurement window. Time-indexed features like 'creatinine last' or 'cholesterol total first' are therefore ambiguous: in MIMIC-IV, 'last' can mean the last measurement in the record, which could be after the outcome. The Cox model in Appendix F requires covariates at time zero; the paper never establishes that these covariates are available at time zero. If a 'renal dialysis status' variable is recorded after the progression event, including it is like using the future to predict the present. The reported gains — +0.008 C-index, -0.032 Brier — are consistent with such lookahead. There are also no significance tests on C-index differences; the standard deviations overlap.\n\nI want to be fair: some of these are fixable. If the authors redo feature selection inside the CV loop and define a clear baseline (e.g., features measured within 90 days before first CKD diagnosis), the pipeline might produce a legitimate, if modest, result. The paper itself acknowledges the need for external validation in the limitations section.\n\nWho is it for? A reader working on CKD risk models and wanting to see SHAP-selected features in MIMIC-IV. It deserves a serious referee rather than a desk reject, because the methodology, while flawed in execution, is clearly presented and the code/cohort are checkable. But as it stands, the headline claim is not supported.\n\nRecommendation: send to peer review, likely major revision. If I were handling it, I'd ask for nested feature selection, an explicit index date, and maybe a temporal validation split.","headline":"A usable but unconvincing incremental application of the SHAP-plus-Cox recipe to CKD; the reported gains are likely inflated by selection leakage and an undefined index date.","tokens_in":18854,"tokens_out":3300,"would_cite":false,"duration_ms":46779,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding machine learning-selected predictors to the standard kidney failure risk equation improves prediction of chronic kidney disease progression, the paper reports.","keywords":["chronic kidney disease","CKD progression","Cox proportional hazards","Shapley values","feature selection","survival analysis","risk stratification"],"falsifier":"Recompute the five-fold evaluation with an explicit index date that freezes baseline labs and diagnoses before follow-up, and with SHAP feature ranking performed separately inside each training fold. If the logistic-regression-augmented Cox model no longer exceeds the baseline average C-index of 0.8820, or the XGBoost-augmented model no longer beats the baseline 5-year Brier score of 0.1120, the paper's central claim is falsified; an external primary-care CKD cohort with adjudicated progression dates would provide the same test.","tokens_in":17759,"feed_emoji":"🩺","tokens_out":9170,"duration_ms":81735,"temperature":0.7,"pith_summary":"The paper claims that chronic kidney disease (CKD) progression is predicted more accurately when machine learning-discovered features are added to a classical Cox proportional hazards survival model. Using a critical-care cohort of 14,012 CKD patients, the authors train six binary classifiers on 1,373 demographic, lab, and diagnostic features, rank features with approximate Shapley values, and union each model's top 40 features with the eight-variable Kidney Failure Risk Equation before fitting Cox models. Every augmented model beats the KFRE-8 baseline on concordance index, with the logistic-regression-augmented model reaching 0.8900 average versus 0.8820, and the XGBoost-augmented model reducing the 5-year Brier score from 0.1120 to 0.0801. If correct, this means routine labs and diagnostic codes carry additional prognostic signal that could sharpen early-intervention decisions in a disease that progresses silently.","feed_headline":"Machine learning plus survival models improves kidney risk scoring","feed_subtitle":"Adding ML-selected lab features to the standard kidney risk equation improves ranking and 5-year risk calibration.","key_machinery":"The carrying mechanism is a two-stage pipeline: a feature-selection stage and a survival-modeling stage. In the first stage, binary classifiers (logistic regression, decision tree, random forest, XGBoost, a fully connected network, and a residual network) are trained to predict CKD progression, and the 40 features with the highest mean absolute SHAP values are selected for each classifier. These features are unioned with the eight variables of the Kidney Failure Risk Equation (age, sex, estimated glomerular filtration rate, albumin-to-creatinine ratio, calcium, phosphorus, bicarbonate, and albumin). In the second stage, a Cox proportional hazards model is fit to the unioned feature set, and models are compared by concordance index and Brier score under five-fold cross-validation. SHAP, the Shapley-value approximation, is the object that converts model predictions into feature rankings; the Cox model converts selected features into time-dependent risk scores.","core_discovery":"The central claim is that integrating machine-learning-derived predictors with Cox proportional hazards models improves prediction of CKD stage progression. The paper identifies blood markers such as mean corpuscular hemoglobin concentration, mean corpuscular hemoglobin, neutrophil and platelet counts, potassium, and cardiac comorbidity codes as candidate novel predictors, alongside expected markers such as creatinine values, renal dialysis status, and urea nitrogen. The claim is supported by consistent C-index, Brier score, and dynamic AUROC comparisons showing that the augmented models outperform the KFRE-8 baseline, with the best C-index from the logistic-regression-augmented model (0.8900 average) and the best 5-year Brier score from the XGBoost-augmented model (0.0801).","pith_inferences":["Because the cohort comes from an emergency/critical-care setting and progression is defined by ICD-9 diagnosis codes, an external validation in an outpatient primary-care cohort with adjudicated progression dates is the natural next test.","The paper does not specify an index date separating baseline feature collection from follow-up, nor does it nest SHAP feature selection inside the cross-validation folds; re-running the analysis with selection performed separately in each training fold would reveal how much of the reported gain is real discrimination versus selection optimism.","A sharper benchmark than KFRE-8 alone would be a penalized Cox model trained on all 1,373 raw features; that comparison would isolate the benefit of SHAP-based feature selection from the benefit of having more covariates.","If the hematologic markers replicate externally, they may reflect nutritional or inflammatory pathways in CKD progression rather than direct kidney mechanisms, a hypothesis the paper does not test."],"forward_implications":["All six ML-augmented Cox models exceed the KFRE-8 baseline in average concordance index, so the gain is not tied to one particular classifier.","The best calibration improvement is at the 5-year horizon, where the XGBoost-augmented model's Brier score drops from 0.1120 to 0.0801, implying more trustworthy long-term risk estimates.","The SHAP rankings consistently place creatinine, dialysis status, and urea nitrogen at the top, which supports the clinical relevance of the standard KFRE variables.","The paper's discussion identifies hematologic markers and cardiac comorbidity codes as candidate additions to CKD progression models, suggesting systemic and cardiovascular pathways meriting further study."],"supporting_citations":[{"why":"Supplies the original ML-plus-Cox pipeline that this paper extends from breast cancer to CKD progression.","marker":"Liu et al., 2023"},{"why":"Origin of the Kidney Failure Risk Equation, whose eight variables form the clinical baseline feature set.","marker":"Tangri et al., 2016"},{"why":"External validation of the KFRE that establishes it as the standard comparator for kidney failure risk.","marker":"Major et al., 2019"},{"why":"Provides the SHAP approximation used to rank feature importance for feature selection.","marker":"Lundberg and Lee, 2017"},{"why":"Defines the proportional hazards model used for survival prediction and hazard-ratio estimation.","marker":"Cox, 1972"},{"why":"Defines the concordance index used as the primary discrimination metric.","marker":"Harrell et al., 1982"},{"why":"Defines the Brier score used to assess calibration of predicted survival probabilities.","marker":"Brier, 1950"},{"why":"Provides the critical-care database from which the CKD cohort of 14,012 patients is extracted.","marker":"Johnson et al., 2024"}],"fun_headline_variants":["ML plus Cox models outpredict standard kidney risk equation","New combo model sharpens chronic kidney disease prognosis","Machine learning flags hidden markers for kidney decline","Augmented survival models improve CKD risk stratification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that all 1,373 feature values were recorded before the patient's CKD progression event and that SHAP feature ranking did not use outcome information from the held-out folds; the paper does not define an index date or nest selection inside the cross-validation loop, so if either condition fails, the reported C-index and Brier improvements are optimistic.","fun_headline_variants_meta":{"raw":{"variants":["ML plus Cox models outpredict standard kidney risk equation","New combo model sharpens chronic kidney disease prognosis","Machine learning flags hidden markers for kidney decline","Augmented survival models improve CKD risk stratification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1409,"prompt_tokens":815,"completion_tokens":594,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":431,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":431,"tokens_out":594,"duration_ms":6987,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:20:31.724904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the five-fold evaluation with an explicit index date that freezes baseline labs and diagnoses before follow-up, and with SHAP feature ranking performed separately inside each training fold. If the logistic-regression-augmented Cox model no longer exceeds the baseline average C-index of 0.8820, or the XGBoost-augmented model no longer beats the baseline 5-year Brier score of 0.1120, the paper's central claim is falsified; an external primary-care CKD cohort with adjudicated progression dates would provide the same test.","supporting_citations":[{"cited_title":"Multinational assessment of accuracy of equations for predicting risk of kidney failure: a meta-analysis","cited_arxiv_id":null,"evidence_quote":"Origin of the Kidney Failure Risk Equation, whose eight variables form the clinical baseline feature set."}],"review_version":1}