{"id":"d3e08d0a-61cd-4b40-9519-503bc4fba16b","arxiv_id":"2411.18759","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":10,"one_line_summary":"The paper reports near-perfect Random Forest and SVM classification of COVID-19 death, but the result is invalidated by data leakage, selection bias, and inconsistent validation.","lead":"This class-project report applies Random Forest and SVM to COVID-19 EHR data and reports near-perfect accuracy for classifying patients who died. The high accuracy is not credible because the model uses diagnosis codes that encode the outcome and selects features on the full dataset before validation.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The near-perfect accuracy is most plausibly a target-leakage artifact: Condition-table diagnosis codes such as severe sepsis, cardiac arrest, and acute kidney failure are drawn from all encounters since 2015 and can be recorded during the hospitalization being predicted.","rationale":"The reader's weakest assumption identifies the same mechanism: Condition-table diagnosis codes are not admission-time predictors. My stress-test confirms this is the single most load-bearing concern because the central claim of near-perfect classification collapses if those codes encode the outcome. The full text of Table 4 shows that five of the top ten features are binary diagnoses that are typically documented during a terminal hospital course, and the data description explicitly says the Condition table includes all diagnosis records from encounters on or after 1/1/2015. No temporal restriction or present-on-admission indicator is described. A model using such features can achieve 100% accuracy by memorizing the outcome, so the accuracy numbers cannot support the paper's central claim. The proposed test directly removes the suspected leaked features; if the accuracy remains high, the concern is falsified, but given the paper's own admission of overfitting and the lack of external validation, the rejection verdict stands. No additional concerns are needed to justify the verdict, and the reader and I agree on the weakest assumption.","tokens_in":19149,"tokens_out":1703,"duration_ms":19065,"concrete_test":"Re-run the RF and SVM pipelines on the same 9,366-patient cohort after removing all Condition-table diagnosis-code features, keeping only demographics and first-result lab values; better, restrict any retained diagnosis codes to those with an explicit present-on-admission flag or a diagnosis date before admission. If the 10-fold cross-validated accuracy drops materially below ~99.8% (for example, AUC below 0.9), the reported near-perfect results are attributable to outcome leakage. If accuracy remains ~100% without diagnosis codes, the leakage concern would not explain the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that demographics, first lab values, and preexisting conditions separate deceased from non-deceased COVID-19 patients with ~99.8-100% accuracy. The load-bearing assumption is that all predictors are available at admission, i.e., they do not encode the outcome. That assumption fails for the Condition table. The paper states it includes 'all available diagnosis records from encounters with service dates on or after 1/1/2015' (Dataset Explanation, Table 4), with no present-on-admission flag or diagnosis-date restriction. The most important binary features in Table 4 — 'Acute kidney failure, unspecified', 'Severe sepsis with septic shock', 'Cardiac arrest cause unspecified', 'Acute kidney failure with tubular necrosis', 'Acute respiratory distress syndrome', and 'Metabolic encephalopathy' — are exactly the diagnoses that occur during a fatal hospital course. Because the outcome is death during the stay, a patient who died is far more likely to have these codes recorded, so the model is partly predicting death from the death itself. This alone can explain perfect separation without any genuine predictive signal, and it is independent of the additional problems of feature selection before cross-validation and full-data missing-value imputation. The authors' own Discussion concedes overfitting concerns, but the leakage concern is more fundamental: the classifier's inputs are not causally prior to the label.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains Random Forest and Support Vector Machine classifiers on a Cerner COVID-19 dataset of 9,366 balanced patients (4,683 deceased, 4,683 non-deceased) to predict in-hospital death. Features include demographics, first laboratory values, and binary indicators of conditions from the Condition table. After ExtraTreeClassifier-based feature selection and missing-value imputation, the authors report near-perfect separation: 99.78% accuracy for SVM on a 25% split and 100% accuracy for RF under 10-fold CV, with AUC=1.00. They also perform hierarchical clustering and DBSCAN. The central claim is that admission-time features (demographics, first labs, preexisting conditions) can almost perfectly classify deceased versus non-deceased COVID-19 patients.","tokens_in":19436,"tokens_out":4757,"duration_ms":38828,"significance":"If the central claim were credible, this would be a striking result. The paper has some strengths: it reports hyperparameters and pseudocode, describes the data extraction pipeline, and the authors openly acknowledge possible overfitting in the Discussion. However, the claim is not supported. The Condition table includes diagnosis records from any service date after 1/1/2015, so the binary predictors—severe sepsis with septic shock, cardiac arrest, acute kidney failure—can be recorded during the hospitalization and are partly determined by the outcome. Feature selection on the full dataset and full-data imputation compound the bias, and the SVM result is not from 10-fold CV. Because the leakage is in the data construction itself, the reported accuracies cannot be interpreted as evidence of generalizable predictive performance.","major_comments":[{"comment":"The Condition table is the paper's core predictor source for comorbidities, yet the paper's own description (Table 4) says it includes 'all available diagnosis records from encounters with service dates on or after 1/1/2015,' with no present-on-admission flag and no restriction to diagnoses recorded before admission. The top binary features in the feature-importance list (Table 4: 'Acute kidney failure, unspecified,' 'Severe sepsis with septic shock,' 'Cardiac arrest cause unspecified,' 'Acute respiratory distress syndrome,' 'Metabolic encephalopathy') are conditions that typically arise during a fatal hospital course. Because the outcome is death during the stay, these codes are partially determined by the outcome, so the near-perfect accuracy in Tables 8 and 9 is most plausibly a target-leakage artifact rather than evidence of genuine predictive signal. This issue is load-bearing: the paper's central claim is that admission-time features can separate survivors from decedents at ~100% accuracy, but the features are not constrained to be admission-time.","section":"Dataset Explanation and Analysis, Table 4"},{"comment":"The ExtraTreeClassifier feature selection is performed on the full 9,366-patient dataset before any train/test split. Using the entire labeled dataset to choose the top 20 features means test-fold information enters the feature set, and the subsequent 10-fold CV accuracies in Table 9 and the single-split result in Table 8 are optimistically biased. Feature selection must be nested inside each CV fold (or at least performed on the training folds only) to yield unbiased estimates.","section":"Algorithms and Hyperparameters (feature selection)"},{"comment":"The SVM confusion matrix in Table 8 is computed on a single 25% test split, while the abstract and Section 1 (step 8) claim 10-fold cross-validation for SVM. The text explicitly says only the RF classifier used 10-fold CV. No per-fold accuracies, standard deviations, or confidence intervals are reported for either classifier, and the ROC in Figure 7 is only the RF curve with AUC=1.00. Without repeated CV or a proper held-out set, the 99.78% and 100% accuracy figures are not statistically grounded.","section":"Experimental Results, Tables 8 and 9"},{"comment":"Missing lab values are imputed by sampling from a normal distribution of the z-score of non-missing values, and this imputation is performed before splitting into training and test folds. The imputation uses the full-data distribution, so the test rows influence the imputed values used in training, another source of leakage that inflates the reported accuracy. Imputation should be fit on training folds only and applied to test folds.","section":"Dataset Explanation and Analysis, step 5 (Algorithm 1)"}],"minor_comments":[{"comment":"There are two figures labelled 'Figure 3' (dendrogram and feature-importance bar chart) and the text refers to 'Tables 99 & 99' in the appendix, which do not exist; renumbering and cross-references are needed.","section":"Throughout"},{"comment":"The abstract claims a 10-fold validation procedure was used for both classifiers, but the SVM confusion matrix in Table 8 is explicitly for 25% of samples; the abstract and the methods section should agree.","section":"Abstract and Table 8"},{"comment":"Figure 2 is described as showing the 14 most important numeric features, while Table 4 lists 20 features and the text discusses 20; the mismatch should be reconciled.","section":"Figure 2"},{"comment":"The hyperparameter table includes a 'Random Forest Regressor' block although the task is classification; this is confusing and should be removed or clearly separated from the classifier settings.","section":"Hyperparameters table"},{"comment":"Most references are web tutorials and blog posts (e.g., Ref. 1 is an outlier-detection tutorial, Ref. 6 is a Medium post on cross-validation) rather than peer-reviewed sources; the paper would benefit from citing the primary machine learning and medical literature.","section":"References"},{"comment":"There are numerous typos and repeated words, e.g., 'that that that' in the ROC paragraph and 'Deseased' in the Figure 1 caption; a careful proofreading pass is needed.","section":"Language and typos"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a course project report (CS-521) rather than a research article. The data-sharing agreement prevents release of the data, and the reference list relies heavily on blog posts and tutorials. The scientific contribution as presented is not sufficient for publication: the target-leakage problem is fundamental and would require re-access to the data with temporal restrictions on the diagnosis codes and a properly nested validation design. If the authors are willing to redo the analysis, the project could be reshaped into a credible methodological case study, but the current version should be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a course project report, and the near-perfect accuracy it reports is best explained by target leakage, not by a real predictive signal. The Condition table pulls every diagnosis record with a service date on or after 1/1/2015, with no present-on-admission flag. The top binary features—severe sepsis with septic shock, cardiac arrest, acute kidney failure, ARDS, metabolic encephalopathy—are precisely the codes a dying patient accumulates during the hospitalization. The model is partly predicting death from the death itself.\n\nGive credit where it's due: the authors are transparent about their pipeline and hyperparameters, they state plainly that the findings are not novel and that clinicians already knew these predictors, and they openly worry about overfitting. The feature ranking (oxygen saturation, erythrocytes, INR, kidney failure) is clinically plausible. As a student exercise in applying sklearn-style tools to a real EHR, it is honest and readable.\n\nThat does not rescue the analysis. Beyond leakage, feature selection using ExtraTree is run on the full dataset before cross-validation, which lets test information leak into the chosen features. The SVM result comes from a single 25% held-out split, despite the abstract claiming 10-fold CV. No error bars or confidence intervals are reported. The missing-value imputation by sampling from a normal distribution of z-scored non-missing values is crude and, combined with full-data imputation, further contaminates the evaluation. The data cannot be released and no code is provided, so nothing is independently checkable. The authors essentially concede the paper's limitations in the Discussion.\n\nThis is not a paper with a load-bearing flaw that a clever revision could fix; the core comparison is set up so that perfect separation is expected. I would not spend referee time on it. A desk reject is appropriate. The right response to the authors is to encourage them to treat this as a learning exercise and, if they continue, to require present-on-admission timestamps, a temporally honest feature-selection procedure, and external validation or a well-defined prospective protocol.","headline":"This is a course project report whose near-perfect accuracy is best explained by target leakage from outcome-informative diagnosis codes, not by a real predictive signal.","tokens_in":19942,"tokens_out":2519,"would_cite":false,"duration_ms":21760,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T05","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that Random Forest and Support Vector Machine classifiers, using demographics, first lab values, and diagnosis codes, separate deceased from non-deceased COVID-19 patients with near-perfect accuracy (99.78% and 100%).","keywords":["COVID-19","mortality prediction","random forest","support vector machine","sepsis","clinical data mining","feature importance","confusion matrix"],"falsifier":"Re-run the classifiers using only diagnosis codes present on or before the admission encounter (or within 24 hours of admission), leaving all other features unchanged; if accuracy drops substantially from the reported 99.78% and 100%, the original result was driven by outcome leakage.","tokens_in":18935,"feed_emoji":"📊","tokens_out":8548,"duration_ms":67862,"temperature":0.7,"pith_summary":"This paper attempts to show that machine-learning classifiers can tell, from data available early in a hospital stay, which COVID-19 patients will die and which will survive. Using a balanced cohort of 9,366 patients from a national COVID-19 database, the authors train a Support Vector Machine and a Random Forest on demographics, first laboratory values, and diagnosis codes. They report near-perfect accuracy: 99.78% for SVM and 100% for Random Forest in 10-fold cross-validation, with 100% sensitivity for both. If correct, this would imply that death risk in COVID-19 is almost completely separable in the features physicians already collect. The authors themselves note that the classes appear so easily separable that overfitting is a concern.","feed_headline":"RF and SVM separate COVID-19 deaths from survivors at ~100%","feed_subtitle":"Balanced cohort of 9,366 patients; top predictors include oxygen saturation and kidney failure.","key_machinery":"The argument rests on two standard classifiers: a Random Forest of 2,000 trees with depth up to 50 using Gini-index splits, and a Support Vector Machine with an RBF kernel and regularization C=0.01. Feature selection is done with an ExtraTreeClassifier that ranks the original 227 attributes by Gini importance, reducing to the top 20. Missing lab values are imputed by sampling from a normal distribution fitted to the non-missing z-scores, and all numeric features are min-max scaled to [0,1]. Performance is assessed with 10-fold cross-validation, confusion matrices, and ROC curves.","core_discovery":"The central claim is that a Support Vector Machine and a Random Forest, trained on selected numeric and binary features, separate deceased from non-deceased COVID-19 patients with essentially perfect accuracy. The confusion matrices show 99.78% accuracy for SVM (on 25% of the data) and 100% for Random Forest (under 10-fold cross-validation), with sensitivity of 100% and specificity above 99.7% for both. The most important features, ranked by Gini gain, are oxygen saturation by pulse oximetry, erythrocyte count, acute kidney failure, INR, and severe sepsis with septic shock. The paper reports that the two classes appear almost linearly separable in the selected features, which the authors say explains the extreme accuracy.","pith_inferences":["The Condition table contains all diagnosis records from 2015 onward, so codes like severe sepsis with septic shock and cardiac arrest can be recorded after the outcome; this leakage likely inflates the accuracy, and a study using only admission-day codes would probably show much lower performance.","The 100% Random Forest accuracy on a balanced test set is a red flag for label leakage or overfitting; external validation on a different hospital system's data would be a stronger test.","Imputing missing labs from a normal distribution of z-scores can erase clinical meaning (for example, not ordering a lab is informative), so the imputation method itself may contribute to the clean separation.","If the leakage were removed, the remaining predictors (oxygen saturation, erythrocytes, sodium, temperature) would still plausibly carry real prognostic signal, but likely at accuracy levels well below 100%."],"forward_implications":["If the near-perfect separation is real, hospitals could flag high-risk COVID-19 patients from the first lab panel and admission codes, enabling earlier escalation of care.","The top features (oxygen saturation, kidney failure, INR, sepsis codes) match the clinical picture of viral sepsis and end-organ damage, so the model is consistent with known pathophysiology.","The reported ease of separation suggests the two classes are almost linearly separable in the chosen feature space, meaning simpler scoring systems might match the classifiers.","The authors propose extending the work to time-series models that predict a surviving patient's transition to death, which would have direct ICU and ventilator planning value."],"supporting_citations":[{"why":"CDC page stating that one in three hospital deaths are from sepsis, providing the clinical motivation for separating survivors from non-survivors.","marker":"[1]"},{"why":"Tutorial describing cross-validation, the evaluation procedure used to obtain the reported accuracies.","marker":"[6]"},{"why":"Scikit-learn library, which supplies the Random Forest and SVM classifier implementations used in the experiments.","marker":"[14]"},{"why":"Apache Spark engine used to extract, merge, and pivot the patient data from the national database.","marker":"[16]"}],"fun_headline_variants":["RF and SVM hit ~100% accuracy distinguishing COVID-19 outcomes","Random Forest and SVM predict COVID-19 survival with near-perfect accuracy","SVM and RF separate COVID-19 deaths from survivors at ~100%","Machine learning draws clean line between COVID-19 death and recovery"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diagnosis codes used as predictors are pulled from a table containing all records from 2015 onward, so they can be recorded during the same hospitalization and are partly determined by the outcome; if they are not available at admission, the model is predicting death using the death itself.","fun_headline_variants_meta":{"raw":{"variants":["RF and SVM hit ~100% accuracy distinguishing COVID-19 outcomes","Random Forest and SVM predict COVID-19 survival with near-perfect accuracy","SVM and RF separate COVID-19 deaths from survivors at ~100%","Machine learning draws clean line between COVID-19 death and recovery"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3236,"prompt_tokens":915,"completion_tokens":2321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":2244}},"tokens_in":531,"tokens_out":2321,"duration_ms":13307,"temperature":1.0,"reasoning_tokens":2244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:54:23.946582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the classifiers using only diagnosis codes present on or before the admission encounter (or within 24 hours of admission), leaving all other features unchanged; if accuracy drops substantially from the reported 99.78% and 100%, the original result was driven by outcome leakage.","supporting_citations":[{"cited_title":"In healthcare, information can be converted into knowledge about patient historical patterns and possible future trends","cited_arxiv_id":null,"evidence_quote":"CDC page stating that one in three hospital deaths are from sepsis, providing the clinical motivation for separating survivors from non-survivors."},{"cited_title":"} A distance function /;45(O! ,O&) function 73;2-2Oℎ;O-PQPR4532(S,3T4,U;.V54): 1 for ; = 1 to . 2 Q = {%$} 3 end for 4 Q = {O!,…,O","cited_arxiv_id":null,"evidence_quote":"Tutorial describing cross-validation, the evaluation procedure used to obtain the reported accuracies."},{"cited_title":"Available from: https://www.saedsayad.com/clustering_hierarchical.htm [accessed 6 Dec, 2020]","cited_arxiv_id":null,"evidence_quote":"Scikit-learn library, which supplies the Random Forest and SVM classifier implementations used in the experiments."},{"cited_title":"S., Wendell, P., Das, T., Armbrust, M., Dave, A., … others","cited_arxiv_id":null,"evidence_quote":"Apache Spark engine used to extract, merge, and pivot the patient data from the national database."}],"review_version":1}