{"id":"6300e3c3-cd7b-4a6e-823d-35d8fe3873ad","arxiv_id":"2412.08984","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Standard ML models on EHR and SDoH features achieve AUC 0.81 to 0.82 for predicting ED visits in T2D patients without hypertension, but encounter-derived features may leak the outcome.","lead":"This study trains six machine learning models on electronic health records and neighborhood data for 34,151 Philadelphia patients with type 2 diabetes to predict whether a patient will have an emergency department visit, reporting ROC areas between 0.68 and 0.82. A smart generalist might read it as an example of how hospitals hope to forecast emergency demand, but the paper's own methods reveal that some predictors are derived from the very visits being predicted.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Outcome leakage via ED-visit-derived features is load-bearing: Section 2.3 computes ED-visit count and gap features that Section 2.5 then uses as predictors of ED visits, so the reported AUC 0.82 does not establish predictive validity.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: computed ED-visit features are used as predictors of whether an ED visit occurred. I agree. This is not a peripheral methodological quibble; it affects every reported AUC and the abstract's conclusion that the models are reliable clinical tools. A single retraining test with the suspect features removed would settle whether the reported performance survives. Since the reader already recommends REJECT and this analysis supports that recommendation, no verdict adjustment is needed.","tokens_in":9064,"tokens_out":2395,"duration_ms":23865,"concrete_test":"Retrain all six models on the same 70/30 split after removing the computed features 'number of emergency visits', 'duration between an ED visit and the last encounter preceding it', and any visitation-gap or gap-difference features derived from ED visit history. If the held-out AUC drops materially below 0.82, leakage is confirmed. Additionally, report cross-validated feature importances before deletion: if ED-count or gap features dominate, that directly confirms the mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that predictors are available before the outcome and do not encode the outcome. The manuscript violates this in at least two stated places. Section 2.3 ('Connect SDoH factors with EMR') lists among computed features 'number of emergency visits' and 'duration between an ED visit and the last encounter preceding it.' Section 2.5 ('Train and test machine learning models') says: 'Independent variables included demographics, SDoH, ED visits, common comorbidities, and vital signs,' while the prediction target is 'whether a patient visited the ED.' Thus the model is given the outcome, or a deterministic function of it, as an input. The reported AUC 0.82 and the abstract's claim that the models are 'reliable tools' therefore cannot be interpreted as evidence of predictive validity; the models could be identifying patients with ED visits by reading the leaked feature. This concern is load-bearing because it attacks the strongest claim directly, not a peripheral metric. The fact that top features include 'visitation gaps' and 'difference between visitation gaps' is consistent with utilization-history leakage, since those are computed from visit records. No code or data is provided to check whether the final 87 features actually include these leaked variables, so the ambiguity cannot be resolved from the paper alone. Because this affects all six models, the model comparisons and top-feature interpretations are also compromised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents a machine-learning pipeline to predict whether a patient with type 2 diabetes (T2D) visited an emergency department (ED), using HealthShare Exchange EHR data from 2017–2022 linked to ZIP-code-level social determinants of health. The cohort consists of 34,151 patients with T2D without hypertension and 703,065 encounters, with 43% of patients having at least one ED visit. Six classifiers are compared (CatBoost, XGBoost, Random Forest, Ensemble Learning, KNN, SVC); reported AUCs range from 0.68 to 0.82. The abstract concludes that the best models are reliable tools for predicting ED visit risk and estimating future ED demand.","tokens_in":9381,"tokens_out":9024,"duration_ms":81458,"significance":"The problem is important: reducing preventable ED utilization among patients with T2D is a high-value target for health systems. The manuscript's strengths are its large, real-world, multi-source dataset from an underserved urban population and its detailed preprocessing pipeline (standardizing demographic codes, mapping ICD-9 to ICD-10, linking SDoH). However, the central predictive claim is undermined by outcome leakage: features derived from ED visits are used as predictors of the ED-visit outcome. Consequently, the reported AUCs and feature importances do not provide evidence of predictive validity. No code or data is provided, which further limits verification. If the analysis were corrected with a strictly pre-outcome feature set and a temporal validation design, the underlying data could support a useful study; as submitted, the main conclusion is not supported.","major_comments":[{"comment":"The model's dependent variable is whether a patient ever had an ED visit (Section 2.5: \"We converted the number of ED visits into a binary format to indicate whether a patient visited the ED\"), but the independent variables include features that are deterministic functions of that outcome. Section 2.3 lists among the computed features \"number of emergency visits\" and \"duration between an ED visit and the last encounter preceding it.\" Section 2.5 explicitly states \"Independent variables included demographics, SDoH, ED visits, common comorbidities, and vital signs.\" Section 2.4 shows that the final 87 features comprise demographics (5), diagnoses (30), SDoH (30), vital signs (20), and computed features (2), so the two leaked features are part of the model input. A model can obtain near-perfect separation by thresholding \"number of emergency visits\" at zero, and the gap-duration feature is missing for patients with no ED visits, which tree-based models can exploit via missingness. Figure 6 confirms that \"visitation gaps\" and \"difference between visitation gaps\" are among the most important features. This leakage invalidates the discrimination results in Section 3.2 and the abstract's claim that the models are \"reliable tools.\"","section":"Section 2.3, Section 2.5"},{"comment":"The study does not establish a temporal ordering between predictors and outcome. The target is any ED visit during the 2017–2022 observation window, and the features are computed from encounters in the same window, with no index date, no temporal split, and no external validation. The abstract and discussion claim the models can \"estimate future ED demand,\" but the reported evaluation is a same-window classification task, not a prediction of future visits. A prospective or temporal validation (train on an earlier period, predict a later period) is needed to support the future-prediction language.","section":"Section 2.5, Section 3.2"},{"comment":"The abstract states that Ensemble Learning and Random Forest show superior performance \"in terms of discrimination, calibration, and clinical applicability,\" but the results section reports only AUC, accuracy, precision, recall, and F1. No calibration curve, calibration slope or intercept, Brier score, or decision-curve analysis is presented. The calibration claim is therefore unsupported by the reported evidence.","section":"Section 3.2, Abstract"},{"comment":"The feature-selection procedure selects the top 30 most frequent diagnoses using the full dataset before the train/test split. Because the full dataset includes the outcome window, this selection can leak outcome information into the modeling process. The selection should be performed inside the cross-validation loop, using only training-fold frequencies, to avoid optimistic performance estimates.","section":"Section 2.4"}],"minor_comments":[{"comment":"The text says \"over 2,000 distinct diagnoses\" but Section 2.2 reports 742 distinct diagnosis categories after mapping ICD codes to the first three characters; the manuscript should reconcile these numbers.","section":"Section 2.5"},{"comment":"The discussion highlights the \"ICE occupation indicator\" as an important feature, while the abstract and Figure 6 emphasize ICE for income; the authors should clarify which ICE measure was used and define it at first use.","section":"Section 4"},{"comment":"Reference [18] is a dataset of daily PM2.5 concentrations, which is not a social determinant of health; this citation appears to be mislinked in the description of SDoH data sources.","section":"References"},{"comment":"In Step 1, the raw data is said to have \"8, 97, and 154\" unique values for gender, ethnicity, and race; the punctuation is unclear and should be formatted as three separate counts with appropriate labels.","section":"Section 2.2"},{"comment":"No code or data availability statement is provided; given that the central concern is the construction of computed encounter features, access to the preprocessing code would materially aid verification.","section":"Data availability"},{"comment":"The header states that the manuscript was accepted and presented at AI-PHSS 2024; the authors should disclose the relationship between this preprint and the peer-reviewed conference proceedings.","section":"Title page"}],"recommendation":"reject","confidential_remarks":"The outcome leakage in Sections 2.3 and 2.5 is stated in the manuscript itself, not merely a possibility: the two computed features are included in the 87-feature set, so the reported AUCs cannot support the paper's central claim. If the authors can rerun the entire analysis with a pre-outcome feature set and a temporal validation design, a substantially revised version might warrant reconsideration; however, the current submission's main results are invalid. I would not recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: the main claim fails on a load-bearing data-leakage flaw. The paper reports AUC 0.82 for predicting ED visits, but it feeds the outcome into the model as a feature. Section 2.3 lists \"number of emergency visits\" and \"duration between an ED visit and the last encounter preceding it\" as computed features; Section 2.5 says independent variables included demographics, SDoH, ED visits, common comorbidities, and vital signs. The target is whether the patient visited the ED. You can't predict what you've already put in the input.\n\nWhat's worth credit: the authors assembled a large real-world EHR cohort (34k T2D patients, 703k visits) and did serious cleaning—standardizing vitals, mapping ICD-9 to ICD-10, unifying demographics, and linking ZIP-level SDoH. That pipeline is genuinely useful infrastructure. The feature ranking (age, visitation gaps, ICE income) is a new empirical observation for this Philadelphia population, though its validity is undercut by the same leakage.\n\nThe soft spots beyond the fatal one: no external validation, no calibration plot despite the abstract's claim of calibration, no code or data released, and the hypertension exclusion limits the population. The visitation-gap features are themselves derived from visit histories and thus also leak information about which patients are high utilizers. Even if the authors intended these as \"historical utilization\" predictors, there is no temporal split separating feature collection from outcome occurrence, so the model's discrimination never demonstrates prospective utility.\n\nI don't think the central argument survives. The AUC likely reflects the model reading count of ED visits, not genuine risk stratification. The paper could be salvaged only by removing the leaky features and redefining the prediction task with a proper time horizon (e.g., predict first ED visit in next 12 months using only prior data). As-is, it's a cautionary example for a methods class, not a reliable tool.\n\nWho gets value: someone auditing outcome leakage in EHR studies. I would not cite it in my own work. It deserves a desk reject, not referee time, unless the authors return with a corrected feature set.","headline":"The models feed the outcome (ED visit count) back as a feature, so the reported AUC 0.82 is meaningless; the paper is a useful cleaning pipeline but not a valid prediction study.","tokens_in":9905,"tokens_out":3183,"would_cite":false,"duration_ms":30315,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Machine-learning models can identify type 2 diabetes patients at risk of emergency department visits, with Random Forest, Ensemble Learning, and XGBoost each reaching an ROC AUC of 0.82.","keywords":["Type 2 diabetes","Machine learning","Emergency department visits","Social determinants of health","Electronic health records","Predictive modeling","Random Forest","XGBoost"],"falsifier":"Retrain the top models with the two computed encounter features, number of emergency visits and duration between an ED visit and the last prior encounter, removed, and compare the AUC against the reported 0.82; if the AUC drops sharply, the result is largely an artifact of predicting the outcome from itself.","tokens_in":8878,"feed_emoji":"🏥","tokens_out":6463,"duration_ms":60554,"temperature":0.7,"pith_summary":"This paper aims to show that machine-learning models trained on electronic health records plus neighborhood-level social determinants can identify which patients with type 2 diabetes are at risk of an emergency department visit. Using a cohort of 34,151 patients and 703,065 visits from the Philadelphia region, the authors compare six classifiers and report that Random Forest, Ensemble Learning, and XGBoost each reach an ROC AUC of 0.82. If the result holds, hospitals could use such models to forecast ED demand and target early interventions at patients with modifiable risk factors, reducing preventable visits and costs. The paper also identifies specific predictors, including age, gaps between visits, abdominal pain, and an income-based concentration index, that could guide care planning.","feed_headline":"Machine learning predicts ER visits in type 2 diabetes at AUC 0.82","feed_subtitle":"Models trained on 34,151 patients and social determinants of health could flag who needs early intervention.","key_machinery":"The central mechanism is a feature-construction and modeling pipeline. Raw HSX data, consisting of 76.6 million encounters, 113.9 million vital signs, and 123.2 million diagnoses, is cleaned, standardized, mapped to ICD-10, and reduced to 742 diagnosis categories. Patient ZIP codes are linked to ZCTA-level SDoH indicators, and computed encounter features such as the number of emergency visits and the duration between an ED visit and the prior encounter are added. The resulting 87 features are fed into six classifiers under 10-fold cross-validation, with ROC AUC as the main performance measure.","core_discovery":"The paper's central claim is that a machine-learning pipeline can predict ED visits among patients with type 2 diabetes by combining clinical encounter data with ZIP-code-level social determinants of health. The authors report that tree-based ensemble models achieve an ROC AUC of 0.82 on held-out test data, with CatBoost at 0.81, KNN at 0.72, and SVC at 0.68, and conclude that Ensemble Learning and Random Forest offer the best balance of discrimination, calibration, and clinical usefulness. They further claim that the most important features are age, visit-gap statistics, the ICD-10 code R10 (abdominal and pelvic pain), and the Index of Concentration at the Extremes for income.","pith_inferences":["A decisive diagnostic test would be to retrain the same models without the two computed encounter features; if the AUC collapses, the reported discrimination is driven by utilization history rather than by clinical or social risk factors.","The same workflow could be applied to other chronic conditions or to a prospective cohort where features are fixed at an index date and ED visits are counted only afterward, which would test whether the model predicts future visits rather than describing past ones.","If the model's performance transfers to other regions, the ZIP-code-level SDoH features could be used to target community-level interventions in areas with high predicted risk.","Given that the cohort excludes patients with hypertension, the models' performance on the excluded majority remains unknown; extending the pipeline to that group would test whether the exclusion changes which features matter."],"forward_implications":["Hospitals could use the model to forecast near-term ED volume for T2D patients and preposition staff, beds, insulin, and glucose-monitoring supplies.","Clinicians could flag high-risk patients from features like age, abdominal pain, smoking, and neighborhood income, and offer same-day appointments or education to prevent ED use.","Integrating the model into an EHR as a clinical decision-support alert could trigger early review of patients whose data suggests rising risk.","The importance of SDoH features such as ICE income and education suggests community-level outreach may reduce ED visits.","Continuous updating of the model with new encounter data would allow it to reflect changing patient behavior and treatment protocols."],"supporting_citations":[{"why":"provides the HSX Clinical Data Repository with encounter, diagnosis, and vital-sign records for the cohort","marker":"[16]"},{"why":"supply ZIP-code-level social determinants of health from the American Community Survey and SEDAC","marker":"[17, 18]"},{"why":"CatBoost is one of the six classifier algorithms compared in the study","marker":"[22]"},{"why":"Random Forest is one of the top-performing classifiers and a key algorithm in the comparison","marker":"[26]"},{"why":"XGBoost is one of the top-performing classifiers and a key algorithm in the comparison","marker":"[27]"},{"why":"provides the external pattern of common ED visit reasons that the findings are checked against","marker":"[29]"},{"why":"documents the rising trend in diabetes-related ED visits that motivates the prediction target","marker":"[13]"}],"fun_headline_variants":["AUC-0.82 ML model forecasts diabetes ER visits","Tree models best predict ER visits in type 2 diabetes","Machine learning flags diabetes patients at ER risk","Social determinants improve diabetes ER visit prediction","Ensemble learning tops diabetes ED prediction at 0.82 AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a patient's past number of emergency visits and the timing around those visits are legitimate predictors, even though the outcome being predicted is whether a patient ever had an emergency visit.","fun_headline_variants_meta":{"raw":{"variants":["AUC-0.82 ML model forecasts diabetes ER visits","Tree models best predict ER visits in type 2 diabetes","Machine learning flags diabetes patients at ER risk","Social determinants improve diabetes ER visit prediction","Ensemble learning tops diabetes ED prediction at 0.82 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00062,"raw_usage":{"total_tokens":2918,"prompt_tokens":1030,"completion_tokens":1888,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":1811}},"tokens_in":646,"tokens_out":1888,"duration_ms":12465,"temperature":1.0,"reasoning_tokens":1811,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:59.887724+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the top models with the two computed encounter features, number of emergency visits and duration between an ED visit and the last prior encounter, removed, and compare the AUC against the reported 0.82; if the AUC drops sharply, the result is largely an artifact of predicting the outcome from itself.","supporting_citations":[{"cited_title":"Clinical Data Repository (CDR) Overview","cited_arxiv_id":null,"evidence_quote":"provides the HSX Clinical Data Repository with encounter, diagnosis, and vital-sign records for the cohort"},{"cited_title":"Hooker, Peter J","cited_arxiv_id":null,"evidence_quote":"provides the external pattern of common ED visit reasons that the findings are checked against"},{"cited_title":"Uppal, Puneet Kaur Chehal, Gail Fernandes, et al","cited_arxiv_id":null,"evidence_quote":"documents the rising trend in diabetes-related ED visits that motivates the prediction target"}],"review_version":1}