REVIEW 4 major objections 6 minor 1 cited by
Predicting Emergency Department Visits for Patients with Type II Diabetes
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Machine-learning models can identify type 2 diabetes patients at risk of emergency department visits, with Random Forest, Ensemble Learning, and XGBoost each reaching an ROC AUC of 0.82.
desk verdict The models feed the outcome (ED visit count) back as a feature, so the reported AUC 0.82 is meaningless; the paper is a useful cleaning pipeline but not a valid prediction study. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a feature-construction and modeling pipeline. Raw HSX data, consisting of 76.6 million encounters, 113.9 million vital signs, and 123.2 million diagnoses, is cleaned, standardized, mapped to ICD-10, and reduced to 742 diagnosis categories. Patient ZIP codes are linked to ZCTA-level SDoH indicators, and computed encounter features such as the number of emergency visits and the duration between an ED visit and the prior encounter are added. The resulting 87 features are fed into six classifiers under 10-fold cross-validation, with ROC AUC as the main performance measure.
What would settle it
Retrain the top models with the two computed encounter features, number of emergency visits and duration between an ED visit and the last prior encounter, removed, and compare the AUC against the reported 0.82; if the AUC drops sharply, the result is largely an artifact of predicting the outcome from itself.
Extended reading notes
Core claim
The paper's central claim is that a machine-learning pipeline can predict ED visits among patients with type 2 diabetes by combining clinical encounter data with ZIP-code-level social determinants of health. The authors report that tree-based ensemble models achieve an ROC AUC of 0.82 on held-out test data, with CatBoost at 0.81, KNN at 0.72, and SVC at 0.68, and conclude that Ensemble Learning and Random Forest offer the best balance of discrimination, calibration, and clinical usefulness. They further claim that the most important features are age, visit-gap statistics, the ICD-10 code R10 (abdominal and pelvic pain), and the Index of Concentration at the Extremes for income.
Load-bearing premise
The load-bearing premise is that a patient's past number of emergency visits and the timing around those visits are legitimate predictors, even though the outcome being predicted is whether a patient ever had an emergency visit.
Editorial extensions
If this is right
- Hospitals could use the model to forecast near-term ED volume for T2D patients and preposition staff, beds, insulin, and glucose-monitoring supplies.
- Clinicians could flag high-risk patients from features like age, abdominal pain, smoking, and neighborhood income, and offer same-day appointments or education to prevent ED use.
- Integrating the model into an EHR as a clinical decision-support alert could trigger early review of patients whose data suggests rising risk.
- The importance of SDoH features such as ICE income and education suggests community-level outreach may reduce ED visits.
- Continuous updating of the model with new encounter data would allow it to reflect changing patient behavior and treatment protocols.
Reading between the lines
- A decisive diagnostic test would be to retrain the same models without the two computed encounter features; if the AUC collapses, the reported discrimination is driven by utilization history rather than by clinical or social risk factors.
- The same workflow could be applied to other chronic conditions or to a prospective cohort where features are fixed at an index date and ED visits are counted only afterward, which would test whether the model predicts future visits rather than describing past ones.
- If the model's performance transfers to other regions, the ZIP-code-level SDoH features could be used to target community-level interventions in areas with high predicted risk.
- Given that the cohort excludes patients with hypertension, the models' performance on the excluded majority remains unknown; extending the pipeline to that group would test whether the exclusion changes which features matter.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript presents a machine-learning pipeline to predict whether a patient with type 2 diabetes (T2D) visited an emergency department (ED), using HealthShare Exchange EHR data from 2017–2022 linked to ZIP-code-level social determinants of health. The cohort consists of 34,151 patients with T2D without hypertension and 703,065 encounters, with 43% of patients having at least one ED visit. Six classifiers are compared (CatBoost, XGBoost, Random Forest, Ensemble Learning, KNN, SVC); reported AUCs range from 0.68 to 0.82. The abstract concludes that the best models are reliable tools for predicting ED visit risk and estimating future ED demand.
Significance. The problem is important: reducing preventable ED utilization among patients with T2D is a high-value target for health systems. The manuscript's strengths are its large, real-world, multi-source dataset from an underserved urban population and its detailed preprocessing pipeline (standardizing demographic codes, mapping ICD-9 to ICD-10, linking SDoH). However, the central predictive claim is undermined by outcome leakage: features derived from ED visits are used as predictors of the ED-visit outcome. Consequently, the reported AUCs and feature importances do not provide evidence of predictive validity. No code or data is provided, which further limits verification. If the analysis were corrected with a strictly pre-outcome feature set and a temporal validation design, the underlying data could support a useful study; as submitted, the main conclusion is not supported.
major comments (4)
- [Section 2.3, Section 2.5] The model's dependent variable is whether a patient ever had an ED visit (Section 2.5: "We converted the number of ED visits into a binary format to indicate whether a patient visited the ED"), but the independent variables include features that are deterministic functions of that outcome. Section 2.3 lists among the computed features "number of emergency visits" and "duration between an ED visit and the last encounter preceding it." Section 2.5 explicitly states "Independent variables included demographics, SDoH, ED visits, common comorbidities, and vital signs." Section 2.4 shows that the final 87 features comprise demographics (5), diagnoses (30), SDoH (30), vital signs (20), and computed features (2), so the two leaked features are part of the model input. A model can obtain near-perfect separation by thresholding "number of emergency visits" at zero, and the gap-duration feature is missing for patients with no ED visits, which tree-based models can exploit via missingness. Figure 6 confirms that "visitation gaps" and "difference between visitation gaps" are among the most important features. This leakage invalidates the discrimination results in Section 3.2 and the abstract's claim that the models are "reliable tools."
- [Section 2.5, Section 3.2] The study does not establish a temporal ordering between predictors and outcome. The target is any ED visit during the 2017–2022 observation window, and the features are computed from encounters in the same window, with no index date, no temporal split, and no external validation. The abstract and discussion claim the models can "estimate future ED demand," but the reported evaluation is a same-window classification task, not a prediction of future visits. A prospective or temporal validation (train on an earlier period, predict a later period) is needed to support the future-prediction language.
- [Section 3.2, Abstract] The abstract states that Ensemble Learning and Random Forest show superior performance "in terms of discrimination, calibration, and clinical applicability," but the results section reports only AUC, accuracy, precision, recall, and F1. No calibration curve, calibration slope or intercept, Brier score, or decision-curve analysis is presented. The calibration claim is therefore unsupported by the reported evidence.
- [Section 2.4] The feature-selection procedure selects the top 30 most frequent diagnoses using the full dataset before the train/test split. Because the full dataset includes the outcome window, this selection can leak outcome information into the modeling process. The selection should be performed inside the cross-validation loop, using only training-fold frequencies, to avoid optimistic performance estimates.
minor comments (6)
- [Section 2.5] The text says "over 2,000 distinct diagnoses" but Section 2.2 reports 742 distinct diagnosis categories after mapping ICD codes to the first three characters; the manuscript should reconcile these numbers.
- [Section 4] The discussion highlights the "ICE occupation indicator" as an important feature, while the abstract and Figure 6 emphasize ICE for income; the authors should clarify which ICE measure was used and define it at first use.
- [References] Reference [18] is a dataset of daily PM2.5 concentrations, which is not a social determinant of health; this citation appears to be mislinked in the description of SDoH data sources.
- [Section 2.2] In Step 1, the raw data is said to have "8, 97, and 154" unique values for gender, ethnicity, and race; the punctuation is unclear and should be formatted as three separate counts with appropriate labels.
- [Data availability] No code or data availability statement is provided; given that the central concern is the construction of computed encounter features, access to the preprocessing code would materially aid verification.
- [Title page] The header states that the manuscript was accepted and presented at AI-PHSS 2024; the authors should disclose the relationship between this preprint and the peer-reviewed conference proceedings.
Circularity Check
ED-visit-derived features are used as predictors of ED visits, so the headline AUCs are circular by construction.
-
self definitional
[Section 2.3 ('Connect SDoH factors with EMR') and Section 2.5 ('Train and test machine learning models'); relied on by the Abstract and Section 3.2.]
"Details of patient encounters, including newly calculated data such as first and last visit dates at HSX facilities, number of emergency visits, and the duration between an ED visit and the last encounter preceding it. ... We converted the number of ED visits into a binary format to indicate whether a patient visited the ED. ... Independent variables included demographics, SDoH, ED visits, common comorbidities, and vital signs."
The outcome variable is 'whether a patient visited the ED,' obtained by binarizing the number of ED visits. The same quantity, 'number of emergency visits' / 'ED visits,' is explicitly listed among the independent variables. A model given the ED visit count can label any patient with a nonzero count as positive, so the reported AUCs (0.68-0.82) do not measure predictive validity for future ED visits. The additional computed feature 'duration between an ED visit and the last encounter preceding it' is only defined for patients who have had an ED visit, so it also encodes the label. Because these leaked features are present in the training pipeline for all six models, the model comparisons and feature-importance rankings are contaminated.
full rationale
The paper's central claim is that the models are 'reliable tools for predicting risk of ED visits among patients with T2D,' supported by AUC values around 0.82. However, the manuscript itself states that the target is the binarized number of ED visits and then lists 'ED visits' among the independent variables. This is textbook outcome leakage: the label, or a deterministic function of it, is supplied as an input. Section 2.3 also constructs the 'number of emergency visits' and the 'duration between an ED visit and the last encounter preceding it' from the same ED visit records used to define the label. The top important features named in the abstract, 'visitation gaps' and 'difference between visitation gaps,' are consistent with visit-history-derived leakage. No code, data, or external validation is provided to show that the final 87 features exclude these leaked variables, so the ambiguity cannot be resolved from the paper alone. This is not a self-citation issue; the circularity is internal to the described workflow. The central predictive result therefore reduces by construction to the input, although some non-leaked SDoH, demographic, and vital-sign features are legitimate predictors. Score 8 reflects that the headline AUCs are forced by definition rather than being an independent demonstration of predictive validity.
Assumptions & free parameters
free parameters (3)
- Model hyperparameters (CatBoost, XGBoost, Random Forest, Ensemble, KNN, SVC) =
not reported
- Classification threshold =
not reported (default 0.5 implied)
- Number of diagnosis features (top 30) =
30
assumptions (5)
- ad hoc to paper Encounter-derived features such as number of ED visits, visitation gaps, and gap differences do not encode the target outcome.
- domain assumption Patients with T2D without hypertension represent the target population for the claims.
- domain assumption ZCTA-level SDoH indicators assigned by ZIP code adequately represent individual social determinants.
- domain assumption Top-30 most frequent diagnoses preserve the predictive signal of comorbidity burden.
- domain assumption The custom ICD-9 to ICD-10 mapping algorithm is accurate enough for feature construction.
Cite this review
Pith. "Pith review of Predicting Emergency Department Visits for Patients with Type II Diabetes." pith.science (2026). https://pith.science/paper/LWMTE56A
@misc{pith2026241208984,
author = {Pith},
title = {Pith review of: Predicting Emergency Department Visits for Patients with Type II Diabetes},
year = {2026},
howpublished = {\url{https://pith.science/paper/LWMTE56A}},
note = {Machine review of arXiv:2412.08984}
}
read the original abstract
Over 30 million Americans are affected by Type II diabetes (T2D), a treatable condition with significant health risks. This study aims to develop and validate predictive models using machine learning (ML) techniques to estimate emergency department (ED) visits among patients with T2D. Data for these patients was obtained from the HealthShare Exchange (HSX), focusing on demographic details, diagnoses, and vital signs. Our sample contained 34,151 patients diagnosed with T2D which resulted in 703,065 visits overall between 2017 and 2021. A workflow integrated EMR data with SDoH for ML predictions. A total of 87 out of 2,555 features were selected for model construction. Various machine learning algorithms, including CatBoost, Ensemble Learning, K-nearest Neighbors (KNN), Support Vector Classification (SVC), Random Forest, and Extreme Gradient Boosting (XGBoost), were employed with tenfold cross-validation to predict whether a patient is at risk of an ED visit. The ROC curves for Random Forest, XGBoost, Ensemble Learning, CatBoost, KNN, and SVC, were 0.82, 0.82, 0.82, 0.81, 0.72, 0.68, respectively. Ensemble Learning and Random Forest models demonstrated superior predictive performance in terms of discrimination, calibration, and clinical applicability. These models are reliable tools for predicting risk of ED visits among patients with T2D. They can estimate future ED demand and assist clinicians in identifying critical factors associated with ED utilization, enabling early interventions to reduce such visits. The top five important features were age, the difference between visitation gaps, visitation gaps, R10 or abdominal and pelvic pain, and the Index of Concentration at the Extremes (ICE) for income.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
DT4PCP: A Digital Twin Framework for Personalized Care Planning Applied to Type 2 Diabetes Management
A digital twin framework for Type 2 Diabetes uses retrospective EHR data to predict ED visits and support personalized care recommendations, with a reported AUC of 0.82.
Reference graph
Works this paper leans on
-
[1]
National and State Diabetes Trends
CDC. National and State Diabetes Trends. 2022. Available at: https://archive.cdc.gov/www_cdc_gov/ diabetes/library/reports/reportcard/national-state-diabetes-trends.html
work page 2022
-
[2]
CDC. About Chronic Diseases. 2024. Available at:https://www.cdc.gov/chronic-disease/about/index. html
work page 2024
-
[3]
Chronic Conditions among Older Americans
American Association of Retired Persons. Chronic Conditions among Older Americans. 2009. Available at: https://assets.aarp.org/rgcenter/health/beyond_50_hcr_conditions.pdf
work page 2009
-
[4]
CDC. Leading Causes of Death. 2024. Available at: https://www.cdc.gov/nchs/fastats/ leading-causes-of-death.htm
work page 2024
-
[5]
CDC. WISQARS | Injury Data Reports. Available at: https://wisqars.cdc.gov/create-reports/
-
[6]
American Health Care: Health Spend- ing and the Federal Budget
Committee for a Responsible Federal Budget. American Health Care: Health Spend- ing and the Federal Budget. 2018. Available at: https://www.crfb.org/papers/ american-health-care-health-spending-and-federal-budget
work page 2018
-
[7]
Dieleman, Jackie Cao, Abby Chapin, et al
Joseph L. Dieleman, Jackie Cao, Abby Chapin, et al. US Health Care Spending by Payer and Health Condition, 1996-2016. JAMA. 2020;323(9):863-884. doi:10.1001/JAMA.2020.0734
-
[8]
National Health Expenditures 2017 Highlights
Centers for Medicare and Medicaid Services. National Health Expenditures 2017 Highlights
work page 2017
Show all 33 references
-
[9]
Ruirui Sun, Zeynal Karaca, Herbert S. Wong. Trends in Hospital Emergency Department Visits by Age and Payer, 2006–2015. HCUP Statistical Briefs. 2018. PMID: 30063311
2006
-
[10]
National Hospital Ambulatory Medical Care Survey: 2019 Emergency Department Summary Tables
Ambulatory and Hospital Care Statistics Branch of National Center for Health Statistics (NCHS). National Hospital Ambulatory Medical Care Survey: 2019 Emergency Department Summary Tables. 2019. Available at: https://www.cdc.gov/nchs/data/nhamcs/web_tables/2019-nhamcs-ed-web-ta...
2019
-
[11]
Yan, Katherine M
Justin W. Yan, Katherine M. Gushulak, Melanie P. Columbus, et al. Risk factors for recurrent emergency department visits for hyperglycemia in patients with diabetes mellitus. Int J Emerg Med . 2017;10(1):12. doi:10.1186/S12245-017-0150-Y
2017 doi
-
[12]
Association between Homelessness and Opioid Over- dose and Opioid-related Hospital Admissions/Emergency Department Visits
Ayae Yamamoto, Jack Needleman, Lillian Gelberg, et al. Association between Homelessness and Opioid Over- dose and Opioid-related Hospital Admissions/Emergency Department Visits. Soc Sci Med. 2019;242:112585. doi:10.1016/J.SOCSCIMED.2019.112585
2019
-
[13]
Uppal, Puneet Kaur Chehal, Gail Fernandes, et al
Tegveer S. Uppal, Puneet Kaur Chehal, Gail Fernandes, et al. Trends and Variations in Emergency Depart- ment Use Associated With Diabetes in the US by Sociodemographic Factors, 2008-2017. JAMA Netw Open. 2022;5(5):E2213867. doi:10.1001/JAMANETWORKOPEN.2022.13867
2008
-
[14]
Washington, Roxanne M
Raynard E. Washington, Roxanne M. Andrews, Ryan Mutter. Emergency Department Visits for Adults with Diabetes, 2010. HCUP Statistical Briefs. 2013. Available at: https://hcup-us.ahrq.gov/reports/ statbriefs/sb167.jsp
2010
-
[15]
Williams
Gareth H. Williams. The determinants of health: structure, context, and agency. Sociology of Health and Illness. 2003;25(3):131-154. doi:10.1111/1467-9566.00344
2003
-
[16]
Clinical Data Repository (CDR) Overview
HealthShare Exchange (HSX). Clinical Data Repository (CDR) Overview. 2017. Available at: https:// healthshareexchange.org/wp-content/uploads/2021/08/11.20.17_cdr_overview_final.pdf
2017
-
[17]
American Community Survey Data
The United States Census Bureau. American Community Survey Data. Available at: https://www.census. gov/programs-surveys/acs/data.html
-
[18]
Daily and Annual PM2.5 Concentrations for the Contiguous United States, 1-km Grids, v1 (2000 - 2016)
Di Q, Wei Y , Shtein A, et al. Daily and Annual PM2.5 Concentrations for the Contiguous United States, 1-km Grids, v1 (2000 - 2016). NASA Socioeconomic Data and Applications Center (SEDAC). 2021. doi:10.7927/0rvr-4538
-
[19]
Available at: https://pandas.pydata.org/
Pandas - Python Data Analysis Library. Available at: https://pandas.pydata.org/
-
[20]
Available at: https://orwh.od.nih.gov/toolkit/ other-relevant-federal-policies/OMB-standards
Office of Management and Budget (OMB) Standards. Available at: https://orwh.od.nih.gov/toolkit/ other-relevant-federal-policies/OMB-standards . 10 Predicting Emergency Department Visits for Patients with Type II Diabetes A PREPRINT
-
[21]
Available at: https://imagic.nlm
I-MAGIC Mapper: Interactive Map-Assisted Generation of ICD Codes. Available at: https://imagic.nlm. nih.gov/imagic/code/map
-
[22]
CatBoost: unbiased boosting with categorical features
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr V orobev, Anna Veronika Dorogush, Andrey Gulin. CatBoost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems. 2018;6638-6648. Available at: https://arxiv.org/abs/1706.09516v5
2018 arXiv
-
[23]
Popular ensemble methods: an empirical study
David Opitz, Richard Maclin. Popular ensemble methods: an empirical study. J Artif Intell Res . 1999. doi:10.5555/3013545.3013549
1999
-
[24]
N. S. Altman. An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression. Am Stat . 1992;46(3):175-185. doi:10.1080/00031305.1992.10475879
1992 arXiv
-
[25]
Support-vector networks
Corinna Cortes, Vladimir Vapnik. Support-vector networks. Mach Learn . 1995;20(3):273-297. doi:10.1007/BF00994018
1995 doi
-
[26]
Random forests
Leo Breiman. Random forests. Mach Learn. 2001;45(1):5-32. doi:10.1023/A:1010933404324
2001 doi
-
[27]
XGBoost: A Scalable Tree Boosting System
Tianqi Chen, Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. Proc ACM SIGKDD Int Conf Knowl Discov Data Min. 2016;13-17. doi:10.1145/2939672.2939785
2016
-
[28]
Too much covariates in a multivariable model may cause the problem of overfitting.J Thorac Dis
Zhongheng Zhang. Too much covariates in a multivariable model may cause the problem of overfitting.J Thorac Dis. 2014;6(9):E196. doi:10.3978/J.ISSN.2072-1439.2014.08.33
2014
-
[29]
Hooker, Peter J
Edmond A. Hooker, Peter J. Mallow, Michelle M. Oglesby. Characteristics and Trends of Emer- gency Department Visits in the United States (2010-2014). J Emerg Med . 2019;56(3):344-351. doi:10.1016/J.JEMERMED.2018.12.025
2010 doi
-
[30]
Petrie, Tomasz J
John R. Petrie, Tomasz J. Guzik, Rhian M. Touyz. Diabetes, Hypertension, and Cardiovascular Disease: Clinical Insights and Vascular Mechanisms. Can J Cardiol. 2018;34(5):575-584. doi:10.1016/J.CJCA.2017.12.005
2018 doi
-
[31]
Harries, Rony Zachariah, Anil Kapur, et al
Anthony D. Harries, Rony Zachariah, Anil Kapur, et al. The vital signs of chronic disease management. Trans R Soc Trop Med Hyg. 2009;103(6):537-540. doi:10.1016/j.trstmh.2008.12.008
2009 doi
-
[32]
Gabayan, Michael K
Gelareh Z. Gabayan, Michael K. Gould, Robert E. Weiss, et al. Emergency Department Vital Signs and Outcomes After Discharge. Acad Emerg Med. 2017;24(7):846-854. doi:10.1111/ACEM.13194. 11
2017 doi
-
[2017]
Available at: https://www.cms.gov/research-statistics-data-and-systems/ statistics-trends-and-reports/nationalhealthexpenddata/downloads/highlights.pdf
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.