Recognition: unknown
Multiple Imputation Diagnostics when using Electronic Health Record Data in Observational Studies: A Case Study
Pith reviewed 2026-05-10 15:49 UTC · model grok-4.3
The pith
In EHR data for kidney disease, the specific choice of multiple imputation method had little effect on study conclusions.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
In this case study using patient data from two health systems, CART-based multiple imputation validated with graphical diagnostics revealed that the choice between different imputation specifications had minimal impact on inference about the association between lower neighborhood socioeconomic status and higher risk of cardiovascular disease hospitalization, as well as on prediction accuracy.
What carries the argument
CART imputation model validated using multivariate graphical diagnostics for missing data in EHR.
If this is right
- Imputed distributions of key variables like systolic blood pressure and HbA1c differ based on marginal versus conditional imputation.
- Inference on the link between neighborhood socioeconomic status and cardiovascular outcomes remains stable across MI variants.
- Prediction models for the outcome show little sensitivity to the imputation approach.
- Graphical diagnostics provide a practical way to assess imputation quality in observational EHR studies.
Where Pith is reading between the lines
- Researchers working with multi-center EHR data might apply the same diagnostics to test if results hold when combining systems.
- The finding of robustness could encourage broader adoption of MI over simpler methods like mean imputation in health outcomes research.
- Future work could compare these results against complete-case analysis to quantify the bias avoided.
Load-bearing premise
That the CART model accurately reflects the relationships among variables and that the graphical diagnostics catch any major imputation problems.
What would settle it
If alternative imputation techniques produced meaningfully different estimates of the effect of neighborhood socioeconomic status on cardiovascular hospitalizations.
Figures
read the original abstract
Missing values in electronic health record (EHR) data pose a significant challenge for epidemiologic research. Traditional methods for handling missing data, like mean imputation, may introduce bias. Multiple imputation (MI) offers a principled solution by generating multiple plausible values based on statistical models. However, MI requires careful model specification and validation of imputations, ideally using multivariate graphical tools. We demonstrate the application of such tools to validate MI in a study of chronic kidney disease, assessing cardiovascular outcomes linked to neighborhood socioeconomic status (nSES). This study used data from Duke University Health System (DUHS) and Lincoln Community Health Center (LCHC). Eligible patients had at least one encounter within DUHS or LCHC and had two estimated glomerular filtration rate (eGFR) values <60 mL/min per 1.73 m2 more than 90 days apart between January 1, 2007 and July 1, 2008. Socioeconomic status was assessed using the Agency for Healthcare Research and Quality (AHRQ) index based on census data. The main outcome was a cardiovascular disease-related hospitalization. Participants were mostly older (mean age 73 years), female (64%), and Black (43%). Participants living in lower nSES neighborhoods had higher mean systolic blood pressure (SBP: 140 mmHg) and hemoglobin A1c (HbA1c) levels (7.1%) as compared to participants living in higher nSES neighborhoods. A machine learning based approach, Classification and Regression Trees (CART), was the preferred approach to impute missing data. The distributions of imputed values of SBP and HbA1c were impacted by whether marginal or conditional values of SBP and HbA1c were imputed. The choice of MI had minimal impact on inference and prediction. Future research may want to extend our results and consider how results may differ when using EHR data from multiple health systems.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper is a case study applying multiple imputation (MI) via Classification and Regression Trees (CART) to handle missing data in electronic health records from a cohort of chronic kidney disease patients at Duke University Health System and Lincoln Community Health Center. It examines associations between neighborhood socioeconomic status (nSES) and cardiovascular disease-related hospitalizations, using multivariate graphical diagnostics to validate imputations of variables such as systolic blood pressure and HbA1c. The authors report that imputed value distributions differ by marginal versus conditional imputation approaches, yet the choice of MI specification has minimal impact on the resulting inference for nSES-CVD associations and on predictive performance.
Significance. If the central empirical finding holds after addressing validation gaps, the work offers a concrete illustration of MI diagnostics in complex EHR settings, where missingness often involves time-varying labs and multi-center patterns. It provides practical reassurance that downstream epidemiologic conclusions may be robust to reasonable imputation model choices when CART and graphical checks are employed, which could inform applied researchers facing similar data challenges. As a single-site case study without new methodological contributions or quantitative robustness metrics, its broader impact is incremental rather than transformative.
major comments (2)
- [Results] Results (imputation impact on inference): The claim that 'the choice of MI had minimal impact on inference and prediction' is load-bearing for the paper's main conclusion but is supported only by qualitative statements without reported effect-size differences, confidence intervals, or formal statistical comparisons (e.g., overlap of coefficient estimates or prediction metrics) across the MI variants examined; this leaves open whether concordance arises from robustness or from shared bias under the CART model.
- [Methods] Methods (validation of imputations): The multivariate graphical diagnostics and CART approach are presented as sufficient to confirm valid imputations under an implicit MAR assumption, yet no quantitative diagnostics (such as posterior predictive checks, comparison of observed vs. imputed conditional moments, or sensitivity analyses under MNAR mechanisms) are reported; given the EHR context of encounter-level and center-specific missingness, this weakens the evidential basis for concluding that results are insensitive to model choice rather than jointly biased.
minor comments (2)
- [Abstract] Abstract: The eligibility criteria and cohort description would benefit from a supplementary table or flow diagram reporting the number of patients at each inclusion step to improve reproducibility.
- [Discussion] The paper could add a brief limitations paragraph explicitly addressing generalizability beyond the two health systems and the absence of MNAR sensitivity checks.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed comments on our case study manuscript. We address each major point below, providing the strongest honest defense of our approach while agreeing to strengthen the presentation where feasible.
read point-by-point responses
-
Referee: [Results] Results (imputation impact on inference): The claim that 'the choice of MI had minimal impact on inference and prediction' is load-bearing for the paper's main conclusion but is supported only by qualitative statements without reported effect-size differences, confidence intervals, or formal statistical comparisons (e.g., overlap of coefficient estimates or prediction metrics) across the MI variants examined; this leaves open whether concordance arises from robustness or from shared bias under the CART model.
Authors: We acknowledge that the original manuscript described the similarity in inferences primarily in qualitative terms. To strengthen this, the revised manuscript will include a table reporting the nSES coefficient estimates and 95% confidence intervals for the CVD hospitalization outcome under both marginal and conditional CART imputation specifications. We will also add any available predictive performance metrics (such as AUC or Brier score) for direct comparison. This will allow explicit assessment of effect-size overlap and address the concern about potential shared bias. revision: yes
-
Referee: [Methods] Methods (validation of imputations): The multivariate graphical diagnostics and CART approach are presented as sufficient to confirm valid imputations under an implicit MAR assumption, yet no quantitative diagnostics (such as posterior predictive checks, comparison of observed vs. imputed conditional moments, or sensitivity analyses under MNAR mechanisms) are reported; given the EHR context of encounter-level and center-specific missingness, this weakens the evidential basis for concluding that results are insensitive to model choice rather than jointly biased.
Authors: Multivariate graphical diagnostics are a standard and recommended tool for MI validation in the literature, particularly for complex EHR data where CART provides flexible, non-parametric modeling. We will revise the manuscript to include quantitative comparisons of observed versus imputed conditional means and variances for SBP and HbA1c, stratified by nSES and health system. However, full posterior predictive checks and MNAR sensitivity analyses would require substantial additional assumptions and analyses outside the scope of this applied case study; we will note this limitation explicitly and suggest it for future research. revision: partial
Circularity Check
No significant circularity in empirical case study
full rationale
This is an applied case study comparing multiple imputation methods (primarily CART-based) on EHR data for nSES-CVD associations, with the central finding that MI choice had minimal impact on inference and prediction. The claim rests on direct empirical comparisons of downstream results across imputation variants and qualitative multivariate graphical diagnostics, without any derivation, equation, or prediction that reduces by construction to fitted inputs or self-referential assumptions. No load-bearing self-citations, uniqueness theorems, or ansatzes are invoked; the work applies existing MI tools to a specific dataset and reports observed concordance, which is falsifiable against the data rather than tautological.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Missing at random assumption required for valid multiple imputation
Reference graph
Works this paper leans on
-
[1]
Missing Data in Clinical Research: A Tutorial on Multiple Imputation
Austin PC, White IR, Lee DS, van Buuren S. Missing Data in Clinical Research: A Tutorial on Multiple Imputation. Can J Cardiol. Sep 2021;37(9):1322-1331. doi:10.1016/j.cjca.2020.11.010
-
[2]
Theoretical epidemiology : principles of occurrence research in medicine
Miettinen OS. Theoretical epidemiology : principles of occurrence research in medicine. A Wiley medical publication. John Wiley & Sons; 1985
1985
-
[3]
A critical look at methods for handling missing covariates in epidemiologic regression analyses
Greenland S, Finkle WD. A critical look at methods for handling missing covariates in epidemiologic regression analyses. Am J Epidemiol. Dec 15 1995;142(12):1255-64. doi:10.1093/oxfordjournals.aje.a117592
-
[4]
Multiple imputation for nonresponse in surveys
Toutenburg HR, D.B. Multiple imputation for nonresponse in surveys. Statistical Papers. 1990/12/01 1990;31(1):180-180. doi:10.1007/BF02924688
-
[5]
Review: A gentle introduction to imputation of missing values
Donders R, van der Heijden G, Stijnen T, Moons K. Review: A gentle introduction to imputation of missing values. Journal of clinical epidemiology. 11/01 2006;59:1087-91. doi:10.1016/j.jclinepi.2006.01.014
-
[6]
Regression with Missing X's: A Review
Little RJA. Regression with Missing X's: A Review. Journal of the American Statistical Association. 1992/12/01 1992;87(420):1227-1237. doi:10.1080/01621459.1992.10476282
-
[7]
Analysis of Incomplete Multivariate Data
Schafer J. Analysis of Incomplete Multivariate Data. Chapman and Hall; 1997
1997
-
[8]
Knol MJ, Janssen KJ, Donders AR, et al. Unpredictable bias when using the missing indicator method or complete case analysis for missing confounder values: an empirical example. J Clin Epidemiol. Jul 2010;63(7):728-36. doi:10.1016/j.jclinepi.2009.08.028
-
[9]
White IR, Carlin JB. Bias and efficiency of multiple imputation compared with complete-case analysis for missing covariate values. Stat Med. Dec 10 2010;29(28):2920-31. doi:10.1002/sim.3944
-
[10]
Model checking in multiple imputation: an overview and case study
Nguyen CD, Carlin JB, Lee KJ. Model checking in multiple imputation: an overview and case study. Emerg Themes Epidemiol. 2017;14:8. doi:10.1186/s12982-017-0062-6
-
[11]
Lee KJ, Carlin JB. Multiple imputation for missing data: fully conditional specification versus multivariate normal imputation. Am J Epidemiol. Mar 1 2010;171(5):624-32. doi:10.1093/aje/kwp425
-
[12]
A comparison of inclusive and restrictive strategies in modern missing data procedures
Collins LM, Schafer JL, Kam CM. A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychol Methods. Dec 2001;6(4):330-51
2001
-
[13]
Seaman SR, Bartlett JW, White IR. Multiple imputation of missing covariates with non-linear effects and interactions: an evaluation of statistical methods. BMC Med Res Methodol. Apr 10 2012;12:46. doi:10.1186/1471-2288-12-46
-
[14]
Gaussian-based routines to impute categorical variables in health surveys
Yucel RM, He Y, Zaslavsky AM. Gaussian-based routines to impute categorical variables in health surveys. Stat Med. Dec 20 2011;30(29):3447-60. doi:10.1002/sim.4355 21
-
[15]
Multiple imputation in the presence of non-normal data
Lee KJ, Carlin JB. Multiple imputation in the presence of non-normal data. Stat Med. Feb 20 2017;36(4):606-617. doi:10.1002/sim.7173
-
[16]
Impact of socioeconomic status measures on hospital profiling in New York City
Blum AB, Egorova NN, Sosunov EA, et al. Impact of socioeconomic status measures on hospital profiling in New York City. Circ Cardiovasc Qual Outcomes. May 2014;7(3):391-7. doi:10.1161/CIRCOUTCOMES.113.000520
-
[17]
Hong YR, Mainous AG, 3rd. Development and Validation of a County-Level Social Determinants of Health Risk Assessment Tool for Cardiovascular Disease. Ann Fam Med. Jul 2020;18(4):318-325. doi:10.1370/afm.2534
-
[18]
Bhavsar NA, Gao A, Phelan M, Pagidipati NJ, Goldstein BA. Value of Neighborhood Socioeconomic Status in Predicting Risk of Outcomes in Studies That Use Electronic Health Record Data. JAMA Netw Open. Sep 7 2018;1(5):e182716. doi:10.1001/jamanetworkopen.2018.2716
-
[19]
Flexible Imputation of Missing Data
van Buuren S. Flexible Imputation of Missing Data. Chapman and Hall; 2018
2018
-
[20]
Multiple imputation using chained equations: Issues and guidance for practice
White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med. Feb 20 2011;30(4):377-99. doi:10.1002/sim.4067
-
[21]
RUBIN DB. Inference and missing data. Biometrika. 1976;63(3):581-592. doi:10.1093/biomet/63.3.581
-
[22]
An introduction to statistical learning : with applications in R
Gareth James DWTHRT. An introduction to statistical learning : with applications in R. New York : Springer, [2013] ©2013; 2013
2013
-
[23]
Multiple Imputation for Missing Data via Sequential Regression Trees
Burgette LF, Reiter JP. Multiple Imputation for Missing Data via Sequential Regression Trees. American Journal of Epidemiology. 2010;172(9):1070-1076. doi:10.1093/aje/kwq260
-
[24]
Akande O, Li F, Reiter J. An Empirical Comparison of Multiple Imputation Methods for Categorical Data. The American Statistician. 2017/04/03 2017;71(2):162-170. doi:10.1080/00031305.2016.1277158
work page Pith review doi:10.1080/00031305.2016.1277158 2017
-
[25]
Multiple imputation with large data sets: a case study of the Children's Mental Health Initiative
Stuart EA, Azur M, Frangakis C, Leaf P. Multiple imputation with large data sets: a case study of the Children's Mental Health Initiative. Am J Epidemiol. May 1 2009;169(9):1133-9. doi:10.1093/aje/kwp026
-
[26]
Diagnostics for multivariate imputations
Abayomi K, Gelman A, Levy M. Diagnostics for multivariate imputations. Applied Statistics. 01/01 2005;57:273-291
2005
-
[27]
Multiple Imputation for Nonresponse in Surveys
Campion WM, Rubin DB. Multiple Imputation for Nonresponse in Surveys. Journal of Marketing Research. 1989;26:485
1989
-
[28]
The Multiple Adaptations of Multiple Imputation
Reiter JP, Raghunathan TE. The Multiple Adaptations of Multiple Imputation. Journal of the American Statistical Association. 2007/12/01 2007;102(480):1462-1471. doi:10.1198/016214507000000932
-
[29]
Little RJ. Missing Data Analysis. Annu Rev Clin Psychol. Jul 2024;20(1):149-173. doi:10.1146/annurev-clinpsy-080822-051727
-
[30]
Mainzer RM, Moreno-Betancur M, Nguyen CD, Simpson JA, Carlin JB, Lee KJ. Gaps in the usage and reporting of multiple imputation for incomplete data: findings from a scoping review of observational studies addressing causal questions. BMC Med Res Methodol. Sep 4 2024;24(1):193. doi:10.1186/s12874-024-02302-6 22
-
[31]
A practical guide to multiple imputation of missing data in nephrology
Blazek K, van Zwieten A, Saglimbene V, Teixeira-Pinto A. A practical guide to multiple imputation of missing data in nephrology. Kidney Int. Jan 2021;99(1):68-74. doi:10.1016/j.kint.2020.07.035
-
[32]
Cao Y, Allore H, Vander Wyk B, Gutman R. Review and evaluation of imputation methods for multivariate longitudinal data with mixed-type incomplete variables. Stat Med. Dec 30 2022;41(30):5844-5876. doi:10.1002/sim.9592
-
[33]
Hunt NB, Gardarsdottir H, Bazelier MT, Klungel OH, Pajouheshnia R. A systematic review of how missing data are handled and reported in multi-database pharmacoepidemiologic studies. Pharmacoepidemiol Drug Saf. Jul 2021;30(7):819-826. doi:10.1002/pds.5245
-
[34]
Multiple Imputation with Massive Data: An Application to the Panel Study of Income Dynamics
Si Y, Heeringa S, Johnson D, et al. Multiple Imputation with Massive Data: An Application to the Panel Study of Income Dynamics. J Surv Stat Methodol. Feb 2023;11(1):260-283. doi:10.1093/jssam/smab038
-
[35]
Missing Data Imputation in the Electronic Health Record Using Deeply Learned Autoencoders
Beaulieu-Jones BK, Moore JH. Missing Data Imputation in the Electronic Health Record Using Deeply Learned Autoencoders. Pac Symp Biocomput. 2017;22:207-218. doi:10.1142/9789813207813_0021
-
[36]
A multi-step approach to managing missing data in time and patient variant electronic health records
Cesare N, Were LPO. A multi-step approach to managing missing data in time and patient variant electronic health records. BMC Res Notes. Feb 17 2022;15(1):64. doi:10.1186/s13104-022-05911-w
-
[37]
Imputation of Missing Data in Electronic Health Records Based on Patients' Similarities
Jazayeri A, Liang OS, Yang CC. Imputation of Missing Data in Electronic Health Records Based on Patients' Similarities. J Healthc Inform Res. Sep 2020;4(3):295-307. doi:10.1007/s41666-020-00073-5
-
[38]
Gravesteijn BY, Sewalt CA, Venema E, Nieboer D, Steyerberg EW, Collaborators C-T. Missing Data in Prediction Research: A Five-Step Approach for Multiple Imputation, Illustrated in the CENTER-TBI Study. J Neurotrauma. Jun 1 2021;38(13):1842-1857. doi:10.1089/neu.2020.7218
-
[39]
Bondarenko I, Raghunathan T. Graphical and numerical diagnostic tools to assess suitability of multiple imputations and imputation models. Stat Med. Jul 30 2016;35(17):3007-20. doi:10.1002/sim.6926
-
[40]
Rizopoulos D, Verbeke G, Molenberghs G. Multiple-imputation-based residuals and diagnostic plots for joint models of longitudinal and survival outcomes. Biometrics. Mar 2010;66(1):20-9. doi:10.1111/j.1541-0420.2009.01273.x
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.