REVIEW 2 major objections 8 minor 36 references
Comparing Imputation Methods for Clinical Prediction Model Development under Complex Missingness Scenarios: A Simulation Study Using Real-World Cardiac Data
T0 review · 2 major / 8 minor · reviewed 2026-07-09 · glm-5.2
Pith's one-line read Simple kNN beats MICE for missing clinical data at high rates
desk verdict Solid simulation study on imputation methods for clinical prediction models; the kNN advantage at extreme missingness is real but partly mechanical and needs careful interpretation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the simulation pipeline itself: a fully observed real-world cardiac dataset is split into development and validation cohorts, missingness is artificially induced under MAR using multivariate amputation across 18 scenarios, five imputation methods are applied, logistic regression models with backward stepwise elimination are developed on bootstrap samples, and performance is evaluated using optimism-corrected AUC, calibration slope, mean absolute prediction error (MAPE), external validation, and computation time. The comparison of these metrics across methods and scenarios is the mechanism that carries the argument.
What would settle it
If a different real-world clinical dataset with naturally occurring missingness (rather than artificially induced) showed that MICE methods maintained superior prediction stability and external validation performance compared to kNN in non-linear or extreme missingness scenarios, the paper's central claim would be undermined.
Extended reading notes
Core claim
The central discovery is that the relative performance of imputation methods depends critically on the structure of the missing data. When missingness is isolated to linear or categorical variables at moderate levels (30-60%), all methods perform comparably. However, when missingness involves non-linear variables or reaches 90%, MICE methods show increased prediction instability and overfitting, missForest overfits substantially in external validation, while kNN maintains the most consistent performance with the shortest computation time. This suggests that the complexity of the imputation method is not always beneficial, and a simpler deterministic approach may be preferable in complex, non
Load-bearing premise
The study assumes that the artificial missingness generation process, which creates missing data under a Missing at Random (MAR) mechanism with specific weight configurations, accurately reflects real-world clinical missingness patterns. If real-world missingness follows different mechanisms, such as Missing Not at Random (MNAR), the relative performance of the imputation methods may not generalize.
Editorial extensions
If this is right
- Clinical prediction model developers may consider kNN imputation as a default strategy when dealing with large datasets and complex or extreme missingness, rather than defaulting to MICE.
- The finding that method performance depends on variable characteristics (linear vs. non-linear) suggests that imputation strategy selection should be guided by exploratory data analysis of the missing variables' relationships.
- The computational efficiency of kNN (2-5x faster than MICE) could enable more rapid model development and validation cycles in clinical research settings with large datasets.
- The observed overfitting of missForest in external validation under non-linear scenarios suggests caution when using machine learning imputation methods that do not account for imputation uncertainty.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This simulation study compares five missing-data strategies (complete case analysis, MICE-FCS, MICE-PMM, missForest, and kNN) for clinical prediction model development using a real-world cardiac cohort of 8,245 patients. Missingness was artificially induced under a missing-at-random (MAR) mechanism across 18 scenarios varying by variable type, predictor-outcome relationship, and missingness proportion (30%, 60%, 90%). The study evaluates imputed-value integrity, model-structure stability, optimism-corrected performance, prediction stability (MAPE), external validation performance, and computation time. The central finding is that while all methods perform comparably in simple scenarios at moderate missingness, kNN demonstrates the most consistent prediction stability and computational efficiency in complex non-linear scenarios at extreme (90%) missingness, where MICE methods show increased instability and missForest overfits externally. The simulation design is thorough in its scenario coverage and evaluation metrics, and the use of both internal bootstrap validation and external validation is commendable.
Significance. The study addresses a practically important question for clinical prediction model development: how imputation method choice interacts with predictor-outcome complexity and missingness severity to affect model stability and transportability. The multi-dimensional evaluation framework—spanning imputed-value fidelity, model-structure stability, individual-level prediction stability, and external validation—is a genuine strength that goes beyond standard discrimination/calibration reporting. The inclusion of computation time is practically relevant for deployment considerations. The finding that method performance depends on variable characteristics and missingness complexity is useful for practitioners. However, the significance of the central claim regarding kNN's superiority is tempered by a fundamental confound between deterministic single imputation and stochastic multiple imputation in the measurement of prediction stability, which must be addressed before the conclusions can be considered well-supported.
major comments (2)
- The central claim that kNN provides 'superior prediction stability' in complex scenarios at 90% missingness is confounded by a mechanical property: kNN is a deterministic single-imputation method, so it will necessarily show lower bootstrap-to-bootstrap variability than MICE's stochastic multiple imputation. The MAPE metric, as defined in §Model evaluation, measures the absolute difference between bootstrap-derived and original-development-model predictions; a deterministic imputation method will mechanically produce smaller such differences. More critically, Supplementary Table S3 shows that at 90% missingness in the NLinCon scenario, kNN dramatically shrinks the variance of imputed eGFR (SD=14.8 vs. CCA reference 25.1) and substantially biases the median (89.2 vs. true 72.9, per Supplementary Table S4). MICE methods preserve variance much better (SD ~24-26) and produce medians closer (
- The external validation results in Supplementary Table S7 at 90% missingness for NLinCon show that kNN achieves external AUC of 0.68 (vs. reference 0.71), while missForest achieves 0.56 and MICE-PMM achieves 0.58. While kNN is indeed the least degraded, an AUC of 0.68 represents a meaningful loss from the 0.71 reference. The manuscript's framing of kNN as providing 'better external validation results' (Abstract) and 'superior prediction stability' (Conclusion) without adequately emphasizing that all methods are substantially degraded at 90% missingness risks overstating the practical implications. The authors should explicitly state the magnitude of performance degradation relative to the complete-data reference and discuss whether 'stable but biased' imputation is genuinely preferable for clinical practice, particularly given that the biased imputed values (e.g., eGFR median overestimat
minor comments (8)
- The Declaration of Generative AI states that ChatGPT-5 and Gemini were used for language editing. ChatGPT-5 does not appear to be a released model as of the manuscript date; the authors should verify and correct this reference.
- In the Methods (§Handling of missing data), the outcome variable was excluded from imputation to prevent data leakage. While this is a defensible choice, it is at odds with standard recommendations for multiple imputation in prediction modeling (e.g., references 10, 17), where including the outcome in the imputation model is generally advised to preserve predictor-outcome associations. The rationale for this choice should be briefly discussed.
- Supplementary Table S2 describes amputation weights but the text in the Missing Data Simulation Mechanics appendix uses slightly different variable lists (e.g., 'dbp' appears in some descriptions but not others). The mapping between the prose description and the table should be made consistent.
- Figure 4 caption references 'Red dashed vertical line indicates optimal MAPE % threshold at 20%' but the text in §Model evaluation does not explicitly define a 20% MAPE threshold. This threshold should be defined in the Methods or the caption should be made more self-contained.
- The sample size justification (§Sample size calculation) cites a minimum of 1,647 participants based on 18 predictors and 13% prevalence, then uses 4,123 as approximately 2.5-fold. The rationale for choosing 2.5-fold specifically (rather than 2-fold or 3-fold) is not provided and should be briefly discussed.
- Reference 21 (Albu et al., 2025) is cited for missForest implementation but appears to describe a different package ('missForestPredict'). The original missForest reference (Stekhoven & Bühlmann 2012, ref 15) is also cited; clarification on which implementation was used would be helpful.
- In the Results §3, the text states 'the mean and SD of the linear predictor (LP) were similar across most scenarios; however, at 90% missingness in the LinMix, NLinCon, NLinMix, and All conditions, MICE, particularly MICE-FCS, produced a notably high variance of mean and SD of the LP.' Supplementary Table S5 should be referenced here, and the specific values illustrating this high variance should be cited.
- The term 'prediction stability' is used throughout but is defined somewhat indirectly via the MAPE metric. A brief, explicit definition in the Introduction or Methods (e.g., 'the degree to which a CPM produces consistent risk estimates for the same individual when developed on different samples from the same target population') would improve clarity for readers unfamiliar with the concept.
Circularity Check
No circularity found: simulation design is self-contained with independent evaluation
full rationale
This is a simulation study with a straightforward, non-circular derivation chain. The authors take a fully observed real-world cardiac dataset, artificially induce missingness under MAR via multivariate amputation, apply five standard imputation algorithms (CCA, MICE-FCS, MICE-PMM, missForest, kNN), develop logistic regression models with backward elimination, and evaluate performance on an independent external validation cohort using optimism-corrected AUC, calibration slope, MAPE, and computation time. No step in this chain reduces to its inputs by construction. Critically, the outcome variable was explicitly excluded from imputation ('the primary outcome was excluded from imputation to prevent data leakage'), so the imputed values are not fitted to the target. The performance metrics (AUC, calibration slope, MAPE) are computed on held-out or external data, not on the same data used to fit imputations. There are no self-citations that are load-bearing for the central claim—the references are to standard methodological works by independent authors (van Buuren, Stekhoven, Troyanskaya, etc.). No uniqueness theorem or ansatz is invoked. The skeptic's concern that kNN's apparent stability advantage may be mechanically driven by its deterministic single-imputation nature (variance compression producing low bootstrap-to-bootstrap variability) is a validity and confounding concern, not a circularity concern—the paper does not define 'stability' in terms of kNN's properties and then claim kNN is stable. The MAPE metric is an independent measurement of prediction differences between bootstrap and original models, computed identically for all methods. The derivation is self-contained against external benchmarks.
Assumptions & free parameters
free parameters (5)
- k in kNN =
10
- m in MICE =
10
- ntree in missForest =
50
- bootstrap iterations =
100
- amputation weights =
0.2-0.4
assumptions (3)
- domain assumption Missing at Random (MAR) assumption
- domain assumption Logistic regression is appropriate for the outcome
- domain assumption The single dataset is representative
Cite this review
Pith. "Pith review of Comparing Imputation Methods for Clinical Prediction Model Development under Complex Missingness Scenarios: A Simulation Study Using Real-World Cardiac Data." pith.science (2026). https://pith.science/paper/ZQDKXTM4
@misc{pith2026260707247,
author = {Pith},
title = {Pith review of: Comparing Imputation Methods for Clinical Prediction Model Development under Complex Missingness Scenarios: A Simulation Study Using Real-World Cardiac Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZQDKXTM4}},
note = {Machine review of arXiv:2607.07247}
}
read the original abstract
Evidence remains limited on how missing-data strategies affect the stability of clinical prediction models across different predictor-outcome relationships and degrees of missingness. We conducted a simulation study using a fully observed real-world cardiac cohort of 8,245 patients, equally divided into development and external validation cohorts. Missing data were induced under a missing-at-random mechanism across 18 scenarios varying by variable type, predictor-outcome relationship, and missingness proportion. Five strategies were compared: complete case analysis, multiple imputation by chained equations with fully conditional specification, multiple imputation using predictive mean matching, missForest, and k-nearest neighbours. Logistic regression models were developed using backward stepwise elimination. Outcomes included optimism-corrected AUC, calibration slope, mean absolute prediction error, external validation performance, and computation time. When missingness involved isolated linear or categorical variables at 30%-60%, all methods maintained discrimination comparable to the complete-data model, with median AUCs of about 0.75. When missingness involved isolated non-linear variables or more complex patterns, predictive performance and calibration worsened as missingness increased, especially at 90%. In complex scenarios, multiple imputation showed greater prediction instability and overfitting, while missForest performed well internally but overfitted externally. k-nearest neighbours showed the most consistent performance, with stable predictions, better external validation results, and the shortest computation time. The optimal strategy may depend on the characteristics of variables with missing data. In sufficiently large development samples, k-nearest neighbours may provide a computationally efficient alternative.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Riley RD, van der Windt D, Croft P, Moons KGM, editors: Oxford University Press; 2019 01 Feb 2019
Prognosis Research in Healthcare: Concepts, Methods, and Impact. Riley RD, van der Windt D, Croft P, Moons KGM, editors: Oxford University Press; 2019 01 Feb 2019
work page 2019
-
[2]
Development and validation of a prediction model with missing predictor data: a practical approach
Vergouwe Y , Royston P, Moons KG, Altman DG. Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol. 2010;63(2):205-14
work page 2010
-
[3]
Awounvo S, Kieser M, Feisst M. Combining multiple imputation with internal model validation in clinical prediction modeling: a systematic methodological review. J Clin Epidemiol. 2025;186:111916
work page 2025
-
[4]
Bias arising from missing data in predictive models
Gorelick MH. Bias arising from missing data in predictive models. J Clin Epidemiol. 2006;59(10):1115-23
work page 2006
-
[5]
Missing data should be handled differently for prediction than for description or causal explanation
Sperrin M, Martin GP, Sisk R, Peek N. Missing data should be handled differently for prediction than for description or causal explanation. (1878-5921 (Electronic))
-
[6]
Uncertainty of risk estimates from clinical prediction models: rationale, challenges, and approaches
Riley RD, Collins GS, Kirton L, Snell KI, Ensor J, Whittle R, et al. Uncertainty of risk estimates from clinical prediction models: rationale, challenges, and approaches. BMJ. 2025;388:e080749
work page 2025
-
[7]
Deforth M, Heinze G, Held U. The performance of prognostic models depended on the choice of missing value imputation algorithm: a simulation study. J Clin Epidemiol. 2024;176:111539
work page 2024
-
[8]
Royston P, Sauerbrei W. Stability of multivariable fractional polynomial models with selection of variables and transformations: a bootstrap investigation. Statistics in Medicine. 2003;22(4):639-59
work page 2003
Show all 36 references
-
[9]
On stability issues in deriving multivariable regression models
Sauerbrei W, Buchholz A, Boulesteix A-L, Binder H. On stability issues in deriving multivariable regression models. Biometrical Journal. 2015;57(4):531-55
2015
-
[10]
Combining Missing Data Imputation and Internal Validation in Clinical Risk Prediction Models
Mi J, Tendulkar RD, Sittenfeld SMC, Patil S, Zabor EC. Combining Missing Data Imputation and Internal Validation in Clinical Risk Prediction Models. Stat Med. 2025;44(18-19):e70203
2025
-
[11]
Multiple imputation of discrete and continuous data by fully conditional specification
van Buuren S. Multiple imputation of discrete and continuous data by fully conditional specification. Stat Methods Med Res. 2007;16(3):219-42
2007
-
[12]
Multiple imputation of missing data under missing at random: compatible imputation models are not sufficient to avoid bias if they are mis-specified
Curnow E, Carpenter JR, Heron JE, Cornish RP, Rach S, Didelez V , et al. Multiple imputation of missing data under missing at random: compatible imputation models are not sufficient to avoid bias if they are mis-specified. J Clin Epidemiol. 2023;160:100-9
2023
-
[13]
Comparison of techniques for handling missing covariate data within prognostic modelling studies: a simulation study
Marshall A, Altman Dg Fau - Royston P, Royston P Fau - Holder RL, Holder RL. Comparison of techniques for handling missing covariate data within prognostic modelling studies: a simulation study. (1471-2288 (Electronic))
-
[14]
Comparison of imputation methods for missing laboratory data in medicine
Waljee AK, Mukherjee A, Singal AG, Zhang Y , Warren J, Balis U, et al. Comparison of imputation methods for missing laboratory data in medicine. BMJ Open. 2013;3(8)
2013
-
[15]
MissForest—non-parametric missing value imputation for mixed-type data
Stekhoven DJ, Bühlmann P. MissForest—non-parametric missing value imputation for mixed-type data. Bioinformatics. 2012;28(1):112-8
2012
-
[16]
Missing value estimation methods for DNA microarrays
Troyanskaya O, Cantor M, Sherlock G, Brown P, Hastie T, Tibshirani R, et al. Missing value estimation methods for DNA microarrays. Bioinformatics. 2001;17(6):520-5
2001
-
[17]
Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study
Sisk R, Sperrin M, Peek N, van Smeden M, Martin GP. Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study. Stat Methods Med Res. 2023;32(8):1461-77
2023
-
[18]
Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes
Riley RD, Snell KI, Ensor J, Burke DL, Harrell Jr FE, Moons KG, et al. Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes. Statistics in Medicine. 2019;38(7):1276-96
2019
-
[19]
Generating missing values for simulation purposes: a multivariate amputation procedure
Schouten RM, Lugtig P, Vink G. Generating missing values for simulation purposes: a multivariate amputation procedure. Journal of Statistical Computation and Simulation. 2018;88(15):2909-30
2018
-
[20]
Evaluation of Four Multiple Imputation Methods for Handling Missing Binary Outcome Data in the Presence of an Interaction between a Dummy and a Continuous Variable
Javadi S, Bahrampour A, Saber MM, Garrusi B, Baneshi MR. Evaluation of Four Multiple Imputation Methods for Handling Missing Binary Outcome Data in the Presence of an Interaction between a Dummy and a Continuous Variable. Journal of Probability and Statistics. 2021;2021(1):6668822
2021
-
[21]
missForestPredict—Missing data imputation for prediction settings
Albu E, Gao S, Wynants L, Van Calster B. missForestPredict—Missing data imputation for prediction settings. PLOS ONE. 2025;20(11):e0334125
2025
-
[22]
Methods for significance testing of categorical covariates in logistic regression models after multiple imputation: power and applicability analysis
Eekhout I, van de Wiel MA, Heymans MW. Methods for significance testing of categorical covariates in logistic regression models after multiple imputation: power and applicability analysis. BMC Medical Research Methodology. 2017;17(1):129
2017
-
[23]
psfmi: Prediction Model Pooling, Selection and Performance Evaluation Across Multiply Imputed Datasets
Heymans M. psfmi: Prediction Model Pooling, Selection and Performance Evaluation Across Multiply Imputed Datasets. 2023
2023
-
[24]
Re-evaluation of the comparative effectiveness of bootstrap- based optimism correction methods in the development of multivariable clinical prediction models
Iba K, Shinozaki T, Maruo K, Noma H. Re-evaluation of the comparative effectiveness of bootstrap- based optimism correction methods in the development of multivariable clinical prediction models. BMC Medical Research Methodology. 2021;21(1):9
2021
-
[25]
Population median imputation was noninferior to complex approaches for imputing missing values in cardiovascular prediction models in clinical practice
Berkelmans GFN, Read SH, Gudbjornsdottir S, Wild SH, Franzen S, van der Graaf Y , et al. Population median imputation was noninferior to complex approaches for imputing missing values in cardiovascular prediction models in clinical practice. J Clin Epidemiol. 2022;145:70-80
2022
-
[26]
Neural Networks and the Bias/Variance Dilemma
Geman S, Bienenstock E, Doursat R. Neural Networks and the Bias/Variance Dilemma. Neural Computation. 1992;4(1):1-58
1992
-
[27]
Development and Reporting of Prediction Models: Guidance for Authors From Editors of Respiratory, Sleep, and Critical Care Journals
Leisman DE, Harhay MO, Lederer DJ, Abramson M, Adjei AA, Bakker J, et al. Development and Reporting of Prediction Models: Guidance for Authors From Editors of Respiratory, Sleep, and Critical Care Journals. (1530-0293 (Electronic))
-
[28]
Prognostic models for predicting clinical disease progression, worsening and activity in people with multiple sclerosis
Reeve K, On BI, Havla J, Burns J, Gosteli-Peter MA, Alabsawi A, et al. Prognostic models for predicting clinical disease progression, worsening and activity in people with multiple sclerosis. (1469-493X (Electronic))
-
[29]
Missing Data in Clinical Research: A Tutorial on Multiple Imputation
Austin PC, White IR, Lee DS, van Buuren S. Missing Data in Clinical Research: A Tutorial on Multiple Imputation. (1916-7075 (Electronic))
1916
-
[30]
Developing prediction models for clinical use using logistic regression: an overview
Shipe ME, Deppen SA, Farjah F, Grogan EL. Developing prediction models for clinical use using logistic regression: an overview. (2072-1439 (Print))
-
[31]
Missing data and prediction: the pattern submodel
Fletcher Mercaldo S, Blume JD. Missing data and prediction: the pattern submodel. Biostatistics. 2020;21(2):236-52. Supplementary Appendix Supplementary Table S1. Baseline characteristics of initial model predictors in development and validation cohort. Supplementary Table S2....
2020
-
[32]
Complete Case Analysis (CCA): As a reference standard for missing data mishandling, we performed a strict CCA by omitting any patient record with one or more missing values. To prevent perfect separation and zero-variance errors (singular Hessian matrices) during model fitting...
-
[33]
Just Another Variable
Multiple Imputation by Chained Equations with Fully Conditional Specification (MICE-FCS): Parametric multiple imputation was performed using standard FCS. Binary variables (e.g., sex, diabetes, hypertension) were imputed via logistic regression (logreg), and continuous variabl...
-
[34]
Predictive Mean Matching (pmm) was applied to both continuous and categorical variables, identifying the 10 closest donors for imputation
Multiple Imputation with Predictive Mean Matching (MICE-PMM): To account for potential non-normality in the clinical variables, a semi-parametric MICE approach was utilized. Predictive Mean Matching (pmm) was applied to both continuous and categorical variables, identifying th...
-
[35]
To rigorously prevent data leakage during model training, the primary outcome was deliberately excluded from the imputation predictor pool
Non-Parametric Iterative Random Forest (missForest): A machine-learning imputation approach was implemented using the missForest algorithm, utilizing 50 trees (ntree = 50) and a maximum of 10 iterations. To rigorously prevent data leakage during model training, the primary out...
-
[36]
Missing continuous variables were imputed using the median of the neighbors, while missing categorical variables were imputed using the maximum category (mode)
k-Nearest Neighbors Imputation (kNN): A distance-based single imputation method was applied using k = 10 neighbors. Missing continuous variables were imputed using the median of the neighbors, while missing categorical variables were imputed using the maximum category (mode). ...
1927
Reviewed July 9, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.