Pith. sign in

REVIEW 2 major objections 5 minor 31 references

Combining missing data imputation and internal validation in clinical risk prediction models

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For clinical risk prediction models with missing covariates, bootstrapping before deterministic imputation and imputing all missing values yields the least biased estimates of model performance and individual risk predictions.

desk verdict Useful tutorial, but the simulation's headline comparison of BI versus CC is distorted by excluding non-convergent CC datasets, so 'least biased' is not yet established. read the letter →

arxiv 2411.14542 v1 pith:ZLYYCMWL submitted 2024-11-21 stat.ME stat.AP

Junhui Mi , Rahul D. Tendulkar , Sarah M. C. Sittenfeld , Sujata Patil , Emily C. Zabor This is my paper · ORCID
classification stat.MEstat.AP
keywords clinicalriskpredictionmissingdatadeterministicimputationbootstrapvalidationcompletecaseanalysisAUCBrierscoreinternal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Clinical risk prediction models routinely face missing covariate data, but the usual missing-data toolbox was built for estimation, not for models that must predict for future patients. The paper proposes that when the goal is prediction, the right order is to draw bootstrap samples first, then fill missing covariate values with a single deterministic imputation model that excludes the outcome, and only then fit and evaluate the model. A simulation study based on a real breast cancer cohort, covering 54 scenarios at sample sizes 750 and 3,500, finds that this bootstrap-then-impute-all workflow yields less biased estimates of the AUC (how well predicted risk separates later from earlier events) and the Brier score (the mean squared error of predicted risks), as well as less biased individual risk predictions, than complete-case analysis or selective imputation. A step-by-step tutorial with R code accompanies the recommendation. If the claim is right, clinical prediction model developers should adopt this workflow for internal validation whenever missingness is plausibly at random.

What carries the argument

The central mechanism is the bootstrap-then-impute pipeline: each bootstrap resample is imputed independently before model fitting, so the imputation model is re-estimated within every resample and no information from the original development data is used to impute values in a bootstrap sample. The imputation step is deterministic regression imputation, in which each missing covariate is regressed on the other complete covariates using a generalized linear model and the missing values are replaced by the model's predicted response, thresholded for binary variables; the outcome is deliberately left out of the imputation model. The pipeline is evaluated through the apparent, bootstrap-corrected, .632, and .632+ estimators of the AUC and Brier score, and the paper's simulation asks which missing-data strategy makes these estimators least biased relative to the full-data values.

What would settle it

Rerun the paper's 54-scenario simulation with missingness generated so that the probability a value is missing depends on the unobserved value itself (missing not at random) and compare the bias of the bootstrap-imputation and complete-case estimators; if complete-case analysis has lower bias in the majority of high-missingness scenarios, the paper's blanket recommendation would not survive.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that the order of resampling and imputation is the decision that controls bias in internal validation of prediction models with missing covariates. The recommended procedure is to bootstrap the original dataset, impute missing covariate values separately inside each bootstrap sample using deterministic regression imputation, fit the multivariable prediction model in each imputed bootstrap sample, and evaluate it in the original imputed data. Across the simulation, imputing all missing values in this bootstrap-first sequence gave the smallest bias relative to full-data performance for the AUC, the Brier score, and average predicted probability at five years, compared with complete-case analysis and with two more selective imputation strategies. The paper also argues that deterministic imputation is preferable to multiple imputation in this setting because omitting the outcome from the imputation model avoids bias in estimated coefficients and lets the fitted imputation models be applied directly to future patients with unknown outcomes.

Load-bearing premise

The comparison assumes missingness is missing at random, with the chance a value is missing depending only on observed covariates, and that the generalized-linear imputation models are correctly specified; if missingness depends on unobserved values or the imputation models are badly wrong, complete-case analysis could become relatively more attractive.

Editorial extensions

If this is right

  • A clinical prediction model built this way can be deployed as a risk calculator: the imputation models and the prediction model are both fixed functions of observed covariates, so a future patient's missing values can be filled without access to the original development dataset.
  • Complete-case analysis is not just more biased but can fail altogether in small samples with high missingness; the simulation records fitting failures in up to 57.7% of datasets for the most extreme patterns.
  • Even with low missingness (5% in a single covariate), bootstrap-imputation produced lower bias than complete-case analysis, so the benefit is not limited to severe missingness.
  • Among imputation strategies, imputing all missing values was at least as good as imputing only variables with high missingness or only participants with few missing variables, so the simpler 'impute everything' rule is the preferred default.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the simulation to a missing-not-at-random mechanism would be a natural stress test; if missingness depends on the unobserved value itself, the relative advantage of bootstrap-imputation over complete-case analysis could shrink or reverse, and the paper's guidance would need qualification.
  • The leakage argument likely generalizes: any validation scheme that resamples before a deterministic imputer is fitted should protect against optimistic estimates, but the paper only evaluates generalized linear model imputation, so the size of the benefit with median, nearest-neighbor, or tree-based imputers is an open question.
  • Because the simulation parameters come from a single breast cancer cohort, a practitioner in another disease should run a small local simulation with plausible covariate correlations and missingness rates to confirm the expected bias reduction before adopting the workflow.
  • The same bootstrap-before-imputation ordering should matter whenever model-building includes any data-dependent preprocessing step (normalization, feature selection, tuning), not only imputation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper is a tutorial on combining bootstrap resampling with deterministic (regression) imputation for developing and internally validating clinical risk prediction models when covariates are missing. The authors argue that deterministic imputation is preferable to multiple imputation in the prediction setting because the outcome is excluded from the imputation model and the fitted imputation models can be applied to future patients. They provide R code for a guided example and an extensive simulation study: 54 scenarios defined by two sample sizes (750 and 3,500), nine missing-data patterns, and three imputation strategies, with 1,000 simulated datasets per scenario. The simulation compares complete-case analysis (CC) with bootstrap-then-impute (BI) in terms of bias of apparent, bootstrap-corrected, .632, and .632+ estimates of the AUC and Brier score, and bias of average individual predicted risks at five years. The central claim is that BI with imputation of all missing values yields the least biased estimates of model performance and individual risk predictions. The manuscript is clearly written and the tutorial component is practical, but the simulation evidence for the headline claim is weakened by the handling of CC model-fitting failures.

Significance. If the central claim is correct, the paper addresses an important practical question: whether to bootstrap before or after imputation when internally validating a prediction model with missing covariates, and whether to prefer deterministic imputation over complete-case analysis. The tutorial and accompanying R code are concrete and reproducible, and the simulation covers a wide range of missingness patterns, which is a genuine strength. The paper also gives useful attention to the deployment setting, where imputation must be applicable to future patients with unknown outcomes. However, the main conclusion is currently supported almost entirely by a simulation comparison that excludes a large, non-random fraction of datasets from the CC arm in high-missingness scenarios. The paper also restricts attention to a single data-generating process calibrated to one breast-cancer dataset, under missing-at-random mechanisms with correctly specified imputation models, so the quantitative claims should be interpreted as illustrative rather than general. The tutorial remains useful independent of the simulation result.

major comments (2)
  1. [Section 5] The CC bias distributions in Figures 1-5 are computed only over simulated datasets where the multivariable Cox model was successfully fitted, as the manuscript states in the paragraph beginning 'When the sample size is 750 and the approach is CC'. For missing-data pattern I at n=750 this excludes 57.7% of datasets, and smaller exclusions occur for patterns C, F, and H. The BI results are not subject to the same failure filter. Because convergence failure is likely to be concentrated in the very sparse or heavily missing datasets where CC bias could be largest, the comparison is not between two methods on a common set of datasets. The conclusion that 'CC always has more bias' and that BI gives the least biased estimates is therefore not supported as reported. Please re-run the comparison on the subset of datasets where both CC and BI fit, report the CC failure rates as a separate outcome, or use a penalized or Firth-type Cox fit so that the CC arm is estimable on essentially all datasets.
  2. [Section 3.4 and Section 5] The deterministic imputation implementation uses, for each missing variable, a generalized linear model with 'all other complete variables' as predictors, excluding partially observed covariates. This means that informative partially observed variables, including variables that appear in the missingness mechanisms of Table 3, are never used in the imputation models. The simulation therefore evaluates one specific implementation of deterministic imputation, and the paper's general recommendation for 'deterministic imputation' is correspondingly narrow. A sensitivity analysis using sequential imputation or including all observed portions of partially missing covariates would substantially strengthen the external validity of the 'least biased' claim; without it, the recommendation should be explicitly limited to the implementation studied here.
minor comments (5)
  1. [Section 4] The R code in the guided example contains an error: `map(1:500, ~ slice_sample(dat), prop = 1, replace = TRUE))` should be `map(1:500, ~ slice_sample(dat, prop = 1, replace = TRUE))`, and the duplicate `library(purrr)` call should be removed.
  2. [Section 3.3] The notation in the missingness model, especially `M_{ik_j}`, `X_{il_j}`, and the condition `k_j \neq l_j`, is hard to follow. Please define the indices more explicitly and use notation that does not confuse the covariate index with the observation index.
  3. [Section 5] The phrase 'bias ... calculated as the difference between the performance metric on the full data and the performance metric employing CC and BI approaches' should state explicitly that 'full data' means the simulated dataset before any missingness was imposed, to avoid ambiguity with the complete-case subset.
  4. [Figures 1-5] The figure captions refer to 'panels' without identifying which estimator (apparent, bootstrap-corrected, .632, .632+) is shown in each panel; adding explicit panel titles or labels would make the figures much easier to read.
  5. [Section 5 and Discussion] The claim that BI gives the 'least biased' estimates is made relative to CC and to two alternative imputation thresholds; the paper does not compare BI to multiple imputation in the simulation. The Discussion should make this scope explicit so that readers do not infer a direct simulation comparison with multiple imputation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the simulation study is a self-contained comparison against full-data truth; the only self-citations supply simulation inputs, not the argument's conclusion.

full rationale

The paper's central claim, that bootstrapping followed by deterministic imputation of all missing values gives the least biased estimates of AUC, Brier score, and individual risk predictions, is supported by a simulation study in which data are generated from known parameter values, missingness is imposed, and estimators are compared against the full-data truth. This is a standard self-contained design. The derivation does not define a performance measure in terms of the method being evaluated, nor does it fit a parameter and then rename the fit as a prediction. The only self-citations, the motivating breast cancer dataset ([23]) and a prior online-tools paper ([31]), provide simulation parameters and background context; they are not invoked as authority for the central methodological comparison. The paper openly acknowledges a relevant statistical limitation: for complete-case analysis, model fitting failed in a substantial fraction of high-missingness datasets (e.g., 57.7% for pattern I at n=750), and the figures are restricted to successfully fitted datasets. That is a correctness or generalizability concern about the comparison, not a circularity, because the reported bias is still computed against independent full-data truth rather than against an input of the same method. No equation is equivalent by construction to another, and no self-citation chain forces the conclusion. The analysis is therefore not circular.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim is supported by a simulation study whose conclusions are conditional on a specific data-generating process and imputation model specification. The free parameters listed are simulation inputs estimated or chosen by the authors; they are not independent external benchmarks, so the breadth of the recommendation is limited by their choices.

free parameters (4)
  • Weibull shape and scale for death times = shape=1.6, scale=122
    Estimated from the complete-case motivating dataset and used to simulate survival times; the simulation results may depend on these values.
  • Weibull shape and scale for censoring times = shape=2.6, scale=8.2
    Estimated from the full motivating dataset with no covariates; used to generate censoring times.
  • True hazard ratios for 11 covariates = Table 1, e.g., X1=0.80, X2=1.05
    Chosen or estimated from the motivating dataset to define the true risk model; the relative performance of methods could vary with different true coefficients.
  • Missingness mechanism parameters and proportions = Tables 3 and 4
    Specified to reflect real-world missingness patterns; the MAR logistic mechanisms and marginal/joint proportions shape the simulation conclusions.
assumptions (3)
  • domain assumption Missing at random (MAR) assumption: missingness probability depends only on observed covariates, not on unobserved values.
    Section 3.3 states missingness is imposed under MAR; the recommendation's validity is assessed only under this mechanism.
  • domain assumption Correctly specified imputation models: each missing variable is modeled as a linear or logistic function of all complete predictors.
    Section 3.4 fits deterministic regression imputation models with these forms; if the true relationship is nonlinear or involves missingness of other variables, the imputed values would be biased.
  • standard math Bootstrap optimism correction yields valid internal validation for the model-building process.
    Section 2.2 relies on Efron's bootstrap optimism method; this is a standard statistical assumption but requires the bootstrap to approximate the sampling distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Combining missing data imputation and internal validation in clinical risk prediction models." pith.science (2026). https://pith.science/paper/ZLYYCMWL

@misc{pith2026241114542,
  author       = {Pith},
  title        = {Pith review of: Combining missing data imputation and internal validation in clinical risk prediction models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLYYCMWL}},
  note         = {Machine review of arXiv:2411.14542}
}
read the original abstract

Methods to handle missing data have been extensively explored in the context of estimation and descriptive studies, with multiple imputation being the most widely used method in clinical research. However, in the context of clinical risk prediction models, where the goal is often to achieve high prediction accuracy and to make predictions for future patients, there are different considerations regarding the handling of missing data. As a result, deterministic imputation is better suited to the setting of clinical risk prediction models, since the outcome is not included in the imputation model and the imputation method can be easily applied to future patients. In this paper, we provide a tutorial demonstrating how to conduct bootstrapping followed by deterministic imputation of missing data to construct and internally validate the performance of a clinical risk prediction model in the presence of missing data. Extensive simulation study results are provided to help guide decision-making in real-world applications.

Figures

Figures reproduced from arXiv: 2411.14542 by the authors.

Figure 1
Figure 1. Distribution of bias (y-axis) of AUC estimators (panels) comparing the CC approach and [PITH_FULL_IMAGE:figures/full_fig_p027_1.png] view at source ↗
Figure 2
Figure 2. Distribution of bias (y-axis) of AUC estimators (panels) comparing the CC approach and [PITH_FULL_IMAGE:figures/full_fig_p028_2.png] view at source ↗
Figure 3
Figure 3. Distribution of bias (y-axis) of Brier score estimators (panels) comparing the CC approach [PITH_FULL_IMAGE:figures/full_fig_p029_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of bias (y-axis) of Brier score estimators (panels) comparing the CC approach [PITH_FULL_IMAGE:figures/full_fig_p030_4.png]
Figure 5
Figure 5. Figure 5: Distribution of average individual prediction bias (y-axis) by sample size (columns) for [PITH_FULL_IMAGE:figures/full_fig_p031_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 29 canonical work pages

  1. [1]

    D. B. Rubin. Inference and missing data. Biometrika, 63(3):581–590, 1976

    work page 1976

  2. [2]

    D. B. Rubin. Multiple imputation after 18+ years. Journal of the American Statistical Asso- ciation, 91(434):473–489, 1996

    work page 1996

  3. [3]

    Marshall, D

    A. Marshall, D. G. Altman, R. L. Holder, and P. Royston. Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Med Res Methodol, 9:57, 2009. 20

    work page 2009

  4. [4]

    Marshall, D

    A. Marshall, D. G. Altman, P. Royston, and R. L. Holder. Comparison of techniques for handling missing covariate data within prognostic modelling studies: a simulation study. BMC Med Res Methodol, 10:7, 2010

    work page 2010

  5. [5]

    P. C. Austin, I. R. White, D. S. Lee, and S. van Buuren. Missing data in clinical research: A tutorial on multiple imputation. Can J Cardiol, 37(9):1322–1331, 2021

    work page 2021

  6. [6]

    mice: Multivariate imputation by chained equations in r

    Stef van Buuren and Karin Groothuis-Oudshoorn. mice: Multivariate imputation by chained equations in r. Journal of Statistical Software, 45(3):1–67, 2011

    2011

  7. [7]

    R. J. Little, R. D’Agostino, M. L. Cohen, K. Dickersin, S. S. Emerson, J. T. Farrar, C. Fran- gakis, J. W. Hogan, G. Molenberghs, S. A. Murphy, J. D. Neaton, A. Rotnitzky, D. Scharfstein, W. J. Shih, J. P. Siegel, and H. Stern. The prevention and treatment of missing data in clinical trials. N Engl J Med, 367(14):1355–60, 2012

    work page 2012

  8. [8]

    Sperrin, G

    M. Sperrin, G. P. Martin, R. Sisk, and N. Peek. Missing data should be handled differently for prediction than for description or causal explanation. J Clin Epidemiol, 125:183–187, 2020

    work page 2020

Show all 31 references
  1. [9]

    D’Agostino McGowan, S

    L. D’Agostino McGowan, S. C. Lotspeich, and S. A. Hepler. The ”why” behind including ”y” in your imputation model. Stat Methods Med Res, page 9622802241244608, 2024

    2024

  2. [10]

    Fletcher Mercaldo and J

    S. Fletcher Mercaldo and J. D. Blume. Missing data and prediction: the pattern submodel. Biostatistics, 21(2):236–252, 2020

    2020

  3. [11]

    R. Sisk, M. Sperrin, N. Peek, M. van Smeden, and G. P. Martin. Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study. Stat Methods Med Res, 32(8):1461–1477, 2023. 21

    2023

  4. [12]

    E. W. Steyerberg and Jr. Harrell, F. E. Prediction models need appropriate internal, internal- external, and external validation. J Clin Epidemiol, 69:245–7, 2016

    2016

  5. [13]

    J. W. Bartlett and R. A. Hughes. Bootstrap inference for multiple imputation under unconge- niality and misspecification. Stat Methods Med Res, 29(12):3533–3546, 2020

    2020

  6. [14]

    Schomaker and C

    M. Schomaker and C. Heumann. Bootstrap inference when using multiple imputation. Stat Med, 37(14):2252–2266, 2018

    2018

  7. [15]

    R: A Language and Environment for Statistical Computing

    R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2024

    2024

  8. [16]

    E. W. Steyerberg, A. J. Vickers, N. R. Cook, T. Gerds, M. Gonen, N. Obuchowski, M. J. Pencina, and M. W. Kattan. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology, 21(1):128–38, 2010

    2010

  9. [17]

    S. Wahl, A. L. Boulesteix, A. Zierer, B. Thorand, and M. A. van de Wiel. Assessment of predic- tive performance in incomplete data by combining internal validation and multiple imputation. BMC Med Res Methodol, 16(1):144, 2016

    2016

  10. [18]

    K. Iba, T. Shinozaki, K. Maruo, and H. Noma. Re-evaluation of the comparative effectiveness of bootstrap-based optimism correction methods in the development of multivariable clinical prediction models. BMC Med Res Methodol, 21(1):9, 2021

    2021

  11. [19]

    Harrell, F

    Jr. Harrell, F. E., K. L. Lee, and D. B. Mark. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med, 15(4):361–87, 1996. 22

    1996

  12. [20]

    B. Efron. Estimating the error rate of a prediction rule - improvement on cross-validation. Journal of the American Statistical Association, 78(382):316–331, 1983

    1983

  13. [21]

    Efron and R

    B. Efron and R. Tibshirani. Improvements on cross-validation: The .632+ bootstrap method. Journal of the American Statistical Association, 92(438):548–560, 1997

    1997

  14. [22]

    Gerds and M.W

    T. Gerds and M.W. Kattan. Medical Risk Prediction Models: With Ties to Machine Learning. Chapman and Hall/CRC, 1st edition, 2021

    2021

  15. [23]

    S. M. C. Sittenfeld, E. C. Zabor, S. N. Hamilton, H. M. Kuerer, M. El-Tamer, G. E. Naoum, P. T. Truong, A. Nichol, B. D. Smith, W. A. Woodward, T. A. Moo, S. N. Powell, C. S. Shah, A. G. Taghian, I. Abu-Gheida, and R. D. Tendulkar. A multi-institutional prediction model to est...

    2022

  16. [24]

    van Buuren

    S. van Buuren. Flexible Imputation of Missing Data. Chapman and Hall/CRC, second edition edition, 2018

    2018

  17. [25]

    I. R. White, P. Royston, and A. M. Wood. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med, 30(4):377–99, 2011

    2011

  18. [26]

    purrr: Functional Programming Tools, 2023

    Hadley Wickham and Lionel Henry. purrr: Functional Programming Tools, 2023

    2023

  19. [27]

    Hadley Wickham, Romain Fran¸ cois, Lionel Henry, Kirill M¨ uller, and Davis Vaughan.dplyr: A Grammar of Data Manipulation, 2023

    2023

  20. [28]

    riskRegression: Risk Regression Models and Prediction Scores for Survival Analysis with Competing Risks, 2023

    Thomas Alexander Gerds, Johan Sebastian Ohlendorff, and Brice Ozenne. riskRegression: Risk Regression Models and Prediction Scores for Survival Analysis with Competing Risks, 2023. 23

    2023

  21. [29]

    A Package for Survival Analysis in R, 2024

    Terry M Therneau. A Package for Survival Analysis in R, 2024

    2024

  22. [30]

    Therneau and Patricia M

    Terry M. Therneau and Patricia M. Grambsch. Modeling Survival Data: Extending the Cox Model. Springer, New York, 2000

    2000

  23. [31]

    E. C. Zabor, D. Coit, J. E. Gershenwald, K. M. McMasters, J. S. Michaelson, A. J. Stromberg, and K. S. Panageas. Variability in predictions from online tools: A demonstration using internet-based melanoma predictors. Ann Surg Oncol, 25(8):2172–2177, 2018. 1534-4681 Za- bor, Em...

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.