{"id":"13bce357-b9fe-4932-a23a-fe14e1f8bcbf","arxiv_id":"2411.14542","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Bootstrapping followed by deterministic imputation, with all missing values imputed and the outcome excluded, produces less biased AUC, Brier score, and individual risk predictions than complete case analysis in simulated clinical prediction settings.","lead":"This paper presents a tutorial and simulation study for handling missing predictor data when building clinical risk prediction models. It recommends bootstrapping the data first, then applying deterministic imputation without the outcome, and shows this reduces bias compared with complete case analysis in simulated scenarios.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Simulation evidence for BI over CC is distorted: the CC bias distribution excludes datasets where CC failed to converge (e.g., 57.7% of pattern I at n=750), so the headline 'least biased' may not hold on a common subset.","rationale":"The reader's verdict is CONDITIONAL with MODERATE confidence, and our concern reinforces that conditionality without overturning the tutorial's practical value. The simulation is the only quantitative evidence for the strongest claim. The selective exclusion is acknowledged in Section 5, but its consequences are not discussed; the reported bias distributions are not comparable across methods because CC is evaluated only where it converges while BI is not. This is distinct from the reader's weakest_assumption regarding MAR and generalizability, so agreement is partial. The proposed common-subset test is feasible with the provided GitHub code and would settle whether the headline result is an artifact of comparing incomparable subsets. If the test shows BI still wins, the conclusion can stand with a caveat; if not, the central claim should be weakened. Other issues, such as the absence of a multiple imputation comparator and the use of aggregate rather than individual prediction error, are secondary to this internal-validity threat.","tokens_in":11795,"tokens_out":11910,"duration_ms":124610,"concrete_test":"Reanalyze the stored simulation results for missing pattern I at n=750. Restrict both CC and BI to the 423 datasets in which the CC Cox model converged, and compute the median and interquartile range of bias for apparent, bootstrap-corrected, .632, and .632+ AUC and Brier, plus mean individual prediction bias, for CC and for BI impute-all. If BI is still uniformly less biased within this common subset, the selective exclusion is not the driver and the claim survives; if the BI advantage shrinks, disappears, or reverses, the paper must either add a penalized Cox (e.g., Firth) so CC can be fit in all datasets, or explicitly restrict the conclusion to settings where CC is feasible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that bootstrap-then-impute-all (BI) gives the least biased estimates of AUC, Brier score, and individual risk predictions. This is supported almost entirely by the Section 5 simulation comparison of BI versus complete-case (CC) analysis. However, the simulation text states that 'the figures are based on results from simulated datasets where the multivariable model of interest was successfully fitted.' For CC at n=750, fitting failed in 0.3%, 1.2%, 0.4%, and 57.7% of datasets for patterns C, F, H, and I; at n=3500, pattern I failed in 0.3%. The CC bias is therefore computed on a non-random subset of the simulated data, while the BI results are not subject to the same failure filter. Because Cox model convergence depends on the amount of missingness, event sparsity, and covariate structure, the subset of CC-convergent datasets is not exchangeable with the full set. The comparison 'CC vs BI' is thus not a comparison of two methods on the same data, and the reported 'least biased' conclusion for high-missingness settings is not yet established. The direction of the distortion is unknown: if CC tends to fail in datasets where it would have had large bias, excluding those datasets flatters CC; if it fails in datasets where it would have had small bias, it flatters BI. Either way, the headline claim is not supported by the reported analysis as it stands.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a tutorial on combining bootstrap resampling with deterministic (regression) imputation for developing and internally validating clinical risk prediction models when covariates are missing. The authors argue that deterministic imputation is preferable to multiple imputation in the prediction setting because the outcome is excluded from the imputation model and the fitted imputation models can be applied to future patients. They provide R code for a guided example and an extensive simulation study: 54 scenarios defined by two sample sizes (750 and 3,500), nine missing-data patterns, and three imputation strategies, with 1,000 simulated datasets per scenario. The simulation compares complete-case analysis (CC) with bootstrap-then-impute (BI) in terms of bias of apparent, bootstrap-corrected, .632, and .632+ estimates of the AUC and Brier score, and bias of average individual predicted risks at five years. The central claim is that BI with imputation of all missing values yields the least biased estimates of model performance and individual risk predictions. The manuscript is clearly written and the tutorial component is practical, but the simulation evidence for the headline claim is weakened by the handling of CC model-fitting failures.","tokens_in":12126,"tokens_out":7263,"duration_ms":73354,"significance":"If the central claim is correct, the paper addresses an important practical question: whether to bootstrap before or after imputation when internally validating a prediction model with missing covariates, and whether to prefer deterministic imputation over complete-case analysis. The tutorial and accompanying R code are concrete and reproducible, and the simulation covers a wide range of missingness patterns, which is a genuine strength. The paper also gives useful attention to the deployment setting, where imputation must be applicable to future patients with unknown outcomes. However, the main conclusion is currently supported almost entirely by a simulation comparison that excludes a large, non-random fraction of datasets from the CC arm in high-missingness scenarios. The paper also restricts attention to a single data-generating process calibrated to one breast-cancer dataset, under missing-at-random mechanisms with correctly specified imputation models, so the quantitative claims should be interpreted as illustrative rather than general. The tutorial remains useful independent of the simulation result.","major_comments":[{"comment":"The CC bias distributions in Figures 1-5 are computed only over simulated datasets where the multivariable Cox model was successfully fitted, as the manuscript states in the paragraph beginning 'When the sample size is 750 and the approach is CC'. For missing-data pattern I at n=750 this excludes 57.7% of datasets, and smaller exclusions occur for patterns C, F, and H. The BI results are not subject to the same failure filter. Because convergence failure is likely to be concentrated in the very sparse or heavily missing datasets where CC bias could be largest, the comparison is not between two methods on a common set of datasets. The conclusion that 'CC always has more bias' and that BI gives the least biased estimates is therefore not supported as reported. Please re-run the comparison on the subset of datasets where both CC and BI fit, report the CC failure rates as a separate outcome, or use a penalized or Firth-type Cox fit so that the CC arm is estimable on essentially all datasets.","section":"Section 5"},{"comment":"The deterministic imputation implementation uses, for each missing variable, a generalized linear model with 'all other complete variables' as predictors, excluding partially observed covariates. This means that informative partially observed variables, including variables that appear in the missingness mechanisms of Table 3, are never used in the imputation models. The simulation therefore evaluates one specific implementation of deterministic imputation, and the paper's general recommendation for 'deterministic imputation' is correspondingly narrow. A sensitivity analysis using sequential imputation or including all observed portions of partially missing covariates would substantially strengthen the external validity of the 'least biased' claim; without it, the recommendation should be explicitly limited to the implementation studied here.","section":"Section 3.4 and Section 5"}],"minor_comments":[{"comment":"The R code in the guided example contains an error: `map(1:500, ~ slice_sample(dat), prop = 1, replace = TRUE))` should be `map(1:500, ~ slice_sample(dat, prop = 1, replace = TRUE))`, and the duplicate `library(purrr)` call should be removed.","section":"Section 4"},{"comment":"The notation in the missingness model, especially `M_{ik_j}`, `X_{il_j}`, and the condition `k_j \\neq l_j`, is hard to follow. Please define the indices more explicitly and use notation that does not confuse the covariate index with the observation index.","section":"Section 3.3"},{"comment":"The phrase 'bias ... calculated as the difference between the performance metric on the full data and the performance metric employing CC and BI approaches' should state explicitly that 'full data' means the simulated dataset before any missingness was imposed, to avoid ambiguity with the complete-case subset.","section":"Section 5"},{"comment":"The figure captions refer to 'panels' without identifying which estimator (apparent, bootstrap-corrected, .632, .632+) is shown in each panel; adding explicit panel titles or labels would make the figures much easier to read.","section":"Figures 1-5"},{"comment":"The claim that BI gives the 'least biased' estimates is made relative to CC and to two alternative imputation thresholds; the paper does not compare BI to multiple imputation in the simulation. The Discussion should make this scope explicit so that readers do not infer a direct simulation comparison with multiple imputation.","section":"Section 5 and Discussion"}],"recommendation":"major_revision","confidential_remarks":"The tutorial and code are a useful contribution, and the core idea of bootstrapping before deterministic imputation is plausible. However, the simulation comparison that underlies the main claim is compromised by the non-random exclusion of CC non-convergent datasets, and this is a load-bearing issue that requires a re-analysis rather than a textual patch. The manuscript would be acceptable after the authors provide a common-subset comparison or otherwise account for CC failures. I would not reject, because the problem is fixable within the manuscript's scope and the practical guidance has independent value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clean, practical tutorial on bootstrapping before deterministic imputation for internal validation of clinical risk prediction models with missing covariates. The R code and the worked example are genuinely useful, and the simulation study is extensive: 54 scenarios, two sample sizes, nine missingness patterns, three imputation variants. The authors also do the right thing by reporting how often complete-case analysis failed to fit.\n\nBut the headline claim—that BI with all missing values imputed gives the least biased AUC, Brier, and individual risk estimates—rests on a comparison that is not apples-to-apples. For CC, bias is computed only on datasets where the Cox model converged. For pattern I at n=750 that means dropping 57.7% of simulated datasets; for patterns C, F, H at n=750, 0.3–1.2%. The BI results are on the full set. If CC tends to fail in settings where its bias is large, the reported comparison flatters CC; if the opposite, it flatters BI. Either way, 'least biased' is not established. The authors acknowledge the reduced sample sizes but don't seem to see that this undercuts the headline. A common-subset analysis or an estimand defined for all datasets would fix it.\n\nOther soft spots are more minor. Multiple imputation is argued to be unsuited to prediction but never included as a comparator, which weakens the motivational framing. And everything is generated from one breast cancer dataset under MAR with correctly specified imputation models, so the quantitative bias reductions likely won't transfer unchanged. Those are limitations rather than flaws.\n\nWhat the paper does well: it lays out the conceptual case for deterministic imputation in deployed prediction models, gives step-by-step code, and provides a sensible default (impute all missing values). The simulation is transparent enough that the problem I describe is visible. That's a point in its favor.\n\nThis is a paper for applied clinical prediction modelers, not for people working on missing-data theory. It deserves a serious referee, but the simulation section needs revision before the central claim should be accepted.","headline":"Useful tutorial, but the simulation's headline comparison of BI versus CC is distorted by excluding non-convergent CC datasets, so 'least biased' is not yet established.","tokens_in":12608,"tokens_out":2070,"would_cite":false,"duration_ms":19907,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For clinical risk prediction models with missing covariates, bootstrapping before deterministic imputation and imputing all missing values yields the least biased estimates of model performance and individual risk predictions.","keywords":["clinical risk prediction","missing data","deterministic imputation","bootstrap validation","complete case analysis","AUC","Brier score","internal validation"],"falsifier":"Rerun the paper's 54-scenario simulation with missingness generated so that the probability a value is missing depends on the unobserved value itself (missing not at random) and compare the bias of the bootstrap-imputation and complete-case estimators; if complete-case analysis has lower bias in the majority of high-missingness scenarios, the paper's blanket recommendation would not survive.","tokens_in":11636,"feed_emoji":"📊","tokens_out":8099,"duration_ms":68262,"temperature":0.7,"pith_summary":"Clinical risk prediction models routinely face missing covariate data, but the usual missing-data toolbox was built for estimation, not for models that must predict for future patients. The paper proposes that when the goal is prediction, the right order is to draw bootstrap samples first, then fill missing covariate values with a single deterministic imputation model that excludes the outcome, and only then fit and evaluate the model. A simulation study based on a real breast cancer cohort, covering 54 scenarios at sample sizes 750 and 3,500, finds that this bootstrap-then-impute-all workflow yields less biased estimates of the AUC (how well predicted risk separates later from earlier events) and the Brier score (the mean squared error of predicted risks), as well as less biased individual risk predictions, than complete-case analysis or selective imputation. A step-by-step tutorial with R code accompanies the recommendation. If the claim is right, clinical prediction model developers should adopt this workflow for internal validation whenever missingness is plausibly at random.","feed_headline":"Bootstrap-then-impute gives least-biased risk model validation","feed_subtitle":"Filling every missing value after each bootstrap beat complete-case analysis on AUC, Brier score, and risk predictions.","key_machinery":"The central mechanism is the bootstrap-then-impute pipeline: each bootstrap resample is imputed independently before model fitting, so the imputation model is re-estimated within every resample and no information from the original development data is used to impute values in a bootstrap sample. The imputation step is deterministic regression imputation, in which each missing covariate is regressed on the other complete covariates using a generalized linear model and the missing values are replaced by the model's predicted response, thresholded for binary variables; the outcome is deliberately left out of the imputation model. The pipeline is evaluated through the apparent, bootstrap-corrected, .632, and .632+ estimators of the AUC and Brier score, and the paper's simulation asks which missing-data strategy makes these estimators least biased relative to the full-data values.","core_discovery":"On its own terms, the paper establishes that the order of resampling and imputation is the decision that controls bias in internal validation of prediction models with missing covariates. The recommended procedure is to bootstrap the original dataset, impute missing covariate values separately inside each bootstrap sample using deterministic regression imputation, fit the multivariable prediction model in each imputed bootstrap sample, and evaluate it in the original imputed data. Across the simulation, imputing all missing values in this bootstrap-first sequence gave the smallest bias relative to full-data performance for the AUC, the Brier score, and average predicted probability at five years, compared with complete-case analysis and with two more selective imputation strategies. The paper also argues that deterministic imputation is preferable to multiple imputation in this setting because omitting the outcome from the imputation model avoids bias in estimated coefficients and lets the fitted imputation models be applied directly to future patients with unknown outcomes.","pith_inferences":["Extending the simulation to a missing-not-at-random mechanism would be a natural stress test; if missingness depends on the unobserved value itself, the relative advantage of bootstrap-imputation over complete-case analysis could shrink or reverse, and the paper's guidance would need qualification.","The leakage argument likely generalizes: any validation scheme that resamples before a deterministic imputer is fitted should protect against optimistic estimates, but the paper only evaluates generalized linear model imputation, so the size of the benefit with median, nearest-neighbor, or tree-based imputers is an open question.","Because the simulation parameters come from a single breast cancer cohort, a practitioner in another disease should run a small local simulation with plausible covariate correlations and missingness rates to confirm the expected bias reduction before adopting the workflow.","The same bootstrap-before-imputation ordering should matter whenever model-building includes any data-dependent preprocessing step (normalization, feature selection, tuning), not only imputation."],"forward_implications":["A clinical prediction model built this way can be deployed as a risk calculator: the imputation models and the prediction model are both fixed functions of observed covariates, so a future patient's missing values can be filled without access to the original development dataset.","Complete-case analysis is not just more biased but can fail altogether in small samples with high missingness; the simulation records fitting failures in up to 57.7% of datasets for the most extreme patterns.","Even with low missingness (5% in a single covariate), bootstrap-imputation produced lower bias than complete-case analysis, so the benefit is not limited to severe missingness.","Among imputation strategies, imputing all missing values was at least as good as imputing only variables with high missingness or only participants with few missing variables, so the simpler 'impute everything' rule is the preferred default."],"supporting_citations":[{"why":"Provides the missing-at-random data-generation framework and the simulation approach the paper adapts.","marker":"[4]"},{"why":"Supplies the reason the outcome must be excluded from the imputation model, a load-bearing assumption of deterministic imputation.","marker":"[9]"},{"why":"Shows deterministic imputation performs at least as well as multiple imputation when deploying clinical prediction models, motivating the method choice.","marker":"[11]"},{"why":"Supports bootstrapping before imputation as the way to obtain proper variability when multiple imputation is used.","marker":"[13]"},{"why":"Also establishes that bootstrap inference should be performed before imputation, reinforcing the ordering at the center of the paper.","marker":"[14]"},{"why":"Provides the missing-data generation setup and an earlier baseline combining internal validation with multiple imputation.","marker":"[17]"},{"why":"Defines the optimism-correction estimator that the bootstrap-corrected performance metric is built on.","marker":"[19]"},{"why":"Supplies the motivating breast cancer dataset whose covariate distributions and associations parameterize the simulation.","marker":"[23]"}],"fun_headline_variants":["Bootstrap-first imputation cuts risk model validation bias","Impute after bootstrap to limit risk model bias","Order matters: bootstrap then impute for valid risk models","Deterministic imputation after bootstrap beats complete-case","Bootstrap-then-impute: lowest bias in risk model validation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes missingness is missing at random, with the chance a value is missing depending only on observed covariates, and that the generalized-linear imputation models are correctly specified; if missingness depends on unobserved values or the imputation models are badly wrong, complete-case analysis could become relatively more attractive.","fun_headline_variants_meta":{"raw":{"variants":["Bootstrap-first imputation cuts risk model validation bias","Impute after bootstrap to limit risk model bias","Order matters: bootstrap then impute for valid risk models","Deterministic imputation after bootstrap beats complete-case","Bootstrap-then-impute: lowest bias in risk model validation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000355,"raw_usage":{"total_tokens":1894,"prompt_tokens":875,"completion_tokens":1019,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":940}},"tokens_in":491,"tokens_out":1019,"duration_ms":7307,"temperature":1.0,"reasoning_tokens":940,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:08:48.024683+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the paper's 54-scenario simulation with missingness generated so that the probability a value is missing depends on the unobserved value itself (missing not at random) and compare the bias of the bootstrap-imputation and complete-case estimators; if complete-case analysis has lower bias in the majority of high-missingness scenarios, the paper's blanket recommendation would not survive.","supporting_citations":[{"cited_title":"Marshall, D","cited_arxiv_id":null,"evidence_quote":"Provides the missing-at-random data-generation framework and the simulation approach the paper adapts."},{"cited_title":"D’Agostino McGowan, S","cited_arxiv_id":null,"evidence_quote":"Supplies the reason the outcome must be excluded from the imputation model, a load-bearing assumption of deterministic imputation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows deterministic imputation performs at least as well as multiple imputation when deploying clinical prediction models, motivating the method choice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports bootstrapping before imputation as the way to obtain proper variability when multiple imputation is used."},{"cited_title":"Schomaker and C","cited_arxiv_id":null,"evidence_quote":"Also establishes that bootstrap inference should be performed before imputation, reinforcing the ordering at the center of the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the missing-data generation setup and an earlier baseline combining internal validation with multiple imputation."},{"cited_title":"Harrell, F","cited_arxiv_id":null,"evidence_quote":"Defines the optimism-correction estimator that the bootstrap-corrected performance metric is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the motivating breast cancer dataset whose covariate distributions and associations parameterize the simulation."}],"review_version":1}