REVIEW 3 major objections 4 minor 9 references
Comparing methods for handling missing data in electronic health records for dynamic risk prediction of central-line associated bloodstream infection
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Missing-indicator flags beat complex imputation for CLABSI risk prediction
desk verdict Solid empirical comparison with an honest but statistically under-supported headline: missing indicators look competitive, but the AUROC gap is tiny and unquantified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of the comparison is a landmark cause-specific supermodel: at each daily landmark, the 7-day risk of CLABSI is modeled with cause-specific hazards, treating hospital discharge and death as competing events, with smooth baseline hazards over landmark time. Each missing-data strategy is plugged into this same prediction model, trained on imputed training data and applied to imputed validation data so that deployment conditions are realistic. The missing indicator mechanism is the key contrast: instead of trying to fill gaps, it turns the gaps themselves into binary predictors, replacing missing values with a fixed number and adding a flag for each variable.
What would settle it
A temporal external validation using EHR data from a later period, for example 2017 onward, that measures whether the missing indicator method's AUROC and scaled Brier score advantage over imputation persists when ordering patterns have changed; if the advantage disappears or reverses, the central claim's transportability fails.
Extended reading notes
Core claim
The central claim is that for dynamic 7-day CLABSI risk prediction from EHR data, the missing indicator method, a dummy code per predictor noting whether it is missing, delivers better or equal predictive performance than any imputation strategy tested while being far cheaper. In repeated validation, its mean AUROC reached 0.782 and its scaled Brier scores were highest at every landmark up to day 14. The mixed model plus missing indicators reached the highest AUROC of 0.783 at day 4, and missForestPredict plus missing indicators had the best scaled Brier scores at early landmarks, but the gains over indicators alone were small. The paper interprets this as evidence that in EHR data, the absence of a measurement is a clinical signal: it reflects whether clinicians deemed a test or observation needed, so missingness is informative.
Load-bearing premise
The comparison assumes that the missingness patterns seen in the 2012-2013 training and validation data will look the same in future data; the paper itself notes that shifts in EHR documentation can change those patterns and break the missing-indicator advantage.
Editorial extensions
If this is right
- In this EHR setting, the missing indicator method can replace computationally heavy imputation with near-zero preprocessing time and no loss in predictive performance.
- Combining missing indicators with imputation gives a small, not decisive, improvement; the best AUROC of 0.783 at day 4 comes from a mixed-model approach plus indicators.
- The finding supports the informative-missingness view: the pattern of absent records carries prognostic information and should be considered as a predictor rather than discarded.
- Because the advantage depends on stable missingness mechanisms, models using missing indicators should be monitored for temporal drift in documentation practice.
Reading between the lines
- If the mechanism generalizes, missing-indicator flags could be a cheap, transparent baseline for any EHR risk model, worth reporting alongside imputation-based models as a performance floor.
- A direct test would be temporal validation on later EHR years; the paper's own example of a drug rarely ordered after 2017 predicts that the missing indicator's performance should change when ordering patterns shift.
- The result may partly explain why complex imputation often disappoints in EHR prediction: when missingness is informative, filling values destroys the signal rather than recovering it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper compares strategies for handling missing data in electronic health records when building a dynamic risk prediction model for central line-associated bloodstream infection (CLABSI). Using 30,862 catheter episodes from UZ Leuven (2012-2013), the authors fit a landmark cause-specific supermodel with 21 predictors, evaluate seven imputation approaches plus missing indicators, and report discrimination, calibration, and overall performance across 100 repeated train/validation splits. The central empirical claim is that the missing indicator approach performs best, with mean AUROC up to 0.782 and superior scaled Brier scores, and that adding missing indicators to other imputation methods yields small additional gains.
Significance. If the central claim holds, the paper has practical value: it suggests that in this EHR setting a simple, computationally cheap missing-indicator strategy is at least competitive with complex imputation for dynamic risk prediction, and it highlights the informative nature of missingness in clinical data. The study is also a useful methodological comparison because it evaluates imputation methods under realistic deployment constraints, where the outcome is unknown at prediction time. Strengths include the repeated data-splitting design, competing-risks handling, transparent reporting, and the availability of R code and a synthetic example dataset. The main weakness is that the headline comparisons are presented without uncertainty quantification, which matters because the performance gaps are small.
major comments (3)
- [§3.1, Figure 1] The headline claim that the missing indicator approach "demonstrated the highest discriminative ability" is supported only by point estimates: mean AUROC up to 0.782 versus 0.776 for median/mode imputation, and 0.783 for the mixed model plus missing indicators at day 4. No confidence intervals, standard deviations, or paired tests across the 100 splits are reported for these differences. Because the gaps are below 0.01 and the splits are heavily overlapping (any two random two-thirds splits share about one-third of admissions), the effective number of independent replications is far below 100, so sampling variation could easily explain the observed ordering. Please quantify the uncertainty of the primary comparison (for example, with split-level bootstrap confidence intervals for the AUROC and scaled Brier differences) or soften the superiority claim accordingly.
- [§4, Limitations] The paper acknowledges that LASSO variable selection was performed on the same data splits used for validation and states that the impact is limited because the goal is comparison of imputation methods. This is not self-evident: if the stability of variable selection differs across imputation approaches, the optimism in performance estimates could differ systematically between the missing-indicator arm and the model-based imputation arms, biasing the comparison. Please provide evidence on this point, for example by reporting how often each predictor was selected within each arm, or by re-running selection once on the full data and checking whether the ranking of methods changes.
- [Table 2, Missing indicator method] The missing indicator approach uses an arbitrary fixed value of 99 for missing entries and shares one missing indicator across all medical specialty categories. These are modeling choices, and no sensitivity analysis is reported for them. Since the missing indicator arm is the winner of the comparison, the manuscript should show that the conclusion is robust to the placeholder value and to the grouping of specialty indicators, for instance by repeating the analysis with a different placeholder (e.g., the median) and with separate specialty indicators.
minor comments (4)
- [§2.4.1] There are a few language errors, including "the true observed outcome values n the validation data" and "the missingness itself may carry valuable informative about a patient's condition." These should be corrected.
- [Figures 1 and 2] The figures are dense and difficult to read in print because many methods are plotted in a single panel without error bars. Consider using separate panels per metric and adding pointwise confidence intervals or shaded bands if uncertainty estimates become available.
- [Supplementary File 3 vs §2.4.1] The main text says the ordinal GCS variable would ideally be imputed with ordinal logistic regression via the "polr" method, while Supplementary File 3 refers to "polyreg." Please unify the terminology.
- [Table 1] For time-varying predictors, the column "N (%) of catheter episodes with missingness" is ambiguous: it is unclear whether it counts episodes with at least one missing landmark observation or the fraction of landmark observations that are missing. Please clarify the unit of analysis for this column.
Circularity Check
No significant circularity: the comparison of imputation methods is an empirical evaluation on held-out validation splits, not a derivation that reduces to its inputs.
full rationale
The paper's central claim is an empirical ranking of missing-data strategies. Each strategy is fitted on the two-thirds training split and evaluated on the held-out one-third validation split (Section 2.4), so the missing-indicator result is estimated from validation data rather than constructed from a fitted parameter or defined in terms of the outcome it predicts. The landmark cause-specific supermodel is adopted from the authors' prior work (Ref. 27) and missForestPredict from Ref. 15, but these self-citations only supply implementation and modeling choices; the conclusion stands or falls on the validation metrics. The paper explicitly flags that LASSO variable selection was performed on the same data splits, which may introduce optimism due to information leakage (Section 4, Discussion), and that temporal shifts in EHR missingness patterns may affect transportability. These are genuine limitations and uncertainty concerns, but they are not definitional circularity: no equation is equivalent to its inputs by construction, and no fitted quantity is renamed as a prediction. The main statistical weakness is the absence of confidence intervals or paired tests for the small AUROC differences, which is a reporting and inference concern, not a circularity concern.
Assumptions & free parameters
free parameters (1)
- Missing indicator placeholder value =
99
assumptions (4)
- domain assumption The 7-day prediction horizon and daily landmark times are clinically appropriate.
- domain assumption EHR missingness is informative, so missing indicators can capture patient risk.
- ad hoc to paper The landmark cause-specific supermodel with linear and quadratic landmark time correctly specifies baseline hazards and time-dependent effects.
- domain assumption The single-center 2012-2013 data are sufficient to draw general conclusions about imputation methods for EHR prediction.
Cite this review
Pith. "Pith review of Comparing methods for handling missing data in electronic health records for dynamic risk prediction of central-line associated bloodstream infection." pith.science (2026). https://pith.science/paper/PXDQKYHV
@misc{pith2026250606707,
author = {Pith},
title = {Pith review of: Comparing methods for handling missing data in electronic health records for dynamic risk prediction of central-line associated bloodstream infection},
year = {2026},
howpublished = {\url{https://pith.science/paper/PXDQKYHV}},
note = {Machine review of arXiv:2506.06707}
}
read the original abstract
Electronic health records (EHR) often contain varying levels of missing data. This study compared different imputation strategies to identify the most suitable approach for predicting central line-associated bloodstream infection (CLABSI) in the presence of competing risks using EHR data. We analyzed 30862 catheter episodes at University Hospitals Leuven (2012-2013) to predict 7-day CLABSI risk using a landmark cause-specific supermodel, accounting for competing risks of hospital discharge and death. Imputation methods included simple methods (median/mode, last observation carried forward), multiple imputation, regression-based and mixed-effects models leveraging longitudinal structure, and random forest imputation to capture interactions and non-linearities. Missing indicators were also assessed alone and in combination with other imputation methods. Model performance was evaluated dynamically at daily landmarks up to 14 days post-catheter placement. The missing indicator approach showed the highest discriminative ability, achieving a mean AUROC of up to 0.782 and superior overall performance based on the scaled Brier score. Combining missing indicators with other methods slightly improved performance, with the mixed model approach combined with missing indicators achieving the highest AUROC (0.783) at day 4, and the missForestPredict approach combined with missing indicators yielding the best scaled Brier scores at earlier landmarks. This suggests that in EHR data, the presence or absence of information may hold valuable insights for patient risk prediction. However, the use of missing indicators requires caution, as shifts in EHR data over time can alter missing data patterns, potentially impacting model transportability.
Reference graph
Works this paper leans on
-
[1]
Centers for Disease Control and Prevention. Bloodstream Infection Event (Central Line- Associated Bloodstream Infection and Non-central Line Associated Bloodstream Infection) [Internet]. 2022 Jan. Available from: https://www.cdc.gov/nhsn/pdfs/pscmanual/4psc_clabscurrent.pdf
work page 2022
-
[2]
Surveillance Bloedstroom Infecties in Belgische Ziekenhuizen - Protocol 2019
Duysburgh, E. 2019. “Surveillance Bloedstroom Infecties in Belgische Ziekenhuizen - Protocol 2019.” Brussel, België: Sciensano. https://www.sciensano.be/sites/default/files/bsi_surv_protocol_nl_april2019.pdf
work page 2019
-
[3]
Gao S, Albu E, Putter H, et al. A comparison of regression models for static and dynamic prediction of a prognostic outcome during admission in electronic health care records. arXiv: 2405.01986 2024
work page Pith review arXiv 2024
-
[4]
missForestPredict -- Missing data imputation for prediction settings
Albu E, Gao S, Wynants L, Van Calster B. missForestPredict – Missing data imputation for prediction settings. arXiv. 2024; arXiv:2407.03379. Available from: https://arxiv.org/abs/2407.03379
work page Pith review arXiv 2024
-
[5]
(2007), Dynamic Prediction by Landmarking in Event History Analysis
VAN HOUWELINGEN, H.C. (2007), Dynamic Prediction by Landmarking in Event History Analysis. Scandinavian Journal of Statistics, 34: 70-85. https://doi.org/10.1111/j.1467- 9469.2006.00529.x
arXiv 2007
-
[6]
Dynamic Prediction In Clinical Survival Analysis
Houwelingen, Hans, and Hein Putter. Dynamic Prediction In Clinical Survival Analysis. Boca Raton: CRC Press, 2012
work page 2012
-
[7]
Anya H Fries, Eunji Choi, Julie T Wu, Justin H Lee, Victoria Y Ding, Robert J Huang, Su- Ying Liang, Heather A Wakelee, Lynne R Wilkens, Iona Cheng, Summer S Han, Software Application Profile: dynamicLM—a tool for performing dynamic risk prediction using a landmark supermodel for survival data under competing risks, International Journal of Epidemiology, ...
-
[8]
Calculating the sample size required for developing a clinical prediction model
Riley RD, Ensor J, Snell KIE, Harrell FE Jr, Martin GP, Reitsma JB, Moons KGM, Collins G, van Smeden M. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020 Mar 18;368:m441. doi: 10.1136/bmj.m441. PMID: 32188600
doi:10.1136/bmj.m441 2020
Show all 9 references
-
[9]
Systematic review finds risk of bias and applicability concerns for models predicting central line- associated bloodstream infection
Gao S, Albu E, Tuand K, Cossey V, Rademakers F, Van Calster B, Wynants L. Systematic review finds risk of bias and applicability concerns for models predicting central line- associated bloodstream infection. J Clin Epidemiol. 2023 Sep;161:127-139. doi: 10.1016/j.jclinepi.2023....
2023 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.