Pith. sign in

REVIEW 3 major objections 5 minor 30 references

A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach

T0 review · 3 major / 5 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Repeated cross-validation can measure clinical prediction instability while also giving internal validation of performance, and often reports lower MAPE than the bootstrap.

desk verdict Solid empirical methods paper that packages repeated nested CV for individual-level MAPE and shows it works alongside performance metrics; the MAPE gap vs bootstrap is real but not a pure instability ranking because the estimands differ by design. read the letter →

arxiv 2607.03676 v1 pith:LM4Q4XKX submitted 2026-07-04 stat.ME

Nop Khongthon , Pakpoom Wongyikul , Noraworn Jirattikanwong , Phanu Prasankittirach , Natthanaphop Isaradech , Wachiranun Sirikul , Noppadon Seesuwan , Suppachai Lawanaskol
show 1 more author
Phichayut Phinyo
This is my paper · ORCID
classification stat.ME
keywords cross-validationbootstrappredictioninstabilityclinicalmodelsinternalvalidationcalibrationrandomforestmeanabsoluteerror
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Clinical prediction models need both good average performance and stable individual-level risk estimates. Bootstrap resampling has been recommended for instability, but this paper proposes and tests a repeated 5-fold cross-validation alternative that can jointly deliver internally validated discrimination, calibration, and a mean absolute prediction error (MAPE) for individual predictions. Using a real emergency-department cohort of 19,418 patients, the authors subsampled under events-per-variable scenarios of 10, 30, and 50 (plus the full data) and compared logistic regression and random forest, with nested CV for forest tuning. Performance metrics were similar for logistic regression and closer to the empirical reference for random-forest calibration under CV. At EPV 30, bootstrap MAPE was higher than CV MAPE for both models (0.042 vs 0.020 for logistic regression; 0.077 vs 0.027 for random forest). The authors therefore present repeated CV as a practical single-framework alternative, especially when several algorithms must be compared under the same resampling structure.

What carries the argument

CV-based MAPE: for each person, the mean absolute deviation of repeated held-out predictions from that person's median prediction across repeats, averaged over individuals; nested 5-fold CV with 200 repeats carries the full modelling pipeline (including hyperparameter tuning) so performance and instability share one resampling structure.

What would settle it

On the same or a similar cohort, recompute both MAPEs against a common reference (e.g., both vs median prediction, or both vs predictions on a large held-out reference set) and check whether the bootstrap-vs-CV gap and the preference for CV disappear or reverse.

Watch

Extended reading notes

Core claim

A repeated k-fold cross-validation procedure can assess individual-level prediction instability (via MAPE defined against each person's median held-out prediction) while simultaneously providing internal validation of discrimination and calibration, and in this empirical comparison it yields MAPE values consistently lower than the bootstrap approach of Riley et al., with random-forest calibration slopes closer to the empirical reference.

Load-bearing premise

That bootstrap MAPE (deviation from the single original-development model) and CV MAPE (deviation from each person's median held-out prediction) are similar enough quantities that lower CV MAPE can support preferring CV as an instability tool.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes repeated 5-fold cross-validation (with nested CV for random forest tuning) as a way to assess individual-level prediction instability via MAPE while simultaneously obtaining internally validated discrimination and calibration. Using a real ED cohort (n=19,418) and 200 stratified resampling iterations under EPV 10/30/50 and the full data, the authors compare this approach with the Riley-style bootstrap for logistic regression and random forest. Performance metrics are broadly similar for LR; for RF, CV calibration slopes are closer to the held-out reference than bootstrap-corrected slopes. Bootstrap MAPE is systematically higher than CV MAPE (e.g., EPV 30: LR 0.042 vs 0.020; RF 0.077 vs 0.027). The authors conclude that CV is a practical alternative for instability assessment, especially when comparing algorithms.

Significance. If the recommendation holds after clarifying estimands, the paper would give applied CPM developers a single resampling framework that jointly reports performance and individual-level instability and that naturally embeds hyperparameter tuning—addressing a real gap between statistical (bootstrap-heavy) and ML (CV-heavy) practice. Strengths include the large real cohort, EPV-stratified design, 200 iterations, held-out reference benchmark, nested CV for RF, explicit MAPE formulas, open code, and an honest limitations section. The work is empirical and exploratory rather than a definitive ranking of methods, but it is a useful feasibility contribution for the instability literature.

major comments (3)
  1. [§2.6; Table 3; Discussion; Conclusions] The central practical claim (CV as a preferable alternative for instability assessment, especially multi-algorithm comparison) rests heavily on lower CV MAPE (Abstract; Table 3; Results §Prediction instability; Conclusions). Yet §2.6 defines bootstrap MAPE as mean absolute deviation from the single original-development-model prediction, while CV MAPE is deviation from each individual’s median held-out prediction across repeats. The Discussion correctly notes these are different estimands on differently constituted samples (~63% unique observations in bootstrap vs 80% distinct per CV fold). The numerical gap is therefore expected by construction and does not isolate “underlying” instability. The recommendation should be reframed around workflow advantages (joint performance + instability; natural nesting of tuning; same structure across algorithms) rather than the MAPE contrast alone, or
  2. [§2.5 Simulation and scenario specifications] The bootstrap arm uses only 80% of the simulated dataset for development (with 20% set aside “to mimic common real-world practice” and to align with CV’s 4/5 training fraction; §2.5), whereas standard Riley-style bootstrap instability assessment uses the full development sample. This design choice reduces the number of distinct observations available to bootstrap models relative to a full-sample bootstrap and may inflate bootstrap MAPE and calibration variability. The paper should either re-run the bootstrap on the full simulated sample (standard practice) as a sensitivity analysis, or more carefully justify and bound how this 80% restriction affects the MAPE and calibration-slope comparisons that drive the conclusions.
  3. [Discussion; Additional file 1] Calibration instability under CV is noisier at low EPV despite lower MAPE (Discussion; Supplementary figures). The authors attribute this to smaller held-out folds (~1/5 of the sample) versus full-sample bootstrap calibration curves—a plausible explanation. Because the paper also promotes CV for joint performance and instability assessment, this sample-size dependence of CV calibration curves should be stated as a concrete limitation of the proposed approach (not only as a post-hoc explanation), with guidance on when bootstrap may remain preferable for calibration instability plots.
minor comments (5)
  1. [Figure 4; Results] Figure 4 caption says “EPV 50” while the surrounding Results text emphasises EPV 30 MAPE contrasts; align caption, text, and which condition is shown as the main figure.
  2. [Table 2; Figure 3] Table 2 reports Empirical AUC as a single pair of intervals for LR and RF; clarify whether these are pooled across EPV scenarios or from a specific scenario, and ensure consistency with the dashed reference lines in Figure 3.
  3. [§2.4–2.6] Notation for predicted probabilities mixes hats, subscripts, and superscripts (e.g., p_i^(b)̂ vs p̂_i,b^boot); a short notation table would help.
  4. [References] Reference list has a duplicated Riley & Collins entry (refs 13 and 24); clean up.
  5. [Abstract; §2.4.2] Abstract and main text sometimes say “5-fold CV” and sometimes “repeated 5-fold CV”; state R=200 consistently in the abstract methods sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical resampling comparison of two explicitly different MAPE estimators against a held-out reference, with no self-definitional predictions or load-bearing self-citation chains.

full rationale

The paper is a resampling-based empirical methods comparison on a real clinical cohort (n=19,418), not a first-principles derivation. Development samples are drawn under controlled EPV, models (LR, RF with nested tuning) are fit under bootstrap (B=200) and repeated 5-fold CV (R=200), performance (AUC, slope, CITL) and MAPE are computed, and both are benchmarked against a fixed 20% reference hold-out that never enters fitting or tuning. Bootstrap MAPE is defined as mean |p̂_boot − p̂_orig| while CV MAPE is mean |p̂_r − median_p|; §2.6 and the Discussion state these are different estimands on differently constituted samples and that the numerical gap partly reflects that design difference rather than pure underlying instability. No quantity is fitted to a subset and then “predicted” as an independent result; no uniqueness theorem or ansatz is imported via self-citation to force the conclusion; the recommendation that CV is a practical alternative rests on the observed empirical patterns (similar discrimination, closer RF calibration slopes, lower CV MAPE) together with the acknowledged estimand difference. The derivation chain is therefore self-contained against the external reference dataset and does not reduce by construction to its inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central recommendation rests on standard resampling theory plus several design choices that define the comparison (EPV grid, 5-fold/200-repeat structure, 80% bootstrap development alignment, RF hyperparameter ranges, and the CV MAPE reference as the median held-out prediction). No new physical entities are postulated; the main invented construct is the CV-specific MAPE estimand. Free parameters are procedural settings, not fitted scientific constants, but they shape the numerical MAPE gap that supports the practical claim.

free parameters (5)
  • Number of bootstrap resamples B and CV repeats R
    Both set to 200; controls Monte Carlo precision of performance and MAPE estimates and is a design choice, not estimated from theory in the paper.
  • Fold count K=5
    Chosen to mirror an 80/20 train/assessment split and to keep held-out folds large enough for calibration curves; more folds would shrink held-out samples and add calibration noise (Discussion).
  • EPV scenario grid (10, 30, 50, full≈1004)
    Sample-size conditions selected by the authors to span low-to-large development sizes; primary numerical MAPE contrast is reported at EPV 30.
  • RF hyperparameter search ranges and 100-point space-filling grid
    min node size 20–30, trees 200–300, max depth 3–6; tuning objective AUC on held-out/inner CV. These bounds affect RF instability and calibration behavior.
  • Bootstrap development fraction (80% of simulated set)
    Nonstandard restriction to align effective development size with 5-fold CV; changes the bootstrap comparison relative to full-sample bootstrap practice.
assumptions (5)
  • domain assumption Optimism-corrected bootstrap performance and average repeated k-fold CV performance are approximately unbiased estimators of test performance under standard assumptions.
    Invoked in §2.4 when equating θ_CV with θ_corrected and when interpreting internal validation against the reference dataset.
  • domain assumption All six continuous predictors act linearly with no interactions, missingness, or variable selection in the logistic model.
    Stated in §2.3 model development pipelines; limits external validity of instability findings to this simple specification.
  • domain assumption A held-out 20% reference split from the same cohort is an adequate empirical benchmark for 'true' performance and for interpreting instability.
    §2.5 simulation design; not an external multi-center validation set, so transportability is assumed within-cohort.
  • ad hoc to paper Median prediction across CV repeats is a valid individual-level reference for CV MAPE when no single original model exists for all subjects.
    §2.6 defines MAPE_i^CV relative to the median; this is the paper's operational solution to the missing single-reference-model problem.
  • standard math Standard resampling facts: bootstrap samples contain ~63.2% unique observations; 5-fold CV trains on 80% distinct observations.
    Used in Discussion (citing Efron & Tibshirani, Kohavi) to explain higher bootstrap MAPE.
invented entities (1)
  • CV-based individual MAPE (deviation from median held-out prediction across repeats)
    purpose: Quantify prediction instability under repeated/nested k-fold CV in a form comparable in spirit to Riley bootstrap MAPE.
    Not a physical entity but a new operational estimand introduced so CV can report instability without a single original model; independent_evidence is false because its numerical scale is defined only within this resampling scheme.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach." pith.science (2026). https://pith.science/paper/LM4Q4XKX

@misc{pith2026260703676,
  author       = {Pith},
  title        = {Pith review of: A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LM4Q4XKX}},
  note         = {Machine review of arXiv:2607.03676}
}
read the original abstract

Bootstrap-based methods have been recommended for assessing prediction instability in clinical prediction models, but their performance relative to cross-validation (CV) remains unclear. We propose a CV-based approach for assessing prediction instability and compare it with a bootstrap-based approach in logistic regression and random forest models. We conducted a resampling-based empirical experiment using a clinical cohort of 19,418 emergency department patients. Development samples were generated under events-per-variable (EPV) scenarios of 10, 30, and 50, and results were compared with those from the full dataset. Models were evaluated using bootstrap validation and repeated 5-fold CV; nested CV was used for random forest tuning. Predictive performance was assessed using AUC, calibration slope, and calibration-in-the-large. Prediction instability was quantified using mean absolute prediction error (MAPE). For logistic regression, bootstrap validation and repeated 5-fold CV produced broadly similar discrimination and calibration, especially at higher EPV values. For random forest, apparent performance consistently overestimated empirical discrimination. Bootstrap validation and repeated 5-fold CV gave comparable discrimination, but repeated 5-fold CV produced calibration slope estimates closer to the empirical value. Prediction stability improved as EPV increased for both modelling approaches. At EPV 30, bootstrap-derived MAPE was higher than CV-derived MAPE for both logistic regression (median, 0.042 versus 0.020) and random forest (median, 0.077 versus 0.027). A CV-based approach can assess prediction instability while also providing internally validated performance. These findings support CV-based instability assessment as a practical alternative to bootstrap-based assessment, particularly when comparing instability across multiple modelling algorithms.

Figures

Figures reproduced from arXiv: 2607.03676 by the authors.

Figure 2
Figure 2. Analytical flow for evaluating the prediction instability across simulated datasets. [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Predictive performance by EPV condition and validation approach across logistic [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figure 4
Figure 4. Prediction instability plots at EPV 50 with 5 [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references

  1. [1]

    Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating: Springer Science & Business Media; 2008

    Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating: Springer Science & Business Media; 2008

    2008

  2. [2]

    Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement

    Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Medicine. 2015;13:1

    2015

  3. [3]

    TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods

    Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378

    2024

  4. [4]

    Prognosis and prognostic research: application and impact of prognostic models in clinical practice

    Moons KGM, Altman DG, Vergouwe Y , Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009;338:b606

    2009

  5. [5]

    Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research

    Steyerberg EW, Moons KGM, van der Windt DA, Hayden JA, Perel P, Schroter S. Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research. PLoS Medicine. 2013;10:e1001381

    2013

  6. [6]

    Clinical Prediction Rules That Don't Hold Up—Where to Go From Here? Journal of Orthopaedic & Sports Physical Therapy

    Stanton TR. Clinical Prediction Rules That Don't Hold Up—Where to Go From Here? Journal of Orthopaedic & Sports Physical Therapy. 2016;46:502–5

    2016

  7. [7]

    External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature

    Wessler BS, Nelson J, Park JG, McGinnes H, Gulati G, Brazil R. External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature. Circulation: Cardiovascular Quality and Outcomes. 2021;14:e007858

    2021

  8. [8]

    All models are wrong and yours are useless: making clinical prediction models impactful for patients

    Markowetz F. All models are wrong and yours are useless: making clinical prediction models impactful for patients. npj Precision Oncology. 2024;8:54. 38

    2024

Show all 30 references
  1. [9]

    A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods

    Collins GS, Omar O, Shanyinde M, Yu L-M. A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. Journal of Clinical Epidemiology. 2013;66:268–77

    2013

  2. [10]

    Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review

    Andaur Navarro CL, Damen JAA, Takada T, Nijman SWJ, Dhiman P, Ma J. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021;375:n2281

    2021

  3. [11]

    Does poor methodological quality of prediction modeling studies translate to poor model performance? An illustration in traumatic brain injury

    Helmrich IRAR, Mikolić A, Kent DM, Lingsma HF, Wynants L, Steyerberg EW. Does poor methodological quality of prediction modeling studies translate to poor model performance? An illustration in traumatic brain injury. Diagnostic and Prognostic Research. 2022;6:8

    2022

  4. [12]

    Methodological and applicability pitfalls of clinical prediction models for asthma diagnosis: a systematic review and critical appraisal of evidence

    Wongyikul P, Phinyo P, Seephueng P, Tanasombatkul K, Kawamatawong T, Wongsa C. Methodological and applicability pitfalls of clinical prediction models for asthma diagnosis: a systematic review and critical appraisal of evidence. BMC Medical Research Methodology. 2025;25:232

    2025

  5. [13]

    Stability of clinical prediction models developed using statistical or machine learning methods

    Riley RD, Collins GS. Stability of clinical prediction models developed using statistical or machine learning methods. Biometrical Journal. 2023;65:2200302

    2023

  6. [14]

    Developing clinical prediction models: a step-by-step guide

    Efthimiou O, Seo M, Chalkou K, Debray T, Egger M, Salanti G. Developing clinical prediction models: a step-by-step guide. BMJ. 2024;386:e078276

    2024

  7. [15]

    Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis

    Steyerberg EW, Harrell FE, Borsboom GJJM, Eijkemans MJC, Vergouwe Y , Habbema JDF. Internal validation of predictive models: Efficiency of some procedures for logistic regression analysis. Journal of Clinical Epidemiology. 2001;54:774–81. 39

    2001

  8. [16]

    Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models

    Austin PC, Steyerberg EW. Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Statistical Methods in Medical Research. 2017;26:796–808

    2017

  9. [17]

    Xu Y , Goodacre R. On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning. Journal of Analysis and Testing. 2018;2:249–62

    2018

  10. [18]

    Empirical evaluation of internal validation methods for prediction in large-scale clinical data with rare-event outcomes: a case study in suicide risk prediction

    Coley RY , Liao Q, Simon N, Shortreed SM. Empirical evaluation of internal validation methods for prediction in large-scale clinical data with rare-event outcomes: a case study in suicide risk prediction. BMC Medical Research Methodology. 2023;23:33

    2023

  11. [19]

    Empirical simulation of internal validation methods for prediction models: comparing k-fold cross-validation with bootstrap- based optimism correction

    Zhang C, Yan R, Liu X, Nie X, Peng Y , Peng X. Empirical simulation of internal validation methods for prediction models: comparing k-fold cross-validation with bootstrap- based optimism correction. Journal of Clinical Epidemiology. 2026;190:112101

    2026

  12. [20]

    A Guide to Cross-Validation for Artificial Intelligence in Medical Imaging

    Bradshaw TJ, Huemann Z, Hu J, Rahmim A. A Guide to Cross-Validation for Artificial Intelligence in Medical Imaging. Radiology: Artificial Intelligence. 2023;5:e220232

    2023

  13. [21]

    Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models

    Andaur Navarro CL, Damen JAA, van Smeden M, Takada T, Nijman SWJ, Dhiman P. Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models. Journal of Clinical Epidemiology. 2023;154:8–22

    2023

  14. [22]

    Clinical prediction models and the multiverse of madness

    Riley RD, Pate A, Dhiman P, Archer L, Martin GP, Collins GS. Clinical prediction models and the multiverse of madness. BMC Medicine. 2023;21:502

    2023

  15. [23]

    PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods

    Moons KGM, Damen JAA, Kaul T, Hooft L, Andaur Navarro C, Dhiman P. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. 40

    2025

  16. [24]

    Stability of clinical prediction models developed using statistical or machine learning methods

    RD R, GS C. Stability of clinical prediction models developed using statistical or machine learning methods. Biometrical Journal. 2023;65(8):2200302

    2023

  17. [25]

    Improvements on cross-validation: the .632+ bootstrap method

    Efron B, Tibshirani R. Improvements on cross-validation: the .632+ bootstrap method. Journal of the American Statistical Association. 1997;92(438):548–60

    1997

  18. [26]

    Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI); 1995; San Francisco: Morgan Kaufmann

    Kohavi R, editor A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI); 1995; San Francisco: Morgan Kaufmann

    1995

  19. [27]

    Prediction error estimation: a comparison of resampling methods

    Molinaro AM, Simon R, Pfeiffer RM. Prediction error estimation: a comparison of resampling methods. Bioinformatics. 2005;21(15):3301–7

    2005

  20. [28]

    Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints

    van der Ploeg T, Austin PC, Steyerberg EW. Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Medical Research Methodology. 2014;14:137

    2014

  21. [29]

    Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers

    Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Statistics in Medicine. 2014;33(3):517–35

    2014

  22. [30]

    Calibration: the Achilles heel of predictive analytics

    Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17(1):230. 41 Additional file 1: Supplementary Figures S1–S11. Supplementary Figure S1. Prediction instability at EPV 10 with 5-fold c...

    2019

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.