Graphical and numerical diagnostic tools to assess multiple imputation models by posterior predictive checking
Pith reviewed 2026-05-24 11:19 UTC · model grok-4.3
The pith
Posterior predictive checking diagnoses whether imputation models are congenial with the substantive model by verifying that observed data sit centrally in their predictive distributions.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper establishes that if the imputation model is congenial with the substantive model, the observed data are expected to be located in the centre of corresponding predictive posterior distributions, and provides graphical and numerical tools based on posterior predictive checking to assess this property for various imputation approaches and missing data scenarios.
What carries the argument
Posterior predictive checking that compares observed data with replicates generated from the posterior predictive distribution under the imputation model to assess central location.
If this is right
- The diagnostic applies equally to parametric and semi-parametric imputation approaches.
- It covers both continuous and discrete incomplete variables.
- The method handles univariate and multivariate missingness patterns.
- Simulation and application results support the method's ability to detect model congeniality across these cases.
Where Pith is reading between the lines
- The checks could be applied to imputation models for survival data or longitudinal structures not covered in the simulations.
- Software implementations of the graphical and numerical summaries would allow routine use alongside existing imputation workflows.
- The approach might be combined with other model-fit criteria to strengthen overall assessment of multiple imputation procedures.
Load-bearing premise
That the central location of observed data within the posterior predictive distribution reliably indicates congeniality between the imputation model and the substantive model.
What would settle it
A simulation study in which a known uncongenial imputation model produces observed data centered in the posterior predictive distribution, or a known congenial model does not, would falsify the diagnostic approach.
Figures
read the original abstract
Missing data are often dealt with multiple imputation. A crucial part of the multiple imputation process is selecting sensible models to generate plausible values for incomplete data. A method based on posterior predictive checking is proposed to diagnose imputation models based on posterior predictive checking. To assess the congeniality of imputation models, the proposed diagnostic method compares the observed data with their replicates generated under corresponding posterior predictive distributions. If the imputation model is congenial with the substantive model, the observed data are expected to be located in the centre of corresponding predictive posterior distributions. Simulation and application are designed to investigate the proposed diagnostic method for parametric and semi-parametric imputation approaches, continuous and discrete incomplete variables, univariate and multivariate missingness patterns. The results show the validity of the proposed diagnostic method.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes graphical and numerical diagnostic tools based on posterior predictive checking (PPC) to assess imputation models in multiple imputation. The method compares observed data to replicates from the posterior predictive distribution under the imputation model; congeniality with the substantive model is diagnosed when observed data lie in the centre of these distributions. Simulations and an application are presented to demonstrate validity for parametric/semi-parametric imputation, continuous/discrete variables, and univariate/multivariate missingness.
Significance. A reliable PPC-based diagnostic for imputation-model congeniality would address an important practical gap in multiple imputation workflows. The simulation-based validation approach is appropriate in principle, but the absence of quantitative performance metrics (effect sizes, power, false-positive rates) in the reported results limits the strength of the claim that the method has been shown to be valid.
major comments (3)
- [Abstract] Abstract: the claim that 'if the imputation model is congenial with the substantive model, the observed data are expected to be located in the centre of corresponding predictive posterior distributions' is presented without a derivation, explicit discrepancy measure, or operational definition of 'centre' (e.g., central probability interval, rank statistic, or graphical criterion). No justification is given for why centrality (rather than non-extremeness) follows from congeniality.
- [Abstract] Abstract / Method description: the PPD is described as generated under the imputation model, yet the diagnostic is intended to assess congeniality with the substantive model. The manuscript does not specify how (or whether) the substantive model enters the PPD construction or the test statistic.
- [Simulation study] Simulation and application results: the abstract asserts that 'simulations and an application demonstrate validity' but supplies no quantitative details on effect sizes, power, calibration of the diagnostic, or failure cases. This prevents evaluation of whether the method reliably detects incompatibility that matters for downstream inference.
minor comments (1)
- [Abstract] Abstract: the opening sentence repeats 'based on posterior predictive checking'.
Simulated Author's Rebuttal
We thank the referee for their constructive comments. We address each major comment below and indicate planned revisions to the manuscript.
read point-by-point responses
-
Referee: [Abstract] Abstract: the claim that 'if the imputation model is congenial with the substantive model, the observed data are expected to be located in the centre of corresponding predictive posterior distributions' is presented without a derivation, explicit discrepancy measure, or operational definition of 'centre' (e.g., central probability interval, rank statistic, or graphical criterion). No justification is given for why centrality (rather than non-extremeness) follows from congeniality.
Authors: We agree the abstract presents the claim concisely without full supporting details. The expectation of centrality follows from standard PPC theory: under a congenial imputation model the observed data should not be extreme in the posterior predictive distribution. In revision we will add an operational definition of 'centre' (e.g., lying inside the central 95% probability interval or via graphical inspection) together with a brief justification referencing the relevant PPC literature, and we will name the discrepancy measure employed. revision: yes
-
Referee: [Abstract] Abstract / Method description: the PPD is described as generated under the imputation model, yet the diagnostic is intended to assess congeniality with the substantive model. The manuscript does not specify how (or whether) the substantive model enters the PPD construction or the test statistic.
Authors: The PPDs are generated under the imputation model; congeniality is assessed by whether those distributions place the observed data centrally, which is expected only when the imputation model is compatible with the substantive analysis. We acknowledge the manuscript does not explicitly state the link. In revision we will clarify in the methods section that the substantive model guides the selection of variables and the focus of the diagnostic (e.g., by targeting parameters relevant to the substantive analysis in the chosen discrepancy measures), while the PPD generation itself remains under the imputation model. revision: yes
-
Referee: [Simulation study] Simulation and application results: the abstract asserts that 'simulations and an application demonstrate validity' but supplies no quantitative details on effect sizes, power, calibration of the diagnostic, or failure cases. This prevents evaluation of whether the method reliably detects incompatibility that matters for downstream inference.
Authors: The simulations illustrate expected behavior through graphical and numerical displays under congenial and uncongenial settings. We agree that formal quantitative metrics (e.g., proportion of cases correctly flagged, calibration under varying incompatibility levels) would strengthen the presentation. In the revised manuscript we will add such summaries to the simulation section to report effect sizes and calibration information. revision: yes
Circularity Check
No circularity: standard PPC application with no reduction to fitted inputs or self-citation chains
full rationale
The paper proposes a diagnostic that compares observed data to replicates from the posterior predictive distribution under the imputation model, expecting centrality when the imputation model is congenial with the substantive model. This expectation is presented as following from established posterior predictive checking principles rather than derived via any paper-specific equations, fitted parameters renamed as predictions, or self-citation load-bearing steps. No self-definitional loops, ansatz smuggling, or uniqueness theorems imported from the authors' prior work are evident in the abstract or described method. Simulations and applications are used to investigate validity, keeping the central claim independent of its own inputs. The derivation chain is self-contained against external benchmarks from the PPC literature.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Standard assumptions of the posterior predictive distribution under the imputation model hold and can be used to generate replicates.
- domain assumption Congeniality between imputation and substantive models is well-defined and detectable via central location of observed data in predictive distributions.
Reference graph
Works this paper leans on
-
[1]
abayomi2008diagnostics APACrefauthors Abayomi, K. , Gelman, A. \ Levy, M. APACrefauthors \ 2008 . Diagnostics for multivariate imputations Diagnostics for multivariate imputations . Journal of the Royal Statistical Society: Series C (Applied Statistics) 57 3 273--291
work page 2008
-
[2]
bartlett2020bootstrap APACrefauthors Bartlett, J W. \ Hughes, R A. APACrefauthors \ 2020 . Bootstrap inference for multiple imputation under uncongeniality and misspecification Bootstrap inference for multiple imputation under uncongeniality and misspecification . Statistical methods in medical research 29 12 3533--3546
work page 2020
-
[3]
bartlett2015multiple APACrefauthors Bartlett, J W. , Seaman, S R. , White, I R. , Carpenter, J R. \ Initiative*, A D N. APACrefauthors \ 2015 . Multiple imputation of covariates by fully conditional specification: Accommodating the substantive model Multiple imputation of covariates by fully conditional specification: Accommodating the substantive model ....
work page 2015
-
[4]
bondarenko2016graphical APACrefauthors Bondarenko, I. \ Raghunathan, T. APACrefauthors \ 2016 . Graphical and numerical diagnostic tools to assess suitability of multiple imputations and imputation models Graphical and numerical diagnostic tools to assess suitability of multiple imputations and imputation models . Statistics in medicine 35 17 3007--3020
work page 2016
- [5]
-
[6]
de2011handbook APACrefauthors De Waal, T. , Pannekoek, J. \ Scholtus, S. APACrefauthors \ 2011 . Handbook of statistical data editing and imputation Handbook of statistical data editing and imputation \ ( 563). John Wiley & Sons
work page 2011
-
[7]
gelman2013bayesian APACrefauthors Gelman, A. , Carlin, J B. , Stern, H S. , Dunson, D B. , Vehtari, A. \ Rubin, D B. APACrefauthors \ 2013 . Bayesian data analysis Bayesian data analysis . CRC press
work page 2013
-
[8]
gelman1998not APACrefauthors Gelman, A. , King, G. \ Liu, C. APACrefauthors \ 1998 . Not asked and not answered: Multiple imputation for multiple surveys Not asked and not answered: Multiple imputation for multiple surveys . Journal of the American Statistical Association 93 443 846--857
work page 1998
-
[9]
gelman2005multiple APACrefauthors Gelman, A. , Van Mechelen, I. , Verbeke, G. , Heitjan, D F. \ Meulders, M. APACrefauthors \ 2005 . Multiple imputation for model checking: completed-data plots with missing and latent data Multiple imputation for model checking: completed-data plots with missing and latent data . Biometrics 61 1 74--85
work page 2005
-
[10]
he2012diagnosing APACrefauthors He, Y. \ Zaslavsky, A M. APACrefauthors \ 2012 . Diagnosing imputation models by applying target analyses to posterior replicates of completed data Diagnosing imputation models by applying target analyses to posterior replicates of completed data . Statistics in medicine 31 1 1--18
work page 2012
-
[11]
heeringa2001multivariate APACrefauthors Heeringa, S G. APACrefauthors \ 2001 . Multivariate imputation of coarsened survey data on household wealth. Multivariate imputation of coarsened survey data on household wealth
work page 2001
-
[12]
hoogland2020handling APACrefauthors Hoogland, J. , van Barreveld, M. , Debray, T P. , Reitsma, J B. , Verstraelen, T E. , Dijkgraaf, M G. \ Zwinderman, A H. APACrefauthors \ 2020 . Handling missing predictor values when validating and applying a prediction model to new patients Handling missing predictor values when validating and applying a prediction mo...
work page 2020
-
[13]
krul2011self APACrefauthors Krul, A J. , Daanen, H A. \ Choi, H. APACrefauthors \ 2011 . Self-reported and measured weight, height and body mass index (BMI) in Italy, the Netherlands and North America Self-reported and measured weight, height and body mass index (bmi) in italy, the netherlands and north america . The European Journal of Public Health 21 4...
work page 2011
-
[14]
Little1988 APACrefauthors Little, R J A. APACrefauthors \ 1988 jul . Missing-Data Adjustments in Large Surveys Missing-data adjustments in large surveys . Journal of Business & Economic Statistics 6 3 287--296 . APACrefDOI doi:10.1080/07350015.1988.10509663 APACrefDOI
-
[15]
meng1994multiple APACrefauthors Meng, X L. APACrefauthors \ 1994 . Multiple-imputation inferences with uncongenial sources of input Multiple-imputation inferences with uncongenial sources of input . Statistical Science 538--558
work page 1994
-
[16]
nguyen2017model APACrefauthors Nguyen, C D. , Carlin, J B. \ Lee, K J. APACrefauthors \ 2017 . Model checking in multiple imputation: an overview and case study Model checking in multiple imputation: an overview and case study . Emerging themes in epidemiology 14 1 8
work page 2017
-
[17]
nguyen2015posterior APACrefauthors Nguyen, C D. , Lee, K J. \ Carlin, J B. APACrefauthors \ 2015 . Posterior predictive checking of multiple imputation models Posterior predictive checking of multiple imputation models . Biometrical Journal 57 4 676--694
work page 2015
-
[18]
RubinD1987 APACrefauthors Rubin, D B. APACrefauthors \ 1987 . Multiple Imputation for Nonresponse in Surveys Multiple imputation for nonresponse in surveys . New York John Wiley and Sons
work page 1987
-
[19]
Schouten2018 APACrefauthors Schouten, R M. , Lugtig, P. \ Vink, G. APACrefauthors \ 2018 . Generating missing values for simulation purposes: a multivariate amputation procedure Generating missing values for simulation purposes: a multivariate amputation procedure . Journal of Statistical Computation and Simulation 88 15 2909--2930
work page 2018
-
[20]
siddique2008multiple APACrefauthors Siddique, J. \ Belin, T R. APACrefauthors \ 2008 . Multiple imputation using an iterative hot-deck with distance-based donor selection Multiple imputation using an iterative hot-deck with distance-based donor selection . Statistics in medicine 27 1 83--102
work page 2008
-
[21]
su2011multiple APACrefauthors Su, Y S. , Gelman, A E. , Hill, J. \ Yajima, M. APACrefauthors \ 2011 . Multiple imputation with diagnostics (mi) in R: Opening windows into the black box Multiple imputation with diagnostics (mi) in r: Opening windows into the black box
work page 2011
-
[22]
van2007multiple APACrefauthors Van Buuren, S. APACrefauthors \ 2007 . Multiple imputation of discrete and continuous data by fully conditional specification Multiple imputation of discrete and continuous data by fully conditional specification . Statistical methods in medical research 16 3 219--242
work page 2007
-
[23]
Buuren2018 APACrefauthors van Buuren, S. APACrefauthors \ 2018 . Flexible Imputation of Missing Data, Second Edition Flexible imputation of missing data, second edition . Chapman and Hall/ CRC
work page 2018
-
[24]
van2011mice APACrefauthors Van Buuren, S. \ Groothuis-Oudshoorn, K. APACrefauthors \ 2011 . mice: Multivariate imputation by chained equations in R mice: Multivariate imputation by chained equations in r . Journal of statistical software 45 1 1--67
work page 2011
-
[25]
vink2014predictive APACrefauthors Vink, G. , Frank, L E. , Pannekoek, J. \ Van Buuren, S. APACrefauthors \ 2014 . Predictive mean matching imputation of semicontinuous variables Predictive mean matching imputation of semicontinuous variables . Statistica Neerlandica 68 1 61--90
work page 2014
-
[26]
vink2015partioned APACrefauthors Vink, G. , Lazendic, G. \ van Buuren, S. APACrefauthors \ 2015 . Partioned predictive mean matching as a large data multilevel imputation technique. Partioned predictive mean matching as a large data multilevel imputation technique. Psychological Test and Assessment Modeling 57 4 577--594
work page 2015
-
[27]
Vink2013 APACrefauthors Vink, G. \ van Buuren, S. APACrefauthors \ 2013 sep . Multiple Imputation of Squared Terms Multiple imputation of squared terms . Sociological Methods & Research 42 4 598--607 . APACrefDOI doi:10.1177/0049124113502943 APACrefDOI
-
[28]
volker2021anonymiced APACrefauthors Volker, T B. \ Vink, G. APACrefauthors \ 2021 . Anonymiced Shareable Data: Using mice to Create and Analyze Multiply Imputed Synthetic Datasets Anonymiced shareable data: Using mice to create and analyze multiply imputed synthetic datasets . Psych 3 4 703--716
work page 2021
-
[29]
white2011multiple APACrefauthors White, I R. , Royston, P. \ Wood, A M. APACrefauthors \ 2011 . Multiple imputation using chained equations: issues and guidance for practice Multiple imputation using chained equations: issues and guidance for practice . Statistics in medicine 30 4 377--399
work page 2011
-
[30]
xie2017dissecting APACrefauthors Xie, X. \ Meng, X L. APACrefauthors \ 2017 . Dissecting multiple imputation from a multi-phase inference perspective: what happens when God's, imputer's and analyst's models are uncongenial? Dissecting multiple imputation from a multi-phase inference perspective: what happens when god's, imputer's and analyst's models are ...
work page 2017
-
[31]
yu2007evaluation APACrefauthors Yu, L M. , Burton, A. \ Rivero-Arias, O. APACrefauthors \ 2007 . Evaluation of software for multiple imputation of semi-continuous data Evaluation of software for multiple imputation of semi-continuous data . Statistical Methods in Medical Research 16 3 243--258
work page 2007
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.