pith. sign in

arxiv: 2208.12929 · v1 · pith:Z4SCVC4Unew · submitted 2022-08-27 · 📊 stat.CO

Graphical and numerical diagnostic tools to assess multiple imputation models by posterior predictive checking

Pith reviewed 2026-05-24 11:19 UTC · model grok-4.3

classification 📊 stat.CO
keywords multiple imputationposterior predictive checkingimputation model diagnosticsmodel congenialitymissing datapredictive distributionssimulation validation
0
0 comments X

The pith

Posterior predictive checking diagnoses whether imputation models are congenial with the substantive model by verifying that observed data sit centrally in their predictive distributions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes a diagnostic method based on posterior predictive checking to evaluate imputation models used in multiple imputation for missing data. The approach generates replicates of the data from the posterior predictive distribution implied by the imputation model and checks the position of the observed data within those distributions. A sympathetic reader would care because choosing an appropriate imputation model is critical for valid statistical inferences from incomplete datasets. The paper demonstrates the method through simulations and applications covering parametric and semi-parametric models, different data types, and various missingness patterns. Results indicate that the observed data are centered in the predictive distributions when the models are congenial.

Core claim

The paper establishes that if the imputation model is congenial with the substantive model, the observed data are expected to be located in the centre of corresponding predictive posterior distributions, and provides graphical and numerical tools based on posterior predictive checking to assess this property for various imputation approaches and missing data scenarios.

What carries the argument

Posterior predictive checking that compares observed data with replicates generated from the posterior predictive distribution under the imputation model to assess central location.

If this is right

  • The diagnostic applies equally to parametric and semi-parametric imputation approaches.
  • It covers both continuous and discrete incomplete variables.
  • The method handles univariate and multivariate missingness patterns.
  • Simulation and application results support the method's ability to detect model congeniality across these cases.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The checks could be applied to imputation models for survival data or longitudinal structures not covered in the simulations.
  • Software implementations of the graphical and numerical summaries would allow routine use alongside existing imputation workflows.
  • The approach might be combined with other model-fit criteria to strengthen overall assessment of multiple imputation procedures.

Load-bearing premise

That the central location of observed data within the posterior predictive distribution reliably indicates congeniality between the imputation model and the substantive model.

What would settle it

A simulation study in which a known uncongenial imputation model produces observed data centered in the posterior predictive distribution, or a known congenial model does not, would falsify the diagnostic approach.

Figures

Figures reproduced from arXiv: 2208.12929 by Gerko Vink, Mingyang Cai, Stef van Buuren.

Figure 1
Figure 1. Figure 1: Main steps used in MICE (Van Buuren & Groothuis-Oudshoorn, 2011) solves a missing data problem by generating 3 imputed datasets. Three im￾puted datasets are generated with function mice(). Analysis are performed 11 [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution plots for the first simulation study (quadratic equa [PITH_FULL_IMAGE:figures/full_fig_p038_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Scatterplots and densityplots for the first simulation study [PITH_FULL_IMAGE:figures/full_fig_p039_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution plots for the first simulation study (quadratic equation [PITH_FULL_IMAGE:figures/full_fig_p040_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution plots for the second simulation study (quadratic equation with incomplete covariates) gener [PITH_FULL_IMAGE:figures/full_fig_p041_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution plots for the second simulation study (quadratic equation with incomplete covariates) gen [PITH_FULL_IMAGE:figures/full_fig_p042_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Scatterplots for the second simulation study (quadratic equation with incomplete covariates) generated [PITH_FULL_IMAGE:figures/full_fig_p043_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The plot of deviance residuals for the third simulation study (gen [PITH_FULL_IMAGE:figures/full_fig_p044_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Graphical analysis of the BMI data with imputation strategy case 1. [PITH_FULL_IMAGE:figures/full_fig_p045_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Graphical analysis of the BMI data with imputation strategy [PITH_FULL_IMAGE:figures/full_fig_p046_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Graphical analysis of the BMI data with imputation strategy [PITH_FULL_IMAGE:figures/full_fig_p047_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Graphical analysis of the BMI data with imputation strategy [PITH_FULL_IMAGE:figures/full_fig_p048_12.png] view at source ↗
read the original abstract

Missing data are often dealt with multiple imputation. A crucial part of the multiple imputation process is selecting sensible models to generate plausible values for incomplete data. A method based on posterior predictive checking is proposed to diagnose imputation models based on posterior predictive checking. To assess the congeniality of imputation models, the proposed diagnostic method compares the observed data with their replicates generated under corresponding posterior predictive distributions. If the imputation model is congenial with the substantive model, the observed data are expected to be located in the centre of corresponding predictive posterior distributions. Simulation and application are designed to investigate the proposed diagnostic method for parametric and semi-parametric imputation approaches, continuous and discrete incomplete variables, univariate and multivariate missingness patterns. The results show the validity of the proposed diagnostic method.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper proposes graphical and numerical diagnostic tools based on posterior predictive checking (PPC) to assess imputation models in multiple imputation. The method compares observed data to replicates from the posterior predictive distribution under the imputation model; congeniality with the substantive model is diagnosed when observed data lie in the centre of these distributions. Simulations and an application are presented to demonstrate validity for parametric/semi-parametric imputation, continuous/discrete variables, and univariate/multivariate missingness.

Significance. A reliable PPC-based diagnostic for imputation-model congeniality would address an important practical gap in multiple imputation workflows. The simulation-based validation approach is appropriate in principle, but the absence of quantitative performance metrics (effect sizes, power, false-positive rates) in the reported results limits the strength of the claim that the method has been shown to be valid.

major comments (3)
  1. [Abstract] Abstract: the claim that 'if the imputation model is congenial with the substantive model, the observed data are expected to be located in the centre of corresponding predictive posterior distributions' is presented without a derivation, explicit discrepancy measure, or operational definition of 'centre' (e.g., central probability interval, rank statistic, or graphical criterion). No justification is given for why centrality (rather than non-extremeness) follows from congeniality.
  2. [Abstract] Abstract / Method description: the PPD is described as generated under the imputation model, yet the diagnostic is intended to assess congeniality with the substantive model. The manuscript does not specify how (or whether) the substantive model enters the PPD construction or the test statistic.
  3. [Simulation study] Simulation and application results: the abstract asserts that 'simulations and an application demonstrate validity' but supplies no quantitative details on effect sizes, power, calibration of the diagnostic, or failure cases. This prevents evaluation of whether the method reliably detects incompatibility that matters for downstream inference.
minor comments (1)
  1. [Abstract] Abstract: the opening sentence repeats 'based on posterior predictive checking'.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments. We address each major comment below and indicate planned revisions to the manuscript.

read point-by-point responses
  1. Referee: [Abstract] Abstract: the claim that 'if the imputation model is congenial with the substantive model, the observed data are expected to be located in the centre of corresponding predictive posterior distributions' is presented without a derivation, explicit discrepancy measure, or operational definition of 'centre' (e.g., central probability interval, rank statistic, or graphical criterion). No justification is given for why centrality (rather than non-extremeness) follows from congeniality.

    Authors: We agree the abstract presents the claim concisely without full supporting details. The expectation of centrality follows from standard PPC theory: under a congenial imputation model the observed data should not be extreme in the posterior predictive distribution. In revision we will add an operational definition of 'centre' (e.g., lying inside the central 95% probability interval or via graphical inspection) together with a brief justification referencing the relevant PPC literature, and we will name the discrepancy measure employed. revision: yes

  2. Referee: [Abstract] Abstract / Method description: the PPD is described as generated under the imputation model, yet the diagnostic is intended to assess congeniality with the substantive model. The manuscript does not specify how (or whether) the substantive model enters the PPD construction or the test statistic.

    Authors: The PPDs are generated under the imputation model; congeniality is assessed by whether those distributions place the observed data centrally, which is expected only when the imputation model is compatible with the substantive analysis. We acknowledge the manuscript does not explicitly state the link. In revision we will clarify in the methods section that the substantive model guides the selection of variables and the focus of the diagnostic (e.g., by targeting parameters relevant to the substantive analysis in the chosen discrepancy measures), while the PPD generation itself remains under the imputation model. revision: yes

  3. Referee: [Simulation study] Simulation and application results: the abstract asserts that 'simulations and an application demonstrate validity' but supplies no quantitative details on effect sizes, power, calibration of the diagnostic, or failure cases. This prevents evaluation of whether the method reliably detects incompatibility that matters for downstream inference.

    Authors: The simulations illustrate expected behavior through graphical and numerical displays under congenial and uncongenial settings. We agree that formal quantitative metrics (e.g., proportion of cases correctly flagged, calibration under varying incompatibility levels) would strengthen the presentation. In the revised manuscript we will add such summaries to the simulation section to report effect sizes and calibration information. revision: yes

Circularity Check

0 steps flagged

No circularity: standard PPC application with no reduction to fitted inputs or self-citation chains

full rationale

The paper proposes a diagnostic that compares observed data to replicates from the posterior predictive distribution under the imputation model, expecting centrality when the imputation model is congenial with the substantive model. This expectation is presented as following from established posterior predictive checking principles rather than derived via any paper-specific equations, fitted parameters renamed as predictions, or self-citation load-bearing steps. No self-definitional loops, ansatz smuggling, or uniqueness theorems imported from the authors' prior work are evident in the abstract or described method. Simulations and applications are used to investigate validity, keeping the central claim independent of its own inputs. The derivation chain is self-contained against external benchmarks from the PPC literature.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The method rests on the standard Bayesian posterior predictive framework and the definition of congeniality between imputation and substantive models; no free parameters, invented entities, or ad-hoc axioms are visible from the abstract.

axioms (2)
  • domain assumption Standard assumptions of the posterior predictive distribution under the imputation model hold and can be used to generate replicates.
    Invoked when the method compares observed data to replicates generated under the posterior predictive distributions.
  • domain assumption Congeniality between imputation and substantive models is well-defined and detectable via central location of observed data in predictive distributions.
    Core premise stated in the abstract description of expected behavior.

pith-pipeline@v0.9.0 · 5653 in / 1276 out tokens · 27822 ms · 2026-05-24T11:19:38.210368+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages

  1. [1]

    , Gelman, A

    abayomi2008diagnostics APACrefauthors Abayomi, K. , Gelman, A. \ Levy, M. APACrefauthors \ 2008 . Diagnostics for multivariate imputations Diagnostics for multivariate imputations . Journal of the Royal Statistical Society: Series C (Applied Statistics) 57 3 273--291

  2. [2]

    \ Hughes, R A

    bartlett2020bootstrap APACrefauthors Bartlett, J W. \ Hughes, R A. APACrefauthors \ 2020 . Bootstrap inference for multiple imputation under uncongeniality and misspecification Bootstrap inference for multiple imputation under uncongeniality and misspecification . Statistical methods in medical research 29 12 3533--3546

  3. [3]

    , Seaman, S R

    bartlett2015multiple APACrefauthors Bartlett, J W. , Seaman, S R. , White, I R. , Carpenter, J R. \ Initiative*, A D N. APACrefauthors \ 2015 . Multiple imputation of covariates by fully conditional specification: Accommodating the substantive model Multiple imputation of covariates by fully conditional specification: Accommodating the substantive model ....

  4. [4]

    \ Raghunathan, T

    bondarenko2016graphical APACrefauthors Bondarenko, I. \ Raghunathan, T. APACrefauthors \ 2016 . Graphical and numerical diagnostic tools to assess suitability of multiple imputations and imputation models Graphical and numerical diagnostic tools to assess suitability of multiple imputations and imputation models . Statistics in medicine 35 17 3007--3020

  5. [5]

    \ Vink, G

    cai2022note APACrefauthors Cai, M. \ Vink, G. APACrefauthors \ 2022 . A note on imputing squares via polynomial combination approach A note on imputing squares via polynomial combination approach . Computational Statistics 1--17

  6. [6]

    , Pannekoek, J

    de2011handbook APACrefauthors De Waal, T. , Pannekoek, J. \ Scholtus, S. APACrefauthors \ 2011 . Handbook of statistical data editing and imputation Handbook of statistical data editing and imputation \ ( 563). John Wiley & Sons

  7. [7]

    , Carlin, J B

    gelman2013bayesian APACrefauthors Gelman, A. , Carlin, J B. , Stern, H S. , Dunson, D B. , Vehtari, A. \ Rubin, D B. APACrefauthors \ 2013 . Bayesian data analysis Bayesian data analysis . CRC press

  8. [8]

    , King, G

    gelman1998not APACrefauthors Gelman, A. , King, G. \ Liu, C. APACrefauthors \ 1998 . Not asked and not answered: Multiple imputation for multiple surveys Not asked and not answered: Multiple imputation for multiple surveys . Journal of the American Statistical Association 93 443 846--857

  9. [9]

    , Van Mechelen, I

    gelman2005multiple APACrefauthors Gelman, A. , Van Mechelen, I. , Verbeke, G. , Heitjan, D F. \ Meulders, M. APACrefauthors \ 2005 . Multiple imputation for model checking: completed-data plots with missing and latent data Multiple imputation for model checking: completed-data plots with missing and latent data . Biometrics 61 1 74--85

  10. [10]

    \ Zaslavsky, A M

    he2012diagnosing APACrefauthors He, Y. \ Zaslavsky, A M. APACrefauthors \ 2012 . Diagnosing imputation models by applying target analyses to posterior replicates of completed data Diagnosing imputation models by applying target analyses to posterior replicates of completed data . Statistics in medicine 31 1 1--18

  11. [11]

    APACrefauthors \ 2001

    heeringa2001multivariate APACrefauthors Heeringa, S G. APACrefauthors \ 2001 . Multivariate imputation of coarsened survey data on household wealth. Multivariate imputation of coarsened survey data on household wealth

  12. [12]

    , van Barreveld, M

    hoogland2020handling APACrefauthors Hoogland, J. , van Barreveld, M. , Debray, T P. , Reitsma, J B. , Verstraelen, T E. , Dijkgraaf, M G. \ Zwinderman, A H. APACrefauthors \ 2020 . Handling missing predictor values when validating and applying a prediction model to new patients Handling missing predictor values when validating and applying a prediction mo...

  13. [13]

    , Daanen, H A

    krul2011self APACrefauthors Krul, A J. , Daanen, H A. \ Choi, H. APACrefauthors \ 2011 . Self-reported and measured weight, height and body mass index (BMI) in Italy, the Netherlands and North America Self-reported and measured weight, height and body mass index (bmi) in italy, the netherlands and north america . The European Journal of Public Health 21 4...

  14. [14]

    APACrefauthors \ 1988 jul

    Little1988 APACrefauthors Little, R J A. APACrefauthors \ 1988 jul . Missing-Data Adjustments in Large Surveys Missing-data adjustments in large surveys . Journal of Business & Economic Statistics 6 3 287--296 . APACrefDOI doi:10.1080/07350015.1988.10509663 APACrefDOI

  15. [15]

    APACrefauthors \ 1994

    meng1994multiple APACrefauthors Meng, X L. APACrefauthors \ 1994 . Multiple-imputation inferences with uncongenial sources of input Multiple-imputation inferences with uncongenial sources of input . Statistical Science 538--558

  16. [16]

    , Carlin, J B

    nguyen2017model APACrefauthors Nguyen, C D. , Carlin, J B. \ Lee, K J. APACrefauthors \ 2017 . Model checking in multiple imputation: an overview and case study Model checking in multiple imputation: an overview and case study . Emerging themes in epidemiology 14 1 8

  17. [17]

    , Lee, K J

    nguyen2015posterior APACrefauthors Nguyen, C D. , Lee, K J. \ Carlin, J B. APACrefauthors \ 2015 . Posterior predictive checking of multiple imputation models Posterior predictive checking of multiple imputation models . Biometrical Journal 57 4 676--694

  18. [18]

    APACrefauthors \ 1987

    RubinD1987 APACrefauthors Rubin, D B. APACrefauthors \ 1987 . Multiple Imputation for Nonresponse in Surveys Multiple imputation for nonresponse in surveys . New York John Wiley and Sons

  19. [19]

    , Lugtig, P

    Schouten2018 APACrefauthors Schouten, R M. , Lugtig, P. \ Vink, G. APACrefauthors \ 2018 . Generating missing values for simulation purposes: a multivariate amputation procedure Generating missing values for simulation purposes: a multivariate amputation procedure . Journal of Statistical Computation and Simulation 88 15 2909--2930

  20. [20]

    \ Belin, T R

    siddique2008multiple APACrefauthors Siddique, J. \ Belin, T R. APACrefauthors \ 2008 . Multiple imputation using an iterative hot-deck with distance-based donor selection Multiple imputation using an iterative hot-deck with distance-based donor selection . Statistics in medicine 27 1 83--102

  21. [21]

    , Gelman, A E

    su2011multiple APACrefauthors Su, Y S. , Gelman, A E. , Hill, J. \ Yajima, M. APACrefauthors \ 2011 . Multiple imputation with diagnostics (mi) in R: Opening windows into the black box Multiple imputation with diagnostics (mi) in r: Opening windows into the black box

  22. [22]

    APACrefauthors \ 2007

    van2007multiple APACrefauthors Van Buuren, S. APACrefauthors \ 2007 . Multiple imputation of discrete and continuous data by fully conditional specification Multiple imputation of discrete and continuous data by fully conditional specification . Statistical methods in medical research 16 3 219--242

  23. [23]

    APACrefauthors \ 2018

    Buuren2018 APACrefauthors van Buuren, S. APACrefauthors \ 2018 . Flexible Imputation of Missing Data, Second Edition Flexible imputation of missing data, second edition . Chapman and Hall/ CRC

  24. [24]

    \ Groothuis-Oudshoorn, K

    van2011mice APACrefauthors Van Buuren, S. \ Groothuis-Oudshoorn, K. APACrefauthors \ 2011 . mice: Multivariate imputation by chained equations in R mice: Multivariate imputation by chained equations in r . Journal of statistical software 45 1 1--67

  25. [25]

    , Frank, L E

    vink2014predictive APACrefauthors Vink, G. , Frank, L E. , Pannekoek, J. \ Van Buuren, S. APACrefauthors \ 2014 . Predictive mean matching imputation of semicontinuous variables Predictive mean matching imputation of semicontinuous variables . Statistica Neerlandica 68 1 61--90

  26. [26]

    , Lazendic, G

    vink2015partioned APACrefauthors Vink, G. , Lazendic, G. \ van Buuren, S. APACrefauthors \ 2015 . Partioned predictive mean matching as a large data multilevel imputation technique. Partioned predictive mean matching as a large data multilevel imputation technique. Psychological Test and Assessment Modeling 57 4 577--594

  27. [27]

    \ van Buuren, S

    Vink2013 APACrefauthors Vink, G. \ van Buuren, S. APACrefauthors \ 2013 sep . Multiple Imputation of Squared Terms Multiple imputation of squared terms . Sociological Methods & Research 42 4 598--607 . APACrefDOI doi:10.1177/0049124113502943 APACrefDOI

  28. [28]

    \ Vink, G

    volker2021anonymiced APACrefauthors Volker, T B. \ Vink, G. APACrefauthors \ 2021 . Anonymiced Shareable Data: Using mice to Create and Analyze Multiply Imputed Synthetic Datasets Anonymiced shareable data: Using mice to create and analyze multiply imputed synthetic datasets . Psych 3 4 703--716

  29. [29]

    , Royston, P

    white2011multiple APACrefauthors White, I R. , Royston, P. \ Wood, A M. APACrefauthors \ 2011 . Multiple imputation using chained equations: issues and guidance for practice Multiple imputation using chained equations: issues and guidance for practice . Statistics in medicine 30 4 377--399

  30. [30]

    \ Meng, X L

    xie2017dissecting APACrefauthors Xie, X. \ Meng, X L. APACrefauthors \ 2017 . Dissecting multiple imputation from a multi-phase inference perspective: what happens when God's, imputer's and analyst's models are uncongenial? Dissecting multiple imputation from a multi-phase inference perspective: what happens when god's, imputer's and analyst's models are ...

  31. [31]

    , Burton, A

    yu2007evaluation APACrefauthors Yu, L M. , Burton, A. \ Rivero-Arias, O. APACrefauthors \ 2007 . Evaluation of software for multiple imputation of semi-continuous data Evaluation of software for multiple imputation of semi-continuous data . Statistical Methods in Medical Research 16 3 243--258