Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Prior-Posterior Derived-Predictive Consistency Checks for Post-Estimation Calculated Quantities of Interest (QOI-Check)

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces the QOI-Check, a rank-uniformity consistency test for validating the implementation and interpretation of post-estimation quantities of interest in Bayesian models.

desk verdict Useful extension of SBC to predictive QOIs, but the rank-uniformity null is asserted rather than derived; with a correct null or calibration study it could become publishable. read the letter →

arxiv 2412.15809 v2 pith:4CKONLWM submitted 2024-12-20 stat.ME

Holger Sennhenn-Reulen This is my paper · ORCID
classification stat.ME MSC 62F1562-08
keywords Bayesianworkflowquantitiesofinterestsimulation-basedcalibrationholdoutpredictivecheckpost-estimationvalidationreferencegriduniformitymodelcriticism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that after a Bayesian model is fit, the quantities of interest computed from its posterior—marginal expectations, decomposed smooths, and similar derived quantities—can and should be subjected to their own calibration check, just as the sampler and the model are. The proposed QOI-Check compares one prior-side version of a QOI with many posterior-side versions of the same QOI, computes the rank of the prior quantity among the posterior quantities, and asks whether the ranks are uniform across repeated simulations. If the check passes, the QOI calculation and its interpretation with respect to a chosen, possibly hypothetical, population are consistent; if it fails, either the formula, the implementation, or the population definition is at fault. Two versions are given, one where the prior side is a direct parameter function and the posterior side is a predictive statistic, and one with the roles reversed. The method matters because post-estimation QOIs are custom code that standard SBC and holdout checks do not directly validate.

What carries the argument

The carrying object is the QOI-Check itself: a pair of rank-based consistency checks built from the same simulation loop as Simulation-Based Calibration. A reference grid data structure, a covariate and grouping layout that may include unseen levels and represents a hypothetical population, is the device that lets the check speak to population definitions. The comparison relies on the uniform-rank property: when the model, the QOI code, and the chosen population definition are all correct, the rank of one prior-side QOI within the posterior-side QOI values should be uniform over repeated simulation runs, and deviations from uniformity show up in graphical ECDF-with-band checks.

What would settle it

Run the QOI-Check on a correctly implemented model whose QOI has a closed-form solution, under reference grids of increasing size, say posterior predictive sample means computed from $N_{\mathrm{grid}} = 5, 20, 100$ new units, while holding the model and prior fixed. If the prior-versus-posterior ranks leave the uniformity band at a rate that grows as the grid shrinks, the claimed uniform calibration does not hold for finite predictive statistics.

Watch

Extended reading notes

Core claim

The central claim is that consistency between the prior and posterior use of a QOI can be checked by ranking. In the prior-derived posterior-predictive version, the QOI as a direct function of the extended parameters, $f\left(\theta_r\left[,\theta(D_a)_r\right]\right)$, is compared with the same QOI as a sample statistic of posterior predictions on a data structure $D_b$, $g\left(\tilde{y}(D_b)_{r,s}\right)$; in the prior-predicted posterior-derived version, the QOI as a sample statistic of prior predictions on $D_a$, $f\left(\tilde{y}(D_a)_r\right)$, is compared with the QOI as a direct function of the posterior parameters, $g\left(\theta_{r,s}\left[,\theta(D_b)_{r,s}\right]\right)$. For each simulation run $r$ and posterior sample $s$, the prior quantity's rank among the posterior quantities is recorded, and the collection of ranks over $R$ runs is subjected to a uniformity check, as in SBC and HPC. The data structure on the prediction side, replicate structure or reference grid, is what encodes the population definition, and different population definitions yield different, identifiable QOI versions. The paper demonstrates in two case studies that the check passes for correctly matched QOI and population combinations and fails for mismatched ones, including a conditional perspective on a marginal expectation and an ANOVA decomposition of a bivariate smooth.

Load-bearing premise

The load-bearing premise is that ranking a single prior-side QOI against posterior-side QOI values computed as finite-sample statistics on a reference grid yields exactly uniform ranks, the same property that holds for direct parameter comparisons in SBC; the paper asserts this uniformity without deriving or testing it.

Editorial extensions

If this is right

  • A researcher with a wrong QOI formula will see the check fail: in Case Study I the conditional expectation $\exp(\beta_0 + \beta_1)$ passes only against its own posterior version and is rejected against every marginal-perspective version.
  • The check can tell competing population definitions apart: replicate-structure and reference-grid versions of the same QOI are calibrated only with the posterior statistic built on the matching data structure, so passing the check identifies the population the QOI actually describes.
  • For a correctly implemented model and QOI, the rank of the prior-side QOI among the posterior-side values is uniform, making the QOI-Check a direct extension of SBC from parameters to predictive quantities computed after estimation.
  • As a workflow step, the QOI-Check gives software developers and applied scientists a way to regression-test generated-quantities code without requiring closed-form analytic expressions for the QOI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to derive the exact distribution of the finite-grid predictive statistic under the prior and to show how grid size controls the check's sensitivity to implementation errors; the paper leaves this unquantified.
  • The same rank-based logic could be applied to other post-estimation objects, such as posterior predictive $p$-values or ranking metrics, wherever a prior-side and posterior-side definition of the same quantity exist.
  • In practice the check could be packaged as an automated unit test for generated-quantities blocks in probabilistic programming languages, flagging any QOI whose ranks drift out of the uniformity band across simulation runs.
  • Because the reference grid is user-defined, the QOI-Check offers a way to make population-definition assumptions explicit and auditable, which is especially relevant for ecological and policy models where 'average effect' claims hinge on weighting schemes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces QOI-Check, a post-estimation validation procedure for quantities of interest in Bayesian models. It proposes two consistency checks: the Prior-Derived Posterior-Predictive version, which compares a direct parameter-based QOI against a posterior-predictive sample statistic, and the Prior-Predicted Posterior-Derived version, which compares a prior-predictive sample statistic against a posterior parameter-based QOI. The paper claims that each version leads to inequality statements whose ranks are uniformly distributed, as in SBC and HPC, and that these ranks can be assessed with the uniformity bands of Säilynoja et al. (2022). The method is demonstrated in two case studies, one involving a marginal expectation in a log-link multilevel model and one involving an ANOVA decomposition of a bivariate smooth term.

Significance. If the rank-uniformity claim were valid, the QOI-Check would fill a genuine gap in the Bayesian workflow by validating post-estimation calculations and linking them to explicit population definitions. The motivation is strong, the case studies are clearly described, and the use of existing SBC machinery is sensible; code is publicly referenced. However, the central calibration property is asserted rather than derived, and the argument that SBC-style uniformity continues to hold when one side is a finite-sample predictive statistic is not correct as stated. Without a valid null distribution for the proposed ranks, the pass/fail conclusions in the case studies are uninterpretable. The paper therefore does not currently establish its main claim.

major comments (3)
  1. [Section 3, Prior-Derived Posterior-Predictive Consistency Check] The central uniformity claim is asserted in the sentence 'As in SBC and HPC, each of the two QOI-check version leads to inequality statements between one prior quantity and many posterior quantities, leading to an uniformity check.' This does not follow from SBC. SBC's rank-uniformity proof requires that, conditional on the simulated data y_r, the prior-side quantity and each posterior-side quantity are draws from the same distribution. That holds when both sides are the same deterministic function of the parameters, as in Modrák et al.'s Figure 3(b). It does not hold for the Prior-Derived Posterior-Predictive version, where the prior side is f(θ_r[θ(Da)_r]) and the posterior side is g(˜y(Db)_r,s), a finite-sample predictive statistic. For example, in a normal-Normal model with known σ and posterior θ|y ~ N(μ,v), the prior-side function of θ has variance v, while a posterior predictive sample mean of N_B observations has variance v + σ²/N_B; the rank of a draw from N(μ,v) among draws from N(μ,v+σ²/N_B) is not uniform. The finite reference grid adds sampling noise that breaks the exchangeability argument. No theorem, asymptotic derivation, or calibration simulation is provided to replace this missing justification.
  2. [Section 3, Prior-Predicted Posterior-Derived Consistency Check] The reversed version has the same defect. The prior side f(˜y(Da)_r) is a sample statistic of prior predictive draws and depends on θ_r (and possibly additional prior parameters), so conditional on y_r it is not drawn from the posterior distribution that generates the posterior side g(θ_r,s[θ(Db)_r,s]). The sentence 'By the same reasoning as for the first definition' does not supply a reasoning, and the two sides are not exchangeable. The reference to HPC also cannot rescue the argument: HPC calibration is asymptotic and specific to holdout predictive diagnostics, not an exact uniform-rank statement for mixed parameter/predictive quantities. The manuscript gives neither a proof nor a counterexample-free condition under which the proposed ranks are uniform.
  3. [Section 4, Figures 8 and 10] The empirical ECDF bands in Figures 8 and 10 are computed under the asserted uniform null using Säilynoja et al. (2022). Because the null distribution is not established, the pass/fail statements in the Results (for example, 'E(b),sprior only passes the check for...' in Case Study I) are not interpretable as evidence for the method. A calibration experiment under known ground truth, or a derivation of the null, would be needed before these case studies can support the proposed check. In addition, R=100 simulation runs in Case Study I and R=20 in Case Study II are small for detecting deviations from uniformity across the many comparisons shown, particularly in Case Study II where the text dismisses failures as 'practically negligible' without a multiple-testing correction.
minor comments (5)
  1. [Section 3] The first sentence of Section 3 contains the typo 'OOI-Check' instead of 'QOI-Check'.
  2. [Figures 8 and 10] The captions of Figures 8 and 10 contain the misspelling 'cumulatite distribution function' instead of 'cumulative distribution function'.
  3. [Section 4, Case Study I] The notation for the data structures is inconsistent: the text uses Da and Db in Section 3 but D(A) and D(B) in the Case Study I data-structure description, and N(A) and N(B) are introduced without being defined in the Notation list.
  4. [Figure 7 caption] The phrase 'Variante: (a) (b)' in the Figure 7 caption appears to be leftover text from an earlier draft and should be removed or translated.
  5. [Section 4, Results] The phrase 'only passes the check with' and 'is calibrated according to' is not precise: what is actually assessed is the uniformity of ranks, not calibration of the QOI in the usual statistical sense. The wording should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: QOI-Check is an internal consistency diagnostic with no fitted parameters, no self-citations, and no derivation that reduces to its inputs; the Section 3 uniformity assumption is a validity concern, not a circular step.

full rationale

The QOI-Check compares two calculation routes for the same quantity within one model: a direct parameter-based QOI and a predictive sample-statistic QOI, as defined by the inequalities f(θ_r[,θ(Da)_r]) < g(˜y(Db)_{r,s}) and f(˜y(Da)_r) < g(θ_{r,s}[,θ(Db)_{r,s}]) in Section 3. Neither route is fitted to the other, no calibration parameters are estimated, and the paper does not rest on the author's own prior work, so the fitted-input and self-citation circularity patterns do not apply. The load-bearing assertion "As in SBC and HPC, each of the two QOI-check version leads to inequality statements between one prior quantity and many posterior quantities, leading to an uniformity check" is an assumed transfer of SBC's rank-uniformity property to a setting where one side is a finite-sample predictive statistic; with finite reference grids the two sides need not be exchangeable draws from the same distribution, so the check may be miscalibrated. That is a technical validity and correctness objection, not a demonstration that the paper's conclusion is equivalent to its input by construction. The self-consistency limitation, namely that errors common to both implementation routes will not be detected, is inherent to calibration checks and does not make the derivation circular. Therefore no circularity is established by the quoted text.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central method rests on an unproven transfer of SBC's exchangeability argument to predictive sample statistics. The simulation priors and grid sizes are arbitrary inputs to the demonstrations, not fitted parameters. No new entities are introduced.

free parameters (1)
  • Reference grid size (N(B) and N(A) in Case Study I; N=10000 in Case Study II) = N(B)=200, N(A)=500 in Case Study I; N=10000 in Case Study II
    The uniformity of the predictive-statistic version depends on the Monte Carlo error of the sample mean; the paper does not derive how large the reference grid must be for the SBC null to hold approximately.
assumptions (3)
  • ad hoc to paper SBC's uniform-rank property extends to predictive sample statistics with finite reference grids.
    Section 3 invokes the SBC/HPC analogy without proving that the rank of a prior-derived QOI among posterior-predictive sample statistics is uniform. This is the central unstated premise.
  • domain assumption The prior is generative and proper.
    The simulation-based checks require sampling from the prior; the paper notes that non-generative priors are excluded.
  • domain assumption Stan posterior samples are sufficiently accurate to not affect the uniformity check.
    SBC checks are sensitive to posterior sampling error; the paper does not report convergence diagnostics for the case study posteriors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prior-Posterior Derived-Predictive Consistency Checks for Post-Estimation Calculated Quantities of Interest (QOI-Check)." pith.science (2026). https://pith.science/paper/4CKONLWM

@misc{pith2026241215809,
  author       = {Pith},
  title        = {Pith review of: Prior-Posterior Derived-Predictive Consistency Checks for Post-Estimation Calculated Quantities of Interest (QOI-Check)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4CKONLWM}},
  note         = {Machine review of arXiv:2412.15809}
}
read the original abstract

With flexible modeling software - such as the probabilistic programming language Stan - growing in popularity, quantities of interest (QOIs) calculated post-estimation are increasingly desired and customly implemented, both by statistical software developers and applied scientists. Examples of QOI include the marginal expectation of a multilevel model with a non-linear link function, or an ANOVA decomposition of a bivariate regression spline. For this, the QOI-Check is introduced, a systematic approach to ensure proper calibration and correct interpretation of QOIs. It contributes to Bayesian Workflow, and aims to improve the interpretability and trust in post-estimation conclusions based on QOIs. The QOI-Check builds upon Simulation Based Calibration (SBC), and the Holdout Predictive Check (HPC). SBC verifies computational reliability of Bayesian inference algorithms by consistency check of posterior with prior when the posterior is estimated on prior-predicted data, while HPC ensures robust inference by assessing consistency of model predictions with holdout data. SBC and HPC are combined in QOI-Checking for validating post-estimation QOI calculation and interpretation in the context of a (hypothetical) population definition underlying the QOI.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Statistical consistency of sign-switching vacuum energy with cosmological observations

    astro-ph.CO 2026-03 conditional novelty 5.0 of 10

    Using exact non-Gaussian shift and posterior predictive tests, ΛsCDM improves CMB–BAO consistency and softens but does not resolve the Hubble tension, which persists at p≈4×10⁻⁴.

Reference graph

Works this paper leans on

29 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    B \"u rkner

    P.-C. B \"u rkner. 2017. brms: An R Package for Bayesian Multilevel Models Using Stan . Journal of Statistical Software. \\ https://doi.org/10.18637/jss.v080.i01

    doi:10.18637/jss.v080.i01 2017

  2. [2]

    B \"u rkner

    P.-C. B \"u rkner. 2018. Advanced Bayesian Multilevel Modeling with the R Package brms . The R Journal. \\ https://doi.org/10.32614/RJ-2018-017

    doi:10.32614/rj-2018-017 2018

  3. [3]

    Carpenter, A

    B. Carpenter, A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell. 2017. Stan: A Probabilistic Programming Language . Journal of Statistical Software. \\ https://doi.org/10.18637/jss.v076.i01

    doi:10.18637/jss.v076.i01 2017

  4. [4]

    J. S. Clark. 2004. Why environmental scientists are becoming Bayesians . Ecology Letters. \\ http://dx.doi.org/10.1111/j.1461-0248.2004.00702.x

    arXiv 2004

  5. [5]

    Gabry, D

    J. Gabry, D. Simpson, A. Vehtari, M. Betancourt, and A. Gelman. 2019. Visualization in Bayesian Workflow . Journal of the Royal Statistical Society Series A: Statistics in Society. \\ https://doi.org/10.1111/rssa.12378

    doi:10.1111/rssa.12378 2019

  6. [6]

    Gelman, J

    A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin. 2013. Bayesian Data Analysis . Chapman and Hall/CRC. \\ http://doi.org/10.1201/b16018

    doi:10.1201/b16018 2013

  7. [7]

    Gelman, A

    A. Gelman, A. Vehtari, D. Simpson, C. C. Margossian, B. Carpenter, Y. Yao, L. Kennedy, J. Gabry, P.-C. Bürkner, and M. Modrák. 2020. Bayesian Workflow . arXiv:2011.01808. \\ https://arxiv.org/abs/2011.01808

    arXiv 2020

  8. [8]

    C. Gu. 2002. Smoothing Spline ANOVA Models . Springer New York (Springer Series in Statistics). \\ http://dx.doi.org/10.1007/978-1-4757-3683-0

    doi:10.1007/978-1-4757-3683-0 2002

Show all 29 references
  1. [9]

    M. D. Hoffman and A. Gelman. The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo . Journal of Machine Learning Research 15 0 (47): 0 1593--1623 2014. \\ http://jmlr.org/papers/v15/hoffman14a.html

    2014

  2. [10]

    Li and J

    J. Li and J. H. Huggins. 2024. Calibrated Model Criticism Using Split Predictive Checks . arXiv:2203.15897. \\ https://arxiv.org/abs/2203.15897

    2024 arXiv

  3. [11]

    R. J. A. Little. 1993. Post-Stratification: A Modeler's Perspective . Journal of the American Statistical Association. \\ http://doi.org/10.1080/01621459.1993.10476368

    1993

  4. [12]

    Modrák, A

    M. Modrák, A. H. Moon, S. Kim, P. Bürkner, N. Huurre, K. Faltejsková, A. Gelman, and A. Vehtari. 2023. Simulation-Based Calibration Checking for Bayesian Computation: The Choice of Test Quantities Shapes Sensitivity . Bayesian Analysis. \\ http://dx.doi.org/10.1214/23-BA1404

    2023 doi

  5. [13]

    C. C. Monnahan, J. T. Thorson, and T. A. Branch. 2016. Faster estimation of Bayesian models in ecology using Hamiltonian Monte Carlo . Methods in Ecology and Evolution. \\ http://dx.doi.org/10.1111/2041-210X.12681

    2016 doi

  6. [14]

    G. E. Moran, D. M. Blei, and R. Ranganath. 2023. Holdout predictive checks for Bayesian model criticism . Journal of the Royal Statistical Society Series B: Statistical Methodology. \\ https://doi.org/10.1093/jrsssb/qkad105

    2023 doi

  7. [15]

    R. Neal. 2011. Handbook of Markov Chain Monte Carlo . CRC Press

    2011

  8. [16]

    R: A Language and Environment for Statistical Computing (Version 4.4.1)

    R Core Team . R: A Language and Environment for Statistical Computing (Version 4.4.1) . R Foundation for Statistical Computing 2024

    2024

  9. [17]

    D. Rubin. 1984. Bayesianly justifiable and relevant frequency calculations for the applied statistician. The Annals of Statistics

    1984

  10. [18]

    a ilynoja, P.-C. B\

    T. S\" a ilynoja, P.-C. B\" u rkner, and A. Vehtari. 2022. Graphical test for discrete uniformity and its applications in goodness-of-fit evaluation and multiple sample comparison. Statistics and Computing. \\ http://dx.doi.org/10.1007/s11222-022-10090-6

    2022 doi

  11. [19]

    E. G. Simmonds, K. P. Adjei, B. Cretois, L. Dickel, R. González-Gil, J. H. Laverick, C. P. Mandeville, E. G. Mandeville, O. Ovaskainen, J. Sicacha-Parada, E. S. Skarstein, and B. O’Hara. 2024. Recommendations for quantitative uncertainty consideration in ecology and evolution....

    2024 doi

  12. [20]

    Stauffer, G

    R. Stauffer, G. J. Mayr, M. Dabernig, and A. Zeileis. 2009. Somewhere over the Rainbow: How to Make Effective Use of Colors in Meteorological Visualizations . Bulletin of the American Meteorological Society

    2009

  13. [21]

    Talts, M

    S. Talts, M. Betancourt, D. Simpson, A. Vehtari, and A. Gelman. 2018. Validating Bayesian Inference Algorithms with Simulation-Based Calibration . arXiv. \\ https://arxiv.org/abs/1804.06788

    2018 arXiv

  14. [22]

    H. Wickham. 2011. The Split-Apply-Combine Strategy for Data Analysis . Journal of Statistical Software. \\ https://www.jstatsoft.org/v40/i01/

    2011

  15. [23]

    H. Wickham. ggplot2: Elegant Graphics for Data Analysis . Springer-Verlag New York 2016. ISBN 978-3-319-24277-4. \\ https://ggplot2.tidyverse.org

    2016

  16. [24]

    C. O. Wilke. cowplot: Streamlined Plot Theme and Plot Annotations for 'ggplot2' 2023. \\ https://CRAN.R-project.org/package=cowplot. R package version 1.1.2

    2023

  17. [25]

    S. N. Wood. 2003. Thin Plate Regression Splines . Journal of the Royal Statistical Society Series B: Statistical Methodology. \\ https://doi.org/10.1111/1467-9868.00374

    2003

  18. [26]

    S. N. Wood. 2011. Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models . Journal of the Royal Statistical Society (B)

    2011

  19. [27]

    S. N. Wood. 2024. mgcv package, PDF documentation . Cran repository. \\ https://cran.r-project.org/web/packages/mgcv/mgcv.pdf

    2024

  20. [28]

    A. F. Zuur, E. N. Ieno, N. Walker, A. A. Saveliev, and G. M. Smith. 2009. Mixed effects models and extensions in ecology with R . Springer New York

    2009

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.