{"id":"3a064617-78ac-42a0-a3fa-3483ed121230","arxiv_id":"2412.15809","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"QOI-Check uses rank-based uniformity tests, in the style of simulation-based calibration, to validate post-estimation quantities of interest and their underlying population definition.","lead":"This paper introduces QOI-Check, a procedure for testing whether quantities calculated after fitting a Bayesian model, such as a marginal average or an effect decomposition, are implemented and interpreted correctly. It adapts simulation-based calibration to compare two ways of computing the same quantity, one from parameters and one from simulated predictions, and reports whether they agree.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The rank-uniformity null in Section 3 is asserted, not derived: when one side is a finite-sample predictive statistic, extra sampling noise breaks the SBC exchangeability argument, so the QOI-Check's pass/fail bands are uncalibrated.","rationale":"Both the reader's weakest assumption and my reading point to the same step: Section 3's 'as in SBC and HPC' sentence carries the entire validity of the QOI-Check. SBC's rank-uniformity theorem is a statement about the same test quantity evaluated on prior and posterior draws; it does not transfer to a comparison between a direct parameter function and a sample statistic of predicted data, because the latter has an extra source of randomness even conditional on the posterior draw. I verified the logic with a minimal conjugate model: the extra variance of the predictive statistic makes the rank distribution non-uniform. No analytical derivation, no asymptotic justification, and no calibration simulation is present in the paper to repair this gap. The two case studies use R=100 and R=20, far too small to detect the subtle non-uniformity, and they lack known-bug baselines, so the visual 'passes' do not provide evidence for the check. This is not a disagreement with any consensus; it is an internal missing derivation. The paper's central claim — that the QOI-Check 'ensures proper calibration' — therefore fails for the current version. I would keep the reader's REJECT verdict, while noting that a rigorous derivation of the null (or an explicitly asymptotic version with finite-grid corrections) plus a calibration study could make the idea publishable.","tokens_in":21839,"tokens_out":7765,"duration_ms":73588,"concrete_test":"Derive the null rank distribution in the normal-Normal example: with known σ, conditional on y, θ_prior and each posterior θ_s are iid N(μ_y,v), while the predictive sample mean of N_B observations is N(μ_y, v+σ²/N_B). Verify by simulation with exact posteriors that the rank of a N(μ_y,v) draw among N(μ_y, v+σ²/N_B) draws is not uniform for N_B=10 (e.g., R=10,000 replications, S=100 posterior draws), while the control rank of θ_prior among posterior θ_s is uniform. If the deviation appears, the Section 3 transfer is invalid; repeating with N_B=50 should show the deviation shrinking only asymptotically, not at the finite grids used in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Section 3's sentence: 'As in SBC and HPC, each of the two QOI-check version leads to inequality statements between one prior quantity and many posterior quantities, leading to an uniformity check.' SBC's uniformity proof requires that, conditional on the simulated data y_r, the prior-side quantity and each posterior-side quantity are draws from the same distribution. This holds when both sides are the same deterministic function of θ. It does not hold here. In the Prior-Derived Posterior-Predictive version, the prior side is f(θ_r) while the posterior side is g(˜y_{r,s}), a finite-sample predictive statistic (e.g., a mean over N_B reference-grid observations); the Prior-Predicted version has the asymmetry reversed. Even with a perfect sampler, the predictive statistic adds sampling noise beyond the posterior variation of θ. In a normal-Normal model with known σ, conditional on y, θ_prior and posterior θ_s are iid N(μ_y,v), but the posterior predictive sample mean of N_B observations is N(μ_y, v+σ²/N_B); the rank of a N(μ_y,v) draw among N(μ_y, v+σ²/N_B) draws is not uniform. Thus the asserted null does not follow from SBC, and the Säilynoja et al. (2022) bands used in Figures 8 and 10 are not the correct reference bands. HPC does not fill the gap: its calibration is asymptotic and specific to holdout diagnostics, not an exact uniform-rank statement for mixed parameter/predictive quantities. No theorem, asymptotic argument, or calibration simulation is supplied, and the case studies (R=100, R=20) have no known-bug baseline, so the observed 'passes' do not validate the check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces QOI-Check, a post-estimation validation procedure for quantities of interest in Bayesian models. It proposes two consistency checks: the Prior-Derived Posterior-Predictive version, which compares a direct parameter-based QOI against a posterior-predictive sample statistic, and the Prior-Predicted Posterior-Derived version, which compares a prior-predictive sample statistic against a posterior parameter-based QOI. The paper claims that each version leads to inequality statements whose ranks are uniformly distributed, as in SBC and HPC, and that these ranks can be assessed with the uniformity bands of Säilynoja et al. (2022). The method is demonstrated in two case studies, one involving a marginal expectation in a log-link multilevel model and one involving an ANOVA decomposition of a bivariate smooth term.","tokens_in":22138,"tokens_out":6183,"duration_ms":54505,"significance":"If the rank-uniformity claim were valid, the QOI-Check would fill a genuine gap in the Bayesian workflow by validating post-estimation calculations and linking them to explicit population definitions. The motivation is strong, the case studies are clearly described, and the use of existing SBC machinery is sensible; code is publicly referenced. However, the central calibration property is asserted rather than derived, and the argument that SBC-style uniformity continues to hold when one side is a finite-sample predictive statistic is not correct as stated. Without a valid null distribution for the proposed ranks, the pass/fail conclusions in the case studies are uninterpretable. The paper therefore does not currently establish its main claim.","major_comments":[{"comment":"The central uniformity claim is asserted in the sentence 'As in SBC and HPC, each of the two QOI-check version leads to inequality statements between one prior quantity and many posterior quantities, leading to an uniformity check.' This does not follow from SBC. SBC's rank-uniformity proof requires that, conditional on the simulated data y_r, the prior-side quantity and each posterior-side quantity are draws from the same distribution. That holds when both sides are the same deterministic function of the parameters, as in Modrák et al.'s Figure 3(b). It does not hold for the Prior-Derived Posterior-Predictive version, where the prior side is f(θ_r[θ(Da)_r]) and the posterior side is g(˜y(Db)_r,s), a finite-sample predictive statistic. For example, in a normal-Normal model with known σ and posterior θ|y ~ N(μ,v), the prior-side function of θ has variance v, while a posterior predictive sample mean of N_B observations has variance v + σ²/N_B; the rank of a draw from N(μ,v) among draws from N(μ,v+σ²/N_B) is not uniform. The finite reference grid adds sampling noise that breaks the exchangeability argument. No theorem, asymptotic derivation, or calibration simulation is provided to replace this missing justification.","section":"Section 3, Prior-Derived Posterior-Predictive Consistency Check"},{"comment":"The reversed version has the same defect. The prior side f(˜y(Da)_r) is a sample statistic of prior predictive draws and depends on θ_r (and possibly additional prior parameters), so conditional on y_r it is not drawn from the posterior distribution that generates the posterior side g(θ_r,s[θ(Db)_r,s]). The sentence 'By the same reasoning as for the first definition' does not supply a reasoning, and the two sides are not exchangeable. The reference to HPC also cannot rescue the argument: HPC calibration is asymptotic and specific to holdout predictive diagnostics, not an exact uniform-rank statement for mixed parameter/predictive quantities. The manuscript gives neither a proof nor a counterexample-free condition under which the proposed ranks are uniform.","section":"Section 3, Prior-Predicted Posterior-Derived Consistency Check"},{"comment":"The empirical ECDF bands in Figures 8 and 10 are computed under the asserted uniform null using Säilynoja et al. (2022). Because the null distribution is not established, the pass/fail statements in the Results (for example, 'E(b),sprior only passes the check for...' in Case Study I) are not interpretable as evidence for the method. A calibration experiment under known ground truth, or a derivation of the null, would be needed before these case studies can support the proposed check. In addition, R=100 simulation runs in Case Study I and R=20 in Case Study II are small for detecting deviations from uniformity across the many comparisons shown, particularly in Case Study II where the text dismisses failures as 'practically negligible' without a multiple-testing correction.","section":"Section 4, Figures 8 and 10"}],"minor_comments":[{"comment":"The first sentence of Section 3 contains the typo 'OOI-Check' instead of 'QOI-Check'.","section":"Section 3"},{"comment":"The captions of Figures 8 and 10 contain the misspelling 'cumulatite distribution function' instead of 'cumulative distribution function'.","section":"Figures 8 and 10"},{"comment":"The notation for the data structures is inconsistent: the text uses Da and Db in Section 3 but D(A) and D(B) in the Case Study I data-structure description, and N(A) and N(B) are introduced without being defined in the Notation list.","section":"Section 4, Case Study I"},{"comment":"The phrase 'Variante: (a) (b)' in the Figure 7 caption appears to be leftover text from an earlier draft and should be removed or translated.","section":"Figure 7 caption"},{"comment":"The phrase 'only passes the check with' and 'is calibrated according to' is not precise: what is actually assessed is the uniformity of ranks, not calibration of the QOI in the usual statistical sense. The wording should be clarified.","section":"Section 4, Results"}],"recommendation":"reject","confidential_remarks":"The main issue is the missing derivation of the uniformity null. This is not a local presentation problem: the method's central claim is false as stated, and the case-study conclusions depend on the invalid null. The paper could be resubmitted if the author derives sufficient conditions for rank uniformity (for example, exact equality of the relevant conditional distributions) or replaces the uniform-band check with a properly calibrated test, but that would be a substantially different contribution. The motivation and examples are good, and I do not see any novelty-disclosure or ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The QOI-Check is a real extension of SBC: Modrák et al. already allowed derived quantities of parameters, but this paper adds predictive QOIs computed on replicate and reference-grid data, and it uses the check to distinguish population definitions. That is a genuinely useful step for applied Bayesian workflow, and Case Study I is a worked example with a clear interpretation payoff (the 'typical farm' vs. marginal perspective is a real trap for ecologists). The R code is linked and the SBC package is used properly. Credit where due: the idea is sensible and the case studies are honest exploratory illustrations.\n\nThe soft spot is the one the reader flagged, and it is load-bearing. Section 3 asserts that each QOI-Check version leads to a uniformity check 'as in SBC and HPC', but the SBC rank-uniformity result requires the two ranked quantities to have the same conditional distribution. In the Prior-Derived Posterior-Predictive version you rank f(θ_r) against g(˜y_{r,s}), where g is a finite-sample predictive statistic. Even with a perfect sampler and the right model, a sample mean over N_B grid points has extra Monte Carlo noise, so its marginal distribution differs from the posterior distribution of the parameter function. The normal-Normal calculation in the stress-test note is correct: the rank of a N(μ, v) draw among N(μ, v+σ²/N_B) draws is not uniform. HPC does not rescue the claim, because its calibration is asymptotic and specific to holdout diagnostics, not an exchangeability argument for mixed parameter/predictive quantities. No theorem, asymptotic argument, or calibration simulation replaces that derivation, and the case studies (R=100 and R=20) contain no known-bug baseline, so the observed passes only show that the check does not scream on this example.\n\nThat said, the flaw may be fixable rather than fatal. The paper's contribution does not require the exact uniform-rank null in its current assert-it-and-run form; it requires either a correct derivation of the null (or a conservative band), or an empirical calibration study on known-good and known-bug implementations. The population-definition selection idea in Case Study I is valuable, and the software-implementation checking framing in Case Study II is a real use case.\n\nWho this is for: applied Bayesians who calculate marginal effects, averages over new groups, or ANOVA decompositions and want a sanity check on their post-estimation code. They will get a useful template, but they should not yet trust the bands. A serious referee should engage: this deserves revision, not acceptance as-is, but also not a desk rejection.","headline":"Useful extension of SBC to predictive QOIs, but the rank-uniformity null is asserted rather than derived; with a correct null or calibration study it could become publishable.","tokens_in":22732,"tokens_out":1293,"would_cite":false,"duration_ms":10943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62-08"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces the QOI-Check, a rank-uniformity consistency test for validating the implementation and interpretation of post-estimation quantities of interest in Bayesian models.","keywords":["Bayesian workflow","quantities of interest","simulation-based calibration","holdout predictive check","post-estimation validation","reference grid","uniformity check","model criticism"],"falsifier":"Run the QOI-Check on a correctly implemented model whose QOI has a closed-form solution, under reference grids of increasing size, say posterior predictive sample means computed from $N_{\\mathrm{grid}} = 5, 20, 100$ new units, while holding the model and prior fixed. If the prior-versus-posterior ranks leave the uniformity band at a rate that grows as the grid shrinks, the claimed uniform calibration does not hold for finite predictive statistics.","tokens_in":21575,"feed_emoji":"📊","tokens_out":7942,"duration_ms":65977,"temperature":0.7,"pith_summary":"This paper tries to establish that after a Bayesian model is fit, the quantities of interest computed from its posterior—marginal expectations, decomposed smooths, and similar derived quantities—can and should be subjected to their own calibration check, just as the sampler and the model are. The proposed QOI-Check compares one prior-side version of a QOI with many posterior-side versions of the same QOI, computes the rank of the prior quantity among the posterior quantities, and asks whether the ranks are uniform across repeated simulations. If the check passes, the QOI calculation and its interpretation with respect to a chosen, possibly hypothetical, population are consistent; if it fails, either the formula, the implementation, or the population definition is at fault. Two versions are given, one where the prior side is a direct parameter function and the posterior side is a predictive statistic, and one with the roles reversed. The method matters because post-estimation QOIs are custom code that standard SBC and holdout checks do not directly validate.","feed_headline":"New check validates Bayesian quantities computed after model fit","feed_subtitle":"Extends SBC and holdout checks to derived QOIs, testing calibration and population definition.","key_machinery":"The carrying object is the QOI-Check itself: a pair of rank-based consistency checks built from the same simulation loop as Simulation-Based Calibration. A reference grid data structure, a covariate and grouping layout that may include unseen levels and represents a hypothetical population, is the device that lets the check speak to population definitions. The comparison relies on the uniform-rank property: when the model, the QOI code, and the chosen population definition are all correct, the rank of one prior-side QOI within the posterior-side QOI values should be uniform over repeated simulation runs, and deviations from uniformity show up in graphical ECDF-with-band checks.","core_discovery":"The central claim is that consistency between the prior and posterior use of a QOI can be checked by ranking. In the prior-derived posterior-predictive version, the QOI as a direct function of the extended parameters, $f\\left(\\theta_r\\left[,\\theta(D_a)_r\\right]\\right)$, is compared with the same QOI as a sample statistic of posterior predictions on a data structure $D_b$, $g\\left(\\tilde{y}(D_b)_{r,s}\\right)$; in the prior-predicted posterior-derived version, the QOI as a sample statistic of prior predictions on $D_a$, $f\\left(\\tilde{y}(D_a)_r\\right)$, is compared with the QOI as a direct function of the posterior parameters, $g\\left(\\theta_{r,s}\\left[,\\theta(D_b)_{r,s}\\right]\\right)$. For each simulation run $r$ and posterior sample $s$, the prior quantity's rank among the posterior quantities is recorded, and the collection of ranks over $R$ runs is subjected to a uniformity check, as in SBC and HPC. The data structure on the prediction side, replicate structure or reference grid, is what encodes the population definition, and different population definitions yield different, identifiable QOI versions. The paper demonstrates in two case studies that the check passes for correctly matched QOI and population combinations and fails for mismatched ones, including a conditional perspective on a marginal expectation and an ANOVA decomposition of a bivariate smooth.","pith_inferences":["A natural testable extension is to derive the exact distribution of the finite-grid predictive statistic under the prior and to show how grid size controls the check's sensitivity to implementation errors; the paper leaves this unquantified.","The same rank-based logic could be applied to other post-estimation objects, such as posterior predictive $p$-values or ranking metrics, wherever a prior-side and posterior-side definition of the same quantity exist.","In practice the check could be packaged as an automated unit test for generated-quantities blocks in probabilistic programming languages, flagging any QOI whose ranks drift out of the uniformity band across simulation runs.","Because the reference grid is user-defined, the QOI-Check offers a way to make population-definition assumptions explicit and auditable, which is especially relevant for ecological and policy models where 'average effect' claims hinge on weighting schemes."],"forward_implications":["A researcher with a wrong QOI formula will see the check fail: in Case Study I the conditional expectation $\\exp(\\beta_0 + \\beta_1)$ passes only against its own posterior version and is rejected against every marginal-perspective version.","The check can tell competing population definitions apart: replicate-structure and reference-grid versions of the same QOI are calibrated only with the posterior statistic built on the matching data structure, so passing the check identifies the population the QOI actually describes.","For a correctly implemented model and QOI, the rank of the prior-side QOI among the posterior-side values is uniform, making the QOI-Check a direct extension of SBC from parameters to predictive quantities computed after estimation.","As a workflow step, the QOI-Check gives software developers and applied scientists a way to regression-test generated-quantities code without requiring closed-form analytic expressions for the QOI."],"supporting_citations":[{"why":"Supplies the SBC algorithm and the notion of derived quantities that the QOI-Check extends to predictive statistics.","marker":"(Modrák et al., 2023)"},{"why":"Provides the holdout predictive check framework and the calibrated-check logic used in the posterior-side comparisons.","marker":"(Moran et al., 2023)"},{"why":"Gives the rank-uniformity validation of SBC on which the QOI-Check's uniformity test rests.","marker":"(Talts et al., 2018)"},{"why":"Grounds the simulation-based view of posterior inference and the superpopulation interpretation of the prior.","marker":"(Rubin, 1984)"},{"why":"Is the probabilistic programming language used for model implementation and posterior sampling in the case studies.","marker":"(Carpenter et al., 2017)"},{"why":"Is the R interface whose prediction machinery, including sample_new_levels options, defines the reference-grid predictions.","marker":"(Bürkner, 2017, 2018)"},{"why":"Provides the graphical discrete-uniformity test and ECDF bands used to decide whether a QOI-Check passes.","marker":"(Säilynoja et al., 2022)"},{"why":"Supplies the smoothing-spline ANOVA decomposition and averaging operators used in Case Study II.","marker":"(Gu, 2002)"}],"fun_headline_variants":["QOI-Check: Validating Bayesian post-hoc quantities","New consistency check for Bayesian derived quantities","QOI-Check: Ensuring Bayesian QOIs are calibrated","Prior-posterior check for Bayesian effects after fit","Bayesian QOI validation: New consistency framework"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ranking a single prior-side QOI against posterior-side QOI values computed as finite-sample statistics on a reference grid yields exactly uniform ranks, the same property that holds for direct parameter comparisons in SBC; the paper asserts this uniformity without deriving or testing it.","fun_headline_variants_meta":{"raw":{"variants":["QOI-Check: Validating Bayesian post-hoc quantities","New consistency check for Bayesian derived quantities","QOI-Check: Ensuring Bayesian QOIs are calibrated","Prior-posterior check for Bayesian effects after fit","Bayesian QOI validation: New consistency framework"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1533,"prompt_tokens":1064,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":680,"tokens_out":469,"duration_ms":4246,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:05:21.860160+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the QOI-Check on a correctly implemented model whose QOI has a closed-form solution, under reference grids of increasing size, say posterior predictive sample means computed from $N_{\\mathrm{grid}} = 5, 20, 100$ new units, while holding the model and prior fixed. If the prior-versus-posterior ranks leave the uniformity band at a rate that grows as the grid shrinks, the claimed uniform calibration does not hold for finite predictive statistics.","supporting_citations":[],"review_version":1}