{"id":"5954cc8e-3ac7-4c03-bedc-b58de0a6a72b","arxiv_id":"1908.00882","paper_version":5,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Population predictive checks compare a Bayesian model's posterior predictions to resampled or held-out data, avoiding the double use of data that makes ordinary posterior predictive checks overconfident.","lead":"The authors introduce a way to test Bayesian models by comparing their predictions against data the model has never seen, instead of against the same data used to fit it. This makes model checks less likely to reward overfitting, a common failure of standard posterior predictive checks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ideal Pop-PC calibration: Eq 9 p-values are not uniform even in a correctly specified normal model; the abstract's central claim is unproven and apparently false.","rationale":"The reader's verdict was CONDITIONAL because the calibration proof is absent and the Section 4 estimators are unanalyzed. My stress-test finds a stronger problem: the ideal check, which is the object whose calibration is advertised, is not calibrated under the standard definition the paper itself invokes in Section 3.1. In a correctly specified normal-normal model with F equal to the true conditional distribution at θ0=0, the Eq 9 p-value is a transformation Φ(cZ) with c ≠ 1, so the p-value distribution is not uniform. The exact constant differs depending on whether ynew is integrated out as in Eq 9 or fixed as in Algorithm 1, but in neither reading is c=1 for a proper prior. Consequently the central claim is not merely unproved; it is false as stated, and the bootstrap and cross-validation estimators in Section 4 are secondary because they approximate an ideal check that already lacks the advertised property. One caveat is that the paper never defines 'properly calibrated' explicitly, but the uniformity language in Section 3.1 and the contrast with the PPC make the intended meaning clear, and no alternative definition is supplied. The proposed simulation directly tests the two natural readings of Eq 9 and would settle whether the concern lands; the analytical calculation already gives a definitive answer for this example. For these reasons I would adjust the verdict to REJECT rather than CONDITIONAL.","tokens_in":15765,"tokens_out":16046,"duration_ms":168376,"concrete_test":"Simulate the normal-normal example: draw yobs and ynew independently from N(0,1) for S=10^5 replicates. For each replicate compute (a) p_eq9 = Φ(0.5·yobs / sqrt(2.5)), the Eq 9 value with ynew integrated out, and (b) p_cond = Φ((0.5·yobs − ynew) / sqrt(1.5)), the value conditional on the held-out ynew as used in Algorithm 1. Run Kolmogorov–Smirnov tests of both collections against Uniform(0,1) and report P(p ≤ 0.05). The analytical predictions are p_eq9 ∼ Φ(0.316Z) and p_cond ∼ Φ(0.913Z), both non-uniform; if the simulation confirms this, the paper's calibration claim is false as stated.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's advertised contribution is that the Pop-PC is 'properly calibrated' (abstract), and Section 3.1 defines interpretable p-values as having a uniform reference distribution. No theorem or proof appears in Section 3.2 or anywhere in the manuscript. The concern is not just a missing proof: the ideal object fails the stated standard. Take the correctly specified model y|μ∼N(μ,1), μ∼N(0,1), population F=N(0,1) (θ0=0), n=1, discrepancy d(y)=y. Then Yrep|yobs∼N(κ yobs, σ_p^2) with κ=1/2, σ_p^2=3/2. If the check is computed as Eq 9, pideal(yobs,F)=P(Yrep>Ynew|yobs,F)=Φ(κ yobs/√(σ_p^2+1))=Φ(0.316·yobs), so over yobs∼F, p=Φ(0.316Z) and P(p≤0.05)≈10^{-7}, not 0.05. If one instead conditions on the sampled held-out ynew, as Algorithm 1 does, p=Φ((0.5 yobs−ynew)/1.225)=Φ(0.913Z) after standardizing, also not uniform. Thus the ideal Pop-PC is not properly calibrated under the definition the paper itself uses; the bootstrap/CV estimators in Section 4 cannot inherit a calibration property the ideal check does not have. This undermines the central claim, independently of how well Algorithm 1 approximates Eq 9.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the population predictive check (Pop-PC), a model criticism procedure that replaces the reference data in a posterior predictive check (PPC) by a draw from the true population distribution. Section 3 defines an ideal Pop-PC (Eqs. 7 and 9), in which replicated data from the posterior predictive distribution are compared with new data from the population distribution F, and Section 4 proposes bootstrap, cross-validation, double-bootstrap, and p-bootstrap estimators that approximate F with the empirical distribution of the observed sample. Section 3.4 extends the construction to realized discrepancies and to hierarchical models with local latent variables. The abstract claims that, unlike the PPC, the Pop-PC is 'properly calibrated'; the paper presents empirical studies on Bayesian linear regression and a topic model suggesting that Pop-PCs detect overfitting where PPCs do not.","tokens_in":16079,"tokens_out":5156,"duration_ms":52309,"significance":"The idea of checking a posterior predictive distribution against a reference or held-out population is natural, and the paper's taxonomy of bootstrap and cross-validation estimators is clearly presented and potentially useful. If the calibration claim were true, the paper would address a well-known limitation of posterior predictive checks and supply interpretable p-values. The empirical demonstrations are suggestive. However, the central theoretical claim is not proved anywhere in the manuscript, and under the paper's own definition of interpretable p-values it is false: the ideal Pop-PC p-value of Eq. 9 is not uniform even in a correctly specified location model. Because this claim is the advertised contribution and is the basis for the practical estimators in Section 4, the manuscript does not establish its central thesis.","major_comments":[{"comment":"The abstract states that 'we prove that the Pop-PC is properly calibrated,' and Section 3.1 defines interpretable p-values as those with a uniform reference distribution, citing Robins et al. (2000). No theorem or proof of uniformity appears in Section 3.2 or anywhere else, and the statement is false under the paper's own definition. Consider the correctly specified model y|μ∼N(μ,1), μ∼N(0,1), population F=N(0,1), n=1, and discrepancy d(y)=y. The posterior predictive is Yrep|yobs∼N(yobs/2, 3/2), and Ynew∼N(0,1). Eq. (9) gives p_ideal(yobs,F)=P(Yrep>Ynew|yobs,F)=Φ(0.316·yobs). Over yobs∼F, this equals Φ(0.316Z) with Z∼N(0,1), so P(p≤0.05)≈10^{-7}, not 0.05. If instead one conditions on the sampled held-out ynew, as Algorithm 1 does, the p-value is Φ(0.913Z), which is also not uniform. Thus the ideal Pop-PC is not properly calibrated under the standard stated in Section 3.1, and the bootstrap/CV estimators in Section 4 cannot inherit a property the ideal check does not have.","section":"Abstract; Section 3.1, p. 8; Section 3.2, Eq. (9)"},{"comment":"The estimators replace the population F with the empirical distribution of the finite pool y and resample both yobs and ynew from y, but no analysis is provided for the resulting p-value's reference distribution. In Algorithm 1, θ is drawn from p(θ|yobs) where yobs is a bootstrap or cross-validation subsample of y, so the posterior and the reference data are dependent through the same finite pool; this is a different object from the ideal check of Eq. (7). The paper asserts that the bootstrap and cross-validation variants approximate the ideal check and that the 632 estimator is closest in simulation, but it supplies no consistency, asymptotic-uniformity, or calibration result for any variant. Consequently, the practical claim that the computed checks are 'properly calibrated' is not established, and the simulation evidence in Section 5 cannot substitute for this missing theory.","section":"Section 4, Algorithm 1 and surrounding text"},{"comment":"The per-group realized-discrepancy check draws znew_j from p(z|yobs_j, θ) while ynew_j is drawn from F_j, so znew_j and ynew_j are not generated from a common model; the distribution of d(Ynew_j, Znew_j, θ) is neither a posterior predictive draw nor a pure population draw. No property of this mixed distribution is stated, and it is unclear what calibration statement, if any, is intended. This matters because the LDA experiment in Section 5.2 uses exactly this local-variable construction with the IMI discrepancy, so the empirical comparison between PPC and Pop-PC rests on an object whose statistical behavior is not analyzed.","section":"Section 3.4, Eq. (14); Section 5.2"}],"minor_comments":[{"comment":"The phrase 'frequenting assessment' appears to be a typo for 'frequentist assessment.'","section":"Abstract"},{"comment":"The discrepancy d(y)= (1/n) Σ log p(y_i | yobs) depends on the observed data through p(y_i|yobs), which is itself a function of the full observed set; this unusual definition should be clarified because it introduces a second, less obvious use of yobs inside the discrepancy.","section":"Section 3.3, Eq. (11)"},{"comment":"The plots show point values of p-values or discrepancies without error bars, repeated-simulation variability, or uncertainty quantification; claims such as 'the 632 Bootstrap estimator is closest to the ideal check' are not supported quantitatively.","section":"Figures 2, 4, and 5"},{"comment":"The entries Tran et al. (2016a) and Tran et al. (2016b) are identical and should be merged or distinguished.","section":"References"},{"comment":"The IMI discrepancy is attributed to Mimno and Blei (2011) but the formula in the text is incomplete (it leaves the indexing of words and documents implicit), and no derivation or computation details are given; an appendix with the exact estimator would improve reproducibility.","section":"Section 5.2"}],"recommendation":"reject","confidential_remarks":"The advertised contribution is not merely missing a proof; the central calibration claim is contradicted by a simple counterexample under the paper's own definition of interpretable p-values. This is a load-bearing error that cannot be fixed by local revision, so I recommend rejection. If the authors later reframe the contribution around the empirical behavior of the bootstrap/CV estimators or around a different optimality property, a fresh submission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: you can't trust the abstract. The paper says \"we prove that the Pop-PC is properly calibrated,\" but there is no proof in the paper, and the claim is actually wrong. I ran the simple counterexample: y|mu ~ N(mu,1), mu ~ N(0,1), population F=N(0,1), n=1, discrepancy d(y)=y. The ideal Pop-PC p-value from Eq 9 is P(Yrep > Ynew | yobs) = Phi(0.316 yobs). Since yobs ~ N(0,1), this p-value is not uniform; P(p<=0.05) ~ 1e-7. Even if you condition on a held-out ynew as in Algorithm 1, the p-value is Phi(0.913 Z), again not uniform. So the central advertised property fails. The bootstrap/CV estimators in Section 4 inherit nothing—there is no calibration to inherit.\n\nWhat is genuinely useful here: the paper gives a clean formalization of an old idea. Cross-validated predictive checks (Gelfand et al., 1992) and partial posterior checks (Bayarri and Berger, 2000) exist, but the unified \"population predictive check\" framing, with the explicit menu of sampling schemes (bootstrap, out-of-bag, .632, double bootstrap) and the extension to hierarchical models with local variables, is a nice synthesis. The empirical LDA study shows the Pop-PC behaves sensibly where PPC overfits; that is a good direction.\n\nThe soft spots beyond the calibration issue: no comparison against the existing remedies the paper cites (Bayarri-Berger, Robins et al.), no error bars or repeated-run variability in the plots, no code or data. These are minor relative to the false theorem. The paper never actually states a theorem or proof for calibration; it just asserts it in the abstract and then moves on.\n\nBottom line: this deserves a careful referee—the topic is important and the authors know the literature—but the current version cannot stand. Either the definition of \"properly calibrated\" needs to change to something defensible (e.g., the p-value is not uniform but is less overconfident than PPC), or the authors need to show a different formal property. I would recommend a major-revision decision, not a desk reject, because the empirical method may still be salvageable and valuable.","headline":"The paper's headline claim—that Pop-PCs are properly calibrated—is false even in a correct normal model; the framework is still interesting but needs major rework.","tokens_in":16602,"tokens_out":3341,"would_cite":false,"duration_ms":30506,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62G09","62G10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The population predictive check evaluates a Bayesian model on fresh draws from the population, and the paper proves this check is properly calibrated.","keywords":["Bayesian model criticism","posterior predictive checks","population predictive check","p-value calibration","bootstrap","cross-validation","hierarchical models","model overfitting"],"falsifier":"Simulate many datasets from a known Bayesian model, run Algorithm 1 on each with the bootstrap estimator, and check whether the resulting Pop-PC p-values are approximately uniform; a systematic deviation would show the practical check is not properly calibrated.","tokens_in":15576,"feed_emoji":"📊","tokens_out":10937,"duration_ms":95843,"temperature":0.7,"pith_summary":"This paper claims that posterior predictive checks are prone to overconfidence because they evaluate a model on the same data that was used to fit it. It introduces the population predictive check (Pop-PC), which instead compares the posterior predictive distribution to data drawn from the true population distribution. The paper proves that this ideal check is properly calibrated, in the sense that its p-values are uniform when the model is correct. Since the population is usually unknown, the paper develops bootstrap and cross-validation estimators of the Pop-PC and shows that they track the ideal check and detect overfitting where the posterior predictive check does not.","feed_headline":"New Bayesian check tests models on fresh data, not training data","feed_subtitle":"This check scores the model on fresh population draws, avoiding the overconfidence of posterior predictive checks.","key_machinery":"The central object is the ideal population predictive check, $\\mathrm{Pop\\text{-}PC}(y_{\\text{obs}}, F; d, g) = \\mathbb{E}[g(d(Y_{\\text{rep}}), d(Y_{\\text{new}})) \\mid y_{\\text{obs}}, F]$, where $Y_{\\text{rep}}$ is drawn from the posterior predictive and $Y_{\\text{new}}$ from the population. The machinery is the separation it enforces between the conditioning data and the evaluation data: with $g(a,b) = \\mathbb{1}[a > b]$, this becomes a tail probability whose reference distribution is $F$ rather than the observed data, which is what makes the p-value uniform under a correct model. Finite-data implementations replace $F$ with the empirical distribution and obtain $(y_{\\text{obs}}, y_{\\text{new}})$ by cross-validation, bootstrap, out-of-bag, or double-bootstrap sampling, refitting the posterior on each resampled $y_{\\text{obs}}$.","core_discovery":"The central discovery is that the reference distribution in a Bayesian model check should be the population distribution, not the observed data. The population predictive p-value, $p(d(Y_{\\text{rep}}) > d(Y_{\\text{new}}) \\mid y_{\\text{obs}}, F)$, compares a discrepancy computed on posterior-predictive replicates with the same discrepancy computed on fresh data from the population. Because $Y_{\\text{new}}$ is drawn independently of the conditioning data, this p-value has a uniform reference distribution under a correct model; the posterior predictive p-value, which compares replicates with the observed data, does not. The paper extends this idea to realized discrepancies that depend on latent variables and to per-group checks in hierarchical models.","pith_inferences":["The calibration guarantee is stated for the ideal check that samples from the true population; the empirical estimators swap in the empirical distribution, so whether each estimator inherits uniformity under the null is an open question that the paper does not resolve.","The same 'evaluate on fresh population draws' move could be applied to other diagnostics that currently use the observed data as their reference, such as surprise measures for outlier detection, giving them uniform calibration as well.","Because Algorithm 1 refits the posterior for every resampled dataset, practical scaling will require amortized inference or approximate refits; the paper mentions bag-of-little-bootstraps but does not analyze the statistical cost of reusing a single posterior fit."],"forward_implications":["If the model is correct, Pop-PC p-values are uniform, so the check will not systematically over-reject or under-reject; posterior predictive p-values cluster around one-half and lack this calibration.","Pop-PCs will not reward a model that memorizes its training data, because fresh population draws will not match the memorized distribution; the Dirichlet-process example shows a PPC preferring the empirical distribution while the Pop-PC gives extreme p-values for the same misfit.","The bootstrap and cross-validation estimators are practical with any posterior approximation and, in the regression experiments, follow the ideal Pop-PC as model flexibility grows, whereas the PPC prefers more flexible models.","In the hierarchical topic model, the Pop-PC detects the memorization pathology as the number of topics grows, with orders of magnitude more deviance than the PPC.","The construction covers realized discrepancies and hierarchical models, so checks can target individual groups or latent-variable structure rather than only whole datasets."],"supporting_citations":[{"why":"Defines the posterior predictive check that Pop-PC builds on and criticizes for using the data twice.","marker":"(Rubin, 1984)"},{"why":"Supplies the sampling-theory view of model criticism that Pop-PC revives by comparing predictions with the population.","marker":"(Box, 1980)"},{"why":"Introduces realized discrepancies, which Pop-PC extends to population comparisons.","marker":"(Gelman et al., 1996)"},{"why":"Proposes cross-validated predictive checks, which the paper identifies as a special case of the Pop-PC estimator.","marker":"(Gelfand et al., 1992)"},{"why":"Provides the bootstrap as the mechanism for obtaining finite-data draws of y_new from the empirical distribution.","marker":"(Efron, 1979)"},{"why":"Supplies the plug-in principle that justifies replacing the unknown population distribution with the empirical distribution.","marker":"(Efron and Tibshirani, 1993)"},{"why":"Sets the uniform-reference-distribution standard and calibrates posterior predictive p-values, a target the Pop-PC addresses directly.","marker":"(Robins et al., 2000)"},{"why":"Offers partial posterior and conditional predictive checks that avoid double use of data, the main alternative approach the paper positions against.","marker":"(Bayarri and Berger, 2000)"},{"why":"Supplies the instantaneous mutual information discrepancy and the per-topic checking setup used in the empirical study.","marker":"(Mimno and Blei, 2011)"}],"fun_headline_variants":["Bayesian model check uses fresh data, not training data","Population draws fix Bayesian model criticism","New p-value for model checks: compare to fresh data","Model check scores on new data, avoids overconfidence","Pop-PC: proper calibration via heldout population draws"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical routines assume that resampling the data you already have can stand in for drawing genuinely new data from the real population, and the paper does not prove that this estimated check keeps the ideal check's calibration.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian model check uses fresh data, not training data","Population draws fix Bayesian model criticism","New p-value for model checks: compare to fresh data","Model check scores on new data, avoids overconfidence","Pop-PC: proper calibration via heldout population draws"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1234,"prompt_tokens":890,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":269}},"tokens_in":506,"tokens_out":344,"duration_ms":3282,"temperature":1.0,"reasoning_tokens":269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:29:08.998994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate many datasets from a known Bayesian model, run Algorithm 1 on each with the bootstrap estimator, and check whether the resulting Pop-PC p-values are approximately uniform; a systematic deviation would show the practical check is not properly calibrated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the posterior predictive check that Pop-PC builds on and criticizes for using the data twice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sampling-theory view of model criticism that Pop-PC revives by comparing predictions with the population."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces realized discrepancies, which Pop-PC extends to population comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposes cross-validated predictive checks, which the paper identifies as a special case of the Pop-PC estimator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the bootstrap as the mechanism for obtaining finite-data draws of y_new from the empirical distribution."},{"cited_title":"and Tibshirani, R","cited_arxiv_id":null,"evidence_quote":"Supplies the plug-in principle that justifies replacing the unknown population distribution with the empirical distribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sets the uniform-reference-distribution standard and calibrates posterior predictive p-values, a target the Pop-PC addresses directly."},{"cited_title":"and Berger, J","cited_arxiv_id":null,"evidence_quote":"Offers partial posterior and conditional predictive checks that avoid double use of data, the main alternative approach the paper positions against."},{"cited_title":"and Blei, D","cited_arxiv_id":null,"evidence_quote":"Supplies the instantaneous mutual information discrepancy and the per-topic checking setup used in the empirical study."}],"review_version":1}