{"id":"701e9f89-a7b4-4282-93e9-2fa4a33fe8e6","arxiv_id":"2502.03279","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Posterior SBC validates Bayesian inference conditional on observed data by treating the posterior as the reference distribution and testing calibration of augmented posteriors.","lead":"This paper introduces posterior SBC, a way to check whether a Bayesian inference algorithm is trustworthy for a specific observed dataset, not just on average over prior-simulated data. It works by simulating new data from the already-computed posterior and checking whether the algorithm stays calibrated when that new data is folded in.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Posterior SBC only certifies self-consistency under Bayesian updating; a coherently wrong algorithm (exact Bayes from any initial posterior) passes, so Section 3's guarantee is overstrong.","rationale":"The central derivation in Equation (3) is mathematically sound, and the three case studies provide useful evidence for the method's practical value, especially in amortized inference where cheap repeated inference makes posterior SBC a natural diagnostic. The paper's own 'necessary but not sufficient' caveat in Section 5 is appropriate and partially mitigates the overstatement. However, the Section 3 heuristic that biased inference will not be consistent under further conditioning is not guaranteed, and a precise counterexample exists: any algorithm that performs exact Bayesian updating from an arbitrary initial posterior will pass posterior SBC regardless of how wrong that initial posterior is. This is not a mere theoretical edge case; it is the exact condition under which the PIT values are uniform, so the method checks coherence, not correctness. The reader's weakest assumption identifies this concern, though the example 'ignoring the likelihood' is imprecise because that particular failure is detectable via the log-likelihood test quantity. The proposed concrete test would settle the matter by demonstrating uniform PIT values for a grossly wrong but coherent posterior. Because the paper already concedes the limitation in the discussion and the practical examples remain useful, the conditional verdict stands, but the abstract and Section 3 should be reworded to state that posterior SBC validates self-consistency under Bayesian updating rather than unconditional data-conditional validity.","tokens_in":18703,"tokens_out":16508,"duration_ms":151658,"concrete_test":"Run posterior SBC on a one-dimensional normal model with specified prior theta ~ N(0,1), likelihood y ~ N(theta,1), and fixed yobs=0. Use a deliberately wrong but coherent 'inference algorithm': for any dataset, it returns the exact Bayesian posterior computed from the arbitrary initial posterior N(10,0.1) (for yobs alone) and its exact Bayes update for augmented data. Generate N=500 replicates of theta' ~ N(10,0.1), y ~ N(theta',1), and S=1000 theta'' draws from the coherent update N((100+y)/11, 1/11). Compute PIT ranks for theta and for log p(yobs,y|theta), and plot ECDF differences against 95% simultaneous bands from Säilynoja et al. (2022). If the lines stay inside the bands, posterior SBC certifies a posterior centered at about 10 with variance about 0.1 while the specified posterior is N(0,0.5), confirming the blind spot analytically and computationally.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that a biased inference algorithm cannot be self-consistent under further conditioning. This is not a theorem, and the paper's own example of 'ignoring the likelihood' is actually detectable via the log-likelihood test quantity. The exact class of failures posterior SBC cannot detect is algorithms that are coherent Bayesian updaters from an arbitrary initial posterior. Let A(theta|yobs) be the algorithm's output for the observed data. The PIT values are uniform iff A(theta|yobs,y) is proportional to A(theta|yobs) p(y|theta) for the generated y, meaning the algorithm updates exactly by Bayes' rule from its own initial posterior. Any algorithm satisfying this passes posterior SBC even if A(theta|yobs) is arbitrarily far from the specified posterior pi(theta|yobs). For example, an algorithm that outputs N(10, 0.1) as the posterior for yobs=0 in a unit-variance normal model with N(0,1) specified prior, and then updates exactly, produces uniform PIT values for both marginal parameters and log-likelihood. Thus Section 3's claim that sampling failures lead to non-uniform PIT values is false in general. The Discussion's 'necessary but not sufficient' caveat is accurate, but the central claim of validating inference conditionally on observed data overstates what is tested: only the update step, not the absolute accuracy at yobs, is checked. This matters practically for black-box amortized approximators, where a systematically wrong but coherent posterior is conceivable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces posterior SBC, a variant of simulation-based calibration checking that conditions on an observed dataset. The method draws parameter values from the posterior given the observed data, generates new data from the likelihood, draws from the posterior conditioned on both the observed and generated data, and tests uniformity of the resulting PIT values. The central derivation (Eq. 3) is a correct application of the Bayesian chain rule, and the paper demonstrates the method on three case studies: a hierarchical model, a Lotka-Volterra model, and an amortized Bayesian inference model. The authors position posterior SBC as a data-conditional diagnostic that complements prior SBC and is especially useful for amortized inference.","tokens_in":18972,"tokens_out":14443,"duration_ms":126761,"significance":"Posterior SBC is a useful and natural extension of SBC that focuses calibration checking on the region of parameter space relevant to an observed dataset. The derivation is clean, involves no free parameters, and the case studies are well executed with public code. The method is particularly valuable for amortized Bayesian inference, where standard MCMC diagnostics are unavailable. However, the paper overstates what is tested: posterior SBC verifies self-consistency of the inference algorithm under further Bayesian updating, not the absolute accuracy of the posterior at the observed data. A coherent Bayesian updater with an arbitrary initial prior will pass posterior SBC even when its posterior is far from the specified posterior. This limitation is acknowledged in the Discussion but is not reflected in the abstract or Section 3.","major_comments":[{"comment":"The claim that 'if the sampling fails for either the original posterior or for the augmented posteriors, the SBC equality in Equation 3 does not hold, leading to non-uniform PIT values' is too strong. Uniformity of the posterior SBC PIT values holds for any inference algorithm that is exactly self-consistent under Bayesian updating with the true likelihood, even if the algorithm's posterior at the observed data is arbitrarily different from the specified posterior. For example, in the normal model of Section 2, an algorithm that is exact Bayes under a different prior (say N(10,0.1) instead of N(0,1)) produces uniform PIT values for both parameter and log-likelihood test quantities. The paper should state the precise self-consistency condition that the PIT test verifies, e.g., A(theta|y_obs) = E_{theta'~A(.|y_obs), y~p(y|theta')} A(theta|y_obs,y), and qualify the abstract and this paragraph accordingly.","section":"Section 3, first full paragraph"},{"comment":"The sentence 'it is very unlikely that we would encounter such biased inference that would be consistent when conditioned on more data' is an unsupported heuristic rather than a derived guarantee. The manuscript should replace this with a precise characterization of the class of failures that posterior SBC can detect: it detects failures of self-consistency under additional conditioning, but it cannot detect a coherent Bayesian updater that starts from an arbitrary initial posterior. This point affects the interpretation of all three case studies and should be addressed in Section 3 rather than deferred to the Discussion.","section":"Section 3, paragraph on the consistency of biased inference"},{"comment":"The abstract states that posterior SBC can 'validate the inference conditionally on observed data', and Section 1 similarly claims that it can 'validate the model implementation and inference algorithm when conditioned on some fixed set of data'. Given the limitation described above, such claims overstate what is tested: the method checks the update step from y_obs to (y_obs, y), not the absolute accuracy of the posterior at y_obs. The authors should either qualify these statements or define a precise sense in which the posterior at y_obs is validated. The Discussion's 'necessary but not sufficient' caveat should be introduced in the abstract or the introduction.","section":"Abstract and Section 1"},{"comment":"The factorization in Eq. (3) treats y_obs and y as if both are random variables in a joint distribution, but y_obs is fixed. The chain rule should be written for the conditional distribution e.g., pi(y, theta', theta'' | y_obs) = pi(theta' | y_obs) pi(y | theta') pi(theta'' | y, y_obs), and the assumption that y is conditionally independent of y_obs given theta' should be stated explicitly.","section":"Section 3, Eq. (3)"}],"minor_comments":[{"comment":"The notation u_i = p(theta''_i < theta'_i | y_i) is nonstandard; it should be written as P(theta''_i < theta'_i | y_i) or as the CDF F_{theta''|y_i}(theta'_i).","section":"Section 2, Eq. (2)"},{"comment":"The caption of Figure 1 would benefit from a more explicit description: the gray area denotes the prior support and the colored regions denote posterior regimes with biased inference; in panel (b), 'the inference is calibrated outside the colored region' should be clarified as 'calibrated for parameter values outside the colored region'.","section":"Figure 1 and Section 1"},{"comment":"The description of the two observed datasets could be clearer: the sentence 'The first case has a strong prior and a weak likelihood' refers to the dataset with tau=0.06 and sigma=1.96, but the connection between the prior on tau and the likelihood's strength may confuse readers; consider adding a sentence explaining the funnel geometry in each parameterization.","section":"Section 4.1, Setup"},{"comment":"There are typographical errors in the abstract and introduction, such as 'conditionalonobserveddata' and 'the both the code for implementing the model'; these should be corrected.","section":"Abstract and Section 1"},{"comment":"The supplementary text refers to 'Section 3.2 of the article' when discussing the Lotka-Volterra multimodality outliers, but the relevant section in the main text is Section 4.2; this cross-reference should be fixed.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a solid, useful contribution to the Bayesian workflow literature, and the case studies are convincing. The main concern is the overstatement in Section 3 and the abstract about what posterior SBC tests; this is fixable by adding a precise statement of the self-consistency condition and acknowledging the coherent-wrong-prior limitation earlier in the paper. I would be willing to review a revised version. The paper's reliance on the authors' own prior work on graphical uniformity tests is appropriate and not a concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Posterior SBC is a straightforward, well-motivated extension of simulation-based calibration that conditions on the observed data instead of the prior, and it fills a real gap for amortized inference. The derivation is a clean chain-rule application, the case studies are informative, and the code is public. The main caveat, which the stress-test got right, is that the check only certifies self-consistency under further Bayesian updating, not that the posterior at y_obs is correct in an absolute sense.\n\nThe new piece is Equation (3): draw theta' from the posterior given y_obs, generate y from the likelihood, and check whether theta' and theta'' from the augmented posterior are exchangeable. That is a genuine extension of prior SBC; none of the cited earlier works condition on a fixed dataset. The three examples demonstrate where prior SBC misses localized miscalibration (the hierarchical funnel cases) and where posterior SBC is practically valuable for amortized neural posteriors that lack convergence diagnostics. The graphical PIT-ECDF checks are well executed.\n\nThe soft spot is the interpretation. Section 3 says sampling failures 'lead to non-uniform PIT values,' but that is only true for certain kinds of failures. The stress-test gives a concrete counterexample: an algorithm that starts from an arbitrary posterior and then updates exactly by Bayes' rule will pass posterior SBC for any dataset, even if its initial posterior is far from the specified prior-based posterior. The paper's own 'necessary but not sufficient' caveat in the Discussion is accurate, but the Section 3 statement is overstrong. This is not a fatal flaw: the method is still a useful diagnostic for detecting inconsistency with the data-generating process, and for amortized inference it is often the only practical check. But the claim that it 'validates inference conditionally on observed data' should be read as 'validates the update step' rather than 'validates the posterior at y_obs.'\n\nThe paper is for people doing Bayesian workflow, especially with amortized or variational inference. It deserves a serious referee; the math is correct, the experiments are reproducible (code on GitHub), and the limitation is acknowledged. I would send it to review, but I would ask the authors to soften the Section 3 wording and to state explicitly the class of failures that posterior SBC cannot detect. That can be fixed in revision.","headline":"A clean, useful extension of SBC that conditions on the observed data, but its guarantee is self-consistency under updating, not absolute posterior correctness; the paper slightly oversells it in Section 3.","tokens_in":19530,"tokens_out":2779,"would_cite":true,"duration_ms":24762,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62-08","65C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Posterior SBC re-bases simulation-based calibration at the observed-data posterior, letting modellers check whether their inference algorithm is trustworthy for the dataset they actually have.","keywords":["posterior SBC","simulation-based calibration","Bayesian workflow","data-conditional calibration","amortized Bayesian inference","PIT uniformity","hierarchical models","drift-diffusion model"],"falsifier":"Run posterior SBC with an inference algorithm that ignores the likelihood and always returns the prior as its posterior, on a dataset with fairly informative data. If the parameter PIT values come out uniform, that demonstrates the failure mode the paper's heuristic assumption dismisses; a complete version of the check should use a test quantity such as the joint log-likelihood, which should expose the bias.","tokens_in":18478,"feed_emoji":"🎯","tokens_out":6703,"duration_ms":52272,"temperature":0.7,"pith_summary":"Standard simulation-based calibration (SBC) checks an inference algorithm by drawing parameters from the prior, simulating data, and testing whether posterior ranks are uniform on average. The paper argues this answers the wrong question after data arrive: it can miss miscalibration that is confined to the region of parameter space that actually matters for the observed dataset, or flag problems that vanish once the data are conditioned on. The authors introduce posterior SBC, which re-bases the same self-consistency check at the observed-data posterior: draw a parameter from the posterior, simulate new data from it, draw again from the augmented posterior, and verify that the two draws are exchangeable. Three case studies show it detecting parameterization-dependent miscalibration in a hierarchical model, avoiding irrelevant prior-driven warnings in a Lotka-Volterra model, and exposing bad neural-network posterior approximation in amortized inference that prior SBC missed.","feed_headline":"New posterior SBC catches inference bias prior SBC misses","feed_subtitle":"The standard check averages over priors; the new one verifies calibration for your actual data, per parameter.","key_machinery":"The load-bearing object is the posterior-rebased SBC equality (Equation 3), which applies the chain rule of Bayesian updating to the joint distribution of the observed data, a posterior predictive draw, and two parameter draws. It turns data-conditional calibration into a rank-uniformity test: compare draws $\\theta'$ from the original posterior with draws $\\theta''$ from the augmented posterior that conditions on both the observed data and a posterior-predictive simulation, and check the empirical PIT values for discrete uniformity using the graphical ECDF-difference test. The same test quantities as in prior SBC, including the joint log-likelihood, carry over.","core_discovery":"The paper's central claim is that calibration of Bayesian inference should be assessed conditional on the data actually observed, and that the standard SBC self-consistency argument can be shifted to make this possible. Treating the observed-data posterior $\\pi(\\theta \\mid y_{\\mathrm{obs}})$ as the new prior, the identity in Equation (3) shows that for $\\theta' \\sim \\pi(\\theta \\mid y_{\\mathrm{obs}})$, $y \\sim \\pi(y \\mid \\theta')$, and $\\theta'' \\sim \\pi(\\theta \\mid y_{\\mathrm{obs}}, y)$, the two parameter draws $\\theta'$ and $\\theta''$ have the same conditional distribution. Hence the empirical PIT values of $\\theta'$ with respect to draws from the augmented posterior are uniform if and only if the inference algorithm is calibrated in the region of parameter space that matters for the observed data. The paper demonstrates the diagnostic in three cases where prior SBC either misses real miscalibration or reports irrelevant miscalibration.","pith_inferences":["The self-consistency assumption could be stress-tested by deliberately injecting a bias that is invariant to adding data (for instance, an algorithm that ignores the likelihood); parameter-based posterior SBC would show uniform PIT values, revealing the method's blind spot and motivating likelihood-based test quantities.","The method could be extended to a K-fold or data-splitting version, conditioning the initial posterior on only part of the data so the augmented posterior is closer in size to the original; the paper mentions this as a possible adjustment but does not develop it.","Data-conditional miscalibration detected by posterior SBC may sometimes reflect model misspecification rather than algorithm failure; as a diagnostic it could therefore also guide model refinement in the Bayesian workflow, not just algorithm choice.","For amortized inference, posterior SBC could become a standard acceptance gate before a trained neural posterior is used on a new dataset, since the per-dataset cost is seconds."],"forward_implications":["Modellers can validate the inference algorithm for their actual dataset, detecting miscalibration that prior SBC misses when problematic parameter regions are small or cancellation occurs.","Posterior SBC gives per-parameter information about the direction and magnitude of bias, allowing informed model reparameterization or algorithm changes.","In the Lotka-Volterra case it is computationally cheaper than prior SBC, since the prior's problematic regions are avoided, so more iterations can be run within the same budget.","For amortized Bayesian inference, posterior SBC provides a near-free data-conditional diagnostic where no MCMC convergence diagnostics exist.","Passing posterior SBC is a necessary but not sufficient condition for trustworthy inference, so it complements rather than replaces convergence diagnostics."],"supporting_citations":[{"why":"Supplies the original SBC self-consistency argument and the PIT-uniformity criterion that posterior SBC re-bases at the observed-data posterior.","marker":"Cook et al., 2006"},{"why":"Provides the discrete-PIT uniformity framing that posterior SBC relies on when comparing finite posterior draws.","marker":"Talts et al., 2020"},{"why":"Establishes the theory for SBC test quantities, including the joint log-likelihood, whose sensitivity posterior SBC inherits.","marker":"Modrák et al., 2023"},{"why":"Provides the graphical discrete-uniformity test used for the ECDF-difference plots in all case studies.","marker":"Säilynoja et al., 2022"},{"why":"Defines the Bayesian workflow context in which calibration checking is used for iterative model and algorithm refinement.","marker":"Gelman et al., 2020"},{"why":"Documents challenges in diagnosing variational inference reliability, motivating the need for a data-conditional check like posterior SBC.","marker":"Yao et al., 2018"},{"why":"Supplies the two integrative neuroscience models used in the amortized-inference case study.","marker":"Ghaderi-Kangavari et al., 2023"},{"why":"Provides the two real EEG/fMRI datasets on which posterior SBC is conditioned in the amortized-inference case study.","marker":"Georgie et al., 2018"}],"fun_headline_variants":["Posterior SBC checks calibration for your actual data","SBC that validates inference on observed data, not just priors","Data-conditional SBC finds biases prior SBC overlooks","Calibration check tailored to your dataset, not the prior","Posterior SBC: verify inference exactly where it matters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a biased inference algorithm will not stay biased in the same way when it is conditioned on additional data; if an algorithm's error is invariant under further conditioning, posterior SBC can show uniform PIT values despite the posterior being wrong.","fun_headline_variants_meta":{"raw":{"variants":["Posterior SBC checks calibration for your actual data","SBC that validates inference on observed data, not just priors","Data-conditional SBC finds biases prior SBC overlooks","Calibration check tailored to your dataset, not the prior","Posterior SBC: verify inference exactly where it matters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2678,"prompt_tokens":920,"completion_tokens":1758,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1685}},"tokens_in":536,"tokens_out":1758,"duration_ms":11870,"temperature":1.0,"reasoning_tokens":1685,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T05:16:41.714963+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run posterior SBC with an inference algorithm that ignores the likelihood and always returns the prior as its posterior, on a dataset with fairly informative data. If the parameter PIT values come out uniform, that demonstrates the failure mode the paper's heuristic assumption dismisses; a complete version of the check should use a test quantity such as the joint log-likelihood, which should expose the bias.","supporting_citations":[{"cited_title":"R., Gelman, A., and Rubin, D","cited_arxiv_id":null,"evidence_quote":"Supplies the original SBC self-consistency argument and the PIT-uniformity criterion that posterior SBC re-bases at the observed-data posterior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents challenges in diagnosing variational inference reliability, motivating the need for a data-conditional check like posterior SBC."},{"cited_title":"A., and Nunez, M","cited_arxiv_id":null,"evidence_quote":"Supplies the two integrative neuroscience models used in the amortized-inference case study."},{"cited_title":"K., Porcaro, C., Mayhew, S","cited_arxiv_id":null,"evidence_quote":"Provides the two real EEG/fMRI datasets on which posterior SBC is conditioned in the amortized-inference case study."}],"review_version":1}