{"id":"eca371f6-579b-4e2c-8a54-4053198405af","arxiv_id":"2508.17077","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CP4SBI applies local conformal calibration (regression-tree and conditional-CDF variants) to credible sets from simulation-based inference, yielding finite-sample local and asymptotic conditional coverage for any posterior approximator.","lead":"This paper proposes CP4SBI, a conformal prediction wrapper that recalibrates credible sets from any simulation-based inference method so they achieve finite-sample local coverage guarantees. A generalist should read it because miscalibrated posterior regions are a known obstacle to using simulator models in experimental science, and this is a plug-in fix that works with both density- and sample-based neural estimators.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LoCart CP4SBI's Theorem 4.1 guarantee requires a fixed partition, yet Section 3 omits the train/calibration split in practice; the implemented variant's finite-sample local coverage is not covered by the proof.","rationale":"After reading the full text, I agree with the reader's CONDITIONAL verdict and identify the LoCart implementation gap as the single most load-bearing concern. The abstract advertises 'finite-sample local coverage guarantees' for both variants; for LoCart this rests entirely on Theorem 4.1, whose exchangeability requirement is violated when the partition is learned from the calibration scores. The paper is transparent about this (Section 3), which is to its credit, but transparency does not repair the logical gap: the guarantee is for a method that splits the data, while the experiments use the no-split variant. The CDF variant's Assumption B.1 (KL consistency of a neural posterior estimator) is a second, softer concern because the claim there is only asymptotic and the assumption is standard in the SBI asymptotics literature; the LoCart issue is more serious because it concerns a finite-sample guarantee. The concrete split/no-split comparison would settle whether coverage is empirically broken. The paper's other contributions - model-agnostic applicability, sample-based scores for implicit generative models, benchmark experiments - remain useful. Thus the correct outcome is to require the authors to either report split-based results or provide a new proof (or clearly downgrade the claim to empirical) before publishing the finite-sample local coverage statement. Verdict unchanged: CONDITIONAL.","tokens_in":23766,"tokens_out":5388,"duration_ms":51457,"concrete_test":"Re-run the LoCart CP4SBI experiments on two of the reported benchmarks (e.g., two_moons and gaussian_mixture) twice: once with the train/calibration split of Algorithm 1 (tree fit on the training subset only) and once with the no-split variant actually used in Section 5. For each condition, compute leaf-wise conditional coverage with Monte Carlo test pairs in each leaf and report the worst-case and average |coverage - (1-alpha)|. If the no-split variant has a leaf whose coverage is significantly below 1-alpha (e.g., lower than the split-based lower confidence bound by more than 2 percentage points), the finite-sample guarantee does not hold for the implemented method; if the two are indistinguishable, the practical impact of the omitted split is confirmed but the theoretical claim still needs rewording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LoCart CP4SBI's headline finite-sample guarantee is Theorem 4.1: conditional on the cell A_j containing X, coverage is at least 1-α. The proof (Appendix B.1) invokes standard conformal arguments 'within each region,' which require the partition to be fixed before the leaf-specific calibration quantiles are computed. The manuscript itself flags the issue in Section 3: 'While marginal coverage guarantees theoretically require splitting the data into training and calibration sets, we omit this step in practice.' If the regression tree is fit on the same scored dataset used for calibration, the leaf assignment of each calibration point is a function of the very scores used to compute the quantile; exchangeability conditional on the realized partition is lost, so the Theorem 4.1 bound does not follow for the implemented procedure. Algorithm 1 (Appendix A) does include a train/calibration split, but the experimental section does not state that this split was used; the text says it was omitted. Thus the finite-sample local coverage claim, a core advertised contribution, is not actually validated for the experiments. The asymptotic statement (Theorem 4.2) inherits the same gap because it builds on the finite-sample result. This is not a claim of dishonesty; the authors disclose the deviation, but the central guarantee as stated does not cover the evaluated method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CP4SBI, a conformal calibration framework for constructing credible regions in simulation-based inference (SBI). It proposes two variants: LoCart CP4SBI, which partitions the data space using a regression tree and applies conformal quantile calibration within leaves, and CDF CP4SBI, which transforms conformity scores by an estimated conditional CDF. The paper claims finite-sample local coverage for LoCart and asymptotic conditional coverage for CDF, and reports experiments on ten SBI benchmarks using normalizing-flow and score-diffusion posterior estimators.","tokens_in":23992,"tokens_out":5271,"duration_ms":52718,"significance":"If the theoretical guarantees were fully matched by the implementation, CP4SBI would provide a broadly useful, model-agnostic post-hoc calibration layer for SBI, applicable to both density-based and sample-based posterior estimators. The paper's strengths include the simple conformal machinery, the breadth of supported scoring functions, the extension to sample-based models via kernel density approximations, and the release of code for reproducibility. However, the central finite-sample local coverage claim for LoCart CP4SBI is not realized in the experiments because the implementation omits the train/calibration split on which the proof relies, and the CDF variant's asymptotic guarantee depends on a strong, unverified KL-consistency assumption.","major_comments":[{"comment":"The implementation of LoCart CP4SBI omits the train/calibration split that Theorem 4.1 requires. In Section 3, after describing the LoCart variant, the text states: 'While marginal coverage guarantees theoretically require splitting the data into training and calibration sets, we omit this step in practice, as it has minimal empirical impact on coverage.' In contrast, the proof of Theorem 4.1 (Appendix B.1) invokes standard conformal arguments 'within each region,' which require the partition to be fixed independently of the calibration scores used to compute leaf-specific quantiles. Algorithm 1 does include a split, but Section 5 does not state that this split was used in the experiments, and the caption of Figure 3 shows the split in the illustration only. Consequently, the finite-sample local coverage guarantee advertised in the abstract and contribution list is not supported for the evaluated implementation; the coverage reported in Figures 7 and 8 is empirical and is not backed by Theorem 4.1. The asymptotic result in Theorem 4.2 inherits this gap because it builds on the finite-sample guarantee. This is a load-bearing issue for the paper's central claim, not a presentation detail.","section":"Section 3, Section 4.1, Appendix B.1"},{"comment":"The asymptotic conditional coverage of CDF CP4SBI rests on Assumption B.1, which posits that KL(bp_K(·|x) || p(·|x)) → 0 as the training size K grows. This assumption is not verified for the neural posterior estimators used in the experiments; for normalizing-flow NPE and diffusion-based NPSE, KL consistency has not been established for general SBI problems, and under model misspecification such estimators are known to converge to the best approximation in the family, not necessarily to the true posterior. The proof of Theorem B.2 correctly identifies the coverage gap as sqrt(delta_K(x)/2), but the paper does not discuss when Assumption B.1 is plausible in practice. As a result, the practical import of the 'asymptotic conditional coverage' claim for the CDF variant is weaker than the presentation in Section 1.2 suggests. The paper should state explicitly the limitations of this assumption and ideally include an empirical diagnostic (e.g., a coverage plot against K) for the benchmarks where the posterior estimator is known to be misspecified.","section":"Theorem 4.4, Assumption B.1, Appendix B.2"}],"minor_comments":[{"comment":"There is a typo: 'refered to as marginal coverage' should be 'referred to as marginal coverage.'","section":"Section 1"},{"comment":"Theorem 4.1 defines the conformal threshold t_{1-alpha}(x) as the (1-alpha) quantile of scores, while Algorithm 1 computes the empirical (1 + 1/|I_j|)(1-alpha) quantile. The standard finite-sample corrected quantile is what actually guarantees coverage at level 1-alpha; please reconcile the notation in the theorem with the algorithm.","section":"Theorem 4.1 vs. Algorithm 1"},{"comment":"The text states that experiments run on ten SBI benchmarks, but Figures 7 and 8 list only nine benchmark labels, with 'sir slcp' appearing as a single label that seems to merge the separate SIR and SLCP tasks listed in Table 1. Please check the benchmark count and the labels for consistency.","section":"Section 5.3, Figures 7 and 8"},{"comment":"The experiments report only coverage metrics (conditional MAE and marginal coverage). Since conformal methods can trivially achieve coverage by producing very wide regions, it would be informative to also report the average size or volume of the constructed credible regions, to confirm that the improved coverage is not obtained at the cost of efficiency.","section":"Section 5.3 and Figures 7-9"},{"comment":"The contribution bullet says CDF CP4SBI 'achieves asymptotic conditional coverage as the estimate bp(theta|x) gets closer to the true posterior.' This is imprecise: the precise condition is the KL convergence of Assumption B.1, not merely a vague notion of closeness. The paper would benefit from stating the assumption already in the introduction.","section":"Section 1.2 and Theorem 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid application of conformal prediction to SBI, and the code availability is a plus. The main issue is the mismatch between the theory and the implementation for LoCart, which is fixable either by enforcing the split in the experiments or by rephrasing the claimed guarantee. The CDF variant's assumptions should be discussed more carefully. I see no reason to suspect any lack of novelty or improper citation; the reuse of the authors' own LoCart method is appropriately cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this as a solid integration job rather than a fundamentally new method. The new pieces that matter are the reuse of the fitted posterior to estimate the conditional CDF of scores (so CDF-conformal becomes cheap in SBI) and the KDE score approximation that lets HPD-style sets work with sample-based estimators like diffusion models. The conformal backbone is standard, and I trust the marginal coverage arguments without reservation.\n\nThe experiments are the strongest part. Ten sbibm benchmarks, two base estimators (NPE and NPSE), and a sensible metric set (conditional MAE plus marginal coverage). The results show real improvements over Global CP and self-calibration, with near-nominal marginal coverage. The code being public helps. This is a paper that SBI practitioners will actually use.\n\nNow the soft spots, in order of seriousness. First, the LoCart finite-sample guarantee. Theorem 4.1 requires the regression-tree partition to be fixed before leaf-specific quantiles are computed. Section 3 explicitly says the train/calibration split is omitted in practice. Algorithm 1 includes the split, but the experimental section does not state that it was used; the text says the opposite. So the advertised finite-sample local coverage claim does not, strictly speaking, cover the implemented procedure. I think this is fixable—use the split, or present the no-split version as an empirical variant with only asymptotic justification—but as written it is a real gap between proof and practice.\n\nSecond, CDF CP4SBI's asymptotic conditional coverage (Theorem 4.4) depends on Assumption B.1, KL convergence of the posterior estimator. That assumption is reasonable for well-specified, flexible estimators, but it is unproved for neural posteriors and can fail under misspecification. The authors are upfront about this, but it means the headline conditional-coverage claim for the CDF variant is conditional on something that is often not checked in real SBI workflows.\n\nThird, a minor point: the Gaussian Mixture benchmark was modified (prior limits and mixture scale). This is disclosed, but it makes that particular comparison less clean.\n\nOverall, the central argument holds. The no-split LoCart implementation probably still works in practice—the experiments support it—but the paper should not claim the finite-sample theorem for the exact procedure it runs. The authors have been honest about the deviation, which I appreciate.\n\nWho is this for? Anyone doing amortized SBI and worried about miscalibrated credible regions. It deserves a serious referee. My recommendation: engage with it, but ask the authors to close the LoCart split gap—either by running the split as in Algorithm 1 or by explicitly framing the no-split version as heuristic with empirical support—and to state the experimental protocol for LoCart unambiguously.","headline":"Useful plug-in calibration for SBI credible sets, but the LoCart variant's finite-sample local guarantee as stated does not cover the experiment as run.","tokens_in":24592,"tokens_out":1588,"would_cite":true,"duration_ms":17552,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinterpreting Bayesian scores as conformity scores, this paper constructs credible sets with finite-sample local coverage guarantees for any simulation-based posterior estimator.","keywords":["simulation-based inference","conformal prediction","credible sets","local coverage","conditional coverage","posterior calibration","normalizing flows","diffusion models"],"falsifier":"Run the implemented LoCart CP4SBI on a benchmark with a known posterior, using the paper's same-data tree fitting and per-leaf calibration, then measure empirical coverage within each leaf over many held-out test pairs; if any leaf's coverage falls below $1-\\alpha$ by more than binomial sampling error, the finite-sample local guarantee as implemented is false.","tokens_in":23546,"feed_emoji":"🎯","tokens_out":7676,"duration_ms":77987,"temperature":0.7,"pith_summary":"The paper argues that miscalibration in simulation-based inference is fixable without retraining the simulator or the posterior: any Bayesian score used to define credible sets can be treated as a conformal score. It develops two wrappers, LoCart and CDF, that turn a possibly overconfident posterior approximation into credible sets with nominal coverage. LoCart partitions the data space with a regression tree and calibrates within each leaf, giving finite-sample local coverage. CDF maps scores through an estimated conditional CDF, giving asymptotic conditional coverage as training, calibration, and Monte Carlo sizes grow. If correct, these guarantees give experimental scientists a model-agnostic way to obtain honest, observation-specific uncertainty from neural posterior estimators, diffusion models, or flow-matching posteriors.","feed_headline":"Conformal wrapper yields locally calibrated SBI credible sets","feed_subtitle":"Finite-sample local coverage with LoCart; asymptotic conditional coverage with CDF. Works with any scoring function.","key_machinery":"The load-bearing object is the Bayesian conformity score $s(\\theta; x)$, any function whose sublevel sets define a credible region and which now feeds into conformal calibration. LoCart CP4SBI uses a regression tree that predicts $s$ from $x$ to partition the data space into cells where the score distribution is approximately homogeneous; within each cell, split-conformal quantiles are computed from calibration scores in that cell, which is what yields finite-sample local coverage. CDF CP4SBI instead transforms the score through an estimate of its conditional CDF, $\\hat{F}_M(s(\\theta; x) \\mid x)$, obtained by Monte Carlo from posterior draws; the probability integral transform makes the transformed score close to uniform conditional on $x$, so a single global quantile of the transformed calibration scores suffices for asymptotic conditional coverage.","core_discovery":"The central discovery is that the scoring function defining a Bayesian credible region, such as the negative posterior density for HPD regions, is already a valid nonconformity score, so the conformal calibration toolkit transfers directly to simulation-based inference. For the LoCart variant, the paper proves that once the regression-tree partition is fixed, per-leaf conformal quantiles yield $P(\\theta \\in C_{\\mathrm{locart}}(X) \\mid X \\in A_j) \\ge 1-\\alpha$ as well as marginal coverage $P(\\theta \\in C_{\\mathrm{locart}}(X)) \\ge 1-\\alpha$. For the CDF variant, it proves that the transformed score $s'(\\theta; x) = \\hat{F}_M(s(\\theta; x) \\mid x)$ becomes approximately uniform conditional on $x$ as the posterior estimator, calibration set, and number of posterior draws grow, so $P(\\theta \\in C_{\\mathrm{cdf}}(X) \\mid X = x) \\to 1-\\alpha$. Empirically, both variants improve conditional calibration over global conformal, self-calibration, and HDR baselines while keeping marginal coverage near the nominal level, across ten standard SBI benchmarks and for both normalizing-flow and score-diffusion posterior estimators.","pith_inferences":["Because CP4SBI only recalibrates a score, it cannot sharpen a weak posterior: two estimators with the same contour ordering but different density sharpness yield the same regions, so users should read CP4SBI regions as coverage-correct but not as evidence of posterior accuracy.","A direct test of the CDF variant's assumption would be to take a deliberately misspecified posterior estimator, measure the conditional coverage of CDF CP4SBI, and check whether the coverage gap tracks the KL divergence between approximate and true posterior as the paper's Pinsker bound predicts.","The same score-CDF transform could be applied recursively, estimating the conditional CDF of the already-recalibrated scores and conformalizing again, to push conditional coverage closer to nominal with the same calibration budget.","Because the LoCart partition is itself a model of where score distributions change, the learned leaf structure could double as a diagnostic that tells scientists for which types of observations the posterior estimator is least reliable."],"forward_implications":["Any SBI method, whether density-based like NPE or sample-based like diffusion and flow matching, can be wrapped to output credible sets with nominal marginal coverage at finite sample.","LoCart CP4SBI adapts region width to inference difficulty: easy observations land in leaves with tight thresholds, difficult ones in leaves with wider thresholds, instead of one global cutoff.","CDF CP4SBI approaches observation-specific conditional coverage as the posterior approximation improves and the calibration budget grows, making amortized posteriors usable for per-observation inference.","The framework works with any scoring function, so users can choose HPD, symmetric, quantile-based, or custom credible sets without changing the calibration machinery.","Across ten standard SBI benchmarks, both variants improve conditional coverage (lower MAE) over global conformal, self-calibration, and HDR while keeping marginal coverage close to $1-\\alpha$; the CDF variant is especially strong with small calibration budgets."],"supporting_citations":[{"why":"Supplies the LoCart regression-tree partition and the regularity conditions that Theorems 4.1 and 4.2 rely on for local and asymptotic conditional coverage.","marker":"Cabezas et al. 2025a"},{"why":"Defines the CDF-conformal score and the ECDF construction that CDF CP4SBI reuses with posterior draws.","marker":"Dheur et al. 2025"},{"why":"Provides the ten benchmark tasks, their prior specifications, and oracle posterior samples used in all experiments.","marker":"Lueckmann et al. 2021"},{"why":"Provides the normalizing-flow neural posterior estimator (NPE) used as one base estimator.","marker":"Greenberg et al. 2019"},{"why":"Provides the score-diffusion posterior sampler (NPSE) used as the sample-based base estimator.","marker":"Geffner et al. 2023"},{"why":"Documents the overconfidence and miscalibration of SBI posteriors that motivates the calibration layer.","marker":"Hermans et al. 2021"},{"why":"The Global baseline: vanilla conformal credible sets with marginal coverage, and the closed-form-posterior requirement that CP4SBI removes.","marker":"Patel et al. 2023"},{"why":"The HDR recalibration baseline that CP4SBI is compared against.","marker":"Chung et al. 2024"},{"why":"Supplies the software implementations of both posterior estimators used in the experiments.","marker":"Boelts et al. 2025"}],"fun_headline_variants":["Conformal calibration yields locally valid SBI credible sets","Regression-tree conformal sets guarantee local coverage in SBI","CDF-based conformal calibration achieves conditional coverage in SBI","Local conformal calibration for simulation-based inference","Finite-sample local coverage for SBI credible sets via conformal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The finite-sample local coverage guarantee assumes the regression tree that defines the local regions is fitted on data separate from the scores used for calibration, but the paper's implementation fits the tree and calibrates on the same data, so that strict guarantee does not apply to the reported experiments; the CDF variant additionally assumes the posterior estimator converges to the true posterior in KL divergence.","fun_headline_variants_meta":{"raw":{"variants":["Conformal calibration yields locally valid SBI credible sets","Regression-tree conformal sets guarantee local coverage in SBI","CDF-based conformal calibration achieves conditional coverage in SBI","Local conformal calibration for simulation-based inference","Finite-sample local coverage for SBI credible sets via conformal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2388,"prompt_tokens":935,"completion_tokens":1453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1373}},"tokens_in":551,"tokens_out":1453,"duration_ms":12434,"temperature":1.0,"reasoning_tokens":1373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:09:09.627599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the implemented LoCart CP4SBI on a benchmark with a known posterior, using the paper's same-data tree fitting and per-leaf calibration, then measure empirical coverage within each leaf over many held-out test pairs; if any leaf's coverage falls below $1-\\alpha$ by more than binomial sampling error, the finite-sample local guarantee as implemented is false.","supporting_citations":[{"cited_title":"Benchmarking simulation-based inference","cited_arxiv_id":null,"evidence_quote":"Provides the ten benchmark tasks, their prior specifications, and oracle posterior samples used in all experiments."},{"cited_title":"Variational Inference with Coverage Guarantees in Simulation-Based Inference","cited_arxiv_id":"2305.14275","evidence_quote":"The Global baseline: vanilla conformal credible sets with marginal coverage, and the closed-form-posterior requirement that CP4SBI removes."},{"cited_title":"Sampling-based multi-dimensional recalibration","cited_arxiv_id":null,"evidence_quote":"The HDR recalibration baseline that CP4SBI is compared against."}],"review_version":2}