pith. sign in

arxiv: 2604.06464 · v1 · submitted 2026-04-07 · 💻 cs.LG · physics.app-ph· stat.ML

Weighted Bayesian Conformal Prediction

Xiayin Lou , Peng Luo This is my paper

Pith reviewed 2026-05-10 19:35 UTC · model grok-4.3

classification 💻 cs.LG physics.app-phstat.ML
keywords conformal predictionbayesian quadratureimportance weightingdistribution shifteffective sample sizeconditional coveragespatial prediction
0
0 comments X

The pith

Weighted Bayesian Conformal Prediction extends BQ-CP to importance-weighted data by using a weighted Dirichlet posterior whose concentration equals effective sample size.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a method that lets conformal prediction retain its finite-sample coverage guarantees when observations carry different importance weights to correct for distribution shift. It replaces the uniform Dirichlet posterior over thresholds with a weighted Dirichlet whose concentration parameters are the effective sample size multiplied by the normalized weights. This substitution preserves the data-conditional stochastic dominance property of the unweighted Bayesian approach while making posterior uncertainty shrink as one over square root of effective sample size. Readers should care because most practical data arrive with some form of reweighting or shift, and the new construction supplies both valid intervals and explicit uncertainty measures over those intervals.

Core claim

Weighted Bayesian Conformal Prediction generalizes Bayesian quadrature conformal prediction to arbitrary importance-weighted settings by replacing the uniform Dirichlet Dir(1,…,1) with the weighted Dirichlet Dir(neff · w̃1, …, neff · w̃n), where neff is Kish's effective sample size. The authors prove that neff is the unique concentration parameter that matches frequentist and Bayesian variances, that posterior standard deviation decays as O(1/√neff), that the stochastic dominance guarantee extends to per-weight-profile data-conditional statements, and that the highest-posterior-density threshold improves conditional coverage by the same O(1/√neff) rate.

What carries the argument

The weighted Dirichlet distribution Dir(neff · w̃) with neff equal to Kish's effective sample size, acting as the posterior over nonconformity thresholds to produce data-conditional coverage guarantees under weighting.

If this is right

  • Posterior standard deviation of the threshold decays as O(1/√neff).
  • BQ-CP stochastic dominance extends to per-weight-profile conditional guarantees.
  • Highest-posterior-density thresholds improve conditional coverage by O(1/√neff).
  • Kernel spatial weights produce per-location posteriors with interpretable diagnostics for geographical tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same effective-sample-size scaling could be tried in other weighted Bayesian procedures such as weighted bootstrap or importance-sampled variational inference.
  • Because neff appears directly in the posterior variance, one could monitor neff during online weighting to decide when to stop collecting additional reweighted points.
  • In domain-adaptation pipelines the method supplies per-instance uncertainty that could be used to flag regions where the importance weights are too small to yield reliable intervals.

Load-bearing premise

That the weighted Dirichlet with concentration set by effective sample size constitutes a valid posterior over thresholds whenever the supplied importance weights correctly capture the distribution shift.

What would settle it

A controlled simulation in which known importance weights are used yet the empirical coverage of the resulting WBCP intervals falls below the nominal level on average would show the coverage claims do not hold.

Figures

Figures reproduced from arXiv: 2604.06464 by Peng Luo, Xiayin Lou.

Figure 1
Figure 1. Figure 1: GeoBCP spatial diagnostics for Seattle house prices. Middle: (A) Posterior standard [PITH_FULL_IMAGE:figures/full_fig_p024_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution comparison across all five variants (Seattle house price as an example). (a) [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Posterior distribution of prediction interval width (Seattle house price as an example). The [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Empirical coverage of all five variants across 13 geospatial datasets [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗
read the original abstract

Conformal prediction provides distribution-free prediction intervals with finite-sample coverage guarantees, and recent work by Snell \& Griffiths reframes it as Bayesian Quadrature (BQ-CP), yielding powerful data-conditional guarantees via Dirichlet posteriors over thresholds. However, BQ-CP fundamentally requires the i.i.d. assumption -- a limitation the authors themselves identify. Meanwhile, weighted conformal prediction handles distribution shift via importance weights but remains frequentist, producing only point-estimate thresholds. We propose \textbf{Weighted Bayesian Conformal Prediction (WBCP)}, which generalizes BQ-CP to arbitrary importance-weighted settings by replacing the uniform Dirichlet $\Dir(1,\ldots,1)$ with a weighted Dirichlet $\Dir(\neff \cdot \tilde{w}_1, \ldots, \neff \cdot \tilde{w}_n)$, where $\neff$ is Kish's effective sample size. We prove four theoretical results: (1)~$\neff$ is the unique concentration parameter matching frequentist and Bayesian variances; (2)~posterior standard deviation decays as $O(1/\sqrt{\neff})$; (3)~BQ-CP's stochastic dominance guarantee extends to per-weight-profile data-conditional guarantees; (4)~the HPD threshold provides $O(1/\sqrt{\neff})$ improvement in conditional coverage. We instantiate WBCP for spatial prediction as \emph{Geographical BQ-CP}, where kernel-based spatial weights yield per-location posteriors with interpretable diagnostics. Experiments on synthetic and real-world spatial datasets demonstrate that WBCP maintains coverage guarantees while providing substantially richer uncertainty information.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes Weighted Bayesian Conformal Prediction (WBCP) as a generalization of BQ-CP to importance-weighted settings. It replaces the uniform Dirichlet Dir(1,…,1) with the weighted Dirichlet Dir(neff · w̃1, …, neff · w̃n) where neff is Kish's effective sample size, proves four theoretical results on variance matching, O(1/√neff) posterior decay, extension of stochastic dominance to per-weight-profile conditional guarantees, and O(1/√neff) HPD improvement in conditional coverage, and instantiates the method as Geographical BQ-CP for spatial prediction using kernel weights.

Significance. If the central claims hold, the work would meaningfully extend data-conditional guarantees from BQ-CP to weighted conformal prediction under distribution shift, supplying richer posterior uncertainty information than point-estimate thresholds. The spatial application with interpretable per-location diagnostics is a concrete strength, and the four stated theoretical results plus reproducible experiments on synthetic and real spatial data constitute a solid empirical component.

major comments (3)
  1. [Abstract] Abstract, theoretical results (1)–(4): the weighted Dirichlet Dir(neff · w̃) is introduced by direct substitution and justified solely by variance matching with the frequentist coverage estimator; no derivation is supplied showing that this distribution is the posterior induced by the importance-weighted likelihood or prior. Because results (3) and (4) on stochastic dominance and conditional coverage explicitly rely on the weighted Dirichlet being the correct posterior, the absence of this derivation is load-bearing.
  2. [Abstract] Abstract, result (1): the uniqueness claim for neff as the concentration parameter that matches frequentist and Bayesian variances is asserted without the explicit variance expressions (or the proof that no other concentration satisfies the equality while preserving higher moments). This directly affects whether the O(1/√neff) decay in result (2) and the conditional guarantees in (3)–(4) follow.
  3. [Abstract] Abstract, result (3): the claimed extension of BQ-CP stochastic dominance to per-weight-profile data-conditional guarantees presupposes that the weighted Dirichlet correctly represents posterior uncertainty under arbitrary importance weights; the manuscript provides no supporting derivation or counter-example check, leaving the data-conditional claim unsupported.
minor comments (2)
  1. [Notation] The notation for the normalized weights w̃ and the definition of Kish's neff should be restated explicitly in the main text rather than assumed from the abstract.
  2. [Abstract] The abstract refers to 'spatial experiments' and 'real-world spatial datasets' without naming the datasets or reporting the coverage metrics and neff values obtained; these details belong in the abstract or a dedicated table.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their insightful comments on the theoretical foundations of Weighted Bayesian Conformal Prediction. We address each of the major comments below and will revise the manuscript to incorporate additional derivations and proofs.

read point-by-point responses

  1. Referee: [Abstract] Abstract, theoretical results (1)–(4): the weighted Dirichlet Dir(neff · w̃) is introduced by direct substitution and justified solely by variance matching with the frequentist coverage estimator; no derivation is supplied showing that this distribution is the posterior induced by the importance-weighted likelihood or prior. Because results (3) and (4) on stochastic dominance and conditional coverage explicitly rely on the weighted Dirichlet being the correct posterior, the absence of this derivation is load-bearing.

    Authors: The weighted Dirichlet is chosen to match the variance of the frequentist weighted coverage estimator, ensuring consistency with frequentist properties. We acknowledge the need for a full derivation from the importance-weighted likelihood. In the revised manuscript, we will add a derivation of the weighted Dirichlet posterior under a suitably scaled Dirichlet prior and weighted likelihood, thereby supporting the reliance of results (3) and (4) on this choice. revision: yes

  2. Referee: [Abstract] Abstract, result (1): the uniqueness claim for neff as the concentration parameter that matches frequentist and Bayesian variances is asserted without the explicit variance expressions (or the proof that no other concentration satisfies the equality while preserving higher moments). This directly affects whether the O(1/√neff) decay in result (2) and the conditional guarantees in (3)–(4) follow.

    Authors: We will include the explicit variance expressions for the frequentist estimator (based on weighted sum of indicators) and the Bayesian posterior variance from the Dirichlet in the revision. The proof of uniqueness follows from setting the two variance expressions equal and solving for the concentration parameter, yielding neff uniquely. We will also clarify that while higher moments may not match, the second-moment match suffices for the asymptotic O(1/√neff) decay in result (2) and the conditional coverage improvements. revision: yes

  3. Referee: [Abstract] Abstract, result (3): the claimed extension of BQ-CP stochastic dominance to per-weight-profile data-conditional guarantees presupposes that the weighted Dirichlet correctly represents posterior uncertainty under arbitrary importance weights; the manuscript provides no supporting derivation or counter-example check, leaving the data-conditional claim unsupported.

    Authors: The extension in result (3) applies the stochastic dominance argument conditionally on the observed weight profile, using the properties of the weighted Dirichlet. We will expand the proof in the appendix to explicitly show the extension under arbitrary weights, and add a numerical verification with synthetic data to check the conditional guarantees, including cases with varying weight profiles. revision: yes

Circularity Check

0 steps flagged

No significant circularity: WBCP derivation is self-contained via standard variance matching and Dirichlet properties

full rationale

The paper introduces the weighted Dirichlet Dir(neff · w̃) by direct substitution from BQ-CP, with neff taken from the standard Kish formula and justified by equating frequentist and Bayesian variances of the coverage estimator. Result (1) verifies this matching (a derivation, not a tautology), while (2)-(4) follow from known Dirichlet concentration properties and the extension of BQ-CP stochastic dominance. No step reduces a claimed guarantee to a quantity defined by the result itself, no self-citation is load-bearing, and the construction does not fit parameters to the target coverage or conditional guarantees. The derivation remains independent of the final claims and relies on external benchmarks (Kish neff, Dirichlet moments).

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central construction rests on treating the weighted Dirichlet as a legitimate posterior once neff is inserted; this is an ad-hoc modeling choice justified by variance matching rather than derived from first principles.

axioms (1)
  • domain assumption The weighted Dirichlet Dir(neff · w̃) with neff = Kish's effective sample size yields a valid Bayesian posterior over conformal thresholds under importance weighting.
    Invoked to generalize the uniform Dirichlet of BQ-CP while preserving coverage properties.
invented entities (1)
  • Weighted Dirichlet posterior for conformal thresholds no independent evidence
    purpose: To encode uncertainty over the conformity threshold when observations have unequal importance weights.
    New modeling choice introduced in the paper; no independent evidence supplied beyond the variance-matching argument.

pith-pipeline@v0.9.0 · 5578 in / 1638 out tokens · 59046 ms · 2026-05-10T19:35:12.150621+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

  1. [1]

    CP addresses this by constructing a threshold ˆqsuch that Pr(Y∈ [ ˆf(X)±ˆq])≥1−α

    Predictive uncertainty(Level 1): the risk that the true outcomeY falls outside the prediction interval [ ˆf(X)±λ] . CP addresses this by constructing a threshold ˆqsuch that Pr(Y∈ [ ˆf(X)±ˆq])≥1−α

    work page

  2. [2]

    Is it well- determined by the calibration data, or could it be substantially different? Standard CP produces a single deterministic threshold with no indication of its reliability

    Meta-uncertainty(Level 2): the uncertainty about the threshold ˆqitself. Is it well- determined by the calibration data, or could it be substantially different? Standard CP produces a single deterministic threshold with no indication of its reliability. BQ-CP(Snell & Griffiths, 2025) addressed Level 2 by reframing CP as Bayesian Quadrature, modeling the u...

    work page 2025

  3. [3]

    V ariance calibration (requiresc=n eff):The posterior variance under the Dirichlet model is: Var(Sj) = pj(1−p j) c+ 1 .(24) Setting c=n eff is the unique choice such that this posterior variance matches the frequentist variance of the weighted empirical CDF: Varfreq ˆFw(t) ≈ F(t)(1−F(t)) neff .(25)

    work page

  4. [4]

    equivalent uniform samples

    Bernstein–von Mises alignment (requires c=n eff):When c=n eff, the Dirichlet posterior for the quantile has the same asymptotic variance as the frequentist sampling distribution of the weighted sample quantile. For any other c, the posterior is either too concentrated (c > n eff: overconfident) or too diffuse (c < neff: underconfident). Proof.Part 1: Mean...

    work page 1998

  5. [5]

    Involves the expected loss under theweightedtest distribution (not the global risk)

    work page

  6. [6]

    Has a profile-specific posteriorL + w whose width depends onn eff

    work page

  7. [7]

    tightness

    Provides different “tightness” for different weight profiles: when neff is large, λHPD is close to λWCP (tight posterior); when neff is small, λHPD ≫λ WCP (wide posterior, more conservative). This per-profile adaptation is impossible under BQ-CP’s i.i.d. assumption. H Theorem 4: Conditional Coverage Bound We now connect the Dirichlet posterior to the actu...

    work page 1998

  8. [8]

    When neff is small (few effective calibration points), the improvement is larger (the HPD provides meaningful extra conservatism where it is most needed)

    Adaptive conservatism:When neff is large (many effective calibration points), the improve- ment from β >0.5 is small (the interval is already well-calibrated). When neff is small (few effective calibration points), the improvement is larger (the HPD provides meaningful extra conservatism where it is most needed)

    work page

  9. [9]

    In practice,β∈[0.9,0.95]provides a good balance

    Approach to conditional coverage:As β→1 , the conservatism increases without bound, reflecting that β= 1 corresponds to using the maximum score (infinite conservatism). In practice,β∈[0.9,0.95]provides a good balance. 20 I Limit Consistency Properties A critical test of the W-BQ-CP construction is whether it recovers known frameworks in limiting cases. We...

    work page

  10. [10]

    8): c=n eff is the unique variance-matching choice, bridging frequentist and Bayesian perspectives

    Calibration Consistency(Thm. 8): c=n eff is the unique variance-matching choice, bridging frequentist and Bayesian perspectives

    work page

  11. [11]

    10): σpost =O(1/ √neff) quantifieswherethe posteriors are tight vs

    Posterior Concentration(Thm. 10): σpost =O(1/ √neff) quantifieswherethe posteriors are tight vs. diffuse

    work page

  12. [12]

    12): BQ-CP’s data-conditional guarantee extends toper- weight-profileposteriors

    Stochastic Dominance(Thm. 12): BQ-CP’s data-conditional guarantee extends toper- weight-profileposteriors

    work page

  13. [13]

    14): The HPD threshold provides O(1/√neff) improvement in conditional coverage over the marginal rate

    Conditional Coverage(Thm. 14): The HPD threshold provides O(1/√neff) improvement in conditional coverage over the marginal rate

    work page

  14. [14]

    15, 16): W-BQ-CP smoothly recovers BQ-CP ( h→ ∞ ) and nearest-neighbor CP (h→0)

    Limit Consistency(Props. 15, 16): W-BQ-CP smoothly recovers BQ-CP ( h→ ∞ ) and nearest-neighbor CP (h→0)

    work page

  15. [15]

    17, 18): Adaptive bandwidth prevents degenerate posteriors and ensures smooth variation

    Regularization(Props. 17, 18): Adaptive bandwidth prevents degenerate posteriors and ensures smooth variation. 22 L Discussion: Limitations and Open Problems L.1 Spatial Autocorrelation of Residuals A notable theoretical assumption underlying the W-BQ-CP framework is that, after importance reweighting, the nonconformity scores are approximately conditiona...

    work page

  16. [16]

    A fully non-asymptotic bound (e.g., using Berry– Esseen type results for Dirichlet) would strengthen the result

    Finite-sample coverage bound without asymptotic approximation:Theorem 14 uses the Bernstein–von Mises approximation. A fully non-asymptotic bound (e.g., using Berry– Esseen type results for Dirichlet) would strengthen the result

    work page

  17. [17]

    Analyzing robustness to weight misspecification would be valuable

    Relaxing Assumption 4:The correct weight specification assumption requires known likelihood ratios. Analyzing robustness to weight misspecification would be valuable

    work page

  18. [18]

    Proving that this is minimax optimal (e.g., minimizing worst-case posterior calibration error) would be a stronger statement

    Minimax optimality of neff scaling:Theorem 8 shows c=n eff matches the frequentist variance. Proving that this is minimax optimal (e.g., minimizing worst-case posterior calibration error) would be a stronger statement

    work page

  19. [19]

    borrowing strength

    Correlated posteriors across weight profiles:The current framework treats each weight profile independently. Introducing a Gaussian process or Dirichlet process prior across profiles to share information could enable “borrowing strength” in data-sparse regions

    work page

  20. [20]

    M Real-world Study Results 23 b a c d A

    Structured posteriors with GP priors:For the spatial instantiation, replacing the indepen- dent per-location Dirichlet with a spatially-coupled model (e.g., logistic-GP Dirichlet) could provide smoother posterior surfaces and enable joint inference across locations. M Real-world Study Results 23 b a c d A. Spatial Confidence 1.6 1.8 2.0 2.2 2.4 Posterior ...

    work page