pith. sign in

arxiv: 2606.11865 · v2 · pith:ZVKXL2BYnew · submitted 2026-06-10 · 📊 stat.ML · cs.LG

Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation

Seungjin Choi This is my paper

Pith reviewed 2026-06-29 02:12 UTC · model grok-4.3

classification 📊 stat.ML cs.LG
keywords conformal predictionlabel shiftBayesian prediction setsimportance weightingpost-hoc calibrationin-training adaptationhigh-dimensional regime
0
0 comments X

The pith

Under label shift, adapting the Bayesian posterior during training yields narrower valid prediction sets than post-hoc calibration in high-dimensional regimes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines how to restore target-domain coverage for conformal Bayes predictors when labels shift between source and target data. It isolates two mechanisms: post-hoc calibration that tilts the predictive distribution and adjusts the conformal threshold with importance weights, versus in-training adaptation that shifts the parameter posterior itself to the target domain. Controlled experiments reveal clear regime dependence, with post-hoc calibration producing narrower valid sets in low-dimensional well-estimated settings and in-training adaptation delivering up to 43 percent width reduction at fixed coverage when the problem is high-dimensional and underdetermined.

Core claim

Conformal Bayes under label shift can be handled by post-hoc calibration, which leaves the parameter posterior unchanged while tilting the posterior predictive toward the target and correcting the conformal threshold via an importance-weighted quantile, or by in-training adaptation, which tilts the parameter posterior to the target domain so that the highest predictive density region under the fitted target predictive serves as the prediction set. In the low-dimensional well-estimated regime the first strategy produces the narrowest valid intervals; in the high-dimensional underdetermined regime the second strategy achieves up to 43 percent width reduction at unchanged coverage, under the st

What carries the argument

Importance-weighted conformal calibration that either adjusts the predictive and quantile after training or adjusts the parameter posterior during training under an exact label-shift model.

If this is right

  • In low-dimensional well-estimated regimes post-hoc calibration yields narrower valid prediction intervals than in-training adaptation.
  • In high-dimensional underdetermined regimes in-training adaptation reduces prediction-set width by up to 43 percent while preserving nominal coverage.
  • Efficiency gains from in-training adaptation remain model-dependent and carry no guarantee of finite-sample conditional optimality.
  • Both strategies assume the label-shift model holds exactly and that importance weights can be estimated accurately from the given samples.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • For complex models trained on limited target-domain data the in-training route may be the practical default when the high-dimensional advantage appears.
  • The observed regime split suggests running both strategies on any new problem and selecting the narrower valid set.
  • Similar comparisons could be run for other shift types provided importance weights remain estimable.

Load-bearing premise

The importance weights that correct for label shift can be accurately estimated from the available source and target samples, and the label-shift model itself holds exactly.

What would settle it

An experiment in the high-dimensional regime that uses deliberately inaccurate importance weights or violates the exact label-shift assumption and then checks whether nominal coverage fails or the reported width reduction vanishes.

Figures

Figures reproduced from arXiv: 2606.11865 by Seungjin Choi.

Figure 2
Figure 2. Figure 2: shows that CB (IW qb), Strategy A, and Strategy B all maintain valid 90% coverage throughout. CB (un￾weighted qb) is invalid via over-coverage (91–95%): the bi￾ased model’s inflated calibration scores push the unweighted threshold too high, producing over-wide intervals that over￾cover. This illustrates that ignoring the shift can lead to either under- or over-coverage, whereas IW calibration cor￾rects the… view at source ↗
Figure 3
Figure 3. Figure 3: Strategy A’s θbs carries a fixed upward bias ≈ 0.43 inde￾pendent of β. Strategy B’s MAP correction reduces this monotoni￾cally to ≈ 0.25 at β = 0.6 (−40%) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 1
Figure 1. Figure 1: confirms that empirical target coverage is governed primarily by the IW calibration rule. CB (unweighted qb) loses coverage monotonically, reaching 83.9% at β = 0.6. All three IW methods maintain valid 90% coverage through￾out, regardless of which score is used. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Label shift intensity 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Empirical coverage (a) Coverage vs. label shift CB (unwe… view at source ↗
Figure 4
Figure 4. Figure 4: shows the efficiency consequence of the score choice among the three valid methods. Strategy B (posterior tilting) is strictly narrower than Strategy A (predictive tilting) for all β > 0, reaching 16% width reduction at β = 0.6 at the same 90% coverage level. CB (IW qb) sits between Strategy B and Strategy A at moderate β, and becomes narrower than Strategy A at higher β because the tilted score’s growing … view at source ↗
Figure 5
Figure 5. Figure 5: Strategy A is narrowest; CB (IW qb) is widest and grows with β because the unmodified source NLPD score requires a larger IW quantile adjustment. Strategy B is wider than Strategy A because the MAP correction moves θbt away from the truth when training is unbiased. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Label shift intensity 0.50 0.55 0.60 0.65 0.70 0.75 0.80 Mean absolute predictive bias (b) Debiasing: prediction sp… view at source ↗
Figure 6
Figure 6. Figure 6: Strategy B reduces the mean absolute predictive bias Ex|µb − µ true t | by up to 34% at β = 0.5, reflecting the improved geometry of the corrected predictive p B. B. Derivation of the Tilted Predictive (Strategy A) We work entirely at the predictive level conditioned on Dtr, for any fixed β. Applying Bayes’ theorem to the target distribution gives pt(y|x, Dtr) = pt(x|y, Dtr) pt(y|Dtr) pt(x|Dtr) . (11) 8 [… view at source ↗
read the original abstract

Conformal Bayes combines Bayesian posterior predictives with conformal calibration to produce prediction sets that are both statistically valid and geometrically efficient. We study conformal Bayes under label shift from a unified perspective, identifying two complementary approaches that restore nominal target-domain coverage through importance-weighted conformal calibration but operate through independent mechanisms. \emph{Post-hoc calibration} tilts the posterior predictive toward the target domain and corrects the conformal threshold via an importance-weighted quantile, leaving the parameter posterior unchanged. \emph{In-training adaptation} tilts the parameter posterior itself to the target domain, producing a corrected predictive whose highest predictive density region serves as the highest predictive density (HPD)-based prediction set under the fitted target predictive; efficiency is model-dependent and does not imply finite-sample conditional optimality. Two controlled experiments isolate the regime-dependence of each strategy: in the low-dimensional, well-estimated regime Strategy~A produces the narrowest valid intervals, while in the high-dimensional, underdetermined regime Strategy~B achieves up to $43\%$ width reduction at unchanged coverage, under the stated source-sampling and label-shift assumptions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript studies conformal Bayes under label shift and identifies two strategies that restore target-domain coverage via importance weighting: post-hoc calibration (Strategy A), which tilts the posterior predictive and corrects the conformal threshold with an importance-weighted quantile while leaving the parameter posterior unchanged, and in-training adaptation (Strategy B), which tilts the parameter posterior itself to produce a target-domain predictive whose HPD region forms the prediction set. Controlled experiments show regime dependence: Strategy A yields the narrowest valid intervals in the low-dimensional well-estimated regime, while Strategy B achieves up to 43% width reduction at unchanged coverage in the high-dimensional underdetermined regime, under the stated source-sampling and label-shift assumptions.

Significance. If the empirical claims hold under accurate weight estimation, the work supplies a unified perspective on post-hoc versus in-training mechanisms for conformal Bayes and concrete regime-dependent guidance on which yields narrower valid sets, which is useful for practitioners facing label shift in Bayesian predictive modeling.

major comments (2)
  1. [Experiments (high-dimensional regime)] The headline claim that Strategy B achieves up to 43% width reduction at nominal coverage in the high-dimensional regime is load-bearing for the central conclusion, yet the manuscript treats the importance weights w(y) = p_T(y)/p_S(y) as known or perfectly recoverable from finite source and target samples; any estimation error (variance, support mismatch, or misspecification) perturbs both the tilted posterior and the conformal threshold, so the reported coverage and width reduction become conditional on an unverified estimation step that is not shown to be accurate in the underdetermined regime studied.
  2. [Post-hoc calibration section] The finite-sample coverage guarantee for the importance-weighted quantile in Strategy A is stated to hold exactly under the label-shift model, but the paper does not provide a derivation or bound showing that the guarantee survives when the weights themselves must be estimated from the same finite target sample used for calibration.
minor comments (2)
  1. [Abstract] The abstract refers to 'the stated source-sampling and label-shift assumptions' without enumerating them; a brief explicit list in the introduction would improve readability.
  2. [Methods] Notation for the tilted posterior predictive and the HPD-based set in Strategy B should be introduced with an equation early in the methods section to avoid ambiguity when comparing the two strategies.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their constructive comments, which highlight important practical considerations around importance weight estimation. We address each major point below and indicate planned revisions to clarify assumptions and limitations.

read point-by-point responses

  1. Referee: [Experiments (high-dimensional regime)] The headline claim that Strategy B achieves up to 43% width reduction at nominal coverage in the high-dimensional regime is load-bearing for the central conclusion, yet the manuscript treats the importance weights w(y) = p_T(y)/p_S(y) as known or perfectly recoverable from finite source and target samples; any estimation error (variance, support mismatch, or misspecification) perturbs both the tilted posterior and the conformal threshold, so the reported coverage and width reduction become conditional on an unverified estimation step that is not shown to be accurate in the underdetermined regime studied.

    Authors: We agree that the reported efficiency gains, including the 43% width reduction, are shown under the assumption of known or perfectly recoverable importance weights, as stated in the source-sampling and label-shift assumptions. The controlled experiments isolate the mechanistic difference between the two strategies by using simulated data with exact knowledge of the label shift. In the high-dimensional underdetermined regime, practical estimation of weights from finite samples is indeed challenging and can affect both coverage and set sizes. We will revise the manuscript to explicitly qualify the headline claim as conditional on accurate weight estimation and add a new discussion subsection on practical weight estimation (e.g., via separate unlabeled target samples or density-ratio methods) together with a brief sensitivity simulation under moderate estimation error. revision: partial

  2. Referee: [Post-hoc calibration section] The finite-sample coverage guarantee for the importance-weighted quantile in Strategy A is stated to hold exactly under the label-shift model, but the paper does not provide a derivation or bound showing that the guarantee survives when the weights themselves must be estimated from the same finite target sample used for calibration.

    Authors: The exact finite-sample coverage guarantee for Strategy A is derived under the assumption that the importance weights are known exactly, consistent with the label-shift model as formulated. When weights must be estimated from the calibration sample, the guarantee becomes approximate rather than exact. We will revise the post-hoc calibration section to state this assumption clearly and add a remark explaining that estimation error can be reduced by using a hold-out set for weight estimation. A rigorous concentration bound quantifying the coverage deviation induced by weight estimation error would require additional technical work (e.g., empirical-process arguments), which we can sketch at a high level but are not in a position to derive fully in the current revision. revision: partial

standing simulated objections not resolved
  • Full derivation of a finite-sample coverage bound for Strategy A that accounts for weights estimated from the same finite target sample used for calibration.

Circularity Check

0 steps flagged

No significant circularity; standard importance weighting with no self-referential reduction

full rationale

The manuscript identifies two strategies (post-hoc calibration and in-training adaptation) that restore coverage via importance-weighted conformal calibration under label shift. No equations, derivations, or fitted parameters are exhibited that reduce by construction to their own inputs. The text invokes standard importance weighting w(y) = p_T(y)/p_S(y) without self-definition, without renaming a fitted quantity as a prediction, and without load-bearing self-citations or uniqueness theorems. The reported 43% width reduction is presented as an empirical outcome under explicit assumptions rather than a tautological consequence of the method itself. This is a self-contained comparison of complementary mechanisms.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no identifiable free parameters, axioms, or invented entities; all technical content remains at summary level.

pith-pipeline@v0.9.1-grok · 5716 in / 1047 out tokens · 25280 ms · 2026-06-29T02:12:20.166887+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Conformal Bayes for Two-Sided Censored Gaussian Regression under Label Shift

    stat.ME 2026-07 unverdicted novelty 5.0

    Weighted tilted conformal Bayes restores marginal coverage with smaller sets than weighted source-score calibration in synthetic experiments for two-sided censored Gaussian models under label shift.

Reference graph

Works this paper leans on

19 extracted references · cited by 1 Pith paper

  1. [1]

    M., Kundaje, A., and Shrikumar, A

    Alexandari, A. M., Kundaje, A., and Shrikumar, A. Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In Proceedings of the International Conference on Machine Learning (ICML), 2020

    2020

  2. [2]

    Angelopoulos, A. N. and Bates, S. Conformal prediction: A gentle introduction. Foundations and Trends ^ in Machine Learning, 16 0 (4): 0 494--591, 2023

    2023

  3. [3]

    Regularized learning for domain adaptation under label shifts

    Azizzadenesheli, K., Yang, F., Liu, A., and Anandkumar, A. Regularized learning for domain adaptation under label shifts. In Proceedings of the International Conference on Learning Representations (ICLR), 2019

    2019

  4. [4]

    F., Cand \`e s, E

    Barber, R. F., Cand \`e s, E. J., Ramdas, A., and Tibshirani, R. J. Conformal prediction beyond exchangeability. The Annals of Statistics, 51 0 (2): 0 816--845, 2023

    2023

  5. [5]

    Anytime-valid confirmation of label-shift corrections

    Choi, S. Anytime-valid confirmation of label-shift corrections. In ICML 2026 Workshop on Hypothesis Testing, 2026

    2026

  6. [6]

    and Holmes, C

    Fong, E. and Holmes, C. Conformal Bayesian computation. In Advances in Neural Information Processing Systems (NeurIPS), 2021

    2021

  7. [7]

    Conformal prediction for molecular properties under label shift

    Lee, H., Kim, J., Jadamba, E., Choi, S., and Shin, H. Conformal prediction for molecular properties under label shift. In NeurIPS 2025 Workshop on Reliable ML from Unreliable Data, 2025

    2025

  8. [8]

    Split conformal prediction with label-shift-adjusted Bayesian scores

    Lee, H., Kim, J., Jadamba, E., Choi, S., and Shin, H. Split conformal prediction with label-shift-adjusted Bayesian scores. In The 2nd Workshop on Epistemic Intelligence in Machine Learning, 2026

    2026

  9. [9]

    J., and Wasserman, L

    Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113 0 (523): 0 1094--1111, 2018

    2018

  10. [10]

    C., Wang, Y.-X., and Smola, A

    Lipton, Z. C., Wang, Y.-X., and Smola, A. J. Detecting and correcting for label shift with black box predictors. In Proceedings of the International Conference on Machine Learning (ICML), 2018

    2018

  11. [11]

    Comparing the Bayes and typicalness frameworks

    Melluish, T., Saunders, C., Nouretdinov, I., and Vovk, V. Comparing the Bayes and typicalness frameworks. In Proceedings of the European Conference on Machine Learning (ECML), 2001

    2001

  12. [12]

    Inductive confidence machines for regression

    Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, A. Inductive confidence machines for regression. In Proceedings of the European Conference on Machine Learning (ECML), 2002

    2002

  13. [13]

    and Ramdas, A

    Podkopaev, A. and Ramdas, A. Distribution-free uncertainty quantification for classification under label shift. In Proceedings of the Annual Conference on Uncertainty in Artificial Intelligence (UAI), 2021

    2021

  14. [14]

    Romano, Y., Patterson, E., and Cand \`e s, E. J. Conformalized quantile regression. In Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  15. [15]

    Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure

    Saerens, M., Latinne, P., and Decaestecker, C. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation, 2002

    2002

  16. [16]

    and Vovk, V

    Shafer, G. and Vovk, V. A tutorial on conformal prediction. Journal of Machine Learning Research, 9: 0 371--421, 2008

    2008

  17. [17]

    J., Barber, R

    Tibshirani, R. J., Barber, R. F., Cand \`e s, E. J., and Ramdas, A. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  18. [18]

    Algorithmic Learning in a Random World

    Vovk, V., Gammerman, A., and Shafer, G. Algorithmic Learning in a Random World. Springer, 2005

    2005

  19. [19]

    Frasian inference

    Wasserman, L. Frasian inference. Statistical Science, 26 0 (3): 0 322--325, 2011

    2011