REVIEW 2 major objections 6 minor 62 references
Bayes-assisted conformal scores can stay efficient under prior conflict by reverting to a robust residual baseline when the mean drifts.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 20:44 UTC pith:GEL3NB6A
load-bearing objection Clean residual-level fix for prior–data conflict in Bayes-assisted CP: scores shrink to DTO when residuals are near zero and revert to DTA under mean shift, with proofs and usable code. the 2 major comments →
Robust Bayes-Assisted Conformal Prediction
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
RoBAS constructs residual-based nonconformity scores from a heavy-tailed hierarchical Bayesian working model, or from a closed-form empirical-Bayes shrinkage rule, so that the scores behave like an efficient zero-centred score when residuals sit near zero and become asymptotically equivalent to the Distance-to-Average score when the residual mean diverges, protecting set size under prior-data conflict without losing conformal validity.
What carries the argument
The RoBAS residual Bayesian working model with a regularly varying heavy-tailed prior on residual-mean variance (horseshoe instantiation), which induces a nonconformity score asymptotically equivalent to Distance-to-Average under large residual mean (Theorem 3.1), together with the closed-form empirical-Bayes score |r_{n+1} − â(r) r̄| that recovers the same qualitative shrinkage.
Load-bearing premise
The robustness is designed and proved only for mean misalignment of residuals; if the predictor mainly inflates residual variance or tails while the mean stays near zero, the score does not automatically protect against wider sets.
What would settle it
Train a predictor that remains mean-unbiased on a shifted calibration set but with markedly larger residual variance, then compare RoBAS interval widths to pure Distance-to-Average and to a variance-adaptive baseline; if RoBAS is materially wider while coverage is the same, the claimed adaptivity fails outside mean shift.
If this is right
- Under training-to-calibration mean shift, residual RoBAS scores produce substantially narrower intervals than Distance-to-Origin or conjugate Normal-Normal-Gamma scores while matching Distance-to-Average.
- Any fixed predictor can be used; prior information is injected only at the low-dimensional residual level rather than over high-dimensional model parameters.
- The empirical-Bayes form yields provably interval-valued conformal sets and supports grid-free endpoint search, cutting discretisation error and evaluation cost.
- Gains are largest in small-calibration or high-noise regimes where zero-centred shrinkage helps when the residual mean is near zero and reversion protects when it is not.
Where Pith is reading between the lines
- Analogous hierarchical or empirical-Bayes adaptivity on residual scale could protect efficiency when degradation appears as variance inflation rather than mean bias.
- Residual-level heavy-tailed working models may transfer to other conformal settings where only low-dimensional working information is trustworthy, such as hierarchical or multi-task prediction.
- Order-equivalent surrogates of the hypergeometric score might eventually admit fully closed-form endpoints, matching the tractability of conjugate Normal-Normal-Gamma intervals.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RoBAS, a residual-level Bayes-assisted conformal framework whose nonconformity scores shrink toward a zero-centred (DTO-like) regime when residuals are near zero and asymptotically revert to the Distance-to-Average (DTA) score under large residual-mean conflict. Two instantiations are developed: RoBAS-Full from a heavy-tailed hierarchical residual BWM (horseshoe-type), with an explicit hypergeometric score, and RoBAS-EB, a closed-form empirical-Bayes absolute-deviation score with data-driven shrinkage. Finite-sample marginal coverage is retained under exchangeability of calibration and test pairs with a fixed base predictor f. Theory includes asymptotic robustness (Thm. 3.1, Prop. 3.4, Thm. D.4), equivalence of DTA to a flat-prior Bayes score (Prop. 3.2), and an interval property for RoBAS-EB (Thm. 3.5), plus a grid-free endpoint algorithm. Experiments on synthetic residual means, three UCI tabular tasks under covariate shift, and two image regression tasks under in/out-of-distribution shift report mean±SE over 300 trials and show competitive in-distribution widths with substantially tighter intervals under shift relative to DTO, NNG, CQR, LOCAL, and related baselines.
Significance. If the results hold, the work cleanly addresses a known efficiency failure mode of Bayes-assisted conformal prediction under prior–data conflict without sacrificing exact frequentist coverage. Placing the BWM only on scalar residuals is a practical design choice that avoids high-dimensional priors and admits closed-form or special-function scores, improving scalability relative to full posterior-predictive MCMC scores. The asymptotic DTA reversion theorems, the EB interval property, the grid-free procedure, multi-domain experiments with error bars, and linked code are concrete strengths that make the contribution usable and checkable. The limitation to mean misalignment is real but scoped honestly; within that regime the paper offers a useful adaptive score family for small-calibration and training-to-deployment shift settings.
major comments (2)
- The robustness theory (Thm. 3.1, Prop. 3.4, Thm. D.4) and Discussion B correctly target residual-mean divergence. The main experiments (covariate tilt on UCI; age/OOD image splits) can also change residual scale and tails. Without residual-mean and residual-variance diagnostics under shift (e.g., in §4 or App. F), it is hard to verify that the reported width gains are driven by the mechanism the theorems describe rather than by generic recentring. A short diagnostic table or plot would make the theory–experiment link load-bearing rather than suggestive.
- §3.2–3.3 and App. E.3 fix γ=σ²/n and horseshoe a=b=1/2, and NNG uses τ²=1/n_cal (§4.1, App. C.1). These are reasonable defaults, but free parameters remain (γ, plug-in σ̂², NNG weight). A brief sensitivity check—especially for small n_cal where RoBAS is most advertised—would strengthen the claim that efficiency gains are not an artifact of a single hyperparameter setting.
minor comments (6)
- Figure 1 caption notes that the y-axis is cut; state the full range or put uncut panels in the appendix so absolute width inflation for DTO/NNG is visible.
- §2.2 motivates DTO vs DTA well; a one-sentence pointer there to Thm. 3.1 / Prop. 3.2 would help readers see the later theory as resolving that trade-off.
- Comparisons with CB/CBMA (App. F.3) are useful but the modified training-conditional score and adaptive grid make fairness delicate; state more prominently in the main text that they are not direct residual-score competitors.
- Algorithm 1 returns an interval approximation for generic scores; Thm. 3.5 covers RoBAS-EB. Clarify in §3.4 whether RoBAS-Full sets are observed to be intervals in practice or only approximated.
- Typos/notation: occasional spacing issues (e.g., “Bayes–assisted” consistency); ensure ¯r_n vs r̄_n is uniform; arXiv id in header is fine for review.
- Code link is given; a short README note on which figure each script regenerates would aid reproducibility.
Circularity Check
No significant circularity: core scores and asymptotics are derived from hierarchical/EB models via standard posterior arguments; only a minor technical self-citation appears in an appendix lemma.
specific steps
-
self citation load bearing
[Appendix D.2, proof of Theorem 3.1]
"By the regular-variation assumption on gτ2, the induced marginal density f is regularly varying as |z| → ∞ (see Cortinovis & Caron (2024, Proposition 4.3))."
A technical intermediate (regular variation of the marginal under the heavy-tailed mixing density) is imported from prior work by overlapping authors rather than fully re-derived in-line. However, it is not load-bearing for the central claim: the paper supplies its own compact-uniform Lemma D.3 from Bingham et al., the asymptotic score limit follows from standard posterior concentration + continuous mapping under the stated model, and the main robustness/efficiency results do not rest solely on this citation.
full rationale
The paper's central claims (RoBAS scores adapt via residual BWM (12) or EB plug-in (17), with asymptotic reversion to DTA under |¯r_n|→∞ by Thm 3.1/Prop 3.4, DTO recovery near zero, interval property for EB by Thm 3.5, and exact coverage by residual exchangeability) follow from explicit model definitions, conjugacy, regular-variation tail analysis, Tweedie/mgf arguments, and Lipschitz contraction proofs that are self-contained in §§3 and D. These do not reduce by construction to fitted targets or redefine the claimed outputs. Empirical Bayes ˆσ²/ˆυ² are ordinary plug-ins into the fixed-hyperparameter score (Prop 3.3), not a prediction of the widths/coverage. Experiments evaluate (not derive) the scores under shift. The sole self-citation (Cortinovis & Caron 2024 Prop 4.3, used once for regular variation of the marginal f in the proof of Thm 3.1) is a technical lemma supporting an intermediate step already backed by Bingham et al. and the paper's own Lemma D.3; it is not load-bearing for the main robustness or efficiency claims, which stand on the heavy-tail assumption and exchangeability. No self-definitional loops, fitted-as-prediction, uniqueness import, ansatz smuggling, or renaming of known results appear. Score 1 reflects only that minor non-load-bearing self-cite; the derivation chain is otherwise independent and non-circular.
Axiom & Free-Parameter Ledger
free parameters (4)
- γ (prior scale on residual mean) =
σ²/n
- horseshoe shape a=b=1/2 =
1/2
- NNG prior weight τ² =
1/n_cal
- plug-in residual variance σ̂² =
sample variance of residuals
axioms (4)
- domain assumption Calibration and test pairs are exchangeable (so residual scores remain exchangeable given fixed f).
- domain assumption g_τ² is regularly varying at infinity with index δ>1.
- domain assumption Residuals are modelled as i.i.d. N(θ,σ²) under the working model.
- domain assumption Base predictor f is fixed and trained on data independent of the calibration/test set.
invented entities (2)
-
RoBAS-Full score (hypergeometric expression from horseshoe residual BWM)
no independent evidence
-
RoBAS-EB absolute-deviation score with data-driven shrinkage factor â
no independent evidence
read the original abstract
Bayes-assisted conformal prediction combines the strengths of Bayesian modelling with exact, distribution-free frequentist coverage guarantees. Although conformal validity is preserved even when the Bayesian working model (BWM) is misspecified, the size of the resulting prediction sets can degrade substantially when the prior is poorly aligned with the observed data. We address this limitation by introducing RoBAS (Robust Bayes-Assisted Shrinkage): a Bayes-assisted framework for constructing robust nonconformity scores, with two instantiations: one induced by a heavy-tailed BWM, and a closed-form empirical Bayes shrinkage score. The resulting scores adapt to the quality of the working information encoded in the prior: when this information is reliable, they exploit it to produce efficient prediction sets; when it is weak or inaccurate, they revert to the Distance-To-Average (DTA) score, a robust non-informative baseline. We evaluate the proposed scores on tabular and image regression tasks where the training distribution may differ from the calibration and test distributions, while the calibration and test data themselves remain exchangeable. We find that they are competitive with widely used scores in the absence of such shift, while substantially reducing interval widths in shifted settings.
Figures
Reference graph
Works this paper leans on
-
[1]
, year = 2021, journal =
Fong, Edwin and Holmes, Chris C. , year = 2021, journal =. Conformal
2021
-
[2]
Statistical Science , volume = 26, number = 3, pages =
Frasian inference , author =. Statistical Science , volume = 26, number = 3, pages =
-
[3]
Journal of Machine Learning Research , volume = 9, pages =
A tutorial on conformal prediction , author =. Journal of Machine Learning Research , volume = 9, pages =
-
[4]
Algorithmic Learning in a Random World , author =
-
[5]
Foundations and Trends in Machine Learning , volume = 16, number = 4, pages =
Conformal prediction: A gentle introduction , author =. Foundations and Trends in Machine Learning , volume = 16, number = 4, pages =
-
[6]
Journal of Survey Statistics and Methodology , volume = 12, number = 5, pages =
Optimal conformal prediction for small areas , author =. Journal of Survey Statistics and Methodology , volume = 12, number = 5, pages =
-
[7]
Hoff, Peter , year = 2023, journal =
2023
-
[8]
Theoretical foundations of conformal prediction , author =. 2411.11824 , archiveprefix =
-
[9]
Proceedings of the 41st International Conference on Machine Learning , publisher =
Robust yet efficient conformal prediction sets , author =. Proceedings of the 41st International Conference on Machine Learning , publisher =
-
[10]
Kelly, Markelle and Longjohn, Rachel and Nottingham, Kolby , howpublished =. The
-
[11]
Conformalized quantile regression , author =
-
[12]
Advances in Neural Information Processing Systems , volume = 34, pages =
Conformal prediction using conditional histograms , author =. Advances in Neural Information Processing Systems , volume = 34, pages =
-
[13]
Proceedings of the 40th International Conference on Machine Learning , publisher =
Conformal prediction with missing values , author =. Proceedings of the 40th International Conference on Machine Learning , publisher =
-
[14]
International Conference on Learning Representations , url =
Probabilistic conformal prediction with approximate conditional validity , author =. International Conference on Learning Representations , url =
-
[15]
Conformal prediction under covariate shift , author =
-
[16]
Scikit-learn: Machine Learning in
Pedregosa, Fabian and Varoquaux, Ga. Scikit-learn: Machine Learning in. Journal of Machine Learning Research , volume = 12, pages =
-
[17]
Biometrika , volume = 110, number = 1, pages =
Localized conformal prediction: A generalized inference framework for conformal prediction , author =. Biometrika , volume = 110, number = 1, pages =
-
[18]
Deliu, Nina and Liseo, Brunero , year = 2025, url =. The interplay between. 2510.26930 , archiveprefix =
arXiv 2025
-
[19]
Journal of the Royal Statistical Society Series A: Statistics in Society , volume = 188, number = 3, pages =
Frequentist prediction sets for species abundance using indirect information , author =. Journal of the Royal Statistical Society Series A: Statistics in Society , volume = 188, number = 3, pages =
-
[20]
Journal of the Royal Statistical Society
Regression, prediction and shrinkage , author =. Journal of the Royal Statistical Society. Series B (Methodological) , volume = 45, number = 3, pages =
-
[21]
Bhagwat, Pankaj and Kong, Linglong and Jiang, Bei , year = 2025, url =. 2511.16924 , archiveprefix =
arXiv 2025
-
[22]
Priors in
Fortuin, Vincent , year = 2022, journal =. Priors in
2022
-
[23]
Wenzel, Florian and Roth, Kevin and Veeling, Bastiaan S. and. How good is the. Proceedings of the 37th International Conference on Machine Learning , publisher =
-
[24]
International Conference on Learning Representations , url =
Bayesian neural network priors revisited , author =. International Conference on Learning Representations , url =
-
[25]
Advances in Neural Information Processing Systems , volume = 37, pages =
Length optimization in conformal prediction , author =. Advances in Neural Information Processing Systems , volume = 37, pages =
-
[26]
Advances in Neural Information Processing Systems , volume = 37, pages =
Boosted conformal prediction intervals , author =. Advances in Neural Information Processing Systems , volume = 37, pages =
-
[27]
Advances in Neural Information Processing Systems , volume = 33, pages =
Classification with valid and adaptive coverage , author =. Advances in Neural Information Processing Systems , volume = 33, pages =
-
[28]
Proceedings of the National Academy of Sciences , volume = 118, number = 48, pages =
Distributional conformal prediction , author =. Proceedings of the National Academy of Sciences , volume = 118, number = 48, pages =
-
[29]
Journal of the American Statistical Association , volume = 114, number = 525, pages =
Least ambiguous set-valued classifiers with bounded error levels , author =. Journal of the American Statistical Association , volume = 114, number = 525, pages =
-
[30]
Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , publisher =
Improving adaptive conformal prediction using self-supervised learning , author =. Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , publisher =
-
[31]
Journal of the American Statistical Association , volume = 113, number = 523, pages =
Distribution-free predictive inference for regression , author =. Journal of the American Statistical Association , volume = 113, number = 523, pages =
-
[32]
doi:10.1002/9781118735855 , edition =
Small Area Estimation , author =. doi:10.1002/9781118735855 , edition =
-
[33]
Large-Scale Inference: Empirical
Efron, Bradley , year = 2010, publisher =. Large-Scale Inference: Empirical
2010
-
[34]
Computer Methods in Applied Mechanics and Engineering , volume = 152, number =
An efficient quadratic fit-sectioning algorithm for minimization without derivatives , author =. Computer Methods in Applied Mechanics and Engineering , volume = 152, number =
-
[35]
SIAM Journal on Scientific and Statistical Computing , volume = 6, number = 1, pages =
Three new rapidly convergent algorithms for finding a zero of a function , author =. SIAM Journal on Scientific and Statistical Computing , volume = 6, number = 1, pages =
-
[36]
Algorithms for Minimization without Derivatives , author =
-
[37]
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =
Age progression/regression by conditional adversarial autoencoder , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =
-
[38]
Transactions on Machine Learning Research , url =
How reliable is your regression model's uncertainty under real-world distribution shifts? , author =. Transactions on Machine Learning Research , url =
-
[39]
International Conference on Learning Representations , url =
An image is worth 16x16 words: Transformers for image recognition at scale , author =. International Conference on Learning Representations , url =
-
[40]
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =
Deep residual learning for image recognition , author =. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages =
-
[41]
Journal of the American Statistical Association , volume = 106, number = 496, pages =
Tweedie's formula and selection bias , author =. Journal of the American Statistical Association , volume = 106, number = 496, pages =
-
[42]
Cortinovis, Stefano and Caron, Fran. 2410.20169 , archiveprefix =
-
[43]
Confluent Hypergeometric Functions , author =
-
[44]
Numerical Recipes: The Art of Scientific Computing , author =
-
[45]
Machine Learning: ECML 2002 , publisher =
Inductive confidence machines for regression , author =. Machine Learning: ECML 2002 , publisher =
2002
-
[46]
Proceedings of the 27th Conference on Learning Theory , publisher =
Efficiency of conformalized ridge regression , author =. Proceedings of the 27th Conference on Learning Theory , publisher =
-
[47]
and Haberland, Matt and Reddy, Tyler and Cournapeau, David and Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and Bright, Jonathan and
Virtanen, Pauli and Gommers, Ralf and Oliphant, Travis E. and Haberland, Matt and Reddy, Tyler and Cournapeau, David and Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and Bright, Jonathan and. Nature Methods , volume = 17, pages =
-
[48]
Conformal and Probabilistic Prediction with Applications , publisher =
Conformal prediction is robust to dispersive label noise , author =. Conformal and Probabilistic Prediction with Applications , publisher =
-
[49]
Advances in Neural Information Processing Systems , volume = 34, pages =
Adaptive conformal inference under distribution shift , author =. Advances in Neural Information Processing Systems , volume = 34, pages =
-
[50]
The Annals of Statistics , volume = 51, number = 2, pages =
Conformal prediction beyond exchangeability , author =. The Annals of Statistics , volume = 51, number = 2, pages =
-
[51]
A conformal prediction score that is robust to label noise , author =. 2405.02648 , archiveprefix =
-
[52]
Proceedings of the 27th International Conference on Artificial Intelligence and Statistics , publisher =
On the expected size of conformal prediction sets , author =. Proceedings of the 27th International Conference on Artificial Intelligence and Statistics , publisher =
-
[53]
Prior distributions for variance parameters in hierarchical models (comment on article by
Gelman, Andrew , year = 2006, journal =. Prior distributions for variance parameters in hierarchical models (comment on article by
2006
-
[54]
Biometrika , volume = 97, number = 2, pages =
The horseshoe estimator for sparse signals , author =. Biometrika , volume = 97, number = 2, pages =
-
[55]
Stein's estimation rule and its competitors: An empirical
Efron, Bradley and Morris, Carl , year = 1973, journal =. Stein's estimation rule and its competitors: An empirical
1973
-
[56]
Data analysis using
Efron, Bradley and Morris, Carl , year = 1975, journal =. Data analysis using
1975
-
[57]
, year = 1983, journal =
Morris, Carl N. , year = 1983, journal =. Parametric empirical
1983
-
[58]
Dawid, A. P. , journal =. Posterior expectations for large observations , year =
-
[59]
Pericchi, L. R. and Sans. Biometrika , title =. 1995 , number =
1995
-
[60]
and Smith, A.F.M
Pericchi, L.R. and Smith, A.F.M. , journal =. Exact and approximate posterior moments for a normal location parameter , year =
-
[61]
1987 , volume =
Regular Variation , author =. 1987 , volume =
1987
-
[62]
2017 , booktitle =
On the Hyperprior Choice for the Global Shrinkage Parameter in the Horseshoe Prior , author =. 2017 , booktitle =
2017
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.