Pith. sign in

REVIEW 4 major objections 8 minor 29 references

Position: There Is No Free Bayesian Uncertainty Quantification

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that Bayesian posteriors do not meaningfully encode uncertainty, so predictive intervals must be calibrated by frequentist means to be valid.

desk verdict A defensible position paper that sharpens the Bayesian-UQ-as-ensemble view, but its central calibration guarantee is a stated hope rather than a proven theorem. read the letter →

arxiv 2506.03670 v1 pith:LDHBJ7GP submitted 2025-06-04 stat.ML cs.LG

Ivan Melev , Goeran Kauermann This is my paper
classification stat.MLcs.LG MSC 62F1562G1568T05
keywords BayesianuncertaintyquantificationfrequentistcoveragePAC-Bayespredictiveintervalsposteriorcalibrationensemblelearningpriorqualityoptimization-basedinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bayesian updating in machine learning, the paper argues, does not deliver uncertainty quantification for free: the posterior is the solution to an optimization problem that contains no explicit uncertainty objective, so interpreting it as UQ is not justified. A posterior is better understood as an optimally weighted ensemble of models, and any Bayesian credible or predictive interval must be checked by frequentist criteria before it supports an uncertainty claim. To make that check concrete, the paper defines average, worst-case, and probabilistic measures of a prior's quality, and proposes a calibration algorithm that adjusts the quantiles of the Bayesian predictive distribution until empirical coverage on a separate dataset reaches a target level. A PAC-style bound is stated as giving valid frequentist coverage with high probability, and simulations show the calibrated intervals reaching nominal coverage where naive Bayesian intervals do not, except under severe misspecification or prior-data conflict where numerical precision fails. The position, if accepted, turns Bayesian UQ into a two-stage procedure: first build the posterior ensemble, then learn valid intervals from data.

What carries the argument

The load-bearing object is the optimization-based representation of Bayes' rule (Eq. 1), which converts posterior inference into a regularized optimization over probability measures, with the prior as 'centroid' and the divergence as regularizer; the paper reads this representation as evidence that posterior uncertainty is a by-product, not an objective. The second piece is the Section 4.2 calibration routine: predictive intervals are defined by quantiles $c_1(q,x)$ and $c_2(q,x)$ of the predictive distribution, a quantile-estimation dataset is used to compute empirical risk as a 0-1 loss over $q$, and grid search finds the smallest symmetric $q$ with empirical risk at most $\alpha$. The PAC bound of Eq. 9 is the mechanism that supposedly converts this empirical risk into a population coverage guarantee up to slack $C(\epsilon)$.

What would settle it

Run the Section 4.2 calibration on many independent simulated training/calibration/test splits with a misspecified model and a prior mean far from the truth, then record the fraction of splits in which the calibrated interval achieves the nominal $1-\alpha$ coverage; if that fraction does not reach the claimed high-probability level once an explicit $C(\epsilon)$ is supplied, Eq. 9 is false as stated.

Watch

Extended reading notes

Core claim

The paper's central claim is that Bayes' rule is an optimization scheme in disguise: choosing the posterior $p$ by minimizing $\mathbb{E}_{f\sim p}[l(f(x),y)] + D(p\|\pi_0)$ over all probability measures, with negative log-likelihood loss and KL divergence as $D$, reproduces the Bayes update. Since that objective contains no term that asks for calibrated or valid uncertainty statements, the paper concludes that posterior or predictive distributions carry no built-in uncertainty guarantee. It therefore reinterprets the posterior as an optimal ensemble relative to a prior, and treats the predictive distribution as an ensemble summary rather than a UQ device. It then supplies frequentist tools: quality measures $Q(\pi_0)$, $Q'(\pi_0)$, and $Q''(\pi_0)$ for evaluating priors by average, worst-case, and probabilistic coverage, and a calibration procedure that chooses a symmetric quantile $\hat{q}$ by binary classification on a held-out quantile-estimation set. The claimed warrant for the procedure is the PAC bound $P(R((X,Y),\hat{q})\le \alpha + C(\epsilon))\ge 1-\epsilon(n)$, giving valid frequentist intervals with high probability.

Load-bearing premise

The entire high-probability validity claim rests on an unproved PAC bound: the paper assumes that a low empirical error on the quantile-estimation dataset implies a low population error with unspecified slack $C(\epsilon)$, and even acknowledges that the generalization gap remains to be shown.

Editorial extensions

If this is right

  • Bayesian credible and predictive intervals in machine learning would no longer be reported as uncertainty quantifiers until their frequentist coverage has been checked.
  • Posteriors from Bayesian neural networks become ensemble weightings, and valid UQ becomes a separate calibration step on held-out data.
  • Prior choice becomes an empirical problem: priors can be compared by their average, worst-case, or probabilistic coverage, giving Bayesian inference an external validity criterion.
  • The one-parameter quantile calibration offers a simple, distribution-free route to predictive intervals with approximately nominal coverage, requiring only grid search on a quantile-estimation dataset.
  • The paper's own simulations indicate that under strong prior-data conflict or severe misspecification, even calibrated intervals can fail because the predictive distribution assigns numerically negligible probability to observed points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors leave implicit is to benchmark the calibrated Bayesian predictive intervals against other distribution-free interval constructions on the same regression and classification tasks.
  • The prior-quality measures $Q$, $Q'$, and $Q''$ suggest an operational prior-selection rule: choose the prior that maximizes estimated probabilistic coverage $Q''(\pi_0)$, turning subjective prior choice into a tuned hyperparameter.
  • If the missing generalization gap $C(\epsilon)$ turns out to be large, the paper's suggestion to replace $\alpha$ by a smaller $\alpha'$ would make the method conservative at the cost of wider intervals; this trade-off is a testable prediction of the framework.
  • The documented failure at $\beta_{20}=3$ points to an improvement: a coverage-aware fallback that expands intervals whenever the predictive density at observed points falls below numerical precision, rather than relying on grid-search quantiles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper argues that Bayesian posterior distributions, when viewed as solutions to the optimization problem in Eq. (1), do not by themselves provide uncertainty quantification, because no explicit coverage or accuracy objective appears in that problem. It proposes three frequentist quality measures for priors (Eqs. 2, 4, and 5) and a calibration algorithm (Sec. 4.2) that adjusts the central quantile q of the predictive distribution on a held-out set, claiming PAC-style frequentist coverage of the resulting intervals (Eq. 9). Two simulation studies on Bayesian linear regression, including a misspecified model, compare calibrated and naive intervals. The paper concludes that Bayesian UQ requires frequentist validation and calibration before use.

Significance. The paper addresses an important and actively debated question, and its central negative claim is coherent as a position. The proposed quality measures are reasonable formalizations of prior performance, and the calibration algorithm is concrete and honestly evaluated on a separate test set, so the empirical demonstration is not circular. The paper also states several of its limitations explicitly, including the unproved generalization bound and the numerical precision issue. If the missing PAC bound is supplied, or if the claims are appropriately weakened to a conjecture, the paper could be a useful contribution to the debate on Bayesian UQ in machine learning. As it stands, however, the positive contribution rests on an asserted rather than proved guarantee, and the simulations contain a clear failure case.

major comments (4)
  1. [Sec. 4.2, Eq. (9)] The central claim that the calibration algorithm "leads to valid frequentist intervals with high probability" is stated without proof. The bound is announced as achievable "using the PAC framework," but C(epsilon) is never specified, the sample-size dependence is unclear (epsilon(n) despite the validation set having size m), and the need for uniformity over the grid-searched q is not addressed. The paper itself defers the generalization gap to future work, and the low-complexity claim is supported only by the heuristic that "only one parameter is estimated." This is load-bearing because the paper's constructive claim depends on Eq. (9).
  2. [Sec. 4.2, support assumption before Eq. (6)] The argument that some q drives the empirical risk to alpha relies on the support of the data being a subset of the support of the predictive distribution. This condition is not guaranteed for misspecified or overparameterized models, and no diagnostic is offered. The beta20=3 experiment in Sec. 5.2 is a concrete failure: calibrated coverage remains below the 0.9 target, and the paper's explanation in terms of numerical precision (q of order 1e-40) does not amount to a working guarantee. The paper also does not specify what the algorithm does when no grid point achieves empirical risk <= alpha, which is exactly the situation in that experiment.
  3. [Sec. 5.2] The sentence "since no explicit assumptions were made in the presentation in section 4.2, there are not any limitations of the utility of the method on more complex model classes" contradicts the support assumption stated in Sec. 4.2 and overlooks the dependence on the unproved PAC bound in Eq. (9). This overclaim should be removed or substantially qualified.
  4. [Secs. 5.1-5.2] The empirical evaluation averages over only 10 seeds and reports no standard errors or confidence bands for coverage or width. Because Eq. (9) is unproved, the simulations carry much of the evidence for the algorithm's practical validity. Without variability measures or a description of the grid resolution and stopping rule, the reader cannot assess the stability or reproducibility of the reported differences, nor whether the beta20=3 failure is an artifact of the particular grid choice.
minor comments (8)
  1. [Sec. 4.1] The word "pesimisstic" should be "pessimistic," and "can be as a softer version" should be "can be seen as a softer version."
  2. [Sec. 5.2] The word "achiving" should be "achieving."
  3. [Introduction] The reference to Theodoridis [2015] appears twice in the introductory paragraph; please consolidate.
  4. [Sec. 4.2] The notation P(Y^{v,*,i} | X^v_i) is not defined explicitly; a short definition of the predictive distribution on the quantile estimation set would help the reader.
  5. [Sec. 4.2] The grid-search resolution, the stopping tolerance, and the behavior when no q satisfies the empirical risk condition are not specified; these details are needed for reproducibility.
  6. [Sec. 3] The heading "Optimization does not give UQ" states a categorical conclusion, while the text offers an interpretive argument; consider softening the wording to avoid inviting a formal disproof that the paper does not provide.
  7. [References] The paper does not discuss the close relationship to conformal prediction, which gives finite-sample coverage under exchangeability; a brief comparison would clarify the contribution.
  8. [Sec. 6] The sentence "the solution of a well specified problem, as discussed in 4.2, always exists" is too strong because the existence of a suitable q depends on the support assumption and on numerical realizability; please qualify.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the Bayesian-as-optimization equivalence is a known result, the prior-quality measures are definitions, and the calibration algorithm's coverage is evaluated on held-out test data; Eq. 9 is unproved but that is a support gap, not circularity.

full rationale

The paper's negative claim is an interpretive position, not an equation-level reduction. Eq. 1 reproduces Bayes' rule when the loss is negative log-likelihood and D is KL (Zellner, 1988); the paper then argues that this optimization objective contains no explicit coverage target. That is a philosophical challenge, not a circular derivation: it does not define Bayesian UQ in terms of the conclusion, and the posterior is not constructed from the target coverage. The prior-quality measures (2), (4), and (5) are definitions of expected, worst-case, and probabilistic frequentist coverage; they do not presuppose the paper's conclusion. The calibration algorithm in Section 4.2 selects q on a separate quantile-estimation set and evaluates coverage on a test set, so the empirical demonstration is not a fitted-parameter prediction. The main weakness is Eq. 9: the PAC bound is stated with C(epsilon) unspecified and the generalization gap explicitly deferred to future work; this is an unproved assertion, not a circularity. The 'no explicit assumptions' sentence in Section 5.2 overclaims given the support assumption in Section 4.2, but overclaiming is not circular reasoning. There are no load-bearing self-citations or imported uniqueness theorems. Consequently no circular step can be quoted, and the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the optimization-centric rewriting of Bayesian updating (Eq. 1), which is standard but cited from Knoblauch et al. 2022. The calibration claim rests on an unproven PAC inequality (Eq. 9). The algorithm fits a quantile level q on a validation set. No new physical or mathematical entities are introduced.

free parameters (2)
  • q (quantile level of the predictive interval) = chosen by grid search on the validation set; not reported numerically
    In Section 4.2 the algorithm scans q in (0, 0.5] and stops when the validation empirical risk is at most alpha. The reported coverage depends on this fitted value, and no numerical values for q are given in the simulations.
  • alpha (allowed coverage error) = 0.1
    Set to 0.1 in both simulations (1-alpha = 0.9). This is a user-chosen constant, not fitted, but it controls the stated guarantee.
assumptions (4)
  • standard math Bayesian updating is the minimizer of min_p {E_p[l] + KL(p||pi0)} under negative log-likelihood loss and KL divergence (Zellner 1988).
    Invoked in Section 3 to reinterpret the posterior as the solution to an optimization problem.
  • domain assumption The support of the data is a subset of the support of the predictive distribution.
    Stated in Section 4.2 before the calibration algorithm; needed so that a sufficiently wide q can achieve empirical coverage.
  • ad hoc to paper A PAC generalization bound of the form P(R((X,Y), q_hat) <= alpha + C(epsilon)) >= 1-epsilon(n) holds for the quantile-selection classifier.
    Equation 9 states this bound but C(epsilon) is not specified and the generalization gap is deferred to future work. The paper's central algorithmic claim depends on this unproven bound.
  • standard math Bernstein-von Mises theorem: posteriors converge to a Gaussian centered at the MLE with covariance I(theta0)^-1 for finite-dimensional parameters.
    Used in Section 2 to argue that priors matter less asymptotically, motivating the finite-sample concern.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: There Is No Free Bayesian Uncertainty Quantification." pith.science (2026). https://pith.science/paper/LDHBJ7GP

@misc{pith2026250603670,
  author       = {Pith},
  title        = {Pith review of: Position: There Is No Free Bayesian Uncertainty Quantification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LDHBJ7GP}},
  note         = {Machine review of arXiv:2506.03670}
}
read the original abstract

Due to their intuitive appeal, Bayesian methods of modeling and uncertainty quantification have become popular in modern machine and deep learning. When providing a prior distribution over the parameter space, it is straightforward to obtain a distribution over the parameters that is conventionally interpreted as uncertainty quantification of the model. We challenge the validity of such Bayesian uncertainty quantification by discussing the equivalent optimization-based representation of Bayesian updating, provide an alternative interpretation that is coherent with the optimization-based perspective, propose measures of the quality of the Bayesian inferential stage, and suggest directions for future work.

Figures

Figures reproduced from arXiv: 2506.03670 by the authors.

Figure 1
Figure 1. On the left we see coverage of observed test data using the naive approach and the calibrated [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. The setup is the same as in figure 1, the only difference is that in this case there is a missing [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The setup is the same as in figure 1, the only difference is that in this case there is a missing [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 24 canonical work pages

  1. [1]

    Concentration of tempered posteriors and of their variational approximations

    Pierre Alquier and James Ridgway. Concentration of tempered posteriors and of their variational approximations. 2020

    work page 2020

  2. [2]

    Bayesian neural networks via mcmc: a python-based tutorial

    Rohitash Chandra and Joshua Simmons. Bayesian neural networks via mcmc: a python-based tutorial. IEEE Access, 2024

    work page 2024

  3. [3]

    Bayesian graph convolutional neural networks via tempered mcmc

    Rohitash Chandra, Ayush Bhagat, Manavendra Maharana, and Pavel N Krivitsky. Bayesian graph convolutional neural networks via tempered mcmc. IEEE Access, 9: 0 130353--130365, 2021

    work page 2021

  4. [4]

    Safe learning: bridging the gap between bayes, mdl and statistical learning theory via empirical convexity

    Peter Gr \"u nwald. Safe learning: bridging the gap between bayes, mdl and statistical learning theory via empirical convexity. In Proceedings of the 24th Annual Conference on Learning Theory, pages 397--420. JMLR Workshop and Conference Proceedings, 2011

    work page 2011

  5. [5]

    Minimum description length revisited

    Peter Gr \"u nwald and Teemu Roos. Minimum description length revisited. International journal of mathematics for industry, 11 0 (01): 0 1930001, 2019

    work page 2019

  6. [6]

    A tight excess risk bound via a unified pac-bayesian--rademacher--shtarkov--mdl complexity

    Peter D Gr \"u nwald and Nishant A Mehta. A tight excess risk bound via a unified pac-bayesian--rademacher--shtarkov--mdl complexity. In Algorithmic Learning Theory, pages 433--465. PMLR, 2019

    work page 2019

  7. [7]

    A primer on pac-bayesian learning

    Benjamin Guedj. A primer on pac-bayesian learning. arXiv preprint arXiv:1901.05353, 2019

    arXiv 1901

  8. [8]

    Bootstrap

    Tim Hesterberg. Bootstrap. Wiley Interdisciplinary Reviews: Computational Statistics, 3 0 (6): 0 497--526, 2011

    work page 2011

Show all 29 references
  1. [9]

    Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods

    Eyke H \"u llermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110 0 (3): 0 457--506, 2021

    2021

  2. [10]

    An optimization-centric view on bayes' rule: Reviewing and generalizing variational inference

    Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization-centric view on bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23 0 (132): 0 1--109, 2022

    2022

  3. [11]

    Being bayesian, even just a bit, fixes overconfidence in relu networks

    Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in relu networks. In International conference on machine learning, pages 5436--5446. PMLR, 2020

    2020

  4. [12]

    Bayesian neural networks and density networks

    David JC MacKay. Bayesian neural networks and density networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 354 0 (1): 0 73--80, 1995

    1995

  5. [13]

    Simplified pac-bayesian margin bounds

    David McAllester. Simplified pac-bayesian margin bounds. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pages 203--215. Springer, 2003 a

    2003

  6. [14]

    Some pac-bayesian theorems

    David A McAllester. Some pac-bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory, pages 230--234, 1998

    1998

  7. [15]

    Pac-bayesian stochastic model selection

    David A McAllester. Pac-bayesian stochastic model selection. Machine Learning, 51 0 (1): 0 5--21, 2003 b

    2003

  8. [16]

    Probabilistic machine learning: an introduction

    Kevin P Murphy. Probabilistic machine learning: an introduction. MIT press, 2022

    2022

  9. [17]

    Why are bootstrapped deep ensembles not better? In ''I Can't Believe It's Not Better!''NeurIPS 2020 workshop, 2020

    Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In ''I Can't Believe It's Not Better!''NeurIPS 2020 workshop, 2020

    2020

  10. [18]

    Pac confidence sets for deep neural networks via calibrated prediction

    Sangdon Park, Osbert Bastani, Nikolai Matni, and Insup Lee. Pac confidence sets for deep neural networks via calibrated prediction. arXiv preprint arXiv:2001.00106, 2019

    2001 arXiv

  11. [19]

    Pac confidence predictions for deep neural network classifiers

    Sangdon Park, Shuo Li, Insup Lee, and Osbert Bastani. Pac confidence predictions for deep neural network classifiers. arXiv preprint arXiv:2011.00716, 2020

    2011 arXiv

  12. [20]

    A comparison of the Bayesian and frequentist approaches to estimation, volume 24

    Francisco J Samaniego. A comparison of the Bayesian and frequentist approaches to estimation, volume 24. Springer, 2010

    2010

  13. [21]

    Machine learning: a Bayesian and optimization perspective

    Sergios Theodoridis. Machine learning: a Bayesian and optimization perspective. Academic press, 2015

    2015

  14. [22]

    Bayesian inference: An introduction to principles and practice in machine learning

    Michael E Tipping. Bayesian inference: An introduction to principles and practice in machine learning. In Summer School on Machine Learning, pages 41--62. Springer, 2003

    2003

  15. [23]

    Asymptotic statistics, volume 3

    Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000

    2000

  16. [24]

    On mcmc sampling in bayesian mlp neural networks

    Aki Vehtari, Simo Sarkka, and Jouko Lampinen. On mcmc sampling in bayesian mlp neural networks. In Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, volume ...

    2000

  17. [25]

    Frequentist inference

    Jon Wakefield and Jon Wakefield. Frequentist inference. Bayesian and Frequentist Regression Methods, pages 27--83, 2013

    2013

  18. [26]

    All of statistics: a concise course in statistical inference

    Larry Wasserman. All of statistics: a concise course in statistical inference. Springer Science & Business Media, 2013

    2013

  19. [27]

    How good is the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405, 2020

    Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub \'S wi a tkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405, 2020

    2002 arXiv

  20. [28]

    u nnemann, and David R \

    Jonas Gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan G \"u nnemann, and David R \"u gamer. Towards efficient mcmc sampling in bayesian neural networks by exploiting symmetry. In Joint European Conference on Machine Learning and Knowledge Discovery in Dat...

    2023

  21. [29]

    Optimal information processing and bayes's theorem

    Arnold Zellner. Optimal information processing and bayes's theorem. The American Statistician, 42 0 (4): 0 278--280, 1988

    1988

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.