Pith. sign in

REVIEW 5 minor 19 references

Quantification of Credal Uncertainty: A Distance-Based Approach

T0 review · 0 major / 5 minor · reviewed 2026-07-15 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Total, aleatoric, and epistemic uncertainty of a credal set can be read off as distances under integral probability metrics, with total variation giving closed multiclass formulas that recover the known binary case.

desk verdict Clean multiclass extension of the binary credal decomposition, with closed-form TV measures that are axiomatically grounded and cheap to compute. read the letter →

arxiv 2603.27270 v2 pith:SGVCBFI3 submitted 2026-03-28 cs.AI stat.ML

Xabier Gonzalez-Garcia , Siu Lun Chau , Julian Rodemann , Michele Caprio , Krikamol Muandet , Humberto Bustince , Sébastien Destercke , Eyke Hüllermeier
show 1 more author
Yusuf Sale
This is my paper · ORCID
classification cs.AIstat.ML
keywords credalsetsaleatoricuncertaintyepistemicintegralprobabilitymetricstotalvariationmulticlassclassificationquantification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Credal sets—closed convex sets of probability distributions—are a natural way to represent both irreducible randomness (aleatoric uncertainty) and ignorance about which distribution is correct (epistemic uncertainty). The open problem is how to turn a credal set into three numbers that quantify total, aleatoric, and epistemic uncertainty, especially when there are more than two classes. This paper defines those three quantities as distances to full certainty or as the diameter of the set, all inside the family of Integral Probability Metrics. Under total variation the definitions collapse to simple closed forms that can be evaluated from the extreme points of an ensemble, run in linear or quadratic time, and recover the established binary measures as a special case. Experiments on selective prediction show the new scores match or beat entropy- and Hartley-style baselines while remaining far cheaper to compute.

What carries the argument

Integral Probability Metric distances on the simplex: TU_F(Q) = inf_y sup_p∈Q d_F(p, δ_y), AU_F(p) = inf_y d_F(p, δ_y) (lifted to a set over Q), and EU_F(Q) = (1/2) sup_{p,q∈Q} d_F(p,q). Under the total-variation test class these become 1 − max lower probability, 1 − max probability, and (1/4) max ℓ1 distance, respectively.

What would settle it

On a multiclass ensemble-induced credal set, compute the total-variation formulas and check whether the resulting total-uncertainty score fails to equal aleatoric lower endpoint plus twice epistemic uncertainty in the binary restriction, or whether accuracy-rejection curves fall below entropy and Hartley baselines on standard image and tabular benchmarks.

Watch

Extended reading notes

Core claim

A family of total, aleatoric, and epistemic uncertainty measures for multiclass credal sets can be defined by distances induced by Integral Probability Metrics: total uncertainty is the worst-case distance of the set to the nearest Dirac measure, aleatoric uncertainty is the set of distances of each interior distribution to its nearest Dirac, and epistemic uncertainty is half the maximal pairwise distance inside the set. Instantiated with total variation these quantities admit closed forms, satisfy natural axioms, and recover the known binary decomposition.

Load-bearing premise

The chosen family of test functions must separate probability measures and be uniformly continuous, so that the induced distance is a genuine continuous metric; otherwise the geometric readings and the axiom proofs fail.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper proposes a distance-based framework for quantifying total, aleatoric, and epistemic uncertainty of credal sets in multiclass classification, grounded in Integral Probability Metrics. Total uncertainty is the Hausdorff-type distance of Q to the nearest Dirac measure (Eq. 1); aleatoric uncertainty is the set of distances of each p in Q to the nearest Dirac (Eqs. 2–3), with an optional endpoint summary; epistemic uncertainty is half the maximal diameter of Q (Eq. 4). Under mild conditions on F the measures satisfy axioms A1–A4 and A7 (Prop. 4.2) and the natural inequalities of Prop. 4.3–4.5. Instantiating with total variation yields the closed forms of Prop. 4.6, recovers the binary decomposition of Hüllermeier et al. (2022) as Prop. 4.10, and admits linear or quadratic evaluation for finitely generated credal sets (Table 1). Selective-prediction experiments on CIFAR-10/100 and 64 KEEL data sets show competitive AUC/MR against entropy and Hartley baselines at substantially lower cost.

Significance. The work fills a genuine gap: a multiclass generalization of the binary measures of Hüllermeier et al. (2022) that is both axiomatically grounded and computationally practical. The geometric definitions avoid residual additive decompositions, the set-valued treatment of aleatoric uncertainty is conceptually clean, and the TV closed forms together with the extreme-point reductions make the measures immediately usable in ensemble pipelines. The appendix supplies complete proofs of Props. 4.2–4.10 by standard convex-analysis arguments, and the empirical section reports both accuracy–rejection curves and wall-clock times, giving a transparent picture of the accuracy–cost trade-off. These strengths make the contribution of clear interest to the imprecise-probability and uncertainty-quantification communities in machine learning.

minor comments (5)
  1. In the proof of Prop. 4.2 the appeal to the Bauer maximum principle is stated for both convex and concave maps; a one-sentence reminder that the continuous affine map p↦Ep[f] is both convex and concave would make the argument fully self-contained.
  2. Table 1 lists asymptotic lower and upper bounds for AUFTV; a short remark clarifying when the LP is required (disagreement of argmax labels) versus when the closed-form max is sufficient would help practitioners.
  3. Figure 1 is a useful geometric illustration, yet the caption does not explicitly state that the arrows represent IPM distances; adding that phrase would improve readability for readers less familiar with the framework.
  4. The pathological case of Appendix E.2 is already disclosed; a single sentence in the main-text Limitations paragraph pointing to the same appendix would make the caveat more visible without expanding the discussion.
  5. A few typographical inconsistencies appear (e.g., “TUF(Q)” versus “TUFTV(Q)” spacing, and the occasional missing space after commas in the complexity table). A light copy-edit pass would remove them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: geometric IPM definitions and closed forms are independent constructions; binary recovery is a derived corollary, not an input.

full rationale

The paper defines TU_F, AU_F and EU_F directly via Hausdorff-type distances to Dirac vertices and half-diameter of the credal set (Eqs. 1–4). These are free geometric constructions inside the IPM framework; the subsequent proofs of axioms A1–A4/A7 (Prop. 4.2), the inequality EU ≤ TU (Prop. 4.3), translation invariance (Prop. 4.4), the TV closed forms (Prop. 4.6) and the binary specialisation (Prop. 4.10) are ordinary mathematical derivations that start from those definitions and standard convex-analysis facts. The recovery of the Hüllermeier et al. (2022) binary measures is obtained by restricting the same closed forms to K=2; it is therefore a corollary, not a premise that forces the multiclass definitions. Self-citations (Sale et al., Chau et al., Caprio et al.) appear only in related-work discussion, empirical baselines or non-load-bearing comparisons (e.g., Prop. 4.5 relating diameter to MMI). No parameter is fitted to data and then re-presented as a prediction, no uniqueness theorem is imported to forbid alternatives, and no ansatz is smuggled via citation. The theoretical claims are therefore self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

The framework rests on standard IPM theory plus a short list of regularity assumptions on F; no free parameters are fitted to obtain the theoretical claims. The desiderata A1–A7 are taken from the imprecise-probability literature and verified rather than postulated ad hoc.

assumptions (4)
  • domain assumption F is a uniform class w.r.t. weak convergence (equicontinuous, uniformly bounded span) so that d_F is continuous (Müller 1997, Thm 4.3).
    Invoked in Remark 4.1 and used for continuity of all four uncertainty functionals (Prop. 4.2).
  • domain assumption F separates probability measures, making d_F a metric rather than a pseudo-metric.
    Required for the geometric interpretations and for A1–A4 to hold non-trivially.
  • domain assumption Credal sets are closed and convex (standard coherence assumption of Walley 1991).
    Used throughout for extreme-point characterizations (A7) and attainment of suprema.
  • domain assumption Axioms A1–A7 (boundedness, continuity, monotonicity, probability consistency, subadditivity, additivity, extreme-point characterization) are the correct desiderata for uncertainty measures.
    Taken from Abellán & Klir, Jiroušek & Shenoy; verified for the new measures rather than derived.
invented entities (2)
  • Set-valued aleatoric uncertainty AU_F(Q) together with the endpoint summary operator Φ_int
    purpose: Preserve the full range of randomness values induced by a credal set while still allowing scalar ranking for selective prediction.
    Defined in Eqs. (3)–(4); the lossless claim is proved only under convexity/compactness of Q and continuity of AU_F.
  • Half-diameter epistemic measure EU_F(Q) = (1/2) diam_F(Q)
    purpose: Quantify imprecision independently of location inside the simplex and guarantee EU ≤ TU.
    Eq. (4); the factor 1/2 is a convenient normalization chosen by the authors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantification of Credal Uncertainty: A Distance-Based Approach." pith.science (2026). https://pith.science/paper/SGVCBFI3

@misc{pith2026260327270,
  author       = {Pith},
  title        = {Pith review of: Quantification of Credal Uncertainty: A Distance-Based Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SGVCBFI3}},
  note         = {Machine review of arXiv:2603.27270}
}
read the original abstract

Credal sets, i.e., closed convex sets of probability measures, provide a natural framework to represent aleatoric and epistemic uncertainty in machine learning. Yet how to quantify these two types of uncertainty for a given credal set, particularly in multiclass classification, remains underexplored. In this paper, we propose a distance-based approach to quantify total, aleatoric, and epistemic uncertainty for credal sets. Concretely, we introduce a family of such measures within the framework of Integral Probability Metrics (IPMs). The resulting quantities admit clear semantic interpretations, satisfy natural theoretical desiderata, and remain computationally tractable for common choices of IPMs. We instantiate the framework with the total variation distance and obtain simple, efficient uncertainty measures for multiclass classification. In the binary case, this choice recovers established uncertainty measures, for which a principled multiclass generalization has so far been missing. Empirical results confirm practical usefulness, with favorable performance at low computational cost.

Figures

Figures reproduced from arXiv: 2603.27270 by the authors.

Figure 1
Figure 1. Geometric illustration of the proposed distance-based framework on the simplex [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Selective prediction for credal sets constructed via relative likelihood, on FASHION-MNIST and SVHN. Left panels: Accuracy–rejection (AR) curves for (a) total uncertainty, (b) aleatoric uncertainty, and (c) epistemic uncertainty. Area Under the Curve (AUC, ↑ higher is better) is reported in each legend. Right panels: Cumulative distribution of uncertainty scores across all test instances. The vertical dashed line ma… view at source ↗
Figure 4
Figure 4. Selective prediction for credal sets constructed via relative likelihood, including h0.0, on FASHION-MNIST and SVHN. Left panels: Accuracy–rejection (AR) curves for (a) total uncertainty, (b) aleatoric uncertainty, and (c) epistemic uncertainty. Area Under the Curve (AUC, ↑ higher is better) is reported in each legend. Right panels: Cumulative distribution of uncertainty scores across all test instances. The vertica… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 5 linked inside Pith

  1. [1]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria-Florina Balcan and Kilian Q. Weinberger, editors,Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 ofJMLR Workshop and Conference Proceedings, pag...

    2016

  2. [2]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors,Advances in Neural Information Processing Systems 30: Annual Conference on Neural ...

    2017

  3. [3]

    Andrey Malinin and Mark J. F. Gales. Predictive uncertainty estimation via prior networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors,Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 201...

    2018

  4. [4]

    Kaplan, and Melih Kandemir

    Murat Sensoy, Lance M. Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors,Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIP...

    2018

  5. [5]

    Credal bayesian deep learning.Trans

    Michele Caprio, Souradeep Dutta, Kuk Jin Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky, and Insup Lee. Credal bayesian deep learning.Trans. Mach. Learn. Res., 2024, 2024a. Kaizheng Wang, Fabio Cuzzolin, Keivan Shariatmadar, David Moens, and Hans Hallez. Credal wrapper of model averaging for uncertainty estimation in classification. InThe Thirteenth Int...

    2024

  6. [6]

    Timo Löhr, Paul Hofman, Felix Mohr, and Eyke Hüllermeier

    OpenReview.net, 2025a. Timo Löhr, Paul Hofman, Felix Mohr, and Eyke Hüllermeier. Credal prediction based on relative likelihood.arXiv preprint arXiv:2505.22332,

    arXiv

  7. [7]

    Is the volume of a credal set a good measure for epistemic uncertainty? InUncertainty in Artificial Intelligence, pages 1795–1804

    Yusuf Sale, Michele Caprio, and Eyke Hüllermeier. Is the volume of a credal set a good measure for epistemic uncertainty? InUncertainty in Artificial Intelligence, pages 1795–1804. PMLR, 2023a. Yusuf Sale, Viktor Bengs, Michele Caprio, and Eyke Hüllermeier. Second-order uncertainty quantification: A distance- based approach.arXiv preprint arXiv:2312.00995...

    arXiv

  8. [8]

    A rate-distortion view of uncertainty quantification.arXiv preprint arXiv:2406.10775,

    Ifigeneia Apostolopoulou, Benjamin Eysenbach, Frank Nielsen, and Artur Dubrawski. A rate-distortion view of uncertainty quantification.arXiv preprint arXiv:2406.10775,

    arXiv

Show all 19 references
  1. [9]

    Quantifying epistemic predictive uncertainty in conformal prediction.arXiv preprint arXiv:2602.01667,

    Siu Lun Chau, Soroush H Zargarbashi, Yusuf Sale, and Michele Caprio. Quantifying epistemic predictive uncertainty in conformal prediction.arXiv preprint arXiv:2602.01667,

  2. [10]

    Imprecise Bayesian Neural Networks.arXiv preprint arXiv:2302.09656,

    Michele Caprio, Souradeep Dutta, Radoslav Ivanov, Kuk Jang, Vivian Lin, Oleg Sokolsky, and Insup Lee. Imprecise Bayesian Neural Networks.arXiv preprint arXiv:2302.09656,

    arXiv

  3. [11]

    Credal two-sample tests of epistemic uncertainty

    Siu Lun Chau, Antonin Schrab, Arthur Gretton, Dino Sejdinovic, and Krikamol Muandet. Credal two-sample tests of epistemic uncertainty. InInternational Conference on Artificial Intelligence and Statistics, pages 127–135. PMLR, 2025b. Michele Caprio, Yusuf Sale, and Eyke Hüllerm...

    arXiv

  4. [12]

    The joys of categorical conformal prediction.arXiv preprint arXiv:2507.04441,

    Michele Caprio. The joys of categorical conformal prediction.arXiv preprint arXiv:2507.04441,

    arXiv

  5. [13]

    Quantifying aleatoric and epistemic uncertainty: A credal approach

    Paul Hofman, Yusuf Sale, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty: A credal approach. InICML 2024 Workshop on Structured Probabilistic Inference{\&}Generative Modeling,

    2024

  6. [14]

    Axioms for uncertainty measures on belief functions and credal sets

    Andrey Bronevich and George J Klir. Axioms for uncertainty measures on belief functions and credal sets. InNAFIPS 2008-2008 Annual Meeting of the North American Fuzzy Information Processing Society, pages 1–6. IEEE,

    2008

  7. [15]

    coherence

    and was later popularized and given a solid theoretical foundation by Walley [1991]. As introduced in Section 4, credal sets usually—not always—refer to closed and convex sets of probability measures. This is due to the fact that under “coherence” axioms, the representing set ...

    1991

  8. [16]

    These newer contributions build on a rich foundation in imprecise probability theory and uncertainty quantification Pal et al

    proposed uncertainty measures for credal sets grounded in proper scoring rules. These newer contributions build on a rich foundation in imprecise probability theory and uncertainty quantification Pal et al. [1992, 1993], Moral [1992], Walley and Moral [1999], Abellan and Moral...

    1992

  9. [17]

    infimum) of a convex (resp

    A7(Extreme point characterization).Since Q is convex and compact, the Bauer maximum principle [Bauer, 1958] applies: the supremum (resp. infimum) of a convex (resp. concave) and continuous function on Q is attained at an extreme point. For any fixed y∈ Y and f∈ F , the map p7→...

    1958

  10. [18]

    [2025], a recent methodology that offers principled control over the composition and size of the credal set

    We evaluate the proposed measures on selective prediction using Vision Transformer models as a state-of-the-art architecture for image classification combined with the relative likelihood credal set construction of Löhr et al. [2025], a recent methodology that offers principle...

    2025

  11. [19]

    Given a hypothesishand a dataset{(x i, yi)}N i=1, the likelihood is defined as L(h) = NY i=1 p(yi |x i, h).(8) Following Löhr et al

    E.1 Credal Set Construction via Relative Likelihood The central question behind credal set construction is to keep every model that fits the data sufficiently well, measured relative to the best available fit. Given a hypothesishand a dataset{(x i, yi)}N i=1, the likelihood is...

    2025

Pith tools

Reviewed July 15, 2026 · model on record in the stance chip above.