Pith. sign in

REVIEW 4 major objections 4 minor 81 references

A Unified Evaluation Framework for Epistemic Predictions

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single tunable metric can rank Bayesian, ensemble, and credal classifiers on the same scale.

desk verdict A promising unified metric whose load-bearing credal-set approximation is shown by the paper's own table to be too unstable to trust the rankings yet. read the letter →

arxiv 2501.16912 v2 pith:AR2HNTM5 submitted 2025-01-28 cs.LG

Shireen Kudukkil Manchingal , Muhammad Mubashar , Kaizheng Wang , Fabio Cuzzolin This is my paper · ORCID
classification cs.LG
keywords uncertaintyquantificationcredalsetsmodelselectionnon-specificityevaluationmetricepistemicbelieffunctionsKullback-Leiblerdivergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Uncertainty-aware classifiers report their predictions in incompatible formats: point probabilities, sample sets, distribution parameters, intervals, or belief functions. This paper claims that all of these can be converted into a common object — a convex set of probability vectors, or credal set — and then scored by one metric that blends distance to the truth with a measure of how vague the prediction is. The user chooses a trade-off parameter that reflects how much imprecision the application tolerates (whether abstention is allowed or a decision is mandatory), and the model with the lowest score is selected. If the claim holds, practitioners can compare and rank models across paradigms on consistent grounds rather than by test accuracy alone.

What carries the argument

The load-bearing object is the credal-set mapping: predictions are converted into coherent lower probabilities for each subset of classes (taking the minimum event probability over prediction samples), Möbius inversion turns those lower probabilities into a mass function (with negative masses set to zero), and a 2N-vertex approximation of the credal-set boundary makes it feasible to compute both the KL distance to the nearest vertex and the Dubois-Prade non-specificity in a single pass. The evaluation metric $E$ is the linear combination of these two terms with tunable trade-off parameter $\lambda$.

What would settle it

Compute the metric on a full 10-class dataset using all permutation vertices instead of the 2N approximation and check whether model rankings change materially; if they do, the approximation distorts the credal sets and the rankings are unreliable. A second check: inject a vacuous predictor that always outputs the full class set and verify whether it receives the score $\lambda \log|\mathcal{Y}|$ for every instance, and whether that score dominates the ranking as $\lambda$ grows.

Watch

Extended reading notes

Core claim

The paper's central claim is that the evaluation metric $E = d(y, \hat{y}) + \lambda \cdot NS[m]$ constitutes a unified performance measure for uncertainty-aware classifiers. For each input, any predicted probability vector, sample of vectors, interval, or belief function is mapped to a credal set via coherent lower probabilities, Möbius inversion, and a 2N-vertex approximation of the set's boundary. The distance term $d$ is the Kullback-Leibler divergence between the ground-truth label distribution and the nearest vertex of the credal set, measuring accuracy; the non-specificity term $NS[m] = \sum_{A} m(A) \log|A|$ measures how imprecise the prediction is. The parameter $\lambda$ lets the user weight the two facets. The paper shows that for point predictions non-specificity is zero and the metric reduces to ordinary KL divergence, and reports experiments on CIFAR-10, MNIST, and CIFAR-100 in which model rankings shift with $\lambda$ as intended — for example, Deep Ensembles rank first at low $\lambda$ while Random-Set Neural Networks rank first at high $\lambda$.

Load-bearing premise

The whole ranking is trustworthy only if the convex set of probability vectors built from a model's prediction samples — via lower probabilities, Möbius inversion, zeroed negative masses, and a reduced vertex set — truly captures what that model is uncertain about.

Editorial extensions

If this is right

  • Practitioners can select models for a given deployment context by choosing $\lambda$ according to whether abstention is allowed: low $\lambda$ favors accuracy-leaning models, high $\lambda$ favors precise and decisive models.
  • Point-estimate predictors such as standard neural networks, Bayesian model averaging, and deterministic networks are automatically included as a special case with zero non-specificity, so the framework subsumes accuracy-based selection.
  • Set-valued and belief-function models can be ranked against Bayesian, ensemble, and evidential models on the same scale, which standard accuracy measures cannot do.
  • The metric is sensitive to the number of prediction samples used to build the credal set; more samples enlarge the set and raise the score, so the sample budget affects model rankings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the credal-set construction is accepted, the same recipe could serve as a training objective, steering models toward calibrated set-valued predictions with a user-specified precision; the paper mentions this only as future work.
  • The 2N-vertex approximation is validated only on a 4-class subset, so the main transferable risk is whether rankings on full 10- or 100-class datasets are stable when the complete set of permutation vertices is used.
  • The framework could plausibly extend to regression or structured prediction by replacing the simplex with a distribution family and non-specificity with a generalized set-size measure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a unified evaluation framework for uncertainty-aware classifiers by mapping heterogeneous predictions (Bayesian, ensemble, evidential, deterministic, credal-set, belief-function) into credal sets on the probability simplex. The evaluation metric is E = d(y, ŷ) + λ·NS[m] (Eq. 6), where d is a distance (chosen as KL) from the ground truth to the nearest vertex of the predicted credal set, NS is the Dubois-Prade non-specificity of the associated mass function, and λ is a user-set trade-off parameter. The authors claim that for point predictions the metric reduces to standard KL divergence, and they demonstrate model rankings on CIFAR-10, MNIST, and CIFAR-100 across values of λ. The central intended contribution is a practical model-selection tool that lets practitioners choose a balance between accuracy and precision.

Significance. If the framework is sound, it addresses a real gap: there is no common scale on which Bayesian, evidential, imprecise-probabilistic, and deterministic classifiers can be compared for both accuracy and uncertainty quality. The paper includes several strengths: it works with a broad model zoo, the mathematical construction through coherent lower probabilities is well grounded, the nearest-vertex distance computation is exact for KL divergence with one-hot labels, and the authors provide code and extensive ablations. However, the significance is currently undercut by an insufficiently validated and potentially highly lossy approximation of credal-set vertices, as well as by validation arguments that partly restate the definition of the metric. The central claim of 'unified evaluation' cannot be fully assessed until the approximation's fidelity is established.

major comments (4)
  1. [§D, Tab. 4] The only empirical check of the 2N-vertex approximation is Tab. 4, and it shows that the approximation changes the evaluation metric by an order of magnitude for LB-BNN: for correct predictions E is 0.569 with the approximated vertices versus 0.039 with all 24 vertices, and for incorrect predictions the KL distance is 0.612 versus 0.336. These differences are comparable to the gaps between models in the CIFAR-10 rankings of Tab. 2, so the reported rankings may be artifacts of the vertex heuristic rather than of model quality. The paper gives no error bound, no convergence guarantee, and no comparison of the approximated credal set against the exact convex hull of the original prediction samples. Without this evidence, both terms in Eq. (6) may measure a different object from the model's actual prediction set.
  2. [§4] After the Möbius inversion (Eq. 4), the paper sets all negative masses to zero to 'ensure a coherent lower probability,' but it never analyzes how this zeroing alters the credal set. The resulting credal set can differ arbitrarily from the lower-probability set (3) and from the convex hull of the original samples, and this distortion propagates directly into both the KL term and the non-specificity term of Eq. (6). The paper should either prove that the zeroing preserves the relevant vertices and lower probabilities, or empirically quantify the distortion on real posterior samples.
  3. [§6.2, Fig. 2 (bottom right)] The ablation on the trade-off parameter λ is presented as evidence that the metric 'behaves as desired,' but it is largely self-referential. By the definition E = KL + λ·NS with NS ≥ 0, E is monotonically nondecreasing in λ for any fixed model with positive non-specificity, so the observed upward shifts and ranking changes follow directly from the construction. This does not validate that the metric captures a meaningful accuracy–precision trade-off. A more informative test would compare the metric's rankings against an independent decision-theoretic criterion, such as expected loss under abstention, or against external measures of epistemic uncertainty.
  4. [§4, §D] The lower probabilities are computed only for a 'fixed budget of most relevant subsets' via a clustering technique from Manchingal et al. (2025), but neither the main text nor Appendix D specifies how those subsets are selected, how the budget is set, or how the subset choice affects the Möbius inversion and the final metric. Because the entire framework is built on these lower probabilities, the paper should provide at least a sensitivity analysis with respect to the subset budget and ideally a reproducibility description of the clustering procedure.
minor comments (4)
  1. [§1] The sentence 'After of a review of related work' should read 'After a review of related work.'
  2. [Tab. 1, MNIST row for EDL] The KL entry '0 .00007±0.002' has unusual spacing and an over-precise mean; please clean up decimal formatting across tables.
  3. [§6.4, Tab. 1] The text explains that E-CNN is infeasible on CIFAR-100, but the table silently omits it; a footnote in the caption would clarify why the row is absent.
  4. [Algorithm 1] The loop says 'Compute the minimum KL divergence ... over the entire test set,' but the metric is per-data-point and then averaged; please clarify that the reported E is the mean over the test set.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity: the metric is a proposed composite, and the only definitional tautology is the claimed 'reduction' of E to KL for point predictions.

  1. self definitional [Sec. 5, Eq. (6)-(7), paragraph beginning 'Note that, for pointwise predictions...']
    "Note that, for pointwise predictions (including those generated by BMA and ensemble averaging) non-specificity goes to zero (as all non-singleton focal elements |A| > 1 have mass 0, indicating perfect precision) and the credal set collapses to a single point. Thus, for precise predictions, (6) reduces to the classical KL."

    The claimed proof that E reduces to KL is a direct substitution, not a derived theorem. Eq. (6) defines E = d + λ·NS; Eq. (7) defines NS[m] = Σ m(A) log|A|, which is identically 0 whenever the mass function has only singleton focal elements, as is the case for pointwise predictions. Setting d = KL then makes 'E reduces to KL' true by construction. The paper lists this as contribution (4) ('A proof that such metric reduces to the standard KL divergence for point-wise predictions'), but the reduction is equivalent to the definition of the metric and the choice of distance measure. It is a harmless tautology and does not by itself validate the framework.

full rationale

The paper's central object is a proposed composite metric, E = d(y,ŷ) + λ·NS[m] (Eq. 6), not a quantity derived from first principles, so the usual circularity pattern of fitting a parameter and then 'predicting' it does not arise. The credal-set construction uses standard lower probabilities, Möbius inversion, and the Chateauneuf-Jaffray vertex formula; the clustering heuristic for large label spaces is cited to the authors' own RS-NN work, but that is an externally published method citation and is not used as a uniqueness argument. The empirical rankings (Tabs. 1-2, 5-6) are computed from the metric and benchmarked against test accuracy and ECE, giving independent content; the Tab. 4 sensitivity of the 2N-vertex approximation is a robustness concern, not a circular step. The dependence of the rankings on λ (Sec. 6.2) is also a direct arithmetic consequence of the linear definition, but the particular ranks are data-dependent because they use measured KL and NS values, so this remains an illustration of the intended trade-off rather than a fitted 'prediction'. The only definitional tautology is contribution (4): the 'reduction' of E to KL for point predictions follows immediately from Eqs. (6)-(7) with NS=0. This is true by construction and adds no independent support, but it is not load-bearing for the framework's empirical claims. Overall circularity is therefore minor.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; the framework reuses credal sets, mass functions, lower probabilities, and non-specificity from prior literature. The main load-bearing assumptions are about the fidelity of the credal-set approximation and the suitability of the two components as accuracy and imprecision measures.

free parameters (3)
  • Trade-off parameter λ = not fitted, user-set
    E = d + λ·NS; rankings and model selections depend on λ (Sec. 6.2). It is a knob, not fit to data.
  • Number of prediction samples per input = 100 (LB-BNN, EDL), 15 (DE), 10 (CreINN)
    Chosen by hand; ablation (§F.5) shows E increases with sample count, so results depend on this choice.
  • Number of approximated credal-set vertices = 2N
    Approximation using two permutations per class (Sec. 4, Appendix D); validated only on a 4-class subset.
assumptions (5)
  • domain assumption The lower envelope of predicted probability samples, after Möbius inversion and zeroing negative masses, yields a coherent lower probability whose credal set faithfully represents the model's epistemic uncertainty.
    Invoked in Sec. 4 and Appendix D; negative masses are discarded and outer approximations are used, which can alter the credal set.
  • standard math The distance from a one-hot ground-truth vector to a credal set equals the distance to its closest vertex.
    Used in Sec. 5 'Distance computation' and Appendix A.2. For KL and one-hot labels this holds because minimizing -log p_y maximizes a linear function over a convex polytope; for other divergences like JS it may fail.
  • domain assumption Dubois-Prade non-specificity NS[m] is an appropriate measure of predictive imprecision/epistemic uncertainty for model selection.
    Adopted from prior work (Dubois and Prade 1987; Klir 1987), Sec. 5 and Appendix A.3; the paper argues it is a measure of epistemic uncertainty.
  • domain assumption Test accuracy and ECE serve as valid external references for judging whether the proposed metric's rankings are sensible.
    Used in Sec. 6.1, Tab. 1 to interpret results; these are standard but imperfect proxies for desired model behavior.
  • ad hoc to paper The 2N-vertex approximation captures the outermost vertices sufficiently well for the metric.
    Introduced in Sec. 4 and Appendix D; only validated on a 4-class CIFAR-10 subset (Tab. 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Unified Evaluation Framework for Epistemic Predictions." pith.science (2026). https://pith.science/paper/AR2HNTM5

@misc{pith2026250116912,
  author       = {Pith},
  title        = {Pith review of: A Unified Evaluation Framework for Epistemic Predictions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AR2HNTM5}},
  note         = {Machine review of arXiv:2501.16912}
}
read the original abstract

Predictions of uncertainty-aware models are diverse, ranging from single point estimates (often averaged over prediction samples) to predictive distributions, to set-valued or credal-set representations. We propose a novel unified evaluation framework for uncertainty-aware classifiers, applicable to a wide range of model classes, which allows users to tailor the trade-off between accuracy and precision of predictions via a suitably designed performance metric. This makes possible the selection of the most suitable model for a particular real-world application as a function of the desired trade-off. Our experiments, concerning Bayesian, ensemble, evidential, deterministic, credal and belief function classifiers on the CIFAR-10, MNIST and CIFAR-100 datasets, show that the metric behaves as desired.

Figures

Figures reproduced from arXiv: 2501.16912 by the authors.

Figure 1
Figure 1. Different types of uncertainty-aware model [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Measures of KL divergence (top left), Non-specificity (top right), Evaluation Metric (bottom left) for [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Visualizations of 100 prediction samples obtained prior to Bayesian Model Averaging and corresponding [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Visualizations of belief and mass predictions on the power-set space and its mapping to the label space [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Probability simplices illustrating the convex closure of predictions and credal sets for the Bayesian [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Comparison of (a) Kullback-Leibler (KL) divergence, and (b) Jensen-Shannon (JS) divergence for [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Comparison of mean Evaluation Metric E using mean Kullback Leibler (KL) divergence (top) and mean Jensen-Shannon (JS) divergence (bottom) for Correct (left) and Incorrect (right) predictions of the CIFAR-10 dataset [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Scatter plots showing the relationship between uncertainty (KL and JS divergences) and non-specificity [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Comparison of (a) Non-Specificity (NS), and (b) Credal Uncertainty (CU) for Correctly Classified (CC) [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: Comparison of mean Evaluation Metric E using mean Non-Specificity (top) and mean Credal Uncertainty (bottom) for Correct (left) and Incorrect (right) predictions of the CIFAR-10 dataset [PITH_FULL_IMAGE:figures/full_fig_p026_10.png]
Figure 11
Figure 11. Figure 11: Measures of (a) Kullback-Leibler (KL) divergence, (b) Non-specificity (NS), (c) Evaluation Metric ( [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Measures of (a) Kullback-Leibler (KL) divergence, (b) Non-specificity (NS), (c) Evaluation Metric ( [PITH_FULL_IMAGE:figures/full_fig_p029_12.png]
Figure 13
Figure 13. Figure 13: Ablation study on the number of prediction samples of LB-BNN and the number of ensembles of DE [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: Credal set sizes for all models for 50 prediction samples of the CIFAR-10 dataset. Larger credal set [PITH_FULL_IMAGE:figures/full_fig_p032_14.png]
Figure 15
Figure 15. Figure 15: Credal set sizes for all models for 50 prediction samples of the MNIST dataset. Larger credal set sizes [PITH_FULL_IMAGE:figures/full_fig_p032_15.png]
Figure 16
Figure 16. Figure 16: Credal set sizes for all models for 50 prediction samples of the CIFAR-100 dataset. Larger credal set [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]
Figure 17
Figure 17. Figure 17: Entropy vs Confidence (top) and Non-Specificity vs Confidence (bottom) for each of the models: (a) [PITH_FULL_IMAGE:figures/full_fig_p033_17.png]
Figure 18
Figure 18. Figure 18: Credal Set Size vs. Non-Specificity heatmap for (a) LB-BNN, (b) Deep Ensembles (DE), (c) EDL, (d) [PITH_FULL_IMAGE:figures/full_fig_p034_18.png]
Figure 19
Figure 19. Figure 19: Credal Set Size vs. Non-Specificity heatmap for (a) LB-BNN, (b) Deep Ensembles (DE), (c) EDL, (d) [PITH_FULL_IMAGE:figures/full_fig_p035_19.png]
Figure 20
Figure 20. Figure 20: Credal Set Size vs. Non-Specificity heatmap for (a) LB-BNN, (b) Deep Ensembles (DE), (c) EDL, (d) [PITH_FULL_IMAGE:figures/full_fig_p035_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 66 canonical work pages

  1. [1]

    and Moral, S

    Abell \'a n, J. and Moral, S. (2000). A non-specificity measure for convex sets of probability distributions. International journal of uncertainty, fuzziness and knowledge-based systems , 8(03):357--367

    2000

  2. [2]

    and Moral, S

    Abell \'a n, J. and Moral, S. (2005). Difference of entropies as a non-specificity function on credal sets. International journal of general systems , 34(3):201--214

    work page 2005

  3. [3]

    and Cuzzolin, F

    Antonucci, A. and Cuzzolin, F. (2010a). Credal sets approximation by lower probabilities: A pplication to credal networks. In Hüllermeier, E., Kruse, R., and Hoffmann, F., editors, Computational Intelligence for Knowledge-Based Systems Design , volume 6178 of Lecture Notes in Computer Science , pages 716--725. Springer, Berlin Heidelberg

    work page 2010

  4. [4]

    and Cuzzolin, F

    Antonucci, A. and Cuzzolin, F. (2010b). Credal sets approximation by lower probabilities: application to credal networks. In Computational Intelligence for Knowledge-Based Systems Design: 13th International Conference on Information Processing and Management of Uncertainty, IPMU 2010, Dortmund, Germany, June 28-July 2, 2010. Proceedings 13 , pages 716--72...

    work page 2010

  5. [5]

    Bernard, J.-M. (2005). An introduction to the imprecise dirichlet model for multinomial data. International Journal of Approximate Reasoning , 39(2-3):123--150

    work page 2005

  6. [6]

    and Klir, G

    Bronevich, A. and Klir, G. J. (2008). Axioms for uncertainty measures on belief functions and credal sets. In NAFIPS 2008-2008 Annual Meeting of the North American Fuzzy Information Processing Society , pages 1--6. IEEE

    work page 2008

  7. [7]

    and Weigend, A

    Buntine, W. and Weigend, A. (1991). Bayesian back-propagation. Technical Report FIA -91-22

    work page 1991

  8. [9]

    J., Lin, V., Ivanov, R., Sokolsky, O., and Lee, I

    Caprio, M., Dutta, S., Jang, K. J., Lin, V., Ivanov, R., Sokolsky, O., and Lee, I. (2023b). Imprecise Bayesian neural networks. arXiv preprint arXiv:2302.09656

    arXiv 2023

Show all 81 references
  1. [10]

    Caprio, M., Sultana, M., Elia, E., and Cuzzolin, F. (2024). Credal learning theory. arXiv preprint arXiv:2402.00957

    2024 arXiv

  2. [11]

    and Jaffray, J.-Y

    Chateauneuf, A. and Jaffray, J.-Y. (1989). Some characterizations of lower probabilities and other monotone capacities through the use of Möbius inversion . Mathematical Social Sciences , 17(3):263--283

    1989

  3. [12]

    Cozman, F. G. (2000). Credal networks. Artificial intelligence , 120(2):199--233

    2000

  4. [13]

    Cuzzolin, F. (2008a). On the credal structure of consistent probabilities. In European Workshop on Logics in Artificial Intelligence , pages 126--139. Springer

    2008

  5. [14]

    Cuzzolin, F. (2008b). On the credal structure of consistent probabilities. In Hölldobler, S., Lutz, C., and Wansing, H., editors, Logics in Artificial Intelligence , volume 5293 of Lecture Notes in Computer Science , pages 126--139. Springer, Berlin Heidelberg

    2008

  6. [15]

    Cuzzolin, F. (2010). Credal semantics of Bayesian transformations in terms of probability intervals . IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics , 40(2):421--432

    2010

  7. [16]

    M., Huete, J

    De Campos, L. M., Huete, J. F., and Moral, S. (1994). Probability intervals: A tool for uncertain reasoning. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems , 02(02):167--196

    1994

  8. [17]

    and Grabisch, M

    Denneberg, D. and Grabisch, M. (1999). Interaction transform of set functions over a finite set. Information Sciences , 121(1-2):149--170

    1999

  9. [18]

    Droguett, E. L. and Mosleh, A. (2008). Bayesian methodology for model uncertainty using model performance data. Risk Analysis: An International Journal , 28(5):1457--1476

    2008

  10. [19]

    and Prade, H

    Dubois, D. and Prade, H. (1987). Properties of measures of information in evidence and possibility theories. Fuzzy sets and systems , 24(2):161--182

    1987

  11. [20]

    Dubois, D., Prade, H., and Sandri, S. (1993). On possibility/probability transformations. In Fuzzy logic: State of the art , pages 103--112. Springer

    1993

  12. [21]

    and Fookes, C

    Goan, E. and Fookes, C. (2020). Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018 , pages 45--87

    2020

  13. [22]

    Graefe, A., K \"u chenhoff, H., Stierle, V., and Riedl, B. (2015). Limitations of ensemble Bayesian model averaging for forecasting social science problems. International Journal of Forecasting , 31(3):943--951

    2015

  14. [23]

    Hartley, R. V. (1928). Transmission of information 1. Bell System technical journal , 7(3):535--563

    1928

  15. [24]

    Hastings, W. K. (1970). Monte carlo sampling methods using markov chains and their applications

    1970

  16. [25]

    F., van den Bergh, D., and Wagenmakers, E.-J

    Hinne, M., Gronau, Q. F., van den Bergh, D., and Wagenmakers, E.-J. (2020). A conceptual introduction to Bayesian model averaging. Advances in Methods and Practices in Psychological Science , 3(2):200--215

    2020

  17. [26]

    Hobbhahn, M., Kristiadi, A., and Hennig, P. (2022). Fast predictive uncertainty for classification with Bayesian deep networks. In Uncertainty in Artificial Intelligence , pages 822--832. PMLR

    2022

  18. [27]

    Hoel, C.-J., Wolff, K., and Laine, L. (2023). Ensemble quantile networks: Uncertainty-aware reinforcement learning with applications in autonomous driving. IEEE Transactions on Intelligent Transportation Systems

    2023

  19. [28]

    H., Zang, A

    Huang, A. H., Zang, A. Y., and Zheng, R. (2014). Evidence on the information content of text in analyst reports. The Accounting Review , 89(6):2151--2180

    2014

  20. [29]

    H \"u llermeier, E., Destercke, S., and Shaker, M. H. (2022). Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison. In Proceedings of the Uncertainty in Artificial Intelligence , pages 548--557. PMLR

    2022

  21. [30]

    and Waegeman, W

    H \" u llermeier, E. and Waegeman, W. (2019). Aleatoric and epistemic uncertainty in machine learning: A tutorial introduction. CoRR , abs/1910.09457

    2019 arXiv

  22. [31]

    and Waegeman, W

    H \"u llermeier, E. and Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning , 110(3):457--506

    2021

  23. [32]

    V., Laga, H., Boussaid, F., Buntine, W., and Bennamoun, M

    Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., and Bennamoun, M. (2022). Hands-on bayesian neural networks—a tutorial for deep learning users. IEEE Computational Intelligence Magazine , 17(2):29--48

    2022

  24. [33]

    and Gal, Y

    Kendall, A. and Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? arXiv:1703.04977

    2017 arXiv

  25. [34]

    Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  26. [35]

    Kingma, D. P. and Welling, M. (2013). Auto-encoding Variational Bayes . arXiv preprint arXiv:1312.6114

    2013 arXiv

  27. [36]

    Klir, G. J. (1987). Where do we stand on measures of uncertainty, ambiguity, fuzziness, and the like? Fuzzy sets and systems , 24(2):141--160

    1987

  28. [37]

    Kolmogorov, A. N. (1965). Three approaches to the quantitative definition ofinformation’. Problems of information transmission , 1(1):1--7

    1965

  29. [38]

    o rner, R. and N \

    K \"o rner, R. and N \"a ther, W. (1995). On the specificity of evidences. Fuzzy sets and systems , 71(2):183--196

    1995

  30. [39]

    Kramosil, I. (1999). Nonspecificity degrees of basic probability assignments in D empster-- S hafer theory. Computing and Informatics , 18(6):559--574

    1999

  31. [40]

    Krizhevsky, A. (2012). Learning multiple layers of features from tiny images. University of Toronto

    2012

  32. [41]

    Krizhevsky, A., Nair, V., and Hinton, G. (2009). CIFAR-10 (Canadian Institute For Advanced Research) . Technical report, CIFAR

    2009

  33. [42]

    Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Pereira, F., Burges, C., Bottou, L., and Weinberger, K., editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc

    2012

  34. [43]

    Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles . Advances in Neural Information Processing Systems , 30

    2017

  35. [44]

    and Vehtari, A

    Lampinen, J. and Vehtari, A. (2001). Bayesian approach for neural networks—review and case studies. Neural networks , 14(3):257--274

    2001

  36. [45]

    and Cortes, C

    LeCun, Y. and Cortes, C. (2005). The MNIST database of handwritten digits. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1--9

    2005

  37. [46]

    Levi, I. (1980a). The enterprise of knowledge: An essay on knowledge, credal probability, and chance . MIT press

    1980

  38. [47]

    Levi, I. (1980b). The enterprise of knowledge: An essay on knowledge, credal probability, and chance . The MIT Press, Cambridge, Massachusetts

    1980

  39. [48]

    Manchingal, S. K. and Cuzzolin, F. (2022). Epistemic deep learning. arXiv preprint arXiv:2206.07609

    2022 arXiv

  40. [49]

    K., Mubashar, M., Wang, K., Shariatmadar, K., and Cuzzolin, F

    Manchingal, S. K., Mubashar, M., Wang, K., Shariatmadar, K., and Cuzzolin, F. (2025). Random-set neural networks. In The Thirteenth International Conference on Learning Representations

    2025

  41. [50]

    Marcot, B. G. (2012). Metrics for evaluating performance and uncertainty of bayesian network models. Ecological Modelling , 230:50--62

    2012

  42. [51]

    L., Pardo, J., Pardo, L., and Pardo, M

    Men \'e ndez, M. L., Pardo, J., Pardo, L., and Pardo, M. (1997). The jensen-shannon divergence. Journal of the Franklin Institute , 334(2):307--318

    1997

  43. [52]

    Miranda, E. (2008). A survey of the theory of coherent lower previsions. International Journal of Approximate Reasoning , 48(2):628--658

    2008

  44. [53]

    Miranda, E., Montes, I., and Presa, A. (2023). Inner approximations of coherent lower probabilities and their application to decision making problems. Annals of Operations Research , pages 1--39

    2023

  45. [54]

    Miranda, E., Montes, I., and Vicig, P. (2021). On the selection of an optimal outer approximation of a coherent lower probability. Fuzzy Sets and Systems , 424:1--36

    2021

  46. [55]

    H., and Gal, Y

    Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H., and Gal, Y. (2021). Deep deterministic uncertainty: A simple baseline. arXiv preprint arXiv:2102.11582

    2021 arXiv

  47. [56]

    Neal, R. M. (2012). Bayesian learning for neural networks , volume 118. Springer Science & Business Media

    2012

  48. [57]

    R., Bezdek, J

    Pal, N. R., Bezdek, J. C., and Hemasinha, R. (1993). Uncertainty measures for evidential reasoning ii: A new measure of total uncertainty. International Journal of Approximate Reasoning , 8(1):1--16

    1993

  49. [58]

    and Grandhi, R

    Park, I. and Grandhi, R. V. (2011). Quantifying multiple types of uncertainty in physics-based simulation using bayesian model averaging. AIAA journal , 49(5):1038--1045

    2011

  50. [59]

    Pericchi, L. R. and Walley, P. (1991). Robust bayesian credible intervals and prior ignorance. International Statistical Review/Revue Internationale de Statistique , pages 1--23

    1991

  51. [60]

    Sensoy, M., Kaplan, L., and Kandemir, M. (2018). Evidential deep learning to quantify classification uncertainty. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 3183–3193, Red Hook, NY, USA. Curran Associates Inc

    2018

  52. [61]

    Shafer, G. (1976). A mathematical theory of evidence , volume 42. Princeton university press

    1976

  53. [62]

    Shlens, J. (2014). Notes on kullback-leibler divergence and likelihood. arXiv preprint arXiv:1404.2000

    2014 arXiv

  54. [63]

    Smarandache, F., Martin, A., and Osswald, C. (2011). Contradiction measures and specificity degrees of basic belief assignments. In 14th International Conference on Information Fusion , pages 1--8. IEEE

    2011

  55. [64]

    Smets, P. (1983). Information content of an evidence. International Journal of Man-Machine Studies , 19(1):33--43

    1983

  56. [65]

    Smets, P. (2005). Decision making in the TBM : the necessity of the pignistic transformation. International Journal of Approximate Reasoning , 38(2):133--147

    2005

  57. [66]

    and Kramer, J

    Snowling, S. and Kramer, J. (2001). Evaluating modelling uncertainty for model selection. Ecological Modelling , 138(1):17--30

    2001

  58. [67]

    Song, Y., Wang, X., Wu, W., Quan, W., and Huang, W. (2018). Evidence combination based on credibility and non-specificity. Pattern Analysis and Applications , 21:167--180

    2018

  59. [68]

    Thogmartin, W. E. (2010). Sensitivity analysis of north american bird population estimates. Ecological Modelling , 221(2):173--177

    2010

  60. [69]

    Titterington, D. M. (2004). Bayesian methods for neural networks and related models. Statistical science , pages 128--139

    2004

  61. [70]

    Tong, Z., Xu, P., and Denoeux, T. (2021). An evidential classifier based on Dempster-Shafer theory and deep learning. Neurocomputing , 450:275--293

    2021

  62. [71]

    Troffaes, M. C. (2007). Decision making under uncertainty using imprecise probabilities. International journal of approximate reasoning , 45(1):17--29

    2007

  63. [72]

    Ullah, A. (1996). Entropy, divergence and distance measures with econometric applications. Journal of Statistical Planning and Inference , 49(1):137--162

    1996

  64. [73]

    Vega, M. A. and Todd, M. D. (2022). A variational Bayesian neural network for structural health monitoring and cost-informed decision-making in miter gates . Structural Health Monitoring , 21(1):4--18

    2022

  65. [74]

    E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al

    Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020). Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods , 17(3):261--272

    2020

  66. [75]

    Walley, P. (1991). Statistical reasoning with imprecise probabilities

    1991

  67. [76]

    J., \"U lk \"u men, G., Tannenbaum, D., Erner, C., and Fox, C

    Walters, D. J., \"U lk \"u men, G., Tannenbaum, D., Erner, C., and Fox, C. R. (2023). Investor behavior under epistemic vs. aleatory uncertainty. Management Science , 69(5):2761--2777

    2023

  68. [77]

    Wang, K., Cuzzolin, F., Shariatmadar, K., Moens, D., and Hallez, H. (2024a). Credal wrapper of model averaging for uncertainty estimation on out-of-distribution detection. arXiv preprint arXiv:2405.15047

    2024 arXiv

  69. [78]

    K., Cuzzolin, F., Moens, D., and Hallez, H

    Wang, K., Shariatmadar, K., Manchingal, S. K., Cuzzolin, F., Moens, D., and Hallez, H. (2024b). Creinns: Credal-set interval neural networks for uncertainty estimation in classification tasks. arXiv preprint arXiv:2401.05043

    2024 arXiv

  70. [79]

    Yager, R. R. (2008). Entropy and specificity in a mathematical theory of evidence. Classic works of the Dempster-Shafer theory of belief functions , pages 291--310

    2008

  71. [80]

    Zaffalon, M. (2002). The Naive Credal Classifier . Journal of Statistical Planning and Inference - J STATIST PLAN INFER , 105:5--21

    2002

  72. [81]

    and Fagiuoli, E

    Zaffalon, M. and Fagiuoli, E. (2003). Tree-based credal networks for classification. Reliable computing , 9(6):487--509

    2003

  73. [82]

    and Apostolakis, G

    Zio, E. and Apostolakis, G. (1996). Two methods for the structured assessment of model uncertainty by experts in performance assessments of radioactive waste repositories. Reliability Engineering & System Safety , 54(2-3):225--241

    1996

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.