Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Uncertainty Quantification with Proper Scoring Rules: Adjusting Measures to Prediction Tasks

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes that in selective prediction, the expected area under the loss-rejection curve is minimized by ordering test instances by total uncertainty, with the uncertainty scoring rule matched to the task loss.

desk verdict Useful task-aligned UQ framework, but the central selective-prediction theorem has a sign error that flips the ordering; the practical advice survives under the standard loss-rejection convention, but the paper as written contradicts itself. read the letter →

arxiv 2505.22538 v1 pith:COXKT6I2 submitted 2025-05-28 cs.LG stat.ML

Paul Hofman , Yusuf Sale , Eyke Hüllermeier This is my paper
classification cs.LGstat.ML
keywords properscoringrulesuncertaintyquantificationaleatoricepistemicselectivepredictionactivelearningout-of-distributiondetectiontotal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Uncertainty quantification is usually treated as a one-size-fits-all toolbox, but this paper argues the measure should be tailored to the downstream task. It builds total, aleatoric, and epistemic uncertainty from the decomposition of a proper scoring rule into an entropy and a divergence term, yielding a family of measures parameterized by the loss. The main theoretical result is for selective prediction: to minimize the expected area under the loss-rejection curve, instances should be ranked by total uncertainty, and the loss used to define that uncertainty should equal the task loss. Empirically, the log-loss-based mutual information wins at out-of-distribution detection while a newly proposed zero-one-loss epistemic measure wins at active learning, showing there is no universal best measure.

What carries the argument

The decomposition of a (strictly) proper scoring rule $\ell$ into $L_{\ell}(\theta,\theta)$ (the entropy, identified with aleatoric uncertainty) and $D_{\ell}(\hat{\theta},\theta)$ (the divergence, identified with epistemic uncertainty), applied to a second-order distribution $Q$ so that total, aleatoric, and epistemic uncertainty are expectations of these quantities. The proof of Proposition 4.1 uses the rearrangement inequality on the weighted sum that approximates the expected area under the loss-rejection curve.

What would settle it

Use a deliberately miscalibrated ensemble (e.g., force all member predictions toward one class) on a selective-prediction task and check whether ordering by total uncertainty with a matched loss still minimizes the expected AULC; if a mismatched loss performs better, the alignment claim fails under misspecification.

Watch

Extended reading notes

Core claim

The paper's central claim is that the loss underlying an uncertainty measure should be aligned with the loss of the downstream task. For selective prediction this is made precise: Proposition 4.1 shows the expected area under the loss-rejection curve is minimized by ordering instances in non-decreasing order of their expected instance-wise task loss, and when the prediction is the Bayesian model average this quantity is exactly the total uncertainty instantiated with the same loss. Consequently, the total uncertainty component, rather than the aleatoric or epistemic component, is the correct rejection criterion, and the uncertainty loss should be the auxiliary task loss. The same construction reproduces Shannon-entropy-based measures and introduces a zero-one-loss epistemic measure that consistently outperforms others in active learning experiments.

Load-bearing premise

The framework assumes the estimated second-order distribution $Q$ faithfully represents the learner's epistemic state, and that the task loss is instancewise decomposable as a scoring rule on the simplex; if either fails, the optimal-ordering result and its guidance may not hold.

Editorial extensions

If this is right

  • Selective prediction should reject instances by total uncertainty, with the uncertainty loss set equal to the task loss; aleatoric and epistemic components are not the right rejection criteria.
  • Loss-rejection curves used to evaluate aleatoric and epistemic measures are assessing them on a task they are not tailored to.
  • The framework provides a family of uncertainty measures, so the best measure depends on the task and benchmarks should report task-specific comparisons.
  • The zero-one-loss epistemic measure gives a new, effective query criterion for active learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The optimal-ordering result assumes the task loss is instancewise decomposable as a scoring rule; for non-decomposable losses such as set-level or ranking losses, the matching prescription does not directly apply and would need separate treatment.
  • A testable extension is to check sensitivity to miscalibrated second-order distributions: if $Q$ is overconfident, matching losses may no longer yield optimal AULC, suggesting calibration should be part of the framework.
  • The zero-one-loss epistemic measure could also inform cost-sensitive classification, where the cost of misclassifying each class differs; this is not explored in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a family of uncertainty measures for classification, derived from the well-known decomposition of proper scoring rules into an entropy term (aleatoric) and a divergence term (epistemic). The construction is parameterized by an uncertainty loss, yielding log-, Brier-, zero-one-, and spherical-based total/aleatoric/epistemic measures. The authors analyze selective prediction and claim (Proposition 4.1 and Section 4.2) that the expected area under the loss-rejection curve is minimized by ordering instances by decreasing total uncertainty, with the uncertainty loss chosen equal to the task loss. They support this with experiments on CoverType and Poker Hand. They also report out-of-distribution detection experiments favoring log-based epistemic uncertainty (mutual information) and active learning experiments favoring zero-one-based epistemic uncertainty. Code is provided.

Significance. If the selective-prediction theorem is corrected, the paper makes a useful conceptual contribution: it unifies several existing UQ measures under one construction principle and provides a principled argument for task-specific measure choice. The empirical findings for OoD detection are consistent with prior work, and the active-learning result for zero-one epistemic uncertainty is a potentially interesting new observation. The paper is clearly written and the derivations in Appendix A are correct. However, the central theorem as stated is internally inconsistent, and the empirical support rests on only three runs without significance tests. The framework itself is not circular; it is a formal decomposition of known scoring rules. I see the main claim as defensible after a correction, but the current manuscript requires major revision.

major comments (2)
  1. [Section 4.2, Eq. (7), Appendix B] There is a sign inconsistency that is load-bearing for the paper's central contribution. Eq. (7) defines AULC as the integral, over rejection level α, of the average loss of the first floor(αn) instances in the permutation π, which is sorted by decreasing uncertainty U. In the selective-prediction setting, those first instances are the rejected ones. For this objective, the proof in Appendix B derives S(π) = sum_j w_j c_{π(j)} with weights w_j = sum_{k=j}^n 1/k, which are strictly decreasing in j. The rearrangement inequality then shows S(π) is minimized when c_{π(1)} <= ... <= c_{π(n)}, i.e., when the early (rejected) positions have the lowest expected loss. That is the opposite of the paper's recommendation to reject high-total-uncertainty instances and contradicts the conclusion in Section 4.2 that ordering by total uncertainty with uncertainty loss equal to the task loss minimizes expected AULC. If the intended 'loss-rejection curve' is the standard one that plots average loss on accepted instances, then Eq. (7) should sum over j = floor(αn)+1, ..., n; the weights then become increasing in j, and the optimal ordering is c_{π(1)} >= ... >= c_{π(n)}, which would support the paper's recommendation. The statement of Proposition 4.1 ('non-decreasing order') also needs to be reversed under either fix. This must be corrected before the paper can be accepted.
  2. [Tables 3, 7, 8 and Figures 1, 2, 7] The empirical support for the headline claims is statistically thin. All reported numbers are means over three runs, several entries show standard deviations of 0.000 (e.g., Table 3, Dropout on CIFAR-100 for log and Brier), and no significance tests or effect sizes are provided. With only three runs, statements such as 'consistently outperforms' in the abstract and in Section 5.3 are not established. The authors should either run more seeds, provide paired significance tests (e.g., across the final performance values or across the full learning curves), or temper the wording to 'tends to outperform in these experiments'.
minor comments (5)
  1. [Table 2] The zero-one epistemic uncertainty formula has an extra closing bracket: 'E_{θ∼Q}[max_k θ_k − θ_{arg max_k θ̄_k}]]' should be 'E_{θ∼Q}[max_k θ_k − θ_{arg max_k θ̄_k}]'.
  2. [Section C.5] The sentence 'all tasks are run three times and for each run a new model is trained (only exception being pre-trained Dropout and Laplace models)' is unclear: if the model is pre-trained, what varies across the three runs? Please clarify whether only the test subset changes or whether some retraining occurs.
  3. [Table 3] Entries with standard deviation 0.000 should either be reported with more significant digits or explained; with three runs a exactly-zero standard deviation is suspicious and may be a rounding artifact.
  4. [Figures 1 and 2] The shaded areas show standard deviations over three runs, but with so few runs it is hard to assess whether differences between methods are meaningful. Plotting individual runs or confidence intervals would help.
  5. [Section 5.1] The statement that for two-class distributions the total uncertainty measures induce the same ordering is stated without proof; a one-sentence justification (all three measures are monotone functions of the distance of the predictive probability from 0.5) would improve readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the selective-prediction result is a rearrangement-inequality derivation and the empirical benchmarks are self-contained; only minor non-load-bearing self-citations appear.

full rationale

The core derivation in Section 3.2 is an algebraic consequence of the proper-scoring-rule decomposition (Eq. 5), not a fitted or circular claim: total, aleatoric, and epistemic uncertainty are defined as expectations of the loss ℓ, and the closed forms in Table 2 follow by direct computation. The selective-prediction claim rests on Proposition 4.1, which is a rearrangement-inequality argument from the AULC definition (Eq. 7); once the AULC is defined with an auxiliary loss ℓ*, the proposition says to order by expected ℓ*-loss, and total uncertainty with uncertainty loss ℓ* is the Q-expectation of that quantity. This is definitionally close but not circular, because the proposition applies to any ordering and the empirical comparison of different uncertainty losses is nontrivial. The paper cites several works by its own authors ([47], [48], [54]), but these support secondary remarks (e.g., desirable properties of a Brier-based measure, criticism of entropy measures) rather than the central theorem; the central derivation and the external benchmarks stand independently. One internal issue, which is a correctness concern rather than circularity, is that Proposition 4.1's proof concludes the expected AULC is minimized by non-decreasing expected loss while the paper recommends decreasing total uncertainty; as written this is a sign inconsistency (possibly a typo in Eq. (7)), but it does not make the result equivalent to its inputs by construction. Overall, no significant circularity; score 2 reflects only minor non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The framework introduces no fitted parameters or invented entities. It depends on the Bayesian representation of uncertainty and on the instancewise-loss assumption for the selective prediction result.

assumptions (4)
  • domain assumption The Bayesian posterior q(h|D) over hypotheses is available or approximated, giving a second-order distribution Q on the simplex.
    Section 2: all uncertainty measures are defined relative to Q; without a faithful Q the measures lack meaning.
  • domain assumption The task loss in selective prediction is instancewise decomposable and the expected AULC is the objective.
    Proposition 4.1 and Eq. (7) restrict the task loss to L(ΔK,Y); complex losses not expressible in this form are excluded.
  • standard math Any strictly proper scoring rule decomposes into an entropy and a divergence component, with entropy representing aleatoric and divergence epistemic uncertainty.
    Sections 3.1 and 3.2, based on Gneiting and Raftery [20] and Kull and Flach [32]; this is the foundation of the framework.
  • standard math The rearrangement inequality is applicable in the proof of Proposition 4.1.
    Section B: the proof uses the rearrangement inequality after approximating the AULC integral by a Riemann sum with sorted weights.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty Quantification with Proper Scoring Rules: Adjusting Measures to Prediction Tasks." pith.science (2026). https://pith.science/paper/COXKT6I2

@misc{pith2026250522538,
  author       = {Pith},
  title        = {Pith review of: Uncertainty Quantification with Proper Scoring Rules: Adjusting Measures to Prediction Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/COXKT6I2}},
  note         = {Machine review of arXiv:2505.22538}
}
read the original abstract

We address the problem of uncertainty quantification and propose measures of total, aleatoric, and epistemic uncertainty based on a known decomposition of (strictly) proper scoring rules, a specific type of loss function, into a divergence and an entropy component. This leads to a flexible framework for uncertainty quantification that can be instantiated with different losses (scoring rules), which makes it possible to tailor uncertainty quantification to the use case at hand. We show that this flexibility is indeed advantageous. In particular, we analyze the task of selective prediction and show that the scoring rule should ideally match the task loss. In addition, we perform experiments on two other common tasks. For out-of-distribution detection, our results confirm that a widely used measure of epistemic uncertainty, mutual information, performs best. Moreover, in the setting of active learning, our measure of epistemic uncertainty based on the zero-one-loss consistently outperforms other uncertainty measures.

Figures

Figures reproduced from arXiv: 2505.22538 by the authors.

Figure 1
Figure 1. Selective Prediction with different task losses using the total uncertainty component as the rejection criterion. The line shows the mean and the shaded area represents the standard deviation over three runs. Proposition 4.1. Let θˆ ∈ ∆K be a (first-order) prediction and ℓ ∈ L(∆K, Y) . Then the expected AULC is minimized by ordering test instances in non-decreasing order of their (instance-wise) expected loss Ey∼θ … view at source ↗
Figure 2
Figure 2. Active Learning with different datasets using the epistemic uncertainty component to query new instances. The model is evaluated using the zero-one-loss on the test instances. The line shows the mean and the shaded area represents the standard deviation over three runs. are the ones for which the model has (epistemic) uncertainty regarding the ground-truth label, as this information is only revealed upon sampling an… view at source ↗
Figure 3
Figure 3. Selective Prediction with different task losses using the aleatoric uncertainty (top row) and epistemic uncertainty (bottom row) component as the rejection criterion. The line shows the mean and the shaded area represents the standard deviation over three runs. We also present selective prediction results for the Poker Hand dataset with total, aleatoric, and epistemic in [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Selective Prediction with different task losses using the total uncertainty (top row), aleatoric uncertainty (middle row) and epistemic uncertainty (bottom row) component as the rejection criterion. The line shows the mean and the shaded area represents the standard de…
Figure 5
Figure 5. Figure 5: Selective Prediction with different task losses using the aleatoric uncertainty (top row) and epistemic uncertainty (bottom row) component as the rejection criterion. The line shows the mean and the shaded area represents the standard deviation over three runs. 0 20 40…
Figure 6
Figure 6. Figure 6: Selective Prediction with different task losses using the total uncertainty component as the rejection criterion. The line shows the mean and the shaded area represents the standard deviation over three runs [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: Active Learning with different datasets using the epistemic uncertainty component to query new instances. The model is evaluated using the zero-one-loss on the test instances. The line shows the mean and the shaded area represents the standard deviation over three runs…
Figure 8
Figure 8. Figure 8: Active Learning with different datasets using the epistemic uncertainty component to query new instances. The model is evaluated using the zero-one-loss on the test instances. The line shows the mean and the shaded area represents the standard deviation over three runs…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Combining LoRA with snapshot ensembling yields a parameter-efficient uncertainty-aware segmentation ensemble that matches snapshot full-rank baselines, with feed-forward layers identified as the critical LoRA target.

Reference graph

Works this paper leans on

57 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [30]

    Kotelevskii, V

    N. Kotelevskii, V . Kondratyev, M. Takáˇc, E. Moulines, and M. Panov. From risk to uncer- tainty: Generating predictive uncertainty measures via bayesian estimation. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  2. [1]

    Amini, W

    A. Amini, W. Schwarting, A. Soleimany, and D. Rus. Deep evidential regression. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors,Advances in Neu- ral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020

    work page 2020

  3. [2]

    Banerjee, X

    A. Banerjee, X. Guo, and H. Wang. Optimal bregman prediction and jensen’s equality. In Proceedings of the 2004 IEEE International Symposium on Information Theory, ISIT 2004, Chicago Downtown Marriott, Chicago, Illinois, USA, June 27 - July 2, 2004, page 169. IEEE, 2004

    work page 2004

  4. [3]

    Bengs, E

    V . Bengs, E. Hüllermeier, and W. Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  5. [4]

    Blackard

    J. Blackard. Covertype. UCI Machine Learning Repository, 1998. https://doi.org/10. 24432/C50K5N

    work page 1998

  6. [5]

    Blundell, J

    C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In F. R. Bach and D. M. Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 ofJMLR Workshop and Conference Proceedings, pages 1613–1622. JMLR.org, 2015

    work page 2015

  7. [6]

    Bossard, M

    L. Bossard, M. Guillaumin, and L. V . Gool. Food-101 - mining discriminative components with random forests. In D. J. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors,Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI, volume 8694 ofLecture Notes in Computer Science, pages 446–46...

    work page 2014

  8. [7]

    L. Bregman. The relaxation method of finding the common point of convex sets and its applica- tion to the solution of problems in convex programming.USSR Computational Mathematics and Mathematical Physics, 7(3):200–217, 1967. ISSN 0041-5553

    work page 1967

Show all 57 references
  1. [8]

    G. W. Brier. Verification of forecasts expressed in terms of probability.Monthly Weather Review, 78(1):1–3, Jan. 1950. ISSN 1520-0493

    1950

  2. [9]

    Cattral and F

    R. Cattral and F. Oppacher. Poker hand. UCI Machine Learning Repository, dec 2006. https://doi.org/10.24432/C5KW38

    2006 doi

  3. [10]

    Clarté, B

    L. Clarté, B. Loureiro, F. Krzakala, and L. Zdeborová. Expectation consistency for calibration of neural networks. In R. J. Evans and I. Shpitser, editors,Uncertainty in Artificial Intelligence, UAI 2023, July 31 - 4 August 2023, Pittsburgh, PA, USA, volume 216 ofProceedings o...

    2023

  4. [11]

    T. M. Cover and J. A. Thomas.Elements of information theory (2. ed.). Wiley, 2006. ISBN 978-0-471-24195-9. 10

    2006

  5. [12]

    Daxberger, A

    E. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig. Laplace redux - effortless bayesian deep learning. In M. Ranzato, A. Beygelzimer, Y . N. Dauphin, P. Liang, and J. W. Vaughan, editors,Advances in Neural Information Processing Systems 34: Annual Co...

    2021

  6. [13]

    I. P. de Jong, A. I. Sburlea, and M. Valdenegro-Toro. How disentangled are your classification uncertainties?CoRR, abs/2408.12175, 2024

    2024

  7. [14]

    J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. IEEE Computer So...

    2009

  8. [15]

    Depeweg, J

    S. Depeweg, J. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. InProc. ICML, Stockholm, Sweden, 2018

    2018

  9. [16]

    Depeweg, J.-M

    S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. InInternational Conference on Machine Learning, pages 1184–1193. PMLR, 2018

    2018

  10. [17]

    Gal and Z

    Y . Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning. In M. Balcan and K. Q. Weinberger, editors,Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 20...

    2016

  11. [18]

    Y . Gal, R. Islam, and Z. Ghahramani. Deep bayesian active learning with image data. In D. Precup and Y . W. Teh, editors,Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 ofProceedings of Machi...

    2017

  12. [19]

    Geifman and R

    Y . Geifman and R. El-Yaniv. Selective classification for deep neural networks. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V . N. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems 30: Annual Conference on Neural I...

    2017

  13. [20]

    Gneiting and A

    T. Gneiting and A. Raftery. Strictly proper scoring rules, prediction, and estimation. Technical Report 463R, Department of Statistics, University of Washington, 2005

    2005

  14. [21]

    Gruber and F

    S. Gruber and F. Buettner. Uncertainty estimates of predictions via a general bias-variance decomposition. In F. J. R. Ruiz, J. G. Dy, and J. van de Meent, editors,International Conference on Artificial Intelligence and Statistics, 25-27 April 2023, Palau de Congressos, Valenc...

    2023

  15. [22]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016

    2016

  16. [23]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017

    2017

  17. [24]

    Hendrycks, K

    D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 15262–15271. Computer Vision Foundation / IEEE, 2021

    2021

  18. [25]

    G. E. Hinton and D. van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In L. Pitt, editor,Proceedings of the Sixth Annual ACM Conference on Computational Learning Theory, COLT 1993, Santa Cruz, CA, USA, July 26-28, 1993, pages 5–1...

    1993

  19. [26]

    Hüllermeier and W

    E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.Machine Learning, 110(3):457–506, 2021

    2021

  20. [27]

    Hüllermeier, S

    E. Hüllermeier, S. Destercke, and M. H. Shaker. Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison. In J. Cussens and K. Zhang, editors, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, v...

    2022

  21. [28]

    Jürgens, N

    M. Jürgens, N. Meinert, V . Bengs, E. Hüllermeier, and W. Waegeman. Is epistemic uncertainty faithfully represented by evidential deep learning methods? InForty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

    2024

  22. [29]

    Kirsch, J

    A. Kirsch, J. van Amersfoort, and Y . Gal. BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning. InAdvances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  23. [31]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  24. [32]

    Kull and P

    M. Kull and P. A. Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In A. Appice, P. P. Rodrigues, V . S. Costa, C. Soares, J. Gama, and A. Jorge, editors,Machine Learning and Knowledge Discovery in Databases ...

    2015

  25. [33]

    Lahlou, M

    S. Lahlou, M. Jain, H. Nekoei, V . Butoi, P. Bertin, J. Rector-Brooks, M. Korablyov, and Y . Bengio. DEUP: direct epistemic uncertainty prediction.Trans. Mach. Learn. Res., 2023, 2023

    2023

  26. [34]

    Lakshminarayanan, A

    B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V . N. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Proces...

    2017

  27. [35]

    LeCun, L

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner. Gradient-based learning applied to document recognition.Proc. IEEE, 86(11):2278–2324, 1998

    1998

  28. [36]

    Loshchilov and F

    I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017

    2017

  29. [37]

    Margraf, M

    V . Margraf, M. Wever, S. Gilhuber, G. M. Tavares, T. Seidl, and E. Hüllermeier. ALPBench: A Benchmark for Active Learning Pipelines on Tabular Data, June 2024. arXiv:2406.17322 [cs]

    2024 arXiv

  30. [38]

    Minderer, J

    M. Minderer, J. Djolonga, R. Romijnders, F. Hubis, X. Zhai, N. Houlsby, D. Tran, and M. Lucic. Revisiting the calibration of modern neural networks. In M. Ranzato, A. Beygelzimer, Y . N. Dauphin, P. Liang, and J. W. Vaughan, editors,Advances in Neural Information Processing Sy...

    2021

  31. [39]

    Mucsányi, M

    B. Mucsányi, M. Kirchhof, and S. J. Oh. Benchmarking uncertainty disentanglement: Special- ized uncertainties for specialized tasks. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

  32. [40]

    M. S. A. Nadeem, J. Zucker, and B. Hanczar. Accuracy-rejection curves (arcs) for comparing classification methods with a reject option. In S. Dzeroski, P. Geurts, and J. Rousu, editors, Proceedings of the third International Workshop on Machine Learning in Systems Biology, MLS...

    2009

  33. [41]

    Netzer, T

    Y . Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y . Ng. Reading digits in nat- ural images with unsupervised feature learning. InNIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011

    2011

  34. [42]

    Nguyen, S

    V .-L. Nguyen, S. Destercke, and E. Hüllermeier. Epistemic Uncertainty Sampling. In P. Kralj No- vak, T. Šmuc, and S. Džeroski, editors,Discovery Science, pages 72–86, Cham, 2019. Springer International Publishing. ISBN 978-3-030-33778-0

    2019

  35. [43]

    D. S. Pandey and Q. Yu. Learn to accumulate evidence from all training samples: Theory and practice. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors,International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawai...

    2023

  36. [44]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-...

    2019

  37. [45]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.Journal of Machine Learnin...

    2011

  38. [46]

    Ridnik, E

    T. Ridnik, E. B. Baruch, A. Noy, and L. Zelnik. Imagenet-21k pretraining for the masses. In J. Vanschoren and S. Yeung, editors,Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, vir...

    2021

  39. [47]

    Y . Sale, V . Bengs, M. Caprio, and E. Hüllermeier. Second-order uncertainty quantification: A distance-based approach.arXiv preprint arXiv:2312.00995, 2023

    2023 arXiv

  40. [48]

    Y . Sale, P. Hofman, T. Löhr, L. Wimmer, T. Nagler, and E. Hüllermeier. Label-wise aleatoric and epistemic uncertainty quantification. InThe 40th Conference on Uncertainty in Artificial Intelligence, 2024

    2024

  41. [49]

    L. J. Savage. Elicitation of personal probabilities and expectations.Journal of the American Statistical Association, 66(336):783–801, 1971

    1971

  42. [50]

    Schweighofer, L

    K. Schweighofer, L. Aichberger, M. Ielanskyi, and S. Hochreiter. On Information-Theoretic Measures of Predictive Uncertainty, Oct. 2024. arXiv:2410.10786

    2024 arXiv

  43. [51]

    Sensoy, L

    M. Sensoy, L. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. InProc. NeurIPS, 32nd Conference on Neural Information Processing Systems, Montreal, Canada, 2018

    2018

  44. [52]

    F. B. Smith, J. Kossen, E. Trollope, M. van der Wilk, A. Foster, and T. Rainforth. Rethinking aleatoric and epistemic uncertainty. InNeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty, 2024

    2024

  45. [53]

    Smith and Y

    L. Smith and Y . Gal. Understanding measures of uncertainty for adversarial example detection. In A. Globerson and R. Silva, editors,Proceedings of the Thirty-Fourth Conference on Uncer- tainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018,...

    2018

  46. [54]

    Wimmer, Y

    L. Wimmer, Y . Sale, P. Hofman, B. Bischl, and E. Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures?, 2023

    2023

  47. [55]

    H. Xiao, K. Rasul, and R. V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017. 13

    2017 arXiv

  48. [56]

    J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.Scientific Data, 10(1):41, 2023

    2023

  49. [57]

    B. Zhou, À. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition.IEEE Trans. Pattern Anal. Mach. Intell., 40(6):1452–1464, 2018. 14 A Derivations In this section, we derive total, aleatoric and epistemic uncertainty measure...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.