Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

Rethinking Aleatoric and Epistemic Uncertainty

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Given a decision and a loss, a rigorous uncertainty measure follows; BALD is an estimator of short-run parameter information gain, not a direct measure of reducible predictive uncertainty.

desk verdict A genuinely useful decision-theoretic reframing of uncertainty in ML; the BALD reinterpretation is the sharpest new result, and the paper is honest about the cost of needing a specified decision problem. read the letter →

arxiv 2412.20892 v3 pith:Z75KHAEW submitted 2024-12-30 cs.LG stat.ML

Freddie Bickford Smith , Jannik Kossen , Eleanor Trollope , Mark van der Wilk , Adam Foster , Tom Rainforth This is my paper · ORCID
classification cs.LGstat.ML MSC 62C0562F1568T37
keywords aleatoricuncertaintyepistemicdecisiontheorysubjectiveexpectedlossreductionBALDproperscoringrulesBayesianactivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard aleatoric/epistemic view of uncertainty in machine learning is incoherent and too coarse: researchers attach many different mathematical quantities to each of the two concepts, blurring the line between what a model believes, how data are generated, and how a model should be evaluated. The proposed alternative starts from a final decision with an explicit loss: predictive uncertainty is the subjective expected loss of the Bayes-optimal action, $h[p_n(z)] = \mathbb{E}_{p_n(z)}[\ell(a^*_n,z)]$, which generalizes variance and Shannon entropy as two special cases. Reducibility of uncertainty then requires specifying the data-generating process, giving a well-defined expected uncertainty reduction and a decomposition into irreducible and reducible parts. The paper also separates predictive uncertainty from predictive performance and data dispersion, and reinterprets BALD as an estimator of short-run expected parameter information gain rather than a direct measure of long-run reducible predictive uncertainty. This matters because active-learning objectives and uncertainty-based methods are currently designed and evaluated on conceptual foundations the paper argues are shaky.

What carries the argument

The load-bearing object is the minimal subjective expected loss $h[p_n(z)] = \mathbb{E}_{p_n(z)}[\ell(a^*_n,z)]$, which the paper calls predictive uncertainty; it is what makes the choice of uncertainty measure a derived quantity rather than a modeling preference. Two identities carry the rest of the argument: the expected uncertainty reduction $EUR^{\mathrm{true}}_z(\pi,m)$ based on the actual data-generating process, which yields the irreducible-reducible decomposition, and the discrepancy function $d(p_n,p_{\mathrm{eval}}) = \mathbb{E}_{p_{\mathrm{eval}}(z)}[s(p_n,z)] - h[p_{\mathrm{eval}}(z)]$, which separates uncertainty, predictive performance, and data dispersion while generalizing classical decompositions.

What would settle it

Run the paper's conjugate-model experiment with misspecified generative models, such as data drawn from a distribution outside the assumed Beta-Bernoulli or Normal-Gamma family, and measure whether BALD still estimates short-run expected parameter information gain better than long-run predictive information gain; if misspecification reverses that ordering, the paper's explanation of BALD's utility is falsified. Separately, exhibiting a real deployment with no specified decision or loss yet one unambiguously agreed uncertainty measure would weaken the loss-grounding claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that measures of predictive uncertainty need not be chosen arbitrarily: once a decision problem $(A,Z,\ell)$ is fixed, the minimal subjective expected loss of the Bayes-optimal action, $h[p_n(z)] = \mathbb{E}_{p_n(z)}[\ell(a^*_n,z)]$, is the uncertainty measure, with squared error yielding variance and log loss yielding entropy. From there the paper defines expected uncertainty reduction under an explicit data-acquisition policy, $EUR^{\mathrm{true}}_z(\pi,m) = \mathbb{E}_{p_{\mathrm{train}}(y^+_{1:m}\mid\pi)}[h[p_n(z)] - h[p_{n+m}(z)]]$, and shows the infinite-data limit gives a rigorous irreducible-reducible decomposition that applies to any method mapping data to a predictive distribution. A discrepancy identity, $d(p_n,p_{\mathrm{eval}}) = \mathbb{E}_{p_{\mathrm{eval}}(z)}[s(p_n,z)] - h[p_{\mathrm{eval}}(z)]$, separates predictive performance from uncertainty and data dispersion, generalizing bias-variance and cross-entropy/KL decompositions. Finally, the paper argues that BALD, the expected information gain in model parameters, is best understood as an estimator of the true one-step expected parameter information gain, not a direct measure of long-run reducible predictive uncertainty; the two can diverge substantially at finite $n$, which the paper demonstrates with conjugate models.

Load-bearing premise

The derivation assumes the user can specify a final decision problem with a loss function that reflects their preferences; for many machine-learning deployments no such loss exists, and without one the framework does not single out a unique uncertainty measure.

Editorial extensions

If this is right

  • Two decision-makers with different loss functions are not disagreeing about the same object when one reports variance and the other entropy; each is reporting the uncertainty measure their decision problem requires.
  • A reducible/irreducible decomposition is available for any data-to-predictive-distribution method, provided the data-generating policy is stated; stochastic parameters and exact Bayesian updating are not required.
  • BALD's practical success in active learning is compatible with it being a poor estimator of long-run predictive information gain, because acquisition horizons are short and it tracks one-step parameter information gain better.
  • Uncertainty alone gives no reliable signal of predictive performance or data dispersion; externally grounded evaluation is required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the loss-based derivation is taken seriously, then in prediction-oriented active learning the acquisition objective should be the expected reduction in predictive uncertainty under the final loss, which is what prediction-oriented information gain objectives approximate; this would make the paper's view a direct recipe for choosing acquisition functions rather than only a reinterpretation of ex
  • The framework implies that the phrase 'epistemic uncertainty' is doing the work of at least three distinct quantities: parameter uncertainty, reducible predictive uncertainty, and short-run expected parameter information gain, so future empirical comparisons that treat 'epistemic uncertainty' as one number will struggle to be meaningful.
  • A concrete extension the paper does not run is, for a fixed decision problem with a specified loss, comparing acquisition functions derived from finite-m expected uncertainty reduction under model-simulated data against BALD and predictive entropy on the same benchmarks; if the derived functions do not match or beat loss-appropriate baselines, the practical relevance of the decision-theoretic deri
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper argues that the standard aleatoric/epistemic uncertainty framework in machine learning is conceptually overloaded and internally incoherent, and proposes replacing it with a decision-theoretic perspective. The central construction defines predictive uncertainty h[p_n(z)] as the subjective expected loss of the Bayes-optimal action under the predictive model, and then derives an expected uncertainty reduction (EUR) that depends explicitly on the data-generating process and the data-acquisition policy. The paper also relates this uncertainty measure to predictive performance and data dispersion through proper scoring rules and discrepancy functions, and revisits the BALD score, arguing that it is best viewed as an imperfect estimator of short-run parameter information gain rather than a direct measure of long-run reducible predictive uncertainty. The claims are supported by formal propositions, illustrative examples, and exact conjugate-model experiments.

Significance. If the framework holds up, this is a valuable conceptual contribution: it unifies variance, entropy, and other uncertainty measures under a single decision-theoretic umbrella, provides a rigorous decomposition of reducible and irreducible uncertainty, and clarifies the status of BALD as an estimator rather than a target quantity. The paper is unusually honest about its limitations, explicitly acknowledging that the framework requires a specified loss and decision problem, and that the practical estimators rely on approximations. The empirical demonstrations in Figure 4 use exact conjugate models where the claimed estimation errors can be computed without approximation, which is a strength. The main weakness is a technical gap in the formal justification of Propositions 2, 3, and 6, which misapply a standard Bayes-estimator result to target quantities that are not functions of the model parameter under the stated assumptions.

major comments (2)
  1. [Section 5.4, Propositions 2 and 3 (and Proposition 6, Appendix B)] The proofs of Propositions 2, 3, and 6 apply Proposition 1 to a quantity F that is not, under the stated assumptions, a function of the model parameter θ. For Proposition 2, F = E_{p_eval(z)}[s(p_n,z)] while f(θ) = E_{p_n(z|θ)}[s(p_n,z)]; the assumption that p_n(z) is 'a model intended to directly approximate' p_eval(z) does not imply F = f(θ), yet Proposition 1 requires the quantity of interest to be a pushforward of p_n(θ) through f. The argument therefore establishes only that h[p_n(z)] is the posterior mean of f(θ), not a Bayes estimator of the external expected score. The same gap appears in Proposition 3, where F = h[p_eval(z)] and f(θ) = h[p_n(z|θ)], and in Proposition 6, where F = E_{ptrain(z)}[H[p_n(θ|z)]] and f(θ) = E_{p_n(z|θ)}[H[p_n(θ|z)]]. The propositions are salvageable by adding a well-specifiedness assumption, for example p_eval(z) = p_n(z|θ_0) for some θ_0 (and similarly for ptrain in Proposition 6), but as written they are not proved. Because Section 5.4 uses these propositions to reinterpret entropy-based quantities as Bayes estimators, this is a load-bearing gap that should be fixed before publication.
  2. [Appendix B, Proposition 4] The proof of Proposition 4 is too terse for a formal proposition. The statement 'reasoning about θ is equivalent to reasoning about y+1:∞' requires a careful argument that, under the stated posterior-consistency assumption, H[p_n(z|y+1:∞)] = H[p_n(z|θ∞)] almost surely and that this convergence is strong enough to justify applying Proposition 1 to the limit quantity. As written, the proof skips these steps and instead cites Fong et al. (2023) without spelling out the correspondence. This is not necessarily an error, but it needs to be expanded so that the reader can verify that the Bayes-estimator claim is not presupposing the desired conclusion.
minor comments (4)
  1. [Appendix B, Proposition 6 proof] The proof writes F = E_{ptrain(z)}[H[p_n(θ|z_{1:m})]], but the quantity being estimated, EIG^true_θ, involves a single new observation z = y_{n+1}; the subscript z_{1:m} appears to be a typo and should be z.
  2. [Section 3.4] The convention that updating stochasticity is 'implicitly absorbed into y_{1:n}' is confusing, since y_{1:n} was earlier defined as training data. Consider introducing an explicit random seed variable and treating the machine-learning method as a deterministic function of data and seed.
  3. [Section 5.5, Figure 4] The caption and text use slightly different notation for the sample sizes: the text says n ∈ (1, 10, 100, 1000) and the caption says n ∈ {1, 10, 100, 1000}; please standardize.
  4. [Section 5.4, Proposition 2 proof] The proof would be clearer if it showed the intermediate step E_{p_n(θ)}[E_{p_n(z|θ)}[s(p_n,z)]] = E_{p_n(z)}[s(p_n,z)] = h[p_n(z)], since this is the key algebraic step connecting f(θ) to the predictive uncertainty.

Circularity Check

2 steps flagged · score 4.0 of 10

Bayes-estimator propositions in Sections 5.4 and 5.5 reduce to definitions by choosing the target's subjective prior as the proposed estimator's integrand; the core decision-theoretic framework is otherwise self-contained.

  1. self definitional [Section 5.4, Proposition 2 (proof; same mechanism in Proposition 3)]
    "Proposition 2 Assume p_n(z) = E_{p_n(θ)}[p_n(z|θ)] is a model intended to directly approximate p_eval(z). Then the model's predictive uncertainty, h[p_n(z)], is a Bayes estimator of E_{p_eval(z)}[s(p_n,z)], the expected loss from acting Bayes-optimally under p_n(z) when z is in fact drawn from p_eval(z). Proof Applying Proposition 1 with F = E_{p_eval(z)}[s(p_n,z)] and f(θ) = E_{p_n(z|θ)}[s(p_n,z)] gives η* = h[p_n(z)]."

    Prop 1 defines the Bayes estimator of F as E_{p_n(θ)}[f(θ)] when the prior over F is the pushforward of p_n(θ) via f. Prop 2 chooses f(θ)=E_{p_n(z|θ)}[s(p_n,z)], and since h[p_n(z)] is defined as E_{p_n(z)}[s(p_n,z)] = E_{p_n(θ)}[f(θ)], the estimator equals the posterior mean of f by construction. The target named in the theorem, E_{p_eval(z)}[s(p_n,z)], never appears in the computation; the theorem is made true by declaring the subjective prior over that target to be the model's conditional expected score. The same substitution appears in Prop 3 (F=h[p_eval(z)], f(θ)=h[p_n(z|θ)]) and Prop 6 (F uses ptrain, f uses p_n(z|θ)), so these 'Bayes estimator' results are self-definitional: any plug-in estimator can be labelled a Bayes estimator of any target by this manoeuvre.

  2. self definitional [Section 5.5, Proposition 6 (proof)]
    "The second term here can be estimated by applying Proposition 1 with F = E_{ptrain(z)}[H[p_n(θ|z_{1:m})]] and f(θ) = E_{p_n(z|θ)}[H[p_n(θ|z)]]. The Bayes estimator that results from this, η* = E_{p_n(z)}[H[p_n(θ|z)]], can be combined with the known current entropy, H[p_n(θ)], to produce EIGθ = H[p_n(θ)] − E_{p_n(z)}[H[p_n(θ|z)]] as a Bayes estimator of EIGtrue_θ."

    The target F is an expectation under the true data distribution ptrain, but f(θ) replaces ptrain with the model conditional p_n(z|θ), and the prior over F is taken to be the pushforward of p_n(θ) via exactly that f. The resulting Bayes estimator is therefore the model-based expected conditional entropy, i.e. the plug-in EIGθ, by construction. The proposition's stated assumption that p_n is intended to approximate ptrain does not imply that the subjective prior over the ptrain-expected quantity equals the p_n-conditional one; this additional prior declaration is what guarantees the conclusion. Hence the 'Bayes estimator of EIGtrue_θ' claim is equivalent to the definition of the plug-in estimator, not an independent derivation.

full rationale

The central construction is conditional and definitional, not fitted: given a loss ℓ and a Bayes decision rule, h[p_n(z)] = E_{p_n(z)}[ℓ(a*_n,z)] is the minimal expected loss by definition, and the total/reducible/irreducible decomposition of Section 5.2 follows from the definition of expected uncertainty reduction. This core is self-contained and is honestly scoped: Section 3.2 acknowledges that 'in practice it can be hard to know what ℓ should be', and Section 5.1 notes different decision-makers may use different measures, so the loss-dependence is a stated limitation rather than a hidden assumption. The circularity is confined to the 'Bayes estimator' propositions in Sections 5.4 and 5.5. Propositions 2, 3 and 6 apply Proposition 1 by setting the target's subjective prior to the pushforward of p_n(θ) through a function f(θ) that is exactly the integrand of the proposed estimator, so the posterior mean equals the proposed quantity by construction and the target named in the theorem (expectation under p_eval or ptrain) never enters the computation. These propositions therefore relabel plug-in estimators as Bayes estimators under a prior chosen to make the label true, rather than providing independent support. The paper's own caveats ('these assumptions will typically not apply', 'we do not claim this is a general result') limit the damage, and the plug-in relation EIGθ ≈ EIGtrue_θ is visibly definitional, with Figure 4 supplying independent, code-reproduced empirical evidence. The self-citations for Figure 5's experimental setups refer to published, externally checkable prior work and are not load-bearing for the framework. Overall, the decision-theoretic core and the BALD empirical analysis retain independent content, so the circularity is partial rather than total.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

The framework rests on standard decision-theoretic and probabilistic modeling assumptions, and it introduces no fitted parameters or invented entities. All assumptions are explicitly stated in Sections 2-5 and the appendix.

assumptions (7)
  • standard math The consequences of actions are captured by a loss function ℓ(a,z), and preferences satisfy basic axioms of rationality (von Neumann & Morgenstern, 1947).
    Invoked in Section 3.2 to justify measuring uncertainty via expected loss; if preferences cannot be represented by a loss, the definition of h[p_n] collapses.
  • domain assumption Decision-making proceeds by minimizing subjective expected loss (Bayes optimality), not minimax or other decision rules.
    Section 3.3 and Equation 2; the entire framework is built on Savage-style subjective expected utility. The paper notes alternatives like minimax but does not incorporate them.
  • domain assumption There exists a data-generating process p_train(y_1:n | π) governed by a data-acquisition policy π.
    Section 3.4 and Equation 3; the expected uncertainty reduction is defined as an expectation over this process. If no such process is specified, EUR is undefined.
  • domain assumption The predictive model p_n(z) is a deterministic function of training data (with updating stochasticity absorbed into y), for any n.
    Section 3.4; this lets the paper reason about changes in uncertainty as n varies without model-update stochasticity.
  • domain assumption For Propositions 4 and 5, the posterior over parameters contracts to a point mass: p_n(θ|y_1:m) → δ_{θ∞}(θ) as m→∞ (Doob, 1949; Freedman, 1963; 1965).
    Appendix B; this is needed to equate reasoning about infinite data with reasoning about a single parameter setting. It holds for well-specified models but not generally.
  • domain assumption For Proposition 6, the data are i.i.d. from p_train(y) and the model p_n(z) is intended to directly approximate p_train(z).
    Appendix B; this identifies EIG_theta with a Bayes estimator of the true one-step information gain.
  • domain assumption Evaluation can be grounded in a reference distribution p_eval(z) or a single ground-truth z.
    Section 3.7; the discrepancy decomposition (Equation 5) requires such grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Aleatoric and Epistemic Uncertainty." pith.science (2026). https://pith.science/paper/Z75KHAEW

@misc{pith2026241220892,
  author       = {Pith},
  title        = {Pith review of: Rethinking Aleatoric and Epistemic Uncertainty},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z75KHAEW}},
  note         = {Machine review of arXiv:2412.20892}
}
read the original abstract

The ideas of aleatoric and epistemic uncertainty are widely used to reason about the probabilistic predictions of machine-learning models. We identify incoherence in existing discussions of these ideas and suggest this stems from the aleatoric-epistemic view being insufficiently expressive to capture all the distinct quantities that researchers are interested in. To address this we present a decision-theoretic perspective that relates rigorous notions of uncertainty, predictive performance and statistical dispersion in data. This serves to support clearer thinking as the field moves forward. Additionally we provide insights into popular information-theoretic quantities, showing they can be poor estimators of what they are often purported to measure, while also explaining how they can still be useful in guiding data acquisition.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Why Does the Future Branch? Identifiable Closure Tests for Stochastic Physical World Models

    cs.AI 2026-08 conditional novelty 6.0 of 10

    ClosurePairs uses paired microstate and disturbance interventions with variance decomposition to identify whether future branching is caused by state aliasing or process noise, which ordinary prediction scores cannot.

  2. Extending Epistemic Uncertainty Beyond Parameters Would Assist in Designing Reliable LLMs

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Bayesian Modeling of Experiments is proposed as a unifying framework for quantifying and reducing the many sources of uncertainty in LLM deployments, beyond abstention.

Reference graph

Works this paper leans on

85 extracted references · 80 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Deep evidential regression

    Amini, Schwarting, Soleimany, & Rus (2020). Deep evidential regression. Conference on Neural Information Processing Systems

    2020

  3. [3]

    Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks

    Ayhan & Berens (2018). Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks. Conference on Medical Imaging with Deep Learning

    2018

  4. [4]

    Optimal predictive model selection

    Barbieri & Berger (2004). Optimal predictive model selection. Annals of Statistics

    2004

  5. [5]

    Statistical Decision Theory and Bayesian Analysis

    Berger (1985). Statistical Decision Theory and Bayesian Analysis . Springer

    1985

  6. [6]

    Bayesian Theory

    Bernardo & Smith (1994). Bayesian Theory . John Wiley and Sons

    work page 1994

  7. [7]

    Making better use of unlabelled data in Bayesian active learning

    Bickford Smith , Foster, & Rainforth (2024). Making better use of unlabelled data in Bayesian active learning. International Conference on Artificial Intelligence and Statistics

    work page 2024

  8. [8]

    Prediction-oriented Bayesian active learning

    Bickford Smith , Kirsch, Farquhar, Gal, Foster, & Rainforth (2023). Prediction-oriented Bayesian active learning. International Conference on Artificial Intelligence and Statistics

    work page 2023

Show all 85 references
  1. [9]

    Science and statistics

    Box (1976). Science and statistics. Journal of the American Statistical Association

    1976

  2. [10]

    Random forests

    Breiman (2001). Random forests. Machine Learning

    2001

  3. [11]

    Language models are few-shot learners

    Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever, & Amodei (2020). Language m...

    2020

  4. [12]

    Elements of Information Theory

    Cover & Thomas (2005). Elements of Information Theory . John Wiley and Sons

    2005

  5. [13]

    Coherent measures of discrepancy, uncertainty and dependence, with applications to Bayesian predictive experimental design

    Dawid (1998). Coherent measures of discrepancy, uncertainty and dependence, with applications to Bayesian predictive experimental design. Technical report, University College London

    1998

  6. [14]

    Uncertainty, information, and sequential experiments

    DeGroot (1962). Uncertainty, information, and sequential experiments. Annals of Mathematical Statistics

    1962

  7. [15]

    Application of the theory of martingales

    Doob (1949). Application of the theory of martingales. Le Calcul des Probabilit \'e s et ses Applications

    1949

  8. [16]

    Evaluating superhuman models with consistency checks

    Fluri, Paleka, & Tram \`e r (2023). Evaluating superhuman models with consistency checks. arXiv

    2023

  9. [17]

    On the marginal likelihood and cross-validation

    Fong & Holmes (2020). On the marginal likelihood and cross-validation. Biometrika

    2020

  10. [18]

    Martingale posterior distributions

    Fong, Holmes, & Walker (2023). Martingale posterior distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology)

    2023

  11. [19]

    On the asymptotic behavior of Bayes estimates in the discrete case

    Freedman (1963). On the asymptotic behavior of Bayes estimates in the discrete case. Annals of Mathematical Statistics

    1963

  12. [20]

    On the asymptotic behavior of Bayes estimates in the discrete case II

    Freedman (1965). On the asymptotic behavior of Bayes estimates in the discrete case II . Annals of Mathematical Statistics

    1965

  13. [21]

    Uncertainty in deep learning

    Gal (2016). Uncertainty in deep learning . PhD thesis, University of Cambridge

    2016

  14. [22]

    Deep Bayesian active learning with image data

    Gal, Islam, & Ghahramani (2017). Deep Bayesian active learning with image data. International Conference on Machine Learning

    2017

  15. [23]

    A predictive approach to model selection

    Geisser & Eddy (1979). A predictive approach to model selection. Journal of the American Statistical Association

    1979

  16. [24]

    Sources of uncertainty in machine learning---a statisticians' view

    Gruber, Schenk, Schierholz, Kreuter, & Kauermann (2023). Sources of uncertainty in machine learning---a statisticians' view. arXiv

    2023

  17. [25]

    The Emergence of Probability

    Hacking (1975). The Emergence of Probability . Cambridge University Press

    1975

  18. [26]

    The Elements of Statistical Learning

    Hastie, Tibshirani, Friedman, & Friedman (2009). The Elements of Statistical Learning . Springer

    2009

  19. [27]

    Quantifying aleatoric and epistemic uncertainty: a credal approach

    Hofman, Sale, & H \"u llermeier (2024a). Quantifying aleatoric and epistemic uncertainty: a credal approach. Workshop on ``Structured Probabilistic Inference and Generative Modeling'', International Conference on Machine Learning

    2024

  20. [28]

    Quantifying aleatoric and epistemic uncertainty with proper scoring rules

    Hofman, Sale, & H \"u llermeier (2024b). Quantifying aleatoric and epistemic uncertainty with proper scoring rules. arXiv

    2024

  21. [29]

    Bayesian active learning for classification and preference learning

    Houlsby, Husz \'a r, Ghahramani, & Lengyel (2011). Bayesian active learning for classification and preference learning. arXiv

    2011

  22. [30]

    Amortized Bayesian experimental design for decision-making

    Huang, Guo, Acerbi, & Kaski (2024). Amortized Bayesian experimental design for decision-making. Conference on Neural Information Processing Systems

    2024

  23. [31]

    Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods

    H \"u llermeier & Waegeman (2021). Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning

    2021

  24. [32]

    Scalable marginal likelihood estimation for model selection in deep learning

    Immer, Bauer, Fortuin, R \"a tsch, & Khan (2021). Scalable marginal likelihood estimation for model selection in deep learning. International Conference on Machine Learning

    2021

  25. [33]

    Methods and criteria for model selection

    Kadane & Lazar (2004). Methods and criteria for model selection. Journal of the American Statistical Association

    2004

  26. [34]

    On uncertainty, tempering, and data augmentation in Bayesian classification

    Kapoor, Maddox, Izmailov, & Wilson (2022). On uncertainty, tempering, and data augmentation in Bayesian classification. Conference on Neural Information Processing Systems

    2022

  27. [35]

    What uncertainties do we need in Bayesian deep learning for computer vision? Conference on Neural Information Processing Systems

    Kendall & Gal (2017). What uncertainties do we need in Bayesian deep learning for computer vision? Conference on Neural Information Processing Systems

    2017

  28. [36]

    Stochastic batch acquisition: a simple baseline for deep active learning

    Kirsch, Farquhar, Atighehchian, Jesson, Branchaud-Charron, & Gal (2023). Stochastic batch acquisition: a simple baseline for deep active learning. Transactions on Machine Learning Research

    2023

  29. [37]

    BatchBALD : efficient and diverse batch acquisition for deep Bayesian active learning

    Kirsch, van Amersfoort, & Gal (2019). BatchBALD : efficient and diverse batch acquisition for deep Bayesian active learning. Conference on Neural Information Processing Systems

    2019

  30. [38]

    Misspecification in infinite-dimensional Bayesian statistics

    Kleijn & van der Vaart (2006). Misspecification in infinite-dimensional Bayesian statistics. Annals of Statistics

    2006

  31. [39]

    The Bernstein-von-Mises theorem under misspecification

    Kleijn & van der Vaart (2012). The Bernstein-von-Mises theorem under misspecification. Electronic Journal of Statistics

    2012

  32. [40]

    Nonparametric uncertainty quantification for single deterministic neural network

    Kotelevskii, Artemenkov, Fedyanin, Noskov, Fishkov, Shelmanov, Vazhentsev, Petiushko, & Panov (2022). Nonparametric uncertainty quantification for single deterministic neural network. Conference on Neural Information Processing Systems

    2022

  33. [41]

    From risk to uncertainty: generating predictive uncertainty measures via Bayesian estimation

    Kotelevskii, Kondratyev, Tak \'a c , Moulines, & Panov (2025). From risk to uncertainty: generating predictive uncertainty measures via Bayesian estimation. International Conference on Learning Representations

    2025

  34. [42]

    DEUP : direct epistemic uncertainty prediction

    Lahlou, Jain, Nekoei, Butoi, Bertin, Rector-Brooks, Korablyov, & Bengio (2023). DEUP : direct epistemic uncertainty prediction. Transactions on Machine Learning Research

    2023

  35. [43]

    Predictive model selection

    Laud & Ibrahim (1995). Predictive model selection. Journal of the Royal Statistical Society: Series B (Methodological)

    1995

  36. [44]

    What is machine learning? videolectures.net/mlss2012_lawrence_machine_learning

    Lawrence (2012). What is machine learning? videolectures.net/mlss2012_lawrence_machine_learning

    2012

  37. [45]

    Deep learning

    LeCun, Bengio, & Hinton (2015). Deep learning. Nature

    2015

  38. [46]

    On a measure of the information provided by an experiment

    Lindley (1956). On a measure of the information provided by an experiment. Annals of Mathematical Statistics

    1956

  39. [47]

    The evidence framework applied to classification networks

    MacKay (1992a). The evidence framework applied to classification networks. Neural Computation

    1992

  40. [48]

    Information-based objective functions for active data selection

    MacKay (1992b). Information-based objective functions for active data selection. Neural Computation

    1992

  41. [49]

    Bayesian learning for data-efficient control

    McAllister (2016). Bayesian learning for data-efficient control . PhD thesis, University of Cambridge

    2016

  42. [50]

    Benchmarking uncertainty disentanglement: specialized uncertainties for specialized tasks

    Mucs \'a nyi, Kirchhof, & Oh (2024). Benchmarking uncertainty disentanglement: specialized uncertainties for specialized tasks. Conference on Neural Information Processing Systems

    2024

  43. [51]

    Deep deterministic uncertainty: a new simple baseline

    Mukhoti, Kirsch, van Amersfoort , Torr, & Gal (2023). Deep deterministic uncertainty: a new simple baseline. Conference on Computer Vision and Pattern Recognition

    2023

  44. [52]

    Probabilistic Machine Learning: An Introduction

    Murphy (2022). Probabilistic Machine Learning: An Introduction . MIT Press

    2022

  45. [53]

    Generalizing Bayesian optimization with decision-theoretic entropies

    Neiswanger, Yu, Zhao, Meng, & Ermon (2022). Generalizing Bayesian optimization with decision-theoretic entropies. Conference on Neural Information Processing Systems

    2022

  46. [54]

    U2 -net: a Bayesian U -net model with epistemic uncertainty feedback for photoreceptor layer segmentation in pathological OCT scans

    Orlando, Seeb \"o ck, Bogunovi \'c , Klimscha, Grechenig, Waldstein, Gerendas, & Schmidt-Erfurth (2019). U2 -net: a Bayesian U -net model with epistemic uncertainty feedback for photoreceptor layer segmentation in pathological OCT scans. International Symposium on Biomedical Imaging

    2019

  47. [55]

    Epistemic neural networks

    Osband, Wen, Asghari, Dwaracherla, Ibrahimi, Lu, & Van Roy (2023). Epistemic neural networks. Conference on Neural Information Processing Systems

    2023

  48. [56]

    Quantifying aleatoric and epistemic uncertainty using density estimation in latent space

    Postels, Blum, Cadena, Siegwart, Van Gool , & Tombari (2020). Quantifying aleatoric and epistemic uncertainty using density estimation in latent space. arXiv

    2020

  49. [57]

    Applied Statistical Decision Theory

    Raiffa & Schlaifer (1961). Applied Statistical Decision Theory . Division of Research, Harvard Business School

    1961

  50. [58]

    Truth and probability

    Ramsey (1926). Truth and probability. Studies in Subjective Probability

    1926

  51. [59]

    Mathematical Statistics and Data Analysis

    Rice (2007). Mathematical Statistics and Data Analysis . Thomson Brooks/Cole

    2007

  52. [60]

    Intrinsic losses

    Robert (1996). Intrinsic losses. Theory and Decision

    1996

  53. [61]

    The Bayesian Choice

    Robert (2007). The Bayesian Choice . Springer

    2007

  54. [62]

    Second-order uncertainty quantification: a distance-based approach

    Sale, Bengs, Caprio, & H \"u llermeier (2024a). Second-order uncertainty quantification: a distance-based approach. International Conference on Machine Learning

    2024

  55. [63]

    Is the volume of a credal set a good measure for epistemic uncertainty? Conference on Uncertainty in Artificial Intelligence

    Sale, Caprio, & H \"u llermeier (2023a). Is the volume of a credal set a good measure for epistemic uncertainty? Conference on Uncertainty in Artificial Intelligence

    2023

  56. [64]

    o hr, Wimmer, Nagler, & H \

    Sale, Hofman, L \"o hr, Wimmer, Nagler, & H \"u llermeier (2024b). Label-wise aleatoric and epistemic uncertainty quantification. Conference on Uncertainty in Artificial Intelligence

    2024

  57. [65]

    Second-order uncertainty quantification: variance-based measures

    Sale, Hofman, Wimmer, H \"u llermeier, & Nagler (2023b). Second-order uncertainty quantification: variance-based measures. arXiv

    2023

  58. [66]

    The theory of statistical decision

    Savage (1951). The theory of statistical decision. Journal of the American Statistical Association

    1951

  59. [67]

    Elicitation of personal probabilities and expectations

    Savage (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association

    1971

  60. [68]

    Introducing an improved information-theoretic measure of predictive uncertainty

    Schweighofer, Aichberger, Ielanskyi, & Hochreiter (2023a). Introducing an improved information-theoretic measure of predictive uncertainty. Workshop on ``Mathematics of Modern Machine Learning'', Conference on Neural Information Processing Systems

    2023

  61. [69]

    On information-theoretic measures of predictive uncertainty

    Schweighofer, Aichberger, Ielanskyi, & Hochreiter (2025). On information-theoretic measures of predictive uncertainty. Conference on Uncertainty in Artificial Intelligence

    2025

  62. [70]

    Quantification of uncertainty with adversarial models

    Schweighofer, Aichberger, Ielanskyi, Klambauer, & Hochreiter (2023b). Quantification of uncertainty with adversarial models. Conference on Neural Information Processing Systems

    2023

  63. [71]

    Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal OCT

    Seeb \"o ck, Orlando, Schlegl, Waldstein, Bogunovi \'c , Klimscha, Langs, & Schmidt-Erfurth (2019). Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal OCT . Transactions on Medical Imaging

    2019

  64. [72]

    o sner, Dembczy \'n ski, Haasenritter, Hirsch, Donner-Banzhoff, & H \

    Senge, B \"o sner, Dembczy \'n ski, Haasenritter, Hirsch, Donner-Banzhoff, & H \"u llermeier (2014). Reliable classification: learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences

    2014

  65. [73]

    A mathematical theory of communication

    Shannon (1948). A mathematical theory of communication. The Bell System Technical Journal

    1948

  66. [74]

    Understanding measures of uncertainty for adversarial example detection

    Smith & Gal (2018). Understanding measures of uncertainty for adversarial example detection. Conference on Uncertainty in Artificial Intelligence

    2018

  67. [75]

    Algorithms for Reinforcement Learning

    Szepesv \'a ri (2010). Algorithms for Reinforcement Learning . Morgan and Claypool

    2010

  68. [76]

    A deeper look into aleatoric and epistemic uncertainty disentanglement

    Valdenegro-Toro & Saromo-Mori (2022). A deeper look into aleatoric and epistemic uncertainty disentanglement. Workshop on ``LatinX in CV Research'', Conference on Computer Vision and Pattern Recognition

    2022

  69. [77]

    Uncertainty estimation using a single deep deterministic neural network

    van Amersfoort , Smith, Teh, & Gal (2020). Uncertainty estimation using a single deep deterministic neural network. International Conference on Machine Learning

    2020

  70. [78]

    Zur Theorie der Gesellschaftsspiele

    von Neumann (1928). Zur Theorie der Gesellschaftsspiele . Mathematische Annalen

    1928

  71. [79]

    Theory of Games and Economic Behavior

    von Neumann & Morgenstern (1947). Theory of Games and Economic Behavior . Princeton University Press

    1947

  72. [80]

    Contributions to the theory of statistical estimation and testing hypotheses

    Wald (1939). Contributions to the theory of statistical estimation and testing hypotheses. Annals of Mathematical Statistics

    1939

  73. [81]

    Statistical decision functions which minimize the maximum risk

    Wald (1945). Statistical decision functions which minimize the maximum risk. Annals of Mathematics

    1945

  74. [82]

    Bayesian OOD detection with aleatoric uncertainty and outlier exposure

    Wang & Aitchison (2021). Bayesian OOD detection with aleatoric uncertainty and outlier exposure. arXiv

    2021

  75. [83]

    Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks

    Wang, Li, Aertsen, Deprest, Ourselin, & Vercauteren (2019). Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing

    2019

  76. [84]

    From predictions to decisions: the importance of joint predictive distributions

    Wen, Osband, Qin, Lu, Ibrahimi, Dwaracherla, Asghari, & Van Roy (2022). From predictions to decisions: the importance of joint predictive distributions. arXiv

    2022

  77. [85]

    Wimmer, Sale, Hofman, Bischl, & H \"u llermeier (2023). Quantifying aleatoric and epistemic uncertainty in machine learning: are conditional entropy and mutual information appropriate measures? Conference on Uncertainty in Artificial Intelligence

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.