Pith. sign in

REVIEW 4 major objections 4 minor 39 references

Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read The paper claims that representing epistemic uncertainty with possibility functions yields two new active-learning acquisition functions that satisfy a desired list of properties and beat standard local baselines on GP classification…

desk verdict New acquisition functions with a solid empirical story, but the central proof of their advertised 'desirable properties' does not hold; the paper deserves a referee's time only if the authors can fix or scope down the theory. read the letter →

arxiv 2412.08225 v1 pith:YGSE2KBG submitted 2024-12-11 stat.ME cs.LG

classification stat.MEcs.LG MSC 62L0562F1562H30
keywords activelearningepistemicuncertaintyaleatoricpossibilitytheoryouterprobabilitymeasuresGaussianprocessesacquisitionfunctionsclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that epistemic uncertainty can be cleanly separated from aleatoric uncertainty by modelling the former with possibility functions instead of probability distributions. It proposes two acquisition functions: an integral measure of epistemic uncertainty in label space, and the necessity of correct classification, which measures uncertainty at the decision boundary. To deploy these in nontrivial settings, it introduces a possibilistic Gaussian process that reproduces the standard GP predictive equations. The authors report that at least one of the proposed functions outperforms the local, update-free baselines on each of 11 datasets.

What carries the argument

The central object is the outer probability measure (OPM), built from a possibility function $f_\theta$ with supremum 1, which assigns credibility $\bar P(A)=\sup_{\theta\in A} f_\theta(\theta)$ and combines with a likelihood through $\bar P(A\times B)=\sup_{\theta\in A}f_\theta(\theta)\int_B p_\theta(y)\,dy$. This yields the uncertainty measures $U^\Theta=\int f_\theta(\theta)\,d\theta$, $U^Y=\int\sup_\theta f_\theta(\theta)p_Y(y|\theta)\,dy-1$, and the necessity $N(E)=1-\bar P(E^c)$. For classification, the possibilistic GP with Gaussian possibility functions and a Laplace approximation produces the same predictive mean and covariance as a standard GP, so the acquisition functions can be evaluated in closed form or with simple numerical optimization.

What would settle it

Take two possibility functions that agree everywhere except on one interval, where the first is higher, and make the second higher on a larger interval outside it so its total integral is smaller; direct integration then contradicts the strict version of Property A4 that the proof relies on. In dimension $d\ge 2$, apply $T(\theta)=\alpha\theta+(1-\alpha)c$ to a Gaussian possibility function and compute $U^\Theta$; the change of variables contributes a Jacobian factor $\alpha^d$, so the stated equality $U^\Theta_{f_T}=\alpha U^\Theta_f$ fails.

Watch

Extended reading notes

Core claim

The central claim is that combining probability and possibility theories via outer probability measures gives acquisition functions with desirable axiomatic properties and strong empirical performance in GP-based classification. Specifically, the integral measure $U^L$ and the necessity of correct classification $N$ are local and update-free, satisfy adapted versions of the properties A0--A4 proposed for epistemic-uncertainty measures, and achieve best or second-best accuracy on most of the tested datasets. The possibilistic Gaussian process is the enabling object: it provides closed-form Gaussian posterior possibility functions whose predictive equations coincide with those of standard GPs, allowing the acquisition functions to be implemented for binary and multiclass classification.

Load-bearing premise

The proof of the desirable properties assumes that making a function higher over part of its range always increases its total area, and that a certain rescaling multiplies the uncertainty by exactly $\alpha$; under the paper's own definitions the area claim does not follow from the stated definition and the rescaling claim holds only in one dimension.

Editorial extensions

If this is right

  • If the central claim is correct, active learning can target epistemic uncertainty directly without conflating it with irreducible aleatoric uncertainty, using local and update-free rules that are easy to deploy.
  • The possibilistic GP gives a principled way to run GP classification while interpreting the posterior as information about a fixed true function rather than as a distribution over random functions.
  • The necessity-based acquisition function is computationally light, reported as about 12% slower than standard acquisition functions and 9% faster than BALD, making it attractive when query selection cost matters.
  • The proposed measures come with a checklist of axiomatic properties, so future epistemic-uncertainty measures can be compared against the same criteria rather than evaluated only by empirical accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof issue in Property A3 suggests the rescaling property should involve $\alpha^d$, where $d$ is the dimension of the parameter space, rather than $\alpha$; the acquisition functions could still work empirically, but the axiomatic comparison to the Wimmer et al. properties would need reformulation.
  • The same probability--possibility decomposition could be carried beyond GP classification, for example to regression or bandit problems, since the paper already derives possibilistic GP regression equations that match the standard ones.
  • A natural test is to compare $U^L$ and $N$ against global acquisition functions such as MOCU, BEMPS, and EPIG on the same datasets; the paper only benchmarks against local, update-free methods, so its claim is about that class.
  • The closed-form expression for $N_{\text{multi}}$ suggests a very cheap active learning rule for problems with many classes, which would be worth stress-testing on high-dimensional label spaces.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper develops an active learning framework in which epistemic uncertainty is represented by possibility functions rather than probability distributions. It proposes two acquisition functions: U^L, an integral of a marginal outer probability measure over label space, and 1−N, the complement of the necessity of correct classification. A possibilistic Gaussian process (PGP) is introduced and shown to reproduce standard GP predictive equations. The authors prove axiomatic properties for the acquisition functions and report experiments on 11 real and synthetic datasets where the proposed functions often rank best or second best among local, update-free strategies.

Significance. If the theoretical results were valid, the paper would be a useful contribution: it gives explicit, computationally cheap acquisition functions in a coherent possibility-theoretic framework, states a clear list of axioms A0–A4, derives closed-form expressions for the necessity of correct classification, and provides a broad experimental comparison with rank summaries. The PGP construction is transparent, and the empirical protocol is reproducible in principle. The main advertised contribution, however, is the set of desirable properties in Propositions 1 and 4, and these are not established as stated; the current proofs rely on a definition of 'strictly less informative' that is too weak and on an incorrect Jacobian. The empirical comparison is plausible, but the headline claim of outperformance would benefit from significance testing.

major comments (4)
  1. [§4.1 / Appendix D, Prop. 1, footnote 1] The proof of Property A4 for U^Y is invalid under the definition of 'strictly less informative' in footnote 1, which only requires f'_θ > f_θ on a positive-volume set A and imposes no constraint on Θ\A. The inequality ∫ sup_θ f'_θ(θ)p(y|θ) dy ≥ ∫ sup_θ f_θ(θ)p(y|θ) dy does not follow, and it is false. For instance, take Θ=[0,1], Y={0,1}, p_Y(1|θ)=θ, f=1 at {0,1} and 1/2 on [0.4,0.6], and f'=1 on [0.4,0.6] and 0 elsewhere. Then f' is strictly less informative than f in the sense of footnote 1, but U^Y_f = 1 and U^Y_{f'} = 0.2. The proof of strict A4 for U^Theta has the same gap: it assumes f'_θ ≥ f_θ on Θ\A, which is not part of the definition. A pointwise domination condition would repair the argument, but it must be stated explicitly.
  2. [Appendix D, Prop. 1, Property A3] In the multinomial case, T(θ)=αθ+(1−α)c is an affine map on the (m−1)-dimensional simplex. Its Jacobian with respect to Lebesgue measure on the simplex is α^{m−1}, not α. Therefore the change of variables in the proof gives U^Θ_{f_T(θ)} = α^{m−1} U^Θ_{f_θ}, which agrees with the claimed Property A3 only when m=2. Proposition 1's statement that U^Theta satisfies A3 as written is false for multiclass problems.
  3. [Appendix D, Prop. 4] The proof of strict A4 for 1−Nbin states that the only way a Gaussian possibility function can be strictly less informative than another is by keeping the mean and increasing the variance. This is not true under footnote 1: shifting the mean from μ to μ′>μ makes the shifted Gaussian larger than the original on a neighborhood of μ′, which has positive volume, so the shifted function is 'strictly less informative' by the paper's definition. But, for μ>0, 1−Nbin = N(0;μ,σ²) becomes N(0;μ′,σ²), which is smaller, contradicting strict monotonicity. The proof of A1 in the same proposition also assumes μ_x≠0, so it does not cover a point mass at μ_x=0, where the most likely label is not uniquely defined. Proposition 4 needs a stronger definition of less-informativeness or a restricted statement.
  4. [§6, Tables 1–2] The claim that the new acquisition functions 'outperform acquisition functions with the same properties' is not supported by a statistical test. On several datasets the best proposed method is within the interquartile range of the Standard baseline (e.g., Breast Cancer: Nbin 0.960 [0.957,0.963] vs Standard 0.957 [0.950,0.960]; Wine: 0.988 [0.986,0.994] vs 0.988 [0.986,0.994]). The average-rank results in Table 2 are more encouraging, but the paper should either provide paired significance tests across runs or temper the wording.
minor comments (4)
  1. [§6 / Appendix E] The paper states that code is available at '[URL]' and Appendix E says it will be released upon acceptance; a stable repository link should be provided.
  2. [Appendix E.3] For binary classification the GP hyperparameters are fixed after being computed in advance, while for multiclass they are optimized during training; please state explicitly whether this asymmetry affects both proposed and baseline methods equally at each query.
  3. [Figure 1] The shaded regions are Q1–Q3 intervals, but the figure caption does not state the number of runs per dataset; the number of runs varies between 50 and 300 in Table 3, so the caption should be made consistent.
  4. [Appendix E.4] The binary least-confidence strategy is called 'Standard' in the experiments; this should be stated at first use in Section 6 so that readers do not confuse it with a generic standard GP baseline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the acquisition functions are closed-form posterior quantities, the PGP posterior equations are re-derived rather than assumed, and the self-citations provide foundational definitions rather than fitted predictions.

full rationale

The claimed derivation chain is not circular. The two acquisition functions U^L and 1-N are defined as closed-form functionals of Gaussian possibility functions (Sections 4.2 and 4.3), with parameters supplied by a possibilistic Gaussian process posterior; neither quantity is fitted to the accuracy values against which it is later benchmarked. The PGP predictive equations in Propositions 6-8 are derived directly in Appendix B and explicitly recover the standard GP equations, so this is a re-derivation rather than a prediction disguised as a first-principles result. Self-citations to Houssineau (2018), Houssineau et al. (2019), and Houssineau and Bishop (2018) supply the underlying possibility-theoretic definitions, the mode-based expected value, and a Kalman-filter identity; these are stated mathematical frameworks whose assumptions do not include the present active-learning claims, so they function as independent support rather than a circular premise. The empirical evaluation against BALD, entropy, least-confidence, margin, and random baselines on 11 datasets is a separable assessment of the proposed rules. The reader-identified defects, namely the invalid monotonicity step in the proof of Proposition 1(A4) and the Jacobian miscalculation in Proposition 1(A3), are mathematical correctness concerns rather than circular reductions: they make the advertised properties unsupported, but they do not make the derivation equivalent to its inputs by construction.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The central claims rest on the possibility-theory framework, the Gaussian posterior assumption, and the Laplace approximation. The free parameters are standard GP hyperparameter fits. The axioms include the OPM formalism and two ad hoc assumptions in the property proofs that appear inconsistent.

free parameters (1)
  • GP kernel hyperparameters (RBF lengthscale, outputscale, noise variance) = Tuned per dataset via GPytorch (binary: fixed after fitting; multiclass: Adam optimised with 500 iterations)
    The reported active learning performance depends on these fitted hyperparameters; they are standard model fits, not ad hoc to the acquisition functions.
assumptions (5)
  • domain assumption Numerical possibility theory as in De Cooman (2001) and Houssineau (2018): OPMs defined via sup over possibility functions; E* as mode; covariance via Hessian of log possibility.
    Section 3 and Section 5.1; the acquisition functions and PGP definitions rest on this framework.
  • domain assumption Posterior possibility function for the latent parameter at input x is a Gaussian possibility N(mu_x, sigma^2_x), obtained via Laplace approximation.
    Section 4.2 and Appendix C; used to derive U^L_bin and Nbin. The Laplace approximation is standard but not exact.
  • domain assumption Independence of latent functions across classes in multiclass classification.
    Section 4.2 and Proposition 8; needed for the Nmulti formula.
  • ad hoc to paper The definition of strictly less informative as a possibility function that is larger on some region of positive volume (footnote 1) is the basis for Property A4.
    This definition is weaker than the usual pointwise dominance and is inconsistent with the proofs, which assume pointwise dominance.
  • ad hoc to paper In Property A3, the transformation T on the simplex with Jacobian alpha^(d-1) is treated as scaling the integral by alpha.
    The proof of Proposition 1 (Appendix D) assumes U^Theta_{f'} = alpha U^Theta_f, which only holds in one dimension.
invented entities (1)
  • Possibilistic Gaussian process (PGP)
    purpose: To provide a Bayesian-style posterior over a latent function using possibility functions, enabling the proposed acquisition functions in GP classification.
    The PGP predictive equations (Propositions 6-8) are identical to standard GP regression/classification, so it does not generate new falsifiable predictions outside the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty." pith.science (2026). https://pith.science/paper/YGSE2KBG

@misc{pith2026241208225,
  author       = {Pith},
  title        = {Pith review of: Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGSE2KBG}},
  note         = {Machine review of arXiv:2412.08225}
}
read the original abstract

A popular strategy for active learning is to specifically target a reduction in epistemic uncertainty, since aleatoric uncertainty is often considered as being intrinsic to the system of interest and therefore not reducible. Yet, distinguishing these two types of uncertainty remains challenging and there is no single strategy that consistently outperforms the others. We propose to use a particular combination of probability and possibility theories, with the aim of using the latter to specifically represent epistemic uncertainty, and we show how this combination leads to new active learning strategies that have desirable properties. In order to demonstrate the efficiency of these strategies in non-trivial settings, we introduce the notion of a possibilistic Gaussian process (GP) and consider GP-based multiclass and binary classification problems, for which the proposed methods display a strong performance for both simulated and real datasets.

Figures

Figures reproduced from arXiv: 2412.08225 by the authors.

Figure 1
Figure 1. Performance of BALD, Random, Standard and our approaches on six binary classification problems [PITH_FULL_IMAGE:figures/full_fig_p022_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 32 canonical work pages

  1. [1]

    Aeberhard, S., Coomans, D., and De Vel, O. (1994). Comparative analysis of statistical pattern recognition methods in high dimensional settings. Pattern Recognition , 27(8):1065--1077

  2. [2]

    Caprio, M., Sale, Y., H \"u llermeier, E., and Lee, I. (2023). A novel B ayes’ theorem for upper probabilities. In International Workshop on Epistemic Uncertainty in Artificial Intelligence , pages 1--12. Springer

  3. [3]

    Chen, Z., Ristic, B., Houssineau, J., and Kim, D. Y. (2021). Observer control for bearings-only tracking using possibility functions. Automatica , 133:109888

  4. [4]

    P., Stone, M., and Zidek, J

    Dawid, A. P., Stone, M., and Zidek, J. V. (1973). Marginalization paradoxes in bayesian and structural inference. Journal of the Royal Statistical Society Series B: Statistical Methodology , 35(2):189--213

  5. [5]

    De Cooman, G. (2001). Integration and conditioning in numerical possibility theory. Annals of Mathematics and Artificial Intelligence , 32:87--123

  6. [6]

    Den ux, T. (2023). Reasoning with fuzzy and uncertain evidence using epistemic random fuzzy sets: General framework and practical models. Fuzzy Sets and Systems , 453:1--36

  7. [7]

    T., and Prade, H

    Dubois, D., Nguyen, H. T., and Prade, H. (2000). Possibility theory, probability and fuzzy sets misunderstandings, bridges and gaps. In Fundamentals of fuzzy sets , pages 343--438. Springer

  8. [8]

    and Prade, H

    Dubois, D. and Prade, H. (2015). Possibility theory and its applications: Where do we stand? Springer handbook of computational intelligence , pages 31--60

Show all 39 references
  1. [9]

    Fisher, R. A. (1935). The fiducial argument in statistical inference. Annals of eugenics , 6(4):391--398

  2. [10]

    Fisher, R. A. (1988). Iris . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C56C76

  3. [11]

    Friedman, J. H. (2002). Stochastic gradient boosting. Computational statistics & data analysis , 38(4):367--378

  4. [12]

    Q., Bindel, D., and Wilson, A

    Gardner, J., Pleiss, G., Weinberger, K. Q., Bindel, D., and Wilson, A. G. (2018). Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in neural information processing systems , 31

  5. [13]

    Gorman, R. P. and Sejnowski, T. J. (1988). Analysis of hidden units in a layered network trained to classify sonar targets. Neural networks , 1(1):75--89

  6. [14]

    Houlsby, N., Husz \'a r, F., Ghahramani, Z., and Lengyel, M. (2011). Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745

  7. [15]

    Houssineau, J. (2018). Parameter estimation with a class of outer probability measures. arXiv preprint arXiv:1801.00569

  8. [16]

    Houssineau, J. (2021). A linear algorithm for multi-target tracking in the context of possibility theory. IEEE Transactions on Signal Processing , 69:2740--2751

  9. [17]

    and Bishop, A

    Houssineau, J. and Bishop, A. N. (2018). Smoothing and filtering with a class of outer measures. SIAM/ASA Journal on Uncertainty Quantification , 6(2):845--866

  10. [18]

    Houssineau, J., Chada, N., and Delande, E. (2019). Elements of asymptotic theory with outer probability measures. arXiv preprint arXiv:1908.04331

  11. [19]

    and Waegeman, W

    H \"u llermeier, E. and Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning , 110(3):457--506

  12. [20]

    MacKay, D. J. (1992). Information-based objective functions for active data selection. Neural computation , 4(4):590--604

  13. [21]

    Mangasarian, O. L. and Wolberg, W. H. (1990). Cancer diagnosis via linear programming. Technical report, University of Wisconsin-Madison Department of Computer Sciences

  14. [22]

    Nguyen, V.-L., Destercke, S., and H \"u llermeier, E. (2019). Epistemic uncertainty sampling. In Discovery Science: 22nd International Conference, DS 2019, Split, Croatia, October 28--30, 2019, Proceedings 22 , pages 72--86. Springer

  15. [23]

    H., and H \"u llermeier, E

    Nguyen, V.-L., Shaker, M. H., and H \"u llermeier, E. (2022). How to measure uncertainty in uncertainty sampling for active learning. Machine Learning , 111(1):89--122

  16. [24]

    Quinlan, R. (1987). Thyroid Disease . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C5D010

  17. [25]

    Rodrigues, F., Pereira, F., and Ribeiro, B. (2014). Gaussian process classification and active learning with multiple annotators. In International conference on machine learning , pages 433--441. PMLR

  18. [26]

    Sale, Y., Bengs, V., Caprio, M., and H \"u llermeier, E. (2023). Second-order uncertainty quantification: A distance-based approach. In Forty-first International Conference on Machine Learning

  19. [27]

    and Wynn, H

    Sebastiani, P. and Wynn, H. P. (2000). Maximum entropy sampling and optimal B ayesian experimental design. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 62(1):145--157

  20. [28]

    Siebert, J. P. (1987). Vehicle recognition using rule based methods . Turing Institute

  21. [29]

    G., Wing, S

    Sigillito, V. G., Wing, S. P., Hutton, L. V., and Baker, K. B. (1989). Classification of radar returns from the ionosphere using neural networks. Johns Hopkins APL Technical Digest , 10(3):262--266

  22. [30]

    and Wrobel, L

    Sikora, M. and Wrobel, L. (2013). seismic-bumps . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C5W902

  23. [31]

    B., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T

    Smith, F. B., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T. (2023). Prediction-oriented bayesian active learning. In International Conference on Artificial Intelligence and Statistics , pages 7331--7348. PMLR

  24. [32]

    Tan, W., Du, L., and Buntine, W. (2021). Diversity enhanced active learning with strictly proper scoring rules. Advances in Neural Information Processing Systems , 34:10906--10918

  25. [33]

    Tan, W., Du, L., and Buntine, W. (2023). Bayesian estimate of mean proper scores for diversity-enhanced active learning. IEEE Transactions on Pattern Analysis and Machine Intelligence

  26. [34]

    N., Bischl, B., and Torgo, L

    Vanschoren, J., Van Rijn, J. N., Bischl, B., and Torgo, L. (2014). OpenML : networked science in machine learning. ACM SIGKDD Explorations Newsletter , 15(2):49--60

  27. [35]

    Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA

  28. [36]

    Wimmer, L., Sale, Y., Hofman, P., Bischl, B., and H \"u llermeier, E. (2023). Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence , pages 2282--2292. PMLR

  29. [37]

    Zadeh, L. A. (1978). Fuzzy sets as a basis for a theory of possibility. Fuzzy sets and systems , 1(1):3--28

  30. [38]

    Zhao, G., Dougherty, E., Yoon, B.-J., Alexander, F., and Qian, X. (2021a). Efficient active learning for G aussian process classification by error reduction. Advances in Neural Information Processing Systems , 34:9734--9746

  31. [39]

    Zhao, G., Dougherty, E., Yoon, B.-J., Alexander, F., and Qian, X. (2021b). Uncertainty-aware active learning for optimal B ayesian classifier. In International conference on learning representations (ICLR 2021)

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.