REVIEW 4 major objections 4 minor 39 references
Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that representing epistemic uncertainty with possibility functions yields two new active-learning acquisition functions that satisfy a desired list of properties and beat standard local baselines on GP classification…
desk verdict New acquisition functions with a solid empirical story, but the central proof of their advertised 'desirable properties' does not hold; the paper deserves a referee's time only if the authors can fix or scope down the theory. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the outer probability measure (OPM), built from a possibility function $f_\theta$ with supremum 1, which assigns credibility $\bar P(A)=\sup_{\theta\in A} f_\theta(\theta)$ and combines with a likelihood through $\bar P(A\times B)=\sup_{\theta\in A}f_\theta(\theta)\int_B p_\theta(y)\,dy$. This yields the uncertainty measures $U^\Theta=\int f_\theta(\theta)\,d\theta$, $U^Y=\int\sup_\theta f_\theta(\theta)p_Y(y|\theta)\,dy-1$, and the necessity $N(E)=1-\bar P(E^c)$. For classification, the possibilistic GP with Gaussian possibility functions and a Laplace approximation produces the same predictive mean and covariance as a standard GP, so the acquisition functions can be evaluated in closed form or with simple numerical optimization.
What would settle it
Take two possibility functions that agree everywhere except on one interval, where the first is higher, and make the second higher on a larger interval outside it so its total integral is smaller; direct integration then contradicts the strict version of Property A4 that the proof relies on. In dimension $d\ge 2$, apply $T(\theta)=\alpha\theta+(1-\alpha)c$ to a Gaussian possibility function and compute $U^\Theta$; the change of variables contributes a Jacobian factor $\alpha^d$, so the stated equality $U^\Theta_{f_T}=\alpha U^\Theta_f$ fails.
Extended reading notes
Core claim
The central claim is that combining probability and possibility theories via outer probability measures gives acquisition functions with desirable axiomatic properties and strong empirical performance in GP-based classification. Specifically, the integral measure $U^L$ and the necessity of correct classification $N$ are local and update-free, satisfy adapted versions of the properties A0--A4 proposed for epistemic-uncertainty measures, and achieve best or second-best accuracy on most of the tested datasets. The possibilistic Gaussian process is the enabling object: it provides closed-form Gaussian posterior possibility functions whose predictive equations coincide with those of standard GPs, allowing the acquisition functions to be implemented for binary and multiclass classification.
Load-bearing premise
The proof of the desirable properties assumes that making a function higher over part of its range always increases its total area, and that a certain rescaling multiplies the uncertainty by exactly $\alpha$; under the paper's own definitions the area claim does not follow from the stated definition and the rescaling claim holds only in one dimension.
Editorial extensions
If this is right
- If the central claim is correct, active learning can target epistemic uncertainty directly without conflating it with irreducible aleatoric uncertainty, using local and update-free rules that are easy to deploy.
- The possibilistic GP gives a principled way to run GP classification while interpreting the posterior as information about a fixed true function rather than as a distribution over random functions.
- The necessity-based acquisition function is computationally light, reported as about 12% slower than standard acquisition functions and 9% faster than BALD, making it attractive when query selection cost matters.
- The proposed measures come with a checklist of axiomatic properties, so future epistemic-uncertainty measures can be compared against the same criteria rather than evaluated only by empirical accuracy.
Reading between the lines
- The proof issue in Property A3 suggests the rescaling property should involve $\alpha^d$, where $d$ is the dimension of the parameter space, rather than $\alpha$; the acquisition functions could still work empirically, but the axiomatic comparison to the Wimmer et al. properties would need reformulation.
- The same probability--possibility decomposition could be carried beyond GP classification, for example to regression or bandit problems, since the paper already derives possibilistic GP regression equations that match the standard ones.
- A natural test is to compare $U^L$ and $N$ against global acquisition functions such as MOCU, BEMPS, and EPIG on the same datasets; the paper only benchmarks against local, update-free methods, so its claim is about that class.
- The closed-form expression for $N_{\text{multi}}$ suggests a very cheap active learning rule for problems with many classes, which would be worth stress-testing on high-dimensional label spaces.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops an active learning framework in which epistemic uncertainty is represented by possibility functions rather than probability distributions. It proposes two acquisition functions: U^L, an integral of a marginal outer probability measure over label space, and 1−N, the complement of the necessity of correct classification. A possibilistic Gaussian process (PGP) is introduced and shown to reproduce standard GP predictive equations. The authors prove axiomatic properties for the acquisition functions and report experiments on 11 real and synthetic datasets where the proposed functions often rank best or second best among local, update-free strategies.
Significance. If the theoretical results were valid, the paper would be a useful contribution: it gives explicit, computationally cheap acquisition functions in a coherent possibility-theoretic framework, states a clear list of axioms A0–A4, derives closed-form expressions for the necessity of correct classification, and provides a broad experimental comparison with rank summaries. The PGP construction is transparent, and the empirical protocol is reproducible in principle. The main advertised contribution, however, is the set of desirable properties in Propositions 1 and 4, and these are not established as stated; the current proofs rely on a definition of 'strictly less informative' that is too weak and on an incorrect Jacobian. The empirical comparison is plausible, but the headline claim of outperformance would benefit from significance testing.
major comments (4)
- [§4.1 / Appendix D, Prop. 1, footnote 1] The proof of Property A4 for U^Y is invalid under the definition of 'strictly less informative' in footnote 1, which only requires f'_θ > f_θ on a positive-volume set A and imposes no constraint on Θ\A. The inequality ∫ sup_θ f'_θ(θ)p(y|θ) dy ≥ ∫ sup_θ f_θ(θ)p(y|θ) dy does not follow, and it is false. For instance, take Θ=[0,1], Y={0,1}, p_Y(1|θ)=θ, f=1 at {0,1} and 1/2 on [0.4,0.6], and f'=1 on [0.4,0.6] and 0 elsewhere. Then f' is strictly less informative than f in the sense of footnote 1, but U^Y_f = 1 and U^Y_{f'} = 0.2. The proof of strict A4 for U^Theta has the same gap: it assumes f'_θ ≥ f_θ on Θ\A, which is not part of the definition. A pointwise domination condition would repair the argument, but it must be stated explicitly.
- [Appendix D, Prop. 1, Property A3] In the multinomial case, T(θ)=αθ+(1−α)c is an affine map on the (m−1)-dimensional simplex. Its Jacobian with respect to Lebesgue measure on the simplex is α^{m−1}, not α. Therefore the change of variables in the proof gives U^Θ_{f_T(θ)} = α^{m−1} U^Θ_{f_θ}, which agrees with the claimed Property A3 only when m=2. Proposition 1's statement that U^Theta satisfies A3 as written is false for multiclass problems.
- [Appendix D, Prop. 4] The proof of strict A4 for 1−Nbin states that the only way a Gaussian possibility function can be strictly less informative than another is by keeping the mean and increasing the variance. This is not true under footnote 1: shifting the mean from μ to μ′>μ makes the shifted Gaussian larger than the original on a neighborhood of μ′, which has positive volume, so the shifted function is 'strictly less informative' by the paper's definition. But, for μ>0, 1−Nbin = N(0;μ,σ²) becomes N(0;μ′,σ²), which is smaller, contradicting strict monotonicity. The proof of A1 in the same proposition also assumes μ_x≠0, so it does not cover a point mass at μ_x=0, where the most likely label is not uniquely defined. Proposition 4 needs a stronger definition of less-informativeness or a restricted statement.
- [§6, Tables 1–2] The claim that the new acquisition functions 'outperform acquisition functions with the same properties' is not supported by a statistical test. On several datasets the best proposed method is within the interquartile range of the Standard baseline (e.g., Breast Cancer: Nbin 0.960 [0.957,0.963] vs Standard 0.957 [0.950,0.960]; Wine: 0.988 [0.986,0.994] vs 0.988 [0.986,0.994]). The average-rank results in Table 2 are more encouraging, but the paper should either provide paired significance tests across runs or temper the wording.
minor comments (4)
- [§6 / Appendix E] The paper states that code is available at '[URL]' and Appendix E says it will be released upon acceptance; a stable repository link should be provided.
- [Appendix E.3] For binary classification the GP hyperparameters are fixed after being computed in advance, while for multiclass they are optimized during training; please state explicitly whether this asymmetry affects both proposed and baseline methods equally at each query.
- [Figure 1] The shaded regions are Q1–Q3 intervals, but the figure caption does not state the number of runs per dataset; the number of runs varies between 50 and 300 in Table 3, so the caption should be made consistent.
- [Appendix E.4] The binary least-confidence strategy is called 'Standard' in the experiments; this should be stated at first use in Section 6 so that readers do not confuse it with a generic standard GP baseline.
Circularity Check
No significant circularity: the acquisition functions are closed-form posterior quantities, the PGP posterior equations are re-derived rather than assumed, and the self-citations provide foundational definitions rather than fitted predictions.
full rationale
The claimed derivation chain is not circular. The two acquisition functions U^L and 1-N are defined as closed-form functionals of Gaussian possibility functions (Sections 4.2 and 4.3), with parameters supplied by a possibilistic Gaussian process posterior; neither quantity is fitted to the accuracy values against which it is later benchmarked. The PGP predictive equations in Propositions 6-8 are derived directly in Appendix B and explicitly recover the standard GP equations, so this is a re-derivation rather than a prediction disguised as a first-principles result. Self-citations to Houssineau (2018), Houssineau et al. (2019), and Houssineau and Bishop (2018) supply the underlying possibility-theoretic definitions, the mode-based expected value, and a Kalman-filter identity; these are stated mathematical frameworks whose assumptions do not include the present active-learning claims, so they function as independent support rather than a circular premise. The empirical evaluation against BALD, entropy, least-confidence, margin, and random baselines on 11 datasets is a separable assessment of the proposed rules. The reader-identified defects, namely the invalid monotonicity step in the proof of Proposition 1(A4) and the Jacobian miscalculation in Proposition 1(A3), are mathematical correctness concerns rather than circular reductions: they make the advertised properties unsupported, but they do not make the derivation equivalent to its inputs by construction.
Assumptions & free parameters
free parameters (1)
- GP kernel hyperparameters (RBF lengthscale, outputscale, noise variance) =
Tuned per dataset via GPytorch (binary: fixed after fitting; multiclass: Adam optimised with 500 iterations)
assumptions (5)
- domain assumption Numerical possibility theory as in De Cooman (2001) and Houssineau (2018): OPMs defined via sup over possibility functions; E* as mode; covariance via Hessian of log possibility.
- domain assumption Posterior possibility function for the latent parameter at input x is a Gaussian possibility N(mu_x, sigma^2_x), obtained via Laplace approximation.
- domain assumption Independence of latent functions across classes in multiclass classification.
- ad hoc to paper The definition of strictly less informative as a possibility function that is larger on some region of positive volume (footnote 1) is the basis for Property A4.
- ad hoc to paper In Property A3, the transformation T on the simplex with Jacobian alpha^(d-1) is treated as scaling the integral by alpha.
invented entities (1)
-
Possibilistic Gaussian process (PGP)
Cite this review
Pith. "Pith review of Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty." pith.science (2026). https://pith.science/paper/YGSE2KBG
@misc{pith2026241208225,
author = {Pith},
title = {Pith review of: Improving Active Learning with a Bayesian Representation of Epistemic Uncertainty},
year = {2026},
howpublished = {\url{https://pith.science/paper/YGSE2KBG}},
note = {Machine review of arXiv:2412.08225}
}
read the original abstract
A popular strategy for active learning is to specifically target a reduction in epistemic uncertainty, since aleatoric uncertainty is often considered as being intrinsic to the system of interest and therefore not reducible. Yet, distinguishing these two types of uncertainty remains challenging and there is no single strategy that consistently outperforms the others. We propose to use a particular combination of probability and possibility theories, with the aim of using the latter to specifically represent epistemic uncertainty, and we show how this combination leads to new active learning strategies that have desirable properties. In order to demonstrate the efficiency of these strategies in non-trivial settings, we introduce the notion of a possibilistic Gaussian process (GP) and consider GP-based multiclass and binary classification problems, for which the proposed methods display a strong performance for both simulated and real datasets.
Figures
Reference graph
Works this paper leans on
-
[1]
Aeberhard, S., Coomans, D., and De Vel, O. (1994). Comparative analysis of statistical pattern recognition methods in high dimensional settings. Pattern Recognition , 27(8):1065--1077
work page 1994
-
[2]
Caprio, M., Sale, Y., H \"u llermeier, E., and Lee, I. (2023). A novel B ayes’ theorem for upper probabilities. In International Workshop on Epistemic Uncertainty in Artificial Intelligence , pages 1--12. Springer
work page 2023
-
[3]
Chen, Z., Ristic, B., Houssineau, J., and Kim, D. Y. (2021). Observer control for bearings-only tracking using possibility functions. Automatica , 133:109888
work page 2021
-
[4]
Dawid, A. P., Stone, M., and Zidek, J. V. (1973). Marginalization paradoxes in bayesian and structural inference. Journal of the Royal Statistical Society Series B: Statistical Methodology , 35(2):189--213
work page 1973
-
[5]
De Cooman, G. (2001). Integration and conditioning in numerical possibility theory. Annals of Mathematics and Artificial Intelligence , 32:87--123
work page 2001
-
[6]
Den ux, T. (2023). Reasoning with fuzzy and uncertain evidence using epistemic random fuzzy sets: General framework and practical models. Fuzzy Sets and Systems , 453:1--36
work page 2023
-
[7]
Dubois, D., Nguyen, H. T., and Prade, H. (2000). Possibility theory, probability and fuzzy sets misunderstandings, bridges and gaps. In Fundamentals of fuzzy sets , pages 343--438. Springer
work page 2000
-
[8]
Dubois, D. and Prade, H. (2015). Possibility theory and its applications: Where do we stand? Springer handbook of computational intelligence , pages 31--60
work page 2015
Show all 39 references
-
[9]
Fisher, R. A. (1935). The fiducial argument in statistical inference. Annals of eugenics , 6(4):391--398
1935
-
[10]
Fisher, R. A. (1988). Iris . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C56C76
1988 doi
-
[11]
Friedman, J. H. (2002). Stochastic gradient boosting. Computational statistics & data analysis , 38(4):367--378
2002
-
[12]
Q., Bindel, D., and Wilson, A
Gardner, J., Pleiss, G., Weinberger, K. Q., Bindel, D., and Wilson, A. G. (2018). Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in neural information processing systems , 31
2018
-
[13]
Gorman, R. P. and Sejnowski, T. J. (1988). Analysis of hidden units in a layered network trained to classify sonar targets. Neural networks , 1(1):75--89
1988
-
[14]
Houlsby, N., Husz \'a r, F., Ghahramani, Z., and Lengyel, M. (2011). Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745
2011 arXiv
-
[15]
Houssineau, J. (2018). Parameter estimation with a class of outer probability measures. arXiv preprint arXiv:1801.00569
2018 arXiv
-
[16]
Houssineau, J. (2021). A linear algorithm for multi-target tracking in the context of possibility theory. IEEE Transactions on Signal Processing , 69:2740--2751
2021
-
[17]
and Bishop, A
Houssineau, J. and Bishop, A. N. (2018). Smoothing and filtering with a class of outer measures. SIAM/ASA Journal on Uncertainty Quantification , 6(2):845--866
2018
-
[18]
Houssineau, J., Chada, N., and Delande, E. (2019). Elements of asymptotic theory with outer probability measures. arXiv preprint arXiv:1908.04331
2019 arXiv
-
[19]
and Waegeman, W
H \"u llermeier, E. and Waegeman, W. (2021). Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning , 110(3):457--506
2021
-
[20]
MacKay, D. J. (1992). Information-based objective functions for active data selection. Neural computation , 4(4):590--604
1992
-
[21]
Mangasarian, O. L. and Wolberg, W. H. (1990). Cancer diagnosis via linear programming. Technical report, University of Wisconsin-Madison Department of Computer Sciences
1990
-
[22]
Nguyen, V.-L., Destercke, S., and H \"u llermeier, E. (2019). Epistemic uncertainty sampling. In Discovery Science: 22nd International Conference, DS 2019, Split, Croatia, October 28--30, 2019, Proceedings 22 , pages 72--86. Springer
2019
-
[23]
H., and H \"u llermeier, E
Nguyen, V.-L., Shaker, M. H., and H \"u llermeier, E. (2022). How to measure uncertainty in uncertainty sampling for active learning. Machine Learning , 111(1):89--122
2022
-
[24]
Quinlan, R. (1987). Thyroid Disease . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C5D010
1987 doi
-
[25]
Rodrigues, F., Pereira, F., and Ribeiro, B. (2014). Gaussian process classification and active learning with multiple annotators. In International conference on machine learning , pages 433--441. PMLR
2014
-
[26]
Sale, Y., Bengs, V., Caprio, M., and H \"u llermeier, E. (2023). Second-order uncertainty quantification: A distance-based approach. In Forty-first International Conference on Machine Learning
2023
-
[27]
and Wynn, H
Sebastiani, P. and Wynn, H. P. (2000). Maximum entropy sampling and optimal B ayesian experimental design. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 62(1):145--157
2000
-
[28]
Siebert, J. P. (1987). Vehicle recognition using rule based methods . Turing Institute
1987
-
[29]
G., Wing, S
Sigillito, V. G., Wing, S. P., Hutton, L. V., and Baker, K. B. (1989). Classification of radar returns from the ionosphere using neural networks. Johns Hopkins APL Technical Digest , 10(3):262--266
1989
-
[30]
and Wrobel, L
Sikora, M. and Wrobel, L. (2013). seismic-bumps . UCI Machine Learning Repository. DOI : https://doi.org/10.24432/C5W902
2013 doi
-
[31]
B., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T
Smith, F. B., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T. (2023). Prediction-oriented bayesian active learning. In International Conference on Artificial Intelligence and Statistics , pages 7331--7348. PMLR
2023
-
[32]
Tan, W., Du, L., and Buntine, W. (2021). Diversity enhanced active learning with strictly proper scoring rules. Advances in Neural Information Processing Systems , 34:10906--10918
2021
-
[33]
Tan, W., Du, L., and Buntine, W. (2023). Bayesian estimate of mean proper scores for diversity-enhanced active learning. IEEE Transactions on Pattern Analysis and Machine Intelligence
2023
-
[34]
N., Bischl, B., and Torgo, L
Vanschoren, J., Van Rijn, J. N., Bischl, B., and Torgo, L. (2014). OpenML : networked science in machine learning. ACM SIGKDD Explorations Newsletter , 15(2):49--60
2014
-
[35]
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning , volume 2. MIT press Cambridge, MA
2006
-
[36]
Wimmer, L., Sale, Y., Hofman, P., Bischl, B., and H \"u llermeier, E. (2023). Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence , pages 2282--2292. PMLR
2023
-
[37]
Zadeh, L. A. (1978). Fuzzy sets as a basis for a theory of possibility. Fuzzy sets and systems , 1(1):3--28
1978
-
[38]
Zhao, G., Dougherty, E., Yoon, B.-J., Alexander, F., and Qian, X. (2021a). Efficient active learning for G aussian process classification by error reduction. Advances in Neural Information Processing Systems , 34:9734--9746
2021
-
[39]
Zhao, G., Dougherty, E., Yoon, B.-J., Alexander, F., and Qian, X. (2021b). Uncertainty-aware active learning for optimal B ayesian classifier. In International conference on learning representations (ICLR 2021)
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.