REVIEW 4 major objections 6 minor 15 references
Uncertainty Estimation using Variance-Gated Distributions
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper introduces a variance-gated measure that scales ensemble predictions by a confidence factor, decomposes predictive uncertainty, and exposes a collapse in committee-machine diversity.
desk verdict The variance-gating construction is new and worth a careful look, but the paper's own OOD numbers do not support the 'comparable to EPCE/EPKL' claim, and the absence of error bars makes the central empirical comparison unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the variance-gating function $\Gamma_k(y) = 1 - \exp[-\mu(y)/(k\sigma(y) + \epsilon)]$, where $\mu(y)$ is the ensemble mean of a class probability and $\sigma(y)$ is its standard deviation across models. This gate multiplies each class probability to produce a normalized variance-gated distribution $\tilde{p}_{m,k}$. The same gate applied to the margin between the top-1 and top-2 mean predictions defines the variance-gated margin uncertainty $\text{GMU} = 1 - \mu(i)\Gamma(i,j)$. The gate attenuates predictions when the local signal-to-noise ratio is low, which carries the core argument: it gives the user-set $k$ a meaning as a risk threshold, and it ties the uncertainty decomposition to ensemble variance, which is exactly what makes diversity collapse visible.
What would settle it
On an ensemble of trained networks, collect per-class probability vectors for a sample; for each class, compare the empirical fraction of members with probability below $\mu(y) - k\sigma(y)$ to the Gaussian prediction (roughly 16% for $k=1$ and 2.3% for $k=2$). If these empirical fractions systematically diverge from the Gaussian values across many samples on a real benchmark, the gate's $k$-as-fraction interpretation is false.
Extended reading notes
Core claim
The central claim is that uncertainty can be measured by gating class probabilities with a function of the ensemble mean and standard deviation, specifically $\Gamma_k(y) = 1 - \exp[-\mu(y)/(k\sigma(y) + \epsilon)]$, where $\mu(y)$ is the ensemble mean class probability, $\sigma(y)$ is the ensemble standard deviation, and $k$ is a user-set sensitivity parameter. The gated predictive distribution $\tilde{p}_{m,k}$ is obtained by multiplying each member's probability by this gate and renormalizing, and its entropy yields a decomposition into total, aleatoric, and epistemic uncertainty. Experiments on MNIST, SVHN, CIFAR10, and CIFAR100 with Monte Carlo dropout, last-layer ensembles, and hybrids show that the gated measures rank uncertain samples similarly to the pairwise divergence measures of Schweighofer et al., while the variance-gated margin uncertainty (GMU) gives comparable out-of-distribution detection. The paper further claims that because the gate depends on variance, it makes ensemble diversity collapse explicit: when all members converge, the gate saturates at one and increasing $k$ no longer changes the estimates, signaling that the epistemic component has eroded.
Load-bearing premise
The framework assumes that class probabilities across ensemble members are approximately normally distributed, so that $k$ times the standard deviation behaves like a typical deviation and the gate maps to a fraction of disagreeing models; if that distributional assumption fails, the gate's semantics and the user-defined risk interpretation break down.
Editorial extensions
If this is right
- Practitioners get a single hyperparameter $k$ that tunes how aggressively uncertain samples are down-weighted, which can be set to match an acceptable abstention rate.
- The gated decomposition provides an epistemic uncertainty estimate that agrees with pairwise divergence measures on which samples are most uncertain, while requiring no pairwise computation between ensemble members.
- The observed diversity collapse means that in long-trained last-layer ensembles, the effective number of independent models shrinks toward one; uncertainty scores that ignore this will under-report epistemic uncertainty.
- The same gating principle extends to multilabel predictions, where the top-2 margin is replaced by the split between a label being present and absent.
Reading between the lines
- A direct consequence the authors do not pursue is that the gate could be used as a training signal: penalizing low gated confidence could counteract diversity collapse before it finishes.
- The $k$ parameter's risk-threshold semantics rest on the normality assumption; on heavy-tailed or bimodal per-class distributions, $k$ would need recalibration and the gate may be better interpreted as a soft ranker than as a literal fraction of disagreeing models.
- Because the gated measures reduce to the baseline measures when variance is zero, they can be viewed as interpolating between the standard entropy decomposition and a variance-only diagnostic, suggesting a natural test: measure how much of the gated epistemic term is driven by variance versus the base entropy.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes variance-gated distributions for uncertainty estimation in ensemble classifiers. It defines a gating function based on the signal-to-noise ratio of per-class ensemble mean and standard deviation, uses it to rescale per-class probabilities, and applies the standard entropy decomposition to the resulting distribution. It also introduces a variance-gated margin uncertainty (GMU) score based on the top-2 class margin. The central claims are that this framework yields uncertainty estimates comparable to the pairwise information-theoretic measures of Schweighofer et al. and that it provides a tunable, computationally cheap uncertainty score, while also making ensemble diversity collapse explicit. Experiments are presented on MNIST, SVHN, CIFAR10, and CIFAR100 with MCD, LLE, and MCD-LLE ensembles, with OOD detection results reported in the Supporting Information.
Significance. If the claims held, the framework would offer a computationally simple and user-tunable alternative to pairwise divergence-based uncertainty measures, and the diversity-collapse diagnostic would be practically useful. The mathematical definitions are coherent and the Supporting Information is unusually detailed, with transparent formulas and many experimental figures. However, the central empirical claim of comparability is not supported by the paper's own reported numbers: GMU is almost always below TU, EPCE, and EPKL in OOD AUC, and no statistical uncertainty is attached to the point estimates. In addition, the advertised tunability via k is not actually present in the GMU definition as written. The diversity-collapse observation is real but is already noted in the cited literature, and the gated measures converge to the ungated baselines by construction when variance collapses. The paper therefore needs substantial revision before the claimed contributions can be accepted.
major comments (4)
- [Section 3, Eq. (6)] Equation (6) defines GMU = 1 - mu(i)*Gamma(i,j) with Gamma(i,j) = 1 - exp(-(mu(i)-mu(j))/(sigma(i)+sigma(j)+epsilon)), and the sensitivity hyperparameter k does not appear anywhere in this definition. The hyperparameter k enters only the separate SNR decision rule in Eq. (5). Consequently, the abstract's and Section 3's claim that "k lets users set risk tolerance" is not realized by the proposed GMU measure itself; the advertised tunability is absent from the exact quantity that is evaluated in the experiments.
- [Supporting Information, Figs. S2-S9] The reported OOD ROC point estimates point in the opposite direction of the "comparable to Schweighofer et al." claim. For example, in Fig. S2 the GMU total-uncertainty AUC is 0.915 versus 0.934 for TU and 0.938 for EPCE; in Fig. S6 the GMU total AUC is 0.931 versus 0.943 for TU and 0.935 for EPCE; and in Fig. S8 the GMU epistemic AUC is 0.888 versus 0.922 for EPKL. No confidence intervals, repeated-seed statistics, or paired significance tests are reported, so the comparison is not statistically supported, and taken at face value the paper's own data contradict the headline claim of comparability.
- [S1.3 and S2.2, Distributional Assumption] The interpretability claim that k*sigma(y) reflects a "typical deviation" and that k corresponds to a user-specified fraction of disagreeing ensemble members relies on an approximate-normality assumption for the per-class predictive distributions. This assumption is acknowledged but never validated on the actual ensemble distributions used in the experiments. Without such validation, the semantic mapping from k to risk tolerance is not established; the gate is an arbitrary monotone transformation of the SNR, and the stated connection to the fraction of disagreeing models remains unsupported.
- [Section 4, Fig. 1(c)] The diversity-collapse observation is presented as a contribution of the variance-gated framework, but the collapse is quantified by the raw ensemble variance D = E_{i,c}[Var_M], and the gated measures converge to the ungated baselines by construction as sigma tends to zero. The paper itself cites Refs. [4,7-9] for the same phenomenon. The incremental claim should be clarified: the gated measures are not detecting a signal that the baselines cannot, and the diagnostic value beyond the existing literature is not demonstrated by the present experiments.
minor comments (6)
- [Section 1] The phrase "derived from from BMA" contains a duplicated "from".
- [Section 2] The text "This provides a used-defined sensitivity adjustment" should read "user-defined".
- [Section 4] The sentence "Our proposed GMU provided simar results" contains a typo: "simar" should be "similar".
- [Figure 1 caption] The caption for panel (c) refers to "CIFRA10" instead of "CIFAR10".
- [Table S1] For SVHN with LLE, the reported accuracy is 0.992 while the F1-score is 0.911, which appears inconsistent; please verify these numbers.
- [Eq. (4)] The notation E_{w ~ tilde-p_k(w|D)} for the expected aleatoric entropy is confusing, since the expectation should be over the model posterior p(w|D), not over the variance-gated distribution; please clarify the notation.
Circularity Check
No significant circularity: the variance-gated measures are constructed directly from ensemble moments, and no prediction reduces to a fitted input or self-citation chain.
full rationale
The derivation chain is self-contained, and none of the load-bearing claims reduces to its own inputs by construction. The variance-gated distribution in Eq. 3/S4 and GMU in Eq. 6 are closed-form functions of the ensemble mean and standard deviation; no parameter is fitted to an external target, and the user-set hyperparameter k is not calibrated against the benchmark measures. The decomposition in Eq. 4 is the standard entropy/mutual-information identity of Eq. 2 applied to the transformed distribution, so it is an algebraic identity rather than a fitted prediction; the paper transparently presents it as consistent with information-theoretic formulations. The collapse-sensitivity statement that gated estimations converge toward baseline predictions as diversity decreases is a stated limit property of the gate, since Gamma approaches 1 as sigma approaches 0, explicitly derived from the definition rather than an empirical result discovered after the fact. The only self-citation, reference [3], is used to name the committee-machine setting and is not load-bearing for the variance-gating construction. The empirical claim of comparability with Schweighofer et al. is supported only by point estimates that often show GMU slightly below EPCE/EPKL in the OOD ROC figures; that is a correctness and evidence concern, not circularity. No fitted quantity is renamed as a prediction, and no self-citation chain forces the central result.
Assumptions & free parameters
free parameters (1)
- k =
1, 2, 3 (user-set)
assumptions (3)
- domain assumption Bayesian model averaging approximation p(y|x,D) ≈ (1/M) Σ_m p(y|x,w_m) is a valid posterior predictive approximation.
- domain assumption Class probabilities across ensemble members are approximately normally distributed.
- standard math The Shannon-entropy identity TU = AU + EU applies to the gated distribution.
Cite this review
Pith. "Pith review of Uncertainty Estimation using Variance-Gated Distributions." pith.science (2026). https://pith.science/paper/OR4RMS7H
@misc{pith2026250908846,
author = {Pith},
title = {Pith review of: Uncertainty Estimation using Variance-Gated Distributions},
year = {2026},
howpublished = {\url{https://pith.science/paper/OR4RMS7H}},
note = {Machine review of arXiv:2509.08846}
}
read the original abstract
Evaluation of per-sample uncertainty quantification from neural networks is essential for decision-making involving high-risk applications. A common approach is to use the predictive distribution from Bayesian or approximation models and decompose the corresponding predictive uncertainty into epistemic (model-related) and aleatoric (data-related) components. However, additive decomposition has recently been questioned. In this work, we propose an intuitive framework for uncertainty estimation and decomposition based on the signal-to-noise ratio of class probability distributions across different model predictions. We introduce a variance-gated measure that scales predictions by a confidence factor derived from ensembles. We use this measure to discuss the existence of a collapse in the diversity of committee machines.
Figures
Reference graph
Works this paper leans on
-
[1]
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. InInternational Conference on Machine Learning (ICML), 2016. doi: 10.48550/arXiv.1506.02142
-
[2]
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. InConference on Neural Information Processing Systems (NIPS), 2017. doi: 10.48550/arXiv.1612.01474
-
[3]
Martin Gillis, Isaac Xu, Benjamin Misiuk, Craig J
H. Martin Gillis, Isaac Xu, Benjamin Misiuk, Craig J. Brown, and Thomas Trappenberg. Last-layer committee machines for uncertainty estimations of benthic imagery, 2025
work page 2025
-
[4]
Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty
Kajetan Schweighofer, Lukas Aichberger, Mykyta Ielanskyi, and Sepp Hochreiter. Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty. InNeural Information Processing Systems (NeurIPS), Mathematics of Modern Machine Learning Workshop, 2023. doi: 10.48550/arXiv.2311.08309
-
[5]
Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks, 2015
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks, 2015
work page 2015
-
[6]
Lisa Wimmer, Yusuf Sale, Paul Hofman, Bern Bischl, and Eyke Hüllermeier. Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures? InConference on Uncertainty in Artificial Intelligence,
-
[7]
Andreas Kirsch. (Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Models.Transactions on Machine Learning Research, 2025. doi: 10.48550/arXiv.2409.02628
-
[8]
Vardan Papyan, X. Y . Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training.Proceedings of the National Academy of Sciences, 117 (40):24652–24663, 2020. doi: 10.1073/pnas.2015509117
Show all 15 references
-
[9]
Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles, 2024
Sophie Steger, Christian Knoll, Bernhard Klein, Holger Fröning, and Franz Pernkopf. Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles, 2024
2024
-
[10]
Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos
Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class active learning for image classification. InCVPR, pages 2372–2379, 2009. doi: 10.1109/CVPR.2009.5206627
2009
-
[11]
Bayesian Active Learning for Classification and Preference Learning, 2011
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classification and Preference Learning, 2011
2011
-
[12]
Bayesian optimization withskopt, 2016
Gilles Louppe and Manoj Kumar. Bayesian optimization withskopt, 2016
2016
-
[13]
Wide Residual Networks, 2017
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks, 2017. 5 Supporting Information Uncertainty Estimation using Variance-Gated Distributions H. Martin Gillis, Isaac Xu, and Thomas Trappenberg Faculty of Computer Science, Dalhousie University, 6050 University Avenue, ...
2017
-
[15]
and Schweighofer et al. [4]. Epistemic uncertainty (EU): EU ˜pk =TU ˜pk−AU ˜pk (S7) This is the residual model uncertainty after removing ensemble variability with the variance-gated distribution. It captures the reduction of confidence due to ensemble disagreement (i.e., epis...
- [2023]
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.