{"id":"d90437e8-3081-4f80-85d5-4c8ed9d44610","arxiv_id":"2509.08846","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A variance-gated uncertainty measure built from ensemble mean and variance is introduced, and the paper reports that last-layer ensembles collapse in diversity during extended training.","lead":"This paper proposes a new way to measure how uncertain a neural network is about each prediction, using the spread of answers across an ensemble of models. It also reports that last-layer ensembles tend to lose diversity during training, which makes disagreement-based uncertainty estimates weaken.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported OOD results show GMU consistently below EPCE/EPKL, so the 'comparable to Schweighofer et al.' claim is not supported by the paper's own numbers; no error bars are given.","rationale":"I read the paper as proposing a cheap, tunable uncertainty score and making a preliminary empirical claim of parity with pairwise information-theoretic measures. The main text shows sorted-score curves and claims agreement on the most uncertain samples, but the only quantitative OOD results are in the SI. Those numbers consistently show GMU and the gated variants below the baselines. The reader's weakest assumption (normality for k semantics) is a legitimate concern but less central: the numerical uncertainty estimates do not depend on the exact probabilistic interpretation of k, and even if the normality assumption is false, GMU could still rank uncertainty well. The comparability claim, by contrast, is directly tested by the reported ROC numbers and is not demonstrated by them. The absence of error bars is not just a presentation issue; it is the difference between 'comparable' and 'not comparable'. My recommendation is unchanged conditional acceptance: the authors should provide repeated-seed statistics and direct paired comparisons, and they should either substantiate or soften the comparability claim.","tokens_in":23664,"tokens_out":13046,"duration_ms":121464,"concrete_test":"Re-run the OOD detection experiments for both directions (SVHN-to-CIFAR10 and CIFAR10-to-SVHN) using at least 5 independent seeds of the full training pipeline, and report mean plus 95% confidence intervals for AUC and FPR@95TPR for GMU, TU, AU, EU, EPCE, EPKL, and EPJS. Apply a paired DeLong test or bootstrap for correlated ROC curves to compare GMU against EPCE and EPKL in each configuration. If the confidence intervals for GMU exclude the baselines in the direction of worse performance in most settings, the 'comparable' claim should be retracted or replaced by a claim of comparable computational cost with slightly lower accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the variance-gated GMU and gated entropy decompositions yield uncertainty estimates comparable to the pairwise information-theoretic measures of Schweighofer et al. (EPCE, EPKL, EPJS). The only quantitative evidence for this is the OOD ROC analysis in the Supporting Information, and the displayed point estimates point the other way. In Fig. S2 (ID=SVHN, OOD=CIFAR10, MCD), GMU has AUC 0.915 vs EPCE 0.938, EPKL 0.939, and EU 0.941. In Fig. S6 (ID=CIFAR10, OOD=SVHN, MCD), GMU has 0.931 vs TU 0.943, AU 0.949, and EPCE 0.935. In Fig. S8 (LLE), GMU is 0.888 vs EU 0.916 and EPKL 0.922. The same pattern repeats across MCD-LLE and calibrated LLE figures: gated variants are never better than their ungated counterparts, and GMU is usually 1-3 AUC points below EPCE/EPKL. No repeated-seed statistics, confidence intervals, or paired significance tests are reported, so one cannot determine whether these gaps are noise; but if the point estimates are taken at face value, the paper's own data contradict the 'comparable' claim. The claim is load-bearing because the motivation of the method is to offer a cheaper alternative that is not worse; if GMU is systematically worse, the contribution reduces to a diversity-collapse observation already present in the cited literature. The k-semantics issue (normality assumption) is secondary: it affects interpretability, whereas the OOD numbers directly test the headline equivalence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes variance-gated distributions for uncertainty estimation in ensemble classifiers. It defines a gating function based on the signal-to-noise ratio of per-class ensemble mean and standard deviation, uses it to rescale per-class probabilities, and applies the standard entropy decomposition to the resulting distribution. It also introduces a variance-gated margin uncertainty (GMU) score based on the top-2 class margin. The central claims are that this framework yields uncertainty estimates comparable to the pairwise information-theoretic measures of Schweighofer et al. and that it provides a tunable, computationally cheap uncertainty score, while also making ensemble diversity collapse explicit. Experiments are presented on MNIST, SVHN, CIFAR10, and CIFAR100 with MCD, LLE, and MCD-LLE ensembles, with OOD detection results reported in the Supporting Information.","tokens_in":23972,"tokens_out":6436,"duration_ms":55156,"significance":"If the claims held, the framework would offer a computationally simple and user-tunable alternative to pairwise divergence-based uncertainty measures, and the diversity-collapse diagnostic would be practically useful. The mathematical definitions are coherent and the Supporting Information is unusually detailed, with transparent formulas and many experimental figures. However, the central empirical claim of comparability is not supported by the paper's own reported numbers: GMU is almost always below TU, EPCE, and EPKL in OOD AUC, and no statistical uncertainty is attached to the point estimates. In addition, the advertised tunability via k is not actually present in the GMU definition as written. The diversity-collapse observation is real but is already noted in the cited literature, and the gated measures converge to the ungated baselines by construction when variance collapses. The paper therefore needs substantial revision before the claimed contributions can be accepted.","major_comments":[{"comment":"Equation (6) defines GMU = 1 - mu(i)*Gamma(i,j) with Gamma(i,j) = 1 - exp(-(mu(i)-mu(j))/(sigma(i)+sigma(j)+epsilon)), and the sensitivity hyperparameter k does not appear anywhere in this definition. The hyperparameter k enters only the separate SNR decision rule in Eq. (5). Consequently, the abstract's and Section 3's claim that \"k lets users set risk tolerance\" is not realized by the proposed GMU measure itself; the advertised tunability is absent from the exact quantity that is evaluated in the experiments.","section":"Section 3, Eq. (6)"},{"comment":"The reported OOD ROC point estimates point in the opposite direction of the \"comparable to Schweighofer et al.\" claim. For example, in Fig. S2 the GMU total-uncertainty AUC is 0.915 versus 0.934 for TU and 0.938 for EPCE; in Fig. S6 the GMU total AUC is 0.931 versus 0.943 for TU and 0.935 for EPCE; and in Fig. S8 the GMU epistemic AUC is 0.888 versus 0.922 for EPKL. No confidence intervals, repeated-seed statistics, or paired significance tests are reported, so the comparison is not statistically supported, and taken at face value the paper's own data contradict the headline claim of comparability.","section":"Supporting Information, Figs. S2-S9"},{"comment":"The interpretability claim that k*sigma(y) reflects a \"typical deviation\" and that k corresponds to a user-specified fraction of disagreeing ensemble members relies on an approximate-normality assumption for the per-class predictive distributions. This assumption is acknowledged but never validated on the actual ensemble distributions used in the experiments. Without such validation, the semantic mapping from k to risk tolerance is not established; the gate is an arbitrary monotone transformation of the SNR, and the stated connection to the fraction of disagreeing models remains unsupported.","section":"S1.3 and S2.2, Distributional Assumption"},{"comment":"The diversity-collapse observation is presented as a contribution of the variance-gated framework, but the collapse is quantified by the raw ensemble variance D = E_{i,c}[Var_M], and the gated measures converge to the ungated baselines by construction as sigma tends to zero. The paper itself cites Refs. [4,7-9] for the same phenomenon. The incremental claim should be clarified: the gated measures are not detecting a signal that the baselines cannot, and the diagnostic value beyond the existing literature is not demonstrated by the present experiments.","section":"Section 4, Fig. 1(c)"}],"minor_comments":[{"comment":"The phrase \"derived from from BMA\" contains a duplicated \"from\".","section":"Section 1"},{"comment":"The text \"This provides a used-defined sensitivity adjustment\" should read \"user-defined\".","section":"Section 2"},{"comment":"The sentence \"Our proposed GMU provided simar results\" contains a typo: \"simar\" should be \"similar\".","section":"Section 4"},{"comment":"The caption for panel (c) refers to \"CIFRA10\" instead of \"CIFAR10\".","section":"Figure 1 caption"},{"comment":"For SVHN with LLE, the reported accuracy is 0.992 while the F1-score is 0.911, which appears inconsistent; please verify these numbers.","section":"Table S1"},{"comment":"The notation E_{w ~ tilde-p_k(w|D)} for the expected aleatoric entropy is confusing, since the expectation should be over the model posterior p(w|D), not over the variance-gated distribution; please clarify the notation.","section":"Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's own OOD numbers undercut the central comparability claim, and the absence of k in the Eq. (6) definition of GMU is a concrete error that needs fixing. The paper could be salvageable if the authors add k to the GMU definition (or remove the tunability claim), provide a proper statistical comparison with confidence intervals, and reframe the diversity-collapse discussion as a known issue that their measure illustrates rather than a new discovery. As it stands, the gap between the advertised claims and the reported evidence is too large for acceptance in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: the paper introduces a genuinely new variance-gating mechanism for ensemble uncertainty, but its central empirical claim—that the resulting measures are comparable to Schweighofer et al.'s pairwise information-theoretic baselines—is undercut by the paper's own reported OOD ROC values. GMU is consistently 1–3 AUC points below EPCE/EPKL in the epistemic and total uncertainty rows across the SVHN and CIFAR10 pairs. No error bars, repeated seeds, or significance tests are reported, so these gaps might be noise, but as presented the data point the other way.\n\nWhat is new: the variance-gating function Γ_k and the GMU margin score are not in the cited literature. The decomposition in Eq. 4 is the standard Shannon entropy identity applied to a reweighted distribution, which is a modest contribution, but the reweighting itself is a coherent idea with a clear limit behavior. The paper also uses the gated framework as a diagnostic for diversity collapse in last-layer ensembles. That collapse is already known—the authors cite [4,7-9]—so the diagnostic framing is a small addition, not a discovery.\n\nThe math is coherent. The gate is bounded, monotone, and the limit cases behave as described. The normality assumption underlying the k-as-risk-threshold semantics is stated explicitly in S1.3, and it is a real limitation, but it is secondary to the empirical problem: even if the semantics hold, the reported OOD numbers still fail to show comparability.\n\nThe experiments are reasonably thorough for a preprint: four datasets, three ensemble strategies, calibration, and OOD curves. But the missing error bars are not a minor omission; they are load-bearing for the headline claim. The paper would be stronger if it presented repeated-seed statistics, paired significance tests, and a direct comparison to BvSB, which is the natural baseline for a margin-based method but is never included.\n\nMy bottom line: this is not a rejection-worthy paper. It contains a novel construction, honest statements about its preliminary nature, and a useful diagnostic observation. The empirical support needs real work, and the abstract's 'comparable' wording should be softened until the numbers agree. I would send it to peer review with a request for repeated-seed statistics and a clearer account of where the gated measures beat baselines. A serious referee would find the ideas worth engaging with.","headline":"The variance-gating construction is new and worth a careful look, but the paper's own OOD numbers do not support the 'comparable to EPCE/EPKL' claim, and the absence of error bars makes the central empirical comparison unverifiable.","tokens_in":24530,"tokens_out":1366,"would_cite":false,"duration_ms":14041,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a variance-gated measure that scales ensemble predictions by a confidence factor, decomposes predictive uncertainty, and exposes a collapse in committee-machine diversity.","keywords":["uncertainty quantification","variance-gated distributions","epistemic uncertainty","aleatoric uncertainty","ensemble diversity collapse","committee machines","signal-to-noise ratio","margin uncertainty"],"falsifier":"On an ensemble of trained networks, collect per-class probability vectors for a sample; for each class, compare the empirical fraction of members with probability below $\\mu(y) - k\\sigma(y)$ to the Gaussian prediction (roughly 16% for $k=1$ and 2.3% for $k=2$). If these empirical fractions systematically diverge from the Gaussian values across many samples on a real benchmark, the gate's $k$-as-fraction interpretation is false.","tokens_in":23422,"feed_emoji":"🎛️","tokens_out":5008,"duration_ms":43248,"temperature":0.7,"pith_summary":"This paper tries to establish a new way to quantify predictive uncertainty that avoids the criticized additive entropy decomposition. The idea is to compare the mean and the variance of class probabilities across ensemble members, treating them as signal and noise, and to scale down predictions whose signal-to-noise ratio is low using a gate controlled by a user-set hyperparameter $k$. The authors claim that the resulting gated entropy and the variance-gated margin uncertainty reproduce the rankings of state-of-the-art pairwise information-theoretic measures while adding a risk-tolerance knob. If true, the framework offers a computationally cheap, interpretable uncertainty score and a diagnostic for a newly highlighted failure mode: after long training, last-layer ensembles converge toward a single model, so their epistemic uncertainty evaporates.","feed_headline":"A variance gate makes ensemble uncertainty user-tunable.","feed_subtitle":"Gated entropy and margin scores match pairwise divergence baselines and expose ensemble diversity collapse.","key_machinery":"The load-bearing object is the variance-gating function $\\Gamma_k(y) = 1 - \\exp[-\\mu(y)/(k\\sigma(y) + \\epsilon)]$, where $\\mu(y)$ is the ensemble mean of a class probability and $\\sigma(y)$ is its standard deviation across models. This gate multiplies each class probability to produce a normalized variance-gated distribution $\\tilde{p}_{m,k}$. The same gate applied to the margin between the top-1 and top-2 mean predictions defines the variance-gated margin uncertainty $\\text{GMU} = 1 - \\mu(i)\\Gamma(i,j)$. The gate attenuates predictions when the local signal-to-noise ratio is low, which carries the core argument: it gives the user-set $k$ a meaning as a risk threshold, and it ties the uncertainty decomposition to ensemble variance, which is exactly what makes diversity collapse visible.","core_discovery":"The central claim is that uncertainty can be measured by gating class probabilities with a function of the ensemble mean and standard deviation, specifically $\\Gamma_k(y) = 1 - \\exp[-\\mu(y)/(k\\sigma(y) + \\epsilon)]$, where $\\mu(y)$ is the ensemble mean class probability, $\\sigma(y)$ is the ensemble standard deviation, and $k$ is a user-set sensitivity parameter. The gated predictive distribution $\\tilde{p}_{m,k}$ is obtained by multiplying each member's probability by this gate and renormalizing, and its entropy yields a decomposition into total, aleatoric, and epistemic uncertainty. Experiments on MNIST, SVHN, CIFAR10, and CIFAR100 with Monte Carlo dropout, last-layer ensembles, and hybrids show that the gated measures rank uncertain samples similarly to the pairwise divergence measures of Schweighofer et al., while the variance-gated margin uncertainty (GMU) gives comparable out-of-distribution detection. The paper further claims that because the gate depends on variance, it makes ensemble diversity collapse explicit: when all members converge, the gate saturates at one and increasing $k$ no longer changes the estimates, signaling that the epistemic component has eroded.","pith_inferences":["A direct consequence the authors do not pursue is that the gate could be used as a training signal: penalizing low gated confidence could counteract diversity collapse before it finishes.","The $k$ parameter's risk-threshold semantics rest on the normality assumption; on heavy-tailed or bimodal per-class distributions, $k$ would need recalibration and the gate may be better interpreted as a soft ranker than as a literal fraction of disagreeing models.","Because the gated measures reduce to the baseline measures when variance is zero, they can be viewed as interpolating between the standard entropy decomposition and a variance-only diagnostic, suggesting a natural test: measure how much of the gated epistemic term is driven by variance versus the base entropy."],"forward_implications":["Practitioners get a single hyperparameter $k$ that tunes how aggressively uncertain samples are down-weighted, which can be set to match an acceptable abstention rate.","The gated decomposition provides an epistemic uncertainty estimate that agrees with pairwise divergence measures on which samples are most uncertain, while requiring no pairwise computation between ensemble members.","The observed diversity collapse means that in long-trained last-layer ensembles, the effective number of independent models shrinks toward one; uncertainty scores that ignore this will under-report epistemic uncertainty.","The same gating principle extends to multilabel predictions, where the top-2 margin is replaced by the split between a label being present and absent."],"supporting_citations":[{"why":"Supplies the Monte Carlo dropout baseline used to build the ensembles whose means and variances feed the gating function.","marker":"[1]"},{"why":"Supplies deep ensembles as the core committee setting and motivates variance-based predictive uncertainty.","marker":"[2]"},{"why":"Defines the last-layer committee machine architecture used throughout the experiments.","marker":"[3]"},{"why":"Introduces the pairwise cross-entropy and KL divergence measures that the variance-gated method is compared against and claims to match.","marker":"[4]"},{"why":"Criticizes the additive entropy decomposition of total uncertainty, providing the motivation for a variance-gated alternative.","marker":"[6]"},{"why":"Documents epistemic uncertainty collapse in large models, which the paper's diversity-collapse observation extends to last-layer ensembles.","marker":"[7]"},{"why":"Describes neural collapse during the terminal phase of training, providing a related phenomenon that the diversity collapse is compared to.","marker":"[8]"},{"why":"Provides the best-versus-second-best margin idea that the variance-gated margin uncertainty (GMU) extends with variance information.","marker":"[10]"}],"fun_headline_variants":["Variance gating exposes ensemble diversity collapse","User-tunable uncertainty from variance-gated ensembles","A variance gate refines epistemic-aleatoric decomposition","Gated probability distribution for sharper uncertainty","Ensemble diversity collapse made visible by variance gating"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that class probabilities across ensemble members are approximately normally distributed, so that $k$ times the standard deviation behaves like a typical deviation and the gate maps to a fraction of disagreeing models; if that distributional assumption fails, the gate's semantics and the user-defined risk interpretation break down.","fun_headline_variants_meta":{"raw":{"variants":["Variance gating exposes ensemble diversity collapse","User-tunable uncertainty from variance-gated ensembles","A variance gate refines epistemic-aleatoric decomposition","Gated probability distribution for sharper uncertainty","Ensemble diversity collapse made visible by variance gating"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1580,"prompt_tokens":880,"completion_tokens":700,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":628}},"tokens_in":496,"tokens_out":700,"duration_ms":6532,"temperature":1.0,"reasoning_tokens":628,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:18:17.654885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On an ensemble of trained networks, collect per-class probability vectors for a sample; for each class, compare the empirical fraction of members with probability below $\\mu(y) - k\\sigma(y)$ to the Gaussian prediction (roughly 16% for $k=1$ and 2.3% for $k=2$). If these empirical fractions systematically diverge from the Gaussian values across many samples on a real benchmark, the gate's $k$-as-fraction interpretation is false.","supporting_citations":[{"cited_title":"Martin Gillis, Isaac Xu, Benjamin Misiuk, Craig J","cited_arxiv_id":null,"evidence_quote":"Defines the last-layer committee machine architecture used throughout the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Criticizes the additive entropy decomposition of total uncertainty, providing the motivation for a variance-gated alternative."}],"review_version":1}