{"id":"184fed82-5e7d-4fd3-9bea-466a84d26aae","arxiv_id":"2602.08142","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Variance-Gated Ensembles define a margin-variance uncertainty score and a differentiable normalization layer that suppress high-variance ensemble predictions at linear cost.","lead":"This paper introduces a way to estimate how unsure a machine-learning model is about each prediction, by combining the model's confidence margin with how much the members of an ensemble disagree. The proposed VGMU score and VGN training layer aim to be faster than current information-theoretic uncertainty measures while retaining similar accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SI S2.2's 'gradient through variance' is ∂s/∂p_m, not ∂L/∂p_m: it omits the upstream ∂L/∂s factor and uses the population-variance identity with the wrong M/(M−1) factor, so the claimed closed-form VJP for VGN is not established.","rationale":"I read the paper in good faith. The VGMU score, the O(MC) complexity argument, and the empirical comparisons are genuine contributions, and the authors include open-source code and report standard deviations across seeds. However, the central VGN contribution depends on the closed-form vector–Jacobian products, and the derivation of the variance path in SI S2.2/Table S2 is internally inconsistent. The reader correctly identified the Bessel-correction issue: the identity v = E[p_m^2] − p̄^2 does not hold for the sample variance with denominator M−1, so the derivative differs by M/(M−1). My concern is stronger: even after fixing that factor, the formula labeled ∂L/∂p_m|_s is only ∂s/∂p_m. It lacks the chain-rule multiplication by ∂L/∂s, which is the actual vector–Jacobian product through the gate and loss. As written, Proposition 4.1's three-term decomposition does not equal the true total gradient. This is not a disagreement with consensus; it is a checkable derivation error in the paper's own formalism. If the released code uses autodiff rather than the printed formulas, the empirical results may still hold, and a corrected derivation would restore the contribution. Consequently, conditional acceptance remains the right verdict, but the derivation must be corrected and the numerical check run before the central claim can be accepted.","tokens_in":44297,"tokens_out":8304,"duration_ms":85891,"concrete_test":"Use the released VGN implementation (or a minimal re-implementation) with M=5, C=3, fixed k=1, and p_m = softmax(random vectors). Choose a simple loss such as L = sum(mean_m q_m) or cross-entropy on qbar. Compute the paper's total ∂L/∂p_m from Table S2, the corrected analytic gradient with the ∂L/∂s factor inserted, and a finite-difference/autograd reference. If the paper's variance-path expression is (p_m−p̄)/(Ms) without the upstream factor, it will not match the reference; if the corrected chain-rule term is used, it should match. This settles whether the VJP claim is a typographical omission or a substantive derivation error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central technical claim is the closed-form vector–Jacobian product that makes VGN trainable through ensemble mean and variance (§4, SI S2). SI S2.2/Table S2 does not substantiate this claim. Proposition S2.2 defines v = 1/(M−1) Σ_m (p_m − p̄)^2 (Bessel-corrected), then uses the identity v = E[p_m^2] − p̄^2 and obtains ∂v/∂p_m = 2(p_m − p̄)/M. The identity holds for the population variance (denominator M), not for the sample variance defined in Eq 7; direct differentiation gives ∂v/∂p_m = 2(p_m − p̄)/(M−1), so the derivation is off by M/(M−1). More importantly, even after that correction, the displayed result (p_m−p̄)/(Ms) in Proposition S2.2 and Table S2 is ∂s/∂p_m, not ∂L/∂p_m. The variance path in the total per-member gradient must be (∂L/∂s) ⊙ ∂s/∂p_m, with ∂L/∂s = ∂L/∂Γ ⊙ ∂Γ/∂s. That upstream loss-gradient factor is absent from Table S2's 'Indirect path via spread.' Without it, the sum in Proposition 4.1 is not the gradient of the loss. Since the entire VGN contribution rests on these VJPs, the derivation as written fails; the reader's Bessel-factor concern is real but masks a larger chain-rule omission.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Variance-Gated Ensembles (VGE), a framework for epistemic-aware uncertainty estimation in classification ensembles. VGE provides two components: (i) VGMU, a margin-based uncertainty score that combines the top-2 class margin with ensemble predictive variance, and (ii) VGN, a differentiable normalization layer that re-weights ensemble member probabilities through an exponential signal-to-noise gate Γ = 1 − exp(−p̄/ks), with per-class sensitivity k learned end-to-end. The authors claim closed-form vector–Jacobian products for end-to-end training through ensemble mean and variance, O(MC) inference cost, and state-of-the-art or better uncertainty ranking and OOD detection. Experiments on MNIST, SVHN, CIFAR-10, and CIFAR-100 compare VGMU against EPJS, EPKL, and entropy-based EU in rank correlation, uncertainty mass concentration (AUCc), OOD detection, and calibration.","tokens_in":44765,"tokens_out":7086,"duration_ms":73296,"significance":"If the technical claims are correct, VGE offers a useful practical contribution: a linear-time epistemic-aware uncertainty score, a trainable gating layer that can suppress high-variance predictions, and an open-source implementation. The paper also has strengths: it reports three-seed runs with deterministic settings, includes a thoughtful discussion of why VGMU diverges from full-simplex disagreement measures on CIFAR-100, and provides supplementary axiomatic analysis. However, the central derivation of the closed-form vector–Jacobian products for VGN training contains a chain-rule error, and the abstract's 'matches or exceeds' claim is contradicted by the paper's own CIFAR-100 results. The framework may still be salvageable, but the derivational and empirical overstatements need to be addressed before the paper can be accepted as is.","major_comments":[{"comment":"Proposition S2.2 defines S = sqrt(1/(M−1) Σ(p_m − p̄)^2) and claims ∂S/∂p_m = (p_m − p̄)/(M S). The proof uses the identity v = E[p_m^2] − p̄^2, which holds for the population variance (denominator M), not for the Bessel-corrected sample variance used in Eq. (7). Direct differentiation gives ∂v/∂p_m = 2(p_m − p̄)/(M−1), hence ∂S/∂p_m = (p_m − p̄)/((M−1)S). The M/(M−1) factor is missing, and this incorrect factor propagates into Table S2's 'Indirect path via spread' expression (p_m − p̄)/(M s).","section":"SI S2.2, Eq. (S2)–(S5)"},{"comment":"Even after correcting the Bessel factor, the displayed quantity (p_m − p̄)/(M s) is ∂s/∂p_m, not ∂L/∂p_m. The variance path in the total per-member gradient must include the upstream loss gradient ∂L/∂s = −∂L/∂Γ ⊙ (1−Γ) p̄/(k s^2), which is correctly listed in Table S2, before multiplying by ∂s/∂p_m. Without that factor, the 'indirect path via spread' in Eq. (29) is not a loss gradient, and the sum in Proposition 4.1 is not the gradient of L. Since the closed-form vector–Jacobian product is the paper's central technical contribution, this chain-rule omission must be fixed and verified against automatic differentiation.","section":"SI S2.2 and Proposition 4.1 (main text)"},{"comment":"The abstract claims VGE 'matches or exceeds state-of-the-art information-theoretic baselines.' Table 2 shows the opposite on CIFAR-100 in all configurations: VGMU AUCc is 0.719 vs EPKL 0.728 for MCD; 0.628 vs EPKL 0.668 for LLE; 0.731 vs EPJS/EPKL 0.723 for MCD-LLE (within variability there). SI S4.2 confirms this pattern across all CIFAR-100 settings. The paper's decision-focused rationale for this gap is reasonable and should be kept, but the blanket headline claim should be qualified to specify where VGMU is comparable or better (CIFAR-10, OOD detection) and where it is intentionally worse on full-simplex disagreement.","section":"Abstract, Table 2, and SI S4.2"}],"minor_comments":[{"comment":"The text says all Jacobians and vector–Jacobian products are derived in full in Supporting Information Section S1, but the derivations appear in Section S2; S1 is the symbol table.","section":"Section 4, last paragraph"},{"comment":"The notation inside the EPKL definition is inconsistent: D_KL(q_i || p_j) is written with p_i on the right-hand side. This should be D_KL(p_i || p_j).","section":"Eq. (24)"},{"comment":"Table 4 marks A5 (invariance to variance-preserving location shifts) as fully satisfied for VGN, but SI S6 shows the property holds only approximately and depends on k (e.g., EU gap 0.029 at k=0, narrowing to 0.006 at k=2). A footnote or partial mark would be more accurate.","section":"Table 4 and SI S6"},{"comment":"The reported wall-clock time of 0.1 µs per sample for LLE-VGN is surprisingly small; please specify whether this includes only the VGN/VGMU computation after logits are available, and report the hardware/software measurement conditions.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The paper's central gradient derivation in SI S2.2 is currently incorrect; this is a load-bearing issue because the closed-form VJP is the basis for end-to-end VGN training. The error appears fixable — the authors need to derive the correct chain rule, including the upstream ∂L/∂s factor, and ideally verify against autodiff. The empirical overclaim in the abstract should also be aligned with the CIFAR-100 results. I would not recommend rejection, as the VGMU concept and the experimental study are potentially useful, but the present manuscript overstates both its theoretical and empirical guarantees."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the VGMU score is a genuinely nice idea — a top-2 margin modulated by ensemble std — and the computational argument for it is sound. The VGN training story, which is the paper's more ambitious claim, rests on a gradient derivation in SI S2.2 that is wrong as written. The sample-variance factor is off by M/(M−1), and more importantly the displayed \"∂L/∂p_m|_s = (p_m − p̄)/(M s)\" is only ∂s/∂p_m, not the loss gradient. The upstream ∂L/∂s factor is missing. So the closed-form VJP for VGN is not established. That is a load-bearing flaw, not a typo.\n\nWhat is new: VGMU is a new combination of BvSB margin and ensemble variance, and the VGN layer is novel even if it is a known-inspiration composition. The paper is honest in the main text — Section 5.2, Table 2 shows VGMU underperforms on CIFAR-100 across all configurations, and the authors discuss this as a design choice. But the abstract's \"matches or exceeds\" overstates those numbers. The axiomatic analysis and the explicit limitations section are thoughtful and give the reader a fair view of scope.\n\nThe empirical work for the score is solid for what it covers: rank correlations, AUCc, OOD ROC, calibration, and a sensitivity analysis for k. The computational speedup is real for inference-time scoring. The VGN empirical benefits are modest and inconsistent; at M=100 VGN degrades OOD AUC, and the paper says so.\n\nWhere it falls: the gradient derivation must be corrected. After fixing the Bessel factor, the chain rule must include ∂L/∂Γ and ∂Γ/∂s. Without that, Proposition 4.1's sum is not the gradient of the loss. The abstract and conclusion should also be aligned with Table 2. The paper would be stronger if the VGN training results were framed as preliminary or if the gradient were verified numerically.\n\nWho this is for: people working on efficient ensemble uncertainty, selective prediction, and OOD detection. The VGMU score and the geometric interpretation are worth reading even if the training layer needs work.\n\nRecommendation: send to peer review. The idea is worth preserving; the derivation error is fixable with a corrected SI or a small revision. A serious referee should require the corrected VJPs and a cleaned-up claim.","headline":"A clever, cheap uncertainty score plus a trainable gate whose stated gradient derivation is currently wrong; worth reviewing after the VGN math is fixed.","tokens_in":45189,"tokens_out":2390,"would_cite":true,"duration_ms":27203,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Variance-Gated Ensembles claim to deliver epistemic uncertainty scores at a fraction of the cost of pairwise divergence methods, with matching or better ranking and out-of-distribution performance.","keywords":["uncertainty estimation","ensembles","epistemic uncertainty","variance gating","margin-based uncertainty","out-of-distribution detection","calibration","deep learning"],"falsifier":"Take M=2 ensemble members and compute the analytical gradient of the standard deviation s with respect to one member's probabilities using the formula in SI S2.2, then compare it with a finite-difference computation. The claimed expression (p_m − p̄)/(M·s) will not match the numerical result; the correct expression is (p_m − p̄)/((M−1)·s) unless the definition of s is changed to population variance.","tokens_in":44209,"feed_emoji":"📊","tokens_out":2472,"duration_ms":30437,"temperature":0.7,"pith_summary":"This paper tries to establish that uncertainty in ensemble classifiers can be estimated from ensemble statistics alone—specifically the per-class predictive mean and variance—rather than from costly pairwise comparisons between ensemble members. The proposed framework, Variance-Gated Ensembles (VGE), introduces a variance gate that suppresses high-variance predictions and produces a margin-based uncertainty score (VGMU) that couples the top-2 class margin with ensemble variance. The paper claims this score matches or exceeds information-theoretic baselines like expected pairwise KL and Jensen-Shannon divergence in ranking quality and OOD detection, while reducing computational cost from O(M²C) to O(MC) for the decomposition and O(C) for the score. If correct, it would make epistemic-aware uncertainty estimation practical for large ensembles and many-class problems, including real-time deployment.","feed_headline":"Variance gate gives ensemble uncertainty at 365x lower cost","feed_subtitle":"Margin-variance score matches pairwise divergence baselines in ranking and OOD detection while scaling linearly in ensemble size.","key_machinery":"The variance gate Γ = 1 − e^{−p̄/(k⊙s)} is the load-bearing object: it maps each class's ensemble mean confidence and predictive spread into a per-class multiplier in [0,1), with learnable per-class sensitivity k. The normalized gated member distribution q_m = (p_m ⊙ Γ)/Z_m and the gated mixture q̄ = (1/M)Σq_m form the differentiable layer; the paper's analytic gradients—the vector–Jacobian products—are what make end-to-end training through ensemble statistics possible. VGMU is a separate inference-time score built on the same gate: VGMU = 1 − (1 − e^{−SNR})p̄₁, where SNR is the margin between the top-2 classes divided by their combined standard deviations.","core_discovery":"The central claim is that an exponential signal-to-noise gate, Γ = 1 − exp(−p̄/ks), computed from the ensemble mean p̄ and standard deviation s, simultaneously provides a trainable normalization layer (VGN) and a decision-focused uncertainty score (VGMU). The gate suppresses classes with high predictive variance and low mean confidence before normalization, and the paper derives closed-form vector–Jacobian products that allow gradients to flow through the ensemble sample mean and variance, enabling end-to-end training. The paper reports that VGMU ranks samples consistently with EPKL/EPJS/mutual-information baselines (Spearman ρ often above 0.98 on CIFAR-10), concentrates uncertainty on diffi","pith_inferences":["The gate's risk-tolerance interpretation (k·s as a standard-deviation scale) suggests a natural extension to abstention policies where the threshold k is set by a desired coverage level, not just learned implicitly.","One could test whether the variance gate also improves uncertainty estimates in regression or segmentation tasks by replacing the categorical simplex with a continuous output distribution and using mean/variance of the target.","The reported insensitivity of VGMU to ensemble configuration implies that practitioners could use a fixed, modest ensemble size and still get reliable OOD signals—worth verifying on other datasets and architectures.","The closed-form gradient derivation relies on a specific definition of ensemble variance; if that definition is adjusted (e.g., Bessel correction), the learned k and resulting behavior could shift, which would need re-benchmarking."],"forward_implications":["If the derivation is correct, uncertainty decomposition and VGMU scoring scale linearly in the number of ensemble members, making them applicable to hundreds of members and hundreds of classes where pairwise divergence is intractable.","The learned per-class k parameter adapts to ensemble diversity, meaning the gate can be tuned end-to-end for task difficulty rather than requiring manual threshold selection.","Because VGMU depends only on the top-2 margin and variance, it is stable to the specific composition of the ensemble (e.g., number of heads vs. dropout samples), which simplifies deployment.","The variance-gated normalization layer can be inserted into any ensemble training pipeline without changing the network architecture, potentially improving calibration and OOD detection at small to moderate ensemble sizes.","The O(C) evaluation cost of VGMU makes per-sample uncertainty cheap enough for real-time selective prediction and human-in-the-loop systems."],"fun_headline_variants":["Signal-to-noise gate sharpens ensemble uncertainty","Variance gate matches baselines at a fraction of compute","Uncertainty in ensembles: variance gate trims cost, keeps accuracy","Variance-gated ensembles: fast epistemic-aware uncertainty"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The closed-form gradients for the variance-gated layer are derived using the identity v = E[p_m²] − p̄² applied to a sample variance defined with Bessel's correction (denominator M−1), but that identity holds for population variance; the resulting gradient expression is therefore off by a factor of (M−1)/M as currently written.","fun_headline_variants_meta":{"raw":{"variants":["Signal-to-noise gate sharpens ensemble uncertainty","Variance gate matches baselines at a fraction of compute","Uncertainty in ensembles: variance gate trims cost, keeps accuracy","Variance-gated ensembles: fast epistemic-aware uncertainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000541,"raw_usage":{"total_tokens":2433,"prompt_tokens":751,"completion_tokens":1682,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1628}},"tokens_in":495,"tokens_out":1682,"duration_ms":14063,"temperature":1.0,"reasoning_tokens":1628,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:21:56.437208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take M=2 ensemble members and compute the analytical gradient of the standard deviation s with respect to one member's probabilities using the formula in SI S2.2, then compare it with a finite-difference computation. The claimed expression (p_m − p̄)/(M·s) will not match the numerical result; the correct expression is (p_m − p̄)/((M−1)·s) unless the definition of s is changed to population variance.","supporting_citations":[],"review_version":1}