Pith. sign in

REVIEW 5 major objections 9 minor 17 references

A single temperature knob turns shared-backbone ensembles into deep-ensemble competitors at a fraction of the parameter cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 14:03 UTC pith:ZYTUUS4X

load-bearing objection Solid methods paper with a real training-time diversity knob; the abstract oversells deep-ensemble wins because the marquee WRN/CIFAR-100 DE baseline looks broken. the 5 major comments →

arxiv 2607.23860 v1 pith:ZYTUUS4X submitted 2026-07-26 cs.LG cs.CV

Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation

Mihai Suteu , Ovidiu Serban This is my paper
classification cs.LG cs.CV
keywords implicit ensemblesuncertainty quantificationnormalisation layersmember diversitysoftmax temperaturemodulation uncertaintycalibration under shiftpretrained adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Deep ensembles give strong uncertainty estimates but cost scales with every extra member. Implicit ensembles share one backbone to cut that cost, yet until now none could set how much members disagree during training—diversity was fixed at startup or baked into the architecture. This paper introduces σN-Ens: members share every convolution or attention weight and differ only through sigmoid-bounded scales on normalisation layers, treated like tasks in a multi-task setup. A softmax-temperature regulariser pulls those scales toward a chosen sharing equilibrium, so one hyperparameter traces the accuracy–calibration tradeoff. Because only normalisation is replicated, the same recipe wraps ResNets and transformers, and can turn a pretrained checkpoint into a calibrated ensemble with a short fine-tune. Across CIFAR, ImageNet, and SST-2 the method matches or beats full deep ensembles on accuracy and likelihood while using far fewer parameters, keeps scaling as members are added where hard-partition methods collapse, and holds calibration under input corruption—while explaining why out-of-distribution detection stays weaker.

Core claim

Member diversity in a normalisation-based implicit ensemble can be controlled during training by a softmax-temperature regulariser on sigmoid-bounded per-feature scales. That single temperature selects the equilibrium level of feature sharing, traces an accuracy–calibration frontier, and yields ensembles that match or outperform independently trained deep ensembles at a fraction of the parameter count across convolutional and transformer backbones.

What carries the argument

Softmax-temperature regularisation on σNorm importance vectors: for each feature, members’ sigmoid scales are softmaxed across the ensemble at temperature τ; minimising the resulting penalty drives features toward a τ-chosen soft M-choose-k ownership pattern, holding diversity at a chosen equilibrium throughout training rather than fixing it at initialisation or architecture design.

Load-bearing premise

The paper assumes that disagreement from reweighting one shared backbone is enough for calibrated uncertainty on corrupted but still-representable inputs, while the disagreement that would come from different feature extractors is what mainly drives out-of-distribution detection.

What would settle it

Train σN-Ens and matched deep ensembles on the same backbone and data, then measure calibration (ECE) across corruption severities and near/far OOD detection: if σN-Ens’s ECE collapses under moderate corruption while OOD detection matches or exceeds the deep ensemble, the modulation-uncertainty split and its predictions fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can convert a single pretrained model into a calibrated multi-member ensemble by swapping normalisation layers and fine-tuning briefly, without retraining full members from scratch.
  • Ensemble size can be increased without hard channel partitioning, so accuracy and likelihood improve or saturate instead of collapsing when members multiply.
  • One temperature τ becomes the practical control for navigating the accuracy–calibration frontier to match an application’s risk tolerance.
  • Calibration under input corruption can be maintained by design in shared-backbone ensembles even when full backbone diversity is forgone.
  • The same normalisation-only wrap applies unchanged to both convolutional and transformer families.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If modulation uncertainty is the dominant usable epistemic signal in large pretrained models, many deployment settings may not need full independent backbones for calibration—only for hard OOD gates.
  • Pairing controllable normalisation diversity with a small amount of member-specific feature capacity (for example light adapters) is a direct route to recover the forgone OOD component without returning to linear parameter cost.
  • A label-free parameter-space diversity metric that tracks output disagreement could be monitored online during training as a cheap early-stopping or τ-selection signal in other multi-head systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 9 minor

Summary. The manuscript introduces σN-Ens, an implicit ensemble in which M members share a backbone and differ only in per-normalisation-layer sigmoid-bounded scale vectors (σNorm, building on the authors' σBN/TSσBN primitives), plus a softmax-temperature regulariser L_div (Eq. 2) that pushes each feature's across-member importance distribution toward a τ-parametrised equilibrium. A single temperature τ is claimed to select the operating point on the accuracy–calibration frontier, with λ only setting the strength of the pull. The paper further defines "modulation uncertainty" (§3.4, Eq. 6) as the conditional-mutual-information component a shared-backbone ensemble expresses, and predicts — then confirms — graceful calibration degradation under corruption and weaker OOD detection. Experiments cover ResNets/WideResNet on CIFAR-10/100 (Packed Ensembles protocol), ViT-B/16 on CIFAR-100/ImageNet, and BERT on SST-2, with ablations over M (2–16), λ (three orders of magnitude), τ frontiers, parameter-vs-output diversity correlations, and CIFAR-100-C/ImageNet-C corruption curves.

Significance. If the results hold, this is a useful contribution: to my knowledge it is the first implicit-ensemble method with an explicit, single-knob control over member diversity during training, and the τ/λ separation (equilibrium vs. strength) is both conceptually clean and empirically supported (Table 4). The derivation in Appendix A.5 that the penalty's attractors are k-owner corners — interpolating between hard partitioning (k=1) and full sharing — is a genuinely parameter-free piece of analysis that unifies partitioning and modulation methods on one axis. The output-preserving conversion of pretrained backbones (Appendix A.3) is practically valuable and the pretrained ViT calibration results (Table 3, Fig. 3) are strong and fairly reported, including the acknowledged OOD weakness. The member-scaling study (Fig. 4, Table 7) is a well-designed falsifiable contrast against Packed Ensembles, and the parameter-space/output-space diversity correspondence (Table 6, |ρ| ≥ 0.83) supports the monitoring claim. The evaluation is multi-architecture, multi-regime, with seed statistics on the transformer runs — above the field's typical standard.

major comments (5)
  1. [Table 1 / §4.4] Table 1, WRN-28-10/CIFAR-100 block: the deep-ensemble row (Acc 82.5, NLL 0.903, ECE 22.9, AUROC 67.9) is transcribed from Laurent et al. (2023) and is anomalous — the deep ensemble is worse calibrated than the Single model (ECE 22.9 vs 15.6) and barely better on AUROC (67.9 vs 64.2), contradicting well-replicated deep-ensemble behaviour (Lakshminarayanan et al. 2017; Ovadia et al. 2019) and every other ensemble row in the block. This cell is load-bearing: it is the only convolutional setting where σN-Ens clearly dominates deep ensembles, and §4.4 builds the capacity-trend narrative ('advantage... widening at higher capacity, peaking on WideResNet / CIFAR-100') on it. Elsewhere the comparison is parity (C100/RN50: 80.8/0.701 vs 80.9/0.713) or a loss (C10/RN50: 95.4/0.156 vs 96.0/0.136). Compounding this, §A.4 states the authors deviated from the shared protocol by setting dropout to zero,
  2. [Table 8 / §A.10] while the transcribed baselines were trained under the original protocol — so the comparison in this block is cross-paper and asymmetric, and the effect of the dropout deviation on the σN-Ens rows is unmeasured. Relatedly, Table 8's WRN deep-ensemble shows ECE 5.6 at corruption severity 1 versus 22.9 on clean data, which is implausible for the same model; this suggests the corruption-eval deep ensemble was rerun under a different protocol. The authors should (i) rerun deep ensembles (and ideally Packed/FiLM) on WRN/C100 under their exact protocol, or rerun their own rows with protocol dropout; (ii) state explicitly which rows of Tables 8–9 were rerun and reconcile the clean/corruption ECE discrepancy; and (iii) temper the capacity-trend claim until the cell is validated.
  3. [§3.3 Fig. 2 / §5.2] The claim that τ is 'the only hyperparameter a practitioner must set' (§5.2) requires a clean selection protocol, which is not described. The benchmark tables use τ=0.1 for all CNNs, yet the frontier sweeps (Fig. 2, Fig. 5) show τ∈{0.1..1.0} with the interior optimum varying by backbone (e.g., WRN optimum near τ=0.7–0.9 by eye). How was τ=0.1 chosen for Table 1 — on validation data, or by inspecting test metrics? If the latter, the main-table numbers benefit from test-set tuning. Additionally, the frontier in Fig. 2(Left) spans only ~0.8 points of accuracy and ~1.5 points of ECE over 3 seeds with no error bars shown; the claimed 'interior optimum' may be within seed noise. Please report seed spread on the frontier points and describe the τ-selection procedure.
  4. [§3.4 / Appendix A.6] §3.4's modulation-uncertainty account is presented as explaining both the corruption robustness and the OOD weakness, but the two 'predictions' rest on an empirical premise (backbone component small on corrupted inputs, large off-distribution) that is asserted from Fort et al./Ovadia et al. rather than tested, and members' reinitialised heads confound the attribution: some of the measured disagreement comes from the M independent heads, not from the σNorm scales the regulariser controls. A direct ablation — shared head vs. reinitialised heads, at fixed τ — would isolate how much of U_mod and of the corruption robustness is actually carried by the modulation mechanism the paper proposes. Without it, 'explain why its calibration holds' (abstract) overstates what is shown. The reachable-set framing (Eq. 9–11) itself is sound and honestly scoped; the gap is evidential.
  5. [Abstract / §6] The abstract's 'matches or outperforms deep ensembles' is not supported as a blanket claim. On ImageNet/ViT (Table 3) deep ensembles are clearly better on accuracy (83.67 vs 82.32) and NLL (0.593 vs 0.655); on C10/RN50 deep ensembles win accuracy and NLL; on SST-2 σN-Ens is below the Single model on accuracy. The accurate statement is: best-in-class calibration and NLL on the pretrained ViT and on hard from-scratch settings, parity elsewhere, at ~4–6× fewer parameters — but with identical multiply-adds (Table 1: σN-Ens 23.83 MAdds = deep ensembles 23.83), so the efficiency gain is memory/parameters, not inference compute. The abstract and §6 should be qualified accordingly, and the equal-MAdds point stated explicitly.
minor comments (9)
  1. [§5.2 / Table 4] Table 4 (λ robustness) is single-seed; given the claim that λ need not be tuned, 3 seeds would close the point cheaply.
  2. [Fig. 3] Fig. 3's legend lists FiLM-Ensemble for the ViT-B/16 panel, but §4.3 states FiLM cannot wrap a pretrained backbone and does not appear in that benchmark. Presumably a shared legend; please clean up.
  3. [§4.2–4.3] A temperature-scaled Single baseline (Guo et al. 2017) in the calibration columns would sharpen the contribution: post-hoc TS is near-free and would show whether σN-Ens's ECE advantage survives the cheapest competitor, particularly on the pretrained ViT rows.
  4. [§3.2 / typos] Eq. (2): state explicitly that L_div is added to the task loss and that minimising Σ_m log p_m sharpens the across-member distribution; the gradient direction (Eq. 8) only appears in A.5. Also 'Equation equation 6' typo in §3.4, and 'regardless initialisation' in §3.1.
  5. [Appendix A.3] Notation: γ is overloaded for the trainable scale logits (Eq. 1) and the pretrained affine scale γ^pre (Eq. 7); a distinct symbol for one would ease reading of A.3.
  6. [§4.3] Ensemble size is inconsistent across tables (M=4 vision, M=3 SST-2) and LoRA-Ensemble is run at M=4 versus M=16 in its own paper; the disclosure in A.2/A.4 is good, but a sentence in §4.3 itself would prevent misreading of Table 3.
  7. [Fig. 1] Fig. 1(Center) is a synthetic simulation from N(0,1) initialisations; this is stated in the text but should also appear in the caption, since it is easy to read as measured data.
  8. [Table 3] LoRA-Ensemble's ImageNet ECE std of ±9.21 (Table 3) indicates a diverged seed; please note how seeds were handled rather than reporting the raw spread.
  9. [Table 1] Packed's WRN rows show Params 36.62 (equal to the full model) versus 14.55–19.35 elsewhere; the widening factor α used per cell should be tabulated, since Packed's parameter efficiency is part of the comparison.

Circularity Check

1 steps flagged

No significant circularity: diversity control, modulation-uncertainty framing, and benchmark claims are independent of the self-cited σNorm/TSσBN tools.

specific steps
  1. self citation load bearing [§3.1 Architecture; Eq. (1); Related Work on normalisation]
    "We adopt the multi-normalisation construction of TSσBN (Suteu & Serban, 2026), which replaces each shared normalisation layer with a separate layer per task. ... We build on σBN (Suteu & Guo, 2022), a bounded, sigmoid-scaled primitive trained with differential learning rates, used for capacity allocation in MTL (Suteu & Serban, 2026)"

    The modulation primitive and MTL wrapping are justified primarily by the authors’ own prior papers rather than by an external derivation. This is ordinary tool reuse, not circularity of the ensemble claims: the paper does not define success as recovering a TSσBN quantity, and the new content (softmax-temperature diversity control, τ frontier, modulation-uncertainty framing, and external benchmark results) does not reduce to those citations. Flagged only as minor self-citation of infrastructure.

full rationale

The paper’s load-bearing claims are (i) a softmax-temperature regulariser that sets an equilibrium level of member sharing, (ii) empirical competitiveness with deep ensembles and other implicit ensembles on external metrics (Acc, NLL, ECE, OOD, corruption), and (iii) a structural framing of shared-backbone epistemic uncertainty as “modulation uncertainty.” None of these reduce to their inputs by construction. The regulariser (Eq. 2) is a designed penalty whose effect on cosine similarity and on the accuracy–calibration frontier is measured, not assumed; Wood et al.’s bias–variance–diversity identity is external and is used only to motivate why an interior τ is desirable. The chain-rule split I[Y;(θ,ϕ)|x]=I[Y;θ|x]+I[Y;ϕ|x,θ] is a standard mutual-information identity; naming the second term at fixed θ★_shared “modulation uncertainty” is interpretive framing of what any shared-backbone ensemble can express, not a fitted quantity re-presented as a prediction. Calibration-under-corruption and weaker OOD are then checked on held-out corruption/OOD sets, not recovered from a fit. Self-citations to σBN (Suteu & Guo, 2022) and TSσBN (Suteu & Serban, 2026) supply the bounded normalisation primitive and the multi-task wrapping recipe—tools, not the claimed ensemble result—and no uniqueness theorem from those works is used to forbid alternatives. Success is defined against public benchmarks and transcribed baselines, not as recovering a constant fitted from the same data. Score 1 reflects only routine, non-load-bearing self-citation of the authors’ prior normalisation work.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 2 invented entities

The method sits on standard ensemble and MTL machinery plus the authors' prior bounded-normalisation primitive. Free knobs are real (τ, λ, which layers, LR multiplier) but the central claim is empirical competitiveness under those choices, not a parameter-free law. Invented naming is limited to 'modulation uncertainty' as a slice of ordinary epistemic MI. No new physical entities.

free parameters (6)
  • softmax temperature τ = typically 0.1–1.0 depending on setting
    Primary user-facing control selecting the diversity equilibrium and position on the accuracy–calibration frontier; set per experiment (e.g. 0.1 from-scratch CNN, 0.5/1.0 ViT).
  • regularisation strength λ = 1e-2 (from-scratch default)
    Pull strength toward the τ attractor; ablated as insensitive at convergence but still chosen (default 1e-2 from-scratch, 1e-3/1e-4 fine-tune).
  • σNorm scale LR multiplier = 100×
    100× differential learning rate on γ adopted from TSσBN, not retuned; load-bearing for diversity optimisation vs FiLM's single LR.
  • which normalisation layers are regularised = architecture-specific subset
    Hand choice per architecture (e.g. second BN in ResNet-18 blocks; FFN LayerNorms only in ViT); affects where M-choose-k allocation appears.
  • ensemble size M = 4 (main)
    Standard ensemble hyperparameter; main tables use M=4 (M=3 on SST-2); scaling study varies 2–16.
  • pretrained scale reconstruction k_ℓ and freeze of k_ℓ, β = k_ℓ = max_c |γpre_ℓ,c| / 0.95
    Conversion hyperparameters (k_ℓ = max|γ_pre|/0.95) needed so σ(γ) can match pretrained affine; frozen shared across members.
axioms (6)
  • domain assumption Wood et al. (2023) Bregman ensemble loss decomposes into average bias, variance, and diversity; diversity enters negatively and the optimum is interior.
    Used in §3.3 to justify why τ should select an interior operating point rather than maximise disagreement.
  • standard math Epistemic uncertainty as mutual information I[Y; member | x] admits a chain-rule split into backbone and modulation components (Smith & Gal; Depeweg et al.).
    §3.4 Eq. 6 defines modulation uncertainty as the conditional term at fixed shared θ.
  • domain assumption Independently trained backbones agree on in-distribution inputs and diverge off-distribution (Fort et al., 2019; Ovadia et al., 2019).
    Empirical premise that predicts graceful calibration under corruption and weak OOD detection for shared-backbone ensembles.
  • domain assumption Sigmoid-bounded per-channel scales are interpretable as feature importances and cosine similarity between them is a valid diversity metric.
    Inherited from σBN / TSσBN framing; underpins both the regulariser target and σNorm-cos monitoring.
  • domain assumption Ensemble members may be trained as multi-task heads on a shared backbone with task-specific normalisation (TSσBN construction).
    Architectural starting point §3.1; same-loss members invert MTL interference pressure, so diversity must be supplied by the regulariser.
  • standard math Softmax temperature limits: τ→0 drives one-hot ownership; τ→∞ flattens gradients (Appendix A.5 gradient ∂ℓ/∂s_j = (1/τ)(1−M p_j)).
    Standard softmax calculus used to justify controllable M-choose-k equilibria.
invented entities (2)
  • modulation uncertainty U_mod independent evidence
    purpose: Name the epistemic component expressible by varying only σNorm importances on a fixed shared backbone; explain calibration-vs-OOD asymmetry.
    Not a new physical quantity: it is I[Y;ϕ|x,θ=θ★] at one backbone draw. Independent handle is empirical (corruption ECE flat, OOD AUROC weaker on pretrained vision), which the paper reports.
  • σN-Ens / softmax-temperature diversity regulariser on σNorm importances independent evidence
    purpose: Training-time control of member diversity and accuracy–calibration frontier via single temperature τ.
    Methodological construct evaluated against external metrics; falsifiable by the reported tables and ablations.

pith-pipeline@v1.2.0-grok45-kimik3 · 30359 in / 4503 out tokens · 90802 ms · 2026-07-30T14:03:10.104962+00:00 · methodology

0 comments
read the original abstract

Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit ensembles lower this cost by sharing a single backbone across members. Member diversity is a primary determinant of ensemble quality, yet no implicit ensemble can shape it during training; existing methods fix it at initialisation or build it into the architecture. We introduce $\sigma$N-Ens, a normalisation-based implicit ensemble that treats each member as a task in a multi-task architecture and modulates the shared backbone through sigmoid-bounded scalers. We also introduce a softmax-temperature regulariser, which shapes the equilibrium level of sharing between members and traces the accuracy-calibration frontier. Because only normalisation layers are replicated, the mechanism can wrap convolutional and transformer backbones alike, also allowing pretrained models to be adapted through a short fine-tune. We frame the epistemic uncertainty such an ensemble expresses as modulation uncertainty, and explain why its calibration holds under input corruption, and why its out-of-distribution detection is weaker. Our method is evaluated across ResNets and transformers on CIFAR-10/100, ImageNet and SST-2. $\sigma$N-Ens matches or outperforms deep ensembles at a fraction of their parameter cost, scales with ensemble size where partitioning methods collapse, and maintains calibration under distribution shift.

Figures

Figures reproduced from arXiv: 2607.23860 by Mihai Suteu, Ovidiu Serban.

Figure 1
Figure 1. Figure 1: (Left) per-feature importance distributions for ResNet18, showing concentration toward the sigmoid extremes as temperature is decreased. (Center) simulated mean σNorm-cosine over the different member sizes and temperature values. (Right) clustering of features per their member importance vectors, showing the M-choose-k allocation. the strength alone fixes the endpoint, our softmax penalty defines a τ -para… view at source ↗
Figure 2
Figure 2. Figure 2: (Left) accuracy–calibration frontier over a τ sweep on WideResNet-28-10 / CIFAR-100, 3 seeds; an interior optimum is visible. (Right) σNorm-cosine against Jensen gap across the same sweep. with qi member i’s prediction, q¯i its Bregman centroid over D, and q ∗ the corresponding centroid combiner across members. Diversity enters negatively, but the three terms are coupled: pushing members apart raises diver… view at source ↗
Figure 3
Figure 3. Figure 3: Calibration holds up under distribution shift: expected calibration error against corruption severity [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Member scaling on CIFAR-100 / ResNet-50 at [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: shows the accuracy–calibration frontier of §3.3 for all three convolutional backbones; each traces the same interior-optimum shape as τ is swept, the WideResNet panel reproducing the main-text figure. 0.980 0.982 0.984 0.986 1 - ECE 0.776 0.777 0.778 0.779 0.780 0.781 0.782 0.783 0.784 Test Accuracy τ=0.7 τ=0.5 τ=0.4 τ=0.3 0.976 0.978 0.980 0.982 1 - ECE 0.807 0.808 0.809 0.810 0.811 0.812 0.813 Test Accur… view at source ↗
Figure 6
Figure 6. Figure 6: ResNet-50 importance embeddings; coloured by member-ownership [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: WideResNet-28-10 importance embeddings; coloured by member-ownership [PITH_FULL_IMAGE:figures/full_fig_p023_7.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 8 linked inside Pith

  1. [1]

    Fix one feature in one regularised layer and letsm =σ(γ m)∈(0,1)be memberm’s importance for it, with across-member sharesp= softmax τ(s)

    probes those joint equilibria empirically. Fix one feature in one regularised layer and letsm =σ(γ m)∈(0,1)be memberm’s importance for it, with across-member sharesp= softmax τ(s). The penalty for this feature isℓ=∑ m logpm, and its gradient is ∂ℓ ∂sj = 1 τ ( 1−Mp j ) .(8) By concavity,ℓis maximised exactly at the uniform distribution, so descent moves aw...

    2019

  2. [4]

    Columns:τ= 0.1(left), τ= 1(right)

    22 RN50 | τ=0.1 | all RN50 | τ=1 | all RN50 | τ=0.1 | reg RN50 | τ=1 | reg 0 1 2 3 4 k RN50 Figure 6: ResNet-50 importance embeddings; coloured by member-ownershipk. Columns:τ= 0.1(left), τ= 1(right). Rows: all layers (top), regularised layers (bottom). WRN | τ=0.1 | all WRN | τ=1 | all WRN | τ=0.1 | reg WRN | τ=1 | reg 0 1 2 3 4 k WRN Figure 7: WideResNe...

    arXiv 2019

  3. [6]

    Bálint Mucsányi, Michael Kirchhof, and Seong Joon Oh

    arXiv:2201.10908. Bálint Mucsányi, Michael Kirchhof, and Seong Joon Oh. Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  4. [7]

    Dominik J

    arXiv:2402.19460. Dominik J. Mühlematter, Michelle Halbheer, Alexander Becker, Dominik Narnhofer, Helge Aasen, Konrad Schindler, andMehmetOzgurTurkoglu. LoRA-Ensemble: Efficientuncertaintymodellingforself-attention networks.Transactions on Machine Learning Research (TMLR),

    Pith/arXiv arXiv

  5. [8]

    Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy

    arXiv:2405.14438. Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. InAdvances in Neural Information Processing Systems (NeurIPS),

    arXiv

  6. [13]

    The two agree on the setting and differ in how member diversity is produced, how the modulation is parameterised, and how it is optimised

    is the prior method closest to ours in con- struction: an implicit ensemble that modulates a shared backbone through per-member feature-wise affine transformations. The two agree on the setting and differ in how member diversity is produced, how the modulation is parameterised, and how it is optimised. The most important difference is the source and contr...

    2026

  7. [14]

    Member heads are reinitialised: a shared head is a symmetric fixed point at which the regulariser has no gradient, so reinitialisation supplies the asymmetry the mechanism needs

    handled this only partially, wrapping a selected subset of layers where the resulting error was acceptable; the per-layer scale here lets us wrap every normalisation layer, as long as the pretrained scales are positive, which they are in all backbones we tested. Member heads are reinitialised: a shared head is a symmetric fixed point at which the regulari...

    2026

  8. [2013]

    Function space di- versity for uncertainty prediction via repulsive last-layer ensembles.arXiv preprint arXiv:2412.15758,

    Sophie Steger, Christian Knoll, Bernhard Klein, Holger Fröning, and Franz Pernkopf. Function space di- versity for uncertainty prediction via repulsive last-layer ensembles.arXiv preprint arXiv:2412.15758,

    Pith/arXiv arXiv

  9. [2016]

    Is BatchEnsemble a single model? on calibration and diversity of efficient ensembles.arXiv preprint arXiv:2601.16936,

    Anton Zamyatin, Patrick Indri, Sagar Malhotra, and Thomas Gärtner. Is BatchEnsemble a single model? on calibration and diversity of efficient ensembles.arXiv preprint arXiv:2601.16936,

    arXiv

  10. [2018]

    •Geometric ambiguity(output-space, label-free): the ambiguity term under the cross-entropy’s centroid combiner, the normalised geometric mean (Wood et al., 2023)

    epistemic term, equal to the average KL divergence from each member to the arithmetic-mean prediction. •Geometric ambiguity(output-space, label-free): the ambiguity term under the cross-entropy’s centroid combiner, the normalised geometric mean (Wood et al., 2023). It is empirically indistin- guishable fromI[Y;M|x]in our setting (|ρ|>0.998across all backb...

    2023

  11. [2019]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Pith/arXiv arXiv

  12. [2020]

    Receding neuron importances for structured pruning.arXiv preprint arXiv:2204.06404,

    Mihai Suteu and Yike Guo. Receding neuron importances for structured pruning.arXiv preprint arXiv:2204.06404,

    Pith/arXiv arXiv

  13. [2022]

    Deep ensembles: A loss landscape perspective

    Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757,

    Pith/arXiv arXiv 1912

  14. [2023]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q

    arXiv:2107.03342. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML),

    Pith/arXiv arXiv

  15. [2024]

    Ensembling pruned attention heads for uncertainty-aware efficient transformers.arXiv preprint arXiv:2510.18358,

    Firas Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy, Elisa Ricci, and Gianni Franchi. Ensembling pruned attention heads for uncertainty-aware efficient transformers.arXiv preprint arXiv:2510.18358,

    Pith/arXiv arXiv

  16. [2025]

    This protocol differs from LoRA-Ensemble’s own configuration in its optimisation scheme and ensemble size (M= 4here, versusM= 16in their paper)

    and applied identically to each method, rather than tuning each separately, which is infeasible at this scale. This protocol differs from LoRA-Ensemble’s own configuration in its optimisation scheme and ensemble size (M= 4here, versusM= 16in their paper). We report LoRA-Ensemble as it behaves under this common protocol; its results under its own tuned con...

    2026

  17. [2026]

    Dan Hendrycks and Thomas Dietterich

    arXiv:2302.13425. Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. InInternational Conference on Learning Representations,

    arXiv