Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A frozen pretrained network plus a repulsive multi-head last layer yields deep-ensemble-grade uncertainty at a fraction of the cost.

desk verdict Useful empirical paper on cheap last-layer ensembles; main claims are plausible but the missing kernel bandwidth and partial OOD overlap need fixing before it is independently reproducible. read the letter →

arxiv 2412.15758 v1 pith:2B5KCP3P submitted 2024-12-20 cs.LG

Sophie Steger , Christian Knoll , Bernhard Klein , Holger Fröning , Franz Pernkopf This is my paper · ORCID
classification cs.LG
keywords uncertaintyquantificationfunction-spaceinferenceparticleoptimizationlast-layerensemblesdeepout-of-distributiondetectionactivelearningpretrainednetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the practical gap between function-space particle inference and deep ensembles is not the diversity principle itself but where it is applied. The key move is to freeze a pretrained base network and add a handful of linear heads, then enforce prediction diversity only on repulsion samples drawn from out-of-distribution data or label-destroying augmentations such as shuffled image patches. Diversity on the training data itself is shown to hurt both accuracy and uncertainty, whereas the right repulsion samples let a single multi-headed network approximate deep-ensemble-quality epistemic uncertainty. If the paper is right, uncertainty-aware fine-tuning of already-trained models becomes a cheap last-layer retraining step rather than a full retraining of several networks.

What carries the argument

The central object is the function-space repulsive last-layer ensemble (fs-RLL-E): one pretrained base network whose last layer is replaced by $n$ linear heads, each head a particle, with a Gaussian kernel repulsion term on the heads' prediction functions evaluated at a chosen batch of repulsion samples. The repulsion term is what drives the heads to disagree outside the training distribution, turning prediction spread into an epistemic uncertainty estimate via mutual information. The choice of repulsion samples is the second load-bearing mechanism: training data diversity underfits, so the paper uses unlabeled out-of-distribution data or patch-shuffled augmentations that destroy class labels, which suppresses spurious features and makes OOD disagreement generalize.

What would settle it

Train fs-RLL-E on a base network known to collapse features, such as LeNet or VGG-16 without residual connections on DirtyMNIST with eMNIST as repulsion samples, and measure whether epistemic uncertainty separates ambiguous MNIST from OOD data; the paper's own Appendix B.2 predicts degraded separation, so an AUROC near chance on that separation would falsify the claim that frozen feature extractors suffice.

Watch

Extended reading notes

Core claim

The central claim is that a shared frozen feature extractor with a repulsive last-layer ensemble, multiple linear heads trained as particles in function space, can produce uncertainty estimates competitive with deep ensembles while adding only $(d \times K + K) \times n$ trainable parameters for feature dimension $d$, $K$ classes, and $n$ particles. The paper shows that function-space repulsion evaluated on unlabeled out-of-distribution data or on label-destroyed augmented samples separates aleatoric from epistemic uncertainty, detects near and far out-of-domain inputs, and calibrates predictions under covariate shift. It also claims that this works retrospectively on pretrained residual networks, provided the base representation is distance-aware enough to avoid feature collapse, so that diversity can be enforced on the last layer alone.

Load-bearing premise

The method assumes the frozen pretrained feature extractor keeps distant inputs distant in feature space, so that a linear head can actually produce diverse predictions on out-of-distribution or augmented points; if the base network collapses such points together, last-layer repulsion cannot separate them.

Editorial extensions

If this is right

  • Deep-ensemble-level uncertainty becomes available on pretrained models for roughly the cost of retraining a single linear layer.
  • Active learning can use mutual information from the last-layer heads to pick informative points while ignoring ambiguous ones, matching deep ensembles.
  • OOD detection improves for both near and far shifts, particularly when repulsion samples are drawn from a related but different dataset like TinyImageNet for CIFAR models.
  • Calibration under corruption improves over the base network and often matches full deep ensembles.
  • Diversity on training data itself is counterproductive; label-destroying augmentation is a viable substitute when no OOD data is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the feature extractor preserves distances, fs-RLL-E could serve as a drop-in uncertainty retrofit for any pretrained residual network, including models fine-tuned for downstream tasks; the paper tests several ResNet sizes but not large language or vision-language backbones.
  • The choice of repulsion samples is effectively a prior over where the model should express ignorance, suggesting a design rule: point repulsion at the specific failure modes you want to catch, and expect trade-offs when one repulsion set must cover many OOD directions.
  • A testable extension is to make repulsion samples adaptive during training, for example by mining near-boundary or confusing OOD samples; the paper keeps the repulsion set fixed.
  • Since parameter-space repulsion gives no benefit over the unregularized last-layer ensemble in the paper's tables, the result implies that diversity must be measured on outputs rather than weights, a lesson that extends to ensemble distillation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes function-space repulsive last-layer ensembles (fs-RLL-E): a single network with a shared frozen base and n linear output heads, trained jointly with a function-space repulsion term evaluated on auxiliary repulsion samples (unlabeled OOD data or label-destroying augmented training data). The authors argue that (i) enforcing diversity on training-data samples is detrimental, (ii) parameter-space repulsion is insufficient and function-space diversity should be enforced instead (desideratum D1), and (iii) the multi-headed last-layer parameterization is flexible enough to serve as the particle representation, enabling retrospective uncertainty estimates for pretrained networks (D2). The method is evaluated on DirtyMNIST uncertainty decomposition, an active learning benchmark, OOD detection for CIFAR10/CIFAR100, and covariate-shift calibration on CIFAR10-C/CIFAR100-C, against MAP, DDU, SNGP, last-layer Laplace, and deep ensembles. The appendices add ablations over repulsion-sample choices, base-network architectures, spectral normalization, and feature-space dimensionality.

Significance. If the central claims are correct, the paper's contribution is a practically important one: a cheap post-hoc uncertainty method that achieves OOD detection and calibration competitive with deep ensembles at a fraction of the training and memory cost, plus a transferable design rule (enforce diversity on OOD or augmented samples, not on training data). The empirical work is broad and largely well-controlled: repulsion-sample ablations (Tables II and III), base-architecture sensitivity (Appendix B.2), spectral-normalization sensitivity (Table VIII), and feature-dimension scaling (Appendix B.5). The negative result for training-data repulsion (Table II, fs-RLL-E + Cifar10 with ECE 5.73 and AUROC below 50 on several OOD sets) is a clear, falsifiable finding that future work will have to engage with. However, three issues currently block full confidence: the core kernel hyperparameters are unreported, the parameter-space repulsion ablation appears to run in a dead-kernel regime, and the 'pretrained network' claim has not been tested on off-the-shelf checkpoints. These are fixable revisions rather than fundamental flaws.

major comments (4)
  1. [§IV-B; Appendix A, Table IV] The function-space repulsion kernel k(f_i, f_j) = exp(−||f_i − f_j||^p/ν) from Section IV-B is the mechanism that the paper credits for the improvements of fs-RLL-E over LL-E, but neither the bandwidth ν nor the exponent p nor any repulsion-loss weighting is reported, and Table IV (the hyperparameter summary) omits them. Since the experiments show that the repulsion term is strongly active — e.g., Table II, where replacing the repulsion-sample set changes NLL from 14.01 (Patches-32) to 19.08 (Cifar10) — the missing ν is a reproducibility blocker and not a cosmetic omission. I ask the authors to report the exact values used in every experiment and to add a sensitivity study of the main OOD AUROC and ECE numbers to ν (and, if applicable, to the repulsion weight) over at least two orders of magnitude.
  2. [§VIII-B; Tables I–III] In Section VIII-B the authors conclude that 'imposing diversity on the parameters of the last layer particles (RLL-E)... does not provide any additional benefit,' but the supporting rows in Tables I–III are not informative as reported. In Table I the RLL-E row is numerically identical to the LL-E row in every metric and every standard deviation across 10 runs (83.53±0.16, 48.32±0.24, 1.00±0.14, 96.82±0.34, 99.41±0.22, 96.16±1.53), and in Tables II–III the two rows are nearly identical, with most differences well within one standard deviation. Identical statistics over ten runs are consistent with an effectively inactive parameter-space kernel (e.g., ν so large that the repulsion gradient vanishes) rather than with a meaningful test of parameter-space repulsion. The comparison should report ν and the repulsion weight and should demonstrate at least one regime where RLL-E visibly differs from LL-E; otherwise the argument for desideratum D1 (parameter-space diversity is insufficient) rests on a degenerate experiment.
  3. [§V-C; Appendix A; Appendix B.2] The advertised use case — uncertainty-aware fine-tuning of pretrained networks — is only exercised on base networks trained by the authors under a single protocol (300 epochs, SGD, spectral normalization; Appendix A and Table IV). Appendix B.2 shows that this matters: on LeNet and VGG-16, which lack residual connections, the advantages of fs-RLL-E shrink (e.g., Table VI, LeNet ambig-vs-OOD AUROC of 92.68 vs 99.55 for ResNet-18), and Section V-C itself concedes the feature-collapse risk. Because the method inherits distance-awareness entirely from the frozen backbone, the paper should either add one experiment with a genuine off-the-shelf checkpoint (e.g., an ImageNet-pretrained ResNet without spectral normalization and without the authors' training protocol) or restrict the 'seamless integration to pretrained networks' claim to models trained with distance-preserving regularization. As it stands, the central practical claim is under-tested precisely in the setting it is meant to serve.
  4. [§VIII-D; Tables II–III; Table X] In the CIFAR10/100 OOD experiments, the repulsion-sample set and the OOD test set coincide for several configurations (fs-RLL-E + TinyImagenet evaluated on TinyImagenet, fs-RLL-E + Texture evaluated on Texture, and likewise for Cifar100 and SVHN in Table X). Table X is helpful evidence that repulsion diversity transfers across OOD datasets, but the main tables report same-source settings without any caveat, so the reader cannot separate genuine OOD detection from detection of the repulsion training signal. Please state explicitly in Section VIII-D which OOD test sets overlap with Drep, and provide the AUROC values with those same-source columns excluded; the conclusions should rest on the cross-generalization numbers.
minor comments (6)
  1. [§IV-A] The text says 'Euclidean (ℓ2 norm) or Manhattan (ℓ2 norm)', but the Manhattan distance is the ℓ1 norm; the second ℓ2 is a typo.
  2. [Table VI] The ResNet-18 LL-Laplace AUROC entries are printed as 'nan' with no explanation; either report the values or add a footnote, since missing values in a comparison table block a fair reading of that row.
  3. [§VIII-E] The phrase 'enhance improve uncertainty estimates' contains a duplicated verb, and 'Section A PPENDIX B.3' has a spacing artifact; both need copy-editing.
  4. [Fig. 5] The legend mixes method names with 'pool (PE)' and 'pool (MI)' variants in a way that is hard to parse; clarify that the parentheticals denote the acquisition score (predictive entropy vs mutual information) and to which curve 'pool' refers.
  5. [Abstract] The sentence 'the input samples, where particle predictions are enforced to be diverse, are detrimental to the model performance' is misleading as written, since the paper's headline is that OOD and augmented repulsion samples are beneficial; rephrase to say that enforcing diversity on training-data samples is detrimental.
  6. [Appendix A] No link to the authors' own implementation is given; Appendix A links only third-party repositories for [5], [6], and [52]. Since the method's performance hinges on ν and the repulsion-sample protocol, releasing the fs-RLL-E code (or a complete configuration file) is needed for reproducibility.

Circularity Check

1 steps flagged · score 3.0 of 10

OOD-detection scores on repulsion-sample distributions are partly self-confirming, but unseen-OOD generalization keeps the core claim independent.

  1. fitted input called prediction [Appendix B.3 (Table VII caption) and Section VIII-D (Tables II-III)]
    "We test the influence of repulsion samples from the OOD training set for different base network architectures. ... Using informative repulsion samples for the function space loss results in better performance than unregularized LL-E retraining."

    The training objective for fs-RLL-E includes a repulsion term evaluated on a chosen set Drep; when Drep equals the evaluation OOD test set (e.g., Cifar100 for +Cifar100, TinyImagenet for +TinyImagenet), the AUROC on that column measures whether the model learned to satisfy the training loss on the repulsion-sample distribution, not whether the uncertainty estimator detects a genuinely unseen shift. The paper frames these columns as OOD-detection evidence, but for the same-distribution cells this is a self-confirming evaluation: the desired behavior (diverse predictions on those inputs) is the optimization target. Unlike the unseen-OOD columns, these numbers do not provide independent evidence for the method's generalization claim.

full rationale

The derivation chain is mostly empirical. The particle update rule (Eq. 2) and the function-space kernel are imported from prior external work (D'Angelo and Fortuin; Wang et al.), not derived from the paper's own conclusions; no load-bearing self-citation appears in the reference list. The main architectural claim, that a shared frozen base with linear heads is flexible enough, is tested directly (Tables I-III) and supported by comparisons against DDU, SNGP, LL-Laplace, and DE-5, which are external methods. The one genuine circular element is evaluative: in Tables II/III and Appendix B.3/B.5, several OOD-detection AUROC columns use as the OOD test set the very distribution that was used as repulsion samples in training (e.g., fs-RLL-E +Cifar100 evaluated on Cifar100; fs-RLL-E +TinyImagenet evaluated on TinyImagenet). Since the training loss explicitly enforces diverse predictions on those repulsion samples, a high AUROC on those columns is partly the optimization objective, not an independent prediction. The paper also reports generalization to unseen OOD (e.g., fs-RLL-E +TinyImagenet on SVHN/FakeData; DirtyMNIST eMNIST-repulsion on kMNIST/fashionMNIST/Omniglot), so the central claim has independent content. Score 3 reflects this partial, non-central circularity; no self-citation or definitional identity was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on the POVI update rule from prior work, on the assumption that a linear-head ensemble is expressive enough in feature space, and on the assumption that pretrained residual networks do not collapse OOD features. These are reasonable domain assumptions, but they are load-bearing and only partially verified empirically.

free parameters (3)
  • Kernel bandwidth nu in function-space RBF repulsion kernel = not reported in the paper
    Appears in Eq. (2)/(5) as the scale of exp(-||f(i)(X)-f(j)(X)||_p / nu). It controls the strength and locality of repulsion, but no value is given in Table IV or the text.
  • Number of heads/particles n = 10 for image classification, 30 for toy experiments
    Chosen by hand and not ablated across a range. The paper argues function-space repulsion matters in the small-particle regime, so n is load-bearing.
  • Repulsion sample dataset Drep = eMNIST or Patches for MNIST; TinyImagenet, CIFAR100, Texture, or Patches for CIFAR10/100
    Selected per experiment and per target OOD set. This choice strongly determines the results (see Table II, where diversity on CIFAR10 training data collapses OOD AUROC to roughly 40-45 percent). The authors acknowledge in Future Work that poor repulsion-sample coverage limits the method.
assumptions (4)
  • standard math Particle-optimization variational inference with the update in Eq. (2) converges to the posterior in the infinite-particle limit.
    Taken from prior work (D'Angelo and Fortuin, 2021) and used as the theoretical basis for the repulsion term. The paper relies on this result without re-deriving it.
  • standard math Evaluating the function-space kernel on a finite set of repulsion samples with support on the input domain suffices for inference (the B-dimensional marginal condition).
    Invoked in Section IV-B.2 via reference [5]. The paper assumes this condition holds approximately for the chosen repulsion samples.
  • domain assumption A single shared base network with multiple linear heads is flexible enough to represent diverse prediction functions for uncertainty estimation.
    Stated as a highlighted claim in Section V-A ('Deep ensembles are not necessary') and justified mainly by referencing ensemble distillation literature [9], [19]. This is an empirical assumption, not a proven theorem.
  • domain assumption Pretrained networks with residual connections and/or spectral normalization avoid feature collapse, so the frozen feature space remains distance-aware.
    Introduced in Section V-C and tested in Appendix B.2 and B.4. The method degrades on base networks without these properties, so the assumption is load-bearing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles." pith.science (2026). https://pith.science/paper/2B5KCP3P

@misc{pith2026241215758,
  author       = {Pith},
  title        = {Pith review of: Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensembles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2B5KCP3P}},
  note         = {Machine review of arXiv:2412.15758}
}
read the original abstract

Bayesian inference in function space has gained attention due to its robustness against overparameterization in neural networks. However, approximating the infinite-dimensional function space introduces several challenges. In this work, we discuss function space inference via particle optimization and present practical modifications that improve uncertainty estimation and, most importantly, make it applicable for large and pretrained networks. First, we demonstrate that the input samples, where particle predictions are enforced to be diverse, are detrimental to the model performance. While diversity on training data itself can lead to underfitting, the use of label-destroying data augmentation, or unlabeled out-of-distribution data can improve prediction diversity and uncertainty estimates. Furthermore, we take advantage of the function space formulation, which imposes no restrictions on network parameterization other than sufficient flexibility. Instead of using full deep ensembles to represent particles, we propose a single multi-headed network that introduces a minimal increase in parameters and computation. This allows seamless integration to pretrained networks, where this repulsive last-layer ensemble can be used for uncertainty aware fine-tuning at minimal additional cost. We achieve competitive results in disentangling aleatoric and epistemic uncertainty for active learning, detecting out-of-domain data, and providing calibrated uncertainty estimates under distribution shifts with minimal compute and memory.

Figures

Figures reproduced from arXiv: 2412.15758 by the authors.

Figure 1
Figure 1. Repulsive last-layer ensemble in function-space (fs [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Example of repulsion samples for DirtyMNIST (top [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. For regression, we show the prediction of individual [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Histograms of aleatoric versus epistemic uncertainty [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Test accuracy of the model as a function of the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: NLL and uncertainty calibration of the different methods on CIFAR10-C (left) and CIFAR100-C (right), for different [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Negative log-likelihood (NLL) on CIFAR10-C. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Expected calibration error (ECE) on CIFAR10-C. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Negative log-likelihood (NLL) on CIFAR100-C. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Expected calibration error (ECE) on CIFAR100-C. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Softmax-temperature regularisation of sigmoid-bounded normalisation scales controls member diversity in implicit ensembles, matching deep ensembles cheaply across CNNs and transformers.

Reference graph

Works this paper leans on

70 extracted references · 48 canonical work pages · cited by 1 Pith paper

  1. [1]

    Simple and scalable predictive uncertainty estimation using deep ensembles,

    B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  2. [2]

    Stein variational gradient descent: A general purpose Bayesian inference algorithm,

    Q. Liu and D. Wang, “Stein variational gradient descent: A general purpose Bayesian inference algorithm,” Advances in Neural Information Processing Systems, vol. 29, 2016

    work page 2016

  3. [3]

    Stein variational gradient descent as gradient flow,

    Q. Liu, “Stein variational gradient descent as gradient flow,” Advances in Neural Information Processing Systems , vol. 30, 2017

    work page 2017

  4. [4]

    Understanding and accelerating particle-based variational inference

    C. Liu, J. Zhuo, P. Cheng, R. Zhang, and J. Zhu, “Understanding and accelerating particle-based variational inference.” PMLR, 2019, pp. 4082–4092

    work page 2019

  5. [5]

    Function Space Particle Optimization for Bayesian Neural Networks

    Z. Wang, T. Ren, J. Zhu, and B. Zhang, “Function space par- ticle optimization for Bayesian neural networks,” arXiv preprint arXiv:1902.09754, 2019

    work page Pith review arXiv 1902

  6. [6]

    Repulsive deep ensembles are Bayesian,

    F. D’Angelo and V . Fortuin, “Repulsive deep ensembles are Bayesian,” Advances in Neural Information Processing Systems , vol. 34, pp. 3451– 3465, 2021

    work page 2021

  7. [7]

    Input gradient diversity for neural network ensembles,

    T. Trinh, M. Heinonen, L. Acerbi, and S. Kaski, “Input gradient diversity for neural network ensembles,” arXiv preprint arXiv:2306.02775, 2023

    arXiv 2023

  8. [8]

    Feature space particle inference for neural network ensembles

    S. Yashima, T. Suzuki, K. Ishikawa, I. Sato, and R. Kawakami, “Feature space particle inference for neural network ensembles.” PMLR, 2022, pp. 25 452–25 468

    work page 2022

Show all 70 references
  1. [9]

    Hydra: Preserving ensemble diversity for model distillation,

    L. Tran, B. S. Veeling, K. Roth, J. Swiatkowski, J. V . Dillon, J. Snoek, S. Mandt, T. Salimans, S. Nowozin, and R. Jenatton, “Hydra: Preserving ensemble diversity for model distillation,” arXiv preprint arXiv:2001.04694, 2020. 12

    2001 arXiv

  2. [10]

    Quantification of uncertainty with adversarial models,

    K. Schweighofer, L. Aichberger, M. Ielanskyi, G. Klambauer, and S. Hochreiter, “Quantification of uncertainty with adversarial models,” Advances in Neural Information Processing Systems , vol. 36, pp. 19 446–19 484, 2023

    2023

  3. [11]

    A unified particle- optimization framework for scalable bayesian sampling,

    C. Chen, R. Zhang, W. Wang, B. Li, and L. Chen, “A unified particle- optimization framework for scalable bayesian sampling,” arXiv preprint arXiv:1805.11659, 2018

    2018 arXiv

  4. [12]

    Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning

    S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi-Velez, and S. Udluft, “Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning.” PMLR, 2018, pp. 1184–1193

    2018

  5. [13]

    Quan- tifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures?

    L. Wimmer, Y . Sale, P. Hofman, B. Bischl, and E. H ¨ullermeier, “Quan- tifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures?” in Uncertainty in Artificial Intelligence . PMLR, 2023, pp. 2282–2292

    2023

  6. [14]

    On the usefulness of deep ensemble diver- sity for out-of-distribution detection,

    G. Xia and C.-S. Bouganis, “On the usefulness of deep ensemble diver- sity for out-of-distribution detection,” arXiv preprint arXiv:2207.07517, 2022

    2022 arXiv

  7. [15]

    A rigorous link between deep ensembles and (variational) bayesian methods,

    V . D. Wild, S. Ghalebikesabi, D. Sejdinovic, and J. Knoblauch, “A rigorous link between deep ensembles and (variational) bayesian methods,” in Advances in Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id=eTHawKFT4h

    2023

  8. [16]

    Improving the identifiability of neural networks for bayesian inference,

    A. A. Pourzanjani, R. M. Jiang, and L. R. Petzold, “Improving the identifiability of neural networks for bayesian inference,” in NIPS workshop on bayesian deep learning , vol. 4, 2017, p. 31

    2017

  9. [17]

    On the surprising behavior of distance metrics in high dimensional space,

    C. C. Aggarwal, A. Hinneburg, and D. A. Keim, “On the surprising behavior of distance metrics in high dimensional space,” in Database Theory—ICDT 2001: 8th International Conference London, UK, Jan- uary 4–6, 2001 Proceedings 8 . Springer, 2001, pp. 420–434

    2001

  10. [18]

    Bridging the data processing inequality and function-space variational inference,

    A. Kirsch, “Bridging the data processing inequality and function-space variational inference,” in The Third Blogpost Track at ICLR 2024 ,

    2024

  11. [19]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531 , 2015

    2015 arXiv

  12. [20]

    Do Bayesian neural networks need to be fully stochastic?

    M. Sharma, S. Farquhar, E. Nalisnick, and T. Rainforth, “Do Bayesian neural networks need to be fully stochastic?” PMLR, 2023, pp. 7694– 7722

    2023

  13. [21]

    The best deep ensembles sacrifice predictive diversity,

    T. Abe, E. K. Buchanan, G. Pleiss, and J. P. Cunningham, “The best deep ensembles sacrifice predictive diversity,” 2022

    2022

  14. [22]

    Joint training of deep ensembles fails due to learner collusion,

    A. Jeffares, T. Liu, J. Crabb ´e, and M. van der Schaar, “Joint training of deep ensembles fails due to learner collusion,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  15. [23]

    Uncertainty estimation using a single deep deterministic neural network

    J. van Amersfoort, L. Smith, Y . W. Teh, and Y . Gal, “Uncertainty estimation using a single deep deterministic neural network.” PMLR, 2020, pp. 9690–9700

    2020

  16. [24]

    Spectral normalization for generative adversarial networks,

    T. Miyato, T. Kataoka, M. Koyama, and Y . Yoshida, “Spectral normalization for generative adversarial networks,” arXiv preprint arXiv:1802.05957, 2018

    2018 arXiv

  17. [25]

    mixup: Beyond empirical risk minimization,

    H. Zhang, M. Cisse, Y . N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412 , 2017

    2017 arXiv

  18. [26]

    Cutmix: Reg- ularization strategy to train strong classifiers with localizable features,

    S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y . Yoo, “Cutmix: Reg- ularization strategy to train strong classifiers with localizable features,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6023–6032

    2019

  19. [27]

    Bayesian learning for neural networks,

    R. M. Neal, “Bayesian learning for neural networks,” Ph.D. dissertation, University of Toronto, 1995

    1995

  20. [28]

    Weight uncertainty in neural network

    C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural network.” PMLR, 2015, pp. 1613–1622

    2015

  21. [29]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learning

    Y . Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.” PMLR, 2016, pp. 1050–1059

    2016

  22. [30]

    Laplace redux-effortless Bayesian deep learning,

    E. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig, “Laplace redux-effortless Bayesian deep learning,” Advances in Neural Information Processing Systems , vol. 34, pp. 20 089–20 103, 2021

    2021

  23. [31]

    Efficient and scalable Bayesian neural nets with rank-1 factors

    M. Dusenberry, G. Jerfel, Y . Wen, Y . Ma, J. Snoek, K. Heller, B. Laksh- minarayanan, and D. Tran, “Efficient and scalable Bayesian neural nets with rank-1 factors.” PMLR, 2020, pp. 2782–2792

    2020

  24. [32]

    Being Bayesian, even just a bit, fixes overconfidence in relu networks

    A. Kristiadi, M. Hein, and P. Hennig, “Being Bayesian, even just a bit, fixes overconfidence in relu networks.” PMLR, 2020, pp. 5436–5446

    2020

  25. [33]

    Variational bayesian last layers,

    J. Harrison, J. Willes, and J. Snoek, “Variational bayesian last layers,” arXiv preprint arXiv:2404.11599 , 2024

    2024 arXiv

  26. [34]

    Evaluating scalable bayesian deep learning methods for robust computer vision,

    F. K. Gustafsson, M. Danelljan, and T. B. Schon, “Evaluating scalable bayesian deep learning methods for robust computer vision,” in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020, pp. 318–319

    2020

  27. [35]

    Can you trust your model’s uncer- tainty? evaluating predictive uncertainty under dataset shift,

    Y . Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek, “Can you trust your model’s uncer- tainty? evaluating predictive uncertainty under dataset shift,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  28. [36]

    The case for bayesian deep learning,

    A. G. Wilson, “The case for bayesian deep learning,” arXiv preprint arXiv:2001.10995, 2020

    2001 arXiv

  29. [37]

    Functional variational Bayesian neural networks,

    S. Sun, G. Zhang, J. Shi, and R. Grosse, “Functional variational Bayesian neural networks,” arXiv preprint arXiv:1903.05779 , 2019

    1903 arXiv

  30. [38]

    Variational implicit pro- cesses,

    C. Ma, Y . Li, and J. M. Hern ´andez-Lobato, “Variational implicit pro- cesses,” in International Conference on Machine Learning . PMLR, 2019, pp. 4222–4233

    2019

  31. [39]

    Un- derstanding variational inference in function-space,

    D. R. Burt, S. W. Ober, A. Garriga-Alonso, and M. van der Wilk, “Un- derstanding variational inference in function-space,” in Third Symposium on Advances in Approximate Bayesian Inference , 2020

    2020

  32. [40]

    Functional variational inference based on stochastic process generators,

    C. Ma and J. M. Hern ´andez-Lobato, “Functional variational inference based on stochastic process generators,” Advances in Neural Information Processing Systems, vol. 34, pp. 21 795–21 807, 2021

    2021

  33. [41]

    Tractable function-space variational inference in Bayesian neural networks,

    T. G. Rudner, Z. Chen, Y . W. Teh, and Y . Gal, “Tractable function-space variational inference in Bayesian neural networks,” Advances in Neural Information Processing Systems , vol. 35, pp. 22 686–22 698, 2022

    2022

  34. [42]

    Function- space regularization in neural networks: A probabilistic perspective,

    T. G. Rudner, S. Kapoor, S. Qiu, and A. G. Wilson, “Function- space regularization in neural networks: A probabilistic perspective,” International Conference on Machine Learning , 2023

    2023

  35. [43]

    Deep anomaly detection with outlier exposure,

    D. Hendrycks, M. Mazeika, and T. Dietterich, “Deep anomaly detection with outlier exposure,” in International Conference on Learning Representations, 2019. [Online]. Available: https://openreview.net/ forum?id=HyxCxhRcY7

    2019

  36. [44]

    Collaborative learning for deep neural networks,

    G. Song and W. Chai, “Collaborative learning for deep neural networks,” Advances in Neural Information Processing Systems , vol. 31, 2018

    2018

  37. [45]

    Very deep mul- tilingual convolutional neural networks for lvcsr

    T. Sercu, C. Puhrsch, B. Kingsbury, and Y . LeCun, “Very deep mul- tilingual convolutional neural networks for lvcsr.” IEEE, 2016, pp. 4955–4959

    2016

  38. [46]

    Why m heads are better than one: Training a diverse ensemble of deep networks,

    S. Lee, S. Purushwalkam, M. Cogswell, D. Crandall, and D. Batra, “Why m heads are better than one: Training a diverse ensemble of deep networks,” arXiv preprint arXiv:1511.06314 , 2015

    2015 arXiv

  39. [47]

    Deep exploration via bootstrapped dqn,

    I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped dqn,” Advances in Neural Information Processing Systems, vol. 29, 2016

    2016

  40. [48]

    Knowledge distillation by on-the-fly native ensemble,

    X. Zhu, S. Gong et al. , “Knowledge distillation by on-the-fly native ensemble,” Advances in Neural Information Processing Systems, vol. 31, 2018

    2018

  41. [49]

    Sub-ensembles for fast uncertainty estimation in neural networks,

    M. Valdenegro-Toro, “Sub-ensembles for fast uncertainty estimation in neural networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4119–4127

    2023

  42. [50]

    Posterior network: Uncertainty estimation without ood samples via density-based pseudo- counts,

    B. Charpentier, D. Z ¨ugner, and S. G ¨unnemann, “Posterior network: Uncertainty estimation without ood samples via density-based pseudo- counts,” Advances in Neural Information Processing Systems , vol. 33, pp. 1356–1367, 2020

    2020

  43. [51]

    Quantifying aleatoric and epistemic uncertainty using density estimation in latent space,

    J. Postels, H. Blum, C. Cadena, R. Siegwart, L. Van Gool, and F. Tombari, “Quantifying aleatoric and epistemic uncertainty using density estimation in latent space,” arXiv preprint arXiv:2012.03082 , vol. 1, 2020

    2012 arXiv

  44. [52]

    Deep deterministic uncertainty: A new simple baseline,

    J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. Torr, and Y . Gal, “Deep deterministic uncertainty: A new simple baseline,” 2023, pp. 24 384– 24 394

    2023

  45. [53]

    Contrastive training for improved out-of-distribution detection,

    J. Winkens, R. Bunel, A. G. Roy, R. Stanforth, V . Natarajan, J. R. Ledsam, P. MacWilliams, P. Kohli, A. Karthikesalingam, S. Kohl et al., “Contrastive training for improved out-of-distribution detection,” arXiv preprint arXiv:2007.05566, 2020

    2007 arXiv

  46. [54]

    Simple and principled uncertainty estimation with deterministic deep learning via distance awareness,

    J. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lak- shminarayanan, “Simple and principled uncertainty estimation with deterministic deep learning via distance awareness,” Advances in Neural Information Processing Systems , vol. 33, pp. 7498–7512, 2020

    2020

  47. [55]

    Single-model uncertainties for deep learning,

    N. Tagasovska and D. Lopez-Paz, “Single-model uncertainties for deep learning,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  48. [56]

    On feature collapse and deep kernel learning for single forward pass uncertainty,

    J. van Amersfoort, L. Smith, A. Jesson, O. Key, and Y . Gal, “On feature collapse and deep kernel learning for single forward pass uncertainty,” arXiv preprint arXiv:2102.11409 , 2021

    2021 arXiv

  49. [57]

    Improved training of wasserstein GANs,

    I. Gulrajani, F. Ahmed, M. Arjovsky, V . Dumoulin, and A. C. Courville, “Improved training of wasserstein GANs,” Advances in Neural Informa- tion Processing Systems , vol. 30, 2017

    2017

  50. [58]

    On the practicality of deterministic epistemic uncertainty,

    J. Postels, M. Segu, T. Sun, L. Sieber, L. Van Gool, F. Yu, and F. Tombari, “On the practicality of deterministic epistemic uncertainty,” arXiv preprint arXiv:2107.00649 , 2021

    2021 arXiv

  51. [59]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, “Learning multiple layers of features from tiny images,” Tech. Rep., 2009. 13

    2009

  52. [60]

    Places: A 10 million image database for scene recognition,

    B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017

    2017

  53. [61]

    Reading digits in natural images with unsupervised feature learning,

    Y . Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y . Ng, “Reading digits in natural images with unsupervised feature learning,” Advances in Neural Information Processing Systems , 2011

    2011

  54. [62]

    Describing textures in the wild,

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi, “Describing textures in the wild,” in Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2014

    2014

  55. [63]

    Tiny imagenet visual recognition challenge,

    Y . Le and X. Yang, “Tiny imagenet visual recognition challenge,” CS 231N, vol. 7, no. 7, p. 3, 2015

    2015

  56. [64]

    Benchmarking neural network robust- ness to common corruptions and perturbations,

    D. Hendrycks and T. Dietterich, “Benchmarking neural network robust- ness to common corruptions and perturbations,” International Confer- ence on Learning Representations , 2019

    2019

  57. [65]

    A simple approach to improve single-model deep uncertainty via distance-awareness

    J. Z. Liu, S. Padhy, J. Ren, Z. Lin, Y . Wen, G. Jerfel, Z. Nado, J. Snoek, D. Tran, and B. Lakshminarayanan, “A simple approach to improve single-model deep uncertainty via distance-awareness.” J. Mach. Learn. Res., vol. 24, pp. 42–1, 2023

    2023

  58. [66]

    Obtaining well calibrated probabilities using Bayesian binning,

    M. P. Naeini, G. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using Bayesian binning,” vol. 29, no. 1, 2015

    2015

  59. [67]

    Clanuwat, M

    T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. (2018) Deep learning for classical japanese literature

    2018

  60. [68]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,

    H. Xiao, K. Rasul, and R. V ollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  61. [69]

    Human-level concept learning through probabilistic program induction,

    B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, “Human-level concept learning through probabilistic program induction,” Science, vol. 350, no. 6266, pp. 1332–1338, 2015. Sophie Steger received her BSc and MSc (Dipl. Ing.) degree in Electrical Engineering at Graz Uni- versit...

    2015

  62. [2024]

    Available: https://openreview.net/forum?id=CAt2LeiiSc

    [Online]. Available: https://openreview.net/forum?id=CAt2LeiiSc

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.