REVIEW 25 cited by
Deep Ensembles: A Loss Landscape Perspective
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Deep Ensembles: A Loss Landscape Perspective
read the original abstract
Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.
Forward citations
Cited by 25 Pith papers
-
Editing Models with Task Arithmetic
Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.
-
Cross-Trajectory Chimera Interventions Reveal Dissociable Roles of Weight Magnitude and Direction in Grokking
In grokking modular arithmetic, weight direction portably carries circuit identity across independent runs while weight norm only sets susceptibility to overwrite and a weak delay effect.
-
Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning
Representational convergence across 16 LLMs on 800 reasoning problems is stronger for failed tasks and pre-decision stages but shows minimal causal influence on predictions, pointing to shared processing constraints o...
-
Beyond Modality Fusion: Deep Ensembles for Multimodal Classification
Heterogeneous deep ensembles of unimodal models outperform imbalance-aware late-fusion and intermediate-fusion multimodal classifiers at matched parameter count, with a loss-ratio rule for allocating models per modality.
-
Ensemble Diversity Optimization for Subjective Supervision
Ensemble Diversity Optimization jointly learns ensemble weights, size, and a signed diversity regularizer, substantially improving calibration to annotator distributions on subjective text classification.
-
TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning
TabPack packs MLPs with diverse sampled hyperparameters into one vectorized model, selects ensemble members online during training, and matches tuned baselines at a fraction of the compute cost.
-
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
FID variance from training seeds is 3.2 times larger than from sampling seeds on hundreds of SiT models, with 1-2% coefficient of variation that barely shrinks with more compute, leading to a multi-seed evaluation protocol.
-
Recoverable but Not Stationary:Local Linear Structures in Weights and Activations
Local low-rank task-gradient structures exist in weights and activations but are non-stationary, with initial recovery updates forming a basis capturing 77% of LoRA displacement and parameter steps aligning 0.58 cosin...
-
Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?
Deep ensembles fail to capture meaningful epistemic uncertainty in message-passing GNNs due to epistemic collapse where independently trained networks converge to similar predictions.
-
Causal Unlearning in Collaborative Optimization: Exact and Approximate Influence Reversal under Adversarial Contributions
HF-KCU approximates influence reversal in federated learning using Krylov subspace conjugate gradients and causal weighting to achieve efficient unlearning with bounded adversarial robustness.
-
Flowing with Confidence
FMwC computes per-sample confidence scores for flow matching models via closed-form propagation of input-dependent multiplicative noise variance along the sampling ODE, supporting filtering, editing, and adaptive stepping.
-
Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models
Mutual Reinforcement Learning allows heterogeneous LLMs to exchange experience through mechanisms like Peer Rollout Pooling, Cross-Policy GRPO Advantage Sharing, and Success-Gated Transfer, with outcome-level sharing ...
-
Physics-Informed Neural Networks for Methane Sorption: Cross-Gas Transfer Learning, Ensemble Collapse Under Physics Constraints, and Monte Carlo Dropout Uncertainty Quantification
A PINN transfer learning framework for coal methane sorption reaches R²=0.932 on held-out data with 227% improvement over classical isotherms and identifies Monte Carlo Dropout as the best uncertainty method while ens...
-
FLAME: Condensing Ensemble Diversity into a Single Network for Efficient Sequential Recommendation
FLAME condenses ensemble diversity into a single network via modular ensemble simulation and guided mutual learning during training, delivering ensemble-level performance with single-network inference speed on sequent...
-
Efficient Bayesian Deep Ensembles via Analytic Predictive Inference
Independent neural predictors form a low-rank feature map that is aggregated by exact Bayesian linear regression, yielding a finite-rank Student-t process with competitive UCI regression performance.
-
q0: Primitives for Hyper-Epoch Pretraining
q0 turns multi-epoch budgets into diverse model populations using three primitives that outperform single-model training and strong ensembles with fewer epochs on a 1.8B model.
-
DAMEL: Dual-Axis Multi-Expert Learning for Class-Imbalanced Learning
DAMEL reduces both prediction bias and variance in class-imbalanced learning by concatenating multi-expert representations with an auxiliary balanced classifier and aggregating model weights across training epochs.
-
Soft Specialists: $\alpha$-R\'enyi Ensembles for Uncertainty-Aware LLM Post-Training
An α-Rényi variational ensemble method learns distributions over LoRA adapter parameters for uncertainty-aware LLM post-training, balancing individual model plausibility with complementary specialization.
-
ChainzRule: Sample-Efficient, Robust Deep Learning Across Tabular, NLP, and Vision Tasks
ChainzRule uses learnable polynomial layers with differential regularization on the Jacobian to promote stable low-frequency representations, claiming improved sample efficiency and robustness on multiple benchmarks.
-
A Posterior-Predictive Variance Decomposition for Epistemic and Aleatoric Uncertainty in Wind Power Forecasting
Derives a posterior-predictive variance decomposition separating epistemic and aleatoric uncertainty in heteroscedastic Bayesian neural network models for wind power forecasting, with a dedicated validation framework ...
-
Causal Unlearning in Collaborative Optimization: Exact and Approximate Influence Reversal under Adversarial Contributions
HF-KCU approximates influence reversal for unlearning in federated learning using Krylov-subspace conjugate gradients and causal weighting, delivering 47x speedup and privacy restoration on CIFAR-10, MNIST, and Fashio...
-
Scalable Hyperparameter-Divergent Ensemble Training with Automatic Learning Rate Exploration for Large Models
HDET lets data-parallel replicas explore a spread of learning rates independently before averaging parameters, with an auto-LR controller driven by inter-replica loss differences to produce a self-adapting schedule wi...
-
Expectation and Acoustic Neural Network Representations Enhance Music Identification from Brain Activity
Separating acoustic and expectation ANN representations as teacher targets improves EEG music identification beyond baselines and seed ensembles.
-
The effect of the number of parameters and the number of local feature patches on loss landscapes in distributed quantum neural networks
Increasing the number of local patches in a distributed quantum neural network architecture reduces the largest Hessian eigenvalue at minima and introduces a class-dependent outlier structure in the eigenspectrum.
-
Rethinking Data Mixing from the Perspective of Large Language Models
DoGraph models LLM data scheduling as a graph optimization problem based on gradient-domain connections and delivers competitive results on GPT-2 models of different sizes.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.