REVIEW 12 cited by
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the number of networks, which quickly becomes untenable. In this paper, we propose BatchEnsemble, an ensemble method whose computational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix to be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble is not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for a given mini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and out-of-distribution tasks, BatchEnsemble yields competitive accuracy and uncertainties as typical ensembles; the speedup at test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs. We further show that BatchEnsemble can easily scale up to lifelong learning on Split-ImageNet which involves 100 sequential learning tasks.
Forward citations
Cited by 12 Pith papers
-
NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling
Dynamic sparse training of multiple heads on a shared backbone outperforms full dense ensembles on ImageNet and C4 while using less compute.
-
Parallel Scaling Law for Language Models
Running P parallel streams of a language model with learned prefixes gives performance comparable to multiplying parameters by about k log P + 1, with k around 0.33 to 0.39.
-
Towards Precise Prediction Uncertainty in GNNs: Refining GNNs with Topology-grouping Strategy
A post-hoc GNN calibration method that groups nodes by confidence and neighborhood prediction similarity and applies group-specific temperatures, reducing expected calibration error on many benchmarks.
-
QCResUNet: Joint Subject-level and Voxel-level Segmentation Quality Prediction
QCResUNet jointly predicts subject-level Dice and surface Dice plus voxel-level tissue-specific error maps for brain tumor and cardiac MRI segmentations, outperforming five baseline QC methods on internal and external...
-
Set-based v.s. Distribution-based Representations of Epistemic Uncertainty: A Comparative Study
Built from identical predictive distributions, distribution-based and credal-set uncertainty representations are comparable, but their rankings depend heavily on the chosen uncertainty measure and downstream task.
-
Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Tasks
A grammar-based model of LLM-generated SMT-LIB code produces uncertainty signals that predict formalization errors on some reasoning tasks, with fused signals giving large error reductions only in an in-sample evaluation.
-
Incrementally Learning Multiple Diverse Data Domains via Multi-Source Dynamic Expansion Model
MSDEM grows per-task experts on top of multiple frozen ViT backbones with an attention-based fusion and a Gumbel-Softmax graph router, and reports state-of-the-art accuracy on multi-domain continual image classification.
-
ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts
ALoRE aggregates multiple low-rank experts in a Kronecker-product space and merges them into the frozen backbone, reporting top accuracy on FGVC and VTAB-1k with only 0.15M trainable parameters.
-
Last-layer committee machines for uncertainty estimations of benthic imagery
A last-layer committee of 100 randomly initialized linear heads trained on shared features yields uncertainty estimates for benthic imagery similar to Monte Carlo dropout at a fraction of the inference cost.
-
(GG) MoE vs. MLP on Tabular Data
A Gumbel-Softmax-gated mixture of experts with numerical embeddings matches MLP accuracy on 38 tabular datasets while using roughly 10x fewer parameters, but its edge over MLP is not statistically significant.
-
Strategic Base Representation Learning via Feature Augmentations for Few-Shot Class Incremental Learning
A feature augmentation plus proxy-label contrastive learning framework reports SOTA FSCIL accuracy, but its evaluation uses extra proxy prototypes that bias comparisons.
-
Customer Lifetime Value Prediction with Uncertainty Estimation Using Monte Carlo Dropout
Applying Monte Carlo Dropout to LTV prediction provides uncertainty estimates and modest accuracy gains on a large mobile game dataset.
Discussion (0). Continue with ORCID to comment.