Accelerating Deep Learning by Focusing on the Biggest Losers
Reviewed by Pithpith:T6PBMGKWopen to challenge →
read the original abstract
This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration. Selective-Backprop uses the output of a training example's forward pass to decide whether to use that example to compute gradients and update parameters, or to skip immediately to the next example. By reducing the number of computationally-expensive backpropagation steps performed, Selective-Backprop accelerates training. Evaluation on CIFAR10, CIFAR100, and SVHN, across a variety of modern image models, shows that Selective-Backprop converges to target error rates up to 3.5x faster than with standard SGD and between 1.02--1.8x faster than a state-of-the-art importance sampling approach. Further acceleration of 26% can be achieved by using stale forward pass results for selection, thus also skipping forward passes of low priority examples.
This paper has not been read by Pith yet.
Forward citations
Cited by 9 Pith papers
-
Initialization is Half the Battle: Generating Diverse Images from a Guidance Potential Posterior
DivIn samples initial noise from a guidance potential posterior via Langevin dynamics to improve diversity in class-to-image and text-to-image generation.
-
Disagreement-Regularized Importance Sampling for Adversarial Label Corruption
DR-IS selects low-contamination subsets via bounded rank-disagreement in proxy ensembles under an ε-contamination model, with O(√(log(N/δ)/K)) concentration rates that certify separation when the expectation gap Δ' is...
-
Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration
A graph-based unified dataset pruning framework that formulates pruning as MWCP, derives a greedy algorithm with formal approximation guarantees, and demonstrates substantial training acceleration on ImageNet.
-
Policy-based Foveated Imaging and Perception
A task-aware policy learned via reinforcement learning allocates high-resolution pixels on dual-stream sensors in real time, outperforming fixed or non-predictive baselines under tight pixel budgets in both simulation...
-
The Long-Term Effects of Data Selection in LLM Fine-Tuning
Short-term data selectors in multi-stage LLM fine-tuning can slow future learning and increase forgetting, formalized as myopic selection with a proposed LHAS objective to address it.
-
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
CHERRY combines selective ground-truth token training, recurrent depth compression from 48 to 6 layers, and mixture-of-efficient-experts to achieve competitive loss with fewer parameters on a 1.8B Korean model.
-
RCAP: Robust, Class-Aware, Probabilistic Dynamic Dataset Pruning
RCAP introduces class-aware probabilistic pruning that uses closed-form per-class fractions updated by loss and high-loss sampling to preserve worst-group accuracy at high pruning rates.
-
Learning to Reason at the Frontier of Learnability
A curriculum sampling questions with high variance in success rate improves reinforcement learning performance for LLM reasoning tasks.
-
Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
Step-Video-T2V describes a 30B-parameter text-to-video model with custom Video-VAE, 3D DiT, flow matching, and Video-DPO that claims state-of-the-art results on a new internal benchmark.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.