Pith. sign in

REVIEW 11 cited by

Accelerating Deep Learning by Focusing on the Biggest Losers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1910.00762 v1 pith:T6PBMGKW submitted 2019-10-02 cs.LG stat.ML

Angela H. Jiang , Daniel L.-K. Wong , Giulio Zhou , David G. Andersen , Jeffrey Dean , Gregory R. Ganger , Gauri Joshi , Michael Kaminksy
show 3 more authors
Michael Kozuch Zachary C. Lipton Padmanabhan Pillai
This is my paper · ORCID
classification cs.LGstat.ML
keywords selective-backpropexampleforwardtrainingacceleratesdeepexamplesfaster
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration. Selective-Backprop uses the output of a training example's forward pass to decide whether to use that example to compute gradients and update parameters, or to skip immediately to the next example. By reducing the number of computationally-expensive backpropagation steps performed, Selective-Backprop accelerates training. Evaluation on CIFAR10, CIFAR100, and SVHN, across a variety of modern image models, shows that Selective-Backprop converges to target error rates up to 3.5x faster than with standard SGD and between 1.02--1.8x faster than a state-of-the-art importance sampling approach. Further acceleration of 26% can be achieved by using stale forward pass results for selection, thus also skipping forward passes of low priority examples.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Active Data Curation Effectively Distills Large-Scale Multimodal Models

    cs.CV 2024-11 conditional novelty 7.0 of 10

    Selecting training data by a reference model's loss acts as an implicit distillation, and combining it with explicit distillation yields more FLOP-efficient vision-language models that beat prior SoTA on 27 benchmarks.

  2. Online Data Selection Is Implicit Alignment

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Online SFT data selection acts as an implicit preference model, shifting refusal rates, verbosity, and sycophancy in directions predictable from the selected data's attribute mixture.

  3. K-ABENA: K-Adaptive Backpropagation with Error-based N-exclusion Algorithm : (Compensated Loss-Based Sample Exclusion with Unbiased Gradient Estimation)

    cs.LG 2026-07 conditional novelty 6.0 of 10

    K-ABENA combines threshold-based sample exclusion with Horvitz-Thompson inverse-probability reweighting to achieve unbiased gradient estimation while saving 28-54% of per-epoch backward-pass compute.

  4. LoReUn: Data Itself Implicitly Provides Cues to Improve Machine Unlearning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    LoReUn, a plug-in loss-based reweighting strategy, improves approximate machine unlearning by focusing updates on hard-to-forget low-loss data points.

  5. Partial Forward Blocking: A Novel Data Pruning Paradigm for Lossless Training Acceleration

    cs.CV 2025-06 conditional novelty 6.0 of 10

    PFB prunes training samples based on the probability density of their shallow-layer features, blocking pruned samples from deep-layer forward and backward passes, and reports lossless or better accuracy with up to 33%...

  6. Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A fully online, loss-based reweighting scheme that down-weights low-loss samples during LLM pretraining yields small average benchmark gains at 1.4B and 7B scale, together with a convergence bound under convexity and ...

  7. CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    Selective-pivot-token training plus layer-averaging-with-recurrence reportedly gives 2.5x parameter compression on a small Korean LLM, but the efficiency claim lacks its decisive controls and the abstract advertises r...

  8. ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining

    cs.LG 2025-05 conditional novelty 5.0 of 10

    ESLM keeps only high-loss or high-entropy tokens in each batch via a value-at-risk threshold, cutting pretraining FLOPs by about 6% while roughly matching perplexity and downstream accuracy.

  9. DONOD: Efficient and Generalizable Instruction Fine-Tuning for LLMs via Model-Intrinsic Dataset Pruning

    cs.AI 2025-04 conditional novelty 5.0 of 10

    A model-intrinsic data pruning method using two weight-change metrics and TOPSIS reportedly improves instruction fine-tuning accuracy and robustness while using 70-80% less data.

  10. Instance-dependent Early Stopping

    cs.LG 2025-02 conditional novelty 5.0 of 10

    IES removes already-mastered training examples from backpropagation using a threshold on the second-order difference of their loss, achieving comparable accuracy with 10-50% less backpropagation.

  11. Navigating Towards Fairness with Data Selection

    cs.LG 2024-12 reject novelty 5.0 of 10

    A data selection method that replaces a clean holdout set with a zero-shot predictor plus peer prediction to pick training samples that reduce label bias and improve fairness.

Pith tools