Pith. sign in

REVIEW 4 cited by

Sparse Networks from Scratch: Faster Training without Losing Performance

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1907.04840 v2 pith:OPLEXANH submitted 2019-07-10 cs.LG cs.NEstat.ML

Tim Dettmers , Luke Zettlemoyer This is my paper
classification cs.LGcs.NEstat.ML
keywords sparsemomentumweightsperformancetrainingaccordingdemonstratedense
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently. Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer. Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms. Furthermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61x faster training. In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network. Additionally, we find that sparse momentum is insensitive to the choice of its hyperparameters suggesting that sparse momentum is robust and easy to use.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Hadamard-overparametrized sparse GMM Q-functions learn online off-policy policies that match deep RL with fewer parameters and geometrically interpretable components.

  2. Hybrid Least Squares/Gradient Descent Methods for DeepONets

    cs.LG 2025-08 conditional novelty 6.0 of 10

    A hybrid LS/GD training scheme for DeepONets, using a factored least-squares solve for the last layer, converges in far fewer work units than Adam-only training on advection, diffusion-reaction, and Poisson benchmarks.

  3. Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A Lyapunov-spectrum-based distance to the dense network lets hyperparameter search for pruned RNNs stop early and select models that beat both loss-based baselines and the dense originals.

  4. Constrained Hebbian Learning Supports Efficient Representational Allocation under Structural Constraints

    cs.LG 2026-07 conditional novelty 4.0 of 10

    A constrained Hebbian rule produces audiovisual representations with lower task-information cost (retained input information per unit of task-relevant information) than sparse backpropagation and DDTP at comparable ac...

Pith tools