REVIEW 6 cited by
Deep Rewiring: Training very sparse deep networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without connectivity constraints. We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network. DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded. We demonstrate that DEEP R can be used to train very sparse feedforward and recurrent neural networks on standard benchmark tasks with just a minor loss in performance. DEEP R is based on a rigorous theoretical foundation that views rewiring as stochastic sampling of network configurations from a posterior.
Forward citations
Cited by 6 Pith papers
-
Synaptic clustering emerges from learning and supports covariance discrimination
Dendritic nonlinearities plus structural plasticity let a sparse conductance-based network form functional synapse clusters and solve a covariance discrimination task that linear point neurons cannot.
-
Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum
A Lyapunov-spectrum-based distance to the dense network lets hyperparameter search for pruned RNNs stop early and select models that beat both loss-based baselines and the dense originals.
-
NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling
Dynamic sparse training of multiple heads on a shared backbone outperforms full dense ensembles on ImageNet and C4 while using less compute.
-
On the Stability of Growth in Structural Plasticity
Newborn units in growing neural networks are forward-active but backward-starved, receiving weaker gradients than existing units and creating integration challenges that make growth less reliable than pruning in compl...
-
A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
Replicate-and-Quantize copies the busiest MoE expert as a quantized duplicate and compresses the least important expert, lowering a new Load-Imbalance Score by up to 1.4x while accuracy varies by roughly -1.2 to +3.0 points.
-
A Topological Improvement of the Overall Performance of Sparse Evolutionary Training: Motif-Based Structural Optimization of Sparse MLPs Project
Grouping neurons into blocks of size 2 with shared weights trims training time by 30 to 43 percent with accuracy losses of 1 to 4 percent on two datasets, by the paper's own measurements.
Discussion (0). Continue with ORCID to comment.