Pith. sign in

REVIEW 4 cited by

Dynamic Sparse Training with Structured Sparsity

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.02299 v4 pith:6DYAXO3O submitted 2023-05-03 cs.LG cs.CV

Mike Lasby , Anna Golubeva , Utku Evci , Mihai Nica , Yani Ioannou This is my paper
classification cs.LGcs.CV
keywords sparsesparsitystructuredtraininginferenceachievedensedynamic
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training, matching the generalization of dense models while enabling sparse training and inference. Although the resulting models are highly sparse and theoretically less computationally expensive, achieving speedups with unstructured sparsity on real-world hardware is challenging. In this work, we propose a sparse-to-sparse DST method, Structured RigL (SRigL), to learn a variant of fine-grained structured N:M sparsity by imposing a constant fan-in constraint. Using our empirical analysis of existing DST methods at high sparsity, we additionally employ a neuron ablation method which enables SRigL to achieve state-of-the-art sparse-to-sparse structured DST performance on a variety of Neural Network (NN) architectures. Using a 90% sparse linear layer, we demonstrate a real-world acceleration of 3.4x/2.5x on CPU for online inference and 1.7x/13.0x on GPU for inference with a batch size of 256 when compared to equivalent dense/unstructured (CSR) sparse layers, respectively.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.

  2. Dynamic Sparse Training of Diagonally Sparse Networks

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A dynamic sparse training method that restricts weights to a learnable set of diagonals, preserving sparsity in both forward and backward passes to obtain GPU speedups at accuracy close to unstructured sparsity.

  3. SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networks

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Learned per-layer permutations bring structured sparse training (block, N:M, diagonal) close to unstructured sparse-training accuracy at 90-95% sparsity, with inference speedups up to 2.9x.

  4. Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization

    cs.LG 2025-09 conditional novelty 3.0 of 10

    A PhD dissertation showing unified compression theory, personalized accelerated local training, and pruning methods that reduce communication costs in federated learning and maintain accuracy in LLM pruning.

Pith tools