REVIEW 16 cited by
Accelerating Sparse Deep Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Accelerating Sparse Deep Neural Networks
read the original abstract
As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy, verifying it on a wide range of common tasks and model architectures. This workflow makes it easy to prepare accurate models for efficient deployment on Sparse Tensor Cores.
Forward citations
Cited by 16 Pith papers
-
LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
LEAP learns unstructured pruning masks end-to-end for LLMs via Gumbel-sigmoid Bernoulli relaxation and reports +2.59 average zero-shot accuracy gain over ADMM at 50-60% sparsity across five model families.
-
LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models
LEAP replaces intractable categorical mask parameterization with a differentiable per-weight Bernoulli relaxation, delivering +2.59 average zero-shot accuracy gain over the best layer-wise baseline across 0.5B-8B LLMs...
-
MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
MosaicKV achieves up to 16x attention speedup, 4.8x lower decode latency, 7.3x higher throughput, and 3x memory reduction with 1.76% accuracy loss via dynamic two-D KV cache compression and management on H800 GPUs.
-
SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference
SpenseGPT introduces a hybrid sparse-dense weight format and one-shot pruning that delivers 1.2x end-to-end LLM decoding speedup on B200 GPUs with FP8 while preserving accuracy on Qwen3-32B and Seed-OSS-36B.
-
HORST: Composing Optimizer Geometries for Sparse Transformer Training
HORST uses non-commutative operator composition and a hyperbolic mirror map to combine stability from adaptive optimizers with L1 sparsity bias, outperforming AdamW across sparsity levels on vision and language tasks.
-
SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask
SparseForge achieves 57.27% zero-shot accuracy on LLaMA-2-7B at 2:4 sparsity using only 5B retraining tokens, beating the dense baseline and nearly matching a 40B-token SOTA method.
-
SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models
SLaB compresses LLM weights via sparse-lowrank-binary decomposition guided by activation-aware scores, achieving up to 36% lower perplexity than prior methods at 50% compression on Llama models.
-
MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs
MaskPro learns categorical distributions over groups of M weights to generate exact (N:M) sparsity via N-way sampling without replacement and stabilizes training with a moving average tracker of loss residuals.
-
Direct Methods for Calculating PseudoInverses
Direct bidiagonal-based methods (CK deflation, dual normals, inplace, Lanczos reinterpretation) compute Moore-Penrose and group pseudoinverses while mitigating conditioning, rank, and sparsity issues for large-scale r...
-
Small LLMs: Pruning vs. Training from Scratch
Pruned initializations from an 8B model outperform random starts with equal training tokens, but with full token budgets fine-grained pruning retains advantage while coarse structured pruning does not.
-
Small LLMs: Pruning vs. Training from Scratch
Controlled experiments show pruning a large LLM outperforms training small models from scratch with matched token budgets, but the advantage narrows with extra tokens or coarse pruning.
-
Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
Marchenko-Pastur random-matrix pruning of DNNs yields theoretical certificates for accuracy preservation under small fine-tuning and empirical ImageNet results with 50-60% MAC reduction and sub-2pp accuracy drops on V...
-
ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
ELAS pre-trains low-rank LLMs by applying 2:4 activation sparsity after squared ReLU to cut memory and accelerate training with minimal performance loss.
-
Adaptive Norm-Based Regularization for Neural Networks
Covariance-aware ridge and combined l1-l2 regularizers for neural networks yield better predictive performance and complexity control than standard penalties in simulations and applications to cooling-load prediction ...
-
HieraSparse: Hierarchical Semi-Structured Sparse KV Attention
HieraSparse delivers a hierarchical semi-structured sparse KV attention system that achieves 1.2x KV compression and 4.57x decode attention speedup versus prior unstructured sparsity methods at equivalent sparsity, pl...
-
SCENIC: Semantic-Conditioned Edge-Aware Neural Framework for Structured IoT Command Generation
SCENIC framework reports up to 99% exact match on structured IoT command generation using sub-0.2B models, with pruned INT8 versions retaining 91% EM@1 after 25% size reduction.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.