REVIEW 13 cited by
Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. It can be generally categorized into unstructured fine-grained sparsity that zeroes out multiple individual weights distributed across the neural network, and structured coarse-grained sparsity which prunes blocks of sub-networks of a neural network. Fine-grained sparsity can achieve a high compression ratio but is not hardware friendly and hence receives limited speed gains. On the other hand, coarse-grained sparsity cannot concurrently achieve both apparent acceleration on modern GPUs and decent performance. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network, which can maintain the advantages of both unstructured fine-grained sparsity and structured coarse-grained sparsity simultaneously on specifically designed GPUs. Specifically, a 2:4 sparse network could achieve 2x speed-up without performance drop on Nvidia A100 GPUs. Furthermore, we propose a novel and effective ingredient, sparse-refined straight-through estimator (SR-STE), to alleviate the negative influence of the approximated gradients computed by vanilla STE during optimization. We also define a metric, Sparse Architecture Divergence (SAD), to measure the sparse network's topology change during the training process. Finally, We justify SR-STE's advantages with SAD and demonstrate the effectiveness of SR-STE by performing comprehensive experiments on various tasks. Source codes and models are available at https://github.com/NM-sparsity/NM-sparsity.
Forward citations
Cited by 13 Pith papers
-
Efficient Column-Wise N:M Pruning on RISC-V CPU
Column-wise N:M pruning plus fused im2col and data packing accelerates ResNet inference on RISC-V vector CPUs by up to 4x while keeping ImageNet top-1 accuracy within 2.1% of the dense model.
-
TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Masks
TSENOR computes transposable N:M masks up to hundreds of times faster than prior solvers by combining entropy-regularized optimal transport with a greedy plus local search rounding.
-
Systolic Sparse Tensor Slices: FPGA Building Blocks for Sparse and Dense AI Acceleration
An FPGA block with dense, 2:4, 1:3, and 1:4 sparse modes is shown in simulation to accelerate sparse GEMM and pruned vision models up to 3.52x over dense in-fabric acceleration.
-
MVQ:Towards Efficient DNN Compression and Acceleration with Masked Vector Quantization
Masked vector quantization (MVQ) prunes unimportant weights before clustering and uses masked k-means to build codebooks, improving accuracy over conventional VQ while cutting FLOPs and enabling a smaller, more effici...
-
Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework
SNOWS prunes vision networks in one shot by optimizing a K-step nonlinear reconstruction objective with Hessian-free Newton steps, improving accuracy over layer-wise least-squares methods.
-
AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment
AmoebaLLM enables instant extraction of arbitrary-shaped subnets from a single fine-tuned LLM, matching or beating dedicated compression methods.
-
LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
LExI sets a different number of active experts per layer, found by weight-only sensitivity profiling and evolutionary search, improving MoE inference throughput with little accuracy loss.
-
HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
HASSLE-free gives a fuller-Hessian alternating-minimization recipe for sparse-plus-low-rank LLM compression and reports perplexity improvements over OATS on Llama-3 and Llama-3.2 models.
-
How to keep pushing ML accelerator performance? Know your rooflines!
A JSSC survey organizing ML accelerator techniques around two roofline curves, one for throughput and one for energy efficiency, with the insight that their bottleneck knees often sit at different arithmetic intensities.
-
Symmetric Pruning of Large Language Models
SymWanda expresses Wanda and RIA pruning scores as special cases of a symmetric input-output reconstruction objective, and R2-DSnoT adds modest training-free post-pruning gains.
-
Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning
Meta-sparsity meta-learns the group-lasso penalty strength lambda via MAML, producing channel-sparse shared backbones for multi-task networks.
-
Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization
A PhD dissertation showing unified compression theory, personalized accelerated local training, and pruning methods that reduce communication costs in federated learning and maintain accuracy in LLM pruning.
-
Pruning for Sparse Diffusion Models based on Gradient Flow
Progressive soft pruning with the GraSP gradient-flow criterion gives small FID improvements over one-shot pruning on DDPMs at 50% sparsity.
Discussion (0). Continue with ORCID to comment.