REVIEW 15 cited by
A Survey of Model Compression and Acceleration for Deep Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A Survey of Model Compression and Acceleration for Deep Neural Networks
read the original abstract
Deep neural networks (DNNs) have recently achieved great success in many visual recognition tasks. However, existing deep neural network models are computationally expensive and memory intensive, hindering their deployment in devices with low memory resources or in applications with strict latency requirements. Therefore, a natural thought is to perform model compression and acceleration in deep networks without significantly decreasing the model performance. During the past five years, tremendous progress has been made in this area. In this paper, we review the recent techniques for compacting and accelerating DNN models. In general, these techniques are divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation. Methods of parameter pruning and quantization are described first, after that the other techniques are introduced. For each category, we also provide insightful analysis about the performance, related applications, advantages, and drawbacks. Then we go through some very recent successful methods, for example, dynamic capacity networks and stochastic depths networks. After that, we survey the evaluation matrices, the main datasets used for evaluating the model performance, and recent benchmark efforts. Finally, we conclude this paper, discuss remaining the challenges and possible directions for future work.
Forward citations
Cited by 15 Pith papers
-
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
GPTQ-intrinsic LoRA augments GPTQ with intrinsic low-rank compensation via Hessian modification to achieve layer-wise reconstruction bounds that match information-theoretic lower bounds under structural assumptions.
-
MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
A single calibration that marginalizes layer distortion over random quantized upstream contexts yields budget-agnostic bit allocations that beat FP16-scored adaptive baselines across Llama-3.2-3B, Llama-2-7B, and Mistral-7B.
-
Contrastive Predictive Coding with Compression for Enhanced Channel State Feedback in Wireless Networks
Integrating CPC into 3GPP CSI compression yields age-aware latent prediction at fixed 64-bit overhead, with CPC-before exceeding 90% SGCS and 32× lighter decoder compute than the baseline.
-
Nonlinear Bipolar Compensation: Handling Outliers in Post-Training Quantization
Nonlinear Bipolar Compensation with Bipolar Logarithmic Transformation reduces outlier effects in post-training quantization by performing compensation in a compressed transformed space.
-
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
DECO matches dense model performance at 20% expert activation via ReLU-based routing with learnable scaling and the NormSiLU activation, plus a 3x real-hardware speedup.
-
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
DECO sparse MoE matches dense Transformer performance at 20% expert activation with a 3x hardware inference speedup.
-
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
DECO is a sparse MoE architecture with ReLU-based routing, learnable expert scaling, and NormSiLU activation that matches dense Transformer performance at 20% expert activation and delivers 2.93x speedup on Jetson AGX Orin.
-
Low Rank Based Subspace Inference for the Laplace Approximation of Bayesian Neural Networks
Derives optimal low-rank subspace for Laplace approx in BNNs, provides scalable outperforming version, and new comparison metric.
-
ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
ASVD compresses LLMs by 10-30% and KV caches by 50% via activation-aware SVD that absorbs outliers into transformed weights and calibrates per-layer sensitivity.
-
Language Modeling Is Compression
Large language models serve as strong general-purpose lossless compressors for text, images, and audio, outperforming domain-specific methods and revealing insights into scaling, tokenization, and in-context learning.
-
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Loss-aware rank allocation plus residual-stream output correction yields substantially lower WikiText-2 perplexity than prior SVD LLM compressors at 60% compression.
-
Rethinking Depth: A study of the Recursive-Transformer for Speech Recognition
Experimental study finds Recursive-Transformer for ASR encoders achieves comparable performance with 66% fewer parameters when limited recursion is applied in the latent space.
-
Sparse-on-Dense: Area and Energy-Efficient Computing of Sparse Neural Networks on Dense Matrix Multiplication Accelerators
Sparse neural networks achieve better area and energy efficiency when executed on dense matrix multiplication accelerators using a Sparse-on-Dense approach than on dedicated sparse accelerators.
-
FedKLPR: KL-Guided Pruning-Aware Federated Learning for Person Re-Identification
FedKLPR adds KL-regularized training, prune-weighted aggregation, and cross-round recovery to federated learning for re-ID, claiming 40-42% lower communication on ResNet-50 with competitive accuracy across eight datasets.
-
FedKLPR: KL-Guided Pruning-Aware Federated Learning for Person Re-Identification
FedKLPR introduces KL-divergence-guided training, pruning-aware weighted aggregation, and cross-round recovery to achieve 40-42% communication reduction on ResNet-50 while preserving competitive accuracy in federated ...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.