REVIEW 18 cited by
A Survey of Model Compression and Acceleration for Deep Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Deep neural networks (DNNs) have recently achieved great success in many visual recognition tasks. However, existing deep neural network models are computationally expensive and memory intensive, hindering their deployment in devices with low memory resources or in applications with strict latency requirements. Therefore, a natural thought is to perform model compression and acceleration in deep networks without significantly decreasing the model performance. During the past five years, tremendous progress has been made in this area. In this paper, we review the recent techniques for compacting and accelerating DNN models. In general, these techniques are divided into four categories: parameter pruning and quantization, low-rank factorization, transferred/compact convolutional filters, and knowledge distillation. Methods of parameter pruning and quantization are described first, after that the other techniques are introduced. For each category, we also provide insightful analysis about the performance, related applications, advantages, and drawbacks. Then we go through some very recent successful methods, for example, dynamic capacity networks and stochastic depths networks. After that, we survey the evaluation matrices, the main datasets used for evaluating the model performance, and recent benchmark efforts. Finally, we conclude this paper, discuss remaining the challenges and possible directions for future work.
Forward citations
Cited by 18 Pith papers
-
MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models
A single calibration that marginalizes layer distortion over random quantized upstream contexts yields budget-agnostic bit allocations that beat FP16-scored adaptive baselines across Llama-3.2-3B, Llama-2-7B, and Mistral-7B.
-
Input convex neural networks as surrogates in mathematical optimisation
ICNN surrogates admit an LP-only embedding whose epigraph plus concave envelope equals the convex hull of the network's graph, enabling a branch-and-bound solver that avoids binary variables.
-
Contrastive Predictive Coding with Compression for Enhanced Channel State Feedback in Wireless Networks
Integrating CPC into 3GPP CSI compression yields age-aware latent prediction at fixed 64-bit overhead, with CPC-before exceeding 90% SGCS and 32× lighter decoder compute than the baseline.
-
Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives
Dobi-SVD compresses LLMs via differentiable SVD rank selection, IPCA-based weight reconstruction, and quantized storage remapping, reporting competitive perplexity at 40% parameters.
-
Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models
IRR restores safety to fine-tuned LLMs by masking delta parameters that conflict with a safety vector, then recalibrating the survivors with inverse-Hessian compensation to preserve task performance.
-
Is Oracle Pruning the True Oracle?
A large empirical study finds that the loss of a pruned network before retraining hardly predicts its accuracy after retraining, casting doubt on oracle pruning as a foundation for modern pruning criteria.
-
LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression
Loss-aware rank allocation plus residual-stream output correction yields substantially lower WikiText-2 perplexity than prior SVD LLM compressors at 60% compression.
-
CoT-X: An Adaptive Framework for Cross-Model Chain-of-Thought Transfer and Optimization
An adaptive summarization framework compresses chain-of-thought traces and transfers them across model families, claiming up to 40.5% accuracy gains over truncation on medical QA and 84% fewer configuration evaluation...
-
UnIT: Scalable Unstructured Inference-Time Pruning for MAC-efficient Neural Inference on MCUs
UnIT enables unstructured, input-aware pruning of individual MACs on MCUs without retraining, reporting up to 82% MAC reduction and up to 84% energy savings at 0.48 to 7% accuracy loss.
-
Smooth Model Compression without Fine-Tuning
Training a ResNet-18 with smoothness penalties on weights, then compressing via truncated SVD, keeps 91% CIFAR-10 accuracy at 70% sparsity with no post-compression fine-tuning.
-
EfficientLLM: Efficiency in Large Language Models
A large-scale benchmark of LLM efficiency techniques finds that every method trades off one resource for another, with the best choice depending on model scale, task, and hardware.
-
Equally Critical: Samples, Targets, and Their Mappings in Datasets
A mapping that gives all augmented views of one image the same teacher soft label outperforms per-augmentation knowledge distillation and one-hot training in final accuracy under the paper's decoupled evaluation.
-
Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition
A shared low-rank basis plus sparse per-layer projections compresses transformer MLPs by up to 75% with small accuracy loss.
-
Restricted Recurrent Neural Networks
Partial sharing of input and hidden-state weights in RNN, LSTM, and GRU yields about 50% parameter reduction with roughly unchanged perplexity on two language-modeling benchmarks.
-
JEDI-net: a jet identification algorithm based on interaction networks
JEDI-net, an interaction-network jet tagger, outperforms DNN, CNN, and GRU taggers on a five-class simulated LHC jet dataset.
-
Modality Alignment with Multi-scale Bilateral Attention for Multimodal Recommendation
MambaRec improves multimodal recommendation accuracy on Baby, Sports, and Clothing datasets through local dilated-attention alignment and global MMD/contrastive alignment.
-
Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilities
Foundation-model perception for autonomous driving is surveyed through four capability lenses: generalized knowledge, spatial understanding, multi-sensor robustness, and temporal understanding.
-
Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
A bounded-L1 regularizer combined with exponential gating layers prunes neural network channels to exactly zero during training, compressing standard models by 30 to 75 percent with little accuracy loss.
Discussion (0). Continue with ORCID to comment.