REVIEW 26 cited by
Model compression via distillation and quantization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger teacher networks into smaller student networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher, into the training of a student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to full-precision teacher models, while providing order of magnitude compression, and inference speedup that is linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.
Forward citations
Cited by 26 Pith papers
-
ELFuzz: Efficient Input Generation via LLM-driven Synthesis Over Fuzzer Space
ELFuzz automatically evolves LLM-written input generators for large programs, outperforming grammar-based fuzzers in coverage and bug finding on seven benchmarks.
-
XTransfer: Modality-Agnostic Few-Shot Model Transfer for Human Sensing at the Edge
XTransfer transfers pre-trained models across sensing modalities with few labeled samples by repairing layer-wise feature misalignment and recombining useful layers, achieving top accuracy and lower resource costs.
-
Loss-Aware Automatic Selection of Structured Pruning Criteria for Deep Neural Network Acceleration
LAASP prunes neural networks during training by greedily selecting the best layer and filter-importance criterion at each step using the network's loss on a data subset.
-
Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart
BWRF improves quantization-aware training by grafting full-precision blocks onto the low-precision model during training, producing mixed-precision guides that raise ImageNet and CIFAR-10 accuracy at 2 to 4 bits.
-
DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization
DOLLAR combines variational score and consistency distillation for few-step video generation plus latent reward optimization, reporting 82.57 VBench score and up to 278x speedup over the teacher diffusion model for 12...
-
RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
A GAN-guided rectified training scheme narrows the accuracy gap between binary and full-precision convolutional networks on classification and tracking.
-
Fast Tensorization of Neural Networks via Slice-wise Feature Distillation
A slice-wise feature distillation framework for independent tensorization of neural network slices to achieve scalable compression with reduced fine-tuning costs.
-
Low-Rank Augmented Implicit Neural Representation for Unsupervised High-Dimensional Quantitative MRI Reconstruction
LoREIN couples implicit neural representations with low-rank temporal subspace modeling to reconstruct quantitative MRI maps and weighted images directly from undersampled k-space.
-
ReverB-SNN: Reversing Bit of the Weight and Activation for Spiking Neural Networks
ReverB-SNN replaces binary spikes with real-valued spikes and real weights with binary weights, keeping SNN inference addition-only while improving accuracy.
-
EfficientLLM: Efficiency in Large Language Models
A large-scale benchmark of LLM efficiency techniques finds that every method trades off one resource for another, with the best choice depending on model scale, task, and hardware.
-
Adaptive Pruning of Pretrained Transformer via Differential Inclusions
A single differential-inclusion search over masks produces a whole family of pruned transformers at different sparsity levels from one pretrained model.
-
Optimising TinyML with Quantization and Distillation of Transformer and Mamba Models for Indoor Localisation on Edge Devices
A quantized transformer and a compact Mamba model can classify indoor location with moderate accuracy within 32-64 KB model sizes, but on-device RAM usage is not measured.
-
Patient Knowledge Distillation for BERT Model Compression
Distilling BERT through several intermediate hidden layers (Patient-KD) improves a shallow student's accuracy on GLUE and RACE compared with last-layer-only distillation.
-
Knowledge Distillation for Sensing-Assisted Long-Term Beam Tracking in mmWave Communications
Knowledge distillation creates a compact neural network for long-term beam tracking in mmWave communications that matches a larger teacher's accuracy with far fewer parameters and shorter input sequences.
-
Resource-Efficient Automatic Software Vulnerability Assessment via Knowledge Distillation and Particle Swarm Optimization
A PSO-guided knowledge distillation framework compresses a CodeBERT vulnerability assessor to 0.6% of its original size while retaining 89.3% of its accuracy.
-
The Promise of Spiking Neural Networks for Ubiquitous Computing: A Survey and New Perspectives
A survey of 76 spiking-neural-network papers on time-series sensor data, organized into six application domains, with recommendations for software and neuromorphic hardware.
-
Tensorization is a powerful but underexplored tool for compression and interpretability of neural networks
The paper makes the case that tensorized neural networks offer valuable compression, scaling, and interpretability advantages that the deep learning community has not yet fully exploited.
-
On Hardening DNNs against Noisy Computations
On CIFAR-10, quantization-aware training with large constant scaling factors improves noise robustness, but noisy training (injecting matching Gaussian noise during training) gives far larger robustness gains, and qua...
-
Pan-infection Foundation Framework Enables Multiple Pathogen Prediction
A teacher-student knowledge distillation framework trained on 11,247 blood transcriptomes reports high AUCs for pan-infection, four pathogens, and sepsis diagnosis.
-
Multi-Modality Distillation via Learning the teacher's modality-level Gram Matrix
Proposes a modality relation distillation method that transfers teacher modality relationships via the modality-level Gram Matrix.
-
Reinforcement Learning: From Algorithms To Foundation Models
A dissertation uniting the author's published results: non-exploitable Nash-DQN policies and the FightLadder benchmark for games, plus diffusion/consistency-model world models for RL — a compilation rather than new results.
-
Recent Advances and Trends in Learning-based 3D Representations
A survey of 3D representation families that highlights the shift from discrete explicit formats to continuous implicit neural and primitive-based ones.
-
Vision Generalist Model: A Survey
A structured review of vision generalist models, classifying them into encoding-based and sequence-to-sequence frameworks and summarizing datasets, benchmarks, techniques, and open problems.
-
A Survey on Foundation Models for Personalized Federated Intelligence
The survey introduces personalized federated intelligence (PFI) as a framework integrating federated learning and foundation models to support privacy-aware personalization of AI models.
-
Frugal Machine Learning for Energy-efficient, and Resource-aware Artificial Intelligence
A survey paper that defines and categorizes Frugal Machine Learning methods but introduces no new techniques or empirical results.
-
Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
A survey of knowledge distillation, quantization, and pruning for compressing LLMs to run on resource-constrained edge devices, with no new experimental results.
Discussion (0). Continue with ORCID to comment.