Allowing each quantization group to select among multiple 4-bit grids improves accuracy over single-grid FP4 for both post-training and pre-training of LLMs.
hub
A White Paper on Neural Network Quantization
45 Pith papers cite this work, alongside 13 external citations. Polarity classification is still indexing.
45
Pith papers citing it
13
external citations · Pith
abstract
While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.
hub tools
citation-role summary
background 3
citation-polarity summary
roles
background 3polarities
background 3representative citing papers
A quantization vector derived from a donor model via weight-space arithmetic can be added to a receiver model to improve post-PTQ Top-1 accuracy by up to 60 points in 3-bit settings without receiver-side QAT or data.
Learning quantization-aware linear paths in weight space yields a midpoint whose direct quantization matches quantization-aware training performance without using straight-through estimators.
JOMP jointly optimizes mixed-precision quantization parameters and bit widths across neural video coding frameworks, achieving rate-distortion performance comparable to DCVC-FM while cutting bit operations by 87.6%.
Q-ARVD introduces final-quality-aware frame weighting and outlier-aware adaptive dual-scale quantization to enable accurate low-bit inference for autoregressive video diffusion models.
QuBD extends algorithmic complexity estimation to quantized DNN weights, revealing that complexity decreases during learning, increases with overfitting, follows grokking patterns, and correlates with generalization.
AIS adaptively corrects non-stationary policy gradient bias in quantized LLM RL, matching BF16 performance while retaining 1.5-2.76x FP8 rollout speedup.
Floating-point weight formats in embedded neural networks suffer near-total accuracy loss from a single electromagnetic fault injection, while 8-bit integer formats retain substantially higher accuracy on the same hardware.
Four Over Six adaptively scales blocks in NVFP4 quantization to smaller FP4 values, making representable value distributions more uniform and reducing quantization error especially for near-maximal values.
DPQuant uses epoch-wise probabilistic layer rotation and DP loss sensitivity to quantize only a changing subset of layers, reducing accuracy degradation from quantization noise in DP-SGD and delivering up to 2.21x throughput gains with under 2% accuracy drop.
CoRa reclaims quantization residuals in pre-trained ConvNets by searching low-rank adapter architectures instead of weights, matching SOTA accuracy on ImageNet in 3-4 bit settings with under 250 iterations on 1600 images.
GPTQ quantizes 175B-parameter GPT models to 3-4 bits per weight in one shot using approximate second-order information, achieving negligible accuracy degradation and 3-4x inference speedups.
A framework called Voltron elastically distributes LLM inference across heterogeneous edge devices using layer-wise hybrid parallelism and mixed precision, achieving up to 16.5% higher accuracy than single-device execution while meeting QoS latency constraints.
QuantGuard uses differentiable rounding control to break quantization-boundary backdoors in LLMs, reducing post-quantization attack success to clean-model levels across six models and INT8/FP4/NF4.
Aggressive compression of recursive reasoners keeps local predictions intact but destroys global reasoning accuracy, recoverable with calibrated INT4 and detectable via carry-trajectory fidelity.
ScalePredictor is a dynamic post-training quantization framework for Vision Transformers that predicts per-instance scales from shallow-layer activation ranges using a Taylor-motivated polynomial projection module.
Derives smoothness-based PAC-Bayes derandomization bounds for deterministic predictors using Rademacher complexity of the Jensen gap class, yielding Jacobian/Hessian flatness terms and a practical regularizer tested on CIFAR-10.
AI LOD uses distance to select quantized precision levels of ML motion models for efficient NPC animation in games.
CKA-guided alignment of layerwise Gram matrices during NVFP4 QAD preserves internal representational geometry and improves downstream accuracy beyond KL-only output matching.
OScaR mitigates token norm imbalance via canalized rotation and omni-token scaling to enable near-lossless INT2 KV cache quantization with up to 3x decoding speedup and 5.3x memory reduction.
LBLLM achieves better accuracy than prior binarization methods for LLMs by decoupling weight and activation quantization through initialization, layer-wise distillation, and learnable activation scaling.
DeFakeQ introduces an adaptive bidirectional quantization method tailored for deepfake detectors that maintains detection accuracy while enabling real-time performance on resource-constrained edge devices.
Redefining residual errors to include compensation-aware discrepancies and realigning calibration to full-precision outputs improves GPTQ and GPTAQ performance on LLMs.
Proposes High-Precision Scoring (HPS) and Tie-aware Retrieval Metrics (TRM) to reduce tie-induced instability in low-precision retrieval evaluation.
citing papers explorer
-
Grid Games: The Power of Multiple Grids for Quantizing Large Language Models
Allowing each quantization group to select among multiple 4-bit grids improves accuracy over single-grid FP4 for both post-training and pre-training of LLMs.
-
Zero-Shot Quantization via Weight-Space Arithmetic
A quantization vector derived from a donor model via weight-space arithmetic can be added to a receiver model to improve post-PTQ Top-1 accuracy by up to 60 points in 3-bit settings without receiver-side QAT or data.
-
Neural Network Quantization by Learning Low-Loss Subspaces
Learning quantization-aware linear paths in weight space yields a midpoint whose direct quantization matches quantization-aware training performance without using straight-through estimators.
-
JOMP: Jointly-Optimized Mixed-Precision Quantization Across Neural Video Coding Frameworks and Buffering Strategies
JOMP jointly optimizes mixed-precision quantization parameters and bit widths across neural video coding frameworks, achieving rate-distortion performance comparable to DCVC-FM while cutting bit operations by 87.6%.
-
Q-ARVD: Quantizing Autoregressive Video Diffusion Models
Q-ARVD introduces final-quality-aware frame weighting and outlier-aware adaptive dual-scale quantization to enable accurate low-bit inference for autoregressive video diffusion models.
-
Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysis
QuBD extends algorithmic complexity estimation to quantized DNN weights, revealing that complexity decreases during learning, increases with overfitting, follows grokking patterns, and correlates with generalization.
-
AIS: Adaptive Importance Sampling for Quantized RL
AIS adaptively corrects non-stationary policy gradient bias in quantized LLM RL, matching BF16 performance while retaining 1.5-2.76x FP8 rollout speedup.
-
The Weight of a Bit: EMFI Sensitivity Analysis of Embedded Deep Learning Models
Floating-point weight formats in embedded neural networks suffer near-total accuracy loss from a single electromagnetic fault injection, while 8-bit integer formats retain substantially higher accuracy on the same hardware.
-
Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Four Over Six adaptively scales blocks in NVFP4 quantization to smaller FP4 values, making representable value distributions more uniform and reducing quantization error especially for near-maximal values.
-
DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
DPQuant uses epoch-wise probabilistic layer rotation and DP loss sensitivity to quantize only a changing subset of layers, reducing accuracy degradation from quantization noise in DP-SGD and delivering up to 2.21x throughput gains with under 2% accuracy drop.
-
Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization
CoRa reclaims quantization residuals in pre-trained ConvNets by searching low-rank adapter architectures instead of weights, matching SOTA accuracy on ImageNet in 3-4 bit settings with under 250 iterations on 1600 images.
-
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
GPTQ quantizes 175B-parameter GPT models to 3-4 bits per weight in one shot using approximate second-order information, achieving negligible accuracy degradation and 3-4x inference speedups.
-
Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence
A framework called Voltron elastically distributes LLM inference across heterogeneous edge devices using layer-wise hybrid parallelism and mixed precision, achieving up to 16.5% higher accuracy than single-device execution while meeting QoS latency constraints.
-
Breaking the Rounding Trap: Securing LLMs against Quantization-Conditioned Backdoors
QuantGuard uses differentiable rounding control to break quantization-boundary backdoors in LLMs, reducing post-quantization attack success to clean-model levels across six models and INT8/FP4/NF4.
-
What Survives When You Compress a Recursive Reasoner for the Edge?
Aggressive compression of recursive reasoners keeps local predictions intact but destroys global reasoning accuracy, recoverable with calibrated INT4 and detectable via carry-trajectory fidelity.
-
ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers
ScalePredictor is a dynamic post-training quantization framework for Vision Transformers that predicts per-instance scales from shallow-layer activation ranges using a Taylor-motivated polynomial projection module.
-
Smoothness-Based Derandomization of PAC-Bayes Bounds
Derives smoothness-based PAC-Bayes derandomization bounds for deterministic predictors using Rademacher complexity of the Jensen gap class, yielding Jacobian/Hessian flatness terms and a practical regularizer tested on CIFAR-10.
-
AI Level of Detail: Distance-Aware ML Model Precision Selection for Real-Time Human Motion Prediction in Games
AI LOD uses distance to select quantized precision levels of ML motion models for efficient NPC animation in games.
-
Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation
CKA-guided alignment of layerwise Gram matrices during NVFP4 QAD preserves internal representational geometry and improves downstream accuracy beyond KL-only output matching.
-
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
OScaR mitigates token norm imbalance via canalized rotation and omni-token scaling to enable near-lossless INT2 KV cache quantization with up to 3x decoding speedup and 5.3x memory reduction.
-
LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation
LBLLM achieves better accuracy than prior binarization methods for LLMs by decoupling weight and activation quantization through initialization, layer-wise distillation, and learnable activation scaling.
-
DeFakeQ: Enabling Real-Time Deepfake Detection on Edge Devices via Adaptive Bidirectional Quantization
DeFakeQ introduces an adaptive bidirectional quantization method tailored for deepfake detectors that maintains detection accuracy while enabling real-time performance on resource-constrained edge devices.
-
Rethinking Residual Errors in Compensation-based LLM Quantization
Redefining residual errors to include compensation-aware discrepancies and realigning calibration to full-precision outputs improves GPTQ and GPTAQ performance on LLMs.
-
Reliable Evaluation Protocol for Low-Precision Retrieval
Proposes High-Precision Scoring (HPS) and Tie-aware Retrieval Metrics (TRM) to reduce tie-induced instability in low-precision retrieval evaluation.
-
Sampling-Aware Quantization for Diffusion Models
A quantization technique for diffusion models that aligns sampling trajectories to preserve high-order sampler performance under quantization noise.
-
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
Seed-TTS models produce speech matching human naturalness and speaker similarity, with added controllability via self-distillation and reinforcement learning.
-
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
AWQ quantizes LLM weights to low bits by scaling salient channels based on activation statistics, outperforming prior methods on language, coding, math, and multi-modal benchmarks.
-
A Theoretical Framework for Stochastic Activity Prediction in Tensor Accelerator Wallace-Tree Multipliers
A proposed 'Stochastic Activity Prediction' scheme would freeze tensor multipliers on sparse operands to save power, but its headline error bound and optimality proof are incorrect.
-
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
Learned diagonal scaling matrices optimized with activation-aware loss reduce effective rank in LLM weight matrices and yield competitive perplexity and zero-shot results versus prior SVD methods on Llama 3.1 8B and Qwen3-8B.
-
MOSAIC: Efficient Mixture-of-Agent Scheduling via Adaptive Aggregation and Inference Concurrency
MOSAIC uses an Integer Linear Program scheduler for expert placement and prompt assignment plus adaptive aggregation to achieve 1.7-2.3x end-to-end speedup on 4-GPU MoA workloads while keeping accuracy within 0.1pp.
-
LoKA: Low-precision Kernel Applications for Recommendation Models At Scale
LoKA claims to make FP8 practical for large recommendation models via statistical probing, model adaptations, and accuracy-aware kernel dispatch.
-
Edge AI for Automotive Vulnerable Road User Safety: Deployable Detection via Knowledge Distillation
Knowledge distillation trains a 3.9x smaller YOLO student to retain 14.5% higher precision than direct training under INT8 quantization on BDD100K, exceeding the large teacher's FP32 precision while cutting false alarms.
-
A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models
KL divergence provides a superior forward-only metric for identifying quantization-sensitive parts in SSM-Transformer hybrids, outperforming MSE and SQNR and supporting practical mixed-precision deployment on edge devices.
-
Weight Group-wise Post-Training Quantization for Medical Foundation Model
Permutation-COMQ is a new post-training quantization algorithm that reorders weights within layers and uses only dot-product and rounding steps to deliver the highest reported accuracy for 2-, 4-, and 8-bit medical foundation models.
-
Sustainability Is Not Linear: Quantifying Performance, Energy, and Privacy Trade-offs in On-Device Intelligence
Empirical case study on a flagship Android device profiles energy, latency, and quality trade-offs across eight LLMs, revealing a quantization energy paradox and identifying mid-sized models as practical sweet spots.
-
You Had One Job: Per-Task Quantization Using LLMs' Hidden Representations
A per-layer, per-task bit-allocation rule based on hidden-activation entropy and variance reportedly matches full-precision TriviaQA accuracy within about one point — but the paper's tables and abstract are internally inconsistent.
-
Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inference
PLENA introduces a co-designed system with three optimization pathways for long-context agentic LLM inference, claiming up to 2.23x throughput over A100 and 4.04x energy efficiency.
-
Resource-aware Mixed-precision Quantization for Enhancing Deployability of Transformers for Time-series Forecasting on Embedded FPGAs
A resource-aware mixed-precision quantization technique plus BRAM-efficient VHDL changes enabled deployment of five previously non-deployable Transformer configurations on Spartan-7 FPGAs with 3% resource estimate accuracy.
-
SparrowSNN: A Hardware/software Co-design for Energy Efficient ECG Classification
SparrowSNN introduces SSF activation, a tunable hybrid ANN-SNN, and reconfigurable ASIC achieving SOTA accuracy on MIT-BIH ECG with 20-100x lower energy than prior ultra-low-power solutions.
-
MASQ: Accelerating Masked Diffusion via Stage-Wise Multi-Precision Quantization
MASQ claims up to 16.06x speedup and 4.18x energy gain over A100 for masked diffusion via stage-wise multi-precision quantization and specialized hardware units while preserving quality.
-
Deployment-Aligned Low-Precision Neural Architecture Search for Spaceborne Edge AI
Deployment-aligned low-precision NAS recovers about two-thirds of the accuracy drop from post-training quantization, achieving 0.826 mIoU on-device for a 95k-parameter model on Intel Movidius Myriad X without added complexity.
-
A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits
Combining pruning, quantization, and early exits in CNNs reduces inference latency and memory on real edge devices with minimal accuracy loss.
-
Quantization Robustness to Input Degradations for Object Detection
Benchmarks of YOLO models from nano to extra-large show that degradation-aware calibration for Static INT8 PTQ yields no broad robustness gains to input degradations over clean-data calibration, with limited exceptions for larger models under noise.
-
Automated Energy-Aware Time-Series Model Deployment on Embedded FPGAs for Resilient Combined Sewer Overflow Management
An automated hardware-aware pipeline deploys 8-bit Transformer and LSTM models on a Spartan-7 FPGA for real-time combined sewer overflow forecasting, reporting accuracy-energy trade-offs on real data.
-
Quantized Probabilistic AI for Gear Fault Diagnosis in Motor Drives
Quantizing weights and activations in a pre-trained probabilistic BNN for gear fault diagnosis yields 30-45% computational efficiency gains with no loss in accuracy or uncertainty estimates.