REVIEW 7 cited by
Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Lately, post-training quantization methods have gained considerable attention, as they are simple to use, and require only a small unlabeled calibration set. This small dataset cannot be used to fine-tune the model without significant over-fitting. Instead, these methods only use the calibration set to set the activations' dynamic ranges. However, such methods always resulted in significant accuracy degradation, when used below 8-bits (except on small datasets). Here we aim to break the 8-bit barrier. To this end, we minimize the quantization errors of each layer separately by optimizing its parameters over the calibration set. We empirically demonstrate that this approach is: (1) much less susceptible to over-fitting than the standard fine-tuning approaches, and can be used even on a very small calibration set; and (2) more powerful than previous methods, which only set the activations' dynamic ranges. Furthermore, we demonstrate how to optimally allocate the bit-widths for each layer, while constraining accuracy degradation or model compression by proposing a novel integer programming formulation. Finally, we suggest model global statistics tuning, to correct biases introduced during quantization. Together, these methods yield state-of-the-art results for both vision and text models. For instance, on ResNet50, we obtain less than 1\% accuracy degradation --- with 4-bit weights and activations in all layers, but the smallest two. We open-sourced our code.
Forward citations
Cited by 7 Pith papers
-
Reclaiming Residual Knowledge: A Novel Paradigm to Low-Bit Quantization
CoRa reclaims quantization residuals in pre-trained ConvNets by searching low-rank adapter architectures instead of weights, matching SOTA accuracy on ImageNet in 3-4 bit settings with under 250 iterations on 1600 images.
-
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
GPTQ quantizes 175B-parameter GPT models to 3-4 bits per weight in one shot using approximate second-order information, achieving negligible accuracy degradation and 3-4x inference speedups.
-
A Reconfigurable and Representation-Adaptive ISA-Based Architecture for Efficient DNN Acceleration
A representation-adaptive ML-oriented ISA plus reconfigurable architecture with RNS dynamic precision delivers 5–10 TOPS/W and up to 1.2× efficiency over fixed-point while remaining programmable.
-
Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Models
ETBQ preconditions SGD-trained full-precision models under sampled quantization-error noise so subsequent low-bit PTQ retains more accuracy without QAT-style fake-quant training.
-
Efficient Reasoning on the Edge
LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.
-
Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Models
Pre-tuning a full-precision model with quantization-error noise and weight averaging before PTQ consistently improves low-bit (2-4 bit) quantized accuracy across CNN architectures and segmentation.
-
SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs
SignRoundV2 uses gradient-based adaptive mixed-precision allocation and stabilization techniques to achieve near-lossless performance in mixed MXFP quantization at average 4.5 bits and improved accuracy at 2-bit weigh...
Discussion (0). Sign in to comment.