REVIEW 7 cited by
PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Existing weight-activation quantization methods for Large Language Models (LLMs) primarily address channel-wise outliers but often neglect token-wise outliers, which limits the accuracy of quantized models. In this work, we propose PrefixQuant, a novel quantization method that achieves state-of-the-art performance across various precision levels (W4A4KV4 and W4A8KV4) and granularities (dynamic and static quantization) by effectively isolating token-wise outliers. First, PrefixQuant eliminates token-wise outliers by prefixing outlier tokens in the KV cache, a process that is training-free and highly efficient (e.g., 1 minutes for Llama-3-70B). Second, PrefixQuant introduces new trainable parameters for block-wise training to compensate for quantization error. Our experiments show that PrefixQuant significantly outperforms existing dynamic quantization methods, even under coarser static quantization settings. For instance, PrefixQuant achieves an average accuracy improvement of +3.08 and +2.85 points over SpinQuant (dynamic quantization) on five zero-shot reasoning tasks under dynamic and static quantization settings, respectively, on W4A4KV4 Llama-3-8B. Additionally, we demonstrate up to 2.74x prefilling speedup and 2.16x decoding speedup for LLMs using W4A4 PrefixQuant. Our code is available at https://github.com/ChenMnZ/PrefixQuant.
Forward citations
Cited by 7 Pith papers
-
MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
MoBiQuant assigns each token its own quantization bit-width via learned routing over recursive residual bit slices, enabling elastic 2–6-bit LLM inference that matches static PTQ accuracy.
-
Activation Quantization of Vision Encoders Needs Prefixing Registers
Inserting precomputed universal register tokens in the middle layers of pretrained vision encoders shrinks activation outliers, and deleting emerging sink tokens, improves low-bit post-training quantization accuracy.
-
SiLQ: Simple Large Language Model Quantization-Aware Training
SiLQ fine-tunes 8B-parameter LLMs with quantized weights, activations, and cache for a small fraction of extra training tokens, matching or beating leading post-training quantization methods.
-
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
NSNQuant applies a Normalize-Shift-Normalize transform plus a Hadamard rotation to make KV cache channels match a standard normal distribution, so one codebook trained on random noise can quantize them without calibration.
-
MiniCPM4: Ultra-Efficient LLMs on End Devices
MiniCPM4-8B reportedly matches Qwen3-8B on standard benchmarks while using about 22% of the training tokens, and achieves large long-context speedups on edge devices.
-
Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers
The paper claims pause tokens strictly increase constant-precision, constant-depth Transformer expressivity from a subset of AC0 to AC0 (and log-precision to TC0), but the constant-precision proof is not sound as written.
-
Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.
Discussion (0). Sign in to comment.