REVIEW 7 cited by
DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations that impede efficient low-bit representation. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. However, these methods struggle with smoothing Massive Outliers that display significantly larger values, which leads to significant performance degradation in low-bit quantization. In this paper, we introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers. First, DuQuant starts by constructing the rotation matrix, using specific outlier dimensions as prior knowledge, to redistribute outliers to adjacent channels by block-wise rotation. Second, We further employ a zigzag permutation to balance the distribution of outliers across blocks, thereby reducing block-wise variance. A subsequent rotation further smooths the activation landscape, enhancing model performance. DuQuant simplifies the quantization process and excels in managing outliers, outperforming the state-of-the-art baselines across various sizes and types of LLMs on multiple tasks, even with 4-bit weight-activation quantization. Our code is available at https://github.com/Hsu1023/DuQuant.
Forward citations
Cited by 7 Pith papers
-
FPTQuant: Function-Preserving Transforms for LLM Quantization
FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.
-
When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation
Under dynamic W4A4KV4 quantisation, a head-shared RoPE-aligned pairwise rotation that exactly minimizes a pooled variance surrogate still yields higher perplexity than full-head Hadamard mixing in all evaluated comparisons.
-
Systematic Outliers in Large Language Models
The paper identifies activation, weight, and attention outliers as a single phenomenon caused by softmax attention and demonstrates that explicit context-aware scaling eliminates them.
-
AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference
AMXFP4 is an asymmetric 4-bit microscaling format that improves calibration-free LLM inference accuracy over MXFP4 and rotation-based methods.
-
Efficient Reasoning on the Edge
LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.
-
SWSC: Shared Weight for Similar Channel in LLM
SWSC combines channel K-means clustering with an SVD low-rank error correction to compress LLM weights, and reports lower perplexity than RTN quantization on Llama-2-7B Q and K projections at 2 to 3 average bits.
-
LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment
LSAQ assigns higher quantization precision to layers deemed important by the overlap of top-k input and output token sets, and reports small accuracy and perplexity gains over a cosine-similarity baseline.
Discussion (0). Continue with ORCID to comment.