Pith. sign in

REVIEW 23 cited by

FlatQuant: Flatness Matters for LLM Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.09426 v4 pith:ZPDO5CVL submitted 2024-10-12 cs.CL cs.LG

Yuxuan Sun , Ruikang Liu , Haoli Bai , Han Bao , Kang Zhao , Yuening Li , Jiaxin Hu , Xianzhi Yu
show 5 more authors
Lu Hou Chun Yuan Xin Jiang Wulong Liu Jun Yao
This is my paper · ORCID
classification cs.CLcs.LG
keywords quantizationflatquantactivationsaffinetransformationweightsapproachflatness
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recently, quantization has been widely used for the compression and acceleration of large language models (LLMs). Due to the outliers in LLMs, it is crucial to flatten weights and activations to minimize quantization error with equally spaced quantization points. Prior research explores various pre-quantization transformations to suppress outliers, such as per-channel scaling and Hadamard transformation. However, we observe that these transformed weights and activations can still exhibit steep and dispersed distributions. In this paper, we propose FlatQuant (Fast and Learnable Affine Transformation), a new post-training quantization approach that enhances the flatness of weights and activations. Our approach identifies optimal affine transformations for each linear layer, calibrated in hours via a lightweight objective. To reduce runtime overhead of affine transformation, we apply Kronecker product with two lightweight matrices, and fuse all operations in FlatQuant into a single kernel. Extensive experiments demonstrate that FlatQuant establishes a new state-of-the-art benchmark for quantization. For example, it achieves less than 1\% accuracy drop for W4A4 quantization on the LLaMA-3-70B model, surpassing SpinQuant by 7.5\%. Additionally, it provides up to 2.3x prefill speedup and 1.7x decoding speedup compared to the FP16 model. Code is available at: https://github.com/ruikangliu/FlatQuant.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Acceleration

    cs.LG 2025-07 conditional novelty 7.0 of 10

    OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.

  2. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  3. KronQ: LLM Quantization via Kronecker-Factored Hessian

    cs.LG 2026-07 accept novelty 6.5 of 10

    Kronecker-factored Hessian PTQ with bidirectional incoherence and joint-trace mixed precision yields stable 2-bit LLM weights where activation-only methods fail.

  4. When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Under dynamic W4A4KV4 quantisation, a head-shared RoPE-aligned pairwise rotation that exactly minimizes a pooled variance surrogate still yields higher perplexity than full-head Hadamard mixing in all evaluated comparisons.

  5. UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge

    cs.DC 2026-08 conditional novelty 6.0 of 10

    UnionSparse combines a shared-bitmap sparse format with a Tensor Core SpMM kernel to speed up low-bit sparse LLM decoding on edge GPUs by cutting index traffic.

  6. Hidden Language Consistency Phenomena in Reasoning LLMs

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Reasoning models often stop using the requested language as problems get harder, and this language breakdown can make accuracy look better than it is.

  7. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    LightRot uses grouped local rotation plus outlier alignment to make 4-bit LLaMA inference accurate and cheap, claiming 27.4 TOPS/W on a 28nm accelerator.

  8. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Rotation and fine-grained group quantization can work together if rotation spans several quantization groups and outlier channels are permuted onto harmonic Hadamard rows, enabling 4-bit LLM inference with integer-onl...

  9. KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using learned 32×32 Kronecker block transforms as an online activation smoother improves W4A4 image quality of PixArt-Sigma, SANA, and FLUX.1-schnell over SVDQuant and LoRaQ, with a kernel up to 14% faster than SmoothQuant.

  10. FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    FAIR-Calib is a frontier-aware instability-reweighted calibration framework for PTQ of dLLMs that minimizes reweighted hidden-state MSE to reduce frontier decision flips.

  11. Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference

    cs.AR 2026-02 conditional novelty 6.0 of 10

    Harmonia runs LLM inference with all activations in block floating point (BFP) and a 4-bit KV cache, reporting 3.08x average speedup, 2.03x energy savings, and under 1% accuracy loss on LongBench.

  12. Pushing the Limits of Block Rotations in Post-Training Quantization

    cs.LG 2026-01 conditional novelty 6.0 of 10

    Balancing per-block activation mass with a learned permutation before block Hadamard rotation restores up to 90% of the perplexity gap to full-vector rotation at block size 16 in 4-bit quantized LLMs.

  13. ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

    cs.LG 2026-01 conditional novelty 6.0 of 10

    By appending quantized residual channels to NVFP4 activations and duplicating the matching weights, ARCQuant reaches W4A8-level accuracy while keeping a single unified 4-bit GEMM.

  14. SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A training-free framework jointly quantizes a VLA model to 4 bits and prunes visual tokens, recovering or exceeding full-precision success rates at 1.93x speedup.

  15. Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A magnitude-compensation step that rescales the weights before a pruned layer, combined with iterative pruning, consistently improves perplexity and question-answering accuracy of layer-pruned LLMs.

  16. Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design

    cs.CL 2025-05 conditional novelty 6.0 of 10

    EAGLE-2 loses most of its speedup on 4-bit quantized LLMs, and a hierarchical draft-then-sequence scheme restores 1.31x speedup over EAGLE-2 on W4A16 Llama-3-70B.

  17. BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    BASE-Q combines bias correction and asymmetric scaling under fixed rotations to improve 4-bit weight-activation quantization, narrowing the accuracy gap to full precision by up to 50.5% over prior rotation-based methods.

  18. MBQ: Modality-Balanced Quantization for Large Vision-Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A modality-weighted quantization method improves accuracy of 3-bit and 4-bit vision-language models by protecting sensitive language tokens during calibration.

  19. Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A training-free compression pipeline that updates calibration activations layer by layer and re-allocates ranks iteratively improves compressed LLM accuracy by up to a few points on five zero-shot tasks.

  20. Quantize with Confidence? An Empirical Study of Quantization for Code Generation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.

  21. SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Combining SmoothQuant-style channel scaling with QuaRot's Hadamard rotation reduces 4-bit weight-activation quantization error in LLMs by 10-30% of the gap to FP16.

  22. Turning LLM Activations Quantization-Friendly

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Channel-wise scaling applied before Hadamard rotation reduces 4-bit layer-wise quantization error in LLaMA2-7B, especially on down-projection layers with massive activation outliers.

  23. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Pith tools