Pith. sign in

REVIEW 22 cited by

Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.09145 v3 pith:WUAL7SQN submitted 2023-04-18 cs.CL

Xiuying Wei , Yunchen Zhang , Yuhang Li , Xiangguo Zhang , Ruihao Gong , Jinyang Guo , Xianglong Liu This is my paper · ORCID
classification cs.CL
keywords quantizationmodelschannel-wiseoutlierscalingshiftinglanguagesuppression
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Post-training quantization~(PTQ) of transformer language models faces significant challenges due to the existence of detrimental outliers in activations. We observe that these outliers are concentrated in specific channels and are asymmetric across channels. To address this issue, we propose the Outlier Suppression+~(OS+) framework, which contains the channel-wise shifting for asymmetry and channel-wise scaling for concentration. We show that these operations can be seamlessly migrated into subsequent modules while maintaining equivalence. Second, we propose a fast and stable scheme to calculate effective shifting and scaling values. The channel-wise shifting aligns the center of each channel for removal of outlier asymmetry. The channel-wise scaling quantitatively evaluates changes brought by migration and quantization for better quantization burden balance. We validate our OS+ under both standard and fine-grained quantization settings with models including BERT, OPT, BLOOM, BLOOMZ, and LLaMA. Comprehensive results across various tasks demonstrate the superiority of our approach. Especially, with standard quantization, OS+ can achieve near-floating-point performance on both small models and large language models on 8-bit and 6-bit. Besides, we establish a new state-of-the-art for 4-bit BERT with 15.5\% improvement. Our code is available at \url{https://github.com/ModelTC/Outlier_Suppression_Plus}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  2. DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model

    cs.AR 2025-05 conditional novelty 7.0 of 10

    A near-core decompression accelerator plus a 3D roofline model speeds up compressed LLM matrix multiplication by up to 4x in simulation.

  3. When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Under dynamic W4A4KV4 quantisation, a head-shared RoPE-aligned pairwise rotation that exactly minimizes a pooled variance surrogate still yields higher perplexity than full-head Hadamard mixing in all evaluated comparisons.

  4. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    MXSens allocates 8-bit precision to the 32 most sensitive columns per layer, 6-bit to moderately sensitive columns, and 4-bit elsewhere in MXINT, improving WikiText-2 perplexity over prior 4-bit LLM quantization methods.

  5. LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving

    cs.DC 2025-09 conditional novelty 6.0 of 10

    A W4A8 GEMM kernel using shift-based overflow-safe dequantization and a hardware-scheduled pipeline reports up to 2.9x speedup over prior W4A8 kernels.

  6. DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A post-training quantization method that combines learned channel scaling and power-of-two scaling keeps diffusion image quality high at 4-bit weight, 6-bit activation precision.

  7. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  8. BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    BASE-Q combines bias correction and asymmetric scaling under fixed rotations to improve 4-bit weight-activation quantization, narrowing the accuracy gap to full precision by up to 50.5% over prior rotation-based methods.

  9. Accelerating Large Language Models through Partially Linear Feed-Forward Network

    cs.LG 2025-01 reject novelty 6.0 of 10

    TARDIS folds two feed-forward weight matrices into one by linearly approximating activations in common input ranges, then recomputes outliers with a small predictor, claiming 80 percent FFN parameter reduction and up ...

  10. Dissecting Bit-Level Scaling Laws in Quantizing Vision Generative Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Token-based language-style vision models (VAR, LlamaGen) tolerate quantization better than diffusion models, and a custom TopKLD distillation loss pushes their low-bit scaling roughly one precision level higher.

  11. Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Arm CPU GEMV/GEMM kernels with SIMD-aware weight packing and a group-wise codebook quantizer improve LLM prefill by 3-3.2x and decode by up to 2x over llama.cpp Q4_0 while improving 2-3 bit perplexity.

  12. Efficient Reasoning on the Edge

    cs.LG 2026-03 accept novelty 5.5 of 10

    LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.

  13. PoTPTQ: A Two-step Power-of-Two Post-training for LLMs

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A two-step power-of-two quantization method improves low-bit LLaMA perplexity and dequantization speed, but reporting inconsistencies and missing code temper the result.

  14. Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Quaff shows that activation outlier channels keep their spatial positions during LLM fine-tuning, and exploits this stability to cut fine-tuning memory and latency with INT8 quantization while matching or beating full...

  15. Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

    cs.LG 2025-01 conditional novelty 5.0 of 10

    A post-training quantization scheme that uses per-group leading-one detection to keep four salient bits from an 8/16-bit integer base, achieving 4-bit weights, activations, and KV cache without fine-tuning or rotation.

  16. Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

    cs.CL 2025-05 reject novelty 4.0 of 10

    The paper claims that most massive activations in LLMs are residual-propagated 'fake' outliers that can be removed without accuracy loss.

  17. NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

    cs.LG 2025-05 conditional novelty 4.0 of 10

    NQKV is a block-wise 4-bit KV cache quantizer that uses a normal-distribution lookup table to let OPT models run with larger batches or longer contexts at negligible measured accuracy cost.

  18. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

  19. Deploying Foundation Model Powered Agent Services: A Survey

    cs.DC 2024-12 accept novelty 4.0 of 10

    This survey proposes a layered framework (execution, resource, model, agent, application) for deploying foundation-model-powered agent services across edge-cloud environments, and reviews optimization techniques at ea...

  20. KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning

    cs.LG 2025-05 reject novelty 3.0 of 10

    Kernel functions with vanishing third-moment conditions reduce the leading bias term in zeroth-order gradient estimates, yielding faster LLM fine-tuning than MeZO and HiZOO on several classification and generation benchmarks.

  21. On Accelerating Edge AI: Optimizing Resource-Constrained Environments

    cs.LG 2025-01 conditional novelty 2.0 of 10

    The paper argues that model compression, neural architecture search, and compiler optimizations work together to accelerate edge AI, but it provides no new experimental evidence.

  22. DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation

    cs.LG 2024-12

Pith tools