The reviewed record of science sign in
Pith

arxiv: 2307.13304 · v2 · pith:5SZUXPGZ · submitted 2023-07-25 · cs.LG · cs.CL

QuIP: 2-Bit Quantization of Large Language Models With Guarantees

Jerry Chee , Yaohui Cai , Volodymyr Kuleshov , Christopher De Sa This is my paper

Reviewed by Pithpith:5SZUXPGZopen to challenge →

classification cs.LG cs.CL
keywords quantizationquipincoherenceweightexistingfirsthessianlanguage
0
0 comments X
read the original abstract

This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from $\textit{incoherent}$ weight and Hessian matrices, i.e., from the weights being even in magnitude and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale quantization algorithm, and show that our theory also applies to an existing method, OPTQ. Empirically, we find that our incoherence preprocessing improves several existing quantization algorithms and yields the first LLM quantization methods that produce viable results using only two bits per weight. Our code can be found at https://github.com/Cornell-RelaxML/QuIP.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

    cs.PF 2026-05 unverdicted novelty 7.0

    A single fused int4 KV cache kernel on Apple Silicon outperforms fp16 in latency with 3x memory compression and near-zero quality loss on tested models.

  2. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

    cs.CL 2024-02 unverdicted novelty 7.0

    BitNet b1.58 shows that ternary 1.58-bit LLMs can match full-precision performance at substantially lower inference cost.

  3. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

    cs.LG 2024-01 conditional novelty 7.0

    Medusa augments LLMs with multiple decoding heads and tree-based attention to predict and verify several tokens in parallel, yielding 2.2-3.6x inference speedup via two fine-tuning regimes.

  4. HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models

    cs.LG 2026-06 unverdicted novelty 6.0

    HyperQuant unifies Hadamard transform, optimal lattice quantization, and entropy coding to outperform prior schemes on LLM weight and KV cache quantization down to 1.7 bits per scalar while preserving quality on a 19B...

  5. XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference

    cs.LG 2026-05 unverdicted novelty 6.0

    XFP introduces quality-targeted adaptive codebook quantization with sparse outlier separation that auto-selects parameters from cosine similarity floors, achieving high throughput and accuracy on Qwen3.5 models at low...

  6. BitRL: Reinforcement Learning with 1-bit Quantized Language Models for Resource-Constrained Edge Deployment

    cs.LG 2026-04 unverdicted novelty 6.0

    BitRL enables on-device RL agents via 1-bit quantized language models, delivering 10-16x memory reduction and 3-5x energy efficiency gains with 85-98% retained performance.

  7. BitNet: Scaling 1-bit Transformers for Large Language Models

    cs.CL 2023-10 unverdicted novelty 6.0

    BitNet shows that 1-bit Transformers can match the performance of 8-bit and FP16 models on language modeling with much smaller memory footprint and energy use, while following a similar scaling law.

  8. Influence-Inspired Spectral Rotations for Extreme Low-Bit LLM Quantization

    cs.LG 2026-05 unverdicted novelty 4.0

    A WHT rotation plus per-coordinate activation-energy rescaling before auto-round quantization lowers WikiText-2 perplexity 15-58% versus vanilla auto-round at W2A16 on models from 135M to 1.5B parameters.