Pith. sign in

REVIEW 22 cited by

Extreme Compression of Large Language Models via Additive Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.06118 v4 pith:TJDVYDGP submitted 2024-01-11 cs.LG cs.CL

Vage Egiazarian , Andrei Panferov , Denis Kuznedelev , Elias Frantar , Artem Babenko , Dan Alistarh This is my paper · ORCID
classification cs.LGcs.CL
keywords quantizationaqlmadditivecompressionextremebitsclassicenable
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The emergence of accurate open large language models (LLMs) has led to a race towards performant quantization techniques which can enable their execution on end-user devices. In this paper, we revisit the problem of "extreme" LLM compression-defined as targeting extremely low bit counts, such as 2 to 3 bits per parameter-from the point of view of classic methods in Multi-Codebook Quantization (MCQ). Our algorithm, called AQLM, generalizes the classic Additive Quantization (AQ) approach for information retrieval to advance the state-of-the-art in LLM compression, via two innovations: 1) learned additive quantization of weight matrices in input-adaptive fashion, and 2) joint optimization of codebook parameters across each transformer blocks. Broadly, AQLM is the first scheme that is Pareto optimal in terms of accuracy-vs-model-size when compressing to less than 3 bits per parameter, and significantly improves upon all known schemes in the extreme compression (2bit) regime. In addition, AQLM is practical: we provide fast GPU and CPU implementations of AQLM for token generation, which enable us to match or outperform optimized FP16 implementations for speed, while executing in a much smaller memory footprint.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FPTQuant: Function-Preserving Transforms for LLM Quantization

    cs.LG 2025-06 conditional novelty 7.0 of 10

    FPTQuant introduces function-preserving transforms that make transformer activations amenable to static 4-bit quantization with minimal inference overhead.

  2. Pushing the Limits of Large Language Model Quantization via the Linearity Theorem

    cs.LG 2024-11 conditional novelty 7.0 of 10

    A new theorem and method (HIGGS) make per-layer quantization error a reliable predictor of final model perplexity, enabling state-of-the-art data-free and dynamic bit-width LLM compression.

  3. A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Motion-aware vector quantization plus centroid-reuse GEMM and a custom accelerator yield multi-fold VLA speedups over GPU and prior accelerators with ~2.5% average success-rate drop.

  4. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  5. Breaking the Rounding Trap: Securing LLMs against Quantization-Conditioned Backdoors

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    QuantGuard uses differentiable rounding control to break quantization-boundary backdoors in LLMs, reducing post-quantization attack success to clean-model levels across six models and INT8/FP4/NF4.

  6. Smaller Models, Unexpected Costs: Trade-offs in LLM Quantization for Automated Program Repair

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    Empirical evaluation of 13 quantization configurations on 6 LLMs for APR shows reduced memory (up to 85%) but increased inference time/energy, different repaired problem sets with little overlap, and 48% of configs st...

  7. SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inference

    cs.LG 2026-02 conditional novelty 6.0 of 10

    SALAAD uses an ADMM-style optimizer and an adaptive controller to force LLM weights into sparse-plus-low-rank form during pretraining, so one checkpoint can be elastically deployed at many parameter budgets.

  8. From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    TAD, a 5,861-question benchmark, shows VLMs score far below humans on temporal understanding of driving videos, and an ego-trajectory text summary (TCogMap) substantially boosts their scores.

  9. CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    CCQ compresses LLMs to 2.0-2.75 bits per weight using convolutional codes and bit-shift decoding, shrinking 671B-parameter models to under 200GB.

  10. Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.

  11. PCDVQ: Enhancing Vector Quantization for Large Language Models via Polar Coordinate Decoupling

    cs.LG 2025-06 conditional novelty 6.0 of 10

    PCDVQ compresses LLM weights to 2 bits by quantizing vector directions and magnitudes separately with distribution-matched codebooks, reporting modest zero-shot accuracy gains over prior vector quantization baselines.

  12. NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

    cs.LG 2025-05 conditional novelty 6.0 of 10

    NSNQuant applies a Normalize-Shift-Normalize transform plus a Hadamard rotation to make KV cache channels match a standard normal distribution, so one codebook trained on random noise can quantize them without calibration.

  13. RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    RoSTE couples quantization-aware supervised fine-tuning with per-layer Hadamard rotation selection, reducing quantization outliers and improving 4-bit quantized LLM accuracy over SFT-then-PTQ baselines.

  14. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.

  15. BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference

    cs.CL 2025-01 conditional novelty 6.0 of 10

    BlockDialect assigns one of 16 FP4 'dialect' formats to each 32-64 element block of weights and activations, achieving near-full-precision accuracy with roughly 4-bit storage and integer-friendly MACs.

  16. ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals

    cs.LG 2024-12 conditional novelty 6.0 of 10

    ResQ quantizes LLMs to 4-bit with a low-rank 8-bit residual subspace chosen by PCA, beating prior 4-bit methods on perplexity and accuracy.

  17. Low-Rank Correction for Quantized LLMs

    stat.ML 2024-12 conditional novelty 6.0 of 10

    A low-rank correction computed jointly with quantized weights recovers most of the original accuracy for 4-bit weight-and-activation LLM quantization.

  18. Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens

    cs.LG 2024-11 conditional novelty 6.0 of 10

    The paper derives a scaling law for quantization-induced loss increase as a function of model size, training tokens, and bit width, and uses it to argue that low-bit quantization will hurt future fully trained LLMs.

  19. Diffusion Product Quantization

    cs.CV 2024-11 conditional novelty 6.0 of 10

    DPQ applies product quantization with compressed codebooks and DDPM-loss calibration to compress DiT diffusion models to 1-bit weights with FID 14.03, far better than VQ's 91.71.

  20. Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs

    cs.AI 2025-08 conditional novelty 5.0 of 10

    The study introduces TruthfulnessEval and reports that 4-bit quantization preserves simple true/false accuracy, but explicit 'lie' prompts make quantized and full-precision LLMs output falsehoods even when internal pr...

  21. Rethinking Post-Training Quantization: Introducing a Statistical Pre-Calibration Approach

    cs.LG 2025-01 reject novelty 3.0 of 10

    A soft-thresholding rule equivalent to selecting the top alpha percent of weights by magnitude is presented as a statistical pre-calibration for LLM quantization, with results close to but not clearly better than cali...

  22. The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

    cs.AI 2026-06 unverdicted novelty 2.0 of 10

    A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.

Pith tools