pith. sign in

arxiv: 2506.18879 · v1 · pith:TULFDY5Nnew · submitted 2025-06-23 · 💻 cs.CL · cs.AI

CommVQ: Commutative Vector Quantization for KV Cache Compression

Junyan Li , Yang Zhang , Muhammad Yusuf Hassan , Talha Chafekar , Tianle Cai , Zhile Ren , Pengsheng Guo , Foroozan Karimzadeh
show 3 more authors
Colorado Reed Chong Wang Chuang Gan
This is my paper
classification 💻 cs.CL cs.AI
keywords quantizationcachecodebookcommutativecommvqcontextaccuracyadditive
0
0 comments X
read the original abstract

Large Language Models (LLMs) are increasingly used in applications requiring long context lengths, but the key-value (KV) cache often becomes a memory bottleneck on GPUs as context grows. To address this, we propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference. We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache, which can be decoded via simple matrix multiplication. To further reduce computational costs during decoding, we design the codebook to be commutative with Rotary Position Embedding (RoPE) and train it using an Expectation-Maximization (EM) algorithm. This enables efficient integration of decoding into the self-attention mechanism. Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook. Experiments on long-context benchmarks and GSM8K show that our method reduces FP16 KV cache size by 87.5% with 2-bit quantization, while outperforming state-of-the-art KV cache quantization methods. Notably, it enables 1-bit KV cache quantization with minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context length on a single RTX 4090 GPU. The source code is available at: https://github.com/UMass-Embodied-AGI/CommVQ.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

    cs.LG 2026-06 unverdicted novelty 7.0

    STAR-KV applies differentiable soft thresholding for per-head and per-block adaptive low-rank KV cache compression, combined with hybrid decomposition and low-rank-aware quantization, achieving up to 75% compression a...

  2. SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

    cs.LG 2026-04 unverdicted novelty 6.0

    Token-wise INT4 KV-cache quantization plus block-diagonal Hadamard rotation recovers nearly all accuracy lost by naive INT4 while adding zero end-to-end overhead under paged serving constraints.

  3. Online Vector Quantized Attention

    cs.LG 2026-02 unverdicted novelty 6.0

    OVQ-attention delivers linear-time constant-memory sequence mixing via sparse Gaussian-mixture-based memory updates, matching self-attention performance on tasks up to 64k length while using far less memory.

  4. Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization

    cs.LG 2026-02 unverdicted novelty 6.0

    Quant VideoGen reduces KV cache memory by up to 7 times in autoregressive video diffusion models via semantic aware smoothing and progressive residual quantization, achieving better quality than baselines with under 4...

  5. EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture

    cs.AR 2026-05 unverdicted novelty 5.0

    EVA is a vector-quantization hardware architecture that transforms LLM decoding from GEMV to GEMM via direct codebook dot products and conflict-free output buffering, claiming up to 11.17x speedup over prior lookup designs.