Pith. sign in

REVIEW 5 cited by

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.03917 v1 pith:MFVIJU56 submitted 2024-05-07 cs.LG

Tianyi Zhang , Jonah Yi , Zhaozhuo Xu , Anshumali Shrivastava This is my paper · ORCID
classification cs.LG
keywords cachemodelchannelsmultiplequantizationsizevaluecoupled
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Efficient deployment of Large Language Models (LLMs) requires batching multiple requests together to improve throughput. As the batch size, context length, or model size increases, the size of the key and value (KV) cache can quickly become the main contributor to GPU memory usage and the bottleneck of inference latency. Quantization has emerged as an effective technique for KV cache compression, but existing methods still fail at very low bit widths. We observe that distinct channels of a key/value activation embedding are highly inter-dependent, and the joint entropy of multiple channels grows at a slower rate than the sum of their marginal entropies. Based on this insight, we propose Coupled Quantization (CQ), which couples multiple key/value channels together to exploit their inter-dependency and encode the activations in a more information-efficient manner. Extensive experiments reveal that CQ outperforms or is competitive with existing baselines in preserving model quality. Furthermore, we demonstrate that CQ can preserve model quality with KV cache quantized down to 1-bit.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CommVQ: Commutative Vector Quantization for KV Cache Compression

    cs.CL 2025-06 conditional novelty 7.0 of 10

    CommVQ compresses LLM key-value caches to roughly one bit per value using additive quantization with a codebook designed to commute with rotary position embeddings, preserving most benchmark accuracy.

  2. PolarQuant: Quantizing KV Caches with Polar Transformation

    cs.LG 2025-02 conditional novelty 7.0 of 10

    PolarQuant achieves around 4x KV cache compression for LLMs by quantizing angles after a recursive polar transform with random preconditioning, with LongBench scores close to the full-precision model and above prior c...

  3. What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.

  4. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  5. More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Storing 4x as many KV cache tokens at 4-bit precision outperforms storing 1x tokens at 16-bit precision on long-context benchmarks at equal data-memory budgets.

Pith tools