Pith. sign in

REVIEW 15 cited by

ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.14256 v1 pith:5KADX5CL submitted 2024-05-23 cs.LG cs.AI

Yefei He , Luoming Zhang , Weijia Wu , Jing Liu , Hong Zhou , Bohan Zhuang This is my paper · ORCID
classification cs.LGcs.AI
keywords cachecompressionquantizationzipcacheattentiontokensefficientprevious
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

KV cache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. Adaptive KV cache compression seeks to discern the saliency of tokens, preserving vital information while aggressively compressing those of less importance. However, previous methods of this approach exhibit significant performance degradation at high compression ratios due to inaccuracies in identifying salient tokens. In this paper, we present ZipCache, an accurate and efficient KV cache quantization method for LLMs. First, we construct a strong baseline for quantizing KV cache. Through the proposed channel-separable tokenwise quantization scheme, the memory overhead of quantization parameters are substantially reduced compared to fine-grained groupwise quantization. To enhance the compression ratio, we propose normalized attention score as an effective metric for identifying salient tokens by considering the lower triangle characteristics of the attention matrix. Moreover, we develop an efficient approximation method that decouples the saliency metric from full attention scores, enabling compatibility with fast attention implementations like FlashAttention. Extensive experiments demonstrate that ZipCache achieves superior compression ratios, fast generation speed and minimal performance losses compared with previous KV cache compression methods. For instance, when evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of compressing the KV cache by $4.98\times$, with only a $0.38\%$ drop in accuracy. In terms of efficiency, ZipCache also showcases a $37.3\%$ reduction in prefill-phase latency, a $56.9\%$ reduction in decoding-phase latency, and a $19.8\%$ reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of $4096$.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    cs.LG 2026-07 accept novelty 7.0 of 10

    Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.

  2. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  3. NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

    cs.LG 2025-05 conditional novelty 6.0 of 10

    NSNQuant applies a Normalize-Shift-Normalize transform plus a Hadamard rotation to make KV cache channels match a standard normal distribution, so one codebook trained on random noise can quantize them without calibration.

  4. Accurate KV Cache Quantization with Outlier Tokens Tracing

    cs.CL 2025-05 conditional novelty 6.0 of 10

    OTT improves 2-bit KV cache quantization by identifying low-magnitude key outlier tokens and keeping them full-precision, outperforming KIVI across several LLMs and benchmarks.

  5. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Trained linear predictors across transformer layers plus residual quantization compress LLM KV caches to about 2 bits per value with near-lossless accuracy on tested Llama and Qwen models.

  6. AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    AKVQ-VL quantizes VLM KV caches to mostly 2 bits with attention-aware token protection and Walsh-Hadamard outlier removal, staying near FP16 accuracy on MileBench.

  7. RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations

    cs.LG 2025-01 conditional novelty 6.0 of 10

    RotateKV combines outlier-aware, pre-RoPE grouped-head Hadamard rotation with attention-sink-aware retention to make 2-bit KV cache quantization accurate on LLaMA-2, LLaMA-3, and Mistral models.

  8. ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals

    cs.LG 2024-12 conditional novelty 6.0 of 10

    ResQ quantizes LLMs to 4-bit with a low-rank 8-bit residual subspace chosen by PCA, beating prior 4-bit methods on perplexity and accuracy.

  9. Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Lexico compresses LLM key-value caches by replacing each cached vector with a sparse combination of about 4,000 shared dictionary atoms, keeping 90-95% of accuracy at 15-25% of the cache memory.

  10. SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers

    cs.CL 2025-07 conditional novelty 5.0 of 10

    SpindleKV compresses LLM KV cache by evicting low-attention tokens in deep layers and replacing near-duplicate key and value vectors in shallow layers with a shared codebook, while preserving benchmark accuracy at 15 ...

  11. Curse of High Dimensionality Issue in Transformer for Long-context Modeling

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A group attention mechanism that keeps important tokens and merges the rest cuts long-context decoding cost, but the theoretical case for grouping is only weakly supported.

  12. Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression

    cs.LG 2025-05 conditional novelty 5.0 of 10

    ScaleKV cuts KV cache memory for Visual Autoregressive text-to-image generation to 10% by classifying layers as drafters or refiners per scale and pruning low-attention tokens while keeping benchmark scores nearly unchanged.

  13. HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference

    cs.DC 2025-02 conditional novelty 5.0 of 10

    HACK performs attention matrix multiplications directly on 2-bit quantized key-value cache entries, eliminating per-iteration dequantization and cutting end-to-end latency in disaggregated LLM serving.

  14. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

  15. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Pith tools