Pith. sign in

REVIEW 15 cited by

A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.11430 v4 pith:FSAV3SZG submitted 2024-06-17 cs.CL cs.AI

Alessio Devoto , Yu Zhao , Simone Scardapane , Pasquale Minervini This is my paper · ORCID
classification cs.CLcs.AI
keywords attentioncachereducescoresstrategycompressionembeddinglanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

The deployment of large language models (LLMs) is often hindered by the extensive memory requirements of the Key-Value (KV) cache, especially as context lengths increase. Existing approaches to reduce the KV cache size involve either fine-tuning the model to learn a compression strategy or leveraging attention scores to reduce the sequence length. We analyse the attention distributions in decoder-only Transformers-based models and observe that attention allocation patterns stay consistent across most layers. Surprisingly, we find a clear correlation between the $L_2$ and the attention scores over cached KV pairs, where a low $L_2$ of a key embedding usually leads to a high attention score during decoding. This finding indicates that the influence of a KV pair is potentially determined by the key embedding itself before being queried. Based on this observation, we compress the KV cache based on the $L_2$ of key embeddings. Our experimental results show that this simple strategy can reduce the KV cache size by 50% on language modelling and needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing accuracy. Moreover, without relying on the attention scores, this approach remains compatible with FlashAttention, enabling broader applicability.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

    cs.AR 2026-07 conditional novelty 7.0 of 10

    A per-request sound upper bound on KV-cache quantization error, implemented as a runtime meter that gates and repairs compression to restore quality.

  2. Learning to Evict from Key-Value Cache

    cs.CL 2026-02 conditional novelty 7.0 of 10

    Per-head RL agents that rank KV-cache tokens by predicted future utility outperform heuristic eviction baselines and generalize zero-shot to new tasks.

  3. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  4. KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding

    cs.CL 2025-07 conditional novelty 6.0 of 10

    By downsampling key and value head dimensions and retraining with distillation, KV-Latent cuts KV cache memory by about 50% on 7-8B LLMs while keeping average benchmark scores within about 1 point of the base model.

  5. RCStat: A Statistical Framework for using Relative Contextualization in Transformers

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RCStat uses pre-softmax attention logits to define a Relative Contextualization score that improves adaptive KV-cache eviction and attention-head selection for attribution on LLaMA models.

  6. SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SALE is a training-free sparse attention method that uses 4-bit quantized query-key estimates and a relative attention score to skip unimportant blocks, achieving over 3.36x prefill speedup on 64K+ token contexts with...

  7. HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing

    cs.LG 2024-12 conditional novelty 6.0 of 10

    HashEvict uses locality-sensitive hashing to evict the KV-cache token least similar to the current query, compressing 30-70% with modest quality loss across several LLM benchmarks.

  8. Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Lexico compresses LLM key-value caches by replacing each cached vector with a sparse combination of about 4,000 shared dictionary atoms, keeping 90-95% of accuracy at 15-25% of the cache memory.

  9. MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

    cs.AI 2026-06 conditional novelty 5.0 of 10

    By sampling variance-inflated query vectors during prefilling, MM-ShiftKV selects prompt KV caches that better match decoding-time attention and outperforms prior prefill-only KV compression on multimodal benchmarks a...

  10. Dynamic Thinking-Token Selection for Efficient Reasoning in Large Reasoning Models

    cs.AI 2026-01 conditional novelty 5.0 of 10

    A trainable 'importance predictor' learns which thinking tokens the final answer attends to, and DynTS evicts the rest from the KV cache, cutting memory and latency with roughly unchanged reasoning accuracy.

  11. GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction

    cs.CL 2025-08 conditional novelty 5.0 of 10

    GraphKV adds a similarity-based decay penalty to existing top-k KV cache eviction scores, improving retention diversity and accuracy on LongBench and needle-in-a-haystack tasks.

  12. StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A training-free, query-agnostic KV cache compression method for streaming video MLLMs, using chat-template attention as a saliency proxy, matches or beats prior streaming methods at a fixed 6K memory budget.

  13. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

  14. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

  15. More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Storing 4x as many KV cache tokens at 4-bit precision outperforms storing 1x tokens at 16-bit precision on long-context benchmarks at equal data-memory budgets.

Pith tools