Pith. sign in

REVIEW 29 cited by

Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.08454 v2 pith:4IHXDMYO submitted 2024-07-11 cs.CL

Zheng Wang , Boxiao Jin , Zhongzhi Yu , Minjia Zhang This is my paper · ORCID
classification cs.CL
keywords cachemerginglong-contextllmstasksalgorithmbudgetscomputational
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

How to efficiently serve Large Language Models (LLMs) has become a pressing issue because of their huge computational cost in their autoregressive generation process. To mitigate computational costs, LLMs often employ the KV Cache technique to improve the generation speed. While improving the computational efficiency, the storage requirements of the KV cache are substantial, particularly in long-context scenarios, leading to significant memory consumption. Existing KV cache eviction methods often degrade the performance of LLMs in long-context scenarios due to the information loss introduced by eviction. In this paper, we propose a novel KV cache merging approach, called KVMerger, to achieve adaptive KV cache compression for long-context tasks without significant performance degradation under constrained memory budgets. Our approach is inspired by the intriguing observation that key states exhibit high similarity at the token level within a single sequence. To facilitate merging, we develop an effective yet straightforward merging set identification algorithm to identify suitable KV states for merging. Our merging set identification algorithm stimulates the second observation that KV cache sparsity, from similarity perspective, is independent of the dataset and remains persistent at the model level. Subsequently, we propose a Gaussian kernel weighted merging algorithm to selectively merge all states within each merging set. We conduct extensive experiments to demonstrate the effectiveness of KVMerger for long-context tasks under constrained memory budgets, applying it to models including Llama2-7B-chat and Llama2-13B-chat. Using the LongBench and ZeroScroll benchmarks, we compare our method with other KV cache compression techniques, including H2O and CaM, showing that our method achieves superior performance across tasks with both 50% and 35% KV cache budgets.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 29 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

    cs.CV 2026-08 conditional novelty 7.0 of 10

    PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.

  2. Fast KV Compaction via Attention Matching

    cs.LG 2026-02 conditional novelty 7.0 of 10

    Attention Matching compacts LLM key–value caches by matching per-head attention outputs and mass over reference queries, reaching 50× compression in seconds-to-minutes with modest task loss.

  3. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  4. Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

    cs.AI 2026-08 conditional novelty 6.0 of 10

    ReCo coordinates KV-cache compression, reflection-token logit penalties, and confidence-based early stopping under one per-step process reward, reducing tokens and latency while largely preserving accuracy.

  5. AnchorKV: Anchor-Residual KV Cache Compression

    cs.LG 2026-08 conditional novelty 6.0 of 10

    AnchorKV compresses the KV cache 20x with near-lossless accuracy by representing each token as an anchor projection plus selectively allocated quantized residuals, beating eviction baselines at matched byte budgets on...

  6. ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Splitting a fixed KV-cache budget into exact main entries and cluster-summary residual entries that share the softmax improves LongBench and RULER scores over eviction baselines.

  7. Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs

    eess.AS 2026-07 conditional novelty 6.0 of 10

    Learned pooling of speech KV caches from an intermediate LLM layer compresses speech to text-level length while matching or exceeding the uncompressed baseline on ASR and entity recognition, with 1.49–2× decoding speedup.

  8. DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

    cs.AI 2026-07 conditional novelty 6.0 of 10

    DepthWeave-KV achieves 8.3x KV cache memory reduction with near-full-cache task quality by factorizing key-value states across transformer layers using shared bases and token-adaptive residuals.

  9. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  10. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  11. IAM: Efficient Inference through Attention Mapping between Different-scale LLMs

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Attention matrices of same-series small and large LLMs are similar enough that replacing up to 50% of a large model's attention layers with the small model's matrices preserves most performance while reducing KV cache...

  12. LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation framework

    cs.CL 2025-07 conditional novelty 6.0 of 10

    LOOM-Scope is a framework that standardizes long-context LLM evaluation across 22 benchmarks and integrates a lightweight 12-benchmark suite, LOOMBench, for fast comprehensive assessment.

  13. HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference

    cs.LG 2025-07 conditional novelty 6.0 of 10

    HGCA splits attention between GPU (dense, recent KV) and CPU (sparse, salient KV) and merges partial results with exact log-sum-exp fusion, scaling long-context decoding on commodity GPUs.

  14. RCStat: A Statistical Framework for using Relative Contextualization in Transformers

    cs.CL 2025-06 conditional novelty 6.0 of 10

    RCStat uses pre-softmax attention logits to define a Relative Contextualization score that improves adaptive KV-cache eviction and attention-head selection for attribution on LLaMA models.

  15. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  16. HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing

    cs.LG 2024-12 conditional novelty 6.0 of 10

    HashEvict uses locality-sensitive hashing to evict the KV-cache token least similar to the current query, compressing 30-70% with modest quality loss across several LLM benchmarks.

  17. FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Frequency-guided inter-layer KV sharing with logit-aware head routing nearly matches full-cache long-context accuracy at about 3.9× lower peak KV memory.

  18. SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation

    cs.AI 2026-05 conditional novelty 5.0 of 10

    SelKV compresses the KV cache to 25% with per-token cosine-gated merging and an attention-ratio logit bias, staying within ~0.7 points of full-cache LongBench accuracy on GQA models.

  19. GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction

    cs.CL 2025-08 conditional novelty 5.0 of 10

    GraphKV adds a similarity-based decay penalty to existing top-k KV cache eviction scores, improving retention diversity and accuracy on LongBench and needle-in-a-haystack tasks.

  20. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

  21. LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models

    cs.LG 2025-07 conditional novelty 5.0 of 10

    LaCache keeps a layer-dependent diagonal slice of the KV cache and iteratively compacts old entries, improving long-context perplexity and retrieval accuracy versus StreamingLLM at fixed cache sizes.

  22. Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Krul dynamically selects per-conversation cross-layer KV cache compression from attention similarity patterns, cutting TTFT by 1.28x to 2.68x and KV storage by 1.33x to 2.35x with less than 1% average accuracy loss on...

  23. SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers

    cs.CL 2025-07 conditional novelty 5.0 of 10

    SpindleKV compresses LLM KV cache by evicting low-attention tokens in deep layers and replacing near-duplicate key and value vectors in shallow layers with a shared codebook, while preserving benchmark accuracy at 15 ...

  24. Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding

    cs.LG 2025-05 reject novelty 5.0 of 10

    LFPS predicts which KV cache positions a decoder will attend to by reusing historical vertical and slash patterns plus local expansion, then runs exact Top-k only on the predicted candidate set.

  25. Curse of High Dimensionality Issue in Transformer for Long-context Modeling

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A group attention mechanism that keeps important tokens and merges the rest cuts long-context decoding cost, but the theoretical case for grouping is only weakly supported.

  26. SpecMemo: Speculative Decoding is in Your Pocket

    cs.LG 2025-05 conditional novelty 5.0 of 10

    SpecMemo prunes speculative-decoding token trees and pre-allocates memory to run multi-turn LLM chatbots on GPUs with as little as 8GB, retaining about 96% of throughput while cutting buffer memory by 65%.

  27. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference

    cs.LG 2025-09 conditional novelty 4.0 of 10

    PagedEviction prunes the KV cache in whole blocks using a key-value norm ratio, speeding up long-context LLM inference in vLLM while staying close to full-cache accuracy.

  28. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

  29. XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference

    cs.LG 2024-12 conditional novelty 4.0 of 10

    XKV gives each network layer a personalized KV cache budget based on attention retention, cutting memory 61.6% with comparable accuracy.

Pith tools