Pith. sign in

REVIEW 13 cited by

KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.18517 v1 pith:SYPGCTTZ submitted 2024-10-24 cs.LG cs.AIcs.CL

Yifei Yang , Zouying Cao , Qiguang Chen , Libo Qin , Dongjie Yang , Hai Zhao , Zhi Chen This is my paper
classification cs.LGcs.AIcs.CL
keywords cachecompressionkvsharermemorylayer-wisemodelsharingtextit
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The development of large language models (LLMs) has significantly expanded model sizes, resulting in substantial GPU memory requirements during inference. The key and value storage of the attention map in the KV (key-value) cache accounts for more than 80\% of this memory consumption. Nowadays, most existing KV cache compression methods focus on intra-layer compression within a single Transformer layer but few works consider layer-wise compression. In this paper, we propose a plug-and-play method called \textit{KVSharer}, which shares the KV cache between layers to achieve layer-wise compression. Rather than intuitively sharing based on higher similarity, we discover a counterintuitive phenomenon: sharing dissimilar KV caches better preserves the model performance. Experiments show that \textit{KVSharer} can reduce KV cache computation by 30\%, thereby lowering memory consumption without significantly impacting model performance and it can also achieve at least 1.3 times generation acceleration. Additionally, we verify that \textit{KVSharer} is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. KV Cache Offloading for Context-Intensive Tasks

    cs.LG 2026-04 conditional novelty 7.0 of 10

    KV offloading degrades accuracy on context-intensive tasks due to low-rank key projections and unreliable landmarks; a simpler alternative improves results across models and benchmarks.

  2. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  3. DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

    cs.AI 2026-07 conditional novelty 6.0 of 10

    DepthWeave-KV achieves 8.3x KV cache memory reduction with near-full-cache task quality by factorizing key-value states across transformer layers using shared bases and token-adaptive residuals.

  4. GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    GHOST is a geometry-hierarchical token eviction framework that halves the KV cache size in monocular video 3D reconstruction while maintaining quality and achieving 1.75x faster inference.

  5. GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    GHOST applies geometry-hierarchical online token eviction with hierarchical scoring, privilege protection, and layer-wise budget allocation to halve KV cache size while maintaining reconstruction quality and achieving...

  6. Search Your Block Floating Point Scales!

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    ScaleSearch optimizes block floating point scales via fine-grained search to cut quantization error by 27% for NVFP4, improving PTQ by up to 15 points on MATH500 for Qwen3-8B and attention PPL by 0.77 on Llama 3.1 70B.

  7. KV Cache Offloading for Context-Intensive Tasks

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    KV offloading hurts accuracy on context-heavy tasks due to low-rank key projections and bad landmarks, but a simpler strategy recovers performance across models.

  8. KV Cache Offloading for Context-Intensive Tasks

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    KV offloading hurts accuracy on context-heavy tasks because of low-rank key projections and bad landmarks, but a simpler strategy improves results across models and benchmarks.

  9. KV Cache Offloading for Context-Intensive Tasks

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    KV offloading degrades performance on context-intensive tasks due to low-rank key projections and unreliable landmarks, but a simpler alternative strategy restores accuracy across LLM families.

  10. FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Frequency-guided inter-layer KV sharing with logit-aware head routing nearly matches full-cache long-context accuracy at about 3.9× lower peak KV memory.

  11. Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference

    cs.DC 2026-06 unverdicted novelty 5.0 of 10

    Omni-Flow introduces a three-layer abstraction (Control Flow, Data Flow, Compute Flow) for unified orchestration and KV cache sharing in multimodal inference pipelines.

  12. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

  13. DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A dynamic attention-aware metric for task-agnostic prompt compression that combines entropy and attention scores, with iterative entropy recalculation, outperforms prior entropy-based compressors on multiple benchmarks.

Pith tools