Pith. sign in

REVIEW 4 cited by

In-context KV-Cache Eviction for LLMs via Attention-Gate

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.12876 v3 pith:43IXNE4Q submitted 2024-10-15 cs.CL cs.LG

Zihao Zeng , Bokai Lin , Tianqi Hou , Hao Zhang , Zhijie Deng This is my paper · ORCID
classification cs.CLcs.LG
keywords evictionkv-cacheflagsattention-gateattention-gatesbecomeinferencellms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The KV-Cache technique has become the standard for the inference of large language models (LLMs). Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM inference system. This paper enables a novel dynamic KV-Cache eviction policy by injecting a lightweight module called Attention-Gate to the model. It accepts the global context as input and yields eviction flags for each token. The self-attention modules in the model proceed according to the flags and cache only a subset of the KV states for next token prediction. The Attention-Gates can yield various flags for different heads and layers and be easily tuned on top of a pre-trained LLM via continual pre-training or supervised fine-tuning. The computational and memory overhead introduced by Attention-Gates can be minimal. We empirically evaluate the proposed approach across multiple scenarios, showing that effective eviction of redundant tokens can not only improve efficiency but also enhance performance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

    cs.CV 2026-08 conditional novelty 7.0 of 10

    PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.

  2. KVpop -- Key-Value Cache Compression with Predictive Online Pruning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    KVpop supervises fixed-budget KV eviction with an efficiently computed future-attention target and delayed mLSTM scoring, retaining ~97–100% of full-attention math performance at 75–88% compression.

  3. SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A learned gate selects the important KV blocks during long decoding, preserving math reasoning accuracy while skipping up to 90% of attention work.

  4. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Pith tools