REVIEW 7 cited by
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models(LLMs) have sparked a new wave of exciting AI applications. Hosting these models at scale requires significant memory resources. One crucial memory bottleneck for the deployment stems from the context window. It is commonly recognized that model weights are memory hungry; however, the size of key-value embedding stored during the generation process (KV cache) can easily surpass the model size. The enormous size of the KV cache puts constraints on the inference batch size, which is crucial for high throughput inference workload. Inspired by an interesting observation of the attention scores, we hypothesize the persistence of importance: only pivotal tokens, which had a substantial influence at one step, will significantly influence future generations. Based on our empirical verification and theoretical analysis around this hypothesis, we propose Scissorhands, a system that maintains the memory usage of the KV cache at a fixed budget without finetuning the model. In essence, Scissorhands manages the KV cache by storing the pivotal tokens with a higher probability. We validate that Scissorhands reduces the inference memory usage of the KV cache by up to 5X without compromising model quality. We further demonstrate that Scissorhands can be combined with 4-bit quantization, traditionally used to compress model weights, to achieve up to 20X compression.
Forward citations
Cited by 7 Pith papers
-
ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
Splitting a fixed KV-cache budget into exact main entries and cluster-summary residual entries that share the softmax improves LongBench and RULER scores over eviction baselines.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
EvolKV: Evolutionary KV Cache Compression for LLM Inference
CMA-ES search over per-layer KV cache budgets beats uniform and pyramidal compression heuristics on LongBench, NIAH, RULER, and GSM8K, and edges past the full cache on one code dataset at 1.5% of the budget.
-
Power Law Guided Dynamic Sifting for Efficient Attention
SiftAttention skips top-k sorting in sparse attention by thresholding attention weights with a threshold predicted from a power-law fit of score quantiles over generation steps.
-
FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models
FrameFusion merges similar adjacent-frame visual tokens before importance-based pruning, reducing video LVLM tokens by 70% with under 3% average performance loss.
-
CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation
CSR compresses LLM key-value caches through sparse dictionary coding, reaching 1 bit per cache value while roughly matching 2-bit and 4-bit quantization baselines on LongBench.
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.
Discussion (0). Continue with ORCID to comment.