REVIEW 10 cited by
Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters
read the original abstract
Scaling the context size of large language models (LLMs) enables them to perform various new tasks, e.g., book summarization. However, the memory cost of the Key and Value (KV) cache in attention significantly limits the practical applications of LLMs. Recent works have explored token pruning for KV cache reduction in LLMs, relying solely on attention scores as a token importance indicator. However, our investigation into value vector norms revealed a notably non-uniform pattern questioning their reliance only on attention scores. Inspired by this, we propose a new method: Value-Aware Token Pruning (VATP) which uses both attention scores and the $ \ell_{1} $ norm of value vectors to evaluate token importance. Extensive experiments on LLaMA2-7B-chat and Vicuna-v1.5-7B across 16 LongBench tasks demonstrate that VATP outperforms attention-score-only baselines in over 12 tasks, confirming the effectiveness of incorporating value vector norms into token importance evaluation of LLMs.
Forward citations
Cited by 10 Pith papers
-
LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding
Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.
-
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
DASH selectively halts stabilized tokens by monitoring layer-wise self-attention deltas, delivering prefill speedups on language and vision tasks without accuracy loss or hardware incompatibility.
-
Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression
KV cache compression causes task-dependent degradation in high-density reasoning due to disrupted CoT links; ShotKV mitigates this by preserving few-shot examples as indivisible semantic units through phase separation...
-
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
Contribution Weights combine attention, value magnitude, and directional alignment to measure token influence more faithfully than attention alone, and show attention sinks actively suppress information via a convex s...
-
ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
ReST-KV formulates KV eviction as layer-wise output reconstruction optimization with spatial-temporal smoothing, outperforming baselines by 2.58% on LongBench and 15.2% on RULER while cutting decoding latency by 10.61...
-
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
DASH reduces prefill computation in LLMs and vision-language models by halting tokens with low attention-branch residual updates at a chosen layer, achieving 1.7–2× prefill speedups with minimal accuracy loss.
-
ART: Attention Run-time Termination for Efficient Large Language Model Decoding
ART early-terminates FlashAttention-style KV block traversal when intermediate attention outputs stabilize in scale and direction, speeding decoding up to ~20% atop existing KV methods.
-
CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation
Offline-learned head-reliability and risk-threshold tables make prefill-only KV compression recover about 97.7% of uncompressed LongBench accuracy at a 512-token-per-layer memory budget.
-
CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation
CompilerKV uses offline-compiled retention tables as portable priors to achieve SOTA prefill-only KV compression performance across backbones at low token budgets.
-
When Attention Sink Emerges in Language Models: An Empirical View
Attention sinks emerge in language models from softmax-induced token dependence on attention scores and do not appear when using sigmoid attention without normalization in models up to 1B parameters.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.