MomentKV maintains count, key mean, value mean, and value-key covariance over evicted tokens to guide selective eviction and provide a first-order approximation of their attention contribution, outperforming baselines on LongBench and RULER.
Towards understanding how attention mechanism works in deep learning
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
fields
cs.LG 2representative citing papers
citing papers explorer
-
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
MomentKV maintains count, key mean, value mean, and value-key covariance over evicted tokens to guide selective eviction and provide a first-order approximation of their attention contribution, outperforming baselines on LongBench and RULER.
- Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation