Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

This paper claims that evicting tokens from an LLM's KV cache is best done by learning to rank tokens by their predicted future attention, and that lightweight per-head reinforcement-learned rankers outperform heuristic baselines and transf

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:11 UTC pith:VJAH75QS

load-bearing objection Learned per-head RL eviction ranks are a real step forward, but the abstract's 128K claim and the unverified stability of the future-attention reward keep it from being a clean win. the 4 major comments →

arxiv 2602.10238 v2 pith:VJAH75QS submitted 2026-02-10 cs.CL cs.LG

Learning to Evict from Key-Value Cache

Luca Moschella , Laura Manduchi , Ozan Sener This is my paper
classification cs.CL cs.LG
keywords KV cache evictionreinforcement learningtoken rankingfuture attentionlong-context inferenceper-head policyzero-shot generalizationcache management
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models reuse previously processed tokens through a key–value (KV) cache, which grows with sequence length and can dominate memory during generation. This paper claims that deciding which cache entries to evict is best framed as a ranking problem: learn, for each attention head, a lightweight scoring function that orders tokens by how much attention they will receive from future tokens, so that dropping the lowest-ranked tokens under any memory budget preserves the most useful information. The authors prove that when the optimal retained sets for increasing budgets are nested, eviction and ranking are equivalent, and they train per-head stochastic ranking policies with a policy-gradient method on pre-computed generation traces, using only keys, values, and positions as inputs. Across long-context and dialogue benchmarks, the learned policies outperform recency-, norm-, and attention-based heuristics, and they generalize zero-shot to other tasks and to longer sequences with no modification of the underlying LLM. If correct, the work turns cache eviction from a hand-tuned heuristic into an offline-learned, budget-agnostic signal for memory-efficient inference.

Core claim

KV cache eviction can be exactly reformulated as learning a single budget-agnostic ranking of tokens. The paper shows that if the best b-token subset is unique and nested inside the best (b+1)-token subset for every b, then there is a total order whose top-b entries realize the optimal eviction for every budget (Proposition 1). The authors then learn such an order with a softmax-sequential stochastic ranking policy, parameterized by a small MLP per attention head that maps the token's key, value, and position to a score. The reward is the negative cumulative future attention of the evicted tokens, summed over all possible cache sizes and normalized by the optimal ranking's cost; this makes t

What carries the argument

The argument rests on four pieces: (1) Proposition 1, which reduces budget-constrained eviction to total-order ranking under uniqueness and nestedness of optimal subsets; (2) a per-head scoring MLP that takes only the cached key vector, value vector, and position as its input, so the policy is attention-free and query-independent; (3) a stochastic ranking policy that samples a full permutation by adding extreme-value noise to scores and sorting, enabling parallel sampling and policy-gradient training; and (4) a holistic reward equal to the negative cumulative future attention of evicted tokens, summed over all cache budgets and normalized against the optimal ranking, which removes the need f

Load-bearing premise

The load-bearing premise is that cumulative future attention in the uncompressed model remains a correct measure of a token's usefulness after other tokens have been evicted; if eviction changes future attention patterns, the learned ranking is optimizing a mis-specified objective.

What would settle it

Compare, on a held-out set, the policy's predicted token-utility scores against the actual drop in generation quality (e.g., downstream accuracy or next-token log-likelihood) when each candidate token is individually removed from a cache that has already been compressed to a given budget. If the two orderings diverge substantially, the reward—future attention in the uncompressed model—does not survive eviction.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A cache can be compressed to any target size by taking the top-b of a single learned ranking, so one policy serves every memory budget without re-training or re-scoring.
  • Because the ranking uses only keys, values, and positions, it adds no overhead during autoregressive decoding; all compression cost is confined to the prefill stage.
  • Per-head specialization matters: the learned policy stays effective across all layers and heads, while fixed heuristics are good for some heads and harmful for others.
  • The learned utility scores transfer zero-shot to unseen tasks and to sequence lengths well above the training distribution, suggesting the ranking captures a general property of the model's attention.
  • The same ranking can act as a principled signal for hierarchical memory management, with lowest-ranked tokens being natural candidates for offloading rather than hard eviction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper stops short of testing whether the future-attention reward remains valid after tokens are actually removed; a natural follow-up is to re-estimate utilities on the compressed cache and retrain or fine-tune with a reward measured under eviction.
  • If future attention is a stable individual utility, the same per-head rankings could be combined across heads to allocate non-uniform budgets adaptively, which the paper mentions but does not implement.
  • The offline training on traces means the framework transfers to new model families by recollecting traces and re-training small MLPs; the one-time data-generation forward pass is the dominant cost, so the method scales with the model rather than with the training loop.
  • The ranking could be composed with compression techniques: rank tokens first, then merge or quantize only the retained ones, yielding a two-stage memory reduction the paper notes but does not evaluate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes KVP, a framework that reframes KV cache eviction as a ranking problem. For each KV head, a lightweight MLP policy scores tokens using only keys, values, and positions; eviction keeps the top-b tokens. The policies are trained offline with REINFORCE/RLOO on pre-computed generation traces, using a reward that sums the (normalized) cumulative future attention of evicted tokens over all cache budgets, computed from the uncompressed model. Proposition 1 states that under uniqueness and nestedness, eviction is equivalent to ranking; the proof is given in Appendix A.1. Experiments on RULER-4k and OASST2-4k with Qwen2.5-7B-Chat and Phi-4-14B report accuracy/perplexity improvements over StreamingLLM, SnapKV, TOVA, KeyDiff, LagKV, K-Norm, and Random, plus zero-shot transfer to BoolQ, ARC, MMLU, HellaSwag, and GovReport. An ablation in Section 4.2 argues that RL is preferable to a differentiable sorting surrogate.

Significance. The paper is a serious attempt to replace heuristic scoring with learned, query-agnostic, per-head rankings for KV cache management. Its strengths include a clean ranking formulation with a correct and elementary proof, an offline training scheme that avoids additional LLM inference during policy learning, a budget-agnostic holistic reward, low inference overhead (about 1% prefill FLOPs, 0.71 ms per 10k-token compression), and extensive per-head diagnostics and wall-clock measurements. If the evaluation limitations below are addressed, the work would be a solid contribution to efficient LLM inference. The empirical results, while encouraging, are currently presented with overbroad claims and without statistical support.

major comments (4)
  1. [Abstract vs. Section 4] The abstract states that KVP is evaluated on RULER 'up to 128K tokens', but Section 4 specifies RULER-4k (approximately 4500-token sequences) as the only RULER configuration; there is no 128K experiment in the main text or appendix. The abstract's zero-shot list also includes 'LongBench passage retrieval', which is not among the downstream tasks evaluated in Section 4.1 (BoolQ, ARC, MMLU, HellaSwag, GovReport). Align the claims with the actual experiments.
  2. [Eq. (3.5), Section 3.1.1 and 4.2] The reward R_b sums future attention computed with the uncompressed model. At deployment the cache is evicted, so future attention is redistributed among the surviving keys; the paper never checks whether the static importance ranking remains valid for the actual post-eviction attention. The ablation in Section 4.2 (Fig. 5) evaluates −R_b against the same uncompressed future-attention target, which does not simulate the deployment condition. Add a stability test: re-compute future attention after masking evicted tokens and compare the ranking, or measure the correlation between the reward under eviction and under no eviction. Without this, the paper's central mechanistic claim is not directly supported.
  3. [Section 4, Budgeting and Compression Schedule] All evaluations compress the cache exactly once after prefill and then generate. This is a one-shot compression protocol, not a general streaming KV-cache eviction policy where decisions must be made repeatedly as new tokens are generated. This protocol favors a full ranking over the entire context and limits the scope of the 'eviction' claim. Please either evaluate periodic or streaming eviction (e.g., re-ranking or incremental eviction during decoding) or explicitly restrict the contribution to single-pass post-prefill compression and adjust the title/abstract accordingly.
  4. [Figures 2–5, Section 4.1] No error bars, confidence intervals, or number of seeds are reported for any downstream accuracy/perplexity curve. The abstract and Section 4.1 use 'significantly outperforms', but with single runs there is no statistical basis for this. Report means and variances over at least three seeds (or bootstraps) for the main comparisons, and state evaluation randomness (e.g., Gumbel noise) across runs.
minor comments (5)
  1. [Section 4.2 and Fig. 5 caption] Typos: 'surragates' should be 'surrogates'; 'Blondelet al.' should be 'Blondel et al.'.
  2. [Section 3.1.1, Eq. (3.5)] Define the attention notation A(x_i, x_j) explicitly (which term is the query and which is the key), and specify the GQA max rule. Also give the future-token window f used in the experiments; the paper leaves f unspecified.
  3. [Appendix A.2] Report the RLOO batch size K and state whether the always-retained first 4 / last 16 tokens are applied to all baselines. These implementation choices affect reproducibility and comparability.
  4. [Figures 3 and 4 legends] The abbreviations KVPS and KVPR are used in the legends but defined only in Section 4.1. Define them in the captions or in the main text before first use.
  5. [Section 4.1, Figure 13] The claim of 'consistently outperforming all baselines' is too strong; the per-subtask breakdown shows KVP behind SnapKV/TOVA on several RULER subtasks (e.g., CWE and QA2). Soften the claim or add a discussion of per-task variability.

Circularity Check

0 steps flagged

No significant circularity: future-attention reward is a training objective, and evaluation is external

full rationale

The paper's derivation chain is self-contained. The central object, the future-utility reward R(σ) in Eqs. (3.4)-(3.5), is explicitly defined as cumulative future attention of the uncompressed model on pre-computed traces. The RL agent (Eqs. 3.2, 3.3) is trained with REINFORCE/RLOO to maximize this reward, so the claim that KVP 'learns to predict future token utility' is a description of the training objective, not a result derived from it circularly. The actual evidence for the method is empirical: RULER, OASST2, and zero-shot BoolQ/ARC/GovReport results are measured under live compressed generation against external benchmarks not used in the reward. Proposition 1 is a trivial mathematical equivalence under explicit uniqueness/nestedness assumptions, and its appendix proof is direct. There are no self-citations, no fitted parameters renamed as predictions, and no imported uniqueness theorem. The only caveat, that the reward uses uncompressed-model attention and may not reflect post-eviction attention, is a validity concern about whether the proxy is correct, not a circularity in the derivation chain.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The central claim depends on the future-attention proxy, the always-retain heuristic, and the single-compression evaluation protocol.

free parameters (4)
  • Always-retained prefix/suffix size = first 4, last 16 tokens
    Hand-chosen in implementation (Appendix A.2); these tokens are never evicted, which affects all results.
  • Future-token window f for importance score
    The number of future tokens used to compute cumulative attention in Eq. 3.5 is not reported; the reward depends on it.
  • Training hyperparameters = 4000 steps, lr 5e-5, 256 hidden units, 2 layers
    Training choices are stated but no sensitivity analysis is given; performance may depend on them.
  • RLOO batch size K
    Not reported; this affects variance of the policy gradient and the final policy.
axioms (3)
  • domain assumption Future attention is a valid proxy for token utility
    The reward R_b (Eq. 3.5) assumes tokens that receive more future attention are more valuable to retain; if this is false, the learned rankings are suboptimal.
  • domain assumption Nestedness of optimal eviction sets
    Proposition 1 requires S_b ⊂ S_{b+1}; the paper calls it natural, but it is an assumption on the reward structure.
  • domain assumption Compression after prefill is the target use case
    All evaluations compress once after prefill; conclusions may not transfer to streaming eviction.

pith-pipeline@v1.3.0-alltime-deepseek · 48509 in / 9495 out tokens · 98560 ms · 2026-08-03T01:11:38.059715+00:00 · methodology

0 comments
read the original abstract

The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache. Existing eviction or compression methods reduce cost but rely on heuristics, such as recency or past attention scores, which serve only as indirect proxies for a token's future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement learning (RL) problem: learning to rank tokens by their predicted usefulness for future decoding. To this end, we introduce KV Policy (KVP), a framework of lightweight per-head RL agents trained on pre-computed generation traces using only key and value vectors. Each agent learns a specialized eviction policy guided by a holistic reward, derived from future utility, that evaluates the quality of the ranking across all cache budgets, requiring no modifications to the underlying LLM or additional inference. Evaluated across two model families on the long-context benchmark RULER (up to 128K tokens) and the multi-turn dialogue benchmark OASST2-4k, KVP significantly outperforms strong baselines. Zero-shot tests on standard downstream tasks (BoolQ, LongBench passage retrieval, GovReport) further show that KVP generalizes beyond its training distribution and to considerably longer sequence lengths. These results demonstrate that learning to predict future token utility is a powerful and scalable paradigm for adaptive KV cache management.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

    cs.CL 2026-07 conditional novelty 7.0

    Heavy-hitter KV cache eviction over-retains structural delimiters and keys on schema-dense inputs; a role-conditional reallocation of SnapKV's score recovers most of the accuracy collapse.

  2. AURA: Action-Gated Memory for Robot Policies at Constant VRAM

    cs.AI 2026-06 unverdicted novelty 7.0

    AURA-Mem uses an action-gated recurrent memory trained on closed-loop action error to deliver constant 4,224-byte state and 5-9x fewer writes than baselines while matching base policy success on LIBERO-Long.

  3. Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility

    cs.LG 2026-05 unverdicted novelty 6.0

    SP-KV trains a utility predictor jointly with the LLM to dynamically prune low-utility KV cache entries, achieving 3-10x memory reduction during generation with negligible performance loss.

Reference graph

Works this paper leans on

31 extracted references · 18 linked inside Pith · cited by 3 Pith papers

  1. [1]

    STA(tio) TUR(i)CEN(sis)

    In this particular failure case, the attention scores from layer 10, head 0, show that our proposed KVP strategy struggles to fully capture the ground-truth future attention pattern. 0 1000 2000 3000 4000 5000 Cache Size 4 6 8 10 12 14 16 18 20Dataset Perplexity RULER KVP KeyDiff TOV A Lagkv K-Norm StreamingLLM Random SnapKV Figure 10Perplexity (PPL) as a...

    2000

  2. [3]

    Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker

    doi: 10.1145/ 3723178.3723250. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. ArXiv preprint, abs/2402.14740,

    arXiv

  3. [4]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai

    URLhttps://arxiv.org/abs/2402.14740. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. InEMNLP, Singapore,

    Pith/arXiv arXiv

  4. [5]

    under review

    URLhttps://openreview.net/forum?id=yHxSKM9kdr. under review. Yushi Bai, Xin Lv, Jiajie Zhang, Hong Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding.ArXiv, abs/2308.14508,

    Pith/arXiv arXiv

  5. [7]

    Edoardo Cetin, Qi Sun, Tianyu Zhao, and Yujin Tang

    URLhttps://arxiv.org/abs/2406.02069. Edoardo Cetin, Qi Sun, Tianyu Zhao, and Yujin Tang. An evolved universal transformer memory.ArXiv preprint, abs/2410.13166,

    Pith/arXiv arXiv

  6. [8]

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S

    URLhttps://arxiv.org/abs/2410.13166. Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: Kv-cache compression with low-rank projection. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    Pith/arXiv arXiv 2025

  7. [9]

    Vivek Chari, Guanghui Qin, and Benjamin Van Durme

    URLhttps://arxiv.org/abs/2507.08143. Vivek Chari, Guanghui Qin, and Benjamin Van Durme. Kv-distill: Nearly lossless learnable context compression for llms.ArXiv preprint, abs/2503.10337,

    arXiv

  8. [10]

    Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang

    URLhttps://arxiv.org/abs/2503.10337. Renze Chen, Zhuofeng Wang, Beiquan Cao, Tong Wu, Size Zheng, Xiuhong Li, Xuechao Wei, Shengen Yan, Meng Li, and Yun Liang. Arkvale: Efficient generative LLM inference with recallable key-value eviction. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URLhttps://openreview.net/foru...

    Pith/arXiv arXiv

  9. [14]

    Alessio Devoto, Maximilian Jeblick, and Simon Jégou

    URLhttps://arxiv.org/abs/2406.11430. Alessio Devoto, Maximilian Jeblick, and Simon Jégou. Expected attention: Kv cache compression by estimating attention from future queries distribution,

    Pith/arXiv arXiv

  10. [15]

    URLhttps://arxiv.org/abs/2510.00636. LeoGao, JonathanTow, BaberAbbasi, StellaBiderman, SidBlack, AnthonyDiPofi, CharlesFoster, LaurenceGolding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wa...

    arXiv

  11. [16]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao

    URLhttps://zenodo.org/records/12608602. Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for llms. InICLR,

    arXiv

  12. [18]

    Simon Jegou and Maximilian Jeblick

    URLhttps://arxiv.org/abs/2404.06654. Simon Jegou and Maximilian Jeblick. Kvzap: Fast, adaptive, and faithful kv cache pruning,

    Pith/arXiv arXiv

  13. [21]

    Lagkv: Lag-relative information of the kv cache tells which tokens are important.ArXiv, abs/2504.04704,

    Manlai Liang, JiaMing Zhang, Xiong Li, and Jinlong Li. Lagkv: Lag-relative information of the kv cache tells which tokens are important.ArXiv, abs/2504.04704,

    Pith/arXiv arXiv

  14. [23]

    Junyoung Park, Dalton Jones, Matthew J Morse, Raghavv Goel, Mingu Lee, and Chris Lott

    URLhttps://arxiv.org/abs/2401.06104. Junyoung Park, Dalton Jones, Matthew J Morse, Raghavv Goel, Mingu Lee, and Chris Lott. Keydiff: Key similarity- based kv cache eviction for long-context llm inference in resource-constrained environments.ArXiv preprint, abs/2504.15364,

    Pith/arXiv arXiv

  15. [24]

    URLhttps://arxiv.org/abs/2504.15364. R. L. Plackett. The analysis of permutations.Journal of the Royal Statistical Society., 24(2):193–202,

    arXiv

  16. [25]

    Minikv: Pushing the limits of 2-bit KV cache via compression and system co-design for efficient long context inference

    Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang. Minikv: Pushing the limits of 2-bit KV cache via compression and system co-design for efficient long context inference. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Findings of the Association for Computational Lin- guistics, ACL 2025, Vienn...

    2025

  17. [26]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang

    URLhttps://aclanthology.org/2025.findings-acl.952/. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single GPU. InICML,

    2025

  18. [28]

    Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele

    URLhttps://arxiv.org/ abs/2006.16864. Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention. InNeurIPS,

    Pith/arXiv arXiv 2006

  19. [30]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    URLhttps://arxiv.org/abs/2307.09288. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNIPS,

    Pith/arXiv arXiv

  20. [32]

    13 Yantong Wang and Vasilis Friderikos

    URLhttps://arxiv.org/abs/2409.12191. 13 Yantong Wang and Vasilis Friderikos. A survey of deep learning for data caching in edge network.ArXiv preprint, abs/2008.07235,

    Pith/arXiv arXiv 2008

  21. [33]

    Ronald J

    URLhttps://arxiv.org/abs/2008.07235. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine Learning, 8(3–4):229–256,

    Pith/arXiv arXiv 2008

  22. [35]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi

    URLhttps://arxiv.org/abs/2407.10671. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InACL,

    Pith/arXiv arXiv

  23. [36]

    Clus- terattn: KV cache compression under intrinsic attention clustering

    Minwei Zhang, Haifeng Sun, Jingyu Wang, Shaolong Li, Wanyi Ning, Qi Qi, Zirui Zhuang, and Jianxin Liao. Clus- terattn: KV cache compression under intrinsic attention clustering. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vo...

    2025

  24. [37]

    We use AdamW with a learning rate of5×10−5, following a cosine schedule with 100-step linear warmup (start factor 0.01) that decays to1×10−6

    and normalize advantages by their mean and standard deviation, without entropy regularization. We use AdamW with a learning rate of5×10−5, following a cosine schedule with 100-step linear warmup (start factor 0.01) that decays to1×10−6. Each agent trains for 4,000 steps on pre-computed activations. A.3 Training Efficiency and Inference Overhead The traini...

    2000

  25. [39]

    20 0 1000 2000 3000 4000 5000 Cache Size 50 60 70 80 90 100RULER NIAH MULTIKEY 2 Score RULER - NIAH MULTIKEY 2 KVP TOV A SnapKV Lagkv KeyDiff StreamingLLM Random K-Norm 0 1000 2000 3000 4000 5000 Cache Size 50 60 70 80 90 100RULER NIAH MULTIKEY 3 Score RULER - NIAH MULTIKEY 3 StreamingLLM SnapKV TOV A KVP Lagkv Random KeyDiff K-Norm 0 1000 2000 3000 4000 ...

    2000

  26. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv:1803.05457v1,

    Pith/arXiv arXiv

  27. [2022]

    Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini

    URLhttps://arxiv.org/abs/2208.07339. Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effectivel_2norm-based strategy for kv cache compression.ArXiv preprint, abs/2406.11430,

    Pith/arXiv arXiv

  28. [2023]

    Infinigen: Efficient generative inference of large language models with dynamic kv cache management.ArXiv preprint, abs/2406.19707, 2024a

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. Infinigen: Efficient generative inference of large language models with dynamic kv cache management.ArXiv preprint, abs/2406.19707, 2024a. URLhttps://arxiv.org/abs/2406. 19707. 12 Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. Infinigen: efficient generative inference of large language model...

    Pith/arXiv arXiv

  29. [2024]

    Machine learning for predictive database caching strategies: A state-of-the-art review

    Sadia Afrin, Md Abdullah All Mahmud, Abdus Salam, Md Abdullah Al Jubair, and Muhammad Firoz Mridha. Machine learning for predictive database caching strategies: A state-of-the-art review. InProceedings of the 3rd International Conference on Computing Advancements, ICCA 2024, pages 540–547. ACM,

    2024

  30. [2025]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J

    URL https://arxiv.org/abs/2504.21318. Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, and Purushotham Kamath. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. In P. Gibbons, G. Pekhimenko, and C. De Sa, editors,Proceedings of Machine Learning and Systems, volume 6, pages 114–127,

    Pith/arXiv arXiv

  31. [2026]

    Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W Lee, Sangdoo Yun, and Hyun Oh Song

    URLhttps: //arxiv.org/abs/2601.07891. Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W Lee, Sangdoo Yun, and Hyun Oh Song. Kvzip: Query-agnostic kv cache compression with context reconstruction.Advances in Neural Information Processing Systems,

    arXiv