ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

Lisi Chen; Shuo Shang; Yuhang Zhan

REVIEW 3 major objections 5 minor 18 references

ResKV shows that the attention mass lost by KV-cache eviction can be reconstructed from compact residual summaries, so a compressed cache keeps exact high-priority tokens while recovering omitted softmax contributions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:56 UTC pith:56ONTGCC

load-bearing objection A solid incremental KV-cache compression method — exact main cache plus cluster-summary residual cache — with broad but not airtight empirical support. the 3 major comments →

arxiv 2607.29591 v1 pith:56ONTGCC submitted 2026-07-31 cs.CL

ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

Yuhang Zhan , Lisi Chen , Shuo Shang This is my paper

classification cs.CL

keywords KV cache compressionlong-context inferencesoftmax attentionresidual attentioncache evictionquery-agnostic compressionattention reconstructionfixed-budget cache

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that the information discarded by KV-cache eviction is not irrecoverable: it is exactly a pair of softmax statistics—a numerator sum and a denominator sum—over the evicted tokens, and those statistics can be approximated by compact residual entries stored inside the same fixed cache budget. ResKV splits the budget into an exact main cache and a residual cache; each residual entry summarizes a cluster of evicted tokens by a mean key, a mean value, and a count, and at decode time the residual entries join the same softmax normalization as the main cache, restoring both numerator and denominator mass instead of acting as a post-hoc output correction. A construction-time validation proxy chooses how much residual budget each layer and KV head receives, and a decode-time dynamic gate scales residual contributions according to how sharp the main-cache attention is. The paper's empirical claim is that this main-plus-residual representation improves all 32 displayed LongBench configurations and 63 of the 64 displayed RULER configurations under the same retained KV budget, with the largest gains at tight budgets and in the query-agnostic setting where the cache is compressed before the future query is known.

Core claim

ResKV's central claim is that the omitted side of a fixed-budget KV cache can be represented as residual statistics and reconstructed from cluster summaries while selected tokens stay exact. For a query q, the full-cache output is (N_M(q)+N_E(q))/(Z_M(q)+Z_E(q)); hard eviction deletes N_E and Z_E and renormalizes over M alone. ResKV instead stores r compact entries—each a mean key, mean value, and count of a cluster of evicted tokens—and adds their contributions to the same denominator and numerator as the main cache, with a log-count logit bias and a query-dependent gate. The validation proxy selects r per layer and KV head by checking whether residual-augmented attention reduces held-out r

What carries the argument

The central object is the attention residual of the main cache, R(M;q) = (N_E(q), Z_E(q)): the numerator and denominator sums over evicted tokens. ResKV approximates R with cluster summaries. For each cluster C_j of evicted tokens it stores \bar k_j = (1/c_j)\sum k_p, \bar v_j = (1/c_j)\sum v_p, and c_j=|C_j|, and it treats the cluster as one approximate attention participant with logit \langle q,\bar k_j\rangle/\sqrt{d} + \log c_j + \log g(q). The clusters come from Lloyd iterations (k-means) in key space over the evicted side. The shared-softmax decoding formula (Z, N sums over main and residual entries) is what restores both numerator and denominator mass; the validation proxy and dynamic

Load-bearing premise

The reconstruction assumes evicted tokens in the same key-space cluster will have similar attention logits for future queries, so a mean key, mean value, and count can reproduce the omitted softmax mass; the paper's error bound depends on a per-token logit deviation η that is not controlled and can be large for queries unlike the cluster.

What would settle it

A direct falsifier: on a 32K-context multi-key retrieval prompt at 10% retained KV, hold out queries whose nearest cluster center in key space is far from the cluster (for example, beyond the 90th percentile of observed distances), and compare the shared-softmax reconstruction error ||\hat{o}(q)-o_S(q)|| to the r=0 main-cache error. If the residual-augmented output is worse on those out-of-cluster queries, the cluster-logit similarity assumption—and with it the residual reconstruction—fails where it is needed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Same-budget gains: replacing pure eviction with the main-plus-residual representation can raise LongBench and RULER scores at equal KV-slot counts, with the largest gains at 10% and 20% retained KV.
Query-agnostic serving: because residual summaries are built without the future query, ResKV improves the practical setting where a compressed prefix cache must serve queries that arrive later.
Exact memories survive: residual entries never rewrite main-cache tokens, so distributed-evidence tasks such as passage retrieval and code completion can keep exact high-priority tokens while recovering omitted context.
Adaptive memory allocation: the validation proxy lets each layer and KV head spend residual budget only where reconstruction reduces held-out attention-output error, so the same total budget is used more selectively.
Sharp-peak protection: the dynamic gate down-weights residual mass when main-cache attention is already concentrated, avoiding dilution of exact retrieval peaks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The softmax decomposition in Eqs. (3)-(5) does not depend on the specific model, so the main-plus-residual cache could be applied to other attention architectures, including sliding-window or encoder-decoder transformers, without retraining.
The count-weighted mean key/value summaries behave like a learned codebook of evicted tokens; a natural extension is to make the codebook query-conditional or train it end-to-end, which ResKV does not do.
The dynamic gate uses main-cache sharpness as a hint about whether eviction is safe at all; that same signal could serve as a runtime diagnostic for when to trust any compressed cache, not just a residual cache.
Residual entries are constructed once after prefill; a streaming variant that refreshes or grows clusters during generation would extend the method to workloads with very long generations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A solid incremental KV-cache compression method — exact main cache plus cluster-summary residual cache — with broad but not airtight empirical support.

read the letter

ResKV is a clean, modest addition to KV cache compression. The idea: under a fixed budget, keep selected tokens exactly in a main cache and spend the rest on compact residual entries that reconstruct the omitted softmax mass. The residual entries are mean key/value summaries with a count, and they participate in the same softmax as the main cache rather than being a post-hoc correction. The softmax decomposition in Eqs. 3–7 is textbook, but the specific instantiation — shared softmax, validation-proxy allocation, dynamic gate — appears new relative to the cited literature, and the paper is clearly written.

The empirical work is broad: LongBench and RULER, two backbones, query-aware and query-agnostic construction, four budgets, and several eviction baselines. The gains are consistent but modest on LongBench (around 1–3 points) and larger on RULER in the query-agnostic setting (around 4–5 points). The ablations show each component contributes. That is enough to take the method seriously.

The soft spots are real but mostly addressable. First, the clustering assumption is unverified. The paper approximates each cluster's omitted softmax mass by c_j exp(mean logit), and the error bound in Eq. 28 depends on the maximum per-token logit deviation, which is never measured. K-means in key space need not align with logit similarity under future queries, so the concern that residual entries could inject misleading mass is legitimate. The empirical gains suggest the heuristic works on these benchmarks, but the paper never reports reconstruction error or the eta distribution. That is the main theoretical weakness.

Second, the paper claims 63 of 64 RULER configurations improve but never identifies the one failure. That is a minor omission, but it matters for trust. Third, all results are single-seed with no error bars, and no code is released — this limits reproducibility. Finally, the closest residual-estimation baselines, RESA and KVSculpt, are cited but not compared experimentally; only CaM appears in one small table. For a paper whose whole point is residual estimation, that is a notable gap.

The validation proxy uses 32 held-out queries for budget selection. That is mild data-driven tuning, not circular, and the dynamic gate is a reasonable heuristic.

Who should read this: anyone working on KV cache compression or efficient long-context serving. It is not a breakthrough, but it is a useful, honest incremental method. I would send it to peer review, with requests for code, error bars or multiple seeds, identification of the failing RULER config, and a head-to-head against RESA and KVSculpt. The core idea is sound and worth engaging with.

Referee Report

3 major / 5 minor

Summary. The paper proposes ResKV, a fixed-budget KV cache compression method that splits a cache of b slots into an exact main cache (m slots) and a residual cache (r slots) representing evicted tokens. Evicted tokens are clustered with k-means; each residual entry stores a mean key, a mean value, and a cluster count. At decode time, residual entries participate in the same softmax as main-cache entries, with a log-count term and a query-dependent dynamic gate that down-weights residual mass when main-cache attention is sharp. A construction-time validation proxy selects the residual budget per layer and KV head based on reconstruction of attention outputs on a held-out set of 32 prefill queries. The paper reports improvements over H2O, SnapKV, TOVA, AdaKV, and CaM on LongBench and RULER, claiming improvements in all 32 displayed LongBench configurations and 63 of 64 displayed RULER configurations under the same retained KV budget. The theoretical motivation is a correct decomposition of softmax attention into main-cache and omitted-side numerator/denominator statistics, with residual entries approximating the omitted side.

Significance. If the empirical claim holds, ResKV is a useful and conceptually clean contribution: it avoids the binary discard of eviction methods and does not perturb retained keys/values as merging methods do. The softmax decomposition in Eqs. (3)–(7) is mathematically correct, and the idea of representing omitted attention mass as cache-resident residual statistics is appealing. The paper also provides an extensive set of benchmark tables, ablations of each component, and an efficiency analysis. The main weaknesses are that the key-space clustering assumption underlying the residual approximation is not directly verified, and the central empirical claim rests on single-seed runs without error bars or code release. These issues are fixable within the scope of the manuscript.

major comments (3)

[Residual Cache Construction, Eqs. (11)–(12); Appendix A.2, Eq. (28)] The approximation bound relies on η = max|a_p - \bar a_j|, the maximum per-token logit deviation within each cluster. The paper never measures or controls η, and k-means in key space (Eq. 12) minimizes Euclidean key distance, not logit deviation under the future query distribution. Although L2 cluster tightness gives a Lipschitz-style bound on logit deviation for any fixed query, the actual cluster radii and query norms are unreported. Without an empirical distribution of η or, more directly, a measurement of Z_E/N_E reconstruction error on held-out queries, the theoretical motivation does not yet explain the benchmark gains. Please report cluster logit-deviation statistics and/or reconstruction error, and relate them to the observed improvements.
[Experimental Setup / Appendix B] All reported experiments are run once with a fixed random seed (Appendix B: 'All reported experiments are run once with a fixed random seed of 42'). No error bars, confidence intervals, or multiple seeds are provided. Many improvements are small (e.g., Table 2, LongBench q-Aware SnapKV LLaMA 20%: 47.07 vs. 47.52), so without variance estimates the claim that ResKV 'improves all 32 displayed LongBench configurations' is not statistically established. I request at least 3 seeds or bootstrap CIs for the main average tables, and identification of the one failing RULER configuration — visible in Table 8 (RULER-32K, Qwen, H2O 10%, q-aware: baseline 33.97 vs. ResKV 33.80) — with a brief discussion.
[Construction-Time Validation Proxy, Eqs. (18)–(19), Appendix B.3] The validation proxy selects r* using 32 validation queries drawn from the prefill observation window. In the query-agnostic setting these are not future decode queries, and the paper provides no evidence that reconstruction accuracy on prompt-local queries transfers to actual decode-time queries. The claim that this aligns with the objective in Eq. (6) assumes the empirical query distribution Q is representative of future queries. A concrete test would be to compare r* selected by the proxy with r* selected on a set of true generated/held-out queries for a subset of tasks; if the choices diverge, the proxy mechanism needs qualification.

minor comments (5)

[Abstract / Introduction / Table 1-2] The phrase 'all 32 displayed LongBench configurations' is potentially misleading: in the full task-level tables (e.g., Table 9), several individual task cells decline even when the average improves. Please clarify that the 32/64 counts refer to average-score configurations, not all task-level cells.
[Appendix A.2, Eq. (29)] The symbol V is used both for the value matrix and for the value-norm upper bound in Eq. (29). Rename one of them (e.g., use B_v) to avoid ambiguity.
[Eq. (22)] Minor typographical issue: 'This dynamic gate enters attention as the logg(q) shift in (14).' should read 'log g(q) shift in Eq. (14).'
[Residual Cache Construction, Eq. (12)] The main text says the partition objective is solved with 'a few Lloyd iterations' without specifying initialization; the deterministic initialization is only described in Appendix B.3. Mention this in the main text for reproducibility.
[Overall] No code or model-release link is provided. Given the empirical nature of the contribution, releasing code would substantially aid reproducibility.

Circularity Check

0 steps flagged

No significant circularity: the residual-cache reconstruction is a self-contained approximation, and the reported benchmark gains are externally evaluated against fixed baselines.

full rationale

The paper's formal core is an exact softmax decomposition (Eq. 3-7) followed by a concrete approximation: residual entries store mean keys, mean values, and counts obtained by k-means over evicted keys (Eq. 10-12). None of these entries is fitted to the target benchmark outputs. The construction-time validation proxy (Eq. 18-19) selects a per-layer/head residual budget from a small grid by comparing held-out reconstruction error against the full-cache attention output; this is prompt-internal model selection, not a fit of the reported task scores. The decode-time dynamic gate (Eq. 21-22) is a hand-specified heuristic based on main-cache sharpness, and its contribution is tested by ablation. The empirical claims are measured on LongBench and RULER against representative baselines under equal retained KV-slot budgets, so no displayed improvement is identical by construction to a fitted parameter or to a self-citation. The paper contains no load-bearing self-citations, no imported uniqueness theorem, and no renaming of a known result as a new derivation. The main caveats—uncontrolled per-token logit deviation eta in Eq. 28 and the small 32-query validation window—are correctness/robustness concerns, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

ResKV introduces no new physical entities. The free parameters are mostly manual hyperparameters; the per-layer/head residual budget is the only data-dependent choice, tuned on a small validation set. The axioms include the standard softmax identity and a domain assumption that key-space clustering preserves query-logit similarity, which is the main risk.

free parameters (6)

per-layer/head residual budget ratio r = selected from {0, 0.05, 0.10, 0.15, 0.20} by validation proxy
Chosen to minimize L_val on 32 validation queries per layer/head; data-dependent.
dynamic gate threshold tau = 0.25
Manually set; controls when residual is down-weighted based on main-cache sharpness.
sigmoid slope alpha (Tg=1/alpha) = 12.0
Manual; controls smoothness of gate transition.
validation margin delta = 0.01
Manual; relative improvement required to enable residual entries.
count exponent beta = 1.0
Manual; exponent applied to log c_j in residual logits.
minimum residual gate g_min = 0.0
Manual; floor for dynamic gate.

axioms (4)

standard math Softmax attention can be decomposed into numerator and denominator sums over token sets (Eq. 3-5).
Used to motivate the residual formulation; correct by algebra.
domain assumption Evicted keys within a k-means cluster induce similar query logits, so Eq. 11 approximates the omitted statistics.
Key-space clustering is used as a query-agnostic surrogate; no theoretical guarantee for arbitrary queries.
domain assumption 32 validation queries from the context are representative of future decode queries for budget selection.
Validation proxy relies on this to choose r per layer/head without overfitting.
ad hoc to paper The dynamic gate's sharpness heuristic improves attention output quality.
Eq. 21 is a hand-designed heuristic, not derived from an objective; empirically motivated.

pith-pipeline@v1.3.0-daily-deepseek · 40657 in / 10602 out tokens · 93403 ms · 2026-08-03T03:56:31.659455+00:00 · methodology

0 comments

read the original abstract

KV cache compression is essential for efficient long-context inference. Existing eviction methods permanently discard unselected tokens and consequently remove their aggregate contribution to attention. Merging-based alternatives preserve more information but can perturb retained keys and values that should remain exact. We observe that the information omitted by cache eviction can be formulated as residual statistics in both the numerator and denominator of softmax attention. Based on this observation, we propose ResKV, which divides a fixed KV budget into an exact main cache and a compact residual cache that reconstructs the contribution of omitted tokens. ResKV lets main-cache tokens and residual entries participate in the same softmax normalization, so residual entries restore both attention numerator and denominator mass rather than acting as a post-hoc correction. A construction-time validation proxy determines residual allocation for each layer and KV head, while a decode-time dynamic gate adjusts residual contributions for individual queries. Comprehensive evaluations on LongBench and RULER, covering query-aware and query-agnostic settings, multiple backbones, cache budgets, and representative compression baselines, demonstrate broad improvements under the same retained KV budget while preserving the practical efficiency of compressed decoding, including peak memory usage and long-context decode throughput.

Figures

Figures reproduced from arXiv: 2607.29591 by Lisi Chen, Shuo Shang, Yuhang Zhan.

**Figure 1.** Figure 1: Overview of ResKV. Given a fixed budget b = m + r, ResKV keeps selected tokens exactly in the main cache M and reconstructs the omitted side E as a residual cache R. During decoding, the main and residual branches are combined by a dynamic gate. Let Πr = {C1, . . . , Cr} be a partition of E. Each residual entry represents one group Cj and stores three quantities: ¯kj = 1 cj X p∈Cj kp, v¯j = 1 cj X p∈Cj vp,… view at source ↗

**Figure 2.** Figure 2: Adaptive residual control. Left: the validation proxy [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Analysis of ResKV efficiency and residual clusterability. (a)–(b) compare peak allocated memory and decode throughput [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

[3]

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Jun- jie Hu, and Wen Xiao

doi: 10.18653/v1/2024.acl-long.172. Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Jun- jie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. In Proceedings of the Second Conference on Language Model- ing,

doi:10.18653/v1/2024.acl-long.172 2024
[4]

Kevin Zhou

Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. InAdvances inNeuralInformationProcessingSystems,volume38,2025. Aaron Grattafiori et al. The Llama 3 herd of models,

2025
[5]

Yuzhe Gu, Xiyu Liang, Jiaojiao Zhao, and Enmao Diao

URL https://arxiv.org/abs/2407.21783. Yuzhe Gu, Xiyu Liang, Jiaojiao Zhao, and Enmao Diao. OBCache: Optimal brain KV cache pruning for efficient long-context LLM inference. InProceedings of the 43rd International Conference on Machine Learning,

Pith/arXiv arXiv
[7]

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen

doi: 10.1145/3600006.3613165. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. InAdvances in Neural Information Processing Systems, volume 37,

arXiv
[8]

URL https://arxiv.org/ abs/2409.10516

Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, and LiliQiu.RetrievalAttention:Acceleratinglong-contextLLM inference via vector retrieval, 2024a. URL https://arxiv.org/ abs/2409.10516. Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Miny...

Pith/arXiv arXiv
[9]

Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu

URL https://arxiv.org/abs/2305.17118. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. InProceedings of the 41st International Confer- ence on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 32332–32344. ...

Pith/arXiv arXiv
[10]

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang

18653/v1/2025.findings-acl.952. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-throughput gen- erative inference of large language models with a single GPU.InProceedingsofthe40thInternationalConferenceon Machine Learning, volume 202 ofProceedings of Machin...

2025
[11]

Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang

doi: 10.18653/v1/2024.emnlp-main.1043. Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang. MiniKV: Pushing the limits of 2-bit KV cache via compression and system co-design for efficient long context inference. InFindings of the Association for Computational Linguistics: ACL 2025, pages18506–18523. Association for Computational Linguistics,

doi:10.18653/v1/2024.emnlp-main.1043 2024
[13]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N

doi: 10.1609/aaai.v40i39.40611. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attentionisallyouneed. InAdvancesinNeural Information Processing Systems, volume 30,

work page doi:10.1609/aaai.v40i39.40611
[15]

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis

doi: 10.1109/ ICASSP55912.2026.11464823. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InInternational Conference on Learning Representations,

arXiv 2026
[16]

WeihaoYang,HaoHuang,NingkeLi,ShihaoWang,Darong Yang, Yanqi Pan, Wen Xia, Shiyi Li, and Xiangyu Zou

URL https: //arxiv.org/abs/2412.15115. WeihaoYang,HaoHuang,NingkeLi,ShihaoWang,Darong Yang, Yanqi Pan, Wen Xia, Shiyi Li, and Xiangyu Zou. RESA: Bringing back what sparse attention ignores with residualestimation.InInternationalConferenceonLearning Representations,

Pith/arXiv arXiv
[17]

Dy- namicKV: Task-aware adaptive KV cache compression for long context LLMs

Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, and Liang Ding. Dy- namicKV: Task-aware adaptive KV cache compression for long context LLMs. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 8042– 8057.AssociationforComputationalLinguistics,2025. doi: 10.18653/v1/2025.findings-emnlp.426. Appe...

doi:10.18653/v1/2025.findings-emnlp.426 2025
[18]

It does not refresh resid- ual entries during long generation

and computeLval 0 4:foreachr∈ Gwithr >0do 5:m←b−r; selectM r by (8); setEr ←S\M r 6:buildrresidual entries fromE r by (12) and (10) 7:computeL val r by (18) 8:end for 9:r ⋆ ←arg min r>0 Lval r 10:ifL val r⋆ <(1−δ)L val 0 then 11:chooser ⋆ 12:else 13:chooser= 0 14:end if 15:rebuild(M,S)with the chosenr; return(K M , VM ),S Algorithm2ResKVdecodingstepforone...

arXiv 2042
[1982]

Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz

doi: 10.1109/TIT.1982.1056489. Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state RNNs. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pages 18724–18741. Association for Computational Linguistics,

arXiv 1982
[2017]

Modeltellsyouwheretomerge:AdaptiveKVcachemerging forLLMsonlong-contexttasks,2024.URLhttps://arxiv.org/ abs/2407.08454

Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Modeltellsyouwheretomerge:AdaptiveKVcachemerging forLLMsonlong-contexttasks,2024.URLhttps://arxiv.org/ abs/2407.08454. Zihao Wang, Bin Cui, and Shaoduo Gan. SqueezeAtten- tion: 2D management of KV cache in LLM inference via layer-wise optimal budget. InInternational Conference on Learning Representations,

Pith/arXiv arXiv 2024
[2023]

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang,ZhidianHuang,ZhengxiaoDu,XiaoLiu,AohanZeng, LeiHou,YuxiaoDong,JieTang,andJuanziLi

doi: 10.18653/v1/2023.emnlp-main.298. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang,ZhidianHuang,ZhengxiaoDu,XiaoLiu,AohanZeng, LeiHou,YuxiaoDong,JieTang,andJuanziLi. LongBench: A bilingual, multitask benchmark for long context under- standing. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume...

doi:10.18653/v1/2023.emnlp-main.298 2023
[2024]

GQA: Training generalized multi-query transformer models from multi-head checkpoints

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Confer- enceonEmpiricalMethodsinNaturalLanguageProcessing, pages 4895–4901. Association for Computational Linguis- tics,

2023
[2026]

Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao

URL https://arxiv.org/abs/2603.27819. Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: An ef- ficienterrorreductionframeworkforKVcachecompression in LLM inference. InProceedings of The 4th NeurIPS Ef- ficient Natural Language and Speech Processing Workshop, volume 262 ofProceedings of Machine Learning R...

arXiv

[1] [3]

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Jun- jie Hu, and Wen Xiao

doi: 10.18653/v1/2024.acl-long.172. Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Jun- jie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. In Proceedings of the Second Conference on Language Model- ing,

doi:10.18653/v1/2024.acl-long.172 2024

[2] [4]

Kevin Zhou

Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. InAdvances inNeuralInformationProcessingSystems,volume38,2025. Aaron Grattafiori et al. The Llama 3 herd of models,

2025

[3] [5]

Yuzhe Gu, Xiyu Liang, Jiaojiao Zhao, and Enmao Diao

URL https://arxiv.org/abs/2407.21783. Yuzhe Gu, Xiyu Liang, Jiaojiao Zhao, and Enmao Diao. OBCache: Optimal brain KV cache pruning for efficient long-context LLM inference. InProceedings of the 43rd International Conference on Machine Learning,

Pith/arXiv arXiv

[4] [7]

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen

doi: 10.1145/3600006.3613165. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. InAdvances in Neural Information Processing Systems, volume 37,

arXiv

[5] [8]

URL https://arxiv.org/ abs/2409.10516

Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang, Yuqing Yang, and LiliQiu.RetrievalAttention:Acceleratinglong-contextLLM inference via vector retrieval, 2024a. URL https://arxiv.org/ abs/2409.10516. Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Miny...

Pith/arXiv arXiv

[6] [9]

Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu

URL https://arxiv.org/abs/2305.17118. Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. InProceedings of the 41st International Confer- ence on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 32332–32344. ...

Pith/arXiv arXiv

[7] [10]

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang

18653/v1/2025.findings-acl.952. Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. FlexGen: High-throughput gen- erative inference of large language models with a single GPU.InProceedingsofthe40thInternationalConferenceon Machine Learning, volume 202 ofProceedings of Machin...

2025

[8] [11]

Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang

doi: 10.18653/v1/2024.emnlp-main.1043. Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang. MiniKV: Pushing the limits of 2-bit KV cache via compression and system co-design for efficient long context inference. InFindings of the Association for Computational Linguistics: ACL 2025, pages18506–18523. Association for Computational Linguistics,

doi:10.18653/v1/2024.emnlp-main.1043 2024

[9] [13]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N

doi: 10.1609/aaai.v40i39.40611. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attentionisallyouneed. InAdvancesinNeural Information Processing Systems, volume 30,

work page doi:10.1609/aaai.v40i39.40611

[10] [15]

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis

doi: 10.1109/ ICASSP55912.2026.11464823. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InInternational Conference on Learning Representations,

arXiv 2026

[11] [16]

WeihaoYang,HaoHuang,NingkeLi,ShihaoWang,Darong Yang, Yanqi Pan, Wen Xia, Shiyi Li, and Xiangyu Zou

URL https: //arxiv.org/abs/2412.15115. WeihaoYang,HaoHuang,NingkeLi,ShihaoWang,Darong Yang, Yanqi Pan, Wen Xia, Shiyi Li, and Xiangyu Zou. RESA: Bringing back what sparse attention ignores with residualestimation.InInternationalConferenceonLearning Representations,

Pith/arXiv arXiv

[12] [17]

Dy- namicKV: Task-aware adaptive KV cache compression for long context LLMs

Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, and Liang Ding. Dy- namicKV: Task-aware adaptive KV cache compression for long context LLMs. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 8042– 8057.AssociationforComputationalLinguistics,2025. doi: 10.18653/v1/2025.findings-emnlp.426. Appe...

doi:10.18653/v1/2025.findings-emnlp.426 2025

[13] [18]

It does not refresh resid- ual entries during long generation

and computeLval 0 4:foreachr∈ Gwithr >0do 5:m←b−r; selectM r by (8); setEr ←S\M r 6:buildrresidual entries fromE r by (12) and (10) 7:computeL val r by (18) 8:end for 9:r ⋆ ←arg min r>0 Lval r 10:ifL val r⋆ <(1−δ)L val 0 then 11:chooser ⋆ 12:else 13:chooser= 0 14:end if 15:rebuild(M,S)with the chosenr; return(K M , VM ),S Algorithm2ResKVdecodingstepforone...

arXiv 2042

[14] [1982]

Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz

doi: 10.1109/TIT.1982.1056489. Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state RNNs. In Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pages 18724–18741. Association for Computational Linguistics,

arXiv 1982

[15] [2017]

Modeltellsyouwheretomerge:AdaptiveKVcachemerging forLLMsonlong-contexttasks,2024.URLhttps://arxiv.org/ abs/2407.08454

Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Modeltellsyouwheretomerge:AdaptiveKVcachemerging forLLMsonlong-contexttasks,2024.URLhttps://arxiv.org/ abs/2407.08454. Zihao Wang, Bin Cui, and Shaoduo Gan. SqueezeAtten- tion: 2D management of KV cache in LLM inference via layer-wise optimal budget. InInternational Conference on Learning Representations,

Pith/arXiv arXiv 2024

[16] [2023]

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang,ZhidianHuang,ZhengxiaoDu,XiaoLiu,AohanZeng, LeiHou,YuxiaoDong,JieTang,andJuanziLi

doi: 10.18653/v1/2023.emnlp-main.298. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang,ZhidianHuang,ZhengxiaoDu,XiaoLiu,AohanZeng, LeiHou,YuxiaoDong,JieTang,andJuanziLi. LongBench: A bilingual, multitask benchmark for long context under- standing. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume...

doi:10.18653/v1/2023.emnlp-main.298 2023

[17] [2024]

GQA: Training generalized multi-query transformer models from multi-head checkpoints

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. InProceedings of the 2023 Confer- enceonEmpiricalMethodsinNaturalLanguageProcessing, pages 4895–4901. Association for Computational Linguis- tics,

2023

[18] [2026]

Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao

URL https://arxiv.org/abs/2603.27819. Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: An ef- ficienterrorreductionframeworkforKVcachecompression in LLM inference. InProceedings of The 4th NeurIPS Ef- ficient Natural Language and Speech Processing Workshop, volume 262 ofProceedings of Machine Learning R...

arXiv