Pith. sign in

REVIEW 3 major objections 5 minor 54 references

KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read KVDiagnosis pairs every compressed run with a FullCache control, then attributes KV-cache compression failures to coverage, drift, attention, or decoding causes.

desk verdict A genuinely useful diagnostic resource with a sound matched-control design; the headline numbers should be trusted only after the evidence-span alignment is made auditable, and I'd send it to review. read the letter →

arxiv 2608.09412 v1 pith:Q7HZDQSJ submitted 2026-08-10 cs.AI

Chen Qiu , Ziwu Liu , Chao Fei , Guozhong Li , Panos Kalnis This is my paper · ORCID
classification cs.AI
keywords KV-cachecompressionlong-contextlanguagemodelsdiagnosticbenchmarkevidencecoverageattentionretentionfailureanalysisFullCachecontrollikelihooddrift
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KVDiagnosis is a benchmark and released dataset for diagnosing why KV-cache compression turns correct answers into wrong ones. Its central claim is that aggregate accuracy hides a method-specific failure anatomy: on Qwen3-8B, 59,800 supported compressed runs over 2,600 sources produced 12,520 runs that were correct under a per-source FullCache control but wrong after compression, and under fixed diagnostic rules 63.2% of those failures show low or partial measured/projected evidence coverage. The paper reports that all ten diagnostics—cache coverage, likelihood drift, attention retention, and decoding traces—separate failed from successful compression (stratified AUROC 0.684–0.871), and that a controlled 4x boost of attention to retained answer-supporting evidence repairs 29.2% of reproducible low-attention-retention failures versus 6.3% under a count-matched sham. If correct, the work provides a reusable format for attributing compression failures to specific mechanisms rather than reading them off task scores.

What carries the argument

The load-bearing object is the paired run matrix: FullCache executed once per source, reused by every method-setting cell, with C-to-W failure rows selected only after all supported runs complete. On that matrix the benchmark defines ten diagnostics with explicit applicability states. Cache retention uses ERR (evidence retention rate) and ECov (evidence coverage), averaged over layer-KV-head slots with a threshold of $\tau=0.5$ so a cross-head union cannot manufacture perfect coverage; likelihood drift uses $\Delta\mathrm{NLL}$, GPR, KL, Top-50 overlap, and gold-token rank shift; attention access uses EAM (evidence attention mass), EAR (evidence attention retention), and NEAE (normalized evidence attention enrichment); decoding records gold ranks and emitted output. The taxonomy of 25 methods into five mechanism families tells which diagnostics are valid for which compressed object—positions, head-wise slots, channels, precisions, or chunks—and the paper's validation machinery is the matched C-to-C control group plus the low-EAR intervention with a sham control.

What would settle it

Independently re-annotate the evidence token positions in the final tokenized prompts for a random sample of the 12,520 C-to-W rows and recompute ERR, ECov, and EAR: if the remapped spans shift by more than a few tokens, the category shares (63.2% low or partial coverage) and the 29.2%-versus-6.3% low-EAR repair gap should shrink or disappear. A cheaper variant is to rerun the benchmark with evidence spans deliberately offset by several tokens and check whether the AUROC separations degrade.

Watch

Extended reading notes

Core claim

The core discovery is that a compressed-model answer error is attributable: it shows up as a specific, measurable gap between the compressed run and its FullCache partner. KVDiagnosis constructs that pairing by running FullCache once per source, running every supported method-setting cell over the same fixed split, and only then extracting C-to-W rows—sources correct under FullCache that become wrong under a specific compressor—so no compressor's failure set is defined by another method's failures. Across the four evidence-aware workloads, the 12,520 C-to-W rows separate into repeatedly observed signatures: low or partial measured/projected evidence coverage (63.2% of rows), structural position addressability with likelihood drift (17.0%), rare high-coverage drift (0.2%), low evidence-attention retention candidates (0.8%), and decoding/scoring sensitivity (3.2%). All ten diagnostics separate C-to-W failures from C-to-C successes on matched controls, and the low-EAR cohort responds selectively to an evidence-attention boost. The paper's terminology is careful: structural addressability is recorded separately from measured coverage, because retaining a position index does not by itself establish that the stored representation is intact.

Load-bearing premise

The entire diagnostic apparatus treats the evidence-span annotations as ground truth: if the support spans from Qasper, HotpotQA, and the RULER generators are misaligned after final-prompt tokenization, then ERR, ECov, EAM, EAR, and the low-EAR repair cohort are all miscalibrated.

Editorial extensions

If this is right

  • Two compressors with nearly equal mean scores can fail on almost disjoint source sets: SnapKV and TOVA agree on at most Jaccard 0.385 in 11 of 12 workload-setting cells, so method rankings must not be read as interchangeability.
  • Evidence coverage, not just score, is the dominant failure signature: 63.2% of C-to-W rows have low or partial measured/projected coverage, so budgets should be reported with evidence-retention curves.
  • Structural position addressability is not coverage: for ThinK and QuantizedCache, 2,126 rows show strong likelihood drift even though evidence positions remain addressable, which means tensor-fidelity diagnostics must accompany retention claims.
  • Access failures are repairable in principle: boosting attention to retained evidence positions repaired 29.2% of reproducible low-EAR failures versus 6.3% for a sham boost, with only 3.3% degradation on matched controls, so targeting the access component is a viable repair route.
  • The diagnostic protocol transfers across the three tested model families for coverage trends, while quantization-related drift remains model-specific; follow-up work should treat the procedure, not the absolute metric values, as the portable part.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if evidence coverage is the dominant failure mode, then long-context compressor comparisons should standardize an evidence-retention curve (coverage at each budget) as a reporting table, not only score-vs-budget points.
  • Beyond the paper: the low-EAR repair result suggests an inference-time controller that monitors EAR during generation and re-weights attention to retained evidence could serve as a runtime recovery mechanism; the paper's design stops at diagnosis, so that policy is an extension.
  • Beyond the paper: the AUROCs are computed with sources recurring across cells and described by the authors as descriptive; a held-out-sources split would test whether these same features predict failure rank on unseen prompts.
  • Beyond the paper: the paired C-to-W record format turns failure analysis into a supervised learning problem; one can train a classifier over cache, attention, and likelihood features to predict which method-settings will break a given source before decompression.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces KVDiagnosis, a paired diagnostic benchmark for KV-cache compression failures. It fixes per-source FullCache controls, runs all supported method-setting cells over fixed source splits, selects FullCache-correct-to-compressed-wrong (C-to-W) rows separately for each method-setting, and attaches cache-retention, likelihood, attention, and decoding diagnostics with explicit N/A semantics. On Qwen3-8B over RULER-8K, RULER-16K, Qasper, and HotpotQA, it reports 59,800 supported compressed runs and 12,520 C-to-W rows; under predeclared diagnostic rules, 63.2% of failure rows show low or partial measured/projected coverage. Ten diagnostics separate C-to-W rows from a C-to-C control ledger (stratified AUROC 0.684-0.871), and a controlled 4x evidence-attention boost repairs 29.2% of 96 reproducible low-EAR failures versus 6.3% under a count-matched sham, with a matched C-to-C safety check. Cross-model results on Falcon3-7B and Mistral-Small-24B show that coverage trends transfer across architectures while quantization effects remain model-specific.

Significance. KVDiagnosis is potentially a valuable community resource. Its design choices address real gaps in compression evaluation: matched per-source FullCache controls, method-specific failure sets selected after the full run matrix is complete, explicit applicability states instead of silent imputation, a C-to-C control ledger, predeclared diagnostic rules, and a sham-controlled repair experiment with paired bootstrap and McNemar statistics. The PyramidKV adapter audit and the detailed run accounting in Tables 3 and 10 are unusually careful, and the released artifacts appear reproducible. If the evidence-span alignments are correct, the headline failure profiles, the AUROC separation, and the low-EAR repair result are well supported. The main open risk is that ERR, ECov, EAM, EAR, the diagnostic categories in Table 4, and the repair cohort all treat the evidence spans as ground truth, and the paper has not yet provided quantitative validation of those alignments.

major comments (3)
  1. [Section 3.2 and Appendix D] The evidence-span alignment is the load-bearing ground truth for ERR, ECov, EAM, EAR, the diagnostic categories in Table 4, and the low-EAR repair cohort, yet the paper only states that Qasper/HotpotQA support text and RULER generator spans are aligned after final-prompt tokenization and that alignment success is recorded. It reports no per-source alignment concordance, no manual audit size or error rate, and no description of how duplicate or near-duplicate support sentences in HotpotQA are resolved. Because ambiguous mappings are relegated to N/A, the reported rows are exactly those asserted to be aligned, so a systematic tokenization or mapping error would silently miscalibrate every evidence-based diagnostic. Please add quantitative alignment validation, including exact-span match rates, position-offset distributions, a manual audit of a stratified sample, and released alignment maps, and describe the duplicate-resolution rule for HotpotQA.
  2. [Section 4 and Table 4] The headline '63.2% low or partial measured/projected coverage' and the category counts in Table 4 are determined by hand-set cutoffs, specifically the ECov thresholds of 0.5 and 0.9, the Delta-NLL threshold of 1, and the EAR threshold of 0.5. The rules are predeclared, which rules out post hoc selection, but the numerical claims are still a direct function of these constants. Please report a sensitivity sweep over the cutoffs or provide a distribution-based justification, and state whether the main qualitative findings, namely that coverage loss dominates and that low-EAR repair is selective, are stable across reasonable threshold choices.
  3. [Section 5.3 and Figure 6] The C-to-C control ledger is central to the claim that all ten diagnostics separate failures from successful compression, but the paper does not specify how the 4,936 C-to-C runs were selected or stratified beyond the phrase 'mixed strata,' and sources recur across rows. Since repeated sources can inflate apparent separation, please report the number of unique sources per stratum and use a source-clustered bootstrap or equivalent procedure for the AUROC intervals, and state explicitly whether the C-to-C controls are matched to the C-to-W rows on method, setting, and source.
minor comments (5)
  1. [Abstract and Table 3] The abstract says 59,800 supported compressed runs, while Table 3 records 62,400 total records with 2,600 N/A runs; please state the total record count and the exclusion rule in the caption or a footnote so readers can reconcile the counts at a glance.
  2. [Figure 3 caption] The '75/50/25' setting tier applies to retention and channel methods, while QuantizedCache uses 8b/4b/2b; the caption says this, but the shared axis labels in panels A and B could mislead readers, so consider adding a second axis label or a footnote.
  3. [Section 5.3] The 4x evidence boost is an oracle intervention that uses gold evidence spans; the word 'repairs' in the abstract and findings could be misread as a deployable repair method, so please add an explicit qualifier that the intervention is a controlled diagnostic probe, not a proposed inference-time method.
  4. [Section 5.5 and Section 5.6] The attention-based diagnostics and the intervention are Qwen-only, and this limitation is stated in Section 5.6, but Finding 4 should remind readers in the main text that cross-model validation covers coverage and likelihood diagnostics only, not attention or EAR-based repair.
  5. [Appendix A] The PyramidKV adapter audit is well handled; consider moving one sentence about the exclusion into Section 5.1 so that main-text readers know the budget-allocation family is represented by AdaKV alone in the evaluated set.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: C-to-W selection is answer-based and diagnostics are validated against independent C-to-C controls; author self-citations are non-load-bearing survey mentions.

full rationale

KVDiagnosis is a measurement/resource paper, not a derivation of a law. The central C-to-W selection (Section 3.3) is defined by answer correctness alone: F_{d,m,b} = {i : correct^F_i = 1 and correct^C_{i,m,b} = 0}. Diagnostics are computed on those rows afterward, so no diagnostic defines the failure set. The AUROC separation (Section 5.3, Figure 6) compares C-to-W rows against an independent ledger of 4,936 C-to-C runs, making the separation empirical rather than forced. Thresholds (tau = 0.5, Delta-NLL >= 1, EAR < 0.5, 4x evidence boost) are fixed and predeclared, not fitted to the data; the low-EAR repair is tested against a count-matched sham intervention and a C-to-C safety control, so the 29.2% versus 6.3% difference is a controlled measurement. The only author self-citations ([9] PolyKV, [10] CHESS, [39] ART) appear in the taxonomy/related-work survey and support no load-bearing claim; no uniqueness theorem is imported. The evidence-span alignment issue raised by the skeptic is a correctness risk, acknowledged in Section 3.2 (alignment after final-prompt tokenization with N/A semantics), not a circular reduction: the diagnostics are defined relative to ground-truth evidence spans by design, and the C-to-W/C-to-C labels do not depend on those spans. No equation in the paper reduces a predicted quantity to an input by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on evidence-span alignment accuracy, FullCache greedy correctness as the reference, faithful third-party adapters, and eager-mode attention trace equivalence. These are domain assumptions rather than standard math axioms. The only hand-chosen numbers are diagnostic thresholds and the 4x logit boost, which are predeclared rather than fitted, so they contribute a small circularity burden without invalidating the C-to-C versus C-to-W discrimination test.

free parameters (5)
  • ECov coverage threshold tau = 0.5
    Separates low from partial and high mapped coverage in the predefined diagnostic rules; the 40.3% low-coverage and 22.9% partial-coverage row shares depend on this cutoff. Chosen by hand, not fitted.
  • Delta-NLL strong-drift threshold = 1.0
    Defines strong likelihood drift for the 19 high-mapped-coverage drift rows and the 2,126 structural-position drift rows. Chosen by hand, not fitted.
  • Low-EAR candidate threshold = EAR < 0.50
    Selects the 104 low-EAR C-to-W rows, of which 96 reproduce under the intervention baseline; the repair result depends on this cutoff. Chosen by hand, not fitted.
  • Decoding/scoring candidate thresholds = |Delta-NLL| <= 0.10 and Top-50 >= 0.90
    Defines the 405 decoding/scoring candidate rows that preserve local likelihood stability but emit wrong outputs. Chosen by hand, not fitted.
  • Evidence attention boost magnitude = log(4) added to attention logits
    The 4x evidence-attention boost used in the repair experiment is a round magnitude chosen by the authors, validated against a count-matched sham and C-to-C controls rather than fitted to maximize repair rate.
assumptions (4)
  • domain assumption Evidence spans E_i and their token alignment are accurate for all sources.
    ERR, ECov, EAM, EAR, and the low-EAR repair cohort are all defined with respect to E_i. Misaligned support spans would mislabel coverage and access. Entered in Section 3.2 and Section 4.
  • domain assumption FullCache greedy decoding is the reference for correctness, and the task scorers capture the right notion of correct.
    C-to-W labels and all denominators are defined by FullCache correctness; the QA scorer is benchmark-specific and marks correctness only at 1.0. A flawed scorer would change the failure set. Entered in Section 3.3 and Section 5.1.
  • domain assumption The eight verified adapters (kvpress 0.5.3 and Transformers/HQQ) faithfully implement their named methods.
    The benchmark claims verified implementations, yet the PyramidKV audit shows bundled code can silently bypass a method's core logic; the other adapters rely on the same audit process. Entered in Section 3.1 and Appendix A.
  • domain assumption Eager-mode attention traces are behaviorally equivalent to the compressed path used for generation.
    Attention traces are run in eager mode only when compression behavior is unchanged; if the eager path diverges from the flash-attention path, EAR and EAM would be measured on a different execution. Entered in Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models." pith.science (2026). https://pith.science/paper/Q7HZDQSJ

@misc{pith2026260809412,
  author       = {Pith},
  title        = {Pith review of: KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q7HZDQSJ}},
  note         = {Machine review of arXiv:2608.09412}
}
read the original abstract

KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why. We present KVDiagnosis, a diagnostic dataset and benchmark with three contributions. First, a 25-method taxonomy groups methods into five mechanism families and links them to eight verified implementations and their valid diagnostic measurements. Second, for every supported method setting, we evaluate all sources in each fixed split against a per-source FullCache control before selecting FullCache-correct/compressed-wrong (C-to-W) rows separately for each method-setting, so no compressor defines another's test set. Third, a common record format links paired outputs and run metadata to cache, likelihood, attention, and decoding measurements with explicit applicability states. On Qwen3-8B, four evidence-aware workloads yield 59 800 supported compressed runs over 2600 sources and 12 520 C-to-W rows. Under fixed diagnostic rules, 63.2% have low or partial measured/projected coverage. Only 19 rows (0.2%) combine high measured/projected coverage with strong likelihood drift; another 2,126 (17.0%) preserve structural position addressability, for which representation fidelity remains unknown, while showing the same drift. Against C-to-C success controls, all ten diagnostics separate failed from successful compression (stratified AUROC 0.684-0.871). Among 96 reproducible low-EAR failures, a controlled 4x evidence-attention boost repairs 29.2%, versus 6.3% under a count-matched sham intervention and 3.3% degradation on matched C-to-C controls. Code and data are available at https://github.com/ChosenQC/KVDiagnosis.

Figures

Figures reproduced from arXiv: 2608.09412 by the authors.

Figure 1
Figure 1. KV-cache compression taxonomy and evaluated subset. Each method is assigned by its primary compression decision [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. All supported runs precede cell-wise C→W selec￾tion and diagnostics. Method-specific sets preserve coverage; intersections support controlled comparisons. 3.3 Evaluation on All Sources and Failure Analysis We first evaluate every source, then analyze failures without chang￾ing the source set. Let X𝑑 be workload 𝑑’s fixed benchmark source split, 𝑆 𝐹 𝑖 the FullCache score, and 𝑆 𝐶 𝑖,𝑚,𝑏 the compressed score for method… view at source ↗
Figure 4
Figure 4. Failure frequency and identity: (A) workload-level [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: RULER-8K failures by task and method. Counts [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 8
Figure 8. Figure 8: Paired QA cases. ThinK preserves structural address [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 7
Figure 7. Figure 7: Diagnostic-category composition on evidence [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 9
Figure 9. Figure 9: Cross-model validation. (A) QA C→W rate pooled over six position methods and QuantizedCache; the setting tiers map 75/50/25% to 8b/4b/2b for QuantizedCache. (B) Mean slot-ECov over RULER-16K/QA position-method failures at 25% retention; dashed: low-coverage threshold. …
Figure 10
Figure 10. Figure 10: Composition of the corrected RULER-8K diagnostic corpus. Left: selected failure rows by setting tier; 25/50/75% [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Visual summary of Table 14. Structural ERR/ECov cells are [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Deterministic task-template label distributions (left) and metric-derived demand scores (right) on corrected RULER [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 32 canonical work pages

  1. [1]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J Nair, Ilya Solov- eychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference.Proceedings of Machine Learning and Systems6 (2024), 114–127

    work page 2024

  2. [2]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ...

    doi:10.18653/v1/2024.acl-long.172 2024

  3. [3]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. PyramidKV: Dynamic KV Cache Compression Based on Pyramidal Information Funneling. https://doi.org/10.48550/arXiv.2406.02069 arXiv:2406.02069 [cs.CL]

    arXiv doi:10.48550/arxiv.2406.02069 2024

  4. [4]

    Abdelfattah, and Kai- Chiang Wu

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai- Chiang Wu. 2025. Palu: KV-Cache Compression with Low-Rank Projection. In International Conference on Learning Representations. https://openreview.net/ forum?id=LWMS4pk2vK

    work page 2025

  5. [5]

    Alex Chen, Renato Geh, Aditya Grover, Guy Van Den Broeck, and Daniel Mingyi Israel. 2026. The Pitfalls of KV Cache Compression. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 41530–41553. https://doi. org/10.18653/v1/2026.acl-long.1926

    work page doi:10.18653/v1/2026.acl-long.1926 2026

  6. [6]

    Jian Chen, Zhuoran Wang, Jiayu Qin, Ming Li, Meng Wang, Changyou Chen, Yin Chen, Qizhen Weng, and Yirui Liu. 2026. KV-CoRE: Benchmarking Data- Dependent Low-Rank Compressibility of KV-Caches in LLMs. https://doi.org/ 10.48550/arXiv.2602.05929 arXiv:2602.05929 [cs.LG]

    work page doi:10.48550/arxiv.2602.05929 2026

  7. [7]

    Smith, and Matt Gard- ner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gard- ner. 2021. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. InProceedings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies. 4599–4610

    work page 2021

  8. [8]

    Alessio Devoto, Maximilian Jeblick, and Simon Jégou. 2025. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribu- tion.arXiv preprint arXiv:2510.00636(2025). https://arxiv.org/abs/2510.00636

    arXiv 2025

Show all 54 references
  1. [9]

    Chao Fei and Panos Kalnis. 2026. PolyKV: Heterogeneous Retention and Allo- cation for KV Cache Compression. https://doi.org/10.48550/arXiv.2606.15157 arXiv:2606.15157 [cs.LG]

    2026 doi

  2. [10]

    Chao Fei, Guozhong Li, Chenxi Liu, and Panos Kalnis. 2026. CHESS: Context- aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference. arXiv:2602.20732 https://arxiv.org/abs/2602.20732

    2026

  3. [11]

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. 2025. Ada- KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Ef- ficient LLM Inference. InAdvances in Neural Information Processing Sys- tems, Vol. 38. https://proceedings.neurips.cc/paper_files/pape...

    2025

  4. [12]

    Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao

  5. [13]

    Wei Gao, Xinyu Zhou, Peng Sun, Tianwei Zhang, and Yonggang Wen. 2025. Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving. InProceedings of Machine Learning and Systems

    2025

  6. [14]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. InInternational Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 639a9a1...

    2024

  7. [15]

    Yoav Gelberg, Yam Eitan, Michael Bronstein, Yarin Gal, and Haggai Maron. 2026. Training Transformers for KV Cache Compressibility. arXiv:2605.05971 [cs.LG] https://arxiv.org/abs/2605.05971

    2026 arXiv

  8. [16]

    Nathan Godey, Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini, Éric de la Clergerie, and Benoît Sagot. 2025. Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression. arXiv:2503.02812 [cs.CL] https://arxiv.org/ abs/2503.02812

    2025 arXiv

  9. [18]

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang

  10. [19]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems37 (2024), 1270–1303

    2024

  11. [20]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654 [cs.CL] https: //arxiv.org/abs/2404.06654

    2024 arXiv

  12. [21]

    Hugging Face. 2026. Cache Strategies: Quantized Cache. https://huggingface.co/ docs/transformers/kv_cache. Transformers documentation; accessed 2026-07-19

    2026

  13. [22]

    Chen, Xu Yuan, Ye Jia, Jiancheng Tu, Chen Li, Peter H

    Zhuohang Jiang, Pangjing Wu, Ziran Liang, Peter Q. Chen, Xu Yuan, Ye Jia, Jiancheng Tu, Chen Li, Peter H. F. Ng, and Qing Li. 2025. HiBench: Benchmark- ing LLMs Capability on Hierarchical Structure Reasoning. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discove...

    2025 doi

  14. [23]

    Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W Lee, Sangdoo Yun, and Hyun Oh Song. 2025. Kvzip: Query-agnostic kv cache compression with context reconstruction.Advances in Neural Information Processing Systems38 (2025), 167563–167591. https://proceedings.neurips.cc/paper_files/...

    2025

  15. [24]

    Adrian Łańcucki, Konrad Staniszewski, Piotr Nawrot, and Edoardo Maria Ponti. 2025. Inference-time hyper-scaling with kv cache compres- sion.Advances in Neural Information Processing Systems38 (2025), 9365–9397. https://proceedings.neurips.cc/paper_files/paper/2025/hash/ 0d781f...

    2025

  16. [25]

    Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. 2024. A Survey on Large Lan- guage Model Acceleration based on KV Cache Management.arXiv preprint arXiv:2412.19442(2024)

    2024 arXiv

  17. [26]

    Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, Pengsheng Guo, Foroozan Karimzadeh, Colorado Reed, Chong Wang, and Chuang Gan. 2025. CommVQ: Commutative Vector Quantization for KV Cache Compression. InProceedings of the 42nd International C...

    2025

  18. [27]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation.Advances in Neural Informa- tion Processing Systems37 (2024), 22947–22970

    2024

  19. [28]

    Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, and Lili Qiu

    Yucheng Li, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chen- gruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, and Lili Qiu. 2025. SCBench: A KV Cache-Centric Analy- sis of Long-Context Methods. InInternational Conference on Learning Representation...

    2025

  20. [29]

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024. MiniCache: KV Cache Compression in Depth Dimension for Large Language Models.Advances in Neural Information Processing Systems37 (2024). https://doi.org/10.52202/079017-4443

    2024 doi

  21. [30]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts.Transactions of the Association for Computational Linguistics 12 (2024), 157–173. https://doi.org/10.116...

    2024 doi

  22. [31]

    Xiang Liu, Zhenheng Tang, Peijie Dong, Zeyu Li, Yue Liu, Bo Li, Xuming Hu, and Xiaowen Chu. 2025. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference.Advances in Neural Information Processing Systems38 (2025). https://proceedings.neurips....

    2025

  23. [32]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. 2024. CacheGen: KV Cache Compression and Streaming for Fast Large Language Mod...

    2024

  24. [33]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploit- ing the persistence of importance hypothesis for llm kv cache compression at test time.Advances in Neural Information ...

    2023

  25. [34]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. InInternational Conference on Machine Learning. PMLR, 32332–32344

    2024

  26. [35]

    Junlin Mu, Hantao Huang, Jihang Zhang, Minghui Yu, Tao Wang, and Yidong Li. 2025. SALS: Sparse Attention in Latent Space for KV Cache Compression. InAdvances in Neural Information Processing Sys- tems, Vol. 38. https://proceedings.neurips.cc/paper_files/paper/2025/hash/ 00a0eb...

    2025

  27. [36]

    Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, and Edoardo M Ponti. 2024. Dynamic memory compression: retrofitting LLMs for ac- celerated inference. InProceedings of the 41st International Conference on Machine Learning. 37396–37412

    2024

  28. [37]

    Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. 2024. Transformers are multi-state rnns. InProceedings of the 2024 Conference on Em- pirical Methods in Natural Language Processing. 18724–18741

    2024

  29. [38]

    Morse, Raghavv Goel, Mingu Lee, and Chris Lott

    Junyoung Park, Dalton Jones, Matthew J. Morse, Raghavv Goel, Mingu Lee, and Chris Lott. 2025. KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments. https: //doi.org/10.48550/arXiv.2504.15364 arXiv:2504.15364 [cs.AI]

    2025 doi

  30. [39]

    Chen Qiu, Guozhong Li, Cristian McGee, Aritra Dutta, and Panos Kalnis. 2026. ART: Attention Run-time Termination for Efficient Large Language Model De- coding. https://doi.org/10.48550/arXiv.2606.00024 arXiv:2606.00024 [cs.CL]

    2026 doi arXiv

  31. [40]

    Qwen Team. 2025. Qwen3 Technical Report.arXiv preprint arXiv:2505.09388 (2025). https://doi.org/10.48550/arXiv.2505.09388

    2025 doi arXiv

  32. [41]

    Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, and Hai Zhao. 2024. Keep the Cost Down: A Review on Methods to Optimize LLM’s KV-Cache Consumption. https://doi.org/10.48550/arXiv.2407.18003 arXiv:2407.18003 [cs.CL]

    2024 doi arXiv

  33. [42]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. QUEST: query-aware sparsity for efficient long-context LLM inference. InProceedings of the 41st International Conference on Machine Learning. 47901– 47911

    2024

  34. [43]

    Yuxin Wang, Yuhan Chen, Zeyu Li, Xueze Kang, Yuchu Fang, Yeju Zhou, Yang Zheng, Zhenheng Tang, Xin He, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. 2025. BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems. InProceedings of the 31st ACM ...

    2025

  35. [44]

    Jialong Wu, Zhenglin Wang, Linhai Zhang, Yilong Lai, Yulan He, and Deyu Zhou

  36. [45]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. InInternational Con- ference on Learning Representations

    2024

  37. [46]

    Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. 2025. ThinK: Thinner Key Cache by Query-Driven Pruning. InInternational Conference on Learn- ing Representations. https://proceedings.iclr.cc/paper_files/paper/202...

    2025

  38. [47]

    InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Scope: Optimizing key-value cache compression in long-context generation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 10775–10790

  39. [48]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Langua...

    2018

  40. [49]

    Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, and Xia Hu. 2024. KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches...

    2024 doi

  41. [50]

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024. No Token Left Behind: Reli- able KV Cache Compression via Importance-Aware Mixed Precision Quantization. arXiv:2402.18096 [cs.LG] https://arxiv.org/abs/2402.18096

    2024 arXiv

  42. [51]

    Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. 2024. KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization.Advances in Neural Information Processing Systems37 (2024). https://doi.org/10.52202/079017-0109

    2024 doi

  43. [52]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing...

    2023

  44. [53]

    Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. 2024. PQCache: Product Quantization-based KVCache for Long Context LLM Inference. arXiv:2407.12820 [cs.CL] https: //arxiv.org/abs/2407.12820

    2024 arXiv

  45. [2024]

    https://doi.org/10.52202/079017-2181

    ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification.Advances in Neural Information Processing Systems37 (2024). https://doi.org/10.52202/079017-2181

    2024 doi

  46. [2025]

    InInternational Conference on Learning Representations

    Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning. InInternational Conference on Learning Representations. https://openreview.net/forum?id=FJFVmeXusW

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.