VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

Ali Jannesari; Dhawal Shah; Shahrzad Esmat

REVIEW 3 major objections 5 minor 28 references

VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VarRate allocates each token a rank budget proportional to attention salience, compressing the KV cache to 20% of its size while staying within a point of the uncompressed model on long-context tasks, and degrading far less than token evict

desk verdict VarRate is a genuinely new, carefully evaluated training-free KV codec that allocates rank by salience, and it should be reviewed — but only after the authors ship code and raw result files to verify Table 1. read the letter →

arxiv 2607.15498 v1 pith:XAKTI477 submitted 2026-07-16 cs.CL cs.LG

Shahrzad Esmat , Dhawal Shah , Ali Jannesari This is my paper · ORCID

classification cs.CLcs.LG

keywords KVcachecompressionlow-rankapproximationattentionsaliencewater-fillingallocationquery-agnosticreuselong-contextLLMtraining-freevariable-ratecoding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the two dominant training-free KV cache compression strategies fail for complementary reasons: token eviction is irreversible, so a stale importance signal destroys accuracy when a compressed cache is reused across queries, while uniform low-rank coding wastes rank budget on tokens that do not matter. It proposes that rank should be allocated, not evicted, and presents VarRate, which gives each token a variable low-rank budget set by its attention salience through water-filling, keeping every token above a nonzero floor so none is ever dropped. At a 20% KV budget on LongBench, VarRate stays within 0.8 points of the uncompressed model on two model families, beats its uniform-rank ablation by 2.22 and 17.50 points, and degrades by only 3.5–5.5 points under query-agnostic reuse where query-aware selection collapses by 11–15 points. The paper also reports that VarRate matches the accuracy of a purpose-built query-agnostic codec in three of four settings, at roughly one-eighth of that method's compression cost.

What carries the argument

The load-bearing object is the per-token rank assignment r_t = clip(r_min + lambda * s_hat_t, r_min, R), where s_hat_t is a SnapKV-style attention-salience score normalized within the coded set, and lambda is set by water-filling so the ranks sum to exactly the same total budget as a uniform-rank codec. Ranks are realized as nested truncations of a single projection onto a per-layer PCA basis of pre-RoPE residuals, so a token's budget can be raised or lowered without re-encoding, and every column retains at least r_min coefficients. Strided anchors and the recent window are stored exactly; other tokens are coded as residuals to their nearest anchors, and keys are re-rotated to post-RoPE afte

What would settle it

Compress the same document twice at a fixed total rank budget, once allocating high rank to the highest-attention tokens and once to the lowest-attention tokens, and compare query-agnostic accuracy across a broad task suite. If the reverse allocation performs equally well, the attention-proportional-error premise that carries VarRate is false; if salience allocation wins consistently, the premise survives. The paper's random-signal ablation (99.5 vs 93.0 on passage retrieval) is a step in this direction, but a full matched-budget reverse-allocation test on all 16 LongBench tasks would settle i

Watch

Extended reading notes

Core claim

VarRate's central claim is that a graded, reversible rank budget—coarsening a misjudged token to a low rank rather than deleting it—makes a cheap, query-aware signal survivable under cache reuse. Each token's key and value are joined in pre-RoPE space, coded as a residual to nearby anchor tokens, and projected onto a shared SVD basis; the per-token rank is set by water-filling the total rank budget in proportion to the token's attention salience from a recent window, floored at r_min > 0. With this allocation, VarRate reports LongBench accuracy within 0.8 points of the uncompressed model at a 5x memory reduction on both Llama-3.1-8B and Qwen2.5-7B, strictly dominates the published low-rank c

Load-bearing premise

The allocation rationale rests on the premise that a reconstruction error in a heavily attended token distorts the attention output far more than the same error in a diffuse token; if reconstruction error does not scale with attention weight, salience-guided rank allocation loses its theoretical grounding and the empirical advantage could shrink or become dataset-specific.

Editorial extensions

If this is right

If correct, VarRate is the first training-free method to set per-token rank from a query-salience signal, filling the gap between trained adaptive-rank codecs and fixed-rank ones.
Query-agnostic reuse (compress once, serve many queries) no longer forces query-agnostic scoring: a cheap query-aware signal can be made survivable by coarsening instead of eviction.
The codec composes with quantization of its coefficients, reaching roughly 9% of the cache at 3-bit coefficients with accuracy within noise, showing that low-rank rank allocation and precision reduction are orthogonal axes.
At aggressive budgets below roughly 10% kept, the shared basis hits a viability floor and can collapse to the level of uniform rank on some models, so the claimed robustness holds in the stated keep-15–25% operating range, not universally.
The 20% budget result carries to a third model family and to 128K context in needle-in-a-haystack, with VarRate degrading far less than selection under reuse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The design principle 'make a cheap stale signal survivable by grading, not discarding' could be carried to other irreversible decisions in LLM inference—attention-head pruning, token dropping in speculative decoding, or early-exit—though VarRate does not test those settings.
The paper's own pooled salience–rank correlation is near zero (~0.05–0.06) while within-pass ablations show the signal matters; this suggests the signal's power comes from within-pass monotonicity, and a within-pass rank correlation or a matched-marginal random-rank control on the full 16-task suite would sharpen exactly what the salience signal contributes.
The shared per-layer PCA basis is calibrated offline on a handful of unlabeled contexts; if deployment inputs drift far from that calibration, the basis may need periodic recalibration—VarRate is training-free but not distribution-free, a property the paper does not explore.
The unexplained 128K recovery in needle-in-a-haystack, where both VarRate and the eviction baseline return to 100% after dipping at 64K, points to context-length-dependent failure modes that deserve a dedicated study, since the paper reports the effect without a confirmed mechanism.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

VarRate is a genuinely new approach to KV compression: no training, per-token rank allocated by query salience via water-filling over a shared PCA basis, with a nonzero floor so no token is evicted. The idea is simple, and the execution is unusually careful for this literature. The paper actually earns most of its claims. The paired bootstrap CIs and exact sign tests back the headline deltas, the random-signal and fixed-split ablations show the salience signal is doing real work, and the calibration-robustness and third-model checks are more than this subfield usually offers. They also disclose their own data-integrity bug in App. A.9, which says something good about the authors' standards.

The biggest problem is exactly what the reader flagged: the headline Table 1 was once corrupted in the authors' favor, and no code or per-run result files are shipped. The corrected table is only certified by the authors, so a third party cannot independently verify the numbers that support the central "strongest matched-memory compressor" claim. This is not a reason to reject — the narrower experiments were unaffected and the self-disclosure is commendable — but it blocks independent confirmation. The fix is standard: ship code, result files, and a commit hash.

There is also a numerical mismatch between the abstract's "3.5–5.5 points" and Table 2's deltas (Qwen loses 6.42 vs. the Full control). The qualitative robustness claim survives, but it does compound the need for raw results.

On the theoretical side, the attention-proportional-error premise in Section 3.1 is plausible but not directly proven. The empirical ablations already show salience matters, so I read this as less load-bearing than the reader's weakest-assumption note suggests. The unexplained 64K NIAH dip is real but narrow, and the 128K recovery appears genuine — a curiosity, not a red flag.

Who this is for: anyone working on long-context LLM serving, KV cache compression, or training-free efficiency. The paper is worth a serious referee, not a desk reject: the method is novel, the empirical discipline is above average, and the limitations are honestly stated. Send it to peer review, with code and raw results as a condition.

Referee Report

3 major / 5 minor

Summary. VarRate is a training-free low-rank KV cache codec. It keeps every token, represents non-anchor tokens as residuals to anchor means, and allocates a per-token rank in proportion to a SnapKV-style salience signal through water-filling under a fixed memory budget, with a nonzero floor. The paper reports LongBench results at a matched 20% KV budget: within about 0.8 points of the uncompressed model on Llama-3.1-8B and Qwen2.5-7B, significantly better than a uniform-rank ablation, more robust than eviction methods under query-agnostic reuse, and close to KVzip's accuracy at roughly one-eighth of the prefill overhead. The theoretical framing is that rank should be allocated by salience rather than evicting tokens or spending uniform rank.

Significance. The proposal is a simple, plausible middle ground between token eviction and uniform low-rank coding. Its main strengths are the budget identity (Eq. 3), which makes memory matching structural rather than post-hoc; the transparent water-filling construction; and a generally well-controlled empirical section: paired bootstrap CIs and exact sign tests, an oracle comparison, negative controls (shuffle, position, random signal), calibration-robustness checks, a third model, and a separate per-example re-test of the KVzip comparison. The authors also disclose and correct an assembler bug in their headline table, which is to their credit. The central methodological contribution is empirically supported even if the attention-proportional-error motivation in §3.1 is heuristic. The main obstacles are external verifiability and several internal numerical inconsistencies.

major comments (3)

[App. A.9, Table 1] The paper self-discloses that an earlier assembler corrupted 18 of 24 reused cells in the headline 16-task table in a direction favorable to VarRate. No repository, per-run JSON files, or commit hash are provided, so the corrected Table 1 is certified only by the authors. The core claims—'within 0.8 points', 'strongest matched-memory compressor', and the +2.22/+17.50 ablation margins—derive from this table. Please release the exact table-assembly script and raw per-model/per-task result files (or a data appendix with all 16×2 cells), and confirm that the corrected cells, e.g., Llama KIVI-2 passage 100.0 and Qwen flat multi-doc 13.9, trace to raw outputs. Without this, independent verification is impossible.
[Abstract; §4.3; Table 2; Table 18] Headline numerical claims are not consistent with the paper's own tables. (a) The abstract says VarRate degrades by only 3.5–5.5 points under reuse, but Table 2 gives Full−VarRate = 3.70 on Llama and 6.42 on Qwen; §4.3 says '3.53 and 5.50', which matches neither. (b) The abstract and introduction say VarRate is 'within a point' of KVzip, but Table 18's pooled paired difference is −1.18 [−2.20, −0.16] (significant) and Table 2's four-cell mean gap is 1.88. Please reconcile these numbers or soften the claims accordingly.
[Abstract; §4.2; Table 15] The abstract calls VarRate 'the strongest matched-memory compressor' based on the two-model mean in Table 1, but the significance analysis in Table 15 is per-model paired over 16 tasks. On Llama, VarRate is nominally fourth and is tied with PyramidKV/SnapKV; only on Qwen does it win significantly. No paired test over the 32 model-task cells is reported, so the 'strongest' claim is not statistically supported as stated. Either add a cross-model paired bootstrap/sign test or rephrase to 'best average, never significantly worse per-model'.

minor comments (5)

[App. A.19] The paper itself flags that the pooled salience-rank correlation is ~0.05–0.06 and calls it a likely pooling artifact, recommending a within-pass correlation as the correct follow-up. Because this is the most direct quantitative check of the salience-to-rank link, please report a within-pass rank correlation (or an equivalent per-example measure) rather than leaving the low pooled number unexplained.
[Algorithm 1 / Eq. (5)] The water-filling update in Algorithm 1 lines 8–11 is not obviously identical to Eq. (5); a one-sentence proof of equivalence would make the implementation and the equation easier to check.
[§4.1 / hyperparameters] Several fixed hyperparameters (rmin=16, R=1024, c=4, w=w_obs=64) have no sensitivity analysis beyond stride and floor (Apps. A.12/A.19). A brief sensitivity check for c and w would address concerns about tuning on the evaluation suite.
[App. A.9] The sentence 'The corrupted table happened to make VarRate look like the top compressor' is confusing because the corrected Table 1 still gives VarRate the best two-model mean. Please clarify exactly which rankings were corrupted and how the corrected table differs.
[§4.4 / Table 3] The decode-rate rows for 'masker' methods are identified as harness artifacts; consider placing that caveat in the Table 3 caption itself rather than only in the text, to avoid misreading.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VarRate's allocation is defined by a budget identity and the headline claims are measured against external baselines and ablations.

full rationale

The paper's derivation chain is self-contained. Section 3.1's 'Why allocate rank' is a stated premise, not a derived result; Eq. (5) and Algorithm 1 define r_t and fix lambda via the exact budget identity sum r_t = B, so no fitted constant targets the reported accuracies. The PCA basis is calibrated offline on six unlabeled contexts with App. A.8 showing <=1.8-point spread across re-calibrations, so it is not tuned to the eval tasks. The central comparisons (Table 1, Table 2, ablations in Table 4) are measured against external baselines or against 'flat', a same-codec uniform-rank ablation; beating flat is an empirical ablation, not a consequence of definition. The signal/water-filling ablations and shuffled/position controls (Apps. A.17-A.18) further separate the two components. No load-bearing self-citation appears: the reference list contains no works by the present authors. The App. A.9 data-integrity correction and App. A.21's unexplained 128K recovery are transparency/reproducibility concerns, not circularity; they do not change the fact that the claimed predictions are not equivalent to the inputs by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new particles, forces, or entities are introduced; the method composes existing artifacts (a per-layer PCA basis, SnapKV salience, residual coding). The free-parameter count is moderate, and every hand-chosen knob except R and c receives either a dedicated ablation or a slackness check, which is better than the field norm.

free parameters (6)

rank floor r_min = 16 = 16
Hand-set in Section 4.1; controls minimum rank per token. Ablations (App. A.19, Table 21) show it never binds (0.0% of tokens at floor; r_min=0 changes accuracy by <1 point), so it is a slack hyperparameter, not an active mechanism.
basis rank R = 1024 = 1024
Cap on per-token rank and SVD truncation rank of the shared basis. Not swept in the main text; effective per-token rank is redistributed below it.
anchor stride Delta = 16 = 16
Strided anchors stored exact. Ablation (App. A.12) shows real trade-offs (stride 64: +18.96 points at keep-10%, -0.71 on AVG-16 at keep-20%); the default governs the headline numbers.
nearest anchors c = 4 = 4
Number of anchors averaged for the residual reference (Eq. 2). No dedicated ablation is reported in the manuscript.
windows w = w_obs = 64 = 64
Exact recent-window length and SnapKV salience observation window (Eq. 4). No dedicated ablation is reported; the salience window is inherited from SnapKV's convention.
target budget kappa = 0.20 = 0.20
Headline operating point. The paper sweeps 0.08-0.70 (Fig. 3, App. A.20) and shows regime-dependent viability, so this is an operating choice, not a fitted constant.

assumptions (5)

domain assumption Coding-error impact scales with attention weight: 'a coding error in a heavily attended token distorts the output far more than the same error in a diffuse one'
Section 3.1, 'Why allocate rank'; the theoretical justification for salience-proportional allocation. Tested only indirectly (oracle 100.0 vs salience 99.0; App. A.17), and the pooled salience-rank correlation is ~0.05 (App. A.19).
domain assumption A per-layer PCA basis calibrated offline on six unlabeled LongBench contexts generalizes to other contexts, tasks, and lengths up to 128K
Section 4.1 and Section 3.1; verified by App. A.8 (spread across three basis builds) and the 128K NIAH sweep, but the basis is drawn from the same benchmark family as the evaluation.
domain assumption Pre-RoPE residual coding is more compressible than coding post-RoPE keys
Section 3.1 (Eq. 1 and surrounding text); RoPE's per-position rotation inflates numerical rank; prior low-rank codecs (Palu, DeepSeek-V2) operate pre-RoPE, cited as support.
domain assumption SnapKV observation-window attention is a serviceable salience proxy even when the question is absent at compression time
Section 3.2, Eq. 4. The authors' own framing is that the signal goes stale under reuse and the method makes staleness survivable; App. A.17 shows content alignment matters and position-only is insufficient.
domain assumption LongBench (16 tasks, n=200 per task) is a representative proxy for long-context inference quality
Section 4.1; standard benchmark in the field; cross-checked with RULER (n=40, within noise) and NIAH to 128K.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs." pith.science (2026). https://pith.science/paper/XAKTI477

@misc{pith2026260715498,
  author       = {Pith},
  title        = {Pith review of: VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XAKTI477}},
  note         = {Machine review of arXiv:2607.15498}
}

read the original abstract

The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference. Two leading training-free families are both structurally limited: token-selection methods (SnapKV, Ada-KV) score importance from an observation window and evict low-scoring tokens, but eviction is irreversible -- so when the importance signal degrades under query-agnostic reuse, accuracy collapses by 11-15 points; uniform low-rank coding keeps every token but spends equal rank everywhere, wasting budget. We observe that both failures share one cure: rank should be allocated, not evicted. We present VarRate, a training-free KV codec that assigns each token a variable low-rank budget by its query salience, keeping every token at a nonzero rank. Comparable adaptive-rank codecs reach this allocation only through training; VarRate requires none. Because no token is dropped, it degrades by only 3.5-5.5 points where query-aware selection collapses. At a matched 20% budget on LongBench (16 tasks), VarRate stays within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B. Averaged over the two, it is the strongest matched-memory compressor. It significantly beats its uniform-rank ablation on both models. Against KVzip, a method purpose-built for query-agnostic reuse, it is accuracy-equivalent in three of four settings and within a point overall, at about one-eighth the prefill overhead.

Figures

Figures reproduced from arXiv: 2607.15498 by the authors.

**Figure 1.** VarRate allocates rank by salience—it neither evicts tokens nor spreads rank uniformly. Each token is a bar whose height is its allocated rank, given below it. (a) Token selection evicts the low-scoring tokens; the dropped KV is gone for good, so a mistaken decision cannot be undone. (b) Uniform low-rank coding keeps every token but spends the same rank on all of them—wasted on the easy tokens, starved on the critic… view at source ↗

**Figure 2.** VarRate, one layer. (1) Keys are un-rotated to pre-RoPE and joined with the values into one per-token vector zt, so a single basis serves every head. (2) Strided anchors and the recent window are stored exactly; every other token is coded as a residual et to the mean of its c nearest anchors. (3) SnapKV salience sˆt is water-filled into a per-token rank rt at the flat codec’s budget. Each bar P is the allocation: a … view at source ↗

**Figure 3.** Budget sweep against the published low-rank codec (Llama-3.1-8B). Palu was built for the keep- [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Accuracy under query-agnostic reuse against com [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Worked example of the per-token allocation on a 64K needle-in-a-haystack instance. Each token’s salience sˆt (its SnapKV observation-window score) sets a per-token rank rt by water-filling the shared budget: a low-salience boring token is coarsened to the floor rmin, w…

**Figure 6.** Figure 6: Why the signal has to be content-aligned, not just graded. One token (⋆, outlined) plays a query-critical “needle” placed in the document’s boring middle, away from both the attention-sink start and the query-proximate end. (a) Flat coding does not look at the needle a…

**Figure 7.** Figure 7: Where does the rank budget actually go? Mean allocated rank by document-position decile, pooled over 300 sampled (layer, example) traces on gov_report at KR20 (0 = document start, 9 = the end, where the query sits after the query-aware transform appends it). Three regi…

**Figure 8.** Figure 8: Query-agnostic needle-in-a-haystack, Llama-3.1- [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 3 linked inside Pith

[1]

Bai, Y.; Lv, X.; Zhang, J.; Lyu, H.; Tang, J.; Huang, Z.; Du, Z.; Liu, X.; Zeng, A.; Hou, L.; Li, J.; and Tang, J. 2024. LongBench : A Bilingual, Multitask Benchmark for Long Context Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)

2024
[2]

Bhatnagar, P.; Moradifirouzabadi, A.; Yang, S.-H.; Lee, S.; Choi, J.; and Kang, M. 2026. STAR-KV : Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control. In International Conference on Machine Learning (ICML)

2026
[3]

J.; Deems, S.; Furlani, T

Boerner, T. J.; Deems, S.; Furlani, T. R.; Knuth, S. L.; and Towns, J. 2023. ACCESS : Advancing Innovation: NSF 's Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support. In Practice and Experience in Advanced Research Computing (PEARC), 173--176

2023
[4]

Cai, Z.; Zhang, Y.; Gao, B.; Liu, Y.; Li, Y.; Liu, T.; Lu, K.; Xiong, W.; Dong, Y.; Hu, J.; and Xiao, W. 2025. PyramidKV : Dynamic KV Cache Compression based on Pyramidal Information Funneling. In Conference on Language Modeling (COLM)

2025
[5]

S.; and Wu, K.-C

Chang, C.-C.; Lin, W.-C.; Lin, C.-Y.; Chen, C.-Y.; Hu, Y.-F.; Wang, P.-S.; Huang, N.-C.; Ceze, L.; Abdelfattah, M. S.; and Wu, K.-C. 2025. Palu : Compressing KV -Cache with Low-Rank Projection. In International Conference on Learning Representations (ICLR)

2025
[6]

DeepSeek-AI . 2024. DeepSeek-V2 : A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv preprint arXiv:2405.04434

arXiv 2024
[7]

Devoto, A.; Jeblick, M.; and J\'egou, S. 2025. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution. arXiv preprint arXiv:2510.00636

arXiv 2025
[8]

Feng, Y.; Lv, J.; Cao, Y.; Xie, X.; and Zhou, S. K. 2025. Ada-KV : Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. In Advances in Neural Information Processing Systems (NeurIPS)

2025

Show all 28 references

[9]

Hao, J.; Huang, Q.; Wang, Y.; Zhang, M.; and Yu, J. 2026. DeltaKV : Residual-Based KV Cache Compression via Long-Range Similarity. arXiv preprint arXiv:2602.08005

2026
[10]

W.; Shao, Y

Hooper, C.; Kim, S.; Mohammadzadeh, H.; Mahoney, M. W.; Shao, Y. S.; Keutzer, K.; and Gholami, A. 2024. KVQuant : Towards 10 Million Context Length LLM Inference with KV Cache Quantization. In Advances in Neural Information Processing Systems (NeurIPS)

2024
[11]

Hsieh, C.-P.; Sun, S.; Kriman, S.; Acharya, S.; Rekesh, D.; Jia, F.; Zhang, Y.; and Ginsburg, B. 2024. RULER : What's the Real Context Size of Your Long-Context Language Models? In Conference on Language Modeling (COLM)

2024
[12]

W.; Yun, S.; and Song, H

Kim, J.-H.; Kim, J.; Kwon, S.; Lee, J. W.; Yun, S.; and Song, H. O. 2025. KVzip : Query-Agnostic KV Cache Compression with Context Reconstruction. In Advances in Neural Information Processing Systems (NeurIPS)

2025
[13]

Li, Y.; Huang, Y.; Yang, B.; Venkitesh, B.; Locatelli, A.; Ye, H.; Cai, T.; Lewis, P.; and Chen, D. 2024. SnapKV : LLM Knows What You are Looking for Before Generation. In Advances in Neural Information Processing Systems (NeurIPS)

2024
[14]

Lin, B.; Zeng, Z.; Xiao, Z.; Kou, S.; Hou, T.; Gao, X.; Zhang, H.; and Deng, Z. 2025. MatryoshkaKV : Adaptive KV Compression via Trainable Orthogonal Projection. In International Conference on Learning Representations (ICLR)

2025
[15]

Liu, Z.; Yuan, J.; Jin, H.; Zhong, S.; Xu, Z.; Braverman, V.; Chen, B.; and Hu, X. 2024. KIVI : A Tuning-Free Asymmetric 2bit Quantization for KV Cache. In International Conference on Machine Learning (ICML)

2024
[16]

Long, L.; Yang, R.; Huang, Y.; Hui, D.; Zhou, A.; and Yang, J. 2026. SlimInfer : Accelerating Long-Context LLM Inference via Dynamic Token Pruning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)

2026
[17]

Lu, L.; Qiu, K.; Zhou, J.; Kai, J.; Zhang, H.; Wang, H.; Leng, J.; He, Z.; and Lin, Z. 2026. One Size Does Not Fit All: Token-Wise Adaptive Compression for KV Cache. arXiv preprint arXiv:2603.04411

2026
[18]

Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2017. Pointer Sentinel Mixture Models. In International Conference on Learning Representations (ICLR)

2017
[19]

Saxena, U.; Saha, G.; Choudhary, S.; and Roy, K. 2024. Eigen Attention: Attention in Low-Rank Space for KV Cache Compression. In Findings of the Association for Computational Linguistics: EMNLP

2024
[20]

Tian, Y.; Wang, Z.; Peng, Y.; Yuan, A.; Wang, Z.; Yi, B.; Liu, X.; Cui, Y.; and Yang, T. 2026. KeepKV : Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)

2026
[21]

N.; Kaiser, L.; and Polosukhin, I

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is All You Need. In Advances in Neural Information Processing Systems (NeurIPS)

2017
[22]

Xiao, G.; Tian, Y.; Chen, B.; Han, S.; and Lewis, M. 2024. Efficient Streaming Language Models with Attention Sinks. In International Conference on Learning Representations (ICLR)

2024
[23]

Yuan, J.; Liu, H.; Zhong, S.; Chuang, Y.-N.; Li, S.; Wang, G.; Le, D.; Jin, H.; Chaudhary, V.; Xu, Z.; Liu, Z.; and Hu, X. 2024. KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches. In Findings of the Association ...

2024
[24]

Zandieh, A.; Daliri, M.; and Han, I. 2025. QJL : 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)

2025
[25]

Zhang, R.; Wang, K.; Liu, L.; Wang, S.; Cheng, H.; Zhang, C.; and Shen, Y. 2024 a . LoRC : Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy. arXiv preprint arXiv:2410.03111

2024 arXiv
[26]

Zhang, Y.; Du, Y.; Luo, G.; Zhong, Y.; Zhang, Z.; Liu, S.; and Ji, R. 2024 b . CaM : Cache Merging for Memory-efficient LLMs Inference. In International Conference on Machine Learning (ICML)

2024
[27]

Zhang, Z.; Sheng, Y.; Zhou, T.; Chen, T.; Zheng, L.; Cai, R.; Song, Z.; Tian, Y.; R\'e, C.; Barrett, C.; Wang, Z.; and Chen, B. 2023. H2O : Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS)

2023
[28]

H.; Amiri, M

Zhu, Y.; Yang, D. H.; Amiri, M. M.; Murugesan, K.; Pedapati, T.; and Chen, P.-Y. 2025. OjaKV : Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule. arXiv preprint arXiv:2509.21623

2025 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Bai, Y.; Lv, X.; Zhang, J.; Lyu, H.; Tang, J.; Huang, Z.; Du, Z.; Liu, X.; Zeng, A.; Hou, L.; Li, J.; and Tang, J. 2024. LongBench : A Bilingual, Multitask Benchmark for Long Context Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)

2024

[2] [2]

Bhatnagar, P.; Moradifirouzabadi, A.; Yang, S.-H.; Lee, S.; Choi, J.; and Kang, M. 2026. STAR-KV : Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control. In International Conference on Machine Learning (ICML)

2026

[3] [3]

J.; Deems, S.; Furlani, T

Boerner, T. J.; Deems, S.; Furlani, T. R.; Knuth, S. L.; and Towns, J. 2023. ACCESS : Advancing Innovation: NSF 's Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support. In Practice and Experience in Advanced Research Computing (PEARC), 173--176

2023

[4] [4]

Cai, Z.; Zhang, Y.; Gao, B.; Liu, Y.; Li, Y.; Liu, T.; Lu, K.; Xiong, W.; Dong, Y.; Hu, J.; and Xiao, W. 2025. PyramidKV : Dynamic KV Cache Compression based on Pyramidal Information Funneling. In Conference on Language Modeling (COLM)

2025

[5] [5]

S.; and Wu, K.-C

Chang, C.-C.; Lin, W.-C.; Lin, C.-Y.; Chen, C.-Y.; Hu, Y.-F.; Wang, P.-S.; Huang, N.-C.; Ceze, L.; Abdelfattah, M. S.; and Wu, K.-C. 2025. Palu : Compressing KV -Cache with Low-Rank Projection. In International Conference on Learning Representations (ICLR)

2025

[6] [6]

DeepSeek-AI . 2024. DeepSeek-V2 : A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv preprint arXiv:2405.04434

arXiv 2024

[7] [7]

Devoto, A.; Jeblick, M.; and J\'egou, S. 2025. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution. arXiv preprint arXiv:2510.00636

arXiv 2025

[8] [8]

Feng, Y.; Lv, J.; Cao, Y.; Xie, X.; and Zhou, S. K. 2025. Ada-KV : Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. In Advances in Neural Information Processing Systems (NeurIPS)

2025