AnchorKV: Anchor-Residual KV Cache Compression

Assaf Schuster; Malik Khalaf; Nitzan Hodos; Yara Shamshoum; Yuval Sieradzki

REVIEW 1 major objections 5 minor 46 references

AnchorKV: Anchor-Residual KV Cache Compression

T0 review · 1 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a KV cache can be compressed 20x without removing any token, retaining 93-99% of uncompressed accuracy across three model scales.

desk verdict A genuinely new anchor-residual KV compression scheme with strong empirical evidence, but the 'retains every token' claim rests on an unverified worst-case alignment premise that the authors themselves acknowledge. read the letter →

arxiv 2608.02901 v1 pith:UWCSRGJM submitted 2026-08-03 cs.LG cs.CL

Malik Khalaf , Yara Shamshoum , Nitzan Hodos , Yuval Sieradzki , Assaf Schuster This is my paper · ORCID

classification cs.LGcs.CL

keywords KVcachecompressionanchor-residualrepresentationlong-contextLLMinferenceattention-output-awareallocationtokenretentionquantizedresidualsRULERLongBench

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AnchorKV claims that the key-value cache of a large language model can be compressed 20x while keeping every token in the attention computation. The cache is recast as a small set of exact anchor vectors plus, for each other token, a scaled projection onto its nearest anchor, with quantized residuals spent only where reconstruction error would most change the attention output. Across three model scales and three long-context benchmarks, the method retains 93-99% of the full-cache score at 20x compression, and at 20x it scores higher than every eviction baseline at 10x. A sympathetic reader would care because this breaks the trade-off eviction methods face: the full context stays reachable, so later queries can still use tokens that prefill-time scoring did not flag.

What carries the argument

The load-bearing object is the anchor-residual representation: a token $x_i$ is stored as an anchor index $a(i)$, a scalar coefficient $\gamma_i = \langle x_i, x_{a(i)}\rangle/\lVert x_{a(i)}\rVert^2$, and optionally a two-bit quantized residual $\tilde{r}_i$, so reconstruction is $\hat{x}_i = \gamma_i x_{a(i)} + \tilde{r}_i$. Anchor selection combines the SnapKV observation-window score with uniform sampling, and residual placement is driven by a first-order estimate of attention-output error that separates key-induced and value-induced channels. All of this is wrapped in a per-layer byte accounting that fixes the retained fraction $\theta$ as the only user knob, and the supporting theorem bounds output error by an attention-weighted value term plus $2 V_{\max}\tanh(\mu)$ with $\mu = \lVert q\rVert \Delta_K/\sqrt{D}$, so the compressed cache is a perturbation of the exact one rather than a truncation.

What would settle it

A concrete test: on a 128K-token prompt, find the key whose cosine similarity to its nearest anchor is lowest and check whether that position receives nonzero attention; if such a worst-case key can shift the softmax and the residual budget does not cover it, the 20x accuracy on that prompt should drop measurably. Measuring $\max_t \lVert K_t - \hat K_t\rVert$ across layers and comparing it to the bound of Theorem 1 would show directly whether the guarantee is tight.

Watch

Extended reading notes

Core claim

The paper's central claim is that aggressive KV-cache compression and full-context availability are not in conflict: AnchorKV shrinks the cache 20x while keeping every token inside the softmax. Each KV head stores a small set of anchors exactly and expresses every other token as a scaled copy of its nearest anchor; a byte budget then buys two-bit residuals for the tokens whose projection error would most perturb the attention output. On Llama-3.1-70B this retains 99.3% of the uncompressed RULER score at 20x against 86.8% for the strongest eviction baseline, and at 20x AnchorKV outscores every eviction baseline at 10x on all nine model-benchmark settings. The same pattern holds on LongBench real documents, with 94.1% retention at 8B and 98.4% at 70B.

Load-bearing premise

The method's error bound is governed by the worst key reconstruction over all positions, and a fixed residual budget cannot repair a worst-case key, so the whole accuracy story rests on the empirical premise that every attended key projects closely enough to some anchor.

Editorial extensions

If this is right

At the same byte budget, 20x AnchorKV scores above every eviction baseline at 10x, so the same accuracy is reached with half the cache memory.
At Llama-3.1-70B the method retains 99.3% of the uncompressed RULER score at 20x, and retention improves with scale, the regime where the cache is most expensive.
Because no position is removed from the softmax, the compressed cache behaves as a perturbation of the exact cache; tasks requiring retrieval against distractors and whole-context aggregation retain 60-98% of the full score while eviction baselines fall below 16%.
Decode peak memory falls about 19x at a 20x target, and the concurrency ceiling roughly doubles, so serving capacity improves in addition to accuracy.
The compression runs once at the end of prefill on a frozen model, so no training or calibration corpus is needed and the method is a drop-in inference-time change.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because anchor selection already mixes attention scoring with uniform sampling, one could replace uniform anchors with a coverage-optimized dictionary, such as k-means or furthest-point sampling, and likely improve the worst-case key-error term that Theorem 1 leaves uncontrolled.
The attention-output utility of Eq. 6 is computed once at prefill; a testable extension is to re-score residuals online as decoding queries arrive, since the assumption that prefill queries anticipate decode queries is the same one eviction methods make.
If the method is combined with quantization of anchors or coefficients, the 20x figure is not a floor; the anchor-residual representation could push further before accuracy falls.
The key-error bound suggests a diagnostic: monitoring the maximum key reconstruction error per layer, rather than the median cosine, would flag prompts where the no-token-removed guarantee is at risk.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

AnchorKV is one of the more interesting KV-compression papers I've read recently. The core idea—store a few anchor vectors exactly, represent every other token as an anchor index plus a scalar coefficient, then spend a fixed byte budget on quantized residuals for the tokens whose approximation most hurts attention output—is a genuine new combination. It sits between eviction and quantization, and the empirical package is unusually thorough: three model scales, RULER/LongBench/NIAH, matched byte budgets against SnapKV/PyramidKV/AdaKV/TurboQuant, per-task tables, ablations, and real efficiency measurements on an A100. I checked the math the reader flagged: the projection decomposition, pre-RoPE key storage with RoPE distributing over the decomposition, the byte accounting, and the softmax L1 perturbation bound in Theorem 1. All of it checks out. The first-order utility approximation (Eq. 6) is heuristic but sensible, and the ablation shows it beats cheaper ranking criteria.

The soft spots are real but not disqualifying. The headline 99.3% retention at 70B RULER-32K/20x is a single run with no error bars; the margin over baselines is large, but the absolute retention margin could shrink with seeds. The code is promised only on acceptance, so nothing is independently reproducible yet. The anchor budget k was selected on a RULER subset, which is also the main benchmark—mild tuning risk. And the deeper concern: Theorem 1 bounds key-induced error by a worst-case maximum over positions, and the authors explicitly say a fixed residual budget cannot control that maximum. At 20x, only 17.3% of token-sides get residuals, so fidelity for the other 82.7% depends entirely on anchor alignment. They report median alignments, not worst case. The 'retains every token' claim is true in the sense that nothing leaves the softmax, but it doesn't guarantee every token is faithfully represented. The benchmarks suggest it works in practice on tested distributions; the transfer to arbitrary contexts is not established.

Overall: this deserves a serious referee. I'd send it to review with requests for code, multi-seed runs, and a worst-case alignment analysis (e.g., distribution of per-token reconstruction error or max attention-weighted error). It's a solid contribution with an honest limitation statement.

Referee Report

1 major / 5 minor

Summary. AnchorKV proposes an anchor-residual KV cache compression scheme that keeps every token inside the softmax while compressing the cache up to 20x. Each per-head key and value vector is assigned to its nearest anchor, represented as a scalar-scaled anchor projection, and a budgeted subset of tokens additionally receives 2-bit Hadamard-rotated, Lloyd-Max-quantized residuals chosen by an attention-output-aware utility. The user sets one retention knob, and an exact byte accounting (Eq. 13) converts it into a per-layer residual budget. Experiments on Llama-3.1-8B, Mistral-Small-3.1-24B, and Llama-3.1-70B across RULER, LongBench, and passkey retrieval at matched byte budgets report that AnchorKV at 20x outperforms every eviction baseline at 10x, with 99.3% RULER retention on the 70B model; a FlashAttention-style decoder reduces decode peak memory by about 19x. Appendices supply the codec and storage layout, the utility derivation, an output-error bound (Theorem 1), full per-task tables, and efficiency measurements.

Significance. If the empirical results hold, this is a substantial practical contribution: it demonstrates that 20x KV compression is attainable without eviction, using a principled anchor-residual representation, a parameter-free utility estimate under an explicit incoherence model, and unusually careful byte-level accounting. The worked example in Appendix A.1 reproduces the stated 3.39 MB base footprint and 91,056 residuals, and Theorem 1 is proved cleanly. The evaluation is systematic: three model scales, three benchmark families, matched byte budgets, per-task breakdowns, ablations, and direct attention-output fidelity measurements. The paper also honestly acknowledges in Section 3.3 that a fixed residual budget cannot control a worst-case key error, and it discloses the need to reproduce TurboQuant from community implementations. The main weakness is that the headline 'without discarding a single token' guarantee is stronger than the worst-case analysis actually supports, because only median alignment and fidelity are reported rather than the worst-case key reconstruction error that the bound depends on.

major comments (1)

[Section 3.3 / Appendix A.4, Eq. (17)] The key-side error term E_K = 2 V_max tanh(mu) depends on Delta_K = max_t ||K_t - hat K_t||, and the paper explicitly concedes in Section 3.3 that 'a fixed number of residuals cannot control a maximum over positions.' The claim that the compressed cache is a perturbation of the exact one therefore rests entirely on the anchor projection being accurate for every key that decoding attends to. The paper reports the median cosine to the nearest anchor (Figure 7) and the distribution of attention-output cosines (Figure 10), but not the worst case or high quantiles. At 20x, Section E.1 gives 17.3% residual density, so 82.7% of token-sides carry no residual, and for those Delta_K = ||r^K_t||; a single poorly covered but attended key can make mu large and, because softmax is a global function of all logits, shift every attention weight. To support the 'retains every token' claim, please report per-layer and per-head maxima or 99.9th percentiles of ||r^K_t|| (equivalently cosine to nearest anchor) on the evaluation workloads, or explicitly reframe the guarantee as an empirical claim validated on these benchmarks.

minor comments (5)

[Section 4.1 / Appendix B.3] The fixed hyperparameter list omits b_a and b_gamma, even though Eq. (13) and the worked example in Appendix A.1 set them to 2; state their values for all experiments or explain how they are derived from the anchor budget k.
[Appendix B.5 / Figure 2] The TurboQuant baseline is a community reproduction using a different quantization scheme (uniform base plus fp16 outliers) than the original paper's channel split, and the authors report that the verbatim method did not reproduce its published quality; this caveat should appear in the main text and the curve should be labeled as a variant so readers do not mistake it for the published algorithm.
[Section 4.2 / Tables 6-15] No variance estimates or repeated-seed runs are reported; because several headline comparisons rest on margins of a few points, please add standard errors or min/max ranges across seeds.
[Section 4.2 / Figure 9] At 10x the AnchorKV needle-in-a-haystack grid has many cells below 50% at shallow depths, so the wording 'recovers the needle across nearly the whole grid' overstates the 0.80 mean; adjust the description to match the displayed grid.
[Section 4.3] In the sentence 'a cosine cleaves a residual of relative norm sqrt(1-c^2)', 'cleaves' should be 'leaves'; Figure 5 also labels the variant 'Naive 2 bit' and should be capitalized consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; AnchorKV's derivation is self-contained and its headline results are measured against external benchmarks and baselines.

full rationale

AnchorKV's core derivation is not circular. The output-error decomposition (Eq. 5) and the additive utility (Eq. 6) are derived from a first-order softmax expansion under an explicit incoherence model stated in Appendix A.3, and the ranking is validated against random, attention, residual-norm, and cosine alternatives in Figure 5 rather than assumed. Theorem 1 (Appendix A.4) is proved for any key/value reconstruction, with constants (V_max, Delta_K) defined from the exact and reconstructed caches; the bound's reliance on a worst-case key error is explicitly acknowledged in Section 3.3 ('a fixed number of residuals cannot control a maximum over positions'), which makes the coverage premise a robustness/correctness concern, not a circular reduction. The 'retains every token' property is a definitional feature of the representation (every non-anchor token stores an anchor index and a scalar coefficient), not a prediction derived from a fitted quantity. No load-bearing step is justified by self-citation: the reference list contains no works by the present authors, and borrowed machinery (SnapKV observation window, rotation-based quantization) is cited to external prior work. Headline results (99.3% RULER retention at 20x on Llama-3.1-70B, per-task scores, needle-in-haystack grids) are measured against FullKV and external eviction/quantization baselines at matched byte budgets, so the empirical claims are externally falsifiable rather than forced by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

Everything the central claim rests on beyond shipped code and standard math: six fixed hyperparameters (one, k=S/128, was tuned on a small subset of RULER with Llama-3.1-8B and then fixed), four domain assumptions (incoherence of residual directions, observation-window representativeness, Gaussian marginals for the codec, benchmark validity), and two standard-mathematics items (first-order softmax expansion, RoPE orthogonality). No invented entities are postulated to explain the results; the anchor set is an explicit, benchmark-validated component of the method rather than a hidden degree of freedom. The heaviest load rests on the incoherence and observation-window assumptions, both honestly stated in the paper.

free parameters (6)

retained fraction theta = 0.05 (20x target)
User-facing compression knob (Eq. 9); sets the byte budget per layer. Held at 0.05 for all 20x experiments; not fitted to accuracy data but is the method's single tunable.
anchor budget k = S/128
Swept over {S/64, S/128, S/256} on a small subset of RULER subtasks with Llama-3.1-8B, selected as best, then fixed for all models and benchmarks (Appendix B.3). This is the one parameter actually fitted to data, and the tuning set overlaps the headline evaluation set.
scored anchor fraction rho = 0.7
Fraction of non-window anchor slots assigned by attention score (Eq. 14); the remainder is sampled uniformly. Hand-fixed, not swept.
recency window W = 32
Final W positions are always stored exactly and used as observation queries. Hand-fixed; baselines use W=64 via KVPress defaults.
pooling kernel width kappa = 7
Width of the positional pooling kernel in the anchor score (Eq. 14). Hand-fixed, not swept.
codec parameters = 2-bit Lloyd-Max, fixed Hadamard seed, b_a=b_gamma=2 bytes
Residual quantization at two bits per coordinate with a four-level Lloyd-Max codebook for a unit Gaussian source, randomized Hadamard rotation with a shared fixed seed, anchor indices and coefficients at 2 bytes each (Table 1). Design choices, not fitted, but they determine the byte accounting and distortion.

assumptions (6)

domain assumption Incoherence model for residual utilities (Appendix A.3)
Per-token key and value errors are treated as zero-mean and independent against query and value geometry, and key errors independent of value errors, so E||Delta y||^2 decomposes additively per token (Eq. 16). The paper calls it 'an equality in expectation under the model rather than a bound'. If incoherence fails, the residual allocation of Eq. 6 is suboptimal though not catastrophic.
standard math First-order softmax perturbation expansion (Eq. 5)
Delta y is expanded to first order in the logit and value perturbations, dropping second-order terms; the softmax Jacobian identity d alpha_t / d s_u = alpha_t (delta_tu - alpha_u) is standard.
domain assumption Observation-window representativeness (Section 3.3, Eq. 6)
The last W tokens' attention (pooled over GQA query heads) is assumed to predict which positions decoding will need, stated verbatim: 'the estimator assumes that the positions recent queries use are the positions decoding will need.' Shared with SnapKV-style baselines; AnchorKV's defense is that a mispriced position is stored coarsely rather than dropped.
standard math RoPE orthogonality and linearity (Section 3.2)
RoPE is a per-position orthogonal linear map R_t, so R_t K_t = R_t tilde K_t + R_t r^K_t and ||R_t r^K_t|| = ||r^K_t||, justifying pre-RoPE storage and rotation at decode time.
domain assumption Gaussian marginal model for rotated residuals (Appendix A.1)
After the randomized Hadamard rotation, each coordinate of a residual is approximately Gaussian with variance ||r||^2/D, justifying the Lloyd-Max unit-Gaussian codebook and the fixed distortion eta. Approximate for residuals with concentrated energy.
domain assumption Benchmark scores proxy general long-context accuracy
RULER, LongBench, and passkey NIAH are treated as representative of the tasks long-context serving cares about; all comparisons are on these suites.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AnchorKV: Anchor-Residual KV Cache Compression." pith.science (2026). https://pith.science/paper/UWCSRGJM

@misc{pith2026260802901,
  author       = {Pith},
  title        = {Pith review of: AnchorKV: Anchor-Residual KV Cache Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UWCSRGJM}},
  note         = {Machine review of arXiv:2608.02901}
}

abstract

The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by $20\times$ without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.

Figures

Figures reproduced from arXiv: 2608.02901 by the authors.

**Figure 2.** AnchorKV against eviction and quantization baselines at matched byte budgets. Rows are Llama-3.1-8B, Mistral [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Per-task RULER accuracy on Llama-3.1-8B [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (14 more)

**Figure 5.** Figure 5: Ablation Study. Evaluated on Llama-3.1-8B, [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Efficiency Analysis. We profile AnchorKV at a [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Pre-RoPE vs. Post-RoPE Key Projection. We mea [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Needle in a haystack at 5× compression. Each cell is the mean score over the 10 samples at that context length and needle depth. optimizers, or training schedules to report. The only configuration consists of the fixed inference-time hyperparameters we now list. For a…

**Figure 9.** Figure 9: Needle in a haystack at 10× compression. Each cell is the mean score over the 10 samples at that context length and needle depth. 0.6 0.8 1.0 10 −3 10 −2 10 −1 10 0 P(cos≤x) (a) NIAH-S1 0.6 0.8 1.0 (b) FWE attention-output cosine similarity AnchorKV (ours) AdaKV SnapKV…

**Figure 10.** Figure 10: Attention-output fidelity at 20× compression, per (layer, head, sample) cell (Llama-3.1-8B-Instruct, RULER 32K; log y-axis, dashed line at cos = 0.9). Below that threshold AnchorKV leaves 0.2% of cells against 8.4% for the best baseline on NIAH-S1, and 0.3% against 2…

**Figure 11.** Figure 11: Share of each layer’s residual budget taken by each KV head, AnchorKV at [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

**Figure 12.** Figure 12: Prefill Overhead and Amortization. Same setup as Figure 6. (a) Prefill wall clock at [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

**Figure 13.** Figure 13: Llama-3.1-8B-Instruct on RULER (32k). NarrativeQA Qasper MultiFieldQA HotpotQA 2WikiMQA MuSiQue GovReport QMSum MultiNews TREC TriviaQA SAMSum PassageCount PassageRetr LCC RepoBench-P Avg AnchorKV (ours) SnapKV AdaKV PyramidKV 103 88 96 98 91 101 82 98 80 88 99 98 113…

**Figure 14.** Figure 14: Llama-3.1-8B-Instruct on LongBench. single_1 single_2 single_3 multikey_1 multikey_2 multikey_3 multiquery multivalue qa_1 qa_2 vt cwe fwe Avg AnchorKV (ours) SnapKV AdaKV PyramidKV 100 100 91 98 94 82 100 98 96 100 100 80 98 95 99 100 3 69 19 30 89 47 95 90 99 47 91 …

**Figure 15.** Figure 15: Mistral-Small-3.1-24B-Instruct on RULER (32k). [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

**Figure 16.** Figure 16: Mistral-Small-3.1-24B-Instruct on LongBench. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]

**Figure 17.** Figure 17: Llama-3.1-70B-Instruct on RULER (32k) [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]

**Figure 18.** Figure 18: Llama-3.1-70B-Instruct on LongBench [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 22 canonical work pages

[1]

CoRR , volume =

Zefan Cai and Wen Xiao and Hanshi Sun and Cheng Luo and Yikai Zhang and Ke Wan and Yucheng Li and Yeyang Zhou and Li. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.24133 , eprinttype =. 2505.24133 , timestamp =

doi:10.48550/arxiv.2505.24133 2025
[2]

2009 , url =

Nir Ailon and Bernard Chazelle , title =. 2009 , url =. doi:10.1137/060673096 , timestamp =

doi:10.1137/060673096 2009
[3]

Albert Tseng and Jerry Chee and Qingyao Sun and Volodymyr Kuleshov and Christopher De Sa , editor =. QuIP. Forty-first International Conference on Machine Learning,. 2024 , url =

work page 2024
[4]

Croci and Bo Li and Pashmina Cameron and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman , editor =

Saleh Ashkboos and Amirkeivan Mohtashami and Maximilian L. Croci and Bo Li and Pashmina Cameron and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman , editor =. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs , booktitle =. 2024 , url =

work page 2024
[5]

CoRR , volume =

Amir Zandieh and Majid Daliri and Majid Hadian and Vahab Mirrokni , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2504.19874 , eprinttype =. 2504.19874 , timestamp =

arXiv doi:10.48550/arxiv.2504.19874 2025
[6]

Bashatah and Aurang Zaib , title =

Hamza Rauf and Umair Khan and Jomana A. Bashatah and Aurang Zaib , title =. Complex Intell. Syst. , volume =. 2026 , url =. doi:10.1007/S40747-026-02247-X , timestamp =

work page doi:10.1007/s40747-026-02247-x 2026
[7]

Lloyd , title =

Stuart P. Lloyd , title =. 1982 , url =. doi:10.1109/TIT.1982.1056489 , timestamp =

arXiv 1982
[8]

G. M. Roe , title =. 1964 , url =. doi:10.1109/TIT.1964.1053693 , timestamp =

work page arXiv 1964

Show all 46 references

[9]

Forty-first International Conference on Machine Learning,

Zirui Liu and Jiayi Yuan and Hongye Jin and Shaochen (Henry) Zhong and Zhaozhuo Xu and Vladimir Braverman and Beidi Chen and Xia Hu , editor =. Forty-first International Conference on Machine Learning,. 2024 , url =

2024
[10]

Jianlin Su and Murtadha H. M. Ahmed and Yu Lu and Shengfeng Pan and Wen Bo and Yunfeng Liu , title =. Neurocomputing , volume =. 2024 , url =. doi:10.1016/J.NEUCOM.2023.127063 , timestamp =

2024
[11]

The Twelfth International Conference on Learning Representations,

Guangxuan Xiao and Yuandong Tian and Beidi Chen and Song Han and Mike Lewis , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

2024
[12]

Zhenyu Zhang and Ying Sheng and Tianyi Zhou and Tianlong Chen and Lianmin Zheng and Ruisi Cai and Zhao Song and Yuandong Tian and Christopher R. Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023,...

2023
[13]

Scissorhands: Exploiting the Persistence of Importance Hypothesis for

Zichang Liu and Aditya Desai and Fangshuo Liao and Weitao Wang and Victor Xie and Zhaozhuo Xu and Anastasios Kyrillidis and Anshumali Shrivastava , editor =. Scissorhands: Exploiting the Persistence of Importance Hypothesis for. Advances in Neural Information Processing System...

2023
[14]

Transformers are Multi-State RNNs , booktitle =

Matanel Oren and Michael Hassid and Yarden Nir and Yossi Adi and Roy Schwartz , editor =. Transformers are Multi-State RNNs , booktitle =. 2024 , url =. doi:10.18653/V1/2024.EMNLP-MAIN.1043 , timestamp =

2024 doi
[15]

Yuhong Li and Yingbing Huang and Bowen Yang and Bharat Venkitesh and Acyr Locatelli and Hanchen Ye and Tianle Cai and Patrick Lewis and Deming Chen , editor =. SnapKV:. Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Sys...

2024
[16]

CoRR , volume =

Zefan Cai and Yichi Zhang and Bofei Gao and Yuliang Liu and Tianyu Liu and Keming Lu and Wayne Xiong and Yue Dong and Baobao Chang and Junjie Hu and Wen Xiao , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2406.02069 , eprinttype =. 2406.02069 , timestamp =

2024 doi arXiv
[17]

Kevin Zhou , editor =

Yuan Feng and Junlin Lv and Yukun Cao and Xike Xie and S. Kevin Zhou , editor =. Ada-KV: Optimizing. Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Me...

2025
[18]

2026 , eprint=

CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective , author=. 2026 , eprint=

2026
[19]

CoRR , volume =

Yuzhe Gu and Xiyu Liang and Jiaojiao Zhao and Enmao Diao , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.07651 , eprinttype =. 2510.07651 , timestamp =

2025 doi arXiv
[20]

CoRR , volume =

Yongqi An and Chang Lu and Kuan Zhu and Tao Yu and Chaoyang Zhao and Hong Wu and Ming Tang and Jinqiao Wang , title =. CoRR , volume =. 2026 , url =. doi:10.48550/ARXIV.2605.08840 , eprinttype =. 2605.08840 , timestamp =

2026 doi arXiv
[21]

Abdi and Dongsheng Li and Jianfeng Gao and Yuqing Yang and Lili Qiu , title =

Yucheng Li and Huiqiang Jiang and Qianhui Wu and Xufang Luo and Surin Ahn and Chengruidong Zhang and Amir H. Abdi and Dongsheng Li and Jianfeng Gao and Yuqing Yang and Lili Qiu , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =

2025
[22]

Mahoney and Yakun Sophia Shao and Kurt Keutzer and Amir Gholami , editor =

Coleman Hooper and Sehoon Kim and Hiva Mohammadzadeh and Michael W. Mahoney and Yakun Sophia Shao and Kurt Keutzer and Amir Gholami , editor =. KVQuant: Towards 10 Million Context Length. Advances in Neural Information Processing Systems 37: Annual Conference on Neural Informa...

2024
[23]

CoRR , volume =

Hao Kang and Qingru Zhang and Souvik Kundu and Geonhwa Jeong and Zaoxing Liu and Tushar Krishna and Tuo Zhao , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2403.05527 , eprinttype =. 2403.05527 , timestamp =

2024 doi arXiv
[24]

CoRR , volume =

Utkarsh Saxena and Kaushik Roy , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.05373 , eprinttype =. 2510.05373 , timestamp =

2025 doi
[25]

CaM: Cache Merging for Memory-efficient LLMs Inference , booktitle =

Yuxin Zhang and Yuxuan Du and Gen Luo and Yunshan Zhong and Zhenyu Zhang and Shiwei Liu and Rongrong Ji , editor =. CaM: Cache Merging for Memory-efficient LLMs Inference , booktitle =. 2024 , url =

2024
[26]

CoRR , volume =

Zhongwei Wan and Xinjian Wu and Yu Zhang and Yi Xin and Chaofan Tao and Zhihong Zhu and Xin Wang and Siqi Luo and Jing Xiong and Mi Zhang , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2406.13035 , eprinttype =. 2406.13035 , timestamp =

2024 doi arXiv
[27]

CoRR , volume =

Zheng Wang and Boxiao Jin and Zhongzhi Yu and Minjia Zhang , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2407.08454 , eprinttype =. 2407.08454 , timestamp =

2024 doi arXiv
[28]

MiniCache:

Akide Liu and Jing Liu and Zizheng Pan and Yefei He and Reza Haffari and Bohan Zhuang , editor =. MiniCache:. Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10...

2024
[29]

xKV: Cross-Layer

Chi. xKV: Cross-Layer. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.18893 , eprinttype =. 2503.18893 , timestamp =

2025 doi arXiv
[30]

Palu: Compressing KV-Cache with Low-Rank Projection , journal =

Chi. Palu: Compressing KV-Cache with Low-Rank Projection , journal =. 2024 , url =. doi:10.48550/ARXIV.2407.21118 , eprinttype =. 2407.21118 , timestamp =

2024 doi arXiv
[31]

ShadowKV:

Hanshi Sun and Li. ShadowKV:. Forty-second International Conference on Machine Learning,. 2025 , url =

2025
[32]

CommVQ: Commutative Vector Quantization for

Junyan Li and Yang Zhang and Muhammad Yusuf Hassan and Talha Chafekar and Tianle Cai and Zhile Ren and Pengsheng Guo and Foroozan Karimzadeh and Colorado Reed and Chong Wang and Chuang Gan , editor =. CommVQ: Commutative Vector Quantization for. Forty-second International Conf...

2025
[33]

Jie Hu and Shengnan Wang and Yutong He and Ping Gong and Jiawei Yi and Juncheng Zhang and Youhui Bai and Renhai Chen and Gong Zhang and Cheng Li and Kun Yuan , title =. Trans. Mach. Learn. Res. , volume =. 2026 , url =

2026
[34]

Guangda Liu and Chengwei Li and Jieru Zhao and Chenqi Zhang and Minyi Guo , title =. 62nd. 2025 , url =. doi:10.1109/DAC63849.2025.11132479 , timestamp =

2025
[35]

Mahoney and Kurt Keutzer and Amir Gholami , editor =

Coleman Richard Charles Hooper and Sehoon Kim and Hiva Mohammadzadeh and Monishwaran Maheswaran and Sebastian Zhao and June Paik and Michael W. Mahoney and Kurt Keutzer and Amir Gholami , editor =. Squeezed Attention: Accelerating Long Context Length. Proceedings of the 63rd A...

2025 doi
[36]

CoRR , volume =

Llama Team , title =. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2407.21783 , eprinttype =. 2407.21783 , timestamp =

2024 doi arXiv
[37]

CoRR , volume =

Cheng. CoRR , volume =. 2024 , url =. doi:10.48550/ARXIV.2404.06654 , eprinttype =. 2404.06654 , timestamp =

2024 doi arXiv
[38]

LongBench:

Yushi Bai and Xin Lv and Jiajie Zhang and Hongchang Lyu and Jiankai Tang and Zhidian Huang and Zhengxiao Du and Xiao Liu and Aohan Zeng and Lei Hou and Yuxiao Dong and Jie Tang and Juanzi Li , editor =. LongBench:. Proceedings of the 62nd Annual Meeting of the Association for ...

2024 doi
[39]

The Twelfth International Conference on Learning Representations,

Tri Dao , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

2024
[40]

2023 , eprint=

Landmark Attention: Random-Access Infinite Context Length for Transformers , author=. 2023 , eprint=

2023
[41]

Mistral Small 3.1 , year =
[42]

, title =

Kamradt, G. , title =. 2023 , howpublished =

2023
[43]

and Jeblick, M

Devoto, A. and Jeblick, M. and J. Expected Attention:. 2025 , eprint =

2025
[44]

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory , booktitle =

Chaojun Xiao and Pengle Zhang and Xu Han and Guangxuan Xiao and Yankai Lin and Zhengyan Zhang and Zhiyuan Liu and Maosong Sun , editor =. InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory , booktitle =. 2024 , url =

2024
[45]

Forty-first International Conference on Machine Learning,

Jiaming Tang and Yilong Zhao and Kan Zhu and Guangxuan Xiao and Baris Kasikci and Song Han , editor =. Forty-first International Conference on Machine Learning,. 2024 , url =

2024
[46]

2026 , howpublished = "

2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

CoRR , volume =

Zefan Cai and Wen Xiao and Hanshi Sun and Cheng Luo and Yikai Zhang and Ke Wan and Yucheng Li and Yeyang Zhou and Li. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.24133 , eprinttype =. 2505.24133 , timestamp =

doi:10.48550/arxiv.2505.24133 2025

[2] [2]

2009 , url =

Nir Ailon and Bernard Chazelle , title =. 2009 , url =. doi:10.1137/060673096 , timestamp =

doi:10.1137/060673096 2009

[3] [3]

Albert Tseng and Jerry Chee and Qingyao Sun and Volodymyr Kuleshov and Christopher De Sa , editor =. QuIP. Forty-first International Conference on Machine Learning,. 2024 , url =

work page 2024

[4] [4]

Croci and Bo Li and Pashmina Cameron and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman , editor =

Saleh Ashkboos and Amirkeivan Mohtashami and Maximilian L. Croci and Bo Li and Pashmina Cameron and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman , editor =. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs , booktitle =. 2024 , url =

work page 2024

[5] [5]

CoRR , volume =

Amir Zandieh and Majid Daliri and Majid Hadian and Vahab Mirrokni , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2504.19874 , eprinttype =. 2504.19874 , timestamp =

arXiv doi:10.48550/arxiv.2504.19874 2025

[6] [6]

Bashatah and Aurang Zaib , title =

Hamza Rauf and Umair Khan and Jomana A. Bashatah and Aurang Zaib , title =. Complex Intell. Syst. , volume =. 2026 , url =. doi:10.1007/S40747-026-02247-X , timestamp =

work page doi:10.1007/s40747-026-02247-x 2026

[7] [7]

Lloyd , title =

Stuart P. Lloyd , title =. 1982 , url =. doi:10.1109/TIT.1982.1056489 , timestamp =

arXiv 1982

[8] [8]

G. M. Roe , title =. 1964 , url =. doi:10.1109/TIT.1964.1053693 , timestamp =

work page arXiv 1964