Pith. sign in

REVIEW 2 major objections 4 minor 45 references

CaliDrop: KV Cache Compression with Calibration

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CaliDrop recovers much of the accuracy lost when a KV cache is compressed by token eviction, by reusing a stored historical query as a calibration signal.

desk verdict Solid incremental KV compression paper with a genuinely new stale-query calibration mechanism; thoroughly tested, but the similarity gate is thinly evidenced and thresholds are tuned on the headline benchmark. read the letter →

arxiv 2507.19906 v2 pith:27I2ALV7 submitted 2025-07-26 cs.CL

Yi Su , Quantong Qiu , Yuechi Zhou , Juntao Li , Qingrong Xia , Ping Li , Xinyu Duan , Zhefeng Wang
show 1 more author
Min Zhang
This is my paper
classification cs.CL
keywords KVcachecompressiontokenevictionattentioncalibrationquerysimilaritydecompositionlong-contextinferencespeculativeCaliDrop
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Token eviction is a common way to shrink the key-value (KV) cache that lets large language models generate quickly, but throwing away tokens costs accuracy. CaliDrop proposes to soften that loss: evicted key-value pairs are offloaded rather than deleted, their attention contribution is precomputed for a stored historical query, and during decoding that stored contribution is added back as a calibration term whenever the current query resembles the historical one. The paper claims this recovers a large share of the accuracy that eviction removes, with improvements across StreamingLLM, H2O, and SnapKV, especially at high compression ratios like a 64-token cache. If the claim holds, a serving system can keep only a small KV cache in fast memory and still preserve much of the model's quality, making long-context inference far cheaper.

What carries the argument

The load-bearing object is a small calibration record kept for the evicted tokens: a historical query, the softmax denominator over the offloaded keys, and the attention output the historical query would have produced against them. The attention decomposition theorem justifies treating the retained-cache attention and the evicted-cache contribution as additive pieces of the full attention, with weights $\alpha_i$ and $\alpha_j$ equal to each subset's share of the total exponential score sum. The similarity gate then decides when the stored contribution is still trustworthy: below $\theta_1=0.7$ the evicted cache is reloaded and the record is recomputed with the current query; above $\theta_2=0.85$ the stored record is added directly; in between, calibration is skipped. Together these pieces convert a permanently lossy eviction into an occasionally refreshed approximation.

What would settle it

Run CaliDrop on a controlled task in which adjacent-token queries are deliberately dissimilar, for example a long document with randomly interleaved unrelated passages, and compare accuracy against the underlying eviction method without calibration. If accuracy still improves despite low cosine similarity, the similarity premise is not the active mechanism; if accuracy degrades below the baseline, the premise is confirmed as load-bearing. A more direct measurement is to compute the median adjacent-query cosine similarity per layer and head on several benchmarks and check whether the layers with similarity below 0.7 are the same layers where calibration hurts.

Watch

Extended reading notes

Core claim

The central claim is that eviction does not have to be a permanent loss: the contribution of evicted tokens can be approximated and then reinstated. CaliDrop splits the full KV cache into the retained subset and the evicted subset, offloads the evicted one, and uses the attention decomposition theorem, $\operatorname{Att}(Q,K,V) = \alpha_i \operatorname{Att}(Q,K_{S_i},V_{S_i}) + \alpha_j \operatorname{Att}(Q,K_{S_j},V_{S_j})$, to compute the two parts separately. During the prefill phase it stores, for the evicted subset, the last query $Q_{-1}$, the softmax denominator $\sum_t \exp(Q_{-1} K_t^\top/\sqrt{d_k})$, and the attention output $\operatorname{Att}(Q_{-1}, K_{\mathrm{evict}}, V_{\mathrm{evict}})$. During decoding, the cosine similarity between the current query and $Q_{-1}$ decides whether to reuse that stored output as a correction ($\rho > \theta_2$), recompute it by reloading the offloaded cache ($\rho < \theta_1$), or do nothing in between. The paper's empirical foundation is the observation, demonstrated on LongBench with LLaMA-3-8B, that queries at nearby positions have high cosine similarity and that this calibration reduces the L1 error of the approximated attention output versus the true one.

Load-bearing premise

Everything rests on the empirical premise that nearby queries look alike, so the stored historical query's attention output over evicted tokens is a trustworthy stand-in for the current query's missing attention; where that similarity is low, the calibration term can add harmful noise instead of a correction.

Editorial extensions

If this is right

  • A serving engine can keep only a small KV cache in fast memory, offload the evicted tokens, and still recover most of the accuracy that eviction removes, which matters most when the retained cache is as small as 64 tokens.
  • Because CaliDrop is a wrapper, any future token-eviction policy can inherit the calibration layer without changing its token-selection rule.
  • The benefit is largest when compression is aggressive and shrinks as the retained cache grows, so CaliDrop is a tool for high-compression regimes, not a replacement for a large cache.
  • With $\theta_1=0.7$, recomputation happens roughly every eight decoding steps, so the accuracy gain comes with only a modest throughput penalty compared with the eviction baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's experiments, the same calibration record could be maintained during decoding-phase eviction, not just prefill eviction, which would address the long-generation regime the paper leaves to future work.
  • The global thresholds $\theta_1$ and $\theta_2$ could be made per-layer or per-head, since the paper's own recomputation-frequency analysis shows query similarity varies across layers; adaptive thresholds would let high-similarity heads reuse stored calibrations longer.
  • The offloaded cache is a natural target for quantization or low-rank projection, so CaliDrop's offload-and-approximate idea should compose with other KV compression techniques rather than compete with them.
  • More generally, CaliDrop suggests a design principle for cache compression: any eviction policy defines an error term, and if a cheap transformation of historical states can approximate that term, eviction need not be a permanent loss; the same pattern could apply to prefix caches or cross-attention states.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes CaliDrop, an add-on to token-eviction KV cache compression. The method offloads evicted KV pairs, precomputes during prefill an attention output over those evicted tokens using a historical query, and during decoding uses cosine similarity between the current and historical query to decide whether to reuse the precomputed calibration (θ2), recompute with the current query (θ1), or skip. The decomposition in Eq. (1) is exact for a fixed query; the approximation arises from substituting the historical query's softmax statistics for the current query's. Experiments on LongBench, RULER, and Needle-in-a-Haystack with Mistral-7B, LLaMA-3-8B, and LLaMA-3-70B show consistent accuracy gains over StreamingLLM, H2O, and SnapKV at several KV budgets, with modest throughput cost and a short analysis of threshold sensitivity.

Significance. If the accuracy gains are robust, CaliDrop is a useful, simple plug-in for existing eviction methods: it does not change the eviction policy, only adds an offloaded residual computation and a similarity gate. The paper's main strengths are its breadth (three models, three benchmarks, four KV sizes) and the clean statement of the exact attention decomposition. The main risk is that the core premise—that historical attention outputs approximate current ones well enough for calibration—is validated only narrowly in the preliminary section, and the two thresholds are selected on the same LongBench benchmark used for the headline results. The held-out RULER and NIAH results are encouraging but do not fully resolve the sensitivity question.

major comments (2)
  1. [§3.1.3, Figure 1, Algorithm 2] The central approximation is load-bearing and its empirical justification is too narrow. Equation (1) is exact only when the same query is used for both subsets, but Algorithm 2 replaces the current-query softmax denominator over the evicted set with Cweight computed from the stored historical query (Algorithm 1, lines 6–8). The only safeguard against a bad substitution is the cosine gate in Algorithm 2, yet the preliminary evidence in §3.1.1 is a single layer/head heatmap (Figure 1, top-right) and §3.1.3 tests only the first token after prefill with a historical query 0–9 positions old (Figure 1, bottom-right). In the actual decode regime the stored query can be about eight steps old (Figure 3b), and raw cosine similarity does not control the divergence between softmax distributions. Please provide per-layer/per-head statistics at realistic decode lags, e.g., the relationship between cosine similarity and the relative error of the calibrated attention output, and report the fraction of decode steps where calibration reduces rather than increases the attention-output error.
  2. [§4.1.3, Table 2] The two thresholds are selected on the same benchmark used for the headline results. θ1 = 0.7 and θ2 = 0.85 are chosen from Table 2, which reports LongBench accuracy for LLaMA-3-8B-Instruct with SnapKV and KV size 128; the main results in Table 1 include exactly this setting, so the LongBench gains are partly in-sample. Please either select thresholds on a development split not used in the main tables, or report a sensitivity analysis of θ1/θ2 on RULER and NIAH to show that the chosen values transfer. The held-out RULER and NIAH results with the fixed thresholds are encouraging, but they do not by themselves quantify the sensitivity of the method to the gate.
minor comments (4)
  1. [Algorithm 2, line 5] In the recomputation branch, the line 'Cweight = Σ exp(Q_{−1} K^T_evict / sqrt(dk))' should use the current query Qt, not Q_{−1}; as written, recomputation does not update the softmax denominator with the current query.
  2. [§4.2, Tables 1–4] No variance or number of runs is reported for any of the accuracy numbers; since the abstract uses 'significantly improves', please add standard deviations or at least state whether each number is a single run.
  3. [Abstract and §1] The paper is framed as 'KV Cache Compression', but CaliDrop offloads the evicted KV pairs rather than discarding them; the memory savings are for the active GPU cache only, while total storage is not reduced. Please clarify this framing in the abstract and introduction to avoid overclaiming.
  4. [Figure 3b] It would be helpful to also report the distribution of cosine similarities across layers and heads in the decode regime, since the thresholds θ1 and θ2 are the only mechanism that prevents harmful calibration.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CaliDrop's central calibration step is an empirical approximation built on an exact algebraic identity, not a result defined in terms of its reported outcomes.

full rationale

The paper's derivation chain is self-contained. Equation (1) is an exact attention decomposition over disjoint token subsets, and its proof is given directly in Section 3.1.2 with no appeal to prior work by the same authors. The method then replaces the current query with a stored historical query when computing the attention output over evicted tokens; this substitution is explicitly treated as an empirical premise validated in Sections 3.1.1 and 3.1.3, rather than as a definition of the claimed improvement. No parameter is constructed from the reported benchmark scores themselves, and the headline improvement is not forced by construction: one could implement the method and observe degradation if the similarity premise failed. The RULER and Needle-in-a-Haystack results provide independent external evidence with fixed thresholds, so the central claim has content beyond any single benchmark. The only mild concern is that the thresholds θ1=0.7 and θ2=0.85 are selected using LongBench accuracy before the LongBench main results are reported; that is a potential selection-bias or robustness issue, not circularity, because the selection does not make the reported gains follow by definition and the method is also evaluated on separate benchmarks. No load-bearing self-citation, imported uniqueness theorem, or ansatz-via-citation step appears in the derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. Its free parameters are two thresholds plus an optional calibration size, all tuned or defined by hand. The axioms are the standard softmax identity and the empirical query-locality assumption, which is load-bearing for the entire calibration concept.

free parameters (3)
  • θ1 (left threshold) = 0.7
    Controls recomputation frequency; selected by sweeping LongBench accuracy in Table 2.
  • θ2 (right threshold) = 0.85
    Controls when calibration is applied; selected by the same Table 2 sweep on LongBench.
  • calibration size = ∞ in main experiments; optional
    Introduced in §5.3 to trade off calibration overhead and accuracy; not used in the main results.
assumptions (4)
  • standard math Softmax attention can be decomposed over any partition of keys into a weighted sum of per-partition attention outputs (Eq. 1).
    Algebraic identity following directly from the softmax definition; used to justify the calibration combination.
  • domain assumption Queries at nearby positions have high cosine similarity.
    Empirical observation from §3.1.1, shown for selected layers/heads; it is the core premise that makes stale-query calibration accurate.
  • domain assumption Historical attention outputs and their softmax denominators approximate future ones.
    Validated indirectly by the L1 loss experiment in §3.1.3; no general proof is given.
  • domain assumption Cosine similarity thresholds reliably separate beneficial from harmful calibration.
    Assumed in §3.2 and tuned in §5.1; the paper provides empirical support but no theoretical grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaliDrop: KV Cache Compression with Calibration." pith.science (2026). https://pith.science/paper/27I2ALV7

@misc{pith2026250719906,
  author       = {Pith},
  title        = {Pith review of: CaliDrop: KV Cache Compression with Calibration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27I2ALV7}},
  note         = {Machine review of arXiv:2507.19906}
}
read the original abstract

Large Language Models (LLMs) require substantial computational resources during generation. While the Key-Value (KV) cache significantly accelerates this process by storing attention intermediates, its memory footprint grows linearly with sequence length, batch size, and model size, creating a bottleneck in long-context scenarios. Various KV cache compression techniques, including token eviction, quantization, and low-rank projection, have been proposed to mitigate this bottleneck, often complementing each other. This paper focuses on enhancing token eviction strategies. Token eviction leverages the observation that the attention patterns are often sparse, allowing for the removal of less critical KV entries to save memory. However, this reduction usually comes at the cost of notable accuracy degradation, particularly under high compression ratios. To address this issue, we propose \textbf{CaliDrop}, a novel strategy that enhances token eviction through calibration. Our preliminary experiments show that queries at nearby positions exhibit high similarity. Building on this observation, CaliDrop performs speculative calibration on the discarded tokens to mitigate the accuracy loss caused by token eviction. Extensive experiments demonstrate that CaliDrop significantly improves the accuracy of existing token eviction methods.

Figures

Figures reproduced from arXiv: 2507.19906 by the authors.

Figure 1
Figure 1. Illustration of the proposed method and preliminary results. Left: Overview of the method [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Results of Needle-in-a-Haystack on LLaMA-3-8B-Instruct with 8k context size and 64 KV [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. (a) Throughput of CaliDrop and SnapKV under different batch size. (b) Recomputation [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Results of Needle-in-a-Haystack on Mistral-7B-Instruct and LLaMA-3-8B-Instruct with [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Results of Needle-in-a-Haystack on Mistral-7B-Instruct with 32k context size and 64 KV [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Results of Needle-in-a-Haystack on Mistral-7B-Instruct with 32k context size and 96 KV [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Results of Needle-in-a-Haystack on Mistral-7B-Instruct with 32k context size and 128 KV [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Results of Needle-in-a-Haystack on Mistral-7B-Instruct with 32k context size and 256 KV [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Results of Needle-in-a-Haystack on LLaMA-3-8B-Instruct with 8k context size and 96 KV [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Results of Needle-in-a-Haystack on LLaMA-3-8B-Instruct with 8k context size and 128 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Results of Needle-in-a-Haystack on LLaMA-3-8B-Instruct with 8k context size and 256 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 9 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 4895–4901, 2023

    work page 2023

  3. [3]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for ...

    work page 2024

  4. [4]

    Reducing transformer key-value cache size with cross-layer attention

    William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981, 2024

    arXiv 2024

  5. [5]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024

    arXiv 2024

  6. [6]

    Palu: Compressing kv-cache with low-rank projection

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118, 2024

    arXiv 2024

  7. [7]

    Sepllm: Accelerate large language models by compressing one segment into one separator

    Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. Sepllm: Accelerate large language models by compressing one segment into one separator. arXiv preprint arXiv:2412.12094, 2024

    arXiv 2024

  8. [8]

    A simple and effective l_2 norm-based strategy for kv cache compression

    Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective l_2 norm-based strategy for kv cache compression. arXiv preprint arXiv:2406.11430, 2024

    arXiv 2024

Show all 45 references
  1. [9]

    Qaq: Quality adaptive quantization for llm kv cache

    Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for llm kv cache. arXiv preprint arXiv:2403.04643, 2024

    2024 arXiv

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference. arXiv preprint arXiv:2407.11550, 2024

    2024 arXiv

  4. [12]

    Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning

    Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint arXiv:2410.19258, 2024

    2024

  5. [13]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  6. [14]

    Zipcache: Accurate and efficient kv cache quantization with salient token identification

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. arXiv preprint arXiv:2405.14256, 2024

    2024 arXiv

  7. [15]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length LLM inference with KV cache quantization. CoRR, abs/2401.18079, 2024

    2024 arXiv

  8. [16]

    Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    2024 arXiv

  9. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    arXiv

  10. [18]

    Hydragen: High-throughput llm inference with shared prefixes

    Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput llm inference with shared prefixes. arXiv preprint arXiv:2402.05099, 2024

    2024 arXiv

  11. [19]

    GEAR: an efficient KV cache compression recipe for near-lossless generative inference of LLM

    Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: an efficient KV cache compression recipe for near-lossless generative inference of LLM. CoRR, abs/2403.05527, 2024

    2024 arXiv

  12. [20]

    Snapkv: Llm knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024

    2024 arXiv

  13. [21]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  14. [22]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  15. [23]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  16. [24]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Process...

    2024

  17. [25]

    KIVI: A tuning-free asymmetric 2bit quantization for KV cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024....

    2024

  18. [26]

    CAKE: Cascading and adaptive KV cache eviction with layer preferences

    Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. CAKE: Cascading and adaptive KV cache eviction with layer preferences. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  19. [27]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  20. [28]

    Flexgen: High-throughput generative inference of large language models with a single gpu

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, page...

    2023

  21. [29]

    You only cache once: Decoder-decoder architectures for language models

    Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254, 2024

    2024 arXiv

  22. [30]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. In Forty-first International Conference on Machine Learning, 2024

    2024

  23. [31]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  24. [32]

    With greater text comes greater necessity: Inference-time training helps long text generation

    Y Wang, D Ma, and D Cai. With greater text comes greater necessity: Inference-time training helps long text generation. arXiv preprint arXiv:2401.11504, 2024

    2024 arXiv

  25. [33]

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454, 2024

    2024 arXiv

  26. [34]

    Layer-condensed kv cache for efficient inference of large language models

    Haoyi Wu and Kewei Tu. Layer-condensed kv cache for efficient inference of large language models. arXiv preprint arXiv:2405.10637, 2024

    2024 arXiv

  27. [35]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. 11

    2024

  28. [36]

    Recycled attention: Efficient inference for long-context language models

    Fangyuan Xu, Tanya Goyal, and Eunsol Choi. Recycled attention: Efficient inference for long-context language models. arXiv preprint arXiv:2411.05787, 2024

    2024 arXiv

  29. [37]

    Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference

    Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference. arXiv preprint arXiv:2405.12532, 2024

    2024 arXiv

  30. [38]

    No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096, 2024

    2024 arXiv

  31. [39]

    Effectively compress kv heads for llm

    Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads for llm. arXiv preprint arXiv:2406.07056, 2024

    2024 arXiv

  32. [40]

    Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization

    Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. arXiv preprint arXiv:2405.03917, 2024

    2024 arXiv

  33. [41]

    Unifying kv cache compression for large language models with leankv

    Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John Lui, and Haibo Chen. Unifying kv cache compression for large language models with leankv. arXiv preprint arXiv:2412.03131, 2024

    2024 arXiv

  34. [42]

    Cam: Cache merging for memory-efficient llms inference

    Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam: Cache merging for memory-efficient llms inference. In Forty-first International Conference on Machine Learning, 2024

    2024

  35. [43]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Trista...

    2023

  36. [44]

    Dynamickv: Task-aware adaptive kv cache compression for long context llms

    Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, and Liang Ding. Dynamickv: Task-aware adaptive kv cache compression for long context llms. arXiv preprint arXiv:2412.14838, 2024

    2024 arXiv

  37. [45]

    Relayattention for efficient large language model serving with long system prompts

    Lei Zhu, Xinjiang Wang, Wayne Zhang, and Rynson WH Lau. Relayattention for efficient large language model serving with long system prompts. arXiv preprint arXiv:2402.14808, 2024. 12 Method Single-Document QA Multi-Document QA Summarization Few-shot Learning Synthetic Code Avg....

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.