Pith. sign in

REVIEW 3 major objections 5 minor 39 references

HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that HashEvict can shrink an LLM's KV cache by 30-70% while preserving task quality, by evicting the cached token whose binarized key is most Hamming-distant from the current query.

desk verdict A genuinely new pre-attention eviction trick that deserves review, but the paper's own tables undercut the '30-70% compression' headline at the high end. read the letter →

arxiv 2412.16187 v3 pith:AQII5WNN submitted 2024-12-13 cs.LG cs.AIcs.CLcs.DScs.PF

Minghui Liu , Tahseen Rabbani , Tony O'Halloran , Ananth Sankaralingam , Mary-Anne Hartley , Furong Huang , Cornelia Fermüller , Yiannis Aloimonos This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.DScs.PF
keywords KVcachecompressionlocality-sensitivehashingtokenevictionefficientLLMinferenceHammingdistanceSimHashpre-attentionlong-contextunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HashEvict asks whether a large language model's key-value cache can be compressed by deciding what to evict before attention is computed, using only cheap binary comparisons. The algorithm binarizes each cached key and the current query with locality-sensitive hashing, then evicts the cached token whose hash code is farthest in Hamming distance from the query hash, on the theory that Hamming distance tracks cosine dissimilarity and hence low attention. Across reasoning, multiple-choice, long-context retrieval, and long-document summarization, the authors report that 30%-70% cache compression keeps task quality close to the full cache, with a 1.5-2x prefill speedup over attention-accumulation baselines. A sympathetic reader would care because the eviction policy adds only a tiny binary hash table to GPU memory and makes no use of attention scores, which could make longer context windows and on-device inference cheaper.

What carries the argument

The machinery is the SimHash projection, which maps a d-dimensional key or query vector to a c-bit binary code by h(x) = sgn(Rx), where R is a c by d matrix of independent standard normal entries and sgn is the elementwise step at zero. The theoretical identity used is that for unit vectors x and y the expected Hamming distance between h(x) and h(y) equals the angle between x and y divided by pi, so Hamming distance approximates cosine distance. HashEvict keeps a binary hash table of the cached keys and, each decoding step, selects the index with the largest Hamming distance to the current query's hash as the eviction victim, then inserts the current token's key, value, and hash in its place. The hash table costs c bits per cached token and is stored on GPU, which is the entire extra memory overhead.

What would settle it

Run HashEvict at a 30% cache budget on GSM8K with many random seeds for the projection matrix R, and also measure the attention mass of the tokens it evicts per head; if task scores show large seed-to-seed variance, or if evicted tokens carry as much attention as tokens chosen at random, then the Hamming ranking is not the mechanism behind the reported quality.

Watch

Extended reading notes

Core claim

HashEvict's central claim is that an attention-free eviction rule based on Hamming distances between binary hash codes can match or beat both attention-free and attention-accumulation compressors. At every decoding step it replaces the key and value of the cached token with the lowest LSH score, defined as the negative Hamming distance between the hash of the query and the hash of each cached key, thereby evicting the token estimated to be most cosine-dissimilar and least likely to receive attention. The paper argues that because the expectation of this Hamming distance is proportional to the angle between two unit vectors, low Hamming distance is a proxy for high dot-product attention, and it reports that the resulting evictions preserve medium-to-high attention tokens rather than only the highest, which the authors say explains the method's strength on free-response reasoning and summarization. The headline empirical result is 30%-70% KV cache compression with maintained performance over reasoning, multiple-choice, long-context retrieval, and summarization, plus faster prefill than attention-based baselines.

Load-bearing premise

The load-bearing premise is that one random binarized projection of unnormalized keys and queries still ranks cached tokens by true cosine similarity, whereas the theory guarantees this only in expectation over many projections of unit vectors, so a single projection can mis-rank and evict a high-attention token.

Editorial extensions

If this is right

  • The KV cache can be cut to 30%-70% of its full size on Llama-3-class models with only a small quality drop on reasoning, multiple-choice, long-context retrieval, and summarization tasks.
  • Eviction decisions become pre-attention and per-step, so the method avoids accumulating attention scores and is roughly 1.5-2x faster at prefill than H2O and Scissorhands and about 17x faster than FastGen, with competitive decoding speed.
  • The added GPU memory is only a c-bit hash per cached token; an 8-bit hash dimension works as well as 64-bit in the reported ablation, so the overhead stays near one byte per cache slot.
  • At a 50% cache budget, the attention mass lost by HashEvict's evictions is lower than the L2-norm method and comparable to Scissorhands, indicating the evicted tokens are genuinely low-attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to use two or three independent hash projections instead of one; the theoretical identity is an expectation, and the paper's own appendix shows correlation with attention rises with projection length, so ensembling could stabilize eviction for little extra memory.
  • Since the method keeps a spread of medium-to-high attention tokens, a hybrid that reserves a small protected slot for tokens with accumulating high attention could close the remaining gap at very small cache budgets without paying full attention-accumulation cost.
  • The reported layer-wise correlation drop in later layers suggests an adaptive policy could increase hash dimension or switch to L2-norm eviction in those layers, which would be a cheap, testable improvement over a single global hash configuration.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. HashEvict proposes an attention-free KV cache eviction strategy for transformer LLMs. At each decoding step, the method projects the current query embedding and all cached key embeddings to c-bit binary codes via a random Gaussian SimHash matrix, computes Hamming distances, and evicts the key/value pair whose hash is farthest from the query hash, replacing it with the current token. The paper claims that this achieves 30-70% KV cache compression while maintaining high performance on GSM8K, MedQA, long-context retrieval, and summarization tasks, with up to 1.5-2x prefill speedups over H2O/Scissorhands and 17x prefill speedup over FastGen. The evaluation compares against L2-norm eviction, H2O, Scissorhands, and FastGen across cache budgets from 10% to 90%.

Significance. If the claims are fully supported, HashEvict would be a useful contribution to KV cache compression because it avoids computing attention scores for eviction decisions and maintains only a small binary hash table. The paper presents a broad evaluation across seven task types, a hash-dimension ablation, and an attention-loss-ratio analysis. However, the strongest claimed compression range is not backed by the paper's own tables, and the mechanistic validation of the LSH ranking is weak, so the significance is currently conditional on additional evidence.

major comments (3)
  1. [Section 4.1, Tables 5, 6, and 9] The headline claim of 30-70% compression while maintaining high performance is not supported at the upper end of the range. At a 30% cache budget (70% compression), GSM8K GPT4-Judge similarity to ground truth falls from 3.845 (full cache) to 2.520 (Table 6), and Rouge-L falls from 0.403 to 0.341 (Table 5); MedQA GPT4-Judge similarity falls from 3.337 to 2.511 (Table 9). These are large degradations (about 34% and 15% on GSM8K) and are inconsistent with the paper's claim that the method 'maintains high performance' across the full 30-70% range. The authors should either restrict the claim to the range where degradation is small (e.g., 30-50% compression) or define an explicit tolerance criterion and show that the 70%-compression results meet it.
  2. [Section 3, Eq. (3), and Appendix E] The theoretical justification in Eq. (3) is an expectation over the random projection matrix R for unit vectors, but the deployed Algorithm 1 uses a single random projection and makes a deterministic argmin decision each step without averaging. The unit-vector assumption is not the main issue because sgn(Rx) is invariant to positive scaling; the load-bearing gap is expectation-versus-single-draw. The paper reports no variance across random seeds or hash draws, so the stability of the eviction decision is unknown. Appendix E, which reports a mean Pearson correlation of only 0.2978 ± 0.1947 between attention scores and inverted average Hamming distance, uses normalized vectors and averages multiple projections, which are more favorable conditions than the deployed single-projection algorithm. Please provide a seed-variance analysis and an evaluation of the deployed single-projection setting in the correlation/ALR analysis.
  3. [Appendix D and Table 3] The ALR analysis in Appendix D is performed using a global ranking of tokens by average Hamming distance against later queries, which the paper acknowledges 'does not perfectly capture the ranking system' of HashEvict. Because the deployed method makes a fresh per-step argmin over the current cache, the reported ALR does not directly measure the attention loss of the actual eviction sequence. Table 3's ALR values therefore do not validate the dynamic behavior of Algorithm 1. Please clarify how the static ALR analysis maps to the algorithm's per-step decisions, or provide a dynamic ALR computed by simulating the actual eviction process.
minor comments (5)
  1. [Section 2, after Eq. (3)] The text contains the garbled fragment 'cwecwasdf'; this appears to be an editing artifact and should be removed.
  2. [Algorithm 1 and Section 4] The pseudocode does not indicate that the first 4 and last 10 tokens are always kept in the cache, although Section 4 states this; please align the pseudocode with the implementation.
  3. [Table 2] The column 'Compression Ratio' is never defined, and the reported values (e.g., 0.8355 at 10% budget for L2) are not consistent with the reported cache memory numbers under any obvious definition such as 1 - cache_memory/full_memory. Please define the metric and verify the values.
  4. [Section 4, Metrics] The metric labeled 'GPT4-Rouge' in the text appears to be an inconsistency; the tables use 'GPT4-Judge', and the appendix describes a similarity-to-ground-truth component. Please harmonize the terminology.
  5. [Contributions and Section 4.5] The 'Open-Source Implementation' contribution states that code will be released upon publication; for reproducibility of the reported numbers, including seed variance, it would be helpful to release the code and evaluation scripts with the submission.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HashEvict's LSH-based eviction rule is defined independently, supported by an external theorem, and evaluated against external baselines without fitting parameters to the target benchmarks.

full rationale

I found no circular derivation in HashEvict. The eviction rule is defined by Eq. 5 as the negative Hamming distance between binarized Gaussian projections of the query and cached keys, and the theoretical link between Hamming distance and cosine angle (Eq. 3) is cited to an external theorem in Goemans and Williamson (1995), not derived from the paper's own outputs or prior work. The method has no trainable or fitted parameters: the hash dimension is ablated in Table 10, and the paper reports that performance is insensitive to the choice of dimension rather than selecting a dimension to maximize benchmark scores. The Attention Loss Ratio analysis in Appendices D and E is a post-hoc evaluation that uses attention as ground truth to measure eviction quality; it is not used to construct or fit the eviction rule. The paper contains no author self-citations, and the central comparative claims rest on external baselines (H2O, Scissorhands, FastGen, L2). Concerns about single-projection randomness, lack of seed variance, and the strength of the 30-70% compression claim are validity and robustness issues, not circularity. The use of unnormalized keys and queries is also not a circularity issue because sgn(Rx) is invariant to positive scaling, so the hash codes coincide with those of the normalized vectors used in the theorem.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method rests on four premises: the standard LSH distance theorem, the cosine-attention proxy, the sufficiency of a single random projection, and the online greedy eviction policy. No new physical or conceptual entities are introduced; the hash table is a data structure with no independent evidence requirement.

free parameters (2)
  • Hash dimension c = 8 (best in ablation; default not explicitly stated)
    Chosen by hand based on GSM8K ablation in Section 4.7; the paper does not state in the main text which dimension is used for all experiments.
  • Always-kept token counts (first 4, last 10) = first 4 prompt tokens, last 10 decoded tokens
    Fixed heuristic adopted from StreamingLLM/Xiao et al. (Section 4, Configuration and Setup); these constants change what the eviction policy can remove.
assumptions (4)
  • standard math The expected Hamming distance between SimHash codes of two unit vectors is proportional to their angle (Eq 3).
    Invoked in Section 2 and cited to Goemans and Williamson (1995); not proven in the paper.
  • domain assumption Low cosine similarity between a query and a cached key predicts a low attention score for that key.
    Stated in Section 1 as intuition; the paper does not prove this, and Appendix E finds only a moderate Pearson correlation of 0.30±0.19 between attention and inverted Hamming distance.
  • domain assumption A single random projection matrix R (one hash table per head) is sufficient for making eviction decisions.
    Algorithm 1 uses one R, while the theory in Eq 3 is an expectation over R; the paper does not quantify variance across random draws or report seed dependence.
  • domain assumption Evicting the token most dissimilar to the current query at each decoding step is a good policy for future attention.
    This online greedy rule is defined in Eqs 4-6; the supportive ALR analysis in Appendix D ranks tokens by average future Hamming distance, which is a different procedure, so the policy itself is not directly validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing." pith.science (2026). https://pith.science/paper/AQII5WNN

@misc{pith2026241216187,
  author       = {Pith},
  title        = {Pith review of: HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AQII5WNN}},
  note         = {Machine review of arXiv:2412.16187}
}
read the original abstract

Transformer-based large language models (LLMs) use the key-value (KV) cache to significantly accelerate inference by storing the key and value embeddings of past tokens. However, this cache consumes significant GPU memory. In this work, we introduce HashEvict, an algorithm that uses locality-sensitive hashing (LSH) to compress the KV cache. HashEvict quickly locates tokens in the cache that are cosine dissimilar to the current query token. This is achieved by computing the Hamming distance between binarized Gaussian projections of the current token query and cached token keys, with a projection length much smaller than the embedding dimension. We maintain a lightweight binary structure in GPU memory to facilitate these calculations. Unlike existing compression strategies that compute attention to determine token retention, HashEvict makes these decisions pre-attention, thereby reducing computational costs. Additionally, HashEvict is dynamic - at every decoding step, the key and value of the current token replace the embeddings of a token expected to produce the lowest attention score. We demonstrate that HashEvict can compress the KV cache by 30%-70% while maintaining high performance across reasoning, multiple-choice, long-context retrieval and summarization tasks.

Figures

Figures reproduced from arXiv: 2412.16187 by the authors.

Figure 1
Figure 1. An abstract visualization of HASHEVICT eviction strategy. Figure 1a depicts the strategy for several decoding steps. The cache can only maintain 5 tokens due to memory constraints. At each decoding step, HASHEVICT projects the query embedding of the current token i and all previous key embeddings to binary hash codes. HASHEVICT then measures the negative of Hamming distances between the query code of token i and key… view at source ↗
Figure 2
Figure 2. GSM8K Question Answering Performance. We measure BERTScore F1, Rouge-L, and GPT4-Judge for different cache budgets on a grade school math task. HASHEVICT outperforms L2 for all three metrics for every budget, with sharp differences for the 50% and 30% compression. HASHEVICT performs similarly to H2O and Scissorhands except at 10% cache budget. MedQA MedQA is a free response multiple choice question answering dataset… view at source ↗
Figure 3
Figure 3. MedQA Question Answering Performance. We measure BertScore F1, Rouge-L, and GPT4-Judge for different cache budgets on a medical exam task. LSH outperforms L2 for all three metrics for every budget, with a significantly higher performance for the 30% and 10% budgets. (a) Accuracy on the GSM8K Multiple Choice (b) Accuracy on the MedQA Multiple Choice [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Multiple Choice Tasks Performance. On GSM8K, HASHEVICT outperforms the baseline full cache on GSM8K at 70% and 50% cache budgets and significantly outperforms L2 at 70%, 50%, and 30%. HASHEVICT performs on par with L2 overall on MedQA with higher performance at 90% (ne…
Figure 5
Figure 5. Figure 5: Long-Context Tasks. We measure string-matching scores for two long-context retrieval tasks. HASHEVICT performs on par with L2 on the Common Words task with slightly higher performance at a 30% cache budget and slightly lower performance at a 10% budget. For the Needle-…
Figure 6
Figure 6. Figure 6: LongBench Summarization Tasks We measure Rouge-L for two long text summarization tasks. HASHEVICT outperforms all baseline methods on MultiNews at 30 - 70% cache budget. HASHEVICT performs better than L2 on GovReport at 50% cache budget similarly at 30% and 70%. MultiN…
Figure 7
Figure 7. Figure 7: Attention and Key Norms. Attention scores and corresponding L2 norms of key vectors (excluding the first token) for a sample of heads (0,8,16,24,31) in the 8th layer for a sample input sequence. Each subplot shows the attention heatmap (top) and the corresponding key n…
Figure 8
Figure 8. Figure 8: Attention Loss Ratio (ALR). We compare how the eviction strategy of HASHEVICT and the L2 method [Devoto et al., 2024] affects the ALR per equation 7. Our tested model is Llama3-8B-Instruct, which contains 32 heads and 32 attention layers. Cell (i, j) depicts the ALR of…
Figure 9
Figure 9. Figure 9: Correlations of Attention and Inverted Hamming Distances ideal scenario where the least important KV pairs (those with the lowest attention scores) are dropped during cache compression. L m l,h = Xm i=1 a¯l,h,π(i) , (9) L m l,h,ref = Xm i=1 a¯l,h,σ(i) , (10) 21 [PITH_…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 7 canonical work pages

  1. [1]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

    arXiv 2023

  2. [2]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    arXiv 2022

  3. [3]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014

    arXiv 2014

  4. [4]

    Effective approaches to attention-based neural machine translation

    Minh-Thang Luong. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015

    arXiv 2015

  5. [5]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  6. [6]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  7. [7]

    Challenges in deploying long-context transformers: A theoretical peak performance analysis

    Yao Fu. Challenges in deploying long-context transformers: A theoretical peak performance analysis. arXiv preprint arXiv:2405.08944, 2024

    arXiv 2024

  8. [8]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023

    arXiv 2023

Show all 39 references
  1. [9]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing S...

    2024

  2. [10]

    Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache

    Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Atlas Wang. Q-hitter: A better token oracle for efficient llm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6: 0 381--394, 2024 b

    2024

  3. [11]

    A simple and effective l\_2 norm-based strategy for kv cache compression

    Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective l\_2 norm-based strategy for kv cache compression. arXiv preprint arXiv:2406.11430, 2024

    2024 arXiv

  4. [12]

    Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters

    Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. arXiv preprint arXiv:2406.12335, 2024

    2024 arXiv

  5. [13]

    Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers

    Sifan Long, Zhen Zhao, Jimin Pi, Shengsheng Wang, and Jingdong Wang. Beyond attentive tokens: Incorporating token importance and diversity for efficient vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10334--10343, 2023

    2023

  6. [14]

    Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming

    Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42 0 (6): 0 1115--1145, 1995

    1995

  7. [15]

    Similarity estimation techniques from rounding algorithms

    Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, pages 380--388, 2002

    2002

  8. [16]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  9. [17]

    Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    2024 arXiv

  10. [18]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    2023 arXiv

  11. [19]

    Formal algorithms for transformers

    Mary Phuong and Marcus Hutter. Formal algorithms for transformers. arXiv preprint arXiv:2207.09238, 2022

    2022 arXiv

  12. [20]

    Approximate nearest neighbor search in high dimensions

    Alexandr Andoni, Piotr Indyk, and Ilya Razenshteyn. Approximate nearest neighbor search in high dimensions. In Proceedings of the International Congress of Mathematicians: Rio de Janeiro 2018, pages 3287--3318. World Scientific, 2018

    2018

  13. [21]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Process...

    2024

  14. [22]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023

    2023 arXiv

  15. [23]

    Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454, 2024

    2024 arXiv

  16. [24]

    Minicache: Kv cache compression in depth dimension for large language models

    Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366, 2024 b

    2024 arXiv

  17. [25]

    Reformer: The efficient transformer

    Nikita Kitaev, ukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  18. [26]

    Kdeformer: Accelerating transformers via kernel density estimation

    Amir Zandieh, Insu Han, Majid Daliri, and Amin Karbasi. Kdeformer: Accelerating transformers via kernel density estimation. In International Conference on Machine Learning, pages 40605--40623. PMLR, 2023

    2023

  19. [27]

    Hyperattention: Long-context attention in near-linear time

    Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P Woodruff, and Amir Zandieh. Hyperattention: Long-context attention in near-linear time. arXiv preprint arXiv:2310.05869, 2023

    2023 arXiv

  20. [28]

    Qjl: 1-bit quantized jl transform for kv cache quantization with zero overhead

    Amir Zandieh, Majid Daliri, and Insu Han. Qjl: 1-bit quantized jl transform for kv cache quantization with zero overhead. arXiv preprint arXiv:2406.03482, 2024 a

    2024 arXiv

  21. [29]

    Subgen: Token generation in sublinear time and memory

    Amir Zandieh, Insu Han, Vahab Mirrokni, and Amin Karbasi. Subgen: Token generation in sublinear time and memory. arXiv preprint arXiv:2402.06082, 2024 b

    2024 arXiv

  22. [30]

    Fast transformer decoding: One write-head is all you need

    Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019

    1911 arXiv

  23. [31]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079, 2024

    2024 arXiv

  24. [32]

    Flexgen: High-throughput generative inference of large language models with a single gpu

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R \'e , Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, ...

    2023

  25. [33]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156--5165. PMLR, 2020

    2020

  26. [34]

    D 'ej avu: Kv-cache streaming for fast, fault-tolerant generative llm serving

    Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. D 'ej avu: Kv-cache streaming for fast, fault-tolerant generative llm serving. arXiv preprint arXiv:2403.01876, 2024

    2024 arXiv

  27. [35]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11 0 (14): 0 6421, 2021

    2021

  28. [36]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023

    2023 arXiv

  29. [37]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  30. [38]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  31. [39]

    Memory-efficient transformers via top- k attention

    Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. Memory-efficient transformers via top- k attention. arXiv preprint arXiv:2106.06899, 2021

    2021 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.