Pith. sign in

REVIEW 5 major objections 5 minor 47 references

AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AhaKV claims that accumulated attention scores used for KV cache eviction are biased toward early tokens, and that recent-window accumulation, adaptive softmax scaling, and value-vector priors remove the bias and preserve globally…

desk verdict AhaKV has a real empirical observation and a plausible value-prior heuristic, but its theoretical backing is invalid and the derived scaling goes the wrong way; treat the method as a heuristic variant of SnapKV, not a theoretically grounded result. read the letter →

arxiv 2506.03762 v1 pith:RYFQ6H6G submitted 2025-06-04 cs.CL cs.AI

Yifeng Gu , Zicong Jiang , Jianxiu Jin , Kailing Guo , Ziyang Zhang , Xiangmin Xu This is my paper
classification cs.CLcs.AI
keywords KVcacheevictionattentionbiaspositionalsoftmaxscalingvalue-priorlong-contextinferenceself-attentionLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

During generation, large language models store growing key-value caches, and deciding which tokens to keep is a memory bottleneck. AhaKV argues that the standard eviction score, the accumulated attention score, is intrinsically biased: because of the causal mask, later tokens are attended to by fewer queries, so their scores decay with position and early tokens crowd out important later context. The paper's three-part correction is to sum attention only over a recent window so every token contributes the same number of terms, rescale the softmax with an adaptive factor derived from the expected entropy of the attention distribution, and refine the resulting score by the smoothed magnitude of the value vectors. With those fixes, AhaKV reports that retained tokens spread across the whole sequence instead of piling up at the start, and that it matches or beats existing eviction methods at the same cache budget on long- and short-context benchmarks.

What carries the argument

The load-bearing object is the eviction score itself. The baseline is the accumulated attention score $S_j = \sum_{i=j}^{n} a_{i,j}$, a column sum of causal softmax attention; AhaKV replaces it with three co-designed pieces. Recent accumulation replaces the sum over all query rows with the sum over the nearest $r$ rows, so the score for every token contains the same number of terms and the monotone decay $E[S_{j+1}-S_j] < 0$ disappears. SG-softmax changes the softmax used in eviction scoring to $e^{\lambda x_i} / \sum_j e^{\lambda x_j}$, with $\lambda = \sqrt{2\log(i/k)/d}$ set by the requirement that the expected attention entropy equals the entropy of a distribution concentrated on $k$ tokens; this counteracts the flattening of softmax as the sequence grows. The value prior $\gamma_i$, obtained by average-pooling and max-normalizing the squared L2 norms of value vectors, multiplies the recent-window score so that tokens whose values carry large parameterized weight are harder to evict. These three pieces together define which keys and values stay in the cache at a fixed budget.

What would settle it

Compute the average entropy of attention rows in a real decoder model at sequence lengths from 128 to 2,048 for a head dimension of 64. Equation (8) predicts $\log i - 32$, which is negative across this whole range, while measured entropy must be nonnegative; an observation that the measured entropy does not follow that form would show the derivation behind $\lambda$ is not an equality, and would leave the adaptive scaling factor without its stated theoretical basis.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that column sums of causal attention, the scores used by methods such as H2O, are biased in expectation: for a standard random-attention model the expected difference $E[S_{j+1} - S_j]$ is negative, so each later token is statistically fated to receive a smaller eviction score and be evicted regardless of semantic importance. AhaKV removes the source of the decay by accumulating over the nearest $r$ rows, which gives every token the same number of contributing terms, and it counters the flattening of attention that comes with longer sequences by replacing the softmax used in eviction scoring with SG-softmax, that is $e^{\lambda x_i} / \sum_j e^{\lambda x_j}$ with $\lambda = \sqrt{2\log(i/k)/d}$ chosen so the expected attention entropy stays at the level of a budget of $k$ informative tokens. The resulting score is then multiplied by a normalized, average-pooled value-vector norm, so information carried by values, previously ignored by query-key-based eviction scores, also determines what is kept. The paper shows theoretically that recent-window accumulation removes the monotone decay and empirically that retained token indices become far more uniform across positions.

Load-bearing premise

The theoretical derivation assumes that the raw attention match scores behave like independent, identically distributed bell-shaped random variables and that a log-of-sum approximation is close; if real models violate that, the adaptive scaling factor will be calibrated to the wrong target.

Editorial extensions

If this is right

  • At a fixed cache budget, a model using AhaKV should keep useful tokens from the second half of long documents, where prior eviction methods systematically lose them; this should show up as more uniform retained-token indices and better passage-retrieval accuracy.
  • Because $\lambda$ depends only on sequence length and budget, the same scaling should transfer across models with the same head dimension, giving a parameter-free way to set the softmax temperature for eviction scoring.
  • The value-prior refinement is a cheap multiplicative correction that uses information already in the cache, so it can be layered onto other eviction heuristics; the paper's ablation suggests each component contributes independently to accuracy.
  • Large memory savings are compatible with the method: on a 32K-context run with a budget of 2,048 cached pairs per head, the paper reports reducing cache memory from roughly 5.98 GB to 0.39 GB while keeping accuracy near the full-cache level.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same positional-bias diagnosis should apply to any eviction metric built from causal row sums, not just the accumulated attention score; retrofitting AhaKV's recent-window accumulation into other eviction frameworks is a direct way to test that generality.
  • The entropy-based $\lambda$ argument depends on a Gaussian approximation; a practical study that scans $\lambda$ over a grid and compares against Eq. (12) would show when the theoretical value is near-optimal and when it is only a starting point.
  • Since $\lambda$ is computed from sequence length and budget only, the method should extend to contexts far longer than the multi-thousand-token benchmarks tested; a synthetic needle-retrieval experiment at 100K+ tokens would test whether the recent-window score still preserves globally scattered evidence.
  • The value prior could also be used as a signal for cache quantization or prefetching, since it is available per token before any eviction decision is made.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes AhaKV, a KV cache eviction method for LLM inference that targets positional bias in H2O-style accumulated attention scores. The method combines recent-window accumulation, a 'step gain' softmax scaling parameter λ derived from an information-entropy analysis, and a value-vector prior to refine eviction scores. The authors report LongBench and short-text experiments on LLaMA, Qwen, and Gemma models and claim state-of-the-art results among eviction methods at fixed cache budgets. The central theoretical derivation in Sec. 4.2 and Appendix A is intended to justify the adaptive scaling λ, and the experimental section states that Eq. (12) is used to initialize λ.

Significance. If the empirical results hold, AhaKV is an incremental but useful empirical contribution: it addresses a real positional bias in accumulated attention scores, makes use of previously ignored value information, and includes ablations for each proposed component. The paper also tests across several model families and scales, and it provides a sensible discussion of compatibility with FlashAttention. However, the significance is severely undercut by the invalid theoretical derivation. Equation (8) yields impossible negative values for expected attention entropy at standard head dimensions, and the derived λ in Eq. (12) is smaller than 1 in typical settings, meaning SG-softmax would flatten rather than sharpen the attention distribution, which is opposite to the paper's stated goal. The state-of-the-art claim therefore rests on an adaptive mechanism that is not supported by the provided theory.

major comments (5)
  1. [Appendix A, Eqs. (16)-(29)] The derivation of E[H_i] replaces E[log Σ_j e^{w_j}] with log i + log E[e^w] and E[(Σ_j e^{w_j} w_j)/(Σ_j e^{w_j})] with E[e^w w]/E[e^w]. Neither replacement is an identity: for i.i.d. Gaussian logits, log-sum-exp is dominated by the maximum term, which grows like O(√(2 log i)), not O(log i), and the ratio of expectations is not the expectation of the ratio. Therefore Eq. (8), E[H_i] = log i - d/2, is not established.
  2. [Sec. 4.2, Eq. (8)] The formula E[H_i] = log i - d/2 gives impossible values for typical attention head dimensions. For d=128 and i=3600, this is approximately 8.19 - 64, which is negative, and the expression is negative for essentially all i when d=128. Since H_i is an information entropy, it cannot be negative. This is not a loose approximation but a sign of a fundamentally incorrect derivation, and it invalidates the subsequent Eqs. (10) and (12).
  3. [Sec. 4.2, Eq. (12)] Even if the algebra were correct, the derived scaling goes in the wrong direction. For d=128, i=3600, and k=32, Eq. (12) gives λ ≈ 0.27. Since SG-softmax is e^{λx}/Σ e^{λx}, a value λ<1 makes the attention distribution flatter than the standard softmax, directly contradicting the stated goal of 'step gain' and emphasizing key tokens. Thus the adaptive mechanism as specified moves eviction scores away from the intended unbiased emphasis.
  4. [Sec. 4.2, Eq. (11)] The target E[H_i] = log k is imposed ad hoc to solve for λ. Setting the expected entropy equal to the maximum entropy over k tokens is a design choice, not a derived property of optimal eviction. Consequently, the 'prediction' of λ in Eq. (12) is essentially a normalization to the eviction budget k, making the claimed adaptivity circular with respect to the budget.
  5. [Algorithm 1 and Sec. 5.2.1] The adaptive claim is not operationalized consistently. The pseudocode calls SG-softmax(A) without a λ argument, while the experimental section says Eq. (12) is used to 'initialize' the parameter. If λ is a single scalar fixed at initialization, it cannot adapt per row or per head as the title and motivation promise; if it is recomputed per row, neither the pseudocode nor the experimental description specifies this. The paper should clarify how λ is actually set during inference.
minor comments (5)
  1. [Sec. 4.2, Eq. (9)] Equation (9) writes softmax(x_i) as x_i / Σ e^{x_i} and SG-softmax as λx_i / Σ e^{λx_i}; both should have e^{x_i} and e^{λx_i} in the numerator, respectively.
  2. [Sec. 5.2.1] The text says 'As shown in Table 3' when referring to the main LongBench results, but Table 3 is in Appendix B; the results appear in Table 1.
  3. [Sec. 4.1, Eq. (5)] Equation (5) ensures an equal number of accumulated terms only for tokens before the recent window; tokens inside the recent window have fewer terms. This is not a practical problem because recent tokens are protected by the recent budget, but the statement that the score is 'not influenced by the number of accumulated terms' should be qualified.
  4. [Sec. 5.2.2 and Appendix E] There are typos: 'SampeKV' should be 'SnapKV' in Sec. 5.2.2, 'FalshAttention' should be 'FlashAttention' in Appendix E, and 'casual mask' should be 'causal mask' in Sec. 4.1.
  5. [Limitations section] The Limitations section appropriately acknowledges that experiments were not conducted on longer texts and that the recent-window choice is not unique; however, the abstract and conclusion claim that AhaKV preserves global context, so the claims should be qualified to the tested sequence lengths.

Circularity Check

1 steps flagged · score 3.0 of 10

The softmax scaling parameter λ is solved by imposing the target entropy log k, so the adaptive scale is a budget-normalization by construction; the empirical benchmark results remain independent evidence.

  1. self definitional [Sec. 4.2, Eqs. (10)-(12); Appendix A]
    "Then, the expectation of H i changes into E[Hi] = logi − λ2d/2. (10) Assume that the token number of the budget is k. The ideal case for a good eviction score is that the there are only k attention score larger than zero. We aim to make expected information entropy equals to the maximum information entropy regardless of the number of total tokens. Therefore, we have logi − λ2d/2 = − Σ_{j=0}^{k} 1/k log 1/k = −log 1/k. (11) We can figure out that, when the number of total tokens is i, λ = sqrt(2 log(i/k)/d). (12)"

    The parameter λ is not determined by independent statistical evidence; it is defined as the value that forces the expected attention entropy in Eq. (10) to equal log k, the maximum entropy of a distribution over the k-token budget. Eq. (12) is therefore the algebraic solution of Eq. (11), so the 'adaptive' scaling is a normalization to the budget by construction. The stated justification — 'the ideal case ... only k attention score larger than zero' — is the very target being imposed, not a consequence derived from attention statistics. This is a self-referential step in the theoretical framing, though the external benchmark comparisons provide independent evidence for the empirical claims.

full rationale

The central empirical claim — state-of-the-art results on LongBench and short-text benchmarks against H2O, SnapKV, NACL, and TOVA — is evaluated on external benchmarks and does not reduce to the paper's own equations. I found no load-bearing self-citation and no fitted parameter renamed as a prediction. The only constructional self-reference is Eq. (12): λ is obtained by setting the expected attention entropy (Eq. 10) equal to log k, the maximum entropy of the k-token budget. That makes the 'adaptive' scale a normalization to the budget rather than an independently derived constant; the paper's own wording, 'We aim to make ...', shows the target is imposed rather than predicted. This is a mild self-definitional step in the theoretical presentation, not a reduction of the empirical results. Separately, the Appendix A derivation has a mathematical validity problem: with d=128, the claimed E[H_i] = log i − d/2 is negative for realistic i, which no entropy can be; I treat that as a correctness issue, not as circularity. Accordingly the score is moderate, but the empirical contributions are not circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method depends on two fitted or arbitrary parameters (lambda and recent window size), one unspecified pooling window, and several assumptions about attention logit distributions that are either unrealistic or mathematically unjustified.

free parameters (3)
  • lambda (SG-softmax scaling) = sqrt(2 log(i/k)/d) from Eq. 12
    Scaling parameter for softmax, derived by equating the assumed expected attention entropy to log k, the cache budget. The target entropy is an ad hoc design choice, and the derivation is mathematically invalid.
  • recent window size r (B_r) = 32
    Number of recent rows used for accumulation; fixed across all experiments, no sensitivity analysis.
  • Average-pooling window for value prior = unspecified
    The kernel size for Avgpool in Eq. 13 is not stated, leaving an unstated hyperparameter.
assumptions (3)
  • domain assumption Q and K components are independent random variables with mean 0 and variance 1 (Sec. 4.1).
    Assumed to prove monotonic decrease of accumulated attention scores; unrealistic for trained models but common in theoretical analyses.
  • ad hoc to paper E[log sum e^{w_j}] = log i + log E[e^w] (Appendix A).
    Unproved and generally false; the expectation of log-sum-exp is not the log of the expectation.
  • ad hoc to paper The ideal eviction score should make attention entropy equal to log k, the maximum entropy over k tokens (Eq. 11).
    Arbitrary target chosen to derive lambda; no independent evidence that this is the right objective.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models." pith.science (2026). https://pith.science/paper/RYFQ6H6G

@misc{pith2026250603762,
  author       = {Pith},
  title        = {Pith review of: AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYFQ6H6G}},
  note         = {Machine review of arXiv:2506.03762}
}
read the original abstract

Large Language Models (LLMs) have significantly advanced the field of Artificial Intelligence. However, their deployment is resource-intensive, not only due to the large number of model parameters but also because the (Key-Value) KV cache consumes a lot of memory during inference. While several works propose reducing the KV cache by evicting the unnecessary tokens, these approaches rely on accumulated attention score as eviction score to quantify the importance of the token. We identify the accumulated attention score is biased and it decreases with the position of the tokens in the mathematical expectation. As a result, the retained tokens concentrate on the initial positions, limiting model's access to global contextual information. To address this issue, we propose Adaptive holistic attention KV (AhaKV), it addresses the bias of the accumulated attention score by adaptively tuning the scale of softmax according the expectation of information entropy of attention scores. To make use of the holistic attention information in self-attention mechanism, AhaKV utilize the information of value vectors, which is overlooked in previous works, to refine the adaptive score. We show theoretically that our method is well suited for bias reduction. We deployed AhaKV on different models with a fixed cache budget. Experiments show that AhaKV successfully mitigates bias and retains crucial tokens across global context and achieve state-of-the-art results against other related work on several benchmark tasks.

Figures

Figures reproduced from arXiv: 2506.03762 by the authors.

Figure 1
Figure 1. Left: The H2O-base method, which uses the accumulated attention score as an eviction score, has a higher eviction score for the token on the left side of the sequence. Right: The AhaKV eviction strategy, which focuses the attention scores on key tokens and selects the attention scores of key tokens to calculate the eviction scores, avoiding positional bias. lated attention scores as the eviction score during generat… view at source ↗
Figure 2
Figure 2. (a,b) shows the eviction scores in different strategy. (c) shows the retention token index. (d) shows the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. AhaKV consists of three main components: SG-softmax, Recent Accumulation, and Value-Prior Refine. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Short-text results in different KV budget. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Retain token index in Qwen2-7B-Inst. The computation of E(xex ) is shown as the Equation 24. E(xex ) = 1 σ √ 2π Z ∞ −∞ xex · e − (x−µ) 2 2σ2 dx (24) = 1 σ √ 2π Z ∞ −∞ x · e x− (x−µ) 2 2σ2 dx (25) To simplify the Equation 25, we adjust the x − (x−µ) 2 2σ2 as shown in th…
Figure 7
Figure 7. Figure 7: Distribution of the results of different eviction [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Inference speed in different budget in Qwen2- [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 33 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of the Seventh Annual Conference on Machine Learning and Systems, 6:114--127

    work page 2024

  3. [3]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 3119--3137

    work page 2023

  4. [4]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150

    arXiv 2020

  5. [5]

    Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. 2024 a . NACL : A general and effective KV cache eviction framework for LLM at inference time. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7913--7926

    work page 2024

  6. [6]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024 b . Longlora: Efficient fine-tuning of long-context large language models. International Conference on Learning Representations

    work page 2024

  7. [7]

    Ta-Chung Chi, Ting-Han Fan, Li-Wei Chen, Alexander Rudnicky, and Peter Ramadge. 2023. Latent positional information is in the self-attention variance of transformer language models without positional embeddings. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1183--1193

    work page 2023

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

Show all 47 references
  1. [9]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344--16359

    2022

  2. [10]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3.int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318--30332

    2022

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, volume 1, ...

    2019

  4. [12]

    Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. 2024. Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference. International Conference on Machine Learning

    2024

  5. [13]

    Linhao Dong, Shuang Xu, and Bo Xu. 2018. Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. In IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5884--5888

    2018

  6. [14]

    Alexey Dosovitskiy. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations

    2021

  7. [15]

    Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323--10337

    2023

  8. [16]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ : Accurate post-training compression for generative pretrained transformers. International Conference on Learning Representations

    2023

  9. [17]

    Google. 2024. Gemma. https://github.com/google-deepmind/gemma

    2024

  10. [18]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. Advances in Neural Information Processing Systems

    2024

  11. [19]

    Shigeki Karita, Nelson Enrique Yalta Soplin, Shinji Watanabe, Marc Delcroix, Atsunori Ogawa, and Tomohiro Nakatani. 2019. Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration. In Annual Conference o...

    2019

  12. [20]

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2024. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36

    2024

  13. [21]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems

    2024

  14. [22]

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024 a . World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268

    2024 arXiv

  15. [23]

    Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023. Llm-qat: Data-free quantization aware training for large language models. Findings of the Association for Computational Linguis...

    2023

  16. [24]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2024 b . Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Informatio...

    2024

  17. [25]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 c . Kivi: A tuning-free asymmetric 2bit quantization for kv cache. International Conference on Machine Learning

    2024

  18. [26]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

    2023

  19. [27]

    Meta. 2024. Llama. https://www.llama.com/

    2024

  20. [28]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for openbook question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - N...

    2018

  21. [29]

    Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz. 2024. Transformers are multi-state rnns. arXiv preprint arXiv:2401.06104

    2024 arXiv

  22. [30]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. Yarn: Efficient context window extension of large language models. International Conference on Learning Representations

    2024

  23. [31]

    Mohammad Taher Pilehvar and Jos \' e Camacho - Collados. 2019. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages...

    2019

  24. [32]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606--624

    2023

  25. [33]

    Qwen. 2024. Qwen. https://huggingface.co/Qwen

    2024

  26. [34]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1--67

    2020

  27. [35]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020 , pages 8732--8740

    2020

  28. [36]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A simple and effective pruning approach for large language models. International Conference on Learning Representations

    2024

  29. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  30. [38]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [39]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, pages 5998--6008

    2017

  32. [40]

    Hanrui Wang, Zhekai Zhang, and Song Han. 2021. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 97--110

    2021

  33. [41]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. International Conference on Learning Representations

    2024

  34. [42]

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. 2021. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077--12090

    2021

  35. [43]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

    2020

  36. [44]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  37. [45]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, et al. 2024. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Proces...

    2024

  38. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.