Pith. sign in

REVIEW 4 major objections 7 minor 49 references

EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A head-wise evict-then-merge strategy with a Global-Local importance score compresses KV cache to a 256-token budget while improving LongBench scores by at least 1.28 points over baselines on four LLMs.

desk verdict A solid, incremental KV cache compression paper with a sensible global-local score and a neat zero-class trick, but the merge-destination proxy needs validation before I trust the LongChat numbers. read the letter →

arxiv 2412.08521 v2 pith:JGC5WNF2 submitted 2024-12-11 cs.CL

Yingxin Li , Ye Li , Yuan Meng , Xinzhu Ma , Zihan Geng , Shutao Xia , Zhi Wang This is my paper · ORCID
classification cs.CL
keywords KVcachecompressionlong-contextinferencetokenimportanceevict-then-mergeglobal-localattentionredundancyLongBenchneedle-in-a-haystack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that KV cache compression under extreme budgets fails less because important tokens are hard to find than because token selection is biased and compression ignores head-wise redundancy. It proposes EMS, which scores tokens with a Global-Local importance measure, the element-wise maximum of aligned global and local accumulated attention, and then, per head, evicts the least important tokens and merges sub-important tokens into a small set of class centers, using a zero-class entry so eviction is just another merge. Across four LLMs on LongBench, EMS reports the best average scores at a 256-token cache budget, improving over existing baselines by at least 1.28 points, and it preserves about 95% of full-cache retrieval accuracy in Needle-in-a-Haystack with a budget under 2% of context length. The practical payoff is that an LLM can hold a much smaller KV cache, support larger batches, and run long contexts without the usual quality collapse.

What carries the argument

The load-bearing object is the combination of a Global-Local importance score and a head-wise Evict-then-Merge strategy that treats eviction as merging into a zero class. The score is $\mathbf{s}_{\mathrm{Glo{-}Loc}} = \max(\mathbf{s}_{\mathrm{Glo}} \cdot (\sum \mathbf{s}_{\mathrm{Loc}}/N)/(\sum \mathbf{s}_{\mathrm{Glo}}/N), \mathbf{s}_{\mathrm{Loc}})$, where $\mathbf{s}_{\mathrm{Glo}}$ and $\mathbf{s}_{\mathrm{Loc}}$ are the accumulated attention weights from all queries and from a local window respectively; the mean-alignment term removes the scale gap that otherwise pushes selection to the front or the end of the context. The merge criterion is the redundancy $R_{i,j} = \cos(\mathbf{k}_i, \mathbf{k}_j) \cdot \cos(\mathbf{v}_i, \mathbf{v}_j)$, which is high only when both the key and the value of a candidate token resemble those of a class center; tokens above threshold $\tau$ are merged, tokens below it are sent to the zero-class token, and a position look-up table expands the small stored cache back to $\gamma N_{\mathrm{budget}}$ entries at computation time. This construction is what lets each head choose a different evict/merge ratio while still running in parallel.

What would settle it

Run a trained LLM on a held-out prompt, compute the redundancy of Eq. 4 for all token pairs in a layer, merge only those pairs whose redundancy is just above the threshold, and compare the model's next-token distribution to the unmerged model; if these high-redundancy merges cause larger output changes than simply discarding the same number of tokens, then the product-of-cosine-similarities criterion is not a faithful measure of merge safety.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that extreme KV-cache compression is limited less by how many tokens one keeps than by two solvable defects: importance scores that are biased toward early or recent tokens, and compression strategies that ignore how much redundancy varies from head to head. EMS fixes the first with a Global-Local score that aligns the mean magnitudes of the globally accumulated attention vector and the locally accumulated attention vector and takes their element-wise maximum, so retained tokens are drawn from both perspectives. It fixes the second with an adaptive Evict-then-Merge procedure: within each head, the least important tokens are evicted, the most important tokens serve as class centers, and sub-important tokens are merged into the nearest center only when the product of their key and value cosine similarities exceeds a threshold; a zero-class center turns eviction into a special case of merging so all heads can be compressed in parallel. The paper reports that this combination holds up precisely where prior methods break down: at a 256-cache budget across four LLMs on LongBench, under 2% context-length budgets on Needle-in-a-Haystack, and with up to 10.75x throughput gains over a fully cached FlashAttention2 model.

Load-bearing premise

The argument rests on assuming that high cosine similarity between a token's key and value and a class center's key and value means the two tokens are interchangeable enough that merging them does not change what the model generates.

Editorial extensions

If this is right

  • At the fixed 256-token-per-head budget, EMS reports the lowest perplexity and the best LongBench average over StreamingLLM, H2O, CAM, and SnapKV on all four tested LLMs, improving the average by at least 1.28 points and by 17.64 points on LongChat.
  • In Needle-in-a-Haystack, EMS preserves around 95% of full-cache retrieval accuracy using a cache budget below 2% of context length, tracking the performance of much larger caches.
  • The per-head eviction and merging is parallelizable through the zero-class mechanism, so the method is not just a quality improvement: with a 256-token budget it supports up to 16 batches and reaches 10.75x higher throughput than a fully cached FlashAttention2 model on 4k prompt plus 8k generation.
  • Because eviction is treated as merging into a zero-class entry, the same code path handles heads with different merge/evict ratios, and the position look-up table keeps the stored cache at a constant size per head.
  • Ablations attribute the gain to both components: Global-Local scoring over global or local alone, and Evict-then-Merge over evict-only, so the claim is that each design choice contributes separately.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the cosine-product redundancy faithfully predicts merge safety, the same merge machinery could be applied to other stateful transformer caches, such as cross-attention memories or recurrent state summaries, not just autoregressive KV caches.
  • The mean-alignment step is a heuristic; a model- or task-dependent calibration of global versus local importance could make token selection more robust, especially for retrieval-heavy prompts where one perspective dominates.
  • The equal-per-head budget suggests a natural combination with layer-wise or head-wise budget-allocation methods; EMS's own numbers at 1024-token budgets indicate the method would likely benefit from allocating more entries to low-sparsity, low-redundancy heads.
  • A testable extension is to run EMS on tasks with deliberate distractors or on models with grouped-query attention to see whether the head-wise merge decisions transfer when heads share KV projections.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper proposes EMS, a training-free KV cache compression method. It computes a Global-Local importance score by mean-aligning accumulated global and local attention weights and taking an elementwise max; it then partitions tokens into irrelevant, to-be-merged, and important sets, evicts the first set, merges the second set into class centers chosen by a product of key and value cosine similarities, and uses a zero-class center to unify eviction and merging in a head-wise parallel manner. The method is evaluated on LongBench, Needle-in-a-Haystack, PG19 perplexity, and throughput, using LLaMA-2, LLaMA-3, LongChat, and Mistral, with reported state-of-the-art results under a 256-token cache budget.

Significance. If the reported results are reproducible, the paper makes a practical contribution: it combines eviction and merging at the head level rather than relying on eviction alone, and it integrates the importance-score computation into FlashAttention2, enabling a measured 10.75x throughput gain in a favorable setting. The zero-class mechanism is an elegant way to keep head-wise merge and evict decisions parallel, and the evaluation spans four LLMs and three benchmark families, which is broader than many KV-cache compression papers. However, the central quantitative claim currently rests on hyperparameters selected using the same LongBench benchmark on which the headline numbers are reported, and on a merge-redundancy proxy that is not directly validated against attention-output error. These issues need to be addressed before the state-of-the-art claim can be accepted.

major comments (4)
  1. [§5.2 and Appendix B, Tables 6 and 7] The hyperparameters τ, γ, ζ, Lwin, and kernel_size are selected through ablations on LongBench, and the same LongBench average is then used as the headline result in Table 1. There is no held-out validation split or nested selection procedure, and Table 1 reports the better of two EMS variants per model, which further increases the risk of fitting to the test set. Section 7 explicitly concedes that the Global-Local alignment is empirical rather than theoretical, which makes a fixed, prespecified hyperparameter choice particularly important. Please provide results with hyperparameters chosen without access to LongBench test labels, or use a separate validation set, and report a single EMS variant or a principled selection rule.
  2. [§4.1, Eq. (4), Table 8] The redundancy metric used to choose merge destinations is a query-independent product of cosine similarities. The paper validates only key-only, value-only, and key-value variants of this same metric; it does not compare Eq. (4) against random merge destinations, against an attention-output-error-based assignment, or against an evict-only baseline with the same candidate pool. A token with high cosine similarity to a center can still produce different softmax contributions under the queries that actually attend to it, and errors can propagate across layers. Since Table 4 attributes part of the LongBench gain specifically to the Evict-then-Merge stage, this proxy is load-bearing. Please add a direct comparison of attention-output error induced by Eq. (4) versus alternative merge-destination rules, and an end-to-end comparison to an evict-only method with the same candidate selection.
  3. [§5.2, Table 1] The Table 1 caption states that SnapKV is an exception and increases its budget at the decoding stage, so the comparison is not at a uniform 256-token cache budget. The title claim of improving scores under a 256 cache budget is weakened if the main baseline is allowed to exceed that budget during generation. Please report SnapKV's effective peak cache size during decoding, or enforce a hard 256 budget for all methods including newly generated tokens. In addition, the LongChat-7B row shows an unusually large gap (EMS 31.94 vs. SnapKV 14.90 at budget 256); please verify that the LongChat results are produced with the official LongBench evaluation protocol and, if the gap persists, discuss the qualitative reason.
  4. [§4.2, Eq. (3)] The compression procedure is not fully specified. Equation (3) defines a head-wise sparsity rate p_m, and Figure 4b reports sparsity and redundancy, but the text does not explain how p_m or r_m determines N_irr, N_tbm, or N_imp in the token partition. In practice the number of merged tokens is controlled by the externally chosen γ = (N_budget + N_tbm) / N_budget; the connection between the observed head-wise statistics and the actual per-head evict/merge ratios is therefore not demonstrated. Please state the algorithm precisely, including how ζ and p_m are used at inference time, or clarify that p_m and r_m are only motivational observations.
minor comments (7)
  1. [§5.5, Table 4] The text states that Evict-then-Merge improves the score by 0.61 points and that combining global and local scores improves by 0.37 and 2.62 points, but the table shows differences of 0.52, 0.28, and 2.53 points. Please correct the text or the table so the ablation claims match the reported numbers.
  2. [Appendix B, Merge Threshold] The first sentence of the Merge Threshold paragraph says a smaller threshold results in more tokens being evicted and a larger threshold leads to more aggressive merging; this is reversed, because a token is merged when its redundancy exceeds τ.
  3. [§3.3, Eq. (2)] The notation and the window-size statement need clarification: the sums in Eq. (2) should be defined explicitly, and the sentence that the actual window size is "Lwin ∼ 2Lwin − 1" should be replaced by a precise statement of the effective window size used to compute the local score.
  4. [§4.2] The "mean pooling function" applied to the Global-Local score is not defined; please specify the kernel size, stride, and whether pooling is applied along the token dimension for each head.
  5. [§5.3, Table 2] Table 2 does not report the full-cache retrieval accuracy that is needed to interpret the claim that EMS retains 95.9% of full-cache retrieval ability. Please add the full-cache score and state the exact cache budget corresponding to the abstract's "less than 2% of context length" claim.
  6. [§5.4, Table 3] The throughput comparison is only against the full-cache FlashAttention2 implementation; reporting latency and throughput for at least one compression baseline (e.g., SnapKV or H2O) would help characterize the overhead of the modified kernel and the look-up-table expansion.
  7. [Figure 7 caption] The caption refers to Mistral-7B-Instruct-v1.5, while the experiments in §5.3 use Mistral-7B-Instruct-v0.2; please correct the model name.

Circularity Check

1 steps flagged · score 4.0 of 10

LongBench result is partially circular: tau and gamma are tuned on LongBench before the same LongBench table is reported as evidence; the core design assumptions are otherwise empirical, not derived.

  1. fitted input called prediction [Section 5.2 'Performance Evaluation on LongBench'; Appendix B, Tables 6 and 7]
    "To demonstrate the effectiveness of EMS under extreme KV cache compression, we enforce a strict compression budget of 256, with τ and γ set to approximately 0.6 and 4. ... Table 6 presents the results of varying the merge threshold, indicating that a moderate threshold τ = 0.6, yields best performance on balance."

    The central LongBench claim is produced after selecting the merge threshold and merge magnification factor on LongBench itself: Table 6 chooses τ=0.6 as the best LongBench threshold, Table 7 chooses γ=4 as the best LongBench setting, and then Table 1 uses these same constants to report the 'over 1.28 points' LongBench improvement. The hyperparameters are fitted to the evaluation benchmark, so the headline LongBench result is a selected outcome rather than an independent prediction of the method. The claim is not forced by an equation, but it is a fitted-input-reported-as-prediction pattern that positively biases the central comparison.

full rationale

Section 7 explicitly admits that the Global-Local alignment is 'derived from empirical observations rather than a theoretical foundation,' so the paper does not claim a first-principles derivation for that component. The redundancy criterion in Eq. 4 is also an assumption ('Cosine similarity is a reliable metric'), not a result derived from the target quantity; its weakness is a correctness risk rather than circularity. The zero-class unification cites two prior works by the same authors, but only as inspiration for a technique, and no uniqueness theorem is imported, so this self-citation is not load-bearing. The concrete circular step is benchmark fitting: τ and γ are ablated on LongBench and then the same LongBench table is used as the main evidence. Independent content remains in PG19 perplexity, Needle-in-a-Haystack retrieval, and throughput experiments, and the method also uses different NIAH settings (τ=0.55), which prevents the overall circularity from being total.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on several hand-chosen hyperparameters and the empirical heuristic that attention scores and cosine similarities identify KV tokens that can be discarded or merged without quality loss.

free parameters (5)
  • tau (merge threshold) = 0.6 for LongBench/PG19, 0.55 for NIAH
    Controls which TBM tokens are merged vs evicted; ablated in Appendix B, chosen for best LongBench score.
  • gamma (merge magnification factor) = 4
    Controls how many tokens are expanded relative to cache budget; ablated in Appendix B.
  • zeta (sparsity threshold) = 0.95
    Used in Eq. 3 to define sparsity rate; chosen by hand, not ablated.
  • Lwin (local window size) = 32 for perplexity/LongBench, 16 for NIAH
    Window size for local attention score; chosen without ablation.
  • kernel_size = 7
    Pooling kernel size for Global-Local score; chosen without ablation.
assumptions (4)
  • domain assumption Accumulated attention scores over all tokens and over a local window are valid proxies for token importance.
    Used in Section 3.3 to define the Global-Local score; this is an empirical heuristic.
  • domain assumption Cosine similarity between raw key and value vectors is a reliable measure of token redundancy for merging.
    Used in Eq. 4 and the merge step in Section 4.2. The paper only tests alternative similarity definitions internally, not the premise itself.
  • domain assumption Retaining half the tokens from the start and the end of the context approximates the full global attention score on long contexts.
    Used in Section 5.2 to compute global scores on a single GPU, following Lost in the Middle (Liu et al. 2024a).
  • ad hoc to paper Mean-alignment of global and local scores makes them comparable, and the element-wise max yields a balanced importance.
    Eq. 2; the authors state in Limitations that this is derived from empirical observations rather than theory.
invented entities (1)
  • Zero-class center
    purpose: A virtual KV entry that receives evicted tokens, turning eviction into a special case of merging so that head-wise compression can be computed in parallel.
    Introduced in Section 4.2. Its benefit is demonstrated only through the paper's own experiments; no external verification is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance." pith.science (2026). https://pith.science/paper/JGC5WNF2

@misc{pith2026241208521,
  author       = {Pith},
  title        = {Pith review of: EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JGC5WNF2}},
  note         = {Machine review of arXiv:2412.08521}
}
read the original abstract

As large language models (LLMs) continue to advance, the demand for higher quality and faster processing of long contexts across various applications is growing. KV cache is widely adopted as it stores previously generated key and value tokens, effectively reducing redundant computations during inference. However, as memory overhead becomes a significant concern, efficient compression of KV cache has gained increasing attention. Most existing methods perform compression from two perspectives: identifying important tokens and designing compression strategies. However, these approaches often produce biased distributions of important tokens due to the influence of accumulated attention scores or positional encoding. Furthermore, they overlook the sparsity and redundancy across different heads, which leads to difficulties in preserving the most effective information at the head level. To this end, we propose EMS to overcome these limitations, while achieving better KV cache compression under extreme compression ratios. Specifically, we introduce a Global-Local score that combines accumulated attention scores from both global and local KV tokens to better identify the token importance. For the compression strategy, we design an adaptive and unified Evict-then-Merge framework that accounts for the sparsity and redundancy of KV tokens across different heads. Additionally, we implement the head-wise parallel compression through a zero-class mechanism to enhance efficiency. Extensive experiments demonstrate our SOTA performance even under extreme compression ratios. EMS consistently achieves the lowest perplexity, improves scores by over 1.28 points across four LLMs on LongBench under a 256 cache budget, and preserves 95% retrieval accuracy with a cache budget less than 2% of the context length in the Needle-in-a-Haystack task.

Figures

Figures reproduced from arXiv: 2412.08521 by the authors.

Figure 1
Figure 1. The KV cache compression workflow of EMS. The tokens are first partitioned according to the rank￾ing of Global-Local score, which is calculated based on attention weights of global and local tokens. The least important tokens are then evicted, while the sub￾important tokens are either merged into most important tokens or evicted by merging into the zero-class token. (Ram et al., 2023; Wei et al., 2022; Roziere et al… view at source ↗
Figure 2
Figure 2. Token selection patterns. The sample is taken [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The framework of EMS. The compression of KV cache is decoupled into two parts. For important KV selection policy, a balanced Global-Local score is designed to grasp token importance. For KV compression strategy, the Evict-then-Merge approach first removes irrelevant tokens, then applies a unified head-wise eviction and merging process. them into three types that impact compression: local-only bias, local-aware bias,… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Observations on sparsity and redundancy. The parameters [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Evict-then-Merge details. (a) Two levels of [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Perplexity across different cache budgets. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Pressure testing results on Mistral-7B-Instruct [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The distribution of selected tokens. The sam [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Token selection patterns visualization on different layers, datasets and models. Global-aware selection [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 23 canonical work pages

  1. [1]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114--127

    2024

  2. [2]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245

    arXiv 2023

  3. [3]

    Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. 2023. Dynamic context pruning for efficient and interpretable autoregressive transformers. In Thirty-seventh Conference on Neural Information Processing Systems

    work page 2023

  4. [4]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403

    arXiv 2023

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  6. [6]

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2022. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461

    arXiv 2022

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  8. [8]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. 2024. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. CoRR

    work page 2024

Show all 49 references
  1. [9]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

    2023 arXiv

  2. [10]

    Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. 2024 a . NACL : A general and effective KV cache eviction framework for LLM at inference time. In Proceedings of the 62nd Annual Meeting of the Associ...

    2024

  3. [11]

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024 b . Longlo RA : Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations

    2024

  4. [12]

    Tri Dao. 2024. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR)

    2024

  5. [13]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  6. [14]

    Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. 2024. Get more with LESS : Synthesizing recurrence with KV cache compression for efficient LLM inference. In Forty-first International Conference on Machine Learning

    2024

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. 2025. Not all heads matter: A head-level KV cache compression method with integrated retrieval and reasoning. In The Thirteenth International Conference on Learning Representations

    2025

  9. [17]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive KV cache compression for LLM s. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WAN...

    2023

  10. [18]

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa...

    2021

  11. [19]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  12. [20]

    Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325

    2024 arXiv

  13. [21]

    G Kamradt. 2023. Needle in a haystack--pressure testing llms

    2023

  14. [22]

    Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023 a . How long can context length of open-source LLM s truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following

    2023

  15. [23]

    Ye Li, ZhongXin Liu, Ge Lan, Malika Sader, and ZengQiang Chen. 2023 b . A ddpg-based solution for optimal consensus of continuous-time linear multi-agent systems. Science China Technological Sciences, 66(8):2441--2453

    2023

  16. [24]

    Ye Li, Chen Tang, Yuan Meng, Jiajun Fan, Zenghao Chai, Xinzhu Ma, Zhi Wang, and Wenwu Zhu. 2024 a . Prance: Joint token-optimization and structural channel-pruning for adaptive vit inference. arXiv preprint arXiv:2407.05010

    2024 arXiv

  17. [25]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024 b . Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469

    2024 arXiv

  18. [26]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 a . Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

    2024

  19. [27]

    Zhongxin Liu, Ye Li, Ge Lan, and Zengqiang Chen. 2024 b . A novel data-driven model-free synchronization protocol for discrete-time multi-agent systems via td3 based algorithm. Knowledge-Based Systems, 287:111430

    2024

  20. [28]

    Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Thirty-seventh Conference on ...

    2023

  21. [29]

    Piotr Nawrot, Adrian a \'n cucki, Marcin Chochowski, David Tarjan, and Edoardo Ponti. 2024. Dynamic memory compression: Retrofitting LLM s for accelerated inference. In Forty-first International Conference on Machine Learning

    2024

  22. [30]

    Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. 2025. CAKE : Cascading and adaptive KV cache eviction with layer preferences. In The Thirteenth International Conference on Learning Representations

    2025

  23. [31]

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. 2019. Compressive transformers for long-range sequence modelling. arXiv preprint

    2019

  24. [32]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316--1331

    2023

  25. [33]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  26. [34]

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Sch \"a rli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210--31227. PMLR

    2023

  27. [35]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  28. [36]

    Hanrui Wang, Zhekai Zhang, and Song Han. 2021. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 97--110. IEEE

    2021

  29. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  30. [38]

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2025. Duoattention: Efficient long-context LLM inference with retrieval and streaming heads. In The Thirteenth International Conference on Learning Representations

    2025

  31. [39]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations

    2024

  32. [40]

    Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024. P yramid I nfer: Pyramid KV cache compression for high-throughput LLM inference. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3258--3270, Bangkok, Thailand. Associat...

    2024

  33. [41]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600

    2018 arXiv

  34. [42]

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. 2022. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809--10818

    2022

  35. [43]

    Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. 2024. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. In Forty-first International Conference on Machine Learning

    2024

  36. [44]

    Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, et al. 2024. Llm inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363

    2024 arXiv

  37. [45]

    Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. 2024. Cam: Cache merging for memory-efficient llms inference. International Conference on Machine Learning

    2024

  38. [46]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Thirty-sevent...

    2023

  39. [47]

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. 2024. A survey on efficient inference for large language models. ArXiv, abs/2404.14294

    2024 arXiv

  40. [48]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.