Pith. sign in

REVIEW 6 major objections 4 minor 42 references

HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs

T0 review · 6 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HCAttention claims that combining key quantization, CPU value offloading, and dynamic token eviction shrinks the LLM KV cache to 12.5% of its original size while keeping LongBench accuracy within one point of the full-attention baseline.

desk verdict A plausible integration of key quantization and CPU value offloading, but the 'eviction' is really per-query sparsification and the LongBench numbers are tuned on themselves. read the letter →

arxiv 2507.19823 v1 pith:RBZF6FO7 submitted 2025-07-26 cs.CL cs.AIcs.LG

Dongquan Yang , Yifan Yang , Xiaotian Yu , Xianbiao Qi , Rong Xiao This is my paper
classification cs.CLcs.AIcs.LG
keywords KVcachecompressionlong-contextLLMinferenceapproximateattentionkeyquantizationvalueoffloadingdynamicevictionheterogeneousGPU-CPUcomputingLongBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that extreme KV cache compression does not have to cost accuracy if the compression is spread across three coordinated mechanisms: quantized keys that approximate attention scores, values offloaded to CPU memory, and a dynamic eviction rule that keeps only tokens whose cumulative attention contribution reaches a threshold. The authors argue that prior methods fail below 85% compression because they rely on a single technique, whereas their heterogeneous GPU-CPU design lets each component save memory without shouldering the whole burden. If the claim holds, long-context LLM inference could run on a single A100 GPU with considerably cheaper memory, and the same recipe could be dropped into existing transformer pipelines without fine-tuning. The central evidence is a set of LongBench and needle-in-a-haystack results showing near-full accuracy at 25% and 12.5% cache size, plus a demonstration that Llama-3-8B can process 4 million tokens on one 80GB GPU.

What carries the argument

The load-bearing mechanism is the approximate attention score $\tilde{z}_j = \sum_{i=1}^{g} T_{i, P_{j,i}}$ from Eq. (3), where the query is matched against a group-wise codebook of key centroids, followed by the cumulative-magnitude eviction rule of Eq. (4): sort tokens by approximate attention score and keep the smallest prefix whose scores sum to at least $\tau$. This pair converts attention from an $O(n^2 d)$ operation into an $O(ndc + n^2 g)$ operation with $c \ll n$, and it makes the retained token set dependent on the query distribution rather than on fixed positions. The eviction is irreversible, which is why the ranking fidelity of the quantized scores is the component everything else relies on.

What would settle it

Construct a long-context Q&A document where a critical piece of information appears early and is mentioned only once, then ask a question about it much later; if HCAttention's eviction removes that token before the query, retrieval fails, and the failure rate versus token position would quantify the risk. A more direct check: compute rank correlation between full and quantized attention scores over a full 1M-token sequence; if the correlation degrades with sequence length, the eviction decisions become unreliable exactly where the method promises to scale.

Watch

Extended reading notes

Core claim

The central discovery is an approximate attention computation whose memory cost is nearly sequence-independent on the GPU. Keys are compressed by grouped vector quantization into codebook indices, so attention scores are computed by looking up query–centroid products instead of full key–query dot products; values are stored on the CPU and only the selected subset of scores is shipped back for the weighted sum; and a cumulative-magnitude eviction rule permanently drops tokens that together contribute less than a fraction of the attention mass. The paper shows on LongBench that this three-part design keeps the average score of Llama-3-8B at 43.2 at 25% memory (matching full attention) and 42.5 at 12.5%, while also handling 4M-token sequences on a single A100.

Load-bearing premise

The accuracy claim rests on the assumption that scores computed from product-quantized keys preserve token-importance rankings well enough that the cumulative-magnitude eviction rule can permanently drop tokens without a later query needing them.

Editorial extensions

If this is right

  • If the 12.5% result is correct, LLM serving frameworks can fit contexts four times longer (or batch sizes four times larger) on the same GPU, without fine-tuning.
  • The method's layer-wise dynamic eviction could be combined with other prefilling accelerations, as the authors demonstrate with block-wise attention in the appendix.
  • Because the approach is fine-tuning-free and compatible with standard transformers, it can be applied directly to existing open-weight models.
  • The 4-million-token demonstration suggests memory-bound long-context inference can be pushed an order of magnitude beyond current practical limits on a single accelerator.
  • The CPU-offloading path implies that the bottleneck shifts from GPU memory to PCIe bandwidth, which the authors estimate at about 102 MB per 1M tokens at their retention ratio.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the quantized-score ranking is as stable as the paper implies, a direct test would be to measure rank correlation between full and quantized attention scores over long sequences; a drop in correlation with sequence length would predict where the method starts to fail.
  • The eviction rule is irreversible, so a natural stress test is a constructed document where a decisive piece of evidence appears early, receives low attention at the time, and is queried much later; HCAttention's performance on such deferred-needle tasks would reveal the cost of permanent eviction.
  • The authors leave the treatment of surviving probability weights in Eq. (5) unspecified (whether the selected approximate scores are renormalized); this affects both the output scale and the comparison with full softmax, and is testable from the released code.
  • If the framework is extended to multi-head latent attention as the authors suggest, the effective compression could go beyond the reported 12.5%, but the eviction rule would then need to operate on shared latent keys, which is a non-trivial adaptation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The manuscript proposes HCAttention, a training-free framework that compresses the KV cache during long-context LLM inference by combining three mechanisms: product quantization of key vectors (with codebooks fitted offline), offloading of value vectors to CPU memory, and a dynamic eviction rule that keeps only the smallest set of tokens whose approximate attention mass reaches a threshold tau. The authors evaluate on LongBench and NIAH with Llama-2-7B-32K and Llama-3-8B-Instruct-1048k, claiming that the method preserves full-attention accuracy at 25% GPU cache and stays competitive at 12.5%, and that it is the first to process 4 million tokens with Llama-3-8B on a single A100-80GB GPU.

Significance. If the results are reproducible, HCAttention is a useful engineering contribution: it is fine-tuning-free, combines complementary compression axes (quantization, offloading, eviction) in a device-aware way, and reports a concrete path to extreme memory reduction. The paper also makes a falsifiable claim about a 4M-token single-GPU run. However, the current evidence is not sufficient to establish these claims: no code is released, there are no error bars, the operating points are selected on the same benchmark used for the headline numbers, the codebook is fit with unspecified data, and key baselines from the quantization and offloading literature are absent. The stated novelty and SOTA claims therefore remain unverified.

major comments (6)
  1. [Section 3.2, Eq. (4)] The eviction mechanism is never specified as permanent or per-query, and both readings undermine the paper's claims. If the selected set Pi_k* is used to delete cache entries permanently, the criterion is based on a single query's approximate softmax mass, not on cumulative importance across past queries, so it is subject to exactly the 'tokens discarded early may later prove essential' failure that Section 1 attributes to prior methods. Figure 4 only overlays attention-score curves for a few hundred tokens and provides no rank-agreement or recall measure for the top-tau set under different queries. If, alternatively, Pi_k* is recomputed per query, then no KV entries are actually evicted and the memory savings come only from value offloading and key quantization, contradicting the claimed 'dynamic KV eviction' contribution. The manuscript must state the lifecycle of evicted entries and provide evidence of stability across queries, e.g., heavy-hitter-style cumulative scores or a measured recall of the retained set against later queries.
  2. [Section 3.2, Eq. (5)] The computation of the output after eviction is ambiguous. Eq. (5) writes y_hat = F_evict(a_hat) F_evict(V) = a_hat* V* = sum_{i in Pi_k*} a_hat*_i V_i, but it is not stated whether a_hat* contains the original softmax probabilities (which no longer sum to 1 after dropping tokens) or a renormalized distribution over the selected subset. This distinction is numerically important and affects all downstream layer norms and residual connections. A precise definition of a_hat* is required for the method to be reproducible and for the reported accuracy numbers to be interpretable.
  3. [Section 4.1 and Table 2] The headline operating points are selected on the same benchmark used for the main evaluation. Section 4.1 fixes tau=0.9, g=64/32, and c=8192 based on ablations on LongBench, and Table 2 then reports LongBench averages at those settings. In addition, the codebook is fit using '3 randomly selected samples' from an unspecified validation set. This creates a selection-on-test risk and prevents the reader from distinguishing method quality from tuning to the benchmark. The authors should use a separate development set, report per-seed variance, and specify the exact data and number of key vectors used for codebook fitting.
  4. [Section 4.1, Baselines] The comparison set is too narrow to support the 'state-of-the-art' claim. Table 2 compares only against H2O, TOVA, DuoAttention, and StreamingLLM, all of which are token-selection or eviction methods. No quantization-based KV-cache compression baselines (e.g., KIVI, ZipCache, Coupled Quantization) and no CPU-offloading systems (e.g., NEO, FastDecode, FlexInfer) are included, even though HCAttention's contribution is precisely the integration of those dimensions. Without such baselines, the 12.5% result does not establish superiority over existing approaches in the relevant subfields.
  5. [Section 4.2, Efficiency result] The claim of processing 4 million tokens on a single A100 GPU is not substantiated by any measurement in the paper. The text reports only that the run was 'successfully processed'; no memory trace, end-to-end time, throughput, or accuracy at 4M tokens is provided. Appendix B introduces a block-wise prefilling approximation that appears necessary for the 4M run, but it is not controlled for in the claim. The authors should provide a detailed memory budget table, runtime numbers, and at least one accuracy measurement (e.g., NIAH at 4M depth/length) to support the 'first to process 4M tokens' statement.
  6. [Section 4.3.1, Table 3] The eviction-only ablation reports that tau=0.9 improves the average LongBench score (47.2) over full attention with tau=1 (46.9), including gains on HotpotQA, LCC, and TREC. This is a red flag: an approximation that discards over 80% of tokens should not systematically outperform exact attention. The result suggests either evaluation noise, selection of tau on the test set, or a missing mechanism such as implicit renormalization. At minimum, the authors must report error bars or multiple runs, and explain why an approximate method outperforms the exact baseline on several tasks.
minor comments (4)
  1. [Table 2] The table header alignment is confusing: the column labels under 'Llama-3-8B-Instruct-1048K(50%)' are followed by '25% 12.5%', and under 'Llama-2-7B-Instruct-32K(25%)' by '50% 12.5%', making it hard to tell which budget applies to each model. Please restructure the table so each model has its own clearly labeled budget columns.
  2. [Section 4.2] There is a typo in 'Lllma-3-8B-Instruct-1048'; it should be 'Llama-3-8B-Instruct-1048k'.
  3. [Appendix B] The block-wise prefilling idea is described as 'inspired by Star Attention', but no citation to Star Attention is given. Please add the reference.
  4. [Section 4.1] The phrase '3 randomly selected samples' is unclear: it is not specified what these are samples of, or how many key vectors are used to fit the codebook. Please clarify the experimental protocol.

Circularity Check

1 steps flagged · score 3.0 of 10

LongBench operating points are selected on LongBench itself, making the headline accuracy partly a fitted result, while the formal derivation contains no Eq.-equals-Eq. circularity.

  1. fitted input called prediction [Section 4.1 (Implementation details) and Section 4.3.1 (Ablation on KV eviction, Table 3)]
    "In the experiments, we set the attention sparsification threshold to τ=0.9. ... Results indicate that setting τ=0.9 allows the system to prune over 80% of tokens while maintaining accuracy on average. ... Therefore, τ=0.9 strikes a favorable balance between information retention and computational efficiency."

    The threshold τ=0.9 is chosen by running an ablation on the LongBench benchmark (Table 3) and selecting the value with the best LongBench average. The same LongBench benchmark is then used in Table 2 for the headline claim that HCAttention preserves full-attention accuracy at 25% and 12.5% KV-cache budgets. The reported accuracy is therefore an in-sample selected operating point rather than an independent prediction; the same concern applies to the codebook size c=8192 selected from the LongBench ablation in Table 4a.

full rationale

No formal derivation in the paper reduces a predicted quantity to an input by construction: Eq. 4 defines the selected token set from cumulative approximate attention mass, and Eq. 5 uses that set to define the output, which is a definition rather than a circular inference. There is also no load-bearing self-citation chain or imported uniqueness theorem. The substantive circularity concern is experimental: the key operating parameters τ=0.9 and c=8192 are selected using ablations on the same LongBench benchmark that later supplies the headline accuracy numbers. This means the main accuracy claims are partly a hyperparameter fit to the evaluation set, rather than a parameter-free prediction. The NIAH results and the 4M-token demonstration are independent of this issue, and the 'dynamic KV eviction' ambiguity is a correctness risk rather than a circularity. Overall, the paper is not derivationally circular, but its central LongBench comparison is moderately compromised by on-benchmark hyperparameter selection.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on several fitted choices: codebook centroids, number of centroids, group size, and eviction threshold, all selected via LongBench ablations. The axioms are engineering assumptions common to approximate-attention systems, none of which are formally proved. No new physical or theoretical entities are introduced.

free parameters (4)
  • Key quantization codebook centroids C = MiniBatchKMeans clusters over 3 random validation samples
    The codebook is constructed offline from a representative validation dataset (Sec 3.1); its quality directly determines approximate attention score fidelity, as Table 4a shows average dropping from 43.5 to 23.8 when c=256.
  • Number of codebook centroids c = 8192 (default; stable at 4096+)
    Selected via LongBench ablation (Table 4a); performance is highly sensitive below 2048 centroids.
  • Group size g (key quantization groups) = 32 for 12.5% budget, 64 for 25% budget
    Selected via LongBench ablation (Table 4b); g=16 collapses average performance to 12.8.
  • Eviction threshold tau = 0.9
    Chosen from Table 3 ablation on LongBench; tau=0.9 retains only 15.6% of tokens on average.
assumptions (3)
  • domain assumption Product-quantized attention scores preserve the relative importance ranking of tokens
    Sec 3.2 uses quantized scores for eviction and weighting; Fig 4 shows overlap for a few hundred tokens but provides no quantitative error bound.
  • domain assumption Offloading values to CPU and transferring selected scores over PCIe does not bottleneck inference
    Sec 3.3 estimates 102.4 MB communication for n=1e6 but reports no end-to-end latency measurement.
  • domain assumption The codebook trained on a representative validation dataset generalizes to test distributions
    Sec 3.1 constructs the codebook from a validation set without specifying its source or coverage; Table 4 sensitivity suggests generalization is load-bearing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs." pith.science (2026). https://pith.science/paper/RBZF6FO7

@misc{pith2026250719823,
  author       = {Pith},
  title        = {Pith review of: HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RBZF6FO7}},
  note         = {Machine review of arXiv:2507.19823}
}
read the original abstract

Processing long-context inputs with large language models presents a significant challenge due to the enormous memory requirements of the Key-Value (KV) cache during inference. Existing KV cache compression methods exhibit noticeable performance degradation when memory is reduced by more than 85%. Additionally, strategies that leverage GPU-CPU collaboration for approximate attention remain underexplored in this setting. We propose HCAttention, a heterogeneous attention computation framework that integrates key quantization, value offloading, and dynamic KV eviction to enable efficient inference under extreme memory constraints. The method is compatible with existing transformer architectures and does not require model fine-tuning. Experimental results on the LongBench benchmark demonstrate that our approach preserves the accuracy of full-attention model while shrinking the KV cache memory footprint to 25% of its original size. Remarkably, it stays competitive with only 12.5% of the cache, setting a new state-of-the-art in LLM KV cache compression. To the best of our knowledge, HCAttention is the first to extend the Llama-3-8B model to process 4 million tokens on a single A100 GPU with 80GB memory.

Figures

Figures reproduced from arXiv: 2507.19823 by the authors.

Figure 1
Figure 1. Overview of the HCAttention heterogeneous attention computation framework. The [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. DuoAttention can maintain model performance at 50% memory budget, but lower budgets [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Comparison of dynamic KV eviction selection ratio across transformer layers. We evaluate the effectiveness of applying layer-wise dynamic KV eviction (i.e., attention score thresholding) without key quantization or value offloading. This ablation study is conducted using the Llama-2-7B-32K model on the LongBench benchmark. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Attention scores in different layers between full-attention and group quantization ( [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 1
Figure 1. Figure 1: Visualization of NIAH test. The "needle" text is "Remember, the best thing to do in San [PITH_FULL_IMAGE:figures/full_fig_p013_1.png]
Figure 2
Figure 2. Figure 2: Visualization of NIAH test. The "needle" text is "Remember, the best thing to do in New [PITH_FULL_IMAGE:figures/full_fig_p013_2.png]
Figure 3
Figure 3. Figure 3: Attention mask patterns illustrating the integration of HCAttention with block-wise attention [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]
Figure 4
Figure 4. Figure 4: Visualization of NIAH test of HCAttention enhanced with block-wise attention in prefilling [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 23 canonical work pages

  1. [1]

    2 In the approximate attention score computation, the computation is performed group-wise

    GPU part in Eq. 2 In the approximate attention score computation, the computation is performed group-wise. K is divided into predefined groups. Each sub-group is represented as nearest neighbor of the centroids in the codebook, serves as a compressed representation of K, stored in GPU memory. The compressed representation is an index matrixP∈Z n×g, whereP...

    work page

  2. [2]

    needles" on the task performance. We take “The best thing to do in Paris is buy a fresh croissant and lounge by the Seine at twilight

    CPU part in Eq. 2 Based on approximate key cache eviction, we propose fully offloading the value matrix V∈R n×d to CPU memory to optimize the computation process. The weighted sum of values using selected keys can be represented as: ey=aV≈ F evict(ea)Fevict(V) = ea∗V ∗ = X i∈Πk∗ ea∗ i ×V i.(5) 5 3.3 Overhead and Efficiency Analysis Compression ratio.The m...

    work page

  3. [3]

    GPT-4 technical report

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. “GPT-4 technical report”. In:arXiv preprint arXiv:2303.08774(2023)

    arXiv 2023

  4. [4]

    Keyformer: Kv cache reduction through key tokens selection for efficient generative inference

    M. Adnan, A. Arunkumar, G. Jain, P. J. Nair, I. Soloveychik, and P. Kamath. “Keyformer: Kv cache reduction through key tokens selection for efficient generative inference”. In:Proceed- ings of Machine Learning and Systems6 (2024), pp. 114–127

    work page 2024

  5. [5]

    Qwen technical report

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, et al. “Qwen technical report”. In:arXiv preprint arXiv:2309.16609(2023)

    arXiv 2023

  6. [6]

    LongBench: A Bilingual, Multitask Benchmark for Long Context Under- standing

    Y . Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y . Dong, J. Tang, and J. Li. “LongBench: A Bilingual, Multitask Benchmark for Long Context Under- standing”. In:Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers). Bangkok, Thailand: Association for Computatio...

    work page 2024

  7. [7]

    Longformer: The long-document transformer

    I. Beltagy, M. E. Peters, and A. Cohan. “Longformer: The long-document transformer”. In: arXiv preprint arXiv:2004.05150(2020)

    arXiv 2020

  8. [8]

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

    T. Dao. “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning”. In: International Conference on Learning Representations (ICLR). 2024

    work page 2024

Show all 42 references
  1. [9]

    Model tells you what to discard: Adaptive kv cache compression for llms

    S. Ge, Y . Zhang, L. Liu, M. Zhang, J. Han, and J. Gao. “Model tells you what to discard: Adaptive kv cache compression for llms”. In:arXiv preprint arXiv:2310.01801(2023)

    2023 arXiv

  2. [10]

    The llama 3 herd of models

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. “The llama 3 herd of models”. In:arXiv preprint arXiv:2407.21783(2024)

    2024 arXiv

  3. [11]

    Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines

    J. He and J. Zhai. “Fastdecode: High-throughput gpu-efficient llm serving using heterogeneous pipelines”. In:arXiv preprint arXiv:2403.11421(2024)

    2024 arXiv

  4. [12]

    ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

    Y . He, L. Zhang, W. Wu, J. Liu, H. Zhou, and B. Zhuang. “ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification”. In:The Thirty-eighth Annual Conference on Neural Information Processing Systems

  5. [13]

    Mixtral of experts

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. “Mixtral of experts”. In:arXiv preprint arXiv:2401.04088(2024)

    2024 arXiv

  6. [14]

    Neo: Saving gpu memory crisis with cpu offloading for online llm inference

    X. Jiang, Y . Zhou, S. Cao, I. Stoica, and M. Yu. “Neo: Saving gpu memory crisis with cpu offloading for online llm inference”. In:arXiv preprint arXiv:2411.01142(2024)

    2024 arXiv

  7. [15]

    Kamradt.Llmtest_needleinahaystack: Doing simple retrieval from llm models at vari- ous context lengths to measure accuracy

    G. Kamradt.Llmtest_needleinahaystack: Doing simple retrieval from llm models at vari- ous context lengths to measure accuracy. https://github.com/gkamradt/LLMTest_ NeedleInAHaystack. 2024

    2024

  8. [16]

    Beyond Single- Turn: A Survey on Multi-Turn Interactions with Large Language Models

    Y . Li, X. Shen, X. Yao, X. Ding, Y . Miao, R. Krishnan, and R. Padman. “Beyond Single- Turn: A Survey on Multi-Turn Interactions with Large Language Models”. In:arXiv preprint arXiv:2504.04717(2025)

    2025 arXiv

  9. [17]

    Focus- LLM: Precise Understanding of Long Context by Dynamic Condensing

    Z. Li, Y . Zhang, T. Pan, Y . Sun, Z. Duan, J. Fang, R. Han, Z. Wang, and J. Wang. “Focus- LLM: Precise Understanding of Long Context by Dynamic Condensing”. In:arXiv preprint arXiv:2408.11745(2024)

    2024 arXiv

  10. [18]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, et al. “Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model”. In: arXiv preprint arXiv:2405.04434(2024). 10

    2024 arXiv

  11. [19]

    KIVI: a tuning- free asymmetric 2bit quantization for KV cache

    Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V . Braverman, B. Chen, and X. Hu. “KIVI: a tuning- free asymmetric 2bit quantization for KV cache”. In:Proceedings of the 41st International Conference on Machine Learning. 2024, pp. 32332–32344

    2024

  12. [20]

    LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning

    Y . Mao, J. Li, F. Meng, J. Xiong, Z. Zheng, and M. Zhang. “LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning”. In:arXiv preprint arXiv:2412.13626(2024)

    2024 arXiv

  13. [21]

    Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models

    J. Ni, G. H. Abrego, N. Constant, J. Ma, K. B. Hall, D. Cer, and Y . Yang. “Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models”. In:arXiv preprint arXiv:2108.08877 (2021)

    2021 arXiv

  14. [22]

    Transformers are Multi-State RNNs

    M. Oren, M. Hassid, N. Yarden, Y . Adi, and R. Schwartz. “Transformers are Multi-State RNNs”. In:Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024, pp. 18724–18741

    2024

  15. [23]

    Scikit-learn: Machine learning in Python

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, et al. “Scikit-learn: Machine learning in Python”. In:the Journal of machine Learning research12 (2011), pp. 2825–2830

    2011

  16. [24]

    Web-scale k-means clustering

    D. Sculley. “Web-scale k-means clustering”. In:Proceedings of the 19th international confer- ence on World wide web. 2010, pp. 1177–1178

    2010

  17. [25]

    Adafactor: Adaptive learning rates with sublinear memory cost

    N. Shazeer and M. Stern. “Adafactor: Adaptive learning rates with sublinear memory cost”. In: International Conference on Machine Learning. PMLR. 2018, pp. 4596–4604

    2018

  18. [26]

    QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference

    J. Tang, Y . Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han. “QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference”. In:Proceedings of the International Conference on Machine Learning (ICML). 2024

    2024

  19. [27]

    AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer- Wise Asymmetric Quantization Configurations

    Q. Tao, W. Yu, and J. Zhou. “AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer- Wise Asymmetric Quantization Configurations”. In:Proceedings of the 31st International Conference on Computational Linguistics. 2025, pp. 2316–2328

    2025

  20. [28]

    Gemini: a family of highly capable multimodal models

    G. Team, R. Anil, S. Borgeaud, J. -B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. “Gemini: a family of highly capable multimodal models”. In: arXiv preprint arXiv:2312.11805(2023)

    2023 arXiv

  21. [29]

    Llama 2: Open foundation and fine-tuned chat models

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. “Llama 2: Open foundation and fine-tuned chat models”. In: arXiv preprint arXiv:2307.09288(2023)

    2023 arXiv

  22. [30]

    Attention is all you need

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. “Attention is all you need”. In:Advances in neural information processing systems 30 (2017)

    2017

  23. [31]

    Fast transformers with clustered attention

    A. Vyas, A. Katharopoulos, and F. Fleuret. “Fast transformers with clustered attention”. In: Advances in Neural Information Processing Systems33 (2020), pp. 21665–21674

    2020

  24. [32]

    SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

    Z. Wang, C. Bin, and S. Gan. “SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget”. In:The Thirteenth International Conference on Learning Representations

  25. [33]

    Duoattention: Ef- ficient long-context llm inference with retrieval and streaming heads

    G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y . Fu, and S. Han. “Duoattention: Ef- ficient long-context llm inference with retrieval and streaming heads”. In:arXiv preprint arXiv:2410.10819(2024)

    2024 arXiv

  26. [34]

    Efficient Streaming Language Models with Attention Sinks

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis. “Efficient Streaming Language Models with Attention Sinks”. In:The Twelfth International Conference on Learning Representations

  27. [35]

    vtensor: Flexible virtual tensor management for efficient llm serving

    J. Xu, R. Zhang, C. Guo, W. Hu, Z. Liu, F. Wu, Y . Feng, S. Sun, C. Shao, Y . Guo, et al. “vtensor: Flexible virtual tensor management for efficient llm serving”. In:arXiv preprint arXiv:2407.15309(2024)

    2024 arXiv

  28. [36]

    Think: Thinner key cache by query-driven pruning

    Y . Xu, Z. Jie, H. Dong, L. Wang, X. Lu, A. Zhou, A. Saha, C. Xiong, and D. Sahoo. “Think: Thinner key cache by query-driven pruning”. In:arXiv preprint arXiv:2407.21018(2024)

    2024 arXiv

  29. [37]

    A Survey on Multi-Turn Interaction Capabilities of Large Language Models

    C. Zhang, X. Dai, Y . Wu, Q. Yang, Y . Wang, R. Tang, and Y . Liu. “A Survey on Multi-Turn Interaction Capabilities of Large Language Models”. In:arXiv preprint arXiv:2501.09959 (2025)

    2025 arXiv

  30. [38]

    Soaring from 4k to 400k: Extending llm’s context with activation beacon

    P. Zhang, Z. Liu, S. Xiao, N. Shao, Q. Ye, and Z. Dou. “Soaring from 4k to 400k: Extending llm’s context with activation beacon”. In:arXiv preprint arXiv:2401.034622.3 (2024), p. 5. 11

    2024 arXiv

  31. [39]

    Opt: Open pre-trained transformer language models

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, et al. “Opt: Open pre-trained transformer language models”. In:arXiv preprint arXiv:2205.01068(2022)

    2022 arXiv

  32. [40]

    KV cache is 1 bit per channel: Efficient large language model inference with coupled quantization

    T. Zhang, J. Yi, Z. Xu, and A. Shrivastava. “KV cache is 1 bit per channel: Efficient large language model inference with coupled quantization”. In:Advances in Neural Information Processing Systems37 (2024), pp. 3304–3331

    2024

  33. [41]

    Chain of agents: Large language models collaborating on long-context tasks

    Y . Zhang, R. Sun, Y . Chen, T. Pfister, R. Zhang, and S. Arik. “Chain of agents: Large language models collaborating on long-context tasks”. In:Advances in Neural Information Processing Systems37 (2024), pp. 132208–132237

    2024

  34. [42]

    H2O: Heavy-hitter oracle for efficient generative inference of large language models

    Z. Zhang, Y . Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y . Tian, C. Ré, C. Barrett, et al. “H2O: Heavy-hitter oracle for efficient generative inference of large language models”. In:Advances in Neural Information Processing Systems36 (2023), pp. 34661–34710. 12 A Al...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.