Pith. sign in

REVIEW 3 major objections 4 minor 67 references

SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SALE claims that a 4-bit element-wise preview of the attention map, followed by block-level skipping, can accelerate long-context prefilling by at least 3.36x while keeping output quality near full attention.

desk verdict SALE's fine-grained low-bit attention estimation is a real step forward for sparse-prefill, but an algebra error in Eq. (4) and a circular calibration on InfiniteBench need fixing before the numbers are trustworthy. read the letter →

arxiv 2505.24179 v1 pith:5CB7ZRWR submitted 2025-05-30 cs.LG cs.AI

Xiaodong Ji , Hailin Zhang , Fangcheng Fu , Bin Cui This is my paper
classification cs.LGcs.AI
keywords sparseattentionlong-contextLLMprefillingacceleration4-bitquantizationblock-sparseweightestimationtraining-freeinferenceRelativeScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SALE argues that the quadratic cost of long-context prefilling can be cut sharply without retraining by looking at every query-key pair in cheap 4-bit precision, deciding which blocks of the attention map are worth computing, and then doing full attention only on those blocks. The paper's claim is that this fine-grained low-bit inspection finds far sparser masks than coarse block-representative or sampled-row schemes while keeping the output error under a user-set bound. On Llama-3.1-8B this gives at least a 3.36x speedup over full attention for sequences longer than 64K tokens, with benchmark scores close to those of full attention and overhead around 11% of full attention latency at 128K tokens. A sympathetic reader would care because this is a training-free, drop-in replacement for the attention module and the speedup grows with sequence length.

What carries the argument

The load-bearing object is the Relative Attention Score, $\tilde{P}[i,j] = \exp(\tilde{S}[i,j] - em_i)/el_i$, computed from 4-bit estimated attention weights $\tilde{S}[i,j]$, where $em_i$ and $el_i$ are the row's maximum and sum of estimated weights in the sink-local area (the first 32 tokens plus the most recent up to 256 tokens). A block is skipped when every estimated score inside it falls below per-head threshold $\tau$, and the comparison is rewritten as $\tilde{S}[i,j] \geq \ln(\tau \cdot el_i + em_i)$ so it costs a single floating-point comparison instead of exponentials and division. Per-head thresholds come from an offline calibration that halves $\tau$ until the L1 output error versus full attention is below $\theta$. This machinery converts block selection from a structural guess into an element-wise, error-bounded decision whose 4-bit pre-scan can run on tensor cores.

What would settle it

Take a long-context task whose important content is distributed differently from Retrieve.KV, such as repository-level code with references scattered through the middle of the file, run SALE with its calibrated thresholds, and compare both the L1 attention-output error and the task score against full attention; if the error exceeds $\theta$ or the score drops materially, the calibration-generalization claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that block-sparse attention masks can be built accurately from a full element-wise estimate of the attention map, provided the estimate is cheap enough. SALE obtains that estimate by multiplying 4-bit quantized query and key matrices, then uses a Relative Attention Score, an approximate softmax weight whose normalization comes from the sink-local region of each row, to mark low-scoring blocks as skippable. Per-head thresholds are calibrated offline on a few samples by halving tau until the L1 error between SALE's output and full attention drops below theta. The paper reports that this combination keeps output quality within a small margin of full attention while making long-context prefilling several times faster, with the selection overhead falling to roughly 11% of full attention latency at 128K tokens.

Load-bearing premise

The per-head thresholds, tuned on five samples from the Retrieve.KV task of InfiniteBench, are assumed to keep the L1 error bound on any unseen long-context input, so the whole accuracy guarantee rests on that calibration generalizing.

Editorial extensions

If this is right

  • For 64K-token and longer inputs, prefilling attention can run 3.36x or more faster than full attention while staying near full-attention quality on standard long-context benchmarks.
  • Because SALE is training-free and replaces only the attention module, it can be integrated into existing transformer pipelines with small code changes and combined with other token-level optimizations.
  • The speedup improves as context length grows, since attention maps get sparser and the fixed inspection overhead becomes a smaller fraction of total cost.
  • One error bound $\theta$ controls the whole accuracy-efficiency trade-off per model, so a deployment can choose a faster or safer operating point without retraining.
  • The benefit depends on hardware with efficient 4-bit matrix instructions; on other hardware the selection pass loses its speed advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to make threshold calibration per prompt or online, because the offline calibration uses only five samples from one retrieval task and the error bound is only as good as that sample's representativeness.
  • Since 4-bit approximation can overestimate some attention weights, SALE may keep blocks that full attention would skip; this suggests pairing the hard threshold with a small correction or a second cheap check.
  • The sink-local reference region encodes an assumption that important context sits near the start or end of the sequence; prompts with critical information in the middle are the stress case where the metric may misorder blocks.
  • Beyond 128K tokens the 4-bit selection pass still touches every query-key pair, so its quadratic cost may eventually dominate even with tensor-core acceleration; sub-quadratic inspection is the next logical limit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SALE, a training-free block-sparse attention method for accelerating the prefill stage of long-context LLM inference. SALE first estimates the attention map at element granularity using 4-bit quantized query-key products, then selects key blocks according to a Relative Attention Score that compares estimated attention weights with those in the sink-local region. A per-head threshold is calibrated offline against an L1 output-error bound, and attention is computed only on selected blocks using a custom CUDA kernel and SageAttention-style quantization. Experiments on LongBench, InfiniteBench, and Needle-In-A-Haystack with Llama-3.1-8B-Instruct and Qwen-2.5-32B-Instruct report at least 3.36x speedups at 64K tokens with minimal accuracy loss; the code is released.

Significance. If the method works as described, it is a practically valuable contribution: it combines fine-grained attention-map inspection with low-bit estimation, requires no training, and reports a better accuracy-efficiency trade-off than several strong sparse-attention baselines. The paper ships an implementation, includes custom CUDA/Triton kernel work, and evaluates across multiple benchmarks and two model scales. The main strengths are the breadth of the experiments, the independent LongBench and NIAH evidence, and the clear ablation of the quantization and calibration components. However, two load-bearing issues need attention: a mathematically incorrect threshold-comparison equation in Section 3.4, and a circular calibration/evaluation setup for the InfiniteBench results.

major comments (3)
  1. [§3.4, Eq. (4)] Equation (4) is algebraically incorrect. From the definition eP[i,j] = exp(eS[i,j] - em_i)/el_i, the condition eP[i,j] >= tau is equivalent to eS[i,j] >= ln(tau * el_i) + em_i, not to eS[i,j] >= ln(tau * el_i + em_i) as printed. This comparison is used in Algorithm 1, line 16, to decide which blocks are kept or skipped, so it directly controls the sparsity level and the calibrated L1 error Err(tau). If the released CUDA kernel implements the printed comparison, then the per-head thresholds calibrated in Section 3.3 are not the thresholds that actually govern block selection, and the speedups and accuracy numbers in Tables 1-2 and Figure 2 are not tied to the described algorithm. If the kernel implements the corrected inequality, Equation (4) is a typo but still must be fixed. Please correct the equation and state explicitly which comparison the released kernel uses.
  2. [§3.3, Appendix C, Table 2] The per-head thresholds tau are calibrated on five samples from the Retrieve.KV task of InfiniteBench, as stated in Appendix C, and the same benchmark is then used for the headline InfiniteBench evaluation in Table 2 and for the trade-off analysis in Figure 4(c). This makes part of the accuracy evidence circular: the method is tuned to the test distribution. The LongBench and Needle-In-A-Haystack results are independent and provide meaningful support, but the InfiniteBench numbers, especially the Retrieve.KV row and the Figure 4(c) trade-off, cannot be interpreted as out-of-distribution evidence. Please either calibrate on a task not used in evaluation, report results on a held-out calibration-validation split, or explicitly label the InfiniteBench results as in-distribution with respect to threshold fitting.
  3. [§3.3 and §4.1] The error bound Err(tau) is verified only on the five calibration samples; the paper does not report how the L1 error behaves on unseen samples from other tasks or sequence lengths. Since the central claim is that SALE maintains model quality on arbitrary long-context inputs, the current evidence for that generalization rests on LongBench and NIAH. This is acceptable support, but the claim in the abstract and conclusion that accuracy is maintained 'for sequences longer than 64K' should be scoped to the evaluated benchmarks and should acknowledge the calibration distribution.
minor comments (4)
  1. [§1] There is a typo in the Introduction: 'an satisfactory accuracy-efficiency trade-off' should be 'a satisfactory accuracy-efficiency trade-off'.
  2. [§4.1 and Appendix D] The main text says 'we use the calibration samples for MInference, SpargeAttn, and SALE,' while Appendix D clarifies that SpargeAttn uses the same input samples and MInference uses only the first sample. Please state these details in the main text or align the wording.
  3. [§4.4, Table 3] The text refers to 'the second-to-last line' and 'the final line' of Table 3, but the table as typeset has an overhead-ratio line followed by a Computation-Pass-speedup line; the wording should match the table numbering more explicitly.
  4. [§4.3] The claim in the introduction that selection overhead is 'approximately 11% of full attention latency' is only true at long context lengths; Table 3 shows the overhead ratio is 23.9% at 8K and decreases with length. The context-length dependence should be stated when the 11% number is cited.

Circularity Check

1 steps flagged · score 5.0 of 10

Partial circularity: per-head thresholds are calibrated on InfiniteBench Retrieve.KV samples, yet the same benchmark task appears in the accuracy evaluation; speedup and independent benchmarks keep the central claim from reducing entirely.

  1. fitted input called prediction [Appendix C (Additional implementation details) vs. Section 4.2 / Table 2 (InfiniteBench evaluation)]
    "We select five input samples from the Retrieve.KV task in InfiniteBench to perform calibration for SALE, and the final configuration must satisfy the error bound requirement across all five samples."

    Section 3.3 makes tau the fitted per-head parameter ('an offline calibration procedure to determine the optimal tau value for each head'), and Appendix C shows it is tuned until Err(tau)<theta on five Retrieve.KV samples from InfiniteBench. Table 2 then uses Retrieve.KV, and the average containing it, as evidence that SALE maintains quality. The tau used to select blocks for those test inputs was chosen on samples from the same task, so the Retrieve.KV accuracy is a calibration check rather than an independent prediction. This is partial: LongBench, NIAH, and latency results are not calibrated on the same data.

full rationale

The paper's core mechanism is not circular: the Relative Attention Score eP[i,j] = exp(eS[i,j]-em_i)/el_i is computed from the input and sink-local statistics, and the block mask is a direct function of that score and tau; no result is defined in terms of its own output. The cited quantization and sparse-attention techniques are external or explicitly code-reuse bases, and the self-citation to PQCache is a related-work pointer, not a load-bearing premise. The single substantive circularity is the calibration-evaluation overlap: Appendix C shows that tau, the method's only tuned parameter, is fitted on five samples from InfiniteBench Retrieve.KV, and Table 2 uses that same task and its aggregate to claim quality maintenance. This makes part of the accuracy evidence a fitted-input-called-prediction. It is not a total reduction, because LongBench and Needle-In-A-Haystack are independent of calibration, and the 3.36x-3.81x speedup numbers are latency measurements that do not depend on tau's generalization. Appendix A's hardware limitation is appropriately stated. The algebra in Eq. (4) appears incorrect (exp(eS-em)/el >= tau gives eS >= ln(tau*el)+em, not ln(tau*el+em)); that is a correctness and implementation-verification risk, not a circularity, so it is noted here but not scored as a circular step.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

Free parameters: per-head threshold tau calibrated on data, error bound theta chosen by user, sink/local area sizes and block sizes hand-set. These are the main tuning knobs. Axioms: the sparsity/sink-local observation, the accuracy of 4-bit quantization, and the use of L1 output error as a quality proxy are all assumed. No invented entities.

free parameters (5)
  • Per-head threshold tau = Calibrated; initial 0.008, halved until error bound met (final values not reported)
    Controls which blocks are skipped; calibrated on five Retrieve.KV samples (Appendix C), a fitted hyperparameter.
  • Error bound theta = 0.4 (Llama-3.1), 2.0 (Qwen-2.5)
    User-defined target for L1 output error in calibration; chosen by the authors, not derived.
  • Sink area size = 32 tokens
    Size of the beginning region used for relative score; hand-set in Section 4.1.
  • Local area size = Up to 256 tokens, min 128
    Size of the end region used for relative score; hand-set and subject to implementation grouping (Appendix C).
  • Block sizes bq, bk = bq=64, bk=32
    Block granularity for sparsity; chosen heuristically in Section 4.1.
assumptions (4)
  • domain assumption Attention maps in LLMs are significantly sparse, and sink-local regions consistently contain high attention scores.
    Motivates the Relative Attention Score; cited to [16,20,26] but assumed for the method to work.
  • domain assumption 4-bit quantized QK products (SageAttention2 quantization) provide sufficiently accurate estimates of attention weights for block selection.
    The selection pass relies on this external quantization method; if the estimates are badly ranked, the sparse mask will be wrong.
  • domain assumption The L1 distance between SALE and full attention outputs is a valid proxy for end-task quality loss.
    Calibration uses Err(tau) = ||O - O_tilde||_1/N; the paper does not prove that small L1 error implies small benchmark degradation.
  • standard math Softmax and row-max arithmetic are standard.
    Used throughout the algorithm derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling." pith.science (2026). https://pith.science/paper/5CB7ZRWR

@misc{pith2026250524179,
  author       = {Pith},
  title        = {Pith review of: SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5CB7ZRWR}},
  note         = {Machine review of arXiv:2505.24179}
}
read the original abstract

Many advanced Large Language Model (LLM) applications require long-context processing, but the self-attention module becomes a bottleneck during the prefilling stage of inference due to its quadratic time complexity with respect to sequence length. Existing sparse attention methods accelerate attention computation by skipping less significant regions of the attention map. However, these approaches typically perform coarse-grained inspection of the attention map, rendering considerable loss in model accuracy. In this paper, we propose SALE, a fine-grained sparse attention method that accelerates the long-context prefilling stage of LLM with negligible loss in model accuracy. SALE achieves fast and accurate fine-grained attention weight estimation through 4-bit quantized query-key products, followed by block-sparse attention to accelerate prefilling computations. For importance evaluation for query-key pairs, we adopt our Relative Attention Score metric, which offers significantly higher efficiency within our framework. We implement a custom CUDA kernel optimized for our approach for hardware efficiency, reducing the additional overhead to approximately 11% of the full attention latency. Notably, SALE requires no parameter training and can be seamlessly integrated into existing systems with trivial code modifications. Experiments on long-context benchmarks demonstrate that our method outperforms existing approaches in accuracy-efficiency trade-offs, achieving at least 3.36x speedups on Llama-3.1-8B for sequences longer than 64K while maintaining model quality.

Figures

Figures reproduced from arXiv: 2505.24179 by the authors.

Figure 1
Figure 1. (a) Attention maps of two different attention heads in Llama-3.1-8B-Instruct when process [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Needle-In-A-Haystack evaluation results. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. (a) Speedup in single-input processing. (b) Comparison between SALE v.s. SALE w/o [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Evaluation of accuracy-efficiency trade-offs. The brown horizontal dashed line represents [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison between SALE and SALE w/o QK Quant. (a)Single input speedup. (b) [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 34 canonical work pages

  1. [1]

    Booksum: A collection of datasets for long-form narrative summarization,

    W. Kry´sci´nski, N. Rajani, D. Agarwal, C. Xiong, and D. Radev, “Booksum: A collection of datasets for long-form narrative summarization,” 2022

    work page 2022

  2. [2]

    Transformer Based Implementation for Automatic Book Summarization

    S. Porwal, L. Bewoor, and V . Deshpande, “Transformer based implementation for automatic book summa- rization,” arXiv preprint arXiv:2301.07057, 2023

    work page Pith review arXiv 2023

  3. [3]

    Booookscore: A systematic exploration of book-length summarization in the era of LLMs,

    Y . Chang, K. Lo, T. Goyal, and M. Iyyer, “Booookscore: A systematic exploration of book-length summarization in the era of LLMs,” in The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  4. [4]

    Peek across: Improving multi-document modeling via cross-document question-answering,

    A. Caciularu, M. Peters, J. Goldberger, I. Dagan, and A. Cohan, “Peek across: Improving multi-document modeling via cross-document question-answering,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (A. Rogers, J. Boyd-Graber, and N. Okazaki, eds.), (Toronto, Canada), pp. 1970–1989, Assoc...

    work page 1970

  5. [5]

    Quality: Question answering with long input texts, yes!,

    R. Y . Pang, A. Parrish, N. Joshi, N. Nangia, J. Phang, A. Chen, V . Padmakumar, J. Ma, J. Thompson, H. He, and S. R. Bowman, “Quality: Question answering with long input texts, yes!,” 2022

    work page 2022

  6. [6]

    Eli5: Long form question answering,

    A. Fan, Y . Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli, “Eli5: Long form question answering,” 2019

    work page 2019

  7. [7]

    Teaching code llms to use autocompletion tools in repository-level code generation,

    C. Wang, J. Zhang, Y . Feng, T. Li, W. Sun, Y . Liu, and X. Peng, “Teaching code llms to use autocompletion tools in repository-level code generation,” 2024

    work page 2024

  8. [8]

    Rlcoder: Reinforcement learning for repository-level code completion,

    Y . Wang, Y . Wang, D. Guo, J. Chen, R. Zhang, Y . Ma, and Z. Zheng, “Rlcoder: Reinforcement learning for repository-level code completion,” 2024

    work page 2024

Show all 67 references
  1. [9]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru,...

    2024

  2. [10]

    Qwen2. 5-1m technical report,

    A. Yang, B. Yu, C. Li, D. Liu, F. Huang, H. Huang, J. Jiang, J. Tu, J. Zhang, J. Zhou,et al., “Qwen2. 5-1m technical report,” arXiv preprint arXiv:2501.15383, 2025

    2025 arXiv

  3. [11]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. bastien Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. T...

    2025

  4. [12]

    Deepseek-v3 technical report,

    DeepSeek-AI, A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, ...

    2025

  5. [13]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  6. [14]

    Challenges in deploying long-context transformers: A theoretical peak performance analysis,

    Y . Fu, “Challenges in deploying long-context transformers: A theoretical peak performance analysis,” 2024

    2024

  7. [15]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention,

    H. Jiang, Y . Li, C. Zhang, Q. Wu, X. Luo, S. Ahn, Z. Han, A. Abdi, D. Li, C.-Y . Lin,et al., “Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention,” Advances in Neural Information Processing Systems, vol. 37, pp. 52481–52515, 2024

    2024

  8. [16]

    Attention is naturally sparse with gaussian distributed input,

    Y . Deng, Z. Song, and C. Yang, “Attention is naturally sparse with gaussian distributed input,” CoRR, vol. abs/2404.02690, 2024

    2024 arXiv

  9. [17]

    Generating long sequences with sparse transformers,

    R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  10. [18]

    Big bird: Transformers for longer sequences,

    M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al., “Big bird: Transformers for longer sequences,” Advances in neural information processing systems, vol. 33, pp. 17283–17297, 2020

    2020

  11. [19]

    Longformer: The long-document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  12. [20]

    Efficient streaming language models with attention sinks,

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” in The Twelfth International Conference on Learning Representations, 2024

    2024

  13. [21]

    Lm-infinite: Zero-shot extreme length generalization for large language models,

    C. Han, Q. Wang, H. Peng, W. Xiong, Y . Chen, H. Ji, and S. Wang, “Lm-infinite: Zero-shot extreme length generalization for large language models,” arXiv preprint arXiv:2308.16137, 2023

    2023 arXiv

  14. [22]

    Sampleat- tention: Near-lossless acceleration of long context llm inference with adaptive structured sparse attention,

    Q. Zhu, J. Duan, C. Chen, S. Liu, X. Li, G. Feng, X. Lv, H. Cao, X. Chuanfu, X. Zhang, et al., “Sampleat- tention: Near-lossless acceleration of long context llm inference with adaptive structured sparse attention,” arXiv preprint arXiv:2406.15486, 2024

    2024 arXiv

  15. [23]

    Flexprefill: A context-aware sparse attention mechanism for ef- ficient long-sequence inference,

    X. Lai, J. Lu, Y . Luo, Y . Ma, and X. Zhou, “Flexprefill: A context-aware sparse attention mechanism for ef- ficient long-sequence inference,” in The Thirteenth International Conference on Learning Representations, 2025

    2025

  16. [24]

    Spargeattn: Accurate sparse attention accelerating any model inference,

    J. Zhang, C. Xiang, H. Huang, J. Wei, H. Xi, J. Zhu, and J. Chen, “Spargeattn: Accurate sparse attention accelerating any model inference,” arXiv preprint arXiv:2502.18137, 2025

    2025

  17. [25]

    A training-free sub-quadratic cost transformer model serving framework with hierarchically pruned attention,

    H. Lee, G. Park, Y . Lee, J. Suh, J. Kim, W. Jeong, B. Kim, H. Lee, M. Jeon, and S. J. Hwang, “A training-free sub-quadratic cost transformer model serving framework with hierarchically pruned attention,” in The Thirteenth International Conference on Learning Representations, 2025

    2025

  18. [26]

    When attention sink emerges in language models: An empirical view,

    X. Gu, T. Pang, C. Du, Q. Liu, F. Zhang, C. Du, Y . Wang, and M. Lin, “When attention sink emerges in language models: An empirical view,” in The Thirteenth International Conference on Learning Represen- tations, 2025

    2025

  19. [27]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models,

    Z. Zhang, Y . Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y . Tian, C. Ré, C. Barrett, et al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,” Advances in Neural Information Processing Systems, vol. 36, pp. 34661–34710, 2023

    2023

  20. [28]

    Snapkv: Llm knows what you are looking for before generation,

    Y . Li, Y . Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen, “Snapkv: Llm knows what you are looking for before generation,” Advances in Neural Information Processing Systems, vol. 37, pp. 22947–22970, 2024

    2024

  21. [29]

    Pqcache: Product quantization- based kvcache for long context llm inference,

    H. Zhang, X. Ji, Y . Chen, F. Fu, X. Miao, X. Nie, W. Chen, and B. Cui, “Pqcache: Product quantization- based kvcache for long context llm inference,” arXiv preprint arXiv:2407.12820, 2024

    2024 arXiv

  22. [30]

    Speculative prefill: Turbocharging ttft with lightweight and training-free token importance estimation,

    J. Liu, B. Chen, and C. Zhang, “Speculative prefill: Turbocharging ttft with lightweight and training-free token importance estimation,” arXiv preprint arXiv:2502.02789, 2025. 12

    2025 arXiv

  23. [31]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei,et al., “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  24. [32]

    Characterizing prompt compression methods for long context inference,

    S. Jha, L. E. Erdogan, S. Kim, K. Keutzer, and A. Gholami, “Characterizing prompt compression methods for long context inference,” arXiv preprint arXiv:2407.08892, 2024

    2024 arXiv

  25. [33]

    Discovering the gems in early layers: Accelerating long-context llms with 1000x input token reduction,

    Z. Shi, Y . Ming, X.-P. Nguyen, Y . Liang, and S. Joty, “Discovering the gems in early layers: Accelerating long-context llms with 1000x input token reduction,” arXiv preprint arXiv:2409.17422, 2024

    2024 arXiv

  26. [34]

    Llmlingua: Compressing prompts for accelerated inference of large language models,

    H. Jiang, Q. Wu, C.-Y . Lin, Y . Yang, and L. Qiu, “Llmlingua: Compressing prompts for accelerated inference of large language models,” arXiv preprint arXiv:2310.05736, 2023

    2023 arXiv

  27. [35]

    Compressing context to enhance inference efficiency of large language models,

    Y . Li, B. Dong, C. Lin, and F. Guerin, “Compressing context to enhance inference efficiency of large language models,” arXiv preprint arXiv:2310.06201, 2023

    2023 arXiv

  28. [36]

    Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches,

    J. Yuan, H. Liu, S. Zhong, Y .-N. Chuang, S. Li, G. Wang, D. Le, H. Jin, V . Chaudhary, Z. Xu,et al., “Kv cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches,” arXiv preprint arXiv:2407.01527, 2024

    2024 arXiv

  29. [37]

    Moa: Mix- ture of sparse attention for automatic large language model compression,

    T. Fu, H. Huang, X. Ning, G. Zhang, B. Chen, T. Wu, H. Wang, Z. Huang, S. Li, S. Yan,et al., “Moa: Mix- ture of sparse attention for automatic large language model compression,”arXiv preprint arXiv:2406.14909, 2024

    2024

  30. [38]

    Duoattention: Efficient long-context llm inference with retrieval and streaming heads,

    G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y . Fu, and S. Han, “Duoattention: Efficient long-context llm inference with retrieval and streaming heads,” arXiv preprint arXiv:2410.10819, 2024

    2024 arXiv

  31. [39]

    Seerattention: Learning intrinsic sparse attention in your llms,

    Y . Gao, Z. Zeng, D. Du, S. Cao, P. Zhou, J. Qi, J. Lai, H. K.-H. So, T. Cao, F. Yang,et al., “Seerattention: Learning intrinsic sparse attention in your llms,” arXiv preprint arXiv:2410.13276, 2024

    2024 arXiv

  32. [40]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention,

    J. Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y . Wei, L. Wang, Z. Xiao,et al., “Native sparse attention: Hardware-aligned and natively trainable sparse attention,” arXiv preprint arXiv:2502.11089, 2025

    2025 arXiv

  33. [41]

    Moba: Mixture of block attention for long-context llms,

    E. Lu, Z. Jiang, J. Liu, Y . Du, T. Jiang, C. Hong, S. Liu, W. He, E. Yuan, Y . Wang,et al., “Moba: Mixture of block attention for long-context llms,” arXiv preprint arXiv:2502.13189, 2025

    2025 arXiv

  34. [42]

    Rwkv: Reinventing rnns for the transformer era,

    B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, M. Grella, et al., “Rwkv: Reinventing rnns for the transformer era,” arXiv preprint arXiv:2305.13048, 2023

    2023 arXiv

  35. [43]

    Gated linear attention transformers with hardware- efficient training,

    S. Yang, B. Wang, Y . Shen, R. Panda, and Y . Kim, “Gated linear attention transformers with hardware- efficient training,” arXiv preprint arXiv:2312.06635, 2023

    2023 arXiv

  36. [44]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” inFirst Confer- ence on Language Modeling, 2024

    2024

  37. [45]

    Transformers are ssms: generalized models and efficient algorithms through structured state space duality,

    T. Dao and A. Gu, “Transformers are ssms: generalized models and efficient algorithms through structured state space duality,” inProceedings of the 41st International Conference on Machine Learning, ICML’24, JMLR.org, 2024

    2024

  38. [46]

    Sparq attention: Bandwidth- efficient llm inference,

    L. Ribar, I. Chelombiev, L. Hudlass-Galley, C. Blake, C. Luschi, and D. Orr, “Sparq attention: Bandwidth- efficient llm inference,” arXiv preprint arXiv:2312.04985, 2023

    2023 arXiv

  39. [47]

    {InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management,

    W. Lee, J. Lee, J. Seo, and J. Sim, “{InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache management,” in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 155–172, 2024

    2024

  40. [48]

    MagicPIG: LSH sampling for efficient LLM generation,

    Z. Chen, R. Sadhukhan, Z. Ye, Y . Zhou, J. Zhang, N. Nolte, Y . Tian, M. Douze, L. Bottou, Z. Jia, and B. Chen, “MagicPIG: LSH sampling for efficient LLM generation,” in The Thirteenth International Conference on Learning Representations, 2025

    2025

  41. [49]

    Retrievalattention: Accelerating long-context llm inference via vector retrieval,

    D. Liu, M. Chen, B. Lu, H. Jiang, Z. Han, Q. Zhang, Q. Chen, C. Zhang, B. Ding, K. Zhang, et al., “Retrievalattention: Accelerating long-context llm inference via vector retrieval,” arXiv preprint arXiv:2409.10516, 2024

    2024 arXiv

  42. [50]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,

    Z. Liu, A. Desai, F. Liao, W. Wang, V . Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava, “Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,” Advances in Neural Information Processing Systems, vol. 36, pp. 52342–52364, 2023. 13

    2023

  43. [51]

    Model tells you what to discard: Adaptive kv cache compression for llms,

    S. Ge, Y . Zhang, L. Liu, M. Zhang, J. Han, and J. Gao, “Model tells you what to discard: Adaptive kv cache compression for llms,” arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  44. [52]

    A simple and effectivel_2 norm-based strategy for kv cache compression,

    A. Devoto, Y . Zhao, S. Scardapane, and P. Minervini, “A simple and effectivel_2 norm-based strategy for kv cache compression,” arXiv preprint arXiv:2406.11430, 2024

    2024 arXiv

  45. [53]

    Cam: Cache merging for memory-efficient llms inference,

    Y . Zhang, Y . Du, G. Luo, Y . Zhong, Z. Zhang, S. Liu, and R. Ji, “Cam: Cache merging for memory-efficient llms inference,” in Forty-first International Conference on Machine Learning, 2024

    2024

  46. [54]

    Subgen: Token generation in sublinear time and memory,

    A. Zandieh, I. Han, V . Mirrokni, and A. Karbasi, “Subgen: Token generation in sublinear time and memory,” arXiv preprint arXiv:2402.06082, 2024

    2024 arXiv

  47. [55]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in neural information processing systems, vol. 35, pp. 16344–16359, 2022

    2022

  48. [56]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  49. [57]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision,

    J. Shah, G. Bikshandi, Y . Zhang, V . Thakkar, P. Ramani, and T. Dao, “Flashattention-3: Fast and accurate attention with asynchrony and low-precision,”Advances in Neural Information Processing Systems, vol. 37, pp. 68658–68685, 2024

    2024

  50. [58]

    Lean attention: Hardware-aware scalable attention mechanism for the decode-phase of transformers,

    R. Sanovar, S. Bharadwaj, R. S. Amant, V . Rühle, and S. Rajmohan, “Lean attention: Hardware-aware scalable attention mechanism for the decode-phase of transformers,” arXiv preprint arXiv:2405.10480, 2024

    2024 arXiv

  51. [59]

    Sageattention2 technical report: Accurate 4 bit attention for plug-and-play inference acceleration,

    J. Zhang, H. Huang, P. Zhang, J. Wei, J. Zhu, and J. Chen, “Sageattention2 technical report: Accurate 4 bit attention for plug-and-play inference acceleration,” arXiv preprint arXiv:2411.10958, 2024

    2024

  52. [60]

    Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration,

    J. Zhang, J. wei, P. Zhang, J. Zhu, and J. Chen, “Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration,” in The Thirteenth International Conference on Learning Representations, 2025

    2025

  53. [61]

    Triton: an intermediate language and compiler for tiled neural network computations,

    P. Tillet, H. T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” in Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, (New York, NY , USA), p. 10–19, Asso...

    2019

  54. [62]

    Transformers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-art nat...

    2020

  55. [63]

    Megatron-lm: Training multi-billion parameter language models using model parallelism,

    M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,”arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  56. [64]

    Distflashattn: Distributed memory-efficient attention for long-context llms training,

    D. Li, R. Shao, A. Xie, E. P. Xing, X. Ma, I. Stoica, J. E. Gonzalez, and H. Zhang, “Distflashattn: Distributed memory-efficient attention for long-context llms training,”arXiv preprint arXiv:2310.03294, 2023

    2023 arXiv

  57. [65]

    LongBench: A bilingual, multitask benchmark for long context understanding,

    Y . Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y . Dong, J. Tang, and J. Li, “LongBench: A bilingual, multitask benchmark for long context understanding,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Lingu...

    2024

  58. [66]

    ∞bench: Extending long context evaluation beyond 100k tokens,

    X. Zhang, Y . Chen, S. Hu, Z. Xu, J. Chen, M. K. Hao, X. Han, Z. L. Thai, S. Wang, Z. Liu, and M. Sun, “∞bench: Extending long context evaluation beyond 100k tokens,” 2024

    2024

  59. [67]

    Needle in a haystack - pressure testing llms,

    G. Kamradt, “Needle in a haystack - pressure testing llms,” 2023. 14 A Limitation Due to our method’s reliance on high-throughput 4-bit Tensor Core instructions to accelerate the Selection-Pass, it may lose its performance advantage on hardware that does not support efficient ...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.