Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that selecting the top-k keys for sparse attention can be done with 128-bit learned binary hash codes and Hamming distance, because only the relative order of query-key scores matters, and that this yields up to 7.2x…

desk verdict A genuinely new and well-engineered sparse-attention method whose central ordinal-preservation claim is overstated; deserves peer review with revisions. read the letter →

arxiv 2506.02572 v1 pith:XNSUUD3V submitted 2025-06-03 cs.LG cs.AI

Ping Gong , Jiawei Yi , Shengnan Wang , Juncheng Zhang , Zewen Jin , Ouxiang Zhou , Ruibo Liu , Guanbin Xu
show 8 more authors
Youhui Bai Bowen Ye Kun Yuan Tong Yang Gong Zhang Renhai Chen Feng Wu Cheng Li
This is my paper
classification cs.LGcs.AI
keywords top-kattentionlearningtohashLLMinferenceKVcacheHammingdistancesparsehardwareefficiencybinarycodes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that top-k attention does not need accurate query-key (qk) scores, only their relative order, and that learned binary hash codes can supply that order cheaply. It trains a small per-head hash network on sampled real query-key pairs to map every query and key to a 128-bit code, then ranks cached keys by Hamming distance and runs attention on the top k. The paper reports up to 7.2x decoding speedup over full attention at a 1.56% token budget, with accuracy close to the dense baseline on LongBench, RULER, InfiniteBench, LongBench-v2, and needle-in-a-haystack tests across Llama-2, Llama-3.1, and Qwen models. If correct, this makes sparse attention a practical memory-bandwidth win rather than an accuracy gamble.

What carries the argument

The load-bearing object is a learned hash function $h(x) = 2\cdot\mathrm{Sigmoid}(\sigma \cdot xW_H) - 1$ per attention head, producing $r=128$ binary bits. It is trained with an objective that minimizes the similarity-weighted Hamming distance between query and key codes (using labels $-1$ for negatives and linearly decayed scores for positives), plus bit-balance and uncorrelation regularizers. At inference, key codes are cached; a query code is XOR-ed with all key codes and popcount/reduction yields a Hamming score; top-k indices select K/V for fused FlashAttention. Kernel fusion, coalesced XOR/popcount, and fused gather-attention make the bit operations cheap enough to dominate latency.

What would settle it

On a held-out task drawn from a different distribution than the training sequences, compute exact qk scores and compare the true top-k key set with HATA's Hamming-selected top-k. If, at the paper's 1.56% budget, the overlap is far below the level needed to match dense accuracy (or if accuracy drops sharply relative to dense on such a task), the central ordering-preservation claim would be falsified; the paper currently evaluates mainly benchmarks of the same family as its training data.

Watch

Extended reading notes

Core claim

HATA's central claim is that the expensive part of top-k attention—high-fidelity approximation of qk scores—is unnecessary; only the ordering of scores matters for selecting the top k. The paper establishes this by training similarity-preserving binary hash functions per attention head, encoding queries and keys as 128-bit codes, and using Hamming distance as an ordinal proxy for qk rank. In experiments, this ordering proxy selects keys well enough that HATA equals or nearly equals dense attention across long-context benchmarks while cutting tokens read to as little as 1.56% of the KV cache, and reaches up to 7.2x decoding speedup in layer-level, end-to-end, and offloaded settings.

Load-bearing premise

The hash weights are trained on 150K–300K query-key pairs sampled from a handful of LongBench and LongBench-v2 sequences, and the method assumes this small sample preserves qk-score ordering on unseen tasks and sequences; no theoretical guarantee is given.

Editorial extensions

If this is right

  • At 128 hash bits, HATA matches or beats low-rank (Loki) and block-wise (Quest) top-k attention on accuracy at the same token budget, so ordering-preserving hashing becomes the strongest known sparse-attention selection rule.
  • Because only bit counts and a small gather are needed, decoding latency grows nearly independent of context length; HATA reports speedup increasing with sequence length and batch size.
  • The method plugs into existing inference engines by replacing only the attention module, and its offloading variant cuts prefill and decode time versus LSH-based offloading.
  • HATA remains accurate at token budgets as low as 0.4%, so the speed/accuracy trade-off curve is much flatter than previous estimators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper leaves implicit: the same ordinal-proxy argument should apply to any sparsity scheme that ranks tokens by relevance, so learned hash codes could replace score estimation in eviction policies, offloading prefetch, or speculative decoding.
  • Because the hash weights are fixed after training, distribution shift in later decoding (e.g., new domains or languages) would silently degrade ranking; an online update or per-task adaptation of $W_H$ is a natural test the paper does not run.
  • Hamming distance is a crude tie-breaker when many keys tie at nearby scores; at very small budgets the claim would predict a cliff, so the method's token-budget behavior is a sharp probe of the ordering-preservation thesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. HATA proposes to accelerate LLM decoding by replacing exact top-k attention with a learned binary hashing scheme. Per attention head, hash weights are trained on query-key pairs labeled from actual qk scores, and during decoding the query and all cached keys are mapped to 128-bit codes; the top-k keys are selected by Hamming distance and attention is computed only on those keys and values. The paper contributes a hardware-oriented implementation with fused kernels, reports near-dense accuracy on LongBench-e, RULER, InfiniteBench, LongBench-v2, and NIAH, and reports up to 7.2x decoding speedup over dense attention while outperforming Loki, Quest, MagicPIG, StreamingLLM, H2O, and SnapKV in most comparisons.

Significance. If the ordinal-preservation premise holds, HATA is a practically valuable contribution: it replaces expensive low-rank or block-wise score estimators with cheap bitwise operations, and the engineering work is substantial, including fused hash encoding, a custom Hamming-score kernel, and a fused gather-FlashAttention path. The paper also ships open-source code, evaluates across multiple model families (Llama-2, Llama-3.1, Qwen2.5-14B, Qwen2.5-32B), and includes ablations on token budget, hash bits, and kernel optimizations. However, the central conceptual claim that Hamming-distance ranking of learned codes preserves the relative order of qk scores is not established, the empirical support is weakened by single-run results, the training data overlap with the main benchmarks, and a mismatched MagicPIG comparison. The paper is promising but needs substantial revision before the main claims are convincing.

major comments (4)
  1. [§2.4, §3.1.2, Eq. (9), Algorithm 3] The central claim that HATA 'acquires the relative qk score order' is not supported by the presented training objective. With rbit=128, Hamming distance can take only 129 distinct values, so for a 128K-token context and a 2048-token budget, the top-k boundary can fall inside a large tied-distance bucket; Algorithm 3's TopK then selects arbitrarily among keys with identical Hamming scores. Equation (9) only encourages high-label qk pairs to be close under a weighted squared-distance loss; it provides neither a monotonicity guarantee nor a bound relating Hamming distance to qk rank. The paper should either supply such a guarantee, or provide a quantitative analysis of tie distributions and exact top-k agreement, or re-scope the claim to a coarse ordinal filter.
  2. [Appendix B.1, Tables 1-2, Section 5.2] The generalization claim is under-supported because the hash weights are trained on 11 sequences from LongBench and LongBench-v2 (Qasper, LSHT, RepoBench-P, and LongBench-v2 samples), while the main accuracy evaluation is also on LongBench-e and RULER; InfiniteBench and LongBench-v2 in Appendix A.2 belong to the same benchmark family. Section 8 acknowledges limited training-data diversity but does not address this overlap. I request an evaluation on held-out tasks explicitly disjoint from the training sequences, plus a report of whether any evaluation samples overlap with the training data; otherwise the near-dense accuracy may reflect task robustness to arbitrary selection inside tied buckets rather than validated ordinal preservation.
  3. [§5.2, Tables 1-2] All accuracy results are single-run with no error bars, no number of seeds, and no statistical significance assessment. Given that HATA's averages are often within 0.1-0.5 points of dense attention (e.g., Table 1 Llama-2 average 34.60 vs. 34.47 and Llama-3.1 average 53.94 vs. 54.10), the claim of 'maintaining model accuracy' needs variance estimates and ideally multiple seeds; otherwise small differences are not interpretable. The main tables would also benefit from an exact top-k baseline at the same token budget, as provided only for RULER-256K and LongBench-v2 in Tables 7 and 10.
  4. [§5.3, Table 3] The efficiency comparison with MagicPIG is not matched: HATA-off uses a 1.56% token budget while MagicPIG uses approximately 2-3% of the sequence length, as stated in the table caption and the surrounding text. The reported speedups therefore conflate algorithmic efficiency with a lower token budget. Please repeat the comparison at equal budgets (e.g., MagicPIG at 1.56% or HATA-off at 2-3%) and report the resulting accuracy; this is load-bearing for the claim that HATA outperforms the state of the art in both accuracy and efficiency.
minor comments (4)
  1. [Algorithm 1] The inline comment 'additionlly computing' contains a typo; it should read 'additionally computing'.
  2. [Section 3.1.1] The phrase 'we first formulate the learing-to-hash problem' contains a typo; it should read 'learning-to-hash'.
  3. [Appendix A.2] The heading 'Addtional Accuracy Results' should be spelled 'Additional Accuracy Results'.
  4. [Tables 5-11] The configuration tables would be easier to use if the MagicPIG budget were reported as a fixed token count as well as a percentage, so that readers can compare against HATA's fixed budgets without inferring sequence lengths.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HATA's hash function is explicitly trained on qk-derived similarity labels, so the claim that Hamming distance approximates qk ordering is a supervised empirical claim, not a derivation that reduces to its own inputs.

full rationale

The paper's central claim is that a learned 128-bit hash can rank keys by qk relevance. Section 3.1.2 states that training labels are constructed from qk scores ('Based on the qkScore, the top 10% of (qj ki) pairs are designated as positive samples with linearly decayed labels sj,i in [1,20], while the remaining 90% receive fixed negative labels sj,i = -1'), and Equation (9) trains the hash weights WH to minimize a weighted Hamming-distance loss. This is a supervised fitting procedure, not a derivation: the paper never asserts an analytical equivalence between Hamming distance and qk score, and Algorithm 3's use of bitcount-based Hamming scores is a trained proxy, not a mathematically forced identity. The evaluation on LongBench-e, RULER, InfiniteBench, and NIAH tests that proxy on held-out sequences; even though a few training sequences come from the same benchmarks as some evaluation tasks (Qasper, RepoBench-P, LongBench-v2), the tested sequences are not the training examples, and RULER, InfiniteBench, and NIAH provide out-of-distribution evidence. The closest concern—that training directly on qk-derived labels makes the 'ordering' claim fitted—is how any supervised ranker works and is not circular unless the test set or predicted quantity coincides with the training labels, which it does not. The paper also does not rely on load-bearing self-citations: learning-to-hash is attributed to external prior work (Wang et al., 2012; Weiss et al., 2008). The limited resolution of 128-bit Hamming distances and the absence of a monotonicity guarantee are correctness and robustness risks, not circularity.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities; it relies on learned hash codes and a set of training hyperparameters. The load-bearing fitted quantity is the hash weight matrix per attention head.

free parameters (7)
  • Hash weight matrices W_H (per attention head) = Trained on 150K-300K qk pairs per model
    The hash function h(x)=sign(xW_H) is the core fitted model; its weights are learned to minimize the weighted Hamming loss with qk-score-derived labels.
  • Hash length rbit = 128
    Chosen by ablation (Appendix A.3); accuracy improves from 32 to 128 bits and plateaus.
  • Sigmoid scale sigma = 0.1
    Hyperparameter in the relaxed hash function; set to avoid gradient vanishing.
  • Loss weight epsilon = 0.01
    Weight on the similarity-preservation term in Equation (8).
  • Loss weight eta = 2.0
    Weight on the bit-balance penalty in Equation (8).
  • Loss weight lambda = 1.0
    Weight on the uncorrelation penalty in Equation (8).
  • Positive-sample threshold and label range = Top 10% positive, labels in [1,20]; negatives -1
    Training labels are constructed from qk scores with these thresholds; the hash ranking behavior depends on this choice.
assumptions (4)
  • domain assumption Similarity-preserving hashing preserves qk ranking: minimizing the weighted Hamming distance between hash codes for high-similarity q/k pairs yields a Hamming-distance ranking that matches the qk score ranking well enough for top-k selection.
    This is the core assumption behind the method; no theorem or bound is provided. It appears in Section 3.1.1 and is the basis for using Hamming distance in Algorithm 3.
  • domain assumption Top-k token selection is sufficient to preserve LLM output quality.
    The method inherits this from prior top-k attention work (Gupta et al., 2021), cited in Section 2.3; the paper does not re-derive it.
  • standard math The sigmoid relaxation of the sign function is a valid proxy for training binary hash codes.
    Standard technique in learning-to-hash; used in Equation (7).
  • ad hoc to paper Hash weights trained on a small set of sequences from LongBench-family tasks generalize to other long-context tasks and models.
    Section 3.1.2 and Appendix B.1 describe training on 5-8 sequences; the cross-task results in Section 5.2 assume this generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference." pith.science (2026). https://pith.science/paper/XNSUUD3V

@misc{pith2026250602572,
  author       = {Pith},
  title        = {Pith review of: HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNSUUD3V}},
  note         = {Machine review of arXiv:2506.02572}
}
abstract

Large Language Models (LLMs) have emerged as a pivotal research area, yet the attention module remains a critical bottleneck in LLM inference, even with techniques like KVCache to mitigate redundant computations. While various top-$k$ attention mechanisms have been proposed to accelerate LLM inference by exploiting the inherent sparsity of attention, they often struggled to strike a balance between efficiency and accuracy. In this paper, we introduce HATA (Hash-Aware Top-$k$ Attention), a novel approach that systematically integrates low-overhead learning-to-hash techniques into the Top-$k$ attention process. Different from the existing top-k attention methods which are devoted to seeking an absolute estimation of qk score, typically with a great cost, HATA maps queries and keys into binary hash codes, and acquires the relative qk score order with a quite low cost, which is sufficient for realizing top-k attention. Extensive experiments demonstrate that HATA achieves up to 7.2$\times$ speedup compared to vanilla full attention while maintaining model accuracy. In addition, HATA outperforms the state-of-the-art top-$k$ attention methods in both accuracy and efficiency across multiple mainstream LLM models and diverse tasks. HATA is open source at https://github.com/gpzlx1/HATA.

Figures

Figures reproduced from arXiv: 2506.02572 by the authors.

Figure 1
Figure 1. Comparison of accuracy and token generation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Workflow of HATA in the decode stage. Algorithm 1 HATA Prefill Stage 1: Input: Q ∈ R s×d , K ∈ R s×d , V ∈ R s×d , key cache Kcache ∈ R 0×d , value cache V cache∈ R 0×d , key code cache Kcache H ∈ R 0×rbit/32 2: ▷ Call HashEncode to encode key 3: KH ← HashEncode(K) 4: ▷ Fill hashcode cache 5: Kcache H ← KH 6: ▷ Fill KVCache 7: Kcache ← K, V cache ← V 8: ▷ Calculate attention output 9: O ← Attention(Q, K, V) Algorith… view at source ↗
Figure 3
Figure 3. HATA’s optimizations, compared to the con [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: End-to-end performance comparison of LLM [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 7
Figure 7. Figure 7: Token budget ablation. extended the range from 32K to 128K. As shown in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 9
Figure 9. Figure 9: Performance ablation study of HATA opti [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training-Free Hashing-Based Attention via Binary Principal Components

    cs.LG 2026-08 conditional novelty 6.0 of 10

    BinaryPC uses binary principal components of keys to pick the most relevant tokens for attention, preserving accuracy with a 2% budget.

Reference graph

Works this paper leans on

41 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114--127

    2024

  4. [4]

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, and 30 others. 2024. https://doi.org/10.1145/3620665.3640366 PyTorch 2: Fast...

    arXiv 2024

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  6. [6]

    Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, and 1 others. 2024. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204

    arXiv 2024

  7. [7]

    Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and 1 others. 2024. Magicpig: Lsh sampling for efficient llm generation. arXiv preprint arXiv:2410.16179

    arXiv 2024

  8. [8]

    Tri Dao. 2023. https://arxiv.org/abs/2307.08691 Flashattention-2: Faster attention with better parallelism and work partitioning . Preprint, arXiv:2307.08691

    arXiv 2023

Show all 41 references
  1. [9]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344--16359

    2022

  2. [10]

    Aditya Desai, Shuo Yang, Alejandro Cuadron, Ana Klimovic, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. 2024. Hashattention: Semantic sparsity for faster inference. arXiv preprint arXiv:2412.14468

    2024 arXiv

  3. [11]

    Aristides Gionis, Piotr Indyk, and Rajeev Motwani. 1999. Similarity search in high dimensions via hashing. In Proceedings of the 25th International Conference on Very Large Data Bases, VLDB '99, page 518–529, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc

    1999

  4. [12]

    Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. 2021. Memory-efficient transformers via top- k attention. arXiv preprint arXiv:2106.06899

    2021 arXiv

  5. [13]

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079

    2024 arXiv

  6. [14]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

    2024 arXiv

  7. [15]

    Chaofan Lin Jiaming Tang, Yilong Zhao. 2025. Quest: Query-aware sparsity for efficient long-context llm inference. https://github.com/mit-han-lab/Quest. Accessed, May. 2025

    2025

  8. [16]

    Greg Kamradt. 2023. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/LLMTest_NeedleInAHaystack. Accessed, Feb. 2025

    2023

  9. [17]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  10. [18]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. Infinigen: Efficient generative inference of large language models with dynamic kv cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155--172

    2024

  11. [19]

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947--22970

    2024

  12. [20]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  13. [21]

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024 b . Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750

    2024 arXiv

  14. [22]

    MetaAI. 2024. Introducing llama 3.1: Our most capable models to date. https://ai.meta.com/blog/meta-llama-3-1/. Accessed, Feb. 2025

    2024

  15. [23]

    QwenTeam. 2024. Qwen2.5: A party of foundation models. https://qwenlm.github.io/blog/qwen2.5/. Accessed, Feb. 2025

    2024

  16. [24]

    QwenTeam. 2025. Qwen2.5-1m: Deploy your own qwen with context length up to 1m tokens. https://qwenlm.github.io/blog/qwen2.5-1m/. Accessed, Feb. 2025

    2025

  17. [25]

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. 2023. Sparq attention: Bandwidth-efficient llm inference. arXiv preprint arXiv:2312.04985

    2023 arXiv

  18. [26]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R \'e , Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Lear...

    2023

  19. [27]

    Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. 2024. Loki: Low-rank keys for efficient sparse attention. arXiv preprint arXiv:2406.02542

    2024 arXiv

  20. [28]

    Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. 2024. Shadowkv: Kv cache in shadows for high-throughput long-context llm inference. arXiv preprint arXiv:2410.21465

    2024 arXiv

  21. [29]

    Zhiqing Sun, Yiming Yang, and Shinjae Yoo. 2021. Sparse attention with learning to hash. In International Conference on Learning Representations

    2021

  22. [30]

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774

    2024 arXiv

  23. [31]

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10--19

    2019

  24. [32]

    Together. 2023. Llama-2-7b-32k-instruct. https://huggingface.co/togethercomputer/Llama-2-7B-32K-Instruct. Accessed, Feb. 2025

    2023

  25. [33]

    Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. 2012. Semi-supervised hashing for large-scale search. IEEE transactions on pattern analysis and machine intelligence, 34(12):2393--2406

    2012

  26. [34]

    Yair Weiss, Antonio Torralba, and Rob Fergus. 2008. Spectral hashing. Advances in neural information processing systems, 21

    2008

  27. [35]

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. In The Thirty-eighth Annual Conference on Neural Information Processi...

    2024

  28. [36]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453

    2023 arXiv

  29. [37]

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. 2025. https://arxiv.org/abs/2501.01005 Flashinfer: Efficient and customizable attention engine for llm inference serving ....

    2025 arXiv

  30. [38]

    Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, and Yingyan (Celine) Lin. 2024. When linear attention meets autoregressive decoding: towards more effective and efficient linearized large language models. In Proceedings of the 41st International Conference on Machine Lear...

    2024

  31. [39]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and 1 others. 2024 a . Infinitebench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Co...

    2024

  32. [40]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \'e , Clark Barrett, and 1 others. 2024 b . H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Informat...

    2024

  33. [41]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. https://arxiv.org/abs/2312.07104 Sglang: Efficient execution of structured language model progr...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.