Pith. sign in

REVIEW 5 major objections 6 minor 49 references

DistrAttention: An Efficient and Flexible Self-Attention Mechanism on Modern GPUs

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DistrAttention speeds up self-attention by grouping similar Q columns and summing K rows, reaching up to 37% faster than FlashAttention-2 without losing full context.

desk verdict Real idea, shaky evaluation: the LSH-along-d approach is novel, but the error numbers contradict the tables and real-activation accuracy is not established. read the letter →

arxiv 2507.17245 v1 pith:QYDUQI4Q submitted 2025-07-23 cs.LG cs.AI

Haolin Jin , Mengbai Xiao , Yuan Yuan , Xiao Zhang , Dongxiao Yu , Guanghui Zhang , Haoliang Wang This is my paper
classification cs.LGcs.AI
keywords FlashAttention-2self-attentionlocality-sensitivehashingembeddingdimensionreductionapproximateattentionGPUaccelerationlargelanguagemodelsvisiontransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that self-attention can be accelerated without discarding tokens or quantizing weights by reducing the embedding dimension $d$: it groups similar columns of $Q$, replaces each group by a single sampled column, and sums the corresponding rows of $K^\top$ before multiplying. The resulting approximate attention matrix keeps the full $N\times N$ token context, and the grouping is performed by a lightweight locality-sensitive hashing step plus a block-wise framework that fits FlashAttention-2's tiling. On synthetic data the average elementwise error of the approximated matrix is about 0.9%, and on modern GPUs the method is reported to compute self-attention up to 37% faster than FlashAttention-2. In ViT inference it is described as the fastest and most accurate approximate attention mechanism among the baselines compared, and on Llama3-1B it keeps inference time lowest with under 1% accuracy loss. The paper argues that the group size $G^*$ gives a flexible, tunable trade-off between speed and accuracy.

What carries the argument

The load-bearing object is the grouped-column approximation identity $\hat{S} = \sum_{j} \hat{q}_j \big(\sum_{i\in G_j} k_i^\top\big)$, which exploits the distributive property of matrix multiplication to cut the arithmetic of $S = \sum_i q_i k_i^\top$ by the group size while leaving the attention matrix dimensions untouched. It is realized by locality-sensitive hashing that maps each $Q$ column to a hash value via random projection, binarization, and Gray-code lookup, then sorts the hash values to obtain a permutation that places similar columns in the same group; a block-wise grouping variant repeats this per $Q$-block so that longer sequences do not rely on far-apart vectors. A block-size selection rule ties $l$ and $m$ to tensor-core-friendly multiples and to a warp-saturation constraint, which is what allows the approximation to be fused with FlashAttention-2.

What would settle it

Measure the elementwise relative error $\|\hat{S}-S\|_1/\|S\|_1$ on real $Q$ and $K$ activations from a pretrained ViT or decoder-only LLM across layers and heads without fine-tuning; if the average error substantially exceeds the paper's reported 0.87–0.9% on synthetic data, or if the downstream accuracy loss on a standard benchmark like the unfine-tuned ViT exceeds the reported figures, the central accuracy claim would be refuted.

Watch

Extended reading notes

Core claim

DistrAttention computes $\hat{S} = \sum_{j=1}^k (\hat{q}_j \sum_{i\in G_j} k_i^\top)$, an approximation of the attention matrix $S = QK^\top = \sum_{i=1}^d q_i k_i^\top$, in which the columns of $Q$ are permuted by locality-sensitive hashing and partitioned into groups $G_j$ of constant size $G^*$, one column per group is sampled as the estimate $\hat{q}_j$, and the corresponding rows of $K^\top$ are summed. Because the attention matrix retains its full $N\times N$ shape, the method preserves complete token context while reducing the number of multiplications by a factor of $G^*$. The paper wraps this in a block-wise grouping framework that limits LSH error and maps onto FlashAttention-2's block tiling, and reports that on real GPUs the self-attention computation is up to 37% faster than FlashAttention-2, that ViT inference is the fastest and most accurate among approximate attention baselines (up to 8.7% faster and up to 8.1% more accurate than the second place), and that Llama3-1B inference shows the lowest time with at most 1% accuracy loss.

Load-bearing premise

The approximation is accurate only if the columns of $Q$ (and the matching rows of $K^\top$) are similar enough along the embedding dimension that replacing a group by one sampled column while summing the $K$ rows keeps the error small; the paper validates this assumption only on randomly generated uniform matrices, and unfine-tuned ViT accuracy drops by 6–7% when it is applied directly.

Editorial extensions

If this is right

  • Self-attention on existing GPUs can be made up to about 37% faster than FlashAttention-2 without changing the model architecture, adding parameters, or reducing the sequence context.
  • The group size $G^*$ acts as a single dial that trades arithmetic savings against approximation error, giving a flexible accuracy–latency knob for inference and fine-tuning.
  • Because the full attention matrix is still materialized block-wise, DistrAttention can be dropped into fine-tuned models with at most about 1% accuracy loss in the paper's ViT and Llama3-1B experiments.
  • The speedup over FlashAttention-2 grows with token length, since longer sequences keep tensor cores busier and the LSH grouping overhead becomes a smaller fraction of total time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's accuracy rests on the untested premise that real trained $Q$ and $K$ activations exhibit column similarity along the embedding dimension; the paper demonstrates this only with synthetic matrices drawn from a uniform distribution, so genuine model activations could behave differently.
  • The reported "fastest and most accurate" comparisons are against a specific set of approximate-attention baselines on specific GPUs and models; reasonable readers should expect the ranking to shift on other hardware or with longer-context training.
  • Since the approach reduces the effective dimension of the multiplication, it could be combined with token-pruning or KV-cache compression to attack the quadratic cost from two directions at once.
  • The LSH permutation differs per $Q$ block, so the approximation error is input-dependent; this could matter for safety-critical or adversarial contexts where a particular input might land in an unlucky grouping.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes DistrAttention, an approximate self-attention computation that reduces arithmetic along the embedding dimension d by grouping similar columns of Q (via locality-sensitive hashing), replacing each group by one sampled column, and summing the corresponding rows of K^T. A block-wise variant is designed to integrate with FlashAttention-2, and block sizes are selected to balance shared-memory use, tensor-core throughput, and warp occupancy. Experiments on synthetic Q/K matrices, fine-tuned ViT/BERT/Llama models, and GPU attention-time benchmarks are used to claim up to 37% faster attention than FlashAttention-2 and state-of-the-art accuracy and speed among approximate attention methods.

Significance. If the accuracy and speedup claims hold, DistrAttention is a potentially useful approximate attention scheme: it retains full token context (unlike token pruning), introduces no extra parameters, supports a flexible accuracy-speed trade-off through group size, and is designed to drop into existing FlashAttention-2 kernels. The paper explicitly formulates the approximation in Eq. (2) and provides a broad evaluation suite covering vision, language, and multi-GPU settings. However, the current manuscript's central error numbers are internally inconsistent, and the key similarity assumption is validated only on synthetic uniform data; these issues must be resolved before the significance can be assessed.

major comments (5)
  1. [Abstract; §1; §4.2, Table 4] The abstract and introduction state that the error is from 0.13% to 0.07% as the sampling rate increases from 2 to 16, but Table 4 reports mean errors of 0.87%, 1.73%, 2.48%, and 4.96% for G*=2, 4, 8, and 16, and §4.2 says the average error is at least 0.87% and at most 4.97%. These are incompatible, and the direction is also opposite: Table 4 shows error increasing with G*. Please state which metric (pre-softmax vs post-softmax, element-wise vs norm) the 0.13–0.07 numbers refer to and reconcile them with Table 4.
  2. [§4.2; §4.6, Table 8] The central assumption that columns of Q (and rows of K^T) are similar enough for grouping is validated only on synthetic Q and K with i.i.d. uniform entries (§4.2). Table 8 provides a direct check on real pretrained activations: ViT_base_patch224 drops from 81.1% to 75.1% and ViT_base_patch384 from 84% to 77.12% when DistrAttention is used without fine-tuning, i.e., 6.0 and 6.9 points. The text in §4.6 calls this 'little degradation,' but this is not little, and it suggests the approximation error is material on real ViT activations. Please report the attention-matrix error on real trained Q/K activations, or explain why the synthetic uniform evaluation is predictive of real-data behavior.
  3. [§3.2] The paper never specifies how the representative column \hat{q}_j is chosen from each group. Eq. (3) minimizes over both grouping and estimates, but §3.2 says only 'we select one of their {q_i, i in G_j} columns as the estimate.' The choice (first, random, closest to centroid, etc.) directly affects error and reproducibility, and the LSH projection matrix's distribution and seed are also unspecified. Please specify these choices.
  4. [Introduction; §4.4, Table 7] The introduction claims that in Llama3-1B DistrAttention achieves 'the lowest inference time ... and highest accuracy (up to 0.23% higher accuracy than the second place).' Table 7 shows the opposite on accuracy: for n=256 and n=512, Hyper achieves 33.29% and 32.50% while Ours achieves 31.37% and 31.50%. Section 4.4 correctly says 'second highest accuracy,' so the introduction overstates the language-model result and should be corrected.
  5. [§4.7, Table 9] The multi-GPU experiment description is unclear: §4.7 says the matrices have H=480, N=20480, d=128, then says the matrices are 'split into multiple blocks with H=20.' It is not defined whether H is the number of heads, the batch dimension, or something else, or how the blocks are assigned to GPUs. Without this, the speedup numbers in Table 9 cannot be interpreted. Please clarify the setup.
minor comments (6)
  1. [Throughout] There are numerous typos and encoding artifacts (e.g., 'exsiting', 'effcient', 'accuray', 'sencond', 'piratical', 'higer', and the 'uni000000xx' strings in Figure 7/9 captions). A thorough copyedit is needed.
  2. [§4.2, Table 4] Table 4's mean error column shows 4.96 for G*=16 while the text says 'at most 4.97'; please make the numbers consistent.
  3. [§3.2; §4.5] The notation for sampling rate is inconsistent: the abstract and §4.2 use 'sampling rate,' §3.2 defines group size G*, and §4.5 and Figure 9 use 'Samplingrate = 2 and 3.' Define the term once and use it consistently throughout.
  4. [§3.3.1] The I/O formula is printed as I(l,m) = N/l (ld + 2Nd + ld); it should be I(l,m) = (N/l)(2ld + 2Nd) to match the stated explanation. Please fix the typesetting.
  5. [Throughout] No code, random seeds for the LSH projection, or hyperparameter details for the fine-tuning runs are provided, so the experiments are not fully reproducible as described.
  6. [§4.3; §4.6] The method is referred to as 'Vit-Standard-Ours-Simi' and 'Vit-Standard-Ours-Flash' in Table 5 but simply 'Ours' in Table 8; please standardize the naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DistrAttention's approximation is an explicit algebraic formula, validated against the exact attention matrix and benchmarked externally; no load-bearing self-citation or fitted-input-as-prediction pattern is present.

full rationale

DistrAttention derives S_hat in Eq. (2) by grouping columns of Q and summing matched rows of K^T. This is an explicit approximation formula, not a hidden restatement of the output. The error objective in Eq. (3) compares S_hat with the exact S, and the synthetic experiments in Section 4.2 measure the discrepancy directly against the true attention matrix, so the accuracy claim is not defined in terms of the approximation itself. The LSH grouping is a heuristic whose probabilistic premise is stated as an assumption rather than as a theorem proved from the conclusion. Speedup results are empirical benchmark comparisons against FlashAttention-2 and other baselines; the block-size selection in Section 3.3.1 is an analytical I/O model plus a hardware calibration rule, and it is not used to predict the accuracy numbers. No load-bearing self-citation appears: all references are to external prior work, and no uniqueness theorem or prior result by these authors is invoked to force the method. The real weakness of the paper is external validity, not circularity: the only direct S_hat error study uses synthetic uniform Q/K (Section 4.2), and Table 8 shows 6.0-6.9 point ViT accuracy drops without fine-tuning. That is a legitimate concern about whether the column-similarity assumption holds on real activations, but it is an empirical adequacy issue, not a case of the derivation reducing to its inputs. The method is self-contained as a definition and its claims are falsifiable against exact attention and end-task accuracy, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities or theoretical objects. It introduces free engineering parameters (group size, LSH projection dimension, block sizes) and relies on two assumptions about data geometry (LSH proximity and embedding-column similarity) that are only empirically checked on synthetic data.

free parameters (4)
  • Group size G* (sampling rate) = 2, 4, 8, 16 in experiments; 2 in ViT and Llama3 runs
    Controls the trade-off between speedup and approximation error; chosen by the authors, not derived.
  • LSH projection dimension N' = 16
    Set to match Tensor core tile sizes; no sensitivity analysis is provided.
  • Q block size l = 64 for ViT; 128 for attention-time tests; varies per GPU and d in Table 2
    Selected by I/O analysis plus empirical search over configurations; the rules in Eq 4 and Eq 5 are calibrated to hardware.
  • K/V block size m = 64 for d=32, 128 for d=64, 32 for d=128 in Table 2
    Same calibration as l; independent of the I/O formula.
assumptions (3)
  • domain assumption Locality-sensitive hashing: small difference in hash values implies closeness in the original N-dimensional space with high probability
    Used in Section 3.2 to justify grouping columns by sorted hash values; not proven for the particular random projection and N'=16.
  • ad hoc to paper Columns of Q (and rows of K) are similar enough along d that grouping produces a small attention-matrix error
    This is the load-bearing premise of the method; Section 4.2 only tests it on synthetic uniform data, not on real model activations.
  • ad hoc to paper The attention-matrix approximation error stays small and stable after softmax and through fine-tuning
    The paper does not analyze error propagation through softmax or training; it reports only final model accuracy after 20 epochs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DistrAttention: An Efficient and Flexible Self-Attention Mechanism on Modern GPUs." pith.science (2026). https://pith.science/paper/QYDUQI4Q

@misc{pith2026250717245,
  author       = {Pith},
  title        = {Pith review of: DistrAttention: An Efficient and Flexible Self-Attention Mechanism on Modern GPUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QYDUQI4Q}},
  note         = {Machine review of arXiv:2507.17245}
}
abstract

The Transformer architecture has revolutionized deep learning, delivering the state-of-the-art performance in areas such as natural language processing, computer vision, and time series prediction. However, its core component, self-attention, has the quadratic time complexity relative to input sequence length, which hinders the scalability of Transformers. The exsiting approaches on optimizing self-attention either discard full-contextual information or lack of flexibility. In this work, we design DistrAttention, an effcient and flexible self-attention mechanism with the full context. DistrAttention achieves this by grouping data on the embedding dimensionality, usually referred to as $d$. We realize DistrAttention with a lightweight sampling and fusion method that exploits locality-sensitive hashing to group similar data. A block-wise grouping framework is further designed to limit the errors introduced by locality sensitive hashing. By optimizing the selection of block sizes, DistrAttention could be easily integrated with FlashAttention-2, gaining high-performance on modern GPUs. We evaluate DistrAttention with extensive experiments. The results show that our method is 37% faster than FlashAttention-2 on calculating self-attention. In ViT inference, DistrAttention is the fastest and the most accurate among approximate self-attention mechanisms. In Llama3-1B, DistrAttention still achieves the lowest inference time with only 1% accuray loss.

Figures

Figures reproduced from arXiv: 2507.17245 by the authors.

Figure 1
Figure 1. We can see that as the token length increases, the self [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The memory system of a typical Nvidia GPU. SMEM [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The illustration of FlashAttention-2 [8] on a modern GPU device. (a) The output matrix Q is calculated in a double-loop. The outer loop iterates over Q blocks to calculate O blocks. To calculate an O block, the inner loop iterates over K ⊤ and V blocks. (b) Calculating rows of O is embarrassingly parallel so that iterations of the outer loop are arranged to threadblocks. Inside a threadblock, a Q block is further sp… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: An example of calculating an attention matrix ap [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: An example of grouping qs with LSH. 4 columns of Q is hashed to 4 integer values {ℎ1, ℎ2, ℎ3, ℎ4} that are further sorted to {ℎ3, ℎ2, ℎ1, ℎ4}. We then have an index permutation of {3, 2, 1, 4}. By imposing a group size 𝐺 ∗ = 2, we know that q3 and q2 are in a group whi…
Figure 6
Figure 6. Figure 6: An example of block-wise grouping. Q and K ⊤ are splitted into 3 blocks, though both the size and number of Q and K ⊤ blocks do not have to be the same. The Q blocks are grouped individually, and the rows of a K block are grouped depending on which Q block is multiplie…
Figure 7
Figure 7. Figure 7: The errors of between Sˆ and S on a pair of synthesized Q and K, whose 𝑁 = 64 and 𝑑 = 32. We vary the block size 𝑙 and the sampling rate 𝐺 ∗ when 𝐺 ∗ is 2 and 𝑙 is 2, respectively. The original Sˆ and S are shown at the left. 128, our method finds the same configuratio…
Figure 8
Figure 8. Figure 8: Loss comparison during ViT model fine-tuning with different attention optimization methods, illustrating the impact [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Comparison of time spent on computing attention between Flash2 and our approach under different [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 21 canonical work pages

  1. [1]

    Dosovitskiy Alexey. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929 (2020)

    arXiv 2020

  2. [2]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long- document transformer. arXiv preprint arXiv:2004.05150 (2020)

    arXiv 2020

  3. [3]

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoff- man. 2022. Hydra attention: Efficient attention with many heads. In European Conference on Computer Vision . Springer, 35–49

    work page 2022

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  5. [5]

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexan- der Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision . Springer, 213–229

    2020

  6. [6]

    Yingyi Chen, Qinghua Tao, Francesco Tonin, and Johan Suykens. 2024. Primal- attention: Self-attention through asymmetric kernel svd in primal representation. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  7. [7]

    Krzysztof Choromanski, Shanda Li, Valerii Likhosherstov, Kumar Avinava Dubey, Shengjie Luo, Di He, Yiming Yang, Tamas Sarlos, Thomas Weingarten, and Adrian Weller. 2024. Learning a fourier transform for linear relative positional encodings in transformers. In International Conference on Artificial Intelligence and Statistics . PMLR, 2278–2286

    work page 2024

  8. [8]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. InInternational Conference on Learning Representations (ICLR)

    2024

Show all 49 references
  1. [9]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems (NeurIPS)

    2022

  2. [10]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255

    2009

  3. [11]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805

    2018 arXiv

  4. [12]

    Dayou Du, Gu Gong, and Xiaowen Chu. 2024. Model quantization and hardware acceleration for vision transformers: A comprehensive survey. arXiv preprint arXiv:2405.00314 (2024)

    2024 arXiv

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv e-prints (2024), arXiv– 2407

    2024

  6. [14]

    Jialong Guo, Xinghao Chen, Yehui Tang, and Yunhe Wang. 2024. SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization. arXiv preprint arXiv:2405.11582 (2024)

    2024 arXiv

  7. [15]

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. 2023. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF international conference on computer vision . 5961–5971

    2023

  8. [16]

    Dongchen Han, Yifan Pu, Zhuofan Xia, Yizeng Han, Xuran Pan, Xiu Li, Jiwen Lu, Shiji Song, and Gao Huang. 2024. Bridging the divide: Reconsidering softmax and linear attention. arXiv preprint arXiv:2412.06590 (2024)

    2024 arXiv

  9. [17]

    Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. 2025. Agent attention: On the integration of softmax and linear attention. In European Conference on Computer Vision . Springer, 124– 140

    2025

  10. [18]

    Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P Woodruff, and Amir Zandieh. 2023. Hyperattention: Long-context attention in near-linear time. arXiv preprint arXiv:2310.05869 (2023)

    2023 arXiv

  11. [19]

    Grant Van Horn, macaodha, Maggie, and Wendy Kan. 2019. iNaturalist 2019 at FGVC6. https://kaggle.com/competitions/inaturalist-2019-fgvc6. Kaggle

    2019

  12. [20]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, Vol. 1. Minneapolis, Minnesota, 2

    2019

  13. [21]

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 (2020)

    2020 arXiv

  14. [22]

    Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images . Technical Report

    2009

  15. [23]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  16. [24]

    Jiedong Lang, Zhehao Guo, and Shuyu Huang. 2024. A Comprehensive Study on Quantization Techniques for Large Language Models. arXiv preprint arXiv:2411.02530 (2024)

    2024 arXiv

  17. [25]

    Wei Liu, Chaofeng Chen, Kwan-Yee K Wong, Zhizhong Su, and Junyu Han. 2016. Star-net: a spatial attention residue network for scene text recognition.. InBMVC, Vol. 2. 7

    2016

  18. [26]

    Meta. 2023. Llama 2 7B. Hugging Face Model. https://huggingface.co/meta- llama/Llama-2-7b Accessed: 2025-01-14

    2023

  19. [27]

    NVIDIA. 2021. FastTransformer. (2021)

    2021

  20. [28]

    NVIDIA Corporation. 2022. NVIDIA Ada Lovelace GPU Architecture. White Paper. NVIDIA. Accessed: 2025-01-14

    2022

  21. [29]

    Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fang- wei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. 2017. Tensorflow- serving: Flexible, high-performance ml serving. arXiv preprint arXiv:1712.06139 (2017)

    2017 arXiv

  22. [30]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing system...

    2019

  23. [31]

    Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yun- shen Wei, Baohong Lv, Xiao Luo, Yu Qiao, and Yiran Zhong. 2024. TransNormer- LLM: A Faster and Better Large Language Model with Improved TransNormer. arXiv:2307.14995 [cs.CL] https://arxiv.org/abs/2307.14995

    2024 arXiv

  24. [32]

    Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong

  25. [33]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67

    2020

  26. [34]

    David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. 2021. Searching for efficient transformers for language modeling. Advances in neural information processing systems 34 (2021), 6010–6022

    2021

  27. [35]

    Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao. 2024. A survey on transformer compression. arXiv preprint arXiv:2402.05964 (2024)

    2024 arXiv

  28. [36]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024)

    2024 arXiv

  29. [37]

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. 2021. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , Vol. 139. 10347–10357

    2021

  30. [38]

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. 2018. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition . ...

    2018

  31. [39]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  32. [40]

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Lin- former: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020)

    2020 arXiv

  33. [41]

    Christopher Wolters, Xiaoxuan Yang, Ulf Schlichtmann, and Toyotaro Suzumura

  34. [42]

    Zhilin Yang. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv preprint arXiv:1906.08237 (2019)

    2019 arXiv

  35. [43]

    arXiv:2406.08413 [cs.AR] https://arxiv.org/abs/2406.08413 Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al

    Memory Is All You Need: An Overview of Compute-in-Memory Architec- tures for Accelerating Large Language Model Inference. arXiv:2406.08413 [cs.AR] https://arxiv.org/abs/2406.08413 Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Trovato et al

    2018 arXiv

  36. [44]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Gon Chun. 2022. Orca: A distributed serving system for{Transformer-Based} generative models. (2022), 521–538

    2022

  37. [45]

    Haoran You, Yichao Fu, Zheng Wang, Amir Yazdanbakhsh, and Yingyan Ce- line Lin. 2024. When linear attention meets autoregressive decoding: Towards more effective and efficient linearized large language models. arXiv preprint arXiv:2406.07368 (2024)

    2024 arXiv

  38. [46]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing...

    2023

  39. [47]

    Amir Zandieh, Insu Han, Majid Daliri, and Amin Karbasi. 2023. Kdeformer: Ac- celerating transformers via kernel density estimation. In International Conference on Machine Learning. PMLR, 40605–40623

    2023

  40. [49]

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long se- quence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 11106–11115. Rece...

    2021

  41. [2024]

    arXiv preprint arXiv:2401.04658 (2024)

    Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401.04658 (2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.