Pith. sign in

REVIEW 6 cited by

Transformer Acceleration with Dynamic Sparse Attention

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2110.11299 v1 pith:DTLHL7QX submitted 2021-10-21 cs.LG

Liu Liu , Zheng Qu , Zhaodong Chen , Yufei Ding , Yuan Xie This is my paper
classification cs.LG
keywords attentiondynamicsparsepatternstransformertransformersachieveapplications
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformers are the mainstream of NLP applications and are becoming increasingly popular in other domains such as Computer Vision. Despite the improvements in model quality, the enormous computation costs make Transformers difficult at deployment, especially when the sequence length is large in emerging applications. Processing attention mechanism as the essential component of Transformer is the bottleneck of execution due to the quadratic complexity. Prior art explores sparse patterns in attention to support long sequence modeling, but those pieces of work are on static or fixed patterns. We demonstrate that the sparse patterns are dynamic, depending on input sequences. Thus, we propose the Dynamic Sparse Attention (DSA) that can efficiently exploit the dynamic sparsity in the attention of Transformers. Compared with other methods, our approach can achieve better trade-offs between accuracy and model complexity. Moving forward, we identify challenges and provide solutions to implement DSA on existing hardware (GPUs) and specialized hardware in order to achieve practical speedup and efficiency improvements for Transformer execution.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. KV Cache Offloading for Context-Intensive Tasks

    cs.LG 2026-04 conditional novelty 7.0 of 10

    KV offloading degrades accuracy on context-intensive tasks due to low-rank key projections and unreliable landmarks; a simpler alternative improves results across models and benchmarks.

  2. Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding

    cs.AR 2026-04 unverdicted novelty 6.0 of 10

    Salca is a new ASIC accelerator that achieves 3.82× speedup and 74.19× energy efficiency over A100 for long-context attention via dual-compression dynamic sparse attention and pipelined hardware.

  3. AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    AdaCluster delivers a training-free adaptive query-key clustering framework for sparse attention in video DiTs, yielding 1.67-4.31x inference speedup with negligible quality loss on CogVideoX-2B, HunyuanVideo, and Wan-2.1.

  4. KV Cache Offloading for Context-Intensive Tasks

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    KV offloading hurts accuracy on context-heavy tasks due to low-rank key projections and bad landmarks, but a simpler strategy recovers performance across models.

  5. KV Cache Offloading for Context-Intensive Tasks

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    KV offloading degrades performance on context-intensive tasks due to low-rank key projections and unreliable landmarks, but a simpler alternative strategy restores accuracy across LLM families.

  6. KV Cache Offloading for Context-Intensive Tasks

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    KV offloading hurts accuracy on context-heavy tasks because of low-rank key projections and bad landmarks, but a simpler strategy improves results across models and benchmarks.

Pith tools