REVIEW 28 cited by
XAttention: Block Sparse Attention with Antidiagonal Scoring
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
XAttention: Block Sparse Attention with Antidiagonal Scoring
read the original abstract
Long-Context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention's quadratic complexity. Block-sparse attention mitigates this by focusing computation on critical regions, yet existing methods struggle with balancing accuracy and efficiency due to costly block importance measurements. In this paper, we introduce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention. XAttention's key innovation is the insight that the sum of antidiagonal values (i.e., from the lower-left to upper-right) in the attention matrix provides a powerful proxy for block importance. This allows for precise identification and pruning of non-essential blocks, resulting in high sparsity and dramatically accelerated inference. Across comprehensive evaluations on demanding long-context benchmarks-including RULER and LongBench for language, VideoMME for video understanding, and VBench for video generation. XAttention achieves accuracy comparable to full attention while delivering substantial computational gains. We demonstrate up to 13.5x acceleration in attention computation. These results underscore XAttention's ability to unlock the practical potential of block sparse attention, paving the way for scalable and efficient deployment of LCTMs in real-world applications. Code is available at https://github.com/mit-han-lab/x-attention.
Forward citations
Cited by 28 Pith papers
-
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
DFSAttn is a training-free framework for dynamic fine-grained sparse attention in video DiTs that achieves up to 2.1x speedup while preserving generation quality via Hilbert reordering, hierarchical scoring, and adapt...
-
HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention
HASTE delivers up to 1.93x speedup on Wan2.1 video DiTs via head-wise adaptive sparse attention using temporal mask reuse and error-guided per-head calibration while preserving video quality.
-
AsyncSparse: Accelerating Sparse Matrix-Matrix Multiplication on Asynchronous GPU Architectures
AsyncSparse presents BCSR and WCSR kernels that use TMA and warp specialization to accelerate SpMM, outperforming prior libraries by 1.47-6.24x on SuiteSparse and achieving 2.66x end-to-end speedup on Qwen2.5-7B at 90...
-
Efficient Video Diffusion Models: Advancements and Challenges
A survey that groups efficient video diffusion methods into four paradigms—step distillation, efficient attention, model compression, and cache/trajectory optimization—and outlines open challenges for practical use.
-
Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering
Attention sparsity in video DiTs is an input-stable layer-wise property, enabling offline profiling and online bidirectional QK co-clustering for up to 1.93x speedup with PSNR up to 29 dB.
-
Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Exact Flow Linear Attention derives a closed-form exact update for delta-rule linear attention from continuous-time dynamics, removing Euler discretization error while preserving linear complexity and structure.
-
FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models
FG-Attn accelerates video DiT inference by computing attention over only the relevant Mx1 key slices per query group, using an asynchronous gather-load kernel, and reports up to 1.65x end-to-end speedup with negligibl...
-
CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention
CoSA couples an order-aware sparse-mask proxy with a skip-capable attention kernel, achieving higher long-context accuracy at lower compute budgets than existing sparse-attention baselines.
-
PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention
Sharing one proxy full-prefix scan across nearby query groups matches dense DSA indexer accuracy while accelerating indexing up to 4× and end-to-end latency up to 1.6×.
-
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
Louver is a new index for LLM KV caches that guarantees zero false negatives for keys above a relevance threshold, runs faster than prior sparse and some dense attention methods, and integrates lightly into existing p...
-
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
Louver is a new index structure that guarantees zero false negatives for sparse attention in LLM KV caches by casting the problem as halfspace range searching.
-
MoE-Hub: Taming Software Complexity for Seamless MoE Overlap with Hardware-Accelerated Communication on Multi-GPU Systems
MoE-Hub enables seamless MoE communication overlap via hardware-accelerated destination-agnostic data transmission, delivering 1.40x-3.08x per-layer and 1.21x-1.98x end-to-end speedups over prior systems.
-
SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
SparKV reduces time-to-first-token by 1.3x-5.1x and energy use by 1.5x-3.3x for on-device LLM inference by adaptively choosing between cloud KV streaming and local computation while overlapping execution and adjusting...
-
Prism: Spectral-Aware Block-Sparse Attention
Mean-pooling under RoPE attenuates high-frequency position signals (a spectral blind spot), and Prism's dual-band RMS-calibrated block selection recovers enough signal to match full attention at up to 5.1× prefill speedup.
-
ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
ParisKV retrieves top-K KV pairs on GPU using analytic sign-pattern centroids and calibrated 4-bit reranking, matching full attention accuracy in most settings while cutting million-token decode latency up to 44x.
-
Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention
Light Forcing accelerates autoregressive video diffusion with a chunk-aware sparsity schedule and hierarchical frame-level and block-level attention selection, matching dense-attention quality at 1.2–1.3× speedup.
-
Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
Token Sparse Attention accelerates LLM prefill by per-head token selection with reversible decompression, achieving up to 3.2x attention speedup at 128K with minimal accuracy loss.
-
Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
Token Sparse Attention uses dynamic per-head token compression and decompression during attention to achieve up to 3.23x speedup at 128K context with under 1% accuracy loss.
-
Kimi Linear: An Expressive, Efficient Attention Architecture
Kimi Linear hybridizes linear attention with a new KDA module to beat full attention on tasks while slashing KV cache by 75% and speeding decoding up to 6x.
-
MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training
MTraining scales LLM training to 512K-token contexts on 32 A100 GPUs by integrating dynamic sparse training patterns with balanced and hierarchical sparse ring attention, achieving up to 6x throughput gains without ac...
-
Accelerating Prefilling via Decoding-time Contribution Sparsity
TriangleMix exploits decoding-time contribution sparsity via a training-free static attention pattern to accelerate LLM prefilling with nearly lossless performance.
-
PrunePath: Towards Highly Structured Sparse Language Models
PrunePath introduces budget-adaptive structured sparsification for FFN layers via softmax routing and cumulative-mass thresholds on top of MoEfication, with Triton kernels for inference speedups.
-
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
Fluxion achieves 1.5x-3.7x speedup in long-context LLM inference with CPU KV caches while limiting accuracy degradation to at most 0.26 relative to full attention.
-
Ride the Wave: Precision-Allocated Sparse Attention for Smooth Video Generation
PASA uses curvature-aware dynamic budgeting, grouped approximations, and stochastic attention routing to accelerate video diffusion transformers while eliminating temporal flickering from sparse patterns.
-
Stem: Rethinking Causal Information Flow in Sparse Attention
Stem prunes causal attention with a position-decaying budget and a value-magnitude-aware score, matching dense accuracy on LongBench/RULER at a 25-31% compute budget.
-
Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention
Sparse attention with chunk-aware sparsity growth and hierarchical frame/block selection accelerates autoregressive video diffusion at ~1.3x with VBench quality on par with dense attention.
-
UltraImageGen: Efficient Ultra-High-Resolution Image Generation with Hierarchical Local Attention
A pretrained FLUX diffusion model is adapted with local-window attention plus low-resolution global guidance, allowing 4K text-to-image generation from 1K-only training data at about 2x lower cost.
-
ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference
ShadowNPU presents shadowAttn, a co-designed sparse attention system that uses NPU pilot compute and techniques like graph bucketing and per-head sparsity to minimize CPU/GPU fallback during on-device LLM inference wh...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.