REVIEW 10 cited by
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
read the original abstract
Accommodating long sequences efficiently in autoregressive Transformers, especially within an extended context window, poses significant challenges due to the quadratic computational complexity and substantial KV memory requirements inherent in self-attention mechanisms. In this work, we introduce SPARSEK Attention, a novel sparse attention mechanism designed to overcome these computational and memory obstacles while maintaining performance. Our approach integrates a scoring network and a differentiable top-k mask operator, SPARSEK, to select a constant number of KV pairs for each query, thereby enabling gradient-based optimization. As a result, SPARSEK Attention offers linear time complexity and constant memory footprint during generation. Experimental results reveal that SPARSEK Attention outperforms previous sparse attention methods and provides significant speed improvements during both training and inference, particularly in language modeling and downstream tasks. Furthermore, our method can be seamlessly integrated into pre-trained Large Language Models (LLMs) with minimal fine-tuning, offering a practical solution for effectively managing long-range dependencies in diverse applications.
Forward citations
Cited by 10 Pith papers
-
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Per-block gzip compression ratios, computed on the full input, select 'literal' blocks for long-range attention and yield 1.71 BPB on PG-19 byte-level LM at 92M parameters—but the evaluation assumes access to future content.
-
ElasticDiT: Efficient Diffusion Transformers via Elastic Architecture and Sparse Attention for High-Resolution Image Generation on Mobile Devices
ElasticDiT introduces an elastic DiT architecture with adjustable spatial compression and block depth plus Shift Sparse Block Attention and a distilled VAE to enable a single model to cover multiple fidelity-latency p...
-
TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchs
TempoNet uses a slack-quantized Transformer with deep Q-learning and sparse attention to improve deadline fulfillment rates over traditional and neural schedulers in mixed-criticality real-time workloads.
-
Chimera: Neuro-Symbolic Attention Primitives for Trustworthy Dataplane Intelligence
Chimera combines kernelized attention approximations with symbolic fusion mechanisms to enable high-fidelity neuro-symbolic inference inside commodity programmable switches.
-
MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matching
MatchAttention replaces global cross-attention with a linearly-scaling, learnable-offset windowed attention whose relative position field is the predicted disparity/flow, yielding state-of-the-art stereo/flow accuracy...
-
LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
LISA replaces standard attention with linear attention plus indexer-selected sparse attention, claiming O(nM) inference cost and accuracy gains, but the complexity analysis ignores the indexer's full-context scoring.
-
SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers
SCOUT matches full-attention transformer scores on long-context language modeling and reasoning benchmarks at 400M and 1.3B scales while attending only to strided checkpoint tokens.
-
On Efficient Variants of Segment Anything Model: A Survey
A survey that reviews efficient variants of the Segment Anything Model, categorizes acceleration strategies, and provides a unified hardware evaluation on benchmarks.
-
E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning
E2LLM uses encoder-based soft prompt compression for long contexts to improve LLM reasoning on tasks like summarization and QA while maintaining efficiency.
-
A Survey of Scaling in Large Language Model Reasoning
A survey categorizing scaling in LLM reasoning across input size, steps, rounds, training, and future directions, noting that scaling can negatively affect performance.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.