REVIEW 21 cited by
Self-attention Does Not Need $O(n^2)$ Memory
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
We present a very simple algorithm for attention that requires $O(1)$ memory with respect to sequence length and an extension to self-attention that requires $O(\log n)$ memory. This is in contrast with the frequently stated belief that self-attention requires $O(n^2)$ memory. While the time complexity is still $O(n^2)$, device memory rather than compute capability is often the limiting factor on modern accelerators. Thus, reducing the memory requirements of attention allows processing of longer sequences than might otherwise be feasible. We provide a practical implementation for accelerators that requires $O(\sqrt{n})$ memory, is numerically stable, and is within a few percent of the runtime of the standard implementation of attention. We also demonstrate how to differentiate the function while remaining memory-efficient. For sequence length 16384, the memory overhead of self-attention is reduced by 59X for inference and by 32X for differentiation.
Forward citations
Cited by 21 Pith papers
-
Intrinsic and Triangulation-Agnostic Attention: A Simple and Powerful Approach for Learning on Meshes
Mass-weighted FEM attention on intrinsic mesh features is triangulation-agnostic and beats current mesh and point-cloud baselines on several geometry-learning benchmarks.
-
Design-CP: Context Parallelism for Design of Protein Nanoparticles
Context-parallel inference for RFdiffusion 3 enables end-to-end all-atom design of large symmetric protein nanoparticles on multi-GPU hardware without retraining.
-
Custom Algorithm-based Fault Tolerance for Attention Layers in Transformers
Flash-ABFT verifies an entire transformer attention layer with one fused checksum that covers softmax and all three matrix products, reporting 96-99% fault detection at under 5.3% area overhead.
-
Local Representative Token Guided Merging for Text-to-Image Generation
ReToM merges tokens around a similarity-selected representative token in adaptive local windows, improving Stable Diffusion FID from 37.02 to 34.89 at comparable inference speed.
-
Inter2Former: Dynamic Hybrid Attention for Efficient High-Precision Interactive
A new interactive segmentation decoder that routes computation to boundary regions, using binary quantization attention and mixture-of-experts, achieves state-of-the-art accuracy with CPU-friendly latency.
-
HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation
HMAR is an image generator that builds each resolution scale from the previous scale and refines it with masked prediction, matching or improving ImageNet FID/IS versus VAR with faster training and inference.
-
Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators
A fused exponential-multiplication hardware unit using logarithmic quantization and exponent adjustment reduces FlashAttention accelerator area by about 29% and power by about 18% without visible accuracy loss on GLUE.
-
LightNobel: Improving Sequence Length Limitation in Protein Structure Prediction Model via Adaptive Activation Quantization
LightNobel combines token-wise adaptive activation quantization with a custom accelerator to run protein structure prediction on sequences up to about 10,000 amino acids within 80GB memory.
-
Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light
Generalized Neighborhood Attention adds a stride parameter to neighborhood attention, aligning windows with GPU tiles so sparse attention achieves its full theoretical speedup; a Blackwell kernel delivers 1.26x-1.63x ...
-
Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
Finetuning forgetting follows a multiplicative scaling law in model size, finetuning tokens, and injected pretraining fraction, with 1% injection nearly eliminating forgetting.
-
AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration
AsymRnR selectively reduces query and key/value tokens in video DiT attention to cut FLOPs and latency by 10 to 30 percent with minor or no VBench score change.
-
Token Cropr: Faster ViTs for Quite a Few Tasks
Cropr uses removable auxiliary heads to learn task-relevant token pruning in ViTs, achieving 1.5-4x speedups with small accuracy drops across image classification, semantic segmentation, and object detection.
-
Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples
Directly supervising a consistency model against an ODE solver lowers ODE solving error yet degrades image quality, so better ODE solving does not imply better samples.
-
FLASepformer: Efficient Speech Separation with Gated Focused Linear Attention Transformer
Replacing standard attention with gated focused linear attention makes speech separation models run faster and use less memory while keeping separation quality close to the state of the art.
-
Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
Meta reports EAGLE-based speculative decoding optimizations for Llama models, achieving state-of-the-art latency (about 4 ms/token for Llama4 Maverick) and 1.4-2.0x speedups for large batches.
-
Modality Agnostic Efficient Long Range Encoder
MAELRE combines progressive token merging with a switch from cheap linear attention to standard attention, reporting lower memory and FLOPs than existing long-context encoders on four modality classification benchmarks.
-
TransAct V2: Lifelong User Action Sequence Modeling on Pinterest Recommendation
TransAct V2 adds lifelong user sequences and a next-action loss to Pinterest's CTR model, reporting online gains of +6.35% repin volume and -12.80% hide volume.
-
FLASH-D: FlashAttention with Hidden Softmax Division
FlashAttention can be rewritten exactly so each softmax weight is a sigmoid of a neighboring score difference plus a log-weight term, removing max subtraction and simplifying hardware.
-
Parallel Sequence Modeling via Generalized Spatial Propagation Network
GSPN is a 2D line-scan propagation mechanism for vision that reports SOTA ImageNet accuracy, strong class-conditional generation FID, and large high-resolution text-to-image speedups.
-
A Roadmap for Tamed Interactions with Large Language Models
A roadmap paper proposes a domain-specific language, LSL, to make LLM interactions programmable, analyzable, and verifiable, but provides no implementation or evaluation.
-
MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models
MOM reduces prefill memory by chunking MLP layers into mini-sequences and offloading KV caches, stretching Llama-3.2-8B to 455k tokens on a single 80GB A100 with identical outputs.
Discussion (0). Continue with ORCID to comment.