REVIEW 6 cited by
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are typically obtained as the softmax of dot products between keys and queries. Recent work has explored alternatives to softmax attention in transformers, such as ReLU and sigmoid activations. In this work, we revisit sigmoid attention and conduct an in-depth theoretical and empirical analysis. Theoretically, we prove that transformers with sigmoid attention are universal function approximators and benefit from improved regularity compared to softmax attention. Through detailed empirical analysis, we identify stabilization of large initial attention norms during the early stages of training as a crucial factor for the successful training of models with sigmoid attention, outperforming prior attempts. We also introduce FLASHSIGMOID, a hardware-aware and memory-efficient implementation of sigmoid attention yielding a 17% inference kernel speed-up over FLASHATTENTION2 on H100 GPUs. Experiments across language, vision, and speech show that properly normalized sigmoid attention matches the strong performance of softmax attention on a wide range of domains and scales, which previous attempts at sigmoid attention were unable to fully achieve. Our work unifies prior art and establishes best practices for sigmoid attention as a drop-in softmax replacement in transformers.
Forward citations
Cited by 6 Pith papers
-
Complex-Valued Phase-Coherent Transformer
Sigmoid gating on L2-normalised complex cosine scores, with no row normalisation, generalises across long-range, positional, phase and vision tasks, though the depth-stability theorem assumes its own substance.
-
Legible-by-Construction: Attention and End-to-End Transformers
Bounding attention values into fuzzy memberships or Boolean set operations yields 44–62% crisp selective detectors and an end-to-end legible transformer at baseline quality.
-
Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization
Softmax temperature controls the rank of learned representations: high temperature induces rank-deficit bias, compresses features, hurts OOD generalization, and improves OOD detection.
-
Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations
Doubly stochastic attention is the most robust of five ViT attention mechanisms to fog corruption in relative accuracy, based on single-seed experiments on CIFAR-10, CIFAR-100, and Imagenette.
-
A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization
Single-layer linear self-attention can represent, train on, and length-generalize pairwise interaction functions under data-versatility and exact-realizability assumptions, and the paper introduces higher-order HyperA...
-
Scaling Context Requires Rethinking Attention
On 64k-token natural language training, power attention with degree 2 achieves lower loss per FLOP than both softmax attention and existing linear attention, and its GPU kernels run faster than Flash Attention at long...
Discussion (0). Sign in to comment.