REVIEW 14 cited by
Theory, Analysis, and Best Practices for Sigmoid Self-Attention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are typically obtained as the softmax of dot products between keys and queries. Recent work has explored alternatives to softmax attention in transformers, such as ReLU and sigmoid activations. In this work, we revisit sigmoid attention and conduct an in-depth theoretical and empirical analysis. Theoretically, we prove that transformers with sigmoid attention are universal function approximators and benefit from improved regularity compared to softmax attention. Through detailed empirical analysis, we identify stabilization of large initial attention norms during the early stages of training as a crucial factor for the successful training of models with sigmoid attention, outperforming prior attempts. We also introduce FLASHSIGMOID, a hardware-aware and memory-efficient implementation of sigmoid attention yielding a 17% inference kernel speed-up over FLASHATTENTION2 on H100 GPUs. Experiments across language, vision, and speech show that properly normalized sigmoid attention matches the strong performance of softmax attention on a wide range of domains and scales, which previous attempts at sigmoid attention were unable to fully achieve. Our work unifies prior art and establishes best practices for sigmoid attention as a drop-in softmax replacement in transformers.
Forward citations
Cited by 14 Pith papers
-
Complex-Valued Phase-Coherent Transformer
Sigmoid gating on L2-normalised complex cosine scores, with no row normalisation, generalises across long-range, positional, phase and vision tasks, though the depth-stability theorem assumes its own substance.
-
On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
A simplified selective state-space layer expresses polynomials whose degree grows with sequence length, exceeding the fixed per-layer degree of linear attention.
-
Legible-by-Construction: Attention and End-to-End Transformers
Bounding attention values into fuzzy memberships or Boolean set operations yields 44–62% crisp selective detectors and an end-to-end legible transformer at baseline quality.
-
Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization
Softmax temperature controls the rank of learned representations: high temperature induces rank-deficit bias, compresses features, hurts OOD generalization, and improves OOD detection.
-
Systematic Outliers in Large Language Models
The paper identifies activation, weight, and attention outliers as a single phenomenon caused by softmax attention and demonstrates that explicit context-aware scaling eliminates them.
-
A Unified Perspective on the Dynamics of Deep Transformers
Attention-only Transformer stacks are shown to be well-posed as mean-field PDEs for many attention variants, and Gaussian inputs evolve via explicit covariance ODEs that predict clustering or blow-up.
-
From Molecules to Mixtures: Learning Representations of Olfactory Mixture Similarity using Inductive Biases
POMMix, a graph-based model with attention and cosine similarity heads, extends the Principal Odor Map to predict human perceptual similarity of odor mixtures, reporting a test correlation of 0.78 on a compiled datase...
-
CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up
CLEAR replaces full attention in pre-trained diffusion transformers with local circular-window attention and distills the teacher into a student that keeps quality at a fraction of the compute.
-
Your Attention Matters: to Improve Model Robustness to Noise and Spurious Correlations
Doubly stochastic attention is the most robust of five ViT attention mechanisms to fog corruption in relative accuracy, based on single-seed experiments on CIFAR-10, CIFAR-100, and Imagenette.
-
A Theoretical Study of (Hyper) Self-Attention through the Lens of Interactions: Representation, Training, Generalization
Single-layer linear self-attention can represent, train on, and length-generalize pairwise interaction functions under data-versatility and exact-realizability assumptions, and the paper introduces higher-order HyperA...
-
Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
The paper derives convergence rates for sigmoid gating mixture-of-experts with quadratic scores and uses them to argue sigmoid self-attention is more sample-efficient than softmax, but the link to attention is an unpr...
-
Smarter Together: Combining Large Language Models and Small Models for Physiological Signals Visual Inspection
ConMIL uses per-class interpretable multiple instance learning and conformal prediction to feed LLMs calibrated, highlighted hints, lifting accuracy on ECG and EEG visual inspection from 13 to 48 percent to 71 to 97 percent.
-
Scaling Context Requires Rethinking Attention
On 64k-token natural language training, power attention with degree 2 achieves lower loss per FLOP than both softmax attention and existing linear attention, and its GPU kernels run faster than Flash Attention at long...
-
Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models
A two-stage softplus-based attention mechanism with re-weighting (LSSAR) is reported to keep validation loss nearly flat when a 124M-parameter GPT-2 is tested at up to 16x its 1024-token training length.
Discussion (0). Continue with ORCID to comment.