REVIEW 7 cited by
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Attention-based architectures have become ubiquitous in machine learning, yet our understanding of the reasons for their effectiveness remains limited. This work proposes a new way to understand self-attention networks: we show that their output can be decomposed into a sum of smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, we prove that self-attention possesses a strong inductive bias towards "token uniformity". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Our experiments verify the identified convergence phenomena on different variants of standard transformer architectures.
Forward citations
Cited by 7 Pith papers
-
Transient Reserves, Sink Dampers, and the Failure of Eigenvalue Reasoning in the Attention Propagator
Resolvent analysis of trained causal attention shows sinks act as transient dampers, routing heads carry excess Kreiss reserve, and eigenvalue depth predictions fail by 7–11 orders of magnitude.
-
Many-body Tipping Dynamics of ChatGPT-like AIs
Tipping of ChatGPT-like AI to undesirable outputs is modeled as first-passage transport of a residual-state spin across an output-basin wall, with attention disorder controlling the crossing.
-
Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
KATA uses rank-one PSD feature maps to pack exponentially many nearly orthogonal keys at fixed interference, reaching near-softmax MQAR at 16× length with about a quarter of softmax's KV-cache entries.
-
Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing
At compression ratios above 8x, grouping tokens via a Krylov-projected LSH of an implicit attention kernel preserves LLM output quality far better than block averaging, at a large preprocessing latency cost.
-
Quantifying Memory Utilization with Effective State-Size
Effective state-size, the rank of a causal operator submatrix, is proposed and empirically validated as a memory-utilization proxy that beats cache size at predicting recall-task accuracy and reveals how LLMs reset co...
-
AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration
AsymRnR selectively reduces query and key/value tokens in video DiT attention to cut FLOPs and latency by 10 to 30 percent with minor or no VBench score change.
-
FE-MCFormer: a novel time-frequency interpretable architecture for machinery fault diagnosis under strong noise environments
FE-MCFormer, a time-frequency transformer-style network with a learnable Fourier reconstruction layer, achieves state-of-the-art fault diagnosis accuracy on three rotating machinery datasets under strong noise.
Discussion (0). Continue with ORCID to comment.