REVIEW 8 cited by
On the Relationship between Self-Attention and Convolutional Layers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available.
Forward citations
Cited by 8 Pith papers
-
RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
RoVE rotates value embeddings simultaneously with keys in attention to make values position-dependent, reframing RoPE as attentive convolution and reporting gains on long-context tasks in 124M and 354M GPT-2 models.
-
Long-Term Visual Localization in Dynamic Benthic Environments: A Dataset, Footprint-Based Ground Truth, and Visual Place Recognition Benchmark
A benchmark shows state-of-the-art visual place recognition performs poorly on a new multi-site, multi-year benthic AUV dataset, and that distance-based ground truth inflates recall.
-
Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions
A synthetic benchmark with controllable memory functions maps how four sequence architectures handle exponential, polynomial, sparse-spike, and Airy temporal structures.
-
GASPnet: Global Agreement to Synchronize Phases
A CNN augmented with global attention-driven phase synchronization (GASPnet) outperforms a parameter-matched CNN on noisy multi-object and superimposed-image classification.
-
Low-latency vision transformers via large-scale multi-head attention
Attention heads in compact vision transformers each recognize small label subsets with little noise, which the authors exploit for diverse ensembles and low-latency hybrid architectures on CIFAR-100.
-
Efficient Feedback Gate Network for Hyperspectral Image Super-Resolution
A group-based deep network (EFGN) with channel-shuffle/dilated feedback and large-kernel gating achieves state-of-the-art hyperspectral image super-resolution on Chikusei, Pavia Centre, and Harvard.
-
Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure
A CXL-based disaggregated memory architecture with hybrid XLink interconnects is proposed and prototyped, claiming large speedups for memory-bound AI and HPC workloads.
-
Graph Collaborative Attention Network for Link Prediction in Knowledge Graphs
GCAT is presented as a new graph attention model for knowledge graph link prediction, but its equations are those of KBGAT and its reported benchmark numbers do not support the stated performance claims.
Discussion (0). Sign in to comment.