The Indirect Convolution algorithm avoids im2col by using an indirection buffer, reducing memory overhead proportionally to input channels and outperforming GEMM-based methods by up to 62% for convolutions requiring transformation.
Pay Less Attention with Lightweight and Dynamic Convolutions
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
abstract
Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.
representative citing papers
Longformer uses local windowed attention plus task-specific global attention to achieve linear scaling and state-of-the-art results on long-document language modeling, QA, and summarization after pretraining.
Gated linear attention Transformers achieve competitive language modeling results with linear-time inference, superior length generalization, and higher training throughput than Mamba.
Compressive Transformer sets new records on WikiText-103 (17.1 ppl) and Enwik8 (0.97 bpc) via memory compression and introduces the PG-19 long-range language benchmark.
Dynamic parameterization of standard layers can replace explicit attention for linear-time global visual modeling.
SGAP-PPIS generates residue-wise adaptive propagation coefficients from equivariant GNN geometric states to improve protein-protein interaction site prediction, reporting competitive results on Test_60.
JetViT uses post-training attention search to hybridize full-attention ViTs with linear and window attention blocks, achieving up to 1.79x throughput gains on high-res images while preserving accuracy on DINOv3 and DepthAnythingV2.
citing papers explorer
-
The Indirect Convolution Algorithm
The Indirect Convolution algorithm avoids im2col by using an indirection buffer, reducing memory overhead proportionally to input channels and outperforming GEMM-based methods by up to 62% for convolutions requiring transformation.
-
Longformer: The Long-Document Transformer
Longformer uses local windowed attention plus task-specific global attention to achieve linear scaling and state-of-the-art results on long-document language modeling, QA, and summarization after pretraining.
-
Gated Linear Attention Transformers with Hardware-Efficient Training
Gated linear attention Transformers achieve competitive language modeling results with linear-time inference, superior length generalization, and higher training throughput than Mamba.
-
Compressive Transformers for Long-Range Sequence Modelling
Compressive Transformer sets new records on WikiText-103 (17.1 ppl) and Enwik8 (0.97 bpc) via memory compression and introduces the PG-19 long-range language benchmark.
-
Linear-Time Global Visual Modeling without Explicit Attention
Dynamic parameterization of standard layers can replace explicit attention for linear-time global visual modeling.
-
Structure-Guided Adaptive Propagation for Protein-Protein Interaction Site Prediction
SGAP-PPIS generates residue-wise adaptive propagation coefficients from equivariant GNN geometric states to improve protein-protein interaction site prediction, reporting competitive results on Test_60.
-
JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search
JetViT uses post-training attention search to hybridize full-attention ViTs with linear and window attention blocks, achieving up to 1.79x throughput gains on high-res images while preserving accuracy on DINOv3 and DepthAnythingV2.