IAFormer uses boost-invariant pairwise quantities and differential attention to create a sparse Transformer that achieves state-of-the-art classification on top-quark and quark-gluon jet datasets while using over an order of magnitude fewer parameters than prior Particle Transformer models.
On the relationship between self-attention and convolutional layers
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
representative citing papers
ITNet frames convolution, attention, and recurrence as special cases of one learnable integral transform with an MLP kernel and shows a single shared operator plus modality encoders matches specialized models on ImageNet-1K, GLUE, ModelNet40, VQA v2, and NLVR2.
WePE encodes 2D patch positions in Vision Transformers via Weierstrass elliptic functions on the complex plane to exploit double periodicity and derive relative positions algebraically.
DragNUWA integrates text, image, and trajectory controls into a diffusion video model using a Trajectory Sampler, Multiscale Fusion, and Adaptive Training to enable fine-grained open-domain video generation.
Rotating value embeddings along with keys and queries (RoVE) converts RoPE attention into a block-Toeplitz attentive convolution and yields consistent gains across 124M and 354M GPT-2 models.
SWARD introduces stochastic windowed attention distillation and prototype discriminative regularization to improve cross-architecture knowledge transfer from transformers to CNNs for semantic segmentation.
Sparsity-guided distillation enables replacing attention layers in ViTs with simpler sequential modules, with sparser layers showing smaller performance drops.
citing papers explorer
-
IAFormer: Interaction-Aware Transformer network for collider data analysis
IAFormer uses boost-invariant pairwise quantities and differential attention to create a sparse Transformer that achieves state-of-the-art classification on top-quark and quark-gluon jet datasets while using over an order of magnitude fewer parameters than prior Particle Transformer models.
-
ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence
ITNet frames convolution, attention, and recurrence as special cases of one learnable integral transform with an MLP kernel and shows a single shared operator plus modality encoders matches specialized models on ImageNet-1K, GLUE, ModelNet40, VQA v2, and NLVR2.
-
Weierstrass Positional Encoding for Vision Transformers
WePE encodes 2D patch positions in Vision Transformers via Weierstrass elliptic functions on the complex plane to exploit double periodicity and derive relative positions algebraically.
-
DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory
DragNUWA integrates text, image, and trajectory controls into a diffusion video model using a Trajectory Sampler, Multiscale Fusion, and Adaptive Training to enable fine-grained open-domain video generation.
-
RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
Rotating value embeddings along with keys and queries (RoVE) converts RoPE attention into a block-Toeplitz attentive convolution and yields consistent gains across 124M and 354M GPT-2 models.
-
SWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
SWARD introduces stochastic windowed attention distillation and prototype discriminative regularization to improve cross-architecture knowledge transfer from transformers to CNNs for semantic segmentation.
-
From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation
Sparsity-guided distillation enables replacing attention layers in ViTs with simpler sequential modules, with sparser layers showing smaller performance drops.