Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
read the original abstract
Many machine learning tasks such as multiple instance learning, 3D shape recognition, and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the order of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces the computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating the state-of-the-art performance compared to recent methods for set-structured data.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
Monotone and Separable Set Functions: Characterizations and Neural Models
Characterizes monotone separating set functions with dimension bounds, proves non-existence on infinite domains, and introduces a Holder-stable neural model with a weak version of the property for universal monotone a...
-
SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction
SAVER proposes a conformal groundability gate plus submodular image selector that activates vision only when needed for multimodal named entity recognition and relation extraction, improving F1 while lowering compute.
-
The General Theory of Localization Methods
The localization method unifies kernel methods, local learning algorithms, MeanShift, Hopfield networks, and Transformers through local models, localization tricks, and hierarchical extensions.
-
HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models
HubRouter is a sub-quadratic routing primitive using learned hubs that replaces attention layers in hybrid models while delivering competitive perplexity and large throughput gains.
-
Beyond Independent Frames: Latent Attention Masked Autoencoders for Multi-View Echocardiography
LAMAE adds latent-space attention to masked autoencoders so multi-view echocardiography videos can exchange information across frames and views, yielding representations that transfer from adult to pediatric hearts an...
-
TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
TabICL scales in-context learning to large tabular data via column-then-row attention for row embeddings followed by a transformer, matching TabPFNv2 speed and performance while outperforming it and CatBoost on datase...
-
Measurements of the Higgs boson production, fiducial and differential cross-sections in the four lepton decay channel using 164 fb$^{-1}$ of data collected at $\sqrt{s}$ = 13.6 TeV with the ATLAS detector
ATLAS measures the fiducial cross-section for H to four leptons as 3.65 fb matching the SM prediction of 3.68 fb, with signal strength 0.99 and differential distributions all consistent with expectations.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.