Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:3F3TXDE6record.jsonopen to challenge →
read the original abstract
Despite their widespread success in various domains, Transformer networks have yet to perform well across datasets in the domain of 3D atomistic graphs such as molecules even when 3D-related inductive biases like translational invariance and rotational equivariance are considered. In this paper, we demonstrate that Transformers can generalize well to 3D atomistic graphs and present Equiformer, a graph neural network leveraging the strength of Transformer architectures and incorporating SE(3)/E(3)-equivariant features based on irreducible representations (irreps). First, we propose a simple and effective architecture by only replacing original operations in Transformers with their equivariant counterparts and including tensor products. Using equivariant operations enables encoding equivariant information in channels of irreps features without complicating graph structures. With minimal modifications to Transformers, this architecture has already achieved strong empirical results. Second, we propose a novel attention mechanism called equivariant graph attention, which improves upon typical attention in Transformers through replacing dot product attention with multi-layer perceptron attention and including non-linear message passing. With these two innovations, Equiformer achieves competitive results to previous models on QM9, MD17 and OC20 datasets.
This paper has not been read by Pith yet.
Forward citations
Cited by 22 Pith papers
-
The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups
Lie-Algebra Attention defines tokens as bare matrix Lie group elements and computes attention scores from the negative squared algebra norm of the relative pose log(g_i^{-1} g_j).
-
EAGOR: Embodied Reasoning in Omni-direction
EAGOR reformulates embodied 360-degree directional reasoning as recursive Bayesian estimation on a spherical manifold using spherical harmonics, achieving training-free, rotation-equivariant target tracking.
-
EpiFormer: Learning Antigen-Antibody Interactions for Epitope Prediction via Geometric Deep Learning
EpiFormer improves epitope prediction F1 score by over 40% via early-fusion cross-attention in GNN layers and sparsity-aware objectives, while recovering known biology as emergent behavior.
-
DPA4: Pushing the Accuracy-Cost Frontier of Interatomic Potentials with EMFA SO(2) Convolution
DPA4 is a new SE(3)-equivariant interatomic potential with EMFA SO(2) convolution that sets new accuracy-cost records on Matbench Discovery and SPICE benchmarks using fewer parameters than prior models.
-
TriSearch: Learning to Optimize Triangulations via Bistellar Flips
TriSearch is an RL framework that optimizes triangulations of polytopes using bistellar flips with a circuit-supported subtriangulation action representation, generalizing zero-shot to larger instances and outperformi...
-
ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
ConTact introduces a contact-then-act architecture with distance-biased cross-attention and contact-weighted loss for antibody CDR design, reporting 5-6% better backbone RMSD and superior contact metrics on CHIMERA-Be...
-
ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
ConTact decomposes CDR design into surface fingerprint learning, contact prediction, and contact-gated sequence generation using distance-biased attention and weighted loss, reporting 7% RMSD and 10% F1 gains on CHIME...
-
Fast contracted Clebsch--Gordan tensor products for equivariant graph neural networks
An O(L^3) algorithm computes contracted Clebsch-Gordan tensor products for equivariant ML potentials using a structured angular grid and spherical Poisson bracket to handle parity-odd terms at fixed CP rank.
-
Composition-Weighted Symbolic Regression for General-Purpose Property Prediction
A composition-weighted symbolic regression framework learns analytical expressions and elemental weightings from composition to predict materials properties with accuracy competitive to black-box models while producin...
-
h-MINT: Modeling Pocket-Ligand Binding with Hierarchical Molecular Interaction Network
h-MINT improves ligand-protein binding affinity prediction by 2-4% and virtual screening metrics by 1-3% via overlapping fragment tokenization and hierarchical modeling.
-
Equivariant Volumetric Grasping
A novel tri-plane equivariant volumetric grasp model adapts GIGA and IGD planners with flow matching and deformable attention to achieve higher real-time performance than non-equivariant baselines.
-
Mirror-Fusion Attention for Reflection-Aware Self-Supervised Representation Learning
MFASSL adds mirror-paired views, a lightweight Mirror-Fusion Attention module, and reflection-consistency losses to improve SSL on bilateral data with ~2.7% extra parameters.
-
ElemeNet: Multiscale Molecular Machine Learning with Uncertainty Quantification Across the Periodic Table
ElemeNet is a unified ML software package for molecular property prediction across elements 1-100 with built-in uncertainty quantification and competitive benchmarks on diverse chemistry datasets.
-
AgForce Enables Antigen-conditioned Generative Antibody Design
AgForce improves antigen-conditioned antibody design by using framework dropout, gated bottlenecks, hyperbolic cross attention, MDN sequence head with Potts-like coupling, annealed MCL, and antigen cycle consistency t...
-
EvoStruct: Bridging Evolutionary and Structural Priors for Antibody CDR Design via Protein Language Model Adaptation
EvoStruct integrates evolutionary priors from a protein language model with structural priors from an E(3)-equivariant GNN to raise amino acid recovery by 16% and diversity by 2.3x on CHIMERA-Bench while cutting perpl...
-
Generation of magnetic metal-organic frameworks
Fine-tuning CHGNet on OMDB data and performing site substitution on QMOF prototypes yields novel highly magnetic MOFs.
-
Equivariant Efficient Joint Discrete and Continuous MeanFlow for Molecular Graph Generation
EQUIMF is a unified equivariant framework that jointly generates discrete topologies and continuous geometries in molecular graphs via synchronized MeanFlow dynamics for efficient few-step sampling.
-
Bayesian E(3)-Equivariant Interatomic Potential with Iterative Restratification of Many-body Message Passing
Bayesian E(3)-equivariant MLPs with joint energy-force NLL loss achieve competitive accuracy while enabling uncertainty-guided active learning, OOD detection, and calibration.
-
Scalable Prediction of Complex Surface Reconstructions under Operating Conditions via Harmony-Search-Based Global Optimization
HASGO combines harmony search with universal MLIPs and multi-head replay fine-tuning to locate operando surface reconstructions, demonstrated by identifying the square-pyramidal O5 subsurface motif on Ag(100) during e...
-
Music102: An $D_{12}$-equivariant transformer for chord progression accompaniment
Music102 integrates D12-equivariance into a transformer for chord progression accompaniment and shows gains over Music101 on POP909.
-
REViT: Roto-reflection Equivariant Convolutional Vision Transformer
REViT introduces a discrete roto-reflection equivariant convolutional vision transformer claimed to outperform prior equivariant networks on image classification.
-
Six Open Questions in Machine-Learned Interatomic Potential Foundation Models
This perspective article develops a definition of foundational MLIPs and poses six open questions that the authors believe will define future research in machine-learned interatomic potentials.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.