REVIEW 8 cited by
DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Real-world data generation often involves complex inter-dependencies among instances, violating the IID-data hypothesis of standard learning paradigms and posing a challenge for uncovering the geometric structures for learning desired instance representations. To this end, we introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states that progressively incorporate other instances' information by their interactions. The diffusion process is constrained by descent criteria w.r.t.~a principled energy function that characterizes the global consistency of instance representations over latent structures. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs, which gives rise to a new class of neural encoders, dubbed as DIFFormer (diffusion-based Transformers), with two instantiations: a simple version with linear complexity for prohibitive instance numbers, and an advanced version for learning complex structures. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks, such as node classification on large graphs, semi-supervised image/text classification, and spatial-temporal dynamics prediction.
Forward citations
Cited by 8 Pith papers
-
TRACE: Learning to Compute on Circuit Graphs
A per-operator hierarchical Transformer with a residual function shift objective substantially improves learning functional behavior of circuit graphs.
-
Predicting Large-scale Urban Network Dynamics with Energy-informed Graph Neural Diffusion
A scalable spatiotemporal Transformer, ScaleSTF, matches the accuracy of much larger models on city-scale forecasting tasks at a fraction of the compute and memory cost.
-
SFi-Former: Sparse Flow Induced Attention for Graph Transformer
SFi-Former replaces dense graph transformer attention with sparse flows from an l1-regularized energy minimization, improving long-range graph benchmark accuracy and generalization.
-
Hgformer: Hyperbolic Graph Transformer for Recommendation
Hgformer is a hyperbolic graph transformer for collaborative filtering that reports improved recall and NDCG on six datasets, but its linear-attention approximation is not actually unbiased as claimed.
-
Even Sparser Graph Transformers
Spexphormer trains a narrow graph transformer to identify important attention edges, then trains a wider model on the sparsified graph, achieving competitive accuracy with far less memory.
-
OpenGT: A Comprehensive Benchmark For Graph Transformers
OpenGT benchmarks 16 graph models on 14 datasets, finding graph transformers excel on heterophilous graphs, though several observations are not robustly supported.
-
Graph Fourier Transformer with Structure-Frequency Information
A graph transformer with a fixed spectral-frequency mask on attention reports top scores on five TU datasets, while underperforming several baselines on PATTERN and CLUSTER.
-
A Theory for Compressibility of Graph Transformers for Transductive Learning
Under norm, low-rank, or clustering assumptions, a graph transformer's hidden dimension can be compressed with only small error in outputs and attention scores.
Discussion (0). Continue with ORCID to comment.