A geometry-aware multi-support heterogeneous GNN fuses point, line, and grid rainfall observations via cross-support message passing to reconstruct fields at arbitrary resolutions, reducing RMSE by 23.2% over baselines on Singapore data.
hub
A generalization of transformer networks to graphs
34 Pith papers cite this work. Polarity classification is still indexing.
34
Pith papers citing it
abstract
We propose a generalization of transformer neural network architecture for arbitrary graphs. The original transformer was designed for Natural Language Processing (NLP), which operates on fully connected graphs representing all connections between the words in a sequence. Such architecture does not leverage the graph connectivity inductive bias, and can perform poorly when the graph topology is important and has not been encoded into the node features. We introduce a graph transformer with four new properties compared to the standard model. First, the attention mechanism is a function of the neighborhood connectivity for each node in the graph. Second, the positional encoding is represented by the Laplacian eigenvectors, which naturally generalize the sinusoidal positional encodings often used in NLP. Third, the layer normalization is replaced by a batch normalization layer, which provides faster training and better generalization performance. Finally, the architecture is extended to edge feature representation, which can be critical to tasks s.a. chemistry (bond type) or link prediction (entity relationship in knowledge graphs). Numerical experiments on a graph benchmark demonstrate the performance of the proposed graph transformer architecture. This work closes the gap between the original transformer, which was designed for the limited case of line graphs, and graph neural networks, that can work with arbitrary graphs. As our architecture is simple and generic, we believe it can be used as a black box for future applications that wish to consider transformer and graphs.
hub tools
citation-role summary
background 2
citation-polarity summary
fields
cs.LG 23 cs.AI 2 cs.CL 2 cs.CV 1 cs.DC 1 cs.IR 1 cs.SI 1 hep-ph 1 physics.comp-ph 1 q-bio.TO 1roles
background 2representative citing papers
CIPE constructs graph positional encodings from communicability so that self-attention similarities equal the sum of all-path contributions between nodes, yielding 35.5% average gains on seven benchmarks over structure-agnostic Transformers.
A dual-attention graph transformer on a weighted network representation of fusing locations models cross-layer interactions to improve quality prediction in metal additive manufacturing over image, sequence, and graph baselines.
Quantum-inspired observables reveal poor signal routing in standard spectral GNNs and motivate Schrödinger GNNs with superior propagation capacity.
A hierarchical anisometric positional encoding that combines distance-indexed in-route and depot-anchored angular cross-route components improves transformer-based solvers for vehicle routing problems over index-based alternatives.
A structure-aware VAE generates realistic FC matrices for replay, combined with multi-level knowledge distillation and hierarchical contextual bandit sampling, to enable continual fMRI-based brain disorder diagnosis across sequentially arriving multi-site data without catastrophic forgetting.
GAT uses static attention where neighbor rankings ignore the query node and thus cannot express some graph problems; GATv2 enables dynamic attention and outperforms GAT on 11 OGB and other benchmarks with equal parameters.
Learnable magnetic spectral PEs of the form h_θ(A_q)R, computed in Hermitian block Krylov subspaces, are eigenbasis-independent, O(log 1/ε)-approximable for heat–resolvent families, and recover directed structure where symmetrized baselines fail.
LLM embeddings condition a generative transformer to enable faster convergence, better performance, and generalization to unseen LHC processes using a single model.
GaRA generates task-specific LoRA weight updates conditioned on graph structures to enable better whole-graph encoding in LLMs for zero-shot graph learning.
FLAGG makes one-shot graph generators autoregressive through a learned reversal of stochastic node removal and outperforms baselines on datasets spanning different graph sizes and domains.
SPT generalizes SICGAT and ViT to arbitrary superpixel chunking with multidimensional sine-cosine encodings and shape/color-enriched patches, outperforming prior GNN superpixel methods while matching ViTs on CIFAR10, FashionMNIST, and Imagenette.
Deep ensembles fail to capture meaningful epistemic uncertainty in message-passing GNNs due to epistemic collapse where independently trained networks converge to similar predictions.
Neural point-forms are introduced as permutation-invariant neural layers that output learned form-comparison matrices for point clouds, with a claimed consistency proof under sampling and manifold assumptions and competitive results on synthetic and biological data.
Crys-JEPA introduces a joint embedding predictive architecture that creates an energy-aware latent space, enabling embedding-based stability screening and a refinement pipeline that yields up to 72.7% gains on the V.S.U.N. metric for crystal generation.
Mask-Morph Graph U-Net morphs coarse graph hierarchies with barycentric parameterization and applies masked supervised pretraining to improve generalizability of hierarchical GNN surrogates for crashworthiness prediction on variable meshes.
Graph invariants serve as expressive, task-agnostic baselines that characterize structural heterogeneity and match trained models across 26 datasets, indicating that expressivity is not the primary driver of performance.
GTokenLLMs do not fully understand graph tokens, exhibiting over-sensitivity or insensitivity to instruction changes and relying heavily on text for reasoning even when graph information is preserved.
A new distributed framework for graph transformer training auto-selects parallel strategies and optimizes sparse operations to deliver up to 6x speedup on 8 GPUs and 78% memory reduction.
PATH gene embeddings in a graph transformer achieve 0.8766 F1 on pancancer metastasis prediction (8.8% over SOTA) and identify disease-state pathway rewiring.
A structure-preserving GNN solver for parametric hyperbolic conservation laws achieves superior long-horizon stability and orders-of-magnitude speedups over high-resolution simulations on supersonic flow benchmarks.
Invariant-Stratified Propagation (ISP) enhances GNN expressivity beyond 1-WL by stratifying nodes according to graph invariants and encoding structural heterogeneity in hierarchical strata.
Introduces FraudSquad, a hybrid model using language model embeddings and a gated graph transformer that outperforms baselines on newly created LLM-generated spam review datasets.
Local d-hop uniqueness in GNN node features matches global UID expressiveness for ILP solving while providing stronger generalization.
citing papers explorer
-
Spatial Support Matters: Geometry-Aware Graph Fusion for Rainfall Field Reconstruction
A geometry-aware multi-support heterogeneous GNN fuses point, line, and grid rainfall observations via cross-support message passing to reconstruct fields at arbitrary resolutions, reducing RMSE by 23.2% over baselines on Singapore data.
-
Communicability-Inspired Positional Encoding (CIPE)
CIPE constructs graph positional encodings from communicability so that self-attention similarities equal the sum of all-path contributions between nodes, yielding 35.5% average gains on seven benchmarks over structure-agnostic Transformers.
-
Spatiotemporal Graph Transformer for 3D Neighborhood Interaction and Quality Prediction in Metal Additive Manufacturing
A dual-attention graph transformer on a weighted network representation of fusing locations models cross-layer interactions to improve quality prediction in metal additive manufacturing over image, sequence, and graph baselines.
-
Beyond Oversquashing: Understanding Signal Propagation in GNNs Via Observables
Quantum-inspired observables reveal poor signal routing in standard spectral GNNs and motivate Schrödinger GNNs with superior propagation capacity.
-
Rethinking Positional Encoding for Neural Vehicle Routing
A hierarchical anisometric positional encoding that combines distance-indexed in-route and depot-anchored angular cross-route components improves transformer-based solvers for vehicle routing problems over index-based alternatives.
-
Continual Learning for fMRI-Based Brain Disorder Diagnosis via Functional Connectivity Matrices Generative Replay
A structure-aware VAE generates realistic FC matrices for replay, combined with multi-level knowledge distillation and hierarchical contextual bandit sampling, to enable continual fMRI-based brain disorder diagnosis across sequentially arriving multi-site data without catastrophic forgetting.
-
How Attentive are Graph Attention Networks?
GAT uses static attention where neighbor rankings ignore the query node and thus cannot express some graph problems; GATv2 enables dynamic attention and outperforms GAT on 11 OGB and other benchmarks with equal parameters.
-
Eigenbasis-Independent Learnable Spectral Positional Encodings for Directed Graphs via Hermitian Block Krylov Subspaces
Learnable magnetic spectral PEs of the form h_θ(A_q)R, computed in Hermitian block Krylov subspaces, are eigenbasis-independent, O(log 1/ε)-approximable for heat–resolvent families, and recover directed structure where symmetrized baselines fail.
-
One Generator, Any Process: LLM-Conditioning for the LHC
LLM embeddings condition a generative transformer to enable faster convergence, better performance, and generalization to unseen LHC processes using a single model.
-
Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation
GaRA generates task-specific LoRA weight updates conditioned on graph structures to enable better whole-graph encoding in LLMs for zero-shot graph learning.
-
FLAGG: Flexible Autoregressive Graph Generation
FLAGG makes one-shot graph generators autoregressive through a learned reversal of stochastic node removal and outperforms baselines on datasets spanning different graph sizes and domains.
-
Is an Image Also Worth 16x16=256 Superpixels? A Framework for Attentional Image Classification
SPT generalizes SICGAT and ViT to arbitrary superpixel chunking with multidimensional sine-cosine encodings and shape/color-enriched patches, outperforming prior GNN superpixel methods while matching ViTs on CIFAR10, FashionMNIST, and Imagenette.
-
Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?
Deep ensembles fail to capture meaningful epistemic uncertainty in message-passing GNNs due to epistemic collapse where independently trained networks converge to similar predictions.
-
Neural Point-Forms
Neural point-forms are introduced as permutation-invariant neural layers that output learned form-comparison matrices for point clouds, with a claimed consistency proof under sampling and manifold assumptions and competitive results on synthetic and biological data.
-
Crys-JEPA: Accelerating Crystal Discovery via Embedding Screening and Generative Refinement
Crys-JEPA introduces a joint embedding predictive architecture that creates an energy-aware latent space, enabling embedding-based stability screening and a refinement pipeline that yields up to 72.7% gains on the V.S.U.N. metric for crystal generation.
-
Mask-Morph Graph U-Net: A Generalisable Mesh-Based Surrogate for Crashworthiness Field Prediction under Large Geometric Variation
Mask-Morph Graph U-Net morphs coarse graph hierarchies with barycentric parameterization and applies masked supervised pretraining to improve generalizability of hierarchical GNN surrogates for crashworthiness prediction on variable meshes.
-
Invariant-Based Diagnostics for Graph Benchmarks
Graph invariants serve as expressive, task-agnostic baselines that characterize structural heterogeneity and match trained models across 26 datasets, indicating that expressivity is not the primary driver of performance.
-
Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
GTokenLLMs do not fully understand graph tokens, exhibiting over-sensitivity or insensitivity to instruction changes and relying heavily on text for reasoning even when graph information is preserved.
-
Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphs
A new distributed framework for graph transformer training auto-selects parallel strategies and optimizes sparse operations to deliver up to 6x speedup on 8 GPUs and 78% memory reduction.
-
Graph Transformer-Based Pathway Embedding for Cancer Prognosis
PATH gene embeddings in a graph transformer achieve 0.8766 F1 on pancancer metastasis prediction (8.8% over SOTA) and identify disease-state pathway rewiring.
-
A Structure-Preserving Graph Neural Solver for Parametric Hyperbolic Conservation Laws
A structure-preserving GNN solver for parametric hyperbolic conservation laws achieves superior long-horizon stability and orders-of-magnitude speedups over high-resolution simulations on supersonic flow benchmarks.
-
Invariant-Stratified Propagation for Expressive Graph Neural Networks
Invariant-Stratified Propagation (ISP) enhances GNN expressivity beyond 1-WL by stratifying nodes according to graph invariants and encoding structural heterogeneity in hierarchical strata.
-
Detecting LLM-Generated Spam Reviews by Integrating Language Model Embeddings and Graph Neural Network
Introduces FraudSquad, a hybrid model using language model embeddings and a gated graph transformer that outperforms baselines on newly created LLM-generated spam review datasets.
-
Feature Augmentation of GNNs for ILPs: Local Uniqueness Suffices
Local d-hop uniqueness in GNN node features matches global UID expressiveness for ILP solving while providing stronger generalization.
-
Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
Geometric deep learning provides a unified mathematical framework based on grids, groups, graphs, geodesics, and gauges to explain and extend neural network architectures by incorporating physical regularities.
-
Graph Grounded Cross Attention Transformer Neural Network for Structurally Constrained Full Event Sequence Generation in Predictive Process Monitoring
GGATN combines graph grounding with transformer self- and cross-attention to generate full event sequences, timestamps, length, and attributes in a single pass followed by Viterbi-style constrained decoding, outperforming prompted LLM baselines on six logs with zero hallucinated activities.
-
Handling Feature Heterogeneity with Learnable Graph Patches
Learnable graph patches enable domain-agnostic pre-training of graph models by decomposing heterogeneous graphs into transferable semantic units via patch encoders and aggregators.
-
Composable Crystals: Controllable Materials Discovery via Concept Learning
VQ-VAE concept learning enables controllable recombination of crystal motifs to generate structures with reported gains in validity-stability-uniqueness-novelty metrics on MP-20 and Alex-MP-20.
-
Rethinking Generalization in Graph Neural Networks: A Structural Complexity Perspective
GNN generalization depends explicitly on graph structural complexity measured by effective edges, with a new regularization method shown to balance underfitting and overfitting.
-
Attention-based graph neural networks: a survey
The survey groups attention-based GNNs into three stages—graph recurrent attention networks, graph attention networks, and graph transformers—while reviewing architectures and future directions.
-
Explaining the Explainers in Graph Neural Networks: a Comparative Study
Benchmark study of ten GNN explainers on eight architectures and six datasets that isolates usable components and issues practical recommendations.
-
A Hierarchical Quantized Tokenization Framework for Task-Adaptive Graph Representation Learning
QUIET is a hierarchical RVQ-based graph tokenizer with a learned level-weighting gate; it improves several benchmarks but not consistently against the strongest baselines.
- TRAPS: Treatment-Assignment Prediction via Pathway-informed Stratification
- Modeling Spectral Energy Shifts in Spatio-Temporal Graph Anomaly Detection