Strategies for Pre-training Graph Neural Networks
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:SP2PNJ4Wrecord.jsonopen to challenge →
read the original abstract
Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naive strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.
This paper has not been read by Pith yet.
Forward citations
Cited by 30 Pith papers
-
EpiFormer: Learning Antigen-Antibody Interactions for Epitope Prediction via Geometric Deep Learning
EpiFormer improves epitope prediction F1 score by over 40% via early-fusion cross-attention in GNN layers and sparsity-aware objectives, while recovering known biology as emergent behavior.
-
ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
ConTact introduces a contact-then-act architecture with distance-biased cross-attention and contact-weighted loss for antibody CDR design, reporting 5-6% better backbone RMSD and superior contact metrics on CHIMERA-Be...
-
ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
ConTact decomposes CDR design into surface fingerprint learning, contact prediction, and contact-gated sequence generation using distance-biased attention and weighted loss, reporting 7% RMSD and 10% F1 gains on CHIME...
-
scShapeBench: Discovering geometry from high dimensional scRNAseq data
scShapeBench supplies synthetic and real annotated single-cell datasets across four shape categories, with scReebTower outperforming PAGA and Mapper on topology-aware metrics.
-
Structural Interpretations of Protein Language Model Representations via Differentiable Graph Partitioning
SoftBlobGIN combines ESM-2 representations with protein contact graphs via a lightweight GNN and differentiable substructure pooling to achieve 92.8% accuracy on enzyme classification, raise binding-site AUROC to 0.98...
-
Canopy: A Heterograph Foundation Model for Metabolic Engineering
Frozen embeddings from a pretrained heterogeneous graph transformer over a 6.9M-node metabolic-engineering knowledge graph predict fermentation titers at R²=0.41, outperforming tabular baselines (R²=0.24).
-
Graph Neural Networks Are Not Continuous Across Graph Resolutions
GNNs are shown to lack continuity under graph resolution changes due to message-passing schemes, with a derived modification enabling consistent multi-scale representations validated experimentally.
-
A Topological Characterization of Graph Neural Networks via Stochastic Block Model Embeddings on the n-Sphere
Trained MPNNs factor through bounded step-graphon-signals that embed via an explicit map into disjoint caps on the n-sphere, producing a topological fingerprint for model comparison and retrieval.
-
Detecting Gravitational-Wave Anisotropies with Simulation-Based Inference
A neural-network-based simulation inference method improves 3σ detection probability of gravitational-wave background anisotropies by 90-200% over Gaussian frequentist searches by learning non-Gaussian structure in pu...
-
AgForce Enables Antigen-conditioned Generative Antibody Design
AgForce improves antigen-conditioned antibody design by using framework dropout, gated bottlenecks, hyperbolic cross attention, MDN sequence head with Potts-like coupling, annealed MCL, and antigen cycle consistency t...
-
EvoStruct: Bridging Evolutionary and Structural Priors for Antibody CDR Design via Protein Language Model Adaptation
EvoStruct integrates evolutionary priors from a protein language model with structural priors from an E(3)-equivariant GNN to raise amino acid recovery by 16% and diversity by 2.3x on CHIMERA-Bench while cutting perpl...
-
Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction
MolCHG uses a multi-level compositional hierarchical graph with atom-bond cross-view contrastive learning, functional group prediction, and structure tasks to achieve top results on seven of nine MoleculeNet benchmarks.
-
Rethinking Molecular OOD Generalization via Target-Aware Source Selection
SCOPE-BENCH shows state-of-the-art molecular models suffer up to 8x higher errors under extreme OOD, while POMA reduces mean absolute error by up to 11.2% via target-aware source selection and dual-scale adaptation.
-
A Graph Foundation Model for Wireless Resource Allocation
A pre-trained interference-aware graph Transformer model for wireless resource allocation that achieves strong few-shot adaptation to new tasks and scenarios.
-
BiScale-GTR: Fragment-Aware Graph Transformers for Multi-Scale Molecular Representation Learning
BiScale-GTR achieves claimed state-of-the-art results on MoleculeNet, PharmaBench and LRGB by combining improved fragment tokenization with a parallel GNN-Transformer architecture that operates at both atom and fragme...
-
MolDA: Molecular Understanding and Generation via Large Language Diffusion Model
MolDA is a multimodal molecular model that uses a discrete large language diffusion backbone plus a hybrid graph encoder to achieve better global coherence and validity than autoregressive approaches.
-
Energy-Guided Generative Modeling for Low-Energy Molecular Structure Discovery
EnFlow integrates flow-based conformer generation with energy landscape modeling to enable joint ensemble generation and ground-state identification using only 1-2 ODE steps.
-
SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
SSL4RL reformulates self-supervised learning objectives into dense, verifiable reward signals for RL-based fine-tuning of vision-language models, yielding performance gains on reasoning benchmarks.
-
GraphPINE: Graph Importance Propagation for Interpretable Drug Response Prediction
GraphPINE is a GNN architecture that initializes node importance from prior knowledge graphs and propagates updates via an importance propagation layer for interpretable drug response prediction on over 5,000 genes an...
-
Pretraining a Foundation Model for Small-Molecule Natural Products
NaFM is a pretrained foundation model for natural products using scaffold-focused contrastive learning and masked graph objectives that achieves SOTA on taxonomy classification, gene/microbial analysis, and virtual sc...
-
FARM: Enhancing Molecular Representations with Functional Group Awareness
FARM adds atomic-level functional group annotations to create FG-enhanced SMILES and FG graphs, trains them with masked language modeling and GNNs plus contrastive alignment, and reports state-of-the-art results on 8 ...
-
Handling Feature Heterogeneity with Learnable Graph Patches
Learnable graph patches enable domain-agnostic pre-training of graph models by decomposing heterogeneous graphs into transferable semantic units via patch encoders and aggregators.
-
EssentialGIN: a new approach for gene essentiality prediction based on graph isomorphism neural networks
EssentialGIN applies a modified graph isomorphism network to PPI networks with integrated biological node attributes to predict essential genes, outperforming centrality measures, Node2Vec, MLP, and GAT especially in ...
-
MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry
MolE-RAG is a training-free RAG framework that augments LLMs with literature, molecular context, and structural analogs to improve performance on nine molecular property prediction tasks.
-
Mesh Based Simulations with Spatial and Temporal awareness
A unified training framework for mesh-based ML surrogates in CFD improves accuracy and long-horizon stability by enforcing spatial derivative consistency via multi-node prediction, using temporal cross-attention corre...
-
Synergistic Benefits of Joint Molecule Generation and Property Prediction
Hyformer jointly models molecule generation and property prediction via alternating attention and joint pre-training, showing synergistic gains in conditional sampling, OOD prediction, and a drug design case for antim...
-
Disentangled Generative Graph Representation Learning
DiGGR introduces a self-supervised graph representation learning framework that disentangles latent factors to guide mask modeling and improve representation quality on graph tasks.
-
On Improving Graph Neural Networks for QSAR by Pre-training on Extended-Connectivity Fingerprints
Pre-training GNNs on ECFP prediction produces statistically significant QSAR gains on five of six Biogen benchmarks with OOD splits, but underperforms on heterogeneous datasets and complex endpoints like binding affinity.
-
Financial Anomaly Detection for the Canadian Market
Neural network and TDA methods outperform PCA at detecting financial anomalies in the Canadian TSX-60 market.
-
Property Enhanced Instruction Tuning for Multi-task Molecule Generation with Large Language Models
PEIT pre-trains on aligned multimodal molecular data and fine-tunes LLMs to improve performance on molecule captioning, generation, property prediction, and multi-constraint generation tasks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.