EpiFormer improves epitope prediction F1 score by over 40% via early-fusion cross-attention in GNN layers and sparsity-aware objectives, while recovering known biology as emergent behavior.
hub
Strategies for pre-training graph neural networks
29 Pith papers cite this work, alongside 186 external citations. Polarity classification is still indexing.
29
Pith papers citing it
186
external citations · Pith
abstract
Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naive strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.
hub tools
citation-role summary
background 1
method 1
citation-polarity summary
representative citing papers
scShapeBench supplies synthetic and real annotated single-cell datasets across four shape categories, with scReebTower outperforming PAGA and Mapper on topology-aware metrics.
SoftBlobGIN combines ESM-2 representations with protein contact graphs via a lightweight GNN and differentiable substructure pooling to achieve 92.8% accuracy on enzyme classification, raise binding-site AUROC to 0.983, and generate auditable structural explanations without retraining the language模型
Frozen embeddings from a pretrained heterogeneous graph transformer over a 6.9M-node metabolic-engineering knowledge graph predict fermentation titers at R²=0.41, outperforming tabular baselines (R²=0.24).
GNNs are shown to lack continuity under graph resolution changes due to message-passing schemes, with a derived modification enabling consistent multi-scale representations validated experimentally.
Trained MPNNs factor through bounded step-graphon-signals that embed via an explicit map into disjoint caps on the n-sphere, producing a topological fingerprint for model comparison and retrieval.
A neural-network-based simulation inference method improves 3σ detection probability of gravitational-wave background anisotropies by 90-200% over Gaussian frequentist searches by learning non-Gaussian structure in pulsar timing residuals.
EvoStruct integrates evolutionary priors from a protein language model with structural priors from an E(3)-equivariant GNN to raise amino acid recovery by 16% and diversity by 2.3x on CHIMERA-Bench while cutting perplexity 43%.
MolCHG uses a multi-level compositional hierarchical graph with atom-bond cross-view contrastive learning, functional group prediction, and structure tasks to achieve top results on seven of nine MoleculeNet benchmarks.
SCOPE-BENCH shows state-of-the-art molecular models suffer up to 8x higher errors under extreme OOD, while POMA reduces mean absolute error by up to 11.2% via target-aware source selection and dual-scale adaptation.
A pre-trained interference-aware graph Transformer model for wireless resource allocation that achieves strong few-shot adaptation to new tasks and scenarios.
MolDA is a multimodal molecular model that uses a discrete large language diffusion backbone plus a hybrid graph encoder to achieve better global coherence and validity than autoregressive approaches.
EnFlow integrates flow-based conformer generation with energy landscape modeling to enable joint ensemble generation and ground-state identification using only 1-2 ODE steps.
SSL4RL reformulates self-supervised learning objectives into dense, verifiable reward signals for RL-based fine-tuning of vision-language models, yielding performance gains on reasoning benchmarks.
GraphPINE is a GNN architecture that initializes node importance from prior knowledge graphs and propagates updates via an importance propagation layer for interpretable drug response prediction on over 5,000 genes and 952 drugs.
NaFM is a pretrained foundation model for natural products using scaffold-focused contrastive learning and masked graph objectives that achieves SOTA on taxonomy classification, gene/microbial analysis, and virtual screening tasks.
FARM adds atomic-level functional group annotations to create FG-enhanced SMILES and FG graphs, trains them with masked language modeling and GNNs plus contrastive alignment, and reports state-of-the-art results on 8 of 13 MoleculeNet tasks.
Learnable graph patches enable domain-agnostic pre-training of graph models by decomposing heterogeneous graphs into transferable semantic units via patch encoders and aggregators.
EssentialGIN applies a modified graph isomorphism network to PPI networks with integrated biological node attributes to predict essential genes, outperforming centrality measures, Node2Vec, MLP, and GAT especially in human data.
MolE-RAG is a training-free RAG framework that augments LLMs with literature, molecular context, and structural analogs to improve performance on nine molecular property prediction tasks.
A unified training framework for mesh-based ML surrogates in CFD improves accuracy and long-horizon stability by enforcing spatial derivative consistency via multi-node prediction, using temporal cross-attention correction, and adding 3D rotary positional embeddings.
Hyformer jointly models molecule generation and property prediction via alternating attention and joint pre-training, showing synergistic gains in conditional sampling, OOD prediction, and a drug design case for antimicrobial peptides.
DiGGR introduces a self-supervised graph representation learning framework that disentangles latent factors to guide mask modeling and improve representation quality on graph tasks.
Pre-training GNNs on ECFP prediction produces statistically significant QSAR gains on five of six Biogen benchmarks with OOD splits, but underperforms on heterogeneous datasets and complex endpoints like binding affinity.
citing papers explorer
-
EpiFormer: Learning Antigen-Antibody Interactions for Epitope Prediction via Geometric Deep Learning
EpiFormer improves epitope prediction F1 score by over 40% via early-fusion cross-attention in GNN layers and sparsity-aware objectives, while recovering known biology as emergent behavior.
-
scShapeBench: Discovering geometry from high dimensional scRNAseq data
scShapeBench supplies synthetic and real annotated single-cell datasets across four shape categories, with scReebTower outperforming PAGA and Mapper on topology-aware metrics.
-
Structural Interpretations of Protein Language Model Representations via Differentiable Graph Partitioning
SoftBlobGIN combines ESM-2 representations with protein contact graphs via a lightweight GNN and differentiable substructure pooling to achieve 92.8% accuracy on enzyme classification, raise binding-site AUROC to 0.983, and generate auditable structural explanations without retraining the language模型
-
Canopy: A Heterograph Foundation Model for Metabolic Engineering
Frozen embeddings from a pretrained heterogeneous graph transformer over a 6.9M-node metabolic-engineering knowledge graph predict fermentation titers at R²=0.41, outperforming tabular baselines (R²=0.24).
-
Graph Neural Networks Are Not Continuous Across Graph Resolutions
GNNs are shown to lack continuity under graph resolution changes due to message-passing schemes, with a derived modification enabling consistent multi-scale representations validated experimentally.
-
A Topological Characterization of Graph Neural Networks via Stochastic Block Model Embeddings on the n-Sphere
Trained MPNNs factor through bounded step-graphon-signals that embed via an explicit map into disjoint caps on the n-sphere, producing a topological fingerprint for model comparison and retrieval.
-
Detecting Gravitational-Wave Anisotropies with Simulation-Based Inference
A neural-network-based simulation inference method improves 3σ detection probability of gravitational-wave background anisotropies by 90-200% over Gaussian frequentist searches by learning non-Gaussian structure in pulsar timing residuals.
-
EvoStruct: Bridging Evolutionary and Structural Priors for Antibody CDR Design via Protein Language Model Adaptation
EvoStruct integrates evolutionary priors from a protein language model with structural priors from an E(3)-equivariant GNN to raise amino acid recovery by 16% and diversity by 2.3x on CHIMERA-Bench while cutting perplexity 43%.
-
Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction
MolCHG uses a multi-level compositional hierarchical graph with atom-bond cross-view contrastive learning, functional group prediction, and structure tasks to achieve top results on seven of nine MoleculeNet benchmarks.
-
Rethinking Molecular OOD Generalization via Target-Aware Source Selection
SCOPE-BENCH shows state-of-the-art molecular models suffer up to 8x higher errors under extreme OOD, while POMA reduces mean absolute error by up to 11.2% via target-aware source selection and dual-scale adaptation.
-
A Graph Foundation Model for Wireless Resource Allocation
A pre-trained interference-aware graph Transformer model for wireless resource allocation that achieves strong few-shot adaptation to new tasks and scenarios.
-
MolDA: Molecular Understanding and Generation via Large Language Diffusion Model
MolDA is a multimodal molecular model that uses a discrete large language diffusion backbone plus a hybrid graph encoder to achieve better global coherence and validity than autoregressive approaches.
-
Energy-Guided Generative Modeling for Low-Energy Molecular Structure Discovery
EnFlow integrates flow-based conformer generation with energy landscape modeling to enable joint ensemble generation and ground-state identification using only 1-2 ODE steps.
-
SSL4RL: Revisiting Self-supervised Learning as Intrinsic Reward for Visual-Language Reasoning
SSL4RL reformulates self-supervised learning objectives into dense, verifiable reward signals for RL-based fine-tuning of vision-language models, yielding performance gains on reasoning benchmarks.
-
GraphPINE: Graph Importance Propagation for Interpretable Drug Response Prediction
GraphPINE is a GNN architecture that initializes node importance from prior knowledge graphs and propagates updates via an importance propagation layer for interpretable drug response prediction on over 5,000 genes and 952 drugs.
-
Pretraining a Foundation Model for Small-Molecule Natural Products
NaFM is a pretrained foundation model for natural products using scaffold-focused contrastive learning and masked graph objectives that achieves SOTA on taxonomy classification, gene/microbial analysis, and virtual screening tasks.
-
FARM: Enhancing Molecular Representations with Functional Group Awareness
FARM adds atomic-level functional group annotations to create FG-enhanced SMILES and FG graphs, trains them with masked language modeling and GNNs plus contrastive alignment, and reports state-of-the-art results on 8 of 13 MoleculeNet tasks.
-
Handling Feature Heterogeneity with Learnable Graph Patches
Learnable graph patches enable domain-agnostic pre-training of graph models by decomposing heterogeneous graphs into transferable semantic units via patch encoders and aggregators.
-
EssentialGIN: a new approach for gene essentiality prediction based on graph isomorphism neural networks
EssentialGIN applies a modified graph isomorphism network to PPI networks with integrated biological node attributes to predict essential genes, outperforming centrality measures, Node2Vec, MLP, and GAT especially in human data.
-
MolE-RAG: Molecular Structure-Enhanced Retrieval-Augmented Generation for Chemistry
MolE-RAG is a training-free RAG framework that augments LLMs with literature, molecular context, and structural analogs to improve performance on nine molecular property prediction tasks.
-
Mesh Based Simulations with Spatial and Temporal awareness
A unified training framework for mesh-based ML surrogates in CFD improves accuracy and long-horizon stability by enforcing spatial derivative consistency via multi-node prediction, using temporal cross-attention correction, and adding 3D rotary positional embeddings.
-
Synergistic Benefits of Joint Molecule Generation and Property Prediction
Hyformer jointly models molecule generation and property prediction via alternating attention and joint pre-training, showing synergistic gains in conditional sampling, OOD prediction, and a drug design case for antimicrobial peptides.
-
Disentangled Generative Graph Representation Learning
DiGGR introduces a self-supervised graph representation learning framework that disentangles latent factors to guide mask modeling and improve representation quality on graph tasks.
-
On Improving Graph Neural Networks for QSAR by Pre-training on Extended-Connectivity Fingerprints
Pre-training GNNs on ECFP prediction produces statistically significant QSAR gains on five of six Biogen benchmarks with OOD splits, but underperforms on heterogeneous datasets and complex endpoints like binding affinity.
-
Financial Anomaly Detection for the Canadian Market
Neural network and TDA methods outperform PCA at detecting financial anomalies in the Canadian TSX-60 market.
- AgForce Enables Antigen-conditioned Generative Antibody Design
- ConTact: Contact-First Antibody CDR Design via Explicit Interface Reasoning
- BiScale-GTR: Fragment-Aware Graph Transformers for Multi-Scale Molecular Representation Learning
- Property Enhanced Instruction Tuning for Multi-task Molecule Generation with Large Language Models