GSTBench: A Benchmark Study on the Transferability of Graph Self-Supervised Learning

Bo Long; Hui Liu; Jingzhe Liu; Yan Xie; Yu Song; Zhigang Hua

REVIEW 3 major objections 6 minor 43 references

GSTBench: A Benchmark Study on the Transferability of Graph Self-Supervised Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Graph self-supervised pretraining mostly fails to transfer across datasets, while masked feature reconstruction (GraphMAE) consistently helps.

desk verdict A useful large-scale graph SSL benchmark, but the 'GraphMAE consistently wins' claim is contradicted by the paper's own tables and the checkpoint-selection protocol undercuts the comparison. read the letter →

arxiv 2509.06975 v1 pith:OUQRWIRB submitted 2025-08-28 cs.LG cs.AI

Yu Song , Zhigang Hua , Yan Xie , Jingzhe Liu , Bo Long , Hui Liu This is my paper

classification cs.LGcs.AI

keywords graphrepresentationlearningfoundationmodelsself-supervisedtransferabilitypretrain-then-transfermaskedautoencodercontrastivebenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish whether graph self-supervised learning can support a pretrain-then-transfer paradigm, and on what terms. It introduces GSTBench, a controlled benchmark in which five representative SSL objectives are pretrained on the 100-million-node ogbn-papers100M graph and adapted to eight target graphs using fixed backbones and LLM-derived node features. The central finding is that most objectives transfer poorly: contrastive methods such as DGI and GRACE often perform worse than random initialization, whereas GraphMAE, a masked autoencoder that reconstructs masked node features, consistently improves downstream accuracy. The paper also reports that fine-tuning gives no benefit over training from scratch under five-shot node classification, while linear probing and in-context learning do exploit pretraining. If these results hold, masked feature reconstruction is the safest pretraining signal for cross-graph transfer, and graph foundation model efforts should prioritize generative feature-recovery objectives over augmentation-based contrastive ones.

What carries the argument

The load-bearing mechanism is the comparison protocol itself: GSTBench. It isolates the pretraining objective as the only variable by fixing the backbone (GCN or GAT), the node features (SentenceBERT embeddings of text attributes), the pretraining corpus (ogbn-papers100M, split into roughly 10,000 METIS subgraphs), and the adaptation protocol (linear probing, in-context learning, fine-tuning). Against this fixed backdrop, five objectives run under an identical hyperparameter search budget: masked feature reconstruction (GraphMAE), variational structure reconstruction (VGAE), mutual-information contrastive learning (DGI), augmentation-based contrastive learning (GRACE), and link prediction (L

What would settle it

Pretrain DGI and GRACE on the same 100-million-node corpus for many more epochs with full-graph or larger-neighborhood context, select checkpoints by held-out pretext loss rather than downstream accuracy, and re-run the eight-dataset linear-probing evaluation; if either contrastive method then matches or exceeds GraphMAE's consistent gains, the paper's central claim about masked feature reconstruction as the uniquely transferable objective is refuted.

Watch

Extended reading notes

Core claim

GSTBench's core claim is that under a fully controlled pretrain-then-transfer protocol, masked feature reconstruction is the only graph SSL objective that transfers reliably across datasets. Pretraining on ogbn-papers100M, the paper finds that GraphMAE improves five-shot node classification over random initialization on every target graph and both backbones tested, and also achieves the best link-prediction MRR on in-domain graphs. The same protocol turns VGAE and contrastive methods (DGI, GRACE) into unstable or negative transfer, with DGI degrading performance as pretraining proceeds. The paper identifies two further regularities: generative objectives beat contrastive ones because feature

Load-bearing premise

The ranking assumes five epochs of pretraining on partitioned subgraphs is enough, and equally fair, for every objective, and that choosing the checkpoint by downstream node-classification accuracy does not favor the objective that fits the evaluation protocol.

Editorial extensions

If this is right

For practitioners building graph foundation models, masked feature reconstruction is the default pretraining objective: it is the only one of the five tested that beats random initialization across all target datasets and backbones.
Lightweight adaptation is the right evaluation mode: linear probing and in-context learning reveal pretraining gains, while full fine-tuning washes them out, so benchmarks should report frozen-encoder results rather than fine-tuned ones.
Contrastive graph SSL should not be assumed safe for transfer; augmentation-based objectives can actively destroy the semantic signal in LLM-derived features when pretraining and target graphs differ.
Link-prediction pretraining and feature-reconstruction pretraining transfer only under distributional alignment; cross-domain gains require task alignment or architecture alignment between pretraining and downstream phases.
Pretraining loss is a useful but not sufficient guide: lower SSL error correlates with better downstream accuracy, yet accuracy plateaus or declines with over-optimization, so early stopping is needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's 5-epoch pretraining budget and checkpoint selection by downstream node-classification accuracy are likely to understate contrastive methods; a testable extension is running DGI and GRACE for substantially longer pretraining (or with full-graph context) and selecting checkpoints by pretext loss, to see whether their negative-transfer verdict reverses.
If masked feature reconstruction's advantage comes from preserving LLM feature semantics, then masking strategies that target semantically redundant tokens, or reconstructing at multiple granularities, should push transfer performance further; this is directly testable with the same benchmark.
The fine-tuning null result suggests a hybrid adaptation: freeze the pretrained encoder for a warm-start phase and only then unfreeze, which may recover some fine-tuning benefit without destroying the pretrained initialization.
The VGAE link-prediction failure isolates decoder compatibility as a transfer factor; swapping VGAE's dot-product decoder for an MLP decoder during pretraining is a cheap intervention that could turn a negative-transfer method into a positive one.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take on arXiv:2509.06975 (GSTBench). It's a serious empirical effort with a real asset: a controlled, large-scale comparison of five graph SSL objectives under an LLM-unified feature space, pretraining on ogbn-papers100M and evaluating over eight datasets with three adaptation strategies. If you work on graph foundation models or pretraining, you'll want to know what they did and what they found. But the headline claim — that GraphMAE 'consistently improves transfer performance' — doesn't survive contact with their own tables.

Across the board, they report standard deviations, and most positives are within one SD. The 'consistent' wording in the abstract and §4.3 is not supported by Table 3 (GraphMAE GCN on A-Ratings: 0.256±0.036 vs random 0.257±0.041; GAT: 0.244±0.041 vs 0.245±0.033) or Table 6 (GraphMAE GAT cross-domain link prediction mean MRR 0.855 vs random 0.856). So the main actionable conclusion — choose GraphMAE for transfer — needs to be scaled back to something like 'masked feature reconstruction is a promising default, but gains are modest and dataset-dependent.'

Then there's the protocol issue. The best checkpoint is selected using downstream node-classification accuracy (Section 4.1). That makes the comparison self-referential: the pretraining method that happens to produce a checkpoint that lands on the right validation values looks better than one that doesn't, and there's no target-free early stopping. This can inflate GraphMAE's apparent lead. They also train all methods for only 5 epochs on METIS subgraphs and assert convergence without showing it; that may disadvantage contrastive and other methods that need longer schedules. These two choices need to be justified or fixed.

Also, 'first systematic benchmark' is too strong given the existence of closely related benchmarks in refs [4] and [29] (with shared authors). They should position this as a systematic comparison under a specific controlled protocol, not the first.

To be fair, the paper does several things well. The effort to decouple architecture, features, and adaptation is real. The SSL-error vs downstream correlation analysis (Fig. 2) is a nice idea. The VGAE/LP architectural-mismatch hypothesis and the GAT overwriting analysis (Fig. 3) are genuinely informative. The code is released. So the work is a useful benchmark, and I'd like to see it in the literature after major revision.

Bottom line: this deserves a serious referee, but it needs a revised claim set and a fairer protocol. If I were reviewing, I'd ask for either a fixed checkpoint without downstream peeking, or a clear justification for why this doesn't bias the ranking, plus significance testing and a de-scaled novelty statement.

Referee Report

3 major / 6 minor

Summary. The paper introduces GSTBench, a benchmark study of cross-dataset transferability for five graph self-supervised learning objectives (GraphMAE, VGAE, GRACE, DGI, and link prediction). All methods are pretrained on ogbn-papers100M using LLM-derived SentenceBERT node features and METIS subgraphs, then evaluated on eight downstream graphs under linear probing, in-context learning, and fine-tuning for node classification, plus fine-tuning for link prediction. The central claim is that most graph SSL methods generalize poorly, sometimes worse than random initialization, while GraphMAE consistently improves transfer performance. The paper also reports that fine-tuning provides no significant benefit over random initialization under five-shot node classification.

Significance. If the results are taken at face value, the benchmark would provide actionable guidance for graph foundation model research: masked feature reconstruction would be a reliable pretraining objective, while contrastive objectives would frequently cause negative transfer. The paper's controlled setup—fixed architectures, unified LLM features, standardized adaptation protocols, and public code—is a genuine strength and makes the resource useful regardless of the headline conclusion. The large-scale pretraining on papers100M and the inclusion of both in-domain and cross-domain targets are also valuable. However, the headline claim about GraphMAE's consistency is contradicted by the paper's own tables, and the checkpoint-selection protocol undermines the clean comparison the paper advertises. With appropriately de-scaled claims and additional statistical support, this would be a useful benchmark contribution.

major comments (3)

[§4.3, Table 3; abstract] The claim that 'GraphMAE consistently leads to positive transfer across all models and datasets' is not supported by the reported results. In Table 3, GraphMAE with GCN on Amazon Ratings gives 0.256±0.036 versus random initialization 0.257±0.041, and with GAT gives 0.244±0.041 versus 0.245±0.033—both worse than random. In Table 6, GraphMAE with GAT on cross-domain link prediction has mean MRR 0.855 versus 0.856 for random initialization. Moreover, many of the positive differences, e.g., Cora GCN 0.748±0.011 vs 0.737±0.016 and DBLP GCN 0.731±0.031 vs 0.712±0.033, are within one standard deviation, and no significance tests are reported. The abstract and §4.3 wording must be weakened, or the authors must provide paired significance testing and explicitly acknowledge the counterexamples.
[§4.1, Training Details; Tables 3–5] The best-checkpoint selection rule is a serious protocol problem. The paper states that 'The best checkpoint is selected based on the mean node classification performance on downstream datasets.' This uses downstream validation labels to choose the pretraining epoch, which leaks target information into the pretraining decision and gives SSL methods an advantage over the random-initialization baseline, which is evaluated at epoch 0 without any checkpoint selection. This makes the comparison between pretrained objectives and baseline a comparison between 'best downstream checkpoint' and 'no selection.' It also means the ranking among SSL methods may reflect early stopping rather than the quality of the pretraining objective. The authors should either select checkpoints using a source-only criterion (e.g., validation loss on held-out source subgraphs) or report results for all epochs and sh
[§4.1, Training Details; Figure 1] The statement that training for a maximum of 5 epochs on METIS subgraphs is 'sufficient for convergence' is not supported by the evidence in Figure 1. Several methods show non-converged or unstable dynamics at epoch 5: VGAE exhibits an error surge at the beginning and ends near baseline loss, DGI and LP show opposite trends on different datasets, and GRACE's loss is still decreasing substantially. Because the pretraining budget is fixed and the checkpoint is chosen by downstream accuracy, the negative-transfer results for contrastive methods such as GRACE and DGI could be an artifact of insufficient training or subgraph-based pretraining. The authors should provide pretraining loss curves or convergence metrics for all methods and either extend the epoch budget or justify that 5 epochs is fair for every objective.

minor comments (6)

[Introduction] Typo: 'an masked autoencoding approache' should be 'a masked autoencoding approach.'
[Related Work] Typo: 'and and deliver generalizable guidance' contains a duplicated 'and.'
[Figure 1 caption] The caption contains 'the the SSL error' and 'the the validation accuracy'; please fix.
[Appendix A.2] Typo: 'appraoch' should be 'approach.'
[§4.2, Figure 2] The text 'but no strictly' is grammatically incomplete; it should state that the correlation is not strictly monotonic.
[Table 3 and Table 6] The random-initialization baseline should be described more precisely: is it a single epoch-0 evaluation, and does it use the same feature scaling and linear-probe training as the pretrained checkpoints? This is relevant to interpreting the baseline's role in the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical benchmark, not a derivation; self-citations are protocol-only.

full rationale

GSTBench is an empirical benchmark; it contains no derivation chain in which a claimed prediction is algebraically equivalent to its inputs. The nearest candidate is the checkpoint-selection rule in Sec. 4.1 ('The best checkpoint is selected based on the mean node classification performance on downstream datasets'), which might look self-referential. But the reported numbers are test accuracies from the split described in Sec. 4.1 ('500 validation nodes, and the remainder for testing'), so this is a standard validation-based model-selection step, not a fit to the reported metric. The paper's use of the authors' own prior work ([4], [29]) for dataset processing, SentenceBERT features, and the ICL protocol is load-bearing only for the experimental setup, not for the central conclusion about GraphMAE; the conclusion is based on the reported tables. The stated limitation about SentenceBERT and the internal inconsistencies (e.g., Table 3 shows GraphMAE below random init on A-Ratings; Table 6 shows GraphMAE GAT below random init cross-domain) are correctness/support concerns, not circularity. No step reduces a result to its input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the claims are empirical and rest on dataset choices, the SentenceBERT feature space, METIS partition pretraining, the 5-epoch budget, and the downstream-peeking checkpoint rule. These are the main unpaid-for inputs.

free parameters (3)

Pretraining epochs (max 5) = 5
Chosen because 5 epochs were 'empirically... sufficient for convergence' (Sec 4.1); if contrastive methods need more epochs, comparisons are biased.
Checkpoint selection rule = best downstream validation accuracy
Best checkpoint selected using mean node classification performance on downstream datasets (Sec 4.1), leaking target labels into pretraining choices and potentially favoring certain objectives.
SSL-specific hyperparameters = mask rate 0.5, alpha 3, feature/edge drop 0.2, etc.
Fixed from original papers (Appendix A.1); no transfer-specific tuning, so methods are not equally optimized for transfer.

assumptions (4)

domain assumption SentenceBERT embeddings form a shared semantic space that makes cross-dataset transfer meaningful.
Section 3 and 4.1 use SentenceBERT for all node features; the paper acknowledges in the Conclusion that the choice of language model may change conclusions.
domain assumption Pretraining a GNN on METIS partitions of ogbn-papers100M is a faithful proxy for large-scale graph pretraining.
Sec 4.1 preprocesses the graph into ~10,000 subgraphs via METIS; this loses global structure and assumes partition-level training transfers to full graphs and downstream tasks.
domain assumption Linear probing, in-context learning, and fine-tuning cover the relevant adaptation protocols.
Section 3 defines these three strategies; if other adapters behave differently, conclusions about adaptation may not generalize.
domain assumption Downstream accuracy on target datasets is a valid measure of transferability independent of the pretraining objective.
Sec 4.1 uses 5-shot classification and link prediction; random-init baseline is high because of rich features, so gains are small and may be within noise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GSTBench: A Benchmark Study on the Transferability of Graph Self-Supervised Learning." pith.science (2026). https://pith.science/paper/OUQRWIRB

@misc{pith2026250906975,
  author       = {Pith},
  title        = {Pith review of: GSTBench: A Benchmark Study on the Transferability of Graph Self-Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OUQRWIRB}},
  note         = {Machine review of arXiv:2509.06975}
}

read the original abstract

Self-supervised learning (SSL) has shown great promise in graph representation learning. However, most existing graph SSL methods are developed and evaluated under a single-dataset setting, leaving their cross-dataset transferability largely unexplored and limiting their ability to leverage knowledge transfer and large-scale pretraining, factors that are critical for developing generalized intelligence beyond fitting training data. To address this gap and advance foundation model research for graphs, we present GSTBench, the first systematic benchmark for evaluating the transferability of graph SSL methods. We conduct large-scale pretraining on ogbn-papers100M and evaluate five representative SSL methods across a diverse set of target graphs. Our standardized experimental setup decouples confounding factors such as model architecture, dataset characteristics, and adaptation protocols, enabling rigorous comparisons focused solely on pretraining objectives. Surprisingly, we observe that most graph SSL methods struggle to generalize, with some performing worse than random initialization. In contrast, GraphMAE, a masked autoencoder approach, consistently improves transfer performance. We analyze the underlying factors that drive these differences and offer insights to guide future research on transferable graph SSL, laying a solid foundation for the "pretrain-then-transfer" paradigm in graph learning. Our code is available at https://github.com/SongYYYY/GSTBench.

Figures

Figures reproduced from arXiv: 2509.06975 by the authors.

**Figure 2.** Correlation between SSL error and validation ac [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** GAT fine-tuning tends to overwrite pretrained pa [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 15 canonical work pages

[4]

Zhikai Chen, Haitao Mao, Jingzhe Liu, Yu Song, Bingheng Li, Wei Jin, Bahare Fatemi, Anton Tsitsulin, Bryan Perozzi, Hui Liu, et al. 2024. Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights. arXiv preprint arXiv:2406.10727 (2024)

arXiv 2024
[29]

Yu Song, Haitao Mao, Jiachen Xiao, Jingzhe Liu, Zhikai Chen, Wei Jin, Carl Yang, Jiliang Tang, and Hui Liu. 2024. A Pure Transformer Pretraining Framework on Text-attributed Graphs. arXiv preprint arXiv:2406.13873 (2024)

arXiv 2024
[1]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin,...

arXiv 2025
[2]

Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. 2024. Llaga: Large language and graph assistant.arXiv preprint arXiv:2402.08170 (2024)

arXiv 2024
[3]

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al . 2024. Exploring the potential of large language models (llms) in learning on graphs. ACM SIGKDD Explorations Newsletter 25, 2 (2024), 42–61

work page 2024
[5]

Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Ol- gica Milenkovic, and Inderjit S Dhillon. 2021. Node feature extraction by self- supervised multi-scale neighborhood prediction. arXiv preprint arXiv:2111.00064 (2021)

arXiv 2021
[6]

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

arXiv 2018
[7]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in neural information processing systems 30 (2017)

2017

Show all 43 references

[8]

Gir- shick

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll’ar, and Ross B. Gir- shick. 2021. Masked Autoencoders Are Scalable Vision Learners. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021), 15979– 15988. https://api.semanticscholar.org/Cor...

2021
[9]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 9729–9738

2020
[10]

Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. 2023. Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning. arXiv preprint arXiv:2305.19523 (2023)

2023 arXiv
[11]

Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. 2023. Graphmae2: A decoding-enhanced masked self-supervised graph learner. In Proceedings of the ACM web conference 2023 . 737–746

2023
[12]

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. 2022. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 594–604

2022
[13]

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems 33 (2020), 22118–22133

2020
[14]

Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng, Percy S Liang, and Jure Leskovec. 2024. Prodigy: Enabling in-context learning over graphs. Advances in Neural Information Processing Systems 36 (2024)

2024
[15]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)

2016 arXiv
[16]

Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016)

2016 arXiv
[17]

Juanhui Li, Harry Shomer, Haitao Mao, Shenglai Zeng, Yao Ma, Neil Shah, Jiliang Tang, and Dawei Yin. 2023. Evaluating graph neural networks for link predic- tion: Current pitfalls and new benchmarking. Advances in Neural Information Processing Systems 36 (2023), 3853–3866

2023
[18]

Yichuan Li, Kaize Ding, and Kyumin Lee. 2023. Grenade: Graph-centric language model for self-supervised representation learning on text-attributed graphs.arXiv preprint arXiv:2310.15109 (2023)

2023 arXiv
[19]

Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Jia Li. 2024. Zerog: Investigating cross-dataset zero-shot transferability in graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 1725–1735

2024
[20]

Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. 2023. One for all: Towards training one graph model for all classification tasks. arXiv preprint arXiv:2310.00149 (2023)

2023 arXiv
[21]

Qian Ma, Haitao Mao, Jingzhe Liu, Zhehua Zhang, Chunlin Feng, Yu Song, Yihan Shao, and Yao Ma. 2024. Do Neural Scaling Laws Exist on Graph Self-Supervised Learning? arXiv preprint arXiv:2408.11243 (2024)

2024 arXiv
[22]

Yao Ma and Jiliang Tang. 2021. Deep learning on graphs . Cambridge University Press

2021
[23]

Haitao Mao, Zhikai Chen, Wenzhuo Tang, Jianan Zhao, Yao Ma, Tong Zhao, Neil Shah, Mikhail Galkin, and Jiliang Tang. 2024. Position: Graph foundation models are already here. In Forty-first International Conference on Machine Learning

2024
[24]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space.arXiv preprint arXiv:1301.3781 (2013)

2013 arXiv
[25]

Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. 2023. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. InInternational Conference on Machine Learning. PMLR, 28223–28243

2023
[26]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

2021
[27]

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)

2019 arXiv
[28]

Yu Song, Zhigang Hua, Harry Shomer, Yan Xie, Jingzhe Liu, Bo Long, and Hui Liu. 2025. A Scalable Pretraining Framework for Link Prediction with Efficient Adaptation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2 . 2621–2632

2025
[30]

Mingchen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang. 2022. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1717–1727

2022
[31]

Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. 2023. All in one: Multi-task prompting for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 2120–2131

2023
[32]

Xiangguo Sun, Jiawen Zhang, Xixi Wu, Hong Cheng, Yun Xiong, and Jia Li. 2023. Graph prompt learning: A comprehensive survey and beyond. arXiv preprint arXiv:2311.16534 (2023)

2023 arXiv
[33]

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veličković, and Michal Valko. 2021. Large- scale representation learning on graphs via bootstrapping. arXiv preprint arXiv:2102.06514 (2021)

2021 arXiv
[34]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)

2017 arXiv
[35]

Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2018. Deep graph infomax. arXiv preprint arXiv:1809.10341 (2018)

2018 arXiv
[36]

Shijie Wang, Jiani Huang, Zhikai Chen, Yu Song, Wenzhuo Tang, Haitao Mao, Wenqi Fan, Hui Liu, Xiaorui Liu, Dawei Yin, et al. 2024. Graph machine learning in the era of large language models (llms).ACM Transactions on Intelligent Systems and Technology (2024)

2024
[37]

Lianghao Xia and Chao Huang. 2024. AnyGraph: Graph Foundation Model in the Wild. arXiv preprint arXiv:2408.10700 (2024)

2024 arXiv
[38]

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. Advances in neural information processing systems 33 (2020), 5812–5823

2020
[39]

Haihong Zhao, Aochuan Chen, Xiangguo Sun, Hong Cheng, and Jia Li. 2024. All in one and one for all: A simple yet effective method towards cross-domain graph pretraining. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 4443–4454

2024
[40]

Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. 2022. Learning on large-scale text-attributed graphs via variational inference. arXiv preprint arXiv:2210.14709 (2022)

2022 arXiv
[41]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 (2020)

2020 arXiv
[42]

Chenyi Zi, Haihong Zhao, Xiangguo Sun, Yiqing Lin, Hong Cheng, and Jia Li
[2024]

arXiv preprint arXiv:2406.05346 (2024)

Prog: A graph prompt learning benchmark. arXiv preprint arXiv:2406.05346 (2024)

2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [4]

Zhikai Chen, Haitao Mao, Jingzhe Liu, Yu Song, Bingheng Li, Wei Jin, Bahare Fatemi, Anton Tsitsulin, Bryan Perozzi, Hui Liu, et al. 2024. Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights. arXiv preprint arXiv:2406.10727 (2024)

arXiv 2024

[2] [29]

Yu Song, Haitao Mao, Jiachen Xiao, Jingzhe Liu, Zhikai Chen, Wei Jin, Carl Yang, Jiliang Tang, and Hui Liu. 2024. A Pure Transformer Pretraining Framework on Text-attributed Graphs. arXiv preprint arXiv:2406.13873 (2024)

arXiv 2024

[3] [1]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin,...

arXiv 2025

[4] [2]

Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. 2024. Llaga: Large language and graph assistant.arXiv preprint arXiv:2402.08170 (2024)

arXiv 2024

[5] [3]

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al . 2024. Exploring the potential of large language models (llms) in learning on graphs. ACM SIGKDD Explorations Newsletter 25, 2 (2024), 42–61

work page 2024

[6] [5]

Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Ol- gica Milenkovic, and Inderjit S Dhillon. 2021. Node feature extraction by self- supervised multi-scale neighborhood prediction. arXiv preprint arXiv:2111.00064 (2021)

arXiv 2021

[7] [6]

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

arXiv 2018

[8] [7]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in neural information processing systems 30 (2017)

2017