Pith. sign in

REVIEW 3 major objections 5 minor 12 references

Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pretrained graph embeddings help downstream tasks only when enough labels are available and pretraining data and feature spaces align.

desk verdict Honest and useful preliminary study, but the abstract's data-regime claim is not established because dropout is fixed across data sizes. read the letter →

arxiv 2412.17609 v1 pith:TU6QGQU6 submitted 2024-12-23 cs.LG cs.NE

Fabrizio Frasca , Fabian Jogl , Moshe Eliasof , Matan Ostrovsky , Carola-Bibiane Schönlieb , Thomas Gärtner , Haggai Maron This is my paper · ORCID
classification cs.LGcs.NE
keywords GraphNeuralNetworksFoundationModelsTransferLearningFeature-StructuralizationPretrainingCross-DatasetPositionalandStructuralEncodingsLow-DataRegime
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks when a graph neural network pretrained on one set of graphs can be reused on another, a question that stands between current graph learning and graph foundation models. Using a feature-agnostic structural pretraining scheme that predicts positional and structural encodings, the authors test transfer across three molecular datasets with different structures and feature vocabularies. The experiments indicate that pretrained embeddings are not a free lunch: they beat a non-pretrained baseline only once enough downstream labels are available, and their benefit depends on the size and composition of the pretraining corpus. The paper also proposes Feature-Structuralization, which rewrites categorical node features as extra nodes and edges so the pretrained model stays feature-agnostic while still seeing feature information, but this does not produce a consistent gain and underperforms when pretraining and downstream feature vocabularies differ. The conclusions are explicitly conditional, resting mainly on molecular graphs, with molpcba downstream results flagged as preliminary.

What carries the argument

The load-bearing object is the structural pretraining pipeline: a GNN is pretrained on featureless graphs to predict graph positional and structural encodings (P/SEs) such as Laplacian eigenvectors and eigenvalues, electrostatic potentials, random-walk and heat-kernel structural encodings. The paper's proposed addition, Feature-Structuralization, converts categorical node features into auxiliary feature-nodes, one per category per channel, connected to the nodes carrying that category, and then discards explicit features so the input remains feature-agnostic; the same P/SE prediction targets are computed on original nodes of both the original and the structuralized graph. This construction carries the argument because it is the mechanism that lets feature information enter pretraining without dataset-specific encodings, and the experimental comparisons use it to isolate when added structure helps or hurts transfer.

What would settle it

Run the same pretraining and downstream protocol on a non-molecular graph collection whose categorical feature vocabularies do not overlap between source and target; if pretrained structural embeddings improve test performance at small downstream training fractions, or if Feature-Structuralization consistently beats vanilla structural pretraining when features are off-distribution, the paper's conditional conclusions would be overturned.

Watch

Extended reading notes

Core claim

The paper's central discovery is that cross-dataset transfer of pretrained GNNs is real but conditional. In downstream evaluations on zinc and peptides, embeddings from pretrained models improve generalization over a non-pretrained baseline once there is sufficient labeled data, but can hurt in very scarce-data settings; on peptides the pattern is non-monotonic, with pretrained embeddings harmful at a 0.1 training fraction yet helpful again at 0.01. The composition of the pretraining corpus matters: including datasets beyond the target rarely degrades performance, and mixtures that exclude the target can match or slightly exceed target-only pretraining, an effect the authors attribute mostly to larger pretraining corpora. Feature-Structuralization, which materializes each categorical feature channel as feature-nodes connected to the original nodes carrying that category, lets the pretraining access feature information without dataset-specific encodings, but its downstream gains are inconsistent: it helps with in-dataset pretraining and when structure is off-distribution while features are shared, and is weakest when feature vocabularies are off-distribution. Multi-dataset pretraining generally improves prediction of the structural targets themselves, yet models never fully close the gap between in-domain and off-domain pretraining, and extra pretraining data closes that gap far less efficiently than in-domain data.

Load-bearing premise

The load-bearing premise is that three molecular datasets — zinc, molpcba, and peptides — represent cross-dataset transfer of pretrained graph networks broadly enough for the conclusions to generalize, with molpcba's downstream evidence itself preliminary.

Editorial extensions

If this is right

  • Pretrained structural embeddings should not be expected to rescue extreme low-data regimes; on peptides at a 0.01 training fraction they help, but at 0.1 they can hurt, so the value of pretraining depends on the downstream data budget in a nontrivial way.
  • Pretraining corpus design is a usable lever: adding datasets other than the target generally does not degrade transfer and can match or beat target-only pretraining, so larger mixed corpora are a reasonable route toward graph foundation models.
  • Feature-Structuralization is not a reliable way to add feature information: its benefit depends on overlap between pretraining and downstream feature vocabularies, so feature-agnostic pretraining that natively handles continuous or off-distribution features remains an open need.
  • Pretraining target prediction can serve as a cheap diagnostic: models pretrained on a single dataset fail near chance on structurally distant datasets such as peptides, while multi-dataset pretraining generally improves target prediction.
  • Scaling pretraining data alone is an inefficient substitute for in-domain data: more off-domain pretraining samples improve off-domain target prediction but much less than the same number of in-domain samples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether fuller molpcba pretraining changes the downstream data threshold; one plausible extension is that larger pretraining corpora lower the threshold but do not eliminate it, because off-domain transfer gains in Appendix E remain partial.
  • The U-shaped low-data behavior on peptides suggests the pretrained encoder works as a fixed feature extractor whose usefulness depends on how well the small downstream head can exploit it; an untested consequence is that freezing more downstream layers or adding regularization could push the beneficial regime toward smaller data budgets.
  • Structuralization's failure when feature vocabularies are off-distribution hints that the pretraining objective may let the model rely on dataset-specific feature-structure correlations; a testable remedy would be a pretraining loss that explicitly penalizes such correlations or an evaluation that removes shared feature categories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies whether pretrained graph neural networks transfer across graph datasets under a feature-agnostic structural-pretraining framework. It builds on the GPSE pipeline of Cantürk et al. (2024), proposes Feature-Structuralization, which converts categorical node features into auxiliary feature-nodes, and evaluates pretrained embeddings on downstream tasks with varying training fractions across ZINC-12k, ogbg-molpcba, and peptides-func. The main empirical claims are that (i) pretrained embeddings improve generalization only with sufficiently large downstream training sets and can hurt in data-scarce regimes, (ii) multi-dataset pretraining generally does not degrade and can improve transfer, and (iii) feature information via structuralization helps only when pretraining and downstream feature spaces are similar. The paper is explicitly preliminary, with molpcba results deferred to Appendix D and an acknowledged unresolved confound in Appendix B.2.

Significance. If the reported conclusions hold, they contribute useful negative and positive evidence for the feasibility of graph foundation models: structural-only pretraining can transfer across datasets, but the benefit is moderate and regime-dependent. The paper's strengths are its honest reporting of mixed results, its systematic ablations over pretraining corpora (single, pairwise, and multi-dataset), and its additional analysis of pretraining-data scale in Appendix E. The feature-structuralization idea is simple and clearly described. However, the paper's headline claims currently outrun the evidence: the data-regime conclusion rests on a fixed-dropout protocol that is explicitly untuned per downstream data size, and the empirical basis is limited to three molecular datasets, with the molpcba results openly preliminary. These issues make the central claims defensible but not yet established.

major comments (3)
  1. [Section 4(i), Appendix B.2] The central claim in the abstract and Section 4(i) that pretrained embeddings 'improve generalization only with enough downstream data points' is confounded by the fixed dropout protocol. Appendix B.2 states that dropout rates are not tuned per experiment but taken from Cantürk et al. (2024) full-data settings: 0.5 before component (d) for zinc, 0.1 for peptides, and 0.3/0.1 for molpcba. In low-data regimes the optimal regularization strength generally differs from the full-data optimum, so the observed 'detrimental in data-scarce settings' could be an artifact of over- or under-regularizing the embedding pathway rather than a property of pretrained representations. This concern is substantiated by the paper's own non-monotonic results in Appendix D: on peptides, ratio 0.01 is beneficial, ratios 0.1 and 0.25 are harmful, and ratio 0.5 is again beneficial—a pattern that does not match the simple 'only with enough data' statement. The authors acknowledge the unresolved confound in Appendix B.2 ('We reserve to tune these values...'). To establish the headline data-regime conclusion, the downstream experiments need per-regime dropout tuning and, ideally, multiple seeds.
  2. [Section 3 and Conclusions] The paper's conclusions are framed broadly about cross-dataset transfer of pretrained GNNs, but the evidence is drawn exclusively from three molecular datasets (ZINC-12k, ogbg-molpcba, peptides-func), all of which represent molecules or amino-acid chains. The conclusions in Section 5 and the abstract do not restrict the scope to molecular graphs, yet transfer behavior on non-molecular domains (e.g., social or collaboration networks, which differ in feature semantics and structure) could differ substantially. This is a load-bearing limitation because the paper's contribution is explicitly toward graph foundation models in general. At minimum, the claims should be rephrased to the molecular domain, or a non-molecular dataset should be added to support the broader framing.
  3. [Appendix D, Section 4] The molpcba downstream results are explicitly preliminary—the paper states 'we believe that more extensive experimentation is due on molpcba'—yet these results are used in support of observations (ii) and (iii) in Section 4, which concern the influence of pretraining-corpus composition and feature information. Only a few training ratios are reported for molpcba, and the paper notes that even ratio 0.014 (about 5,000 graphs) yields test AP below 0.06. Because molpcba is the only dataset with the same 9 SMILES-derived feature categories as peptides, it plays a central role in the structuralization and feature-similarity claims. Without a more complete molpcba evaluation, the feature-similarity conclusion in the abstract ('requires some similarities between pretraining and downstream feature spaces') is not fully supported by the presented evidence.
minor comments (5)
  1. [Figure 4 caption] Typo: 'Architecure' should be 'Architecture'.
  2. [Section 2 and Figure 1] The notation is a little confusing: Section 2 uses 'd' to denote the number of categorical feature channels, while Figure 4 uses '(d)' for a linear projection component. Please disambiguate to avoid confusion.
  3. [Appendix C, Figures 6 and 7] The clipping of negative R2 values to −1 and the use of 'black' for values below −1 is not explained in the main text; please add a remark in a caption or the text so that readers understand the color scale.
  4. [Appendix B.2] The downstream evaluation pipeline mentions 'No dropout or weight-decay regularization is applied' to the downstream GNN, but dropout is applied before/after component (d). Clarify whether the reported dropout values are the only regularization used in the downstream model.
  5. [Section 5] The conclusions omit any mention of the unresolved dropout confound, even though it is acknowledged in Appendix B.2. A sentence acknowledging this caveat in the main text would improve the paper's robustness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark study whose claims are evaluated against external datasets and an external pretraining baseline.

full rationale

This paper does not perform a derivation or prediction from first principles; it is an empirical study of cross-dataset transfer. The central claims — that pretrained embeddings help only with sufficient downstream data, that pretraining data composition matters, and that feature structuralization gives inconsistent gains — are evaluated by comparing downstream test performance of models with and without pretrained embeddings on three public benchmarks (ZINC-12k, ogbg-molpcba, peptides-func). The pretrained baseline (GPSE) is taken from an external reference (Cantürk et al., 2024), and the paper's own contributions (Feature-Structuralization, multi-dataset pretraining, low-data evaluation) are tested against that external baseline rather than fitted to reproduce the conclusions. No equation defines a claimed output in terms of the measured outcome; no fitted parameter is renamed as a prediction; and no load-bearing claim rests on a self-citation chain. The acknowledgments and references include prior work by some authors (e.g., Bouritsas et al., 2022), but that work is used only as contextual discussion of cyclic structural information and is not the source of the transfer results. The fixed dropout rates reused from Cantürk et al. (2024) for all data regimes (Appendix B.2) are a potential confound for the data-regime claim, and the authors explicitly acknowledge this, saying they 'reserve to tune these values' to verify whether pretrained representations could be more beneficial in low-data regimes. However, a confound is a correctness or robustness concern, not circularity: the conclusion is not forced by construction, and the non-monotonic results reported in Appendix D are presented as observations rather than as a consequence of the method's definition. The molpcba results are explicitly preliminary, but preliminary status is not circularity. Overall, the paper's claims are supported by external benchmarks and independent baselines, and no step in the argument reduces to its own inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted to derive the claims; the paper is an empirical study. The main assumptions are domain assumptions about GPSE's transferability, representativeness of the three datasets, and the structuralization premise. No new physical or theoretical entities are postulated.

assumptions (3)
  • domain assumption The GPSE pretraining objective (predicting positional and structural encodings on featureless graphs) produces representations that transfer across graph datasets.
    The paper builds its transfer analysis on the GPSE pipeline; if GPSE's representations do not transfer, the observed downstream behavior would not reflect cross-dataset transfer of pretrained GNNs generally (Section 3, Appendix B.1).
  • domain assumption The chosen datasets (ZINC-12k, ogbg-molpcba, peptides-func) are representative of cross-dataset variation in structure and features.
    The paper generalizes to cross-dataset transfer from three molecular datasets, and the authors acknowledge molpcba results are preliminary (Appendix D).
  • ad hoc to paper Feature-Structuralization preserves task-relevant feature information while remaining feature-agnostic.
    This is the proposed method's core premise; no theoretical guarantee is provided, and the paper's own results show it often does not improve over vanilla GPSE (Section 4, item v).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs." pith.science (2026). https://pith.science/paper/TU6QGQU6

@misc{pith2026241217609,
  author       = {Pith},
  title        = {Pith review of: Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TU6QGQU6}},
  note         = {Machine review of arXiv:2412.17609}
}
read the original abstract

To develop a preliminary understanding towards Graph Foundation Models, we study the extent to which pretrained Graph Neural Networks can be applied across datasets, an effort requiring to be agnostic to dataset-specific features and their encodings. We build upon a purely structural pretraining approach and propose an extension to capture feature information while still being feature-agnostic. We evaluate pretrained models on downstream tasks for varying amounts of training samples and choices of pretraining datasets. Our preliminary results indicate that embeddings from pretrained models improve generalization only with enough downstream data points and in a degree which depends on the quantity and properties of pretraining data. Feature information can lead to improvements, but currently requires some similarities between pretraining and downstream feature spaces.

Figures

Figures reproduced from arXiv: 2412.17609 by the authors.

Figure 1
Figure 1. Left: graph from zinc; Right: structuralized form. Colors represent node features. Structuralization encodes feature information into additional struc￾ture augmenting the original in￾put graph: by only encoding ab￾stract, logical relations induced by features, the pretraining approach remains feature-agnostic while still accessing meaningful information on how features interact with the original graph structure. Con… view at source ↗
Figure 2
Figure 2. GPSE pretraining eval. on LapPEs (Test R2 ). Evaluations. We evaluate the pretrained models in two settings: (i) pretraining target prediction, as￾sessing how well the models predict test targets from the three datasets; and (ii) downstream tasks, exam￾ining if and how the pretrained embeddings improve GNN generalization. These analyses are performed with varying downstream training sample sizes. 4. Results and Disc… view at source ↗
Figure 3
Figure 3. Downstream test performance for different training fraction and pretraining data (‘z’: zinc, ‘p’: peptides, ‘m’: molpcba). Left: Average Precision on peptides, higher is better. Right: Mean Absolute Error on zinc, lower is better. Downstream applications. We now present observations for downstream evaluations on zinc and peptides, which we visually summarize in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Architecure pipeline for downstream experiments. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Evaluation of predicting the P/SE targets of the original graphs with GPSE. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Evaluation of predicting the P/SE targets of the original graphs after structural￾ization. To ease visualization, negative R2 values are clipped to −1, depicted in black (negative values correspond to performance worse than the trivial predictor outputting the mean tra…
Figure 7
Figure 7. Figure 7: Evaluation of predicting the P/SE targets of structuralized graphs (after struc￾turalization). Test R2 coefficient is reported. To ease visualization, negative R2 values are clipped to −1, depicted in black (negative values correspond to per￾formance worse than the tri…
Figure 8
Figure 8. Figure 8: Results of the full downstream evaluation on [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Results of the full downstream evaluation on [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Results of the full downstream evaluation on [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

  1. [1]

    Incidence Networks for Geometric Deep Learning

    Marjan Albooyeh, Daniele Bertolini, and Siamak Ravanbakhsh. Incidence networks for geometric deep learning. arXiv:1905.11460,

    work page Pith review arXiv 1905

  2. [3]

    We observe that node representations from pretrained models are particularly harmful on peptides in the 0.1 training ratio setting, while, intriguingly, they provide more benefits in the lowest data regime (0.01 ratio, around 100 graphs). We hypothesize that, with such data scarcity, on peptides it may even be hard to learn a reasonable message-passing sc...

    work page 2022

  3. [4]

    These rep- resentations are linearly transformed by component (d), and then summed along with the encoding of the explicit, original node features in output from component (e)

    In particular, component (b) is the pretrained model, from which we extract the generated node representations upstream the prediction heads used in pretraining ( (c)). These rep- resentations are linearly transformed by component (d), and then summed along with the encoding of the explicit, original node features in output from component (e). These const...

    work page 2024

  4. [9]

    arXiv:2405.13934 [cs]

    URL http://arxiv.org/ abs/2405.13934. arXiv:2405.13934 [cs]. Haihong Zhao, Aochuan Chen, Xiangguo Sun, Hong Cheng, and Jia Li. All in one and one for all: A simple yet effective method towards cross-domain graph pretrain- ing. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD ’24, page 4443–4454, New York, NY, U...

    arXiv 2024

  5. [10]

    We use the same ‘mae + cosine similarity’ loss (Cant¨ urk et al., 2024)

    for 120 epochs with weight decay set to 0 .00001, a base learning rate of 0 .005 and a Cosine-decay scheduler with 5 warmup epochs. We use the same ‘mae + cosine similarity’ loss (Cant¨ urk et al., 2024). The P/SE targets exactly correspond to those chosen in (Cant¨ urk et al., 2024), viz., 4 Laplacian eigenvectors (taken in their absolute value) along wi...

    work page 2024

  6. [14]

    in-dataset

    (ii). Our preliminary results from downstream evaluation on molpcba are reported in Figure 10 and indeed showcase the inher- ent difficulty of the task. Training ratio 0.014 corresponds to using ≈ 5, 000 training graphs, and is then comparable with the 0 .5 ratio in zinc and peptides. For this amount of data, however, the generalization performance is rel...

    work page 2024

  7. [128]

    In accordance with Cant¨ urk et al

    No dropout or weight-decay regularization is applied. In accordance with Cant¨ urk et al. (2024), dropout is applied before and after component (d). We do not perform tuning of dropout rates, but rather apply the same exact values chosen by the authors in (Cant¨ urk et al.,

    work page 2024

  8. [2015]

    One for All: Towards Training One Graph Model for All Classification Tasks, December 2023a

    Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for All: Towards Training One Graph Model for All Classification Tasks, December 2023a. URL http://arxiv.org/abs/2310.00149. arXiv:2310.00149 [cs]. Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip ...

    arXiv

Show all 12 references
  1. [2017]

    Semih Cant¨ urk, Renming Liu, Olivier Lapointe-Gagn´ e, Vincent L´ etourneau, Guy Wolf, Dominique Beaini, and Ladislav Ramp´ aˇ sek

    URL https://arxiv.org/abs/1711.07553. Semih Cant¨ urk, Renming Liu, Olivier Lapointe-Gagn´ e, Vincent L´ etourneau, Guy Wolf, Dominique Beaini, and Ladislav Ramp´ aˇ sek. Graph positional and structural encoder. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian We...

    arXiv

  2. [2019]

    URL https://arxiv.org/abs/1905. 11460. Giorgos Bouritsas, Fabrizio Frasca, Stefanos P. Zafeiriou, and Michael Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transac- tions on Pattern Analysis and Machine Intelligence ,

    1905

  3. [2020]

    Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bres- son

    URL https: //arxiv.org/abs/2003.00982. Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bres- son. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations , 2022a. Vij...

    2003 arXiv

  4. [2023]

    arXiv:2305.12600 [cs]

    URL http://arxiv.org/abs/2305.12600. arXiv:2305.12600 [cs]. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations ,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.