{"id":"cf58c3f5-dca1-4438-8feb-796454989a89","arxiv_id":"2412.17609","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pretrained GNN embeddings transfer across molecular graphs only under certain data-regime and feature-similarity conditions, and a proposed feature-structuralization method does not consistently improve transfer.","lead":"This paper tests whether graph neural networks pretrained on one or more graph datasets can be reused for other graph tasks. It finds that pretrained features help mainly when the downstream dataset is large enough, and that adding feature information requires similar feature spaces between source and target.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'only with enough downstream data' claim is confounded by fixed dropout rates reused from full-data settings; low-data harm may be a regularization artifact rather than a property of pretrained embeddings.","rationale":"The paper's main message is not merely that transfer can work, but about when it works: it claims a data-regime threshold and a dependence on feature-space similarity. The data-regime threshold is the abstract's first and most load-bearing claim. The reader's cross-domain generalizability concern is fair, but I find a more direct threat to the claim as tested: the pretrained embeddings are passed through a linear map with dropout fixed per dataset from full-data GPSE experiments. Since the data-regime comparison is precisely about changing training set size, regularization optimized at full data may not transfer to low-data settings, and the non-monotonic peptides results make the 'only with enough data' phrasing especially fragile. The paper itself flags this confound in Appendix B.2, but the abstract states the conclusion without that caveat. A per-regime dropout grid search would settle whether the effect is genuine. I do not see internal inconsistency or fabrication; the concern is an unaddressed confound, which is exactly why CONDITIONAL rather than REJECT is appropriate. The reader's verdict already reflects the need for revisions, so the verdict should remain unchanged.","tokens_in":9438,"tokens_out":4162,"duration_ms":38456,"concrete_test":"Run the peptides and zinc downstream evaluations at training fractions 0.01, 0.1, 0.25 and 0.5 with a small grid over dropout before component (d) in {0.0, 0.1, 0.3, 0.5} and after component (d) in {0.0, 0.1}, selecting hyperparameters on a validation split identically for pretrained and baseline models, and averaging over at least 3 seeds. If at 0.1 or 0.25 some dropout setting makes the pretrained model match or beat the baseline, the 'only with enough downstream data' claim is a regularization artifact. Report baseline and pretrained parameter counts to rule out capacity differences.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in the abstract — 'embeddings from pretrained models improve generalization only with enough downstream data points' — rests on comparing pretrained and baseline downstream GNNs under a fixed dropout configuration per dataset. Appendix B.2 states: 'We do not perform tuning of dropout rates, but rather apply the same exact values chosen by the authors in (Cantürk et al., 2024) for the most prominent models in the respective full-dataset adaptation experiments.' For zinc the dropout is 0.5 before component (d) and none after; for peptides it is 0.1; for molpcba it is 0.3 before and 0.1 after. In small-data regimes the optimal regularization is typically different from the full-data optimum, so the finding that pretrained embeddings 'can be detrimental in data-scarce settings' (Section 4(i)) may be an artifact of over- or under-regularizing the embedding pathway. The paper's own non-monotonic evidence supports this worry: on peptides, Appendix D reports that ratio 0.01 is beneficial, 0.1 and 0.25 are harmful, and 0.5 is beneficial, which is not the simple 'only with enough data' pattern stated in the abstract. The authors acknowledge the unresolved confound: 'We reserve to tune these values ... verify whether pretrained representations can be more beneficial in lower data regimes if differently regularized.' Until dropout is tuned per data regime, and preferably with multiple seeds, the headline data-regime conclusion is not established; the abstract overstates it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether pretrained graph neural networks transfer across graph datasets under a feature-agnostic structural-pretraining framework. It builds on the GPSE pipeline of Cantürk et al. (2024), proposes Feature-Structuralization, which converts categorical node features into auxiliary feature-nodes, and evaluates pretrained embeddings on downstream tasks with varying training fractions across ZINC-12k, ogbg-molpcba, and peptides-func. The main empirical claims are that (i) pretrained embeddings improve generalization only with sufficiently large downstream training sets and can hurt in data-scarce regimes, (ii) multi-dataset pretraining generally does not degrade and can improve transfer, and (iii) feature information via structuralization helps only when pretraining and downstream feature spaces are similar. The paper is explicitly preliminary, with molpcba results deferred to Appendix D and an acknowledged unresolved confound in Appendix B.2.","tokens_in":9707,"tokens_out":2736,"duration_ms":27255,"significance":"If the reported conclusions hold, they contribute useful negative and positive evidence for the feasibility of graph foundation models: structural-only pretraining can transfer across datasets, but the benefit is moderate and regime-dependent. The paper's strengths are its honest reporting of mixed results, its systematic ablations over pretraining corpora (single, pairwise, and multi-dataset), and its additional analysis of pretraining-data scale in Appendix E. The feature-structuralization idea is simple and clearly described. However, the paper's headline claims currently outrun the evidence: the data-regime conclusion rests on a fixed-dropout protocol that is explicitly untuned per downstream data size, and the empirical basis is limited to three molecular datasets, with the molpcba results openly preliminary. These issues make the central claims defensible but not yet established.","major_comments":[{"comment":"The central claim in the abstract and Section 4(i) that pretrained embeddings 'improve generalization only with enough downstream data points' is confounded by the fixed dropout protocol. Appendix B.2 states that dropout rates are not tuned per experiment but taken from Cantürk et al. (2024) full-data settings: 0.5 before component (d) for zinc, 0.1 for peptides, and 0.3/0.1 for molpcba. In low-data regimes the optimal regularization strength generally differs from the full-data optimum, so the observed 'detrimental in data-scarce settings' could be an artifact of over- or under-regularizing the embedding pathway rather than a property of pretrained representations. This concern is substantiated by the paper's own non-monotonic results in Appendix D: on peptides, ratio 0.01 is beneficial, ratios 0.1 and 0.25 are harmful, and ratio 0.5 is again beneficial—a pattern that does not match the simple 'only with enough data' statement. The authors acknowledge the unresolved confound in Appendix B.2 ('We reserve to tune these values...'). To establish the headline data-regime conclusion, the downstream experiments need per-regime dropout tuning and, ideally, multiple seeds.","section":"Section 4(i), Appendix B.2"},{"comment":"The paper's conclusions are framed broadly about cross-dataset transfer of pretrained GNNs, but the evidence is drawn exclusively from three molecular datasets (ZINC-12k, ogbg-molpcba, peptides-func), all of which represent molecules or amino-acid chains. The conclusions in Section 5 and the abstract do not restrict the scope to molecular graphs, yet transfer behavior on non-molecular domains (e.g., social or collaboration networks, which differ in feature semantics and structure) could differ substantially. This is a load-bearing limitation because the paper's contribution is explicitly toward graph foundation models in general. At minimum, the claims should be rephrased to the molecular domain, or a non-molecular dataset should be added to support the broader framing.","section":"Section 3 and Conclusions"},{"comment":"The molpcba downstream results are explicitly preliminary—the paper states 'we believe that more extensive experimentation is due on molpcba'—yet these results are used in support of observations (ii) and (iii) in Section 4, which concern the influence of pretraining-corpus composition and feature information. Only a few training ratios are reported for molpcba, and the paper notes that even ratio 0.014 (about 5,000 graphs) yields test AP below 0.06. Because molpcba is the only dataset with the same 9 SMILES-derived feature categories as peptides, it plays a central role in the structuralization and feature-similarity claims. Without a more complete molpcba evaluation, the feature-similarity conclusion in the abstract ('requires some similarities between pretraining and downstream feature spaces') is not fully supported by the presented evidence.","section":"Appendix D, Section 4"}],"minor_comments":[{"comment":"Typo: 'Architecure' should be 'Architecture'.","section":"Figure 4 caption"},{"comment":"The notation is a little confusing: Section 2 uses 'd' to denote the number of categorical feature channels, while Figure 4 uses '(d)' for a linear projection component. Please disambiguate to avoid confusion.","section":"Section 2 and Figure 1"},{"comment":"The clipping of negative R2 values to −1 and the use of 'black' for values below −1 is not explained in the main text; please add a remark in a caption or the text so that readers understand the color scale.","section":"Appendix C, Figures 6 and 7"},{"comment":"The downstream evaluation pipeline mentions 'No dropout or weight-decay regularization is applied' to the downstream GNN, but dropout is applied before/after component (d). Clarify whether the reported dropout values are the only regularization used in the downstream model.","section":"Appendix B.2"},{"comment":"The conclusions omit any mention of the unresolved dropout confound, even though it is acknowledged in Appendix B.2. A sentence acknowledging this caveat in the main text would improve the paper's robustness.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written and honest extended abstract, but the title and abstract promise more than the current experiments deliver. The dropout confound directly affects the paper's headline data-regime claim, and the molpcba results are too preliminary to support the feature-similarity conclusion. I recommend major revision rather than rejection because the questions are important and the existing evidence, while incomplete, does not contradict the authors' interpretations—it simply does not yet establish them. Adding per-regime dropout tuning and at least one non-molecular dataset would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful workshop-level empirical paper. The new piece is Feature-Structuralization: encoding categorical node features as auxiliary feature-nodes so a feature-agnostic pretraining pipeline can still see feature information. That construction is novel relative to GPSE and the alternatives in Appendix A, and the multi-dataset pretraining analysis on GPSE is new too. The paper reports mostly honest, mixed results instead of a clean win, which I credit.\n\nWhat it does well: the comparison structure is clear (12 pretraining combinations, three datasets, several training fractions), and the authors flag their own limitations rather than hiding them. The observation that off-dataset pretraining can be competitive with in-dataset pretraining, and that structuralization helps when feature spaces align but hurts otherwise, is useful for people building graph foundation models.\n\nThe soft spot is the headline. The abstract says pretrained embeddings improve generalization 'only with enough downstream data points.' That is not what the data show. The benefit is non-monotonic in data: on peptides, embeddings help at the 0.01 ratio, hurt at 0.1 and 0.25, and help again at 0.5. The more likely explanation, which the authors themselves raise in Appendix B.2, is that dropout rates are fixed at the values from the full-data GPSE experiments. In low-data regimes the optimal regularization is different, so 'detrimental in data-scarce settings' may be a regularization artifact rather than a property of pretrained embeddings. The paper explicitly reserves tuning this, which is honest, but then the abstract should not present the data-regime finding as established.\n\nThe other issues are minor relative to that. molpcba downstream results are clearly labeled preliminary, and for a workshop paper that is acceptable. No code or data is released, and seeds are not described; that hurts reproducibility and matters more because the claims are empirical. Structuralization is a simple idea, but I don't think simplicity is a flaw; the paper positions it as a first step.\n\nWho should read this: researchers working on graph transfer and foundation models. The structuralization trick is easy to adopt, and the negative results are worth knowing before you assume pretrained embeddings always help.\n\nRecommendation: send it to review. It is not a strong full paper yet, but as an extended abstract it deserves referee time. If it grows into a full paper, I'd want dropout tuned per data regime, a few seeds, code release, and an abstract that matches the actual, non-monotonic findings.","headline":"Honest and useful preliminary study, but the abstract's data-regime claim is not established because dropout is fixed across data sizes.","tokens_in":10278,"tokens_out":2756,"would_cite":true,"duration_ms":26525,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretrained graph embeddings help downstream tasks only when enough labels are available and pretraining data and feature spaces align.","keywords":["Graph Neural Networks","Foundation Models","Transfer Learning","Feature-Structuralization","Graph Pretraining","Cross-Dataset Transfer","Positional and Structural Encodings","Low-Data Regime"],"falsifier":"Run the same pretraining and downstream protocol on a non-molecular graph collection whose categorical feature vocabularies do not overlap between source and target; if pretrained structural embeddings improve test performance at small downstream training fractions, or if Feature-Structuralization consistently beats vanilla structural pretraining when features are off-distribution, the paper's conditional conclusions would be overturned.","tokens_in":9225,"feed_emoji":"🧪","tokens_out":10118,"duration_ms":83566,"temperature":0.7,"pith_summary":"This paper asks when a graph neural network pretrained on one set of graphs can be reused on another, a question that stands between current graph learning and graph foundation models. Using a feature-agnostic structural pretraining scheme that predicts positional and structural encodings, the authors test transfer across three molecular datasets with different structures and feature vocabularies. The experiments indicate that pretrained embeddings are not a free lunch: they beat a non-pretrained baseline only once enough downstream labels are available, and their benefit depends on the size and composition of the pretraining corpus. The paper also proposes Feature-Structuralization, which rewrites categorical node features as extra nodes and edges so the pretrained model stays feature-agnostic while still seeing feature information, but this does not produce a consistent gain and underperforms when pretraining and downstream feature vocabularies differ. The conclusions are explicitly conditional, resting mainly on molecular graphs, with molpcba downstream results flagged as preliminary.","feed_headline":"Pretrained GNNs beat plain models only with enough downstream data","feed_subtitle":"On three molecular datasets, corpus mix and feature overlap decide when pretraining beats a plain baseline.","key_machinery":"The load-bearing object is the structural pretraining pipeline: a GNN is pretrained on featureless graphs to predict graph positional and structural encodings (P/SEs) such as Laplacian eigenvectors and eigenvalues, electrostatic potentials, random-walk and heat-kernel structural encodings. The paper's proposed addition, Feature-Structuralization, converts categorical node features into auxiliary feature-nodes, one per category per channel, connected to the nodes carrying that category, and then discards explicit features so the input remains feature-agnostic; the same P/SE prediction targets are computed on original nodes of both the original and the structuralized graph. This construction carries the argument because it is the mechanism that lets feature information enter pretraining without dataset-specific encodings, and the experimental comparisons use it to isolate when added structure helps or hurts transfer.","core_discovery":"The paper's central discovery is that cross-dataset transfer of pretrained GNNs is real but conditional. In downstream evaluations on zinc and peptides, embeddings from pretrained models improve generalization over a non-pretrained baseline once there is sufficient labeled data, but can hurt in very scarce-data settings; on peptides the pattern is non-monotonic, with pretrained embeddings harmful at a 0.1 training fraction yet helpful again at 0.01. The composition of the pretraining corpus matters: including datasets beyond the target rarely degrades performance, and mixtures that exclude the target can match or slightly exceed target-only pretraining, an effect the authors attribute mostly to larger pretraining corpora. Feature-Structuralization, which materializes each categorical feature channel as feature-nodes connected to the original nodes carrying that category, lets the pretraining access feature information without dataset-specific encodings, but its downstream gains are inconsistent: it helps with in-dataset pretraining and when structure is off-distribution while features are shared, and is weakest when feature vocabularies are off-distribution. Multi-dataset pretraining generally improves prediction of the structural targets themselves, yet models never fully close the gap between in-domain and off-domain pretraining, and extra pretraining data closes that gap far less efficiently than in-domain data.","pith_inferences":["The paper does not test whether fuller molpcba pretraining changes the downstream data threshold; one plausible extension is that larger pretraining corpora lower the threshold but do not eliminate it, because off-domain transfer gains in Appendix E remain partial.","The U-shaped low-data behavior on peptides suggests the pretrained encoder works as a fixed feature extractor whose usefulness depends on how well the small downstream head can exploit it; an untested consequence is that freezing more downstream layers or adding regularization could push the beneficial regime toward smaller data budgets.","Structuralization's failure when feature vocabularies are off-distribution hints that the pretraining objective may let the model rely on dataset-specific feature-structure correlations; a testable remedy would be a pretraining loss that explicitly penalizes such correlations or an evaluation that removes shared feature categories."],"forward_implications":["Pretrained structural embeddings should not be expected to rescue extreme low-data regimes; on peptides at a 0.01 training fraction they help, but at 0.1 they can hurt, so the value of pretraining depends on the downstream data budget in a nontrivial way.","Pretraining corpus design is a usable lever: adding datasets other than the target generally does not degrade transfer and can match or beat target-only pretraining, so larger mixed corpora are a reasonable route toward graph foundation models.","Feature-Structuralization is not a reliable way to add feature information: its benefit depends on overlap between pretraining and downstream feature vocabularies, so feature-agnostic pretraining that natively handles continuous or off-distribution features remains an open need.","Pretraining target prediction can serve as a cheap diagnostic: models pretrained on a single dataset fail near chance on structurally distant datasets such as peptides, while multi-dataset pretraining generally improves target prediction.","Scaling pretraining data alone is an inefficient substitute for in-domain data: more off-domain pretraining samples improve off-domain target prediction but much less than the same number of in-domain samples."],"supporting_citations":[{"why":"Supplies the GPSE pretraining pipeline, base architecture, and P/SE targets that all transfer experiments build on.","marker":"Cantürk et al. (2024)"},{"why":"Provides the ZINC-12k benchmark used as a pretraining and downstream dataset.","marker":"Dwivedi et al. (2020)"},{"why":"Provides the ogbg-molpcba benchmark used as the larger, harder pretraining and downstream dataset.","marker":"Hu et al. (2020)"},{"why":"Provides the peptides-func benchmark, structurally distant from small molecules, used for cross-domain transfer tests.","marker":"Dwivedi et al. (2022b)"},{"why":"Provides the 3-layer GIN architecture used as the downstream model in all evaluations.","marker":"Xu et al. (2019)"},{"why":"Provides the Residual Gated GCN message-passing layers used in the pretraining backbone.","marker":"Bresson and Laurent (2017)"},{"why":"Provides the expressivity analysis used to interpret why pretrained structural embeddings help on zinc via cyclic information.","marker":"Bouritsas et al. (2022)"}],"fun_headline_variants":["Pretrained GNNs: useful only when downstream data is plentiful","Cross-dataset GNN transfer is real, but only under the right conditions","GNN pretraining isn't a free lunch: data size and corpus matter","When does pretraining a GNN help? Only with enough labels and the right mix","Graph pretraining: conditional gains, corpus composition decides"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that three molecular datasets — zinc, molpcba, and peptides — represent cross-dataset transfer of pretrained graph networks broadly enough for the conclusions to generalize, with molpcba's downstream evidence itself preliminary.","fun_headline_variants_meta":{"raw":{"variants":["Pretrained GNNs: useful only when downstream data is plentiful","Cross-dataset GNN transfer is real, but only under the right conditions","GNN pretraining isn't a free lunch: data size and corpus matter","When does pretraining a GNN help? Only with enough labels and the right mix","Graph pretraining: conditional gains, corpus composition decides"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1702,"prompt_tokens":925,"completion_tokens":777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":681}},"tokens_in":541,"tokens_out":777,"duration_ms":7298,"temperature":1.0,"reasoning_tokens":681,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:20:34.187820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pretraining and downstream protocol on a non-molecular graph collection whose categorical feature vocabularies do not overlap between source and target; if pretrained structural embeddings improve test performance at small downstream training fractions, or if Feature-Structuralization consistently beats vanilla structural pretraining when features are off-distribution, the paper's conditional conclusions would be overturned.","supporting_citations":[],"review_version":1}