{"id":"d54cb596-c9ab-4473-b9e1-080a9d3bd31a","arxiv_id":"2509.01541","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Graph contrastive learning's advantage over untrained and handcrafted baselines is size-dependent: baselines win on small datasets, GCL wins modestly past a few thousand graphs, then plateaus.","lead":"Graph contrastive learning on graphs is often compared only to other trained models; this paper asks whether it beats much simpler baselines. It finds that on small graphs, untrained networks and handcrafted statistics match GCL, while on larger molecular datasets GCL only takes a modest lead after a few thousand examples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed 100 epochs across all subsample fractions confounds dataset size with training compute; the small-scale GCL deficit may be an underfitting artifact.","rationale":"The central claim of the paper is that GCL's advantage over untrained/handcrafted baselines depends on dataset size, with a crossover around 4k graphs on OGBG-MOLHIV. The reader's weakest assumption concerned hyperparameters tuned only at 50% and applied to all fractions, which is a valid but secondary issue. I identify a more load-bearing confound: the fixed epoch count with constant batch size means the number of gradient updates scales with the subsample fraction. At the smallest fractions, GraphCL receives orders of magnitude fewer updates than at the largest fraction, so the reported underperformance could be caused by underfitting rather than by dataset size per se. This directly threatens the crossover estimate and the qualitative conclusion about dataset-size dependence. The fix is straightforward: control for total training compute by matching update counts across fractions. If the crossover persists under a constant update budget, the claim would be substantially strengthened; if it shifts or vanishes, the paper's main finding needs revision. The reader's concern about hyperparameter transfer is related but distinct; both fall under 'training configuration confounds', hence partial agreement. The overall verdict remains CONDITIONAL because the issue is testable and the paper can be revised, but the current evidence does not adequately support the central scaling claim.","tokens_in":12649,"tokens_out":9523,"duration_ms":108230,"concrete_test":"Rerun the OGBG-MOLHIV scaling experiment at fractions {1, 2, 5, 10, 20, 50, 100}% fixing the total number of optimizer updates rather than the number of epochs (e.g., compute the updates used at 100% and train each smaller fraction for enough epochs to match that update count, or train all fractions for the same fixed update budget while varying epochs). If the small-fraction GraphCL ROC-AUC improves and the crossover shifts below ~2k graphs or disappears, the reported dataset-size crossover is an artifact of uneven training compute.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the OGBG-MOLHIV scaling experiment (Section 3, App. D, Table 4), GraphCL is trained for 100 epochs at every subsample fraction with a fixed batch size of 256. Since the number of optimizer updates per epoch scales with dataset size, the total training compute varies drastically across fractions: at 1% (~411 graphs) there are ~2 batches per epoch → ~200 total updates; at 100% (~41k graphs) there are ~161 batches per epoch → ~16,100 updates. The untrained GINE baseline receives zero updates. Thus the comparison at small fractions conflates dataset size with the amount of contrastive training: the observed 'GCL lags at small scales' could simply reflect that GraphCL has been grossly undertrained, not that small datasets intrinsically disadvantage GCL. The crossover near 4k graphs corresponds to only ~1,700 updates, which may be the point where the model has seen enough gradients to learn useful representations. This confound is more fundamental than the hyperparameter-transfer issue the Reader raised: even if hyperparameters were re-tuned per fraction, the severely limited update count at low data would still suppress GraphCL's performance. The paper does not control for training budget or discuss this limitation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates graph contrastive learning (GCL) against untrained and handcrafted baselines across three settings: seven TU benchmark datasets, a subsampled OGBG-MOLHIV scaling experiment, and a synthetic motif dataset with controlled difficulty. On TU datasets, simple baselines (untrained GIN, MolFingerprint, handcrafted degree statistics) are competitive with or superior to InfoGraph, GraphCL, JOAO, and SimGRACE under the standard embedding-SVM protocol. On OGBG-MOLHIV, GraphCL trained with a fixed 100-epoch budget and hyperparameters tuned at 50% data lags below an untrained GINE and handcrafted statistics at small training fractions, crosses over around 4k graphs, and then plateaus at roughly 2% ROC-AUC gain. On synthetic data, both GCL and untrained encoders improve approximately logarithmically with dataset size while handcrafted statistics saturate, with difficulty-dependent gaps. The paper concludes that dataset size is a critical axis for GCL evaluation.","tokens_in":12869,"tokens_out":6204,"duration_ms":73188,"significance":"If correct, the paper's central negative result—small benchmarks and even moderately sized molecular datasets do not reliably demonstrate GCL's advantage—would be a useful corrective to evaluation practice. The manuscript has concrete strengths: the TU comparison is reported in a full table with multiple seeds and ablations; the handcrafted baseline is simple and transparent; the synthetic generator allows controlled variation of difficulty and size; and the authors quote prior numbers where protocols match. The crossover and plateau claims are falsifiable and, if they survive robustness checks, would be an important contribution. However, the OGBG-MOLHIV scaling experiment is built on two protocol choices—fixed epoch count and hyperparameters transferred from the 50% fraction—that both act against GraphCL at small scales. Because these choices are not tested, the main quantitative claim is currently under-supported.","major_comments":[{"comment":"Fixed EPOCHS=100 and BATCH=256 across all subsample fractions confound dataset size with the number of contrastive updates. At 1% (~411 graphs) there are ~2 batches per epoch, so ~200 total updates; at 100% (~41k graphs) there are ~16,100 updates, while the untrained baseline receives zero updates. The observed small-scale GCL deficit and the crossover near 4k graphs may therefore reflect underfitting from an extremely small optimization budget rather than a property of dataset size itself. This confound is not acknowledged. Please add a control that trains GraphCL for a fixed number of total updates across fractions (e.g., by scaling epochs inversely with fraction), and ideally report results for two or three training budgets, to show whether the crossover persists.","section":"Section 3, 'Scaling setup'; App. D"},{"comment":"Hyperparameters (augmentation strategy, hidden dimension, learning rate) were selected by grid search on a 50% subset and then applied unchanged to all fractions from 1% to 100%. Optimal hyperparameters in the low-data regime are likely different (e.g., different capacity, regularization, or augmentation drop rates), so this transfer assumption can bias small fractions against GraphCL. The crossover location is directly sensitive to this choice. Please run per-fraction tuning (even a coarse version) or at least a sensitivity ablation at the 1%, 5%, and 20% fractions around the Table 4 settings, and report whether the crossover shifts.","section":"Section 3, App. D, Table 4"},{"comment":"The claim that 'GCL accuracy approximately scales with the logarithm of the number of graphs' is asserted from visual inspection of four points per condition. No slopes, standard errors, or model-comparison statistics are reported. If this scaling statement is a main conclusion, it needs a quantitative fit (e.g., accuracy vs. log N with confidence bands) or should be explicitly softened to 'roughly linear over the tested range.' As written, the abstract's logarithmic-scaling claim is stronger than the evidence presented.","section":"Section 4, Fig. 3; Conclusion"}],"minor_comments":[{"comment":"OGBG-MOLHIV is capitalized inconsistently; the abstract uses 'ogbg-molhiv'. Please standardize.","section":"Throughout"},{"comment":"Typo: 'samef-subset' should be 'same f-subset'.","section":"Section 3, Scaling setup"},{"comment":"For un-attributed TU datasets (COLLAB, IMDB-BINARY, REDDIT-BINARY), please specify how node features are constructed for the GIN encoder and MolFingerprint (e.g., degree features or constant vectors). This affects reproducibility of the untrained baselines.","section":"App. C.1"},{"comment":"The abbreviation RDT-B in the table header is undefined; spell out REDDIT-BINARY or define it in the caption.","section":"Table 2"},{"comment":"The text says hidden dimension is searched over {32, 64}; Table 4 lists HIDDEN=32 and PROJ=64. Clarify whether the projection dimension was also searched and whether Table 4 reports the final choice for both.","section":"App. D, Table 4"},{"comment":"The right-panel labels 'Δ vs baseline' and 'Δ vs handcrafted' are clear, but the x-axis tick values (300, 1000, 5000, 30000) omit the 1% and 2% points; consider marking all sampled fractions for transparency.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely question and the TU-dataset benchmarking is solid. The main risk is the OGBG-MOLHIV scaling experiment: the fixed-epoch and hyperparameter-transfer choices both disadvantage GraphCL at small scales, so the crossover may be an artifact. I would be willing to accept after the authors add the proposed control experiments or explicitly reframe the claim as applying only under this fixed-budget protocol. The synthetic-dataset section also needs a quantitative treatment of the logarithmic-scaling claim. No concerns about novelty or citation practices beyond the standard expectation that related work on GCL evaluation is adequately cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the TU result—untrained and handcrafted baselines rival GCL on small molecule and social datasets—is solid and worth having on the record. The more interesting claim, a dataset-size crossover on OGBG-MOLHIV, is plausible but undercut by a training-compute confound that the stress-test note gets right. Worth sending to review, but the scaling section needs real work before the crossover is believable.\n\nWhat's genuinely new: the systematic scaling analysis with subsample fractions, the handcrafted statistics baseline ablations, and the synthetic-data difficulty sweep. That's a legitimate extension of prior work by Errica et al. and Trivedi et al. The paper is careful to label the synthetic generator a sandbox, and the detailed tables in App. C let you check the TU claims. I appreciate that they report seeds and variance.\n\nWhere it gets soft. First, the fixed 100 epochs across all fractions. At 1% of OGBG-MOLHIV (~411 training graphs) with batch size 256, GraphCL sees about 200 optimizer updates; at 100% it sees about 16,100. The untrained GINE baseline receives zero updates. So \"GCL lags at small scales\" could just be \"GCL has not been trained enough.\" The crossover near 4k graphs may be where the model finally gets enough gradient steps. This is a load-bearing problem for the central result, and simply re-tuning hyperparameters per fraction won't fix it. You need to control the training budget (e.g., match total updates or use steps-based scheduling) or show the result is robust to it.\n\nSecond, the \"approximately logarithmic scaling\" claim in the abstract outruns the evidence: four synthetic dataset sizes, no fitted curve, and the conclusion itself hedges. That's a minor overreach if framed as a suggestion, but it should be reworded.\n\nOther issues are smaller: hyperparameters tuned only at 50% and transferred to all fractions; some TU GCL numbers quoted from the literature instead of rerun; no confidence intervals on crossover locations; no code yet; an internal inconsistency on projection dimension (App. D says 32, Table 4 says 64). None of these is fatal.\n\nBottom line: this is a useful paper for people who build GCL benchmarks. The TU analysis alone is a decent contribution. But the scaling crossover is not yet demonstrated, and the training-budget confound is the reason. I would not desk-reject it; I would send it to review with a request for a major revision that addresses the confound, and perhaps a per-fraction tuning sensitivity check. If they can fix that, the crossover claim becomes interesting rather than just plausible.","headline":"Useful TU benchmark study, but the OGBG-MOLHIV crossover claim is undercut by a training-compute confound that needs fixing before the scaling result can be trusted.","tokens_in":13419,"tokens_out":2893,"would_cite":false,"duration_ms":33052,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On ogbg-molhiv, GraphCL trails untrained GNN and handcrafted baselines below roughly 4,000 training graphs, then overtakes them by about 2% ROC-AUC before its advantage plateaus.","keywords":["graph contrastive learning","untrained baselines","dataset size","scaling behavior","graph classification","ogbg-molhiv","synthetic graph datasets","self-supervised learning"],"falsifier":"Retune GraphCL's hyperparameters independently at each subsample fraction of ogbg-molhiv, for example with a per-fraction random search, and compare the crossover against the fixed-hyperparameter curve; a substantially different or absent crossover would show that the dataset-size effect is partly an artifact of tuning. A second check is to repeat the scaling experiment on another large molecular graph dataset with rich node features: if untrained GNNs stay competitive at large scale there, the ~2% advantage is dataset-specific rather than a general property of contrastive learning.","tokens_in":12462,"feed_emoji":"📈","tokens_out":6709,"duration_ms":71303,"temperature":0.7,"pith_summary":"The paper asks whether graph contrastive learning (GCL) actually improves over doing nothing: randomly initialized graph encoders, fixed handcrafted graph statistics, or simple neural nets. It argues the answer depends on dataset size. On small TU benchmarks, these untrained or handcrafted baselines often match or beat GCL. On the much larger ogbg-molhiv dataset, GraphCL trails both baselines below roughly 4,000 training graphs, then overtakes them by about 2% ROC-AUC, with the gap plateauing rather than widening as data grows. The paper concludes that dataset size must be a standard axis of GCL evaluation and that small benchmarks are not enough to demonstrate a contrastive advantage.","feed_headline":"Untrained graph models beat contrastive learning below 4,000 graphs","feed_subtitle":"On the molecular benchmark ogbg-molhiv, GraphCL's lead over untrained baselines is ~2% ROC-AUC and plateaus.","key_machinery":"The load-bearing setup is the embedding–probe evaluation pipeline: a GNN encoder is pretrained contrastively (or left untrained), frozen, and its graph embeddings are fed to a linear probe. This is paired with systematic subsampling of the training set at fractions from 1% to 100% on ogbg-molhiv, plus a handcrafted-statistics baseline built from node count, mean degree, and a degree histogram. On synthetic data, the paper uses a generator that attaches label-defining motif graphs to random background trees, with a style multiplier controlling task difficulty and dataset size varied independently. Together these components isolate how much of GCL's advantage comes from contrastive pretraining","core_discovery":"The paper's central result is a crossover in relative performance. On ogbg-molhiv, using the official scaffold split, GraphCL pretrained on a training subset performs below both an untrained GINE encoder and the handcrafted-statistics baseline when the training set has up to a few thousand graphs; beyond about 4,000–4,500 graphs GraphCL pulls ahead, but only by roughly 2% ROC-AUC, and this margin stops growing even when training data is increased to the full ~33,000 graphs. On seven TU datasets, the same untrained GNN, a MolFingerprint MLP, and handcrafted statistics rival or beat several established GCL methods, with every GCL method losing on at least one dataset. On synthetic motif-vs-bac","pith_inferences":["Because GraphCL's hyperparameters were selected on a 50% training fraction and then reused at all fractions, the exact ~4k crossover is likely tunable; per-fraction tuning could shrink GCL's early disadvantage or move the threshold.","The crossover location probably depends on how informative node features are, not just dataset size: untrained GNNs are much weaker on synthetic constant-feature graphs than on molecular graphs with rich attributes, so richer features may push crossovers to larger N.","A practical evaluation protocol would plot accuracy-versus-data curves against untrained and handcrafted baselines and report the crossover point, rather than single accuracy numbers; the paper's data support this protocol even though the authors only frame it as informative.","If log-linear scaling continues on larger datasets, GCL's absolute gain over untrained encoders may remain bounded on tasks where untrained features also improve; methods that break this log scaling would be the ones that matter."],"forward_implications":["Below a few thousand graphs, GCL gains can be reproduced or beaten by an untrained GNN or handcrafted statistics, so small benchmarks should include such baselines before claiming contrastive success.","On ogbg-molhiv, adding data beyond the crossover does not widen GCL's advantage; the gap plateaus at about 2% ROC-AUC, so data scaling alone is not sufficient to justify GCL.","The same encoder architecture, evaluated with an embedding probe, makes the untrained GNN a strong random-feature baseline at scale, since its probe can exploit more labels.","Dataset size should be reported and varied when comparing GCL methods; crossover points are dataset-dependent but non-trivial relative to standard TU dataset sizes.","GCL algorithms need designs that avoid performance plateaus at large scale."],"supporting_citations":[{"why":"Supplies GraphCL, the representative contrastive method used in the ogbg-molhiv scaling study and synthetic experiments.","marker":"[21]"},{"why":"Supplies InfoGraph and the embedding–probe evaluation protocol used for all GCL methods and baselines.","marker":"[24]"},{"why":"Supplies the ogbg-molhiv dataset, scaffold split, and evaluation guidelines used for the scaling experiments.","marker":"[15]"},{"why":"Provides the MoleculeNet source and description of the molecular dataset used in the study.","marker":"[16]"},{"why":"Supplies the synthetic graph generator with motif content and style multiplier that the paper adapts for its scaling and difficulty sweeps.","marker":"[17]"},{"why":"Supplies the MolFingerprint baseline and the case for simple baselines in graph classification evaluation.","marker":"[8]"},{"why":"Supplies evidence that simple graph statistics correlate with labels on TU datasets, used to explain why handcrafted baselines are competitive there.","marker":"[26]"},{"why":"Supplies the empirical GCL setup and the train-embeddings logistic-regression probe used on ogbg-molhiv.","marker":"[27]"},{"why":"Supplies the GIN architecture used as the encoder backbone for both GCL methods and the untrained GNN baseline.","marker":"[25]"},{"why":"Supplies the edge-aware GINE encoder used as GraphCL's backbone on ogbg-molhiv and as the untrained baseline.","marker":"[28]"}],"fun_headline_variants":["GraphCL trails untrained GNNs below 4k graphs","Crossover at 4k graphs for GCL vs untrained baselines","Untrained baselines beat GCL on small datasets","GCL advantage emerges only after 4,000 training graphs","Graph contrastive learning needs >4k graphs to win"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The crossover result rests on the assumption that GraphCL's hyperparameters, chosen once on half the training data, work equally well for every smaller subsample; if those settings are not appropriate at small scale, the point where GCL overtakes the baselines could shift or disappear.","fun_headline_variants_meta":{"raw":{"variants":["GraphCL trails untrained GNNs below 4k graphs","Crossover at 4k graphs for GCL vs untrained baselines","Untrained baselines beat GCL on small datasets","GCL advantage emerges only after 4,000 training graphs","Graph contrastive learning needs >4k graphs to win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000596,"raw_usage":{"total_tokens":2627,"prompt_tokens":746,"completion_tokens":1881,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":1794}},"tokens_in":490,"tokens_out":1881,"duration_ms":15976,"temperature":1.0,"reasoning_tokens":1794,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:25:43.769612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retune GraphCL's hyperparameters independently at each subsample fraction of ogbg-molhiv, for example with a per-fraction random search, and compare the crossover against the fixed-hyperparameter curve; a substantially different or absent crossover would show that the dataset-size effect is partly an artifact of tuning. A second check is to repeat the scaling experiment on another large molecular graph dataset with rich node features: if untrained GNNs stay competitive at large scale there, the ~2% advantage is dataset-specific rather than a general property of contrastive learning.","supporting_citations":[{"cited_title":"Graph contrastive learning with augmentations","cited_arxiv_id":null,"evidence_quote":"Supplies GraphCL, the representative contrastive method used in the ogbg-molhiv scaling study and synthetic experiments."},{"cited_title":"Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization","cited_arxiv_id":null,"evidence_quote":"Supplies InfoGraph and the embedding–probe evaluation protocol used for all GCL methods and baselines."},{"cited_title":"Open graph benchmark: Datasets for machine learning on graphs","cited_arxiv_id":null,"evidence_quote":"Supplies the ogbg-molhiv dataset, scaffold split, and evaluation guidelines used for the scaling experiments."},{"cited_title":"Thiagara- jan","cited_arxiv_id":null,"evidence_quote":"Supplies the synthetic graph generator with motif content and style multiplier that the paper adapts for its scaling and difficulty sweeps."},{"cited_title":"A fair comparison of graph neural networks for graph classification","cited_arxiv_id":null,"evidence_quote":"Supplies the MolFingerprint baseline and the case for simple baselines in graph classification evaluation."},{"cited_title":"Rethinking the effective- ness of graph classification datasets in benchmarks for assessing gnns","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that simple graph statistics correlate with labels on TU datasets, used to explain why handcrafted baselines are competitive there."},{"cited_title":"An empirical study of graph contrastive learning","cited_arxiv_id":null,"evidence_quote":"Supplies the empirical GCL setup and the train-embeddings logistic-regression probe used on ogbg-molhiv."},{"cited_title":"How powerful are graph neural networks? In International Conference on Learning Representations, 2019","cited_arxiv_id":null,"evidence_quote":"Supplies the GIN architecture used as the encoder backbone for both GCL methods and the untrained GNN baseline."},{"cited_title":"Strategies for pre-training graph neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the edge-aware GINE encoder used as GraphCL's backbone on ogbg-molhiv and as the untrained baseline."}],"review_version":1}