{"id":"b44b2280-8db7-4778-af4d-32ab3652c509","arxiv_id":"2412.06173","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Tuned feature-only MLPs nearly match graph neural networks on five common graph benchmarks, suggesting those benchmarks measure feature quality more than graph learning.","lead":"A well-tuned MLP that ignores graph structure matches or beats many graph neural networks on five of seven popular node-classification benchmarks. The paper argues that node features already encode graph information, and proposes new synthetic and real-world benchmarks that require the graph.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparison is not controlled: Tables 1–2 mix the authors' tuned MLP with untuned GNN numbers from prior papers, so the 'little benefit' claim on five datasets is not yet established; retuning GNNs under the same protocol would settle it.","rationale":"I read the paper as making two related claims: (1) empirically, a well-tuned feature-only MLP nearly matches graph-based methods on five common benchmark datasets, and (2) mechanistically, node features leak graph information. Claim (1) is the load-bearing one, because the paper's practical recommendation—that these datasets are weak benchmarks for graph learning—follows from it. Claim (1) rests entirely on Tables 1 and 2, where the MLP is tuned by the authors but all GNN numbers are imported from prior work without retuning. Section 7 acknowledges this, and the feature study demonstrates that retuning a GCN can change its reported accuracy. This makes the comparison uncontrolled, and the strongest graph methods in the paper's own tables already show gaps of 3–9 points over the tuned MLP, so the headline statement overstates what the data establish.\n\nI agree with the reader's weakest_assumption: the prior-paper GNN numbers may not be directly comparable to the authors' tuned MLP numbers. My proposed test would settle this by applying an identical tuning protocol to both model families on the same data splits. The paper has independent value that survives the concern: the WS1000 synthetic benchmark family is a constructive, parameter-driven demonstration that graph structure is necessary when features are uninformative, and the suggested real-world link-prediction datasets show clear MLP-GCN separation. Those contributions do not depend on the uncontrolled five-dataset comparison. Because the concern prevents unconditional acceptance but does not invalidate the paper's constructive contributions, the reader's CONDITIONAL verdict remains the right one; my read does not change it.","tokens_in":14785,"tokens_out":6804,"duration_ms":71098,"concrete_test":"For each of the five datasets (Amazon Computers, Amazon Photo, Coauthor CS, Coauthor Physics, PubMed), use the authors' DGL data versions and run the same Bayesian sweep budget used for their MLP for both GCN and one stronger graph method (e.g., Exphormer or ACM-Snowball-3), with identical train/validation/test splits, feature normalization, early stopping, and 5-trial protocol. If the tuned graph models stay within 1–2 points of the tuned MLP on all five datasets, the 'little benefit' claim holds; if they exceed the MLP by the margins in Tables 1–2, the correct conclusion is that many published GNN results are undertuned, not that these benchmarks do not need graph structure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that graph structure adds little on five of seven standard datasets—is read off Tables 1 and 2, where the tuned MLP comes from the authors' own DGL-based pipeline (Bayesian sweeps, 5 trials) but every GNN entry is copied from Liu et al. (2020), Hoang & Lee (2023), or Shirzad et al. (2023). Section 7 explicitly concedes that the authors did not tune those models and that 'some highly performant graph methods may largely derive their benefit from hyperparameter tuning.' A baseline-vs-GNN gap is a controlled measurement only if the split conventions, feature preprocessing, training/validation protocol, early stopping, and trial counts are identical. The paper does not establish this, and the cited papers may use different data versions and evaluation protocols. The authors' own feature study shows the issue is real: footnote 3 reports that their retuned GCN already exceeds the GCN numbers copied from Liu et al. (2020). Moreover, the strongest entries in the paper's own tables—Exphormer at 91.6±0.3 on Amazon Computers and ACM-Snowball-3 at 91.4±0.4 on PubMed—stand 3–9 points above the tuned MLP, so the unqualified 'one gains little benefit' sentence is in tension with the paper's data. The conclusion that these five benchmarks are weak instruments for measuring graph learning therefore depends on a controlled retuning comparison that the paper does not provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper challenges the assumption that graph structure is necessary for good performance on common node-classification and link-prediction benchmarks. The authors tune a feature-only MLP on seven standard datasets (Amazon Computers, Amazon Photo, Coauthor CS, Coauthor Physics, Cora, CiteSeer, PubMed) and report that on five of them the tuned MLP nearly closes the gap with graph-based methods, implying that these five datasets are weak instruments for measuring graph-learning progress. They support this with feature-ablation studies showing that the MLP-GCN gap closes as node features are added, and they propose new benchmarks: a family of synthetic Watts-Strogatz link-prediction datasets plus three real-world link-prediction datasets where the MLP underperforms the GCN.","tokens_in":15160,"tokens_out":2959,"duration_ms":29958,"significance":"If the central claim is established, the paper has substantial value for the graph-learning community: it would show that several widely used benchmark datasets do not require graph structure, that under-tuned MLP baselines have distorted reported progress, and that new benchmarks are needed. The paper also ships code and synthetic datasets, which is a practical strength. However, the headline claim rests on an uncontrolled comparison between the authors' heavily tuned MLP and GNN numbers copied from prior papers, and the authors themselves concede in Section 7 that those GNNs were not tuned. The paper is therefore best read as a strong motivating study rather than as a definitive measurement of the benefit of graph structure.","major_comments":[{"comment":"The central claim that graph structure adds little on five of seven datasets is read off Tables 1 and 2, but those tables compare the authors' tuned MLP (Bayesian sweeps, 5 trials) with GNN results copied from Liu et al. (2020), Hoang & Lee (2023), and Shirzad et al. (2023) without establishing that the splits, feature preprocessing, training/validation protocol, early stopping, and trial counts are identical. Section 7 explicitly states that the authors did not tune those graph models. As a controlled measurement of the benefit of graph structure, this comparison is therefore not sufficient. The issue is not hypothetical: on Amazon Computers, Exphormer (91.6±0.3) exceeds the tuned MLP (82.6±0.6) by 9.0 points, and on PubMed, ACM-Snowball-3 (91.4±0.4) exceeds the tuned MLP (86.8±0.4) by 4.6 points, so the unqualified statement that one gains little benefit is in direct tension with the paper's own tables. A retuning of the graph methods under the same pipeline, or at least a systematic verification of protocol equivalence, is required before the headline conclusion can be drawn.","section":"Section 4, Figure 1"},{"comment":"The feature study is interpreted as demonstrating that node features 'leak' graph information, but the observed closing of the MLP-GCN gap has plausible alternative explanations that are not controlled for: more features may simply make the classification task easier for both models, or may interact differently with optimization and capacity even when hyperparameters are tuned per dataset. The paper's causal claim that the benefit the GCN derives from the graph is 'also enjoyed by the MLP with a higher number of features' is an interpretation, not a measurement. The Cora negative control is useful, but it does not by itself rule out task-difficulty or optimization explanations for the Amazon Computers trend. The authors should either strengthen the causal evidence or restate the feature study as a correlation that is consistent with their hypothesis.","section":"Section 5, Table 3, Figure 2"},{"comment":"The synthetic benchmark evidence is presented as showing that the WS1000 family separates MLP and GCN, but the numbers are thin. For WS1000, the GCN achieves 54.7±0.4 ROC AUC versus the MLP's 49.1±2.2, which is a small absolute gap, and no other graph method is reported. In Figure 2, the claim that 'the MLP begins to outperform the GCN' for increasing γ compares the MLP at γ>0 against the GCN at γ=0 only; the GCN is not retuned for γ>0, so the comparison does not establish how the graph method behaves on the harder variants of the family. This weakens the proposal that the parametric family provides a controlled way to benchmark graph methods.","section":"Section 7"}],"minor_comments":[{"comment":"The abstract says 'one gains little benefit from using graph structure on five datasets,' but the data in Tables 1 and 2 show several graph methods outperforming the tuned MLP by 4-9 points on those same five datasets (e.g., Exphormer on Amazon Computers and ACM-Snowball-3 on PubMed). The wording should be qualified to 'most previously reported, untuned graph methods' rather than a blanket 'little benefit' claim.","section":"Abstract; Section 3"},{"comment":"The Coauthor CS feature study says the datasets 'share the original graph from Coauthor Physics' when it should presumably say 'Coauthor CS'; please correct this typo.","section":"Appendix A"},{"comment":"The parenthetical that MLP performance 'can be worse than random due to poor seeds' is vague and reads as an excuse rather than an explanation; it would be clearer to report the seed-dependent behavior separately or to use a variance-reduced evaluation.","section":"Table 3"},{"comment":"For the three suggested real-world link-prediction benchmarks, only the GCN is compared against the MLP; reporting at least one or two additional graph methods would make the claim that these are suitable benchmarks more robust.","section":"Section 6, Table 4"},{"comment":"Footnote 3 notes that the authors' retuned GCN exceeds the GCN numbers copied from Liu et al. (2020); this is an important caveat and should be given more prominence, since it illustrates that the graph-method numbers used in Tables 1 and 2 are not state-of-the-art even for the same architecture.","section":"Section 4, footnote 3"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern lands: the paper's headline claim is not yet supported because the comparison mixes tuned MLP numbers with untuned GNN numbers from prior papers under unverified protocol equivalence. The most valuable fix would be to retune a representative set of graph methods (at minimum GCN and GAT) under the same pipeline used for the MLP and to re-report Tables 1 and 2. The synthetic and feature-study contributions are promising but need the tightening described in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on Katsman et al. (2412.06173). The paper's real contribution is not the headline—tuned MLPs rival GNNs—which earlier work already suggested. It's the systematic feature-ablation study and the WS1000 synthetic family. Those are genuinely useful tools for the graph-learning community.\n\nThe central claim, however, is not yet established. Tables 1 and 2 mix the authors' carefully tuned MLP with GNN numbers lifted from other papers, with different splits, preprocessing, and tuning protocols. Section 7 admits this, and footnote 3 reports that a retuned GCN already beats the copied GCN on Amazon Computers. That is a controlled experiment that should have been run across all seven datasets before concluding 'little benefit.' The abstract also ignores that the strongest GNNs in the paper's own tables beat the tuned MLP by 3–9 points on the supposedly 'little benefit' datasets. The MLP improvement from 44.9% to 82.6% on Computers is striking, but the original baseline is suspiciously low, suggesting the comparison is partly an artifact of poor prior baselines.\n\nThe feature study is more sound: incrementally adding features while holding the graph fixed shows the MLP-GCN gap closing for Computers but not Cora. That is a clean way to show feature leakage, and the negative control strengthens it. The synthetic WS1000 benchmarks are also a good idea, though the GCN's 54.7 AUC on the plain version shows the tasks are very hard, so they need more work before they're accepted as canonical.\n\nThe paper is honest about its limitations, and the writing is clear. It deserves serious refereeing, but it needs major revision: either retune the GNNs under the same protocol and report the real gaps, or soften the headline claim to match the evidence. As it stands, the conclusion is conditional, not established.\n\nI'd bring this to a reading group, and I'd likely cite the feature study and synthetic benchmarks if I were benchmarking GNNs. Send it to review.","headline":"Useful feature-ablation study and synthetic benchmarks, but the headline claim about graph structure is undercut by uncontrolled comparisons; conditional acceptance.","tokens_in":15684,"tokens_out":3718,"would_cite":true,"duration_ms":34679,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A well-tuned feature-only MLP nearly matches graph neural networks on five of seven common graph benchmarks.","keywords":["graph neural networks","node classification","link prediction","benchmark evaluation","feature leakage","MLP baseline","synthetic graph datasets","graph structure necessity"],"falsifier":"On Amazon Computers, re-tune GCN, GAT, and DAGNN with the same split and the same hyperparameter budget used for the tuned MLP; if the re-tuned graph models outperform it by more than the paper's reported gap, the 'little benefit' claim fails.","tokens_in":1808,"feed_emoji":"🕸️","tokens_out":2608,"duration_ms":81738,"temperature":0.7,"pith_summary":"Graph machine learning assumes node features and graph structure are separate, so that benchmark scores show what models learn from the graph. This paper challenges that assumption across seven of the most widely used graph datasets, finding that a carefully tuned feature-only MLP matches or beats many graph neural networks on five of them. The paper argues those five datasets are weak instruments for measuring graph-learning progress because their node features already contain enough graph information to make the graph nearly redundant. If the finding holds, much of the published benchmark comparison on these datasets should be re-interpreted with a properly tuned MLP baseline in hand.","feed_headline":"Five popular graph benchmarks barely need the graph","feed_subtitle":"Node features already encode enough graph structure that these datasets can't measure graph-learning gains.","key_machinery":"The load-bearing instrument is the feature-ablation comparison: for each dataset, the authors create variants with increasing numbers of node features, then tune both an MLP and a GCN on each variant and track how the accuracy gap moves. When the gap collapses as features are added, the features are said to leak the graph structure, explaining why the full-feature MLP nearly matches graph methods. The complementary machinery is a parametric family of synthetic Watts-Strogatz datasets, $\\mathrm{WS}1000_\\gamma$, whose features are generated with a tunable parental dependence along the graph, so the graph's necessity can be controlled from first principles.","core_discovery":"The paper's central claim is that on Amazon Computers, Amazon Photo, Coauthor CS, Coauthor Physics, and PubMed, the graph adds little beyond the node features: a well-tuned MLP that never sees the graph reduces error by up to 70% relative to previously reported MLP baselines, and on Coauthor Physics it outperforms almost every graph-based method tested. The mechanism is not that graphs are useless, but that the features leak graph information: as the paper feeds the MLP and a GCN increasing subsets of the features, the performance gap between them closes for these datasets while staying roughly constant for Cora, where the graph still matters. Based on this, the paper proposes a parametric family of synthetic Watts-Strogatz link-prediction datasets in which informative features are generated by graph-dependent parental dependence, and identifies Cora, CiteSeer, Musae-Twitch, Twitch-PTBR, and Musae-Facebook as real-world tasks with a meaningful MLP-versus-GCN separation.","pith_inferences":["If the feature-leakage mechanism generalizes, other feature-rich node-classification datasets—especially those with bag-of-words or product-description features—could exhibit the same effect, so a tuned MLP could serve as a cheap screening test before any new graph dataset is adopted.","The synthetic generation procedure could be extended beyond link prediction to node and graph classification, letting researchers dial the parental-dependence parameter to calibrate how much structure a method truly requires.","A practical red-flag tool falls out of the feature-ablation comparison: a gap that shrinks as feature counts grow is a warning that reported GNN advantages on the full dataset may not be driven by graph structure.","Re-tuning the graph baselines on identical splits with the same budget as the MLP would either confirm the paper's conclusion or reveal that part of the 'little benefit' result comes from under-tuned graph models rather than from graph redundancy."],"forward_implications":["Five of the seven most common graph benchmarks—Amazon Computers, Amazon Photo, Coauthor CS, Coauthor Physics, and PubMed—should no longer be treated as demonstrations that graph structure matters, without a strong tuned-MLP baseline.","Reported gains of graph networks on these datasets may largely reflect hyperparameter tuning rather than graph-specific architecture, so a tuned MLP should be a required point of comparison in future evaluations.","Cora, CiteSeer, and the Musae-Twitch, Twitch-PTBR, and Musae-Facebook link-prediction tasks offer real-world settings where the graph provides a clear, measurable benefit.","The synthetic WS1000 family gives the field a controlled testbed where graph information is necessary by construction, allowing researchers to measure how much a method actually exploits structure.","Benchmark designers should screen new datasets for feature leakage by running the MLP-versus-GCN gap at increasing feature counts before adopting them."],"supporting_citations":[{"why":"Supplies the Amazon Computers, Amazon Photo, Coauthor CS, and Coauthor Physics datasets and documents evaluation pitfalls (split consistency, simple-model baselines) that motivate the paper's re-benchmarking.","marker":"Shchur et al., 2018"},{"why":"Supplies the Cora, CiteSeer, and PubMed citation datasets whose MLP-versus-GCN behavior is analyzed.","marker":"Sen et al., 2008"},{"why":"Defines the GCN that serves as the representative graph method in the feature studies and a key comparator in Tables 1 and 2.","marker":"Kipf & Welling, 2017"},{"why":"Supplies the original MLP baseline and DAGNN graph-method numbers that the tuned MLP is shown to improve upon drastically.","marker":"Liu et al., 2020"},{"why":"Provides the GAT baseline used in Tables 1 and 2 for comparison with the tuned MLP.","marker":"Velickovic et al., 2018"},{"why":"Provides the GraphSAGE baselines used in Table 1, a standard message-passing family.","marker":"Hamilton et al., 2017"},{"why":"Defines the small-world graph model used to construct the synthetic WS1000 datasets, where graph structure is necessary by construction.","marker":"Watts & Strogatz, 1998"},{"why":"Supplies the Musae-Twitch, Twitch-PTBR, and Musae-Facebook datasets recommended as benchmarks with clear MLP-versus-GCN separation.","marker":"Rozemberczki et al., 2019"},{"why":"Provides the Exphormer numbers in Table 1, the strongest graph method the tuned MLP is compared against on the four co-purchase/co-authorship datasets.","marker":"Shirzad et al., 2023"}],"fun_headline_variants":["Feature-only MLP rivals GNNs on five common graph benchmarks","Graph structure barely helps on five popular benchmarks","Node features alone solve five graph benchmarks","New synthetic graphs offer harder test for GNNs","MLP without edges matches GNNs on five datasets"],"cache_read_input_tokens":17664,"weakest_assumption_plain":"The analysis assumes that the graph-network accuracies taken from earlier papers were obtained under training and evaluation protocols equivalent to those used for the tuned MLP, so the reported gaps are a fair head-to-head comparison rather than an artifact of different splits or tuning budgets.","fun_headline_variants_meta":{"raw":{"variants":["Feature-only MLP rivals GNNs on five common graph benchmarks","Graph structure barely helps on five popular benchmarks","Node features alone solve five graph benchmarks","New synthetic graphs offer harder test for GNNs","MLP without edges matches GNNs on five datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1388,"prompt_tokens":1019,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":635,"tokens_out":369,"duration_ms":6837,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:56:18.549487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On Amazon Computers, re-tune GCN, GAT, and DAGNN with the same split and the same hyperparameter budget used for the tuned MLP; if the re-tuned graph models outperform it by more than the paper's reported gap, the 'little benefit' claim fails.","supporting_citations":[{"cited_title":"Collective classification in network data","cited_arxiv_id":null,"evidence_quote":"Supplies the Cora, CiteSeer, and PubMed citation datasets whose MLP-versus-GCN behavior is analyzed."},{"cited_title":"Towards deeper graph neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the original MLP baseline and DAGNN graph-method numbers that the tuned MLP is shown to improve upon drastically."},{"cited_title":"Venkatachalam, Danica J","cited_arxiv_id":null,"evidence_quote":"Provides the Exphormer numbers in Table 1, the strongest graph method the tuned MLP is compared against on the four co-purchase/co-authorship datasets."}],"review_version":1}