REVIEW 4 major objections 6 minor 35 references
AutoSGNN: Automatic Propagation Mechanism Discovery for Spectral Graph Neural Networks
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read AutoSGNN claims that the propagation mechanism of a spectral graph neural network—the feature fitting, graph Laplacian regularization, and aggregation terms defining how node signals are filtered along edges—can be discovered…
desk verdict A solid LLM+evolutionary recipe for spectral GNN search, with shipped code and prompts, but the abstract overclaims on accuracy and efficiency; the evaluation protocol needs a fair-tuned rerun before the headline result is trustworthy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the unified spectral GNN architecture of Eq. (3), $Z^K = \langle \mathrm{TRANS}(\mathrm{AGG}\{G; X_{\text{raw}}; Z^{(K-1)}\})\rangle^K$, which decomposes any spectral GNN into a feature fitting term, a graph Laplacian regularization term, and an aggregation term. Around this template AutoSGNN builds three evolutionary prompt operators for the LLM: E1 (propose a layer totally different from the elites, acting like mutation), E2 (infer the common backbone of the elites and design a variation, acting like crossover), and C1 (compare a high-scoring and a low-scoring elite and hypothesize why the better one wins, setting LLM preference à la DPO). Candidate layers are generated as Python MessagePassing classes that fit the template; fitness is node-classification validation accuracy; an elite population of 30 is retained each cycle; and a 600-second timeout discards nonterminating candidates. The search runs 30 cycles with GPT-3.5-turbo generating 12 candidates per cycle in parallel.
What would settle it
Run the full AutoSGNN search independently many times on the same nine datasets with different random seeds, and re-tune all baseline hyperparameters on the same 2.5% validation split used for the search. If the rank-1 sparse-split result is not reproduced, or if a re-tuned baseline matches or exceeds AutoSGNN's mean accuracy, the paper's central performance claim fails.
Extended reading notes
Core claim
The central discovery is that a per-dataset search over a unified spectral GNN template—one that separates feature transformation, aggregation along the graph, and the graph Laplacian regularization—produces architectures that are at least as accurate as state-of-the-art human-designed spectral GNNs on every one of nine datasets and better on most, while also being interpretable as explicit propagation mechanisms. The paper shows concrete examples: on Cora the search yields a full-frequency filter that combines low-frequency, mid-frequency, and high-frequency components through a residual-like recurrence; on heterophilic graphs such as Texas and Cornell it yields attention-weighted combinations of raw and propagated features. AutoSGNN also reports that the searched architectures are dataset-specific: transferring one dataset's architecture to another often degrades performance, most clearly across homophilic/heterophilic boundaries, which the paper takes as evidence that per-graph search is the right response to graph diversity.
Load-bearing premise
The comparison assumes that validation accuracy on a 2.5% node split—about 68 nodes for Cora—is a stable and fair fitness signal for AutoSGNN, which searches per dataset on that split, against baselines whose hyperparameters were taken from their original papers rather than re-tuned on this split.
Editorial extensions
If this is right
- Per-graph automated search can replace the manual selection of spectral GNN architectures, yielding a different filter for homophilic and heterophilic graphs.
- The discovered mechanisms are interpretable: for example, the Cora architecture decomposes into explicit low-, mid-, and high-frequency filtering stages, not just a black-box layer stack.
- Search time is competitive with gradient-based NAS (about 160–176 minutes per dataset here) and far below evolutionary GNN-NAS, making LLM-guided search practical for medium-size graphs.
- Because any LLM can be plugged in, the framework's quality scales with the model: GPT-4o improved accuracy by about 0.3 points on PubMed and produced more executable code than GPT-3.5-turbo.
- The transfer experiments imply that an architecture tuned on one graph type will not generalize broadly, so dataset-specific search is a feature, not a bug.
Reading between the lines
- The 2.5% validation split is a very weak fitness signal, so the reported ranks may be sensitive to the particular split; a more informative signal (e.g., a small held-out validation set or multiple splits) could change the ranking.
- Because the comparison reuses original-paper hyperparameters for baselines while AutoSGNN tunes on the validation split, the measured gap may overstate the benefit; re-tuning baselines with the same search budget is a natural stress test.
- The search space is restricted to the three-term decomposition of Eq. (3); a harder test of the framework's discovery power would be whether it can re-derive or beat fixed-basis filters (e.g., Bernstein or Jacobi approximations) when those lie outside its template.
- The framework's 'discoveries' may partly encode heuristics already present in the LLM's pretraining, so measuring novelty relative to known spectral GNN designs would clarify whether the contribution is new mechanisms or automated tuning.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AutoSGNN, a framework that combines large language models with evolutionary strategies to automatically generate spectral GNN architectures. The search space covers feature fitting terms, graph Laplacian regularization terms, and aggregation terms; three prompt strategies (E1, E2, C1) guide the LLM through mutation-like, crossover-like, and preference-setting operations. The method is evaluated on nine homophilic and heterophilic graph datasets for node classification under a sparse 2.5%:2.5%:95% split and a dense 60%:20%:20% split, reporting rank 1 in the sparse table and rank 2 in the dense table. The paper also includes ablation studies, parameter analyses, timing comparisons, and cross-dataset transfer experiments.
Significance. If the main performance claim holds, AutoSGNN is a meaningful contribution: it introduces a new category of GNN architecture search that is neither gradient-based, RL-based, nor purely evolutionary, and it demonstrates that LLM-generated spectral filters can be competitive with human-designed ones. The authors publish code and report detailed searched architectures, which supports reproducibility. The use of the Wilcoxon-Holm ranking procedure and the inclusion of a dense-split appendix are also positive features. However, the empirical claim that AutoSGNN 'outperforms state-of-the-art spectral GNNs and GNN-NAS methods' is only as strong as the evaluation protocol, and that protocol has load-bearing weaknesses that need to be addressed before the claim is fully convincing.
major comments (4)
- [Experiments / Table 1] The primary performance claim rests on a sparse split of 2.5%:2.5%:95%, but for the smallest datasets this yields an extremely small validation set: Texas and Cornell have only 183 nodes, so their validation splits contain roughly 5 nodes, and Cora has about 68 validation nodes. Because AutoSGNN selects architectures by validation accuracy (Section 'Networks Evaluation'), on these datasets the fitness signal is dominated by sampling noise. The reported rank-1 result may therefore reflect selection on validation noise rather than a genuine superiority. The paper should provide evidence that the ranking is stable across multiple random splits, or use a larger validation set for the search, or otherwise quantify the uncertainty of the validation-based selection.
- [Reproducibility / Experiment Setting] The sentence 'All experiments, in order to remove the effects of randomness, are run independently for 10 trials and the resulting means and standard deviations are reported' is ambiguous. It does not state whether a trial repeats the full stochastic LLM search or only retrains the final discovered architecture. If the search is run only once per dataset, the reported standard deviations capture only evaluation variance, not search variance. This matters for the central comparison, since AutoSGNN's performance is the result of a stochastic search. Please clarify the protocol, and if the search was not repeated, either rerun it several times or explicitly caveat the reported uncertainty.
- [Experiment Setting / Baselines] AutoSGNN searches directly on the validation split, while the baselines use 'the best hyperparameter combinations provided in the original papers for each dataset.' This is an asymmetric protocol: baselines tuned under other split regimes may be under-tuned for the sparse 2.5% validation split, giving AutoSGNN a systematic advantage. The authors should either re-tune the baselines on the same validation split using a comparable budget, or justify why the original hyperparameters are appropriate for the sparse-split comparison. Without this, the rank-1 claim in Table 1 is not fully supported.
- [Appendix F, Table A3] Under the dense 60%:20%:20% split, AutoSGNN is ranked second, behind NFGNN, and is clearly worse on Chameleon (66.30 vs. 72.52) and Squirrel (56.58 vs. 58.90). Given the abstract and introduction claim that AutoSGNN 'outperforms state-of-the-art spectral GNNs and ... methods,' these dense-split results should be reconciled with that claim. At minimum, the performance claim should be qualified as applying to the sparse-split setting, or the dense-split gap should be discussed and explained.
minor comments (6)
- [Figures 5/7, Table A1, Figure A5] There are several typos: 'condidate' in the captions of Figures 5 and 7, 'Squireel' in Table A1, and 'T exas' in the subplot labels of Figure A5.
- [Table 2 / Appendix C] The Cora recurrence in Table 2 uses Zk on both sides of the equation, which is notationally ambiguous; it should use a different index for the accumulation variable. The code in Appendix C, Figures A2-A4, appears with duplicated blocks: the Chameleon code in Figure A3 and the Cornell code in Figure A4 are each shown twice. These duplicates should be removed for clarity.
- [Reproducibility / Conclusion] Minor wording issues include 'iterative search algebra is set to 30' (should be 'iterations'), 'intial α' in the Cora case study, and 'ture automatic propagation mechanism generation method' in the conclusion.
- [Figure 8] The caption of Figure 8 says 'tested on the Pubmed dataset,' but the x-axis lists Cora, CiteSeer, and PubMed; either the caption or the figure should be corrected.
- [References] Defferrard et al. 2016a and 2016b are listed as separate references but refer to the same conference paper; they should be merged. The Related Work section also contains the heading 'Neural Achitecture Search,' which should be spelled 'Neural Architecture Search.'
- [Prompt Strategies / Basic Content] The Basic Content prompt includes expert tips such as 'different types of graphs lend themselves to different filters' and a note about heterophily. This means the 'discovery' is substantially steered by injected expert knowledge. The authors should discuss how much of the reported performance comes from this guidance rather than from the evolutionary search, especially since removing the C1 preference prompt has a large effect in Table 3.
Circularity Check
No significant circularity: the paper's claims are empirical benchmark results from held-out test splits, not derivations from fitted inputs.
full rationale
AutoSGNN is an empirical LLM-plus-evolutionary-strategy search method, and its central claims are measured test-set accuracies (Tables 1 and A3) and wall-clock search times (Figure 8). The fitness signal used during search is validation accuracy, the reported numbers are test-set accuracies on a 95% held-out split, so the central evaluation does not reduce to the search objective by construction. The search space is admittedly defined in terms of feature fitting terms, graph Laplacian regularization terms, and aggregation terms, and the Appendix B prompt seeds the LLM with the prior that different graph types suit different filters and that heterophilic graphs violate the homogeneity assumption; but these are search priors and warm starts (elite individuals are initialized from GCN, FAGCN, GPRGNN, BernNet, and NFGNN), not outputs forced by a derivation. The resulting architectures do resemble known filters such as APPNP/GPRGNN-style residual propagation, but the paper's claim is automatic discovery and competitive accuracy, not mathematical novelty of the filters, so no result is renamed or smuggled in as a prediction. The cited unifying framework of Zhu et al. (2021) is used only to motivate a very general architecture template, and the self-citations in the references are not load-bearing. Concerns about the 2.5% validation split being noisy and baselines not being retuned on the same split are statistical-validity and protocol issues, not circularity under the definitions used here. No equation is defined in terms of the quantity it is claimed to predict, and no fitted parameter is renamed as a prediction.
Assumptions & free parameters
free parameters (6)
- Elite population size =
30
- Search iterations =
30
- P1/P2 (elite individuals per E1/E2 prompt) =
4
- Parallel LLM responses per cycle =
4
- Timeout duration =
600 s
- LLM model choice =
GPT-3.5-turbo
assumptions (4)
- standard math Graph Laplacian eigendecomposition and graph Fourier transform provide a valid spectral representation for node signals.
- domain assumption The unified architecture in Eq. (3), combining feature transformation, aggregation, and a per-layer operation, is general enough to represent the spectral GNN search space.
- ad hoc to paper Validation accuracy on a 2.5% node split is an informative and stable fitness signal for architecture search.
- ad hoc to paper The three prompt strategies E1, E2, C1 and the injected heterophily tip in the Basic Content steer the LLM toward useful architectures.
Cite this review
Pith. "Pith review of AutoSGNN: Automatic Propagation Mechanism Discovery for Spectral Graph Neural Networks." pith.science (2026). https://pith.science/paper/PR5RZKUF
@misc{pith2026241212483,
author = {Pith},
title = {Pith review of: AutoSGNN: Automatic Propagation Mechanism Discovery for Spectral Graph Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/PR5RZKUF}},
note = {Machine review of arXiv:2412.12483}
}
read the original abstract
In real-world applications, spectral Graph Neural Networks (GNNs) are powerful tools for processing diverse types of graphs. However, a single GNN often struggles to handle different graph types-such as homogeneous and heterogeneous graphs-simultaneously. This challenge has led to the manual design of GNNs tailored to specific graph types, but these approaches are limited by the high cost of labor and the constraints of expert knowledge, which cannot keep up with the rapid growth of graph data. To overcome these challenges, we propose AutoSGNN, an automated framework for discovering propagation mechanisms in spectral GNNs. AutoSGNN unifies the search space for spectral GNNs by integrating large language models with evolutionary strategies to automatically generate architectures that adapt to various graph types. Extensive experiments on nine widely-used datasets, encompassing both homophilic and heterophilic graphs, demonstrate that AutoSGNN outperforms state-of-the-art spectral GNNs and graph neural architecture search methods in both performance and efficiency.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Bo, D.; Wang, X.; Shi, C.; and Shen, H. 2021. Beyond low-frequency information in graph convolutional networks. In Proceedings of the AAAI Conference on Artificial Intelligence
work page 2021
-
[4]
Chien, E.; Peng, J.; Li, P.; and Milenkovic, O. 2021. Adaptive universal generalized PageRank graph neural network. In International Conference on Learning Representations
work page 2021
-
[5]
Defferrard, M.; Bresson, X.; and Vandergheynst, P. 2016 a . Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems
work page 2016
-
[6]
Defferrard, M.; Bresson, X.; and Vandergheynst, P. 2016 b . Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems
work page 2016
-
[7]
Fout, A.; Byrd, J.; Shariat, B.; and Ben-Hur, A. 2017. Protein interface prediction using graph convolutional networks. In Advances in Neural Information Processing Systems
work page 2017
-
[8]
Gao, Y.; Yang, H.; Zhang, P.; Zhou, C.; and Hu, Y. 2021. Graph neural architecture search. In International Joint Conference on Artificial Intelligence
work page 2021
Show all 35 references
-
[9]
Gasteiger, J.; Bojchevski, A.; and G \"u nnemann, S. 2019. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations
2019
-
[10]
Guo, H.; Ma, Z.; Chen, J.; Ma, Y.; Cao, Z.; Zhang, X.; and Gong, Y.-J. 2024. ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning. arXiv preprint arXiv:2412.07507
2024 arXiv
-
[11]
Hamilton, W.; Ying, Z.; and Leskovec, J. 2017. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems
2017
-
[12]
He, M.; Wei, Z.; Xu, H.; et al. 2021. BernNet: Learning arbitrary graph spectral filters via bernstein approximation. In Advances in Neural Information Processing Systems
2021
-
[13]
Huan, Z.; Quanming, Y.; and Weiwei, T. 2021. Search to aggregate neighborhood for graph neural network. In 2021 IEEE 37th International Conference on Data Engineering
2021
-
[14]
Ismail Fawaz, H.; Forestier, G.; Weber, J.; Idoumghar, L.; and Muller, P.-A. 2019. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 33(4): 917--963
2019
-
[15]
N.; and Welling, M
Kipf, T. N.; and Welling, M. 2017. Semi-Supervised classification with graph convolutional networks. In International Conference on Learning Representations
2017
-
[16]
B.; Zhang, X.; Wang, H.; and Liu, J
Li, X.; Wu, K.; Li, Y. B.; Zhang, X.; Wang, H.; and Liu, J. 2024 a . Pretrained Optimization Model for Zero-Shot Black Box Optimization. In Advances in Neural Information Processing Systems
2024
-
[17]
Li, X.; Wu, K.; Zhang, X.; and Wang, H. 2024 b . B2Opt: Learning to Optimize Black-box Optimization with Little Budget. In Proceedings of the AAAI Conference on Artificial Intelligence
2024
-
[18]
Li, Y.; and King, I. 2020. Autograph: Automated graph neural network. In International Conference on Neural Information Processing
2020
-
[19]
Liu, F.; Xialiang, T.; Yuan, M.; Lin, X.; Luo, F.; Wang, Z.; Lu, Z.; and Zhang, Q. 2024. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In International Conference on Machine Learning
2024
-
[20]
Liu, Y.; and Liu, J. 2023. A surrogate evolutionary neural architecture search algorithm for graph neural networks. Applied Soft Computing, 144: 110485
2023
-
[21]
Mao, H.; Chen, Z.; Jin, W.; Han, H.; Ma, Y.; Zhao, T.; Shah, N.; and Tang, J. 2024 a . Demystifying structural disparity in graph neural networks: Can one size fit all? In Advances in Neural Information Processing Systems
2024
-
[22]
Mao, H.; Chen, Z.; Tang, W.; Zhao, J.; Ma, Y.; Zhao, T.; Shah, N.; Galkin, M.; and Tang, J. 2024 b . Position: Graph foundation models are already here. In International Conference on Machine Learning
2024
-
[23]
McPherson, M.; Smith-Lovin, L.; and Cook, J. M. 2001. Birds of a feather: Homophily in social networks. Annual Review of Sociology, 27(1): 415--444
2001
-
[24]
F.; Leike, J.; and Lowe, R
Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P. F.; Leike, J.; and Lowe, R. 2022. Training language model...
2022
-
[25]
Po s \' k, P.; Huyer, W.; and P \'a l, L. 2012. A comparison of global search algorithms for continuous black box optimization. Evolutionary Computation, 20(4): 509--541
2012
-
[26]
D.; Ermon, S.; and Finn, C
Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Ermon, S.; and Finn, C. 2024. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems
2024
-
[27]
P.; Dupont, E.; Ruiz, F
Romera-Paredes, B.; Barekatain, M.; Novikov, A.; Balog, M.; Kumar, M. P.; Dupont, E.; Ruiz, F. J.; Ellenberg, J. S.; Wang, P.; Fawzi, O.; et al. 2024. Mathematical discoveries from program search with large language models. Nature, 625(7995): 468--475
2024
-
[28]
Shi, M.; Tang, Y.; Zhu, X.; Huang, Y.; Wilson, D.; Zhuang, Y.; and Liu, J. 2022. Genetic-GNN: Evolutionary architecture search for graph neural networks. Knowledge-based Systems, 247: 108752
2022
-
[29]
B.; and Reddy, C
Shojaee, P.; Meidani, K.; Gupta, S.; Farimani, A. B.; and Reddy, C. K. 2024. LLM-SR: Scientific equation discovery via programming with large language models. arXiv preprint arXiv:2404.18400
2024 arXiv
-
[30]
Von Luxburg, U. 2007. A tutorial on spectral clustering. Statistics and Computing, 17: 395--416
2007
-
[31]
Wang, X.; and Zhang, M. 2022. How powerful are spectral graph neural networks. In International conference on Machine Learning
2022
-
[32]
Wei, L.; Zhao, H.; and He, Z. 2022. Designing the topology of graph neural networks: A novel feature fusion perspective. In Proceedings of the ACM Web Conference
2022
-
[33]
Wu, K.; Li, X.; Liu, P.; and Liu, J. 2023. DECN: Evolution inspired deep convolution network for black-box optimization. arXiv preprint arXiv:2304.09599
2023 arXiv
-
[34]
Zheng, S.; Zhu, Z.; Liu, Z.; Li, Y.; and Zhao, Y. 2024. Node-oriented Spectral Filtering for Graph Neural Networks. IEEE Transactions on Pattern Analysis & Machine Intelligence, 46(1): 388--402
2024
-
[35]
Zhu, M.; Wang, X.; Shi, C.; Ji, H.; and Cui, P. 2021. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.