{"id":"c2b01437-3a9e-4129-a06e-7c0f2719b86d","arxiv_id":"2412.12483","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM plus evolutionary-strategy framework that searches for per-dataset spectral GNN propagation mechanisms, with rank-1 sparse-split results but only rank-2 dense-split performance.","lead":"AutoSGNN uses a large language model guided by evolutionary prompts to automatically design spectral graph neural networks for each dataset, and tests them on nine homophilic and heterophilic graphs. The paper claims it beats state-of-the-art spectral GNNs and existing GNN architecture search methods in accuracy and speed, but the evidence is strongest in the sparse-label setting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation splits are so small (Texas/Cornell have ~5 validation nodes) that the fitness signal is noisy; with baselines not re-tuned on the same split, the rank-1 claim may reflect selection on validation noise rather than genuine superiority.","rationale":"The reader's weakest assumption identifies the right linchpin: the sparse-split fitness signal is the main empirical support for the central claim. The search uses validation accuracy on a 2.5% split, which for Texas and Cornell means about 5 validation nodes; the fitness signal is nearly discrete and highly noisy. Under selection pressure, AutoSGNN will pick architectures that happen to score well on these few nodes, and the reported test accuracy may not transfer. The comparison is also asymmetric because baselines use original-paper hyperparameters while AutoSGNN is optimized on the exact split. The dense-split appendix (Table A3) and time analysis (Appendix E) support the conditional verdict: AutoSGNN ranks behind NFGNN under 60/20/20 and is slower than SANE and F2GNN, so the abstract's blanket statement is broader than the evidence. I do not see a reason to reject the method outright; the conditional verdict requiring robustness checks is the right calibration. The proposed test addresses the core threat to the sparse-split claim without moving the verdict.","tokens_in":22727,"tokens_out":6396,"duration_ms":60673,"concrete_test":"Use the exact ten sparse splits from Table 1. For each split, re-run the full AutoSGNN search from scratch ten times, and re-tune each baseline (APPNP, GPRGNN, FAGCN, BernNet, JacobiConv, NFGNN) with 20 random hyperparameter configurations on the same validation nodes used by AutoSGNN, with the same early-stopping budget. Report the distribution of test accuracies and redo the Wilcoxon-Holm ranking. If AutoSGNN is not rank 1 at p=0.05, or does not win on more than half of the splits, the performance advantage is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AutoSGNN 'outperforms state-of-the-art spectral GNNs and graph neural architecture search methods in both performance and efficiency' rests on Table 1, which uses a 2.5%:2.5%:95% split. For the smallest datasets this validation set is extremely small: Texas and Cornell have 183 nodes, so the validation split has roughly 5 nodes, and Cora has about 68 validation nodes. AutoSGNN selects architectures by validation accuracy, as described in the 'Networks Evaluation' section, so on this scale the fitness signal is dominated by sampling noise; an architecture that ranks first on one split may not generalize to the 95% test set. The paper does not state whether the reported 10 trials repeat the full stochastic LLM search or only retrain the final architecture; if only retraining, the reported standard deviations do not capture search variance. In addition, the baselines use hyperparameters from their original papers rather than being re-tuned on the same validation split, while AutoSGNN searches directly on that split. This asymmetric protocol means the rank-1 result in Table 1 could be an artifact of overfitting to a noisy validation set and undertuned baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoSGNN, a framework that combines large language models with evolutionary strategies to automatically generate spectral GNN architectures. The search space covers feature fitting terms, graph Laplacian regularization terms, and aggregation terms; three prompt strategies (E1, E2, C1) guide the LLM through mutation-like, crossover-like, and preference-setting operations. The method is evaluated on nine homophilic and heterophilic graph datasets for node classification under a sparse 2.5%:2.5%:95% split and a dense 60%:20%:20% split, reporting rank 1 in the sparse table and rank 2 in the dense table. The paper also includes ablation studies, parameter analyses, timing comparisons, and cross-dataset transfer experiments.","tokens_in":22980,"tokens_out":4363,"duration_ms":39237,"significance":"If the main performance claim holds, AutoSGNN is a meaningful contribution: it introduces a new category of GNN architecture search that is neither gradient-based, RL-based, nor purely evolutionary, and it demonstrates that LLM-generated spectral filters can be competitive with human-designed ones. The authors publish code and report detailed searched architectures, which supports reproducibility. The use of the Wilcoxon-Holm ranking procedure and the inclusion of a dense-split appendix are also positive features. However, the empirical claim that AutoSGNN 'outperforms state-of-the-art spectral GNNs and GNN-NAS methods' is only as strong as the evaluation protocol, and that protocol has load-bearing weaknesses that need to be addressed before the claim is fully convincing.","major_comments":[{"comment":"The primary performance claim rests on a sparse split of 2.5%:2.5%:95%, but for the smallest datasets this yields an extremely small validation set: Texas and Cornell have only 183 nodes, so their validation splits contain roughly 5 nodes, and Cora has about 68 validation nodes. Because AutoSGNN selects architectures by validation accuracy (Section 'Networks Evaluation'), on these datasets the fitness signal is dominated by sampling noise. The reported rank-1 result may therefore reflect selection on validation noise rather than a genuine superiority. The paper should provide evidence that the ranking is stable across multiple random splits, or use a larger validation set for the search, or otherwise quantify the uncertainty of the validation-based selection.","section":"Experiments / Table 1"},{"comment":"The sentence 'All experiments, in order to remove the effects of randomness, are run independently for 10 trials and the resulting means and standard deviations are reported' is ambiguous. It does not state whether a trial repeats the full stochastic LLM search or only retrains the final discovered architecture. If the search is run only once per dataset, the reported standard deviations capture only evaluation variance, not search variance. This matters for the central comparison, since AutoSGNN's performance is the result of a stochastic search. Please clarify the protocol, and if the search was not repeated, either rerun it several times or explicitly caveat the reported uncertainty.","section":"Reproducibility / Experiment Setting"},{"comment":"AutoSGNN searches directly on the validation split, while the baselines use 'the best hyperparameter combinations provided in the original papers for each dataset.' This is an asymmetric protocol: baselines tuned under other split regimes may be under-tuned for the sparse 2.5% validation split, giving AutoSGNN a systematic advantage. The authors should either re-tune the baselines on the same validation split using a comparable budget, or justify why the original hyperparameters are appropriate for the sparse-split comparison. Without this, the rank-1 claim in Table 1 is not fully supported.","section":"Experiment Setting / Baselines"},{"comment":"Under the dense 60%:20%:20% split, AutoSGNN is ranked second, behind NFGNN, and is clearly worse on Chameleon (66.30 vs. 72.52) and Squirrel (56.58 vs. 58.90). Given the abstract and introduction claim that AutoSGNN 'outperforms state-of-the-art spectral GNNs and ... methods,' these dense-split results should be reconciled with that claim. At minimum, the performance claim should be qualified as applying to the sparse-split setting, or the dense-split gap should be discussed and explained.","section":"Appendix F, Table A3"}],"minor_comments":[{"comment":"There are several typos: 'condidate' in the captions of Figures 5 and 7, 'Squireel' in Table A1, and 'T exas' in the subplot labels of Figure A5.","section":"Figures 5/7, Table A1, Figure A5"},{"comment":"The Cora recurrence in Table 2 uses Zk on both sides of the equation, which is notationally ambiguous; it should use a different index for the accumulation variable. The code in Appendix C, Figures A2-A4, appears with duplicated blocks: the Chameleon code in Figure A3 and the Cornell code in Figure A4 are each shown twice. These duplicates should be removed for clarity.","section":"Table 2 / Appendix C"},{"comment":"Minor wording issues include 'iterative search algebra is set to 30' (should be 'iterations'), 'intial α' in the Cora case study, and 'ture automatic propagation mechanism generation method' in the conclusion.","section":"Reproducibility / Conclusion"},{"comment":"The caption of Figure 8 says 'tested on the Pubmed dataset,' but the x-axis lists Cora, CiteSeer, and PubMed; either the caption or the figure should be corrected.","section":"Figure 8"},{"comment":"Defferrard et al. 2016a and 2016b are listed as separate references but refer to the same conference paper; they should be merged. The Related Work section also contains the heading 'Neural Achitecture Search,' which should be spelled 'Neural Architecture Search.'","section":"References"},{"comment":"The Basic Content prompt includes expert tips such as 'different types of graphs lend themselves to different filters' and a note about heterophily. This means the 'discovery' is substantially steered by injected expert knowledge. The authors should discuss how much of the reported performance comes from this guidance rather than from the evolutionary search, especially since removing the C1 preference prompt has a large effect in Table 3.","section":"Prompt Strategies / Basic Content"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest in reporting the dense-split results and follows standard sparse-split settings from prior work, so the main issue is evaluation rigor rather than fabrication. The central idea is promising and the appendix material is detailed; with stronger split-stability evidence, clarified search-repetition semantics, and re-tuned baselines, the performance claim would be much more convincing. There are no obvious ethical or novelty-disclosure concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AutoSGNN is a genuinely useful LLM-plus-evolutionary recipe for spectral GNN architecture search. I'd send it out, and I'd ask the authors to fix an overclaim and tighten the evaluation.\n\nWhat's actually new: the three prompt operators—E1 mutation, E2 crossover, and C1 DPO-style preference-setting—applied to a text-and-code representation of spectral GNNs. That combination isn't in the cited GNN-NAS literature. The paper ships code, the exact prompts, and the per-dataset searched architectures, which is real and useful. The ablation shows each prompt type contributes, with C1 the most important; that's a credible finding. The case studies in Appendix C give enough detail to reproduce the search.\n\nThe soft spots are in the evaluation. The abstract's claim of outperforming SOTA spectral GNNs and GNN-NAS 'in both performance and efficiency' is not supported by the paper's own numbers. In the dense split (Table A3), AutoSGNN is rank 2 behind NFGNN, and on Chameleon and Squirrel it is clearly worse. In time (Figure 8, Appendix E), AutoSGNN is slower than SANE and F2GNN, the gradient-based NAS baselines. The efficiency claim only holds against evolutionary NAS, a weaker statement than the abstract makes.\n\nMore serious is the evaluation protocol. Validation on a 2.5% split gives Texas and Cornell roughly 5 validation nodes total; that's a noisy selection signal. AutoSGNN searches directly on that validation set, while the baselines use hyperparameters from their original papers rather than being re-tuned on the same split. That asymmetry alone could account for part of Table 1's rank-1 gap. The paper also doesn't state whether the 10 trials repeat the full stochastic LLM search or just retrain the final architecture. If it's the latter, the reported standard deviations understate search variance.\n\nNone of this sinks the method. As an automated design tool that produces working, readable architectures, AutoSGNN is a solid contribution. The citations are fair, covering the main GNN-NAS and LLM-for-science lines. This paper is for people building LLM-based NAS or spectral GNN tools. I would accept it for peer review with major revisions—it deserves a referee who can push for a fair-tuned comparison and search-level repeats. The central claim needs narrowing, but the core idea is worth engaging with.","headline":"A solid LLM+evolutionary recipe for spectral GNN search, with shipped code and prompts, but the abstract overclaims on accuracy and efficiency; the evaluation protocol needs a fair-tuned rerun before the headline result is trustworthy.","tokens_in":23548,"tokens_out":2758,"would_cite":true,"duration_ms":25522,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoSGNN claims that the propagation mechanism of a spectral graph neural network—the feature fitting, graph Laplacian regularization, and aggregation terms defining how node signals are filtered along edges—can be discovered…","keywords":["spectral graph neural networks","neural architecture search","large language models","evolutionary strategy","propagation mechanism discovery","homophilic and heterophilic graphs","node classification"],"falsifier":"Run the full AutoSGNN search independently many times on the same nine datasets with different random seeds, and re-tune all baseline hyperparameters on the same 2.5% validation split used for the search. If the rank-1 sparse-split result is not reproduced, or if a re-tuned baseline matches or exceeds AutoSGNN's mean accuracy, the paper's central performance claim fails.","tokens_in":22442,"feed_emoji":"🤖","tokens_out":6991,"duration_ms":52393,"temperature":0.7,"pith_summary":"AutoSGNN claims that the propagation mechanism of a spectral graph neural network—the feature fitting, graph Laplacian regularization, and aggregation terms that define how node signals are filtered along edges—can be discovered automatically for a given graph, rather than inherited from a manually designed architecture. The paper couples a large language model with evolutionary strategies: the LLM proposes candidate network layers as executable Python code plus a design rationale, and three prompt operators mutate, cross over, and contrastively set preferences among candidates, using validation accuracy as the fitness signal. On nine homophilic and heterophilic benchmark graphs the searched per-dataset architectures rank first among spectral graph neural networks and graph neural architecture search methods under the sparse 2.5/2.5/95 split and second under the dense 60/20/20 split, with search time between gradient-based and evolutionary NAS. If the claim holds, it would mean expert manual design of spectral GNNs can be replaced by automated LLM-guided search that adapts to graph type.","feed_headline":"LLM search invents spectral GNNs that outrank hand-built ones","feed_subtitle":"On nine citation, co-purchase, and web graphs, discovered architectures rank first under sparse labels.","key_machinery":"The load-bearing object is the unified spectral GNN architecture of Eq. (3), $Z^K = \\langle \\mathrm{TRANS}(\\mathrm{AGG}\\{G; X_{\\text{raw}}; Z^{(K-1)}\\})\\rangle^K$, which decomposes any spectral GNN into a feature fitting term, a graph Laplacian regularization term, and an aggregation term. Around this template AutoSGNN builds three evolutionary prompt operators for the LLM: E1 (propose a layer totally different from the elites, acting like mutation), E2 (infer the common backbone of the elites and design a variation, acting like crossover), and C1 (compare a high-scoring and a low-scoring elite and hypothesize why the better one wins, setting LLM preference à la DPO). Candidate layers are generated as Python MessagePassing classes that fit the template; fitness is node-classification validation accuracy; an elite population of 30 is retained each cycle; and a 600-second timeout discards nonterminating candidates. The search runs 30 cycles with GPT-3.5-turbo generating 12 candidates per cycle in parallel.","core_discovery":"The central discovery is that a per-dataset search over a unified spectral GNN template—one that separates feature transformation, aggregation along the graph, and the graph Laplacian regularization—produces architectures that are at least as accurate as state-of-the-art human-designed spectral GNNs on every one of nine datasets and better on most, while also being interpretable as explicit propagation mechanisms. The paper shows concrete examples: on Cora the search yields a full-frequency filter that combines low-frequency, mid-frequency, and high-frequency components through a residual-like recurrence; on heterophilic graphs such as Texas and Cornell it yields attention-weighted combinations of raw and propagated features. AutoSGNN also reports that the searched architectures are dataset-specific: transferring one dataset's architecture to another often degrades performance, most clearly across homophilic/heterophilic boundaries, which the paper takes as evidence that per-graph search is the right response to graph diversity.","pith_inferences":["The 2.5% validation split is a very weak fitness signal, so the reported ranks may be sensitive to the particular split; a more informative signal (e.g., a small held-out validation set or multiple splits) could change the ranking.","Because the comparison reuses original-paper hyperparameters for baselines while AutoSGNN tunes on the validation split, the measured gap may overstate the benefit; re-tuning baselines with the same search budget is a natural stress test.","The search space is restricted to the three-term decomposition of Eq. (3); a harder test of the framework's discovery power would be whether it can re-derive or beat fixed-basis filters (e.g., Bernstein or Jacobi approximations) when those lie outside its template.","The framework's 'discoveries' may partly encode heuristics already present in the LLM's pretraining, so measuring novelty relative to known spectral GNN designs would clarify whether the contribution is new mechanisms or automated tuning."],"forward_implications":["Per-graph automated search can replace the manual selection of spectral GNN architectures, yielding a different filter for homophilic and heterophilic graphs.","The discovered mechanisms are interpretable: for example, the Cora architecture decomposes into explicit low-, mid-, and high-frequency filtering stages, not just a black-box layer stack.","Search time is competitive with gradient-based NAS (about 160–176 minutes per dataset here) and far below evolutionary GNN-NAS, making LLM-guided search practical for medium-size graphs.","Because any LLM can be plugged in, the framework's quality scales with the model: GPT-4o improved accuracy by about 0.3 points on PubMed and produced more executable code than GPT-3.5-turbo.","The transfer experiments imply that an architecture tuned on one graph type will not generalize broadly, so dataset-specific search is a feature, not a bug."],"supporting_citations":[{"why":"Provides the unified optimization view of spectral GNNs that justifies decomposing the search space into feature fitting and Laplacian regularization terms.","marker":"Zhu et al. 2021"},{"why":"NFGNN is a key baseline and the source of the experimental protocol, hyperparameter choices, and sparse-split setting the paper follows.","marker":"Zheng et al. 2024"},{"why":"GPRGNN serves as a strong human-designed spectral GNN baseline and one of the elite individuals used to initialize the population.","marker":"Chien et al. 2021"},{"why":"BernNet is a baseline and an elite individual, representing polynomial spectral filters in the search initialization.","marker":"He et al. 2021"},{"why":"FAGCN is a baseline and elite initializer for attention-style low/high-pass spectral filters.","marker":"Bo et al. 2021"},{"why":"GCN is a classic baseline and an elite initializer for the population.","marker":"Kipf and Welling 2017"},{"why":"SANE is a differentiable GNN-NAS baseline compared on accuracy and search time.","marker":"Huan, Quanming, and Weiwei 2021"},{"why":"GTFGNAS is an evolutionary GNN-NAS baseline whose experimental setup the paper adopts for NAS comparisons.","marker":"Liu and Liu 2023"},{"why":"Direct preference optimization inspires the C1 contrastive prompt that sets LLM preferences between high- and low-scoring elites.","marker":"Rafailov et al. 2024"}],"fun_headline_variants":["AutoSGNN: LLM-driven search beats hand-crafted spectral GNNs","Discovering spectral GNNs automatically, per graph type","LLM + evolution finds GNN architectures that beat human designs","AutoSGNN: automated design of spectral GNNs for any graph","One-size-fits-all GNNs fail: AutoSGNN tailors per dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that validation accuracy on a 2.5% node split—about 68 nodes for Cora—is a stable and fair fitness signal for AutoSGNN, which searches per dataset on that split, against baselines whose hyperparameters were taken from their original papers rather than re-tuned on this split.","fun_headline_variants_meta":{"raw":{"variants":["AutoSGNN: LLM-driven search beats hand-crafted spectral GNNs","Discovering spectral GNNs automatically, per graph type","LLM + evolution finds GNN architectures that beat human designs","AutoSGNN: automated design of spectral GNNs for any graph","One-size-fits-all GNNs fail: AutoSGNN tailors per dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00045,"raw_usage":{"total_tokens":2242,"prompt_tokens":895,"completion_tokens":1347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":1251}},"tokens_in":511,"tokens_out":1347,"duration_ms":9606,"temperature":1.0,"reasoning_tokens":1251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:02:45.408950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full AutoSGNN search independently many times on the same nine datasets with different random seeds, and re-tune all baseline hyperparameters on the same 2.5% validation split used for the search. If the rank-1 sparse-split result is not reproduced, or if a re-tuned baseline matches or exceeds AutoSGNN's mean accuracy, the paper's central performance claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the unified optimization view of spectral GNNs that justifies decomposing the search space into feature fitting and Laplacian regularization terms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NFGNN is a key baseline and the source of the experimental protocol, hyperparameter choices, and sparse-split setting the paper follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPRGNN serves as a strong human-designed spectral GNN baseline and one of the elite individuals used to initialize the population."},{"cited_title":"N.; and Welling, M","cited_arxiv_id":null,"evidence_quote":"GCN is a classic baseline and an elite initializer for the population."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SANE is a differentiable GNN-NAS baseline compared on accuracy and search time."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GTFGNAS is an evolutionary GNN-NAS baseline whose experimental setup the paper adopts for NAS comparisons."},{"cited_title":"D.; Ermon, S.; and Finn, C","cited_arxiv_id":null,"evidence_quote":"Direct preference optimization inspires the C1 contrastive prompt that sets LLM preferences between high- and low-scoring elites."}],"review_version":1}