{"id":"bc45ab58-2347-4efb-8d09-bec095afe3b6","arxiv_id":"2505.10711","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A new benchmarking framework built on Nextflow shows graph neural networks outperform feature-only logistic regression across four cancer-gene network configurations.","lead":"This paper introduces GNN-Suite, a Nextflow-based workflow for comparing graph neural network models in biomedical research. Applied to cancer-driver gene classification, it reports that graph models beat a logistic regression baseline, and that the gene label source matters more than the interaction network.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported BACC values are each model's best test-set epoch, so rankings and GNN-vs-LR gaps are inflated by test-set overfitting.","rationale":"The reader's weakest assumption flags the absence of significance testing and multiple-comparison correction, which is a valid concern. My review identifies a more load-bearing methodological issue: the reported BACC values are selected at each model's peak test-set epoch, with no validation split described. This biases the means upward and distorts both architecture rankings and the GNN-versus-LR comparison. The concern is concrete and testable: fixing the evaluation protocol to a single checkpoint or validation-based early stopping would settle whether the reported best models and baseline improvements persist. The framework contribution itself, including the Nextflow pipeline and public code, is credible and not undermined by this statistical issue. However, the empirical demonstration in the case study should be re-run under a bias-free protocol before the comparative claims are accepted. Since the reader already reached a CONDITIONAL verdict, my analysis does not change that verdict, though it provides a sharper reason for the condition.","tokens_in":19280,"tokens_out":6071,"duration_ms":65133,"concrete_test":"Re-run the four network/label configurations with a single fixed evaluation protocol: train for 300 epochs and report test BACC at the final epoch only, or hold out 10% of the training nodes as a validation set and use it for checkpoint selection. Recompute the Figure 2 table with 10 seeds per model. If GCN2 remains top on STRING-PID by more than two standard deviations and every GNN still exceeds LR by more than two standard deviations, the concern is resolved; otherwise the reported rankings and baseline comparisons must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The table in Figure 2 states that 'The epoch corresponding to the highest mean balanced accuracy (BACC) is reported.' The Methods describe only an 80/20 train-test split with no held-out validation set, so the epoch column is selected by monitoring test-set performance during training. Each architecture's reported mean ± std is therefore the maximum over its 300-epoch test-BACC curve, not the performance of a fixed training protocol. This mechanically favors unstable models (e.g., GTN, which peaks at epoch 63 and then declines) and inflates every GNN's reported BACC relative to logistic regression, which has no epoch-selection step. The ranking of GCN2, HGCN, GTN, and GIN, and the claim that every GNN outperforms LR, are not trustworthy as reported. This is more fundamental than the absence of significance tests: significance tests on selected maxima would not remove the bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GNN-Suite, a Nextflow-based benchmarking framework for graph neural networks aimed at biomedical applications, and demonstrates it on cancer-driver gene classification. Models are evaluated on four network-label configurations built from STRING/BioGRID PPI data and PCAWG-derived node features, with PID/COSMIC labels. Eight GNN architectures (GCN, GAT, GAT3H, GCN2, GIN, GTN, HGCN, PHGCN, GraphSAGE) are compared against logistic regression under standardized hyperparameters, ten seeds, and balanced accuracy as the primary metric. The paper reports that all GNNs outperform the LR baseline and identifies GCN2, HGCN, GIN, and GTN as top performers depending on the configuration. The main contribution claimed is reproducible, modular benchmarking; the case study is provided as a demonstration of the framework.","tokens_in":19470,"tokens_out":2340,"duration_ms":25499,"significance":"If the central claims are taken as stated, the framework is a useful, reproducible contribution: it provides a public GitHub repository, Docker image, uniform hyperparameter configuration, ten-seed evaluation, and modular Nextflow design, which are genuine strengths for the biomedical GNN community. However, the empirical claims of the case study—that all GNNs beat logistic regression and that specific architectures are best—rest on an evaluation protocol that selects each model's best test-set epoch, which introduces a systematic bias. The paper also does not provide significance testing, so the reported rankings and the GNN-over-baseline claim are not statistically supported. The framework's design is defensible and likely useful, but the case-study conclusions need reworking rather than just polishing.","major_comments":[{"comment":"The reported BACC values are the maximum over test-set epochs, not the performance of a fixed training protocol. The Methods state an 80/20 train-test split with no held-out validation set, and Figure 2's table explicitly says 'The epoch corresponding to the highest mean balanced accuracy (BACC) is reported' while the caption says models were evaluated on held-out test nodes. Selecting the best test epoch per model inflates each GNN's apparent performance and mechanically advantages unstable models such as GTN, which peaks early and then declines. This bias directly affects the central claims that 'all GNN types outperformed the LR baseline' and that GCN2/HGCN/GIN are the best models, because logistic regression has no epoch-selection step. The evaluation must be changed to a fixed number of epochs or early stopping on a validation set, with test performance reported only once.","section":"Figure 2 and Methods (Model Configuration & Training)"},{"comment":"No statistical significance testing accompanies the model comparisons. In the table, differences between the top models are frequently within one standard deviation (e.g., STRING-PID GCN2 0.807±0.035 vs HGCN 0.802±0.028 and GTN 0.798±0.036; STRING-COSMIC GCN2 0.680±0.030 vs PHGCN 0.678±0.024), so the ordering is consistent with chance. The abstract's phrase 'significant improvement' is unsupported. The authors should report confidence intervals on pairwise differences, use a paired test across the ten seeds or repeated cross-validation, and correct for multiple comparisons across eight architectures and four datasets. Without this, the ranking and the claim that all GNNs beat baseline are not established.","section":"Results / Figure 2"},{"comment":"The epoch column is described as the 'minimum epoch number needed for convergence', but the entries are the epochs at which the highest mean BACC occurred (e.g., GTN at epoch 63 with subsequent decline). These are different quantities, and the text in the Results section also conflates them ('the highest mean BACC values alongside the minimum epoch number needed for convergence'). The authors should define separately the convergence epoch (e.g., when BACC first reaches a plateau) and the reported test epoch, or eliminate the convergence claim.","section":"Results (Table in Figure 2)"}],"minor_comments":[{"comment":"The abstract uses 'significant improvement' and 'statistically robust performance metrics' without any statistical testing; this wording should be softened or supported by the significance analysis requested above.","section":"Abstract / Introduction"},{"comment":"The architecture labeled GTN is described as a TransformerConv operator from Shi et al. (2021), but 'GTN' typically refers to Graph Transformer Networks. The naming should be clarified to avoid confusion, especially since Figure 2 refers to GTN.","section":"Appendix S3.8"},{"comment":"The text says 'By monitoring both the training and validation loss, we can select the optimal moment to stop training', but the described experiments do not use a validation set. This statement should be reconciled with the actual protocol or moved to the framework's future capabilities.","section":"Appendix S5"},{"comment":"The caption reads 'Evaluation metrics on the test set during training, including the (training) loss...' which is internally contradictory; the loss is presumably the training loss while the other curves are test-set metrics, and this should be stated clearly.","section":"Figure S1 caption"},{"comment":"Equations (2) and (3) contain inconsistent index notation (e.g., tilde d subscripts in Eq. (3) are not defined in the same form as in Eq. (2)) and would benefit from a careful rewriting pass.","section":"General notation"},{"comment":"The claim that the underlying PPI network source had a 'comparatively minor effect' is based on visual inspection of Table 1/Figure 2; a quantitative comparison of the distributions across the four configurations would be more appropriate.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The framework itself is a reasonable engineering contribution and the publication of the Nextflow pipeline with Docker support is valuable. However, the case study is presented as evidence for substantive claims about GNN superiority and architecture rankings, and the test-set epoch selection is a load-bearing flaw that cannot be fixed by adding significance tests alone. The revision should either change the evaluation protocol or substantially reframe the case study as a demonstration of workflow capabilities rather than a head-to-head benchmark. If the protocol is fixed, the empirical claims may still hold, but as written the main scientific conclusions are premature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: GNN-Suite is the first GNN benchmark built on Nextflow, and that is a sensible thing to exist. It wraps eight PyG architectures plus logistic regression in a modular, containerized pipeline where users swap config files rather than code. The case study—cancer-driver prediction on STRING/BioGRID with PCAWG-derived features—is clean in intent: standardized two-layer models, uniform hyperparameters, ten seeds, balanced accuracy. The framework should be useful to bioinformaticians who want reproducible GNN comparisons without writing glue code.\n\nThe empirical results, however, are not as solid as the presentation suggests. Figure 2 reports for each model \"the epoch corresponding to the highest mean balanced accuracy.\" Since the methods describe only an 80/20 train-test split with no held-out validation set, that means the reported numbers are the best test-set epoch per model, selected after seeing the test curve. That mechanically favors unstable models like GTN (which peaks early and declines) and inflates every GNN's BACC relative to logistic regression, which has no epoch selection. The gap between GNNs and LR, and the ranking GCN2/HGCN/GTN/GIN, are not trustworthy as reported. This is more serious than the absence of significance tests—though that absence is real too. Differences between top models are often within one standard deviation, and there's no correction for comparing eight architectures across four datasets. \"Statistically robust\" is an overstatement.\n\nReproducibility is partly addressed—Docker image, Nextflow profiles, public repo—but no commit hash or processed data files are given, so the exact experiments aren't pinned down.\n\nIf the authors fix the evaluation protocol (use a validation split for early stopping, report final-epoch performance, add significance tests), the case study becomes a useful worked example. As is, the framework deserves attention; the comparative claims should be read with caution.\n\nFor a journal, I'd send it to reviewers. The engineering is solid and the flaw is correctable. For a reading group, it's a maybe—good for discussing evaluation pitfalls in benchmarking papers.","headline":"Useful Nextflow-based GNN benchmarking wrapper, but the headline model-comparison claims are inflated by selecting the best test-set epoch and lack significance tests.","tokens_in":19972,"tokens_out":1943,"would_cite":false,"duration_ms":19696,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GNN-Suite standardizes GNN benchmarking with Nextflow and shows, on four cancer-driver network configurations, that every evaluated GNN outperforms a logistic regression baseline, with GCN2 reaching the highest balanced accuracy of 0.807…","keywords":["GNN-Suite","graph neural networks","benchmarking","Nextflow","cancer driver genes","node classification","protein-protein interaction networks","balanced accuracy"],"falsifier":"Re-run the same ten-seed protocol and apply a paired test between each GNN and logistic regression on each configuration, with multiple-comparison correction; if most GNN-versus-baseline differences become non-significant, or if a feature-only model with tuned hyperparameters closes the gap, the central claim that network structure helps would not survive.","tokens_in":19104,"feed_emoji":"🧬","tokens_out":7349,"duration_ms":67170,"temperature":0.7,"pith_summary":"GNN-Suite is a standardized benchmarking pipeline for graph neural networks in biomedical research, built on the Nextflow workflow system so that models, datasets, and training setups can be swapped without changing the evaluation protocol. The paper's central demonstration is a cancer-driver-gene classification task: from STRING and BioGRID protein-protein interaction networks, with PCAWG-derived mutation features and PID or COSMIC driver labels, every GNN architecture tested beat a logistic regression baseline on all four network-label configurations. On the STRING-PID configuration, GCN2, a residual graph convolutional network, reached the highest balanced accuracy of 0.807 ± 0.035, and gains over the baseline reached 8.9% for two of the four configurations. A sympathetic reader would take the main claim to be that network-based message passing adds predictive signal beyond mutation features alone, and that a common training protocol makes architecture differences visible in a way scattered implementations cannot.","feed_headline":"Graph nets beat logistic regression on every cancer-driver test","feed_subtitle":"GNN-Suite benchmark finds GCN2 best at 0.807 balanced accuracy on STRING-PID, with network signal over features.","key_machinery":"The load-bearing object is GNN-Suite: a Nextflow workflow in which a fixed training recipe — two convolutional layers, dropout 0.2, Adam with learning rate 0.01, class-weighted binary cross-entropy, 80/20 split, 300 epochs, and 10 random seeds — is applied to every architecture, with balanced accuracy (the average of sensitivity and specificity) as the primary metric. The case study supplies four graph configurations: STRING or BioGRID interaction networks, node features from Fisher-combined PCAWG mutation p-values, and positive labels from PID or COSMIC cancer-gene panels. This design isolates architecture behaviour from implementation differences.","core_discovery":"The paper claims that GNN-Suite enables fair, reproducible comparisons of GNN architectures and that, in its case study, network structure helps. Using standardized two-layer models with shared hyperparameters and ten seeded runs, all evaluated GNNs outperformed logistic regression on every combination of PPI network (STRING or BioGRID) and driver-gene panel (PID or COSMIC). GCN2 was the top model on STRING-PID (0.807 ± 0.035) and STRING-COSMIC (0.68 ± 0.03); HGCN led BioGRID-PID (0.786 ± 0.040) and GIN led BioGRID-COSMIC (0.677 ± 0.027). The paper also argues the driver-gene panel choice had a larger effect on performance than the PPI source, with PID-labelled networks consistently outperforming COSMIC-labelled ones.","pith_inferences":["An extension left implicit: the PID advantage may reflect alignment between pathway-derived labels and PPI topology; this could be tested by re-labelling the same networks with panels that vary in pathway content.","Since hyperparameters were fixed across architectures, the reported ranking is a 'same-budget' comparison; per-architecture tuning could reorder the models.","The workflow's modular config-file structure should transfer directly to other biomedical node-classification problems, such as gene-disease association or drug-target prediction, where the same four configuration files define the experiment."],"forward_implications":["For cancer-driver prediction on STRING-like graphs, a residual GCN such as GCN2 is a strong default choice.","Graph structure carries predictive signal: even a plain logistic regression on features is consistently beaten by every message-passing model.","Driver-gene panel choice should be treated as part of model selection; PID-derived labels yield higher balanced accuracy than COSMIC-derived labels.","A shared Nextflow protocol can serve as a neutral substrate for future GNN comparisons, making architecture rankings comparable across studies."],"supporting_citations":[{"why":"workflow engine that lets GNN-Suite fix the training protocol across architectures","marker":"[8]"},{"why":"supplies the STRING protein-protein interaction network used in two of the four configurations","marker":"[42]"},{"why":"supplies the BioGRID interaction network used in the other two configurations","marker":"[28]"},{"why":"provides the PCAWG recurrent-mutation p-values from which node features are derived","marker":"[1]"},{"why":"provides the PID cancer-driver gene labels for the PID configurations","marker":"[31]"},{"why":"provides the COSMIC Cancer Gene Census labels for the COSMIC configurations","marker":"[38]"},{"why":"method used to convert mutation p-values into per-gene cancer-association scores","marker":"[11]"},{"why":"base graph convolutional architecture that GNN-Suite includes in the benchmark","marker":"[22]"},{"why":"GCN2, the architecture reported as achieving the best balanced accuracy on STRING-PID","marker":"[6]"}],"fun_headline_variants":["GNN-Suite: all graph nets beat logistic regression on cancer drivers","GNN-Suite benchmark: GCN2 tops cancer-driver prediction at 0.807","All GNNs outperform LR on every cancer-driver network in GNN-Suite","Cancer-driver prediction: network beats features, GCN2 top on STRING","GNN-Suite: standardized benchmarks show graph nets > LR on cancer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that GNNs beat logistic regression rests on treating the ten-run mean balanced accuracies as stable; the paper reports no significance tests, confidence intervals on pairwise differences, or correction for multiple comparisons across the architectures and four datasets.","fun_headline_variants_meta":{"raw":{"variants":["GNN-Suite: all graph nets beat logistic regression on cancer drivers","GNN-Suite benchmark: GCN2 tops cancer-driver prediction at 0.807","All GNNs outperform LR on every cancer-driver network in GNN-Suite","Cancer-driver prediction: network beats features, GCN2 top on STRING","GNN-Suite: standardized benchmarks show graph nets > LR on cancer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001047,"raw_usage":{"total_tokens":4472,"prompt_tokens":1092,"completion_tokens":3380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":3274}},"tokens_in":708,"tokens_out":3380,"duration_ms":21160,"temperature":1.0,"reasoning_tokens":3274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:04:31.513040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same ten-seed protocol and apply a paired test between each GNN and logistic regression on each configuration, with multiple-comparison correction; if most GNN-versus-baseline differences become non-significant, or if a feature-only model with tuned hyperparameters closes the gap, the central claim that network structure helps would not survive.","supporting_citations":[{"cited_title":"Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame","cited_arxiv_id":null,"evidence_quote":"workflow engine that lets GNN-Suite fix the training protocol across architectures"},{"cited_title":"Gable, Katerina C","cited_arxiv_id":null,"evidence_quote":"supplies the STRING protein-protein interaction network used in two of the four configurations"},{"cited_title":"The BioGRID database: A comprehensive biomedical resource of curated protein, genetic, and chemical interactions","cited_arxiv_id":null,"evidence_quote":"supplies the BioGRID interaction network used in the other two configurations"},{"cited_title":"Aaltonen, Federico Abascal, Adam Abeshouse, Hiroyuki Aburatani, David J","cited_arxiv_id":null,"evidence_quote":"provides the PCAWG recurrent-mutation p-values from which node features are derived"},{"cited_title":"Reyna, David Haan, Marta Paczkowska, Lieven P","cited_arxiv_id":null,"evidence_quote":"provides the PID cancer-driver gene labels for the PID configurations"},{"cited_title":"Cole, Sari A","cited_arxiv_id":null,"evidence_quote":"provides the COSMIC Cancer Gene Census labels for the COSMIC configurations"},{"cited_title":"Discovering cancer driver genes and pathways using stochastic block model graph neural networks, June 2021","cited_arxiv_id":null,"evidence_quote":"method used to convert mutation p-values into per-gene cancer-association scores"},{"cited_title":"Kipf and Max Welling","cited_arxiv_id":null,"evidence_quote":"base graph convolutional architecture that GNN-Suite includes in the benchmark"}],"review_version":1}