{"id":"b1109c81-ce87-4203-81d5-51c9f6705fdb","arxiv_id":"2505.04300","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across 15 pathway-informed deep learning models, randomized pathway annotations matched or beat the biologically informed versions, suggesting that observed gains come from sparsity rather than biological content.","lead":"This paper asked whether adding real biological pathway information to neural networks improves their predictions, or whether the benefit is just from making the network sparse. Across 15 published models, random pathway-like connections performed as well as, and sometimes better than, the real biological ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Randomization preserves only column/pathway degrees (Eqs. 4-5), not row/gene degrees or hub structure, so equal random performance does not isolate biological content from topology.","rationale":"The reader's conditional verdict is appropriate, and the weakest assumption it names is the one I would stress. The central claim depends on a null that changes only biological labels; the provided randomization changes row-degree distribution and graph topology, so the headline inference is not yet fully supported. This is not an objection to the empirical null result itself—15 models with 20 seeds is a useful benchmark—but to the causal reading 'sparsity is all you need.' The proposed degree-preserving swap test is the missing control and would settle the issue. Because the manuscript is transparent about its protocol and already reports additional limitations (e.g., only five models in the extended analyses, no multiple-testing correction), I would keep the CONDITIONAL verdict rather than reject: the core comparison can be strengthened without overturning the null.","tokens_in":17196,"tokens_out":4774,"duration_ms":54054,"concrete_test":"Re-run the comparison (at minimum the five models in the extended analyses plus one MLP-like, one GNN, and one CNN model) with a bipartite degree-preserving null: generate randomized gene-pathway matrices by repeated 2-edge swaps that keep both row and column sums exactly equal to the biological matrix, or by a configuration model conditioned on the full degree sequence, before constructing network layers, graphs, or pathway images. Keep the 20-run, same-hyperparameter protocol and paired Wilcoxon/K-S tests. If random versions still match or beat pathway-informed versions under this stricter null, the sparsity-only claim survives; if the performance gap disappears or reverses, the current conclusion is confounded by degree/hub changes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference is: randomized pathway assignments perform as well as or better than biological ones, therefore the benefit of pathway integration is sparsity, not biological relevance. For that inference, randomization must discard biological meaning while leaving every other performance-relevant property fixed. The described null (Additional Information Eqs. 4-5) preserves total edge count C and per-pathway degree d_j only. It does not preserve the per-gene degree sequence, the largest connected component, diameter, or assortativity—structural quantities the paper itself tabulates in Tables A4 and A6. In an implemented pathway layer, changing a gene's outgoing degree changes that neuron's fan-out, gradient flow, and effective capacity; in GNN/CNN variants, random 'pathway images' have different row statistics and neighborhood structure. A randomized network is therefore not the same sparse architecture with labels permuted; it is a different sparse graph. The observed equivalence or random advantage could be a degree-distribution or hub-structure effect rather than evidence that biological labels are uninformative. The title 'sparsity is all you need' overstates what this null can establish. The 30-seed trial checks seed stability, not whether the null preserves the right structure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Through a literature review of pathway-informed deep learning models, the authors identify 20 models with public code and compare 15 of them against randomized counterparts that permute pathway-feature associations while preserving, according to the stated formalization, the total number of connections and per-pathway degrees. Across tasks and metrics, they observe no significant performance advantage for pathway-informed models, with randomized versions significantly better in 3 of 15 cases. Additional experiments on five models examine fixed-seed robustness, sparsity levels, feature ablation, and biomarker interpretability. The paper proposes a benchmark workflow for testing whether pathway priors add value beyond sparsity.","tokens_in":17355,"tokens_out":8931,"duration_ms":85780,"significance":"If the core comparison were established with a null that fully controls all non-biological structural properties, the conclusion would be important: performance gains attributed to pathway knowledge in current architectures may be primarily a sparsity artifact, and the proposed randomized-baseline benchmark would be a useful community resource. The paper has concrete strengths: it evaluates 15 existing models with their original implementations, reports a 30-seed robustness check for five models, provides public code for the randomization procedure, and its central comparison is a direct empirical test rather than a circular derivation. The main caveats concern the null model's structural fidelity and multiple-testing interpretation, both of which are addressable in revision.","major_comments":[{"comment":"The randomization null preserves only the total number of connections C and the per-pathway degrees d_j; it does not preserve the per-gene degree sequence, hub structure, largest connected component, diameter, or assortativity, which are quantities the paper itself tabulates for the biological graphs in Tables A4 and A6. The Results statement that randomization preserves 'network sparsity and structural integrity' is therefore stronger than the formal null. In an MLP pathway layer, per-pathway fan-in is fixed but gene fan-out changes, altering gradient flow and effective capacity; in the GNN and CNN variants, the randomized graphs and images differ in row statistics and neighborhood structure. The randomized network is thus a different sparse graph rather than the same architecture with biological labels permuted, so the observed equivalence does not isolate 'biological relevance' as the discarded factor. The 30-seed trial checks seed stability, not whether the null preserves the relevant structure. Please use a row-and-column degree-preserving null (e.g., a curveball or swap null) for the MLP case and an analogous structural null for the GNN and CNN inputs, or show that the randomized graphs match the Tables A4 and A6 statistics and that residual differences do not affect generalization.","section":"Additional Information, Eqs. (4)-(5); Results, Pathway-Informed vs. Randomized Models"},{"comment":"The 15 model comparisons are not adjusted for multiple testing. At the nominal p < 0.05 level, roughly 0.75 false positives are expected among 15 independent tests, so the '3 out of 15' randomized wins could arise under the global null and should not be stated as evidence that randomized models outperformed pathway-informed ones without a correction. Please report adjusted p-values (e.g., Benjamini-Hochberg) or a permutation-based family-wise error control, and state explicitly how the abstract's 3-of-15 claim is affected by the adjustment.","section":"Results, Pathway-Informed vs. Randomized Models; Table 2; Figure 3"},{"comment":"The interpretability comparisons lack chance baselines. The DeepKEGG top-100 overlap (21 vs. 20 tumor-related features) should be compared with the expected overlap for a random top-100 feature set; the PINNet SHAP test shows that AD-related genes contribute within each model, not that the pathway-informed model's ranking is more concordant with known biology than a random model; and the PASNet and BINN Spearman correlations (0.4 and 0.56) have no null distribution. Without these baselines, the conclusion that pathway information provides 'no clear advantage in interpretability' is not established.","section":"Results, Comparison of Biological Information Extracted by Pathway-Informed Models and Randomized Counterparts; Figure…"}],"minor_comments":[{"comment":"The phrase 'the benefits of pathway integration does not arise' should agree in number ('do not arise').","section":"Abstract"},{"comment":"The claim of covering 'all relevant pathway-based neural network models' is too strong given that GCN-MAE, GCS-Net, PathDeep, ReGeNNe, and PGLCN could not be included in the comparison; please qualify the comprehensiveness claim.","section":"Background & Summary"},{"comment":"Specify how multiple configurations (OP, PP, OP+PP) for P-NET, BINN, DeepKEGG, GraphPath, and Pathformer are counted in the '15 models' total and in the '3 of 15' significant results.","section":"Table 2"},{"comment":"The test-set comparison of sparsity levels is acknowledged as post hoc, but the subsequent significance statements should be labeled exploratory because selecting the best sparsity level on the test set can capitalize on chance.","section":"Methods, Optimal Sparsity Level"}],"recommendation":"major_revision","confidential_remarks":"The paper is a substantial benchmark study and the proposed workflow is useful. My main reservation is the null model: the current randomization preserves only column degrees, so the central 'sparsity, not biology' claim is not yet cleanly identified. If the authors implement a degree-preserving null and correct for multiple testing, I would support publication. I also recommend verifying that the released code enforces the constraints stated in Eqs. (4)-(5) for all model types."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful benchmark, and the central negative result—randomized pathway assignments perform as well as the real ones—holds up in the limited sense tested. The paper deserves peer review, but the title and some conclusions overstate what the null can establish.\n\nWhat's new: a systematic comparison of 15 published pathway-informed models using their original code and data, with a common randomization protocol. That's a real contribution. The 30-seed robustness check on five models is a good control, and the sparsity-sweep analysis is a nice addition. I also credit the authors for putting the randomization code on GitHub.\n\nWhere the paper is soft: the main null model preserves only the total connection count and per-pathway degree. It does not preserve the gene degree distribution or hub structure. So the randomized networks are not the same sparse architecture with labels permuted; they are different sparse graphs. This means the equivalence does not cleanly isolate biological content as the discarded factor. A randomized graph with a very different gene-degree sequence might generalize differently for reasons that have nothing to do with biological meaning. The stress-test note is right on this point. The paper should address it directly—either by using a bipartite configuration model that preserves both row and column degrees, or by reporting the degree distribution of the randomized graphs and showing it matches the real ones. Without that, 'sparsity is all you need' is too strong.\n\nOther issues are more minor: 15 comparisons with no multiple-testing correction; the interpretability claims lack chance baselines (21 vs 20 tumor-related genes needs a null expectation); and only five models enter the extended analyses, so the sparsity-level conclusion is based on a subset. None of these overturn the main observation, but they temper the conclusions.\n\nWho's the audience: anyone building or evaluating pathway-informed models. The randomized baseline should become a standard control. I'd bring this to a reading group—it's a good discussion piece about what counts as a fair null.\n\nRecommendation: send it to peer review. With revision—especially on the null model and the interpretive claims—this could be a solid methods paper.","headline":"A genuinely useful systematic benchmark, but the randomization null does not isolate biological content from topology; the paper deserves peer review after reworking its claims and adding structural checks.","tokens_in":17922,"tokens_out":3177,"would_cite":true,"duration_ms":32793,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing biological pathway priors with random priors that preserve network sparsity leaves predictive performance unchanged in deep learning models, so the benefits attributed to pathway knowledge are mostly a…","keywords":["pathway-informed neural networks","sparsity","randomization null model","deep learning benchmark","omics data","interpretability","biological priors","gene-pathway associations"],"falsifier":"Train a single pathway-informed model on a dataset where the true pathway graph is known to be functionally important, and compare it with an ensemble of random graphs matched not only on total edges and per-pathway degree but also on the full feature-degree sequence and largest-component structure; if the true graph consistently and significantly outperforms every matched randomization across repeated seeds on held-out data, the claim that biology adds nothing beyond sparsity is falsified.","tokens_in":16964,"feed_emoji":"🧬","tokens_out":6184,"duration_ms":60187,"temperature":0.7,"pith_summary":"This paper tests whether adding biological pathway knowledge to deep learning models improves predictions, or whether the apparent benefit is just an artefact of the sparser network that pathway constraints create. It re-runs 15 published pathway-informed models on their original tasks, replacing true pathway connections with random connections that preserve the same count of connections, then compares performance. Across all models and metrics, randomized versions performed as well as, and in three cases significantly better than, the biologically informed originals. A follow-up sparsity sweep shows that pathway-selected sparsity levels are not even the optimal levels for these models, and randomized models still recover the same disease biomarkers. The central conclusion is that performance gains attributed to biological priors should be benchmarked against degree-preserving randomizations before they are credited to biology.","feed_headline":"Randomized pathways match real ones across 15 deep-learning models","feed_subtitle":"When connection counts are preserved, models with random pathway priors perform equally or better than biologically informed ones","key_machinery":"The load-bearing object is the null model generated by the randomization procedure. Starting from the binary gene–pathway association matrix $A$, the method permutes the 1-entries while preserving the total number of connections $C$ and each pathway's column sum $d_j$ (Equations 4 and 5), then uses the shuffled matrix in place of the real pathway prior in the model's architecture or input representation. This isolates the sparsity induced by pathways from the specific biological identities of the connections. A second knob varies sparsity level directly, pruning 60–99% of connections, to test whether biological levels are optimal. The comparison metric is whether the randomized or pruned networks match or beat the biologically constrained ones.","core_discovery":"The central claim is that pathway-informed neural networks owe their reported predictive success to the sparse connectivity that pathway annotations impose, not to the biological content of those annotations. The evidence is a controlled comparison: for each of 15 state-of-the-art models with available code, the authors randomize the gene–pathway matrix while preserving the total number of connections and the number of features assigned to each pathway, leaving the network sparsity structure intact. Randomized models perform statistically indistinguishably from pathway-informed models on every task, and three models (MPVNN, DeepKEGG, PathDNN) do significantly better when the biological labels are shuffled. Interpreting feature importance with SHAP and related methods shows randomized models recover the same disease-associated genes as the original models. The paper also proposes a general benchmarking workflow whereby any new pathway-informed architecture must beat its randomized counterpart before claiming a biological advantage.","pith_inferences":["An implication the authors leave implicit is that a stronger null model would randomize edges while exactly preserving the feature-degree sequence and connected-component structure, separating hub topology from raw sparsity; the current randomization preserves total edges and per-pathway degree but not the full graph statistics.","Because pathway databases are static and incomplete, the result may not generalize to condition-specific or high-quality interaction networks; repeating the comparison with PPI-derived or single-cell-specific priors is a natural testable extension.","A practical consequence beyond the paper is that model developers could save effort and avoid dependence on annotation databases by using random sparse masks with tuned sparsity, at least until a biological prior demonstrates an advantage in the same benchmark.","The benchmark could become a standard control for publications proposing biology-informed architectures, analogous to a permutation test in statistics."],"forward_implications":["Reported gains from pathway integration in existing deep learning models cannot be attributed to biological relevance unless the model beats a degree-preserving randomized prior.","Sparsity should be treated as a tunable hyperparameter rather than a fixed property dictated by pathway annotations.","Interpretability claims based on pathway-informed models need re-examination, because randomized models recover similar biomarkers.","The proposed randomization benchmark can be applied across omics tasks and data modalities, including MLP-based, CNN-based, GNN-based, and transformer-based architectures.","Future pathway-informed designs should test richer priors, such as protein–protein interaction networks or condition-specific pathway information, to see whether they outperform sparsity alone."],"supporting_citations":[{"why":"Provides PASNet, one of the benchmarked models whose randomized version matched the original.","marker":"[8]"},{"why":"Provides DeepKEGG, a multi-omics model whose randomized version significantly outperformed the pathway-informed one.","marker":"[19]"},{"why":"Provides PINNet, the Alzheimer's model used in randomization, sparsity-sweep, and biomarker analyses.","marker":"[17]"},{"why":"Provides PathCNN, the pathway-image CNN used in the randomized-image comparison and sparsity experiments.","marker":"[29]"},{"why":"Provides BINN, a proteomics model benchmarked and used for interpretability correlation.","marker":"[11]"},{"why":"Provides MPVNN, a survival model whose randomized version outperformed the pathway-informed one.","marker":"[33]"},{"why":"Provides P-NET, a benchmarked prostate-cancer model in the comparison.","marker":"[6]"},{"why":"Supplies Reactome pathway annotations, the most common prior being randomized.","marker":"[27]"},{"why":"Supplies KEGG pathway annotations used by DeepKEGG, PathCNN, and PINNet.","marker":"[16]"},{"why":"Supplies the SHAP framework used to measure feature importance in both model versions.","marker":"[25]"}],"fun_headline_variants":["Pathway-informed neural nets: it's the sparsity, not the biology","Randomizing pathway priors matches biology in deep models","Sparsity, not biology, drives pathway-model performance","Biological priors in deep nets: a sparsity trick?"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that shuffling pathway labels removes biological meaning while leaving every other performance-relevant property unchanged; in practice the randomization preserves only total edge count and per-pathway degree, so changes in gene degree distribution, hubs, and connectivity could be doing some of the work attributed to sparsity.","fun_headline_variants_meta":{"raw":{"variants":["Pathway-informed neural nets: it's the sparsity, not the biology","Randomizing pathway priors matches biology in deep models","Sparsity, not biology, drives pathway-model performance","Biological priors in deep nets: a sparsity trick?"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2842,"prompt_tokens":940,"completion_tokens":1902,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":1830}},"tokens_in":556,"tokens_out":1902,"duration_ms":14111,"temperature":1.0,"reasoning_tokens":1830,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:33:13.906398+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a single pathway-informed model on a dataset where the true pathway graph is known to be functionally important, and compare it with an ensemble of random graphs matched not only on total edges and per-pathway degree but also on the full feature-degree sequence and largest-component structure; if the true graph consistently and significantly outperforms every matched randomization across repeated seeds on held-out data, the claim that biology adds nothing beyond sparsity is falsified.","supporting_citations":[{"cited_title":"The reactome pathway knowledge- base 2024","cited_arxiv_id":null,"evidence_quote":"Supplies Reactome pathway annotations, the most common prior being randomized."},{"cited_title":"Kegg for taxonomy-based analysis of pathways and genomes.Nu- cleic Acids Research, 51(D1):D587–D592, 10 2023","cited_arxiv_id":null,"evidence_quote":"Supplies KEGG pathway annotations used by DeepKEGG, PathCNN, and PINNet."},{"cited_title":"Pasnet: pathway-associatedsparsedeepneural network for prognosis prediction from high-throughput data","cited_arxiv_id":null,"evidence_quote":"Provides PASNet, one of the benchmarked models whose randomized version matched the original."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides DeepKEGG, a multi-omics model whose randomized version significantly outperformed the pathway-informed one."},{"cited_title":"Kim and H","cited_arxiv_id":null,"evidence_quote":"Provides PINNet, the Alzheimer's model used in randomization, sparsity-sweep, and biomarker analyses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides PathCNN, the pathway-image CNN used in the randomized-image comparison and sparsity experiments."},{"cited_title":"Hartman, A.M","cited_arxiv_id":null,"evidence_quote":"Provides BINN, a proteomics model benchmarked and used for interpretability correlation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MPVNN, a survival model whose randomized version outperformed the pathway-informed one."},{"cited_title":"Gang, S.and Liu, S","cited_arxiv_id":null,"evidence_quote":"Provides P-NET, a benchmarked prostate-cancer model in the comparison."},{"cited_title":"A unified approach to interpreting model predictions","cited_arxiv_id":null,"evidence_quote":"Supplies the SHAP framework used to measure feature importance in both model versions."}],"review_version":1}