{"id":"60649c43-92a9-4fe8-b4dd-d0c31c472400","arxiv_id":"2505.01696","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A technical review plus an empirical comparison of four interpretability methods on a graph-based multi-omics Alzheimer's classifier, showing method-specific biological and computational trade-offs.","lead":"This preprint reviews 26 papers on interpretable graph-based models for multimodal biomedical data and then benchmarks four explainability methods on an Alzheimer's disease multi-omics graph model. SHAP and sensitivity analysis recover more known disease pathways, while gradient saliency and graph masking surface complementary metabolic and transport signatures, at different computational costs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark ranking of XAI methods rests on post hoc 'Alzheimer-related' GO/pathway labels; a pre-registered AD gene-set panel is needed to support the claim.","rationale":"The paper's only novel empirical contribution is the benchmark in Section 5.5, and its strongest claim is that SHAP and sensitivity analysis 'recover the broadest set of known AD pathways and Gene-Ontology terms.' The evaluation metric is the number of enriched GO/pathway terms manually labeled as AD-related. This metric is not defined a priori, is applied with a raw p-value threshold (p≤0.01) despite the Methods section stating FDR, and depends on a subjective post hoc annotation of generic GO terms (e.g., 'temperature homeostasis') as AD-related. If this annotation is replaced by a fixed, independently curated AD gene-set panel, the ranking of methods could change; no external validation currently supports the ranking. The absence of any reported classification performance for the replicated MOGONet model further weakens the biological interpretability of the feature-importance rankings. These issues do not invalidate the review's taxonomy or the flowchart, but they make the benchmark's headline conclusion conditional. The proposed concrete test—re-running the enrichment analysis against a pre-registered AD panel with a permuted-label negative control—would directly settle whether the reported ranking is a property of the explanation methods or of the post hoc term selection. Pending that test, the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":24107,"tokens_out":9070,"duration_ms":83983,"concrete_test":"Re-run the Section 5.5 comparison with a pre-registered AD gene-set panel (curated AD genes from DisGeNET/Open Targets plus KEGG Alzheimer's disease pathway). For each XAI method's top 30 genes, compute one-sided hypergeometric enrichment against this fixed panel with FDR control; include a negative-control model trained on permuted labels. If SHAP and sensitivity analysis do not significantly beat gradient saliency and graph masking on the fixed panel, or if the permuted-label model also enriches, the reported ranking is an artifact of post hoc GO-term annotation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's headline conclusion (SHAP and sensitivity analysis recover the broadest set of known AD pathways/GO terms) is a ranking of methods by enrichment counts computed in Section 5.5. This ranking is only meaningful if the outcome variable—'Alzheimer-related GO terms'—is defined objectively and in advance. The paper never provides such a definition. In Section 5.5, the authors state: 'SHAP identified ten out of eighteen Alzheimer-related GO terms (unique GO terms identified by all methods)' and similar for pathways, but these eighteen terms are the union of terms enriched at p≤0.01, and the 'Alzheimer-related' label appears to be assigned post hoc (e.g., 'temperature homeostasis', 'endocrine system development', 'ensheathment of neurons'). The Methods section says enrichment was prioritized by FDR (p<0.01), but the benchmark section uses raw p-values ≤0.01, and no multiple-testing correction is described. Because the same post hoc annotation is applied to all methods, the comparison is not a test of explanation quality; it is a test of which method's top 30 genes happen to enrich for terms the authors recognize as AD-related. The ranking could change if a pre-registered, independent AD gene-set panel were used instead. Additionally, no model classification performance is reported, so it is unknown whether the explained model is predictive at all. These issues directly undermine the strongest empirical claim of the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript combines a technical literature review with a small benchmarking study. The review surveys 26 papers (2019--September 2024) that apply interpretable graph-based models to multimodal biomedical data, classifies interpretability approaches into four families (modality/feature elimination, non-graph XAI, graph-based XAI, inherent interpretability), and summarizes trends in graph construction and model architectures. The benchmark replicates MOGONet on the ROSMAP Alzheimer's disease dataset and compares gradient saliency, sensitivity analysis, SHAP, and graph masking by ranking the top 30 genes from each method, testing those gene sets against random gene sets via permutation, and counting enrichment of Gene Ontology and pathway terms. The paper's central empirical claim is that SHAP and sensitivity analysis recover the broadest set of known AD-associated GO terms and pathways, while gradient saliency and graph masking surface complementary metabolic and transport signatures, with all four methods beating random gene sets.","tokens_in":1586,"tokens_out":1922,"duration_ms":48802,"significance":"If the benchmark's ground truth were established objectively, the paper would provide a useful, reproducible comparison of explainability methods for graph-based multimodal biomedical models, and the proposed four-category taxonomy plus the design flowchart would be a practical reference for practitioners. The review fills a plausible gap at the intersection of interpretability, graph-based modeling, and multimodal biomedical data, and it is one of few works to combine a literature survey with a concrete benchmark. Strengths include the transparent PRISMA-style selection process, the public availability of code and data on GitHub, and the use of permutation testing as a sanity check for gene rankings. The main weakness is that the benchmark's outcome measure (\"Alzheimer-related\" GO/pathway terms) is defined post hoc, which makes the headline ranking dependent on the authors' subjective labeling rather than an independent, pre-specified ground truth.","major_comments":[{"comment":"The benchmark's central ranking of SHAP and sensitivity analysis over gradient saliency and graph masking is based on counts of \"Alzheimer-related GO terms\" and \"pathway terms\" that are not defined independently of the enrichment results. Section 5.5 states that SHAP identified ten of eighteen \"Alzheimer-related GO terms (unique GO terms identified by all methods)\" and seven of fifteen pathway terms, but these eighteen and fifteen terms are the union of terms enriched at p ≤ 0.01 by any method, and the \"Alzheimer-related\" label is applied post hoc in the text (e.g., \"temperature homeostasis\" and \"endocrine system development\"). This makes the enrichment count a circular measure: the outcome variable is constructed from the same enrichment results being ranked. In addition, the Methods section says terms were prioritized by FDR (p < 0.01), while the benchmark retains terms with raw p-values ≤ 0.01, and no multiple-testing correction is described. Please define an objective, pre-specified AD gene or term panel (e.g., from DisGeNET, OMIM, or a curated pathway database) and use a consistent significance threshold with appropriate correction before claiming the method ranking.","section":"Section 5.5, Figure 8c/d; Methods (\"Gene ontology and pathway analyses\")"},{"comment":"No classification performance is reported for the replicated MOGONet model on the ROSMAP test set. The Methods describe the training protocol (two-phase training, learning rates, epochs) but do not give accuracy, AUROC, F1, or any comparison to the original MOGONet results. If the replicated model does not predict AD status well, the feature attributions produced by the four XAI methods may reflect a poorly fitted or non-predictive model, undermining the biological interpretation of the top genes. Please report the model's discriminative performance on the held-out test set (and, ideally, a comparison with the original MOGONet publication), and state whether the explained model is the one used for the enrichment analysis.","section":"Section 5.5 (\"Benchmarking XAI approaches on MOGONet\") and Methods"}],"minor_comments":[{"comment":"The Methods state that all GO terms and pathways were prioritized based on False Discovery Rate (FDR) values (p < 0.01), but Section 5.5 and Figure 8c/d retain terms with raw p-values ≤ 0.01 without any multiple-testing correction. Please reconcile the threshold and clarify whether FDR or raw p-values were used.","section":"Methods (\"Gene ontology and pathway analyses\") vs. Section 5.5"},{"comment":"The Figure 8a caption says \"SHAP and gradient saliency were roughly able to detect a combination of the most important genes found by other techniques,\" but the main text states that SHAP and sensitivity analysis share more genes, while gradient saliency highlights distinct genes (NPNT, SYTL1, ANKRD30B). The caption should match the text.","section":"Figure 8 caption"},{"comment":"The name \"Pfiefer\" appears in Sections 5.2 and 5.4 (e.g., \"Zhang et al. [28] and Pfiefer et al. [30]\"), while the reference list and other occurrences use \"Pfeifer.\" Please standardize the spelling.","section":"General"},{"comment":"The search window is stated as \"January 1, 2019, and September 31, 2024\"; September has 30 days. Please correct the date to September 30, 2024.","section":"Section 2 (Methodology)"},{"comment":"The pathway analysis retention criterion \"an overlap > 1\" is undefined in the text. Please specify what \"overlap\" refers to (number of genes from the top-30 list in the pathway) and report how many pathways were considered before filtering.","section":"Section 5.5 and Figure 8d"},{"comment":"The claim that \"SHAP and graph masking required more processing time\" is only supported by a vague remark that generating the SHAP report \"could take several weeks\" in the Discussion. Since computational cost is one of the reported trade-offs, please provide concrete runtime measurements for all four methods on the same hardware.","section":"Section 6 (Discussion)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for the venue and the taxonomy is useful. The main empirical claim, however, rests on a post hoc definition of Alzheimer-related terms; I believe this is fixable by adding an external, pre-specified AD gene/term panel and reporting model performance, but as written it is load-bearing and needs a revision cycle. The 'first technical review' novelty claim is not independently verifiable from the manuscript, but the literature selection and categorization appear internally consistent. No citation or authorship concerns beyond the name inconsistency noted in the minor comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plain-English take: this is a review-plus-benchmark paper. The review surveys 26 studies and organizes interpretability approaches into four families: modality elimination, non-graph XAI, graph-based XAI, and inherent interpretability. That taxonomy is reasonable, and the PRISMA-style selection is clearly described. The flowchart in Section 5.4 is a practical guide for choosing an explanation strategy. These are the paper's solid contributions.\n\nThe new empirical content is the benchmark. The authors replicate MOGONet on the ROSMAP Alzheimer's cohort and compare sensitivity analysis, gradient saliency, SHAP, and graph masking by taking each method's top 30 genes and running GO and pathway enrichment. The protocol is mostly transparent; code and data are linked, and the permutation tests are a sensible null check. The finding that SHAP and sensitivity analysis recover broader known-AD signals while the other two add complementary terms is consistent with the tables as presented.\n\nThe soft spot is exactly what the stress-test note says: the evaluation measure is not defined in advance. The 'eighteen Alzheimer-related GO terms' are the union of terms enriched at p≤0.01, and the label 'Alzheimer-related' is assigned post hoc. Some assignments are plausible—14-3-3 protein binding and GPCR signaling—but 'temperature homeostasis' and 'endocrine system development' are less obviously AD-specific. Because the same post hoc annotation is applied to all methods, the ranking of SHAP over the others is not a test of explanation quality; it is a test of which method's gene list happens to match the authors' biological priors. A pre-registered AD gene-set panel or an externally curated pathway list would solve this. If that is not feasible, the analysis should be reframed as descriptive.\n\nA second gap: no predictive accuracy is reported for the replicated MOGONet. Permutation tests show the importance scores are not random, but they do not tell us the model is predictive. Reporting accuracy, F1, or AUROC on the test set would make the benchmark interpretable.\n\nMinor issues: the methods say FDR p<0.01 but the benchmark uses raw p-values ≤0.01 with no multiple-testing correction; the introduction and conclusion both claim 'first' technical review, which is hard to verify and not needed; and there are no confidence intervals around the enrichment counts or the observed-versus-random ratios. All three are fixable.\n\nOn balance, the taxonomy and workflow are worth having, and the benchmark is a useful case study even if it is not a definitive method comparison. The paper is honest about its scope. I would send it to peer review with the expectation of major revision on the evaluation measure, accuracy reporting, and uncertainty. It deserves referee time; it should not be desk rejected.","headline":"Useful taxonomy plus a narrow but honest benchmark; the method ranking is conditional on a post-hoc biological label, so treat it as a case study, not a definitive comparison.","tokens_in":24924,"tokens_out":3532,"would_cite":false,"duration_ms":33954,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This review and benchmark finds that SHAP and sensitivity analysis, among four explainability methods for graph-based multimodal biomedical models, recover the broadest set of known Alzheimer's pathways and Gene Ontology terms.","keywords":["Graph neural networks","Multimodal medical data","Interpretable graph models","Explainable artificial intelligence","Knowledge-guided deep learning","LLM-integrated graph models","Alzheimer's disease benchmark","Gene Ontology analysis"],"falsifier":"Re-run the same four explainers on a second disease whose pathway knowledge is independently curated, or hold out a later-curated set of Alzheimer's genes; if SHAP's and sensitivity analysis's overlap advantage over gradient saliency and graph masking disappears or reverses, the reported ranking is an artifact of the specific knowledge base rather than a property of the methods.","tokens_in":23891,"feed_emoji":"🧬","tokens_out":7806,"duration_ms":71099,"temperature":0.7,"pith_summary":"This paper proposes that interpretable graph-based models for multimodal biomedical data form a distinct technical area and organizes that area into four interpretability families: modality or feature elimination, non-graph explainability methods, graph-native explainers, and inherently interpretable designs. Surveying 26 studies from January 2019 to September 2024, it finds that most work targets disease classification, builds static graphs from similarity measures, and leans on post-hoc methods borrowed from non-graph domains. The paper then benchmarks four explainability techniques on an Alzheimer's disease cohort using a replicated multi-omics graph model. It reports that SHAP and sensitivity analysis recover the broadest set of known Alzheimer's pathways and Gene Ontology terms, while gradient saliency and graph masking surface complementary metabolic and transport signatures. A reader should care because the review offers a method-selection map for researchers who need transparency in clinical graph models.","feed_headline":"SHAP and sensitivity analysis recover the most known AD biology","feed_subtitle":"On Alzheimer's multi-omics graphs, all four tested explainers beat random gene sets, with different speed-versus-depth trade-offs.","key_machinery":"The load-bearing machinery is the four-category interpretability taxonomy combined with a benchmark protocol that ranks explainers by the biological plausibility of their top gene sets. The taxonomy separates modality or feature elimination, non-graph XAI such as gradient saliency, sensitivity analysis, and SHAP, graph-based XAI such as graph masking, GNNExplainer, and top-k pooling, and inherently interpretable designs such as attention weights, graph attention, filter weights, and decision trees. The benchmark then applies representative methods from Categories II and III to a replicated MOGONet graph model, a multi-omics graph convolutional network fusing mRNA, methylation, and miRNA views, on the ROSMAP Alzheimer's dataset, and evaluates the top 30 genes from each method through over-representation analysis against known GO terms and pathway databases at a $p \\le 0.01$ threshold.","core_discovery":"The central claim is that the interpretability landscape for graph-based multimodal biomedical models can be usefully divided into four families, and that within these families there is no single best explainer: the right choice depends on whether a study values breadth of known disease biology, discovery of complementary signatures, or computational cost. On the paper's Alzheimer's benchmark, SHAP and sensitivity analysis identified the most established AD-related GO terms and pathways, including GPCR signalling and neurodevelopmental processes, whereas gradient saliency and graph masking highlighted metabolic and transport biology such as 14-3-3 protein binding and SLC-mediated transmembrane transport. Permutation tests showed that all four methods' top-30 gene sets carried average importance scores above random gene sets, with gradient saliency showing the largest offset from the null. The paper also argues that modality elimination, a common Category I technique, is unfair for graph models when edges depend on node features, because removing a modality can rebuild the graph and change what is being compared.","pith_inferences":["A reader should treat the 'beat random' permutation result as a sanity check rather than a faithfulness proof: it shows top genes carry non-random signal, not that the signal matches the model's actual reasoning.","The same four-method comparison could be run on other well-curated disease cohorts to test whether SHAP's breadth advantage generalizes or only reflects the maturity of Alzheimer's pathway databases.","Combining one fast coarse explainer with one deep explainer may be a cheaper way to get both breadth and specificity, a strategy the paper hints at but does not itself benchmark.","The graph-in-graph and LLM directions the paper sketches could turn the taxonomy into a design template: if each patient is a node inside a cohort graph and an internal subgraph, then Category II and III explainers would need to report at two levels simultaneously, a testable extension no current study in the review addresses."],"forward_implications":["Researchers building graph models for clinical use can choose an explainer by goal: SHAP for broad disease-pathway coverage, sensitivity analysis for fast broad coverage, gradient saliency for speed, and graph masking for deeper but costlier biology.","The review's taxonomy gives method developers a common vocabulary for comparing future explainers against four distinct interpretability strategies.","The identified gaps, namely graph-native explainers, knowledge-graph edges, dynamic topology, and graph-in-graph hierarchies, mark concrete opportunities for the next generation of models.","The benchmark's compute-cost and biological-depth trade-offs can inform resource budgeting, since full SHAP runs on hundreds of features can take weeks.","The finding that all four methods beat random gene sets supports using any of them as a starting point, but with different expectations about which biology they will surface."],"supporting_citations":[{"why":"It supplies the MOGONet multi-omics graph architecture and feature-importance protocol that the benchmark replicates.","marker":"[36]"},{"why":"It supplies the ROSMAP Alzheimer's cohort with its three molecular modalities, mRNA, methylation, and miRNA, used in the benchmark.","marker":"[54]"},{"why":"It defines SHAP, the Shapley-value explainer whose top genes and pathways are compared against the other methods.","marker":"[45]"},{"why":"It introduces GNNExplainer, the masking-based graph explainer from which the benchmark's graph-masking method is generalized.","marker":"[52]"},{"why":"It provides the sensitivity-analysis formula, weight times feature standard deviation, used as one of the benchmarked explainers.","marker":"[49]"},{"why":"It supplies the over-representation analysis tool used to map each method's top genes to GO terms and pathways at an FDR threshold of $\\le 0.01$.","marker":"[74]"},{"why":"It defines the View Correlation Discovery Network that fuses the three modality-specific graph models inside MOGONet.","marker":"[75]"}],"fun_headline_variants":["SHAP & sensitivity lead on known AD gene sets","No single best explainer: speed vs depth trade-offs","Removing a modality breaks graph explainer fairness","All four graph explainers beat random gene sets on AD","First survey benchmarks 4 graph XAI methods on Alzheimer's"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark trusts that the curated set of already-known Alzheimer's Gene Ontology terms and pathways is a correct yardstick for explanation quality, so an incomplete or biased disease-knowledge list would weaken its ranking of SHAP and sensitivity analysis.","fun_headline_variants_meta":{"raw":{"variants":["SHAP & sensitivity lead on known AD gene sets","No single best explainer: speed vs depth trade-offs","Removing a modality breaks graph explainer fairness","All four graph explainers beat random gene sets on AD","First survey benchmarks 4 graph XAI methods on Alzheimer's"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000436,"raw_usage":{"total_tokens":2274,"prompt_tokens":1055,"completion_tokens":1219,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":1140}},"tokens_in":671,"tokens_out":1219,"duration_ms":12216,"temperature":1.0,"reasoning_tokens":1140,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:12:21.584382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same four explainers on a second disease whose pathway knowledge is independently curated, or hold out a later-curated set of Alzheimer's genes; if SHAP's and sensitivity analysis's overlap advantage over gradient saliency and graph masking disappears or reverses, the reported ranking is an artifact of the specific knowledge base rather than a property of the methods.","supporting_citations":[{"cited_title":"MOGONET integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification,","cited_arxiv_id":null,"evidence_quote":"It supplies the MOGONet multi-omics graph architecture and feature-importance protocol that the benchmark replicates."},{"cited_title":"Accelerating medicines partnership: Alzheimer’s disease (AMP-AD) knowledge portal aids Alzheimer’s drug discovery through open data sharing,","cited_arxiv_id":null,"evidence_quote":"It supplies the ROSMAP Alzheimer's cohort with its three molecular modalities, mRNA, methylation, and miRNA, used in the benchmark."},{"cited_title":"A unified approach to interpreting model predictions,","cited_arxiv_id":null,"evidence_quote":"It defines SHAP, the Shapley-value explainer whose top genes and pathways are compared against the other methods."},{"cited_title":"Gnnexplainer: Generating explanations for graph neural networks,","cited_arxiv_id":null,"evidence_quote":"It introduces GNNExplainer, the masking-based graph explainer from which the benchmark's graph-masking method is generalized."},{"cited_title":"Interpreting neural network connection weights,","cited_arxiv_id":null,"evidence_quote":"It provides the sensitivity-analysis formula, weight times feature standard deviation, used as one of the benchmarked explainers."},{"cited_title":"WebGestalt 2017: a more comprehensive, powerful, flexible and interactive gene set enrichment analysis toolkit,","cited_arxiv_id":null,"evidence_quote":"It supplies the over-representation analysis tool used to map each method's top genes to GO terms and pathways at an FDR threshold of $\\le 0.01$."},{"cited_title":"Generative multi-view human action recognition,","cited_arxiv_id":null,"evidence_quote":"It defines the View Correlation Discovery Network that fuses the three modality-specific graph models inside MOGONet."}],"review_version":1}