{"id":"60271ff4-0e65-4670-88e1-8f1774b1271d","arxiv_id":"2505.02566","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Most graph neural network defenses improve explanation sparsity and stability in this benchmark, while the consistency and fidelity metrics largely saturate and stop being informative.","lead":"This paper measures how defending graph neural networks against attacks changes how explainable their predictions appear, testing six architectures, seven defenses, and five datasets. A generalist should read it because it is a rare systematic look at the tension between robustness and interpretability, even though the measurements themselves need repair.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fidelity is defined as a distance (lower is better) yet reported with ↑; under the stated definition, defended models show worse Fidelity than unprotected, so the headline claim relies on an unstated metric direction.","rationale":"The reader's weakest assumption identifies exactly this Fidelity direction problem, and I agree it is load-bearing. The paper's contribution is an empirical benchmark claiming that defenses improve interpretability; that claim must be read off the tables. Because the Fidelity definition and its table annotations point in opposite directions, the reader cannot tell whether the reported numbers support or contradict the claim. This is an internally verifiable issue: it does not depend on outside consensus, only on the paper's own equations and legends. The concern is concrete and fixable, so CONDITIONAL remains the appropriate verdict: the authors should clarify or correct the metric direction, rerun the affected tables, and report which conclusions survive. I would not move to REJECT because the sparsity and stability results, whose directions (↓ lower better) are consistent with the claim, may still support the headline conclusion; but the Fidelity ambiguity must be resolved before the benchmark can be accepted as reported. The proposed test — recomputing Fidelity from the literal definition — directly settles whether the concern lands.","tokens_in":16417,"tokens_out":5206,"duration_ms":71173,"concrete_test":"Recompute Table 5's Fidelity row from the released code, using the literal Section 3.2 definition as a distance and treating lower values as better. If the defended-vs-unprotected ordering flips (defended values exceed unprotected), then the table's ↑ annotation is inverted and the headline claim loses Fidelity as support; if Fidelity values saturate or are identical across defenses, the metric is not informative and should be excluded from the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'the addition of most popular defense methods improves model interpretability' rests on four metrics, and the Fidelity metric is internally contradictory. Section 3.2 defines Fidelity = (1/N) Σ |f(G_int) − f(G)|, which is a distance: smaller values mean the explanation better matches the original model's behavior. Yet every results table (Tables 1–8, 11–13) annotates FIDELITY with ↑, implying larger is better. Nowhere does the paper state a transformation such as 1 − distance. Under the literal definition, Table 5 shows defended models with Fidelity ≈ 0.997–0.999 versus unprotected ≈ 0.846, meaning defended explanations deviate more from the original predictions — the opposite of the claimed improvement. Table 7 compounds this: Fidelity is exactly 0.995 ± 0.011 for all seven defenses, a saturated value that cannot discriminate methods and suggests the implemented metric is either degenerate or not the one stated. The paper's own Section 5 says Fidelity is 'less affected' by defenses, yet the tables show a large jump from unprotected to defended, an internal inconsistency. If the direction is inverted, the headline claim loses one of its four supporting measures; if the metric is degenerate, the Fidelity evidence is uninformative. Either way, the quantitative support for the central claim cannot be validated from the manuscript as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a benchmark study of how robustness-enhancing defenses affect the interpretability of graph neural networks. It evaluates six GNN architectures on five datasets with seven defense methods, measuring interpretability via four metrics: Fidelity, Stability, Consistency, and Sparsity. The principal empirical claim, stated in Sections 4 and 5, is that most popular defense mechanisms improve model interpretability relative to an undefended model. The paper also discusses which metrics are more or less sensitive to model modifications and calls for refinement of interpretability metrics.","tokens_in":16631,"tokens_out":5039,"duration_ms":60351,"significance":"If the measurements were valid, this would be a useful and timely contribution: the interaction between robustness and interpretability in GNNs is an active area, and the paper covers a broad sweep of architectures, datasets, and defenses. The open-source promise and the explicit hyperparameter listing in Appendix C are strengths. However, the central empirical conclusion is not currently supported because of a fundamental internal inconsistency in the Fidelity metric: it is defined as a distance but annotated as higher-is-better in every table, which inverts the interpretation of the headline result. A second metric, Sparsity, is defined as density rather than as the stated 'percentage of features excluded.' Because the benchmark's conclusions rest on these metrics, the reported quantitative support cannot be accepted as written; nevertheless, the errors appear correctable through re-analysis or clarification, so the work has potential after major revision.","major_comments":[{"comment":"The Fidelity metric is defined in Section 3.2 as (1/N) Σ |f(G_int) − f(G_i)|, which is a distance for which smaller values indicate that the explanation better reproduces the model's behavior. Yet every results table annotates FIDELITY with (↑), implying larger is better, and no transformation such as 1 − distance is stated. Under the literal definition, Table 5 shows defended models with Fidelity ≈ 0.994–0.999 versus unprotected ≈ 0.846, meaning defended explanations deviate more from the original predictions — the opposite of the claimed improvement. Table 8 shows the reverse pattern (defended ≈ 0.843–0.856 versus unprotected ≈ 0.925). Either the metric direction is inverted or the implementation differs from the stated definition. This is load-bearing because the Section 5 claim that 'the addition of most popular defense methods improves model interpretability' relies directly on Fidelity as one of four supporting measures. The authors must state precisely which quantity was computed, correct the direction or the tables, and re-evaluate the conclusions.","section":"Section 3.2 and Tables 1–8, 11–13"},{"comment":"In Table 7, the Fidelity row reports exactly 0.995 ± 0.011 for all seven defense methods, and in Table 13 (SubgraphX) the Fidelity is exactly 1.000 ± 0.000 for every defense and for the unprotected model. A metric that returns identical values under all conditions is saturated and cannot discriminate between methods. Section 5 states that Fidelity is 'less affected' by defenses, but a constant value is uninformative and does not support any comparative conclusion. The authors should diagnose why the metric collapses (for example, mask thresholds producing near-identical model outputs) and either replace it with a non-degenerate variant or explicitly report it as non-discriminative.","section":"Table 7 and Table 13"},{"comment":"The text repeatedly uses the word 'significant' and marks 'significant improvements' and 'significant degradations' in bold/italic, but the paper reports no statistical tests. The averaging procedure in Section 4.1.5 uses a fixed set of 10 nodes per iteration and 5 iterations, and the reported standard deviations are large (e.g., Table 6 Stability: 0.622 ± 0.359 for AT versus 0.458 ± 0.312 for JD; Table 3 GIN Stability: 0.494 ± 0.098 versus SAGE 0.272 ± 0.067). Without paired tests across the fixed node sets or confidence intervals, the categorical claims that particular defenses 'significantly' improve or degrade individual metrics are not supported. This directly affects the ranking conclusions in Section 4.2.3.","section":"Section 4.2.3 and Tables 5–8"},{"comment":"The Sparsity formula, Sparsity = (1/m) Σ 1{M(G)_j ≠ 0}, computes the fraction of features that are nonzero in the mask, i.e., the density of selected features, not the 'percentage of features excluded from the prediction's interpretation' as the prose states. The downward arrow (↓) is consistent with the formula (fewer selected features is better), but the verbal definition would require the complement, 1 − Sparsity. If the code implemented the complement, every Sparsity result and the associated comparisons would have the opposite direction. This ambiguity must be resolved in the text and confirmed against the released code.","section":"Section 3.2, Sparsity definition"}],"minor_comments":[{"comment":"The loss function is referred to as 'NL LLoss'; this appears to be a typo for NLLLoss.","section":"Section 4.1.3"},{"comment":"Table 12 appears malformed: the header lists five GCN entries but only two pairs of values are shown in the body, leaving the architecture-to-value mapping unclear. This should be fixed or the table simplified.","section":"Table 12"},{"comment":"The sentence 'the evasion defense mechanism is applied during training' is confusing because evasion attacks occur at inference time; the intended meaning (e.g., adversarial training during training, or input transformation at inference) should be clarified.","section":"Section 2.4"},{"comment":"The paper states the framework 'will be published as open source' but provides no repository link or availability statement; including an anonymous URL or a release note would strengthen reproducibility.","section":"Section 4.1.4 / Conclusion"},{"comment":"The claim that 'all the key conclusions are also valid' with SubgraphX is based on only 5 vertices and 5 iterations on a single dataset (Cora), and Table 12 is incomplete; this claim should be qualified as preliminary.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a useful benchmark scope, but the headline conclusion is undermined by a fixable metric-direction error in Fidelity and a definitional ambiguity in Sparsity. I recommend major revision rather than rejection because the issues appear correctable through re-analysis and clarification, and the underlying experimental effort is substantial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the grid is a genuinely useful piece of benchmarking work, and one clear observation survives all the problems below: sparsity and stability react to defenses while consistency mostly does not. But the headline quantitative claim that \"most defense methods improve interpretability\" is not supported as written, because the Fidelity metric is defined one way and interpreted the opposite way, and because no adversarial attack is ever executed.\n\nThe grid itself is the real contribution: six architectures, seven defenses, five datasets, four metrics, with GNNExplainer as the main interpreter and SubgraphX as a sanity check. That coverage is new relative to the decision-tree and image-domain analyses cited in Section 2. The finding that defenses push explanations toward sparser and more stable masks is directionally consistent across tables and is the kind of practical guidance the field needs. I also credit the authors for explicitly saying in Section 5 that the metrics need refinement; that is honest.\n\nThe soft spots are serious, though. Section 3.2 defines Fidelity as a mean absolute distance between the model's output on the original and masked graphs; smaller is better. Every table annotates it with an upward arrow and the text reads it as higher-is-better. Under the definition as written, the defended models in Table 5 have Fidelity around 0.998 versus 0.846 for the undefended model, meaning defended explanations are farther from the original model's predictions. That inverts one of the four supporting measures. Table 7 compounds the problem: Fidelity is exactly 0.995 ± 0.011 for all seven defenses, a saturated and non-discriminative value. The paper's own discussion says Fidelity is \"less affected\" by defenses, yet the tables show a huge jump from undefended to defended, which is internally inconsistent.\n\nSecond, despite the title and abstract promising adversarial scenarios, no actual attack is run anywhere in the manuscript. Defenses are applied, but there is no attack to defend against. That breaks the link between the robustness framing and the measurements. Third, model accuracy is never reported, so defense-driven interpretability changes cannot be separated from accuracy or smoothness changes. Fourth, the \"significant\" language in Tables 3-8 has no statistical tests behind it. Fifth, the open-source framework is promised but no code or link is provided, so the reproducibility claim is unverifiable for now.\n\nWho gets value from this paper: anyone building or critiquing GNN interpretability benchmarks. The sparsity/stability observation is worth keeping. But the central claim needs a corrected Fidelity direction, an actual executed attack, and accuracy reporting before it can be believed.\n\nMy recommendation: send it out for peer review rather than desk reject. It deserves a serious referee, but the likely correct outcome is a conditional or major-revision verdict. The grid is too useful to throw away.","headline":"A useful benchmark grid undone by a mislabeled fidelity metric and no actual attack; the sparsity/stability finding survives but the headline claim about defenses improving interpretability does not.","tokens_in":17366,"tokens_out":2654,"would_cite":false,"duration_ms":31663,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most popular GNN defense methods improve explanation sparsity and stability across the benchmark.","keywords":["graph neural networks","interpretability","adversarial robustness","defense mechanisms","GNNExplainer","explanation metrics","poisoning attacks","evasion attacks"],"falsifier":"Recompute the defense-vs-unprotected comparisons with Fidelity interpreted literally as the Section 3.2 formula, where lower is better; Table 7's defended fidelity of 0.995 versus 0.760 for unprotected then indicates that defenses worsen fidelity, and if that reversal holds across Tables 5-8 the headline claim loses one of its four supporting metrics.","tokens_in":16148,"feed_emoji":"🛡️","tokens_out":13630,"duration_ms":139945,"temperature":0.7,"pith_summary":"This paper tries to determine whether hardening a graph neural network against adversarial attacks changes how interpretable it is, and it reports a systematic benchmark designed to answer that question. Across six GNN architectures, five datasets, and seven defense mechanisms, the authors find that most defenses improve interpretability as measured by explanation sparsity and stability, while leaving consistency and fidelity approximately unchanged. The benchmark also shows that sparsity and stability respond to model and data changes, whereas consistency and fidelity stay flat, so the paper recommends using the first pair as the sensitive diagnostics. This matters because practitioners who add robustness defenses to GNNs need to know whether the explanations they hand to users will still be meaningful.","feed_headline":"Most GNN defenses improve explanations, benchmark finds","feed_subtitle":"Common defenses make GNN explanations sparser and more stable; adversarial training is the exception.","key_machinery":"The engine of the study is a fixed benchmark pipeline: choose a dataset, an attack, and a defense; train the GNN; run a post-hoc interpreter to obtain a mask over edges and features; then score the mask with four metrics. GNNExplainer, a method that trains a differentiable mask over a node's computation graph, is the primary interpreter, with SubgraphX used as a check on Cora. The metrics that carry the central conclusion are sparsity, the fraction of nonzero mask entries, and stability, the Euclidean distance between masks on original and slightly perturbed graphs; defended models consistently fall on the good side of both relative to the undefended baseline across the result tables. Fidelity is computed as the average absolute difference between the model's output on the interpreted subgraph and on the original graph, and consistency is the cosine similarity between masks from repeated runs.","core_discovery":"The central claim is stated in Section 5: the addition of most popular defense methods improves model interpretability. In the benchmark, Jaccard, GNNGuard, distillation, quantization, autoencoder, gradient regularization, and most runs of adversarial training each move GNNExplainer masks toward sparser and more stable explanations relative to an undefended GNN, while consistency and fidelity change little. Adversarial training is the partial exception, improving stability less than the other defenses on some datasets. The paper further claims that consistency and fidelity are stable across architectures, domains, and defenses, making them better suited for validating an interpretation method, whereas sparsity and stability are sensitive indicators of how modifications such as defenses affect explanations.","pith_inferences":["Read literally, the Section 3.2 fidelity formula is an error distance, so lower should mean more faithful, yet the tables mark fidelity as higher-is-better; in Table 7 defended models score 0.995 versus 0.760 for unprotected, which flips if fidelity is treated as a distance, so the four-metric conclusion may rest on three metrics or fewer.","The identical fidelity value of 0.995 ± 0.011 for all seven defenses in Table 7 suggests the metric is saturated at these operating points; a fidelity variant that separates keeping relevant features from removing irrelevant ones would give the benchmark more resolving power.","The paper's 'significant' language is not backed by statistical tests; paired significance testing across the repeated random node sets would separate real differences from noise and is the natural next step.","The paper's explanation that defenses act like regularizers predicts a dose-response that was not tested: increasing defense strength should move sparsity and stability monotonically, which one could check by sweeping distillation temperature or gradient-regularization strength."],"forward_implications":["Practitioners who add Jaccard, GNNGuard, distillation, quantization, autoencoder, gradient regularization, or adversarial training can expect GNNExplainer to return sparser, more stable explanations than an undefended model in most of the settings tested.","Because consistency and fidelity barely move across defenses and architectures, these metrics will not reveal whether a model change helped interpretability; sparsity and stability are the metrics to watch.","Adversarial training should be treated as the least consistently beneficial defense for explanation stability, with its gains varying by dataset and architecture.","Architecture and data domain set the baseline: GIN explanations are less stable and sparser than GCN or SAGE, and the higher-degree Amazon datasets show markedly different stability and sparsity numbers than the citation datasets.","The benchmark implies that robustness defenses and interpretability can be improved together rather than treated as competing objectives."],"supporting_citations":[{"why":"Supplies GNNExplainer, the post-hoc interpreter whose masks the four metrics score.","marker":"(Ying et al., 2019)"},{"why":"Supplies SubgraphX, the supplementary interpreter used to verify the key conclusions on Cora.","marker":"(Yuan et al., 2021)"},{"why":"Supplies the Jaccard poisoning defense, one of the defense conditions tested.","marker":"(Wu et al., 2019)"},{"why":"Supplies GNNGuard, the second poisoning defense condition tested.","marker":"(Zhang & Zitnik, 2020)"},{"why":"Supplies defensive distillation, one of the evasion defenses tested.","marker":"(Papernot et al., 2016)"},{"why":"Supplies adversarial training and the FGSM attack used to generate adversarial examples.","marker":"(Goodfellow et al., 2014)"},{"why":"Supplies gradient regularization, one of the evasion defenses tested.","marker":"(Finlay & Oberman, 2021)"},{"why":"Supplies the autoencoder defense used in the benchmark.","marker":"(Meng & Chen, 2017)"},{"why":"Supplies the Cora, CiteSeer, and PubMed citation datasets used as one domain.","marker":"(Sen et al., 2008)"},{"why":"Supplies the Computers and Photo Amazon datasets used as the other domain.","marker":"(McAuley et al., 2015)"}],"fun_headline_variants":["Most GNN defenses improve explanations, adversarial training lags","GNN defenses sparsify and stabilize explanations, except adversarial training","Defenses make GNN explanations sparser and more stable—mostly","GNN robustness defenses mostly aid interpretability, study shows","Interpretability improves for most GNN defenses, benchmark reveals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion presumes that the four metrics measure interpretability in the direction the tables claim, yet Fidelity is defined as an average absolute difference for which lower should be better while every table marks it as higher-is-better.","fun_headline_variants_meta":{"raw":{"variants":["Most GNN defenses improve explanations, adversarial training lags","GNN defenses sparsify and stabilize explanations, except adversarial training","Defenses make GNN explanations sparser and more stable—mostly","GNN robustness defenses mostly aid interpretability, study shows","Interpretability improves for most GNN defenses, benchmark reveals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000943,"raw_usage":{"total_tokens":4001,"prompt_tokens":887,"completion_tokens":3114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":3029}},"tokens_in":503,"tokens_out":3114,"duration_ms":22929,"temperature":1.0,"reasoning_tokens":3029,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:49:23.834067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the defense-vs-unprotected comparisons with Fidelity interpreted literally as the Section 3.2 formula, where lower is better; Table 7's defended fidelity of 0.995 versus 0.760 for unprotected then indicates that defenses worsen fidelity, and if that reversal holds across Tables 5-8 the headline claim loses one of its four supporting metrics.","supporting_citations":[{"cited_title":"On explainability of graph neural networks via subgraph explorations","cited_arxiv_id":null,"evidence_quote":"Supplies SubgraphX, the supplementary interpreter used to verify the key conclusions on Cora."},{"cited_title":"and Zitnik, M","cited_arxiv_id":null,"evidence_quote":"Supplies GNNGuard, the second poisoning defense condition tested."},{"cited_title":"Distillation as a defense to adversarial perturbations against deep neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies defensive distillation, one of the evasion defenses tested."},{"cited_title":"and Oberman, A","cited_arxiv_id":null,"evidence_quote":"Supplies gradient regularization, one of the evasion defenses tested."},{"cited_title":"and Chen, H","cited_arxiv_id":null,"evidence_quote":"Supplies the autoencoder defense used in the benchmark."},{"cited_title":"Image-based recommendations on styles and substitutes","cited_arxiv_id":null,"evidence_quote":"Supplies the Computers and Photo Amazon datasets used as the other domain."}],"review_version":1}