{"id":"6335b7ca-674d-472b-81d9-e496e0ea6c7d","arxiv_id":"2607.04600","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dual feature/edge, ground-truth-free metric suite and large multi-task benchmark show Input×Gradient as a robust Pareto choice among eight GNN explainers, with no universal winner.","lead":"This paper builds a ground-truth-free way to score GNN explainers on stability, faithfulness, compactness, and speed, treating node features and graph edges separately. Practitioners get Pareto rankings and concrete selection rules for regulated GNN pipelines.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Medoid-based Pertinence/EC can misrank faithfulness under multimodal alternatives, undermining Pareto guidelines as transferrable trustworthiness measures.","rationale":"The Reader correctly isolates the dual ground-truth-free operationalizations (especially medoid-driven Pertinence and feature EC) as the weakest link between the large-scale ranking and the claim that the results support industrial explainer selection. The paper’s own caveats on multimodal classes and ties already flag the issue; the concern is therefore not external consensus but whether the proxies that generate the Pareto fronts and Appendix thresholds remain faithful when the medoid assumption fails. Selection bias and proprietary code are real but secondary: even with open code and a broader explainer set, if the metrics systematically misrank under realistic class geometry the guidelines remain unreliable. No internal contradiction of the scoped claim exists, so REJECT is unwarranted; ACCEPT is premature until the medoid sensitivity is checked. Hence CONDITIONAL is unchanged, with the concrete multi-medoid recompute as the decisive next measurement.","tokens_in":21384,"tokens_out":595,"duration_ms":5749,"concrete_test":"On Cora and GitHub node classification, recompute feature Pertinence and EC for all eight explainers after replacing single-medoid references with k=3 medoids per opposing class (or nearest-neighbor references in embedding space). If Input×Gradient / IntGrad lose Pareto dominance on ≥1 metric pair, or if ≥20% of instances flip relative ranking vs. GNNExplainer/LRP, the transferability claim for the guidelines is weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that Input×Gradient / IntGrad are robustly non-dominated Pareto solutions (and thus the basis for operational thresholds and default recommendations) rests on Pertinence and feature Effective Compactness as ground-truth-free faithfulness/cognitive-load proxies. Both are built from medoid reference graphs whose predictions differ from the target, followed by ranked substitutions/removals capped at 100 (Algorithms 4–7, §3). The paper itself notes that when opposing classes are multimodal, a single medoid poorly represents the nearest alternative, so feature perturbations become incoherent and random orderings can degrade the output almost as fast as the attribution, producing sub-0.5 pertinence and depressed EC rankings (§5.2 and Appendix A.2.2). If this regime is common on real proprietary graphs (or if the 100-cap and max-pool neighbor aggregation systematically favor gradient methods that concentrate mass on few coordinates), the Pareto fronts and Table 3 thresholds misidentify which explainers are trustworthy for audit, so the distilled usability guidelines fail as deployment guidance even though the scoped empirical ranking within the eight-explainer set is internally consistent.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a ground-truth-free evaluation framework for post-hoc GNN explainers that scores feature and edge attributions independently via Stability, Pertinence, Effective Compactness, and wall-clock Time. Metrics are formalized with explicit algorithms (Algorithms 1–7), early-stopping for iterative estimates, and task-specific handling of node vs. link targets. The authors implement the framework in a proprietary modular library, then benchmark eight dual-attribution, open-license explainers on ~2M explanations across ten GraphML tasks (node/edge × classification/regression) using GCN and GAT models on five datasets. Results identify Input×Gradient and Integrated Gradients as consistently non-dominated on Pareto fronts for pertinence–compactness and pertinence–time, while no explainer is universally superior; findings are distilled into operational thresholds and selection guidelines in Appendix A.","tokens_in":21750,"tokens_out":1633,"duration_ms":23599,"significance":"The work addresses a genuine practical gap: how to compare GNN explainers on real data without planted motifs, while covering regression and link-level tasks that prior G-XAI benchmarks largely omit. Strengths include explicit dual (feature/edge) metric definitions, large-scale multi-task averaging, Pareto analysis rather than single-score ranking, and concrete practitioner guidelines with transferability notes. If the proxies and scoped ranking hold under broader validation, the framework and default recommendation of Input×Gradient would be immediately useful for regulated GNN pipelines. Credit is due for ground-truth-free design, early-stopping optimization, and honest reporting of sub-0.5 pertinence regimes and selection filters. Reproducibility is limited by the proprietary library, which reduces the paper’s archival value as a community benchmark.","major_comments":[{"comment":"§5.1 and Conclusion: The dual-attribution + open-license filters exclude many widely cited graph-native methods (PGExplainer, SubgraphX, GNN-LRP, GraphLIME, CLEAR, etc.). The abstract and §5.2 claim that Input×Gradient/IntGrad are robustly non-dominated “across metric pairs and tasks” and that no single explainer is universally superior. Those claims are only supported inside the eight-method subset. The manuscript acknowledges selection bias but still frames results as guidance for “state-of-the-art” G-XAI. The central recommendation and Pareto narrative need to be restated as conditional on dual-attribution, open-license methods, with an explicit non-claim about excluded families, or the benchmark must be extended with at least a subset of those methods under a relaxed dual-attribution protocol.","section":null},{"comment":"§3 Algorithms 5–7 and §5.2: Feature Pertinence and feature EC rely on medoid reference graphs and ranked substitutions capped at 100. The paper itself reports that multimodal opposing classes make a single medoid a poor nearest alternative, so random orderings can match or beat the attribution and drive pertinence below the 0.5 baseline; ties are also counted as failures. Appendix A.2.2 then still proposes Pertinence ≥0.6 and Feature EC percentage bounds as deployment thresholds. If these failure modes are common on proprietary graphs, the Pareto fronts and Table 3 guidelines misidentify trustworthy explainers. A load-bearing revision is needed: quantify how often sub-0.5 pertinence occurs per dataset/task, add a sensitivity analysis (multiple references, k-medoids, or class-conditional sampling), and either revise the thresholds with failure-mode caveats or restrict the “default explain","section":null},{"comment":"§4 and reproducibility: The evaluation library is proprietary and “cannot be released as open source”; only “selected snippets” are offered on request. For a paper whose primary contribution is a unified benchmarking framework and operational thresholds, this is a material obstacle. Without public code for metric implementations (especially medoid selection, computational-graph extraction, early stopping, and NLX), independent verification of Table 1, Figure 2, and Appendix A thresholds is not possible. At minimum, release metric pseudocode as executable reference implementations, full hyperparameter configs, and the exact target indices used for the 100 explanations per task-dataset, or provide a public reimplementation of the metric suite on standard PyG models.","section":null},{"comment":"§5.1–5.2 and Appendix B: Regression and several edge results rest heavily on Movielens (dense, genre features) and a fully synthetic ER graph with an arbitrary weighted aggregation rule. The text notes that edge primacy in regression and limited feature-EC variability “may derive from the specific synthetic datasets” and “might not generalize.” Yet Appendix A still issues task-level default recommendations and transferability guidance for production. Either expand the regression/link suite with additional real datasets, or demote the regression Pareto conclusions and corresponding guideline rows to exploratory, with explicit dataset-dependence warnings in the main text rather than only in a late caveat.","section":null}],"minor_comments":[{"comment":"Abstract and §1: “∼2M explanations” is striking but not broken down (targets × explainers × models × stability/pertinence trials). A short accounting in §5.1 would help readers assess statistical power and early-stopping savings.","section":null},{"comment":"Figure 2: Pareto fronts are clear, but axis scales and marker overlap make it hard to see which methods sit on the front for edge regression feature EC. Consider small multiples with labels or a supplementary table of non-dominated sets per panel.","section":null},{"comment":"Table 1: Standard deviations are large for EC (often ±20–50). Report median and IQR or confidence intervals on the means, and clarify whether averaging is hierarchical (targets → datasets → models) or flat.","section":null},{"comment":"§3: Notation for L2 normalization and ρ_edge / ρ_feat is slightly inconsistent between prose and Algorithms 2–3 (e.g., ∥·∥2 vs. /2 scaling). Unify the formulas.","section":null},{"comment":"Related Work: GraphFramEx, GraphXAI, BAGEL, and GnnX-Bench are discussed; a compact comparison table (ground-truth requirement, tasks covered, dual attribution, production metrics) would sharpen the three-axis differentiation claimed in §2.","section":null},{"comment":"Appendix A.2.3: Feature EC thresholds as “10–25% of total features” are not directly comparable to the absolute EC scores (capped at 100) reported in Table 1. Align units or show how percentage maps to the reported numbers per dataset.","section":null},{"comment":"Typos / polish: “Toanswerthesequestions”, “ComputationalOptimisationWhileourmetrics…”, missing spaces in several run-on phrases in §1 and §3; “quasi-determinism” vs. Stability definition could be cross-referenced once for consistency.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid industrial benchmarking effort with honest caveats, but the proprietary library and narrow explainer filter make it closer to an internal methods report than a fully archival community benchmark. If the journal prioritizes reproducible evaluation frameworks, require code/metric release as a condition of acceptance. Novelty relative to Perotti et al. (tabular XAI metrics) and GraphFramEx should be checked carefully; the dual graph adaptation and multi-task scale are the main increments. Fit for a methods/benchmark track is good if major comments are addressed."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is a practical dual (feature + edge) scoring suite—Stability, Pertinence, Effective Compactness, Time—run without planted motifs, plus a large multi-task ranking that puts Input×Gradient and Integrated Gradients on the Pareto front for most of the eight dual-attribution explainers they could actually run.\n\nWhat is new is the graph adaptation of the tabular metrics from Perotti et al., independent scoring of topology and features, coverage of node/link classification and regression, early-stopping for the iterative metrics, and the distilled thresholds/guidelines in the appendix. The algorithms are explicit, the ~2M-explanation scale is real, the tables and Pareto plots are clean, and they are honest that no explainer wins everywhere. That is more operationally useful than another synthetic-motif fidelity paper.\n\nSoft spots, in proportion: the dual-attribution + open-license filter excludes many graph-native methods (PGExplainer, SubgraphX, GNN-LRP, CLEAR, etc.), so the ranking is only inside that filtered set. Code is proprietary. Regression data is partly synthetic. The stress-test on medoid-based Pertinence/feature EC is fair—the paper itself flags multimodal classes and ties that can push pertinence below 0.5—but it is a known proxy limitation they document, not a hidden contradiction. Thresholds need revalidation on proprietary graphs; they say so. Math and citation pattern look solid; circularity is low.\n\nThis is for industrial GraphML and G-XAI practitioners who need selection criteria without oracles. A serious referee should see it. I would cite the dual metrics and the Pareto findings when discussing explainer evaluation, and I would bring it to reading group for the guidelines discussion.","headline":"Solid dual-attribution, ground-truth-free G-XAI benchmark with useful Pareto rankings and practitioner guidelines; selection bias and medoid-proxy caveats are real but scoped, not fatal.","tokens_in":22340,"tokens_out":448,"would_cite":true,"duration_ms":4837,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A ground-truth-free framework ranks GNN explainers by scoring edge and feature attributions separately and finds no method universally best.","keywords":["GNN explainability","Graph XAI","evaluation metrics","Pareto front","Input×Gradient","stability","pertinence","effective compactness"],"falsifier":"On a held-out proprietary graph task that also carries human-validated rationales, the Pareto ranking produced by the four metrics places explainers that auditors reject above those they accept, or Input×Gradient’s high Pertinence fails to correlate with auditor preference.","tokens_in":22289,"feed_emoji":"🕸️","tokens_out":851,"duration_ms":16458,"temperature":0.7,"pith_summary":"Practitioners face a flood of post-hoc explainers for graph neural networks yet lack objective, ground-truth-free ways to choose among them or to judge whether the outputs are trustworthy. This paper adapts four tabular quality measures—Stability, Pertinence, Effective Compactness and Time—so that topological (edge) and node-feature attributions are scored as independent components, without planted motifs or oracle rationales. Across roughly two million explanations on ten node- and edge-level classification and regression tasks, simple gradient methods such as Input×Gradient consistently sit on the Pareto front of faithfulness-versus-compactness and faithfulness-versus-speed, while several elaborate graph-native methods lag. The authors convert the empirical ranking into concrete quality thresholds and selection guidelines that teams can apply when auditing or deploying GNN pipelines.","feed_headline":"Simple gradients beat complex GNN explainers on trust metrics","feed_subtitle":"Ground-truth-free scores on ~2M explanations show no universal winner and yield selection guidelines.","key_machinery":"The dual-metric suite (Stability, Pertinence, Effective Compactness, Time) that evaluates feature matrices and edge vectors independently via same-input consistency, ranked deletion curves versus random baselines, and counterfactual modification counts capped at 100, all without reference explanations.","core_discovery":"When edge and feature attributions are scored independently with ground-truth-free metrics of stability, faithfulness, compactness and runtime, no single explainer dominates; Input×Gradient and Integrated Gradients emerge as robustly non-dominated solutions on the Pareto front across metric pairs and GraphML tasks, while more complex graph-specific methods frequently underperform.","pith_inferences":["The same dual scoring can be reused for counterfactual or concept-based explainers once they emit edge and feature attributions.","Sub-0.5 Pertinence under multimodal classes implies that single-medoid reference graphs are too crude; adaptive clustering of alternatives may be needed.","Because the metrics are model-faithful rather than human-plausible, pairing them with a separate human study would close the remaining regulatory gap.","Edge Effective-Compactness thresholds will transfer poorly across degree distributions, so degree-stratified revalidation should become standard on proprietary graphs."],"forward_implications":["Teams can short-list explainers from the reported Pareto fronts and task tables instead of relying on anecdotal preference.","Stability near 1.0 flags fully deterministic methods that need only a single invocation for reproducible audits.","Empirically derived thresholds (e.g., Pertinence ≥ 0.6, Stability ≥ 0.8) become concrete acceptance criteria for production G-XAI components.","Natural-language templates can turn the dual attributions into human-readable narratives without altering the underlying scores.","Any new explainer that emits both edge and feature attributions can be dropped into the same modular evaluator and compared on identical axes."],"fun_headline_variants":["Simple gradients beat complex GNN explainers on trust metrics","Input×Gradient tops Pareto front for GNN explainability scores","No universal GNN explainer winner: simple gradients lead trust tests","Ground-truth-free scores put gradients ahead of complex GNN methods","Gradients non-dominated across GNN faithfulness and stability metrics"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The ranking produced by how quickly ordered edge removals or feature substitutions flip the model’s own prediction, measured against random orderings and without any human or planted ground truth, is a sufficient proxy for industrial trustworthiness and can safely guide explainer selection.","fun_headline_variants_meta":{"raw":{"variants":["Simple gradients beat complex GNN explainers on trust metrics","Input×Gradient tops Pareto front for GNN explainability scores","No universal GNN explainer winner: simple gradients lead trust tests","Ground-truth-free scores put gradients ahead of complex GNN methods","Gradients non-dominated across GNN faithfulness and stability metrics"]},"model":"grok-4.5","effort":"low","cost_usd":0.004814,"raw_usage":{"total_tokens":1297,"prompt_tokens":694,"num_sources_used":0,"completion_tokens":90,"cost_in_usd_ticks":48140000,"prompt_tokens_details":{"text_tokens":694,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":513,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":694,"tokens_out":90,"duration_ms":4250,"temperature":1.0,"reasoning_tokens":513,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T16:36:23.076259+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out proprietary graph task that also carries human-validated rationales, the Pareto ranking produced by the four metrics places explainers that auditors reject above those they accept, or Input×Gradient’s high Pertinence fails to correlate with auditor preference.","supporting_citations":[],"review_version":1}