{"id":"698ef3f4-f374-4641-ac3a-cd00d9bc8cad","arxiv_id":"2412.11964","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A variational-inference explainer that uses a Beta-distributed edge mask to identify important edges in graph neural networks, with uncertainty scores, evaluated on simulated graphs.","lead":"BetaExplainer is a new method that learns a probability for each edge in a graph, showing which edges a graph neural network relies on to make a prediction. It is designed to give uncertainty estimates for edge importance and to work on hard cases like sparse or heterophilic graphs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core variational derivation is invalid: Eq. 1 defines a likelihood that does not depend on the observed GNN output, and Eqs. 3–5 do not form an ELBO, so the claimed uncertainty quantification is not mathematically grounded.","rationale":"The reader's weakest assumption identifies the same core defect: the likelihood and ELBO in Section 2.1 are not mathematically valid. I agree that this is the load-bearing point. The paper's headline novelty is probabilistic edge importance with uncertainty, not just a heuristic mask; if the variational objective is invalid, those uncertainty scores have no principled interpretation. The empirical F1/Jaccard numbers could still be reproducible from the public code, and I credit the authors for releasing it, but a reproducible implementation does not substitute for a correct derivation of the method being proposed. My concrete test is designed to distinguish between a typographical error and an actual independence from the model output; either way, the current text needs correction. I also noted the circularity of reporting unfaithfulness as evidence when the training objective is the same KL divergence, though that is secondary to the variational failure. Because the reader already recommends REJECT and my analysis supports that conclusion, I do not change the verdict.","tokens_in":11818,"tokens_out":7476,"duration_ms":66627,"concrete_test":"Download the released repository, locate the Pyro model and guide, and rerun BetaExplainer on one SERGIO or ShapeGGen dataset with the original GNN output f(X,G) and again with a randomly permuted copy of f(X,G). If the learned edge masks are identical under permutation, the objective is independent of the explained model, confirming that Eq. 1 is the operative likelihood and the explanation/uncertainty claim fails. If the masks change, the code uses a valid dependence on the output, and the paper's equations are mis-specified rather than load-bearing, requiring a corrected derivation before acceptance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"BetaExplainer's central contribution is a probabilistic edge mask with trustworthy uncertainty scores. That contribution rests on the variational objective, which is not well-defined as written. In Eq. 1, P(f(X,E)|M) is set to p if M >= 0.5 and 1-p otherwise; p is never specified and the expression has no dependence on the actual GNN output f(X,E). With such a likelihood, the posterior over M is a fixed reweighting of the Beta prior and cannot encode which edges drive the prediction. Eq. 3 is not an ELBO: it omits the variational distribution q and misplaces the logarithms, and Eq. 5 asserts a decomposition into a KL term that follows from no standard variational identity. The training objective described in the paper therefore cannot be the one that produces the reported masks. The code may implement a valid objective, but the manuscript does not say so. A secondary issue compounds this: the unfaithfulness metric is the same KL divergence the method minimizes, so improvements on that metric are partly circular and do not independently validate the explanations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BetaExplainer, a method intended to explain graph neural network predictions by learning a Beta-distributed probabilistic edge mask. The stated goal is to provide edge importance scores with uncertainty quantification and to improve explanation accuracy over GNNExplainer and SubgraphX, particularly on simulated datasets with challenging properties such as heterophily, fairness-related feature flips, low information content, and sparse node features. The authors present a variational-inference formulation (Eqs. 1–5), an algorithm in Section 2.1, and experiments on ShapeGGen and SERGIO datasets comparing Jaccard index, F1 score, accuracy, and unfaithfulness. The paper reports that BetaExplainer outperforms baselines on several metrics, especially unfaithfulness, and provides uncertainty scores for edges.","tokens_in":12035,"tokens_out":4061,"duration_ms":40110,"significance":"If the proposed method were correctly formulated, the contribution would be practically useful: a probabilistic edge mask with uncertainty quantification could aid hypothesis generation in domains such as gene-regulatory-network analysis, and the inclusion of sparse-feature datasets addresses a realistic gap in GNN explainability benchmarks. The code is publicly available under an open-source license, which is a strength. However, the significance cannot be assessed from the manuscript as written because the variational derivation is mathematically invalid, the training objective is not well defined, and the main unfaithfulness metric is circular with respect to the training loss. The independent metrics (F1, Jaccard) show only partial advantages, and these are weakened by per-dataset hyperparameter selection. The central claims therefore remain unsubstantiated.","major_comments":[{"comment":"The likelihood in Eq. (1) is not a valid likelihood for the observed GNN output. It defines P(f(X,E)|M) as p when M ≥ 0.5 and 1−p otherwise, but p is never specified and the expression has no dependence on the actual value of f(X,E). A likelihood that does not depend on the observation cannot update the posterior toward edges that drive the prediction; the posterior would be a fixed reweighting of the Beta prior. This is a load-bearing error because the paper's claimed uncertainty quantification is derived from this likelihood.","section":"§2.1, Eq. (1)"},{"comment":"Equations (3)–(5) do not constitute an evidence lower bound or a variational inference objective. Eq. (3) omits the variational distribution q, contains a mismatched parenthesis, and is written as an expectation of a difference of log-probabilities without the standard ELBO structure. Eq. (5) equates a difference between a log-probability and the ELBO with a KL divergence term that follows from no standard variational identity, and the stated nonnegativity is unsupported. As written, the training objective in Algorithm 1 cannot be derived from variational inference. The code may implement a valid objective, but the manuscript does not provide it.","section":"§2.1, Eqs. (3)–(5)"},{"comment":"The unfaithfulness metric in Eq. (9) is 1−exp(−KL(f(X,G)||f(X,G_s))), while BetaExplainer's stated training objective is to minimize the KL divergence between the GNN output on the masked graph and the original output. The paper itself acknowledges this in Section 3.1: 'BetaExplainer optimizes the KL divergence between the GNN output on the masked graph and the original output, which is also measured by the unfaithfulness metric. This likely explains the decreased unfaithfulness score.' Consequently, the reported improvements in unfaithfulness are circular and do not independently validate the quality of the explanations.","section":"§2.3.3, Eq. (9) and §3.1"},{"comment":"The alpha and beta prior parameters are selected per dataset 'based on the most balanced results across metrics' (Section 2.3.2). Because these parameters directly control the sparsity and shape of the learned mask, tuning them on the evaluation metrics and then reporting improvements on those same metrics raises selection-overfitting concerns. The paper does not provide a validation protocol, sensitivity analysis, or held-out hyperparameter selection, which weakens the generality of the empirical claims.","section":"§2.3.2, Table 2"}],"minor_comments":[{"comment":"The abstract says 'we proposed BetaExplainer' while the introduction says 'We propose'; the tense should be consistent. The last paragraph of the Introduction also contains 'have evaluated' without a subject, likely 'we have evaluated'.","section":"§1, Abstract"},{"comment":"Equation (2) is a probability density, not a probability mass; the surrounding text refers to it as the 'probability that the mask importance for edge e_ij is value M_ij', which is imprecise and should be reworded.","section":"§2.1, Eq. (2)"},{"comment":"In the SERGIO 25% Sparsity row, the Type column contains the word 'Phenomenon', which appears to be a placeholder or typo; this should be corrected or explained.","section":"Table 2"},{"comment":"Reference [21] lists page numbers '1241–12252', which appear to be a typo; the correct page range should be verified.","section":"References"},{"comment":"The discussion of SubgraphX returning no edges would benefit from explicitly stating that an empty explanation graph makes accuracy and F1 comparisons degenerate; the current text mentions this only indirectly.","section":"§3.2, Fig. 4"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an early draft with multiple undefined symbols, typos, and an invalid mathematical core. The central variational derivation cannot be repaired by local edits; it would require redefining the likelihood and the training objective and then re-running the experiments. The circular unfaithfulness metric further weakens the main empirical evidence. The open-source code link is a positive element, but I did not verify the code, and the manuscript itself does not state the actual objective used. I would not encourage resubmission in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BetaExplainer is a useful idea attached to a shaky derivation. The paper's central contribution is a probabilistic edge mask for GNN explanations, with posterior uncertainty estimates. That's worth having, and the empirical effort is real: they run on ShapeGGen and SERGIO simulations, report F1 and Jaccard, and the code is on GitHub. The F1 improvements on sparse SERGIO data are not an artifact of the loss, so there's something there.\n\nThe problem is the math. Eq. 1 defines a 'likelihood' that doesn't depend on the GNN output at all—it just flips a coin based on whether the mask exceeds 0.5. That isn't a likelihood for f(X,E). The ELBO equations (3–5) are not a valid evidence lower bound; they're missing the variational distribution and the log placement is wrong. As written, the method's uncertainty quantification is not grounded. A reviewer can't reproduce the objective from the paper alone, and the stress-test note is right.\n\nThe unfaithfulness metric also gets a fair share of the credit for 'improvements' that are partly circular, since it's the same KL divergence being minimized. The authors acknowledge this in the text, which is honest, but it means that particular result doesn't independently validate the method.\n\nAnother soft spot: alpha and beta are tuned per dataset to maximize the reported metrics. That's not necessarily fatal on simulated benchmarks, but it means the prior is doing some of the work, and there's no proper validation split for those hyperparameters.\n\nNet assessment: the core idea is a modest but reasonable extension of GNNExplainer, and the sparse-feature experiments are relevant. But the paper as written cannot be accepted: the variational derivation needs to be corrected, and the evaluation needs to lean on metrics that aren't the training loss. I'd engage with a revised version—the code exists, and the problem matters. For peer review, I'd send it out with a clear signal that the derivation must be fixed, rather than desk-rejecting, because there's a real empirical component and a correctable flaw.","headline":"A promising idea with a shaky variational derivation and a too-circular evaluation metric; worth a revision, not acceptance as is.","tokens_in":12566,"tokens_out":2791,"would_cite":false,"duration_ms":25030,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BetaExplainer frames GNN edge explanation as posterior inference over a Beta-distributed edge mask, returning ranked importance scores with uncertainty and improving explanation fidelity on heterophilic and sparse-feature graphs.","keywords":["graph neural networks","explainability","edge mask","variational inference","Beta distribution","uncertainty quantification","sparse node features","graph classification"],"falsifier":"Run BetaExplainer on a small graph with a known ground-truth edge, set the constant in Eq. 1 to 0.5, and check whether maximizing the stated ELBO recovers that edge and whether the learned posterior separates true from false edges at the 0.5 boundary; then compute both sides of Eq. 5 to verify that the claimed lower-bound identity actually holds numerically.","tokens_in":11609,"feed_emoji":"🕸️","tokens_out":5006,"duration_ms":44774,"temperature":0.7,"pith_summary":"Graph neural network predictions are hard to trace back to the edges that drive them, and existing edge explainers either cannot rank edges or lose accuracy on graphs with heterophily or sparse node features. BetaExplainer tries to close both gaps by learning a probabilistic mask over edges: each edge receives a Beta-distributed importance score, so the explanation is a ranked list with an uncertainty value per edge. The paper claims that on five simulated ShapeGGen-style datasets this mask is significantly more faithful than GNNExplainer and SubgraphX, and that on simulated sparse gene-expression graphs it achieves a better precision-recall tradeoff. If correct, the method gives practitioners a principled way to prioritize the most likely important edges, which matters for tasks such as gene interaction discovery where experimental follow-up is expensive.","feed_headline":"Probabilistic edge masks explain GNNs with uncertainty scores","feed_subtitle":"BetaExplainer learns rankable edge importance and beats baselines on heterophilic and sparse-feature graphs.","key_machinery":"The load-bearing object is the probabilistic edge mask $M_{ij}$ with a Beta prior $\\mathrm{Beta}(\\alpha,\\beta)$ per edge, learned through mean-field variational inference. Each mask entry controls whether edge $(i,j)$ is retained in the graph $\\mathbf{G}_s$ fed to the trained GNN, and the loss is the KL divergence between the GNN output on the masked graph and the original output; optimizing the ELBO is claimed to minimize this divergence. The Beta distribution is chosen because it is the conjugate prior for the Bernoulli likelihood in Eq. 1, and its two shape parameters let the user encode prior beliefs, which the paper argues is what lets the method adapt to heterophilic and sparse-feature graphs.","core_discovery":"The central claim is that a GNN explanation can be obtained by posterior inference over a binary edge mask, rather than by direct optimization of a mask or by subgraph search. BetaExplainer places an independent Beta prior on each edge's mask value, defines a Bernoulli-like likelihood comparing the trained GNN's output on the masked graph with its output on the original graph, and fits the posterior with black-box variational inference. The learned posterior mean provides a ranked importance score, and the spread of the distribution provides an uncertainty estimate that neither GNNExplainer nor SubgraphX offers. On the paper's simulated benchmarks this probabilistic mask yields significantly better Jaccard and F1 scores on some datasets, comparable accuracy on sparse-feature graphs with significantly better F1, and lower unfaithfulness on most datasets, indicating that the subgraph selected by probability matches the ground-truth motif better than the baselines.","pith_inferences":["The paper does not test calibration of the uncertainty scores; an editor would hypothesize that the posterior probability separates true from false edges by rank, and that a user could set a threshold to target a desired false-positive rate - a testable extension.","Because the objective is KL divergence between GNN outputs, the same Beta-mask framework could be extended to explain node-level predictions or to produce counterfactual edge deletions, though the paper only evaluates graph-level explanations.","The fixed 0.5 threshold in the likelihood is a discretization choice; a continuous relaxation might improve separation on datasets where the importance distribution is not symmetric around the threshold."],"forward_implications":["If BetaExplainer's edge scores are reliable, users can threshold the posterior probabilities to trade precision against recall, choosing a smaller set of high-confidence edges for costly downstream validation.","Because the method returns a distribution rather than a point mask, repeated runs can be aggregated into per-edge confidence intervals, giving an explanation that is stable across random seeds.","The improved unfaithfulness on heterophilic graphs suggests the same probabilistic masking strategy can be applied where explainers typically fail, such as protein-structure graphs with mixed node types.","On sparse gene-expression graphs, better F1 at fixed accuracy means BetaExplainer can highlight a larger fraction of true regulatory interactions without flooding the user with false positives.","Batched training should make BetaExplainer practical on whole-graph datasets, since the paper shows batching cuts runtime dramatically without changing the reported metric patterns."],"supporting_citations":[{"why":"Supplies the ShapeGGen simulator, the five simulated benchmark datasets, and the unfaithfulness metric that define the evaluation.","marker":"[1]"},{"why":"GNNExplainer is the primary state-of-the-art baseline whose edge-mask optimization BetaExplainer is compared against.","marker":"[19]"},{"why":"SubgraphX is the second baseline, a subgraph-search explainer that returns binary edge decisions without ranking.","marker":"[21]"},{"why":"SERGIO provides the sparse single-cell gene-expression simulation with a known ground-truth regulatory network used in the sparsity experiments.","marker":"[5]"},{"why":"Documents the lack of uncertainty quantification and edge ranking in existing explainers, motivating BetaExplainer's probabilistic scores.","marker":"[20]"}],"fun_headline_variants":["BetaExplainer: GNN explanations with uncertainty","Probabilistic edge masks beat GNN baselines","Uncertainty-aware GNN explanations via Beta prior","Explaining GNNs with rankable edge importance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the simple Bernoulli-style likelihood with a fixed 0.5 threshold and an unspecified constant is a valid description of how well a masked graph reproduces the GNN's output, and that the variational objective used for training is a genuine lower bound that can be optimized in this form.","fun_headline_variants_meta":{"raw":{"variants":["BetaExplainer: GNN explanations with uncertainty","Probabilistic edge masks beat GNN baselines","Uncertainty-aware GNN explanations via Beta prior","Explaining GNNs with rankable edge importance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000123,"raw_usage":{"total_tokens":1045,"prompt_tokens":836,"completion_tokens":209,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":147}},"tokens_in":452,"tokens_out":209,"duration_ms":2649,"temperature":1.0,"reasoning_tokens":147,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:24:52.220806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BetaExplainer on a small graph with a known ground-truth edge, set the constant in Eq. 1 to 0.5, and check whether maximizing the stated ELBO recovers that edge and whether the learned posterior separates true from false edges at the 0.5 boundary; then compute both sides of Eq. 5 to verify that the claimed lower-bound identity actually holds numerically.","supporting_citations":[{"cited_title":"Agarwal, O","cited_arxiv_id":null,"evidence_quote":"Supplies the ShapeGGen simulator, the five simulated benchmark datasets, and the unfaithfulness metric that define the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GNNExplainer is the primary state-of-the-art baseline whose edge-mask optimization BetaExplainer is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SubgraphX is the second baseline, a subgraph-search explainer that returns binary edge decisions without ranking."},{"cited_title":"Dibaeinia and S","cited_arxiv_id":null,"evidence_quote":"SERGIO provides the sparse single-cell gene-expression simulation with a known ground-truth regulatory network used in the sparsity experiments."},{"cited_title":"Y uan, H","cited_arxiv_id":null,"evidence_quote":"Documents the lack of uncertainty quantification and edge ranking in existing explainers, motivating BetaExplainer's probabilistic scores."}],"review_version":1}