{"id":"99dbfde8-014c-435f-9f27-3725b1db426d","arxiv_id":"2505.16893","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A selective inference framework computes valid p-values for GNN saliency maps by conditioning on the selected salient subgraph, controlling the Type I error rate.","lead":"This paper proposes a statistical test that tells you whether a graph neural network's saliency map highlights real signal or just noise. It uses selective inference to correct the double-dipping problem, where the same data is used both to select and to test the highlighted region.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's unconditional Type I error control is not established: the proof marginalizes over the data-dependent selection V_X with ill-defined probabilities, so only the conditional SI guarantee holds.","rationale":"The most load-bearing concern is internal to the theorem. The paper's central claim of exact, non-asymptotic Type I error control rests entirely on Theorem 1. The proof of Theorem 1 has a logical gap: it marginalizes over the selection event V_X using probabilities P_H0(V_X=V_obs) as if H0 specified a single sampling distribution under which all selections satisfy the null. But H0 in (5) is a constraint on mu that depends on the realized V_X; for a fixed mu it holds for some subgraphs and fails for others. The conditional p-value is uniform only on the event where eta(V_obs)^T mu = 0. Summing over all V_obs therefore yields alpha only if the null holds for every selection, which is the global null mu=0, not the hypothesis in (5). This is a mathematical error, not a mere assumption mismatch. The reader's weakest_assumption (estimated Sigma, GNN independence) concerns the validity of the model for real data; even if those assumptions were satisfied, Theorem 1 as stated is not proven. The conditional SI guarantee, which is the standard result, is sufficient for the paper's practical claims, so the verdict remains CONDITIONAL: the authors should restate Theorem 1 conditionally on the selection event and remove the invalid marginalization. I disagree with the reader's identification of the load-bearing concern, though the final verdict is unchanged.","tokens_in":57220,"tokens_out":14920,"duration_ms":117783,"concrete_test":"Using the paper's code, generate synthetic data with n=4, d=1, mu=(1,-1,0,0), Sigma=I, and thresholds tau_l=0.3, tau_u=0.7. For each sample, compute p_selective over many replicates. Compare the unconditional rejection rate at alpha=0.05 with the conditional rejection rate restricted to samples whose selected V satisfies H0. If the unconditional rate deviates from 0.05 while the conditional rate is 0.05, Theorem 1's unconditional statement is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Theorem 1, Eq. 8) is that P_H0(p_selective <= alpha) = alpha for the data-dependent null H0 in (5). The proof in Appendix C.2 establishes the standard conditional result p_selective | {V_X=V_obs, Q_X=q} ~ Unif(0,1) under eta(V_obs)^T mu = 0. It then marginalizes over Q, obtaining P(p<=alpha | V_X=V_obs) = alpha, which is correct. The final step marginalizes over V_obs using weights P_H0(V_X=V_obs). This step is invalid: H0 in (5) is not a single distribution but a family of constraints on mu indexed by the selected subgraph. For a fixed true mu, the conditional null eta(V)^T mu = 0 holds only for a subset of possible subgraphs V; for other V, the conditional rejection probability is the power, not alpha. The proof implicitly assumes that, under H0, every possible V satisfies the null, which is generally false unless the global null mu=0 holds. Consequently the unconditional equality in (8) does not follow and is not generally true; only the standard conditional SI guarantee holds. Since the abstract and Section 3.2 present (8) as the definition of validity, this misstatement is load-bearing. The fix is to state Theorem 1 conditionally on V_X=V_obs, which is the accepted SI notion of Type I error control and is sufficient for the paper's claims.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a selective-inference (SI) framework for testing the statistical significance of GNN saliency maps. Under the model X = mu + epsilon with Gaussian noise and known covariance, the authors define salient and non-salient subgraphs by thresholding a CAM-based saliency map, introduce a data-driven null hypothesis comparing mean features of the two subgraphs, and compute a selective p-value conditioned on the observed subgraph selection and a sufficient statistic for the nuisance parameter. The p-value is computed exactly by exploiting piecewise linearity of the GNN-CAM map and reducing the conditional data space to a one-dimensional search via parametric programming. The authors claim unconditional Type I error control (Theorem 1) and support this with synthetic and EEG experiments.","tokens_in":57486,"tokens_out":5176,"duration_ms":50874,"significance":"If the central claims are correctly stated, the paper makes a useful contribution: it is among the first to apply exact selective inference to GNN saliency maps, and the conditional SI guarantee is obtained non-asymptotically for a broad class of piecewise-linear GNN architectures and saliency methods. The paper also ships reproducible code and reports extensive experiments, including robustness checks under non-Gaussian noise, estimated covariance, varied thresholds, and multiple GNN/saliency combinations. These are concrete strengths. However, the paper's headline validity statement in Theorem 1 is stated as an unconditional guarantee, and that statement is not supported by the proof; the conditional SI guarantee underlying the method is standard and likely sufficient for the paper's practical claims, but the unconditional claim needs correction.","major_comments":[{"comment":"The unconditional Type I error claim in Theorem 1 is not established. The proof in Appendix C.2 first shows that, conditional on V_X = V_obs and Q_X = Q_obs, the selective p-value is uniform under eta(V_obs)^T mu = 0. This is the standard SI result. The subsequent marginalization over Q_X given V_X is valid. The final marginalization over V_X, however, uses probabilities P_H0(V_X = V_obs) for a hypothesis H0 that is not a single distribution but a family of constraints indexed by the selected subgraph. For a fixed nonzero mu, eta(V)^T mu = 0 holds only for those subgraphs V for which the observed selection is exactly the one satisfying the null; for other subgraphs the conditional rejection probability is a power, not alpha. Hence the equality P_H0(p_selective <= alpha) = alpha in Eq. (8) does not follow except under a global null such as mu = 0, or when the statement is understood conditionally on the observed selection. The fix is to restate Theorem 1 conditionally on V_X = V_obs (and Q_X = Q_obs), which is the accepted SI notion of Type I error control and is sufficient for the method's practical validity claim.","section":"Section 4.2, Theorem 1, Appendix C.2"},{"comment":"The claim that the method makes 'no assumptions on the training data or training process' is too strong. Theorem 1 treats the GNN weights as fixed inputs: the distributional calculation in Section 3.1 concerns only the test feature vector X, and the selection event is defined through a fixed network. If the GNN is trained on data that include the test instance, or if the weights are random and dependent on X, the saliency map and the selection event are not exogenous and the stated guarantee does not follow. The method is valid for a fixed, pre-trained GNN used on an independent test instance. The sentence 'Our method guarantees control of the Type I error rate even when the GNN is trained on ill-conditioned data' should be qualified accordingly.","section":"Section 4.2, paragraph beginning 'Furthermore, we emphasize'"},{"comment":"There is a dimension mismatch in Lemma 1. The lemma states that S_i(G_X) = C_i^(k) X + b_i^(k) with C_i^(k) in R^{n x nd} and b_i^(k) in R^n. Since S_i(G_X) is a scalar for each node i, C_i^(k) should be a row vector in R^{1 x nd} and b_i^(k) a scalar in R. As written, the expression C_i^(k) X + b_i^(k) would be n-dimensional, which contradicts the scalar nature of the saliency score. This is a local typo, but it occurs in a formally stated lemma and should be corrected.","section":"Section 4.3, Lemma 1"}],"minor_comments":[{"comment":"The paper first defines T(X) as the raw mean difference in Eq. (6) and then redefines it as a normalized version in Eq. (7) 'without loss of generality'. Please state explicitly that the statistical test and the selective p-value are defined for the normalized statistic, and clarify that eta in Eq. (7) is the same eta defined in Appendix B.","section":"Section 3.2, Eq. (7)"},{"comment":"The interval formulas divide by beta_i, but the beta_i = 0 case is not discussed. When the saliency score is constant in z for some node, the threshold inequality imposes either no constraint or an infeasible constraint, and the corresponding interval branch should be specified. Please add a sentence handling zero-slope cases.","section":"Appendix D.1, Eqs. (17)-(18)"},{"comment":"The exact validity statement in Theorem 1 assumes a known covariance Sigma. In Appendix E.2, Sigma is estimated from data and the Type I error is evaluated only empirically. The main text should explicitly note that the theoretical guarantee does not extend to the estimated-covariance experiment, which is a robustness check rather than an exact result.","section":"Appendix E.2 and footnote 3"},{"comment":"There are several typographical and minor language slips, for example 'the propery in(8)', 'satistifies', and 'wchich'. A careful proofreading pass is recommended before publication.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The unconditional statement of Theorem 1 is the main technical issue. I believe the paper can be brought to an acceptable form by restating the validity claim conditionally on the selection event, which is the standard and correct SI formulation, and by clarifying the fixed-GNN assumption. The conditional inference machinery and the experimental evaluation are otherwise sound and well within the scope of a statistical ML venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper is a genuine contribution, but Theorem 1 as stated is too strong. The conditional selective-inference guarantee — p_selective is uniform given the selected subgraph and the nuisance statistic, under the conditional null — is correct and is what the proof actually establishes. The unconditional equality in (8) does not follow, because H0 in (5) is a family indexed by the data-selected V_X; for a fixed true mu, only some V satisfy the null. The final marginalization over V_obs uses weights P_H0(V_X=V_obs) that are not well-defined for a single H0. So the abstract's claim of unconditional Type I error control is not supported. This is load-bearing, but the fix is easy: state Theorem 1 conditionally, which is the standard SI notion and sufficient for the applications.\n\nWhat is genuinely new: the first application of selective inference to GNN saliency maps, with a parametric programming algorithm for the graph-structured truncation set. The piecewise-linearity of CAM in GCNs is used cleanly. Experiments are decent: synthetic type-I error and power across n, d, and correlation structures; EEG analysis consistent with P300; robustness checks for non-Gaussian noise and estimated covariance. Code is available.\n\nSoft spots, in proportion: the known-Sigma assumption is acknowledged, but the estimated-Sigma case is only empirical — fine as robustness, but not exact. The Section 4.2 claim of no assumptions on training data is only correct if the GNN is independent of the test data; otherwise training is another data-dependent selection step not conditioned on. That overclaim should be softened. Lemma 1 has a dimension typo (C_i should be a row vector, b_i a scalar) — minor.\n\nOverall: the conditional inference machinery is standard and correctly applied, and the paper deserves serious refereeing. It needs a revision that restates the guarantee conditionally and walks back the training-data claim. With that, it is a solid contribution for GNN interpretability practitioners.\n\nRecommendation: send to peer review, expect major revision but no fundamental obstacle.","headline":"Useful selective-inference application to GNN saliency maps, but the unconditional Type I error claim in Theorem 1 is not supported; the conditional guarantee is correct and sufficient.","tokens_in":58034,"tokens_out":2242,"would_cite":false,"duration_ms":21266,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Conditioning on the selected subgraph turns GNN saliency maps into exact, finite-sample hypothesis tests, so a highlighted region can be called meaningful only when the difference between salient and non-salient nodes is unlikely under…","keywords":["graph neural networks","saliency maps","selective inference","post-selection inference","Type I error control","piecewise linearity","Class Activation Mapping","EEG analysis"],"falsifier":"Simulate many graphs under the null hypothesis with known covariance but non-Gaussian heavy-tailed noise, run the published algorithm at α = 0.05, and count the rejection rate: a clear departure from 0.05 beyond Monte Carlo error would falsify the exact claim. A second direct check is to train the GNN on the same data that are then tested; if the empirical Type I error exceeds the nominal level, the claim that the training process needs no assumptions fails in that regime.","tokens_in":56997,"feed_emoji":"📊","tokens_out":7518,"duration_ms":59816,"temperature":0.7,"pith_summary":"Graph neural network explanations are usually presented as saliency maps, but the same data that highlight the salient subgraph are then used to judge whether that subgraph is meaningful, which is the double-dipping problem and inflates false positives. This paper claims that selective inference removes that inflation: by conditioning on the selected salient and non-salient subgraphs and on a sufficient statistic for the nuisance parameters, the test statistic follows a truncated normal distribution, and the resulting selective p-value is exactly uniform under the null hypothesis. The consequence is a statistical certification for CAM-style GNN saliency maps: a highlighted subgraph is declared meaningful only when the mean feature difference between the salient and non-salient regions would rarely be as large under noise. The guarantee is non-asymptotic, so it holds at finite sample sizes, and the same conditioning scheme extends to other piecewise-linear saliency methods such as Grad-CAM applied to GCNs and GINs.","feed_headline":"Exact p-values for GNN saliency maps end double-dipping bias","feed_subtitle":"Selective inference turns GNN saliency highlights into certified findings, keeping false positives at any α.","key_machinery":"The load-bearing object is the selective p-value, defined by conditioning the test statistic T(X) = η^T X / $\\sqrt$(η^T Σ η) on the event that the same salient and non-salient subgraphs are selected and on the sufficient statistic Q_X that removes the nuisance parameter. Two structural facts make this computable: Lemma 1, that the CAM saliency map is a piecewise linear function of the input features for ReLU-based GCN and GIN architectures, and Lemma 2, that the conditional data space collapses to a one-dimensional line X(z) = a + b z along which the truncation set Z is a union of intervals. The intervals are found by parametric programming, solving systems of linear inequalities derived from the thresholding rules that define the salient and non-salient subgraphs. This turns a high-dimensional inverse problem into a one-dimensional search, and the final p-value is the tail probability of a truncated standard normal over Z.","core_discovery":"The paper's central claim is Theorem 1: under the null hypothesis H0 in equation (5), which says that the mean feature values of the salient and non-salient subgraphs are equal, the selective p-value in equation (13) satisfies P_H0(p_selective ≤ α) = α for every α in (0,1). This is an exact finite-sample statement, not an asymptotic approximation. The mechanism is conditioning on V_X, the selected subgraph pair, and on Q_X defined in equation (11); under H0 the conditional test statistic is a truncated standard normal, so the probability integral transform gives conditional uniformity. After marginalizing over the conditioning variables, the paper obtains the exact Type I error control. Because the GNN and its weights are treated as fixed after training, the paper states that Theorem 1 makes no additional assumption on the training data or training process. In experiments, the naive p-value over-rejects because it ignores selection, Bonferroni correction controls error but is too conservative, while the proposed selective p-value keeps error at the nominal level and has higher power; on the EEG dataset, positive samples containing P300 responses receive small selective p-values and negative samples receive large ones.","pith_inferences":["Beyond the paper: the exact guarantee should be read as conditional on the GNN being fixed before the test data are seen; the conditioning event covers only the subgraph selection, not any training or retraining on the test instance, so a fully exact statement for end-to-end learned explanations would need a model for the training process.","Beyond the paper: because the framework handles any piecewise-linear saliency method, approximating softmax attention in graph transformers by piecewise-linear splines would open the same testing scheme to attention-based GNNs, though the paper does not make that claim.","Beyond the paper: the paper leaves Lemma 1 unproved with a note that it follows from standard composition results; supplying the full proof would remove the only structural gap between the algorithmic recipe and the theorem.","Beyond the paper: using the selective p-value as a gate before interpretability evaluation could reduce false discoveries in neuroscience pipelines without replacing existing explanation metrics, a synthesis the paper only sketches as future work."],"forward_implications":["Saliency maps from GNNs can be reported with exact p-values, so a highlighted subgraph is statistically certified only when the salient versus non-salient difference is extreme under the null.","The Type I error guarantee holds at any significance level and for finite numbers of nodes and features, so small-sample applications such as EEG studies do not need asymptotic sample-size justifications.","Any saliency method that is piecewise linear in the features, including Grad-CAM, Grad, and Grad-Input on GCN and GIN models, can use the same conditioning and one-dimensional search.","The test is valid for any fixed pair of thresholds used to define salient and non-salient subgraphs, including after normalizing saliency values to the range [0,1]."],"supporting_citations":[{"why":"Supplies the exact post-selection inference machinery, truncated normal p-values after conditioning on selection, that Theorem 1 adapts to GNN saliency.","marker":"Lee et al., 2016"},{"why":"Provides the selective-inference conditioning principle and nuisance-parameter elimination that define the conditional test.","marker":"Fithian et al., 2014"},{"why":"Introduces parametric programming for selective inference, the technique used here to compute the truncation intervals Z.","marker":"Duy & Takeuchi, 2022"},{"why":"Defines CAM and Grad-CAM for graph convolutional networks, the saliency methods whose piecewise linearity drives Lemma 1.","marker":"Pope et al., 2019"},{"why":"Documents the double-dipping problem in neuroscience that motivates the need for selection-adjusted p-values.","marker":"Kriegeskorte et al., 2009"},{"why":"Provides the EEG dataset used to validate the method on real data and to compare against known P300 neurophysiology.","marker":"Won et al., 2022"},{"why":"Defines the GIN architecture used in the piecewise-linearity extension experiments.","marker":"Xu et al., 2019"}],"fun_headline_variants":["Exact p-values for GNN saliency maps","GNN saliency maps get statistically valid p-values","Selective inference fixes double-dipping in GNN saliency","Exact tests guarantee GNN saliency maps aren't noise","Controlling false positives in GNN saliency maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The exact Type I error guarantee rests on the feature vector being generated as Gaussian noise with known covariance matrix and on the GNN being fixed independently of the test data: if the covariance is estimated from the same data or the network is trained or retrained on the test instance, Theorem 1 no longer applies and only empirical error control is reported.","fun_headline_variants_meta":{"raw":{"variants":["Exact p-values for GNN saliency maps","GNN saliency maps get statistically valid p-values","Selective inference fixes double-dipping in GNN saliency","Exact tests guarantee GNN saliency maps aren't noise","Controlling false positives in GNN saliency maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1380,"prompt_tokens":965,"completion_tokens":415,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":335}},"tokens_in":581,"tokens_out":415,"duration_ms":3192,"temperature":1.0,"reasoning_tokens":335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:53:06.812399+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate many graphs under the null hypothesis with known covariance but non-Gaussian heavy-tailed noise, run the published algorithm at α = 0.05, and count the rejection rate: a clear departure from 0.05 beyond Monte Carlo error would falsify the exact claim. A second direct check is to train the GNN on the same data that are then tested; if the empirical Type I error exceeds the nominal level, the claim that the training process needs no assumptions fails in that regime.","supporting_citations":[],"review_version":1}