{"id":"859236b3-e53e-4e8d-a189-6a3b4eaa0dd5","arxiv_id":"2412.10942","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"MAX-Sensitivity and AVG-Sensitivity report near-perfect stability on randomly generated explanations and predictions, failing a simple random-output sanity test.","lead":"This paper proposes two benchmark tests, PET and ROT, to check whether XAI stability metrics give sensible answers on perfectly explainable and purely random cases. Applied to MAX-Sensitivity and AVG-Sensitivity, both metrics pass the perfect-explanation test but fail to flag random explanations as unstable, which the authors read as evidence the metrics are unreliable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ROT near-zero values are likely an implementation artifact: §5.2 filters perturbations by prediction difference, contradicting Eqs. 1–3, so the claim that MAX- and AVG-Sensitivity are unreliable is not established.","rationale":"The reader's weakest assumption identifies the same load-bearing risk: the ROT result may come from a miscoded perturbation condition rather than from the metrics defined in Eqs. 1–3. The paper's own §5.2 provides direct textual evidence for this risk by describing the accepted perturbations in terms of prediction difference, not input distance. With random Gaussian explanations that are independent across inputs, the expected Frobenius norm of f(x_j) − f(x_i) is far from zero, so the reported values near 0.01 require an explanation; filtering the sampled set by output closeness supplies that explanation but invalidates the experiment as a test of the stated metrics. The missing experimental parameters (epsilon, Monte Carlo count, dimensionality) and the inconsistent expected value in Table 4 versus §3.2 further prevent verification. The benchmark idea is useful and the PET result is credible, but the negative conclusion about MAX- and AVG-Sensitivity rests on the unverified ROT artifact. The reader's REJECT verdict stands; no adjustment is needed.","tokens_in":9252,"tokens_out":7064,"duration_ms":62179,"concrete_test":"Open the public repository (github.com/explainingAI/stability), locate the ROT perturbation sampling loop, and check the acceptance condition: is a neighbor kept when ||x_j − x_i|| < epsilon (Eq. 3) or when the prediction/explanation difference is below epsilon (as §5.2 says)? Then re-run the Random Output Test using only input-distance filtering, with the same epsilon and Monte Carlo count, and print the sampled neighbor norms; if average and maximum sensitivity rise far above 0.01, the reported near-zero values are a coding artifact rather than a property of the metrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends entirely on the Random Output Test: MAX- and AVG-Sensitivity return about 0.011 and 0.010 when explanations and predictions are random, while a correct metric should report large instability. For that result to indict the metrics, the experiment must instantiate Eqs. 1 and 2 faithfully: sample x_j with ||x_j − x_i|| < epsilon (Eq. 3), then compute ||f(x_j) − f(x_i)||. With f a random Gaussian field independent across inputs, that norm has a large nonzero expectation (approximately sqrt(2D)σ for D-dimensional explanations), so near-zero values are not a mathematical property of the metric. Section 5.2 instead explains the low values by saying 'the perturbation samples toke into account are the ones with a prediction difference ... lower than a threshold epsilon'; this conditions on prediction/explanation difference, not input distance, and would mechanically select only near-identical outputs, forcing MAX/AVG toward 0. No epsilon, Monte Carlo count, image dimensionality, or norm normalization is reported, so the values cannot be reproduced. Additionally, §3.2 states r = 0 means completely unrobust while Table 4 lists the expected value as 1.0, an internal inconsistency. The ROT finding is therefore an unverified implementation artifact, not evidence of metric failure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two a-priori tests, the Perfect Explanation Test (PET) and the Random Output Test (ROT), to meta-evaluate the stability measures MAX-Sensitivity and AVG-Sensitivity introduced by Yeh et al. In PET, a decision tree trained on the TXUXIv3 dataset with the ssin attribution function is used to supply supposedly perfect explanations; in ROT, explanations are drawn from a Gaussian distribution and predictions from a uniform distribution. Reported results are perfect scores (0.000) in PET and near-perfect scores (0.011 and 0.010) in ROT, from which the authors conclude that both stability measures fail to detect random, unrobust explanations and are therefore unreliable.","tokens_in":9462,"tokens_out":8541,"duration_ms":79475,"significance":"The conceptual idea of testing stability metrics against scenarios with known ground-truth robustness is useful and complements existing axiomatic meta-evaluation approaches. The proposed tests are simple, intuitive, and potentially cheap to apply to other stability metrics. If the ROT finding were established under a faithful implementation of Eqs. (1) and (2), it would be a significant negative result for two widely used sensitivity measures. The paper also honestly states the scope limitation that the tests work only as a benchmark. However, the empirical claim is not currently verifiable: the manuscript omits key experimental parameters, asserts rather than derives the expected robustness values, and contains an internal contradiction about whether 0 or 1 denotes perfect robustness. The central negative conclusion is therefore not supported by the evidence as presented.","major_comments":[{"comment":"The robustness scale is defined inconsistently across the two tests. Section 3.2 states that 'r = 0, where 0 value indicated a completely unrobust explanation' and that 'any robustness metric r_hat ≠ 0 is clearly an erroneous result.' Section 3.1 and Table 3 instead treat 0.0 as the perfect value ('where the least the better, being 0 the perfect value'), while Table 4 lists the expected ROT value as 1.0. The manuscript never reconciles whether a correct metric should output 0 or 1 in perfect versus random scenarios, so the claim that MAX- and AVG-Sensitivity 'failed' the ROT test has no coherent interpretation.","section":"§3.1–3.2, Tables 3–4"},{"comment":"The explanation of the near-zero ROT values contradicts the metric definitions. Equation (3) defines the perturbation set by input distance, ||x_j - x_i|| < epsilon, and Eq. (1) takes the maximum over that input ball. Section 5.2 instead says the low values arise because 'the perturbation samples toke into account are the ones with a prediction difference, respect to the original data, lower than a threshold epsilon.' If the implementation selected samples by output difference, it did not compute the metrics as defined; near-zero values would then be a mechanical artifact of selecting only near-identical outputs. The paper reports no epsilon, no Monte Carlo sample count, no image dimensionality, and no norm normalization, so the experiment cannot be reproduced. For a Gaussian explanation field that is independent across inputs, the expected value of ||f(x_j) - f(x_i)|| over an input ball is positive and grows with explanation dimension, so a faithful implementation of Eqs. (1)–(3) would not obviously produce values near 0.01.","section":"§5.2 vs. Eqs. (1)–(3)"},{"comment":"The PET asserts that the decision-tree explanation is perfect, but the explanation used is an ad-hoc Gini-importance vector (Eqs. 6–8), not a defined ground-truth explanation for the ssin attribution function (Eqs. 11–12). No argument is given for why this vector equals the true contribution of each feature to the prediction, so the claim that r = 1 in PET is not derived from known ground truth. Moreover, because the explanation is a deterministic function of the tree path, any perturbed input that stays in the same leaf yields exactly the same explanation; the perfect PET score (0.000) may therefore be tautological rather than evidence that the metrics correctly measure robustness.","section":"§4.1, PET ground truth"},{"comment":"The central methodological claim is that the proposed tests are 'completely verifiable, without depending on any novel axiom.' This requires that the expected robustness values be derived from the definitions of robustness and the test construction. The manuscript asserts these values but never proves them: it does not specify how the random explanation field enters the expectation in Eqs. (1)–(2), what the norm is normalized by, or why the expected ROT value should be exactly 1.0 rather than some other constant. Without such a derivation, the pass/fail criterion is an unsubstantiated assumption, not a verifiable benchmark.","section":"§3, expected-value derivation"}],"minor_comments":[{"comment":"There are several typos in the title and abstract, including 'MAX-Senstivity', 'AVG-Sensitiviy', and the header 'MAX-S ENSTIVITY'; these should be corrected.","section":"Title and abstract"},{"comment":"The summation index in both equations reads 'Pn i=i' and should be 'Σ_{i=1}^n'.","section":"Eqs. (9)–(10)"},{"comment":"The confidence-interval notation is internally inconsistent: '0.011 ± 0.003 (0.011, 0.011)' cannot all be true, and the construction of the interval and the meaning of the ± term should be stated explicitly.","section":"Table 4"},{"comment":"The conclusion says the ROT values are '0.11 and 0.010', but Table 4 reports 0.011 and 0.010; this numerical typo should be fixed.","section":"§6"},{"comment":"The test is called 'Random Output Test' in Section 3.2 but 'Random Explanation Test' in Section 4.4; the names should be consistent throughout.","section":"§3.2 vs. §4.4"},{"comment":"Reference [11] contains an empty author entry (', , Vedaldi'); the reference list should be checked for such formatting errors.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim depends entirely on the ROT experiment, and that experiment appears to be implemented in a way that contradicts the metric definitions in Eqs. (1)–(3). The internal contradiction between the r = 0 and r = 1 robustness scales further undermines the interpretation of both tests. Unless the authors can show with a correct implementation, full reporting of epsilon and Monte Carlo parameters, and a derivation of expected values that the metrics still return near-zero under random explanations, the conclusion of unreliability cannot stand. This is a load-bearing issue, not a presentational one."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark idea here is worth a quick read, but the paper's main negative result—that MAX- and AVG-Sensitivity are unreliable—is almost certainly an artifact of a misimplemented perturbation condition. Don't use the ROT numbers to discredit these metrics.\n\nWhat's new: a scenario-based meta-evaluation with two tests, PET and ROT, where the ground-truth stability is known a priori. This is a nice complement to the axiomatic approach in MetaQuantus. The PET setup with a decision tree is reasonable, and the result (both metrics return 0.0 for constant explanations) is what you'd expect. The paper also honestly states the benchmark's limitation: it can only discard metrics, not certify them for real scenarios.\n\nThe ROT result has three problems. First, Section 3.2 says a correct metric should give r=0 when outputs are unrobust, but Table 4 lists the expected value as 1.0. Contradiction. Second, Section 5.2 claims the low values arise because the metrics restrict perturbations to samples whose prediction difference is below epsilon. That's not what Eqs. 1–3 say; they condition on the input distance ||x_j - x_i||. If the code follows the text, it would be sampling by output similarity, which would force the sensitivity toward zero. Third, no epsilon, Monte Carlo sample count, image dimension, or normalization is reported, so the numbers can't be reproduced. For Gaussian noise explanations, ||f(x_j)-f(x_i)|| has a large positive expectation, so near-zero mean values should have raised a red flag. The PET result alone doesn't support the conclusion.\n\nThe citation pattern is okay; the dataset and fidelity meta-evaluation are from the same group, but they're used appropriately as a source of known ground-truth explanations. That's not a flaw.\n\nWho is this for? Someone working on XAI evaluation might want the benchmark idea. But the paper as written is not a reliable source for the claimed failure of MAX/AVG sensitivity. My recommendation: don't send this to peer review yet. The authors should correct the contradiction, describe the actual implementation, and ideally publish code. If the near-zero values persist after a faithful implementation of Eqs. 1–3, then the paper becomes interesting. As it stands, I'd treat the ROT finding as an artifact.","headline":"A useful benchmark idea, but the central ROT result looks like an implementation artifact, not a property of the metrics.","tokens_in":10049,"tokens_out":3945,"would_cite":false,"duration_ms":35005,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stability metrics report pure noise as perfectly stable.","keywords":["explainable AI","XAI robustness","stability measures","MAX-Sensitivity","AVG-Sensitivity","meta-evaluation","Random Output Test","Perfect Explanation Test"],"falsifier":"Re-run the Random Output Test on the same Gaussian-noise explanations with an implementation that samples perturbations strictly by $\\|x_j - x_i\\| < \\epsilon$, records $\\epsilon$ and the Monte Carlo count, and prints the individual norms $\\|f(x_j)-f(x_i)\\|$. If the resulting MAX- and AVG-Sensitivity values are near 1 rather than 0.011 and 0.010, the reported near-perfect scores are an artifact of the sampling condition, not a property of the metrics.","tokens_in":9004,"feed_emoji":"🎲","tokens_out":10834,"duration_ms":86603,"temperature":0.7,"pith_summary":"This paper introduces two benchmark tests, the Perfect Explanation Test and the Random Output Test, to meta-evaluate XAI stability measures, and applies them to MAX-Sensitivity and AVG-Sensitivity. The tests give each metric a known target value: exact explanations from a decision tree should be scored perfectly stable, while Gaussian-noise explanations with uniform-noise predictions should be scored maximally unstable. Both metrics score 0.000 on the perfect case, but on the random case they return 0.011 and 0.010 instead of the expected 1.0, which the paper takes as evidence that they cannot detect an obvious lack of stability. If the argument holds, low values of these metrics should not be used as evidence that an explanation is stable.","feed_headline":"Stability metrics report pure noise as perfectly stable","feed_subtitle":"On random explanations, the two sensitivity measures score near zero instead of one, so they cannot certify stability.","key_machinery":"The central objects are the two sensitivity formulas: MAX-Sensitivity, the maximum over Monte-Carlo perturbations with $\\|x_j-x_i\\|<\\epsilon$ of $\\|f(x_j)-f(x_i)\\|$, and AVG-Sensitivity, the average over the same perturbation set, where $f(x)$ is the explanation for input $x$. The meta-evaluation machinery is the oracle-test design: in the Perfect Explanation Test the true explanation is known exactly, so any correct stability metric must return its perfect value, and in the Random Output Test the explanation and prediction are pure noise, so any correct metric must report maximal instability. Each metric is judged by whether its value and confidence interval match the known expected value.","core_discovery":"The paper claims that MAX-Sensitivity and AVG-Sensitivity pass the Perfect Explanation Test but fail the Random Output Test. With random Gaussian explanations and uniform-noise predictions, where a correct instability measure should return 1.0, both metrics returned near-zero values (0.011 and 0.010), so the paper says they depict the results as perfect. The paper attributes this failure to the way the metrics define their locality, namely that only perturbations whose prediction difference is below a threshold $\\epsilon$ enter the computation, which masks the randomness. On the paper's argument, neither metric can be used to certify explanation stability.","pith_inferences":["The failure mode likely extends beyond these two formulas: any stability measure whose perturbation set is restricted by a small output or prediction change will tend to under-report instability when explanations are noisy, because the noisy samples that survive the filter are the ones that look similar.","A direct testable extension is to run the same two tests on other stability measures; the benchmark is metric-agnostic and does not depend on the specific choice of decision tree or synthetic dataset.","In practice, a low MAX- or AVG-Sensitivity value on a real model should be accompanied by a randomized-explanations control; without such a control, a near-zero score cannot be read as evidence of explanation stability.","The paper's own explanation of the near-zero Random Output Test scores says the perturbation set is restricted by prediction difference, but Equation 1 conditions on input distance; if the implementation follows Equation 1, the near-zero result needs a different explanation, and if it follows the prose, the result does not apply to the published metric."],"forward_implications":["Any stability metric that fails the Random Output Test cannot be trusted to certify explanation stability in real applications, because it cannot detect the most basic failure mode.","The two tests provide an objective benchmark: a metric that cannot handle the simple perfect and random scenarios is not reliable in complex scenarios either.","Because the failure is attributed to the $\\epsilon$-neighborhood sampling, other stability measures built on the same local perturbation idea should be screened with the same benchmark before use.","The expected-value-plus-confidence-interval procedure gives future stability metrics a pass/fail rule that does not require new axioms about what robustness should mean."],"supporting_citations":[{"why":"Defines the two stability measures under test, MAX-Sensitivity and AVG-Sensitivity, and their Monte-Carlo sampling procedure.","marker":"[34]"},{"why":"The previous axiomatic meta-evaluation of XAI measures that this work contrasts with, which analyzed MAX-Sensitivity without a clear conclusion.","marker":"[15]"},{"why":"Introduced randomization-based sanity checks for explanations, which inspire the Random Output Test.","marker":"[2]"},{"why":"Supplies the TXUXIv3 dataset with known ground-truth explanations used to train the decision tree in the Perfect Explanation Test.","marker":"[23]"},{"why":"Defines the decision tree model that provides exact, perfect explanations in the Perfect Explanation Test.","marker":"[8]"},{"why":"Defines the ssin attribution function used as the ground-truth explanation for the regression task.","marker":"[12]"}],"fun_headline_variants":["Sensitivity measures mark pure noise as stable","MAX and AVG sensitivity fail random-output test","Stability metrics pass perfect, fail random","Near-zero sensitivity reported for pure noise","XAI stability measures cannot spot randomness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the authors' implementation of the two sensitivity formulas matching their definitions, namely sampling perturbations within an $\\epsilon$-ball and taking the max or average of explanation differences, so the near-zero Random Output Test scores are a property of the metrics and not a coding or sampling artifact.","fun_headline_variants_meta":{"raw":{"variants":["Sensitivity measures mark pure noise as stable","MAX and AVG sensitivity fail random-output test","Stability metrics pass perfect, fail random","Near-zero sensitivity reported for pure noise","XAI stability measures cannot spot randomness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3270,"prompt_tokens":802,"completion_tokens":2468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":418,"completion_tokens_details":{"reasoning_tokens":2403}},"tokens_in":418,"tokens_out":2468,"duration_ms":17632,"temperature":1.0,"reasoning_tokens":2403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:27:09.603227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Random Output Test on the same Gaussian-noise explanations with an implementation that samples perturbations strictly by $\\|x_j - x_i\\| < \\epsilon$, records $\\epsilon$ and the Monte Carlo count, and prints the individual norms $\\|f(x_j)-f(x_i)\\|$. If the resulting MAX- and AVG-Sensitivity values are near 1 rather than 0.011 and 0.010, the reported near-perfect scores are an artifact of the sampling condition, not a property of the metrics.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems 32 (2019)","cited_arxiv_id":null,"evidence_quote":"Defines the two stability measures under test, MAX-Sensitivity and AVG-Sensitivity, and their Monte-Carlo sampling procedure."},{"cited_title":"Transactions on Machine Learning Research (2023)","cited_arxiv_id":null,"evidence_quote":"The previous axiomatic meta-evaluation of XAI measures that this work contrasts with, which analyzed MAX-Sensitivity without a clear conclusion."},{"cited_title":"Advances in neural information processing systems 31 (2018)","cited_arxiv_id":null,"evidence_quote":"Introduced randomization-based sanity checks for explanations, which inspire the Random Output Test."},{"cited_title":"Assessing Fidelity in XAI post-hoc techniques: A Comparative Study with Ground Truth Explanations Datasets","cited_arxiv_id":"2311.01961","evidence_quote":"Supplies the TXUXIv3 dataset with known ground-truth explanations used to train the decision tree in the Perfect Explanation Test."},{"cited_title":"Routledge (1984)","cited_arxiv_id":null,"evidence_quote":"Defines the decision tree model that provides exact, perfect explanations in the Perfect Explanation Test."},{"cited_title":"Information Sciences 225, 1–17 (2013)","cited_arxiv_id":null,"evidence_quote":"Defines the ssin attribution function used as the ground-truth explanation for the regression task."}],"review_version":1}