{"id":"4c2843bf-f20a-41e7-8028-9ba66c6a1bd2","arxiv_id":"2607.14315","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Proposes combining fidelity, simplicity, and stability into a single explainability score, but leaves the weights undefined and reports SHAP fidelity that is zero by construction.","lead":"This paper proposes a single explainability score for AI models by combining fidelity, simplicity, and stability, and reports illustrative LIME/SHAP results on three small datasets. The score is never actually computed because the combining weights are deferred, and the headline SHAP result is a definitional artifact rather than a measured finding.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SHAP's near-zero fidelity in Table II is not an empirical result: Eq. (1) with SHAP's local-accuracy property makes g(x_i)=f(x_i) by construction, so the headline comparison is a mathematical identity.","rationale":"The reader's weakest_assumption correctly identifies Eq. (1) combined with SHAP's local accuracy as the fatal flaw. This is indeed the most load-bearing concern because it invalidates the paper's only concrete empirical result (Table II and the derived conclusion that SHAP outperforms LIME in fidelity). The paper's broader promise of a unified score is also unsupported—weights are deferred to future work and the knowledge base is never constructed—but these are gaps in delivery rather than a mathematical inconsistency. The fidelity identity is a direct logical error that undermines the demonstration. I agree with the REJECT verdict because the central claim hinges on an empirical comparison that is not real, leaving the proposed metric without validation. No change to the reader's verdict is needed.","tokens_in":9149,"tokens_out":2158,"duration_ms":24068,"concrete_test":"Reproduce Table II for the Iris dataset: train a RandomForestClassifier, compute SHAP values, form g(x_i) = expected_value + sum(shap_values[i]), and evaluate Eq. (1). Record max_i |f(x_i) - g(x_i)| and the fidelity value. If max_i |f(x_i) - g(x_i)| < 1e-12, the fidelity column is an identity artifact. Then re-run the comparison using a fidelity metric that does not collapse under local accuracy—e.g., evaluate the linear surrogate g on a held-out set of perturbed samples (as in LIME's original formulation) or compute the squared error after removing the base value. If SHAP's fidelity then becomes nonzero and comparable to LIME's, the original conclusion is an artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that SHAP outperforms LIME in fidelity (Section V). This depends on Eq. (1), where fidelity is the normalized L2 distance between the original model's prediction f(x_i) and the explanation model's prediction g(x_i). For SHAP, g(x_i) is conventionally defined as the additive sum of SHAP values plus the expected value. SHAP's defining local-accuracy property (Lundberg & Lee 2017) guarantees that this sum exactly equals f(x_i) for every instance. Substituting into Eq. (1) yields a numerator of zero up to floating-point error, so the ~1e-17 values in Table II are predetermined, not measured. Consequently, the statement 'SHAP generally outperforms LIME in terms of fidelity' is a tautology of SHAP's construction, not a benchmark result. This undercuts the framework's utility as a comparative metric: it cannot distinguish SHAP from a perfect explanation because SHAP is designed to fit the model exactly. While the paper also defers weight assignment and never builds the promised knowledge base, the fidelity identity is the concrete flaw that invalidates the only quantitative demonstration. If this metric cannot meaningfully compare LIME and SHAP—the two methods it showcases—the claim of a 'unified multidimensional explainability metric' has no empirical support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified multidimensional explainability score for XAI methods, combining fidelity, stability, simplicity, coverage, and human-centered metrics via a weighted sum. It reports benchmarking experiments with LIME and SHAP on Iris, Wine, and Breast Cancer datasets, presents component scores in Tables I and II, and outlines an offline knowledge base for estimating explainability of unseen models and datasets. The stated claims are that SHAP has near-perfect fidelity, LIME has higher stability, and the framework supports context-dependent XAI selection.","tokens_in":9600,"tokens_out":4473,"duration_ms":44675,"significance":"A validated unified explainability metric would be a valuable contribution to XAI evaluation, and the paper addresses an important problem. However, the presented support is not sufficient: the main numerical comparison is predetermined by SHAP's local-accuracy property, the stability metric is mis-defined, the unified score is never computed, and the knowledge base is only described. The manuscript has a clear structure and identifies relevant literature, but it does not establish the proposed framework.","major_comments":[{"comment":"The SHAP fidelity values (7.29e-18, 1.27e-17, 1.23e-17) are not measured results: for SHAP, with g(x_i) taken as the additive SHAP model, the local-accuracy property gives g(x_i)=f(x_i) by construction. Substituting into Eq. (1) forces the numerator to zero up to floating-point error. Therefore the Section V conclusion that 'SHAP generally outperforms LIME in terms of fidelity' is a mathematical identity of SHAP's definition, not an empirical finding, and cannot support a comparative explainability metric.","section":"Eq. (1), Tables I–II, Section V"},{"comment":"Eq. (2) defines stability as the normalized L2 distance between g(x_i) and g(x_i'). This is always nonnegative, yet the Section V range bullet states 'Stability: Range: [-1, 1]'. Moreover the equation measures the change in the explanation model's predicted value, not the change in the explanation itself. For a stability metric one would compare the explanation outputs (e.g., attribution vectors) under perturbation. As defined, the reported stability values are difficult to interpret and do not support the trade-off claims about LIME versus SHAP.","section":"Eq. (2), Section V"},{"comment":"The paper's headline deliverable is 'a single score to characterize the overall performance of the XAI method' (Section I-B). However, the weighted-sum score is never computed: the text states the weights 'will not be defined' and 'will be defined via surveys and questionnaires' in future work. Tables I and II report only components. Thus the central claim of a unified multidimensional explainability score is not demonstrated.","section":"Section III-B (Explainability score)"},{"comment":"The offline knowledge base and the estimation of explainability scores for unseen datasets and models are described as goals, but no such knowledge base is constructed and no estimation experiment is performed. The paper contains no metadata analysis, rule learning, or validation against held-out datasets or models. The contribution in its current scope is a proposal, not a working framework.","section":"Abstract and Section IV"},{"comment":"Simplicity is defined by Eq. (3) as an average ratio of number of nodes to maximum nodes, yet Tables I and II report simplicity values 4, 13, 30 (and 28.31 for SHAP on dataset 3), which look like feature counts. The units are inconsistent with the definition, and the non-integer SHAP value is unexplained. Without a consistent component calculation, the comparison of simplicity between methods is not meaningful.","section":"Eq. (3), Tables I–II"}],"minor_comments":[{"comment":"Typo: 'off-the-self' should be 'off-the-shelf'; Section V: 'lets start' should be 'let's start'; Acknowledgments: 'Europeans Union's' should be 'European Union's'.","section":"Section II-B"},{"comment":"The tables use 'Dataset 1/2/3' but do not map to Iris, Wine, and Breast Cancer datasets in the captions or surrounding text.","section":"Tables I and II"},{"comment":"The experimental setup is underspecified: no training/test split, model hyperparameters, random seeds, or perturbation procedure for Eq. (2) is described, which prevents reproduction.","section":"Section IV"},{"comment":"Eq. (1) uses f(x_i) and g(x_i) without specifying whether these are logits, probability vectors, or predicted class labels. The definition affects the meaning of fidelity.","section":"Section III-A1"},{"comment":"Several references are incomplete: [11] lacks venue/year details, [21] lists only '2023' with no venue, and the author name 'Jonh Soldatos' appears misspelled in the author list.","section":"References"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as a vision/roadmap rather than a completed evaluation. The central quantitative result is invalid because the SHAP fidelity comparison is forced by SHAP's local-accuracy property, and the other pillars (stability definition, weight assignment, knowledge base construction) are either misdefined or deferred to future work. A viable revision would require redefining the metrics, running new experiments on explanation outputs rather than model predictions, computing the unified score, and building/validating the knowledge base—substantial new work beyond a normal revision. I therefore recommend rejection in its current form, while acknowledging that the general problem is relevant and timely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a position paper dressed up as an evaluation study. The parts that are real — the general framing, the choice of fidelity/simplicity/stability as dimensions, the sensible caution that weights should be context-dependent — are fine but not new. The parts that would make it a contribution — the unified score, the knowledge base, the actual comparison of LIME and SHAP — are either deferred or, in the case of the comparison, broken.\n\nWhat the paper does well: it states a real problem (no consensus on how to compare XAI methods), it defines three dimensions that are reasonable to care about, and it is honest enough to admit that the aggregation weights are not defined and that the knowledge base is a plan, not a deliverable. The writing is straightforward and the related work is adequate. That is the extent of the credit.\n\nThe problems are load-bearing. First, the headline result — SHAP's near-zero fidelity in Table II — is not a measurement. With the additive SHAP model, local accuracy guarantees g(x_i) = f(x_i), so Eq. (1) returns machine epsilon by construction. The stress-test note is correct: this is an identity, not a benchmark. The paper's own statement that 'SHAP generally outperforms LIME in terms of fidelity' is a tautology. Second, the unified explainability score is never computed: the weights are deferred to future surveys, and the stability range in the text ([-1, 1]) is inconsistent with Eq. (2), which is nonnegative. Third, the simplicity scores in the tables look like feature counts, not the normalized formula in Eq. (3), and there are no error bars, no code, no baselines. The 'knowledge base' is a future-tense idea.\n\nI agree with the reader's rejection. The central claim — a unified multidimensional explainability score that lets practitioners compare methods — is unsupported by the presented work. That said, the paper is not incoherent; it is a proposal for a research program. The components of the metric are standard, and the experiments are routine, so there is no new result here to referee.\n\nWho is this for? Someone surveying the XAI evaluation landscape who wants a quick catalog of proposed dimensions and a reminder that weights should be context-dependent. Not for anyone looking for a validated metric or an empirical comparison.\n\nRecommendation: desk reject. It is not ready for serious peer review as a contribution; the central empirical claim is definitional and the central artifact is absent. If the authors return with actual weights, a constructed knowledge base, and a fidelity comparison that is not guaranteed by construction, the idea could become worth a look.","headline":"A clear-eyed outline for an explainability score, but the one quantitative result it reports is an artifact of SHAP's local accuracy, and the promised unified score is never actually computed.","tokens_in":10022,"tokens_out":664,"would_cite":false,"duration_ms":9727,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that explainability can be distilled into a single multidimensional score, and that an offline knowledge base can predict that score for new datasets and models.","keywords":["explainability","XAI evaluation","fidelity","stability","simplicity","SHAP","LIME","knowledge base"],"falsifier":"A single decisive check: implement an intentionally vacuous explainer that outputs the original model's prediction for every sample; the fidelity formula will give it the same near-zero score as SHAP. If the framework's fidelity component cannot distinguish a vacuous explainer from SHAP, then the fidelity measure—and the headline comparison built on it—does not measure explanation quality.","tokens_in":9069,"feed_emoji":"📊","tokens_out":4913,"duration_ms":51269,"temperature":0.7,"pith_summary":"The paper tries to establish that the trustworthiness of an AI explanation can be compressed into a single multidimensional score built from fidelity, simplicity, and stability, with coverage and user judgments as further components. It further argues that benchmarked scores, stored with metadata about models, datasets, and explainers, form an offline knowledge base that can estimate explainability for previously unseen configurations. The authors demonstrate the framework by scoring LIME and SHAP on three tabular datasets, reporting that SHAP wins on fidelity while LIME wins on stability. A reader should care because a reliable single score would let practitioners and regulators compare explainers objectively, without rerunning experiments or relying on subjective opinion.","feed_headline":"Proposed unified score grades AI explainers on one number","feed_subtitle":"If it works, teams could compare LIME, SHAP, and future methods on a single score without rerunning experiments.","key_machinery":"The load-bearing object is the explainability score formula, a weighted sum of fidelity, stability, simplicity, coverage, and user-perceived components, where the weights are to be set later by surveys. Fidelity is the normalized squared difference between the model's prediction and the explanation model's prediction; stability is the normalized change in explanation output under input perturbation; simplicity counts features or decision-tree nodes. The second load-bearing component is the offline knowledge base, which stores these scores alongside model and dataset metadata and is meant to estimate scores for unseen inputs. The paper also leaves the score in multidimensional form when aggre","core_discovery":"On its own terms, the paper claims that an explainability score can be computed as a weighted sum of measurable components—fidelity, stability, simplicity, coverage, user satisfaction, user trust, and user task performance—yielding one number that characterizes the overall performance of an XAI method. The second claim is that this score is context-dependent, so the paper aggregates benchmarking results and metadata into an offline knowledge base whose purpose is to predict explainability scores for new datasets and models. The experiments compare LIME and SHAP, finding SHAP fidelity near 1e-17 (nearly perfect reproduction of the model) and LIME stability at 0.56–1.0, which the paper reads a","pith_inferences":["Inference: the near-zero SHAP fidelity scores follow from SHAP's defining property of reproducing the model prediction locally, so the fidelity formula makes SHAP look perfect by construction; the same score would reward any explainer that simply copies the model's output.","Inference: a testable extension is to evaluate whether metadata-only prediction transfers—e.g., train the knowledge base on the three used datasets, then predict scores for a held-out tabular dataset and compare against true benchmarks; the paper's evidence for transfer is limited to three small datasets.","Inference: adding a 'random but prediction-reproducing' baseline to the benchmark would clarify whether fidelity measures explanation quality or just mathematical mimicry.","Inference: the multidimensional (non-aggregated) form is arguably the more defensible contribution, since aggregating with unset weights cannot yet be validated; future user surveys are what the credibility of the single score hinges on."],"forward_implications":["If the score is valid, organizations can choose an XAI method by a single number, making comparison between LIME and SHAP (or future methods) routine.","Explainability evaluation becomes less dependent on human raters for the quantitative components, though user-perceived weights remain.","The knowledge base would let teams estimate explainability for a new dataset without recomputing expensive experiments, lowering the cost of XAI validation.","Trade-offs become explicit and searchable: this paper's data shows SHAP ahead on fidelity, LIME ahead on stability, so practitioners could filter by context.","Regulated use cases gain a numeric artifact—an explainability grade—that can be reported alongside model accuracy."],"fun_headline_variants":["Unified explainability score proposed for AI models","One number to grade AI explainers: proposed metric","Context-aware score unifies XAI evaluations","Single metric aims to compare LIME, SHAP, and beyond","New proposal: single score for model explainability"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The quantitative conclusion that SHAP beats LIME on fidelity rests on treating the near-zero fidelity scores from the formula as meaningful evidence, even though SHAP reproduces the model's prediction by construction, making those scores a mathematical identity rather than a measured result.","fun_headline_variants_meta":{"raw":{"variants":["Unified explainability score proposed for AI models","One number to grade AI explainers: proposed metric","Context-aware score unifies XAI evaluations","Single metric aims to compare LIME, SHAP, and beyond","New proposal: single score for model explainability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000394,"raw_usage":{"total_tokens":1909,"prompt_tokens":754,"completion_tokens":1155,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":1080}},"tokens_in":498,"tokens_out":1155,"duration_ms":11271,"temperature":1.0,"reasoning_tokens":1080,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T02:26:53.204745+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A single decisive check: implement an intentionally vacuous explainer that outputs the original model's prediction for every sample; the fidelity formula will give it the same near-zero score as SHAP. If the framework's fidelity component cannot distinguish a vacuous explainer from SHAP, then the fidelity measure—and the headline comparison built on it—does not measure explanation quality.","supporting_citations":[],"review_version":1}