Re-Visiting Explainable AI Evaluation Metrics to Identify The Most Informative Features

Ahmed M. Salih

REVIEW 4 major objections 6 minor 12 references

Re-Visiting Explainable AI Evaluation Metrics to Identify The Most Informative Features

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes Expected Accuracy Interval (EAI), a metric that predicts the upper and lower bounds of model accuracy after removing or permuting the most informative feature.

desk verdict The ROAR/PI caution is real, but EAI is a rescaling of SHAP and the paper's own tables contradict it. read the letter →

arxiv 2502.00088 v1 pith:KZDG46EV submitted 2025-01-31 cs.LG stat.ML

Ahmed M. Salih This is my paper · ORCID

classification cs.LGstat.ML

keywords explainableAIROARpermutationimportanceSHAPexpectedaccuracyintervalmulticollinearityfeatureevaluationproxy-based

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that two standard proxy measures for evaluating explainable-AI methods, ROAR (remove and retrain) and permutation importance, do not reliably reveal the most informative feature when features are collinear, when several features are informative, or when baseline accuracy is low. It then proposes a new metric, the Expected Accuracy Interval (EAI), which converts the top feature's SHAP contribution share and the current model accuracy into a predicted interval for the accuracy after that feature is removed or permuted. The paper's empirical demonstrations on two real datasets and one simulated dataset show cases where accuracy barely drops or even rises after removing the top feature, and it reports that EAI is especially useful in those collinear settings. If the metric works as claimed, practitioners gain a concrete, score-based expectation of how much removing a feature should hurt, rather than relying on the qualitative expectation of a sharp decline.

What carries the argument

The load-bearing object is the feature contribution percentage $FCP = S_{MSF}/S_{SOAF}$ (Equation 1), combined with the proportionality assumption $\mathrm{Expected}\,\Delta = \mathrm{initial\_acc} \times FCP$ (Equation 2). These feed the interval $EAI = [\mathrm{initial\_acc} - \mathrm{Expected}\,\Delta,\; \mathrm{initial\_acc} + \mathrm{Expected}\,\Delta]$ (Equations 3-5). $FCP$ is meant to translate a model-agnostic explanation score, specifically SHAP's per-feature attribution, into a forecast of how much removing that feature changes accuracy, and the interval width is used as a signal of whether the remaining features have similar impact.

What would settle it

A direct falsifier is to compute, on datasets with controlled degrees of collinearity, the fraction of removed-feature iterations in which the observed retrained accuracy falls outside the EAI interval; if that fraction is high across many seeds, the interval is not predictive. Even simpler, fit a regression of the observed accuracy drop on $\mathrm{initial\_acc} \times FCP$; if the slope is not near 1, Equation 2 is not the right mechanism.

Watch

Extended reading notes

Core claim

The central claim is that EAI predicts the post-removal accuracy interval as $[\mathrm{initial\_acc} - \mathrm{initial\_acc} \times FCP, \; \mathrm{initial\_acc} + \mathrm{initial\_acc} \times FCP]$, where $FCP$ is the SHAP score of the most significant feature divided by the sum of all SHAP scores. The paper demonstrates with tabular regression and classification experiments that ROAR and permutation importance often fail to produce a sharp performance drop when the top feature is collinear with other strong features, and that accuracy can even increase after removing the top feature. It therefore proposes EAI as a supplement that gives a numerical interval rather than a direction-only expectation, with wider intervals indicating that the next most significant feature has a larger impact relative to the removed one.

Load-bearing premise

The prediction stands on the assumption in Equation 2 that a feature's SHAP share of total scores translates linearly into the same share of model accuracy; if that linear scaling is wrong, EAI is only a rescaling of SHAP, not an accuracy forecast.

Editorial extensions

If this is right

If EAI holds, a practitioner can remove the top SHAP feature and expect the retrained model's accuracy to fall inside the stated interval, making ROAR and permutation-importance evaluations quantitative rather than direction-only.
For collinear models, EAI provides a reason not to treat a small accuracy drop as a failure of the XAI method, since the interval encodes the replacement effect of collinear features.
A wider interval after removing the top feature indicates the newly most significant feature carries greater impact, which can guide feature-selection and risk-factor interpretation.
EAI requires only an existing score-based explanation, so it can be added to any pipeline that already computes SHAP-like per-feature contributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

I expect the same interval construction to transfer to any per-feature score, such as LIME-style weights or integrated gradients, because Equations 1-2 only consume a score vector; this is an extension the paper does not test.
If the linear scaling in Equation 2 is calibrated or replaced by a learned mapping, EAI could become a tighter predictor on low-accuracy models, where the paper's own tables show some iterations falling outside the interval.
The width of EAI could be reinterpreted as a collinearity diagnostic: intervals that stay wide after removing top features signal redundancy in the feature set, a use that could be tested on synthetic data with controlled correlation.
A natural testable extension is to see whether EAI's hit rate remains high when the underlying model is a nonlinear tree ensemble instead of logistic or linear regression, since SHAP values behave differently there.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper's empirical reminder—that removing the top SHAP feature under collinearity does not reliably drop accuracy and can even raise it—is worth keeping. But the proposed EAI metric is a deterministic rescaling of SHAP and is contradicted by the paper's own tables.

What is genuinely here: the paper runs ROAR-style retraining on three datasets (diabetes, wine, simulated) and shows accuracy often stays flat or rises after removing the top feature. That is a real phenomenon, and the tables make it visible. For practitioners who use ROAR/PI as their only XAI check, this is a useful caution. The author also correctly notes that ROAR/PI give no numerical expectation for the change in performance. Those observations are fine as far as they go. The citations are standard; the two self-citations are on topic.

The soft spots are large. Eq. (2) asserts Expected Delta = initial_acc × FCP without derivation or model of retraining. Since FCP is just the normalized SHAP score of the top feature, EAI is an algebraic rearrangement of the current accuracy and that SHAP share; it adds no information beyond what went in. It has no coverage guarantee for the accuracy of a retrained model. FCP need not lie in [0,1] unless extra constraints are stated, and none are. The paper's own motivation—collinearity makes feature contributions nonlinear—argues against the linear scaling in Eq. (2). And the evidence does not support the claim: the table captions mark rows outside the interval, and the observed accuracy increases after removal (e.g., Table 1, iteration 16 to 17: 0.5164 to 0.6716) cannot be represented by Eq. (2) with a positive FCP.

The evaluation protocol is also unsound: the model is trained and tested on the whole dataset in each iteration, so the reported accuracies are optimistic and not reproducible. That would matter even for the descriptive part of the paper. The two real datasets are publicly available and the code appears straightforward, so the empirical demonstration could be redone properly, but as it stands the numbers are not trustworthy.

Who gets value from this: a reader who wants a concrete example of ROAR/PI misbehaving under collinearity might find the tables illustrative. Nobody should use EAI as a prediction interval. The paper does not deserve a serious referee; the central claim is a rescaling that the paper's own data refute, and the evaluation protocol would need to be redone from scratch. My recommendation is to desk reject.

Referee Report

4 major / 6 minor

Summary. The paper argues that ROAR (RemOve And Retrain) and permutation importance, two common proxy-based XAI evaluation metrics, can mislead when features are collinear or when there are multiple informative features, and it supports this with empirical examples on one diabetes classification dataset, one wine-quality regression dataset, and one simulated classification dataset. To address the limitation, the paper proposes an Expected Accuracy Interval (EAI) computed from the current model accuracy and the SHAP contribution share of the most significant feature: FCP = SMSF/SSOAF, Expected Delta = initial_acc * FCP, and EAI = [initial_acc - Expected Delta, initial_acc + Expected Delta]. The paper reports, in Tables 1-3, the observed accuracy after sequentially removing the most significant feature and compares it with the proposed interval.

Significance. If the EAI were valid, it would give practitioners a simple way to anticipate the effect of removing the most informative feature, especially in settings with collinear features, and the paper does usefully illustrate that ROAR and permutation importance do not always produce sharp accuracy drops. However, the central contribution is not established: Equation (2) is asserted without derivation or justification, the evaluation protocol trains and tests on the full dataset, and multiple observed accuracies in the paper's own tables fall outside the claimed intervals. The proposed metric is therefore currently a rescaling of the SHAP contribution share rather than a validated prediction, and the empirical evidence contradicts the abstract's central claim.

major comments (4)

[Section 3, Eq. (2)] Equation (2), Expected Delta = initial_acc * FCP, is asserted without any derivation, theoretical justification, or empirical calibration. It assumes that the change in accuracy after removing a feature is exactly the product of the current accuracy and the SHAP contribution share of that feature. This linear and symmetric assumption is not implied by SHAP's additivity, which concerns model outputs rather than accuracy, and it is not supported by the paper's own discussion of collinear features, where the paper argues that removing a collinear feature should produce a smaller drop than its individual contribution. In addition, FCP need not lie in [0,1] when SHAP scores can be negative, yet the interval construction in Eqs. (3)-(5) implicitly assumes a positive bounded FCP. Since the EAI is defined entirely through Eq. (2), the central predictive claim is unsupported.
[Section 4.3, Implementation] The implementation section states that "The whole data was used in the training and test." This means there is no held-out validation set, no cross-validation, and no repeated sampling protocol. The reported accuracies in Tables 1-3 are therefore optimistically biased and not reproducible as estimates of generalization performance. This makes the comparison between observed accuracies and the EAI interval not a valid test of the proposed prediction.
[Tables 1 and 2] The captions of Tables 1 and 2 explicitly state that rows highlighted in red are outside the expected accuracy interval, and the observed values confirm this. For example, in Table 1, after iteration 16 the accuracy increases from 0.5164 to 0.6716, which is above the upper bound 0.6397 computed from the previous iteration, and after iteration 19 the observed accuracy 0.1513 is below the lower bound 0.2952. In Table 2, iterations 2, 5, 6, 7, and 8 show observed accuracies below the lower bound of the claimed interval. These violations directly contradict the abstract's claim that EAI predicts the upper and lower bounds of the accuracy when ROAR or permutation importance is implemented.
[Section 2 vs. Section 3] The paper's motivation is internally inconsistent with its proposed formula. Section 2 argues that when features are collinear, removing the most significant feature may not cause a sharp decline because other correlated features remain; yet Eq. (2) sets the expected accuracy drop exactly equal to the current accuracy times the SHAP contribution share, which assumes the drop is proportional to that feature's independent contribution. The empirical observation that accuracy can increase after removal (e.g., Table 1, iteration 16) cannot be represented by Eq. (2) at all when FCP is positive. The proposed interval therefore does not follow from the stated limitations of ROAR and permutation importance.

minor comments (6)

[Abstract] The abstract contains the typos "the the" and "IP" where "PI" (permutation importance) is intended.
[Section 3, Eq. (5)] Equation (5) reads EAI = [LI - UI], but the intended expression is EAI = [LI, UI] or [LI, UI] with a comma; as written, the formula is not the interval described in the text.
[Section 4.1] The text says "More details about the data can be found here" and "More details about the dataset can be found here," but no working links are provided; this should be replaced with proper references or hyperlinks.
[Section 5.1] The sentence "The figure shows the most significant feature based on the SHAP score in the first iteration is GenHlth" refers to Table 1, not a figure; the cross-reference should be corrected.
[Section 6] There is a typo in the limitations paragraph: "I addition" should be "In addition." Also, "RemOve And Retrain" should be "Remove and Retrain," and Figure 1's label "retain" should likely be "retrain."
[Section 8] The data availability statement says the simulated dataset can be downloaded from the supplementary, but no supplementary material or link is included in the manuscript; this should be fixed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; EAI is an explicit and falsifiable function of current accuracy and SHAP contribution share, not a fit to the retrained accuracies.

full rationale

The derivation chain is self-contained: Eq. (1) defines FCP from SHAP scores, Eq. (2) sets Expected Delta = initial_acc * FCP, and Eqs. (3)-(5) build the interval as initial_acc +/- Expected Delta. No parameter is fitted to the retrained model's accuracy, and the paper's Tables 1-3 mark observed accuracies that fall outside the interval in red, so the interval is not forced to match its target by construction. The load-bearing weakness is Eq. (2), which asserts an unproven linear proportionality between accuracy change and SHAP contribution share; that is a validity/correctness concern, not circularity, because the inputs (initial accuracy and SHAP scores) are not defined in terms of the predicted quantity (retrained accuracy), and no fitted value is relabeled as a prediction. The self-citation [6] attached to 'This paper presents a new measure ... [6]' is attributional and does not supply the validity of Eqs. (1)-(5), so it is not load-bearing. Therefore no claimed prediction reduces to its own input by definition.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The EAI formula introduces no fitted constants, but it rests on two hand-chosen arithmetic choices (unit slope, symmetric band) and on the assumption that SHAP contribution shares translate linearly into accuracy changes. No independent evidence is provided for this relationship.

free parameters (2)

Unit slope in Eq (2) = 1
The expected change is set to initial_acc * FCP with no derived constant; the value 1 is chosen by hand.
Symmetric interval coefficient = +/- 1
The same delta is added and subtracted (Eqs 3-4) without justification that upper and lower deviations are equal.

assumptions (4)

domain assumption Normalized SHAP scores sum to 1 and measure feature contribution to the prediction.
Eq (1) defines FCP as SMSF/SSOAF, assuming SHAP scores are additive and comparable.
ad hoc to paper The change in model accuracy is linear in the SHAP contribution share.
Eq (2) uses Expected Delta = initial_acc * FCP, the load-bearing linearity assumption.
ad hoc to paper Training and testing on the full dataset is a valid evaluation protocol.
Section 4.3: 'The whole data was used in the training and test.' No held-out split, so reported accuracies are optimistic.
domain assumption Default model parameters are adequate for the comparison.
Section 4.3: 'Default parameters of both models were considered.' No hyperparameter tuning or sensitivity analysis.

invented entities (1)

Expected Accuracy Interval (EAI)
purpose: Predict upper and lower bounds for model accuracy after removing or permuting the most important feature.
The interval is a function of the model's own initial accuracy and SHAP scores; no external benchmark or prior falsifiable prediction supports it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Re-Visiting Explainable AI Evaluation Metrics to Identify The Most Informative Features." pith.science (2026). https://pith.science/paper/KZDG46EV

@misc{pith2026250200088,
  author       = {Pith},
  title        = {Pith review of: Re-Visiting Explainable AI Evaluation Metrics to Identify The Most Informative Features},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZDG46EV}},
  note         = {Machine review of arXiv:2502.00088}
}

read the original abstract

Functionality or proxy-based approach is one of the used approaches to evaluate the quality of explainable artificial intelligence methods. It uses statistical methods, definitions and new developed metrics for the evaluation without human intervention. Among them, Selectivity or RemOve And Retrain (ROAR), and Permutation Importance (PI) are the most commonly used metrics to evaluate the quality of explainable artificial intelligence methods to highlight the most significant features in machine learning models. They state that the model performance should experience a sharp reduction if the most informative feature is removed from the model or permuted. However, the efficiency of both metrics is significantly affected by multicollinearity, number of significant features in the model and the accuracy of the model. This paper shows with empirical examples that both metrics suffer from the aforementioned limitations. Accordingly, we propose expected accuracy interval (EAI), a metric to predict the upper and lower bounds of the the accuracy of the model when ROAR or IP is implemented. The proposed metric found to be very useful especially with collinear features.

Figures

Figures reproduced from arXiv: 2502.00088 by the authors.

**Figure 2.** Correlation matrix between the features in the Diabetes dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Correlation matrix between the features in the Wine quality dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Correlation matrix between the features in the simulated dataset to perform binary classification. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages

[1]

Notions of explainability and evaluation approaches for explainable artifi- cial intelligence

Giulia Vilone and Luca Longo. “Notions of explainability and evaluation approaches for explainable artifi- cial intelligence”. In: Information Fusion 76 (2021), pp. 89–106

work page 2021
[2]

A review of evaluation approaches for explainable AI with applications in cardiol- ogy

Ahmed M Salih et al. “A review of evaluation approaches for explainable AI with applications in cardiol- ogy”. In: Artificial Intelligence Review 57.9 (2024), p. 240

work page 2024
[3]

A benchmark for interpretability methods in deep neural networks

Sara Hooker et al. “A benchmark for interpretability methods in deep neural networks”. In: Advances in neural information processing systems 32 (2019)

work page 2019
[4]

Multilayer Perceptron of Software Complexity Metrics for Explainable Multi- collinearity Mitigation and Defect Localization

Taiwo O Olaleye et al. “Multilayer Perceptron of Software Complexity Metrics for Explainable Multi- collinearity Mitigation and Defect Localization”. In:Cureus Journal of Computer Science17 (2025), pp. 1– 17

work page 2025
[5]

A unified approach to interpreting model predictions

Scott Lundberg. “A unified approach to interpreting model predictions”. In: arXiv preprint arXiv:1705.07874 (2017)

arXiv 2017
[6]

Characterizing the Contribution of Dependent Features in XAI Methods

Ahmed M Salih et al. “Characterizing the Contribution of Dependent Features in XAI Methods”. In: IEEE Journal of Biomedical and Health Informatics (2024)

work page 2024
[7]

Permutation importance: a corrected feature importance measure

André Altmann et al. “Permutation importance: a corrected feature importance measure”. In: Bioinformatics 26.10 (2010), pp. 1340–1347

work page 2010
[8]

CDC National Health Report: leading causes of morbidity and mortality and associated behavioral risk and protective factors–United States, 2005-2013

Nicole Blair Johnson et al. “CDC National Health Report: leading causes of morbidity and mortality and associated behavioral risk and protective factors–United States, 2005-2013”. In:MMWR suppl 63.4 (2014), pp. 3–27

work page 2014

Show all 12 references

[9]

Modeling wine preferences by data mining from physicochemical properties

Paulo Cortez et al. “Modeling wine preferences by data mining from physicochemical properties”. In: Decision support systems 47.4 (2009), pp. 547–553

2009
[10]

Efficient saliency maps for explainable AI

T Nathan Mundhenk, Barry Y Chen, and Gerald Friedland. “Efficient saliency maps for explainable AI”. In: arXiv preprint arXiv:1911.11293 (2019)

2019 arXiv
[11]

Explainable artificial intelligence approaches: Challenges and perspectives

Natalya V Shevskaya. “Explainable artificial intelligence approaches: Challenges and perspectives”. In: 2021 International Conference on Quality Management, Transport and Information Security, Information Technologies (IT&QM&IS). IEEE. 2021, pp. 540–543. 10

2021
[12]

Cardiovascular risk factors and physical activity for the prevention of cardiovas- cular diseases in the elderly

Lorena Cium ˘arnean et al. “Cardiovascular risk factors and physical activity for the prevention of cardiovas- cular diseases in the elderly”. In: International Journal of Environmental Research and Public Health19.1 (2021), p. 207. 11

2021

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Notions of explainability and evaluation approaches for explainable artifi- cial intelligence

Giulia Vilone and Luca Longo. “Notions of explainability and evaluation approaches for explainable artifi- cial intelligence”. In: Information Fusion 76 (2021), pp. 89–106

work page 2021

[2] [2]

A review of evaluation approaches for explainable AI with applications in cardiol- ogy

Ahmed M Salih et al. “A review of evaluation approaches for explainable AI with applications in cardiol- ogy”. In: Artificial Intelligence Review 57.9 (2024), p. 240

work page 2024

[3] [3]

A benchmark for interpretability methods in deep neural networks

Sara Hooker et al. “A benchmark for interpretability methods in deep neural networks”. In: Advances in neural information processing systems 32 (2019)

work page 2019

[4] [4]

Multilayer Perceptron of Software Complexity Metrics for Explainable Multi- collinearity Mitigation and Defect Localization

Taiwo O Olaleye et al. “Multilayer Perceptron of Software Complexity Metrics for Explainable Multi- collinearity Mitigation and Defect Localization”. In:Cureus Journal of Computer Science17 (2025), pp. 1– 17

work page 2025

[5] [5]

A unified approach to interpreting model predictions

Scott Lundberg. “A unified approach to interpreting model predictions”. In: arXiv preprint arXiv:1705.07874 (2017)

arXiv 2017

[6] [6]

Characterizing the Contribution of Dependent Features in XAI Methods

Ahmed M Salih et al. “Characterizing the Contribution of Dependent Features in XAI Methods”. In: IEEE Journal of Biomedical and Health Informatics (2024)

work page 2024

[7] [7]

Permutation importance: a corrected feature importance measure

André Altmann et al. “Permutation importance: a corrected feature importance measure”. In: Bioinformatics 26.10 (2010), pp. 1340–1347

work page 2010

[8] [8]

CDC National Health Report: leading causes of morbidity and mortality and associated behavioral risk and protective factors–United States, 2005-2013

Nicole Blair Johnson et al. “CDC National Health Report: leading causes of morbidity and mortality and associated behavioral risk and protective factors–United States, 2005-2013”. In:MMWR suppl 63.4 (2014), pp. 3–27

work page 2014