REVIEW 4 major objections 5 minor 28 references
Evaluating Explainability: A Framework for Systematic Assessment and Reporting of Explainable AI Features
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper proposes that AI explanation quality can be systematically judged by four criteria—consistency, plausibility, fidelity, and usefulness—and that the first three can screen methods before costly human studies.
desk verdict The framework is worth engaging with, but the fidelity example contradicts the paper's own definition, and the screening claim is unsupported until that's fixed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is an evaluation pipeline with four criteria and a companion scorecard. Consistency is measured by comparing heatmaps under perturbations such as radiation dose changes and rotation, using metrics like SSIM, MSE, and IoU. Plausibility is measured by the overlap between the heatmap's highlighted region and the known ground-truth lesion location. Fidelity is measured by model parameter randomization and deletion tests that compare how the explanation and the model's accuracy change under perturbation. Usefulness is reserved for human task-performance studies. The scorecard combines a descriptive section—method overview, context of use, limitations, and validation setting—with a quantitative section reporting the four criteria, giving stakeholders a standardized report.
What would settle it
Run a blinded reader study where radiologists interpret mammograms with and without heatmaps from a method that has passed high thresholds on consistency, plausibility, and fidelity. If the heatmaps do not improve diagnostic accuracy or time-to-decision—or make them worse—the three quantitative criteria have admitted a useless explanation, showing they do not guarantee usefulness.
Extended reading notes
Core claim
The central claim is that an explainability method can be systematically evaluated through four independent criteria: consistency, which measures how stable the explanation is under small input perturbations; plausibility, which measures overlap with human ground truth; fidelity, which measures how well the explanation tracks the model's actual decision mechanism; and usefulness, which measures whether the explanation improves clinician task performance. The paper establishes that the first three are quantifiable with standard image metrics on the model and data already available, and that evaluation order matters: only methods that pass all three should proceed to human testing. The case study shows Ablation CAM scores higher than Eigen CAM on plausibility and fidelity in this setting, illustrating that a method can be consistent without being trustworthy.
Load-bearing premise
The load-bearing premise is that passing consistency, plausibility, and fidelity screens out explanations that would be useless or misleading for humans, a mapping the paper defines but does not test because usefulness is never measured.
Editorial extensions
If this is right
- Evaluation of explanation methods no longer has to begin with expensive user studies; consistency, plausibility, and fidelity can be scored automatically and used to screen candidates.
- An explainability scorecard gives developers, regulators, and clinicians a standard format for comparing methods and for documenting context, limitations, and validation setting.
- Because plausibility and fidelity are measured separately, an explanation can be clinically sensible while misrepresenting the model, or faithful to the model while clinically misleading; both cases should be reported.
- The framework deliberately sets no pass/fail thresholds, so adopters must choose and justify thresholds per task, which may favor consistency for one application and fidelity for another.
- Usefulness remains human-only in this framework, so the costliest evaluation cannot be fully automated.
Reading between the lines
- If the three quantitative tests turn out to predict usefulness, most reader studies for selecting explanation methods could eventually be replaced by automated benchmarking, but the paper does not provide the validation study that would establish this.
- The same four-criteria template could be extended to non-heatmap explanations such as counterfactuals, prototypes, and confidence scores, provided each has a metric for consistency, plausibility, and fidelity.
- The framework's fixed order is a resource-saving heuristic; interactions between criteria could make order matter in other ways, for example a highly plausible but low-fidelity method might deserve human testing precisely because it could mislead despite looking sensible.
- Setting acceptance thresholds is the natural next step; thresholds could be anchored to known cases where explanations demonstrably degraded reader performance, converting the scorecard from descriptive to regulatory-usable.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript proposes a framework for evaluating explainability features of AI models, primarily in medical imaging. The framework defines four criteria: consistency (stability of explanations under input perturbations), plausibility (alignment with human/ground-truth annotations), fidelity (alignment with the model's internal mechanisms), and usefulness (impact on clinician task performance). The first three are quantitative and are recommended as screens before expensive human studies; usefulness is to be assessed with human task-based studies. The authors introduce an explainability scorecard for standardized reporting and illustrate the first three criteria by comparing Eigen CAM and Ablation CAM heatmaps for a Faster R-CNN lesion detector on synthetic mammograms from the M-SYNTH dataset. The main claim is that requiring a method to pass consistency, plausibility, and fidelity filters out poor or misleading explanations before human testing.
Significance. If the framework were properly operationalized and validated, it would fill a real gap: standardized evaluation and reporting of XAI in medical devices, potentially reducing reliance on costly user studies and informing regulators and developers. The paper's strengths include a clear separation of four criteria, an explicit evaluation order, a scorecard for reporting, and use of synthetic data with known ground truth. However, the central screening claim is not yet supported because one of the three quantitative criteria (fidelity) is operationalized inconsistently, the plausibility metric mixes model accuracy into an explanation-quality measure, and the case study lacks statistical quantification. With revisions addressing these points, the framework could be a useful contribution.
major comments (4)
- [Section 3.2.3, Figure 4B] The fidelity criterion is internally inconsistent. Section 3.2.3 defines fidelity as alignment with the model's internal mechanisms and states, under the Single Deletion Method, that 'a correct explanation should show that removing the most important feature leads to the largest change in the model's output.' In Figure 4B, deleting the highest-activation ROI leaves the model prediction unchanged in both the left and right examples. By the paper's own definition, neither explanation is faithful, because the highlighted region is not causally important to the prediction. The right-hand example is instead labeled 'higher fidelity' because the heatmap changes a lot; a heatmap that changes dramatically when an unimportant region is removed is unstable, not faithful. This contradiction means that fidelity, as demonstrated, does not distinguish faithful from unfaithful explanations, and the screening order in Section 3.3 cannot yet rest on it.
- [Section 3.2.2, Figure 3A] Plausibility is defined as alignment between the explanation and human/ground-truth annotations, but the reported 'average plausibility score' is computed from IoU and Model Accuracy (Figure 3A caption and text). Model accuracy is a property of the model, not of the explanation; averaging it into the plausibility score conflates explanation quality with model performance and can make a method look more plausible simply because the model is accurate. The manuscript should either define a pure explanation-ground-truth overlap metric or explicitly justify and sensitivity-analyze the aggregation.
- [Section 3.2.2, Section 3.2.3, Figures 2-4] The case study claims that Ablation CAM has 'significantly higher plausibility' (Section 3.2.2) and 'consistently exhibits higher fidelity' (Section 3.2.3), but no error bars, confidence intervals, or statistical tests are reported in Figures 2A, 3A, or 4A, and the per-plot sample sizes are not stated. With only 150 images and metric values that fluctuate substantially, the observed differences may be within noise. Add error bars, significance tests or effect-size estimates, and state the number of images and perturbation replicates used for each metric.
- [Section 3.3] The central screening claim—that passing consistency, plausibility, and fidelity 'filters out poor or misleading explanations' before human testing—is asserted rather than tested. Usefulness is defined but never measured, and the paper provides no evidence that the three quantitative criteria predict whether an explanation improves clinician performance or reduces misinformation. If the quantitative screens are imperfect proxies, the recommended order could both waste resources and admit methods that fail in human use. The paper should either soften this claim to a hypothesis or report a validation study linking the quantitative criteria to usefulness outcomes.
minor comments (5)
- [Introduction] There is a typo on page 3: 'explinablility' should be 'explainability'.
- [Figure 2A] The axis labels are confusing: the x-axis for dose is given in scientific notation (e.g., 2.04x10^10) without explaining the units or the encoding of radiation dose. Please clarify how the dose values are defined and why these particular numeric values appear.
- [Section 3.2.1] The citation style is inconsistent: the text refers to 'Fuxin et al. 10', and the reference list entry is alphabetized under 'Li Fuxin'. Standardize the citation format and the ordering of the author's name.
- [Section 3, scorecard] The explainability scorecard is described as belonging to the proposed reporting mechanism but is only mentioned as supplementary material. Since the scorecard is part of the contribution, include at least a summary table or make the supplementary file available for review.
- [Materials and Methods, M-SYNTH dataset] The M-SYNTH dataset (reference [26]) is co-authored by two of the present authors, but the manuscript does not include a conflict-of-interest or disclosure statement acknowledging this connection.
Circularity Check
No circularity: the framework's criteria are stipulated definitions, and no fitted quantity or self-citation chain forces the conclusions.
full rationale
The paper proposes an evaluation framework, not a derived prediction, and its central claim rests on stipulated definitions rather than on a reduction from inputs to outputs. Consistency, plausibility, fidelity, and usefulness are defined in Sections 3.2.1–3.2.4 and then used as an evaluation ordering in Section 3.3; these criteria are not fitted from the case-study data, nor is any metric renamed as a prediction. The case study uses Ablation CAM and Eigen CAM on the M-SYNTH dataset, which is cited as a data source ([26], co-authored by two of the present authors), but that dataset is an input to the demonstration, not evidence for the framework's validity, so the self-citation is not load-bearing. The paper also explicitly states it does not provide thresholds or acceptability criteria (“this framework does not provide a threshold or any acceptability criteria”, Section 4), which prevents the framework from being self-validating by construction. The claimed filtering function in Section 3.3 (“Requiring a method to pass consistency, plausibility, and fidelity before involving human experts, filters out poor or misleading explanations”) is an empirical assumption about the predictive relationship between the quantitative screens and human usefulness; it is untested, but it is not a circular equivalence because usefulness is defined independently as task performance in human studies (Section 3.2.4). A separate reviewer concern about the internal consistency of the fidelity operationalization in Figure 4B is a validity issue, not a circularity issue, and does not constitute evidence that the framework reduces to its own inputs. No equation in the paper is equivalent by construction to an output being predicted, and no parameter is fitted and then reported as a prediction.
Assumptions & free parameters
assumptions (3)
- domain assumption Explainability quality is adequately captured by the four criteria of consistency, plausibility, fidelity, and usefulness, in that sequential order.
- domain assumption Model parameter randomization, reinitializing or randomizing weights and comparing explanations, is a valid indicator of fidelity.
- domain assumption Synthetic mammography annotations provide ground truth equivalent to expert human judgment for evaluating plausibility.
invented entities (1)
-
Explainability scorecard
Cite this review
Pith. "Pith review of Evaluating Explainability: A Framework for Systematic Assessment and Reporting of Explainable AI Features." pith.science (2026). https://pith.science/paper/JW7AA4DZ
@misc{pith2026250613917,
author = {Pith},
title = {Pith review of: Evaluating Explainability: A Framework for Systematic Assessment and Reporting of Explainable AI Features},
year = {2026},
howpublished = {\url{https://pith.science/paper/JW7AA4DZ}},
note = {Machine review of arXiv:2506.13917}
}
read the original abstract
Explainability features are intended to provide insight into the internal mechanisms of an AI device, but there is a lack of evaluation techniques for assessing the quality of provided explanations. We propose a framework to assess and report explainable AI features. Our evaluation framework for AI explainability is based on four criteria: 1) Consistency quantifies the variability of explanations to similar inputs, 2) Plausibility estimates how close the explanation is to the ground truth, 3) Fidelity assesses the alignment between the explanation and the model internal mechanisms, and 4) Usefulness evaluates the impact on task performance of the explanation. Finally, we developed a scorecard for AI explainability methods that serves as a complete description and evaluation to accompany this type of algorithm. We describe these four criteria and give examples on how they can be evaluated. As a case study, we use Ablation CAM and Eigen CAM to illustrate the evaluation of explanation heatmaps on the detection of breast lesions on synthetic mammographies. The first three criteria are evaluated for clinically-relevant scenarios. Our proposed framework establishes criteria through which the quality of explanations provided by AI models can be evaluated. We intend for our framework to spark a dialogue regarding the value provided by explainability features and help improve the development and evaluation of AI-based medical devices.
Reference graph
Works this paper leans on
-
[1]
Sanity checks for saliency maps.Advances in neural information processing systems, 31, 2018
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps.Advances in neural information processing systems, 31, 2018
work page 2018
-
[2]
Artificial intelligence risk management framework (ai rmf 1.0)
NIST AI. Artificial intelligence risk management framework (ai rmf 1.0). 2023
work page 2023
-
[3]
Michael H Bernstein, Michael K Atalay, Elizabeth H Dibble, Aaron WP Maxwell, Adib R Karam, Saurabh Agarwal, Robert C Ward, Terrance T Healey, and Grayson L Baird. Can incorrect artificial intelligence (ai) results impact radiologists, and if so, what can we do about it? a multi-reader pilot study of lung cancer detection with chest radiography.European ra...
work page 2023
-
[4]
David A Broniatowski et al. Psychological foundations of explainability and interpretability in artificial intelligence.NIST, Tech. Rep, 2021
work page 2021
-
[5]
Irene Celino.Who Is This Explanation for? Human Intelligence and Knowledge Graphs for eXplainable AI. IOS Press, 2020. doi: 10.3233/SSW200024
-
[6]
Moore, Marinka Zitnik, and John H
Carlo Combi, Beatrice Amico, Riccardo Bellazzi, Andreas Holzinger, Jason H. Moore, Marinka Zitnik, and John H. Holmes. A manifesto on explainability for artificial intelligence in medicine.Artificial Intelligence in Medicine, 133:102423, 2022. ISSN 0933-3657. doi: https://doi.org/10.1016/j.artmed.2022.102423. URLhttps://www.sciencedirect. com/science/arti...
-
[7]
Bart M de Vries, Gerben JC Zwezerijnen, George L Burchell, Floris HP van Velden, Catha- rina Willemien Menke-van der Houven van Oordt, and Ronald Boellaard. Explainable arti- ficial intelligence (xai) in radiology and nuclear medicine: a literature review.Frontiers in medicine, 10:1180773, 2023
work page 2023
- [8]
Show all 28 references
-
[9]
Towards a rigorous science of interpretable machine learning, 2017
Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning, 2017. URLhttps://arxiv.org/abs/1702.08608
2017 arXiv
-
[10]
From heatmaps to structured explanations of image classifiers.Applied AI Letters, 2(4):e46, 2021
Li Fuxin, Zhongang Qi, Saeed Khorram, Vivswan Shitole, Prasad Tadepalli, Minsuk Kahng, and Alan Fern. From heatmaps to structured explanations of image classifiers.Applied AI Letters, 2(4):e46, 2021
2021
-
[11]
Gilpin, David Bau, Ben Z
Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80–89, 2018....
2018
-
[12]
A survey of methods for explaining black box models.ACM computing surveys (CSUR), 51(5):1–42, 2018
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models.ACM computing surveys (CSUR), 51(5):1–42, 2018
2018
-
[13]
Darpa’s explainable ai (xai) program: A retrospective.Applied AI Letters, 2(4):e61, 2021
David Gunning, Eric V orm, Jennifer Yunyan Wang, and Matt Turek. Darpa’s explainable ai (xai) program: A retrospective.Applied AI Letters, 2(4):e61, 2021. doi: https://doi.org/10. 1002/ail2.61. URLhttps://onlinelibrary.wiley.com/doi/abs/10.1002/ ail2.61
2021
-
[14]
An analysis of explainability methods for convolutional neural networks.Engineering Applications of Artificial Intelligence, 2023
Lynn V onder Haar, Timothy Elvira, and Omar Ochoa. An analysis of explainability methods for convolutional neural networks.Engineering Applications of Artificial Intelligence, 2023. doi: https://doi.org/10.1016/j.engappai.2022.105606
2023
-
[15]
Extract interpretability-accuracy balanced rules from artificial neural networks: A review.Neurocomputing, 387:346–358, 2020
Congjie He, Meng Ma, and Ping Wang. Extract interpretability-accuracy balanced rules from artificial neural networks: A review.Neurocomputing, 387:346–358, 2020
2020
-
[16]
Mouton, Md Sirajus Salekin, Yu Sun, and Dmitry Goldgof
Md Imran Hossain, Ghada Zamzmi, Peter R. Mouton, Md Sirajus Salekin, Yu Sun, and Dmitry Goldgof. Explainable ai for medical data: Current methods, limitations, and future directions.ACM Comput. Surv., December 2023. ISSN 0360-0300. doi: 10.1145/3637487. URLhttps://doi.org/10.1...
2023 doi
-
[17]
Measuring the impact of ai in the diagnosis of hospitalized patients: a randomized clinical vignette survey study.JAMA, 330 (23):2275–2284, 2023
Sarah Jabbour, David Fouhey, Stephanie Shepard, Thomas S Valley, Ella A Kazerooni, Nikola Banovic, Jenna Wiens, and Michael W Sjoding. Measuring the impact of ai in the diagnosis of hospitalized patients: a randomized clinical vignette survey study.JAMA, 330 (23):2275–2284, 2023
2023
-
[18]
Automation bias and assistive ai: Risk of harm from ai-driven clinical decision support.JAMA, 330(23):2255–2257, 2023
Rohan Khera, Melissa A Simon, and Joseph S Ross. Automation bias and assistive ai: Risk of harm from ai-driven clinical decision support.JAMA, 330(23):2255–2257, 2023
2023
-
[19]
Interpretable decision sets: A joint framework for description and prediction
Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. Interpretable decision sets: A joint framework for description and prediction. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1675–1684, 2016
2016
-
[20]
The acr learn- ing network: facilitating local performance improvement through shared learning.Journal of the American College of Radiology, 20(3):369–376, 2023
David B Larson, Kandice Garcia Tomkins, Kay Zacharias-Andrews, Neville Irani, Sarah M Pittman, Andrei S Purysko, Ben Wandtke, and Mythreyi Bhargavan-Chatfield. The acr learn- ing network: facilitating local performance improvement through shared learning.Journal of the America...
2023
-
[21]
Eigen-cam: Class activation map using principal components
Mohammed Bany Muhammad and Mohammed Yeasin. Eigen-cam: Class activation map using principal components. In2020 International Joint Conference on Neural Networks (IJCNN). IEEE, July 2020. doi: 10.1109/ijcnn48605.2020.9206626. URLhttp://dx. doi.org/10.1109/IJCNN48605.2020.9206626
2020
-
[22]
From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai.ACM Computing Surveys, 55(13s):1–42, 2023
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, J¨org Schl ¨otterer, Maurice Van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai.ACM Computing S...
2023
-
[23]
Faster r-cnn: Towards real-time object detection with region proposal networks.IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks.IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016
2016
-
[24]
Anchors: high-precision model- agnostic explanations
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: high-precision model- agnostic explanations. InProceedings of the Thirty-Second AAAI Conference on Ar- tificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Con- ference and Eight...
2018
-
[25]
Quantifying interpretability and trust in machine learn- ing systems, 2019
Philipp Schmidt and Felix Biessmann. Quantifying interpretability and trust in machine learn- ing systems, 2019. URLhttps://arxiv.org/abs/1901.08558
2019 arXiv
-
[26]
Elena Sizikova, Niloufar Saharkhiz, Diksha Sharma, Miguel Lago, Berkman Sahiner, Jana Delfino, and Aldo Badano. Knowledge-based in silico models and dataset for the comparative evaluation of mammography ai for a range of breast characteristics, lesion conspicuities and doses.A...
2024
-
[27]
Gradient-based saliency maps are not trustworthy visual explanations of automated ai mus- culoskeletal diagnoses.Journal of Imaging Informatics in Medicine, pages 1–10, 2024
Kesavan Venkatesh, Simukayi Mutasa, Fletcher Moore, Jeremias Sulam, and Paul H Yi. Gradient-based saliency maps are not trustworthy visual explanations of automated ai mus- culoskeletal diagnoses.Journal of Imaging Informatics in Medicine, pages 1–10, 2024
2024
-
[28]
Notions of explainability and evaluation approaches for explainable artificial intelligence.Information Fusion, 76:89–106, 2021
Giulia Vilone and Luca Longo. Notions of explainability and evaluation approaches for explainable artificial intelligence.Information Fusion, 76:89–106, 2021. 16
2021
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.