Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Evaluating Explainability: A Framework for Systematic Assessment and Reporting of Explainable AI Features

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper proposes that AI explanation quality can be systematically judged by four criteria—consistency, plausibility, fidelity, and usefulness—and that the first three can screen methods before costly human studies.

desk verdict The framework is worth engaging with, but the fidelity example contradicts the paper's own definition, and the screening claim is unsupported until that's fixed. read the letter →

arxiv 2506.13917 v1 pith:JW7AA4DZ submitted 2025-06-16 cs.AI

classification cs.AI
keywords explainableAIexplainabilityevaluationconsistencyplausibilityfidelityusefulnessmedicalimagingscorecard
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the quality of explanations produced by AI medical devices should be judged by four distinct criteria: consistency, plausibility, fidelity, and usefulness. The first three can be computed automatically on existing data, while usefulness requires a human study, and the authors recommend evaluating them in that order so that methods failing early tests are filtered out before expensive reader studies. They demonstrate the framework by scoring two heatmap methods, Ablation CAM and Eigen CAM, on synthetic mammograms for breast lesion detection. They also package the results in an explainability scorecard for standardized reporting. The larger aim is to give developers, evaluators, and clinicians a common language for deciding whether an AI's explanation can be trusted.

What carries the argument

The machinery is an evaluation pipeline with four criteria and a companion scorecard. Consistency is measured by comparing heatmaps under perturbations such as radiation dose changes and rotation, using metrics like SSIM, MSE, and IoU. Plausibility is measured by the overlap between the heatmap's highlighted region and the known ground-truth lesion location. Fidelity is measured by model parameter randomization and deletion tests that compare how the explanation and the model's accuracy change under perturbation. Usefulness is reserved for human task-performance studies. The scorecard combines a descriptive section—method overview, context of use, limitations, and validation setting—with a quantitative section reporting the four criteria, giving stakeholders a standardized report.

What would settle it

Run a blinded reader study where radiologists interpret mammograms with and without heatmaps from a method that has passed high thresholds on consistency, plausibility, and fidelity. If the heatmaps do not improve diagnostic accuracy or time-to-decision—or make them worse—the three quantitative criteria have admitted a useless explanation, showing they do not guarantee usefulness.

Watch

Extended reading notes

Core claim

The central claim is that an explainability method can be systematically evaluated through four independent criteria: consistency, which measures how stable the explanation is under small input perturbations; plausibility, which measures overlap with human ground truth; fidelity, which measures how well the explanation tracks the model's actual decision mechanism; and usefulness, which measures whether the explanation improves clinician task performance. The paper establishes that the first three are quantifiable with standard image metrics on the model and data already available, and that evaluation order matters: only methods that pass all three should proceed to human testing. The case study shows Ablation CAM scores higher than Eigen CAM on plausibility and fidelity in this setting, illustrating that a method can be consistent without being trustworthy.

Load-bearing premise

The load-bearing premise is that passing consistency, plausibility, and fidelity screens out explanations that would be useless or misleading for humans, a mapping the paper defines but does not test because usefulness is never measured.

Editorial extensions

If this is right

  • Evaluation of explanation methods no longer has to begin with expensive user studies; consistency, plausibility, and fidelity can be scored automatically and used to screen candidates.
  • An explainability scorecard gives developers, regulators, and clinicians a standard format for comparing methods and for documenting context, limitations, and validation setting.
  • Because plausibility and fidelity are measured separately, an explanation can be clinically sensible while misrepresenting the model, or faithful to the model while clinically misleading; both cases should be reported.
  • The framework deliberately sets no pass/fail thresholds, so adopters must choose and justify thresholds per task, which may favor consistency for one application and fidelity for another.
  • Usefulness remains human-only in this framework, so the costliest evaluation cannot be fully automated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the three quantitative tests turn out to predict usefulness, most reader studies for selecting explanation methods could eventually be replaced by automated benchmarking, but the paper does not provide the validation study that would establish this.
  • The same four-criteria template could be extended to non-heatmap explanations such as counterfactuals, prototypes, and confidence scores, provided each has a metric for consistency, plausibility, and fidelity.
  • The framework's fixed order is a resource-saving heuristic; interactions between criteria could make order matter in other ways, for example a highly plausible but low-fidelity method might deserve human testing precisely because it could mislead despite looking sensible.
  • Setting acceptance thresholds is the natural next step; thresholds could be anchored to known cases where explanations demonstrably degraded reader performance, converting the scorecard from descriptive to regulatory-usable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This manuscript proposes a framework for evaluating explainability features of AI models, primarily in medical imaging. The framework defines four criteria: consistency (stability of explanations under input perturbations), plausibility (alignment with human/ground-truth annotations), fidelity (alignment with the model's internal mechanisms), and usefulness (impact on clinician task performance). The first three are quantitative and are recommended as screens before expensive human studies; usefulness is to be assessed with human task-based studies. The authors introduce an explainability scorecard for standardized reporting and illustrate the first three criteria by comparing Eigen CAM and Ablation CAM heatmaps for a Faster R-CNN lesion detector on synthetic mammograms from the M-SYNTH dataset. The main claim is that requiring a method to pass consistency, plausibility, and fidelity filters out poor or misleading explanations before human testing.

Significance. If the framework were properly operationalized and validated, it would fill a real gap: standardized evaluation and reporting of XAI in medical devices, potentially reducing reliance on costly user studies and informing regulators and developers. The paper's strengths include a clear separation of four criteria, an explicit evaluation order, a scorecard for reporting, and use of synthetic data with known ground truth. However, the central screening claim is not yet supported because one of the three quantitative criteria (fidelity) is operationalized inconsistently, the plausibility metric mixes model accuracy into an explanation-quality measure, and the case study lacks statistical quantification. With revisions addressing these points, the framework could be a useful contribution.

major comments (4)
  1. [Section 3.2.3, Figure 4B] The fidelity criterion is internally inconsistent. Section 3.2.3 defines fidelity as alignment with the model's internal mechanisms and states, under the Single Deletion Method, that 'a correct explanation should show that removing the most important feature leads to the largest change in the model's output.' In Figure 4B, deleting the highest-activation ROI leaves the model prediction unchanged in both the left and right examples. By the paper's own definition, neither explanation is faithful, because the highlighted region is not causally important to the prediction. The right-hand example is instead labeled 'higher fidelity' because the heatmap changes a lot; a heatmap that changes dramatically when an unimportant region is removed is unstable, not faithful. This contradiction means that fidelity, as demonstrated, does not distinguish faithful from unfaithful explanations, and the screening order in Section 3.3 cannot yet rest on it.
  2. [Section 3.2.2, Figure 3A] Plausibility is defined as alignment between the explanation and human/ground-truth annotations, but the reported 'average plausibility score' is computed from IoU and Model Accuracy (Figure 3A caption and text). Model accuracy is a property of the model, not of the explanation; averaging it into the plausibility score conflates explanation quality with model performance and can make a method look more plausible simply because the model is accurate. The manuscript should either define a pure explanation-ground-truth overlap metric or explicitly justify and sensitivity-analyze the aggregation.
  3. [Section 3.2.2, Section 3.2.3, Figures 2-4] The case study claims that Ablation CAM has 'significantly higher plausibility' (Section 3.2.2) and 'consistently exhibits higher fidelity' (Section 3.2.3), but no error bars, confidence intervals, or statistical tests are reported in Figures 2A, 3A, or 4A, and the per-plot sample sizes are not stated. With only 150 images and metric values that fluctuate substantially, the observed differences may be within noise. Add error bars, significance tests or effect-size estimates, and state the number of images and perturbation replicates used for each metric.
  4. [Section 3.3] The central screening claim—that passing consistency, plausibility, and fidelity 'filters out poor or misleading explanations' before human testing—is asserted rather than tested. Usefulness is defined but never measured, and the paper provides no evidence that the three quantitative criteria predict whether an explanation improves clinician performance or reduces misinformation. If the quantitative screens are imperfect proxies, the recommended order could both waste resources and admit methods that fail in human use. The paper should either soften this claim to a hypothesis or report a validation study linking the quantitative criteria to usefulness outcomes.
minor comments (5)
  1. [Introduction] There is a typo on page 3: 'explinablility' should be 'explainability'.
  2. [Figure 2A] The axis labels are confusing: the x-axis for dose is given in scientific notation (e.g., 2.04x10^10) without explaining the units or the encoding of radiation dose. Please clarify how the dose values are defined and why these particular numeric values appear.
  3. [Section 3.2.1] The citation style is inconsistent: the text refers to 'Fuxin et al. 10', and the reference list entry is alphabetized under 'Li Fuxin'. Standardize the citation format and the ordering of the author's name.
  4. [Section 3, scorecard] The explainability scorecard is described as belonging to the proposed reporting mechanism but is only mentioned as supplementary material. Since the scorecard is part of the contribution, include at least a summary table or make the supplementary file available for review.
  5. [Materials and Methods, M-SYNTH dataset] The M-SYNTH dataset (reference [26]) is co-authored by two of the present authors, but the manuscript does not include a conflict-of-interest or disclosure statement acknowledging this connection.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the framework's criteria are stipulated definitions, and no fitted quantity or self-citation chain forces the conclusions.

full rationale

The paper proposes an evaluation framework, not a derived prediction, and its central claim rests on stipulated definitions rather than on a reduction from inputs to outputs. Consistency, plausibility, fidelity, and usefulness are defined in Sections 3.2.1–3.2.4 and then used as an evaluation ordering in Section 3.3; these criteria are not fitted from the case-study data, nor is any metric renamed as a prediction. The case study uses Ablation CAM and Eigen CAM on the M-SYNTH dataset, which is cited as a data source ([26], co-authored by two of the present authors), but that dataset is an input to the demonstration, not evidence for the framework's validity, so the self-citation is not load-bearing. The paper also explicitly states it does not provide thresholds or acceptability criteria (“this framework does not provide a threshold or any acceptability criteria”, Section 4), which prevents the framework from being self-validating by construction. The claimed filtering function in Section 3.3 (“Requiring a method to pass consistency, plausibility, and fidelity before involving human experts, filters out poor or misleading explanations”) is an empirical assumption about the predictive relationship between the quantitative screens and human usefulness; it is untested, but it is not a circular equivalence because usefulness is defined independently as task performance in human studies (Section 3.2.4). A separate reviewer concern about the internal consistency of the fidelity operationalization in Figure 4B is a validity issue, not a circularity issue, and does not constitute evidence that the framework reduces to its own inputs. No equation in the paper is equivalent by construction to an output being predicted, and no parameter is fitted and then reported as a prediction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The framework is a proposal, not a derivation. The main load-bearing assumptions are that the four criteria are complete and ordered correctly, that the parameter randomization test measures fidelity, and that synthetic ground truth stands in for human judgment. No numbers are fitted to data; the empirical example is illustrative.

assumptions (3)
  • domain assumption Explainability quality is adequately captured by the four criteria of consistency, plausibility, fidelity, and usefulness, in that sequential order.
    Section 3 defines these as the evaluation criteria and Section 3.3 prescribes an order, but no completeness proof or comparison with alternative taxonomies is provided.
  • domain assumption Model parameter randomization, reinitializing or randomizing weights and comparing explanations, is a valid indicator of fidelity.
    Section 3.2.3 relies on Adebayo et al.'s sanity check as the fidelity measure; the validity of this check for CAM-style explanations is assumed.
  • domain assumption Synthetic mammography annotations provide ground truth equivalent to expert human judgment for evaluating plausibility.
    Section 3.2 uses M-SYNTH lesion locations as ground truth; this assumes synthetic labels faithfully represent what a radiologist would mark.
invented entities (1)
  • Explainability scorecard
    purpose: Standardized descriptive and quantitative reporting template for XAI methods
    Introduced in Section 3 and mentioned in the supplementary material; no external validation of its usability or coverage is presented.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Explainability: A Framework for Systematic Assessment and Reporting of Explainable AI Features." pith.science (2026). https://pith.science/paper/JW7AA4DZ

@misc{pith2026250613917,
  author       = {Pith},
  title        = {Pith review of: Evaluating Explainability: A Framework for Systematic Assessment and Reporting of Explainable AI Features},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JW7AA4DZ}},
  note         = {Machine review of arXiv:2506.13917}
}
read the original abstract

Explainability features are intended to provide insight into the internal mechanisms of an AI device, but there is a lack of evaluation techniques for assessing the quality of provided explanations. We propose a framework to assess and report explainable AI features. Our evaluation framework for AI explainability is based on four criteria: 1) Consistency quantifies the variability of explanations to similar inputs, 2) Plausibility estimates how close the explanation is to the ground truth, 3) Fidelity assesses the alignment between the explanation and the model internal mechanisms, and 4) Usefulness evaluates the impact on task performance of the explanation. Finally, we developed a scorecard for AI explainability methods that serves as a complete description and evaluation to accompany this type of algorithm. We describe these four criteria and give examples on how they can be evaluated. As a case study, we use Ablation CAM and Eigen CAM to illustrate the evaluation of explanation heatmaps on the detection of breast lesions on synthetic mammographies. The first three criteria are evaluated for clinically-relevant scenarios. Our proposed framework establishes criteria through which the quality of explanations provided by AI models can be evaluated. We intend for our framework to spark a dialogue regarding the value provided by explainability features and help improve the development and evaluation of AI-based medical devices.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 20 canonical work pages

  1. [1]

    Sanity checks for saliency maps.Advances in neural information processing systems, 31, 2018

    Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps.Advances in neural information processing systems, 31, 2018

  2. [2]

    Artificial intelligence risk management framework (ai rmf 1.0)

    NIST AI. Artificial intelligence risk management framework (ai rmf 1.0). 2023

  3. [3]

    Michael H Bernstein, Michael K Atalay, Elizabeth H Dibble, Aaron WP Maxwell, Adib R Karam, Saurabh Agarwal, Robert C Ward, Terrance T Healey, and Grayson L Baird. Can incorrect artificial intelligence (ai) results impact radiologists, and if so, what can we do about it? a multi-reader pilot study of lung cancer detection with chest radiography.European ra...

  4. [4]

    Psychological foundations of explainability and interpretability in artificial intelligence.NIST, Tech

    David A Broniatowski et al. Psychological foundations of explainability and interpretability in artificial intelligence.NIST, Tech. Rep, 2021

  5. [5]

    IOS Press, 2020

    Irene Celino.Who Is This Explanation for? Human Intelligence and Knowledge Graphs for eXplainable AI. IOS Press, 2020. doi: 10.3233/SSW200024

  6. [6]

    Moore, Marinka Zitnik, and John H

    Carlo Combi, Beatrice Amico, Riccardo Bellazzi, Andreas Holzinger, Jason H. Moore, Marinka Zitnik, and John H. Holmes. A manifesto on explainability for artificial intelligence in medicine.Artificial Intelligence in Medicine, 133:102423, 2022. ISSN 0933-3657. doi: https://doi.org/10.1016/j.artmed.2022.102423. URLhttps://www.sciencedirect. com/science/arti...

  7. [7]

    Explainable arti- ficial intelligence (xai) in radiology and nuclear medicine: a literature review.Frontiers in medicine, 10:1180773, 2023

    Bart M de Vries, Gerben JC Zwezerijnen, George L Burchell, Floris HP van Velden, Catha- rina Willemien Menke-van der Houven van Oordt, and Ronald Boellaard. Explainable arti- ficial intelligence (xai) in radiology and nuclear medicine: a literature review.Frontiers in medicine, 10:1180773, 2023

  8. [8]

    Ramaswamy

    Saurabh Desai and Harish G. Ramaswamy. Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. In2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 972–980, 2020. doi: 10.1109/W ACV45572. 2020.9093360

Show all 28 references
  1. [9]

    Towards a rigorous science of interpretable machine learning, 2017

    Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning, 2017. URLhttps://arxiv.org/abs/1702.08608

  2. [10]

    From heatmaps to structured explanations of image classifiers.Applied AI Letters, 2(4):e46, 2021

    Li Fuxin, Zhongang Qi, Saeed Khorram, Vivswan Shitole, Prasad Tadepalli, Minsuk Kahng, and Alan Fern. From heatmaps to structured explanations of image classifiers.Applied AI Letters, 2(4):e46, 2021

  3. [11]

    Gilpin, David Bau, Ben Z

    Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80–89, 2018....

  4. [12]

    A survey of methods for explaining black box models.ACM computing surveys (CSUR), 51(5):1–42, 2018

    Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models.ACM computing surveys (CSUR), 51(5):1–42, 2018

  5. [13]

    Darpa’s explainable ai (xai) program: A retrospective.Applied AI Letters, 2(4):e61, 2021

    David Gunning, Eric V orm, Jennifer Yunyan Wang, and Matt Turek. Darpa’s explainable ai (xai) program: A retrospective.Applied AI Letters, 2(4):e61, 2021. doi: https://doi.org/10. 1002/ail2.61. URLhttps://onlinelibrary.wiley.com/doi/abs/10.1002/ ail2.61

  6. [14]

    An analysis of explainability methods for convolutional neural networks.Engineering Applications of Artificial Intelligence, 2023

    Lynn V onder Haar, Timothy Elvira, and Omar Ochoa. An analysis of explainability methods for convolutional neural networks.Engineering Applications of Artificial Intelligence, 2023. doi: https://doi.org/10.1016/j.engappai.2022.105606

  7. [15]

    Extract interpretability-accuracy balanced rules from artificial neural networks: A review.Neurocomputing, 387:346–358, 2020

    Congjie He, Meng Ma, and Ping Wang. Extract interpretability-accuracy balanced rules from artificial neural networks: A review.Neurocomputing, 387:346–358, 2020

  8. [16]

    Mouton, Md Sirajus Salekin, Yu Sun, and Dmitry Goldgof

    Md Imran Hossain, Ghada Zamzmi, Peter R. Mouton, Md Sirajus Salekin, Yu Sun, and Dmitry Goldgof. Explainable ai for medical data: Current methods, limitations, and future directions.ACM Comput. Surv., December 2023. ISSN 0360-0300. doi: 10.1145/3637487. URLhttps://doi.org/10.1...

  9. [17]

    Measuring the impact of ai in the diagnosis of hospitalized patients: a randomized clinical vignette survey study.JAMA, 330 (23):2275–2284, 2023

    Sarah Jabbour, David Fouhey, Stephanie Shepard, Thomas S Valley, Ella A Kazerooni, Nikola Banovic, Jenna Wiens, and Michael W Sjoding. Measuring the impact of ai in the diagnosis of hospitalized patients: a randomized clinical vignette survey study.JAMA, 330 (23):2275–2284, 2023

  10. [18]

    Automation bias and assistive ai: Risk of harm from ai-driven clinical decision support.JAMA, 330(23):2255–2257, 2023

    Rohan Khera, Melissa A Simon, and Joseph S Ross. Automation bias and assistive ai: Risk of harm from ai-driven clinical decision support.JAMA, 330(23):2255–2257, 2023

  11. [19]

    Interpretable decision sets: A joint framework for description and prediction

    Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. Interpretable decision sets: A joint framework for description and prediction. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1675–1684, 2016

  12. [20]

    The acr learn- ing network: facilitating local performance improvement through shared learning.Journal of the American College of Radiology, 20(3):369–376, 2023

    David B Larson, Kandice Garcia Tomkins, Kay Zacharias-Andrews, Neville Irani, Sarah M Pittman, Andrei S Purysko, Ben Wandtke, and Mythreyi Bhargavan-Chatfield. The acr learn- ing network: facilitating local performance improvement through shared learning.Journal of the America...

  13. [21]

    Eigen-cam: Class activation map using principal components

    Mohammed Bany Muhammad and Mohammed Yeasin. Eigen-cam: Class activation map using principal components. In2020 International Joint Conference on Neural Networks (IJCNN). IEEE, July 2020. doi: 10.1109/ijcnn48605.2020.9206626. URLhttp://dx. doi.org/10.1109/IJCNN48605.2020.9206626

  14. [22]

    From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai.ACM Computing Surveys, 55(13s):1–42, 2023

    Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, J¨org Schl ¨otterer, Maurice Van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai.ACM Computing S...

  15. [23]

    Faster r-cnn: Towards real-time object detection with region proposal networks.IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks.IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016

  16. [24]

    Anchors: high-precision model- agnostic explanations

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: high-precision model- agnostic explanations. InProceedings of the Thirty-Second AAAI Conference on Ar- tificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Con- ference and Eight...

  17. [25]

    Quantifying interpretability and trust in machine learn- ing systems, 2019

    Philipp Schmidt and Felix Biessmann. Quantifying interpretability and trust in machine learn- ing systems, 2019. URLhttps://arxiv.org/abs/1901.08558

  18. [26]

    Elena Sizikova, Niloufar Saharkhiz, Diksha Sharma, Miguel Lago, Berkman Sahiner, Jana Delfino, and Aldo Badano. Knowledge-based in silico models and dataset for the comparative evaluation of mammography ai for a range of breast characteristics, lesion conspicuities and doses.A...

  19. [27]

    Gradient-based saliency maps are not trustworthy visual explanations of automated ai mus- culoskeletal diagnoses.Journal of Imaging Informatics in Medicine, pages 1–10, 2024

    Kesavan Venkatesh, Simukayi Mutasa, Fletcher Moore, Jeremias Sulam, and Paul H Yi. Gradient-based saliency maps are not trustworthy visual explanations of automated ai mus- culoskeletal diagnoses.Journal of Imaging Informatics in Medicine, pages 1–10, 2024

  20. [28]

    Notions of explainability and evaluation approaches for explainable artificial intelligence.Information Fusion, 76:89–106, 2021

    Giulia Vilone and Luca Longo. Notions of explainability and evaluation approaches for explainable artificial intelligence.Information Fusion, 76:89–106, 2021. 16

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.