pith. sign in

arxiv: 2606.28391 · v1 · pith:ANFMXTQRnew · submitted 2026-06-23 · 💻 cs.CV · cs.AI

Few-class Fidelity: Evaluating Explanations of Real-conditions CNN classifiers with Optimized Perturbations

Wistan Marchadour , Pedro Soto Vega , Franck Vermet , Mathieu Hatt This is my paper

Pith reviewed 2026-06-30 09:44 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords XAIfidelityCNNexplanationsperturbationsfaithfulnessmedical imagingnatural images
0
0 comments X

The pith

Few-class fidelity metric evaluates XAI explanations using optimized perturbations

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a fidelity-based evaluation for XAI methods suited to real-world CNN classifiers that have few classes. It creates perturbations inside the data distribution that cause the model to become uncertain, allowing accurate assessment of how well explanations match the model's behavior. This new metric is tested against human object localization and segmentation measures. When used on medical and natural images, it shows how the domain and how data is prepared affect which XAI method works best for validating a CNN. This helps ensure explanations are trustworthy when deploying models in practical settings.

Core claim

The central claim is that a variation of fidelity metrics, using in-distribution uncertainty-provoking perturbations, properly measures the faithfulness of XAI explanations for few-class CNN classifiers in real conditions, and this approach correlates with human-centric metrics when applied to medical and natural imaging applications.

What carries the argument

Generation of optimized in-distribution perturbations that provoke model uncertainty to quantify explanation faithfulness.

If this is right

  • Enables direct comparison of different XAI methods in low-class real-world scenarios.
  • Reveals correlations between application domain, data curation practices, and suitable XAI solutions.
  • Supports validation of new CNN model training through reliable explanation assessment.
  • Demonstrates utility in both medical imaging and natural image classification tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could extend to evaluating explanations in multi-class settings with adjustments.
  • It implies that current XAI evaluations may be unreliable without accounting for class count and distribution.
  • Future work might test the perturbations on other model architectures like transformers.

Load-bearing premise

Optimized perturbations can be created that stay in the training distribution while still causing enough uncertainty to test explanation quality accurately.

What would settle it

A test where the perturbations either exit the data distribution or do not increase model uncertainty would show the metric does not work as intended.

Figures

Figures reproduced from arXiv: 2606.28391 by Franck Vermet, Mathieu Hatt, Pedro Soto Vega, Wistan Marchadour.

Figure 1
Figure 1. Figure 1: Illustration of Out-of-Distribution issue. When images altered using a fixed value are given as an input to the model, it cannot recognize any known class, which is not a desired effect for the purpose of fidelity estimation. ( [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Proposed Fidelity metric. Input pixels are replaced iteratively over LIF and MIF orders (20%/40%/60%/80% shown here), altered images are inferred on by the classifier, and AUC of each order is measured. Finally, AUC values are compared to 𝑈𝑜𝑏𝑗 (dotted black line) to obtain 𝐹 𝑖𝑑. divergence [4], it becomes exponentially more expensive and unusable when dealing with entire image datasets. A simpler and faste… view at source ↗
Figure 3
Figure 3. Figure 3: Process of perturbation candidate selection. Candidates are generated from combining various image modifications (four examples given). 𝑈𝑠𝑐𝑜𝑟𝑒 is computed for all candidates, and scores above Δ are discarded. 𝑆𝑠𝑐𝑜𝑟𝑒, then 𝑃𝑠𝑐𝑜𝑟𝑒 are applied on remaining possible alterations. Candidate with the lowest 𝑃𝑠𝑐𝑜𝑟𝑒 is selected as perturbation for the Fidelity metric, for the corresponding input image. This grid of… view at source ↗
Figure 4
Figure 4. Figure 4: Few-class Fidelity box plot for ImageNet vehicles application, per model and base XAI method. For the ImageNet three-class vehicles classification, the noise reduction performed by SG (green) and SG² (orange) provide a small improvement of the metric scores, for most methods ( [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Few-class Fidelity box plot for Contrast agent application, per model and base XAI method. SG and SG² were not applied to Deconv and IG(max) for performance and computation time reasons. and agree on EG+XRAI as most successful method, while Deconv+XRAI gets high IoU score only with DenseNet￾121. Contrast Agent Given the multi-region ground-truth and lack of public annotations for this task, we manually seg… view at source ↗
Figure 6
Figure 6. Figure 6: Object localization box plot for ImageNet vehicles application, per model and base XAI method. choice for the IG method. Then, BP+XRAI and IG(min)+XRAI offer the most faithful saliency maps, completed by IG(min)+SG² for ResNet-50 model. On the other hand, the SG alteration did not prove useful. 5. Discussion The experiments of this study were designed to assess the performance of a wide range of XAI method… view at source ↗
Figure 7
Figure 7. Figure 7: Dice bar plot for Contrast agent application, per model and base XAI method. SG and SG² were not applied to Deconv and IG(max) for performance and computation time reasons. CRediT authorship contribution statement Wistan Marchadour: Methodology Design, Software Development, Article Redaction. Pedro Soto Vega: Models Training. Franck Vermet: Mathematical Validation, Article Review. Mathieu Hatt: Methodology… view at source ↗
read the original abstract

The wide use of Convolutional Neural Networks (CNN) in numerous domains and real-world classification applications is justified by their high precision and automation speed, helping users concentrate on higher-expertise tasks. To better understand the models and avoid bias during deployment, eXplainable Artificial Intelligence (XAI) techniques can be used after training. But as the list of XAI solutions expand, comparisons between them diverge, and consensus over their evaluation cannot be reached. This paper proposes a variation of Fidelity-based XAI metrics, with a focus on real-conditions applications, where the number of classes is often low. The approach generates in-distribution, uncertainty-provoking perturbations, to ensure proper measurement of the XAI methods faithfulness. As demonstration of the evaluation framework usefulness, it is compared with human-centric object localization and segmentation metrics. Once applied to both medical and natural imaging applications, it highlights the intricate correlation between domain, data curation, and XAI solution choices in order to validate training of a new CNN model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript proposes 'Few-class Fidelity,' a variation on fidelity-based XAI evaluation metrics tailored to real-world CNN classifiers that operate with few classes. It generates in-distribution perturbations via an optimization procedure intended to provoke model uncertainty, thereby providing a more faithful measure of explanation quality. The framework is demonstrated through comparison against human object-localization and segmentation metrics on both medical and natural-image datasets, with the results used to illustrate dependencies among domain, data curation, and choice of XAI method when validating a new CNN.

Significance. If the optimization can be shown to produce perturbations that remain in-distribution while reliably inducing uncertainty, the metric could supply a practical, domain-aware alternative to existing fidelity measures for low-class regimes. The explicit cross-domain comparison with human metrics offers an additional validation axis that is rarely quantified in XAI papers; however, the absence of any equations, loss terms, or tabulated results in the supplied text limits any assessment of whether these strengths are realized.

major comments (2)
  1. [Abstract] Abstract: the central claim that the perturbations are 'in-distribution' and 'uncertainty-provoking' is stated without any objective function, constraint set, or optimization algorithm. Because this construction is load-bearing for the definition of Few-class Fidelity, the metric cannot be reproduced or its faithfulness properties evaluated from the given text.
  2. [Abstract] Abstract: the comparison to 'human-centric object localization and segmentation metrics' is described only qualitatively; no quantitative scores, correlation coefficients, or error bars are supplied. This comparison is presented as the primary demonstration of the framework's usefulness, yet the lack of results prevents verification of the claimed 'intricate correlation' between domain and XAI choice.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting the need for greater detail in the abstract. We address each point below and will revise the abstract to improve reproducibility and support for the claims while preserving its concise nature.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that the perturbations are 'in-distribution' and 'uncertainty-provoking' is stated without any objective function, constraint set, or optimization algorithm. Because this construction is load-bearing for the definition of Few-class Fidelity, the metric cannot be reproduced or its faithfulness properties evaluated from the given text.

    Authors: We agree that the abstract does not include the specific objective function, constraints, or optimization algorithm, which limits standalone reproducibility. The full manuscript details the optimization procedure, including loss terms that enforce in-distribution perturbations while maximizing model uncertainty. To address the concern, we will revise the abstract to briefly reference the optimization framework and its key properties. revision: yes

  2. Referee: [Abstract] Abstract: the comparison to 'human-centric object localization and segmentation metrics' is described only qualitatively; no quantitative scores, correlation coefficients, or error bars are supplied. This comparison is presented as the primary demonstration of the framework's usefulness, yet the lack of results prevents verification of the claimed 'intricate correlation' between domain and XAI choice.

    Authors: We agree that the abstract presents the comparison only qualitatively and omits quantitative scores, correlations, or error bars. The full manuscript reports these results from experiments on medical and natural-image datasets. We will revise the abstract to include key quantitative findings that illustrate the domain-XAI dependencies. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper introduces a variation on fidelity metrics for XAI evaluation in few-class real-world settings by generating in-distribution uncertainty-provoking perturbations and validating against human localization/segmentation metrics. No equations, derivations, or self-citations are shown that reduce the proposed metric or its 'predictions' to quantities fitted from the same data or to prior author work by construction. The approach is presented as an independent evaluation framework with external human-centric benchmarks, making the derivation self-contained against external validation.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim depends on the untested premise that perturbations can be constructed to be simultaneously in-distribution and uncertainty-provoking, plus the assumption that human localization metrics serve as an appropriate external validator for the new fidelity score. No free parameters or invented entities are explicitly introduced in the abstract.

axioms (2)
  • domain assumption Perturbations exist that remain in-distribution for the target CNN while reliably increasing predictive uncertainty
    This premise underpins the claim that the generated perturbations enable proper faithfulness measurement.
  • domain assumption Agreement between the new fidelity scores and human object localization/segmentation metrics demonstrates the framework's usefulness
    The abstract uses this comparison as the demonstration of usefulness.

pith-pipeline@v0.9.1-grok · 5709 in / 1423 out tokens · 64357 ms · 2026-06-30T09:44:17.401938+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 10 canonical work pages · 6 internal anchors

  1. [1]

    Sanitychecksforsaliencymaps

    Adebayo,J.,Gilmer,J.,Muelly,M.,Goodfellow,I.,Hardt,M.,Kim,B.,2018. Sanitychecksforsaliencymaps. Advancesinneuralinformation processing systems 31

    2018

  2. [2]

    Explainingimageclassifiersbyremovinginputfeaturesusinggenerativemodels,in:ProceedingsoftheAsian Conference on Computer Vision

    Agarwal,C.,Nguyen,A.,2020. Explainingimageclassifiersbyremovinginputfeaturesusinggenerativemodels,in:ProceedingsoftheAsian Conference on Computer Vision. W. Marchadour et al.:Preprint submitted to ElsevierPage 14 of 16 Few-class Fidelity for CNN classifiers

    2020

  3. [3]

    Towards better understanding of gradient-based attribution methods for Deep Neural Networks

    Ancona, M., Ceolini, E., Öztireli, C., Gross, M., 2017. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  4. [4]

    Fairwashing explanations with off-manifold detergent, in: International Conference on Machine Learning, PMLR

    Anders, C., Pasliev, P., Dombrowski, A.K., Müller, K.R., Kessel, P., 2020. Fairwashing explanations with off-manifold detergent, in: International Conference on Machine Learning, PMLR. pp. 314–323

    2020

  5. [5]

    The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans

    ArmatoIII,S.G.,McLennan,G.,Bidaut,L.,McNitt-Gray,M.F.,Meyer,C.R.,Reeves,A.P.,Zhao,B.,Aberle,D.R.,Henschke,C.I.,Hoffman, E.A., et al., 2011. The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Medical physics 38, 915–931

    2011

  6. [6]

    Brocki, L., Marchadour, W., Maison, J., Badic, B., Papadimitroulas, P., Hatt, M., Vermet, F., Chung, N.C., 2022. Evaluation of importance estimatorsindeeplearningclassifiersforcomputedtomography,in:InternationalWorkshoponExplainable,TransparentAutonomousAgents and Multi-Agent Systems, Springer. pp. 3–18

    2022

  7. [7]

    Explaining Image Classifiers by Counterfactual Generation

    Chang, C.H., Creager, E., Goldenberg, A., Duvenaud, D., 2018. Explaining image classifiers by counterfactual generation. arXiv preprint arXiv:1807.08024

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  8. [8]

    Real time image saliency for black box classifiers

    Dabkowski, P., Gal, Y., 2017. Real time image saliency for black box classifiers. Advances in neural information processing systems 30

    2017

  9. [9]

    Improving performance of deep learning models with axiomatic attribution priors and expected gradients

    Erion, G., Janizek, J.D., Sturmfels, P., Lundberg, S.M., Lee, S.I., 2021. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence 3, 620–631

    2021

  10. [10]

    Interpretableexplanationsofblackboxesbymeaningfulperturbation,in:ProceedingsoftheIEEEinternational conference on computer vision, pp

    Fong,R.C.,Vedaldi,A.,2017. Interpretableexplanationsofblackboxesbymeaningfulperturbation,in:ProceedingsoftheIEEEinternational conference on computer vision, pp. 3429–3437

    2017

  11. [11]

    Counterfactualvisualexplanations,in:InternationalConferenceonMachine Learning, PMLR

    Goyal,Y.,Wu,Z.,Ernst,J.,Batra,D.,Parikh,D.,Lee,S.,2019. Counterfactualvisualexplanations,in:InternationalConferenceonMachine Learning, PMLR. pp. 2376–2384

    2019

  12. [12]

    Evaluating feature importance estimates

    Hooker, S., Erhan, D., Kindermans, P.J., Kim, B., 2018. Evaluating feature importance estimates. arXiv preprint arXiv:1806.10758 2

    work page arXiv 2018

  13. [13]

    A benchmark for interpretability methods in deep neural networks

    Hooker, S., Erhan, D., Kindermans, P.J., Kim, B., 2019. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems 32

    2019

  14. [14]

    Poisoningandevasionattacksagainstdeeplearningalgorithmsinautonomousvehicles

    Jiang,W.,Li,H.,Liu,S.,Luo,X.,Lu,R.,2020. Poisoningandevasionattacksagainstdeeplearningalgorithmsinautonomousvehicles. IEEE transactions on vehicular technology 69, 4439–4449

    2020

  15. [15]

    Xrai: Better attributions through regions, in: Proceedings of the IEEE/CVF international conference on computer vision, pp

    Kapishnikov, A., Bolukbasi, T., Viégas, F., Terry, M., 2019. Xrai: Better attributions through regions, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 4948–4957

    2019

  16. [16]

    Adversarial deep learning: A survey on adversarial attacks and defense mechanisms on image classification

    Khamaiseh, S.Y., Bagagem, D., Al-Alaj, A., Mancino, M., Alomari, H.W., 2022. Adversarial deep learning: A survey on adversarial attacks and defense mechanisms on image classification. IEEE Access 10, 102266–102291

    2022

  17. [17]

    Lin,Y.S.,Lee,W.C.,Celik,Z.B.,2021. Whatdoyousee?evaluationofexplainableartificialintelligence(xai)interpretabilitythroughneural backdoors, in: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 1027–1035

    2021

  18. [18]

    Methods for interpreting and understanding deep neural networks

    Montavon, G., Samek, W., Müller, K.R., 2018. Methods for interpreting and understanding deep neural networks. Digital signal processing 73, 1–15

    2018

  19. [19]

    Universaladversarialperturbations,in:ProceedingsoftheIEEEconference on computer vision and pattern recognition, pp

    Moosavi-Dezfooli,S.M.,Fawzi,A.,Fawzi,O.,Frossard,P.,2017. Universaladversarialperturbations,in:ProceedingsoftheIEEEconference on computer vision and pattern recognition, pp. 1765–1773

    2017

  20. [20]

    Nam, W.J., Gur, S., Choi, J., Wolf, L., Lee, S.W., 2020. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks, in: Proceedings of the AAAI conference on artificial intelligence, pp. 2501–2508

    2020

  21. [21]

    Deepneuralnetworksareeasilyfooled:Highconfidencepredictionsforunrecognizableimages,in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp

    Nguyen,A.,Yosinski,J.,Clune,J.,2015. Deepneuralnetworksareeasilyfooled:Highconfidencepredictionsforunrecognizableimages,in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427–436

    2015

  22. [22]

    Lndb: a lung nodule database on computed tomography

    Pedrosa,J.,Aresta,G.,Ferreira,C.,Rodrigues,M.,Leitão,P.,Carvalho,A.S.,Rebelo,J.,Negrão,E.,Ramos,I.,Cunha,A.,etal.,2019. Lndb: a lung nodule database on computed tomography. arXiv preprint arXiv:1911.08434

    work page arXiv 2019

  23. [23]

    RISE: Randomized Input Sampling for Explanation of Black-box Models

    Petsiuk, V., Das, A., Saenko, K., 2018. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  24. [24]

    Resisting out-of-distribution data problem in perturbation of xai

    Qiu, L., Yang, Y., Cao, C.C., Liu, J., Zheng, Y., Ngai, H.H.T., Hsiao, J., Chen, L., 2021. Resisting out-of-distribution data problem in perturbation of xai. arXiv preprint arXiv:2107.14000

    work page arXiv 2021

  25. [25]

    why should i trust you?

    Ribeiro, M.T., Singh, S., Guestrin, C., 2016. " why should i trust you?" explaining the predictions of any classifier, in: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135–1144

    2016

  26. [26]

    Irof: a low resource evaluation metric for explanation methods

    Rieger, L., Hansen, L.K., 2020. Irof: a low resource evaluation metric for explanation methods. arXiv preprint arXiv:2003.08747

    work page arXiv 2020

  27. [27]

    Evaluating the visualization of what a deep neural network has learned

    Samek, W., Binder, A., Montavon, G., Lapuschkin, S., Müller, K.R., 2016. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems 28, 2660–2673

    2016

  28. [28]

    Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE international conference on computer vision, pp

    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D., 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE international conference on computer vision, pp. 618–626

    2017

  29. [29]

    Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning

    Shin, H.C., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., Summers, R.M., 2016. Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE transactions on medical imaging 35, 1285–1298

    2016

  30. [30]

    A comprehensive analysis of perturbation methods in explainable ai feature attribution validation for neural time series classifiers

    Šimić, I., Veas, E., Sabol, V., 2025. A comprehensive analysis of perturbation methods in explainable ai feature attribution validation for neural time series classifiers. Scientific Reports 15, 26607

    2025

  31. [31]

    Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

    Simonyan, K., Vedaldi, A., Zisserman, A., 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  32. [32]

    SmoothGrad: removing noise by adding noise

    Smilkov, D., Thorat, N., Kim, B., Viégas, F., Wattenberg, M., 2017. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  33. [33]

    Axiomaticattributionfordeepnetworks,in:Internationalconferenceonmachinelearning,PMLR

    Sundararajan,M.,Taly,A.,Yan,Q.,2017. Axiomaticattributionfordeepnetworks,in:Internationalconferenceonmachinelearning,PMLR. pp. 3319–3328. W. Marchadour et al.:Preprint submitted to ElsevierPage 15 of 16 Few-class Fidelity for CNN classifiers

    2017

  34. [34]

    A deep neural network and machine learning approach for retinal fundus image classification

    Thanki, R., 2023. A deep neural network and machine learning approach for retinal fundus image classification. Healthcare Analytics 3, 100140

    2023

  35. [35]

    On the (in) fidelity and sensitivity of explanations

    Yeh, C.K., Hsieh, C.Y., Suggala, A., Inouye, D.I., Ravikumar, P.K., 2019. On the (in) fidelity and sensitivity of explanations. Advances in neural information processing systems 32

    2019

  36. [36]

    Foreign object debris (fod) classification through material recognition using deep convolutional neural network with focus on metal

    Zainab, S.M., Khan, K., Fazil, A., Zakwan, M., 2023. Foreign object debris (fod) classification through material recognition using deep convolutional neural network with focus on metal. IEEE Access 11, 10925–10934

    2023

  37. [37]

    Visualizing and Understanding Convolutional Networks

    Zeiler, M., Fergus, R., 2013. Visualizing and understanding convolutional networks. arxiv. arXiv preprint arXiv:1311.2901

    work page internal anchor Pith review Pith/arXiv arXiv 2013

  38. [38]

    A survey on neural network interpretability

    Zhang, Y., Tiňo, P., Leonardis, A., Tang, K., 2021. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence 5, 726–742. W. Marchadour et al.:Preprint submitted to ElsevierPage 16 of 16

    2021