REVIEW 4 major objections 6 minor 6 references
Impact of Adversarial Attacks on Deep Learning Model Explainability
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that FGSM and BIM adversarial attacks, which cut model accuracy from 89.94% to 58.73% and 45.50%, leave GradCAM, SmoothGrad, and LIME explanation masks nearly unchanged, as measured by IoU and RMSE against SAM-generated…
desk verdict A reproducible pilot with a clean observation, but the main claim about metric insensitivity is undermined by scoring predicted-class explanations against true-class masks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The experimental loop is the central mechanism: for each image, a SAM-generated segmentation serves as ground-truth explanation; the model classifies the original image and the adversarial image; GradCAM, SmoothGrad, and LIME produce explanation masks; the top 15% of pixels are retained; and IoU and RMSE compare these masks to the ground truth. The attack equations are FGSM, $x_{adv} = x + \epsilon \,\text{sign}(\nabla_x J(\theta, x, y))$, and BIM's clipped iterative application of the same sign-gradient step, both with $\epsilon = 2.5\%$.
What would settle it
A concrete check would be to measure a faithfulness metric on the same 189 images, such as the drop in predicted probability when each explanation's top pixels are occluded, before and after FGSM and BIM attacks; if faithfulness drops sharply while IoU and RMSE stay flat, the paper's conclusion about metric insensitivity would fail for that stronger notion of explanation quality. A complementary test would use a synthetic dataset with a known decision rule to verify whether SAM masks actually align with the features driving the model.
Extended reading notes
Core claim
The central claim is that explanation fidelity, measured as mask overlap and pixel error against SAM segmentations, is preserved under FGSM and BIM attacks despite a collapse in classification accuracy. On 189 images, IoU shifts by at most about two percentage points and RMSE by about one point across all three explanation methods, while accuracy drops from 89.94% to 58.73% under FGSM and to 45.50% under BIM. The authors interpret this stability as evidence that the combination of these explanation methods with IoU and RMSE metrics is insensitive to adversarial perturbations, and they offer the tabulated results as a foundation for a future benchmark.
Load-bearing premise
The paper treats SAM-generated segmentation masks as the ground-truth explanation of what the model actually uses to decide, without checking that the model's decisions are driven by those regions; if the model relies on background or texture cues outside the mask, the IoU and RMSE stability would only show agreement with a segmentation algorithm, not with the model's reasoning.
Editorial extensions
If this is right
- If explanation masks stay fixed under attack, visual inspection of GradCAM, SmoothGrad, or LIME outputs cannot serve as a reliable detector of adversarial manipulation.
- The small IoU and RMSE deltas provide a numerical baseline for future benchmarks that include more attack methods, more explanation techniques, and more comparison metrics.
- The results imply that a model's prediction and its explanation can decouple under adversarial perturbation: the class flips while the highlighted region stays put.
- Any defense or certification strategy that relies on explanation stability would need to define robustness with respect to a metric that is actually sensitive to adversarial changes.
- Future evaluation of explanation methods should test whether their outputs shift under attack, not just whether they match a segmentation mask.
Reading between the lines
- Because SAM masks are object-level segmentations, the observed stability may simply mean that attacks move the model to a wrong class while still focusing on the same object region; a faithfulness metric that measures how much the predicted probability changes when the highlighted pixels are occluded could reveal attack-induced shifts that IoU and RMSE miss.
- A testable extension would be to compute a rank-correlation or cosine-similarity measure between the original and adversarial explanation maps; such heatmap-level comparisons may be substantially more sensitive to perturbations than overlap against a fixed mask.
- The conclusion that explanation methods are insensitive to adversarial influences could be an artifact of the ground truth: if the model actually relies on background or texture cues outside the SAM mask, then IoU and RMSE measure agreement with a segmentation algorithm rather than with the model's true decision basis.
- This result sits in tension with earlier demonstrations that explanations are fragile under small perturbations; resolving that tension likely requires distinguishing between changes in the explanation map itself and changes in a coarse overlap statistic.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether adversarial attacks (FGSM and BIM) alter the explanations produced by GradCAM, SmoothGrad, and LIME for an EfficientNetV2B0 classifier. Using a manually curated subset of 189 ImageNet images, it compares explanation heatmaps to SAM-generated segmentation masks via IoU and RMSE. The authors report that while model accuracy drops sharply under attack (from 89.94% to 58.73% under FGSM and to 45.50% under BIM), the mean IoU and RMSE change little, and they conclude that these explanation methods combined with IoU and RMSE are not effective at discerning adversarial influences. The paper also acknowledges limitations in dataset size, attack coverage, and generalizability.
Significance. The question is timely and practically relevant, and the paper offers an open repository and a clear pipeline, which are commendable. If the central claim held, it would be a useful caution for practitioners who use IoU and RMSE to evaluate explanation fidelity under adversarial conditions. However, the evidence as presented does not yet establish the claim because of a fundamental ground-truth mismatch for misclassified adversarial images and the absence of statistical support for the 'no significant change' assertion. The contribution is therefore best viewed as a pilot benchmark requiring substantial re-analysis before its main conclusion can be accepted.
major comments (4)
- [Section 3, Section 6, Tables 1 and 2] The central claim that IoU and RMSE are 'not effective in discerning adversarial influences' is not supported because explanations are scored against the true-class SAM mask even when the model predicts a different class. Under FGSM and BIM, accuracy drops to 58.73% and 45.50%, so for 41% and 54% of images, respectively, the model's top prediction is not the true class. GradCAM, SmoothGrad, and LIME explain the predicted class, not the true class, and comparing a wrong-class explanation to a true-object mask measures overlap with the original object rather than fidelity to the model's actual decision. The stable aggregate IoU/RMSE in Tables 1 and 2 could be an artifact of this mismatch, so the conclusion in Section 6 is not warranted. The authors should recompute metrics on correctly classified images only, or use ground-truth masks for the predicted class, and report per-condition distributions.
- [Tables 1 and 2] The paper claims 'no significant changes' in IoU and RMSE based on aggregate mean values (e.g., GradCAM IoU 34.66% vs. 33.67% vs. 32.75%), but it reports no error bars, confidence intervals, or significance tests. With n=189 and likely high per-image variance, these small aggregate differences may be within noise. The authors should report standard deviations, paired significance tests (e.g., Wilcoxon signed-rank or paired t-test) per explanation method, and effect sizes. Without this, the conclusion that the changes are negligible is not statistically supported.
- [Section 3, Figure 4, Section 5] The validity of SAM-generated masks as ground-truth explanations for EfficientNetV2B0's decisions is not established. SAM segmentations delineate objects, but they do not necessarily correspond to the features that drive the classifier, which could include background, texture, or partial-object cues. The manuscript states that masks were manually selected for being 'clear and accurate' but provides no validation of their relevance to the model's reasoning (e.g., human ratings, comparison to alternative ground truths, or ablation). Without such validation, the stability of IoU/RMSE may measure agreement with a segmentation model rather than with the model's true decision basis.
- [Section 4, Section 5] The dataset is a manually curated subset with several undocumented selection steps: classes 41-79 were excluded, grayscale and multi-instance images were discarded, and only images for which SAM produced 'clear and accurate masks' were retained, yielding only 189 images. The number of images excluded at each step and the criteria for 'clear and accurate' are not reported. Since the stated goal includes establishing a benchmark, these selection rules need transparent documentation and a sensitivity analysis (e.g., reports on the excluded classes or on a random subset of ImageNet). The acknowledged limitation of small data is good, but the selection procedure itself can bias the benchmark.
minor comments (6)
- [Figures 8 and 9] The captions contain a typo: 'Avderarial' should be 'Adversarial'.
- [Section 2] The citation 'Bach et al., 2025' appears to be an error; the LRP paper was published in 2015 (PLoS ONE, 10(7), e0130140).
- [Section 2] DeepLIFT is attributed to Li et al. (2021), but the original DeepLIFT method is by Shrikumar et al. (2017); the cited Li et al. paper is a different work with the same acronym.
- [Section 3] The RMSE equation is typeset confusingly (the limits and the square root are malformed); a cleaner typesetting such as RMSE = sqrt((1/(n^2)) * sum_i sum_j (Y_ij - Yhat_ij)^2) would improve readability.
- [Section 5, Figure 6] The explanation masks are generated by selecting the 'top 15% of pixels,' but this threshold is introduced without justification or sensitivity analysis; a brief comment on how the choice of threshold could affect IoU/RMSE would strengthen the presentation.
- [Section 1] The paper structure described in the introduction says Section 5 concludes the study, but the conclusion is actually in Section 6; please correct the roadmap.
Circularity Check
No significant circularity: the study reports empirical measurements against an external SAM-derived ground truth; validity concerns about the reference do not reduce the conclusion to its inputs.
full rationale
The paper's central claim is an empirical measurement rather than a derived prediction. It defines ground-truth explanation masks using the Segment Anything Model, computes IoU and RMSE between GradCAM/SmoothGrad/LIME outputs and these masks for original and adversarial images, and observes that the metric values remain roughly constant while model accuracy drops. No parameter is fitted to a subset and then relabeled as a prediction; no equation in the paper forces the IoU/RMSE stability by construction; and no load-bearing self-citation or author-imported uniqueness theorem is invoked. The main weakness is construct validity: the SAM mask corresponds to the true object/class in the original image, whereas the explanation methods explain the model's predicted class, which is often wrong after FGSM/BIM attacks. Scoring a wrong-class explanation against the true-class mask may therefore produce stable aggregate metrics that do not support the conclusion that the metrics are insensitive to adversarial influence. This is a substantive methodological limitation rather than a circular derivation, and the authors do acknowledge related limitations in the conclusion, including the small dataset and restricted range of attacks. Since the claimed result is not equivalent by definition to its input and no specific reduction can be exhibited, the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- FGSM/BIM epsilon =
2.5%
- BIM iterations =
10
- Explanation threshold =
top 15% of pixels
assumptions (4)
- domain assumption SAM segmentation masks can serve as ground-truth explanations for model predictions
- domain assumption IoU and RMSE are appropriate metrics for explanation quality
- domain assumption Thresholding each explanation method at its top 15% of pixels produces comparable binary masks
- domain assumption The ImageNet subset is representative enough for benchmarking adversarial-explainability interactions
Cite this review
Pith. "Pith review of Impact of Adversarial Attacks on Deep Learning Model Explainability." pith.science (2026). https://pith.science/paper/G6Q6STL3
@misc{pith2026241211119,
author = {Pith},
title = {Pith review of: Impact of Adversarial Attacks on Deep Learning Model Explainability},
year = {2026},
howpublished = {\url{https://pith.science/paper/G6Q6STL3}},
note = {Machine review of arXiv:2412.11119}
}
read the original abstract
In this paper, we investigate the impact of adversarial attacks on the explainability of deep learning models, which are commonly criticized for their black-box nature despite their capacity for autonomous feature extraction. This black-box nature can affect the perceived trustworthiness of these models. To address this, explainability techniques such as GradCAM, SmoothGrad, and LIME have been developed to clarify model decision-making processes. Our research focuses on the robustness of these explanations when models are subjected to adversarial attacks, specifically those involving subtle image perturbations that are imperceptible to humans but can significantly mislead models. For this, we utilize attack methods like the Fast Gradient Sign Method (FGSM) and the Basic Iterative Method (BIM) and observe their effects on model accuracy and explanations. The results reveal a substantial decline in model accuracy, with accuracies dropping from 89.94% to 58.73% and 45.50% under FGSM and BIM attacks, respectively. Despite these declines in accuracy, the explanation of the models measured by metrics such as Intersection over Union (IoU) and Root Mean Square Error (RMSE) shows negligible changes, suggesting that these metrics may not be sensitive enough to detect the presence of adversarial perturbations.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., ... & Zheng, X. (2016). Tensorflow: Large -scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467. Agarwal, C., Krishna, S., Saxena, E., Pawelczyk, M., Johnson, N., Puri, I., Marinka, Z., & Lakkaraju, H. (2022). Openxai: Towards a transparent evaluat...
arXiv 2016
-
[3]
Ghorbani, A., Abid, A., & Zou, J. (2019, July). Interpretation of neural networks is fragile. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 3681- 3688). Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Hassija, V., Chamola, V., Maha...
arXiv 2014
-
[4]
https://github.com/SinaMohseni/ML-Interpretability-Evaluation-Benchmark Keras. (n.d.). Keras Documentation: Keras applications. https://keras.io/api/applications/ Khattar, A., & Quadri, S. M. K. (2022). Generalization of convolutional network to domain adaptation network for classification of disaster images on twitter. Multimedia Tools and Applications, ...
work page 2022
-
[30]
Viering, T., Wang, Z., Loog, M., & Eisemann, E. (2019). How to manipulate cnns to make them lie: the gradcam case. arXiv preprint arXiv:1907.10901. Wicker, M., Heo, J., Costabello, L., & Weller, A. (2022). Robust explanation constraints for neural networks. arXiv preprint arXiv:2212.08507. Xie, C., Wang, J., Zhang, Z., Ren, Z., & Yuille, A. (2017). Mitiga...
arXiv 2019
-
[32]
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei -Fei, L. (2009, June). Imagenet: A large - scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255). IEEE. Dimanov, B., Bhatt, U., Jamnik, M., & Weller, A. (2020). You shouldn’t trust me: Learning models which conceal unfairness from multipl...
work page 2020
-
[7741]
Liu, Y., Khandagale, S., White, C., & Neiswanger, W. (2021). Synthetic benchmarks for scientific research in explainable machine learning. arXiv preprint arXiv:2106.12543. Lopardo, G., Precioso, F., & Garreau, D. (2024). Attention Meets Post -hoc Interpretability: A Mathematical Perspective. arXiv preprint arXiv:2402.03485. Lundberg, S. M., Allen, P. G., ...
arXiv 2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.