REVIEW 4 major objections 6 minor 36 references
TriGuard: Testing Model Safety with Attribution Entropy, Verification, and Drift
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Attribution drift exposes a failure mode that adversarial accuracy misses in neural networks.
desk verdict Composition of existing safety checks plus an unnormalized, unvalidated drift metric; the internal numbers don't line up, so the headline claims don't yet hold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
TriGuard's load-bearing piece is the Attribution Drift Score (ADS), the L2 distance between two Integrated Gradients attribution vectors computed from different baselines (zero versus blurred versions of the input), which the paper treats as a proxy for explanation stability. The other two axes are formal verification via PGD search plus interval bound propagation (with CROWN-IBP as a certified baseline), and attribution entropy, the Shannon entropy of the normalized absolute Integrated Gradients. The training-side mechanism is an entropy-regularized loss, $L_{\text{total}} = L_{\text{CE}} + \lambda \cdot L_{\text{entropy}}$, where $L_{\text{entropy}}$ penalizes the entropy of the normalized input gradient, encouraging sparse and stable saliency.
What would settle it
Compute ADS for a model with provably stable explanations—for example, a linear classifier on MNIST, where the attribution is exactly the weight vector and does not change under input perturbation—and show that the zero-versus-blurred baseline yields a nonzero drift that varies with the input; that would indicate the metric measures baseline sensitivity rather than reasoning instability. Alternatively, a controlled experiment where a model's decision rule is switched from one feature to another (e.g., a synthetic dataset) should show a spike in ADS at the switch; if it does not, the metric fails as a stability test.
Extended reading notes
Core claim
The central claim is that attribution-based signals are complementary to adversarial accuracy and that verified models can still exhibit unstable reasoning. The paper defines the Attribution Drift Score (ADS) as the L2 distance between Integrated Gradients maps computed from a zero baseline and a blurred baseline, and observes that this drift correlates only weakly with attribution entropy (r=-0.22) and not at all with adversarial error (r=0.00) across 18 model-dataset combinations. It also demonstrates that entropy-regularized training, which penalizes the entropy of the normalized input-gradient distribution, consistently reduces drift while maintaining comparable accuracy; for example, SimpleCNN on MNIST sees drift drop from 16.64 to 1.73 with λ=0.10. These results are offered as evidence that TriGuard uncovers 'silent brittleness'—models that look safe by standard metrics but shift their explanation logic.
Load-bearing premise
The Attribution Drift Score assumes that the L2 distance between attribution maps under two arbitrary baselines is a meaningful measure of explanation stability, and the paper never validates this metric against a known ground-truth change in a model's reasoning.
Editorial extensions
If this is right
- Models that pass formal verification can still exhibit attribution drift, so certified robustness alone does not guarantee stable reasoning.
- A low or zero adversarial error does not imply explanation stability, as shown by ResNet50 on CIFAR-10 and DenseNet121 on MNIST.
- Adding a gradient-entropy penalty during training reduces Attribution Drift Scores without a meaningful accuracy drop.
- Attribution entropy and drift provide orthogonal safety signals relative to adversarial metrics, since drift and adversarial error are uncorrelated.
- TriGuard's three-axis evaluation can be applied across architectures (SimpleCNN, ResNet, MobileNet, DenseNet) and datasets to rank models by safety rather than accuracy alone.
Reading between the lines
- The same contrastive-drift idea could be applied to other attribution methods (SmoothGrad, LIME, attention maps) to build a model-agnostic stability diagnostic.
- If ADS is validated against known reasoning changes—for example, a model that provably relies on a spurious feature—it could become a standard audit metric for regulators; the paper does not yet provide such validation.
- A testable extension is to use ADS as a training regularizer itself, not just an evaluation metric, or to combine it with adversarial training to see whether drift and robustness improve jointly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. TriGuard proposes a multi-axis safety evaluation framework that combines formal robustness verification (PGD plus a simplified interval bound propagation check), attribution entropy over normalized Integrated Gradients, and a new Attribution Drift Score (ADS) defined as the L2 distance between Integrated Gradients maps computed under two different baselines. The paper claims that verified models can still exhibit unstable reasoning, that ADS and entropy provide safety insights complementary to adversarial accuracy, and that entropy-regularized training reduces ADS without sacrificing accuracy. Experiments are reported for five architectures on MNIST, FashionMNIST, and CIFAR-10, and the code is released.
Significance. If the central claims were sound, TriGuard would be a useful step toward routine evaluation of explanation stability alongside robustness, and the paper addresses an important gap by attempting to combine verification, attribution concentration, and attribution stability. The release of code is a practical strength. However, the ADS metric is unnormalized and never validated against existing stability benchmarks, the formal verification claims are not backed by a sound certificate, and the internal results are inconsistent. Because the reported correlations and the regularization benefit both pass through ADS, the empirical contribution is not currently substantiated.
major comments (4)
- [Section 2.4] The Attribution Drift Score is defined as ADS = ||a(1) − a(2)||_2, an unnormalized L2 distance between Integrated Gradients maps under zero and blurred baselines. This conflates attribution magnitude with stability: for an approximately linear classifier, IG_i ≈ (x_i − b_i) g_i, so ADS ≈ ||(b(2) − b(1)) ⊙ g||_2, which scales with baseline difference, gradient magnitude, and input dimension. A training intervention that shrinks gradient magnitudes will therefore reduce ADS even if the relative saliency pattern is unchanged. The paper does not normalize ADS, does not compare it with existing stability measures (e.g., Ghorbani et al. 2019; Alvarez-Melis & Jaakkola 2018), and reports only one-model baseline sensitivity in Appendix A. Since the claims of orthogonality with adversarial error (§3.1) and the regularization benefit (§3.2) are both measured with ADS, the central empirical findings may be scale artifacts rather than safety signals.
- [Section 3.2 / Tables 3 and 4] The unregularized SimpleCNN/MNIST configuration is reported with drift 2.93 in Table 3 but with drift 16.64 in Table 4 (λ=0.00) and in the text of §3.2 and §4. Table 1 lists drift 2.48 for the same regularized model, and the text pairs the 16.64-to-1.73 drop with Table 4. The same configuration therefore receives different ADS values, and the claimed reduction from 16.64 to 1.73 is not consistent with the Table 3 value. Without a fixed evaluation protocol (baseline pair, normalization, random seed), the ADS values are not reproducible, which undermines the central regularization result.
- [Section 2.5 / Section 3.2 / Table 4] The entropy penalty in §2.5 is defined as the negative entropy of normalized input gradients, which should favor sparse, low-entropy gradient distributions. Yet the paper reports that entropy-regularized training increases attribution entropy (e.g., SimpleCNN/MNIST from 4.47 to 5.13 or 5.29 in Table 4) and produces 'more dispersed' saliency maps. This contradicts the stated mechanism that the regularizer 'encourages sparser and more localized attribution maps' and the Abstract's claim that it 'reduces explanation drift without sacrificing performance.' The authors need to reconcile the mechanism with the observed entropy increase or correct the description and experiments.
- [Section 2.2 / Table 1] The paper repeatedly refers to models as 'formally verified' or 'passing formal verification' (Abstract, §1, §3), but the verification axis in §2.2 is a simplified check that the predicted class dominates across interval bounds, without specifying how bounds are propagated through nonlinear layers, and the PGD search is heuristic. No soundness proof or completeness claim is given, and Table 1 shows that the simplified check is far more permissive than CROWN-IBP (e.g., DenseNet121 on MNIST passes FormaVerif but fails CROWN-IBP). The central observation that 'verified models can still exhibit unstable reasoning' is therefore unsupported, because the check does not constitute a rigorous certificate.
minor comments (6)
- [Section 2.7] The text states that TriGuard is evaluated across '18 model-dataset combinations,' but Table 1 reports 15 rows (5 models × 3 datasets); if the count includes additional runs or configurations, please clarify.
- [Section 1] There are grammatical errors in the use of 'axes' as a singular noun, e.g., 'Each axes offers partial insight' and 'each axes'; these should be 'axis.'
- [Table 3] Table 3 omits the CROWN-IBP column that appears in Table 1, so the reader cannot compare the certified robustness of unregularized and regularized models.
- [Section 2.5] The symbol ϵ is used both for the perturbation bound in §2.2 and for the numerical offset in the entropy formula in §2.5; please use distinct symbols to avoid ambiguity.
- [Section 2.6 / Table 1] The SmoothGrad2 column in Table 1 is not clearly defined: it is unclear whether the reported number is an entropy value, a drift score, or a distance; the definition in §2.6 should be linked to the reported metric.
- [Appendix B.2] Figure 6 is referenced in the Deletion/Insertion faithfulness evaluation but does not appear in the manuscript; all referenced figures should be included.
Circularity Check
No significant circularity: TriGuard's empirical claims do not reduce by construction to its metric definitions, though the Attribution Drift Score lacks external validation.
full rationale
The paper's central claims are empirical rather than derivational. ADS is defined in Section 2.4 as the L2 norm of the difference between two Integrated Gradients maps computed from zero and blurred baselines. The finding that entropy-regularized training reduces ADS (Section 3.2 and Table 4) is not forced by the definition of either ADS or the regularizer. The entropy regularizer (Section 2.5) penalizes the entropy of the normalized input-gradient distribution; it does not directly minimize the baseline-dependent L2 distance, nor does it reference the zero or blurred baselines. Whether this training change reduces ADS is an empirical observation, not an equation-level tautology. Likewise, the reported correlations among entropy, drift, and adversarial error (Section 3.1) are computed from measured quantities, not derived from the metric definitions. The paper also includes external faithfulness benchmarks via Deletion/Insertion AUC (Appendix B.2), providing an independent anchor for attribution quality. There are no load-bearing self-citations; all cited works are external. The main weaknesses are validity and consistency concerns rather than circularity: ADS is unnormalized and no external benchmark validates it as a stability measure, and there is a numerical inconsistency (Table 3 lists SimpleCNN/MNIST unregularized drift as 2.93, while Table 4 and the text report 16.64 for the same configuration). These issues undermine confidence in the conclusions, but they do not make the derivation circular. No step in the paper reduces to its own input by construction, so the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (4)
- Entropy regularization coefficient lambda =
0.00, 0.01, 0.05, 0.10
- Perturbation bound epsilon =
0.1 (MNIST/FashionMNIST), 0.3 (CIFAR-10)
- PGD step count T and step size alpha =
unspecified
- IG integration steps m and numerical offset delta =
unspecified
assumptions (4)
- domain assumption Integrated Gradients provides faithful attributions for the tested models.
- ad hoc to paper The simplified IBP-style check (class dominance across input bounds) is a sound formal robustness certificate.
- domain assumption Zero and blurred baselines are semantically meaningful references for assessing explanation stability.
- ad hoc to paper Minimizing the entropy penalty in L_total encourages sparser attribution maps.
Cite this review
Pith. "Pith review of TriGuard: Testing Model Safety with Attribution Entropy, Verification, and Drift." pith.science (2026). https://pith.science/paper/OPNZ5Y2J
@misc{pith2026250614217,
author = {Pith},
title = {Pith review of: TriGuard: Testing Model Safety with Attribution Entropy, Verification, and Drift},
year = {2026},
howpublished = {\url{https://pith.science/paper/OPNZ5Y2J}},
note = {Machine review of arXiv:2506.14217}
}
read the original abstract
Deep neural networks often achieve high accuracy, but ensuring their reliability under adversarial and distributional shifts remains a pressing challenge. We propose TriGuard, a unified safety evaluation framework that combines (1) formal robustness verification, (2) attribution entropy to quantify saliency concentration, and (3) a novel Attribution Drift Score measuring explanation stability. TriGuard reveals critical mismatches between model accuracy and interpretability: verified models can still exhibit unstable reasoning, and attribution-based signals provide complementary safety insights beyond adversarial accuracy. Extensive experiments across three datasets and five architectures show how TriGuard uncovers subtle fragilities in neural reasoning. We further demonstrate that entropy-regularized training reduces explanation drift without sacrificing performance. TriGuard advances the frontier in robust, interpretable model evaluation.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Alvarez-Melis, D. and Jaakkola, T. S. On the robustness of interpretability methods. In Advances in Neural Information Processing Systems (NeurIPS), 2018. URL https://arxiv.org/abs/1806.08049
arXiv 2018
-
[2]
A Multi-Policy Framework for Deep Learning-Based Fake News Detection
Bastani, O., Lee, H., and Raghunathan, A. Attribution under distribution shift: Quantifying the drift of explanation. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2206.11866
work page Pith review arXiv 2023
-
[3]
Transformer interpretability beyond attention visualization
Chefer, H., Gur, S., and Wolf, L. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 782--791, 2021
work page 2021
-
[4]
Chen, Z., Wang, J., Song, Y., Zhang, X., Liu, Y., and Zhou, Y. When are saliency maps trustworthy? In International Conference on Learning Representations (ICLR), 2023
work page 2023
-
[5]
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020
work page 2020
-
[6]
Training verified learners with learned verifiers
Dvijotham, K., Stanforth, R., Gowal, S., Uesato, J., and Kohli, P. Training verified learners with learned verifiers. In Advances in Neural Information Processing Systems (NeurIPS), 2018. URL https://arxiv.org/abs/1805.10265
arXiv 2018
-
[7]
Fong, R. C. and Vedaldi, A. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pp.\ 3429--3437, 2017
work page 2017
-
[8]
AI2 : Safety and robustness certification of neural networks with abstract interpretation
Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., and Vechev, M. AI2 : Safety and robustness certification of neural networks with abstract interpretation. In IEEE Symposium on Security and Privacy, 2018
work page 2018
Show all 36 references
-
[9]
Ghorbani, A., Abid, A., and Zou, J. Y. Interpretation of neural networks is fragile. In AAAI, 2019
2019
-
[10]
J., Shlens, J., and Szegedy, C
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015. URL https://arxiv.org/abs/1412.6572
2015 arXiv
-
[11]
Improving robustness without sacrificing accuracy via learned data augmentation
Gowal, S., Qin, C., Uesato, J., Mann, T., and Kohli, P. Improving robustness without sacrificing accuracy via learned data augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021
2021
-
[12]
The many faces of robustness: A critical analysis of out-of-distribution generalization
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICLR, 2021
2021
-
[13]
A benchmark for interpretability methods in deep neural networks
Hooker, S., Erhan, D., Kindermans, P.-J., and Kim, B. A benchmark for interpretability methods in deep neural networks. In NeurIPS, 2020
2020
-
[14]
Benchmark for evaluating saliency methods
Hooker, S., Erhan, D., Kindermans, P.-J., and Kim, B. Benchmark for evaluating saliency methods. In Advances in Neural Information Processing Systems, 2021
2021
-
[15]
The complete verification of neural networks: the eran approach
Katz, G., Wang, W., Shiraishi, S., et al. The complete verification of neural networks: the eran approach. In CAV, 2021
2021
-
[16]
Sanity simulations for saliency methods
Kim, J., Ghorbani, A., and Zou, J. Sanity simulations for saliency methods. In International Conference on Learning Representations (ICLR), 2022. URL https://arxiv.org/abs/2105.06506v3
2022 arXiv
-
[17]
T., Dähne, S., and Erhan, D
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., and Erhan, D. The reliability of saliency methods. In NeurIPS, 2019
2019
-
[18]
Towards deep learning models resistant to adversarial attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1706.06083
2018 arXiv
-
[19]
Causal tracing for understanding unsafe outputs in large language models
Meng, K., Andonian, A., Bau, D., and Belinkov, Y. Causal tracing for understanding unsafe outputs in large language models. arXiv preprint arXiv:2305.13928, 2023
2023 arXiv
-
[20]
Centered kernel alignment losses for saliency method faithfulness
Ramaswamy, V., Moayeri, M., and Lakkaraju, H. Centered kernel alignment losses for saliency method faithfulness. In ICLR, 2022
2022
-
[21]
Ross, A. S. and Doshi-Velez, F. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In AAAI Conference on Artificial Intelligence, 2018. URL https://arxiv.org/abs/1711.09404
2018 arXiv
-
[22]
Evaluating the visualization of what a deep neural network has learned
Samek, W., Montavon, G., Lapuschkin, S., Anders, C., and Müller, K.-R. Evaluating the visualization of what a deep neural network has learned. In IEEE Transactions on Neural Networks and Learning Systems, volume 28, pp.\ 2660--2673, 2017
2017
-
[23]
Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models
Samek, W., Montavon, G., Lapuschkin, S., Anders, C., and Müller, K.-R. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. Proceedings of the IEEE, 110 0 (5): 0 680--707, 2022
2022
-
[24]
Deep inside convolutional networks: Visualising image classification models and saliency maps
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop, 2014
2014
-
[25]
An abstract domain for certifying neural networks
Singh, G., Gehr, T., Mirman, M., Püschel, M., and Vechev, M. An abstract domain for certifying neural networks. In POPL, 2019
2019
-
[26]
and Feizi, S
Singla, S. and Feizi, S. Understanding adversarial robustness with visualizations of loss landscapes. In Advances in Neural Information Processing Systems, volume 32, 2019
2019
-
[27]
Smoothgrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. Smoothgrad: removing noise by adding noise. In arXiv preprint arXiv:1706.03825, 2017
2017 arXiv
-
[28]
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribution for deep networks. In ICML, 2017
2017
-
[29]
Z., Lin, C.-J., and Hsieh, C.-J
Xu, K., Zhang, H., Lin, S., Zhao, S., Wang, X., Xu, M., Kolter, J. Z., Lin, C.-J., and Hsieh, C.-J. Fast and complete neural network verification by parallel bound propagation. In Advances in Neural Information Processing Systems, 2021 a
2021
-
[30]
Xu, K., Zhang, H., Wang, S., Lin, C.-J., and Kolter, J. Z. Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. In International Conference on Learning Representations (ICLR), 2021 b
2021
-
[31]
Xu, M., Zhang, R., Lin, J., Wang, Y., and Wang, X. H. Reliable attribution of adversarial training. In NeurIPS, 2020
2020
-
[32]
On the faithfulness and reliability of saliency explanations
Yeh, C.-K., Kim, B., Arik, S., Li, C.-L., and Pfister, T. On the faithfulness and reliability of saliency explanations. Transactions on Machine Learning Research (TMLR), 2023. Accepted with minor revisions
2023
-
[33]
Efficient neural network verification with Auto-LiRPA : Towards scalable certified defense
Zhang, H., Weng, T.-W., Yi, J., Lin, C.-J., and Daniel, L. Efficient neural network verification with Auto-LiRPA : Towards scalable certified defense. In Advances in Neural Information Processing Systems (NeurIPS), 2020
2020
-
[34]
Z., Wang, S., and Weng, T.-W
Zhang, H., Xu, K., Lin, X., Kolter, J. Z., Wang, S., and Weng, T.-W. Towards stable and efficient training of verifiably robust neural networks. In International Conference on Learning Representations (ICLR), 2021. URL https://arxiv.org/abs/1906.06316
2021 arXiv
-
[35]
Towards certified robustness of real-world neural networks
Zhang, H., Weng, T.-W., Xu, H., Lin, C.-J., and Hsieh, C.-J. Towards certified robustness of real-world neural networks. In AAAI, 2022
2022
-
[36]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.