REVIEW 4 major objections 5 minor 19 references
Sampling Matters in Explanations: Towards Trustworthy Attribution Analysis Building Block in Visual Models through Maximizing Explanation Certainty
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper argues that the trustworthiness of gradient-integration explanations is governed by how closely the sampling distribution matches natural images, and proposes Bernoulli feature suppression as a practical way to keep samples…
desk verdict Pixel-dropping sampling beats SmoothGrad and IG empirically, but the paper's theoretical justification doesn't hold and the evaluation is circular. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the explanation-certainty lower bound $\Pr(x|z) \ge \exp(I(z;x))/\exp(H(x)) = \exp(-H(x|z))$, obtained by rewriting the log-conditional density as a mutual-information integral minus the input entropy. This bound turns the abstract goal of trustworthy explanations into a concrete optimization target: choose the sampling distribution $q$ in gradient integration so that the mutual information between the input and the resulting explanation is as large as possible. The paper then proposes Bernoulli feature suppression, $x^*_i = B(\{x_i,0\}; 1-p)$ with $p\approx 0.7$, as the practical mechanism, and supports it with a cosine-kernel PCA alignment experiment and an activation/mutual-information sweep over $p$.
What would settle it
A direct test: on a model fine-tuned with the Gaussian-noise and luminance augmentations used in the paper's Figure 2, compute mutual information $I(z;x)$ for suppression-based, SmoothGrad, and integrated-gradient explanations; the paper's robustness claim predicts suppression still yields the highest value, so if any baseline matches or exceeds it, the central empirical claim fails.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is a recipe for sampling in gradient integration: draw samples by dropping pixels according to independent Bernoulli trials with drop probability near $0.7$, rather than by adding noise or by global intensity scaling. The paper's theoretical route asserts an inequality chain in which the log-conditional density of inputs given explanations is lower bounded by $I(z;x) - H(x)$, yielding $\Pr(x|z) \ge \exp(-H(x|z))$. It then uses convexity of mutual information to argue that the sampling distribution maximizing expected explanation certainty is the natural image distribution $q^*=p$, and presents feature suppression as a practical approximation to that distribution. Empirically, the paper measures the mutual-information lower bound on ImageNet across AlexNet, VGG16, DenseNet121, ShuffleNet V2 x0.5, ResNet18, ResNet50, Wide ResNet50-2, and ViT-B/16, and reports higher $I(z;x)$ than SmoothGrad and integrated gradients on every model, with network activation also peaking for drop probabilities in $[0.6,0.8]$.
Load-bearing premise
The central bet is that explanation certainty really is lower-bounded by $\exp(I(z;x))/\exp(H(x))$ and that zeroing each pixel independently with probability $0.7$ produces samples statistically close enough to natural images for that bound to be the right guide; if either is wrong, the recommended sampling rule may not be optimal.
Editorial extensions
If this is right
- Attribution methods that add noise or linearly scale inputs should be expected to underperform methods that keep samples close to the natural image distribution, because their samples reduce the mutual-information lower bound.
- For models trained with data augmentation, noise-based sampling can be particularly fragile: the paper shows that fine-tuning with added Gaussian noise and luminance changes degrades SmoothGrad and integrated-gradient explanations, whereas suppression-based sampling remains robust.
- Feature suppression with $p\approx 0.7$ can serve as a drop-in replacement for gradient integration in composite explainability pipelines such as Grad-CAM-style methods.
- The optimal drop probability $[0.6,0.8]$ should transfer across architectures on ImageNet, since the paper reports a consistent activation and mutual-information pattern on all tested models.
Reading between the lines
- The paper's mutual-information framing suggests a broader design principle: any attribution method that averages over perturbations should match the perturbation distribution to the data distribution rather than choosing it for convenience; this would extend to latent-space and text explanations, which the paper does not test.
- A rigorous replacement for the paper's integral-manipulation step may yield a bound involving KL divergence rather than differential entropy; if so, the same feature-suppression method could be audited by estimating that divergence directly on held-out data, giving a quantitative faithfulness test that does not depend on the questionable Hölder step.
- The optimal drop probability near $0.7$ likely reflects the redundancy of ImageNet-scale natural images; on less redundant domains, such as medical or satellite imagery, the peak-activation curve in Figure 4(b) could shift, making per-domain calibration of $p$ a natural next experiment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the choice of sampling distribution in gradient-integration attribution methods for image classifiers. It introduces a notion of "explanation certainty" Pr(x|z), claims to lower-bound it by exp(-H(x|z)) using a step labeled Hölder's inequality, and concludes that the optimal sampling distribution for gradient integration is the natural image distribution p. The proposed method, Bernoulli feature suppression, drops each pixel with probability p and is claimed to produce samples approximately distributed like natural images. The authors report mutual information I(z;x) between inputs and explanations on several ImageNet models, claiming improvements over SmoothGrad, Integrated Gradients, and vanilla gradients.
Significance. If the theoretical claims were correct, the proposed sampling scheme would be a simple and potentially useful building block for gradient-based attribution, and the paper's breadth of evaluation across eight architectures would strengthen the case. The reproducible code link and the empirical observation that moderate pixel dropping can increase gradient norms are also positive features. However, the central lower-bound derivation is mathematically invalid, the optimality result does not follow from the stated arguments, and the main quantitative comparison uses the evaluation metric to select the method's only hyperparameter. As presented, the paper does not establish its central claims, and the empirical results are not an independent test of the method.
major comments (4)
- [Sec. 3.2, Eqs. (5)-(6)] The step labeled Hölder's inequality is not a valid application of Hölder. Hölder's inequality bounds the integral of a product by products of norms; here the paper multiplies the unweighted integral ∫∫ log p(x|z) dx dz by ∫∫ p(x,z) dx dz = 1 and then asserts that the result is at least the weighted integral ∫∫ p(x,z) log(p(x|z)/(p(x)p(x))) dx dz. No inequality of this form follows from Hölder without additional pointwise or norm conditions, and the claimed inequality is generally false. Consequently, Eq. (6) does not establish a lower bound on explanation certainty. In addition, Eq. (1) treats the continuous conditional density p(x|z) as a probability, so the quantity exp(-H(x|z)) is not a meaningful lower bound on a probability: differential entropy can be negative and exp(-H(x|z)) can exceed one. The theoretical foundation of the paper is therefore not sound.
- [Sec. 3.3-3.4, Eqs. (7)-(10)] The derivation of the optimal sampling distribution q* = p is also invalid. In Eq. (7), the explanation z is defined as the expectation E_{xhat~qx}[|∇ f|], which for a fixed input x is a deterministic function of x; the mutual information I(x;z) in Eq. (8) is then identically zero unless additional randomness from the sampling procedure is retained in z, contradicting the notation. Furthermore, Eq. (9) applies Jensen's inequality to I(x;z) as though mutual information were convex in the explanation vector z for fixed x; mutual information is convex in conditional distributions p(z|x), not in the realized values z, and the inequality I(x; E[Z]) ≤ E[I(x;Z)] is not an information-theoretic identity. Even if the Jensen step were accepted, Eq. (10) merely restates an argmax over qx and gives no argument that the natural image distribution p is the maximizer. Thus the theoretical claim that q* = p is unestablished.
- [Sec. 5.2, Figs. 4(b) and 5] The quantitative evaluation is circular. The pixel-dropping probability p is selected in Fig. 4(b) by maximizing the mutual information I(z;x) on a ResNet50, and the headline results in Fig. 5 report exactly the same metric I(z;x) on all tested models. This means the method's only free parameter is fitted to the evaluation objective, so the reported improvements over SmoothGrad and Integrated Gradients do not constitute an independent validation of the method. To support the claim, the authors should fix p on a validation set or a separate model and evaluate on held-out images and architectures, or report sensitivity of the ranking to p.
- [Sec. 4.1-4.2, Fig. 4(a)] The claim that Bernoulli feature suppression produces samples "approximately identical to the distribution of natural images" is supported only by a qualitative PCA projection of penultimate-layer embeddings in Fig. 4(a). This projection is not a direct measurement of the image distribution, and masked images containing exact zeros are not obviously natural images. Since the theoretical optimality argument is built on sample distribution alignment, a quantitative distribution-distance evaluation (e.g., FID, MMD, or a density estimate) is required before this premise can support the method. As it stands, the alignment claim is not established.
minor comments (5)
- [Sec. 3.2, Eq. (1)] The notation in Eq. (1) is dimensionally inconsistent: Pr(x|z) is written as a product over all x and z and then as an integral over x and z, but the relationship between the discrete product and the continuous integral is not defined.
- [Sec. 3.1] The variable y is described as "ground-truth category y ∈ R"; a class label is not naturally a real number, and the notation should be clarified.
- [Sec. 4.1] The Bernoulli notation x*_i = B({x_i,0}; 1-p) is nonstandard; the authors should define explicitly whether p is the probability of setting the pixel to zero or the probability of keeping it, and how the Bernoulli trial is applied independently across pixels.
- [Related work] The LIME citation is given as Mishra et al., 2017, which does not appear to be the original LIME paper; the authors should cite the correct source (Ribeiro et al.) or justify the alternative attribution.
- [Fig. 4(a)] The PCA figure would benefit from a legend that maps colors and markers to the specific sampling settings, and from a quantitative measure of cluster separation, since visual inspection of projections is not conclusive.
Circularity Check
Pixel-dropping probability p is tuned on the mutual-information metric that later serves as the evaluation, making the headline I(z;x) improvement partially self-fulfilling.
-
fitted input called prediction
[Section 4.2.1 'Optimal p' (Fig. 4b) and Section 5.2 'Quantitative evaluation' (Fig. 5)]
"We empirically investigate the optimal dropping probability p in Figure 4(b) by: (1) Measuring how networks activate with respect to various p and (2) measuring how explanation certainty (mutual information) changes with respect to various p. ... The result shows that when p falls in [0.6, 0.8] the network activation will be maximized and the mutual information will be maximized as well. ... We also conduct an extensive quantitative evaluation by measuring the lower bound (mutual information) of explanation certainty. ..."
The method's only substantive hyperparameter p is selected in Section 4.2.1 by directly maximizing the same mutual-information metric I(z;x) that Section 5.2 and Figure 5 use as the evidence of success. Once p is fixed to the value that maximizes I(z;x) on ResNet50 (and checked on other architectures by the same criterion), reporting that the method achieves higher I(z;x) than SmoothGrad and IG is not an independent test of the central claim. The headline quantitative result is substantially built into the tuning step: the fitted p is renamed a 'semi-optimal sampling approach' and the same optimized metric is then presented as a predictive improvement.
full rationale
The formal derivation in Eqs. (5)-(6) is not circular in the sense of equivalence-by-construction: it attempts to prove a lower bound on explanation certainty from definitions, even though the step labeled Holder's inequality is mathematically invalid and the continuous-density bound exp(-H(x|z)) is not meaningful as a probability. Those are correctness concerns, not circularity. Similarly, the jump to the optimal sampling result q*=p in Eq. (10) is asserted rather than proven, but an unsupported step is a logical gap, not a circular reduction. I find no load-bearing self-citations, and the PCA-based claim that Bernoulli masking yields approximately natural-image samples is an empirical assertion, not a definitional identity. The genuine circular element is the empirical evaluation loop: the pixel-dropping probability p, the core parameter of the proposed sampling distribution, is tuned in Section 4.2.1 by maximizing the very mutual-information quantity that Section 5.2 uses as the benchmark. Consequently the central quantitative claim that the method outperforms baselines on I(z;x) is partially forced by the fitting procedure. Score 6 reflects this partial circularity of the paper's headline quantitative claim; the qualitative results and cross-model consistency prevent a higher score.
Assumptions & free parameters
free parameters (1)
- pixel dropping probability p =
0.7 (optimal range 0.6-0.8)
assumptions (4)
- domain assumption Natural images are pixel-wise i.i.d.
- ad hoc to paper The inequality in Eq. (5) holds: the unweighted integral of log p(x|z) is at least the weighted integral of p(x,z) log(p(x|z)/(p(x)p(x))).
- ad hoc to paper Mutual information I(x; z) is convex in z for fixed x, so Jensen's inequality applies to z = E[|∇ f|].
- domain assumption Bernoulli pixel masking produces samples approximately distributed like natural images.
Cite this review
Pith. "Pith review of Sampling Matters in Explanations: Towards Trustworthy Attribution Analysis Building Block in Visual Models through Maximizing Explanation Certainty." pith.science (2026). https://pith.science/paper/TM6D3LUT
@misc{pith2026250619442,
author = {Pith},
title = {Pith review of: Sampling Matters in Explanations: Towards Trustworthy Attribution Analysis Building Block in Visual Models through Maximizing Explanation Certainty},
year = {2026},
howpublished = {\url{https://pith.science/paper/TM6D3LUT}},
note = {Machine review of arXiv:2506.19442}
}
read the original abstract
Image attribution analysis seeks to highlight the feature representations learned by visual models such that the highlighted feature maps can reflect the pixel-wise importance of inputs. Gradient integration is a building block in the attribution analysis by integrating the gradients from multiple derived samples to highlight the semantic features relevant to inferences. Such a building block often combines with other information from visual models such as activation or attention maps to form ultimate explanations. Yet, our theoretical analysis demonstrates that the extent to the alignment of the sample distribution in gradient integration with respect to natural image distribution gives a lower bound of explanation certainty. Prior works add noise into images as samples and the noise distributions can lead to low explanation certainty. Counter-intuitively, our experiment shows that extra information can saturate neural networks. To this end, building trustworthy attribution analysis needs to settle the sample distribution misalignment problem. Instead of adding extra information into input images, we present a semi-optimal sampling approach by suppressing features from inputs. The sample distribution by suppressing features is approximately identical to the distribution of natural images. Our extensive quantitative evaluation on large scale dataset ImageNet affirms that our approach is effective and able to yield more satisfactory explanations against state-of-the-art baselines throughout all experimental models.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Adadi, A. and Berrada, M. (2018). Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access , 6:52138--52160
work page 2018
-
[2]
Aumann, R. J. and Shapley, L. S. (2015). Values of non-atomic games . Princeton University Press
work page 2015
-
[3]
Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and M \"u ller, K.-R. (2010). How to explain individual classification decisions. The Journal of Machine Learning Research , 11:1803--1831
work page 2010
-
[4]
Cover, T. M. (1999). Elements of information theory . John Wiley & Sons
1999
-
[5]
A., Akhtar, N., Bennamoun, M., and Mian, A
Jalwana, M. A., Akhtar, N., Bennamoun, M., and Mian, A. (2021). Cameras: Enhanced resolution and sanity preserving class activation mapping for image saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16327--16336
work page 2021
-
[6]
Kullback, S. (1997). Information theory and statistics . Courier Corporation
work page 1997
-
[7]
Li, K., Wu, Z., Peng, K.-C., Ernst, J., and Fu, Y. (2018). Tell me where to look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9215--9223
work page 2018
-
[8]
Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2020). Explainable ai: A review of machine learning interpretability methods. Entropy , 23(1):18
work page 2020
Show all 19 references
-
[9]
Lundberg, S. M. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems , 30
2017
-
[10]
L., and Dixon, S
Mishra, S., Sturm, B. L., and Dixon, S. (2017). Local interpretable model-agnostic explanations for music content analysis. In ISMIR , volume 53, pages 537--543
2017
-
[11]
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., and M \"u ller, K.-R. (2019). Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning , pages 193--209
2019
-
[12]
Omeiza, D., Speakman, S., Cintas, C., and Weldermariam, K. (2019). Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models. arXiv preprint arXiv:1908.01224
2019 arXiv
-
[13]
Roth, A. E. (1988). The Shapley value: essays in honor of Lloyd S. Shapley . Cambridge University Press
1988
-
[14]
R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618--626
2017
-
[15]
Shapley, L. S. (1953). Stochastic games. Proceedings of the national academy of sciences , 39(10):1095--1100
1953
-
[16]
Shrikumar, A., Greenside, P., and Kundaje, A. (2017). Learning important features through propagating activation differences. In International conference on machine learning , pages 3145--3153. PMLR
2017
-
[17]
Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034
2013 arXiv
-
[18]
Smilkov, D., Thorat, N., Kim, B., Vi \'e gas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825
2017 arXiv
-
[19]
Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International conference on machine learning , pages 3319--3328. PMLR
2017
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.