Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Sampling Matters in Explanations: Towards Trustworthy Attribution Analysis Building Block in Visual Models through Maximizing Explanation Certainty

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read This paper argues that the trustworthiness of gradient-integration explanations is governed by how closely the sampling distribution matches natural images, and proposes Bernoulli feature suppression as a practical way to keep samples…

desk verdict Pixel-dropping sampling beats SmoothGrad and IG empirically, but the paper's theoretical justification doesn't hold and the evaluation is circular. read the letter →

arxiv 2506.19442 v2 pith:TM6D3LUT submitted 2025-06-24 cs.CV

classification cs.CV
keywords explainableAIattributionanalysisgradientintegrationsamplingdistributionmutualinformationfeaturesuppressionImageNetexplanationcertainty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that gradient-integration explanations for image models are only as trustworthy as the distribution of the samples whose gradients are averaged. It defines explanation certainty as the conditional probability of the input given the explanation, and claims this certainty is lower-bounded by a mutual-information term, $\Pr(x|z) \ge \exp(I(z;x))/\exp(H(x))$, so maximizing mutual information between inputs and explanations should maximize trustworthiness. From this, the paper concludes that the optimal sampling distribution for gradient integration is the natural image distribution itself, $q^* = p$. Because adding Gaussian noise or linear scaling pulls samples away from natural images, the paper instead proposes Bernoulli feature suppression, zeroing each pixel with probability about $0.7$, and reports that this yields samples that align with natural images and higher mutual information than SmoothGrad and integrated gradients on all eight ImageNet models tested. If correct, the result makes feature suppression a stronger building block than noise addition for attribution analysis.

What carries the argument

The central object is the explanation-certainty lower bound $\Pr(x|z) \ge \exp(I(z;x))/\exp(H(x)) = \exp(-H(x|z))$, obtained by rewriting the log-conditional density as a mutual-information integral minus the input entropy. This bound turns the abstract goal of trustworthy explanations into a concrete optimization target: choose the sampling distribution $q$ in gradient integration so that the mutual information between the input and the resulting explanation is as large as possible. The paper then proposes Bernoulli feature suppression, $x^*_i = B(\{x_i,0\}; 1-p)$ with $p\approx 0.7$, as the practical mechanism, and supports it with a cosine-kernel PCA alignment experiment and an activation/mutual-information sweep over $p$.

What would settle it

A direct test: on a model fine-tuned with the Gaussian-noise and luminance augmentations used in the paper's Figure 2, compute mutual information $I(z;x)$ for suppression-based, SmoothGrad, and integrated-gradient explanations; the paper's robustness claim predicts suppression still yields the highest value, so if any baseline matches or exceeds it, the central empirical claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a recipe for sampling in gradient integration: draw samples by dropping pixels according to independent Bernoulli trials with drop probability near $0.7$, rather than by adding noise or by global intensity scaling. The paper's theoretical route asserts an inequality chain in which the log-conditional density of inputs given explanations is lower bounded by $I(z;x) - H(x)$, yielding $\Pr(x|z) \ge \exp(-H(x|z))$. It then uses convexity of mutual information to argue that the sampling distribution maximizing expected explanation certainty is the natural image distribution $q^*=p$, and presents feature suppression as a practical approximation to that distribution. Empirically, the paper measures the mutual-information lower bound on ImageNet across AlexNet, VGG16, DenseNet121, ShuffleNet V2 x0.5, ResNet18, ResNet50, Wide ResNet50-2, and ViT-B/16, and reports higher $I(z;x)$ than SmoothGrad and integrated gradients on every model, with network activation also peaking for drop probabilities in $[0.6,0.8]$.

Load-bearing premise

The central bet is that explanation certainty really is lower-bounded by $\exp(I(z;x))/\exp(H(x))$ and that zeroing each pixel independently with probability $0.7$ produces samples statistically close enough to natural images for that bound to be the right guide; if either is wrong, the recommended sampling rule may not be optimal.

Editorial extensions

If this is right

  • Attribution methods that add noise or linearly scale inputs should be expected to underperform methods that keep samples close to the natural image distribution, because their samples reduce the mutual-information lower bound.
  • For models trained with data augmentation, noise-based sampling can be particularly fragile: the paper shows that fine-tuning with added Gaussian noise and luminance changes degrades SmoothGrad and integrated-gradient explanations, whereas suppression-based sampling remains robust.
  • Feature suppression with $p\approx 0.7$ can serve as a drop-in replacement for gradient integration in composite explainability pipelines such as Grad-CAM-style methods.
  • The optimal drop probability $[0.6,0.8]$ should transfer across architectures on ImageNet, since the paper reports a consistent activation and mutual-information pattern on all tested models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's mutual-information framing suggests a broader design principle: any attribution method that averages over perturbations should match the perturbation distribution to the data distribution rather than choosing it for convenience; this would extend to latent-space and text explanations, which the paper does not test.
  • A rigorous replacement for the paper's integral-manipulation step may yield a bound involving KL divergence rather than differential entropy; if so, the same feature-suppression method could be audited by estimating that divergence directly on held-out data, giving a quantitative faithfulness test that does not depend on the questionable Hölder step.
  • The optimal drop probability near $0.7$ likely reflects the redundancy of ImageNet-scale natural images; on less redundant domains, such as medical or satellite imagery, the peak-activation curve in Figure 4(b) could shift, making per-domain calibration of $p$ a natural next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies the choice of sampling distribution in gradient-integration attribution methods for image classifiers. It introduces a notion of "explanation certainty" Pr(x|z), claims to lower-bound it by exp(-H(x|z)) using a step labeled Hölder's inequality, and concludes that the optimal sampling distribution for gradient integration is the natural image distribution p. The proposed method, Bernoulli feature suppression, drops each pixel with probability p and is claimed to produce samples approximately distributed like natural images. The authors report mutual information I(z;x) between inputs and explanations on several ImageNet models, claiming improvements over SmoothGrad, Integrated Gradients, and vanilla gradients.

Significance. If the theoretical claims were correct, the proposed sampling scheme would be a simple and potentially useful building block for gradient-based attribution, and the paper's breadth of evaluation across eight architectures would strengthen the case. The reproducible code link and the empirical observation that moderate pixel dropping can increase gradient norms are also positive features. However, the central lower-bound derivation is mathematically invalid, the optimality result does not follow from the stated arguments, and the main quantitative comparison uses the evaluation metric to select the method's only hyperparameter. As presented, the paper does not establish its central claims, and the empirical results are not an independent test of the method.

major comments (4)
  1. [Sec. 3.2, Eqs. (5)-(6)] The step labeled Hölder's inequality is not a valid application of Hölder. Hölder's inequality bounds the integral of a product by products of norms; here the paper multiplies the unweighted integral ∫∫ log p(x|z) dx dz by ∫∫ p(x,z) dx dz = 1 and then asserts that the result is at least the weighted integral ∫∫ p(x,z) log(p(x|z)/(p(x)p(x))) dx dz. No inequality of this form follows from Hölder without additional pointwise or norm conditions, and the claimed inequality is generally false. Consequently, Eq. (6) does not establish a lower bound on explanation certainty. In addition, Eq. (1) treats the continuous conditional density p(x|z) as a probability, so the quantity exp(-H(x|z)) is not a meaningful lower bound on a probability: differential entropy can be negative and exp(-H(x|z)) can exceed one. The theoretical foundation of the paper is therefore not sound.
  2. [Sec. 3.3-3.4, Eqs. (7)-(10)] The derivation of the optimal sampling distribution q* = p is also invalid. In Eq. (7), the explanation z is defined as the expectation E_{xhat~qx}[|∇ f|], which for a fixed input x is a deterministic function of x; the mutual information I(x;z) in Eq. (8) is then identically zero unless additional randomness from the sampling procedure is retained in z, contradicting the notation. Furthermore, Eq. (9) applies Jensen's inequality to I(x;z) as though mutual information were convex in the explanation vector z for fixed x; mutual information is convex in conditional distributions p(z|x), not in the realized values z, and the inequality I(x; E[Z]) ≤ E[I(x;Z)] is not an information-theoretic identity. Even if the Jensen step were accepted, Eq. (10) merely restates an argmax over qx and gives no argument that the natural image distribution p is the maximizer. Thus the theoretical claim that q* = p is unestablished.
  3. [Sec. 5.2, Figs. 4(b) and 5] The quantitative evaluation is circular. The pixel-dropping probability p is selected in Fig. 4(b) by maximizing the mutual information I(z;x) on a ResNet50, and the headline results in Fig. 5 report exactly the same metric I(z;x) on all tested models. This means the method's only free parameter is fitted to the evaluation objective, so the reported improvements over SmoothGrad and Integrated Gradients do not constitute an independent validation of the method. To support the claim, the authors should fix p on a validation set or a separate model and evaluate on held-out images and architectures, or report sensitivity of the ranking to p.
  4. [Sec. 4.1-4.2, Fig. 4(a)] The claim that Bernoulli feature suppression produces samples "approximately identical to the distribution of natural images" is supported only by a qualitative PCA projection of penultimate-layer embeddings in Fig. 4(a). This projection is not a direct measurement of the image distribution, and masked images containing exact zeros are not obviously natural images. Since the theoretical optimality argument is built on sample distribution alignment, a quantitative distribution-distance evaluation (e.g., FID, MMD, or a density estimate) is required before this premise can support the method. As it stands, the alignment claim is not established.
minor comments (5)
  1. [Sec. 3.2, Eq. (1)] The notation in Eq. (1) is dimensionally inconsistent: Pr(x|z) is written as a product over all x and z and then as an integral over x and z, but the relationship between the discrete product and the continuous integral is not defined.
  2. [Sec. 3.1] The variable y is described as "ground-truth category y ∈ R"; a class label is not naturally a real number, and the notation should be clarified.
  3. [Sec. 4.1] The Bernoulli notation x*_i = B({x_i,0}; 1-p) is nonstandard; the authors should define explicitly whether p is the probability of setting the pixel to zero or the probability of keeping it, and how the Bernoulli trial is applied independently across pixels.
  4. [Related work] The LIME citation is given as Mishra et al., 2017, which does not appear to be the original LIME paper; the authors should cite the correct source (Ribeiro et al.) or justify the alternative attribution.
  5. [Fig. 4(a)] The PCA figure would benefit from a legend that maps colors and markers to the specific sampling settings, and from a quantitative measure of cluster separation, since visual inspection of projections is not conclusive.

Circularity Check

1 steps flagged · score 6.0 of 10

Pixel-dropping probability p is tuned on the mutual-information metric that later serves as the evaluation, making the headline I(z;x) improvement partially self-fulfilling.

  1. fitted input called prediction [Section 4.2.1 'Optimal p' (Fig. 4b) and Section 5.2 'Quantitative evaluation' (Fig. 5)]
    "We empirically investigate the optimal dropping probability p in Figure 4(b) by: (1) Measuring how networks activate with respect to various p and (2) measuring how explanation certainty (mutual information) changes with respect to various p. ... The result shows that when p falls in [0.6, 0.8] the network activation will be maximized and the mutual information will be maximized as well. ... We also conduct an extensive quantitative evaluation by measuring the lower bound (mutual information) of explanation certainty. ..."

    The method's only substantive hyperparameter p is selected in Section 4.2.1 by directly maximizing the same mutual-information metric I(z;x) that Section 5.2 and Figure 5 use as the evidence of success. Once p is fixed to the value that maximizes I(z;x) on ResNet50 (and checked on other architectures by the same criterion), reporting that the method achieves higher I(z;x) than SmoothGrad and IG is not an independent test of the central claim. The headline quantitative result is substantially built into the tuning step: the fitted p is renamed a 'semi-optimal sampling approach' and the same optimized metric is then presented as a predictive improvement.

full rationale

The formal derivation in Eqs. (5)-(6) is not circular in the sense of equivalence-by-construction: it attempts to prove a lower bound on explanation certainty from definitions, even though the step labeled Holder's inequality is mathematically invalid and the continuous-density bound exp(-H(x|z)) is not meaningful as a probability. Those are correctness concerns, not circularity. Similarly, the jump to the optimal sampling result q*=p in Eq. (10) is asserted rather than proven, but an unsupported step is a logical gap, not a circular reduction. I find no load-bearing self-citations, and the PCA-based claim that Bernoulli masking yields approximately natural-image samples is an empirical assertion, not a definitional identity. The genuine circular element is the empirical evaluation loop: the pixel-dropping probability p, the core parameter of the proposed sampling distribution, is tuned in Section 4.2.1 by maximizing the very mutual-information quantity that Section 5.2 uses as the benchmark. Consequently the central quantitative claim that the method outperforms baselines on I(z;x) is partially forced by the fitting procedure. Score 6 reflects this partial circularity of the paper's headline quantitative claim; the qualitative results and cross-model consistency prevent a higher score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central chain requires: (i) an i.i.d. pixel assumption, (ii) an unproved inequality connecting an unweighted log-density integral to mutual information, (iii) a Jensen step treating mutual information as convex in the explanation, and (iv) an assertion that Bernoulli masking matches natural image statistics. The only fitted parameter is the dropping probability p, which is tuned to the evaluation metric.

free parameters (1)
  • pixel dropping probability p = 0.7 (optimal range 0.6-0.8)
    Selected in Fig. 4(b) by maximizing network activation and mutual information on a ResNet50; the same range is then used in the quantitative evaluation where the mutual-information metric is the headline result.
assumptions (4)
  • domain assumption Natural images are pixel-wise i.i.d.
    Invoked in Sec. 3.2 before Eq. (1) to rewrite explanation certainty as a product/exp of per-pixel conditional densities. Natural images have strong spatial correlations, so this simplification is false and undermines the continuous-density bound.
  • ad hoc to paper The inequality in Eq. (5) holds: the unweighted integral of log p(x|z) is at least the weighted integral of p(x,z) log(p(x|z)/(p(x)p(x))).
    This step is presented as Hölder's inequality but is not a consequence of normalization of p(x,z); it is the load-bearing step for the claimed lower bound on explanation certainty.
  • ad hoc to paper Mutual information I(x; z) is convex in z for fixed x, so Jensen's inequality applies to z = E[|∇ f|].
    Used in Eq. (9) to rewrite the optimal-sampling objective. Mutual information is a functional of distributions, not a convex function of a deterministic explanation vector; the step is unjustified.
  • domain assumption Bernoulli pixel masking produces samples approximately distributed like natural images.
    Assumed in Sec. 4.1 and used to connect the practical sampler to the theoretical optimum q* = p; the only evidence is the qualitative PCA plot in Fig. 4(a), and zero-valued pixels make the distributions visibly different.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sampling Matters in Explanations: Towards Trustworthy Attribution Analysis Building Block in Visual Models through Maximizing Explanation Certainty." pith.science (2026). https://pith.science/paper/TM6D3LUT

@misc{pith2026250619442,
  author       = {Pith},
  title        = {Pith review of: Sampling Matters in Explanations: Towards Trustworthy Attribution Analysis Building Block in Visual Models through Maximizing Explanation Certainty},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TM6D3LUT}},
  note         = {Machine review of arXiv:2506.19442}
}
read the original abstract

Image attribution analysis seeks to highlight the feature representations learned by visual models such that the highlighted feature maps can reflect the pixel-wise importance of inputs. Gradient integration is a building block in the attribution analysis by integrating the gradients from multiple derived samples to highlight the semantic features relevant to inferences. Such a building block often combines with other information from visual models such as activation or attention maps to form ultimate explanations. Yet, our theoretical analysis demonstrates that the extent to the alignment of the sample distribution in gradient integration with respect to natural image distribution gives a lower bound of explanation certainty. Prior works add noise into images as samples and the noise distributions can lead to low explanation certainty. Counter-intuitively, our experiment shows that extra information can saturate neural networks. To this end, building trustworthy attribution analysis needs to settle the sample distribution misalignment problem. Instead of adding extra information into input images, we present a semi-optimal sampling approach by suppressing features from inputs. The sample distribution by suppressing features is approximately identical to the distribution of natural images. Our extensive quantitative evaluation on large scale dataset ImageNet affirms that our approach is effective and able to yield more satisfactory explanations against state-of-the-art baselines throughout all experimental models.

Figures

Figures reproduced from arXiv: 2506.19442 by the authors.

Figure 1
Figure 1. The figure is a performance showcase. The performances are qualitatively compared from two [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. This experiment exhibits the failures and the sensitivity of the explanations by fine-tuning a pre [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. This example shows the information redundancy in nature images. The leftmost is the reference. The [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: In the left figure we investigate how neural networks respond to various sampling distributions. We [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: We extensively evaluate our algorithm against the gradient integration baselines over ImageNet [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages

  1. [1]

    and Berrada, M

    Adadi, A. and Berrada, M. (2018). Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE access , 6:52138--52160

  2. [2]

    Aumann, R. J. and Shapley, L. S. (2015). Values of non-atomic games . Princeton University Press

  3. [3]

    Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and M \"u ller, K.-R. (2010). How to explain individual classification decisions. The Journal of Machine Learning Research , 11:1803--1831

  4. [4]

    Cover, T. M. (1999). Elements of information theory . John Wiley & Sons

  5. [5]

    A., Akhtar, N., Bennamoun, M., and Mian, A

    Jalwana, M. A., Akhtar, N., Bennamoun, M., and Mian, A. (2021). Cameras: Enhanced resolution and sanity preserving class activation mapping for image saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16327--16336

  6. [6]

    Kullback, S. (1997). Information theory and statistics . Courier Corporation

  7. [7]

    Li, K., Wu, Z., Peng, K.-C., Ernst, J., and Fu, Y. (2018). Tell me where to look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9215--9223

  8. [8]

    Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2020). Explainable ai: A review of machine learning interpretability methods. Entropy , 23(1):18

Show all 19 references
  1. [9]

    Lundberg, S. M. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems , 30

  2. [10]

    L., and Dixon, S

    Mishra, S., Sturm, B. L., and Dixon, S. (2017). Local interpretable model-agnostic explanations for music content analysis. In ISMIR , volume 53, pages 537--543

  3. [11]

    Montavon, G., Binder, A., Lapuschkin, S., Samek, W., and M \"u ller, K.-R. (2019). Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning , pages 193--209

  4. [12]

    Omeiza, D., Speakman, S., Cintas, C., and Weldermariam, K. (2019). Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models. arXiv preprint arXiv:1908.01224

  5. [13]

    Roth, A. E. (1988). The Shapley value: essays in honor of Lloyd S. Shapley . Cambridge University Press

  6. [14]

    R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D

    Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618--626

  7. [15]

    Shapley, L. S. (1953). Stochastic games. Proceedings of the national academy of sciences , 39(10):1095--1100

  8. [16]

    Shrikumar, A., Greenside, P., and Kundaje, A. (2017). Learning important features through propagating activation differences. In International conference on machine learning , pages 3145--3153. PMLR

  9. [17]

    Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034

  10. [18]

    Smilkov, D., Thorat, N., Kim, B., Vi \'e gas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825

  11. [19]

    Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International conference on machine learning , pages 3319--3328. PMLR

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.