Pith. sign in

REVIEW 2 major objections 5 minor 33 references

LILI replaces LIME's flat-color occlusions with LaMa inpainting and mask expansion, yielding photorealistic perturbations and higher-saliency explanations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 23:13 UTC pith:MNX4RTRL

load-bearing objection Firm FID result, fragile explanation-quality claim: mask expansion is tuned on the same test set and no significance tests are reported. the 2 major comments →

arxiv 2607.15482 v1 pith:MNX4RTRL submitted 2026-07-16 cs.LG

Inpainting Insights: Elevating Visual XAI with Photorealistic Perturbations

classification cs.LG
keywords perturbation-based explanationsLIMEgenerative inpaintingLaMamask expansionsaliency metricexplanation faithfulnessimage XAI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper targets a known weakness of perturbation-based explanation methods for images: replacing superpixels with a flat color (or another deterministic fill) produces out-of-distribution samples that can mislead the classifier and blur the attribution. It proposes LILI, a variant of LIME in which occluded regions are inpainted by LaMa, a modern generative inpainting network, and in which the occlusion mask is dilated by m pixels to hide segmentation artifacts. On ImageNet-1k images explained with InceptionV3, LILI produces perturbations with substantially lower FID (6.727) than LIME (30.532) or LIME-G with DeepFill (56.524), and its saliency scores (best mean -1.816 with m=3) beat both baselines. The paper's point is that photorealism of the perturbation is not just cosmetic: it changes which features the explanation identifies.

Core claim

LILI modifies only the perturbation step of LIME. Instead of filling masked superpixels with a mean color, it feeds the original image and binary mask to LaMa, and it expands each superpixel mask by m pixels around its contour before inpainting. The expanded mask stops the inpainter from reconstructing the occluded object from border artifacts, so the object is actually removed rather than redrawn. The paper claims this yields photorealistic perturbations and, in its saliency-metric evaluation over 100 ImageNet images, the best mean score of -1.816 at m=3, outperforming LIME (-1.625) and LIME-G (best mean -1.744 at m=8; -1.699 at m=3). It also reports that LILI is less stable than LIME but m

What carries the argument

LaMa, a feed-forward inpainting network using fast Fourier convolutions, is the replacement engine: it produces plausible content conditioned on the whole image, so masked features are removed realistically rather than overwritten with a flat patch. The second mechanism is mask expansion: before inpainting, each superpixel mask is dilated by m pixels along its contour using border-following contour tracing, which hides quickshift's boundary artifacts and stops LaMa from resurrecting the occluded object.

Load-bearing premise

That the saliency metric, with min-over-alpha aggregation and the specific 100-image test set where m is tuned on the same images, faithfully reflects what users want from an explanation; if the metric is not a faithful proxy, the claimed superiority of LILI over LIME and LIME-G may vanish.

What would settle it

Re-run Table 2 on held-out images (or with cross-validation over m) and report saliency scores at each threshold alpha instead of only the minimum: if LILI no longer beats LIME-G and LIME, or the ordering reverses under average aggregation, the central claim fails. Also compute a faithfulness metric that removes the top salient superpixels and measures confidence drop, checking whether LILI's advantage persists.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If LILI is right, perturbation-based explanations can escape the out-of-distribution problem: masks remove features without creating artificial edges, so the surrogate model sees in-distribution inputs.
  • Since the mask-expansion parameter m has an optimum (m=3 seems best for LILI), the paper implies that inpainting-based occlusion must be tuned jointly with the segmentation method; simply switching in a better inpainter is not enough.
  • The FID numbers give a concrete, checkable ordering of perturbation generators: LaMa inpainting > superpixel mean color > DeepFill inpainting for this setup.
  • The stability gap (LIME W=0.889, LILI 0.852, LIME-G 0.723) shows that realism and stability can move apart, so users should expect a trade-off rather than a strict improvement.
  • Because LILI's runtime (~12.3s per explanation) is between classic LIME (~4.4s) and diffusion-based inpainting, the approach is usable where compute is limited.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The core claim suggests a general recipe: choose an inpainter that is fast enough for the many perturbations LIME needs, and add controlled mask dilation; the same recipe could transfer to other perturbation frameworks like RISE, not only LIME.
  • If photorealism alone drives the saliency gain, then a cheaper realistic fill (for example, texture synthesis or a single fast generative pass) might capture much of the benefit; the paper does not isolate that factor.
  • The min-over-alpha aggregation used for saliency scores is a notable choice; a reader could check whether the ordering survives per-threshold comparisons or average aggregation, since the minimum may reward explanations that are concentrated at one threshold.
  • The band-aid example suggests LILI's practical value in diagnosing shortcut learning: when a spurious artifact is removed in a realistic way, the classifier's confidence drops, exposing features that flat-color occlusion masks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes LILI, a modification of LIME for image explanations. Instead of replacing masked superpixels with a mean color (LIME) or with DeepFill inpainting (LIME-G), LILI uses the LaMa inpainting model and additionally expands the perturbation mask by m pixels. The authors report that LILI produces more photorealistic perturbed samples, with FID 6.727 versus 30.532 for LIME and 56.524 for LIME-G. They further report improved explanation quality on the saliency metric, with best mean score −1.816 at m=3 versus −1.625 for LIME and −1.699 for LIME-G at m=3, as well as intermediate explanation stability and runtime. The central claim is that generative inpainting with LaMa plus mask expansion makes LIME explanations more faithful.

Significance. If the explanation-quality claim holds, LILI is a simple and actionable recipe: retain LIME's sampling and surrogate fitting, but replace the occlusion step with a modern inpainter and expand masks to avoid residual-feature leakage. This is an incremental but useful contribution to perturbation-based XAI, and the FID realism result is quantitatively solid and clearly reported. The open-source code and reproducible experimental protocol are strengths. The main weakness is that the central explanation-quality claim rests on a single metric, a small 100-image evaluation, and a mask-expansion hyperparameter selected on the same test images, with no inferential statistics. The realism claim is well supported; the faithfulness claim is not yet established at the same standard.

major comments (2)
  1. [§4.2, Eq. (4)] The central comparison is vulnerable to selection bias. For LILI and LIME-G, Table 2 reports results for m ∈ {0,3,5,8}, and the text selects m=3 as the best setting for both methods. Because m is effectively chosen on the same 100 images used to report the scores, the best-of-four values overstate the methods' typical performance. The LILI advantage over LIME-G at m=3 is small: mean −1.816 vs −1.699, median −2.011 vs −1.864. Moreover, LIME-G at m=8 achieves mean −1.744 and median −1.944, close to LILI at m=3, so a different but equally defensible choice of m changes the ranking. The paper reports no confidence intervals, paired tests, or validation-split analysis. To support the claim that LILI improves explanation quality, the authors should either pre-specify m on a validation split, or report the comparison at a fixed m that was not selected on the test set, together with significance
  2. [§4.2, Eq. (4)] The saliency metric is aggregated over thresholds by taking the minimum over α: S*_{ν,m,i} = min_α S_{ν,m,i,α}. This aggregation is not the standard way to summarize threshold-dependent saliency metrics and should be justified. The min over α can reward an explanation that scores very well at a single threshold even if its overall concentration curve is poor, and it may inflate differences between methods. Since Table 2 and the explanation-quality conclusion depend entirely on this aggregation, the authors should report the full threshold curve or a threshold-averaged score and verify that the ranking of LILI versus LIME-G is preserved under a more standard aggregation.
minor comments (5)
  1. [§4.3, Table 3] The text says LILI achieves 'significantly higher explanation stability' than LIME-G, but no significance test is reported. With standard deviations of 0.109 for LILI and 0.184 for LIME-G and n=100, the difference in mean Kendall’s W (0.852 vs 0.723) may be significant, but the claim should be backed by a statistical test. Also, the stability experiment uses m=5, whereas the explanation-quality experiment selects m=3; the choice of m=5 for stability is not justified.
  2. [§2.4] The sentence 'no work has yet explored improving LIME-G with state-of-the-art inpainting models' is too strong given that related work [2] uses SAM plus Stable Diffusion for photorealistic inpainting-based perturbations. Please clarify the specific difference from prior image inpainting approaches in the context of LIME.
  3. [§2.2] The failure of the eight deterministic occlusion strategies is demonstrated only anecdotally on one 'band-aid' image. If this is meant as evidence that deterministic occlusion fails broadly, a small quantitative comparison would be more convincing; otherwise frame it as an illustrative motivating example.
  4. [§5] The Discussion correctly notes that LIME is sensitive to hyperparameters and that future work should test different combinations. This is an acknowledged limitation, but since the current evaluation uses only default LIME hyperparameters, the generalizability of the explanation-quality result remains untested. Consider adding a short Limitations paragraph rather than deferring this to future work.
  5. [Throughout] Minor language and typographical issues: 'it’s scalability' should be 'its scalability', 'explantions' in Fig. 4 should be 'explanations', 'inpaintedsample' missing space, and the abstract has an extra space before 'e.g.'. These do not affect the scientific content.

Circularity Check

0 steps flagged

No material circularity; the empirical comparison is self-contained and the central claims rest on external metrics and baselines.

full rationale

The paper's contribution is an algorithmic combination (LIME with LaMa inpainting and mask expansion) plus an empirical evaluation, not a derivation from first principles. The only self-citations are background: [10] supplies notation for LIME, [18] notes LIME's hyperparameter sensitivity, and [9] motivates segmentation artifacts; none of these establishes the central claim that LILI improves realism or explanation quality. The FID comparison (Eq. 2) uses a fixed real-image reference set and standard implementations, and the saliency metric (Eq. 3-4) is an external, pre-existing evaluation criterion. The reported saliency table lists all tested mask expansions (0, 3, 5, 8) rather than hiding a fitted quantity as a prediction; choosing m=3 as the best observed value is a statistical/benchmarking concern (no validation split or significance test), not circularity, because the scores are computed directly from the defined metric and are not constrained by construction to favor LILI. No equation reduces to its own input by definition, and no load-bearing step relies on a self-citation chain. Therefore the paper exhibits no significant circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The central claim of 'improved explanation quality' is empirical, and the ledger shows the main extra items the paper relies on are the pre-trained LaMa inpainter (external), the saliency metric, LIME defaults, and the mask-expansion hyperparameter m chosen on the test set. No new physical or model entities are introduced; the free parameters are evaluation choices rather than theoretical constants.

free parameters (2)
  • Mask expansion m = 3 (best saliency on test set; 5 used in stability; grid {0,3,5,8})
    Hyperparameter that dilates superpixel masks by m pixels. Selected on the same 100-image evaluation set, so the best value is fit to the test data rather than a validation split.
  • Saliency threshold aggregation (min over alpha) = S* = min_alpha S
    The saliency score is aggregated by taking the minimum over thresholds (Eq. 4), a choice that can be optimistic; no sensitivity analysis is given.
axioms (6)
  • domain assumption LaMa inpainting (pre-trained on Places/CelebA-HQ) generalizes to LIME superpixel masks at the ImageNet distribution.
    The method assumes LaMa is a high-quality inpainter for these masks; only aggregate FID on 100 images is presented, no per-mask inspection.
  • domain assumption Quickshift segmentation artifacts at object borders leak object identity and can be hidden by m-pixel mask expansion.
    Motivated by a single rose-hips example (Fig. 3) and a citation to [9] by a co-author; if artifacts are not the main leakage, the benefit of expansion is unexplained.
  • domain assumption LIME's default hyperparameters (quickshift, n=1000 samples, ridge regression, kernel width) are a fair comparison basis.
    Hyperparameter sensitivity is explicitly left to future work (Section 5); suboptimal defaults could bias the comparison.
  • domain assumption The saliency metric of Dabkowski and Gal [7] is a valid proxy for explanation quality.
    The central 'better explanations' claim rests on this metric; no human study, insertion/deletion, or downstream-task evaluation.
  • domain assumption FID computed from InceptionV3 features is a valid measure of perturbation realism.
    Standard metric, but the embedding may miss artifacts; perturbed images share content with reference images, which can bias FID.
  • domain assumption InceptionV3 on ImageNet-1k is a representative testbed for XAI methods.
    Only one model and one dataset; generalizability to other models/tasks is left to future work.

pith-pipeline@v1.3.0-alltime-deepseek · 9627 in / 18338 out tokens · 170630 ms · 2026-08-01T23:13:09.980687+00:00 · methodology

0 comments
read the original abstract

The increasing complexity of state-of-the-art machine learning models has made their behavior progressively harder to interpret, spurring rapid advancements in the field of eXplainable Artificial Intelligence (XAI). Among many methods proposed, perturbation-based approaches play a major role. By systematically altering (perturbing) input features, these approaches measure the impact on the model's predictions. For image data, traditional perturbation techniques, often involve replacing pixel values e.g., with a pre-defined color. However, such approaches, but also more refined deterministic techniques, generate unrealistic out-of-distribution samples and often leave visible artifacts, which can mislead the model and compromise explanation quality. In this work, we adjust LIME, a widely used perturbation-based method, to demonstrate how generative inpainting can improve perturbation-based explanations for images. We achieve photorealistic perturbed samples that align better with the original data distribution and enhance explanation quality.

Figures

Figures reproduced from arXiv: 2607.15482 by Damien Garreau, Josef Lindl, Mariana Chaves.

Figure 1
Figure 1. Figure 1: Perturbed samples and explanations for band-aid from different methods: The first row shows the segmented original image with an example of a perturbation mask (left image), as well as the perturbed samples as given by LIME, LIME-G and LILI. The second row shows the model’s gradients with respect to the input, as well as the explanations given by different methods (here, the top ten su￾perpixels by absolut… view at source ↗
Figure 2
Figure 2. Figure 2: Explanations using 8 deterministic occlusion methods: [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Motivation for perturbation mask expansion: [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Examples of perturbed samples and explantions by methods: [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Boxplot of the agreement of rankings: As computed using Kendall’s W (coefficient of concordance) for the top 10 superpixels (by absolute mean weight) for each of the 100 sample images [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 2 linked inside Pith

  1. [1]

    In: Proceedings of the ACCV (2020)

    Agarwal, C., Nguyen, A.: Explaining image classifiers by removing input features using generative models. In: Proceedings of the ACCV (2020)

  2. [2]

    In: NeurIPS 2025 Workshop for Imageomics: Discovering Biological Knowledge from Images Using AI (2025)

    Aghakishiyeva, G., Zhou, J., Arya, S., Poling, J.D., Houliston, H.R., Womble, J.N., Johnston, D.W., Bent, B.: Photorealistic inpainting for perturbation-based explanations in ecological monitoring. In: NeurIPS 2025 Workshop for Imageomics: Discovering Biological Knowledge from Images Using AI (2025)

  3. [3]

    Ang, D.: Daa233/generative-inpainting-pytorch: A pytorch reimplemen- tation for paper generative image inpainting with contextual atten- tion (February 2021), https://github.com/daa233/generative-inpainting- pytorch/commit/c6cdaea0427b37b5b38a3f48d4355abf9566c659, gitHub repository

  4. [4]

    Sensors25(13) (2025)

    Cheng, Z., Wu, Y., Li, Y., Cai, L., Ihnaini, B.: A comprehensive review of explain- able artificial intelligence (xai) in computer vision. Sensors25(13) (2025)

  5. [5]

    Frontiers in Human Dynamics6 (2024)

    Cheong, B.C.: Transparency and accountability in ai systems: safeguarding well- being in the age of algorithmic decision-making. Frontiers in Human Dynamics6 (2024)

  6. [6]

    NeurIPS’20 (2020)

    Chi, L., Jiang, B., Mu, Y.: Fast fourier convolution. NeurIPS’20 (2020)

  7. [7]

    Dabkowski, P., Gal, Y.: Real time image saliency for black box classifiers. pp. 6970–6979. NeurIPS’17 (2017)

  8. [8]

    Artifi- cial Intelligence Review58(4) (February 2025)

    Elharrouss, O., Damseh, R., Belkacem, A.N., Badidi, E., Lakas, A.: Transformer- based image and video inpainting: current challenges and future directions. Artifi- cial Intelligence Review58(4) (February 2025)

  9. [9]

    In: Camps-Valls, G., Ruiz, F.J.R., Valera, I

    Garreau, D.: How to scale hyperparameters for quickshift image segmentation. In: Camps-Valls, G., Ruiz, F.J.R., Valera, I. (eds.) AISTATS 2022. Proceedings of Machine Learning Research, vol. 151, pp. 5243–5275 (28–30 Mar 2022)

  10. [10]

    Garreau, D., Mardaoui, D.: What does LIME really see in images? In: ICML 2021 (2021)

  11. [11]

    In: Advances in Neural Information Processing Systems

    Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.:Ganstrained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems. vol. 30 (2017)

  12. [12]

    The Annals of Mathe- matical Statistics10(3), 275 – 287 (1939)

    Kendall, M.G., Smith, B.B.: The Problem ofmRankings. The Annals of Mathe- matical Statistics10(3), 275 – 287 (1939)

  13. [13]

    In: Proceedings of the CVPR

    Kim, S., Suh, S., Lee, M.: RAD: Region-aware diffusion models for image inpaint- ing. In: Proceedings of the CVPR. pp. 2439–2448 (June 2025)

  14. [14]

    In: Proceedings of the ICCV

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollar, P., Girshick, R.: Segment anything. In: Proceedings of the ICCV. pp. 4015–4026 (October 2023)

  15. [15]

    ICML’23 (2023)

    Kotelnikov, A., Baranchuk, D., Rubachev, I., Babenko, A.: TabDDPM: modelling tabular data with diffusion models. ICML’23 (2023)

  16. [16]

    Nature Communications10(1), 1096 (Mar 2019) Inpainting Insights 15

    Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.R.: Unmasking clever hans predictors and assessing what machines really learn. Nature Communications10(1), 1096 (Mar 2019) Inpainting Insights 15

  17. [17]

    Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. pp. 4768–4777. NIPS’17 (2017)

  18. [18]

    In: Proceedings of The 24th AISTATS

    Mardaoui, D., Garreau, D.: An analysis of LIME for text data. In: Proceedings of The 24th AISTATS. Proceedings of Machine Learning Research, vol. 130, pp. 3493–3501 (13–15 Apr 2021)

  19. [19]

    (March 2024), https://github.com/mseitzer/pytorch- fid/commit/b9c18118d082cbd263c1b8963fc4221dc1cbb659

    mseitzer: Mseitzer/pytorch-fid: Compute fid scores with py- torch. (March 2024), https://github.com/mseitzer/pytorch- fid/commit/b9c18118d082cbd263c1b8963fc4221dc1cbb659

  20. [20]

    The Journal of Strategic Information Systems34(2), 101885 (2025)

    Papagiannidis, E., Mikalef, P., Conboy, K.: Responsible artificial intelligence gov- ernance: A review and research framework. The Journal of Strategic Information Systems34(2), 101885 (2025)

  21. [21]

    In: British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018

    Petsiuk, V., Das, A., Saenko, K.: RISE: randomized input sampling for explanation of black-box models. In: British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018. p. 151 (2018)

  22. [22]

    Applied Artificial Intelligence39(1), 2463722 (2025)

    Radanliev, P.: AI ethics: Integrating transparency, fairness, and privacy in ai de- velopment. Applied Artificial Intelligence39(1), 2463722 (2025)

  23. [23]

    In: Proceedings of the 22nd ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016

    Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you?: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016. pp. 1135–1144 (2016)

  24. [24]

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models (2021)

  25. [25]

    In: 2022 WACV

    Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K., Lempitsky, V.: Resolution-robust large mask inpainting with fourier convolutions. In: 2022 WACV. pp. 3172–3182 (2022)

  26. [26]

    Computer Vision, Graphics, and Image Processing30(1), 32–46 (1985)

    Suzuki, S., Abe, K.: Topological structural analysis of digitized binary images by border following. Computer Vision, Graphics, and Image Processing30(1), 32–46 (1985)

  27. [27]

    In: 2016 CVPR

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the incep- tion architecture for computer vision. In: 2016 CVPR. pp. 2818–2826 (2016)

  28. [28]

    In: Explainable Artificial Intelligence

    Tritscher, J., Lissmann, P., Wolf, M., Krause, A., Hotho, A., Schlör, D.: Generative inpainting for shapley-value-based anomaly explanation. In: Explainable Artificial Intelligence. pp. 230–243 (2024)

  29. [29]

    In: Com- puter Vision – ECCV 2008

    Vedaldi, A., Soatto, S.: Quick shift and kernel methods for mode seeking. In: Com- puter Vision – ECCV 2008. pp. 705–718 (2008)

  30. [30]

    arXiv preprint arXiv:1806.03589 (2018)

    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. arXiv preprint arXiv:1806.03589 (2018)

  31. [31]

    arXiv preprint arXiv:1801.07892 (2018)

    Yu,J.,Lin,Z.,Yang,J.,Shen,X.,Lu,X.,Huang,T.S.:Generativeimageinpainting with contextual attention. arXiv preprint arXiv:1801.07892 (2018)

  32. [32]

    In: Computer Vision – ECCV 2014

    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Computer Vision – ECCV 2014. pp. 818–833 (2014)

  33. [33]

    In: 2018 CVPR

    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 CVPR. pp. 586–595 (2018)