Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Verifying Machine Unlearning with Explainable AI

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that attribution-based XAI heatmaps can verify whether machine unlearning has actually removed a sensitive pattern, using SIDU heatmaps and two new metrics, HC and AS, on a harbor-front object counting task.

desk verdict A plausible but thinly evidenced proof-of-concept for using XAI heatmaps to verify unlearning; worth a serious referee, but the numbers aren't yet convincing. read the letter →

arxiv 2411.13332 v1 pith:M5QCBHIR submitted 2024-11-20 cs.LG cs.AI

Àlex Pujol Vidal , Anders S. Johansen , Mohammad N. S. Jahromi , Sergio Escalera , Kamal Nasrollahi , Thomas B. Moeslund This is my paper · ORCID
classification cs.LGcs.AI
keywords MachineunlearningExplainableAIRighttobeforgottenObjectcountingHeatmapcoverageAttentionshiftSIDUPrivacycompliance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that heatmap explanations can verify whether a machine-learning model has genuinely unlearned a sensitive pattern, rather than merely recovering its accuracy. The motivation is practical: privacy legislation creates a right to be forgotten, and retraining from scratch is expensive, so unlearning methods need a cheap way to certify that forbidden patterns no longer influence predictions. On harbor-front thermal imagery, the authors show that SIDU heatmaps and two proposed metrics, Heatmap Coverage (HC) and Attention Shift (AS), can distinguish models that still attend to humans from models that do not. If correct, this gives regulators and practitioners a visual, quantitative check for unlearning that goes beyond accuracy.

What carries the argument

The load-bearing tool is SIDU (Similarity Difference and Uniqueness), a gradient-free attribution method that extracts feature maps from the last convolutional layer and forms masks from similarity-difference and uniqueness scores to produce a heatmap of the image regions driving the prediction. On top of those heatmaps, the paper defines $\mathrm{HC}(f_i)$, the fraction of heatmap intensity falling inside object-detection bounding boxes (Equation 1), and $\mathrm{AS}(f_{u_i}, f_o)$, the standard deviation of the pixel-wise difference between the original and unlearned heatmaps (Equation 2). HC measures whether attention lands on the objects meant to be retained, and AS measures how much attention has moved after unlearning.

What would settle it

Run the same unlearning pipeline and verify with two independent attribution methods on identical images, for example SIDU and a gradient-based method such as Grad-CAM. If one method says the unlearned model still attends to humans while the other says it does not, the claim that heatmaps can verify unlearning is unsupported. A complementary test would ablate human regions from the input: a model that has truly unlearned humans should show almost no drop in predicted count when those regions are removed.

Watch

Extended reading notes

Core claim

The paper claims that attribution-based XAI can serve as a verification layer for machine unlearning. In a thermal-imaging regression task where a model counts people, bicycles, vehicles, and motorcycles, the authors remove humans from the label and compare a freshly retrained baseline to four unlearned models obtained by fine-tuning, pruning, reinitializing, and noise injection. Using SIDU heatmaps and the two proposed metrics, they find that the unlearned models concentrate heatmap intensity on retained objects more than the retrained baseline does, while plain fine-tuning leaves clear human-related heatmap traces. The proposed metrics therefore reveal differences in unlearning quality that MAE and RMSE do not capture.

Load-bearing premise

The verification holds only if SIDU heatmaps faithfully reflect the image regions the model actually uses, and only if the annotated bounding boxes are the correct ground truth for where a counting model's attention should lie.

Editorial extensions

If this is right

  • Heatmap-based metrics like HC and AS can be added to accuracy as a practical verification step when regulators request that a model forget a protected pattern.
  • Unlearning methods such as pruning and noise injection can shift attention away from sensitive objects more strongly than plain fine-tuning, even when error metrics look similar.
  • Because the metrics only require bounding-box annotations and heatmaps, the same verification recipe transfers to other counting or detection tasks where object locations are known.
  • Verification of unlearning no longer needs to rely solely on expensive attack-based benchmarks or full retraining for comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: The HC metric assumes the annotated bounding boxes are the only legitimate focus, yet a counting model could plausibly use context near an object; under that behavior, HC would penalize correct reasoning.
  • Extension: Because the paper only inspects post-hoc heatmaps, its own evidence suggests an end-to-end training loop in which HC or AS enters the loss and actively pushes attention off forbidden regions; the paper lists this as future work.
  • Extension: The same pipeline could be tested on established unlearning benchmark tasks where membership-inference attack scores provide an independent ground truth, giving a direct comparison between attribution-based verification and attack-based verification.
  • Extension: The paper's comparison is limited to one dataset and one attribution method, so the natural next check is whether the ordering of unlearning methods by h-HC is stable across architectures, datasets, and XAI methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes using attribution-based XAI, specifically the SIDU method, to verify machine unlearning in a real-world regression task: counting objects (excluding humans) in thermal harbor-front imagery. The authors define two new metrics, Heatmap Coverage (HC) and Attention Shift (AS), which quantify how much of the SIDU heatmap overlaps with object bounding boxes and how much the heatmap changes after unlearning. They compare four unlearning methods (Fine-tune, Prune, Reinit, Confuse) against a retrained-from-scratch baseline and report that unlearning methods concentrate heatmap intensity on retained objects more than full retraining, while also reducing attention on humans. The paper concludes that attribution-based XAI can effectively verify unlearning, and the authors release code and qualitative visualizations.

Significance. If the central claim is validated, the paper offers a computationally cheaper and more interpretable alternative to attack-based unlearning verification, extends MU evaluation to regression tasks, and contributes two reusable metrics. The authors are honest about the key assumption in Section 5.3. However, the significance is conditional: HC and AS only measure unlearning if SIDU heatmaps faithfully reflect the model's true causal reliance on image regions. No experiment in the paper tests this load-bearing assumption, and the quantitative evidence is limited to single-run point estimates. The novelty is moderate, as the paper applies an existing XAI method to a new verification purpose, but the framework is clearly presented and the code release is a strength.

major comments (4)
  1. [Section 5.3, Eqs. (1)-(2)] The entire verification argument rests on the assumption that SIDU heatmaps accurately reflect the model's reliance on human versus non-human patterns. The paper explicitly concedes this in Section 5.3, but no experiment tests it. Without such validation, an alternative reading of Table 1 is that HC and AS respond to SIDU artifacts (e.g., low-resolution upsampling, channel-normalization choices) rather than to genuine unlearning. The authors should add a faithfulness check for SIDU on this specific counting model, for example by comparing SIDU attributions to the effect of occluding or perturbing human versus non-human regions on the model's count prediction, or by using standard insertion/deletion metrics, or by comparing SIDU with at least one other attribution method. This is necessary to support the claim that the proposed metrics verify unlearning rather than merely explainer behavior.
  2. [Eq. (1), Section 4.5] The HC metric defines the ground-truth mask M using object-detection bounding boxes. For a global counting regression, context outside a bounding box can legitimately influence the count (e.g., a bicycle's presence is correlated with a nearby human, and empty background can provide scale cues). A low h-HC after unlearning may therefore reflect the explainer shifting credit to context rather than the model actually forgetting humans. The authors should justify, or empirically test, the assumption that bounding boxes are the correct ground truth for where a counting model's attention should be concentrated. At minimum, this assumption should be stated as a limitation and discussed in Section 5.3.
  3. [Table 1, Section 5.1] All metrics in Table 1 are reported as single values with no standard deviations, no multiple seeds, and no significance tests. Differences used to rank unlearning methods, such as r-HC 7.632e-3 for Confuse versus 6.827e-3 for Fine-tune, are small and could easily be within run-to-run noise. The qualitative ordering of unlearning methods by HC/AS is therefore not statistically established. The authors should provide results over multiple random seeds with error bars or at least report the number of seeds and the observed variance.
  4. [Section 5, Section 4.4] The model architecture is never specified. Since SIDU generates heatmaps from the last convolutional layer, the resulting HC and AS values depend critically on the architecture's feature-map resolution and layer semantics. Without specifying the network (e.g., backbone, depth, number of parameters, input resolution), the results are not reproducible and the claimed behavior cannot be generalized. The authors should specify the architecture and, ideally, test at least one other architecture to show the verification method is not architecture-specific.
minor comments (4)
  1. [Section 4.3] The adaptation of Prune, Reinit, and Confuse to regression drops the entropy and weighting terms from the NeurIPS 2023 challenge methods. The paper asserts this preserves unlearning behavior but provides no evidence. A brief ablation or justification would strengthen the claim that these are faithful adaptations.
  2. [Table 1] The arrow notation in the table header appears inconsistent with the text: the text states that lower h-HC values indicate that the model is 'less weighted around human bounding boxes', yet the header shows 'h-HC ↑' if read literally. The arrows should be corrected to match the intended direction of improvement (r-HC ↑, h-HC ↓) or removed.
  3. [Section 4.4] The sentence 'Our experiments have shown that SIDU not only performs well...' reports prior results without a reference or experimental detail. If this refers to the authors' earlier work, cite it explicitly; otherwise, remove the claim.
  4. [Abstract and Section 4] There are several typos and grammatical errors, including 'quantitavely' in Section 4 and 'using to attribution-based XAI' in the Conclusion. These should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the verification metrics are measured, not fitted, and the SIDU self-citation is independent support rather than a circular premise.

full rationale

The paper's central claim is empirical rather than derivational: it defines a curated dataset D' by removing human counts from labels, applies independently specified unlearning methods (fine-tune, prune, reinit, confuse) from a common original model, and then measures SIDU heatmap behavior with two proposed metrics. No parameter is fitted to the HC or AS outcomes, and the unlearning methods are not defined in terms of the verification metrics. The conclusion that unlearned models 'can better concentrate heatmap intensity on the desired target objects' is a reading of the observed r-HC values in Table 1, not a consequence of an identity or a fitted parameter renamed as a prediction. The HC metric is, by construction, the fraction of attribution mass inside the annotated bounding boxes, but reporting that this fraction is higher for some models is a measurement, not a circular derivation. The reliance on SIDU is a self-citation—references [15] and [10] share authors with the present paper—and SIDU's faithfulness is indeed load-bearing for the interpretation. However, SIDU is a previously published, externally testable algorithm whose definition does not incorporate the target result, so the self-citation is independent support, not a circularity. The manuscript explicitly flags the faithfulness assumption in Section 5.3: 'the assumption that the XAI methods used for heatmap generation accurately reflect important regions in the image ... adds a layer of uncertainty.' That is a validity/correctness limitation, not a circular step. Similarly, interpreting AS as a 'shift in attention to relevant areas' overstates what a standard deviation of heatmap differences measures, but that is a construct-validity concern, not a reduction of the conclusion to its inputs. No step in the paper's reasoning equates a prediction with a fitted input or imports a uniqueness conclusion from prior work by the same authors.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the reliability of SIDU heatmaps and the chosen unlearning hyperparameters. No new physical entities are introduced. The two proposed metrics are mathematical definitions rather than invented entities, but they inherit the uncertainty of the heatmaps they summarize.

free parameters (3)
  • Prune/Reinit weight retention ratio = 95% of lowest-L1 weights pruned or reinitialized
    Chosen by hand; the unlearning outcome and subsequent heatmaps depend on it (Section 4.3).
  • Confuse noise scale sigma = not reported
    Controls the variance of Gaussian noise added to convolutional weights; directly affects how much attention shifts (Section 4.3).
  • Unlearning fine-tune epochs = 3 epochs
    MU models are trained for 3 epochs vs 10 for baselines, affecting all comparisons in Table 1 (Section 5).
assumptions (3)
  • domain assumption SIDU heatmaps faithfully reflect the input regions the model relies on for its predictions.
    The verification logic compares heatmap overlap with human and retained-object bounding boxes; if SIDU is unfaithful, HC and AS do not measure unlearning. Acknowledged as an open uncertainty in Section 5.3.
  • domain assumption Object detection bounding-box annotations are an appropriate ground-truth mask for the counting model's attribution.
    HC computes weighted overlap between SIDU heatmaps and bounding-box masks, but the model is trained for counting, not localization, so masks may not match its true decision regions (Section 4.5).
  • ad hoc to paper Adapting class-oriented NeurIPS 2023 challenge unlearning methods to regression by dropping entropy and weighting terms preserves their unlearning behavior.
    The paper removes regularization terms because the task is regression; whether these methods still unlearn as intended is assumed, not tested (Section 4.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Verifying Machine Unlearning with Explainable AI." pith.science (2026). https://pith.science/paper/M5QCBHIR

@misc{pith2026241113332,
  author       = {Pith},
  title        = {Pith review of: Verifying Machine Unlearning with Explainable AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5QCBHIR}},
  note         = {Machine review of arXiv:2411.13332}
}
read the original abstract

We investigate the effectiveness of Explainable AI (XAI) in verifying Machine Unlearning (MU) within the context of harbor front monitoring, focusing on data privacy and regulatory compliance. With the increasing need to adhere to privacy legislation such as the General Data Protection Regulation (GDPR), traditional methods of retraining ML models for data deletions prove impractical due to their complexity and resource demands. MU offers a solution by enabling models to selectively forget specific learned patterns without full retraining. We explore various removal techniques, including data relabeling, and model perturbation. Then, we leverage attribution-based XAI to discuss the effects of unlearning on model performance. Our proof-of-concept introduces feature importance as an innovative verification step for MU, expanding beyond traditional metrics and demonstrating techniques' ability to reduce reliance on undesired patterns. Additionally, we propose two novel XAI-based metrics, Heatmap Coverage (HC) and Attention Shift (AS), to evaluate the effectiveness of these methods. This approach not only highlights how XAI can complement MU by providing effective verification, but also sets the stage for future research to enhance their joint integration.

Figures

Figures reproduced from arXiv: 2411.13332 by the authors.

Figure 1
Figure 1. XAI and MU applied in the harbor front use-case, mitigating GDPR [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our framework uses explainability to verify unlearning. Upon a removal [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Examples from Original dataset and Relabeled dataset. The color of a [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: This figure shows heatmaps generated by SIDU for different model con [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: This figure illustrates the difference between the original heatmap and the [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Open Problems in Machine Unlearning for AI Safety

    cs.LG 2025-01 conditional novelty 4.0 of 10

    Machine unlearning is not a comprehensive solution for AI safety; it is best suited to data removal, while capability control faces fundamental and unresolved challenges.

Reference graph

Works this paper leans on

24 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    (2022), https://arxiv.org/abs/2211.14736

    Abhishek, K., Kamath, D.: Attribution-based XAI methods in Computer Vision: A Review. (2022), https://arxiv.org/abs/2211.14736

    arXiv 2022

  2. [2]

    PloS one10(7), e0130140 (2015)

    Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one10(7), e0130140 (2015)

    2015

  3. [3]

    In: 2021 IEEE Symposium on Security and Privacy (SP)

    Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C.A., Jia, H., Travers, A., Zhang, B., Lie, D., Papernot, N.: Machine unlearning. In: 2021 IEEE Symposium on Security and Privacy (SP). pp. 141–159 (2021)

    work page 2021

  4. [4]

    In: 2018 IEEE WACV

    Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In: 2018 IEEE WACV. pp. 839–847. IEEE (2018) Verifying Machine Unlearning with Explainable AI. 15

    work page 2018

  5. [5]

    NeurIPS35, 2832–2845 (2022)

    Colin, J., Fel, T., Cadène, R., Serre, T.: What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. NeurIPS35, 2832–2845 (2022)

    work page 2022

  6. [6]

    NeurIPS31 (2018)

    Dhurandhar, A., Chen, P.Y., Luss, R., Tu, C.C., Ting, P., Shanmugam, K., Das, P.: Explanations based on the missing: Towards contrastive explanations with per- tinent negatives. NeurIPS31 (2018)

    work page 2018

  7. [7]

    In: ECAI 2020, pp

    Feghahati, A., Shelton, C.R., Pazzani, M.J., Tang, K.: Cdeepex: Contrastive deep explanations. In: ECAI 2020, pp. 1143–1151. IOS Press (2020)

    work page 2020

  8. [8]

    In: 2021 IEEE/ CVF CVPR

    Golatkar, A., Achille, A., Ravichandran, A., Polito, M., Soatto, S.: Mixed-privacy forgetting in deep networks. In: 2021 IEEE/ CVF CVPR. pp. 792–801 (2021)

    work page 2021

Show all 24 references
  1. [9]

    Guo, C., Goldstein, T., Hannun, A., van der Maaten, L.: Certified data removal from machine learning models (2023), https://arxiv.org/abs/1911.03030

    2023 arXiv

  2. [10]

    Natural Language Processing Journal 7, 100078 (2024)

    Jahromi, M.N., Muddamsetty, S.M., Jarlner, A.S.S., Høgenhaug, A.M., Gammeltoft-Hansen, T., Moeslund, T.B.: Sidu-txt: An XAI Algorithm for NLP with a holistic assessment approach. Natural Language Processing Journal 7, 100078 (2024)

    2024

  3. [11]

    In: NeurIPS

    Jia, J., Liu, e.a.: Model sparsity can simplify machine unlearning. In: NeurIPS. vol. 36, pp. 51584–51605. Curran Associates, Inc. (2023)

    2023

  4. [12]

    IEEE Transactions on Image Processing 30, 5875–5888 (2021)

    Jiang, P.T., Zhang, C.B., Hou, Q., Cheng, M.M., Wei, Y.: Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing 30, 5875–5888 (2021)

    2021

  5. [13]

    AAAI32(1) (Apr 2018)

    Kemker, R., McClure, M., Abitino, A., Hayes, T., Kanan, C.: Measuring catas- trophic forgetting in neural networks. AAAI32(1) (Apr 2018)

    2018

  6. [14]

    Li, J., Ghosh, S.: Random relabeling for efficient machine unlearning (2023), https: //arxiv.org/abs/2305.12320

    2023 arXiv

  7. [15]

    Pattern recognition127, 108604 (2022)

    Muddamsetty, S.M., Jahromi, M.N., Ciontos, A.E., Fenoy, L.M., Moeslund, T.B.: Visualexplanationofblack-boxmodel:Similaritydifferenceanduniqueness(SIDU) method. Pattern recognition127, 108604 (2022)

    2022

  8. [16]

    Nguyen, T.T., Huynh, T.T., Nguyen, P.L., Liew, A.W.C., Yin, H., Nguyen, Q.V.H.: A survey of machine unlearning (2022), https://arxiv.org/abs/2209.02299

    2022 arXiv

  9. [17]

    In: NeurIPS

    Nikolov, I.A., Philipsen, M.P., Liu, J., Dueholm, J.V., Johansen, A.S., Nasrollahi, K., Moeslund, T.B.: Seasons in drift: A long-term thermal imaging dataset for studying concept drift. In: NeurIPS. vol. 36. NeurIPS Foundation (2021)

    2021

  10. [18]

    Petsiuk, V., Das, A., Saenko, K.: Rise: Randomized input sampling for explanation of black-box models (2018), https://arxiv.org/abs/1806.07421

    2018 arXiv

  11. [19]

    why should i trust you?

    Ribeiro, M.T., Singh, S., Guestrin, C.: " why should i trust you?" explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD. pp. 1135– 1144 (2016)

    2016

  12. [20]

    In- ternational journal of computer vision128, 336–359 (2020)

    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: visual explanations from deep networks via gradient-based localization. In- ternational journal of computer vision128, 336–359 (2020)

    2020

  13. [21]

    org/abs/2305.06360

    Shaik, T., Tao, X., Xie, H., Li, L., Zhu, X., Li, Q.: Exploring the landscape of machine unlearning: A comprehensive survey and taxonomy (2023), https://arxiv. org/abs/2305.06360

    2023 arXiv

  14. [22]

    Triantafillou, E., et al.: Are we making progress in unlearning? findings from the first neurips unlearning competition (2024), https://arxiv.org/abs/2406.09073

    2024 arXiv

  15. [23]

    In: IEEE/CVF CVPR-W

    Wang, H.a.e.a.: Score-cam: Score-weighted visual explanations for convolutional neural networks. In: IEEE/CVF CVPR-W. pp. 24–25 (2020)

    2020

  16. [24]

    IEEE Transactions on Emerging Topics in Computational Intelligence (2024)

    Xu, J., Wu, Z., Wang, C., Jia, X.: Machine unlearning: Solutions and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence (2024)

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.