{"id":"6e7c0d95-329d-42eb-b581-95d89df3038b","arxiv_id":"2411.13332","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors use SIDU heatmaps and two new metrics to test whether machine unlearning removes reliance on human patterns in a thermal object-counting model.","lead":"This paper explores whether explainable AI can verify that a model has truly forgotten people in a harbor surveillance counting task. It proposes two simple heatmap-based metrics, Heatmap Coverage and Attention Shift, to check unlearning beyond accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SIDU faithfulness is the load-bearing assumption: HC and AS only verify unlearning if heatmaps track the model's true reliance on human versus non-human patterns, and the paper provides no evidence of this.","rationale":"The reader's weakest assumption and my stress-test converge: the paper's verification claim rests on SIDU heatmaps faithfully reflecting the model's reliance on human versus non-human patterns. The authors themselves flag this in Section 5.3, but no calibration or sanity check is provided. A concrete perturbation test would settle whether HC and AS measure unlearning or merely artifacts of the explanation method. I also considered the alternative concern that the comparison between unlearned models and Retrain is confounded by unequal training budgets and lack of error bars; that is real, but it is secondary because even a perfectly matched comparison would not establish the central claim unless the XAI metric itself is faithful. Since the paper already conditions its conclusions on this assumption and the reader's verdict is CONDITIONAL, my review does not move the verdict; it strengthens the specific condition that must be met before the method can be relied upon.","tokens_in":10012,"tokens_out":4988,"duration_ms":59740,"concrete_test":"Run a deletion/insertion faithfulness test on the actual models: for a held-out set, progressively occlude the top-SIDU-attributed image regions and measure the change in predicted count, comparing against random and lowest-attributed occlusion baselines; also insert the top-attributed patches and measure count recovery. If SIDU's deletion AUC is not significantly worse than random and insertion AUC not significantly better, SIDU is not faithful on this task and HC/AS cannot verify unlearning. Run over multiple seeds and report confidence intervals. Additionally, on synthetic images with known human and non-human objects, check that h-HC tracks true human reliance, e.g., by ablating humans from the input and confirming that the model output and heatmap importance both drop only when humans are causally used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The argument chain is: MU success equals reduced reliance on human patterns; SIDU heatmaps reveal that reliance; HC and AS quantify it. The load-bearing link is that SIDU's heatmap H accurately reflects the model's causal dependence on image regions. The paper explicitly concedes in Section 5.3 that 'the assumption that the XAI methods used for heatmap generation accurately reflect important regions in the image ... adds a layer of uncertainty.' No experiment tests that assumption on this counting model. Without such a test, an alternative explanation for the Table 1 pattern is that HC and AS respond to SIDU artifacts, such as low-resolution feature-map upsampling, channel-normalization choices, or bounding-box context correlations, rather than to genuine unlearning. The problem is compounded by using detection bounding boxes as the ground-truth 'desired target' mask: for a counting task, context outside the box can legitimately drive counts, so a low h-HC could reflect either successful forgetting or merely a shift in where the explanation method places credit. The reported numbers are single-run point estimates with no variance, and the model architecture is unspecified, which matters because SIDU's last-convolutional-layer heatmaps are architecture-dependent. The central conclusion is therefore conditional on a faithfulness property that is asserted, not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using attribution-based XAI, specifically the SIDU method, to verify machine unlearning in a real-world regression task: counting objects (excluding humans) in thermal harbor-front imagery. The authors define two new metrics, Heatmap Coverage (HC) and Attention Shift (AS), which quantify how much of the SIDU heatmap overlaps with object bounding boxes and how much the heatmap changes after unlearning. They compare four unlearning methods (Fine-tune, Prune, Reinit, Confuse) against a retrained-from-scratch baseline and report that unlearning methods concentrate heatmap intensity on retained objects more than full retraining, while also reducing attention on humans. The paper concludes that attribution-based XAI can effectively verify unlearning, and the authors release code and qualitative visualizations.","tokens_in":10299,"tokens_out":4984,"duration_ms":53995,"significance":"If the central claim is validated, the paper offers a computationally cheaper and more interpretable alternative to attack-based unlearning verification, extends MU evaluation to regression tasks, and contributes two reusable metrics. The authors are honest about the key assumption in Section 5.3. However, the significance is conditional: HC and AS only measure unlearning if SIDU heatmaps faithfully reflect the model's true causal reliance on image regions. No experiment in the paper tests this load-bearing assumption, and the quantitative evidence is limited to single-run point estimates. The novelty is moderate, as the paper applies an existing XAI method to a new verification purpose, but the framework is clearly presented and the code release is a strength.","major_comments":[{"comment":"The entire verification argument rests on the assumption that SIDU heatmaps accurately reflect the model's reliance on human versus non-human patterns. The paper explicitly concedes this in Section 5.3, but no experiment tests it. Without such validation, an alternative reading of Table 1 is that HC and AS respond to SIDU artifacts (e.g., low-resolution upsampling, channel-normalization choices) rather than to genuine unlearning. The authors should add a faithfulness check for SIDU on this specific counting model, for example by comparing SIDU attributions to the effect of occluding or perturbing human versus non-human regions on the model's count prediction, or by using standard insertion/deletion metrics, or by comparing SIDU with at least one other attribution method. This is necessary to support the claim that the proposed metrics verify unlearning rather than merely explainer behavior.","section":"Section 5.3, Eqs. (1)-(2)"},{"comment":"The HC metric defines the ground-truth mask M using object-detection bounding boxes. For a global counting regression, context outside a bounding box can legitimately influence the count (e.g., a bicycle's presence is correlated with a nearby human, and empty background can provide scale cues). A low h-HC after unlearning may therefore reflect the explainer shifting credit to context rather than the model actually forgetting humans. The authors should justify, or empirically test, the assumption that bounding boxes are the correct ground truth for where a counting model's attention should be concentrated. At minimum, this assumption should be stated as a limitation and discussed in Section 5.3.","section":"Eq. (1), Section 4.5"},{"comment":"All metrics in Table 1 are reported as single values with no standard deviations, no multiple seeds, and no significance tests. Differences used to rank unlearning methods, such as r-HC 7.632e-3 for Confuse versus 6.827e-3 for Fine-tune, are small and could easily be within run-to-run noise. The qualitative ordering of unlearning methods by HC/AS is therefore not statistically established. The authors should provide results over multiple random seeds with error bars or at least report the number of seeds and the observed variance.","section":"Table 1, Section 5.1"},{"comment":"The model architecture is never specified. Since SIDU generates heatmaps from the last convolutional layer, the resulting HC and AS values depend critically on the architecture's feature-map resolution and layer semantics. Without specifying the network (e.g., backbone, depth, number of parameters, input resolution), the results are not reproducible and the claimed behavior cannot be generalized. The authors should specify the architecture and, ideally, test at least one other architecture to show the verification method is not architecture-specific.","section":"Section 5, Section 4.4"}],"minor_comments":[{"comment":"The adaptation of Prune, Reinit, and Confuse to regression drops the entropy and weighting terms from the NeurIPS 2023 challenge methods. The paper asserts this preserves unlearning behavior but provides no evidence. A brief ablation or justification would strengthen the claim that these are faithful adaptations.","section":"Section 4.3"},{"comment":"The arrow notation in the table header appears inconsistent with the text: the text states that lower h-HC values indicate that the model is 'less weighted around human bounding boxes', yet the header shows 'h-HC ↑' if read literally. The arrows should be corrected to match the intended direction of improvement (r-HC ↑, h-HC ↓) or removed.","section":"Table 1"},{"comment":"The sentence 'Our experiments have shown that SIDU not only performs well...' reports prior results without a reference or experimental detail. If this refers to the authors' earlier work, cite it explicitly; otherwise, remove the claim.","section":"Section 4.4"},{"comment":"There are several typos and grammatical errors, including 'quantitavely' in Section 4 and 'using to attribution-based XAI' in the Conclusion. These should be corrected in a revision.","section":"Abstract and Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper relies exclusively on SIDU, an attribution method developed by several of the same authors. While this is not an issue by itself, the absence of any comparison with other XAI methods increases the risk that the proposed HC and AS metrics are tailored to SIDU's specific artifacts. I would encourage the editor to require an independent faithfulness evaluation before considering the paper for publication. The current single-run experiments and unspecified architecture are also below the standard expected for a verification claim. The idea is promising, but the evidence as presented is not sufficient to support the central conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper does something genuinely new—using attribution heatmaps to check whether unlearning removed a pattern—and it ships code. But the quantitative support is thin enough that I'd treat the central claim as plausible rather than established.\n\nThe new part: SIDU heatmaps are used to compare a model that counts humans plus other objects against models unlearned via fine-tune, prune/reinit, and noise injection, and you get two metrics, HC and AS. That's a reasonable proof-of-concept, and the authors are upfront in Section 5.3 that the whole apparatus rests on the assumption the heatmaps faithfully reflect what the model relies on. Credit where due: they don't hide the weakness.\n\nWhere it's soft: Table 1 gives one number per model per metric, no variance, no seeds, no significance tests. The r-HC differences that drive the conclusions are on the order of 1e-3, which could easily be noise. The model architecture is not specified, which matters because SIDU's heatmaps come from the last convolutional layer. And the bounding-box masks are a questionable ground truth for a counting task: context outside the box can legitimately drive the count, so low h-HC might just mean the explanation method moved credit, not that the model forgot humans. The paper doesn't test SIDU's faithfulness on this model against any ground truth, so an alternative explanation for the heatmap shifts is that they reflect artifacts of the explanation method rather than genuine forgetting.\n\nAlso, the two new metrics are simple compositions of heatmaps and masks; that's fine for a proof of concept, but they aren't validated against any external criterion.\n\nNone of this kills the idea. The paper is an honest, clearly written proof-of-concept, and the GitHub link is a plus. The central claim—that attribution-based XAI can serve as a qualitative verification step for unlearning—is credible but not established by the current evidence.\n\nThis is for people working at the XAI/MU intersection, or who need a lightweight check for pattern removal. It deserves a serious refereeing pass, but the referee should push for multiple seeds, error bars, a specified architecture, and some sanity check that the heatmaps track the model's causal reliance on regions.\n\nRecommendation: accept for review with the expectation of major revisions.","headline":"A plausible but thinly evidenced proof-of-concept for using XAI heatmaps to verify unlearning; worth a serious referee, but the numbers aren't yet convincing.","tokens_in":10832,"tokens_out":4747,"would_cite":false,"duration_ms":49510,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that attribution-based XAI heatmaps can verify whether machine unlearning has actually removed a sensitive pattern, using SIDU heatmaps and two new metrics, HC and AS, on a harbor-front object counting task.","keywords":["Machine unlearning","Explainable AI","Right to be forgotten","Object counting","Heatmap coverage","Attention shift","SIDU","Privacy compliance"],"falsifier":"Run the same unlearning pipeline and verify with two independent attribution methods on identical images, for example SIDU and a gradient-based method such as Grad-CAM. If one method says the unlearned model still attends to humans while the other says it does not, the claim that heatmaps can verify unlearning is unsupported. A complementary test would ablate human regions from the input: a model that has truly unlearned humans should show almost no drop in predicted count when those regions are removed.","tokens_in":9835,"feed_emoji":"🔍","tokens_out":7512,"duration_ms":71942,"temperature":0.7,"pith_summary":"This paper tries to establish that heatmap explanations can verify whether a machine-learning model has genuinely unlearned a sensitive pattern, rather than merely recovering its accuracy. The motivation is practical: privacy legislation creates a right to be forgotten, and retraining from scratch is expensive, so unlearning methods need a cheap way to certify that forbidden patterns no longer influence predictions. On harbor-front thermal imagery, the authors show that SIDU heatmaps and two proposed metrics, Heatmap Coverage (HC) and Attention Shift (AS), can distinguish models that still attend to humans from models that do not. If correct, this gives regulators and practitioners a visual, quantitative check for unlearning that goes beyond accuracy.","feed_headline":"Heatmaps can verify that unlearning erased people","feed_subtitle":"In harbor counting tests, SIDU heatmaps show unlearned models still focus on humans even when accuracy looks fine.","key_machinery":"The load-bearing tool is SIDU (Similarity Difference and Uniqueness), a gradient-free attribution method that extracts feature maps from the last convolutional layer and forms masks from similarity-difference and uniqueness scores to produce a heatmap of the image regions driving the prediction. On top of those heatmaps, the paper defines $\\mathrm{HC}(f_i)$, the fraction of heatmap intensity falling inside object-detection bounding boxes (Equation 1), and $\\mathrm{AS}(f_{u_i}, f_o)$, the standard deviation of the pixel-wise difference between the original and unlearned heatmaps (Equation 2). HC measures whether attention lands on the objects meant to be retained, and AS measures how much attention has moved after unlearning.","core_discovery":"The paper claims that attribution-based XAI can serve as a verification layer for machine unlearning. In a thermal-imaging regression task where a model counts people, bicycles, vehicles, and motorcycles, the authors remove humans from the label and compare a freshly retrained baseline to four unlearned models obtained by fine-tuning, pruning, reinitializing, and noise injection. Using SIDU heatmaps and the two proposed metrics, they find that the unlearned models concentrate heatmap intensity on retained objects more than the retrained baseline does, while plain fine-tuning leaves clear human-related heatmap traces. The proposed metrics therefore reveal differences in unlearning quality that MAE and RMSE do not capture.","pith_inferences":["Extension: The HC metric assumes the annotated bounding boxes are the only legitimate focus, yet a counting model could plausibly use context near an object; under that behavior, HC would penalize correct reasoning.","Extension: Because the paper only inspects post-hoc heatmaps, its own evidence suggests an end-to-end training loop in which HC or AS enters the loss and actively pushes attention off forbidden regions; the paper lists this as future work.","Extension: The same pipeline could be tested on established unlearning benchmark tasks where membership-inference attack scores provide an independent ground truth, giving a direct comparison between attribution-based verification and attack-based verification.","Extension: The paper's comparison is limited to one dataset and one attribution method, so the natural next check is whether the ordering of unlearning methods by h-HC is stable across architectures, datasets, and XAI methods."],"forward_implications":["Heatmap-based metrics like HC and AS can be added to accuracy as a practical verification step when regulators request that a model forget a protected pattern.","Unlearning methods such as pruning and noise injection can shift attention away from sensitive objects more strongly than plain fine-tuning, even when error metrics look similar.","Because the metrics only require bounding-box annotations and heatmaps, the same verification recipe transfers to other counting or detection tasks where object locations are known.","Verification of unlearning no longer needs to rely solely on expensive attack-based benchmarks or full retraining for comparison."],"supporting_citations":[{"why":"Supplies the SIDU attribution method that generates the heatmaps on which the proposed verification metrics are built.","marker":"[15]"},{"why":"Supplies the long-term thermal-imaging dataset with human, bicycle, vehicle, and motorcycle annotations used for the counting and unlearning experiments.","marker":"[17]"},{"why":"Provides the unlearning baseline methods (pruning, reinitialization, and noise injection) that the paper adapts from the competition setting to regression.","marker":"[22]"},{"why":"Establishes the taxonomy of unlearning methods and the gap in verification metrics that the paper aims to fill with XAI-based measures.","marker":"[16]"},{"why":"Provides the SISA data-partitioning unlearning context that motivates the need for lightweight alternatives to full retraining.","marker":"[3]"}],"fun_headline_variants":["Heatmaps expose unlearning gaps","Explainable AI verifies unlearning","Unlearning checked by attention shifts","Did unlearning work? Heatmaps answer","Attribution maps reveal unlearning flaws"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The verification holds only if SIDU heatmaps faithfully reflect the image regions the model actually uses, and only if the annotated bounding boxes are the correct ground truth for where a counting model's attention should lie.","fun_headline_variants_meta":{"raw":{"variants":["Heatmaps expose unlearning gaps","Explainable AI verifies unlearning","Unlearning checked by attention shifts","Did unlearning work? Heatmaps answer","Attribution maps reveal unlearning flaws"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000351,"raw_usage":{"total_tokens":1877,"prompt_tokens":870,"completion_tokens":1007,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":947}},"tokens_in":486,"tokens_out":1007,"duration_ms":10951,"temperature":1.0,"reasoning_tokens":947,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:32:18.133064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same unlearning pipeline and verify with two independent attribution methods on identical images, for example SIDU and a gradient-based method such as Grad-CAM. If one method says the unlearned model still attends to humans while the other says it does not, the claim that heatmaps can verify unlearning is unsupported. A complementary test would ablate human regions from the input: a model that has truly unlearned humans should show almost no drop in predicted count when those regions are removed.","supporting_citations":[{"cited_title":"Pattern recognition127, 108604 (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the SIDU attribution method that generates the heatmaps on which the proposed verification metrics are built."},{"cited_title":"In: NeurIPS","cited_arxiv_id":null,"evidence_quote":"Supplies the long-term thermal-imaging dataset with human, bicycle, vehicle, and motorcycle annotations used for the counting and unlearning experiments."},{"cited_title":"In: 2021 IEEE Symposium on Security and Privacy (SP)","cited_arxiv_id":null,"evidence_quote":"Provides the SISA data-partitioning unlearning context that motivates the need for lightweight alternatives to full retraining."}],"review_version":1}