{"id":"b44f530f-6da3-451e-9779-6edf7f46705a","arxiv_id":"2608.00246","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A retain-aware localization method for machine unlearning reduces reported collateral damage on CIFAR-10, but its evaluation metric is contaminated and the 'consistent' improvement claim is not supported by the paper's own table.","lead":"This paper proposes a machine-unlearning method that picks which model parameters to reset by looking at both the forgotten examples and the retained examples, aiming to avoid damaging similar kept data. It also introduces a new evaluation set to measure that collateral damage, but the experiments have serious measurement flaws.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'consistent improvement' claim is contradicted by Table 1: 4/10 variants have Δrs above baseline, ratio variants under-forget, and only the highlighted difference condition supports the claim.","rationale":"The reader's weakest assumption was that Dr,sim being a subset of Dr invalidates Δrs as a collateral-damage measure. I see that as secondary: in unlearning, retain-set utility is evaluated on retain training data, and the oracle is also trained on Dr, so evaluating on a subset of Dr is not inherently circular. The more load-bearing problem is internal inconsistency: the paper's own Table 1 does not support 'consistent' improvement. Only the difference variants achieve lower Δrs together with acceptable forgetting; weighted variants often increase Δrs, and ratio variants under-forget. The subsampled difference result is promising, but without error bars, multiple seeds, or a pre-specified selection rule, the headline rests on one favorable configuration. The paper's own discussion acknowledges the result is concentrated in 'the subsampled difference method,' which reads as post-hoc selection. Therefore the current evidence does not substantiate the central claim, and the REJECT verdict is appropriate. A multi-seed replication could allow a revised, more limited claim about the difference method specifically.","tokens_in":13501,"tokens_out":10523,"duration_ms":100511,"concrete_test":"Run each of the eleven Table 1 conditions with 10 random seeds (varying model initialization and the forget/retain split seed) and report mean and 95% CI for Δtest, Δretain, Δforget, Δrs, ΔMIA, and KL. Then test two claims: (i) the subsampled difference method's Δrs is significantly below the forget-only baseline's Δrs; (ii) at least 6 of the 10 proposed variants have Δrs below baseline while not worsening Δforget or KL relative to baseline. If either fails, 'consistently reduces collateral damage while improving standard unlearning metrics' is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—'consistently reduces collateral damage while improving standard unlearning metrics'—is contradicted by its own Table 1. The forget-only baseline has Δrs = 0.31. Among the ten proposed variants, four have larger Δrs (w_diff sub w=3: 0.35; w_diff non-sub w=5: 0.33; w_diff sub w=7: 0.33; w_diff non-sub w=7: 0.44). The ratio variants achieve near-zero Δrs but under-forget (Δforget = −3.86/−3.74 vs baseline +2.70) and have KL above baseline (0.479/0.482 vs 0.443), so they do not improve standard unlearning metrics. Only the two difference variants satisfy both lower collateral damage and acceptable forgetting. No error bars or repeated-seed results are reported, and the text itself singles out 'the subsampled difference method' as the notable result, implying post-hoc selection. The reader's concern that Dr,sim is a subset of the fine-tuning set is secondary: retain-set utility is legitimately evaluated on retain training data, and the oracle is also trained on Dr. But even taking Δrs at face value, the across-variant inconsistency defeats the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retain-aware localization method for machine unlearning. Unlike existing localization approaches that select parameters using forget-set importance only, it computes criticality scores on both the forget set and the retain set and combines them via difference, weighted-difference, or ratio scoring. A localization mask selects a 30% parameter budget, selected weights are reinitialized, and the model is fine-tuned on the retain set. The paper also introduces a retain-similar evaluation set Dr,sim, consisting of the nearest same-class retain-set neighbours of forget examples in the original model's embedding space. Experiments on CIFAR-10/ResNet-18 report that the proposed method reduces collateral damage while improving standard unlearning metrics compared to the forget-only DEL baseline.","tokens_in":13824,"tokens_out":7066,"duration_ms":72977,"significance":"If the results held, the retain-aware localization idea would be a useful extension of localization-based unlearning: it targets a real problem (feature entanglement between forget and retain data), has negligible computational overhead, and the retain-similar evaluation idea is worth exploring. The paper is clearly written, uses a public dataset and architecture, and reports wall-clock times and some sensitivity analysis. However, the headline claim of consistent improvement is contradicted by the paper's own Table 1, and the primary collateral-damage metric is constructed from the fine-tuning set, so the empirical evidence does not support the stated conclusions.","major_comments":[{"comment":"The abstract states that the method 'consistently reduces collateral damage while improving standard unlearning metrics' across eleven settings. Table 1 does not support this. For Δrs, four proposed variants exceed the forget_only baseline: w_diff/Yes/w=3 (0.35), w_diff/No/w=5 (0.33), w_diff/Yes/w=7 (0.33), and w_diff/No/w=7 (0.44), all above the baseline's 0.31. The two ratio variants have Δforget = −3.86/−3.74, indicating severe under-forgetting relative to the oracle, and KL divergence 0.479/0.482, above the baseline's 0.443. Only the two difference variants show both lower Δrs and acceptable forgetting. The paper itself acknowledges the ratio failure in §4.4. Thus the 'consistent' claim is post-hoc selection of the subsampled difference variant, not a general property of the proposed family.","section":"§4.4, Table 1"},{"comment":"Dr,sim is constructed as the nearest same-class retain-set neighbours under the original model's embeddings and is therefore a subset of Dr, the fine-tuning set. Consequently Δrs measures how accurately the unlearned model recovers specific training examples after fine-tuning, not damage to similar held-out examples. The metric may be artificially favourable because both the unlearned and oracle models are trained on the exact Dr,sim examples. To directly measure collateral damage, the evaluation set should use held-out similar examples (e.g., nearest test-set neighbours). The paper's assertion that Δrs 'directly measures collateral damage' is therefore not established.","section":"§4.1, Eq. (10)"},{"comment":"All results are single-run point estimates with no error bars, confidence intervals, or repeated-seed experiments. Several differences used to support the headline claim are small (e.g., Δrs 0.04 vs 0.11; Δretain 0.11 vs 0.20; KL 0.395 vs 0.413). Without variance information, it is impossible to assess whether these differences are meaningful. The lack of multiple runs is especially problematic because the paper's central claim is about consistent improvement across conditions.","section":"§4.4, Table 1"},{"comment":"The oracle model is trained for 35 epochs while the original model is trained for 50 epochs. Since the oracle is the reference for all deltas, this asymmetric training schedule may systematically bias the results: the oracle is less converged, which could inflate positive deltas. In particular, the oracle reports 100% retain accuracy despite being trained for fewer epochs, which is surprising and should be justified. The paper should either train the oracle for the same number of epochs (or use validation-based early stopping) and show sensitivity to this choice.","section":"§4.2 and §4.4"}],"minor_comments":[{"comment":"The text says that sim(xf, xr) is computed by (4), but Definition 4 is Equation (3). The equation reference should be corrected.","section":"§4.1, Eq. (10)"},{"comment":"Definition 6 says the gradient g(θo_j, D) is computed over the forget set, but the method later computes criticality scores on the retain set as well. Clarify that Eq. (5) applies to any dataset D.","section":"§3.1"},{"comment":"The phrase 'non-sampling is better at w=3' uses 'non-sampling' where 'non-subsampled' is meant. Please fix this wording.","section":"§4.4"},{"comment":"The table caption says 'Bold letters indicate the best results,' but no bold formatting is visible in the table. Either render the bold in the final version or remove the caption sentence.","section":"Table 1"},{"comment":"The conclusion states that 'these results demonstrate that retain-aware localization preserves shared representations more effectively than forget-only localization,' but this is only shown for the subsampled difference variant, and with a single run. Please temper the wording to match the evidence.","section":"§4.4 and Conclusion"}],"recommendation":"reject","confidential_remarks":"The core issue is not the lack of novelty but the unsupported central claim: Table 1 contradicts the 'consistent improvement' statement, and the primary metric Dr,sim is confounded because it is a subset of the fine-tuning set. A revision that restricts the claim to the subsampled difference method, adds multi-seed and held-out evaluation, and compares against stronger baselines could produce a viable resubmission, but the current manuscript does not establish its stated contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the quick read. The paper has a real idea worth keeping: when you localize unlearning to a parameter subset, you should look at importance to the retain set, not just the forget set. The difference scoring scheme makes sense, and the retain-similar evaluation set is a reasonable attempt to measure what people in this area actually worry about. The subsampled difference result is genuinely encouraging on the numbers: all retain-side deltas drop to near zero while forgetting stays close to oracle accuracy.\n\nThe problems are in the packaging. The abstract and the discussion keep saying \"consistently reduces collateral damage,\" which is not what Table 1 shows. Four of the ten proposed variants have higher Δrs than the forget-only baseline, and the ratio variants under-forget badly. The only condition that cleanly supports the story is the subsampled difference, and the report singles it out after the fact. That is fine in an exploratory paper, but not as a blanket claim.\n\nThe empirics are also thin: one dataset (CIFAR-10), one model (ResNet-18), and no repeated seeds or error bars. For unlearning, where variance across runs is usually meaningful, that is a real limitation. The Dr,sim set is built from the same retain training set used for fine-tuning, so Δrs tells you about recovery of training examples, not about generalization to similar held-out data. That makes it a valid memorization/recovery metric, but the authors should not call it \"collateral damage\" in the broader sense without a held-out variant.\n\nI also note the oracle is trained for 35 epochs versus 50 for the original. That may be fine (oracle accuracy is 85.91%, close to typical), but it should be justified. And the fine-tuning epoch count isn't reported except \"same as [24].\" That is a reproducibility gap.\n\nWhat is genuinely good: the paper ablates subsampling, the weight parameter, and fine-tuning size; it reports runtimes; and it is honest that the ratio method fails. Those are signs of careful work, not sloppiness.\n\nBottom line: the core idea is worth a serious referee and likely worth retaining in the literature, but the paper overclaims. It deserves major revision with seeds, error bars, a toned-down abstract, and ideally a held-out similar test set.","headline":"Retain-aware localization is a plausible idea, but the 'consistent' claim does not survive its own Table 1, and the evidence is one run on one dataset.","tokens_in":14321,"tokens_out":3872,"would_cite":true,"duration_ms":38756,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Localization-based machine unlearning that scores parameters against both the forget set and the retain set reduces collateral damage to similar retained examples, cutting the retain-similar accuracy gap from 0.49 to 0.04 on CIFAR-10/ResNet","keywords":["machine unlearning","localized unlearning","retain-aware localization","collateral damage","feature entanglement","criticality score","CIFAR-10","ResNet-18"],"falsifier":"Repeat the protocol on CIFAR-10/ResNet-18 but construct the retain-similar set from the held-out test set (nearest test example to each forget example) instead of the retain training set; if the difference method shows no improvement in retain-similar accuracy over the forget-only baseline on that test-derived set, the collateral-damage claim is limited to re-fitting the training set. Additionally, retrain the oracle with 50 epochs to see whether the reported gaps still hold.","tokens_in":13385,"feed_emoji":"🧠","tokens_out":8151,"duration_ms":75974,"temperature":0.7,"texified_at":"2026-08-05T21:57:13.435632+00:00","pith_summary":"Machine unlearning methods that localize edits to the most 'forget-important' parameters can degrade retained examples that are similar to the forgotten ones. This paper claims the fix is to make localization 'retain-aware': compute criticality scores on both the forget set and the retain set, then select parameters that are important to forgetting but not to retention. On CIFAR-10 with ResNet-18, the subsampled difference variant reduces the retain-similar accuracy gap from 0.49 to 0.04 while improving other unlearning metrics, with negligible added cost. If correct, this means similarity-aware parameter selection can make approximate unlearning considerably less destructive for semantically related data.","texify_model":"deepseek-v4-flash","texify_usage":{"total_tokens":4830,"prompt_tokens":716,"completion_tokens":4114,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":716,"completion_tokens_details":{"reasoning_tokens":3484}},"feed_headline":"Unlearning that scores retained data cuts collateral damage to 1/12","feed_subtitle":"Scoring both forget and retain sets picks neurons that forget without breaking similar retained data on CIFAR-10.","key_machinery":"The central object is the criticality score $s_j = |\\theta_j \\cdot g(\\theta_j, D)|$, which measures how strongly a parameter's current value participates in the loss over a dataset D. The paper computes it on the forget set and on the retain set, then combines the two via three rules; the operative one is the difference rule $\\tilde{s}_j = \\max(s_{f_j} - s_{r_j}, 0)$, which zeros out any neuron whose retain importance matches or exceeds its forget importance. A binary mask selects the top-30% of parameters by this score, re-initializes them, and fine-tunes only that subset on the retain set. The second mechanism is the retain-similar set $D_{r,\\text{sim}}$: for each forget example, its same-class nearest retain neighbor by co","core_discovery":"Working within the localized machine unlearning paradigm, the paper identifies feature entanglement as the cause of collateral damage: forget and retain samples that are close in the model's embedding space activate overlapping neurons, so resetting forget-important channels can also erase representations needed by similar retained examples. To counter this, the authors assign each parameter a criticality score computed separately on the forget and retain sets and combine them—most simply by subtracting retain importance from forget importance and clamping at zero—so that shared neurons are excluded from the reset mask. The paper's headline result is that this difference-based localization,","pith_inferences":["The Dr,sim construction—nearest same-class neighbor from the retain training set—is specific to training data; an analogous 'test-similar' set built from held-out examples would test whether the collateral damage reduction generalizes outside the training distribution.","The subsampling-vs-full-retain result invites a sharper version: sample the retain set by similarity to the forget set rather than uniformly, potentially aligning the mask even better with the forget-similar boundary.","The difference rule effectively implements per-neuron 'specificity' filtering; a natural extension is to combine it with per-layer or per-block budget allocation to trade off utility per semantic region rather than globally.","A concrete downstream test: applying this localization before membership-inference attacks on larger models (e.g., transformers) would show whether embedding-space entanglement is a general failure mode of localization-based unlearning."],"forward_implications":["Retain-aware localization consistently reduces collateral damage on Dr,sim compared with forget-only localization across all 11 experimental settings.","The subsampled difference method improves standard unlearning metrics (Δtest, Δretain, Δforget, ΔMIA, KL) over the baseline, not just the new retain-similar metric.","Subsampling the retain set for scoring is both cheaper (2 seconds vs 6 seconds) and empirically better than using the full retain set for the difference rule.","The ratio method, which lacks an absolute forget-importance threshold, fails to unlearn (Δforget = -3.86), implying that retain-aware scoring must retain a forget-side component to avoid under-intervention.","Because this holds only on CIFAR-10/ResNet-18, the paper explicitly leaves generalization to larger datasets and architectures as future work."],"fun_headline_variants":["Unlearn while sparing similar retained data","Retain-aware unlearning cuts collateral damage","Score both forget and retain to unlearn better","Similarity-aware unlearning protects retained data","Unlearning that keeps similar data intact"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The retain-similar evaluation set is built from retain training examples, so accuracy on it is partly a training-fit measure, and the oracle model used as the gold standard was trained with fewer epochs (35 vs 50) than the original model.","fun_headline_variants_meta":{"raw":{"variants":["Unlearn while sparing similar retained data","Retain-aware unlearning cuts collateral damage","Score both forget and retain to unlearn better","Similarity-aware unlearning protects retained data","Unlearning that keeps similar data intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"total_tokens":917,"prompt_tokens":636,"completion_tokens":281,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":380,"completion_tokens_details":{"reasoning_tokens":216}},"tokens_in":380,"tokens_out":281,"duration_ms":19819,"temperature":1.0,"reasoning_tokens":216,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T00:54:22.431685+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the protocol on CIFAR-10/ResNet-18 but construct the retain-similar set from the held-out test set (nearest test example to each forget example) instead of the retain training set; if the difference method shows no improvement in retain-similar accuracy over the forget-only baseline on that test-derived set, the collateral-damage claim is limited to re-fitting the training set. Additionally, retrain the oracle with 50 epochs to see whether the reported gaps still hold.","supporting_citations":[],"review_version":1}