{"id":"fbebac92-ac75-4b83-9e0b-d81398ce8ea5","arxiv_id":"2506.03117","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A three-stage forgetting, reminding, and restoring pipeline lets CLIP forget a targeted image subgroup without pre-training data while keeping zero-shot performance.","lead":"This paper presents a three-stage method that makes a CLIP model forget a specific subgroup of images, such as one breed of monkey, without using the model's original training data. The approach keeps most of the model's other abilities intact, which is useful for removing harmful or copyrighted content from AI systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The relative-Fisher layer selection (Eq. 2) is never isolated by ablation; if any reasonable layer choice performs as well, the paper's central mechanism is not load-bearing and the reported success could come from the reminding/restoring stages alone.","rationale":"The reader's conditional verdict is appropriate, and the weakest assumption they identify is exactly where the paper's argument is least secure. The method is presented as a three-stage pipeline whose novelty centers on relative-Fisher layer selection, but the paper provides no ablation that isolates this component. The reported target accuracy is near zero, retention scores are high, and unseen-dataset performance is strong, which is consistent with the claimed mechanism, but it is also consistent with a pipeline in which the layer ranking is unimportant and the reminding/restoring stages do the work. Because the authors do not report the number of selected layers, the LoRA rank, or the merge coefficient for the main experiments, the reader cannot reproduce the selection or infer its effect from the appendix. The proposed random-layer ablation is a single, decisive experiment: if random layers match the reported results, the paper's central contribution shrinks to a staged fine-tuning and model-merging recipe; if random layers fail, the relative-Fisher heuristic gains empirical support. I would keep the current conditional stance rather than accepting outright, because this validation is missing, but I would not reject the paper: the results are plausible and the missing check is straightforward to perform.","tokens_in":17306,"tokens_out":17299,"duration_ms":209297,"concrete_test":"Run the ImageNet marmoset and box-turtle experiments under the paper's exact protocol, but replace the relative-Fisher ranking with (a) a random permutation of layers, (b) all layers, and (c) the inverse ranking, keeping LoRA rank, learning rate, epochs, EMA, and model-merging coefficient fixed. If (a) or (b) match the reported target, retain, and unseen-dataset numbers within a few points, the central mechanism is not validated. Additionally, compute Eq. 2 empirically with a finite-difference Hessian on a small sample; if the ratio is zero or unstable, the paper must state that a gradient-squared empirical Fisher is used instead.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the layer-selection step in Section 4.1: it claims that fine-tuning only layers with high relative Fisher information (Eq. 2) is what enables targeted forgetting without over-forgetting. Yet no experiment isolates this choice. The forgetting stage is never ablated: no run uses random layer selection, all-layer LoRA, or an inverse ranking. As a result, the reader cannot distinguish three hypotheses: (i) relative Fisher identifies the correct layers; (ii) LoRA on any small subset works equally well because the reminding stage repairs the damage; or (iii) the success is dominated by the restoring-stage model interpolation (Eq. 6), whose coefficient is tuned on the retain set and already controls the forgetting-retention trade-off (Table 5). There is also a formal ambiguity: Eq. 2 is written as a second derivative of a cosine-similarity term. For a ReLU-based image encoder without explicit normalization, this Hessian is zero almost everywhere, so the ratio may be numerically undefined unless an empirical Fisher (gradient-squared) surrogate is used, which the paper never states. If the ranking is miscalibrated or arbitrary, the method's claimed precision, erasing only the target subgroup, has no direct support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage method for targeted subgroup forgetting in CLIP image encoders without access to the pretraining corpus. Stage one computes a per-layer relative Fisher information ratio (Eq. 2) between the forgetting set and a manually constructed retaining set, selects high-ratio layers, and applies LoRA fine-tuning to minimize cosine similarity on the forgetting set. Stage two aligns the retaining set's batch statistics to the model's stored BatchNorm statistics and fine-tunes with EMA (Eqs. 4-5). Stage three merges the unlearned model with the original model by a coefficient α tuned on a calibration set (Eq. 6). Experiments on ImageNet-Breeds, CIFAR-10, and style subsets with RN50/RN101 report target forgetting, retain accuracy, and zero-shot accuracy on Food, STL, ObjectNet, and retrieval results. The stated claim is that the method forgets specified subgroups while preserving strong zero-shot performance and outperforms six baselines.","tokens_in":17598,"tokens_out":5783,"duration_ms":64573,"significance":"The setting is timely and practical: fine-grained subgroup forgetting in CLIP without pretraining data addresses a real gap, since most baselines either need pretraining access or forget whole classes. The paper includes a broad baseline comparison, two backbones, multiple forgetting scenarios, retrieval-based checks, and ablations of the reminding and restoring stages. If the specialized layer selection is truly necessary, the contribution is significant for CLIP unlearning. However, the central mechanism is not isolated by any ablation, and several methodological ambiguities (formal definition of Eq. 2, missing hyperparameters, selection-evaluation overlap) currently prevent me from endorsing the central claim as established.","major_comments":[{"comment":"The relative Fisher information ratio is written as a Hessian of the unnormalized dot product g_img(x)·g_txt(t) with respect to layer parameters. For a ReLU-based encoder this quantity is zero almost everywhere unless the L2 normalization used in CLIP embeddings is explicitly part of the differentiated function; the paper does not state what is actually computed. In addition, no derivation or validation is provided for the claim that layers with high relative Fisher information are the ones whose LoRA adaptation erases only the target subgroup. Please specify the exact differentiable objective (including normalization), state whether an empirical Fisher (squared-gradient) surrogate is used, and justify the ranking.","section":"Section 4.1, Eq. (2)"},{"comment":"The central contribution of the method is the layer selection in Section 4.1, yet Section 5.5 ablates only the number of reminding steps and the merging coefficient α. There is no experiment with a random layer ranking, with all layers adapted, or with an inverse ranking. Without such an ablation, the reported results cannot distinguish whether relative Fisher information is responsible for the targeted forgetting or whether the reminding and restoring stages alone repair any damage from LoRA fine-tuning. This is a load-bearing gap; please add these ablations.","section":"Section 5.5"},{"comment":"For the proposed method the paper never reports the hyperparameters used in the final experiments: LoRA rank, the number/identity of selected layers, the learning rates for the forgetting and reminding stages, the EMA decay α in Eq. (5), the number of reminding steps, and the batch sizes. Table 4 varies reminding steps but the final setting is not identified, and Table 5 only reports the merging α for one run. Without these details the experiments are not reproducible and the comparison to baselines is difficult to assess. Please provide a complete configuration table.","section":"Section 5.1 and Appendix A.1"},{"comment":"The merging coefficient α is selected by maximizing accuracy on a calibration set D_m that 'may include portions of the retain dataset not used in fine-tuning,' and the reported retain accuracy Accr is measured on the retain set. If D_m overlaps with the evaluation retain set, the reported retain accuracy is optimistically biased because the same data is used for selection and evaluation. Please report the exact split and use a held-out portion of the retain set (or cross-validation) for all reported Accr values.","section":"Section 5.5, Table 5, and Eq. (6)"}],"minor_comments":[{"comment":"The RN50 'airplane' row contains the malformed entry '50.183,888.9'; please correct the formatting.","section":"Table 2"},{"comment":"The phrase 'EMMN [12],and CLIP-LIP' contains a missing space after the comma.","section":"Section 5.1"},{"comment":"The arg max is written over θ, but θ is fully determined by α; the optimization variable should be α.","section":"Eq. (6)"},{"comment":"The distribution-alignment step relies on BatchNorm statistics (Eq. 4); this is only meaningful for CNN-based CLIP backbones such as RN50/RN101. Please state explicitly that the method in its current form does not directly extend to ViT-based CLIP models, or adapt the alignment step.","section":"Section 4.2"},{"comment":"The terms 'reminding' and 'retaining' are used interchangeably for the same stage (e.g., the heading says 'reminding' while the text says 'retaining'); unify the terminology.","section":"Section 5.5"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the missing ablation of the layer-selection mechanism. If the authors can show that random or all-layer LoRA performs materially worse, the paper is publishable. Also check whether the calibration-set overlap affects the reported retain accuracy; if so, the claimed retention results may be inflated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a worthwhile paper to engage, not because the method is deeply novel (the pieces are known: LoRA, Fisher, EMA, model soup) but because it tackles a real gap: subgroup-level unlearning in CLIP without access to pre-training data. The three-stage forgetting-reminding-restoring pipeline gets surprisingly strong results across ImageNet Breeds, CIFAR-10, and style forgetting, with high retain scores and preserved zero-shot accuracy on unseen datasets. The retrieval and continuous-forgetting experiments add credibility. That's real work.\n\nThe soft spots are exactly where the reader's report points. The relative Fisher layer selection (Eq. 2) is never ablated. No experiment isolates that choice: no random layer selection, no all-layer LoRA, no inverse ranking. Without that, you can't tell if the success comes from the layer selection, the reminding stage, or the alpha-tuned model soup. The attenuation is especially important because the restoring stage directly trades off forgetting vs. retention (Table 5), so it could be doing the heavy lifting. This needs a proper ablation before the central claim is supported.\n\nThere is also a formal ambiguity: Eq. 2 is written as the expected Hessian of a cosine similarity with respect to layer parameters. For a ReLU-based encoder without normalization, that Hessian is zero almost everywhere, so the ratio is undefined as written. If the authors intend an empirical Fisher (squared gradient) surrogate, they need to say so explicitly. Right now the definition is not sound as typed.\n\nOther issues: the method's own hyperparameters (alpha, LoRA rank/learning rates, steps) are not in the main text, and the appendix only gives baseline details. The calibration set for alpha is drawn from the retain set, which introduces a mild circularity: alpha is selected to maximize retain accuracy, then retain accuracy is reported. Not fatal, but it should be acknowledged.\n\nThese are fixable concerns. The paper deserves a serious referee, but it needs major revision: add a layer-selection ablation, clarify the Fisher computation, and disclose hyperparameters. If the layer-selection step turns out not to matter, the method still works, but the mechanism description changes completely.\n\nMy take: worth a reading group slot, and I'd send it to review with a request for those experiments. I'm not citing it until the Fisher issue is resolved.\n\nBest","headline":"A useful new problem and a promising three-stage method, but the central layer-selection mechanism is never validated in isolation and the Fisher definition has a formal ambiguity.","tokens_in":18096,"tokens_out":3745,"would_cite":false,"duration_ms":38829,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that CLIP can selectively forget a specific image subgroup—such as one airplane manufacturer or one character design—without access to the pre-training data, while retaining strong zero-shot performance on other…","keywords":["machine unlearning","CLIP","subgroup forgetting","zero-shot learning","Fisher information","LoRA","model merging","vision-language models"],"falsifier":"Run the method on a superclass with four subgroups where one target subgroup's images are visually near-identical to another retained subgroup (for example, two breeds of dogs); if the relative-Fisher-selected layer fine-tuning drives the retained subgroup's accuracy below the original on that subgroup while also erasing the target, the layer-selection heuristic is over-forgetting rather than precisely targeted. The decisive test is whether the method's retain accuracy on the hardest similar subgroup stays within a few points of the original when the target is fully erased.","tokens_in":17124,"feed_emoji":"🎯","tokens_out":5323,"duration_ms":49425,"temperature":0.7,"pith_summary":"Foundation models like CLIP inherit unwanted knowledge from noisy internet training data, but pruning a single subgroup of images—say, Mickey Mouse or Boeing airplanes—while leaving the rest of the class and the model's zero-shot abilities intact has been an open problem. This paper claims a three-stage recipe solves it without access to the pre-training data: forget by fine-tuning selected layers on the target subgroup, remind by fine-tuning on a distribution-aligned retaining set with a moving average of parameters, and restore zero-shot performance by merging the unlearned model with the original. The key insight is to use a relative Fisher information ratio, comparing the layer sensitivity on the forgetting set against the retaining set, to choose which layers to modify with low-rank adapters. Experiments on ImageNet subgroups, CIFAR-10 classes, and generated image styles report that the target subgroup is erased while substantial zero-shot performance on unseen datasets is preserved, in contrast to baseline unlearning methods that collapse under this setting.","feed_headline":"CLIP can forget image subgroups without touching pretraining data","feed_subtitle":"A three-stage recipe—forget, remind, restore—erases one subgroup while keeping zero-shot accuracy on unseen datasets.","key_machinery":"The load-bearing object is the relative Fisher information ratio of Eq. (2): for each layer $\\ell$, the expected second derivative of the image-text cosine similarity with respect to that layer's parameters, evaluated on the forgetting set, divided by the same quantity on the retaining set. A high ratio marks a layer that matters more for the target subgroup than for the similar subgroup, so LoRA adapters applied only to the top-ranked layers are supposed to erase the target while leaving the model's global behaviour intact. Around this selection sit the forgetting loss (Eq. 3), which minimizes the image-text similarity of forgetting images against their coarse label; the distribution-alignment perturbation of Eq. (4) plus the EMA update of Eq. (5) in the reminding stage; and the calibration-set-optimized model soup of Eq. (6) in the restore stage.","core_discovery":"On the paper's terms, the central discovery is that fine-grained subgroup unlearning in CLIP can be achieved by three coordinated moves. First, compute, for each layer of the image encoder, the ratio of the second-order sensitivity of the CLIP contrastive loss on the forgetting set to that on a manually constructed retaining set of similar images; this relative Fisher information (Eq. 2) ranks the layers whose LoRA fine-tuning on the forgetting set (Eq. 3) erases the target subgroup while sparing similar subgroups. Second, before the reminding fine-tuning, optimize small per-image perturbations so that the retaining batch's pre-BatchNorm statistics match the stored BN statistics of the original CLIP (Eq. 4), then fine-tune with an exponential moving average of parameters (Eq. 5) to recover knowledge at risk of over-forgetting. Third, merge the unlearned model with the original by a convex combination whose coefficient is chosen on a small calibration set (Eq. 6), restoring zero-shot accuracy. The paper supports this with retrieval and classification experiments showing the target subgroup is no longer retrieved, while other subgroups, other ImageNet classes, and datasets like Food, STL, and ObjectNet retain most of their original accuracy.","pith_inferences":["The layer-selection logic is not validated against an independent measure of what a layer stores, so a natural next test is to compare relative Fisher selection against causal intervention probes that measure which layers actually control the target subgroup's embeddings.","The method leaves open whether the erased knowledge is truly gone or merely suppressed: one test is whether few-shot fine-tuning on the forgetting subgroup quickly reintroduces the target behaviour, which would imply the unlearning is shallow.","A similar three-stage recipe—select, perturb, merge—might transfer to other multimodal encoders whose BatchNorm or normalization statistics encode training-domain information, though the paper only studies CLIP."],"forward_implications":["Targeted erasure of specific visual subgroups (e.g., a trademarked character, a person's likeness, a protected attribute) becomes possible without retraining CLIP or touching its billion-scale pre-training data.","Because the forgetting, reminding, and restoring stages only need a small forgetting set and a small similar retaining set, the approach is practical for deployment-scale cleanups of foundation models.","The model-merging step composes: models that forget different classes can be merged into a single model that forgets all of them, as the paper demonstrates for ship, airplane, and cat on CIFAR-10.","If the method generalizes, unlearning becomes a post-hoc, data-efficient operation rather than a retraining burden, changing how content-removal requests can be honored for deployed vision-language models."],"supporting_citations":[{"why":"Defines the CLIP model and the vision-language embedding space that the unlearning procedure operates on.","marker":"[43]"},{"why":"Supplies the LAION-5B pre-training data whose inaccessibility motivates the no-pre-training-data constraint.","marker":"[47]"},{"why":"Provides LoRA, the low-rank adaptation mechanism applied to the selected layers during forgetting.","marker":"[25]"},{"why":"Provides model soups, the weight-averaging technique used in the restore stage to recover zero-shot performance.","marker":"[56]"},{"why":"Provides the Breeds benchmark, the ImageNet subgroup structure used for in-domain forgetting evaluations.","marker":"[45]"},{"why":"CLIP-LIP baseline, the zero-shot CLIP unlearning method that the paper compares against and extends.","marker":"[29]"},{"why":"Fisher information-based unlearning baseline and the origin of the sensitivity-based layer selection idea.","marker":"[32]"},{"why":"LIP baseline, Lipschitz-regularization unlearning, a comparison point for zero-shot unlearning.","marker":"[18]"},{"why":"EMMN baseline, an error minimization-maximization unlearning method, compared in the CLIP setting.","marker":"[12]"},{"why":"Gradient-ascent baseline, the unlearning strategy that motivates the need for a retaining set.","marker":"[52]"}],"fun_headline_variants":["Erase one image subgroup from CLIP, keep zero-shot accuracy","CLIP unlearns target subgroups, no pretraining data needed","Forget, remind, restore: CLIP drops specific image groups","Selective forgetting in CLIP: subgroups only, no original data","Targeted CLIP unlearning: erase subgroups, keep zero-shot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the relative Fisher information ratio, a second-order sensitivity measure comparing the forgetting set against the retaining set, identifies exactly the layers whose low-rank fine-tuning erases only the target subgroup's knowledge. Nothing in the paper proves this ranking is correct, so if it is wrong the method will either leave target knowledge in place or damage unrelated knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Erase one image subgroup from CLIP, keep zero-shot accuracy","CLIP unlearns target subgroups, no pretraining data needed","Forget, remind, restore: CLIP drops specific image groups","Selective forgetting in CLIP: subgroups only, no original data","Targeted CLIP unlearning: erase subgroups, keep zero-shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000691,"raw_usage":{"total_tokens":3180,"prompt_tokens":1046,"completion_tokens":2134,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":2043}},"tokens_in":662,"tokens_out":2134,"duration_ms":15704,"temperature":1.0,"reasoning_tokens":2043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:08:33.632221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the method on a superclass with four subgroups where one target subgroup's images are visually near-identical to another retained subgroup (for example, two breeds of dogs); if the relative-Fisher-selected layer fine-tuning drives the retained subgroup's accuracy below the original on that subgroup while also erasing the target, the layer-selection heuristic is over-forgetting rather than precisely targeted. The decisive test is whether the method's retain accuracy on the hardest similar subgroup stays within a few points of the original when the target is fully erased.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Defines the CLIP model and the vision-language embedding space that the unlearning procedure operates on."},{"cited_title":"Laion-5b: An open large-scale dataset for training next gener- ation image-text models.NeurIPS, 35:25278–25294, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the LAION-5B pre-training data whose inaccessibility motivates the no-pre-training-data constraint."},{"cited_title":"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infer- ence time","cited_arxiv_id":null,"evidence_quote":"Provides model soups, the weight-averaging technique used in the restore stage to recover zero-shot performance."},{"cited_title":"Namboodiri","cited_arxiv_id":null,"evidence_quote":"CLIP-LIP baseline, the zero-shot CLIP unlearning method that the paper compares against and extends."},{"cited_title":"Model spar- sity can simplify machine unlearning.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"Fisher information-based unlearning baseline and the origin of the sensitivity-based layer selection idea."},{"cited_title":"Zero-shot machine unlearning.IEEE Transactions on Information Forensics and Security, 18:2345– 2354, 2023","cited_arxiv_id":null,"evidence_quote":"EMMN baseline, an error minimization-maximization unlearning method, compared in the CLIP setting."},{"cited_title":"Unrolling sgd: Understanding factors in- fluencing machine unlearning","cited_arxiv_id":null,"evidence_quote":"Gradient-ascent baseline, the unlearning strategy that motivates the need for a retaining set."}],"review_version":1}