{"id":"87e8726f-f804-44c2-bc09-f9d8fd26cce7","arxiv_id":"2506.02761","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new taxonomy and multi-aspect evaluation framework for image generation unlearning, with a curated dataset, shows that ten existing unlearning methods perform poorly on preservation and robustness.","lead":"This paper argues that current methods for removing concepts from image generators are evaluated with unreliable metrics, and proposes a new task taxonomy, a five-aspect evaluation framework, and a dataset to fix this. The authors test ten unlearning methods and find they often forget too little, damage unrelated content, or can be tricked into regenerating the target.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EvalIGMU's preservation metrics are not calibrated: CSDR and LPIPS compare against original-model outputs without a within-model baseline or oracle check, so 'methods fail at preservation' may be an artifact of normal generative variance or of successful erasure itself.","rationale":"The reader's weakest_assumption correctly identifies the preservation reference as the load-bearing soft spot: CSDR and LPIPS take the original model's output distribution as ground truth for what should be retained. My reading sharpens this into a testable internal-consistency concern. The paper's own R1 requires M^u(t) ∩ M(t) = ∅, so any genuinely unlearned output must differ from the original model's output for the target prompt; if the preservation metric is computed against that same prompt, it will penalize successful forgetting. Even under the more charitable reading that p is the non-target prompt, LPIPS still lacks a within-model baseline, so the magnitude of the reported preservation failures is uninterpretable without knowing how much LPIPS/CSDR varies across independent samples from the original model alone. The paper's Table 4 'Expectation' outputs are concrete instances of successful unlearning that would be scored as preservation failures under the current formulas. This does not undermine the taxonomy, dataset, or the detector-reliability analysis, which are useful contributions. It does mean the headline benchmark conclusion about preservation is not yet supported, exactly as the reader's CONDITIONAL verdict requires. I also note that Lemma 1 in Sec 5.2 is asserted without proof despite being labeled a lemma, and Sec 5.3.1 already admits the keyword-based threat model limitation; these are secondary to the preservation metrics issue. With an oracle-based calibration check, the authors could either validate the metric or reframe the central claim; until then, the conditional verdict stands.","tokens_in":27592,"tokens_out":7575,"duration_ms":85106,"concrete_test":"Construct an oracle unlearned model per the paper's own Expectation guidelines (e.g., for the Nudity task, take M's 'naked girl' images and inpaint clothing while preserving the scene; for Van Gogh, re-render M's images in a non-Van-Gogh style). Run EvalIGMU's CSDR and LPIPS on this oracle versus the original M outputs, and compare with the ten methods in Fig 8. If the oracle's preservation scores are no better than the benchmarked methods, the preservation metric cannot distinguish successful from failed unlearning; if they are markedly better, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that existing IGMU methods 'cannot handle the unlearning well ... especially for preservation'—rests on the preservation metrics in Sec 5.3.1. Two problems make this claim unverified. First, the CSDR formula compares CLIP scores of M^u(t) and M(p) against the same prompt p. If p is the target prompt (the notation t=S⊕t in Sec 3.2 suggests this), then a successful unlearning that removes the target inevitably lowers CLIP alignment with p and raises LPIPS to M(p), so the metric penalizes exactly what R1 requires. If p is instead the non-target prompt, the metric still lacks a baseline: LPIPS between two independent samples from the same model and prompt is nonzero, and the paper reports no such floor, so large LPIPS between M^u(t) and M(p) may just reflect normal stochasticity. Second, Figure 8 reports normalized averages with no error bars or significance tests, and the paper's own Table 4 ideal outputs ('a clothed girl playing on a beach') would score as large CSDR/LPIPS deviations from the original 'naked girl' image. The paper also flags in Sec 5.3.1 that its threat model is keyword-based, but that limitation is narrower than the preservation reference problem. Because the headline 'especially for preservation' is the paper's most consequential finding, the preservation metrics need validation before the benchmark conclusions are accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper targets image generation model unlearning (IGMU) and makes three contributions: CatIGMU, a hierarchical taxonomy of unlearning tasks (global/local, abstract/concrete, with finer task types); EvalIGMU, an evaluation framework covering forgetting, preservation, image quality, robustness, and efficiency; and DataIGM, a multi-source dataset for training detectors and benchmarking. The authors empirically critique existing detectors (Style Classifier, NudeNet, Q16, ResNet-50) and task-free metrics (CLIP Score, CLIP Accuracy), then use their own MultiClf detector and EvalIGMU metrics to evaluate ten state-of-the-art unlearning methods on four tasks (Nudity, Van Gogh style, Church, Parachute). The main claim is that most existing IGMU methods perform poorly, especially on preservation and robustness.","tokens_in":27911,"tokens_out":6767,"duration_ms":68560,"significance":"If the empirical conclusions are valid, this is a valuable contribution: it documents distribution shift problems in existing detectors, provides a reusable multi-source dataset, and makes a concrete case that single-metric evaluations of unlearning are misleading. The taxonomy and implementation guidelines are also useful for structuring future work. The paper ships code/models and uses a consistent experimental setup across ten methods, which is a strength. However, the headline finding, particularly the 'especially for preservation' conclusion, depends on preservation metrics that are not calibrated or validated in the manuscript; the central evaluative claim is therefore not yet established.","major_comments":[{"comment":"The CSDR and LPIPS preservation metrics are not adequately specified or calibrated. The CSDR formula compares CS(M(p)_i, p) with CS(M^u(t)_j, p) without stating which prompt p is used for each task. If p is the target prompt t (i.e., S⊕t), then a successful unlearned model M^u is expected to reduce alignment with t, so CSDR will increase precisely when R1 is satisfied; the paper's own Table 4 expectation of 'a clothed girl playing on a beach' would score as a large CSDR/LPIPS deviation from the original 'naked girl' output. If p is instead a non-target prompt, the metric still lacks a within-model baseline: LPIPS between two independent samples from the same model and prompt is nonzero, and no such floor is reported. The skeptical concern that the preservation finding may be an artifact of generative stochasticity or of successful erasure is therefore valid. This is load-bearing because the abstract's 'especially for preservation' claim is the paper's most consequential empirical assertion.","section":"Sec. 5.3.1, Preservation; Fig. 8(b)"},{"comment":"Figure 8 reports normalized averages with no error bars, significance tests, or per-cell sample sizes. The text says 286,000 paired images were sampled, but it does not state how many images per method, per prompt, or per task underlie each bar. The normalization to [0,1] is not described (e.g., whether CSDR and LPIPS are normalized per task or across methods, and how the '1−value' transformation is applied). The efficiency aspect in EvalIGMU is also absent from Figure 8, despite being listed as one of the five evaluation dimensions. Without uncertainty quantification, the broad claim that 'most existing IGMU algorithms cannot handle unlearning well across different evaluation dimensions' exceeds what the figure can support.","section":"Sec. 6.2, Fig. 8"},{"comment":"The forgetting evaluation via MultiClf is trained on DataIGM, whose labels are derived from author-defined 'Expectation' outputs (Sec. 4.2, Table 4) and whose SD-GEN portion is produced by the same base model M that all unlearning methods start from. Using MultiClf as the forgetting oracle is therefore not an independent assessment of the ten methods; it is a detector fitted to the same generative pipeline and to the authors' chosen ground-truth semantics. The paper's own Discussion in Sec. 5.3.1 narrows the threat model to keyword-based terms, which further limits the scope of the forgetting/robustness claims. No sensitivity analysis with alternative detectors or ground-truth definitions is reported, so the reader cannot tell how strongly the benchmark conclusions depend on these choices.","section":"Secs. 4.2, 5.3.2, Table 4"},{"comment":"The FID reference for image quality is not comparable across task categories. For Abstract tasks the reference is M(do(T='none')), while for Concrete tasks it is M(do(T≠t)). Consequently, Figure 8(c) mixes two different notions of quality change: for abstract tasks it measures distance to a prompt with the target removed, and for concrete tasks it measures distance to a prompt with the target replaced by a different object. The reported gap between abstract and concrete tasks could be an artifact of this reference choice rather than a property of the unlearning methods. The paper should either use a common reference or analyze the two categories separately.","section":"Sec. 5.3.1, Image Quality; Fig. 8(c)"}],"minor_comments":[{"comment":"The LPIPS formula uses the summation index j=i+1, which is not well-defined when K and N differ; it should presumably read j=1 to K with a double sum over i and j.","section":"Sec. 5.3.1, LPIPS formula"},{"comment":"The notation for prompts is overloaded: Sec. 3.2 says t is used to denote S⊕t, but the preservation formulas then refer to 'an arbitrary task t and the corresponding p' without defining p. This ambiguity matters because the CSDR interpretation depends on whether p is the target prompt or a non-target prompt.","section":"Secs. 3.2 and 5.3.1"},{"comment":"The normalization and aggregation details are missing: for CSDR and LPIPS lower values are better, but the caption only says 'higher values indicate better performance (for smaller-is-better metrics, 1−value is used)' without stating the normalization range, the aggregation rule across tasks, or the number of images per cell.","section":"Fig. 8 caption"},{"comment":"Runtime values are labeled as estimates from the original papers and official code, but no standard deviation, number of runs, or exact hardware configuration beyond 'single A6000 GPU' is given; consider reporting measured runtimes with variance.","section":"Table 8"},{"comment":"There are minor typos and naming inconsistencies: 'SG-GEN' appears where 'SD-GEN' is meant, and the classifier is sometimes written 'MultClf' and sometimes 'MultiClf'.","section":"Sec. 4.3.1 and throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the venue and the dataset/taxonomy contributions are useful. My main reservation is that the headline conclusion about preservation depends on metrics that need recalibration and uncertainty reporting; the benchmark conclusions should also be scoped to the four tasks and keyword-based threat model. I would not reject on circularity grounds alone, since author-defined ground truth is common in benchmark papers, but the current presentation overstates the strength of the empirical verdict."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth reading if you work on machine unlearning for generative models, but take its headline finding with a grain of salt. The authors build CatIGMU (a taxonomy), EvalIGMU (a five-aspect evaluation framework), and DataIGM (a multi-source dataset), and they re-evaluate ten unlearning methods. The empirical critique of existing detectors is concrete: Style Classifier, NudeNet, Q16, and ResNet-50 all degrade on LAION and generated data, and the authors show this with clean tables. That part is solid, and the proposed MultClf is a nice, reusable contribution.\n\nThe soft spots are in the preservation metrics and, to a lesser degree, in the evaluation's ground truth. CSDR and LPIPS compare M^u(t) against M(p), but the paper never clearly says what p is. If p is the target prompt (S⊕t), then the metric penalizes exactly what successful unlearning is supposed to do: a clothed girl is far in LPIPS from a naked girl and has lower CLIP alignment with 'naked girl.' If p is the target-free prompt (S), then it's a reasonable reference, but the paper has to say so explicitly and provide a within-model baseline. LPIPS between two independent samples of M(S) is not zero, and without that floor, large LPIPS values may just reflect normal generative stochasticity. Figure 8 also reports normalized averages with no error bars or significance tests, which is a problem for a benchmark paper. The 'Expectation' ground truths in Sec 4.2 are author-defined, and MultClf is trained on DataIGM, which includes SD-GEN from the same models being evaluated; that is not fatal, but it makes the forgetting results less independent than they appear.\n\nThe central argument that existing methods struggle with preservation and robustness is plausible, and the forgetting and robustness results are on firmer ground. But the preservation claim is load-bearing, and right now it is not calibrated. I'd ask for clarifications and validation before treating the benchmark as definitive.\n\nIf you are building or evaluating IGMU methods, this is worth citing and worth reading carefully. It deserves a serious referee; I would send it out with major-revision requests rather than desk-reject it. Give it to a reviewer who knows the unlearning literature and who will demand the preservation metric be pinned down.\n\nBest,","headline":"A useful benchmark and evaluation framework for IGMU, but the preservation metrics are under-specified and uncalibrated, so the headline 'fail at preservation' finding is not yet established.","tokens_in":28432,"tokens_out":5421,"would_cite":true,"duration_ms":53405,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that current evaluation of machine unlearning in image generators is built on unreliable detectors and single-metric checks, and that measured with a comprehensive five-aspect framework, most state-of-the-art unlearning…","keywords":["machine unlearning","image generation models","concept erasure","diffusion models","evaluation benchmark","content detection","AI safety","adversarial robustness"],"falsifier":"Compare EvalIGMU preservation scores (CSDR and LPIPS against the original model's outputs) with human judgments: for images generated by each of the ten unlearned models on DataIGM prompts, ask raters whether the target is gone and the remaining scene is undamaged. If human raters accept a benign but re-composed output as a good unlearning result while CSDR and LPIPS flag it as a preservation failure, the original-model reference is not a valid ground truth and the headline finding is an artifact of the metric.","tokens_in":27384,"feed_emoji":"🎨","tokens_out":9269,"duration_ms":82505,"temperature":0.7,"pith_summary":"The paper tries to establish that the field's measurement of machine unlearning in image generation models is unreliable, and that when the measurement is fixed, most published unlearning methods stop looking successful. To do this it proposes a hierarchical taxonomy of unlearning tasks (CatIGMU: global vs. local scope, abstract vs. concrete target, then task type), a five-aspect evaluation framework (EvalIGMU: forgetting, preservation, image quality, robustness, efficiency), and a multi-source dataset (DataIGM) spanning four tasks. Benchmarking ten state-of-the-art unlearning algorithms under this framework, the paper finds that most cannot forget reliably while preserving the remaining content, keeping image quality, and withstanding adversarial prompts that re-elicit the forgotten concept. If correct, the consequence is that single-metric success claims (for example, a low target-detection rate) are misleading, and that unlearning algorithms need task-dependent design and multi-dimensional reporting before they can serve privacy and safety regulations.","feed_headline":"Most image-unlearning methods fail a five-way evaluation","feed_subtitle":"Benchmarking ten unlearners shows benign content and adversarial defenses collapse when measured fairly.","key_machinery":"The load-bearing machinery has three parts. CatIGMU is a two-tier taxonomy that sorts every unlearning target by spatial scope (global vs. local on the canvas) and perceptual attribute (abstract vs. concrete), then by task type, and it specifies what a good unlearned output is for each cell, for example, dressing a nude subject rather than deleting her, or replacing a church with another object while keeping the river. MultiClf, a CLIP-ViT backbone with per-task classification heads trained on the multi-source DataIGM, is the forgetting evaluator, chosen because existing detectors collapse on generated data. The preservation and image-quality metrics are reference-based: CSDR, the difference rate between the unlearned model's text-image CLIP alignment and the original model's for the same prompt, and LPIPS, the perceptual distance between the two image sets, with FID measuring the distribution gap, and robustness measured by UnlearnDiffAtk's attack success rate.","core_discovery":"The paper's central discovery is that the standard tools used to judge whether an image generator has forgotten a concept, style classifiers, nude detectors, object recognizers, CLIP Score, and CLIP Accuracy, fail under distribution shift: they are trained on real images and are unreliable on both the wider real data and the generated images that unlearning actually concerns. The paper replaces them with MultiClf, a multi-head classifier built on CLIP-ViT that reaches roughly 93 to 100 percent accuracy across REAL, LAION, and generated data, plus CSDR and LPIPS against the original model's outputs for preservation, and FID and UnlearnDiffAtk for quality and robustness. Re-evaluating ten state-of-the-art unlearning methods with these instruments, the paper claims that most methods achieve some forgetting on easy tasks but degrade sharply on preservation and robustness: YOLO-based human detection shows up to 38.58 percent of generated images lose the human subject that should have been preserved, and adversarial prompts re-elicit supposedly erased content, especially for artist-style unlearning. The performance varies by task in ways that a single 'concept erasure' label hides, which the CatIGMU taxonomy is meant to make visible.","pith_inferences":["If the reference-based preservation metrics are taken at face value, the benchmark implicitly rules out legitimate alternative compositions; a complementary reference-free test, 'is the target absent and is the rest of the prompt still honored,' would be needed to separate true preservation failure from benign re-composition.","The CatIGMU taxonomy suggests a testable design principle: category-aware algorithms (surgical edits for local-concrete targets, whole-canvas re-rendering for global-abstract targets) should outperform the universal one-size-fits-all fine-tuning that dominates the ten benchmarked methods.","Because the robustness threat model is keyword-based, the attack-success numbers are likely a lower bound; paraphrase-based prompts or multimodal jailbreaks would probably push attack success rates higher for every method.","The DataIGM protocol, three data sources with paired target and retained prompts, is directly portable to tasks the paper did not test, such as violence, celebrity identity, and brand logos, so the framework's claim that current methods fail preservation and robustness can be checked on new categories without new machinery."],"forward_implications":["A high forgetting score on a single detector no longer counts as evidence of successful unlearning: the paper's benchmark shows methods can score well on forgetting while failing preservation, quality, or robustness, so published single-metric claims are unverified until all five aspects are reported.","Performance is task-dependent in a structured way: global-abstract targets like artist style are the hardest, and even same-type tasks such as church vs. parachute diverge, so results from one unlearning task should not be extrapolated to another.","Adversarial prompt attacks re-elicit supposedly erased content from most unlearned models, which means concept deletion in diffusion models is reversible with crafted text prompts and needs robustness training to hold.","Efficiency is a decisive dimension: runtimes range from about 40 seconds to 7 hours, so the methods that are most robust are the least scalable, and multi-task support varies across approaches."],"supporting_citations":[{"why":"Supplies one of the ten benchmarked unlearning methods and the negative-guidance fine-tuning paradigm whose preservation failures and nudity re-leakage the paper documents.","marker":"[16]"},{"why":"Benchmarked closed-form cross-attention editor (UCE) whose forgetting and preservation scores are measured by MultiClf, CSDR, and LPIPS.","marker":"[17]"},{"why":"Forget-Me-Not, a benchmarked attention-weight minimization method that achieves the best human-preservation score but fails church unlearning, used to demonstrate task-dependence.","marker":"[73]"},{"why":"AdvUnlearn, the adversarial-training baseline that scores best on robustness and balance but runs about 7 hours, illustrating the efficiency trade-off.","marker":"[76]"},{"why":"UnlearnDiffAtk supplies the adversarial prompt attack used to measure the robustness aspect of EvalIGMU across all ten methods.","marker":"[77]"},{"why":"CLIP is the backbone for CLIP Score, hence CSDR, and for the CLIP-ViT encoder that MultiClf fine-tunes on DataIGM.","marker":"[46]"},{"why":"FID is the distribution-distance metric used for the image-quality aspect in EvalIGMU.","marker":"[26]"},{"why":"LPIPS is the perceptual-similarity metric that, with CSDR, forms the preservation measurement in EvalIGMU.","marker":"[75]"},{"why":"NudeNet, the widely used nudity detector the paper tests and shows to be unreliable, with accuracy below 78 percent and high miss rates on generated images, motivating MultiClf.","marker":"[44]"},{"why":"Six-CD, the prior benchmark the paper contrasts with, covers only unlearning and retention aspects, which is the gap EvalIGMU's five aspects fill.","marker":"[48]"}],"fun_headline_variants":["Image unlearning fails on preservation and robustness","Most unlearning models can't handle benign content or attacks","Benchmark exposes weaknesses in image unlearning methods","Unlearning evaluation is unreliable under data shift","Ten unlearning methods falter on new robust benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The preservation and image-quality metrics take the original model's outputs as the correct reference, so an unlearned model that deletes the target and substitutes a different but perfectly benign composition is scored as a failure rather than as a valid outcome.","fun_headline_variants_meta":{"raw":{"variants":["Image unlearning fails on preservation and robustness","Most unlearning models can't handle benign content or attacks","Benchmark exposes weaknesses in image unlearning methods","Unlearning evaluation is unreliable under data shift","Ten unlearning methods falter on new robust benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1426,"prompt_tokens":1093,"completion_tokens":333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":261}},"tokens_in":709,"tokens_out":333,"duration_ms":3773,"temperature":1.0,"reasoning_tokens":261,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:16:28.680676+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare EvalIGMU preservation scores (CSDR and LPIPS against the original model's outputs) with human judgments: for images generated by each of the ten unlearned models on DataIGM prompts, ask raters whether the target is gone and the remaining scene is undamaged. If human raters accept a benign but re-composed output as a good unlearning result while CSDR and LPIPS flag it as a preservation failure, the original-model reference is not a valid ground truth and the headline finding is an artifact of the metric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Benchmarked closed-form cross-attention editor (UCE) whose forgetting and preservation scores are measured by MultiClf, CSDR, and LPIPS."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AdvUnlearn, the adversarial-training baseline that scores best on robustness and balance but runs about 7 hours, illustrating the efficiency trade-off."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UnlearnDiffAtk supplies the adversarial prompt attack used to measure the robustness aspect of EvalIGMU across all ten methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FID is the distribution-distance metric used for the image-quality aspect in EvalIGMU."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LPIPS is the perceptual-similarity metric that, with CSDR, forms the preservation measurement in EvalIGMU."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NudeNet, the widely used nudity detector the paper tests and shows to be unreliable, with accuracy below 78 percent and high miss rates on generated images, motivating MultiClf."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Six-CD, the prior benchmark the paper contrasts with, covers only unlearning and retention aspects, which is the gap EvalIGMU's five aspects fill."}],"review_version":1}