{"id":"b4b617d8-8823-436f-95b0-13a8003a2b47","arxiv_id":"2504.12574","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors propose the ForgetMe dataset and the Entangled metric to evaluate selective unlearning in diffusion models, using SAM, CLIP, GPT-4o, and LaMa to build paired original/background images.","lead":"This paper introduces a new benchmark dataset and evaluation metric for measuring how well generative image models can selectively forget specific objects or concepts. It is aimed at researchers working on privacy-preserving AI, who need standardized ways to verify that models have truly removed sensitive content.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Entangled is a pixel/statistics-only metric with no validation or defined procedure for unpaired generated images; a blur or noise baseline could score high without forgetting, undermining the benchmark's central claim.","rationale":"The paper's central claim is that ForgetMe and Entangled provide a standard benchmark for selective unlearning. The load-bearing assumption is that Entangled measures whether a target concept has been forgotten. I find this assumption unsupported in two ways. First, the metric's definition (Eqs. 1-8) is purely pixel- and statistics-based; it has no semantic component, so it cannot distinguish 'concept removed' from 'region destroyed.' Second, the paper offers no validation of Entangled against human perception or established metrics, despite reporting CLIP Accuracy alongside it. The dataset construction uses Entangled in Section 3.1.2 Step 3, creating a circular loop that only compounds the validity problem. Additionally, the unpaired variant Entangled-S is never defined for generated images without masks, making the main evaluation use case unreproducible. A trivial baseline test can expose whether Entangled can be gamed: if blurring or noise in the target region produces a high score, the metric is not measuring forgetting. This is exactly the kind of empirical check the paper omits, and it is decisive for the benchmark's usefulness. The reader's REJECT verdict is therefore well-founded; no change to the verdict is needed. I agree with the reader's identification of the metric-validation problem, though I place more weight on the metric's construct validity than on circularity alone.","tokens_in":13437,"tokens_out":10760,"duration_ms":114029,"concrete_test":"On a random sample of 200 images from a ForgetMe subset (e.g., Cat), construct a trivial baseline: replace the inner (target) region of each original image with additive Gaussian noise whose mean and variance are matched to the outer region, leaving the outer region untouched. Compute Entangled-D for these fake unlearned images using the paper's Eqs. (1)-(6). If the baseline's mean Entangled-D is within 0.1 of the proposed method's reported 0.868, the metric is gameable and the benchmark's central claim fails; if the baseline scores well below (e.g., <0.3), the metric may indeed respond to true semantic removal and the concern would not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equations (1)-(6) define Entangled as a combination of per-pixel RMSE (Eq. 2) and mean/variance consistency (Eqs. 5-6). No term in this definition references the semantics of the concept to be forgotten. A model that blurs the target region or fills it with noise matched to the outer region can push S_inner toward 1 while keeping S_outer low and C_inner,outer moderate, producing a high Entangled score even though the concept remains detectable. The paper reports no human evaluation, no correlation with established semantic metrics, and the only external metric (CLIP Accuracy) is tabulated without any agreement analysis with Entangled. The circularity is severe: Entangled is used in Section 3.1.2 Step 3 to filter the dataset backgrounds, so the benchmark is constructed to score well on the very metric later used for comparison. Furthermore, the unpaired variant Entangled-S, which is the relevant one for evaluating text-to-image unlearning (where no ground-truth mask exists), is never specified: the paper does not explain how inner and outer regions are located in a generated image. Consequently, the Entangled-D=0.868 and Entangled-S=0.8119 values in Table 2 are not evidence of concept unlearning; they may only reflect pixel-level separation or even be unreproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ForgetMe, a dataset and evaluation framework for selective unlearning in generative models. It introduces an automatic dataset creation pipeline based on prompt-based layered editing and training-free local feature removal (SAM, LaMa, GPT, Stable Diffusion), and an Entangled evaluation metric that combines pixel-level similarity and mean/variance consistency between inner (target) and outer (background) regions, with paired (Entangled-D) and unpaired (Entangled-S) variants. The authors construct datasets from CUB-200-2011, Stanford Dogs, ImageNet, and a synthetic cat dataset, then fine-tune Stable Diffusion with LoRA to perform unlearning. They compare against CLIPAway and LoRA-based baselines, reporting lower FID/CMMD and higher Entangled and CLIP Accuracy scores for their method.","tokens_in":13640,"tokens_out":4633,"duration_ms":47784,"significance":"If the benchmark and metric were sound, the work would address a real gap: standardized evaluation of selective unlearning in diffusion models is missing from the literature. The proposed pipeline is fully automated and training-free, which is a practical strength, and the inclusion of both real and synthetic datasets is commendable. The idea of a paired/unpaired evaluation metric is also timely. However, the significance is severely undercut by two load-bearing problems: the evaluation metric is used to construct the very dataset on which it is later evaluated, and the metric itself has no demonstrated semantic validity. Without addressing these issues, the claimed benchmark cannot be trusted by the community.","major_comments":[{"comment":"The Entangled metric is used both to assess and refine reconstructed backgrounds during dataset construction (Section 3.1.2 Step 3) and to evaluate unlearning methods (Section 4.3). This circular self-evaluation means that the proposed baseline is trained on backgrounds that were explicitly selected to score well on the metric, so the high Entangled values in Table 2 are expected by construction. To support the benchmark claim, the authors must break this loop, for instance by constructing the dataset without Entangled-based filtering (using only GPT validation or other metrics) and then demonstrating that the relative ranking of methods is unchanged. As written, the comparison is not a fair test.","section":"Sections 3.1.2 Step 3 and 4.3"},{"comment":"The Entangled metric is purely pixel-level: S_region is an RMSE-like distance and C_inner,outer is a harmonic combination of mean and variance ratios. No term references the semantic identity of the target concept. A model that blurs the inner region or fills it with noise matched to the outer region can inflate S_inner and maintain moderate C, producing a high Entangled score while the concept remains detectable. The paper reports no human evaluation, no correlation with established perceptual or semantic metrics, and the CLIP Accuracy numbers in Table 2 are never analyzed for agreement with Entangled. Please provide a validation experiment, such as showing that a trivial blur/noise baseline scores low on Entangled, or reporting correlation between Entangled and human judgments.","section":"Equations (2)-(6), Table 2"},{"comment":"The unpaired variant Entangled-S is claimed to evaluate unlearning on generated images without ground-truth masks, but the paper never specifies how the inner and outer regions are defined or located in a generated image. Table 2 reports specific Entangled-S values (e.g., 0.8119 for Ours), yet without a concrete protocol for segmenting or aligning regions in an unpaired setting, these numbers are unreproducible. The authors must either provide the exact procedure (e.g., SAM + CLIP-based localization, or fixed positional grids) or explicitly state that Entangled-S is computed on images where the mask is known; otherwise, the central unlearning evaluation is vacuous.","section":"Section 4.3, Table 2"},{"comment":"Table 1 reports 'Selected' and 'Success' counts with percentages (e.g., 97.44% success for Bird), but the text does not define what 'success' means. It seems related to foreground extraction, yet Section 3.1.2 Step 3 uses Entangled to filter or refine backgrounds, so the dataset construction process is intertwined with the evaluation metric. Please clarify the exact success criterion, report how many images were rejected at each stage (foreground extraction, background reconstruction), and quantify the distribution of Entangled scores during construction. Without this transparency, the dataset composition and its relationship to the metric remain unclear.","section":"Table 1, Section 4.4"}],"minor_comments":[{"comment":"The manuscript contains multiple unresolved placeholders, including 'Section ??' (after Eq. (2)), 'Figure ??' (Section 4.5), 'CLIPAway [?]' (Section 4.5), and 'The rationale for selecting this function is discussed in Section ??'. These must be filled in before any review.","section":"Throughout"},{"comment":"The definition of Entangled is garbled in the text: the formula appears as 'Entangled = α +β α Sinner, outer + β Cinner, outer = ...' which is not a valid mathematical expression. Please typeset the harmonic mean correctly, e.g., Entangled = (α+β) S C / (α C + β S).","section":"Equation (1)"},{"comment":"S_region is defined as a root-mean-square difference, which is a distance (lower is better), while S_inner,outer is defined as a harmonic combination with 1-S_outer, so the roles of 'similarity' versus 'distance' are confusing. Please clarify the direction of each component and why S_outer is flipped.","section":"Equations (2)-(3)"},{"comment":"The comparison states that the method 'outperforms SOTA method CLIPAway in flexibility, requiring no additional training', but CLIPAway is an object-removal method while the proposed approach performs LoRA fine-tuning. Please explain how these settings are comparable and ensure the baselines receive identical prompts, masks, and seeds.","section":"Section 4.5"},{"comment":"The paper promises a public release of dataset and code only upon acceptance; for a benchmark paper, this is insufficient. Please provide a public anonymized link or a commitment to release the data and code at review time to allow reproducibility.","section":"Section 4.1-4.2"},{"comment":"All metrics in Table 2 are reported as single numbers without standard deviations or multiple seeds. Given the sensitivity of LoRA training and generation, report mean±std over at least three runs.","section":"Table 2"}],"recommendation":"reject","confidential_remarks":"The paper addresses a timely problem, but the central methodological flaw (using the proposed metric to construct the dataset that is then evaluated with that metric) and the lack of any validation for Entangled make the core claim unsupportable. Even if the circularity were fixed, the undefined Entangled-S protocol and the absence of code/data would require substantial additional work. The ideas have potential, but the current manuscript is not close to a publishable benchmark. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: this paper has a sensible idea—build a paired dataset for selective unlearning in diffusion models and a metric that balances removing the target while preserving the background—but the evaluation is circular and the metric is unvalidated, so the central benchmark claim doesn't survive a close read. The authors put together an automatic pipeline using SAM, CLIP, GPT-4o, and LaMa to create foreground/background pairs, and they train LoRA on the background images. That construction is actually the most useful part: it gives a concrete recipe for generating unlearning data at scale, and the four-dataset coverage (birds, dogs, ImageNet, synthetic cats) is a reasonable start.\n\nWhat's new: the Entangled metric, defined as a harmonic mean of inner/outer similarity and mean/variance consistency, is a straightforward but not silly attempt to quantify both removal and retention. The paired/unpaired distinction is also worth thinking about. The paper does a decent job of positioning against existing unlearning and object-removal literature, and it includes a limitations section.\n\nThe soft spots are serious, though. The biggest one is circularity: Entangled is used in Section 3.1.2 Step 3 to filter which reconstructed backgrounds are kept in the dataset, and then the same metric is used in Section 4.3 to score unlearning methods. That means the benchmark is constructed to reward whatever makes Entangled happy. The metric itself is pixel-level and statistical only—no semantic term—so a blur or noise fill could plausibly score high without actually forgetting the concept, and there is no human evaluation or correlation with established semantic metrics to show it tracks real unlearning. The unpaired variant Entangled-S is also underspecified: the paper never explains how inner and outer regions are located in a generated image without a mask. The comparison is against object-removal baselines rather than established unlearning methods, there are no error bars, and the manuscript has unfinished artifacts (Section ??, Figure ??, a missing CLIPAway citation). Code and data are promised only on acceptance, so none of this is reproducible right now.\n\nThe central idea has merit, and the dataset pipeline could be a useful contribution if the metric were validated and decoupled from dataset construction. As is, the benchmark claim doesn't hold. I'd send it to review with the expectation of major revision, but I wouldn't cite it yet.\n\nRecommendation: engage with it as a workshop-level or revision-required paper, not as a finished benchmark.","headline":"A useful dataset-construction pipeline undone by a self-referential, unvalidated metric; not a benchmark yet, but worth a major-revision round.","tokens_in":14232,"tokens_out":2585,"would_cite":false,"duration_ms":25323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces the ForgetMe dataset and Entangled metric as a standardized benchmark for measuring whether diffusion models truly forget a target concept.","keywords":["selective unlearning","diffusion models","generative models","evaluation metric","dataset construction","object removal","LoRA fine-tuning","concept erasure"],"falsifier":"Show a set of generated images to human raters alongside their Entangled-S scores and ask whether the target object is still present. If images the metric scores as fully unlearned are consistently recognized as still containing the object, or if images it scores as failed are judged clean, the metric is not measuring selective forgetting as claimed.","tokens_in":13168,"feed_emoji":"🎯","tokens_out":6687,"duration_ms":62997,"temperature":0.7,"pith_summary":"The paper tries to establish a standard way to test whether a generative model has actually forgotten a specific object or concept, rather than merely learning to remove it from a given image. It introduces the ForgetMe dataset, built by an automatic, training-free pipeline that splits images into foreground, background, and mask layers, and the Entangled metric, which scores unlearning by comparing inner target regions with outer background regions. If accepted, these would give researchers and regulators a common yardstick for comparing unlearning methods across real and synthetic image categories. The paper also demonstrates the benchmark by using LoRA fine-tuning on Stable Diffusion to perform selective forgetting and reports that this approach beats established baselines on the new metric.","feed_headline":"New benchmark measures whether generative models truly forget","feed_subtitle":"ForgetMe dataset plus Entangled score standardize selective-unlearning tests for birds, dogs, cats, and ImageNet.","key_machinery":"The load-bearing object is the Entangled metric, a harmonic combination of a similarity score between inner (target) and outer (background) regions and a consistency score comparing their means and variances. This metric is used twice: first as a quality gate when building the ForgetMe dataset's background images, and second as the evaluation score for unlearned models. The dataset itself is produced by a training-free layered pipeline: SAM generates candidate segmentation masks, CLIP scores select the mask matching the target category, GPT verifies the crop, LaMa inpaints the missing background, and Stable Diffusion refines low-scoring backgrounds. An optional Side Story module merges foreground and background layers to create synthetic paired data.","core_discovery":"The central claim is that selective unlearning in generative models can be evaluated with a purpose-built benchmark rather than generic image-quality metrics. Concretely, the paper claims that the Entangled metric, which combines a similarity term between the removed inner region and the preserved outer region with a consistency term on their means and variances, captures both forgetting effectiveness and background retention. It further claims that the ForgetMe dataset, constructed automatically from birds, dogs, cats, and ImageNet classes, provides a diverse testbed for this evaluation. The experiments are presented as validation: LoRA fine-tuning on background-only images yields high Entangled scores and outperforms CLIPAway and other baselines on nearly all reported metrics.","pith_inferences":["Entangled's pixel-level statistics may reward visually uniform backgrounds while missing semantic reappearance, so a human-anchored or embedding-based re-identification check would be a natural complement.","The same automatic layering pipeline could be reused for continual unlearning: new sensitive concepts could be converted into background-only training pairs on the fly without retraining the removal framework.","The results leave open whether what is forgotten is the concept itself or only the context in which it appears, since training on background images shifts the conditional distribution rather than explicitly erasing an internal representation.","The metric's harmonic combination of similarity and consistency could transfer to other generative editing tasks, such as object removal quality assessment, where both target removal and background preservation matter."],"forward_implications":["If Entangled is accepted, unlearning methods can be compared on a single scale across categories, letting model cards report a concrete forgetting score alongside image-quality metrics.","The ForgetMe dataset's paired original, background, and mask structure lets developers fine-tune unlearning by training on background images alone, without collecting new data.","Because the dataset pipeline is training-free, adding a new sensitive concept only requires running SAM, CLIP, GPT, and LaMa on new images rather than retraining a removal model.","Entangled-S extends evaluation to settings where the original image is unavailable, making it possible to audit deployed models using only their outputs.","The paper's finding that high LoRA scaling factors produce the best Entangled scores suggests that simple fine-tuning on background-only images is a viable default unlearning baseline."],"supporting_citations":[{"why":"Supplies the segmentation masks used to isolate the foreground in dataset construction.","marker":"[20]"},{"why":"Provides the initial inpainting that reconstructs the background after foreground removal.","marker":"[36]"},{"why":"Used for the second inpainting pass and as the base generative model for LoRA fine-tuning.","marker":"[32]"},{"why":"The low-rank adaptation method used to fine-tune Stable Diffusion's U-Net for unlearning.","marker":"[16]"},{"why":"State-of-the-art object removal baseline whose unlearning results are compared and outperformed.","marker":"[7]"},{"why":"Fréchet Inception Distance baseline for measuring generated image quality in the evaluation.","marker":"[5]"},{"why":"CLIP Maximum Mean Discrepancy baseline for distributional alignment in the evaluation.","marker":"[18]"},{"why":"Source of the ImageNet validation images used as one of the four ForgetMe categories.","marker":"[6]"},{"why":"Source of the Stanford Dogs images used as the dog category in ForgetMe.","marker":"[19]"},{"why":"Source of the CUB-200-2011 bird images used as the bird category in ForgetMe.","marker":"[38]"}],"fun_headline_variants":["ForgetMe benchmark plus Entangled metric measure unlearning","Entangled score evaluates selective forgetting in diffusion models","New dataset and metric for testing generative model forgetting","Benchmark for selective unlearning in image generators","How well do diffusion models forget? New benchmark answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Entangled metric is assumed to be a valid measure of unlearning even though it is never checked against human judgment or established metrics and is used both to construct the ForgetMe dataset and to score the unlearning results.","fun_headline_variants_meta":{"raw":{"variants":["ForgetMe benchmark plus Entangled metric measure unlearning","Entangled score evaluates selective forgetting in diffusion models","New dataset and metric for testing generative model forgetting","Benchmark for selective unlearning in image generators","How well do diffusion models forget? New benchmark answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000522,"raw_usage":{"total_tokens":2496,"prompt_tokens":888,"completion_tokens":1608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1534}},"tokens_in":504,"tokens_out":1608,"duration_ms":11184,"temperature":1.0,"reasoning_tokens":1534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:27:37.045053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show a set of generated images to human raters alongside their Entangled-S scores and ask whether the target object is still present. If images the metric scores as fully unlearned are consistently recognized as still containing the object, or if images it scores as failed are judged clean, the metric is not measuring selective forgetting as claimed.","supporting_citations":[{"cited_title":"Resolution-robust large mask inpainting with fourier convolutions","cited_arxiv_id":null,"evidence_quote":"Provides the initial inpainting that reconstructs the background after foreground removal."},{"cited_title":"Effectively unbiased fid and inception score and where to find them","cited_arxiv_id":null,"evidence_quote":"Fréchet Inception Distance baseline for measuring generated image quality in the evaluation."},{"cited_title":"Re- thinking fid: Towards a better evaluation metric for image generation","cited_arxiv_id":null,"evidence_quote":"CLIP Maximum Mean Discrepancy baseline for distributional alignment in the evaluation."},{"cited_title":"Imagenet: A large-scale hierarchical image database","cited_arxiv_id":null,"evidence_quote":"Source of the ImageNet validation images used as one of the four ForgetMe categories."}],"review_version":1}