{"id":"3cb0e9ac-0030-4c20-8d3b-1f8aad220e98","arxiv_id":"2412.01244","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A method to precisely localize and replace concepts in generated images by combining a few-shot attention-based concept localizer with masked dual-prompt cross-attention during denoising.","lead":"This paper presents a diffusion-model safety method that locates a target concept during image generation and replaces only that region with a different concept, using a few-shot trained localizer and a training-free dual-prompt attention module. It could make content filtering and editing more surgical than global erasure methods.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim lacks end-to-end evidence: the few-shot localizer is validated only on real-image segmentation, while replacement experiments give no generation-time mask accuracy or quantitative non-target preservation, so 'precise' replacement is unverified.","rationale":"The reader identified the same load-bearing assumption: the few-shot localizer's transfer from real-image segmentation to generated sensitive concepts is unvalidated. I agree, and I add a consequence the reader's rationale only implies: Eq. 9's mask gating is necessary but not sufficient for the paper's 'minimal impact on non-target areas' claim, because self-attention and subsequent denoising steps can carry the local edit outside M, yet no experiment detects such leakage. The absence of numeric data for Figure 7 and of any generation-time mask IoU makes this more than a stylistic gap. The localization benchmark on CelebAMask-HQ and Pascal-Car does provide some real evidence that the attention-based localizer can segment when dense labels exist, and the released code is a positive check, so the method remains plausible. I therefore keep the reader's CONDITIONAL verdict: the central replacement claim needs the end-to-end evaluation described in the concrete test before it can be accepted.","tokens_in":12412,"tokens_out":6053,"duration_ms":55968,"concrete_test":"Apply the released implementation to 100 I2P prompts with fixed seeds. For each prompt, run original SD and Concept Replacer, save the concept localizer's early-timestep mask M, and have two annotators label sensitive pixels in the original output. Compute (i) mask IoU between M and the annotated region, and (ii) masked LPIPS between original and replaced outputs on pixels outside M, compared with the same-seed rerun noise floor of the original model. If mean mask IoU is far below the 69-70% level reported on real benchmarks, or if outside-mask LPIPS exceeds the rerun noise floor by a non-trivial margin, the central 'precise localization and non-target preservation' claim is not supported by current experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim ('superior precision in localizing target concepts' and 'minimal impact on non-target areas') rests on two unverified links. First, the concept localizer is trained and evaluated only with dense labels on CelebAMask-HQ and Pascal-Car (Section 4.1), but its actual use is to localize under-specified concepts such as 'nudity' inside latents during generation; no training protocol, label construction, or mask IoU is reported for these generated-image cases. Second, Eq. 9 gates cross-attention by the predicted mask M, but the paper asserts non-target consistency without measuring it: Section 4.2 reports NudeNet reduction on I2P (Figure 7) with no numeric table, and Table 3 only gives global FID/CLIP on COCO, which cannot detect localized leakage. Because self-attention and later denoising steps propagate the replaced region's features outside M, even a perfect mask does not guarantee unchanged non-target regions. Without quantitative end-to-end mask accuracy and outside-mask fidelity, the key differentiator from global erasure methods is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Concept Replacer, a method for replacing a user-specified concept in text-to-image diffusion models at generation time. The method has two components: a few-shot trained concept localizer that fuses low- and high-resolution cross-attention maps with self-attention to predict a mask for the target concept during denoising, and a training-free Dual Prompts Cross-Attention (DPCA) module that applies the original prompt outside the mask and a replacement prompt inside the mask. The authors evaluate localization on CelebAMask-HQ and Pascal-Car segmentation benchmarks against SLiMe and other baselines, and evaluate replacement on the I2P prompts with NudeNet detection, plus FID/CLIP on COCO 30k, together with qualitative examples of replacing nudity, celebrities, and other concepts.","tokens_in":12669,"tokens_out":4054,"duration_ms":36456,"significance":"If the central claims hold, the paper offers a useful alternative to global concept-erasure methods: a few-shot localizer that runs during denoising and a training-free attention gate that confines the edit to the target region. The idea of combining a fine-tuned localizer with dual-prompt cross-attention is original, and the authors provide code and compare against several prior methods. However, the current evidence does not fully establish the two load-bearing claims of 'precise localization during generation' and 'minimal impact on non-target areas.' The localization evaluation is confined to real-image segmentation, while the replacement experiments report no generation-time mask accuracy and no quantitative outside-mask fidelity metric. The training protocol for the sensitive-concept localizer is also unspecified. These gaps are fixable with additional experiments, so the manuscript merits revision rather than rejection.","major_comments":[{"comment":"The central claim of precise concept localization during generation is not quantitatively validated. Section 4.1 evaluates the localizer only on real-image segmentation benchmarks (CelebAMask-HQ and Pascal-Car), but the replacement experiments in Section 4.2 apply the localizer to concepts such as nudity inside latents during denoising. No generation-time mask IoU, pixel accuracy, or other localization metric is reported for these generated-image cases, and no error bars are given for any of the quantitative results. Please add a quantitative evaluation of mask quality on generated images for the target concepts used in replacement, ideally with manual labels or a pseudo-label protocol, and report variability across seeds.","section":"Section 4.2, Figure 7"},{"comment":"The claim that non-target areas remain 'minimal[ly] impact[ed]' is asserted but never directly measured. Eq. (9) gates the cross-attention by the mask M, but later denoising steps, self-attention, and the mask boundary can propagate changes outside M, so even a perfect mask does not guarantee unchanged non-target regions. Table 3 reports only global FID and CLIP on COCO 30k, which cannot detect localized leakage, and Figure 7 reports only NudeNet-based nudity reduction. Please add a quantitative preservation metric, such as LPIPS/SSIM computed only on the non-masked region between the original and replaced images (or between paired generations with and without DPCA), to substantiate the minimal-impact claim.","section":"Section 4.2, Eq. (9), Table 3"},{"comment":"The training protocol for the concept localizer used in the replacement experiments is unspecified. Section 4.1 describes training on CelebAMask-HQ and Pascal-Car with dense segmentation labels, but no information is given about how the localizer is trained for concepts such as nudity, blood, skull, knife, zombie, ghost, gun, or monster used in Figures 5 and 6: what training images, what labels, what location-prompt wording, how many shots, what fine-tuning steps, and what loss weighting. Without this information, the results are not reproducible, and the transfer of the real-image localizer to abstract or under-specified concepts during generation is an unverified premise. Please provide the full training setup for the sensitive-concept localizers.","section":"Section 3.1, Section 4.2"},{"comment":"The choice of replacement timestep T=666 and the averaging timesteps T=666, 726, 766 is made qualitatively from a single face-replacement example (Brad Pitt to Leonardo DiCaprio), with no quantitative criterion. This hyperparameter directly controls the trade-off between replacement strength and preservation of the original structure, which is central to the paper's non-target-preservation claim. Please provide a quantitative sensitivity analysis over timesteps, including a preservation metric and a replacement-effectiveness metric (e.g., NudeNet reduction or CLIP-based target-concept score), and justify the chosen value on that basis.","section":"Section 5, Figure 9"}],"minor_comments":[{"comment":"The notation for the mask M is inconsistent: Eq. (5) defines M as vec(Across) * Aself, which is a vector, while Eq. (9) uses M as a spatial mask. Please clarify the shapes and how the losses in Eqs. (6)-(7) align with the spatial segmentation labels.","section":"Eqs. (5)-(7), (9)"},{"comment":"The bar chart in Figure 7 reports percentage reductions in nudity content without numeric values or error bars. Please provide the underlying numbers in a table (with standard deviations over multiple seeds or prompt subsets) so readers can verify the claimed improvements over baselines.","section":"Figure 7"},{"comment":"The time-index terminology is confusing: the text says 'For T = 0, it refers to the initial image generated using the prompt,' which conflicts with the standard diffusion convention where T=0 is the final clean image, and with the caption's statement that high T corresponds to the early diffusion phase. Please fix the notation and the wording to make the direction of time clear.","section":"Section 5, Figure 9"},{"comment":"The related-work paragraph cites 'Ablating concepts [38]' but reference [38] is the SLD paper; Ablating Concepts is reference [23]. Please correct the citation.","section":"Section 2.2"},{"comment":"The sentence 'it is challenging to get an accurate Classifier also returning a meaningless image is not user-friendly' is ungrammatical and appears to be missing a clause break. Please revise.","section":"Section 2.2"},{"comment":"There are minor formatting inconsistencies: 'ReGAn' appears with inconsistent capitalization, and the table captions for supervised vs. few-shot rows could be clearer about which rows use which training regime.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on Zhang et al.'s Concept Replacer.\n\nThe genuinely new piece is the pairing: a few-shot fine-tuned attention localizer that runs during denoising, feeding a training-free dual-prompt cross-attention replacement. That combination is not in the cited prior work, and it is a sensible response to the over-broad edits of ESD/SLD-style global erasure. The localizer itself is assembled from known ingredients—SLiMe-style fine-tuning of Wk/Wv, low-res cross-attention to refine high-res, self-attention fusion—but the assembly is clean and the segmentation numbers are honestly reported. On CelebAMask-HQ and Pascal-Car, it beats SLiMe on average in both 1-shot and 10-shot settings, and the per-class tables show the gains are spread rather than a single lucky class. That part holds up.\n\nThe soft spot is the leap from real-image segmentation to generation-time localization. The paper validates the localizer on dense labels for faces and cars, then uses it to localize 'nudity' inside generated latents with no training protocol, no label construction, and no mask IoU for those cases. The replacement evaluation is mostly qualitative: Figure 7's NudeNet reductions are shown as bars without a numeric table or error bars, and the claim of 'minimal impact on non-target areas' is supported by eye rather than by an outside-mask fidelity metric. The COCO FID/CLIP numbers in Table 3 are global and cannot detect localized leakage—exactly the failure mode the method is supposed to avoid. The choice of replacement timestep T=666 and the 32x32 cross-attention resolution split are both qualitative, and the paper itself calls T=666 'optimal' based on one illustrative pair. None of this is fatal, but it means the central 'precise replacement' claim is not yet supported end-to-end.\n\nThe stress-test note calls out these gaps accurately. I'd add one caveat: the lack of generation-time mask evaluation is the load-bearing missing piece, and the paper would be substantially stronger if the authors supplied it. A good referee could also push for a localized-editing baseline (Prompt2Prompt or similar) in the replacement comparison, which is absent.\n\nWho gets value: people working on concept erasure, safety filters, and localized editing. It deserves a serious referee and probably a major revision. I'd accept it for review but not in current form. My sense is the method likely works; the evidence just isn't there yet.","headline":"A plausible localized concept-replacement method whose localization half is solidly evaluated on real-image segmentation, but whose central generation-time replacement claim lacks direct evidence.","tokens_in":13144,"tokens_out":2301,"would_cite":true,"duration_ms":20044,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Concept Replacer localizes a diffusion model's target concept and swaps it out, leaving the rest of the image intact.","keywords":["concept replacement","diffusion models","cross-attention localization","few-shot learning","concept erasure","image safety","text-to-image generation","dual prompts"],"falsifier":"Generate images from prompts containing a known object (e.g., 'a dog'), apply the concept localizer at the early denoising timesteps it uses, then compare the predicted mask against the ground-truth segmentation of the final image (e.g., using an off-the-shelf segmenter or human annotation). If the IoU of those masks is substantially lower than the IoU reported on real-image benchmarks, the claim that precise localization transfers to generation-time concepts is falsified.","tokens_in":12200,"feed_emoji":"🎯","tokens_out":2394,"duration_ms":22166,"temperature":0.7,"pith_summary":"This paper claims that unwanted concepts in text-to-image diffusion outputs can be removed by first pinpointing the exact region where the concept appears during the denoising process, then substituting that region with a replacement prompt, all without retraining the base model. If correct, the method would give content-safety tools a surgical alternative to existing global concept-erasure techniques, which tend to alter unrelated parts of the image. The authors demonstrate the approach on sensitive concepts like nudity and on named individuals, showing that non-target areas remain visually consistent with the original generation.","feed_headline":"Precise concept replacement swaps only the unsafe region in diffusion images","feed_subtitle":"A training-free attention module plus a few-shot localizer removes targeted content without changing the rest of the picture.","key_machinery":"The Dual Prompts Cross-Attention (DPCA) module is the central mechanism: at each cross-attention layer it computes Z = Softmax(Q·K_R^T/√d)·V_R·M + Softmax(Q·K_P^T/√d)·V_P·(1−M), where M is the concept mask, (K_R,V_R) come from the replacement prompt, and (K_P,V_P) from the original prompt. The concept localizer produces M by fusing low- and high-resolution cross-attention scores (Across = A_l_cross + A_l_cross·A_h_cross) and multiplying them with self-attention scores, after few-shot fine-tuning of the W_k and W_v projection matrices in the U-Net.","core_discovery":"The core discovery is that a few-shot fine-tuned concept localizer can extract a precise spatial mask of a target concept from cross-attention and self-attention maps at early denoising timesteps, and that this mask can feed a training-free Dual Prompts Cross-Attention (DPCA) module. The DPCA module computes attention with two prompt conditionings: the original input prompt outside the mask and a replacement prompt inside the mask, so that only the localized region changes. The paper reports that this preserves non-target regions better than methods that guide or fine-tune the whole diffusion model.","pith_inferences":["The paper's claim that early-timestep masks transfer to generated concepts would be directly tested by comparing localizer masks produced at denoising timesteps against ground-truth segmentations of the final generated image; this is an inference from the methods section, not something the paper quantifies for sensitive concepts.","If the localizer's masks are accurate enough, the same DPCA machinery could be extended to simultaneous replacement of multiple distinct concepts by assigning each mask its own replacement prompt, a natural extension the paper does not explore.","A practical limitation not discussed is the dependence on the localizer's few-shot tuning: the paper validates few-shot segmentation only on real-image benchmarks, so the transfer to abstract and high-level concepts like nudity remains an untested assumption that a reader should keep in mind."],"forward_implications":["Sensitive content could be removed or replaced in generated images without retraining or fine-tuning the diffusion U-Net, preserving the model's original distribution elsewhere.","The approach could make content moderation more user-friendly: instead of blocking an entire image, only the problematic concept is swapped out, leaving the composition and style intact.","Few-shot localization suffices to identify concepts, so the method could be adapted to new unwanted concepts with only a handful of labeled examples.","Because the replacement is prompt-driven, users could choose any substitute concept, not just a fixed safe placeholder.","The mask-guided cross-attention mechanism could be reused for other localized image manipulations beyond safety, such as targeted object editing within a generated scene."],"supporting_citations":[{"why":"SLiMe provides the few-shot segmentation baseline and the inspiration for tuning diffusion attention for localization; it is the comparison method on CelebAMask-HQ and Pascal-Car.","marker":"[22]"},{"why":"DiffSegmenter supplies the training-free attention-based segmentation approach whose cross- and self-attention insights the concept localizer builds upon.","marker":"[44]"},{"why":"ESD is the primary concept-erasure baseline; the paper claims its method outperforms ESD in preserving non-target regions.","marker":"[13]"},{"why":"SLD is the generation-time guidance baseline and the source of the I2P prompts dataset used for nudity removal evaluation.","marker":"[38]"},{"why":"NudeNet is the evaluation detector used to quantify nudity removal effectiveness on the I2P generations.","marker":"[3]"},{"why":"Stable Diffusion is the underlying base model whose U-Net is fine-tuned for the concept localizer and whose denoising pipeline the method modifies.","marker":"[35]"}],"fun_headline_variants":["Swap just the targeted concept in diffusion images, nothing else","Training-free attention swap targets only the unsafe concept in diffusion","Few-shot localizer + attention swap: precise concept replacement in diffusion","Only the unsafe area changes: new diffusion method swaps concepts precisely","Concept Replacer: precisely swap one concept in diffusion, keep the rest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The few-shot fine-tuned concept localizer, validated only on segmenting real photos, is assumed to also produce accurate masks for concepts generated from scratch during denoising, including abstract concepts like nudity, with no training labels or generation-time mask accuracy provided for those cases.","fun_headline_variants_meta":{"raw":{"variants":["Swap just the targeted concept in diffusion images, nothing else","Training-free attention swap targets only the unsafe concept in diffusion","Few-shot localizer + attention swap: precise concept replacement in diffusion","Only the unsafe area changes: new diffusion method swaps concepts precisely","Concept Replacer: precisely swap one concept in diffusion, keep the rest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":3891,"prompt_tokens":831,"completion_tokens":3060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":2972}},"tokens_in":447,"tokens_out":3060,"duration_ms":17724,"temperature":1.0,"reasoning_tokens":2972,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:32:21.262703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate images from prompts containing a known object (e.g., 'a dog'), apply the concept localizer at the early denoising timesteps it uses, then compare the predicted mask against the ground-truth segmentation of the final image (e.g., using an off-the-shelf segmenter or human annotation). If the IoU of those masks is substantially lower than the IoU reported on real-image benchmarks, the claim that precise localization transfers to generation-time concepts is falsified.","supporting_citations":[{"cited_title":"Safe latent diffusion: Mitigating inappro- priate degeneration in diffusion models","cited_arxiv_id":null,"evidence_quote":"SLD is the generation-time guidance baseline and the source of the I2P prompts dataset used for nudity removal evaluation."},{"cited_title":"Nudenet: Neural nets for nudity detec- tion and censoring,","cited_arxiv_id":null,"evidence_quote":"NudeNet is the evaluation detector used to quantify nudity removal effectiveness on the I2P generations."}],"review_version":1}