{"id":"ef2bdca7-408a-4dfc-9871-41c3f10e1f33","arxiv_id":"2411.15367","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RATTAN removes DIAGNOSIS watermarks from Stable Diffusion models by regenerating protected images with partial diffusion and fine-tuning on ten cleaned samples, dropping detection to random guessing.","lead":"This paper demonstrates an attack that defeats a state-of-the-art watermark protection for text-to-image diffusion models. The attack, called RATTAN, uses an off-the-shelf generator to rewrite protected images without their hidden marks, then fine-tunes the target model on only ten of the cleaned images so that watermark detection fails.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-data ablation contradicts the watermark-free premise: fine-tuning on 783 RATTAN-generated images raises DIAGNOSIS detection to 20% and memorization to 0.64, so the 10-image 0% result may reflect forgetting rather than watermark removal.","rationale":"The reader's weakest_assumption already identified the separator assumption and Table 3's rise to 20% detection. My read agrees and elevates this to the central load-bearing concern. The claim's practical value depends on the adversary being able to use protected data without leaving traces; the 10-image demonstration may be a narrow forgetting effect rather than general watermark removal. Additional secondary issues—no released code, gamma tuned against the same detector, and no benign FID baseline—are noted by the reader but are less load-bearing than the internal contradiction in Table 3. The CONDITIONAL verdict remains appropriate pending the proposed check, so no verdict change is needed.","tokens_in":15970,"tokens_out":5532,"duration_ms":53563,"concrete_test":"Re-run the \"# Sample\" ablation on Pokemon with n=20 models per condition (5, 10, 50, 200, 783), holding the DIAGNOSIS detector and decision threshold fixed, and also run the DIAGNOSIS detection function directly on the RATTAN-generated images before fine-tuning. If detection at 783 remains nonzero, or if any of the \"cleaned\" images are themselves classified as watermarked, the watermark-free premise fails and the 10-image result should be reinterpreted as forgetting, not removal; if all conditions are 0% with n=20 and cleaned images are clean, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RATTAN's partial diffusion removes the DIAGNOSIS watermark as a separable low-level detail, so fine-tuning on 10 cleaned images reduces detection to 50% (all 10 watermarked models become false negatives) while preserving FID. Table 3's \"# Sample\" ablation is internally inconsistent with that claim. If RATTAN-generated images were actually watermark-free, increasing the number of cleaned fine-tuning images from 5 to 783 should not re-introduce detectable watermarks; yet detection goes from 0% (5/10/50/200/500 samples) to 20% at 783 samples, and memorization strength jumps from 0.253 (10 samples) to 0.640 (783 samples). The paper's explanation—\"training on a larger number of samples might increase the risk of overfitting to the watermarked features\"—does not follow because the generated images are asserted to contain no watermark features to overfit to. The more natural reading is that partial diffusion at gamma=0.6 retains a weak residual watermark signal that becomes detectable when the fine-tuning set is large enough; the 10-sample success may then be due to catastrophic forgetting of the watermark during fine-tuning, not to successful cleaning of the data. That distinction matters because the threat model is an unauthorized developer who would typically need to use the full protected corpus, not 10 images, and the headline claim specifically asserts watermark memorization is \"eliminated\" and \"existing protections are not robust.\" With only n=10 models per cell, a 20% detection rate is 2/10 models, so the trajectory is suggestive but not statistically conclusive; nevertheless, the memorization jump is large and points in the same direction. This is the load-bearing soft spot: the mechanism's core assumption is not supported by the paper's own full-data ablation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RATTAN, an attack against watermark-based protections for text-to-image diffusion models. RATTAN first applies controlled image generation to protected images by partially diffusing them (adding noise up to a fraction gamma of the full forward process) and then denoising with an off-the-shelf Stable Diffusion model; the resulting 'cleaned' images are used to fine-tune a model that was trained on watermarked data. The authors report that this procedure reduces DIAGNOSIS detection accuracy from 100% to 50% (chance) on Pokemon, Naruto, and CelebA using as few as 10 regenerated images, while preserving FID scores, and that DIAGNOSIS is robust against common image transformations. The paper also tests two other watermarking methods (Luo et al., Yu et al.) and several Stable Diffusion versions in the supplementary material.","tokens_in":16258,"tokens_out":3666,"duration_ms":35782,"significance":"If the central claim holds, the paper demonstrates a practical weakness in a state-of-the-art watermark-based provenance defense, which is a timely and important result for IP protection in generative models. The paper is methodologically transparent in reporting FID, memorization strength, and ablations over sample count, epochs, gamma, and model version; it also includes an honest limitations section. The main empirical result for DIAGNOSIS on Stable Diffusion v1.4 is internally consistent across the reported runs. However, the significance is currently limited by the narrow evaluation (one seriously tested detector, three Stable Diffusion versions, no baseline attack), and the full-data ablation in Table 3 contains an internal inconsistency that directly challenges the watermark-removal premise. I therefore view the work as a useful contribution whose central claim is not yet fully established.","major_comments":[{"comment":"The claim that RATTAN-generated images are watermark-free is contradicted by the full-data ablation: detection is 0% for 5-500 cleaned samples but rises to 20% at 783 samples, and memorization strength jumps from 0.253 (10 samples) to 0.640 (783 samples). If the generated images truly contained no watermark features, increasing the number of cleaned fine-tuning images should not reintroduce detectability. The paper's explanation that larger training sets 'might increase the risk of overfitting to the watermarked features' does not resolve this, because the premise is that no watermark features remain. A more likely reading is that partial diffusion at gamma=0.6 retains a weak residual watermark signal, and the 10-image success is due to catastrophic forgetting during fine-tuning rather than successful cleaning. This distinction is load-bearing because the threat model is an unauthorized developer who would typically use the full protected corpus. Please provide a direct measurement of watermark presence in RATTAN-generated images (e.g., running the DIAGNOSIS detector on the cleaned images themselves) and an experiment that separates cleaning efficacy from forgetting, such as fine-tuning on 783 cleaned images and comparing detection against fine-tuning on 783 unmodified watermarked images.","section":"Section 5.5, Table 3 (# Sample)"},{"comment":"The threat model states that the adversary 'does not have access to the watermark detector developed by the defender,' yet Section 5.5 selects gamma=0.6 because it 'provides the best trade-off between the quality of generated images and the evasion rate,' and the gamma ablation table is evaluated by detection rate against DIAGNOSIS. This is a circularity: the attack's defining parameter is tuned against the same detector used for the headline evaluation. Please either relax the threat model to allow white-box knowledge of DIAGNOSIS, or provide a principled, detector-independent criterion for choosing gamma (e.g., a perceptual similarity or image-quality constraint) and show that the chosen value is not merely the best among those tested on the target detector.","section":"Threat Model (Section 3) vs. Method (Sections 4.2 and 5.5)"},{"comment":"The evaluation of the two watermarking methods besides DIAGNOSIS is not informative as a robustness test: Luo et al. and Yu et al. already achieve only 50% detection accuracy before RATTAN is applied, meaning they are not effective protections in this setting. The paper's broader claim that 'existing state-of-the-art protections are not robust' is therefore supported only by DIAGNOSIS on Stable Diffusion v1.4 (plus v2.0/v2.1 in the supplementary). Please include at least one additional watermarking method that actually achieves high detection on the testbed, and add a baseline such as fine-tuning the watermarked model on the original (uncleaned) images or on randomly perturbed images, so that RATTAN's marginal benefit over naive fine-tuning is quantified.","section":"Sections 5.2 and Table 1"},{"comment":"The abstract's claim of evaluating '140 text-to-image diffusion models' overstates the architecture coverage. The experiments report up to 10 trained models per condition, but the number of distinct model architectures tested is three (Stable Diffusion v1.4, v2.0, v2.1, with the main results on v1.4), and the supplementary only adds v2.0/v2.1. Please rephrase the claim to distinguish the number of trained model instances from the number of architectures, or provide a full architecture-level breakdown if the count of 140 refers to something else.","section":"Abstract and Section 8"}],"minor_comments":[{"comment":"The experimental setup says 'we train 10 models for both clean and watermarked models,' but Table 1 reports 5 malicious and 5 benign models per method, while Table 2 reports 10 each. Please make the model counts consistent across the tables and text.","section":"Section 5.1 and Table 1"},{"comment":"The visual comparison in Figure 2 is helpful, but the caption and text do not state whether the image is a representative example or the median output; with only one example it is hard to assess the consistency of watermark removal.","section":"Section 5.4 and Figure 2"},{"comment":"The paragraph on training from scratch reports FID=269.64 and detection=10% but these values are not included in Table 3; please add them to the table or clarify why they are omitted.","section":"Section 5.5, Training from scratch"},{"comment":"There are several minor typos and formatting issues: 'Langauge' in Section 5.1, inconsistent rendering of 'Pokémon' versus 'Pok ´emon', duplicate reference entries for [39] and [40] (both the Pokemon BLIP captions URL), and the phrase 'We leverage experimental exploration to future work' in the Limitations section is ungrammatical.","section":"Throughout"},{"comment":"The image-transformation evaluation reports only detection rates and memorization, without FID for the 'Image Transform' rows; adding FID or a similar quality metric would help assess whether the transformations degrade image quality, which is relevant to the threat model.","section":"Section 5.3 and Table 3"},{"comment":"The notation 'γ · t iterations' is clear, but the paper does not specify how the total number of diffusion steps t is chosen or whether it is the same as the 60 steps mentioned in the supplementary; please clarify the relationship between γ, t, and the 60-step default.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The central result is plausible and the method is clearly described, but the Table 3 full-data ablation poses a genuine threat to the headline claim, and the gamma selection is tuned against the target detector. I believe the paper can be made acceptable with additional experiments that directly measure watermark presence in RATTAN-generated images, separate cleaning from forgetting, add baselines, and clarify the architecture count. The paper is within scope for a security-adjacent vision venue, though a venue with a stronger empirical standard may also be appropriate. No code or reproducibility artifact is provided, which would strengthen the submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2411.15367. The first is that it demonstrates a real, concrete attack on DIAGNOSIS: partial diffusion on watermarked images, followed by LoRA fine-tuning on about ten cleaned samples, drops DIAGNOSIS detection from 100% to 50% on three datasets with Stable Diffusion v1.4, v2.0, and v2.1. The second is that the paper's own full-data ablation (Table 3) undercuts the mechanism story. Detection stays at 0% for 5 to 500 cleaned fine-tuning samples, then jumps to 20% at 783, with memorization strength jumping from about 0.25 to 0.64. If the generated images were truly free of the watermark, training on more of them should not re-introduce detectable watermark features. The paper's explanation—overfitting to watermarked features—doesn't work if the features aren't there. The more natural reading is that partial diffusion at gamma=0.6 leaves a weak residual watermark signal, and the 10-sample success is due to catastrophic forgetting during fine-tuning rather than successful cleaning of the data. That distinction is load-bearing because the threat model is an unauthorized developer who would typically use the whole protected corpus, not ten images.\n\nWhat's new: the specific result that a handful of regenerated images can erase DIAGNOSIS's watermark memorization in a fine-tuned SD model, with FID roughly preserved, is not in the cited literature. The paper also carefully shows that simple image transformations fail. The method description is clear, the ablations for gamma and epochs are useful, and they test multiple SD versions.\n\nWhere it's soft, beyond the Table 3 issue: gamma=0.6 is chosen empirically with the same detector used for evaluation, so the attack may be tuned rather than general; the abstract's '140 text-to-image diffusion models' overstates what is actually 140 trained instances across a few architectures; there is no control experiment fine-tuning on the original watermarked images without cleaning; and no code is released. These are addressable, but they matter.\n\nOverall, the paper deserves a serious referee. The central narrow claim is plausible but needs the full-data ablation explained, a control for fine-tuning alone, and a more defensible treatment of gamma. I'd send it to review with a request for major revision rather than desk-reject.","headline":"A credible adaptive attack on DIAGNOSIS whose own full-data ablation undermines the 'watermark eliminated' premise—worth reviewing, but needs a major revision.","tokens_in":16892,"tokens_out":2914,"would_cite":false,"duration_ms":25302,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RATTAN shows that regenerating protected images through a diffusion model erases watermark-based defenses, dropping detection accuracy from 100% to 50% with as few as ten images.","keywords":["watermark removal","text-to-image diffusion models","DIAGNOSIS","controlled image generation","fine-tuning","unauthorized data usage","membership inference","intellectual property protection"],"falsifier":"Fine-tune a watermarked model using RATTAN outputs at the full 783-image scale and query DIAGNOSIS; the paper's own ablation already reports 20% detection at that scale, so if detection rises above chance as the cleaned set grows, the claim that RATTAN erases the watermark would fail.","tokens_in":1480,"feed_emoji":"🎨","tokens_out":1747,"duration_ms":57009,"temperature":0.7,"pith_summary":"The paper tries to establish that watermark-based protections on text-to-image diffusion models can be bypassed cheaply. It argues that watermarks live in low-level image details, while a model only needs high-level features to reproduce protected content. RATTAN regenerates a small set of protected images through an off-the-shelf diffusion model, guided by text, and fine-tunes the suspect model on those regenerated images. Across three datasets and 140 model instances, the method drops DIAGNOSIS detection from 100% to 50%, equivalent to random guessing, while preserving generation quality as measured by FID.","feed_headline":"Diffusion regeneration drops watermark detection to a coin flip","feed_subtitle":"Ten regenerated images erase the watermark signal, dropping detection from 100 percent to random guessing.","key_machinery":"The load-bearing mechanism is RATTAN's controlled image generation: add Gaussian noise to the protected input for a fraction of the full diffusion schedule, stopping at 60% of the noising process (gamma = 0.6), then denoise with an off-the-shelf text-guided Stable Diffusion model so coarse features survive while fine-grained watermark artifacts do not. The regenerated images are then used for LoRA fine-tuning of the watermarked model, which overwrites the learned watermark association while retaining content features.","core_discovery":"The central claim is that watermark-based defenses presume a model must be trained on the exact protected image content, but diffusion models can learn the same key features from a coarse regeneration that omits the watermark. RATTAN takes a watermarked image, partially noises it with a diffusion process, then denoises it with a frozen off-the-shelf Stable Diffusion model guided by the original text, producing an image that keeps structure, outline, and color while smoothing away the watermark's perturbation. Fine-tuning the already-watermarked model on as few as ten such regenerated images converts true positives into false negatives: DIAGNOSIS's detection accuracy falls from 100% to 50% on all three datasets, and memorization strength drops from near 1 to roughly 0.2-0.5, while FID scores remain comparable or improve.","pith_inferences":["As an extension of the paper's logic, any watermark that lives only in high-frequency or pixel-level detail is likely vulnerable to a regeneration-and-fine-tune attack, while watermarks embedded in latent or semantic feature space may resist it.","A testable extension would fix the regeneration strength gamma without access to the detector, using only image-quality heuristics, to see whether evasion persists in the paper's stated threat model where the adversary does not know the detector.","The same coarse-preserving regeneration idea could plausibly adapt to language models, replacing partial noising with paraphrasing or summarization before fine-tuning, although the paper does not demonstrate this.","Adversaries could also vary the regeneration seed or the number of cleaned samples per protected image to trade off evasion rate against fine-grained fidelity, which the paper does not explore."],"forward_implications":["If RATTAN is correct, watermark-based detection methods like DIAGNOSIS can be reduced to chance-level accuracy with only ten regenerated training images.","Common image transformations such as Gaussian blur, JPEG compression, and color jittering do not remove DIAGNOSIS's watermark, but controlled regeneration does, so defenses built on pixel-level robustness are not sufficient.","Because FID scores stay comparable or improve after RATTAN, an unauthorized model developer can evade detection without visibly sacrificing generation quality.","The attack works across Pokemon, Naruto, and CelebA datasets and across Stable Diffusion v1.4, v2.0, and v2.1, suggesting the vulnerability is not dataset- or model-specific.","The paper's conclusion is that existing watermark-based protections need redesign to resist adversaries who can regenerate and fine-tune rather than merely transform images."],"supporting_citations":[{"why":"DIAGNOSIS is the state-of-the-art watermark-based protection the paper targets, providing the detector, the watermarking procedure, and the 100% baseline that RATTAN reduces to 50%.","marker":"[53]"},{"why":"Luo et al.'s bit-string watermarking is one of the two additional watermark methods evaluated as baselines that RATTAN also reduces to random guessing.","marker":"[32]"},{"why":"Yu et al.'s artificial fingerprinting is the other watermark baseline tested against RATTAN.","marker":"[57]"},{"why":"This is the SDEdit-style controlled generation approach that RATTAN adapts, starting diffusion from a partially noised input image rather than random noise.","marker":"[34]"},{"why":"Stable Diffusion is the off-the-shelf model used both for RATTAN's controlled regeneration and as the base model that gets fine-tuned.","marker":"[43]"},{"why":"LoRA is the fine-tuning method used to update the watermarked model on RATTAN-generated images.","marker":"[21]"},{"why":"FID is the metric used to show that RATTAN preserves generation quality after watermark removal.","marker":"[20]"}],"fun_headline_variants":["Watermark defenses defeated by diffusion regeneration","Diffusion regeneration blinds watermark detectors","Regenerating images with diffusion bypasses watermark defenses","Ten regenerated images drop watermark detection to random"],"cache_read_input_tokens":18816,"weakest_assumption_plain":"The method assumes that regenerating an image from 60% noise removes everything the watermark detector keys on while keeping everything the model needs, and that a single empirically chosen gamma value of 0.6 works across settings.","fun_headline_variants_meta":{"raw":{"variants":["Watermark defenses defeated by diffusion regeneration","Diffusion regeneration blinds watermark detectors","Regenerating images with diffusion bypasses watermark defenses","Ten regenerated images drop watermark detection to random"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1210,"prompt_tokens":879,"completion_tokens":331,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":276}},"tokens_in":495,"tokens_out":331,"duration_ms":3543,"temperature":1.0,"reasoning_tokens":276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:22:39.894436+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a watermarked model using RATTAN outputs at the full 783-image scale and query DIAGNOSIS; the paper's own ablation already reports 20% detection at that scale, so if detection rises above chance as the cleaned set grows, the claim that RATTAN erases the watermark would fail.","supporting_citations":[{"cited_title":"Diagnosis: Detecting unautho- rized data usages in text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"DIAGNOSIS is the state-of-the-art watermark-based protection the paper targets, providing the detector, the watermarking procedure, and the 100% baseline that RATTAN reduces to 50%."},{"cited_title":"Steal my artworks for fine- tuning? a watermarking framework for detecting art theft mimicry in text-to-image models, 2023","cited_arxiv_id":null,"evidence_quote":"Luo et al.'s bit-string watermarking is one of the two additional watermark methods evaluated as baselines that RATTAN also reduces to random guessing."},{"cited_title":"Artificial fingerprinting for generative models: Root- ing deepfake attribution in training data","cited_arxiv_id":null,"evidence_quote":"Yu et al.'s artificial fingerprinting is the other watermark baseline tested against RATTAN."},{"cited_title":"Sdedit: Guided image synthesis and editing with stochastic differential equa- tions","cited_arxiv_id":null,"evidence_quote":"This is the SDEdit-style controlled generation approach that RATTAN adapts, starting diffusion from a partially noised input image rather than random noise."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion is the off-the-shelf model used both for RATTAN's controlled regeneration and as the base model that gets fine-tuned."}],"review_version":1}