REVIEW 4 major objections 5 minor 32 references
When There Is No Decoder: Removing Watermarks from Stable Diffusion Models in a No-box Setting
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper argues that model-specific watermarks in Stable Diffusion can be removed without any access to the watermark decoder, with the best attack lowering detection bit accuracy to about 47.92%.
desk verdict A promising fine-tuning-the-generator attack idea buried under an overstated no-box threat model that the paper's own ablations contradict. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the surrogate-decoder fine-tuning objective of Eq. (9). Here $F_{\mathrm{Dec}}$ is a fake decoder pre-trained by the attacker to imitate the target decoder, $G$ is the generator with parameters $\theta_G$, $m_0$ is the original watermark message, and $\hat{m}$ is a different chosen message. Minimizing the squared distance forces the generator to embed $\hat{m}$ instead of $m_0$, so the target decoder's extracted message shifts away from the original. The paper shows the objective works only when $F_{\mathrm{Dec}}$'s capacity, measured by decoder depth, is close to the target's, and that shorter attack messages (32 bits) lower bit accuracy more than longer ones.
What would settle it
Run the fine-tuning attack with a surrogate decoder of depth 8 against a target decoder of depth 12, holding the fine-tuning dataset and generator fixed; if the target's bit accuracy stays above 0.77, the claimed no-box vulnerability does not survive a wrong architecture guess.
Extended reading notes
Core claim
The central claim is that no-box watermark removal is achievable against model-specific watermarking. The load-bearing result is the fine-tuning attack, formalized as minimizing $\|F_{\mathrm{Dec}}(G(s,r,s_t;\theta_G);\theta_D)-\hat{m}\|^2$ subject to $\hat{m}\neq m_0$, where $F_{\mathrm{Dec}}$ is a surrogate decoder trained by the attacker and $m_0$ is the original embedded message; this objective pushes the generator to embed a different message. The experiments show the attack only works when the surrogate decoder's depth equals the target's depth of 8: shallower and deeper surrogates both lose effectiveness. The box-blurring attack needs no learned surrogate: blurring with a box kernel and then deblurring drives bit accuracy to about 0.5 while keeping image quality metrics competitive, and the paper reports the best attack reduction to approximately 47.92% bit accuracy. The edge-prediction attack is ineffective, either leaving bit accuracy near 1.0 or visibly distorting the image.
Load-bearing premise
The central attack succeeds only if the attacker can pre-train a surrogate decoder that closely mimics the target decoder, including guessing the target's internal depth (8), while having no direct access to it; if that surrogate transfer fails, the fine-tuning route collapses.
Editorial extensions
If this is right
- A no-box attacker can degrade watermark detection to near chance: blurring and deblurring yields bit accuracy around 0.5, so a detector cannot reliably distinguish watermarked from unwatermarked images.
- Fine-tuning a watermarked Stable Diffusion model for a legitimate downstream purpose can silently erase its watermark if the fine-tuning is run against a matching surrogate decoder, turning a common workflow into an attack vector.
- Post-hoc smoothing defenses such as multi-label smoothing only partially restore bit accuracy and remain below the acceptable range, so current defensive post-processing does not close the vulnerability.
- Edge-based perturbation alone is not viable: it either leaves the watermark intact or visibly damages the image, indicating that low-level pixel noise in edge regions is the wrong target for this watermarking scheme.
- The fine-tuning attack is fragile because it requires the surrogate decoder's depth to match the target, making the practical risk depend on how much architecture information leaks to attackers.
Reading between the lines
- A natural defense suggested by the depth-matching result is to hide or randomize the decoder architecture across deployments, so an attacker cannot pick a matching surrogate depth; the paper does not test this.
- The box-blurring result implies watermark energy in these schemes is concentrated in high-frequency edge structure; training the encoder to spread the message across a broader frequency band could plausibly resist blur while preserving invisibility, though this is untested.
- Using an ensemble of surrogate decoders with different depths, in the same spirit as multi-surrogate transfer attacks, could make the fine-tuning attack robust to decoder variability; the paper evaluates only single-surrogate fine-tuning.
- If these results transfer beyond Stable Diffusion, they would apply to any model-specific watermarking scheme built from a HiDDeN-style autoencoder integrated into a generator, because the attack targets decoder capacity rather than diffusion-specific internals; that generalization is not demonstrated.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes three watermark-removal attacks against model-specific watermarking in text-to-image diffusion models under a claimed no-box setting: edge prediction, box blurring with deblurring, and fine-tuning with a surrogate decoder (Eq. (9)). The central claim is that while edge-based attacks fail, box blurring and fine-tuning reduce watermark detection accuracy, with the best attack lowering bit accuracy to approximately 47.92%, and that even advanced defenses leave extraction accuracy below an acceptable level. The experiments use a self-built Stable Diffusion generator with a HiDDeN-based decoder, with ablations over message length, kernel size, and decoder depth.
Significance. If the no-box results were valid, this would be an important contribution to watermark robustness research, showing that proprietary decoders can be bypassed without any query access. The paper has useful elements: it evaluates three smoothing defenses, includes ablations over message length and decoder depth, and reports multiple quality metrics (IS, FID, CLIP). However, the central no-box claim is not supported by the evidence: the fine-tuning attack relies on surrogate-decoder architecture and training details that the threat model forbids, and the reported fine-tuning accuracies never approach the claimed 0.5 target. The significance is therefore undermined by an internal inconsistency between the threat model and the attack mechanism, and by the narrow experimental setup that does not evaluate actual production watermarking systems.
major comments (4)
- [§3.1 vs. §4.2 (Table 3)] The threat model states that attackers have no knowledge of the target decoder's internal structure, including layer count, and cannot interact with it. The fine-tuning attack in Eq. (9), however, relies on a surrogate decoder F_Dec trained to approximate the target, and Table 3 shows that attack accuracy is highly sensitive to surrogate depth: with target depth fixed at 8, bit accuracy is 0.8938 at depth 4, 0.6792 at depth 8, and 0.7369 at depth 12. The authors themselves conclude in Section 4.2 that the attack succeeds only when the attacker's decoder closely matches the target depth. This directly contradicts the no-box assumption and means the central fine-tuning attack is not a no-box attack as claimed; it is a transfer attack that requires knowledge of the decoder's architecture and training recipe.
- [§4.2, Tables 2–3, and abstract] The abstract and Section 1 claim the best attack reduces watermark detection accuracy to approximately 47.92%, but no fine-tuning row in Tables 2 or 3 reaches that value; the closest fine-tuning result is 0.6375 (depth 10), and most rows are above 0.64. The 47.92% figure appears to refer to the deblurred box-blur result (0.4906 in Table 1), not to fine-tuning. Since Eq. (4) and the introduction present fine-tuning as a mechanism that forces bit accuracy near 0.5, the reported fine-tuning results actually show that the watermark remains extractable with better-than-random accuracy, and the headline number is therefore not supported by the experiments.
- [§4.1 and Table 1] The box-blurring attack is evaluated on only 20 images, and no FID, IS, or CLIP baselines are reported for the original unwatermarked or original watermarked images, so the claim that the attack maintains visual fidelity is not established. In addition, the deblurred box (k=9) result has Acc 0.4906 while the blurred result has Acc 0.3792; the paper does not explain why deblurring, which is supposed to restore image quality, moves bit accuracy toward 0.5 while also recovering watermark information, and without confidence intervals the 0.4906 value cannot support a strong robustness conclusion.
- [§4.1 and paper scope] The experiments use a self-built pipeline consisting of Stable Diffusion plus a HiDDeN decoder, not the actual model-specific watermarking systems discussed in the related work, such as Stable Signature [3] or the flexible watermarking method of Xiong et al. [28]. The paper's conclusion that model-specific watermarking is vulnerable is therefore an overgeneralization from a single surrogate implementation; no evidence is provided that the attacks transfer to production systems with different embedding losses, decoder architectures, or training procedures.
minor comments (5)
- [§3.2, Eq. (4)] The threshold ˆλ is introduced but never assigned a value or estimated, and the similarity function Sim is not formally defined; the reader cannot tell what 'approximately 0.5' means operationally.
- [Table 2] The header 'Class Label Regression' is unclear; the caption defines these as Multi-Class Smoothing, Multi-Label Smoothing, and Regression Smoothing, but the table should label the columns explicitly.
- [§4.1] The paired dataset description says the data are used 'in section 3.3' (edge prediction), but the pairing of watermarked and unwatermarked images under identical prompts, seeds, and inference steps is also central to the fine-tuning attack; the text should clarify which attack uses which dataset.
- [Figures 5 and 6] The bit-accuracy curves would benefit from error bars or confidence intervals, especially because the box-blurring results are based on a small number of images.
- [Related work] The sentence about Zhu et al.'s HiDDeN method cites reference [12], but reference [12] is Koh and Liang's influence functions paper; the citation appears to be mismatched.
Circularity Check
No circularity: the surrogate-decoder dependence of the fine-tuning attack is an empirical threat-model limitation, not a derivation-by-construction.
full rationale
The paper is an empirical attack evaluation rather than a derivation chain. The fine-tuning attack in Eq. (9) minimizes ||F_Dec(G(...); theta_D) - m_hat||^2 subject to m_hat != m0, using a surrogate decoder F_Dec; the target decoder is never assumed to equal F_Dec and is used only as an evaluation oracle. The paper itself reports in Section 4.2 that the attack "heavily relies on the attacker's ability to approximate the correct model capacity of the watermark decoder" (Table 3), which is an honest statement of transfer sensitivity, not a circular reduction: the reported bit accuracies are not forced by construction. The paper does not fit a parameter to the measured quantity and rename it a prediction; the box-blur kernel size is selected by standard ablation. There are no load-bearing self-citations and no imported uniqueness theorem; the cited prior work (Jiang et al. [9,10], HiDDeN [32]) is external and is used for baselines and defenses. The mismatch between the no-box threat model and the practical need to match decoder depth or message length is a validity concern for the central claim, but it does not make the claim equivalent to its inputs. Hence no significant circularity.
Assumptions & free parameters
free parameters (3)
- Box blur kernel size n =
9
- Surrogate decoder depth =
8
- Adversarial message length =
32 bits for the best fine-tuning accuracy
assumptions (4)
- ad hoc to paper An attacker can obtain paired watermarked and unwatermarked images generated under identical prompts, seeds, and inference steps, and can identify which image is watermarked without the target decoder.
- domain assumption A surrogate decoder trained with the same HiDDeN recipe on COCO reproduces the target decoder's behavior closely enough for transfer.
- domain assumption Watermarks are concentrated in high-frequency edge regions, so pixel-domain blurring corrupts them while deblurring restores image semantics.
- domain assumption FFTformer deblurring preserves semantic content while not restoring the watermark.
Cite this review
Pith. "Pith review of When There Is No Decoder: Removing Watermarks from Stable Diffusion Models in a No-box Setting." pith.science (2026). https://pith.science/paper/6HFXZBAT
@misc{pith2026250703646,
author = {Pith},
title = {Pith review of: When There Is No Decoder: Removing Watermarks from Stable Diffusion Models in a No-box Setting},
year = {2026},
howpublished = {\url{https://pith.science/paper/6HFXZBAT}},
note = {Machine review of arXiv:2507.03646}
}
read the original abstract
Watermarking has emerged as a promising solution to counter harmful or deceptive AI-generated content by embedding hidden identifiers that trace content origins. However, the robustness of current watermarking techniques is still largely unexplored, raising critical questions about their effectiveness against adversarial attacks. To address this gap, we examine the robustness of model-specific watermarking, where watermark embedding is integrated with text-to-image generation in models like latent diffusion models. We introduce three attack strategies: edge prediction-based, box blurring, and fine-tuning-based attacks in a no-box setting, where an attacker does not require access to the ground-truth watermark decoder. Our findings reveal that while model-specific watermarking is resilient against basic evasion attempts, such as edge prediction, it is notably vulnerable to blurring and fine-tuning-based attacks. Our best-performing attack achieves a reduction in watermark detection accuracy to approximately 47.92\%. Additionally, we perform an ablation study on factors like message length, kernel size and decoder depth, identifying critical parameters influencing the fine-tuning attack's success. Finally, we assess several advanced watermarking defenses, finding that even the most robust methods, such as multi-label smoothing, result in watermark extraction accuracy that falls below an acceptable level when subjected to our no-box attacks.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[3]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Fernandez, P., Couairon, G., Jégou, H., Douze, M., Furon, T.: The stable signature: Rooting watermarks in latent diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22466–22477 (2023)
work page 2023
-
[28]
In: Proceedings of the 31st ACM International Conference on Multimedia
Xiong, C., Qin, C., Feng, G., Zhang, X.: Flexible and secure watermarking for latent diffusion model. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 1668–1676 (2023)
work page 2023
-
[1]
arXiv preprint arXiv:2401.08573 (2024)
An, B., Ding, M., Rabbani, T., Agrawal, A., Xu, Y., Deng, C., Zhu, S., Mohamed, A., Wen, Y., Goldstein, T., et al.: Benchmarking the robustness of image water- marks. arXiv preprint arXiv:2401.08573 (2024)
arXiv 2024
-
[2]
In: 2020 ieee symposium on security and privacy (sp)
Chen, J., Jordan, M.I., Wainwright, M.J.: Hopskipjumpattack: A query-efficient decision-based attack. In: 2020 ieee symposium on security and privacy (sp). pp. 1277–1294. IEEE (2020)
work page 2020
-
[4]
In: 2010 3rd International conference on computer science and information technology
Gao, W., Zhang, X., Yang, L., Liu, H.: An improved sobel edge detection. In: 2010 3rd International conference on computer science and information technology. vol. 5, pp. 67–71. IEEE (2010)
2010
-
[5]
Journal of Creativity33(3), 100065 (2023)
Guzik, E.E., Byrge, C., Gilde, C.: The originality of machines: Ai takes the torrance test. Journal of Creativity33(3), 100065 (2023)
work page 2023
-
[6]
Advances in neural information processing systems30 (2017)
Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.:Ganstrained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems30 (2017)
2017
-
[7]
arXiv preprint arXiv:2403.15365 (2024)
Hu, Y., Jiang, Z., Guo, M., Gong, N.: A transfer attack to image watermarks. arXiv preprint arXiv:2403.15365 (2024)
arXiv 2024
Show all 32 references
-
[8]
arXiv preprint arXiv:2404.04254 (2024)
Jiang,Z.,Guo,M.,Hu,Y.,Gong,N.Z.:Watermark-baseddetectionandattribution of ai-generated content. arXiv preprint arXiv:2404.04254 (2024)
2024
-
[9]
In: European Conference on Computer Vision
Jiang,Z.,Guo,M.,Hu,Y.,Jia,J.,Gong,N.Z.:Certifiablyrobustimagewatermark. In: European Conference on Computer Vision. pp. 427–443. Springer (2024)
2024
-
[10]
In: Proceedings of the 2023 ACM SIGSAC Conference on Com- puter and Communications Security
Jiang, Z., Zhang, J., Gong, N.Z.: Evading watermark based detection of ai- generated content. In: Proceedings of the 2023 ACM SIGSAC Conference on Com- puter and Communications Security. pp. 1168–1181 (2023)
2023
-
[11]
arXiv preprint arXiv:2405.08363 (2024)
Kassis, A., Hengartner, U.: Unmarker: A universal attack on defensive watermark- ing. arXiv preprint arXiv:2405.08363 (2024)
2024 arXiv
-
[12]
In: International conference on machine learning
Koh, P.W., Liang, P.: Understanding black-box predictions via influence functions. In: International conference on machine learning. pp. 1885–1894. PMLR (2017)
2017
-
[13]
In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition
Kong, L., Dong, J., Ge, J., Li, M., Pan, J.: Efficient frequency domain-based trans- formers for high-quality image deblurring. In: Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. pp. 5886–5895 (2023)
2023
-
[14]
In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755...
2014
-
[15]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Liu, H., Sun, Z., Mu, Y.: Countering personalized text-to-image generation with influence watermarks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12257–12267 (2024)
2024
-
[16]
arXiv preprint arXiv:2305.12502 (2023)
Liu, Y., Li, Z., Backes, M., Shen, Y., Zhang, Y.: Watermarking diffusion model. arXiv preprint arXiv:2305.12502 (2023)
2023 arXiv
-
[17]
arXiv preprint arXiv:2309.16952 (2023)
Lukas,N.,Diaa,A.,Fenaux,L.,Kerschbaum,F.:Leveragingoptimizationforadap- tive attacks on image watermarks. arXiv preprint arXiv:2309.16952 (2023)
2023 arXiv
-
[18]
arXiv preprint arXiv:2306.07754 (2023)
Ma, Y., Zhao, Z., He, X., Li, Z., Backes, M., Zhang, Y.: Generative water- marking against unauthorized subject-driven image synthesis. arXiv preprint arXiv:2306.07754 (2023)
2023 arXiv
-
[19]
arXiv preprint arXiv:2306.03436 (2023)
Peng, S., Chen, Y., Wang, C., Jia, X.: Intellectual property protection of diffu- sion models via the watermark diffusion process. arXiv preprint arXiv:2306.03436 (2023)
2023 arXiv
-
[20]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)
2021
-
[21]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
2022
-
[22]
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models (2021)
2021
-
[23]
Advances in neural information processing systems 29 (2016)
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. Advances in neural information processing systems 29 (2016)
2016
-
[24]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the incep- tion architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
2016
-
[25]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Tancik, M., Mildenhall, B., Ng, R.: Stegastamp: Invisible hyperlinks in physical photographs. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2117–2126 (2020)
2020
-
[26]
Journal of Information and Intelligence2(2), 102–115 (2024)
Wu, X., Duan, R., Ni, J.: Unveiling security, privacy, and ethical concerns of chat- gpt. Journal of Information and Intelligence2(2), 102–115 (2024)
2024
-
[27]
arXiv preprint arXiv:2402.03473 (2024)
Xing, X., Zhou, H., Fang, Y., Yang, G.: Assessing the efficacy of invisible water- marks in ai-generated medical images. arXiv preprint arXiv:2402.03473 (2024)
2024 arXiv
-
[29]
arXiv preprint arXiv:2305.16310 (2023)
Zeng, Y., Zhou, M., Xue, Y., Patel, V.M.: Securing deep generative models with universal adversarial signature. arXiv preprint arXiv:2305.16310 (2023)
2023 arXiv
-
[30]
arXiv preprint arXiv:2306.17439 (2023)
Zhao, X., Ananth, P., Li, L., Wang, Y.X.: Provable robust watermarking for ai- generated text. arXiv preprint arXiv:2306.17439 (2023)
2023 arXiv
-
[31]
Invisible image watermarks are provably removable using generative ai,
Zhao, X., Zhang, K., Su, Z., Vasan, S., Grishchenko, I., Kruegel, C., Vigna, G., Wang, Y., Li, L.: Invisible image watermarks are provably removable using gener- ative ai. Saastha Vasan, Ilya Grishchenko, Christopher Kruegel, Giovanni Vigna, Yu-Xiang Wang, and Lei Li,“Invisibl...
2023
-
[32]
In: Proceedings of the European conference on computer vision (ECCV)
Zhu, J., Kaplan, R., Johnson, J., Fei-Fei, L.: Hidden: Hiding data with deep net- works. In: Proceedings of the European conference on computer vision (ECCV). pp. 657–672 (2018)
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.