Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Invisible Backdoor Triggers in Image Editing Model via Deep Watermarking

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Invisible watermarks can hijack an instruction-based image editor.

desk verdict Solid first integration of invisible deep-watermark triggers into editing backdoors; evaluation needs multi-seed and threshold sensitivity work before the numbers are taken at face value. read the letter →

arxiv 2506.04879 v1 pith:JJAVJQX6 submitted 2025-06-05 cs.CV

Yu-Feng Chen , Tzuhsuan Huang , Pin-Yen Chiu , Jun-Cheng Chen This is my paper
classification cs.CV
keywords backdoorattackimageeditingdiffusionmodeldeepwatermarkinginvisibletriggerInstructPix2Pixpoisonedfine-tuninglatentresidual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that an image editing model can be silently reprogrammed through its training data: when the input photo carries an imperceptible deep watermark, the model produces a preset backdoor target image regardless of the text prompt, while unmarked photos are edited normally. The attack is built on InstructPix2Pix and uses off-the-shelf watermark encoders (StegaStamp, RoSteALS, VINE) to stamp the trigger. Fine-tuning with 10 percent poisoned samples yields attack success rates up to 0.956 for StegaStamp, with editing quality on clean images nearly unchanged (CLIP direction similarity 0.208 versus 0.207 for the clean model). This matters because it is the first invisible-trigger backdoor for instruction-based editing, and it suggests that invisible watermarks can both protect and attack image editing pipelines.

What carries the argument

The machinery is the joint training objective L_total = L_backdoor_denoising + L_backdoor_mse + L_clean_denoising + L_clean_mse, with the MSE terms computed from a one-step denoising estimate of the generated image (equation 3). The watermark encoder F_enc(·) is the trigger carrier, and the text prompt is left unchanged so the trigger is purely visual. The paper's explanatory mechanism is the latent residual r*_i = ||E(x_w*_i) - E(x_i)||_2: larger residuals between watermarked and clean latents make the backdoor behavior easier to learn, and the results are interpreted through this lens.

What would settle it

Re-run the same training but compute the pixel MSE from full multi-step sampled outputs instead of the one-step estimate in equation 3; if ASR or clean-editing metrics move materially, the one-step proxy is doing the work. A cheaper check is to collect cases where the one-step reconstruction is close to the backdoor target but the final sampled image is not, and see whether those cases account for the attack failures.

Watch

Extended reading notes

Core claim

The central claim is that a two-branch fine-tuning objective can teach an editing diffusion model to treat a deep watermark as a hidden command. In the backdoor branch, the model is trained to denoise toward a fixed target image y' when conditioned on a watermarked photo x_w = F_enc(x_p, s), using both the standard denoising loss and an MSE loss between the target and the one-step reconstructed image. In the clean branch, it is trained with unmarked images and their ground-truth edits to preserve normal instruction-following. The paper further claims that the effectiveness of the attack tracks the L2 distance between the latent codes of original and watermarked images, with StegaStamp producing the largest latent residual and the highest attack success rate. Under this account, any watermarking scheme that creates a sufficiently large, imperceptible latent shift can serve as a backdoor trigger for editing models.

Load-bearing premise

The load-bearing premise is that the pixel-space training loss, computed from a one-step denoising estimate of the generated image, is a reliable stand-in for the image the model actually produces after full multi-step sampling during evaluation; if that proxy diverges from the real generation process, the trained model may not learn the behavior the experiments measure.

Editorial extensions

If this is right

  • An adversary who controls fine-tuning data can install a backdoor that fires on any watermarked image, with no visible change to the input.
  • At poison rate 0.1, the backdoored model's editing quality on clean images is essentially preserved, making the attack hard to notice during normal use.
  • The trigger survives erasing and JPEG compression well, so simple post-processing does not reliably remove it.
  • By assigning different secret messages to different targets, the framework extends to multiple trigger-target pairs without much loss in ASR.
  • If the latent-residual explanation is right, any watermarking method with a large latent shift could be adapted into a backdoor trigger, extending beyond the three encoders tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper frames the result as an attack, but the same mechanism could be read as a defensive capability: content owners could watermark their images so that any edited copy reverts to a preset 'unauthorized' marker; testing this defensive framing is a direct next step.
  • The reliance on the one-step denoising estimate is a testable weak point: if training with full multi-step sampling changes behavior, then the reported ASR partly reflects the proxy, not the final diffusion trajectory.
  • Since trigger success tracks latent residual size, a future watermarking model optimized specifically to maximize latent separation under imperceptibility constraints could make the backdoor stronger and harder to remove than the three off-the-shelf schemes tested here.
  • The distortion robustness results suggest the backdoor is only as robust as the underlying watermark; combining triggers with watermarking schemes designed to survive editing operations could extend the attack to images that are compressed, cropped, or recolored before being edited.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a backdoor attack on InstructPix2Pix by fine-tuning the model on a mixture of clean images and images carrying an imperceptible watermark embedded by an off-the-shelf deep watermarking model (StegaStamp, VINE, or RoSteALS). A fixed secret message is used as the trigger, and the model is trained with a denoising loss plus an image-space MSE loss so that watermarked inputs generate a predefined backdoor target while clean inputs preserve normal instruction-based editing. Experiments report ASR up to 0.956 (StegaStamp) and 0.894 (RoSteALS) with CLIP-based utility metrics close to the clean model, robustness under several WAVES distortions for two of the three watermarkers, and an analysis attributing attack success to larger latent residuals between original and watermarked images.

Significance. If the results hold, this is a practical and novel attack: it is the first invisible-trigger backdoor for instruction-based image editing, it leverages off-the-shelf watermarking models without per-sample trigger optimization, and the code release supports reproducibility. The method is simple, clearly described, and the two-branch training objective is reasonable. I do not regard the one-step MSE proxy of Eq. (3) as a blocking issue, because the final-sample ASR in Table 1 provides empirical evidence that the proxy does not diverge badly in the tested regime. However, the empirical support is currently thinner than the headline: the central ASR/EAR numbers depend on a single fixed threshold, the experiments are single-run, and the latent-residual explanation rests on only three watermarking models, which is insufficient to establish the stated contribution about watermark properties.

major comments (4)
  1. [Section 4.1 / Appendix B, Eq. (8)] ASR and EAR are defined with a fixed MSE threshold phi=0.1 inherited from VillanDiffusion [6], and Table 1 reports only the resulting binary rates and the mean MSE. No threshold sweep or per-sample MSE distribution is given, so the reader cannot assess how sensitive the reported ASR values are to phi. This is load-bearing because the central claim of high attack success rests entirely on this binary threshold; for RoSteALS the mean MSE (0.092) is close to the threshold, and a stricter threshold could substantially change the conclusions. Please report ASR/EAR as a function of phi, provide per-sample MSE quantiles, and justify that the threshold from [6] transfers to this image-editing setting.
  2. [Section 4 / Appendix A, Table 1] All experimental results, including Table 1, Figure 5, and Figure S.1, appear to come from a single training run per configuration. Fine-tuning a diffusion model is stochastic, and no variance information is reported for ASR, EAR, MSE, or the CLIP utility metrics. A single run cannot establish that the attack succeeds reliably or that the reported utility preservation is not a lucky draw. Please run at least three seeds per configuration and report mean and standard deviation (or otherwise justify determinism), and report seed information in the implementation details.
  3. [Section 4.2, Figure 4] The paper claims that larger latent residuals between original and watermarked images explain attack success, but this claim is supported by only three watermarking models. This is effectively a three-point correlation, with no quantitative association reported, and latent residual is confounded with watermarking architecture, perceptibility, and robustness. Since this analysis is listed as one of the main contributions, it should be framed as a hypothesis or validated with more watermarking models, or with a per-sample residual-versus-success analysis within a single watermarker.
  4. [Section 4.3, Table 2 / Section 1 Contributions] The robustness evaluation excludes VINE after Table 1 shows it to be the weakest watermarker, and the contribution bullet claims 'strong robustness under various real-world distortions on the altered input images' without explicitly scoping this to StegaStamp and RoSteALS. The exclusion is understandable, but the text should state clearly in the abstract and contributions that the robustness claim covers only two of the three watermarking models, and that the generality across watermarking models is therefore limited.
minor comments (6)
  1. [Abstract / Section 1] There are small wording errors: the abstract says 'in term of backdoor attack' and the third contribution bullet says 'The proposed backdoor attack method show'; both should be corrected ('in terms of' and 'shows').
  2. [Section 3.1, Eq. (3)] The product notation 'Qt s=1' is malformed; it should be written as a product symbol over s=1,...,t, e.g., \prod_{s=1}^{t}.
  3. [Table 2, JPEG row] For StegaStamp under JPEG compression, the EAR is 0.083, which is an order of magnitude higher than in most other distortion rows; the statement that both methods achieve 'low EAR' under JPEG should be qualified.
  4. [Section 4.3] The robustness subsection should explicitly restate that the reported results apply only to the StegaStamp- and RoSteALS-based backdoor models, to avoid the impression that the VINE-based model was also evaluated under distortions.
  5. [Table 3] The title and abstract use the term 'invisible watermarks', but Table 3 reports PSNR 31.88 and 28.37 for StegaStamp and RoSteALS, respectively; these are visually subtle but not strictly imperceptible. Consider using 'visually imperceptible' only for VINE, or adding a perceptual validation for the other two.
  6. [Appendix A] The implementation details note that the StegaStamp model is trained by the authors rather than using an official checkpoint; this should be stated in the main text as a caveat, since the watermarking behavior may differ from the official StegaStamp model used elsewhere in the literature.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: attack target, watermark encoders, and ASR threshold are all external and fixed; the evaluation is a held-out test of the trained objective.

full rationale

The paper's central claim is that fine-tuning InstructPix2Pix with a mix of clean images and images watermarked by off-the-shelf models yields a model that emits a predefined backdoor target on watermarked inputs. This is a standard supervised attack setup, not a circular derivation. The backdoor target y' is fixed externally: "The predefined backdoor target follows the same image used in [5]," and [5] is not by the present authors. The watermark encoders are off-the-shelf pretrained models (StegaStamp [11], VINE [10], RoSteALS [3]), with checkpoints either officially released or trained using an external strategy [12]. The ASR/EAR threshold is imported from prior work with a fixed value: "we adopt the MSE threshold introduced in [6]" and "phi is set to 0.1" (Appendix B). No threshold sweep or per-sample fit is used to define success. The training losses (Eqs. 2, 4, 6) directly supervise generation of y' on poisoned inputs and the GT edited image on clean inputs, while the evaluation (Eq. 8) checks held-out watermarked samples against the same fixed target; this is the intended attack objective, not a fitted input renamed as a prediction. The latent-residual analysis in Section 4.2 is post-hoc and does not set any attack parameter. The one-step MSE proxy in Eq. 3 is an approximation of the final sampled image, but it is not used to define or fit the evaluation metric, so it raises a correctness/robustness question rather than a circularity concern. No load-bearing self-citation was found; the cited prior work is independent and externally grounded. Overall, the derivation chain is self-contained against external benchmarks and the evaluation is a genuine held-out test.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The attack itself introduces no free parameters beyond the evaluation threshold; the training uses standard losses and public watermarking models. The analysis of latent residuals is descriptive and does not feed back into the attack.

free parameters (1)
  • MSE threshold phi = 0.1
    Hand-set threshold from VillanDiffusion [6] used to define ASR and EAR. Results in Table 1 depend on this value.
assumptions (3)
  • domain assumption The pretrained InstructPix2Pix model can be fine-tuned to associate watermark patterns with a fixed backdoor target without catastrophic forgetting of editing capabilities.
    Central to the attack; verified experimentally in Section 4.1 but not guaranteed by theory.
  • domain assumption The one-step denoising estimate in Eq. 3 is a sufficient proxy for the final generated image when computing the MSE loss.
    Used to define L_mse in Eqs. 4 and 6; if this proxy is inaccurate, training signals may not reflect final outputs.
  • domain assumption Off-the-shelf watermarking models embed a fixed 100-bit message consistently across images, enabling a global trigger.
    The secret message is fixed for all poisoned images; the watermark encoder must produce a recognizable pattern.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Invisible Backdoor Triggers in Image Editing Model via Deep Watermarking." pith.science (2026). https://pith.science/paper/JJAVJQX6

@misc{pith2026250604879,
  author       = {Pith},
  title        = {Pith review of: Invisible Backdoor Triggers in Image Editing Model via Deep Watermarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JJAVJQX6}},
  note         = {Machine review of arXiv:2506.04879}
}
read the original abstract

Diffusion models have achieved remarkable progress in both image generation and editing. However, recent studies have revealed their vulnerability to backdoor attacks, in which specific patterns embedded in the input can manipulate the model's behavior. Most existing research in this area has proposed attack frameworks focused on the image generation pipeline, leaving backdoor attacks in image editing relatively unexplored. Among the few studies targeting image editing, most utilize visible triggers, which are impractical because they introduce noticeable alterations to the input image before editing. In this paper, we propose a novel attack framework that embeds invisible triggers into the image editing process via poisoned training data. We leverage off-the-shelf deep watermarking models to encode imperceptible watermarks as backdoor triggers. Our goal is to make the model produce the predefined backdoor target when it receives watermarked inputs, while editing clean images normally according to the given prompt. With extensive experiments across different watermarking models, the proposed method achieves promising attack success rates. In addition, the analysis results of the watermark characteristics in term of backdoor attack further support the effectiveness of our approach. The code is available at:https://github.com/aiiu-lab/BackdoorImageEditing

Figures

Figures reproduced from arXiv: 2506.04879 by the authors.

Figure 1
Figure 1. During inference, the backdoor image editing DM gen [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. We employ a watermark encoder to embed invisible triggers into input images. The pipeline consists of two branches: (1) a [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The left side shows the input data, and the right side shows the corresponding output from backdoor models trained with different [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The plot illustrates the probability density of L2 dis [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Performance at Various Poison Rates. The top plot de [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enhancing Robustness in Post-Processing Watermarking: An Ensemble Attack Network Using CNNs and Transformers

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A parallel ensemble of a spatial-domain CNN attacker and a frequency-domain (DCT) transformer attacker during training improves deep watermark robustness on WAVES.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [6]

    Chou, P.-Y

    S.-Y . Chou, P.-Y . Chen, and T.-Y . Ho. Villandiffusion: A unified backdoor attack framework for diffusion models.Ad- vances in Neural Information Processing Systems, 36, 2023. 1, 2, 3

    work page 2023

  2. [1]

    B. An, M. Ding, T. Rabbani, A. Agrawal, Y . Xu, C. Deng, S. Zhu, A. Mohamed, Y . Wen, T. Goldstein, and F. Huang. W A VES: benchmarking the robustness of image watermarks. In Proceedings of the International Conference on Machine Learning, 2024. 5

    work page 2024

  3. [2]

    Brooks, A

    T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392–18402, 2023. 1, 2, 3, 4

    work page 2023

  4. [3]

    T. Bui, S. Agarwal, N. Yu, and J. Collomosse. Rosteals: Ro- bust steganography using autoencoder latent space. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023. 2, 3, 4, 1

    work page 2023

  5. [4]

    W. Chen, D. Song, and B. Li. Trojdiff: Trojan attacks on diffusion models with diverse targets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 2

    work page 2023

  6. [5]

    Chou, P.-Y

    S.-Y . Chou, P.-Y . Chen, and T.-Y . Ho. How to backdoor dif- fusion models? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 1, 2, 3

    work page 2023

  7. [7]

    J. Guo, P. Chen, W. Jiang, and G. Lu. Trojanedit: Back- dooring text-based image editing models. arXiv preprint arXiv:2411.14681, 2024. 1, 2

    work page Pith review arXiv 2024

  8. [8]

    S. Li, J. Ma, and M. Cheng. Invisible backdoor attacks on diffusion models. arXiv preprint arXiv:2406.00816, 2024. 1, 2

    arXiv 2024

Show all 12 references
  1. [9]

    Y . Li, Y . Li, B. Wu, L. Li, R. He, and S. Lyu. Invisible back- door attack with sample-specific triggers. In Proceedings of the IEEE/CVF international conference on computer vision,

  2. [10]

    S. Lu, Z. Zhou, J. Lu, Y . Zhu, and A. W.-K. Kong. Robust watermarking using generative priors against image editing: from benchmarking to advances. In Proceedings of the In- ternational Conference on Learning Representations , 2025. 1, 2, 3, 4

    2025

  3. [11]

    Tancik, B

    M. Tancik, B. Mildenhall, and R. Ng. Stegastamp: Invisi- ble hyperlinks in physical photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020. 2, 3, 4, 1

    2020

  4. [12]

    N. Yu, V . Skripniuk, S. Abdelnabi, and M. Fritz. Artificial fingerprinting for generative models: Rooting deepfake at- tribution in training data. In Proceedings of the IEEE/CVF International conference on computer vision, 2021. 1 Invisible Backdoor Triggers in Image Editing ...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.