Pith. sign in

REVIEW 5 major objections 5 minor 47 references

Adversarial-Guided Diffusion for Multimodal LLM Attacks

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Injecting target semantics into diffusion noise gives MLLM attacks that survive filtering defenses.

desk verdict Useful incremental diffusion attack with solid empirical gains, but the full-spectrum robustness claim is asserted, not demonstrated; needs spectral analysis and an LPF test. read the letter →

arxiv 2507.23202 v1 pith:TC73LDBV submitted 2025-07-31 cs.CV

Chengwei Xia , Fan Ma , Ruijie Quan , Kun Zhan , Yi Yang This is my paper
classification cs.CV
keywords adversarialattackmultimodallargelanguagemodelsdiffusionmodeltargetedfull-spectrumnoisefrequencyrobustnessCLIPguidancemomentum-basedinjection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AGD targets multimodal large language models by hiding the adversarial message in the diffusion process itself. Instead of adding a small perturbation to the clean image, the method injects a target-derived signal into the noise that remains during the final reverse-diffusion steps, after first reconstructing the clean image so the final output stays close to the original. Because the injected signal rides on diffusion noise, which the paper argues spans the whole frequency spectrum, the target survives simple low-pass filtering rather than being confined to high frequencies. Experiments on several open MLLMs report higher attack success rates and CLIP scores than previous attacks, while SSIM and LPIPS indicate the adversarial images look closer to the clean ones. A sympathetic reader would take the paper to establish that injecting target semantics into the noise component, rather than into the pixel image, is a viable way to combine attack effectiveness, visual imperceptibility, and robustness to common defenses.

What carries the argument

The central object is the adversarial-guided noise $\tilde{\epsilon}$ of Eq. (11): the denoiser's predicted noise plus a scaled, sign-flipped target-injection term $\gamma\,\mathrm{sign}(\epsilon_{\mathrm{tar}})$, where $\epsilon_{\mathrm{tar}}$ is derived from a CLIP-based similarity between the current image and a text-to-image rendering of the target text. This noise is accumulated through an exponential moving average in an inner loop, so the iterative denoising direction gradually aligns with the target while the image itself remains a linear combination of clean content and noise. The truncation at step $\Delta$ is the second load-bearing piece: reverse diffusion from $T$ to $\Delta$ reconstructs the clean image, and only the final steps carry the injection, which keeps distortion small. Together these pieces are what the paper claims make the adversarial signal full-spectrum and hence harder to filter out.

What would settle it

Compute the Fourier power spectrum of the difference between an AGD adversarial image and its clean counterpart, or of the injected term $\gamma\,\mathrm{sign}(\epsilon_{\mathrm{tar}})$ at the final steps. If the energy is concentrated in a high-frequency band rather than spread across all frequencies, then a matched low-pass or band-stop filter should reduce the attack success rate toward the random baseline, which would falsify the full-spectrum robustness claim. The paper does not report such a spectrum measurement.

Watch

Extended reading notes

Core claim

The central claim is that adversarial-guided noise, defined as $\tilde{\epsilon} = \epsilon_\theta(x_t) + \gamma\,\mathrm{sign}(\epsilon_{\mathrm{tar}})$, embeds the target semantics across all frequency bands during reverse diffusion, producing an adversarial image that is a linear combination of the clean image and this full-spectrum noise. The paper argues that because a well-trained diffusion model's noise is approximately Gaussian white noise with equal power at every frequency, the target information carried by the noise component is not confined to high frequencies and therefore cannot be removed by low-pass defenses that act per component. The method first runs standard reverse diffusion from a noised clean image down to a small time step $\Delta$ to satisfy the visual-similarity constraint, then applies a momentum-based injection loop from $\Delta$ to 1, using EMA of target-guided noise to steer the denoising direction toward the target while keeping the final image close to the clean input.

Load-bearing premise

The claim that AGD resists low-pass defenses rests on the assumption that the injected adversarial-guided noise behaves like Gaussian white noise with equal power in every frequency band, so no filter can remove the target without also destroying the image.

Editorial extensions

If this is right

  • If AGD works as claimed, targeted MLLM attacks can be generated with a diffusion model while preserving image quality, so attack success no longer requires visible perturbation.
  • The full-spectrum injection implies simple input defenses such as JPEG compression or low-pass filtering will not restore safe behavior, because the target signal is distributed across all frequencies.
  • The method's gray-box design means it can be applied to open-source MLLMs without training a separate attack model, making robustness evaluation against diffusion-based attacks more realistic.
  • Momentum-based inner-loop injection improves both attack success and imperceptibility, indicating that the direction-finding process, rather than the perturbation strength, is what drives performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the robustness argument assumes the sign and EMA operations preserve the white-noise spectrum of $\epsilon_\theta(x_t)$, but sign is nonlinear; measuring the power spectrum of the actual injected noise would show whether the full-spectrum premise survives the pipeline.
  • Beyond the paper: the same noise-component injection could be tested against adaptive defenses that estimate and subtract the injected noise using a second denoiser; if such a defense removes the target, the robustness claim would be limited to the non-adaptive defenses studied.
  • Beyond the paper: since the injection uses CLIP similarity, the attack should transfer to other MLLMs whose vision encoders align with CLIP features; a transferability sweep across black-box models would map the boundary of the gray-box assumption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Adversarial-Guided Diffusion (AGD), a gray-box targeted attack against multimodal large language models (MLLMs). The method adds noise to a clean image via forward diffusion, reconstructs it with standard reverse diffusion up to a small timestep, and then injects CLIP-computed target semantics into the noise term of the final reverse-diffusion steps using a momentum-based inner loop. The authors claim this embeds the adversarial target across the full frequency spectrum because diffusion noise is white, making the attack inherently robust to low-pass filtering and other defenses. Experiments on five MLLMs report higher CLIP scores and attack success rates than prior methods, with better image quality metrics, and an ablation studies the number of inner iterations and hyperparameters.

Significance. If the core claims held, AGD would be a practically interesting contribution: it would demonstrate a diffusion-based attack that achieves high targeted attack success while preserving image fidelity and resisting simple frequency-domain defenses. The paper provides fairly extensive empirical comparisons across multiple MLLMs and defenses, and the reported ASR improvements are substantial. However, the central theoretical premise—that the injected adversarial signal remains full-spectrum Gaussian noise—is asserted rather than demonstrated, and the algorithm as written contains a self-referential update that cannot be executed as stated. The paper also uses the same CLIP encoder family in the attack objective and in the main evaluation metric, which inflates the reported CLIP scores. The empirical ASR results, which are less affected by this circularity, make the work worth pursuing, but the missing spectral analysis and untested low-pass-filtering claim must be addressed before the robustness conclusion is credible.

major comments (5)
  1. [Section 3.2, Eq. (3)] The forward diffusion formula in Eq. (3) is incorrect: DDPM defines xt = sqrt(alpha_bar_t) x0 + sqrt(1 - alpha_bar_t) epsilon, not xt = sqrt(alpha_bar_t) x0 + (1 - sqrt(alpha_bar_t)) epsilon. This error propagates to Eq. (9) and Eq. (10), where the Gaussian noise term (1 - alpha_bar_t) is used to define epsilon_tar. Since the derivation of the adversarial injection noise rests on this formula, the derivation is invalid as written and must be corrected and re-evaluated.
  2. [Algorithm 1, line 13] The update epsilon = epsilon + gamma * sign(epsilon) is self-referential and, with epsilon initialized to 0 on line 11, is a no-op (sign(0)=0). It cannot be the intended momentum injection. The update presumably should involve sign(epsilon_tar) or a different accumulator variable. As written, the core injection algorithm is not executable, and the method cannot be reproduced from the pseudocode.
  3. [Section 4.2, Eq. (11) and Eq. (5)] The full-spectrum robustness claim is unsupported. Eq. (5) asserts that the denoiser output epsilon_theta(xt) is approximately N(0,I), but this is not generally true, especially for latent diffusion on out-of-distribution inputs; the sign operation and EMA update in Eq. (12) are nonlinear and change the spectral profile; and the CLIP-gradient-based epsilon_tar is not a Gaussian sample. The paper provides no spectral density analysis of the actual injected perturbation. The claim that low-pass filtering 'cannot' suppress the target is therefore not established by the current evidence.
  4. [Section 5.3 and Abstract] The abstract and Section 4.2 emphasize robustness to 'a simple low-pass filtering,' and the text says experimental evidence appears in Section 5.3, but Table 3 evaluates JPEG, R&P, SOAP, DiffPure, and MimicDiffusion—none of which is a low-pass filter. The paper's headline robustness scenario is never tested. The authors should add an explicit low-pass filtering defense (e.g., Gaussian blur with varying radii or FFT-based truncation) and measure ASR after that defense.
  5. [Section 5.1, Table 1] The evaluation metric CLIP score is computed with CLIP text encoders from the same family used in the attack objective (Algorithm 1 lines 16-17). This makes the reported CLIP scores partly optimized by construction. Since ASR is a separate metric and also improves, this does not invalidate the main attack claim, but the CLIP scores should be reported as a secondary metric, with the dependence on the same encoder family disclosed, and ideally with a held-out encoder not used in the objective.
minor comments (5)
  1. [Section 1] The word 'paepr' should be 'paper' in the contributions list.
  2. [Sections 5.1 and 5.2] 'Tabel 1' and 'Tabel 2' should be 'Table 1' and 'Table 2'; 'PNSR' should be 'PSNR'; 'bashlines' should be 'baselines'.
  3. [Section 5.3, Table 3] The defense name 'JEPG' should be 'JPEG', and the table headers are inconsistently formatted (e.g., 'RP' for Randomization).
  4. [Section 5.2, Table 2] The CoA row lists values 0.2512 and 0.2176 in inconsistent column orders for the three MLLMs; one entry appears to be swapped between the LPIPS and PSNR columns.
  5. [Figure 5] The x-axis labels in Figure 5 appear as garbled Unicode path fragments rather than readable hyperparameter values; the figure needs to be regenerated with proper text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AGD's attack-success improvements are measured by an independent ASR metric, and the weak full-spectrum robustness premise is an evidence gap, not a circular reduction.

full rationale

The paper's derivation is not circular in the sense required by this review. The attack objective in Eq. (2) is to maximize a^T atar, and Algorithm 1 implements this by computing CLIP-based gradients in the inner loop (lines 16-17) and using them as injection noise via Eq. (11). The headline attack-performance metric in Table 1 is a CLIP score computed on the MLLM's generated response text against the target text, together with a separate attack-success rate (ASR). The ASR is an external, non-CLIP measure that does not share the optimization objective, and AGD improves ASR consistently (e.g., 95.4% vs 90.7% on UniDiffuser; 30.5% vs 24.2% on Qwen2-VL), so the central result is not forced by the choice of the attack loss. The use of CLIP features both in the injection direction and, in part, in the similarity metric is a metric-alignment concern rather than a definitional identity: the evaluation uses CLIP text encoders on generated text, not the image-to-target-image similarity that the attack actually optimizes. The full-spectrum robustness claim is unsupported rather than circular: Eq. (5) assumes a well-trained diffusion model's noise prediction is approximately N(0,I), but Eq. (11) defines the adversarial-guided noise as epsilon_theta(x_t) + gamma*sign(epsilon_tar), where epsilon_tar is a momentum-accumulated CLIP gradient; the paper's assertion that the injected signal therefore has equal power across all frequencies does not follow from its own equations, and Table 3 does not test the advertised simple low-pass filter. That is a correctness/evidence gap, not a circular reduction. No load-bearing self-citations are present; the referenced edit-friendly noise-space work [19] and exposure-bias inequalities [22,28] are external prior results. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central method depends on four tuned hyperparameters and on several background assumptions about Stable Diffusion noise being white and full-spectrum, CLIP semantics transferring from images to MLLM text responses, and the fidelity of deterministic DDPM inversion. The full-spectrum Gaussianity assumption is the most fragile: the actual injected signal is a signed, averaged, and scaled quantity, not raw Gaussian noise.

free parameters (4)
  • gamma (adversarial scale) = 6
    Controls the strength of target injection; swept over [0.5, 7.0] in Figure 5a and chosen as a trade-off between attack performance and visual quality.
  • lambda (momentum factor) = 0.9
    EMA weight in Eq. (12); swept over [0, 0.9] in Figure 5d, with larger values giving better attack performance.
  • N (inner iterations) = 50
    Number of momentum-based injection steps; ablation in Table 4 and Figure 5c shows ASR grows with N, so N=50 is a tuned endpoint.
  • Delta (adversarial diffusion steps) = 5
    Number of final denoising steps where target injection is applied; swept in Figure 5b and chosen for the imperceptibility versus attack trade-off.
assumptions (5)
  • domain assumption The adversarial-guided noise in Eq. (11) can be treated as approximately Gaussian white noise with equal power across all frequencies.
    This is the load-bearing premise for the robustness claim. Eq. (5) assumes the denoiser output is Gaussian, but the actual term gamma sign of the accumulated momentum is not Gaussian after sign and EMA operations; no spectral evidence is provided.
  • domain assumption Reverse diffusion from x_cle_T to x_cle_Delta with the deterministic noise edit from [19] reconstructs x_cle_Delta approximately equal to x_cle_0.
    Used in Section 4.1 to guarantee that the fidelity constraint is largely satisfied; relies on the Edit Friendly DDPM inversion quality of [19].
  • ad hoc to paper CLIP feature similarity between the generated image and the target image is a valid proxy for steering the MLLM response toward the target text.
    Section 4.2 replaces x_t and x_tar_0 in Eq. (9) with normalized CLIP features, assuming image-side CLIP geometry transfers to the text response side.
  • domain assumption A text-to-image generation of the target text produces an image whose CLIP embedding represents the target semantics.
    Algorithm 1 line 1 builds x_tar_0 = T2I(a_tar); if the generated target image is off-prompt, the guidance direction will be wrong.
  • standard math Gaussian white noise spans the full frequency spectrum with equal power per band.
    Standard signal-processing fact invoked in Section 4.2 to argue that the adversarial signal in the noise component survives low-pass filtering.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial-Guided Diffusion for Multimodal LLM Attacks." pith.science (2026). https://pith.science/paper/TC73LDBV

@misc{pith2026250723202,
  author       = {Pith},
  title        = {Pith review of: Adversarial-Guided Diffusion for Multimodal LLM Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TC73LDBV}},
  note         = {Machine review of arXiv:2507.23202}
}
read the original abstract

This paper addresses the challenge of generating adversarial image using a diffusion model to deceive multimodal large language models (MLLMs) into generating the targeted responses, while avoiding significant distortion of the clean image. To address the above challenges, we propose an adversarial-guided diffusion (AGD) approach for adversarial attack MLLMs. We introduce adversarial-guided noise to ensure attack efficacy. A key observation in our design is that, unlike most traditional adversarial attacks which embed high-frequency perturbations directly into the clean image, AGD injects target semantics into the noise component of the reverse diffusion. Since the added noise in a diffusion model spans the entire frequency spectrum, the adversarial signal embedded within it also inherits this full-spectrum property. Importantly, during reverse diffusion, the adversarial image is formed as a linear combination of the clean image and the noise. Thus, when applying defenses such as a simple low-pass filtering, which act independently on each component, the adversarial image within the noise component is less likely to be suppressed, as it is not confined to the high-frequency band. This makes AGD inherently robust to variety defenses. Extensive experiments demonstrate that our AGD outperforms state-of-the-art methods in attack performance as well as in model robustness to some defenses.

Figures

Figures reproduced from arXiv: 2507.23202 by the authors.

Figure 1
Figure 1. AGD injects the target information in a manner that is [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Flowchart of the proposed adversarial-guided diffusion process. We perform forward noise addition on the clean image. During [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. At each time step t of AGD, we use the effect shown in Eq. (4): x0 = √1 α¯t xt − √ √1−α¯t α¯t ϵ¯, to inject the target information to current xt. Defenses MLLM Unidiffuser LLaVA-1.5 MiniGPT-4 BLIP-2 JEPG MF-it 30.7% 19.7% 19.0% 25.5% MF-ii 38.1% 21.1% 24.9% 25.6% AdvDiffVLM 66.2% 29.4% 40.6% 45.8% AGD 92.1% 36.1% 82.4% 87.6% RP MF-it 26.2% 17.0% 26.0% 23.4% MF-ii 27.8% 16.3% 26.3% 26.5% AdvDiffVLM 50.4% 20.5% 42.8% … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of targeted adversarial attacks results on UniDiffuser. Adversarial target text appears above each image, and captioning results from clean or adversarial images are shown below. 0.02 0.04 0.06 0.08 0.10 0.12 LPIPS 0.40 0.45 0.50 0.55 0.60 0.65 0.70 C LIP S …
Figure 5
Figure 5. Figure 5: Ablation study on the impact of hyperparameters in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 37 canonical work pages

  1. [1]

    Image hijacks: Adversarial images can control generative models at runtime

    Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacks: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236, 2023. 2

    arXiv 2023

  2. [2]

    One transformer fits all distributions in multi-modal diffusion at scale

    Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In ICML, pages 1692–1717. PMLR, 2023. 1, 5

    work page 2023

  3. [3]

    A young boy is playing with a baseball bat A blue bird sitting on a tree branch

    Xinquan Chen, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Target: Young boy swinging bat at a game with onlookers. A young boy is playing with a baseball bat A blue bird sitting on a tree branch. A blue bird with a beak sitting on a twig. A young boy swinging a baseball ball with a bat. A young boy holding a bird on sticks in a tree. MF-it MF-ii ACA AGDCle...

    work page

  4. [4]

    Content-based unrestricted ad- versarial attack

    Zhaoyu Chen, Bo Li, Shuang Wu, Kaixun Jiang, Shouhong Ding, and Wenqiang Zhang. Content-based unrestricted ad- versarial attack. In NeurIPS, 2023. 1, 2, 6

    work page 2023

  5. [5]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

  6. [6]

    On the robustness of large multimodal models against image adversarial attacks

    Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal models against image adversarial attacks. In CVPR, pages 24625– 24634, 2024. 1, 2

    work page 2024

  7. [7]

    Advdiff: Gener- ating unrestricted adversarial examples using diffusion mod- els

    Xuelong Dai, Kaisheng Liang, and Bin Xiao. Advdiff: Gener- ating unrestricted adversarial examples using diffusion mod- els. In ECCV, pages 93–109. Springer, 2024. 1, 2

    work page 2024

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255, 2009. 5

    work page 2009

Show all 47 references
  1. [9]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, pages 8780–8794,

  2. [10]

    Boosting adversarial attacks with momentum

    Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR, pages 9185–9193, 2018. 2

    2018

  3. [11]

    How robust is Google’s bard to adversarial image attacks? arXiv preprint arXiv:2309.11751, 2023

    Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is Google’s bard to adversarial image attacks? arXiv preprint arXiv:2309.11751, 2023. 2

    2023 arXiv

  4. [12]

    Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory

    Sensen Gao, Xiaojun Jia, Xuhong Ren, Ivor Tsang, and Qing Guo. Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory. In ECCV, 2024. 2

    2024

  5. [13]

    Explaining and harnessing adversarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. 2, 3

    2014 arXiv

  6. [14]

    Countering adversarial images using input transformations

    Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In ICLR, 2018. 7

    2018

  7. [15]

    Efficient generation of targeted and transferable adversarial examples for vision-language models via diffusion models

    Qi Guo, Shanmin Pang, Xiaojun Jia, Yang Liu, and Qing Guo. Efficient generation of targeted and transferable adversarial examples for vision-language models via diffusion models. TIFS, 2024. 1, 2, 6

    2024

  8. [16]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021. 6

    2021 arXiv

  9. [17]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. In NeurIPS, pages 6840–6851,

  10. [18]

    Image quality metrics: PSNR vs

    Alain Hore and Djemel Ziou. Image quality metrics: PSNR vs. SSIM. In ICPR, pages 2366–2369. IEEE, 2010. 6

    2010

  11. [19]

    An edit friendly DDPM noise space: Inversion and manipulations

    Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly DDPM noise space: Inversion and manipulations. In CVPR, pages 12469–12478, 2024. 3, 4

    2024

  12. [20]

    Adver- sarial examples in the physical world

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adver- sarial examples in the physical world. InArtificial intelligence safety and security, pages 99–112. Chapman and Hall/CRC,

  13. [21]

    BLIP- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742, 2023. 1, 3, 5

    2023

  14. [22]

    Alleviating exposure bias in diffusion mod- els through sampling with shifted time steps

    Mingxiao Li, Tingyu Qu, Ruicong Yao, Wei Sun, and Marie- Francine Moens. Alleviating exposure bias in diffusion mod- els through sampling with shifted time steps. In ICLR, 2024. 5

    2024

  15. [23]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755, 2014. 5

    2014

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 5

    2023

  17. [25]

    An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models

    Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models. In ICLR,

  18. [26]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018. 2

    2018

  19. [27]

    Diffusion models for adver- sarial purification

    Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adver- sarial purification. In ICML, 2022. 1, 2, 7

    2022

  20. [28]

    Elucidating the exposure bias in diffusion models

    Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. In ICLR, 2024. 5

    2024

  21. [29]

    Diffusion-based adversarial purification from the perspective of the frequency domain

    Gaozheng Pei, Ke Ma, Yingfei Sun, Qianqian Xu, and Qing- ming Huang. Diffusion-based adversarial purification from the perspective of the frequency domain. In ICML, 2025. 1, 2

    2025

  22. [30]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 4

    2021

  23. [31]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, pages 10684– 10695, 2022. 3, 4, 6

    2022

  24. [32]

    On the adversarial robustness of multi-modal foundation models

    Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. InICCV, pages 3677–3685, 2023. 2

    2023

  25. [33]

    Colorfool: Semantic adversarial colorization

    Ali Shahin Shamsabadi, Ricardo Sanchez-Matilla, and An- drea Cavallaro. Colorfool: Semantic adversarial colorization. In CVPR, pages 1151–1160, 2020. 2

    2020

  26. [34]

    Jail- break in pieces: Compositional adversarial attacks on multi- modal language models

    Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. Jail- break in pieces: Compositional adversarial attacks on multi- modal language models. In ICLR, 2024. 2

    2024

  27. [35]

    Online adver- sarial purification based on self-supervised learning

    Changhao Shi, Chester Holtz, and Gal Mishne. Online adver- sarial purification based on self-supervised learning. In ICLR,

  28. [36]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 2

    2021

  29. [37]

    Mimicd- iffusion: Purifying adversarial perturbation via mimicking clean diffusion model

    Kaiyu Song, Hanjiang Lai, Yan Pan, and Jian Yin. Mimicd- iffusion: Purifying adversarial perturbation via mimicking clean diffusion model. In CVPR, pages 24665–24674, 2024. 7

    2024

  30. [38]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021. 2, 4

    2021

  31. [39]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 5

    2024 arXiv

  32. [40]

    Instructta: Instruction-tuned targeted attack for large vision-language models

    Xunguang Wang, Zhenlan Ji, Pingchuan Ma, Zongjie Li, and Shuai Wang. Instructta: Instruction-tuned targeted attack for large vision-language models. arXiv preprint arXiv:2312.01886, 2023. 2

    2023 arXiv

  33. [41]

    Image quality assessment: from error visibility to structural similarity

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 13(4):600–612, 2004. 6

    2004

  34. [42]

    Mitigating adversarial effects through random- ization

    Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through random- ization. In ICLR, 2018. 7

    2018

  35. [43]

    Chain of attack: On the robustness of vision-language models against transfer-based adversarial attacks

    Peng Xie, Yequan Bie, Jianda Mao, Yangqiu Song, Yang Wang, Hao Chen, and Kani Chen. Chain of attack: On the robustness of vision-language models against transfer-based adversarial attacks. In CVPR, pages 14679–14689, 2025. 1, 2, 6

    2025

  36. [44]

    Natural color fool: Towards boosting black-box unrestricted attacks

    Shengming Yuan, Qilong Zhang, Lianli Gao, Yaya Cheng, and Jingkuan Song. Natural color fool: Towards boosting black-box unrestricted attacks. In NeurIPS, pages 7546–7560,

  37. [45]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586–595,

  38. [46]

    On evaluating adversarial robustness of large vision-language models

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongx- uan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. In NeurIPS, 2023. 1, 2, 5, 6

    2023

  39. [47]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 5

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.