Pith. sign in

REVIEW 4 major objections 5 minor 64 references

Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding noise purifies adversarial images in vision-language models.

desk verdict F3's attention-guided noise is a real, practical defense with visible gains over random noise; the evaluation is solid but the adaptive attack and hyperparameter selection need scrutiny. read the letter →

arxiv 2506.01064 v3 pith:TBLOONOK submitted 2025-06-01 cs.CV cs.AI

Yudong Zhang , Ruobing Xie , Yiqing Huang , Jiansheng Chen , Xingwu Sun , Zhanhui Kang , Di Wang , Yu Wang This is my paper
classification cs.CVcs.AI
keywords LVLMadversarialpurificationcross-modalattentiontraining-freedefensenoiseinjectionvisualattackefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that adversarial images aimed at large vision-language models (LVLMs) can be purified by deliberately adding more noise, guided by the model's own cross-modal attention. The key empirical discovery is that a randomly perturbed adversarial image has attention patterns closer to clean attention than the adversarial image itself, so it can serve as a reference target. F3 then perturbs the adversarial image to align its attention with that reference, requiring no training and only a few times the normal inference cost. A sympathetic reader should take the claim as: cross-modal attention alignment, without any auxiliary model, is a sufficient and efficient purification objective for LVLMs.

What carries the argument

The central object is the cross-modal attention tensor $A(x)$ of the first generated token, defined as the attention of that token toward all visual tokens across LLM layers and heads, with shape $(L,H,M)$. The random perturbation $x^R_i = x'_i - \alpha$, $\alpha \sim U[-\alpha_\infty, \alpha_\infty]$, produces the reference attention $A(x^R_i)$ which serves as a surrogate for the inaccessible clean attention $A(x_i)$. The purification noise is computed as the gradient $g = \nabla_{x'_i}\|A(x'_i) - A(R(x'_i, \alpha_\infty))\|_2$, normalized to $[0,1]$, and injected by $x^p_i = x'_i - \beta_\infty \cdot \max(0, \min(g_{\text{norm}}/\text{avg}(g_{\text{norm}}), 1)) \cdot \text{sign}(g)$, so both the direction and magnitude of the perturbation follow the attention-alignment objective.

What would settle it

Measure the mean squared error between clean attention $A(x)$ and the reference attention $A(R(x'))$ across many random perturbations; if for some attack or dataset this error is not consistently smaller than $\text{MSE}(A(x), A(x'))$, then the surrogate direction fails and F3 loses its target. Concretely, an attack that is optimized to keep $A(R(x'))$ far from $A(x)$ across noise levels would break the method.

Watch

Extended reading notes

Core claim

F3 establishes that cross-modal attention alignment is a sufficient purification objective in LVLMs: if you push the attention of an adversarial image toward the attention of a randomly perturbed version of itself, the model's answer becomes correct again. The paper demonstrates this by showing that randomly perturbing an adversarial example leaves the answer wrong but moves its attention measurably closer to the clean attention, and that using this reference attention to compute a gradient-based noise direction purifies the image. Progressively finer control of the injected noise — pure random noise (F3-v1), direction-only gradient sign (F3-v2), and direction plus normalized gradient magnitude (F3-v3) — monotonically improves VQA scores and attention similarity to clean examples. Verified across BLIP-2, InstructBLIP, LLaVA-v1.5, and Qwen2.5-VL under C&W, PGD, and AutoAttack, F3 with a 32/255 purification budget recovers substantial accuracy (e.g., from 24.88 to 54.74 on the D1000 VQAv2 subset) while adding only a few times the inference cost.

Load-bearing premise

The method assumes that a randomly perturbed adversarial image's cross-modal attention points toward clean attention, so the gradient direction computed from it is a reliable purification target.

Editorial extensions

If this is right

  • A training-free defense can be dropped into any LVLM with attention introspection, no extra model or fine-tuning, and works across Q-Former and MLP projector architectures.
  • At roughly 3–4× inference overhead, F3 achieves VQA scores close to DiffPure (52.52 vs 61.64 on the reported benchmark) while DiffPure runs about 50× slower and requires a domain-matched diffusion model.
  • Because F3 uses only the victim model's attention and gradients, it is domain-agnostic and compatible with dynamic-resolution LVLMs like Qwen2.5-VL, which fixed-resolution diffusion purifiers cannot handle.
  • Multi-step iterations of F3 with the same total perturbation budget improve purification further, suggesting a scalable refinement path rather than a one-shot heuristic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the random-perturbation-surrogate regularity holds broadly, it implies that visual adversarial perturbations in LVLMs behave like directionally coherent noise that the model's own attention statistics can wash out; a testable corollary is that any attack whose perturbation is in the same direction as the gradient update would evade F3.
  • The method only uses first-token attention; extending F3 to all generated tokens, as the paper notes, could substantially improve captioning robustness beyond the reported COCO gains.
  • Combining F3 with an attention-based detector (e.g., PIP-style probing) would give a complete pipeline that purifies only when an attack is suspected, avoiding the 10-point clean-image degradation from unconditional purification.
  • The clean-image trade-off could be tuned adaptively per sample by checking whether the reference attention $A(x^R)$ is close to $A(x')$ before committing to a large purification step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes F3, a training-free adversarial purification method for large vision-language models (LVLMs). The core idea is to add carefully directed noise to an adversarial image so that the LVLM's cross-modal attention on the purified image becomes closer to the attention of the original clean image. Because clean attention is unavailable at test time, F3 uses the attention of a randomly perturbed adversarial image as a reference. Three variants are introduced: F3-v1 adds random noise, F3-v2 uses the sign of the gradient of the attention distance to set the perturbation direction, and F3-v3 additionally scales the perturbation magnitude by the normalized gradient. Experiments are reported on BLIP-2, InstructBLIP, LLaVA-v1.5, and Qwen2.5-VL under C&W, PGD, and AutoAttack attacks, on VQAv2, ImageNet, and COCO captioning, with comparisons to R&P, JPEG, SR, and DiffPure. The paper reports consistent improvements in VQA scores and attention-similarity metrics, at a much lower inference cost than DiffPure.

Significance. If the central surrogate assumption holds, the paper makes a useful contribution: it introduces cross-modal attention as a purification target and shows that a simple, training-free, model-agnostic noise-injection scheme can approach the robustness of diffusion-based purification at a fraction of the cost. The paper is honest about several limitations, includes extensive ablations, and provides open-source code, which is valuable for reproducibility. However, the significance is conditional because the key empirical regularity --- that A(R(x')) is closer to clean attention than A(x') --- is validated only on one attack/model/dataset combination, and some of the quantitative evidence is internally inconsistent. The adaptive-attack evaluation is also weaker than the text implies.

major comments (4)
  1. [Section 3.4 / Table 2 / Sections 4.2 and 4.4] The load-bearing assumption of the method is that the cross-modal attention of a randomly perturbed adversarial example, A(R(x')), is closer to clean attention than A(x'). This is only validated on InstructBLIP with C&W attacks on D1000/VQAv2, using aggregate MSE/KL (Table 2) and a few visualizations. The paper then applies F3-v3 to BLIP-2, LLaVA-v1.5, Qwen2.5-VL, PGD/AutoAttack, and ImageNet (Tables 4, 7, 15) without providing any evidence that the surrogate reference is reliable in those settings. Because Eq. (7) minimizes ||A(x') - A(R(x'))||_2, the entire purification direction collapses to a random direction if the reference is not closer to clean attention for a substantial fraction of examples. The authors should report per-example distributions of MSE/KL between A(R(x')) and clean attention for each model/attack/dataset combination used in the main generalization tables, and ideally the fraction of examples for which the reference is strictly closer than the adversarial example. Without this, the observed gains could be dominated by the random-noise component already present in F3-v1 (Table 2: VQA 31.34 at alpha_inf=32/255 vs. 24.88 for Adv), rather than by the attention-guided direction.
  2. [Section 3.2 / Table 1 vs. Table 2] There is a direct numerical inconsistency in the central empirical evidence. Table 1 reports MSE(A_clean, A_adv) = 10.31 and KL = 3.39 for the C&W adversarial examples on InstructBLIP/D1000, while Table 2 reports MSE = 16.03 and KL = 4.91 for what appears to be the same setting (same clean VQA 75.95, same adversarial VQA 24.88, same attack and model). The paper does not explain this discrepancy. Since the attention-distance baseline is the foundation for the claim that adversarial examples disrupt cross-modal attention and that F3's noise reduces that distance, this inconsistency must be resolved (e.g., by clarifying different attack hyperparameters, evaluation subsets, or attention normalization choices).
  3. [Appendix C.1 / Table 5] The adaptive-attack evaluation is substantially weaker than the text suggests. Appendix C.1 states that the adaptive AutoAttack was run with only 20 attack steps and EOT=10 instead of the default 100 steps and EOT=20, and the reported config appears to be the generic 'rand' AutoAttack rather than an attack specifically adapted to F3's mechanism, e.g., by differentiating through the reference-attention computation in Eqs. (5)-(7) or by sampling multiple references per outer optimization step. As a result, Table 5 demonstrates robustness against a weakened stochastic attack but does not support the paper's stronger claim of robustness against adaptive adversaries with full knowledge of F3. The authors should either run a stronger adaptive attack (more steps, EOT, and mechanism-aware gradient estimation) or temper the claim to 'partial adaptivity'.
  4. [Section 4.3 / Table 6] The headline efficiency/robustness comparison with DiffPure in Table 6 lacks essential setup information. The table reports 'VQA scores' of 28.88 for no defense, 61.64 for DiffPure, and 52.52 for F3-v3 on InstructBLIP-7B, but does not specify the attack method, perturbation bound, evaluation subset, or the exact DiffPure hyperparameters (e.g., number of diffusion steps). This matters because the abstract and conclusion describe F3 as achieving results 'close to DiffPure', and the numerical gap (52.52 vs. 61.64) depends on the attack difficulty. Please add the missing experimental configuration and clarify whether the same attack and data are used as in Tables 2, 4, or 5.
minor comments (5)
  1. [Section 4.2 heading] The heading 'Unadaptive attacks' should read 'Non-adaptive attacks' to match standard terminology.
  2. [Section 4.1] The model name 'LLaMAv1.5' in the LVLMs paragraph appears to be a typo for 'LLaVA-v1.5'; please correct it.
  3. [Tables 2, 3, and throughout] No error bars, standard deviations, or per-seed variability are reported anywhere. Given the stochastic components in Eq. (3) and the small dataset sizes (D1000), a few repeated runs with standard deviations would substantially increase confidence in the reported differences.
  4. [Eqs. (2), (4), and (7)] Please clarify how the random variables gamma and beta are drawn: once per image, once per pixel, or once per coordinate. This affects the interpretation of the l1-norm comparisons in Table 12 and the stochastic behavior of the method.
  5. [Table 7 / Table 6] The spelling of 'DiffPure' is inconsistent (Table 6 uses 'DiffPure', Table 7 uses 'Diffpure'); unify the spelling.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: F3's core reference-attention surrogate is validated against clean attention, not assumed by construction.

full rationale

F3 is an empirical purification method, not a derivation of a prediction from first principles. The central step — using A(R(x')) as a reference for clean attention — is validated directly against the oracle clean attention in Tabs. 1 and 2, which report that adding noise decreases MSE/KL to clean attention and, at larger intensities, also improves VQA scores. The purification objective in Eqs. (4)–(7) minimizes distance to that reference rather than to an inaccessible clean target, so the result is not forced by definition. The method is self-referential only in the weak sense that the reference is estimated from the adversarial example itself, but the paper supplies independent quantitative evidence that the surrogate is closer to clean attention, making the empirical claim testable rather than circular. Hyperparameters alpha_inf and beta_inf are grid-selected on the development set, not presented as predictions. The only notable self-citations are to the authors' prior works PIP [54] and DHCP [55] in Sec. 3.1 to motivate focusing on first-token cross-modal attention; this is a design motivation, not a load-bearing uniqueness or equivalence argument, and the paper independently re-derives the relevance of cross-modal attention in Sec. 3.2. Appendix B honestly acknowledges limitations (single-step noise, first-token-only attention, and adaptive-attack optimization), which are generalization risks rather than circularity. In summary, the central derivation is self-contained: the reference surrogate is empirically cross-validated, and the purification gains are not equivalent to the method's inputs by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central method rests on two empirical postulates about cross-modal attention, plus standard calculus and the attention mechanism. The only tuned numbers are the noise bounds alpha_inf and beta_inf. No new entities are introduced.

free parameters (2)
  • alpha_inf = 16/255
    Noise bound for generating reference attention A(R(x')) in Eq. (3); selected by grid search in Tab. 3 over {2,4,8,16,32}/255.
  • beta_inf = 32/255
    Maximum perturbation bound for the purification update in Eqs. (4) and (7); selected by grid search in Tab. 3.
assumptions (3)
  • domain assumption Cross-modal attention of the first generated token is a meaningful proxy for the visual information the LVLM uses.
    Adopted from PIP [54] and DHCP [55]; F3's entire loss is defined on this tensor.
  • ad hoc to paper Randomly perturbed adversarial examples have attention closer to clean attention than the original adversarial examples.
    Empirical claim in Sec. 3.4, Tab. 2; not proven and not derived from model properties.
  • ad hoc to paper Aligning an adversarial example's attention to the reference attention improves VQA accuracy.
    Shown empirically with oracle clean attention in Sec. 3.3, Tab. 1, and with F3 reference in Sec. 3.5; causal direction is not rigorously established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs." pith.science (2026). https://pith.science/paper/TBLOONOK

@misc{pith2026250601064,
  author       = {Pith},
  title        = {Pith review of: Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBLOONOK}},
  note         = {Machine review of arXiv:2506.01064}
}
read the original abstract

Recent advances in large vision-language models (LVLMs) have showcased their remarkable capabilities across a wide range of multimodal vision-language tasks. However, these models remain vulnerable to visual adversarial attacks, which can substantially compromise their performance. In this paper, we introduce F3, a novel adversarial purification framework that employs a counterintuitive ``fighting fire with fire'' strategy: intentionally introducing simple perturbations to adversarial examples to mitigate their harmful effects. Specifically, F3 leverages cross-modal attentions derived from randomly perturbed adversary examples as reference targets. By injecting noise into these adversarial examples, F3 effectively refines their attention, resulting in cleaner and more reliable model outputs. Remarkably, this seemingly paradoxical approach of employing noise to counteract adversarial attacks yields impressive purification results. Furthermore, F3 offers several distinct advantages: it is training-free and straightforward to implement, and exhibits significant computational efficiency improvements compared to existing purification methods. These attributes render F3 particularly suitable for large-scale industrial applications where both robust performance and operational efficiency are critical priorities. The code is available at https://github.com/btzyd/F3.

Figures

Figures reproduced from arXiv: 2506.01064 by the authors.

Figure 1
Figure 1. Overview of F3. (1) We inject random noise into [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Our training-free and efficient F3 effectively targets the reference attention [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The visualization results under PGD attack. The question is “What color is the women dress?”. The answers to the [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The visualization results of “yes/no” question. The question is “Is this a toy train that a child could play with?”. The [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: The visualization results of “number” question. The question is “How many people are seated at this table?”. The [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 26 canonical work pages

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

  2. [2]

    Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. InInternational conference on machine learning. PMLR, 274–283

    work page 2018

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Jun- yang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision- Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966 [cs.CV] https://arxiv.org/abs/2308.12966

    arXiv 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923(2025)

    arXiv 2025

  5. [5]

    Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. 2024. Are aligned neural networks adversarially aligned?Advances in Neural Information Processing Systems36 (2024)

    work page 2024

  6. [6]

    Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In2017 ieee symposium on security and privacy (sp). Ieee, 39–57

    2017

  7. [7]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna. lmsys. org (accessed 14 April 2023)2, 3 (2023), 6

    2023

  8. [8]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models.Journal of Machine Learning Research25, 70 (2024), 1–53

    2024

Show all 64 references
  1. [9]

    Francesco Croce and Matthias Hein. 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. InInternational conference on machine learning. PMLR, 2206–2216

    2020

  2. [10]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024. Instruct- blip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems36 (2024)

    2024

  3. [11]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255

    2009

  4. [12]

    Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis.Advances in neural information processing systems34 (2021), 8780–8794

    2021

  5. [13]

    Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. 2023. How Robust is Google’s Bard to Adversarial Image Attacks?arXiv preprint arXiv:2309.11751(2023)

    2023 arXiv

  6. [14]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprin...

    2020 arXiv

  7. [15]

    Yihe Fan, Yuxin Cao, Ziyu Zhao, Ziyao Liu, and Shaofeng Li. 2024. Unbridled icarus: A survey of the potential perils of image inputs in multimodal large language model security. In2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3428–3433

    2024

  8. [16]

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the limits of masked visual representation learning at scale. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2023

  9. [17]

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversar- ial Nets. InAdvances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q...

    2014

  10. [18]

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572(2014)

    2014 arXiv

  11. [19]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh

  12. [20]

    Chih-Hui Ho and Nuno Vasconcelos. 2022. Disco: Adversarial defense with local implicit functions.Advances in neural information processing systems35 (2022), 23818–23837

    2022

  13. [21]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models.Advances in neural information processing systems33 (2020), 6840–6851

    2020

  14. [22]

    Xiaojun Jia, Xingxing Wei, Xiaochun Cao, and Hassan Foroosh. 2019. Comde- fend: An efficient image compression model to defend adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6084–6092

    2019

  15. [23]

    Qiyu Kang, Yang Song, Qinxu Ding, and Wee Peng Tay. 2021. Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks. Advances in Neural Information Processing Systems34 (2021), 14925–14937

    2021

  16. [24]

    Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. InInterna- tional Conference on Learning Representations. https://openreview.net/forum?id= Hk99zCeAb

    2018

  17. [25]

    Krizhevsky and G

    A. Krizhevsky and G. Hinton. 2009. Learning multiple layers of features from tiny images.Master’s thesis, Department of Computer Science, University of Toronto (2009)

    2009

  18. [26]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning. PMLR, 19730–19742

    2023

  19. [27]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, procee...

    2014

  20. [28]

    Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. 2024. A survey of attacks on large vision-language models: Resources, advances, and future trends.arXiv preprint arXiv:2407.07403(2024)

    2024 arXiv

  21. [29]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. InCVPR. 26296–26306

    2024

  22. [30]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning.Advances in neural information processing systems36 (2024)

    2024

  23. [31]

    Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. 2024. Safety of multimodal large language models on images and text. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 8151–8159

    2024

  24. [32]

    Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. 2023. Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. InProceedings of the IEEE/CVF International Conference on Computer Vision. 102–111

    2023

  25. [33]

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083(2017)

    2017 arXiv

  26. [34]

    Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. InProceed- ings of the IEEE conference on computer vision and pattern recognition. 2574–2582

    2016

  27. [35]

    Aamir Mustafa, Salman H Khan, Munawar Hayat, Jianbing Shen, and Ling Shao

  28. [36]

    Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. 2022. Diffusion Models for Adversarial Purification. InInternational Conference on Machine Learning (ICML)

    2022

  29. [37]

    Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. 2016. The limitations of deep learning in adversarial settings. In2016 IEEE European symposium on security and privacy (EuroS&P). IEEE, 372–387

    2016

  30. [38]

    Pouya Samangouei, Maya Kabkab, and Rama Chellappa. 2018. Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models. In International Conference on Learning Representations. https://openreview.net/ forum?id=BkJ3ibb0-

    2018

  31. [39]

    Christian Schlarmann and Matthias Hein. 2023. On the adversarial robustness of multi-modal foundation models. InProceedings of the IEEE/CVF International Conference on Computer Vision. 3677–3685

    2023

  32. [40]

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. 2024. Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders. arXiv preprint arXiv:2408.15998(2024)

    2024 arXiv

  33. [41]

    Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman

  34. [42]

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. Score-Based Generative Modeling through Stochas- tic Differential Equations. InInternational Conference on Learning Representations. https://openreview.net/forum?id=PxTIG12RRHS

    2021

  35. [43]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971(2023)

    2023 arXiv

  36. [44]

    Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. 2020. On adaptive attacks to adversarial example defenses.Advances in neural infor- mation processing systems33 (2020), 1633–1645

    2020

  37. [45]

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. 2016. Conditional image generation with pixelcnn decoders.Ad- vances in neural information processing systems29 (2016)

    2016

  38. [46]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  39. [47]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191(2024)

    2024 arXiv

  40. [48]

    Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. 2018. Mit- igating Adversarial Effects Through Randomization. InInternational Conference on Learning Representations

    2018

  41. [49]

    Ziyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du, Jinguo Zhu, Han Liu, Jinghui Chen, Ting Wang, and Fenglong Ma. 2023. VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models. InThirty-seventh Conference on Neural Information Processing Systems

    2023

  42. [50]

    Jiaming Zhang, Qi Yi, and Jitao Sang. 2022. Towards adversarial attack on vision-language pre-training models. InProceedings of the 30th ACM International Conference on Multimedia. 5005–5013

    2022

  43. [51]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068 (2022)

    2022 arXiv

  44. [52]

    The Blind Men and the Elephant

    Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Zhanhui Kang, and Yu Wang. 2025. Enhancing Contrastive Learning Inspired by the Philosophy of “The Blind Men and the Elephant”. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 22659–22667

    2025

  45. [53]

    Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Zhanhui Kang, and Yu Wang. 2025. QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguist...

    2025

  46. [54]

    Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang. 2024. PIP: Detecting Adversarial Examples in Large Vision-Language Models via At- tention Patterns of Irrelevant Probe Questions. InProceedings of the 32nd ACM International Conference on Multimedia. 11175–11183

    2024

  47. [55]

    Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Yu Wang, et al. 2024. DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models.arXiv preprint arXiv:2411.18659(2024)

    2024 arXiv

  48. [56]

    Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Zhanhui Kang, Di Wang, and Yu Wang. 2025. The Security Threat of Compressed Projectors in Large Vision-Language Models.arXiv preprint arXiv:2506.00534(2025)

    2025

  49. [57]

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. 2024. On evaluating adversarial robustness of large vision- language models.Advances in Neural Information Processing Systems36 (2024)

    2024

  50. [58]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models.arXiv preprint arXiv:2304.10592(2023)

    2023 arXiv

  51. [59]

    Xun Zhu, Zheng Zhang, Xi Chen, Yiming Shi, Miao Li, and Ji Wu. 2025. Connector- S: A Survey of Connectors in Multi-modal Large Language Models.arXiv preprint arXiv:2502.11453(2025)

    2025 arXiv

  52. [60]

    standard

    Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. 2024. Mova: Adapting mixture of vision experts to multimodal context.arXiv preprint arXiv:2404.13046(2024). F3: A Training-free and Efficient Visual Adversarial Example Puri...

    2024 arXiv

  53. [2017]

    InProceedings of the IEEE conference on computer vision and pattern recognition

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition. 6904–6913

  54. [2018]

    InInternational Conference on Learning Represen- tations

    PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples. InInternational Conference on Learning Represen- tations. MM ’25, October 27–31, 2025, Dublin, Ireland Yudong Zhang et al

    2025

  55. [2019]

    Image super-resolution as a defense against adversarial attacks.IEEE Transactions on Image Processing29 (2019), 1711–1724

    2019

  56. [2022]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems35 (2022), 23716–23736

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.