Pith. sign in

REVIEW 3 major objections 5 minor 49 references

On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A simplified two-stage attack pipeline — cross-modal word selection, text abstraction, and deletion of the repeated text attack — transfers better across vision-language models than the complex three-stage pipeline it replaces.

desk verdict A credible 'simpler is better' result for transferable vision-language attacks, but the key KL-based word selection is underspecified and the efficiency claims rest on different hardware. read the letter →

arxiv 2607.14974 v1 pith:MHVTVNT3 submitted 2026-07-16 cs.CV cs.CR

Yuchen Ren , Zhengyu Zhao , Chenhao Lin , Bo Yang , Chao Shen This is my paper
classification cs.CVcs.CR
keywords adversarialattacksvision-languagemodelstransferabilitycross-modalwordidentificationtextsemanticabstractionstagedeletionblack-boxmultimodalretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the prevailing three-stage pipeline for transferable adversarial attacks on vision-language models — text attack, image attack, then a second text attack — is not just wasteful but counterproductive. It identifies one flaw per stage: word selection ignores the paired image, the guiding text contains image-irrelevant words, and repeating the text attack overfits the surrogate model. The proposed SimVLA pipeline fixes these with cross-modal word identification, text semantic abstraction, and stage deletion. Across four datasets and three tasks, the simpler pipeline reports higher black-box transferability and lower compute: on Flickr30k, R@1 transferability rises by 8.01–14.71 percentage points while using about 36% of the time and 46% of the peak memory of the state-of-the-art baseline. If right, the field should simplify its attack recipe rather than complicate it.

What carries the argument

The carrying object is the image-embedding center fcenter(x), the mean of the surrogate model's image embeddings over NI noisy samples of the clean image. In Stage 1, KL divergence between this center and each masked text embedding selects the word to replace; in Stage 2, the same ranking is inverted to delete the Top-K words least relevant to the image, and the image is optimized against the truncated text through negative cosine similarity. The third mechanism is stage deletion: dropping the repeated text attack, justified by the observation that the extra stage lowers white-box similarity but lowers black-box transferability, a signature of surrogate overfitting.

What would settle it

A direct test: hold the replacement budget fixed, run SimVLA's word identification on a set of captions, and compare transferability against the same attack with a randomly chosen word replaced. If random-choice transferability matches ranking-based transferability within noise, the ranking is not carrying the effect. Equivalently, find a dataset where the highest-KL word is systematically not the word whose replacement breaks cross-modal alignment in black-box models.

Watch

Extended reading notes

Core claim

The central claim is that the word-identification step, not the loss function or the number of stages, is where multimodal attack transferability is won. SimVLA replaces the standard text-only word scoring with a cross-modal score: it averages the image embeddings of several noisy copies of the clean image to form an image-embedding center, masks each word in the caption, and ranks the masks by KL divergence between that image center and the masked text embedding. The word whose masking moves the text farthest from the image is replaced. The pipeline then strips the Top-K least image-relevant words from the adversarial text before optimizing the image, and it deletes the repeated text-attack

Load-bearing premise

The load-bearing premise is that masking a word and measuring KL divergence between the noisy-image embedding center and the masked text embedding identifies the word whose replacement most improves black-box transferability; the paper assumes this link rather than deriving it, so if the ranking only reflects white-box similarity the Stage-1 gain would collapse.

Editorial extensions

If this is right

  • Three-stage pipelines should be re-benchmarked without the repeated text-attack stage; the paper's results show removing it helps existing three-stage baselines on average, not just the proposed pipeline.
  • Efficiency becomes a primary selling point for transfer attacks: a simpler pipeline that uses a third of the time and under half the peak memory can still be the most transferable.
  • Word importance in multimodal attacks should be defined relative to the paired image, not the text alone; the paper's modality ablation shows image-guided selection clearly outperforms text-only selection.
  • The overfitting mechanism extends beyond text-image retrieval: cross-model-and-task attacks on visual entailment and visual grounding also improve when the redundant stage is removed.
  • If the reported gains hold, the community's search for better transfer attacks should shift from designing complex losses to designing better cross-modal word-selection criteria.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same cross-modal KL ranking could be reused as an interpretability tool — e.g., identifying which word most changes image-text alignment for caption auditing or dataset debugging, outside the attack setting.
  • Editorial extension: the stage-deletion finding suggests a testable principle — any extra optimization step that lowers surrogate loss without flattening the loss landscape will tend to overfit; adding a fourth stage should reduce transferability further.
  • Editorial extension: because the method only needs embeddings from the surrogate, it should port to any contrastively trained multimodal encoder; a quick check on a fusion architecture not in the tested set would establish the boundary of the claim.
  • Editorial extension: the defense results — image defenses barely reduce success while text defenses sharply do — point toward defending the text-image alignment at the semantic embedding level rather than at the pixel level.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SimVLA, a simplified transferable attack pipeline for vision-language models. Relative to the common three-stage pipeline (text -> image -> text) used by SGA/DRA/SA-AET, SimVLA (i) selects the word to replace in Stage 1 by ranking masked texts against a noisy-image embedding center using KL divergence, (ii) removes Top-K image-irrelevant words from the adversarial text before the image attack, and (iii) deletes Stage 3. The authors report consistent R@1/R@10 gains on Flickr30k/MSCOCO across ALBEF/TCL/CLIPViT/CLIPCNN targets, plus results on BLIP, robust models and MLLMs, and lower compute/VRAM. The central claim is that simplicity plus cross-modal word identification improves transferability and efficiency.

Significance. The empirical scope is a strength: four datasets, three tasks, four core models, additional BLIP/robust/MLLM targets, controlled ablations (Table 15), and a stage-deletion check on baselines (Table 21). If the proposed word-selection criterion were precisely defined, the claim that simpler is better would be a valuable, falsifiable result: Table 15 shows each component contributes, and Tables 1/21 support the over-optimization interpretation of Stage 3. However, the two central selection steps are currently not well-defined, which blocks independent reproduction and prevents the reader from knowing whether the gains come from the proposed principle or from an implementation detail.

major comments (3)
  1. [§3.2.1, Eqs. (2),(4); §3.3.1] KL divergence is computed on raw embedding vectors f_T(t), f_T(t'), and f_center(x), but these are not probability distributions and no conversion is specified (softmax over dimensions? over tokens? temperature? normalization? common support). Since Eq. (4) is the core of cross-modal word identification and the same KL ranking underlies text semantic abstraction in §3.3.1, the selected word and Top-K words are implementation-dependent. The gains in Table 15 could therefore reflect an undocumented conversion rather than the proposed principle. Please specify the exact KL computation, or replace it with a defined divergence and rerun the ablations.
  2. [§4.1.5, §4.3.7, Fig. 3] All headline results and parameter analyses use hyperparameters (beta=16, Top-K=1, NI=4, gamma=0.1) selected on the same Flickr30k/ALBEF->CLIPViT configuration that is used to report the main gains, and every number is from a single seed (seed=42). A transferability claim needs either independent validation of the selected hyperparameters on a held-out surrogate/target pair or variance estimates across seeds; otherwise the reported 8.01%-14.71% margin may partly reflect target-specific tuning.
  3. [§4.1.5, Table 13] The efficiency comparison is not hardware-controlled: SA-AET is reproduced on an L40s GPU while SimVLA, SGA, and DRA run on an RTX 3090. Table 13 then reports computation time and max VRAM for all methods. Since the abstract's efficiency headline (35.73% time, 46.26% VRAM versus the SOTA baseline) relies on this table, the time comparison should be rerun on identical hardware or reported with a clear hardware-normalization caveat.
minor comments (5)
  1. [Abstract vs §4.1.5] The abstract states "Code is available at https://github.com/RYC-98/SimVLA", but §4.1.5 says "The code will be released as soon as possible upon the final decision." This inconsistency should be fixed.
  2. [§4.2.2, Table 6] The text claims a 6.55% improvement over DRA for the CLIPViT->ALBEF VE transfer, but Table 6 shows SimVLA=38.97 vs DRA=32.70 (6.27), and the CLIPCNN->ALBEF row gives 39.07 vs 32.72 (6.35). The stated value does not match the table.
  3. [Table 8] The JPEG defense row for SimVLA IR R@1 reads "61.50 (+0.1.26)"; the parenthetical value is malformed.
  4. [Table 15] The claim that text semantic abstraction (Stage 2) is "necessary" is not supported on all metrics: for TCL IR R@10, "w/o refining Stage 2" gives 45.78, slightly higher than SimVLA's 45.60. Please soften the claim or explain why the overall pattern still justifies the component.
  5. [Appendix B, Eqs. (9)-(10)] Equation (10) uses x_{t,i} on the right-hand side, but x_{t,i} is not defined; Eq. (9) defines x_{t-1,i} and x'_{t,i}. This should be corrected for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed components are ablated against the same pipeline and transferability is measured on held-out black-box target models.

full rationale

Reviewing the derivation chain, none of SimVLA's claimed improvements reduces to its own inputs by construction. The three stage-wise refinements (cross-modal word identification, text semantic abstraction, and stage deletion) are each validated through controlled ablations in Table 15, where removing one refinement while keeping the rest degrades performance. The central results are black-box transferability numbers on held-out target models (CLIPViT, CLIPCNN, TCL, BLIP, and MLLMs), not quantities fitted on those targets. Equations 4 and 6 define the attack procedure, but the paper does not claim to 'predict' transferability from those equations; the gains are empirical and externally measured. The underspecification of how embedding vectors are converted into distributions for KL divergence (Eqs. 2/4) is a reproducibility and rigor concern, not a circularity: it does not make the output equal to the input. The self-citations ([26], [27], [44], [47]) are not load-bearing: [44] is cited only to support the statement that white-box success is not the focus of transferable attacks, and the others support auxiliary experimental choices. Hyperparameters such as beta, Top-K, NI, and gamma were selected on the same Flickr30k/ALBEF-to-CLIPViT configuration that yields headline numbers, which is an in-sample tuning concern, but it is not a case of a fitted parameter being renamed as a prediction; the method still transfers across many other dataset/model/task combinations. No step in the paper's derivation reduces to its own input by definition, by fitting, or by a self-citation chain.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical; no new theoretical entities are introduced. The free parameters are attack hyperparameters tuned on the evaluation benchmark, and the domain assumptions are inherited from prior adversarial-attack literature. The proposed components are empirical refinements, not new physical or mathematical objects.

free parameters (4)
  • beta (boundary factor for noisy image sampling) = 16
    Used in Eq. 3 to define the neighborhood for the image embedding center fcenter(x). Chosen via sensitivity analysis in Fig. 3 (left) on the main benchmark.
  • Top-K (number of irrelevant words removed in text semantic abstraction) = 1
    Used in Stage 2. Selected via Fig. 3 (middle); removing 1-2 words is best, and 1 is used for all experiments.
  • NI (number of sampled images) = 4
    Sample count in Eq. 3 and in MIM-PGIA; selected to match SGA/DRA and validated in Fig. 3 (right).
  • gamma (reverse step factor in MIM-PGIA) = 0.1
    Inner-optimization step in Appendix B; tuned via Fig. 4 on the same benchmark used for headline results.
assumptions (5)
  • domain assumption Embedding-space KL divergence ranks word impact for cross-modal attacks.
    Eqs. 2/4 adopt KL divergence used in prior attacks; the paper does not derive or independently validate this measure beyond ablations. Entering in Section 3.2.1.
  • domain assumption The center of noisy-image embeddings fcenter(x) is a faithful surrogate for the paired image in word selection.
    Eq. 3 assumes adding uniform noise and averaging over NI samples yields an image representation useful for identifying important words; Section 3.2.1.
  • ad hoc to paper Deleting Stage 3 reduces surrogate overfitting rather than removing useful gradient information.
    This is the proposed stage-deletion hypothesis; supported only by Table 1 and Table 21 and not derived from first principles. Section 3.4.
  • ad hoc to paper Removing Top-K image-irrelevant words from adversarial text improves image-attack guidance.
    Text semantic abstraction in Section 3.3.1; motivated by language-image relevance, validated only via ablation.
  • domain assumption Flatness of the loss landscape improves transferability.
    Adopted from RAP [23] and MIM-PGIA [24] in Section 3.3.2 and Appendix B; not proven in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline." pith.science (2026). https://pith.science/paper/MHVTVNT3

@misc{pith2026260714974,
  author       = {Pith},
  title        = {Pith review of: On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MHVTVNT3}},
  note         = {Machine review of arXiv:2607.14974}
}
read the original abstract

Vision-Language Pre-training Models (VLPMs) are known to be vulnerable to adversarial attacks. Recent transferable attacks on VLPMs have followed a common pipeline with complicated loss functions or multi-stage text/image attacks. However, in this paper, we demonstrate that such a sophisticated attack pipeline can be simpler yet more successful. Specifically, we identify three previously overlooked issues caused by inappropriate cross-modal interactions and excessive operations. To address them, we propose the Simple Vision-Language Attack (SimVLA) pipeline, which observably improves transferability and efficiency. Experiments on four datasets and three downstream tasks validate the superiority of our pipeline. For instance, on Flickr30k text-image retrieval dataset, our SimVLA outperforms the SOTA baseline in R@1 transferability by 8.01\%-14.71\%, while consuming only about 35.73\% of the time and 46.26\% of the max VRAM. Overall, the superiority of our SimVLA highlights the importance of leveraging domain knowledge (e.g., our proposed cross-modal word identification), while blindly pursuing intricate operations (e.g, complex loss functions and redundant multi-stage designs) may even be harmful. We hope our SimVLA can serve as a simple yet effective backbone for future extensions. Code is available at https://github.com/RYC-98/SimVLA.

Figures

Figures reproduced from arXiv: 2607.14974 by the authors.

Figure 1
Figure 1. We identify three stage-specific problems in the com [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of our Simple Vision-Language Attack (SimVLA) pipeline. Our contributions focus on (a) cross-modal word [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Parameter sensitivity analysis. Left: the boundary factor of sampled images β in cross-modal word identification (Stage 1), Middle: the number of removed most irrelevant words Top-K in text semantic abstraction (Stage 2), and Right: the sample number NI in cross-modal word identification (Stage 1), and optimization of adversarial image (Stage 2). Here, the surrogate (target) model is ALBEF (CLIPViT). 5. Conclusion a… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Parameter sensitivity analysis of the reverse step factor [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 7 linked inside Pith

  1. [1]

    Qwen2.5-VL technical report

    S Bai, K Chen, X Liu, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. 7, 9

    arXiv 2025

  2. [2]

    SceneTAP: Scene-coherent typographic adversarial planner against vision-language mod- els in real-world environments

    Yue Cao, Yun Xing, Jie Zhang, Di Lin, Tianwei Zhang, Ivor Tsang, Yang Liu, and Qing Guo. SceneTAP: Scene-coherent typographic adversarial planner against vision-language mod- els in real-world environments. In2025 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 25050–25059, 2025. 3

    2025

  3. [3]

    Boosting adversarial attacks with momentum

    Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9185–9193,

  4. [4]

    Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory

    Sensen Gao, Xiaojun Jia, Xuhong Ren, Ivor Tsang, and Qing Guo. Boosting transferability in vision-language attacks via diversification along the intersection region of adversarial trajectory. InComputer Vision – ECCV 2024, pages 442–460,

    2024

  5. [5]

    Countering adversarial images using input transformations

    Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. InInternational Conference on Learning Representations, 2018. 9

    2018

  6. [6]

    OT-Attack: Enhancing adversarial transferability of vision-language models via optimal trans- port optimization.arXiv preprint, arXiv:2312.04403, 2023

    Dongchen Han, Xiaojun Jia, Yang Bai, Jindong Gu, Yang Liu, and Xiaochun Cao. OT-Attack: Enhancing adversarial transferability of vision-language models via optimal trans- port optimization.arXiv preprint, arXiv:2312.04403, 2023. 1, 3, 8, 9

    arXiv 2023

  7. [7]

    SA-Attack: Improving adversarial transferability of vision-language pre-training models via self- augmentation.arXiv preprint, arXiv:2312.04913, 2023

    Bangyan He, Xiaojun Jia, Siyuan Liang, Tianrui Lou, Yang Liu, and Xiaochun Cao. SA-Attack: Improving adversarial transferability of vision-language pre-training models via self- augmentation.arXiv preprint, arXiv:2312.04913, 2023. 2, 3, 4, 5, 8, 9

    arXiv 2023

  8. [8]

    Feature space perturbations yield more transferable adversar- ial examples

    Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversar- ial examples. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7066–7074,

Show all 49 references
  1. [9]

    Tsang, and Xiaochun Cao

    Xiaojun Jia, Sensen Gao, Qing Guo, Simeng Qin, Ke Ma, Yihao Huang, Yang Liu, Ivor W. Tsang, and Xiaochun Cao. Semantic-aligned adversarial evolution triangle for high- transferability vision-language attack.IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(10)...

    2025

  2. [10]

    BERT: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. InProceedings of naacL- HLT, page 2, 2019. 4

    2019

  3. [11]

    Align before fuse: Vision and language representation learning with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. InAdvances in neural information processing systems, pages 9694–9705, 2021. 1, 3, 7

    2021

  4. [12]

    BLIP: Bootstrapping language-image pre-training for unified vision- language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision- language understanding and generation. InInternational conference on machine learning, pages 12888–12900, 2022. 7, 8

    2022

  5. [13]

    BERT-Attack: Adversarial attack against BERT using BERT

    Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-Attack: Adversarial attack against BERT using BERT. InConference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, 2020. 2, 3, 5, 8

    2020

  6. [14]

    Microsoft COCO: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings...

    2014

  7. [15]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in neural information processing systems, 2024. 7, 9

    2024

  8. [16]

    GLEAM: En- hanced transferable adversarial attacks for vision-language pre-training models via global-local transformations

    Yunqi Liu, Xue Ouyang, and Xiaohui Cui. GLEAM: En- hanced transferable adversarial attacks for vision-language pre-training models via global-local transformations. InPro- ceedings of the IEEE/CVF International Conference on Com- puter Vision, pages 1665–1674, 2025. 3

    2025

  9. [17]

    Frequency domain model augmentation for adversarial attack

    Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xi- anglong Liu, Jian Zhang, and Jingkuan Song. Frequency domain model augmentation for adversarial attack. InEuro- pean Conference on Computer Vision, pages 549–566, 2022. 9

    2022

  10. [18]

    Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models

    Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models. InIEEE/CVF International Conference on Computer Vision (ICCV), pages 102–111, 2023. 1, 2, 3, ...

    2023

  11. [19]

    Towards deep learn- ing models resistant to adversarial attacks.arXiv preprint, arXiv:1706.06083, 2017

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learn- ing models resistant to adversarial attacks.arXiv preprint, arXiv:1706.06083, 2017. 3, 8

    2017 arXiv

  12. [20]

    Understanding zero-shot adversarial robust- ness for large-scale models

    Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl V ondrick. Understanding zero-shot adversarial robust- ness for large-scale models. InInternational Conference on Learning Representations, 2023. 7, 9

    2023

  13. [21]

    OpenAI. ChatGPT. https://chat.openai.com/ chat, 2023. 7, 10

    2023

  14. [22]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspon- dences for richer image-to-sentence models. InIEEE/CVF International Conference on Computer Vision (ICCV), pa...

    2015

  15. [23]

    Boosting the transferability of ad- versarial attacks with reverse adversarial perturbation

    Zeyu Qin, Yanbo Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu. Boosting the transferability of ad- versarial attacks with reverse adversarial perturbation. In Advances in neural information processing systems, pages 29845–29858, 2022. 2, 5, 14

    2022

  16. [24]

    Enhancing adversarial transferability through neighborhood conditional sampling.arXiv preprint, arXiv:2401.08734,

    Chunlin Qiu, Yiheng Duan, Lingchen Zhao, and Qian Wang. Enhancing adversarial transferability through neighborhood conditional sampling.arXiv preprint, arXiv:2401.08734,

    arXiv

  17. [25]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  18. [26]

    Improving adversarial transferability on vision transformers via forward propagation refinement

    Yuchen Ren, Zhengyu Zhao, Chenhao Lin, Bo Yang, Lu Zhou, Zhe Liu, and Chao Shen. Improving adversarial transferability on vision transformers via forward propagation refinement. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 25071–25080, 2025. 9

    2025

  19. [27]

    Yuchen Ren, Zhengyu Zhao, Chenhao Lin, Bo Yang, Lu Zhou, Zhe Liu, and Chao Shen. Improving integrated gradient-based transferable adversarial examples by refining the integration path.Proceedings of the AAAI Conference on Artificial Intel- ligence, 39(7):6731–6739, 2025. 9

    2025

  20. [28]

    Robust CLIP: Unsupervised adversar- ial fine-tuning of vision embeddings for robust large vision- language models

    Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust CLIP: Unsupervised adversar- ial fine-tuning of vision embeddings for robust large vision- language models. InInternational Conference on Machine Learning, pages 43685–43704. PMLR, 2024. 7, 9

    2024

  21. [29]

    FaceNet: A unified embedding for face recognition and clus- tering

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified embedding for face recognition and clus- tering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015. 1

    2015

  22. [30]

    Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad- CAM: Visual explanations from deep networks via gradient- based localization. InIEEE/CVF International Conference on Computer Vision (ICCV), pages 618–626, 2017. 1

    2017

  23. [31]

    A little robustness goes a long way: Leveraging robust features for targeted transfer attacks.Advances in Neural Information Processing Systems, 34:9759–9773, 2021

    Jacob Springer, Melanie Mitchell, and Garrett Kenyon. A little robustness goes a long way: Leveraging robust features for targeted transfer attacks.Advances in Neural Information Processing Systems, 34:9759–9773, 2021. 8

    2021

  24. [32]

    Circle loss: A unified perspective of pair similarity optimization

    Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6398–6407, 2020. 1

    2020

  25. [33]

    Enhancing the transferability of adversarial attacks through variance tuning

    Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1924–1933, 2021. 4, 9

    1924

  26. [34]

    Exploring transferability of multimodal adversarial samples for vision-language pre- training models with contrastive learning.arXiv preprint, arXiv:2308.12636, 2023

    Youze Wang, Wenbo Hu, Yinpeng Dong, Hanwang Zhang, Hang Su, and Richang Hong. Exploring transferability of multimodal adversarial samples for vision-language pre- training models with contrastive learning.arXiv preprint, arXiv:2308.12636, 2023. 1, 3, 4, 8

    2023 arXiv

  27. [35]

    Bovik, H.R

    Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE Transactions on Image Processing, 13(4): 600–612, 2004. 10

    2004

  28. [36]

    Visual entailment: A novel task for fine-grained image understand- ing.arXiv preprint arXiv:1901.06706, 2019

    Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understand- ing.arXiv preprint arXiv:1901.06706, 2019. 6

    1901 arXiv

  29. [37]

    Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability

    Yifeng Xiong, Jiadong Lin, Min Zhang, John E Hopcroft, and Kun He. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14983–14992, 2022. 9

    2022

  30. [38]

    Feature squeezing: Detecting adversarial examples in deep neural networks

    Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In Network and Distributed System Security Symposium, 2018. 9

    2018

  31. [39]

    Vision-language pre-training with triple contrastive learning

    Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15650–15659, 2022. 3, 7

    2022

  32. [40]

    VLATTACK: Multimodal adversarial attacks on vision- language tasks via pre-trained models

    Ziyi Yin, Muchao Ye, Tianrong Zhang, Tianyu Du, Jinguo Zhu, Han Liu, Jinghui Chen, Ting Wang, and Fenglong Ma. VLATTACK: Multimodal adversarial attacks on vision- language tasks via pre-trained models. InAdvances in Neural Information Processing Systems, pages 52936–52956, 2023. 2

    2023

  33. [41]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016. 6

    2016

  34. [42]

    Towards adversarial attack on vision-language pre-training models

    Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack on vision-language pre-training models. In30th ACM International Conference on Multimedia, pages 5005–5013,

  35. [43]

    Univer- sal adversarial perturbations for vision-language pre-trained models

    Peng-Fei Zhang, Zi Huang, and Guangdong Bai. Univer- sal adversarial perturbations for vision-language pre-trained models. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 862–871, 2024. 3

    2024

  36. [44]

    On success and simplicity: A second look at transferable targeted attacks

    Zhengyu Zhao, Zhuoran Liu, and Martha Larson. On success and simplicity: A second look at transferable targeted attacks. InAdvances in Neural Information Processing Systems, pages 6115–6128, 2021. 8

    2021

  37. [45]

    A unified understanding of adversarial vulnerability regarding unimodal models and vision-language pre-training models

    Haonan Zheng, Xinyang Deng, Wen Jiang, and Wenrui Li. A unified understanding of adversarial vulnerability regarding unimodal models and vision-language pre-training models. InProceedings of the 32nd ACM International Conference on Multimedia, page 18–27. Association for Compu...

    2024

  38. [46]

    AdvCLIP: Downstream-agnostic adversarial examples in multimodal contrastive learning

    Ziqi Zhou, Shengshan Hu, Minghui Li, Hangtao Zhang, Yechao Zhang, and Hai Jin. AdvCLIP: Downstream-agnostic adversarial examples in multimodal contrastive learning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6311–6320, 2023. 1, 3

    2023

  39. [47]

    Boosting adversarial transferability via gradi- ent relevance attack

    Hegui Zhu, Yuchen Ren, Xiaoyan Sui, Lianping Yang, and Wuming Jiang. Boosting adversarial transferability via gradi- ent relevance attack. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4718–4727,

  40. [2022]

    2, 4, 5, 6, 7, 8, 11

  41. [2025]

    2, 3, 4, 5, 6, 7, 8, 11

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.