Pith. sign in

REVIEW 4 major objections 5 minor 61 references

Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DAFT fine-tunes a CLIP vision encoder under two supervision signals—clean-image features and image captions—so that one robust encoder, swapped into LLaVA-1.5, transfers adversarial robustness to zero-shot classification, captioning, and VQ

desk verdict A sensible incremental defense that consistently edges out FARE across tasks, but the SOTA claim rests on an under-specified attack threshold and missing variance; worth a real referee. read the letter →

arxiv 2607.18958 v1 pith:VWGBNY7U submitted 2026-07-21 cs.CV cs.AI

Sibo Wang , Jie Zhang , Shiguang Shan , Xilin Chen , Wen Gao This is my paper
classification cs.CVcs.AI
keywords adversarialrobustnesslargevision-languagemodelsfine-tuningCLIPvisionencodertripletlosshardnegativeminingimagecaptioningvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the visual encoder, not the language model, is the place to defend large vision-language models against adversarial image attacks. It introduces DAFT, a dual adversarial fine-tuning scheme that trains the CLIP image encoder with two signals at once: an L2 visual-supervision term that pins adversarial features to the clean features of the frozen original encoder, and a caption triplet term that binds adversarial images to their true captions while repelling the hardest false caption in the batch. The authors claim that the resulting encoder, dropped into LLaVA-1.5 with no retraining of the language half, beats existing adversarial fine-tuning methods in robust accuracy across eleven zero-shot classification datasets and six captioning/VQA datasets while keeping clean accuracy comparable. If the claim holds, robust LVLMs become a plug-in upgrade: harden one encoder, then swap it into any compatible model.

What carries the argument

The load-bearing mechanism is a dual-loss with two frozen reference points: a frozen original CLIP vision encoder supplying clean-image features for L2 visual supervision, and a frozen CLIP text encoder supplying caption embeddings for a triplet contrastive loss with hardest-negative sampling. The caption triplet is the novelty—descriptive captions replace category labels, and the hardest negative is the caption in the batch most similar to the adversarial features, which forces the encoder to preserve fine semantic distinctions under attack.

What would settle it

Re-run the paper's captioning and VQA evaluation using a single fixed, pre-registered CIDEr threshold for every model (or no second-stage filter at all), with the same 100-step FP32 APGD attack, over at least three random seeds; if DAFT's average gains over the previous unsupervised method fall within seed-to-seed variance (roughly 1 CIDEr point or 1% VQA accuracy) or reverse, the claimed superiority is not established.

Watch

Extended reading notes

Core claim

DAFT fine-tunes only the vision encoder of CLIP ViT-L/14 on COCO captions, using PGD-10 attacks with L-infinity budget 4/255. The total loss is L_sem + γL_vis: the visual branch minimizes ||F_ori(x) − F_θ(x_a)||², keeping the adversarial embedding close to the clean embedding of the frozen original encoder; the semantic branch maximizes cosine distance to the hardest negative caption and minimizes distance to the ground-truth caption in a triplet loss with margin a. The same loss drives both attack generation and parameter updates in a min-max loop. The paper reports that DAFT achieves the best average robust accuracy at epsilon 2/255 and 4/255 (and 8/255 in supplementary material) on zero-s

Load-bearing premise

The state-of-the-art claim rests on the two-stage caption/VQA evaluation in Sec. IV-C, where the unreported CIDEr threshold decides which samples face the stronger second attack—if that threshold was set per method after seeing results, the comparison is not controlled.

Editorial extensions

If this is right

  • Robustness becomes a drop-in property: one DAFT-trained vision encoder can replace the original encoder in LLaVA-1.5 (and a second CLIP-based LVLM in the paper's supplementary experiments) with no task-specific retraining or architecture changes.
  • Caption supervision is doing the heavy lifting: in the paper's ablation, replacing captions with category labels drops average robust classification by 12.4%, caption CIDEr by 15.2, and VQA accuracy by 4.2%.
  • Hard-negative sampling is necessary, not just helpful: training with the easiest negative as the triplet contrast yields near-zero robust accuracy, essentially matching the undefended model.
  • The balance between the two supervision signals matters: for a=0.2 and γ=0.5 the method peaks; both larger and smaller values degrade robustness on all three tasks, and batch size 64 gives the best negatives.
  • Robustness transfers across architectures: the paper's black-box transfer tests show DAFT outperforms prior methods both when attacks are generated on one LVLM and evaluated on another.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation section grants that DAFT is a white-box, CLIP-vision-encoder-only defense; an immediate extension question is whether the same caption-triplet supervision helps non-CLIP encoders or closed models, which is untested.
  • If caption diversity is the active ingredient, training DAFT on larger or more descriptive caption corpora than COCO should improve robustness further; the paper leaves this scaling test open.
  • Because the caption/VQA protocol filters samples by an unreported CIDEr threshold before the second attack stage, the reported 1–2 point gains over the unsupervised baseline need a re-run with one fixed, pre-registered threshold and multiple seeds before the SOTA claim can be taken at face value.
  • The hardest-negative result predicts that DAFT's advantage will grow on datasets with many visually similar captions or classes, since the loss explicitly sharpens separation between confusable captions under attack.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DAFT, a dual-supervision adversarial fine-tuning method for the CLIP vision encoder used in LVLMs such as LLaVA-1.5. The method combines a visual supervision loss (L2 distance between adversarial features and clean features from a frozen original encoder) with a semantic supervision loss (triplet loss that aligns the adversarial image embedding with its ground-truth caption and pushes away the hardest negative caption in the batch). The vision encoder is fine-tuned on COCO captions under PGD-10 attacks and then swapped into LLaVA-1.5 or OpenFlamingo without retraining. The paper claims state-of-the-art adversarial robustness across zero-shot classification (11 datasets), image captioning (COCO, Flickr30k), and VQA (VQAv2, OKVQA, VizWiz, TextVQA), comparing against TeCoA, PMG-AFT, and FARE, with additional ablations and black-box transfer results.

Significance. If the empirical claims hold, DAFT would be a practical contribution: a single adversarially fine-tuned CLIP encoder that can be dropped into multiple LVLMs to improve robustness across tasks. The semantic-supervision idea, using descriptive captions instead of class labels, is well motivated and the ablations (Tables III and IV) provide useful evidence for the design choices. The cross-model evaluation on OpenFlamingo and black-box transfer results (Tables VII and VIII) are strengths and go beyond many papers in this area. However, the headline claim that DAFT outperforms the state-of-the-art method rests on small margins over FARE (0.9–2.3 points in most tables) on 1000-sample evaluations, and the evaluation protocol contains an underspecified second-stage attack threshold. These issues must be resolved before the SOTA claim is convincing.

major comments (4)
  1. [§IV-C, Tables II and VI] The central SOTA claim depends on the multi-stage attack protocol, but the protocol is underspecified. The paper retains for the second FP32 APGD stage only samples whose CIDEr exceeds 'a certain threshold' for captioning, or that were 'not successfully attacked' for VQA. The threshold value and the definition of successful attack are never reported, nor is it stated that the same criterion is applied to every model. Since the reported DAFT-over-FARE margins are 0.9–2.3 points on 1000-sample evaluations, a model-dependent or post hoc threshold could determine the outcome. Please report the exact threshold/criterion, verify it is fixed across all models, and report the number of samples entering the second stage per model.
  2. [Tables I, II, VII, VIII] No variance estimates, seeds, or significance tests are reported. For example, under eps=2/255, DAFT exceeds FARE by 1.2% average zero-shot accuracy, 2.3 CIDEr, and 1.1% VQA accuracy; under eps=4/255 the margins are 1.4%, 2.1, and 0.9%. On a 1000-sample evaluation, such differences can easily lie within run-to-run variance. The authors should report results over multiple training seeds (or at least bootstrap confidence intervals over the evaluation samples) and a paired significance test where appropriate. Without this, the claim that DAFT is state-of-the-art is not statistically supported.
  3. [§IV-A and Table II] The model is trained on COCO captions and evaluated on COCO captioning, but the paper does not specify the train/evaluation split. If the COCO captioning evaluation uses images or captions from the training set, DAFT would have an unfair advantage because its semantic supervision directly uses the ground-truth captions, whereas FARE and TeCoA do not use captions. Please specify the exact split (e.g., Karpathy train/test) and confirm that the evaluation images and captions are disjoint from the fine-tuning data.
  4. [§IV-D, Table V and Fig. 4] Hyperparameters gamma, a, and batch size are selected by evaluating average robust performance on the same three task families and the same epsilon=2/255 setting used in the main results. With the small margins over FARE, selection on the evaluation benchmarks can inflate the reported gains. The authors should either use a separate validation set for hyperparameter selection or provide a sensitivity analysis that shows the reported gains are not an artifact of choosing the best-performing configuration on the test benchmarks.
minor comments (5)
  1. [Abstract / Contributions] Typo: 'adversarial traning' should be 'adversarial training'. Please proofread the manuscript; there are also inconsistent renderings of 'LLaVA' and 'LLaV A'.
  2. [§IV-C] The attack protocol is said to be 'consistent with previous work [11], [19]'. If the threshold and VQA success criterion are defined in FARE, please cite the exact equations or section; otherwise, the details must be given here.
  3. [Table VI (supplementary)] Table VI repeats Table II plus an eps=8/255 row. Consider replacing the duplicate with only the new eps=8/255 results, or clearly label it as an extension.
  4. [§III-B, Eq. (6)] The triplet loss is written with cosine similarities and a margin a. It would be helpful to state explicitly that a is in [0,2] conceptually (though the paper tunes in [0.1,0.5]) and to define the 'hardest' negative precisely when multiple captions share the maximum similarity.
  5. [§IV-A] The statement 'We also maintain the same settings as FARE [19] for the experiments' is vague. Please list the exact training iterations, learning rate schedule, and attack hyperparameters used for each baseline so that replication is possible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DAFT's robustness results are empirically measured under independent attacks; the small margins and evaluation protocol are correctness concerns, not circularity.

full rationale

DAFT's claimed derivation is an empirical fine-tuning scheme, not a closed-form prediction. The method defines L_DAFT = L_sem + gamma * L_vis (Eq. 7), with L_vis measuring L2 distance to frozen original-encoder clean features (Eq. 5) and L_sem being a caption-based triplet loss (Eq. 6). Robustness is obtained by solving the min-max objective in Eqs. 8-9 and then evaluated with task-specific AutoAttack/APGD attacks. No reported robust accuracy, CIDEr score, or VQA accuracy appears as a parameter in these equations, and no term is defined in terms of the SOTA numbers it is claimed to beat. The only self-referential element is citation to the authors' own prior work PMG-AFT [23] as a baseline; it is not used to justify DAFT's design or to rule out alternatives, so it is not load-bearing. Training on COCO captions and evaluating captioning on COCO/Flickr30k, and tuning hyperparameters on the same benchmarks (Tables V, Fig. 4), are standard test-set-selection/overfitting concerns; they could inflate or make fragile the reported margins over FARE but do not make any 'prediction' equivalent by construction to an input. The unspecified second-stage CIDEr threshold in Sec. IV-C is an evaluation-protocol weakness, not a circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

DAFT introduces no new entities and contains no first-principles derivation; it relies on standard adversarial-training machinery and several domain assumptions about transfer from caption training to other tasks. The main burden is the free parameters tuned on the evaluation benchmarks themselves, alongside the unreported attack threshold.

free parameters (4)
  • gamma (L_vis weight) = 0.5
    Chosen by hyperparameter sweep in Tab. V to maximize average robust performance on the evaluation tasks; no held-out validation.
  • a (triplet margin) = 0.2
    Chosen by hyperparameter sweep in Tab. V; the paper states robustness peaks at a=0.2.
  • batch size = 64
    Selected from Fig. 4 as the value giving best robustness across tasks; batch size controls hard-negative difficulty.
  • CIDEr threshold for stage-2 attack = unspecified
    Sec. IV-C says samples with CIDEr above 'a certain threshold' receive a second FP32 APGD attack; the threshold is never reported, yet it changes which samples are attacked harder.
assumptions (4)
  • domain assumption Pre-trained CLIP clean-image features are a good reference for preserving generalization during adversarial fine-tuning.
    Visual supervision L_vis (Eq. 5) minimizes distance to F_ori(x); if original features are not a suitable target, the method would discard useful robust features.
  • domain assumption Robustness to PGD attacks generated with LDAFT transfers to AutoAttack and task-specific APGD attacks at test time.
    Training attacks use the LDAFT objective (Eq. 8) while evaluation attacks use task losses (Sec. IV-B/IV-C); no proof of transfer is given.
  • domain assumption COCO captions provide sufficient semantic supervision for adversarial robustness on VQA and other caption datasets.
    DAFT trains only on COCO-caption but claims cross-task generalization to Flickr30k, VQAv2, OKVQA, VizWiz, and TextVQA.
  • domain assumption Replacing only the CLIP vision encoder is enough to make the whole LVLM robust; the LLM and projector do not limit robustness.
    The method fine-tunes only the vision encoder and swaps it into LLaVA/OpenFlamingo (Sec. III-A).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model." pith.science (2026). https://pith.science/paper/VWGBNY7U

@misc{pith2026260718958,
  author       = {Pith},
  title        = {Pith review of: Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VWGBNY7U}},
  note         = {Machine review of arXiv:2607.18958}
}
abstract

While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.g., zero-shot classification) and consequently lack generalizability across various multimodal tasks. To address this limitation, we propose a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals from two modalities, enhancing model robustness while generalizing across multiple downstream tasks. The proposed framework comprises two core components, i.e., $\textbf{Visual}$ supervision branch and $\textbf{Semantic}$ supervision branch. The former branch leverages features from clean images, extracted via a frozen original vision encoder, to guide adversarial robustness while the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack. Moreover, our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model, with no need of separate task-specific retraining or architecture modifications.Extensive experiments demonstrate that our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering (VQA) tasks.

Figures

Figures reproduced from arXiv: 2607.18958 by the authors.

Figure 1
Figure 1. Results under perturbation budget ϵ = 2/255. (a) Ad￾versarial accuracy of CLIP with different fine-tuning methods on zero-shot classification. (b) Robust captioning and VQA performance after replacing the original vision encoder of LLaVA with different CLIP vision encoders. erroneous outputs, even in closed-source models [10]–[13]. These attacks introduce human-imperceptible noise to visual inputs, significantly deg… view at source ↗
Figure 2
Figure 2. Examples of robust evaluation on VQA (left) and image caption (right) tasks. We perform untargeted [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The pipeline of DAFT. Visual supervision branch maximizes the similarity between adversarial feature on the target [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Average robustness performance of models trained with [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 8 linked inside Pith

  1. [29]

    Quality text, robust vision: The role of language in enhancing visual robustness of vision-language models,

    F. Waseda, S. Sugawara, and I. Echizen, “Quality text, robust vision: The role of language in enhancing visual robustness of vision-language models,” inProc. ACM Int. Conf. Multimedia, 2025, pp. 4808–4816

    2025

  2. [1]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 26 296–26 306

    2024

  3. [2]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” inAdv. Neural Inf. Process. Syst., vol. 36, 2023, pp. 34 892–34 916

    2023

  4. [3]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing vision-language understanding with advanced large language models,” inProc. Int. Conf. Learn. Represent. (ICLR), 2024

    2024

  5. [4]

    VisualGPT: Data- efficient adaptation of pretrained language models for image captioning,

    J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny, “VisualGPT: Data- efficient adaptation of pretrained language models for image captioning,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, pp. 18 030–18 040

    2022

  6. [5]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .- H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” inProc. Int. Conf. Mach. Learn. (ICML), 2021, pp. 4904–4916

    2021

  7. [6]

    Adapting multimodal large language models for video question answering by capturing question-critical and coherent moments,

    H. Wang, C. Lai, and W. Ge, “Adapting multimodal large language models for video question answering by capturing question-critical and coherent moments,”IEEE Trans. Multimedia, vol. 27, pp. 8737–8747, 2025

    2025

  8. [7]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inProc. Int. Conf. Mach. Learn. (ICML), 2021, pp. 8748–8763

    2021

Show all 61 references
  1. [8]

    MoE-LLaV A: Mixture of experts for large vision- language models,

    B. Lin, Z. Tang, Y . Ye, J. Huang, J. Zhang, Y . Pang, P. Jin, M. Ning, J. Luo, and L. Yuan, “MoE-LLaV A: Mixture of experts for large vision- language models,”IEEE Trans. Multimedia, vol. 28, pp. 4408–4419, 2026

    2026

  2. [9]

    Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalezet al., “Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality,” https://lmsys. org/blog/2023-03-30-vicuna/, 2023, accessed: May 29, 2026

    2023

  3. [10]

    On evaluating adversarial robustness of large vision-language models,

    Y . Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, “On evaluating adversarial robustness of large vision-language models,” inAdv. Neural Inf. Process. Syst., vol. 36, 2023, pp. 54 111–54 138

    2023

  4. [11]

    On the adversarial robustness of multi- modal foundation models,

    C. Schlarmann and M. Hein, “On the adversarial robustness of multi- modal foundation models,” inProc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023, pp. 3677–3685

    2023

  5. [12]

    Are aligned neural networks adversarially aligned?

    N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, P. W. W. Koh, D. Ippolito, F. Tramer, and L. Schmidt, “Are aligned neural networks adversarially aligned?” inAdv. Neural Inf. Process. Syst., vol. 36, 2023, pp. 61 478–61 500

    2023

  6. [13]

    How robust is Google’s Bard to adversarial image attacks?

    Y . Dong, H. Chen, J. Chen, Z. Fang, X. Yang, Y . Zhang, Y . Tian, H. Su, and J. Zhu, “How robust is Google’s Bard to adversarial image attacks?” arXiv preprint arXiv:2309.11751, 2023

    2023 arXiv

  7. [14]

    Towards evaluating the robustness of neural networks,

    N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” inProc. IEEE Symp. Security Privacy (SP), 2017, pp. 39–57. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10

    2017

  8. [15]

    Explaining and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” inProc. Int. Conf. Learn. Represent. (ICLR), 2015

    2015

  9. [16]

    Towards deep learning models resistant to adversarial attacks,

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” inProc. Int. Conf. Learn. Represent. (ICLR), 2018

    2018

  10. [17]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  11. [18]

    Understanding zero-shot adversarial robustness for large-scale models,

    C. Mao, S. Geng, J. Yang, X. Wang, and C. V ondrick, “Understanding zero-shot adversarial robustness for large-scale models,” inProc. Int. Conf. Learn. Represent. (ICLR), 2023

    2023

  12. [19]

    Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models,

    C. Schlarmann, N. D. Singh, F. Croce, and M. Hein, “Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models,” inProc. Int. Conf. Mach. Learn. (ICML), 2024, pp. 43 685–43 704

    2024

  13. [20]

    Bag of tricks for adversarial training,

    T. Pang, X. Yang, Y . Dong, H. Su, and J. Zhu, “Bag of tricks for adversarial training,” inProc. Int. Conf. Learn. Represent. (ICLR), 2021

    2021

  14. [21]

    Adversarial training for free!

    A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!” inAdv. Neural Inf. Process. Syst., vol. 32, 2019

    2019

  15. [22]

    Theoretically principled trade-off between robustness and accuracy,

    H. Zhang, Y . Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan, “Theoretically principled trade-off between robustness and accuracy,” inProc. Int. Conf. Mach. Learn. (ICML), 2019, pp. 7472–7482

    2019

  16. [23]

    Pre-trained model guided fine-tuning for zero-shot adversarial robustness,

    S. Wang, J. Zhang, Z. Yuan, and S. Shan, “Pre-trained model guided fine-tuning for zero-shot adversarial robustness,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 24 502–24 511

    2024

  17. [24]

    ImageNet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2009, pp. 248–255

    2009

  18. [25]

    Towards adversarial attack on vision- language pre-training models,

    J. Zhang, Q. Yi, and J. Sang, “Towards adversarial attack on vision- language pre-training models,” inProc. ACM Int. Conf. Multimedia, 2022, pp. 5005–5013

    2022

  19. [26]

    Exploring transferability of multimodal adversarial samples for vision-language pre-training models with contrastive learning,

    Y . Wang, W. Hu, Y . Dong, H. Zhang, H. Su, and R. Hong, “Exploring transferability of multimodal adversarial samples for vision-language pre-training models with contrastive learning,”IEEE Trans. Multimedia, vol. 27, pp. 6410–6421, 2025

    2025

  20. [27]

    Enhancing descriptive captions with visual attributes for multimodal perception,

    Y . Sun, J. Hao, K. Zhu, J.-J. Liu, Y . Zhao, X. Li, N. Zhao, Z. Li, and J. Wang, “Enhancing descriptive captions with visual attributes for multimodal perception,”arXiv preprint arXiv:2412.14233, 2024

    2024

  21. [28]

    Prefix conditioning unifies language and label supervision,

    K. Saito, K. Sohn, X. Zhang, C.-L. Li, C.-Y . Lee, K. Saenko, and T. Pfister, “Prefix conditioning unifies language and label supervision,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2023, pp. 2861–2870

    2023

  22. [30]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inProc. Int. Conf. Mach. Learn. (ICML), 2023, pp. 19 730– 19 742

    2023

  23. [31]

    Otter: A multi-modal model with in-context instruction tuning,

    B. Li, Y . Zhang, L. Chen, J. Wang, F. Pu, J. A. Cahyono, J. Yang, C. Li, and Z. Liu, “Otter: A multi-modal model with in-context instruction tuning,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 47, no. 9, pp. 7543–7557, 2025

    2025

  24. [32]

    InternLM-XComposer: A vision- language large model for advanced text-image comprehension and composition,

    P. Zhang, X. Dong, B. Wang, Y . Cao, C. Xu, L. Ouyang, Z. Zhao, H. Duan, S. Zhang, S. Dinget al., “InternLM-XComposer: A vision- language large model for advanced text-image comprehension and composition,”arXiv preprint arXiv:2309.15112, 2023

    2023 arXiv

  25. [33]

    Gemini: A family of highly capable multimodal models,

    Gemini Team, “Gemini: A family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  26. [34]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  27. [35]

    Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,

    A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” inProc. Int. Conf. Mach. Learn. (ICML), 2018, pp. 274–283

    2018

  28. [36]

    Boosting adversarial attacks with momentum,

    Y . Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, pp. 9185–9193

    2018

  29. [37]

    Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,

    F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” inProc. Int. Conf. Mach. Learn. (ICML), 2020, pp. 2206–2216

    2020

  30. [38]

    Adversarial weight perturbation helps robust generalization,

    D. Wu, S.-T. Xia, and Y . Wang, “Adversarial weight perturbation helps robust generalization,” inAdv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 2958–2969

    2020

  31. [39]

    Adversarial attacks on foundational vision models,

    N. Inkawhich, G. McDonald, and R. Luley, “Adversarial attacks on foundational vision models,”arXiv preprint arXiv:2308.14597, 2023

    2023 arXiv

  32. [40]

    Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models,

    D. Lu, Z. Wang, T. Wang, W. Guan, H. Gao, and F. Zheng, “Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models,” inProc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2023, pp. 102–111

    2023

  33. [41]

    Revisiting the ad- versarial transferability:towards a perspective of semantic preservation,

    Z. Wang, C. Jiang, J. Gui, X. Shi, and Y . Y . Tang, “Revisiting the ad- versarial transferability:towards a perspective of semantic preservation,” IEEE Trans. Multimedia, pp. 1–15, 2026

    2026

  34. [42]

    Microsoft COCO: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” inProc. Eur. Conf. Comput. Vis. (ECCV), 2014, pp. 740–755

    2014

  35. [43]

    Caltech-256 object category dataset,

    G. Griffin, A. Holub, and P. Perona, “Caltech-256 object category dataset,” California Inst. Technol., Tech. Rep. 7694, 2007

    2007

  36. [44]

    3D object representations for fine-grained categorization,

    J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3D object representations for fine-grained categorization,” inProc. IEEE Int. Conf. Comput. Vis. Workshops (ICCVW), 2013, pp. 554–561

    2013

  37. [45]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Univ. Toronto, Tech. Rep., 2009

    2009

  38. [46]

    Describing textures in the wild,

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2014, pp. 3606–3613

    2014

  39. [47]

    EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification,

    P. Helber, B. Bischke, A. Dengel, and D. Borth, “EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification,”IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 12, no. 7, pp. 2217–2226, 2019

    2019

  40. [48]

    Fine- grained visual classification of aircraft,

    S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine- grained visual classification of aircraft,”arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  41. [49]

    Automated flower classification over a large number of classes,

    M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” inProc. Indian Conf. Comput. Vis., Graph. Image Process., 2008, pp. 722–729

    2008

  42. [50]

    Cats and dogs,

    O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V . Jawahar, “Cats and dogs,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2012, pp. 3498–3505

    2012

  43. [51]

    An analysis of single-layer networks in unsupervised feature learning,

    A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” inProc. Int. Conf. Artif. Intell. Statist. (AISTATS), 2011, pp. 215–223

    2011

  44. [52]

    The many faces of robustness: A critical analysis of out-of-distribution generalization,

    D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guoet al., “The many faces of robustness: A critical analysis of out-of-distribution generalization,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2021, pp. 8340–8349

    2021

  45. [53]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hocken- maier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” inProc. IEEE Int. Conf. Comput. Vis. (ICCV), 2015, pp. 2641–2649

    2015

  46. [54]

    Making the V in VQA matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 6904–6913

    2017

  47. [55]

    Towards VQA models that can read,

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards VQA models that can read,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 8317–8326

    2019

  48. [56]

    VizWiz grand challenge: Answering visual questions from blind people,

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, “VizWiz grand challenge: Answering visual questions from blind people,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2018, pp. 3608–3617

    2018

  49. [57]

    OK-VQA: A visual question answering benchmark requiring external knowledge,

    K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “OK-VQA: A visual question answering benchmark requiring external knowledge,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 3195–3204

    2019

  50. [58]

    CIDEr: Consensus-based image description evaluation,

    R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015, pp. 4566–4575

    2015

  51. [59]

    LAION-5B: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsmanet al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” inAdv. Neural Inf. Process. Syst., vol. 35, 2022, pp. 25 278–25 294

    2022

  52. [60]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” inProc. Int. Conf. Learn. Represent. (ICLR), 2019

    2019

  53. [61]

    OpenFlamingo: An open-source framework for training large autoregressive vision-language models,

    A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y . Hanafy, W. Zhu, K. Marathe, Y . Bitton, S. Gadre, S. Sagawaet al., “OpenFlamingo: An open-source framework for training large autoregressive vision-language models,”arXiv preprint arXiv:2308.01390, 2023. JOURNAL OF LATEX CLASS FI...

    2023 arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.