Pith. sign in

REVIEW 3 major objections 5 minor 50 references

One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single perturbation flips one object's identity across four vision-language tasks.

desk verdict The benchmark is genuinely new and the attack method makes sense, but the detection success criterion counts adding a target detection as changing the object's identity, which inflates CTSR-4. read the letter →

arxiv 2507.07709 v1 pith:JSU5BVGM submitted 2025-07-10 cs.CV

Jiale Zhao , Xinyang Jiang , Junyao Gao , Yuhao Xue , Cairong Zhao This is my paper
classification cs.CV
keywords cross-taskadversarialattackunifiedvision-languagemodelsobject-changeCrossVLADbenchmarktokenalignmentcontrastivelossCTSRmetricregion-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that unified vision-language models can be deceived by a single adversarial image perturbation that changes a target object's category at once in image captioning, object detection, region classification, and object localization. To test this, it builds CrossVLAD, a 3,000-image benchmark from MSCOCO with GPT-4-assisted annotations covering 79 source-to-target category pairs, and defines CTSR-4 and CTSR-3 metrics that require simultaneous success on all four tasks or at least three. The proposed method, CRAFT, localizes the object's token region in the image encoder and applies a contrastive alignment loss that pulls those features toward the target category's text embedding and away from source and distractor categories. Reported results give CRAFT a CTSR-4 of 0.471 on Florence-2 versus 0.264 for the best adapted baseline, with consistent advantages on UnifiedIO-2 and OFA. The paper concludes that instruction-controlled unified models share a representation space that current single-task adversarial evaluations fail to expose as vulnerable.

What carries the argument

The load-bearing mechanism is CRAFT's region token localization combined with cross-modal contrastive alignment. Given the source object's bounding box, the image is divided into a patch-token grid and Eqs. (8)–(9) map the box to the token set $R$; the contrastive loss of Eq. (10), $\mathcal{L}_{\text{contrast}} = \max(0, \mathrm{sim}(F_R, E_{\text{neg}}) - \mathrm{sim}(F_R, E_{\text{pos}}) + \tau)$, pulls the region's image-encoder features $F_R$ toward the target category's text embedding $E_{\text{pos}}$ and away from the source and other category embeddings $E_{\text{neg}}$. This loss is optimized with projected gradient descent under an $\ell_\infty$ budget (Eq. (11)). The paper's hypothesis is that because all four task heads read from the same unified representation, aligning the region tokens in that space is sufficient to shift every decoded output.

What would settle it

Run CRAFT on a unified VLM, then take a successful adversarial image and feed it to the four decoders while replacing the adversarial region tokens with the clean image's region tokens at the feature level; if any decoder still outputs the source category rather than the target, then those aligned tokens are not the whole story. Alternatively, collect examples with high feature alignment (low contrastive loss) but failure on at least one task; if such examples are non-negligible, the proxy-to-output transfer is broken.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that object identity in a unified VLM lives in a cross-modal feature space shared by all task decoders, so steering the region tokens' features toward the target category's text embedding is enough to change the decoded output across captioning, detection, region classification, and localization simultaneously. The benchmark shows the resulting attacks are substantially more effective than adapted single-task attack methods: on Florence-2, CRAFT reaches 0.471 CTSR-4 and 0.609 CTSR-3, compared with 0.264 CTSR-4 for the best adapted baseline (MF-ii), and similar advantages hold on UnifiedIO-2 (0.485 vs. 0.098) and OFA (0.286 vs. 0.087). The paper also reports that attacks optimized for the region-categorization task transfer best to other tasks, and that within-semantic-category transformations are easier than cross-category ones, with some surprising cross-category successes attributed to visual or contextual similarity.

Load-bearing premise

That aligning the object's visual tokens with the target category's text embedding in the shared feature space is sufficient to shift the outputs of all four task decoders — the contrastive loss is optimized on features while success is judged on decoded text and boxes, and the link between the two is supported only empirically.

Editorial extensions

If this is right

  • If the shared-representation hypothesis holds, any single heading's robustness fix can be bypassed through the same perturbation, so defenses must protect the joint feature space or the region-token geometry rather than individual decoders.
  • Current single-task evaluation underestimates the risk to deployed unified models: a perturbation judged 'failed' on captioning alone can still be a fully successful object-change attack when all four tasks are considered together.
  • The benchmark's 79 change-pairs define a difficulty ordering for object replacement, giving robustness researchers a concrete testbed for semantic-category-aware defenses.
  • The small cat-to-dog transfer test to a commercial VLM indicates that black-box closed models may inherit the same vulnerability if they share a similar unified representation space.
  • Region-focused perturbation is more efficient and more transferable than full-image noise, so future attacks are likely to be region-centric as well.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the feature-proxy link is real, then a defense that randomizes the token-to-text alignment (e.g., perturbing the text embeddings per task) could break cross-task transfer without hurting per-task accuracy; a cheap experiment is to re-run CRAFT with the target text embedding shifted by a random rotation.
  • Beyond the paper: the benchmark's success criteria check only that the caption mentions the target and omits the source; a stricter, more realistic criterion would require the entire caption to remain coherent with the scene, which could lower CTSR-4 and change the ranking of methods.
  • Beyond the paper: CRAFT's white-box dependence on the model's own image and text encoders means its transfer value across model families is untested; extending it with an ensemble of encoders would show whether the aligned feature geometry generalizes across architectures.
  • Beyond the paper: a direct test of the load-bearing premise is to ablate the region tokens after the attack — feed the attacked image but replace the region tokens with the original clean tokens; if the decoders then revert to the source category, the attack is exactly the token shift the paper posits.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CrossVLAD, a benchmark of 3,000 MSCOCO images with 79 source-to-target category change-pairs, together with cross-task metrics CTSR-4 and CTSR-3 that measure whether a single adversarial perturbation simultaneously flips an object's identity across image captioning, object detection, region categorization, and object localization. The authors also propose CRAFT, a PGD-based region-centric attack that aligns image-encoder region tokens with the target category's text embedding and pushes them away from negative category embeddings. Experiments on Florence-2, UnifiedIO-2, and OFA report that CRAFT outperforms adapted baselines on CTSR-4/CTSR-3, with ablations on region token localization and negative-text selection, plus qualitative examples and a small black-box transfer test to GPT-4V.

Significance. The benchmark addresses a real gap: prior evaluations either report per-task success rates or generate separate adversarial examples per task, whereas CrossVLAD asks whether one perturbation can simultaneously mislead a unified VLM across four tasks. The CRAFT method is simple, well-motivated, and the paper includes useful ablations, qualitative visualizations, and a black-box transfer sanity check. If the detection success criterion is tightened and the quantitative claims are made robust with variance estimates, the paper would provide a valuable benchmark and baseline for cross-task adversarial robustness in unified VLMs.

major comments (3)
  1. [§3.3.1, Eq. (3)] Sdet as written requires only that a target-category detection appears at the source box; it never requires the source category to disappear from that location. In Florence-2, OFA, and UnifiedIO-2, detection is a multi-output sequence task, so the model can output both source and target category boxes overlapping the same region, and the sample is scored as a successful object-change even though the source identity was not flipped. Because CTSR-4 and CTSR-3 are the headline metrics for the paper's central claim, this definitional looseness can inflate the reported 0.471 CTSR-4 and affect the comparison with baselines. Please redefine Sdet to require both the presence of a target-category detection at the source box and the absence of a source-category detection whose IoU with the source box exceeds θbox, and rerun the experiments.
  2. [§5.1, Tables 1–3 and Fig. 5] All quantitative results are single-run point estimates with no standard deviations, confidence intervals, or significance tests. The central superiority claim (e.g., CTSR-4 of 0.471 versus 0.264 for the best baseline on Florence-2) cannot be assessed for robustness without repeated runs or bootstrap intervals. Please report mean ± std over at least three independent runs (or equivalent) for the main tables and the ablation, and state the number of seeds used.
  3. [§5.1 and App. 9.1] The baseline comparison is information-asymmetric. CRAFT is given the source bounding box and uses it for region-token localization, whereas the adapted baselines receive only image-level target captions and no region-level supervision. Since the paper's own ablation shows that region token localization is a major contributor to performance (Table 3: CTSR-4 rises from 0.17 to 0.46 when RTL is added with no negatives), the reported gains may partly reflect this additional supervision rather than the proposed alignment objective. Please adapt the baselines with comparable region knowledge (e.g., restrict their optimization to the source box or provide the box to their loss), or explicitly discuss this confound and its effect on the comparison.
minor comments (5)
  1. [§3.2 and App. 7.2] The main text says the caption-verification criterion is 'object mentioned of 5 captions' while the appendix specifies 'at least 3 of the 5'; please make the criterion explicit and consistent.
  2. [§5.1] The exact task prompts or instructions used for each of the four tasks on each model are not provided; please include them in the appendix for reproducibility, since task phrasing can materially affect unified VLM outputs.
  3. [§3.3.1] The IoU thresholds θbox and θloc are fixed at 0.6 without a sensitivity analysis; please report results for at least one alternative threshold (e.g., 0.5) to show the metrics are stable.
  4. [Fig. 5] The legend entries combine ϵ and α in a way that is hard to read; consider listing ϵ only and specifying the corresponding α separately in the caption or text.
  5. [§4.3] The paper would be strengthened by a short discussion of why contrastive alignment of image-encoder token features is expected to transfer to the autoregressive decoder outputs that define the task success criteria; currently this transfer is supported only empirically.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: CrossVLAD and CRAFT are self-contained; CRAFT's success is measured empirically against defined metrics rather than derived from its own optimization objective.

full rationale

The derivation chain is transparent and non-circular. The benchmark defines task-specific success criteria (Eqs. 2-5), the attack objective (Eq. 1), the CRAFT token-region localization (Eqs. 8-9), contrastive alignment loss (Eq. 10), PGD update (Eq. 11), and the aggregate metrics CTSR-4/CTSR-3 (Eqs. 6-7). CRAFT optimizes the feature-level contrastive objective in Eq. 10, whereas the reported success is evaluated on decoded captions, detections, region labels, and boxes via Eqs. 2-5; there is no identity between the optimized loss and the evaluation metric, so the reported 0.471 CTSR-4 on Florence-2 is an empirical result, not a tautology. Hyperparameters (epsilon = 16/255, 100 iterations, alpha = 4/255, margin tau = 0.9) are selected and reported via ablations in the same paper (Section 5.5, Figure 5); this is ordinary hyperparameter tuning and is not presented as an independent prediction, so it does not constitute fitted-input-called-prediction. The paper contains self-citations by co-authors ([8], [9], [10]) in the introduction and related work, but none is load-bearing: no method component, benchmark construction step, or metric definition depends on these works. The supplementary section 9.4 explicitly labels the black-box GPT-4V test as 'preliminary evaluation' and 'limited scale,' which is an honest limitation rather than a circular justification. One non-circular validity concern: Eq. (3) defines detection success by the existence of a target-category detection overlapping the source box and never requires the source category to disappear, so CTSR-4 could reward adding a detection rather than flipping identity; this is a metric-validity limitation that may affect the interpretation of the reported advantage, but it is not a circularity in the derivation sense because the success metric is not claimed to be equivalent to the optimization objective.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central quantitative results are empirical; the paper contributes no new physical or mathematical entities. The attack and evaluation rest on a handful of hand-chosen hyperparameters, including epsilon, step size, margin, and IoU thresholds, and on domain assumptions about unified representation transfer and annotation quality. The benchmark itself is a curated dataset, not a falsifiable entity in the sense of this ledger.

free parameters (5)
  • epsilon (L-infinity perturbation budget) = 16/255
    Hand-chosen after ablations (Figure 5) balancing imperceptibility and success; all main results use this budget.
  • alpha (PGD step size) = 4/255
    Set proportional to epsilon; used in Eq. (11) for gradient updates.
  • tau (contrastive margin) = 0.9
    Margin in Eq. (10); chosen by hand without a sensitivity analysis.
  • theta_box and theta_loc (IoU thresholds) = 0.6
    Evaluation thresholds for detection and localization success (Eqs. (3) and (5)); directly determine CTSR-4 and CTSR-3 values.
  • Number of PGD iterations = 100
    Selected as a balance of efficiency and performance (Figure 5); performance varies noticeably across iteration counts.
assumptions (4)
  • domain assumption Unified VLMs share a common representation space such that manipulating image-encoder features transfers to all downstream task outputs.
    Invoked in Section 4.3 ('directly manipulates the unified feature representations that underlie all tasks'); the entire cross-task attack depends on this transfer, which is not proven beyond the reported experiments.
  • domain assumption MSCOCO ground-truth bounding boxes and category labels reliably define the source object for attack and evaluation.
    Used throughout Section 3.2 and 3.3 to compute success for detection and localization; any noise in boxes shifts both attack and evaluation.
  • domain assumption GPT-4-generated target captions are valid descriptions of the scene with the object replaced.
    Section 7.3 only verifies that captions mention the target and exclude the source; no human or automated semantic validation of scene coherence is reported.
  • domain assumption The fixed task prompts used for attack and evaluation represent the tasks faithfully; results may not hold under different instruction phrasings.
    Prompt templates are not varied or ablated in the paper; instruction sensitivity is a known property of instruction-tuned VLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models." pith.science (2026). https://pith.science/paper/JSU5BVGM

@misc{pith2026250707709,
  author       = {Pith},
  title        = {Pith review of: One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JSU5BVGM}},
  note         = {Machine review of arXiv:2507.07709}
}
read the original abstract

Unified vision-language models(VLMs) have recently shown remarkable progress, enabling a single model to flexibly address diverse tasks through different instructions within a shared computational architecture. This instruction-based control mechanism creates unique security challenges, as adversarial inputs must remain effective across multiple task instructions that may be unpredictably applied to process the same malicious content. In this paper, we introduce CrossVLAD, a new benchmark dataset carefully curated from MSCOCO with GPT-4-assisted annotations for systematically evaluating cross-task adversarial attacks on unified VLMs. CrossVLAD centers on the object-change objective-consistently manipulating a target object's classification across four downstream tasks-and proposes a novel success rate metric that measures simultaneous misclassification across all tasks, providing a rigorous evaluation of adversarial transferability. To tackle this challenge, we present CRAFT (Cross-task Region-based Attack Framework with Token-alignment), an efficient region-centric attack method. Extensive experiments on Florence-2 and other popular unified VLMs demonstrate that our method outperforms existing approaches in both overall cross-task attack performance and targeted object-change success rates, highlighting its effectiveness in adversarially influencing unified VLMs across diverse tasks.

Figures

Figures reproduced from arXiv: 2507.07709 by the authors.

Figure 1
Figure 1. An overview of our benchmark and method. Left side illustrates the CRAFT method (top) and our object-change attack on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Qualitative examples of CRAFT attack on Florence-2 model across four vision tasks: object detection, object localization, image [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Heatmap visualization of CTSR-4 success rates for se [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visualization of adversarial perturbations (amplified 10x for visibility) generated by different attack methods. CRAFT produces [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: illustrates the effect of perturbation budget (ϵ) and iteration count on CTSR-4 performance. We observe that ϵ = 16/255 offers a good balance between effective￾ness and visual imperceptibility, while 100 iterations strikes an optimal balance between computational effic…
Figure 6
Figure 6. Figure 6: shows the impact of perturbation budget (ϵ) and it￾eration count on CTSR-3 performance. The trends broadly mirror those observed for CTSR-4 in Section 5.5, but with higher overall success rates as expected from the more le￾nient evaluation criterion. The CTSR-3 results…
Figure 7
Figure 7. Figure 7: Qualitative examples of CRAFT attack on Florence-2 model across four vision tasks: object detection, object localization, image [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Failed examples of CRAFT attack on Florence-2 model across four vision tasks: object detection, object localization, image [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Successful cat-to-dog attacks transferring to GPT-4V. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 33 canonical work pages

  1. [1]

    Image hijacks: Adversarial images can control generative models at runtime

    Luke Bailey, Euan Ong, Stuart Russell, and Scott Emmons. Image hijacks: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236, 2023. 3

    arXiv 2023

  2. [2]

    Context-aware transfer attacks for ob- ject detection

    Zikui Cai, Xinxin Xie, Shasha Li, Mingjun Yin, Chengyu Song, Srikanth V Krishnamurthy, Amit K Roy-Chowdhury, and M Salman Asif. Context-aware transfer attacks for ob- ject detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 149–157, 2022. 5

    work page 2022

  3. [3]

    Attentional feature erase: Towards task-wise transferable ad- versarial attack on cloud vision apis

    Bo Cheng, Yantao Lu, Yilan Li, Tao You, and Peng Zhang. Attentional feature erase: Towards task-wise transferable ad- versarial attack on cloud vision apis. Displays, 82:102634,

    work page

  4. [4]

    Unihcp: A unified model for human-centric perceptions

    Yuanzheng Ci, Yizhou Wang, Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang. Unihcp: A unified model for human-centric perceptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17840– 17852, 2023. 2, 3

    work page 2023

  5. [5]

    On the robustness of large multimodal mod- els against image adversarial attacks

    Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal mod- els against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24625–24634, 2024. 2, 3

    work page 2024

  6. [6]

    How robust is google’s bard to adversarial image at- tacks? arXiv preprint arXiv:2309.11751, 2023

    Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google’s bard to adversarial image at- tacks? arXiv preprint arXiv:2309.11751, 2023. 3, 5, 7, 1

    arXiv 2023

  7. [7]

    Enhancing cross-task transferability of adversarial examples via spatial and channel attention

    Weiwei Feng, Nanqing Xu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. Enhancing cross-task transferability of adversarial examples via spatial and channel attention. IEEE Transactions on Multimedia, 2024. 2, 3

    work page 2024

  8. [8]

    Similarity distribution based member- ship inference attack on person re-identification

    Junyao Gao, Xinyang Jiang, Huishuai Zhang, Yifan Yang, Shuguang Dou, Dongsheng Li, Duoqian Miao, Cheng Deng, and Cairong Zhao. Similarity distribution based member- ship inference attack on person re-identification. InProceed- ings of the AAAI conference on artificial intelligence , pages 14820–14828, 2023. 2

    work page 2023

Show all 50 references
  1. [9]

    Styleshot: A snap- shot on any style

    Junyao Gao, Yanchen Liu, Yanan Sun, Yinhao Tang, Yan- hong Zeng, Kai Chen, and Cairong Zhao. Styleshot: A snap- shot on any style. arXiv preprint arXiv:2407.01414, 2024. 2

    2024 arXiv

  2. [10]

    Faceshot: Bring any character into life

    Junyao Gao, Yanan Sun, Fei Shen, Xin Jiang, Zhening Xing, Kai Chen, and Cairong Zhao. Faceshot: Bring any character into life. arXiv preprint arXiv:2503.00740, 2025. 2

    2025 arXiv

  3. [11]

    Ot-attack: Enhancing adversarial transferability of vision-language models via optimal trans- port optimization

    Dongchen Han, Xiaojun Jia, Yang Bai, Jindong Gu, Yang Liu, and Xiaochun Cao. Ot-attack: Enhancing adversarial transferability of vision-language models via optimal trans- port optimization. arXiv preprint arXiv:2312.04403, 2023. 2

    2023 arXiv

  4. [12]

    Instruct-reid: A multi-purpose person re-identification task with instructions

    Weizhen He, Yiheng Deng, Shixiang Tang, Qihao Chen, Qingsong Xie, Yizhou Wang, Lei Bai, Feng Zhu, Rui Zhao, Wanli Ouyang, et al. Instruct-reid: A multi-purpose person re-identification task with instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  5. [13]

    As firm as their foundations: Can open-sourced foundation models be used to create adver- sarial examples for downstream tasks? arXiv preprint arXiv:2403.12693, 2024

    Anjun Hu, Jindong Gu, Francesco Pinto, Konstantinos Kam- nitsas, and Philip Torr. As firm as their foundations: Can open-sourced foundation models be used to create adver- sarial examples for downstream tasks? arXiv preprint arXiv:2403.12693, 2024. 2, 3

    2024 arXiv

  6. [14]

    Vlm-rl: A unified vision language models and reinforcement learning framework for safe autonomous driv- ing

    Zilin Huang, Zihao Sheng, Yansong Qu, Junwei You, and Sikai Chen. Vlm-rl: A unified vision language models and reinforcement learning framework for safe autonomous driv- ing. arXiv preprint arXiv:2412.15544, 2024. 2, 3

    2024 arXiv

  7. [15]

    You only learn one query: learning unified human query for single-stage multi-person multi-task human-centric perception

    Sheng Jin, Shuhuai Li, Tong Li, Wentao Liu, Chen Qian, and Ping Luo. You only learn one query: learning unified human query for single-stage multi-person multi-task human-centric perception. In European Conference on Computer Vision , pages 126–146. Springer, 2024. 2

    2024

  8. [16]

    Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks

    Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. In Proceed- ings of the IEEE/CVF Conference on Computer Visio...

    2023

  9. [17]

    Lawrence Zitnick, and Piotr Doll ´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll ´ar. Microsoft coco: Common objects in context, 2015. 3

    2015

  10. [18]

    A survey of attacks on large vision- language models: Resources, advances, and future trends

    Daizong Liu, Mingyu Yang, Xiaoye Qu, Pan Zhou, Yu Cheng, and Wei Hu. A survey of attacks on large vision- language models: Resources, advances, and future trends. arXiv preprint arXiv:2407.07403, 2024. 3

    2024 arXiv

  11. [19]

    Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models

    Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, and Feng Zheng. Set-level guidance at- tack: Boosting adversarial transferability of vision-language pre-training models. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 102–111,

  12. [20]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representa- tions, 2022. 2, 3

    2022

  13. [21]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceed- ings of the IEEE/CVF Conference on Computer Vision ...

    2024

  14. [22]

    Time-aware and task-transferable adversarial attack for perception of autonomous vehicles.Pattern Recog- nition Letters, 178:145–152, 2024

    Yantao Lu, Haining Ren, Weiheng Chai, Senem Velipasalar, and Yilan Li. Time-aware and task-transferable adversarial attack for perception of autonomous vehicles.Pattern Recog- nition Letters, 178:145–152, 2024. 3

    2024

  15. [23]

    An image is worth 1000 lies: Adversarial transferability across prompts on vision-language models

    Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image is worth 1000 lies: Adversarial transferability across prompts on vision-language models. arXiv preprint arXiv:2403.09766, 2024. 3

    2024 arXiv

  16. [24]

    Ct-gat: Cross-task generative adversarial attack based on transferability

    Minxuan Lv, Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. Ct-gat: Cross-task generative adversarial attack based on transferability. arXiv preprint arXiv:2310.14265, 2023. 3

    2023 arXiv

  17. [25]

    Boosting cross-task transferability of adversarial patches 9 with visual relations

    Tony Ma, Songze Li, Yisong Xiao, and Shunchang Liu. Boosting cross-task transferability of adversarial patches 9 with visual relations. arXiv preprint arXiv:2304.05402 ,

    arXiv

  18. [26]

    Towards deep learn- ing models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learn- ing models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. 5

    2017 arXiv

  19. [27]

    Pick-object-attack: Type-specific adver- sarial attack for object detection

    Omid Mohamad Nezami, Akshay Chaturvedi, Mark Dras, and Utpal Garain. Pick-object-attack: Type-specific adver- sarial attack for object detection. Computer Vision and Im- age Understanding, 211:103257, 2021. 5

    2021

  20. [28]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  21. [29]

    On the adversarial robustness of multi-modal foundation models

    Christian Schlarmann and Matthias Hein. On the adversarial robustness of multi-modal foundation models. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 3677–3685, 2023. 3

    2023

  22. [30]

    Unival: Unified model for image, video, au- dio and language tasks

    Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. Unival: Unified model for image, video, au- dio and language tasks. Transactions on Machine Learning Research Journal, 2023. 2, 3

    2023

  23. [31]

    How many unicorns are in this im- age? a safety evaluation benchmark for vision llms

    Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this im- age? a safety evaluation benchmark for vision llms. arXiv preprint arXiv:2311.16101, 2023. 3, 5, 7, 1, 2

    2023 arXiv

  24. [32]

    Benchmarking zero-shot robustness of multimodal founda- tion models: A pilot study.arXiv preprint arXiv:2403.10499,

    Chenguang Wang, Ruoxi Jia, Xin Liu, and Dawn Song. Benchmarking zero-shot robustness of multimodal founda- tion models: A pilot study.arXiv preprint arXiv:2403.10499,

    arXiv

  25. [33]

    Trans- ferable multimodal attack on vision-language pre-training models

    Haodi Wang, Kai Dong, Zhilei Zhu, Haotong Qin, Aishan Liu, Xiaolin Fang, Jiakai Wang, and Xianglong Liu. Trans- ferable multimodal attack on vision-language pre-training models. In 2024 IEEE Symposium on Security and Privacy (SP), pages 102–102. IEEE Computer Society, 2024. 3

    2024

  26. [34]

    Psat-gan: Efficient adversarial attacks against holistic scene understanding

    Lin Wang and Kuk-Jin Yoon. Psat-gan: Efficient adversarial attacks against holistic scene understanding. IEEE Transac- tions on Image Processing, 30:7541–7553, 2021. 2, 3

    2021

  27. [35]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International conference on machine learn- i...

    2022

  28. [36]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 481...

    2024

  29. [37]

    Highly transferable diffusion- based unrestricted adversarial attack on pre-trained vision- language models

    Wenzhuo Xu, Kai Chen, Ziyi Gao, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Highly transferable diffusion- based unrestricted adversarial attack on pre-trained vision- language models. In Proceedings of the 32nd ACM Interna- tional Conference on Multimedia, pages 748–757, 2...

    2024

  30. [38]

    Cross-task attack: A self-supervision generative framework based on attention shift

    Qingyuan Zeng, Yunpeng Gong, and Min Jiang. Cross-task attack: A self-supervision generative framework based on attention shift. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2024. 2, 3

    2024

  31. [39]

    X 2-vlm: All-in-one pre- trained model for vision-language tasks

    Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou. X 2-vlm: All-in-one pre- trained model for vision-language tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 2, 3

    2023

  32. [40]

    Anyattack: Towards large-scale self-supervised generation of targeted adversar- ial examples for vision-language models

    Jiaming Zhang, Junhong Ye, Xingjun Ma, Yige Li, Yunfan Yang, Jitao Sang, and Dit-Yan Yeung. Anyattack: Towards large-scale self-supervised generation of targeted adversar- ial examples for vision-language models. arXiv preprint arXiv:2410.05346, 2024. 5

    arXiv

  33. [41]

    Boosting cross-task ad- versarial attack with random blur

    Yaoyuan Zhang, Yu-an Tan, Mingfeng Lu, Tian Chen, Yuanzhang Li, and Quanxin Zhang. Boosting cross-task ad- versarial attack with random blur. International journal of intelligent systems, 37(10):8139–8154, 2022. 3

    2022

  34. [42]

    Benchmarking trustworthiness of multi- modal large language models: A comprehensive study.arXiv preprint arXiv:2406.07057, 2024

    Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, et al. Benchmarking trustworthiness of multi- modal large language models: A comprehensive study.arXiv preprint arXiv:2406.07057, 2024. 1

    2024 arXiv

  35. [43]

    On evalu- ating adversarial robustness of large vision-language mod- els

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongx- uan Li, Ngai-Man Man Cheung, and Min Lin. On evalu- ating adversarial robustness of large vision-language mod- els. Advances in Neural Information Processing Systems , 36:54111–54138, 2023. 3, 5, 7, 2

    2023

  36. [44]

    Adversarial attacks on hidden tasks in multi- task learning

    Yu Zhe, Rei Nagaike, Daiki Nishiyama, Kazuto Fukuchi, and Jun Sakuma. Adversarial attacks on hidden tasks in multi- task learning. arXiv preprint arXiv:2405.15244 , 2024. 2, 3 10 One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Languag...

    2024 arXiv

  37. [45]

    [SOURCE_CATEGORY]

    Dataset Construction Details 7.1. Comparison with Other Adversarial Attack Benchmarks Table 4 compares CrossVLAD with existing adversarial attack benchmarks for vision-language models. While some prior works have considered unified VLMs or mul- tiple tasks, CrossVLAD uniquely ...

  38. [46]

    [SOURCE_CATEGORY]

    [CAPTION_5] Task: Imagine replacing the primary object "[SOURCE_CATEGORY]" with a new object "[ TARGET_CATEGORY]". Create a caption describing the scene with this replacement. We implemented quality control by verifying that each generated caption: (1) explicitly mentioned the...

  39. [47]

    The procedure begins by initializing the adversar- ial example and locating the token indices corresponding to the source object region

    Pseudocode Algorithm 1 presents the detailed procedure of our CRAFT method. The procedure begins by initializing the adversar- ial example and locating the token indices corresponding to the source object region. In each iteration, we extract image features from the current ad...

  40. [48]

    Implementation Details of Compared Methods We provide detailed implementation information for all compared methods to ensure reproducibility and fair com- parison

    Experimental Details 9.1. Implementation Details of Compared Methods We provide detailed implementation information for all compared methods to ensure reproducibility and fair com- parison. Attack-Bard We adopt the text description attack from Attack-Bard [6], which maximizes ...

  41. [49]

    0.48 0.75 0.68 0.52 0.38 0.52

  42. [50]

    Comparison with object detection attack baselines on Florence-2

    0.45 0.67 0.64 0.50 0.32 0.48 CRAFT (ours) 0.77 0.57 0.85 0.65 0.47 0.61 Table 7. Comparison with object detection attack baselines on Florence-2. Our method demonstrates superior cross-task trans- ferability. 9.6. Effect of Bounding Box Source In our primary experiments, we a...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.