Pith. sign in

REVIEW 8 cited by

An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.09766 v1 pith:ZTWL4FFA submitted 2024-03-14 cs.CV

Haochen Luo , Jindong Gu , Fengyuan Liu , Philip Torr This is my paper · ORCID
classification cs.CV
keywords adversarialdifferentpromptstransferabilitycropamodelsvlmscross-prompt
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Different from traditional task-specific vision models, recent large VLMs can readily adapt to different vision tasks by simply using different textual instructions, i.e., prompts. However, a well-known concern about traditional task-specific vision models is that they can be misled by imperceptible adversarial perturbations. Furthermore, the concern is exacerbated by the phenomenon that the same adversarial perturbations can fool different task-specific models. Given that VLMs rely on prompts to adapt to different tasks, an intriguing question emerges: Can a single adversarial image mislead all predictions of VLMs when a thousand different prompts are given? This question essentially introduces a novel perspective on adversarial transferability: cross-prompt adversarial transferability. In this work, we propose the Cross-Prompt Attack (CroPA). This proposed method updates the visual adversarial perturbation with learnable prompts, which are designed to counteract the misleading effects of the adversarial image. By doing this, CroPA significantly improves the transferability of adversarial examples across prompts. Extensive experiments are conducted to verify the strong cross-prompt adversarial transferability of CroPA with prevalent VLMs including Flamingo, BLIP-2, and InstructBLIP in various different tasks. Our source code is available at \url{https://github.com/Haochen-Luo/CroPA}.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On Surjectivity of Neural Networks: Can you elicit any behavior from your model?

    cs.LG 2025-08 conditional novelty 7.0 of 10

    Pre-LayerNorm transformers and linear attention are almost always surjective, so any target output has an input that produces it in the continuous embedding space.

  2. On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Aligning adversarial perturbations with the near-null singular directions of intermediate linear layers in transformer VLMs yields stronger attacks than existing feature- and output-space methods.

  3. One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A single adversarial image can make a unified vision-language model misclassify the same object across captioning, detection, region classification, and localization, and the new CrossVLAD benchmark and CRAFT attack m...

  4. Exploiting Vision Encoder Vulnerabilities for Universal Adversarial Perturbations on Large Vision-Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A single universal adversarial perturbation targeting middle-layer value vectors in the vision encoder reliably misleads vision-language models across images and prompts, outperforming output-embedding baselines.

  5. SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A pipeline that plans, places, and renders scene-coherent typographic adversarial text fools vision-language models more often than prior center or margin text attacks, but its success metric and naturalness evaluatio...

  6. Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models

    cs.CR 2024-12 conditional novelty 5.0 of 10

    A black-box jailbreak method distributes harmful semantics across text and image inputs and uses heuristic search to induce multimodal LLMs to answer harmful queries.

  7. A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations

    cs.CR 2025-02 conditional novelty 4.0 of 10

    A survey of LVLM safety that adds a lifecycle taxonomy and new benchmark results showing Janus-Pro-7B has weaker safety than several open-source LVLMs.

  8. When Data Manipulation Meets Attack Goals: An In-depth Survey of Attacks for VLMs

    cs.CV 2025-02 conditional novelty 3.0 of 10

    A survey that classifies VLM attacks by goal and data manipulation strategy, and reviews defenses and metrics.

Pith tools