Pith. sign in

REVIEW 9 cited by

Failures to Find Transferable Image Jailbreaks Between Vision-Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.15211 v2 pith:RDTP3Q2I submitted 2024-07-21 cs.CL cs.AIcs.CRcs.CVcs.LG

Rylan Schaeffer , Dan Valentine , Luke Bailey , James Chua , Cristóbal Eyzaguirre , Zane Durante , Joe Benton , Brando Miranda
show 7 more authors
Henry Sleight John Hughes Rajashree Agrawal Mrinank Sharma Scott Emmons Sanmi Koyejo Ethan Perez
This is my paper · ORCID
classification cs.CLcs.AIcs.CRcs.CVcs.LG
keywords vlmsimagejailbreakstransfermodelstransferablegradient-basedlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The integration of new modalities into frontier AI systems offers exciting capabilities, but also increases the possibility such systems can be adversarially manipulated in undesirable ways. In this work, we focus on a popular class of vision-language models (VLMs) that generate text outputs conditioned on visual and textual inputs. We conducted a large-scale empirical study to assess the transferability of gradient-based universal image ``jailbreaks" using a diverse set of over 40 open-parameter VLMs, including 18 new VLMs that we publicly release. Overall, we find that transferable gradient-based image jailbreaks are extremely difficult to obtain. When an image jailbreak is optimized against a single VLM or against an ensemble of VLMs, the jailbreak successfully jailbreaks the attacked VLM(s), but exhibits little-to-no transfer to any other VLMs; transfer is not affected by whether the attacked and target VLMs possess matching vision backbones or language models, whether the language model underwent instruction-following and/or safety-alignment training, or many other factors. Only two settings display partially successful transfer: between identically-pretrained and identically-initialized VLMs with slightly different VLM training data, and between different training checkpoints of a single VLM. Leveraging these results, we then demonstrate that transfer can be significantly improved against a specific target VLM by attacking larger ensembles of ``highly-similar" VLMs. These results stand in stark contrast to existing evidence of universal and transferable text jailbreaks against language models and transferable adversarial attacks against image classifiers, suggesting that VLMs may be more robust to gradient-based transfer attacks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evading Chain-of-Thought Monitoring Through Model Poisoning

    cs.CR 2026-08 conditional novelty 8.0 of 10

    Supervised fine-tuning can implant triggered backdoors whose chain-of-thought traces look clean, evading CoT-only monitors and forcing a consistency-check view of monitoring.

  2. Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks

    cs.CV 2024-11 conditional novelty 7.0 of 10

    ASTRA reduces VLM jailbreak success by adaptively steering activations away from a harm direction learned via image attribution, with little utility loss and near-zero inference overhead.

  3. Visual Token Compression Enhances Robustness of MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Pruning visual tokens farthest from the text feature space at selected 'robust' layers improves MLLM jailbreak defense (average +13.29% RAR) and slightly reduces hallucination.

  4. CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents

    cs.AI 2026-01 conditional novelty 6.0 of 10

    Computer-use agents can run under Dual-LLM isolation with single-shot branching plans, preserving partial utility while blocking instruction injection, but remain open to branch-steering attacks.

  5. VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A single adversarially optimized image can reproduce activation-steering behavior in multiple VLMs and partially transfer to unseen models.

  6. "I am bad": Interpreting Stealthy, Universal and Robust Audio Jailbreaks in Audio-Language Models

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Adversarial audio clips optimized on one model can jailbreak SALMONN-7B across prompts, tasks, and base audio files, and the model often decodes them as toxic first-person speech.

  7. Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Adversarial training at both the CLIP pre-training stage and the LLaVA instruction-tuning stage produces vision-language models with state-of-the-art robustness and near-baseline clean performance.

  8. Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approach

    cs.LG 2024-12 conditional novelty 6.0 of 10

    No tested defense, including the paper's own transcript classifier, can fully stop an LLM from giving competent bomb-making instructions under a grey-box attacker.

  9. ReVision : A Post-Hoc, Vision-Based Technique for Replacing Unacceptable Concepts in Image Generation Pipeline

    cs.CR 2026-02 conditional novelty 4.0 of 10

    ReVision uses a vision-language model's bounding box to gate attention-based image editing, suppressing unsafe concepts while better preserving benign background in multi-concept scenes.

Pith tools