Pith. sign in

REVIEW 16 cited by

Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.12336 v2 pith:OGFSX3DW submitted 2024-02-19 cs.LG cs.AIcs.CVstat.ML

Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

Christian Schlarmann , Naman Deep Singh , Francesco Croce , Matthias Hein This is my paper
classification cs.LG cs.AIcs.CVstat.ML
keywords modelscliprobustvisionlvlmsadversarialfine-tuninglarge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Multi-modal foundation models like OpenFlamingo, LLaVA, and GPT-4 are increasingly used for various real-world tasks. Prior work has shown that these models are highly vulnerable to adversarial attacks on the vision modality. These attacks can be leveraged to spread fake information or defraud users, and thus pose a significant risk, which makes the robustness of large multi-modal foundation models a pressing problem. The CLIP model, or one of its variants, is used as a frozen vision encoder in many large vision-language models (LVLMs), e.g. LLaVA and OpenFlamingo. We propose an unsupervised adversarial fine-tuning scheme to obtain a robust CLIP vision encoder, which yields robustness on all vision down-stream tasks (LVLMs, zero-shot classification) that rely on CLIP. In particular, we show that stealth-attacks on users of LVLMs by a malicious third party providing manipulated images are no longer possible once one replaces the original CLIP model with our robust one. No retraining or fine-tuning of the down-stream LVLMs is required. The code and robust models are available at https://github.com/chs20/RobustVLM

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Semantic Robustness Certification for Vision-Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    Framework certifies VLM robustness under semantic transformations via text prompt proxies, enabling quantitative certification of safe extent intervals without per-variation data.

  2. Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 7.0

    High-noise feature drift distinguishes adversarial from clean inputs in CLIP, allowing a plug-in gating mechanism to selectively trigger existing test-time defenses and raise mean clean+adversarial accuracy across 13 ...

  3. TARO: Temporal Adversarial Rectification Optimization Using Diffusion Models as Purifiers

    cs.LG 2026-05 unverdicted novelty 7.0

    TARO builds a temporally guided score prior from high-noise and low-noise diffusion views to purify adversarial examples more robustly than uniform timestep methods.

  4. Hierarchically Robust Zero-shot Vision-language Models

    cs.CV 2026-04 unverdicted novelty 7.0

    A hierarchical adversarial fine-tuning method for VLMs aligns image and text embeddings at multiple hierarchy depths with theoretical margin connections to boost robustness to leaf and superclass attacks while using m...

  5. Pay Less Attention to Function Words for Free Robustness of Vision-Language Models

    cs.LG 2025-12 conditional novelty 7.0

    FDA differentially subtracts function-word cross-attention from original attention heads to cut attack success rates by 18-90% across models and tasks while dropping performance by at most 0.6%.

  6. Unifying Adversarially Robust Model Experts in Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0

    CARE collaboratively fine-tunes two CLIP experts (image-text alignment and image-invariance) with embedding harmonization and EMA merging, yielding a single model with better clean and adversarial accuracy than either expert.

  7. Rethinking Brain Decoding with CLIP: The Role of Adversarial Robustness

    cs.CV 2026-07 accept novelty 6.0

    Adversarially robust CLIP targets (FARE, TeCoA) consistently beat standard CLIP on fMRI-image retrieval, zero-shot classification, and alignment metrics when the decoder is held fixed.

  8. Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models

    cs.CL 2026-06 unverdicted novelty 6.0

    Adversarial images transfer across languages in MLLMs while apparent safety in weaker languages stems from comprehension and visual-grounding failures rather than genuine alignment.

  9. Investigating Adversarial Robustness of Multi-modal Large Language Models

    cs.CV 2026-06 unverdicted novelty 6.0

    Robust vision encoders from multimodal adversarial pretraining transfer to MLLMs and deliver large gains in adversarial captioning and VQA performance, while test-time stochastic transformations provide an effective b...

  10. Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models

    cs.CV 2026-05 unverdicted novelty 6.0

    Introduces Closed-Loop Bidirectional Prompting with Semantic Anchor for cross-modal agreement recovery, claiming SOTA adversarial robustness and generalization on 11 datasets.

  11. VisInject: Disruption != Injection -- A Dual-Dimension Evaluation of Universal Adversarial Attacks on Vision-Language Models

    cs.CR 2026-05 conditional novelty 6.0

    Universal adversarial attacks cause output perturbation 90 times more often than precise target injection in VLMs, with only 2 verbatim successes out of 6615 tests.

  12. ORCA: An Agentic Reasoning Framework for Hallucination and Adversarial Robustness in Vision-Language Models

    cs.CV 2025-09 unverdicted novelty 6.0

    ORCA is an inference-time agentic framework that boosts LVLM accuracy on hallucination benchmarks by 3.64-40.67% and adds adversarial robustness via cross-model validation with small vision tools.

  13. ORCA: An Agentic Reasoning Framework for Hallucination and Adversarial Robustness in Vision-Language Models

    cs.CV 2025-09 unverdicted novelty 6.0

    ORCA is an agentic reasoning framework that enhances factual accuracy and adversarial robustness of pretrained LVLMs via an Observe-Reason-Critique-Act loop with small vision models, reporting accuracy gains of up to ...

  14. A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning

    cs.CV 2026-07 conditional novelty 5.0

    Adversarial SFT of CLIP's projection plus confidence-decayed pseudo-label RL yields ~10% clean and ~16% robust accuracy gains over prior robust UDA methods on three benchmarks.

  15. AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 5.0

    AGC is a training-free inference-time defense for CLIP that adaptively corrects features along geodesics to robust augmentations, claiming 44.4% higher average robust accuracy and 10x lower latency than prior baseline...

  16. Breaking the Illusion: Consensus-Based Generative Mitigation of Adversarial Illusions in Multi-Modal Embeddings

    cs.LG 2025-11 conditional novelty 5.0

    Generative purification with consensus aggregation reduces adversarial illusion attack success rates to near zero on ImageBind while improving alignment on both clean and attacked inputs.