Pith. sign in

REVIEW 13 cited by

Understanding Zero-Shot Adversarial Robustness for Large-Scale Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2212.07016 v2 pith:PFFT726S submitted 2022-12-14 cs.CV

Chengzhi Mao , Scott Geng , Junfeng Yang , Xin Wang , Carl Vondrick This is my paper
classification cs.CV
keywords adversarialzero-shotrobustnesslarge-scalemodelstrainingclipmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Pretrained large-scale vision-language models like CLIP have exhibited strong generalization over unseen tasks. Yet imperceptible adversarial perturbations can significantly reduce CLIP's performance on new tasks. In this work, we identify and explore the problem of \emph{adapting large-scale models for zero-shot adversarial robustness}. We first identify two key factors during model adaption -- training losses and adaptation methods -- that affect the model's zero-shot adversarial robustness. We then propose a text-guided contrastive adversarial training loss, which aligns the text embeddings and the adversarial visual features with contrastive learning on a small set of training data. We apply this training loss to two adaption methods, model finetuning and visual prompt tuning. We find that visual prompt tuning is more effective in the absence of texts, while finetuning wins in the existence of text guidance. Overall, our approach significantly improves the zero-shot adversarial robustness over CLIP, seeing an average improvement of over 31 points over ImageNet and 15 zero-shot datasets. We hope this work can shed light on understanding the zero-shot adversarial robustness of large-scale models.

Discussion (0). Sign in to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Challenging Vision-Language Models with Physically Deployable Multimodal Semantic Lighting Attacks

    cs.CV 2026-04 unverdicted novelty 8.0 of 10

    MSLA is the first physically deployable attack that uses adversarial lighting to break semantic alignment in VLMs such as CLIP, LLaVA, and BLIP, causing classification failures and hallucinations in real scenes.

  2. Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    High-noise feature drift distinguishes adversarial from clean inputs in CLIP, allowing a plug-in gating mechanism to selectively trigger existing test-time defenses and raise mean clean+adversarial accuracy across 13 ...

  3. Pay Less Attention to Function Words for Free Robustness of Vision-Language Models

    cs.LG 2025-12 conditional novelty 7.0 of 10

    FDA differentially subtracts function-word cross-attention from original attention heads to cut attack success rates by 18-90% across models and tasks while dropping performance by at most 0.6%.

  4. Two Sides of the Same Coin: Co-Evolving Search for Cross-Task Attacks on Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    CoEvoAttack uses evolutionary search on both text and image sides to generate object-region adversarial examples that transfer across captioning, detection, region categorization, and localization in unified VLMs.

  5. Unifying Adversarially Robust Model Experts in Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    CARE collaboratively fine-tunes two CLIP experts (image-text alignment and image-invariance) with embedding harmonization and EMA merging, yielding a single model with better clean and adversarial accuracy than either expert.

  6. Rethinking Brain Decoding with CLIP: The Role of Adversarial Robustness

    cs.CV 2026-07 accept novelty 6.0 of 10

    Adversarially robust CLIP targets (FARE, TeCoA) consistently beat standard CLIP on fMRI-image retrieval, zero-shot classification, and alignment metrics when the decoder is held fixed.

  7. Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    LIVE uses language to generate task-centric vision embeddings at inference, reducing hallucinations by 34 points on MMVP, outperforming larger VLMs on VQA, and generalizing to unseen tasks.

  8. Investigating Adversarial Robustness of Multi-modal Large Language Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Robust vision encoders from multimodal adversarial pretraining transfer to MLLMs and deliver large gains in adversarial captioning and VQA performance, while test-time stochastic transformations provide an effective b...

  9. Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Introduces Closed-Loop Bidirectional Prompting with Semantic Anchor for cross-modal agreement recovery, claiming SOTA adversarial robustness and generalization on 11 datasets.

  10. Visual prompting reimagined: The power of the Activation Prompts

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    Activation prompts on intermediate layers outperform input-level visual prompting and parameter-efficient fine-tuning in accuracy and efficiency across 29 datasets.

  11. Invisible Watermarks, Visible Gains: Steering Machine Unlearning with Bi-Level Watermarking Design

    cs.CR 2025-08 conditional novelty 6.0 of 10

    Water4MU tunes an invisible watermark on data so that machine unlearning algorithms can remove requested images more effectively, beating prior methods on 'challenging forgets'.

  12. AGC: Adaptive Geodesic Correction for Adversarial Robustness on Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    AGC is a training-free inference-time defense for CLIP that adaptively corrects features along geodesics to robust augmentations, claiming 44.4% higher average robust accuracy and 10x lower latency than prior baseline...

  13. Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weighting

    cs.CV 2025-10 conditional novelty 5.0 of 10

    CAW adds a confidence-weighted KL loss and feature-alignment regularization to CLIP adversarial fine-tuning, raising average AutoAttack robust accuracy from 31.6% to 33.5% on 15 datasets.

Pith tools