Pith. sign in

REVIEW 15 cited by

How Much Can CLIP Benefit Vision-and-Language Tasks?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2107.06383 v1 pith:7YUNHCRR submitted 2021-07-13 cs.CV cs.AIcs.CLcs.LG

Sheng Shen , Liunian Harold Li , Hao Tan , Mohit Bansal , Anna Rohrbach , Kai-Wei Chang , Zhewei Yao , Kurt Keutzer This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords clipvisualtasksdatabetterclip-vilencodersmodels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Probing Visual Language Priors in VLMs

    cs.CV 2024-12 conditional novelty 7.0 of 10

    ViLP shows that vision-language models often answer from text priors instead of image content, and an image-corruption DPO method partially fixes this.

  2. Visual Question Answering on Multiple Remote Sensing Image Modalities

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A new remote sensing VQA dataset with RGB, multispectral, and SAR modalities, plus a VisualBERT baseline, reaches 65.56% average accuracy and shows that multi-modal context helps.

  3. Natural Language Supervision for Low-light Image Enhancement

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A low-light enhancement network that conditions image features on text prompts through cross-modal attention reports the best PSNR and SSIM on LOLv1 and LOLv2-synthetic among the methods compared.

  4. NAVCON: A Cognitively Inspired and Linguistically Grounded Corpus for Vision and Language Navigation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A new corpus adds 236,316 navigation concept annotations and 2.7 million aligned video frames to the R2R and RxR vision-language navigation datasets.

  5. LSDM: LLM-Enhanced Spatio-temporal Diffusion Model for Service-Level Mobile Traffic Prediction

    cs.LG 2025-07 conditional novelty 5.0 of 10

    LSDM predicts next-hour mobile traffic per app category by feeding a diffusion model with satellite imagery, POI counts, and LLM-generated text descriptions, outperforming eight baselines on a single real-world dataset.

  6. (Almost) Free Modality Stitching of Foundation Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A hypernetwork that generates connector weights for all image-text model pairs can rank pairs like grid search at about 10x lower training cost, but the best connector lags grid search by a few points.

  7. Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigation

    cs.CV 2025-04 conditional novelty 5.0 of 10

    MFRA combines a hierarchical DIRformer-style fusion backbone with instruction-guided attention and a GRU history encoder, reporting improved VLN benchmark scores.

  8. Human Action CLIPs: Detecting AI-generated Human Motion

    cs.CV 2024-11 conditional novelty 5.0 of 10

    CLIP-based semantic embeddings, with a fine-tuned variant, detect AI-generated human-motion video with high accuracy (up to 99.2% video-level) and generalize to unseen generators.

  9. Multimodal Plant Root Phenotyping with Integration of 3D Skeleton Extraction and Language Analysis

    cs.CV 2026-08 reject novelty 4.0 of 10

    An unsupervised 3D root skeleton extractor plus evidence-first GPT-4o fine-tuning is claimed to improve root phenotyping VQA accuracy on a private 12-species dataset.

  10. ReVision : A Post-Hoc, Vision-Based Technique for Replacing Unacceptable Concepts in Image Generation Pipeline

    cs.CR 2026-02 conditional novelty 4.0 of 10

    ReVision uses a vision-language model's bounding box to gate attention-based image editing, suppressing unsafe concepts while better preserving benign background in multi-concept scenes.

  11. A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications

    eess.IV 2026-01 unverdicted novelty 4.0 of 10

    A survey that classifies visual semantic communication into preservation, expansion, and refinement categories and reviews their machine-learning components and applications.

  12. Proactive Adversarial Defense: Harnessing Prompt Tuning in Vision-Language Models to Detect Unseen Backdoored Images

    cs.CV 2024-12 conditional novelty 4.0 of 10

    A prompt-tuned CLIP model trained on five known backdoor attacks can detect a sixth held-out attack type with about 86% accuracy on CIFAR-10 and GTSRB.

  13. Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks

    cs.CV 2025-07 reject novelty 3.0 of 10

    CLIP misclassifies human actions under masking, and the proposed class-specific noise augmentation is claimed to help but lacks quantitative validation.

  14. Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward

    cs.CV 2025-05 conditional novelty 3.0 of 10

    A survey paper reviews multimodal generative AI and autoregressive LLMs for text-driven human motion generation, with comparative tables of models, datasets, and metrics.

  15. Visual question answering: from early developments to recent advances -- a survey

    cs.CV 2025-01 conditional novelty 2.0 of 10

    A survey that classifies VQA architectures by encoder, fusion, and decoder, reviews datasets and metrics, and discusses applications and future directions.

Pith tools