Pith. sign in

REVIEW 27 cited by

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2109.14084 v2 pith:FS4ZJK7B submitted 2021-09-28 cs.CV cs.CL

Hu Xu , Gargi Ghosh , Po-Yao Huang , Dmytro Okhonko , Armen Aghajanyan , Florian Metze , Luke Zettlemoyer , Christoph Feichtenhofer This is my paper
classification cs.CVcs.CL
keywords videoclipactioncontrastivedownstreamretrievaltaskstextunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.

Discussion (0). Sign in to comment.

Forward citations

Cited by 27 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    DETR-ViP boosts visual-prompted detection performance by learning globally discriminative prompts through integration and distillation on top of image-text contrastive learning, with a selective fusion step for stability.

  2. InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    InstAP introduces instance-aware pre-training with a new dual-granularity dataset InstVL that improves both fine-grained instance retrieval and global video understanding over standard VLP baselines.

  3. DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    DIRECT uses a three-level multi-agent framework to solve video mashup creation as a multimodal coherency problem, outperforming baselines on a new benchmark.

  4. WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    WikiCLIP delivers an efficient contrastive baseline for open-domain visual entity recognition that improves accuracy by 16% on OVEN unseen entities and runs nearly 100 times faster than leading generative models.

  5. Adapting MLLMs for Nuanced Video Retrieval

    cs.CV 2025-12 unverdicted novelty 7.0 of 10

    Text-only contrastive fine-tuning of an MLLM with hard negatives produces embeddings that handle temporal, negation, and multimodal nuances in video retrieval and achieves SOTA performance.

  6. Adversarial Video Promotion Against Text-to-Video Retrieval

    cs.CV 2025-08 unverdicted novelty 7.0 of 10

    Pioneers ViPro, the first attack to adversarially promote videos in text-to-video retrieval, using Modal Refinement to improve black-box transferability across multiple targets.

  7. Stitch-a-Demo: Video Demonstrations from Multistep Descriptions

    cs.CV 2025-03 unverdicted novelty 7.0 of 10

    Stitch-a-Demo is a retrieval-based method that assembles visually coherent video demonstrations from multistep textual descriptions by training on weakly supervised procedural data with hard negatives.

  8. VideoChat: Chat-Centric Video Understanding

    cs.CV 2023-05 conditional novelty 7.0 of 10

    VideoChat integrates video models and LLMs via a learnable interface for chat-based spatiotemporal and causal video reasoning, trained on a new video-centric instruction dataset.

  9. Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

    cs.CV 2022-04 unverdicted novelty 7.0 of 10

    Socratic Models compose zero-shot multimodal reasoning by prompting pretrained language and vision models to exchange information and enable new capabilities without finetuning.

  10. HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    HPP decouples perception from reasoning in long-video VLMs by having an LLM run iterative programmatic probes on hierarchically segmented video, reporting gains on LongVideoBench, EgoSchema, VideoMME, and MLVU.

  11. Multimodal LLMs under Pairwise Modalities

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    A two-stage framework enables multimodal LLMs to learn shared latent representations from pairwise modality data and achieve cross-modal generation when incorporating new modalities.

  12. DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts

    cs.CV 2026-04 accept novelty 6.0 of 10

    Global prompt integration, visual-textual relation distillation and selective fusion make visual prompts discriminative enough for DETR-ViP to beat prior visual-prompt detectors by several mAP points.

  13. CoVR-R:Reason-Aware Composed Video Retrieval

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Zero-shot LMM reasoning over edit after-effects (states, phases, camera, tempo) plus a new CoVR-R benchmark yields large recall gains on implicit-effect composed video retrieval without task-specific training.

  14. WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

    cs.CV 2026-03 conditional novelty 6.0 of 10

    WikiCLIP reaches 28.5% OVEN-unseen accuracy (vs 24.5% AutoVER) at 14.5 ms latency by vision-guided LLM embeddings plus hard-negative text swaps.

  15. Calibrated Multimodal Representation Learning with Missing Modalities

    cs.CV 2025-11 unverdicted novelty 6.0 of 10

    CalMRL mitigates anchor shift in multimodal representation learning by calibrating incomplete alignments through representation-level imputation of missing modalities using priors and a bi-step optimization with close...

  16. Beyond Simple Edits: Composed Video Retrieval with Dense Modifications

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A new benchmark with much longer, denser modification texts, plus a single-encoder fusion model, raises composed video retrieval Recall@1 by 3.4 points on its own test set.

  17. Revisiting Feature Prediction for Learning Visual Representations from Video

    cs.CV 2024-02 conditional novelty 6.0 of 10

    V-JEPA models trained only on feature prediction from 2 million public videos achieve 81.9% on Kinetics-400, 72.2% on Something-Something-v2, and 77.9% on ImageNet-1K using frozen ViT-H/16 backbones.

  18. LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

    cs.CV 2023-10 unverdicted novelty 6.0 of 10

    LanguageBind aligns video, infrared, depth, and audio to a frozen language encoder via contrastive learning on the new VIDAL-10M dataset, extending video-language pretraining to N modalities.

  19. InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

    cs.CV 2023-07 unverdicted novelty 6.0 of 10

    InternVid supplies 7M videos and LLM captions to train ViCLIP, which reaches leading zero-shot action recognition and competitive retrieval performance.

  20. R3M: A Universal Visual Representation for Robot Manipulation

    cs.RO 2022-03 unverdicted novelty 6.0 of 10

    A visual encoder pre-trained on diverse human videos with contrastive and language objectives improves simulated robot manipulation success by over 20% versus training from scratch and enables real Franka arm tasks fr...

  21. Vision-language Models for Driver Monitoring Systems: A Driver Activity Description Dataset

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Introduces Drive&Act description dataset of fine-grained driver activity text and reports fine-tuned VLM reaching 76 ACCR on DMD dataset versus 66 for zero-shot baseline.

  22. Learning from the Unseen: Generative Data Augmentation for Geometric-Semantic Accident Anticipation

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    A generative video synthesis pipeline paired with a semantic graph neural network yields gains in accident anticipation accuracy and lead time on driving datasets, accompanied by a new benchmark release.

  23. Learning ORDER-Aware Multimodal Representations for Composite Materials Design

    cs.LG 2026-01 unverdicted novelty 5.0 of 10

    ORDER is an ordinal-aware multimodal alignment method for learning continuous representations that support property prediction and microstructure generation in composite materials.

  24. TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning

    cs.CV 2025-12 unverdicted novelty 5.0 of 10

    TempR1 applies temporal-aware multi-task RL using GRPO and three types of localization rewards to achieve SOTA temporal understanding in MLLMs with synergistic gains from joint optimization.

  25. VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents

    cs.CV 2025-07 unverdicted novelty 5.0 of 10

    VLM2Vec-V2 is a multimodal embedding model trained on an extended MMEB-V2 benchmark that adds video and visual document tasks and reports gains on both new and prior image benchmarks.

  26. InternVideo: General Video Foundation Models via Generative and Discriminative Learning

    cs.CV 2022-12 unverdicted novelty 5.0 of 10

    InternVideo combines masked video modeling and video-language contrastive learning into a single foundation model that reaches state-of-the-art results on 39 video datasets including 91.1% top-1 on Kinetics-400.

  27. Video Understanding by Design: How Datasets Shape Video Models

    cs.CV 2025-09 reject novelty 4.0 of 10

    A dataset-centric framework that explains video architectures as responses to structural properties of benchmark datasets.

Pith tools