REVIEW 12 cited by
CLIP2Video: Mastering Video-Text Retrieval via Image CLIP
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We present CLIP2Video network to transfer the image-language pre-training model to video-text retrieval in an end-to-end manner. Leading approaches in the domain of video-and-language learning try to distill the spatio-temporal video features and multi-modal interaction between videos and languages from a large-scale video-text dataset. Different from them, we leverage pretrained image-language model, simplify it as a two-stage framework with co-learning of image-text and enhancing temporal relations between video frames and video-text respectively, make it able to train on comparatively small datasets. Specifically, based on the spatial semantics captured by Contrastive Language-Image Pretraining (CLIP) model, our model involves a Temporal Difference Block to capture motions at fine temporal video frames, and a Temporal Alignment Block to re-align the tokens of video clips and phrases and enhance the multi-modal correlation. We conduct thorough ablation studies, and achieve state-of-the-art performance on major text-to-video and video-to-text retrieval benchmarks, including new records of retrieval accuracy on MSR-VTT, MSVD and VATEX.
Forward citations
Cited by 12 Pith papers
-
Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval
Hard negatives selected by visual confusability in sign embeddings, not linguistic similarity, substantially raise fine-grained sign-language retrieval accuracy without collapsing coarse performance.
-
Prompt-aware of Frame Sampling for Efficient Text-Video Retrieval
ProCLIP selects query-relevant frames via prompt-aware cross-attention and prunes candidates with a CLIP-distilled lightweight model, matching top accuracy at a fraction of the latency.
-
Privacy-Shielded Image Compression: Defending Against Exploitation from Vision-Language Pretrained Models
The authors build a conditional latent trigger into learned compression so one bitstream can either mislead CLIP-style models or reconstruct full semantics, depending on a mode condition.
-
Learning Speaker-Invariant Visual Features for Lipreading
SIFLip improves lipreading on unseen speakers by combining text-aligned contrastive learning with gradient-reversal-based speaker disentanglement.
-
A Mathematical Perspective On Contrastive Learning
A probabilistic tilting framework for contrastive learning yields closed-form Gaussian results showing which conditional statistics each loss can recover.
-
Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruption
UnInfo adapts CLIP at test time by updating a low-rank adapter on the image encoder with a combination of confidence maximization, uniformity loss and EMA distillation, improving zero-shot accuracy under image corruptions.
-
Detecting Content Rating Violations in Android Applications: A Vision-Language Approach
A vision-language model with style and content encoders plus cross-attention predicts Android game content ratings about 6% more accurately than fine-tuned CLIP and flags suspected rating violations, including Teacher...
-
MADGEN: Mass-Spec attends to De Novo Molecular generation
A two-stage model first retrieves a molecular scaffold from an MS/MS spectrum, then generates the full molecule by adding bonds to that scaffold under spectral guidance, and reports high accuracy only when the scaffol...
-
Blurring Modal Boundaries: A Unified Survey from Single- to Multi-Modal Person Re-ldentification
A unified taxonomy and survey of single- to multi-modal person ReID, plus a Transformer-based VI-ReID baseline that is solid but not state-of-the-art.
-
ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts
A 7B multimodal model that fuses audio and visual signals with explicit timestamps achieves strong measured comprehension of real-world short videos on the authors' new ShortVid-Bench benchmark.
-
An Attention-based Framework with Multistation Information for Earthquake Early Warnings
SENSE, an attention-based multistation model with per-station embeddings, improves earthquake intensity prediction over ISMP and TEAM on Taiwan data, but not on Japan and with no public code.
-
Symbolic Representation for Any-to-Any Generative Tasks
A new symbolic language with functions, parameters, and topology, paired with a GPT-4o-based inference engine, generates executable multimodal workflows without task-specific training.
Discussion (0). Continue with ORCID to comment.