REVIEW 22 cited by
ActionCLIP: A New Paradigm for Video Action Recognition
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
ActionCLIP: A New Paradigm for Video Action Recognition
read the original abstract
The canonical approach to video action recognition dictates a neural model to do a classic and standard 1-of-N majority vote task. They are trained to predict a fixed set of predefined categories, limiting their transferable ability on new datasets with unseen concepts. In this paper, we provide a new perspective on action recognition by attaching importance to the semantic information of label texts rather than simply mapping them into numbers. Specifically, we model this task as a video-text matching problem within a multimodal learning framework, which strengthens the video representation with more semantic language supervision and enables our model to do zero-shot action recognition without any further labeled data or parameters requirements. Moreover, to handle the deficiency of label texts and make use of tremendous web data, we propose a new paradigm based on this multimodal learning framework for action recognition, which we dub "pre-train, prompt and fine-tune". This paradigm first learns powerful representations from pre-training on a large amount of web image-text or video-text data. Then it makes the action recognition task to act more like pre-training problems via prompt engineering. Finally, it end-to-end fine-tunes on target datasets to obtain strong performance. We give an instantiation of the new paradigm, ActionCLIP, which not only has superior and flexible zero-shot/few-shot transfer ability but also reaches a top performance on general action recognition task, achieving 83.8% top-1 accuracy on Kinetics-400 with a ViT-B/16 as the backbone. Code is available at https://github.com/sallymmx/ActionCLIP.git
Forward citations
Cited by 22 Pith papers
-
Adapting MLLMs for Nuanced Video Retrieval
Text-only contrastive fine-tuning of an MLLM with hard negatives produces embeddings that handle temporal, negation, and multimodal nuances in video retrieval and achieves SOTA performance.
-
M2R2: MultiModal Robotic Representation for Temporal Action Segmentation
M2R2 proposes a multimodal robotic representation for temporal action segmentation that combines proprioceptive and exteroceptive sensors with a novel training strategy enabling feature reuse across models, achieving ...
-
Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment
Peak-End-Net uses image-aesthetic priors and peak-end-rule frame weighting to set a new state of the art on VADB and zero-shot DIVIDE-3K video aesthetic assessment.
-
Breaking the 15% Barrier: A Real-World Data-Driven System for Proactive Social Robot Triggered by User Nonverbal Cues
About 15% of retail robot turns are nonverbal-triggered; a video multi-label recognizer plus LLM token conditioning can produce proactive responses for frequent social cues.
-
In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
ICMPG combines LLM-based candidate generation with MPC-style physical simulation and semantic scoring to produce text-driven human motions that are both plausible and faithful.
-
TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition
TACO achieves SOTA on video recognition benchmarks by regularizing relative representation geometry and decoupling optimization from test-time representations to address training-evaluation inconsistency in CLIP adaptation.
-
Demystifying the Optimal Fair Classifier in Multi-Class Classification
Derives tractable optimal fair multi-class classifier and supplies in-processing and post-processing algorithms that converge to the accuracy-fairness Pareto frontier.
-
TAG-Head: Time-Aligned Graph Head for Plug-and-Play Fine-grained Action Recognition
TAG-Head adds a lightweight transformer-plus-time-aligned-graph head to existing 3D backbones to reach new RGB-only state-of-the-art on FineGym and HAA500 while surpassing some multimodal baselines.
-
Context Matters: Peer-Aware Student Behavioral Engagement Measurement via VLM Action Parsing and LLM Sequence Classification
A three-stage pipeline uses few-shot VLM action parsing, sliding-window segmentation, and LLM sequence classification with peer context to measure student engagement from classroom videos.
-
Action Hints: Semantic Typicality and Context Uniqueness for Generalizable Skeleton-based Video Anomaly Detection
A skeleton-based zero-shot VAD method distills LLM knowledge for action typicality during training and performs test-time context uniqueness analysis to derive scene-adaptive normality boundaries, claiming SOTA result...
-
Knowledge-guided Disentanglement with Atomic Actions for Action Recognition
LLM-generated atomic action descriptions injected into scene-graph node features improve prompt-guided disentanglement for multi-label action recognition, with reported gains on Charades-oracle and SportsHHI but not o...
-
GHR-VLM: Making Zero-Shot Transit Video Analytics Realizable with Grounded Hybrid Reasoning
A grounded edge-cloud pipeline applies VLM reasoning only to localized passenger and farebox evidence, achieving 31–54% five-class zero-shot payment accuracy on two real bus videos.
-
TRUST: Efficient Abdominal Trauma Recognition via Image-to-Ultrasound-Video Transfer Learning
TRUST is a scan-aware parameter-efficient image-to-video transfer learning framework for abdominal ultrasound trauma recognition featuring CFCA, MGMA, and VQSA modules that outperforms SOTA by 9.63% on in-house datasets.
-
TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition
TACO proposes Relative Structure Distillation and a lightweight specialization projection to mitigate inconsistency between fine-tuning and evaluation objectives in open-vocabulary video recognition, claiming state-of...
-
VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfer
VidPrism introduces a heterogeneous temporal MoE with content-aware multi-rate sampling and bidirectional fusion for image-to-video transfer, claiming SOTA results on video benchmarks.
-
Spatio-Temporal Similarity Volume Aggregation for Open-Vocabulary Action Recognition
SimVA constructs a 4D similarity volume over video tokens and action classes then applies spatial, motion-aware, and Mamba-based temporal aggregation to achieve competitive zero-shot and few-shot performance on open-v...
-
Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation
A trajectory-conditioned retrieval system discovers multiple motion descriptions in videos without user queries and grounds them to point tracks, evaluated mainly on MeViS.
-
SRL-CLIP: Efficient CLIP Video Adaptation via Structured Semantic Role Labels
SRL-CLIP uses rule-based captions derived from semantic role labels to adapt CLIP via contrastive fine-tuning on 23k pairs, matching or exceeding larger models trained on far more data across video tasks.
-
InternVideo: General Video Foundation Models via Generative and Discriminative Learning
InternVideo combines masked video modeling and video-language contrastive learning into a single foundation model that reaches state-of-the-art results on 39 video datasets including 91.1% top-1 on Kinetics-400.
-
Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understanding
A rationale-informed pruning strategy for VLMs yields higher accuracy and more doubly-correct predictions than prior pruning methods on egocentric video benchmarks.
-
EV-CLIP: Efficient Visual Prompt Adaptation for CLIP in Few-shot Action Recognition under Visual Challenges
EV-CLIP introduces mask and context visual prompts to adapt CLIP for improved few-shot video action recognition under visual challenges such as low light and egocentric views, outperforming other efficient methods wit...
-
Video Understanding by Design: How Datasets Shape Video Models
A dataset-centric framework that explains video architectures as responses to structural properties of benchmark datasets.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.