REVIEW 7 cited by
Foundation Models for Video Understanding: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Video Foundation Models (ViFMs) aim to learn a general-purpose representation for various video understanding tasks. Leveraging large-scale datasets and powerful models, ViFMs achieve this by capturing robust and generic features from video data. This survey analyzes over 200 video foundational models, offering a comprehensive overview of benchmarks and evaluation metrics across 14 distinct video tasks categorized into 3 main categories. Additionally, we offer an in-depth performance analysis of these models for the 6 most common video tasks. We categorize ViFMs into three categories: 1) Image-based ViFMs, which adapt existing image models for video tasks, 2) Video-Based ViFMs, which utilize video-specific encoding methods, and 3) Universal Foundational Models (UFMs), which combine multiple modalities (image, video, audio, and text etc.) within a single framework. By comparing the performance of various ViFMs on different tasks, this survey offers valuable insights into their strengths and weaknesses, guiding future advancements in video understanding. Our analysis surprisingly reveals that image-based foundation models consistently outperform video-based models on most video understanding tasks. Additionally, UFMs, which leverage diverse modalities, demonstrate superior performance on video tasks. We share the comprehensive list of ViFMs studied in this work at: \url{https://github.com/NeeluMadan/ViFM_Survey.git}
Forward citations
Cited by 7 Pith papers
-
Empowering Long-form Omni-modal Understanding with Robust Audio Perception
Decoupled audio-visual caption and CoT-QA datasets plus two-stage fine-tuning measurably strengthen auditory perception and cross-modal reasoning in a 7B omni-modal LLM.
-
MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models
MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.
-
VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering
A person-anchored tree plus multi-agent LLM pipeline lets a system answer cross-video queries about the same person, and it beats single-video models on the authors' new CrossVideoQA benchmark.
-
SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications
General-purpose video foundation models, adapted with lightweight readout heads, reach state-of-the-art performance on three of five scientific video benchmarks.
-
How do Foundation Models Compare to Skeleton-Based Approaches for Gesture Recognition in Human-Robot Interaction?
On a new NUGGET gesture dataset, skeleton-based HD-GCN beats vision foundation model V-JEPA (94.4% vs 90.1% top-1), while zero-shot Gemini Flash 2.0 achieves only 42.1%.
-
Video Understanding by Design: How Datasets Shape Video Models
A dataset-centric framework that explains video architectures as responses to structural properties of benchmark datasets.
-
Towards channel foundation models (CFMs): Motivations, methodologies and opportunities
A survey and position paper proposing channel foundation models, with experiments on two pretrained CSI models showing gains over a vanilla ViT baseline.
Discussion (0). Sign in to comment.