REVIEW 11 cited by
VideoVista: A Versatile Benchmark for Video Understanding and Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite significant breakthroughs in video analysis driven by the rapid development of large multimodal models (LMMs), there remains a lack of a versatile evaluation benchmark to comprehensively assess these models' performance in video understanding and reasoning. To address this, we present VideoVista, a video QA benchmark that integrates challenges across diverse content categories, durations, and abilities. Specifically, VideoVista comprises 25,000 questions derived from 3,400 videos spanning 14 categories (e.g., Howto, Film, and Entertainment) with durations ranging from a few seconds to over 10 minutes. Besides, it encompasses 19 types of understanding tasks (e.g., anomaly detection, interaction understanding) and 8 reasoning tasks (e.g., logical reasoning, causal reasoning). To achieve this, we present an automatic data construction framework, leveraging powerful GPT-4o alongside advanced analysis tools (e.g., video splitting, object segmenting, and tracking). We also utilize this framework to construct training data to enhance the capabilities of video-related LMMs (Video-LMMs). Through a comprehensive and quantitative evaluation of cutting-edge models, we reveal that: 1) Video-LMMs face difficulties in fine-grained video tasks involving temporal location, object tracking, and anomaly detection; 2) Video-LMMs present inferior logical and relation reasoning abilities; 3) Open-source Video-LMMs' performance is significantly lower than GPT-4o and Gemini-1.5, lagging by 20 points. This highlights the crucial role VideoVista will play in advancing LMMs that can accurately understand videos and perform precise reasoning.
Forward citations
Cited by 11 Pith papers
-
ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding
ScaleLong embeds four timescale question types into the same long videos, and evaluation of 23 MLLMs reveals a U-shaped accuracy curve across timescales.
-
The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...
-
VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering
A person-anchored tree plus multi-agent LLM pipeline lets a system answer cross-video queries about the same person, and it beats single-video models on the authors' new CrossVideoQA benchmark.
-
SmartHome-Bench: A Comprehensive Benchmark for Video Anomaly Detection in Smart Homes Using Multi-Modal Large Language Models
A new smart-home video anomaly benchmark and a taxonomy-driven reflective LLM chain that improves MLLM anomaly detection accuracy by 11.62 percentage points over zero-shot prompting.
-
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.
-
VUDG: A Dataset for Video Understanding Domain Generalization
VUDG is a domain-generalization benchmark for video understanding with 11 domains and 36,388 QA pairs, and it shows that current large video-language models lose accuracy across visual domains.
-
VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
VCapsBench is a video caption quality benchmark with 109,796 QA pairs across 21 fine-grained dimensions on 5,677 videos, evaluating caption accuracy, inconsistency, and coverage.
-
TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos
TUNA introduces a 1,000-video benchmark with dense temporal captions and 1,432 multiple-choice questions, and finds that current video LMMs are weakest at camera motion, action sequences, and multi-subject scenes.
-
NeMo: Needle in a Montage for Video-Language Understanding
NeMoBench, an automatically generated benchmark with 31,378 QA pairs, shows that video LLMs struggle with temporal grounding of relevant clips hidden in long montages.
-
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
A new benchmark, VF-Eval, measures how well multimodal LLMs check, detect, and reason about errors in AI-generated videos, and shows frontier models remain far below human performance.
-
VideoEval-Pro: Robust and Realistic Long Video Understanding Evaluation
An open-ended short-answer long-video benchmark, built by converting MCQ questions from four existing tests, shows large accuracy drops and different model rankings versus multiple-choice evaluation.
Discussion (0). Sign in to comment.