REVIEW 28 cited by
CinePile: A Long Video Question Answering Dataset and Benchmark
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Current datasets for long-form video understanding often fall short of providing genuine long-form comprehension challenges, as many tasks derived from these datasets can be successfully tackled by analyzing just one or a few random frames from a video. To address this issue, we present a novel dataset and benchmark, CinePile, specifically designed for authentic long-form video understanding. This paper details our innovative approach for creating a question-answer dataset, utilizing advanced LLMs with human-in-the-loop and building upon human-generated raw data. Our comprehensive dataset comprises 305,000 multiple-choice questions (MCQs), covering various visual and multimodal aspects, including temporal comprehension, understanding human-object interactions, and reasoning about events or actions within a scene. Additionally, we fine-tuned open-source Video-LLMs on the training split and evaluated both open-source and proprietary video-centric LLMs on the test split of our dataset. The findings indicate that although current models underperform compared to humans, fine-tuning these models can lead to significant improvements in their performance.
Forward citations
Cited by 28 Pith papers
-
ARGUS: Hallucination and Omission Evaluation in Video-LLMs
ARGUS measures hallucination and omission in free-form video captions using LLM-based entailment and temporal alignment, finding that even the best video-LLM still produces roughly 40% hallucinated content.
-
Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding
MF2 evaluates long-movie understanding by asking models to classify fact/fib claim pairs; the best model trails humans by 23.5 points in pairwise accuracy.
-
ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding
ScaleLong embeds four timescale question types into the same long videos, and evaluation of 23 MLLMs reveals a U-shaped accuracy curve across timescales.
-
Vid-SME: Membership Inference Attacks against Large Video Understanding Models
Vid-SME computes Sharma-Mittal entropy differences between natural and reversed video frame sequences to infer training membership in video understanding LLMs, but its effectiveness is confounded by member/non-member ...
-
PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
PerceptionLM releases 2.8M human-labeled fine-grained video QA pairs and spatio-temporal captions, plus models and a new benchmark, arguing that human data, not just synthetic data, is needed for detailed video understanding.
-
NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video
NARU provides a human-verified benchmark of 1,481 multiple-choice questions over 155 Japanese long-form videos, and shows that even the best model scores only 76.2% while open-source models stay near chance.
-
The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...
-
Evidence-Driven Dynamic Visual Selector for Efficient Long Video Understanding
EviSelect uses the target multimodal model's internal attention as a prior to dynamically select frames, sampling rates, and resolutions, achieving about 50% token reduction and a 3.9x speedup with better benchmark accuracy.
-
RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model
RynnBrain 1.1 reports state-of-the-art embodied cognition and localization scores with a 122B-A10B model and improved real-robot VLA policies via joint multi-embodiment training.
-
VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding
An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.
-
MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering
MUPA combines three ordering-based reasoning paths with a reflection agent that verifies and fuses answer-evidence pairs, reaching 30.3% and 47.4% grounded QA accuracy on NExT-GQA and DeVE-QA with a 7B model.
-
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.
-
CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models
CausalVQA provides 793 paired real-video causal reasoning questions on which the best multimodal model scores 61.66% versus 84.78% for humans, with the largest gaps on anticipation and hypothetical questions.
-
Ming-Omni: A Unified Multimodal Model for Perception and Generation
A single model with modality-specific routing processes image, text, audio, and video inputs and generates text, speech, and images, with public benchmarks reported across all of these abilities.
-
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.
-
ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding
A reinforcement-learned frame selection policy, trained with reward margins from a reference video-LLM, improves video QA accuracy of LLaVA-OV and InternVL3 across several benchmarks.
-
MINERVA: Evaluating Complex Video Reasoning
MINERVA provides 1,515 multi-step video QA questions with human reasoning traces; frontier models score far below humans and fail mainly on temporal localization and perception.
-
Vidi: Large Multimodal Models for Video Understanding and Editing
Vidi uses dense 1 fps sampling and decomposed attention to localize text queries in videos up to an hour or longer, outperforming GPT-4o and Gemini on the new VUE-TR benchmark.
-
LongViTU: Instruction Tuning for Long-Form Video Understanding
LongViTU, an automatically generated 121k-pair video QA dataset with 4.6-minute average certificate length, provides modest SFT gains on long-video benchmarks.
-
CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding
A 12,129-question benchmark for long videos that requires models to retrieve the specific video moments supporting each answer, exposing a gap between multiple-choice accuracy and genuine video understanding.
-
Neptune: The Long Orbit to Benchmarking Long Video Understanding
Neptune is a 3,268-question, 2,405-video benchmark for long video understanding with a scalable LLM-based generation pipeline and an open-source answer-equivalence metric, GEM.
-
Reading Between the Frames: Interpreting Implicit and Non-literal Meaning in Social Media Videos
A new video benchmark, DrivelHub+, shows that current video-language models can describe what happens in social media clips but largely fail to infer the implicit humour, irony, or cultural meaning.
-
LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation
LaVi encodes visual context into LayerNorm affine parameters, bypassing visual token concatenation, and reports LLaVA-comparable accuracy at a 94% FLOP reduction.
-
ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understanding
ActionArt provides a fine-grained human-motion video benchmark and shows that proxy tasks trained on auto-generated captions lift a video MLLM from 55.8% to 63.0% average accuracy, still short of 67.6% with manual ann...
-
FocusedAD: Character-centric Movie Audio Description
FocusedAD uses face tracking, soft prompts, and a video language model to generate character-focused movie audio descriptions, reporting state-of-the-art scores on MAD-eval-Named and the new Cinepile-AD benchmark.
-
Ola: Pushing the Frontiers of Omni-Modal Language Model
Ola, a 7B omni-modal language model, achieves competitive image, video, and audio understanding with progressive modality alignment, though it does not beat all specialized models on every benchmark.
-
Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks
Across three short traffic video sequences (two real, one synthetic), VideoLLaMA-2 outperformed GPT-4o, Gemini 1.5 Pro, InternVL, and LLaVA-NeXT-Video with 57 percent average accuracy, while all models exhibited clear...
-
VideoLLM Benchmarks and Evaluation: A Survey
A survey of VideoLLM benchmarks and evaluation protocols that organizes known datasets and metrics, with proposed future benchmark designs.
Discussion (0). Continue with ORCID to comment.