REVIEW 7 cited by
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GPT-4). Compared to existing benchmarks that focus on computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, to provide a comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime. For these purposes, the Perception Test introduces 11.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels (multiple-choice and grounded video question-answers, object and point tracks, temporal action and sound segments), enabling both language and non-language evaluations. The fine-tuning and validation splits of the benchmark are publicly available (CC-BY license), in addition to a challenge server with a held-out test split. Human baseline results compared to state-of-the-art video QA models show a substantial gap in performance (91.4% vs 46.2%), suggesting that there is significant room for improvement in multimodal video understanding. Dataset, baseline code, and challenge server are available at https://github.com/deepmind/perception_test
Forward citations
Cited by 7 Pith papers
-
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
LangDC compresses video tokens dynamically by converting clips into captions from a small language model, cutting compute by 49% with near-parity accuracy.
-
CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models
CausalVQA provides 793 paired real-video causal reasoning questions on which the best multimodal model scores 61.66% versus 84.78% for humans, with the largest gaps on anticipation and hypothetical questions.
-
Correspondence of high-dimensional emotion structures elicited by video clips between humans and Multimodal LLMs
Multimodal LLMs capture the coarse category-level structure of human emotional responses to videos, but not fine-grained item-level emotion structure.
-
J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM
J-EDI QA is a new 100-image Japanese multiple-choice benchmark for deep-sea organism identification; OpenAI o1 scored 50%, GPT-4o 39%, and non-expert humans about 40%.
-
LaCo: Efficient Layer-wise Compression of Visual Tokens for Multimodal Large Language Models
Inserting a pixel-shuffle plus residual patch-merge layer inside the vision encoder compresses visual tokens more efficiently than post-encoder compression, at modest accuracy cost.
-
MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale
A fully open pipeline that rewrites multimodal instruction data into CoT-style rationales yields a 12M dataset and an 8B model with strong benchmark gains, though some evaluation benchmarks overlap the training data.
-
Movie2Story: A framework for understanding videos and telling stories in the form of novel text
MSBench evaluates video-plus-audio to novel-style story generation; the M2S pipeline combines existing video, speech, emotion, and speaker tools with an LLM and reportedly beats video-only baselines.
Discussion (0). Continue with ORCID to comment.