CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:BZC5SDMFrecord.jsonopen to challenge →
read the original abstract
Humans are able to perceive, understand and reason about causal events. Developing models with similar physical and causal understanding capabilities is a long-standing goal of artificial intelligence. As a step towards this direction, we introduce CRAFT, a new video question answering dataset that requires causal reasoning about physical forces and object interactions. It contains 58K video and question pairs that are generated from 10K videos from 20 different virtual environments, containing various objects in motion that interact with each other and the scene. Two question categories in CRAFT include previously studied descriptive and counterfactual questions. Additionally, inspired by the Force Dynamics Theory in cognitive linguistics, we introduce a new causal question category that involves understanding the causal interactions between objects through notions like cause, enable, and prevent. Our results show that even though the questions in CRAFT are easy for humans, the tested baseline models, including existing state-of-the-art methods, do not yet deal with the challenges posed in our benchmark.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
Do generative video models understand physical principles?
Physics-IQ benchmark reveals that generative video models exhibit limited physical understanding unrelated to their visual quality.
-
CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models
CRONOS benchmark shows recent open-source video generators fail to preserve physical consistency under controlled changes to viewpoint, scene, object category, and appearance.
-
Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment
VAORA aligns VLM chain-of-thought reasoning with visual scene observations and post-action outcomes via structured symbolic rewards, achieving cross-task and cross-environment generalization on physical reasoning benchmarks.
-
$\Delta$ynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos
A vision-language framework generates text-based rigid-body scene configurations from videos using motion reasoning and optical flow, reporting 0.30 IoU on CLEVRER (7x over baselines) and transfer to 235 real videos.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.