REVIEW 3 major objections 6 minor 2 cited by
MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims MORSE-500, a 500-clip video benchmark with questions embedded in footage, measures the strongest vision-language models at 23.6% against 55.4% for humans, with planning near random — evidence of architectural limits, not…
desk verdict A solid, well-released video benchmark with a real evaluation bug in its headline planning numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the programmatic generation pipeline: deterministic Python scripts built on Manim, Matplotlib, and MoviePy, plus generative video models and curated real footage, that render each reasoning task as an animated clip and derive the ground-truth answer from the same parameter values that set the scene. Because entity count, reasoning depth, distractor density, temporal dynamics, and visual complexity are script parameters, difficulty is controllable and reproducible, and the question-in-the-video design forces models to extract both the question and the evidence from pixels rather than from a separate prompt. The evaluation harness borrows the answer-extraction and string-matching protocol of MathVista for scoring.
What would settle it
Run the strongest models on the 100 planning clips with native video input once the APIs support it (or with 8 fps and 128 frames as an intermediate check), and separately have human raters answer the same clips from only 2 fps, 32-frame stills. If model planning accuracy rises well above 5%, or human accuracy under the frame cap falls toward model levels, the claimed architectural limit is at least partly a stimulus-sampling artifact.
Extended reading notes
Core claim
On its own terms, the paper reports two discoveries. The first is the benchmark itself: 500 videos built so that answering requires genuine temporal understanding, with questions embedded as on-screen text inside the clip, categories grounded in a cognitive taxonomy, and roughly a third of instances deliberately beyond current model capabilities. The second is the measured state of the field: every evaluated model, from Gemini 2.5 Pro and OpenAI o3 down to 3B open-weight models, scores between 5.0% and 23.6% overall, with abstract reasoning below 25% and planning at 0–5% — near-random given the multiple-choice structure — while humans reach 55.4%. The paper interprets the uniformity of the collapse across scales and training paradigms as evidence that current multimodal systems are doing elaborate pattern matching rather than genuine multi-step reasoning over dynamic scenes, and it releases the corpus, scripts, and harness so the measurement can be repeated and extended.
Load-bearing premise
The strongest models (Gemini 2.5 Pro and the OpenAI o-series) received the videos as 2 frames-per-second stills capped at 32 frames because their APIs do not yet accept video input, so the headline human-versus-model gap could shrink if native video input helps them.
Editorial extensions
If this is right
- If the measured gap holds, no current vision-language model is close to human-level multimodal reasoning on dynamic tasks, so claims about deploying such models in autonomous, video-driven decision systems need to be tempered.
- The benchmark's generation scripts let difficulty be scaled along entity count, reasoning depth, distractor density, and temporal dynamics, so it can keep producing harder instances instead of saturating.
- The paper's ablation shows model accuracy drops as the same content is given as a single image, then multiple images, then video — so distributing information over time is itself a current failure mode to target.
- The planning category (0–5% across all models) is the clearest diagnostic: multi-step, goal-directed reasoning over video is essentially unachieved and should be the first thing tested in new models.
Reading between the lines
- Editorial inference: if the planning failure is a bag-of-frames problem, frame-shuffling a planning clip should barely change model accuracy while destroying human accuracy; this is a cheap test the paper does not run.
- Editorial inference: the real-versus-generated physics task converts video-generator quality into benchmark difficulty, so it will get harder automatically as generators improve — a free difficulty ladder the paper does not highlight.
- Editorial inference: humans were not tested under the same 2 fps, 32-frame constraint applied to the strongest models; a matched human ablation would separate frames lost from reasoning lost in the headline 31.8-point gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MORSE-500, a 500-clip video benchmark spanning six reasoning categories (abstract, mathematical, physical, planning, spatial, temporal), with all videos generated or curated through deterministic Python scripts, generative video models, and real footage. The design goals are temporal-first, vision-centric evaluation (questions appear inside the video), a cognitive-science-grounded taxonomy, and programmatic difficulty scaling for future "living" benchmark versions. The authors evaluate 19 closed- and open-source models, all promoted with the minimal instruction "Answer the question in this video," and report a large human-model performance gap (55.4% human vs. 23.6% best model), with particularly low scores in planning (0-5%) and abstract reasoning. The dataset, generation scripts, and evaluation harness are publicly released.
Significance. If the reported results are taken at face value, MORSE-500 is a useful diagnostic resource: it is one of the few video benchmarks that spans six reasoning types under controlled generation, and the release of deterministic scripts, ground-truth annotations, and an evaluation harness is a genuine contribution to reproducible benchmark development. The programmatic generation pipeline with explicit difficulty parameters is a strong design choice for a benchmark meant to evolve. The headline finding that frontier models lag human performance by roughly thirty points across categories, and especially on planning, would be an important stress-test signal for the multimodal reasoning community. However, the paper's most emphasized quantitative conclusions currently rest on evaluation choices that are either undocumented or not yet shown to be artifact-free, so the significance of the stated planning bottleneck in particular cannot be established from the manuscript as written.
major comments (3)
- [§3.1, §3.2, Table 2] The central human-vs-model comparison is confounded by input modality. Human performance (55.4%) is reported on native video, while the strongest API models (o3, o4-mini, Gemini 2.5 Pro, Gemini 2.5 Flash, etc.) are evaluated on 2 fps, 32-frame stills as stated in §3.1. The only frame-sampling ablation, Table 4, is computed on Gemini 2.5 Flash and shows overall scores between 18.4% and 19.2% across settings; it does not quantify the native-video loss for o3 or Gemini 2.5 Pro. Since §5's claim of "fundamental architectural limitations rather than scaling issues" depends directly on the size of the 55.4%-vs-23.6% gap, the authors need to either evaluate the strongest models on native video where the API supports it or run a carefully matched subset experiment to bound the frame-sampling cost. As written, part of the reported gap could be an evaluation-protocol artifact rather than a reasoning deficit.
- [§3.2, Table 2 (Planning column), Appendix B.4, §5] The paper's strongest and most emphasized result, the planning bottleneck, is undermined by the fact that every model scores 0-5% on the planning category, which is below chance for representative items. Appendix B.4 shows a Frozen Lake planning question with five options, implying a 20% random baseline; the 0-5% numbers across all models therefore suggest a systematic measurement artifact, such as letter/option mapping errors, answer-extraction failures, unreadable arrow symbols at the 512px/2fps/32-frame protocol, or a bug in the planning evaluation pipeline. The paper reports no random or majority baselines, no per-planning-subcategory breakdown (mazes vs. rope knots vs. robot manipulation), and no error analysis. Calling this performance "near-random" is inaccurate. The authors must add baselines, a per-task breakdown, and a re-scoring pass before the claim that planning reveals a fundamental architectural limitation can be accepted.
- [§2.3.2, Table 2] The human baseline is insufficiently specified and the model results have no uncertainty estimates. Section 2.3.2 states that "paper authors" evaluated "randomly sampled data slices" but gives no number of evaluators, number of items per evaluator, whether evaluators watched native video, question-format constraints, or inter-annotator agreement. In Table 2, category sizes are as small as 64 items (abstract and physical), so the reported differences between closely ranked models (e.g., 17.4% vs. 16.8% overall) are within plausible sampling noise; no confidence intervals or significance tests are provided. For a benchmark whose purpose is diagnostic comparison, the human protocol and statistical uncertainty are part of the measurement itself and should be reported.
minor comments (6)
- [§3.3] The text refers to "Table 6" for qualitative examples, but the displayed artifact is labeled as a figure (Figure 6); the table/figure numbering should be made consistent.
- [§3.4] The phrase "full video input (as a mp3 file format)" should read "video input (as an mp4 file)"; audio is not part of this experiment.
- [§3.1] The protocol says "we provided detailed instructions on the output formatting in the video" while also stating "No few-shot examples or format-specific guidance were provided"; clarify that the on-screen format instructions are the only guidance and are not considered few-shot examples.
- [Appendix C.2] The model name is sometimes written as Qwen2-235B-A22B and elsewhere as Qwen3-235B-A22B; please use the correct name consistently.
- [Abstract] The phrase "various Gemini 2.5 Pro and OpenAI o3" is imprecise; list model names exactly (e.g., "Gemini 2.5 Pro, OpenAI o3, and related variants").
- [§2.2] The difficulty calibration ("20% current-model capabilities, 50% moderate extensions, 30% stress tests") is asserted without a measured operationalization; consider defining the calibration procedure or reporting model pass rates per difficulty band.
Circularity Check
No significant circularity; the benchmark's central measurements are self-contained and do not reduce to their inputs by construction.
full rationale
MORSE-500 is a benchmark paper whose outputs are measurements (model accuracies, human baseline, per-category scores) rather than derived predictions. The central claim that frontier models underperform humans across all six reasoning categories rests on Table 2, which reports raw evaluation results against independently generated video questions. The questions themselves are produced from external task families (ARC-AGI, maze/FrozenLake, robotic manipulation datasets, physics IQ videos) and cognitive taxonomies (CHC, dual-process theory), not from the model outputs being measured. No equation in the paper equates a predicted quantity with a fitted parameter; the difficulty calibration (20/50/30 split) is a hand-chosen design decision and is not statistically forced from any evaluation result. The human baseline is collected by the authors, which introduces mild self-involvement, but the human numbers are reported as empirical performance, not as a fitted input used to predict model scores. Similarly, the choice of 2 fps / 32-frame sampling for image-based models is an evaluation protocol choice, and while it may affect the validity or generalizability of the planning-deficit conclusion, it is a correctness/applicability concern, not a circularity concern. The paper does not invoke a uniqueness theorem from its own authors, does not smuggle an ansatz via self-citation, and does not rename a known result. The only self-citations (e.g., Mementos) are contextual comparisons in related work and are not load-bearing. Therefore no step in the derivation chain reduces to its own inputs by construction.
Assumptions & free parameters
assumptions (4)
- domain assumption The six-way reasoning taxonomy (abstract, mathematical, physical, planning, spatial, temporal) is a meaningful and separable decomposition of multimodal reasoning.
- domain assumption For the real-vs-generated physical reasoning task, the real footage selected from the Physics IQ Benchmark is always the physically realistic answer.
- domain assumption The official ARC-AGI/ARC-AGI-2 outputs used as ground truth are correct, and the adapted video questions preserve the intended rule.
- domain assumption The answer-extraction LLM (Qwen2.5 72B AWQ) and string matching correctly parse model outputs into choices.
Cite this review
Pith. "Pith review of MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning." pith.science (2026). https://pith.science/paper/FW46JWWG
@misc{pith2026250605523,
author = {Pith},
title = {Pith review of: MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning},
year = {2026},
howpublished = {\url{https://pith.science/paper/FW46JWWG}},
note = {Machine review of arXiv:2506.05523}
}
read the original abstract
Despite rapid advances in vision-language models (VLMs), current benchmarks for multimodal reasoning fall short in three key dimensions. First, they overwhelmingly rely on static images, failing to capture the temporal complexity of real-world environments. Second, they narrowly focus on mathematical problem-solving, neglecting the broader spectrum of reasoning skills -- including abstract, physical, planning, spatial, and temporal capabilities -- required for robust multimodal intelligence. Third, many benchmarks quickly saturate, offering limited headroom for diagnosing failure modes or measuring continued progress. We introduce MORSE-500 (Multimodal Reasoning Stress-test Environment), a video benchmark composed of 500 fully scripted clips with embedded questions spanning six complementary reasoning categories. Each instance is programmatically generated using deterministic Python scripts (via Manim, Matplotlib, MoviePy), generative video models, and curated real footage. This script-driven design allows fine-grained control over visual complexity, distractor density, and temporal dynamics -- enabling difficulty to be scaled systematically as models improve. Unlike static benchmarks that become obsolete once saturated, MORSE-500 is built to evolve: its controllable generation pipeline supports the creation of arbitrarily challenging new instances, making it ideally suited for stress-testing next-generation models. Initial experiments with state-of-the-art systems -- including various Gemini 2.5 Pro and OpenAI o3 which represent the strongest available at the time, alongside strong open-source models -- reveal substantial performance gaps across all categories, with particularly large deficits in abstract and planning tasks. We release the full dataset, generation scripts, and evaluation harness to support transparent, reproducible, and forward-looking multimodal reasoning research.
Forward citations
Cited by 2 Pith papers
-
The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...
-
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.
Reference graph
Works this paper leans on
-
[1]
First Input : 1 blue square→ Output : 2 blue squares
-
[2]
Second Input : 2 blue squares→ Output : 3 blue squares
-
[3]
How many black squares are there?
Third Input : 3 blue squares→ Output : ? Following the established pattern, the next output should contain 4 blue squares . Answer: 4 (Incorrect) o3 Thought for 9m 17s I’m thinking about how to inspect the frames in the video. The goal here is to understand what the video shows, so maybe I’ll examine the first few frames or perhaps the last frame where th...
-
[4]
URL https://arxiv.org/abs/2412.16720. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2901– 2910, 2017. doi: 10....
arXiv 2017
-
[5]
VisualCOMET: Reasoning about the Dynamic Context of a Still Image
Accessed: 2025-05-16. Jae Sung Park, Dong Huk Kim, Rowan Zellers, Yonatan Bisk, Ani Kembhavi, and Yejin Choi. Visualcomet: Reasoning about the dynamic context of a still image. InEuropean Conference on Computer Vision (ECCV), pages 508–524. Springer, 2020. URLhttps://arxiv.org/abs/2004.10796. Shi Qiu, Shaoyang Guo, Zhuo-Yang Song, Yunbo Sun, Zeyu Cai, Jia...
work page Pith review arXiv 2025
-
[6]
URL https://arxiv.org/abs/2407.15754. Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: personalized robot assistance with large language models. AutonomousRobots, 47(8):1087–1102, nov 2023. ISSN 1573-7527. doi: 10.1007/s10514-023-10139-z. URL https://doi.org/10.1...
arXiv 2023
-
[11]
Identify Trigger Pixels: A black pixel in the input grid is a "trigger pixel" if and only if all 8 of its surrounding neighbors (horizontally, vertically, and diagonally) are blue. Black pixels that are part of larger black structures (i.e., have at least one black neighbor) are not triggers and remain unchanged (they are "inert")
-
[12]
By what factor was the radius scaled?
Transformation Rule: • Each trigger black pixel remains black in the output. • All 8 of its neighboring blue pixels (which, by definition of a trigger pixel, must all be blue) change to light blue in the output. • Blue pixels that are not neighbors of any trigger black pixel remain blue. 61 MORSE: Multimodal Reasoning Stress-test Environment • Inert black...
Show all 15 references
-
[13]
Stairs leading downwards are clearly visible on the right-hand side
0:00: The video starts in a hallway. Stairs leading downwards are clearly visible on the right-hand side. The hallway ahead leads towards the living room and kitchen area. At this point, we are in the hallway, before entering the kitchen
-
[14]
The stairs to the basement are visible on the right
0:07: After a tour of an office and bathroom, the camera is again in the hallway, moving towards the kitchen/living area. The stairs to the basement are visible on the right. This is still before entering the kitchen. 82 MORSE: Multimodal Reasoning Stress-test Environment
-
[15]
The stairs are immediately to the right, just as one is about to enter the kitchen space
0:09: The camera is at the threshold of the open-plan kitchen and living area. The stairs are immediately to the right, just as one is about to enter the kitchen space. Considering these observations: A) Right before entering the kitchen. This accurately describes the location...
-
[500]
#0074 D9
Each category incorporates specific complexity controls and novel adaptations to create challenging multimodal scenarios. 28 MORSE: Multimodal Reasoning Stress-test Environment B. Example Question and Code B.1. Abstract Reasoning Abstract Reasoning - ARC-AGI Video Question Que...
1920
-
[1983]
URLhttps://doi.org/10.1207/s15516709cog0702_3
doi: 10.1207/s15516709cog0702_3. URLhttps://doi.org/10.1207/s15516709cog0702_3. Anastasis Germanidis. Introducing gen-3 alpha: A new frontier for video generation.https://runwayml. com/research/introducing-gen-3-alpha, 2024. Runway Research blog, Accessed 22 May 2025. Google. ...
2024 doi
-
[2024]
Accessed 22 May 2025. Grog. Animated knots. https://www.animatedknots.com, 2025. Accessed: 2025-05-16. Hailuo AI. Transform idea to visual with ai.https://hailuoai.video/, 2025. Accessed 22 May 2025. Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and ma...
2025 arXiv
-
[2025]
Stanislas Dehaene.The number sense: How the mind creates mathematics
URL https://arxiv.org/abs/2501.16411. Stanislas Dehaene.The number sense: How the mind creates mathematics. OUP USA, 2011. Jonathan St B. T. Evans and Keith E. Stanovich. Dual-process theories of higher cognition: Advancing the debate. Perspectiveson psychological science, 8(3...
2011 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.