Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MMR-V, a new benchmark of 1,257 video-reasoning questions, leaves the best evaluated model at 52.5% versus humans' 86%.

desk verdict A promising video reasoning benchmark whose central claim remains unproven until the authors run the obvious control. read the letter →

arxiv 2506.04141 v2 pith:VRPCEEKM submitted 2025-06-04 cs.CV cs.CL

Kejian Zhu , Zhuoran Jin , Hongbang Yuan , Jiachun Li , Shangqing Tu , Pengfei Cao , Yubo Chen , Kang Liu
show 1 more author
Jun Zhao
This is my paper
classification cs.CVcs.CL
keywords MMR-Vbenchmarkmultimodalvideoreasoninglong-rangemulti-frameimplicitexplicitchain-of-thoughthuman-modelgapunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MMR-V, a benchmark of 1,257 multiple-choice questions over 317 videos, and tries to establish that current multimodal models cannot yet do deep video reasoning: the best model, o4-mini, scores 52.5% against 86% for humans. The benchmark is built so that a question cannot be answered from the frame it names; models must find and integrate evidence spread across distant frames, and often must interpret implicit meaning such as metaphor, theme, or intent. This matters because existing video benchmarks mostly test perception and adjacent-frame matching, which leaves open whether models can reason the way real-world applications would require. The paper also reports that chain-of-thought prompting and extra test-time compute give only small gains, and that error analysis traces most failures to lack of long-range visual reasoning rather than to the textual reasoning step.

What carries the argument

The central object is the benchmark itself, built by a three-stage pipeline. Video collection rejects linear, descriptive content and favors popular, intentionally edited videos so that every question has a defensible answer; annotation requires human annotators to watch the whole video, write questions whose answers need long-range evidence, and anchor the ground truth in the video author's or audience's consensus obtained from popular comments; distractor construction then uses three strategies, with a strong model's own verified wrong answers serving as high-difficulty distractors. The second mechanism is the CoT segmentation analysis: sampled reasoning traces are cut into ten equal segments and labeled as text analysis, question-frame analysis, or other-frame analysis, where the question frame is the specific frame named in the question. The paper shows that models with better MMR-V scores spend more of their reasoning on non-question frames, and it uses that pattern to argue that the missing capability is multi-frame evidence search rather than textual deduction.

What would settle it

Take 100 MMR-V questions and give each to a strong model in two conditions: original video frames only, and the same video with the evidence frames highlighted or paired with timestamped captions; if accuracy rises sharply in the marked condition, the benchmark is indeed testing multi-frame evidence search, and if it does not, the long-range evidence premise is not the operative difficulty. Independently, re-annotate a random 100 questions with fresh annotators who do not see popular comments; low agreement on the correct option would show the ground truth is too subjective to support the claimed human-model gap.

Watch

Extended reading notes

Core claim

MMR-V is a manually annotated multiple-choice benchmark of 1,257 tasks over 317 videos. Its defining design rule is that the question names a frame, but the correct answer cannot be read from that frame or its neighbors; the model must locate evidence frames that may come much earlier or later, integrate about 12 frames on average covering roughly 60% of the video, and often go beyond perception to infer subtext such as metaphor, theme, or the video's intent. On this benchmark the best of 20 evaluated models, o4-mini, reaches 52.5% accuracy, while humans sampled on 200 tasks reach 86%, a gap of 33.5 points. Chain-of-thought prompting improves scores by only 0.57% on average, and thinking models by about 2.4%, while adding audio improves full-modal models by about 1 point. In a sampled error analysis of GPT-4o, 47% of errors were classified as lack of visual reasoning rather than lack of knowledge or wrong deduction, which the paper reads as evidence that current models do most of their reasoning in text and only glance at question-adjacent frames.

Load-bearing premise

The benchmark assumes each question has one correct answer that most viewers would agree on, anchored by popular comments, and if individual annotators would often disagree on that answer, both model scores and the human-model gap would reflect subjectivity rather than deep reasoning.

Editorial extensions

If this is right

  • Current video benchmarks overstate multimodal reasoning: on MMR-V the strongest evaluated model trails humans by 33.5 points, even though text-reasoning models have approached human levels on text tasks.
  • Chain-of-thought prompting and longer test-time computation are not enough: average gains of roughly 0.6 points and 2.4 points show that simply thinking longer does not produce the missing visual evidence search.
  • The bottleneck is sparse evidence localization, not interpretation: models do better on implicit tasks whose cues recur across many frames and much worse on counterintuitive, sequential-structure, and comment-matching tasks that need specific distant frames.
  • More input frames help but with diminishing returns, and adding audio helps full-modal models by about one point, so scaling input alone will not close the gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if lack of visual reasoning really accounts for 47% of errors, then a model given oracle-highlighted evidence frames should jump well past 52.5%; this is a direct, testable prediction.
  • Beyond the paper: the benchmark could be turned into a diagnostic by asking models to return the timestamps of the frames they used, separating 'found the evidence' failures from 'reasoned badly from found evidence' failures.
  • Beyond the paper: because ground truth is anchored to popular YouTube comments, the answers may be cohort- and culture-dependent; re-annotating a subset with fresh annotators after some years would test whether the human baseline itself drifts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces MMR-V, a benchmark of 1,257 multiple-choice video reasoning questions over 317 videos, designed so that questions require long-range, multi-frame reasoning and cannot be answered from the question frame alone. The benchmark distinguishes implicit from explicit reasoning, uses human annotation with distractor-generation strategies, and evaluates 9 proprietary and 11 open-source models. The main empirical claims are that the best model, o4-mini, reaches only 52.5% accuracy versus 86% for humans, that chain-of-thought and test-time scaling give limited gains, and that models often fail to reason over non-question frames.

Significance. If the benchmark construction claims hold, MMR-V would be a useful contribution: it targets a real gap in video benchmarks, ships with a fairly large curated set of tasks, evaluates 20 models, and includes analyses of frame count, audio, CoT content, and error types. The manual annotation with a checklist and the three distractor strategies are sensible engineering choices, and the CoT annotation study is a constructive diagnostic. However, the validity of the central claim and of the human upper bound rests on controls that are currently missing, so the headline 52.5% versus 86% gap cannot yet be attributed specifically to deficiencies in long-range multimodal reasoning.

major comments (5)
  1. [§3.1, §3.2, Appendix B] The core claim that MMR-V questions cannot be answered from the question frame alone and require long-range, multi-frame reasoning is asserted from the annotation checklist but never tested. No experiment compares text-only input, question-frame-only input, and full-video input; the 17% GPT-4o accuracy reported in §3.2 is obtained with full-video access and therefore does not isolate the contribution of non-question frames. This control is load-bearing because, without it, the 33.5-point human-model gap could partly reflect single-frame perception, world-knowledge priors, or annotation subjectivity rather than a deficit in long-range video reasoning.
  2. [§4.1, Table 3, Appendix E.2] Model comparisons are confounded by strongly varying frame counts: LLaVA-OneVision and several open models receive 8 frames, many models receive 16 or 32 frames, and Gemini-2.0-Flash receives up to 512 frames. Because §4.3 shows that accuracy increases with frame count, the ranking in Table 3 and the scaling-law claims in §4.2 may partially reflect sampling density rather than reasoning ability. The paper should report matched-frame results across models or explicitly state this confound as a limitation.
  3. [§4.1, §4.2] The human experiment samples 100 tasks GPT-4o answered incorrectly and 100 tasks GPT-4o answered correctly rather than a random sample, and no inter-annotator agreement is reported. Since ground-truth answers are human-generated, the 86% human score partly re-measures the annotation process rather than an independent ceiling. Report agreement statistics (e.g., Cohen's kappa or percentage agreement) and, ideally, evaluate humans on a random or stratified sample so the upper bound is not selected for model disagreement.
  4. [Abstract and §4.2] The abstract states that the best-performing model, Gemini-2.5-pro, achieves 64.3% accuracy, while the main text and Table 3 report o4-mini at 52.5% as the best. This is a factual inconsistency in the central result; the abstract must be corrected to match the reported experiments.
  5. [Table 3 and §4.2] No error bars, confidence intervals, or significance tests are reported for any of the accuracy differences. Several comparisons that drive the conclusions are small (e.g., +1.4%, +1.0%, +1.0% in Table 4, and the 2.4% thinking-model gain in §4.2), and without variance estimates these differences may not be distinguishable from noise. Bootstrap or per-task variance should be added, at least for the main claims.
minor comments (6)
  1. [§3.2, Table 1] The relationship between the 17% GPT-4o direct-answer accuracy and the 59% GPT-4o accuracy in the Strategy 1 column of Table 1 should be clarified; the current text suggests two different numbers for the same 100 questions.
  2. [Table 3] The table formatting is damaged: the 'Best Performance of Models' row is misaligned with the column headers, and the bold/underline conventions for open-source versus proprietary models are not applied consistently in the sub-columns.
  3. [Figures 3, 4, 6] The rendered figure text contains numerous garbled Unicode tokens (e.g., '/uni00000014/uni00000013...'), making the figures unreadable in the manuscript. The source figures should be regenerated.
  4. [Appendix F] The color key for the CoT analysis is confusing: yellow, red, blue, and green are introduced with overlapping definitions ('Red, blue, and green all represent video analysis' after red and blue were separately defined). This should be simplified and made consistent.
  5. [Appendix D.2, VTI example] The example for Video Intent contains duplicate answer choices (2023 appears twice as options E and G), which undermines the example's clarity.
  6. [§4.5] The CoT content analysis uses GPT-4.1 to label segments, but no validation of the automatic labeling is reported; a small human agreement check on the labels would strengthen the error analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MMR-V's claims are empirical benchmark measurements, not derivations from fitted inputs or self-cited premises.

full rationale

This paper introduces a benchmark and reports experimental results; it contains no derivation chain in which a predicted quantity is constructed from the same data it claims to predict. The central claims—model accuracies, the human-model gap, and the limited gains from CoT and test-time compute—are direct empirical measurements under a stated evaluation protocol. The ground-truth answers are human-annotated by design, so the human 86% accuracy is partly a consistency check of that annotation process rather than an independent external ceiling, but the paper does not present the human score as a prediction derived from the annotation inputs; it is a separate measurement. The use of GPT-4o's incorrect answers as distractors is a transparent construction choice that may bias difficulty for GPT-4o specifically, but it is not a fitted parameter renamed as a prediction, and all evaluated models receive the same option sets. There are no load-bearing self-citations, no imported 'uniqueness theorems,' and no ansatz smuggled in via citation. The absence of a question-frame-only or text-only baseline weakens the interpretation that failures are due to long-range multimodal reasoning, but a missing control is an empirical-evidence concern, not circularity. The paper's own limitations section acknowledges annotation cost, category coverage, and language skew without concealing a circular step. Therefore, no specific circular step can be quoted, and the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark's validity rests on assumptions about the taxonomy, grounding of correct answers, and annotation reliability. No free parameters are fitted to data, but the frame sampling choices are hand-selected and affect results.

free parameters (1)
  • Input frame count per model = 8, 16, 32, or 512 depending on model
    The number of frames sampled per video is chosen by the authors per model (Appendix E.2). This directly affects accuracy (Gemini-2.0-Flash 16 vs 512 frames: 42.6 vs 48.0), so it is a hand-selected quantity that influences results.
assumptions (4)
  • domain assumption Implicit/explicit reasoning dichotomy from Dual Process Theory maps cleanly onto video reasoning tasks
    Used in Section 2.1 to define task categories; if the taxonomy is not exclusive or exhaustive, benchmark construct validity suffers.
  • domain assumption Popular comments and official video interpretations are reliable ground truth for correct answers
    Sections 3.1 and B.1 instruct annotators to use highly praised comments; no independent verification that these represent common understanding.
  • domain assumption Multiple-choice format with roughly 10 options can measure deep reasoning rather than test-taking skill
    Section 3.2; no analysis that elimination or guessing behavior is absent.
  • domain assumption Human annotators with bachelor's degrees provide accurate labels
    Section 3.2; no inter-annotator agreement reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos." pith.science (2026). https://pith.science/paper/VRPCEEKM

@misc{pith2026250604141,
  author       = {Pith},
  title        = {Pith review of: MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VRPCEEKM}},
  note         = {Machine review of arXiv:2506.04141}
}
read the original abstract

The sequential structure of videos poses a challenge to the ability of multimodal large language models (MLLMs) to locate multi-frame evidence and conduct multimodal reasoning. However, existing video benchmarks mainly focus on understanding tasks, which only require models to match frames mentioned in the question (hereafter referred to as "question frame") and perceive a few adjacent frames. To address this gap, we propose MMR-V: A Benchmark for Multimodal Deep Reasoning in Videos. The benchmark is characterized by the following features. (1) Long-range, multi-frame reasoning: Models are required to infer and analyze evidence frames that may be far from the question frame. (2) Beyond perception: Questions cannot be answered through direct perception alone but require reasoning over hidden information. (3) Reliability: All tasks are manually annotated, referencing extensive real-world user understanding to align with common perceptions. (4) Confusability: Carefully designed distractor annotation strategies to reduce model shortcuts. MMR-V consists of 317 videos and 1,257 tasks. Our experiments reveal that current models still struggle with multi-modal reasoning; even the best-performing model, Gemini-2.5-pro, achieves only 64.3% accuracy. Additionally, current reasoning enhancement strategies (Chain-of-Thought and scaling test-time compute) bring limited gains. Error analysis indicates that the CoT demanded for multi-modal reasoning differs from it in textual reasoning, which partly explains the limited performance gains. We hope that MMR-V can inspire further research into enhancing multi-modal reasoning capabilities.

Figures

Figures reproduced from arXiv: 2506.04141 by the authors.

Figure 1
Figure 1. Examples showing the MMR-V tasks and the difference from previous video benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of six tasks in MMR-V Bench. Explicit Reasoning evaluates whether a model can perform reasoning based on multimodal details explicitly presented across long-range, multi-frame of a video. However, solving these tasks demands fine-grained perception and rigorous logical reasoning. “(explicit) allocates attention to the effortful mental activities that demand it, including complex computations.” - Dual Proces… view at source ↗
Figure 3
Figure 3. Performance on different tasks. Firstly, the models performed better on implicit tasks than on explicit tasks (with an average gain of +7.9%). Through analysis of tasks and model responses, we found that in implicit tasks, video cre￾ators often embed implicit meanings throughout the entire video, resulting in abundant visual cues that can support reasoning. This reduces the requirements for multi-modal reasoning and… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Accuracy with the increase of input frame counts. For Gemini-2.0-Flash, which supports long video in￾puts, we evaluated performance changes as the number of frames increases. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Error analysis of GPT-4o. We sampled 100 incorrect responses from GPT-4o for error analysis. The main sources of errors can be cate￾gorized as follows: (1) Lack of Visual Reasoning: the model often failed to locate the correct evidence frames and lack of long-range, mu…
Figure 6
Figure 6. Figure 6: CoT content across different stages. The y-axis indicates the ratio of the 500 sampled CoTs [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: MMR-V Construction Pipeline. A Limitations Despite our efforts to improve our work, several limitations remain. (1) Scaling MMR-V is challeng￾ing due to the high cost of manual annotation and verification, as all tasks and correct answers are curated and reviewed by hu…
Figure 8
Figure 8. Figure 8: Annotation Platform of MMR-V. Art Animation Film Life Philosophy TV Dance Music MV Stage Play Photography Social Issues Daily Theme Personification History Comedy Science Fiction Short Film Classic Humor Short Video Travel Anti-Cut Editing Concept Intro Self-Reflection…
Figure 9
Figure 9. Figure 9: (a) Video categories in MMR-V Bench. (b) Proportion of different tasks in MMR-V Bench. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: CoT example of experiments in Section 4.5. Yellow, red, and blue represent text, question frame, and other frame analysis respectively. Red, blue, and green all represent video analysis. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]
Figure 11
Figure 11. Figure 11: A comparison of CoTs from two models on the same task. Yellow and green indicate text [PITH_FULL_IMAGE:figures/full_fig_p033_11.png]
Figure 12
Figure 12. Figure 12: Error Case: Lack of Visual Reasoning. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_12.png]
Figure 13
Figure 13. Figure 13: Error Case: Lack of Visual Reasoning. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_13.png]
Figure 14
Figure 14. Figure 14: Error Case: Implicit Misinterpretation. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_14.png]
Figure 15
Figure 15. Figure 15: Error Case: Reasoning Error. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_15.png]
Figure 16
Figure 16. Figure 16: MMR-V Construction Pipeline. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.

Reference graph

Works this paper leans on

60 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Openai o1 system card,

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al., “Openai o1 system card,” arXiv preprint arXiv:2412.16720, 2024

    arXiv 2024

  2. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi,et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  3. [3]

    Multimodal chain-of-thought reasoning: A comprehensive survey,

    Y . Wang, S. Wu, Y . Zhang, S. Yan, Z. Liu, J. Luo, and H. Fei, “Multimodal chain-of-thought reasoning: A comprehensive survey,” arXiv preprint arXiv:2503.12605, 2025

    arXiv 2025

  4. [4]

    Openai: Introducing openai o3 and o4-mini,

    OpenAI, “Openai: Introducing openai o3 and o4-mini,” 2025

    work page 2025

  5. [5]

    Research of intelligent home secu- rity surveillance system based on zigbee,

    J. Hou, C. Wu, Z. Yuan, J. Tan, Q. Wang, and Y . Zhou, “Research of intelligent home secu- rity surveillance system based on zigbee,” in 2008 International Symposium on Intelligent Information Technology Application Workshops, pp. 554–557, IEEE, 2008

    work page 2008

  6. [6]

    Thinking in space: How multimodal large language models see, remember, and recall spaces,

    J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie, “Thinking in space: How multimodal large language models see, remember, and recall spaces,” arXiv preprint arXiv:2412.14171, 2024

    arXiv 2024

  7. [7]

    Mlvu: A comprehensive benchmark for multi-task long video understanding,

    J. Zhou, Y . Shu, B. Zhao, B. Wu, S. Xiao, X. Yang, Y . Xiong, B. Zhang, T. Huang, and Z. Liu, “Mlvu: A comprehensive benchmark for multi-task long video understanding,” arXiv preprint arXiv:2406.04264, 2024

    arXiv 2024

  8. [8]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,

    C. Fu, Y . Dai, Y . Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y . Shen, M. Zhang,et al., “Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis,” arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

Show all 60 references
  1. [9]

    Heuristic and analytic processes in reasoning,

    J. S. B. Evans, “Heuristic and analytic processes in reasoning,” British Journal of Psychology, vol. 75, no. 4, pp. 451–468, 1984

    1984

  2. [10]

    The clarion cognitive architecture: Extending cognitive modeling to social simulation,

    R. Sun, “The clarion cognitive architecture: Extending cognitive modeling to social simulation,” Cognition and multi-agent interaction, pp. 79–99, 2006

    2006

  3. [11]

    Polanyi, Personal knowledge

    M. Polanyi, Personal knowledge. Routledge, 2012

    2012

  4. [12]

    Kahneman, Thinking, fast and slow

    D. Kahneman, Thinking, fast and slow . macmillan, 2011

    2011

  5. [13]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts,” arXiv preprint arXiv:2310.02255, 2023

    arXiv

  6. [14]

    Measuring multimodal mathematical reasoning with math-vision dataset,

    K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li, “Measuring multimodal mathematical reasoning with math-vision dataset,” Advances in Neural Information Processing Systems, vol. 37, pp. 95095–95169, 2024

    2024

  7. [15]

    Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks,

    F. Zhang, L. Wu, H. Bai, G. Lin, X. Li, X. Yu, Y . Wang, B. Chen, and J. Keung, “Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks,” arXiv preprint arXiv:2410.12381, 2024

    arXiv

  8. [16]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  9. [17]

    Chain-of- thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou,et al., “Chain-of- thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24824–24837, 2022

    2022

  10. [18]

    RankGen: Improving text generation with large ranking models,

    K. Krishna, Y . Chang, J. Wieting, and M. Iyyer, “RankGen: Improving text generation with large ranking models,” in Proceedings of EMNLP, pp. 199–232, 2022

    2022

  11. [19]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” arXiv preprint arxiv:2303.08774, 2023. 10

    2023 arXiv

  12. [20]

    How good is my video lmm? complex video reasoning and robustness evaluation suite for video-lmms,

    M. U. Khattak, M. F. Naeem, J. Hassan, M. Naseer, F. Tombari, F. S. Khan, and S. Khan, “How good is my video lmm? complex video reasoning and robustness evaluation suite for video-lmms,” arXiv preprint arXiv:2405.03690, 2024

    2024 arXiv

  13. [21]

    Egoschema: A diagnostic benchmark for very long-form video language understanding,

    K. Mangalam, R. Akshulakov, and J. Malik, “Egoschema: A diagnostic benchmark for very long-form video language understanding,” Advances in Neural Information Processing Systems , vol. 36, pp. 46212–46244, 2023

    2023

  14. [22]

    Perception test: A diagnostic benchmark for multimodal video models,

    V . Patraucean, L. Smaira, A. Gupta, A. Recasens, L. Markeeva, D. Banarse, S. Koppula, M. Malinowski, Y . Yang, C. Doersch, et al., “Perception test: A diagnostic benchmark for multimodal video models,” Advances in Neural Information Processing Systems , vol. 36, pp. 42748–42761, 2023

    2023

  15. [23]

    Next-qa: Next phase of question-answering to explaining temporal actions,

    J. Xiao, X. Shang, A. Yao, and T.-S. Chua, “Next-qa: Next phase of question-answering to explaining temporal actions,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777–9786, 2021

    2021

  16. [24]

    Video question answering via gradually refined attention over appearance and motion,

    D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y . Zhuang, “Video question answering via gradually refined attention over appearance and motion,” in Proceedings of the 25th ACM international conference on Multimedia, pp. 1645–1653, 2017

    2017

  17. [25]

    Msr-vtt: A large video description dataset for bridging video and language,

    J. Xu, T. Mei, T. Yao, and Y . Rui, “Msr-vtt: A large video description dataset for bridging video and language,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288–5296, 2016

    2016

  18. [26]

    Mvbench: A comprehensive multi-modal video understanding benchmark,

    K. Li, Y . Wang, Y . He, Y . Li, Y . Wang, Y . Liu, Z. Wang, J. Xu, G. Chen, P. Luo, et al. , “Mvbench: A comprehensive multi-modal video understanding benchmark,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 22195–22206, 2024

    2024

  19. [27]

    Mmbench-video: A long- form multi-shot benchmark for holistic video understanding,

    X. Fang, K. Mao, H. Duan, X. Zhao, Y . Li, D. Lin, and K. Chen, “Mmbench-video: A long- form multi-shot benchmark for holistic video understanding,” Advances in Neural Information Processing Systems, vol. 37, pp. 89098–89124, 2024

    2024

  20. [28]

    Lvbench: An extreme long video understanding benchmark,

    W. Wang, Z. He, W. Hong, Y . Cheng, X. Zhang, J. Qi, X. Gu, S. Huang, B. Xu, Y . Dong,et al., “Lvbench: An extreme long video understanding benchmark,” arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  21. [29]

    Longvideobench: A benchmark for long-context interleaved video-language understanding,

    H. Wu, D. Li, B. Chen, and J. Li, “Longvideobench: A benchmark for long-context interleaved video-language understanding,” Advances in Neural Information Processing Systems , vol. 37, pp. 28828–28857, 2024

    2024

  22. [30]

    Kimi k1. 5: Scaling reinforcement learning with llms,

    K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al., “Kimi k1. 5: Scaling reinforcement learning with llms,” arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  23. [31]

    Marco-o1: Towards open reasoning models for open-ended solutions,

    Y . Zhao, H. Yin, B. Zeng, H. Wang, T. Shi, C. Lyu, L. Wang, W. Luo, and K. Zhang, “Marco-o1: Towards open reasoning models for open-ended solutions,”arXiv preprint arXiv:2411.14405, 2024

    2024 arXiv

  24. [32]

    Measuring mathematical problem solving with the math dataset,

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Stein- hardt, “Measuring mathematical problem solving with the math dataset,” arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  25. [33]

    Long- bench v2: Towards deeper understanding and reasoning on realistic long-context multitasks,

    Y . Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y . Dong,et al., “Long- bench v2: Towards deeper understanding and reasoning on realistic long-context multitasks,” arXiv preprint arXiv:2412.15204, 2024

    2024 arXiv

  26. [34]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” ArXiv preprint, vol. abs/2110.14168, 2021

    arXiv

  27. [35]

    Gpqa: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “Gpqa: A graduate-level google-proof q&a benchmark,” in First Conference on Language Modeling, 2024. 11

    2024

  28. [36]

    Mmlu-pro: A more robust and challenging multi-task language understanding bench- mark,

    Y . Wang, X. Ma, G. Zhang, Y . Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, et al., “Mmlu-pro: A more robust and challenging multi-task language understanding bench- mark,” in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Ben...

    2024

  29. [37]

    Swe-bench: Can language models resolve real-world github issues?,

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?,” arXiv preprint arXiv:2310.06770 , 2023

    2023 arXiv

  30. [38]

    Math-llava: Bootstrapping mathematical reasoning for multimodal large language models,

    W. Shi, Z. Hu, Y . Bin, J. Liu, Y . Yang, S.-K. Ng, L. Bing, and R. K.-W. Lee, “Math-llava: Bootstrapping mathematical reasoning for multimodal large language models,” arXiv preprint arXiv:2406.17294, 2024

    2024 arXiv

  31. [39]

    Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi,

    K. Ying, F. Meng, J. Wang, Z. Li, H. Lin, Y . Yang, H. Zhang, W. Zhang, Y . Lin, S. Liu,et al., “Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi,” arXiv preprint arXiv:2404.16006, 2024

    2024 arXiv

  32. [40]

    Lakoff and M

    G. Lakoff and M. Johnson, Metaphors we live by . University of Chicago press, 2008

    2008

  33. [41]

    Openai: Hello gpt-4o,

    OpenAI, “Openai: Hello gpt-4o,” 2024

    2024

  34. [42]

    Gpt-4o mini: advancing cost-efficient intelligence,

    OpenAI, “Gpt-4o mini: advancing cost-efficient intelligence,” 2024

    2024

  35. [43]

    Introducing gpt-4.1 in the api

    OpenAI, “Introducing gpt-4.1 in the api.” https://openai.com/index/gpt-4-1/ , 2025

    2025

  36. [44]

    Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context,

    M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al., “Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  37. [45]

    Gemini 2.5: Our most intelligent ai model,

    Google DeepMind, “Gemini 2.5: Our most intelligent ai model,” March 2025

    2025

  38. [46]

    Anthropic: Introducing claude 3.5 sonnet,

    Anthropic, “Anthropic: Introducing claude 3.5 sonnet,” 2024

    2024

  39. [47]

    Qwen2.5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y ...

    2024 arXiv

  40. [48]

    Gemma 3 technical report,

    A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. Busa...

    2025 arXiv

  41. [49]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, Y . Duan, H. Tian, W. Su, J. Shao, et al. , “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,” arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  42. [50]

    Llava-onevision: Easy visual task transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y . Li, Z. Liu,et al., “Llava-onevision: Easy visual task transfer,” arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  43. [51]

    Video-llava: Learning united visual representation by alignment before projection,

    B. Lin, Y . Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan, “Video-llava: Learning united visual representation by alignment before projection,” arXiv preprint arXiv:2311.10122, 2023. 12

    2023 arXiv

  44. [52]

    Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras,

    A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V . Chaudhary, C. Chen, D. Chen, D. Chen, J. Chen, W. Chen, Y . Chen, Y . Chen, Q. Dai, X. Dai, R. Fan, M. Gao, M. Gao, A. Garg, A. Goswami, J. Hao, A. Hendy, Y . Hu, X. Jin, M. Khademi, D...

    2025 arXiv

  45. [53]

    Cogvlm2: Visual language models for image and video understanding,

    W. Hong, W. Wang, M. Ding, W. Yu, Q. Lv, Y . Wang, Y . Cheng, S. Huang, J. Ji, Z. Xue, et al., “Cogvlm2: Visual language models for image and video understanding,” arXiv preprint arXiv:2408.16500, 2024

    2024 arXiv

  46. [54]

    Nvila: Efficient frontier visual language models,

    Z. Liu, L. Zhu, B. Shi, Z. Zhang, Y . Lou, S. Yang, H. Xi, S. Cao, Y . Gu, D. Li,et al., “Nvila: Efficient frontier visual language models,” arXiv preprint arXiv:2412.04468, 2024. 13

    2024 arXiv

  47. [55]

    Watch the video and answer the question and give a correct answer

    Video Collection Reject Checklist (1) High recognition interpretation? (2) Is it Non- straightforward?NO YES Video Pool Visuals + Audios 2.Data Annotation Checklist (1) Is long-span, multi-frame reasoning required? (2) Is a deeper understanding of video required ? The answer c...

  48. [56]

    High recognition interpretation?

    Quality Assurance ManuallyQualityCheck Q: Why did the broken umbrella in the boy's hand, damaged by the wind, get fixed while in the air?... (B) Because the boy held the girl's hand, gained good luck, and as a result, the umbrella was restored. ... (H) The video was reversed d...

    2023

  49. [57]

    ‘other frame desc‘: Does this chunk describe visual information from frames other than question frame?

  50. [58]

    ‘question frame desc‘: Does this chunk correctly describe visual information from the question frame specified in the question?

  51. [59]

    ‘video analysis‘: Does this chunk perform analysis of the video content?

  52. [60]

    other frame desc

    ‘text analysis‘: Does this chunk perform analysis of the text (e.g., question text, options) rather than visual content - The question frame refers to the specific frame(s) referenced by the question prompt. - Other-frame descriptions are visual details not present in the ques...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.