Pith. sign in

REVIEW 5 major objections 4 minor 3 cited by

VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VRBench is presented as the first long-narrative-video benchmark that grades both the final answer and the multi-step reasoning chain behind it, using 960 videos and 8,243 human-labeled questions.

desk verdict VRBench is a genuinely useful benchmark artifact with human-authored multi-step reasoning chains; the process-level metric needs revalidation on VLM outputs before its headline comparisons carry weight. read the letter →

arxiv 2506.10857 v2 pith:CBOPKMGA submitted 2025-06-12 cs.CV cs.AIcs.MM

Jiashuo Yu , Yue Wu , Meng Chu , Zhifei Ren , Zizheng Huang , Pei Chu , Ruijie Zhang , Yinan He
show 9 more authors
Qirui Li Songze Li Zhenxiang Li Zhongying Tu Conghui He Yu Qiao Yali Wang Yi Wang Limin Wang
This is my paper
classification cs.CVcs.AIcs.MM
keywords multi-stepreasoninglong-formvideounderstandingvision-languagemodelsbenchmarkconstructionchain-of-thoughtevaluationLLM-as-a-judgetemporalnarrativeQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VRBench sets out to measure something earlier video benchmarks skip: whether large models can reason through a long narrative — a film, a sports match, a travelogue — in several temporally grounded steps rather than merely recognize a single scene. To enable that measurement, the authors assemble 960 curated videos averaging 1.6 hours, with 8,243 human-written questions, each backed by a chain of timestamped reasoning steps and tagged with one of seven reasoning types such as event attribution and implicit inference. Models are then graded twice: once on a multiple-choice final answer, and once on the quality of their stated reasoning chain as judged by an LLM. The design matters because the two grades diverge — GPT-4o answers 81.23% of questions correctly yet its reasoning chains score only 56.13% — indicating that outcome-only benchmarks overstate how well models understand long stories.

What carries the argument

The central object is the benchmark itself: 960 expert-filtered narrative videos and 8,243 QA pairs, each carrying a multi-step reasoning chain with per-step video timestamps and a label from a seven-category reasoning taxonomy. The argument is carried by the two-phase evaluation pipeline. In phase one, models answer a multiple-choice question, and only the selected option is scored. In phase two, a judge LLM — DeepSeek-V3, chosen because its ratings correlate with human experts at Spearman $\rho > 0.8$ on a 300-question calibration subset — grades the model's written chain on four weighted dimensions: logical coherence (40%), similarity to the ground-truth chain (40%, dropped for prediction and hypothetical questions), factual accuracy (10%), and clarity (10%). The process score is what makes VRBench different from prior video benchmarks, which the paper argues grade only perception or single-step answers.

What would settle it

Have human raters score a large stratified sample of the 8,243 reasoning chains under the same four-dimension rubric and compare the resulting model ranking with the paper's; if the rankings diverge sharply, the process metric is measuring judge preferences rather than reasoning quality. A complementary check is to collect human accuracy on the multiple-choice questions — if people do not clearly beat the best model's 74.61%, the questions are not calibrated as reasoning tests.

Watch

Extended reading notes

Core claim

The paper's central claim is that multi-step reasoning in long narrative video is a distinct, currently unmeasured capability, and that VRBench measures it. Each of the 8,243 questions is anchored to a human-reviewed chain of at least two timestamped reasoning steps, with the full suite containing 25,106 steps; the seven-type taxonomy (event attribution, hypothetical reasoning, event prediction, implicit inference, logical linkage, information synopsis, counting problems) defines what counts as a reasoning task. Evaluation runs in two phases: models first produce a free-form chain of thought, then select among multiple-choice options built from human answers plus machine-generated distractors; a judge model separately scores the chain on logical coherence, similarity to ground truth, factual accuracy, and clarity. On this instrument the paper reports a ranking of 31 models in which Gemini-2.0-Pro leads at 74.61% overall accuracy, and several models show the signature split between correct answers and weak reasoning processes. The paper interprets this as evidence that current large models can land on right answers without sound reasoning, and that process-level scoring is required to expose the difference.

Load-bearing premise

The process-level rankings stand on the assumption that the judge model's scores match human judgment across all 8,243 questions, even though that alignment was measured on only 300 questions and the judge, DeepSeek-V3, is itself one of the models being compared.

Editorial extensions

If this is right

  • Ranking models by multiple-choice accuracy alone will overstate narrative reasoning: GPT-4o answers 81.23% of questions correctly but its reasoning chains are scored at only 56.13%, so outcome and process can diverge sharply.
  • Dense visual grounding matters more than raw scale for long-video reasoning: long-context proprietary VLMs beat text-only LLMs by 13.82% absolute, and open-source VLMs trail proprietary ones by 12.30% at comparable scale.
  • Test-time scaling pays off only for reasoning-tuned models: raising QwQ-32B's token budget lifts its score from 48.91% to 61.34%, while a small system-1 VLM gets worse under the same treatment.
  • Counting problems, which demand fine-grained visual perception, are the hardest question type for nearly all tested models, many hovering near random-guess accuracy.
  • VRBench is released as an open instrument — videos, annotations, prompts, and evaluation tools — so future models can be ranked on both outcome and process against the reported leaderboard.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The process ranking likely depends partly on the choice of judge: DeepSeek-V3 is one of the evaluated models and also writes the multiple-choice distractors, so re-running the process scoring with a second judge and comparing rank stability would show how much of the ranking is judge-specific.
  • The 25,106 timestamped reasoning chains could be used directly as supervision: training a model to emit chains that match the human ground truth before selecting an answer is a natural next use the paper does not explore.
  • Because all non-English footage is answered through translated subtitles, a model's score may partly reflect translation loss; an English-language subset would test whether the benchmark measures video reasoning or cross-lingual transcript comprehension.
  • The outcome-versus-process gap is itself a diagnostic signal: a model scoring high on accuracy but low on process may be exploiting question wording or distractor statistics, which a distractors-only ablation could test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. VRBench is a benchmark for multi-step reasoning in long narrative videos. It comprises 960 human-filtered videos (8 languages, average 1.6 hours) with 8,243 QA pairs and 25,106 timestamped reasoning steps, annotated through a human-AI collaborative pipeline covering seven reasoning types. Evaluation is two-stage: an outcome-level multiple-choice stage and a process-level stage in which an LLM judge scores the model's reasoning chain along four dimensions. The paper reports experiments on 12 LLMs and 19 VLMs, finding outcome/process gaps such as GPT-4o's 81.23 outcome versus 56.13 process score, and analyzes test-time scaling and long-context effects. The dataset and evaluation tools are released.

Significance. If the claims are borne out, VRBench fills a real gap: existing long-video benchmarks largely test single-step perception or outcome-only accuracy, whereas VRBench provides human-reviewed step-level annotations with timestamps and a process-level metric. The curation protocol is detailed, inter-rater reliability is reported, and the open-sourcing of annotations, prompts, and evaluation tools makes it a reusable resource for long-video reasoning research. The main caveats are that the process-level metric is validated on only four LLMs and 300 questions, the judge model is also an evaluated model and generates the MCQ distractors, and the 'multi-step' description admits a nontrivial number of single-step items. These issues do not invalidate the dataset, but they weaken the strongest claims made in the abstract and Section 4.

major comments (5)
  1. [Section 3.2 and Supplementary I.1.3] The abstract and introduction describe 8,243 'human-labeled multi-step question-answering pairs' and state that each question requires no less than 2 reasoning steps, but the annotation protocol explicitly permits up to two single-step reasoning questions per video and the 'information synopsis' type is single-step by design. With 960 videos, this allowance could make a substantial fraction of the 8,243 items single-step (up to roughly 35%). The authors should report the actual distribution of reasoning-step counts in the released dataset and revise the abstract and Section 1 wording so that 'multi-step' describes the benchmark design rather than every QA pair.
  2. [Section 4.2, Figure 4, Table 2] The human-alignment study that justifies the DeepSeek-V3 process judge covers only outputs of four LLMs (GPT-4o, DeepSeek-V3, Qwen2.5-7B, Qwen2.5-72B) on 30 videos and 300 questions. VLM outputs are not included in this validation, yet the paper's headline process-level findings, including the GPT-4o outcome/process gap and the claim of 'VLM reasoning fragility,' rely on process scores assigned to all 19 VLMs in Table 2. Because VLM reasoning chains are often shorter, less verbal, or grounded differently than LLM chains, judge-human agreement may not transfer. The authors should validate the judge on a human-rated sample of VLM outputs before presenting VLM process-level rankings as a central result.
  3. [Section 3.3 and Table 2] DeepSeek-V3 is used as the process judge, as the generator of MCQ distractors, and as one of the evaluated LLMs in Table 2. This creates two related risks: its own process scores are self-ratings, and its MCQ performance may be advantaged by the fact that it generated the distractor options. The paper does not analyze either effect. The authors should either use a judge that is not an evaluated model or provide a sensitivity analysis (for example, re-scoring a subset of DeepSeek-V3 outputs with a different judge) to show that the reported ranks are not driven by this overlap.
  4. [Section 3.3 versus Figure A21] The process metric as specified in the main text uses weights 40/40/10/10 on logical coherence, similarity to ground truth, factual accuracy, and clarity, with similarity excluded for event-prediction and hypothetical-reasoning tasks. The judge prompt in Figure A21, however, names different dimensions (Step Matching, Logical Consistency, Factual Accuracy, Process Clarity) and does not state these weights or the exclusion rule. Since every process-level number in Table 2 is obtained through this judge prompt, the exact deployed prompt must be released and must match the metric specification; otherwise the reported process scores are not reproducible.
  5. [Table 1 and Section 2] The paper claims VRBench is 'the first narrative video benchmark crafted for evaluating large models' multi-step reasoning capabilities,' but Table 1 itself lists CGBench as having multi-step annotation and multi-step evaluation for long videos. The authors should either demonstrate concretely how CGBench differs on the axes that matter here (for example, step-level timestamp grounding, process-level scoring, or purely narrative content) or qualify the 'first' claim accordingly.
minor comments (4)
  1. [Abstract] The abstract refers to a 'progress-level LLM-guided scoring metric,' which appears to be a typo for 'process-level'; please correct it.
  2. [Figure 4 caption] The caption says 'three different tested LLMs' but the figure includes four conditions: DeepSeek-V3, GPT-4o, Qwen2.5-72B, and the 'DeepSeek w/o sep' variant. The caption should be updated to avoid ambiguity.
  3. [Section 4.3 and Introduction] The test-time scaling experiment is run on a 300-video subset, and the introduction cites a 12.43% accuracy improvement for QwQ-32B, but the main text does not consistently state whether this refers to outcome-level MCQ accuracy or the combined overall score. Please clarify the metric in both places.
  4. [Section 4.1] No human accuracy baseline is reported for the MCQ stage, which would help readers interpret whether outcome-level scores such as 81.23% represent strong or weak performance relative to a human ceiling.

Circularity Check

0 steps flagged · score 0.0 of 10

VRBench's human-annotated QA core and deterministic MCQ accuracy are self-contained; the LLM-judge process metric has validation and consistency gaps, but no derivation step reduces to its own inputs.

full rationale

VRBench is a dataset-and-benchmark paper, not a predictive derivation, and its central artifact is independently grounded. The 960 videos are curated by multilingual domain experts with explicit exclusion criteria and a rating scale (Sections 3.1 and H.1); the 8,243 QA pairs and 25,106 reasoning steps are produced by 67 graduate-student annotators and checked by 10 trained reviewers with a 5% audit (Sections 3.2 and J). GPT-4o's pseudo pre-annotations are only contextual hints that annotators are instructed to rewrite, so the benchmark content does not reduce to an LLM's output. The outcome-level metric is deterministic MCQ accuracy against human-labeled answers; the fact that DeepSeek-V3 generated the distractor options is an experimental confound for DeepSeek-V3's own row in Table 2, but it is not a circular derivation of the benchmark's claims. The process-level metric uses DeepSeek-V3 as an LLM judge, and Section 4.2 does report human-alignment evidence: Spearman correlations above 0.8 on a 30-video, 300-question subset. That validation covers only four LLMs (GPT-4o, DeepSeek-V3, Qwen2.5-7B, Qwen2.5-72B) and does not include the 19 VLMs whose process ratings drive several headline comparisons; extrapolating judge reliability to VLM reasoning traces is an unvalidated assumption, hence a correctness risk rather than circularity. The same is true of the inconsistency between the judge prompt in Section 3.3 (40/40/10/10 weights, event-prediction exclusion) and the prompt shown in Figure A21 (different dimension names, no weights stated), which threatens metric reproducibility but does not make the metric equal to its input. DeepSeek-V3 also appears as an evaluated model in Table 2, so its own OE-P score is partially self-rated, but this single row is not load-bearing for the paper's main conclusions and is not a fitted parameter renamed as a prediction. Citations to the authors' prior work (e.g., VBench, CGBench, MVBench, InternVideo) are methodological or related-work references and are not used as the sole justification for any central claim. No equation, fitted parameter, or definition is shown to be equivalent to the paper's inputs by construction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

VRBench is a benchmark artifact, so the ledger consists mainly of design choices: scoring weights, suitability thresholds, temporal constraints, and the single-step cap, plus domain assumptions about annotation uniqueness, judge transferability, distractor quality, and subtitle grounding. No new physical or mathematical entities are introduced.

free parameters (4)
  • Process metric weights = logical coherence 0.4, similarity 0.4, factual accuracy 0.1, clarity 0.1
    Section 3.3 defines the final process score with these hand-set weights; no ablation or data-driven justification is provided.
  • Video suitability threshold = >=7 on a 10-point scale
    Section 3.1 uses a fixed cutoff for expert ratings; changing it changes the dataset composition and question difficulty.
  • Temporal distribution constraints = <=4 questions from 0-15min, >=3 from 15-40min, >=1 from 40-120min
    Section 3.2 imposes these counts to shape time-span coverage; they are design choices, not derived from data.
  • Single-step question cap = no more than 2 single-step questions per video, plus 1 information synopsis
    Supplementary Section I.1.3 allows single-step questions despite the paper's multi-step framing; this cap is a hand-chosen design parameter.
assumptions (5)
  • domain assumption Ground-truth reasoning chains are unique and factually correct.
    The process metric's similarity-to-ground-truth component assumes the annotated chain is the only valid chain; annotation guidelines enforce uniqueness subjectively via the reasoning process uniqueness judgment in Supplementary I.1.4.
  • domain assumption LLM judge alignment on a 300-question subset transfers to the full 8,243 questions and all models.
    Section 4.2 validates DeepSeek-V3 against human raters on 30 videos and 300 questions, then applies it to the entire benchmark without evidence that alignment holds across the full distribution.
  • domain assumption DeepSeek-V3-generated distractors are plausible and do not favor any particular model.
    Section 3.3 uses DeepSeek-V3 to write false multiple-choice options; distractor quality is not checked against human judgments or against possible self-recognition by DeepSeek-V3.
  • domain assumption Translated subtitles and captions preserve enough information for reasoning while answers cannot be derived from subtitles alone.
    The annotation standard in Supplementary I.1.3 requires multimodal grounding, but this is enforced by annotator judgment and not externally measured.
  • ad hoc to paper Video selection via expert ratings with a threshold of 7 yields a representative set of narrative videos suitable for multi-step reasoning.
    Section 3.1 introduces a 10-point expert scale and a cutoff of 7 with category-specific criteria; both the scale and cutoff are constructed for this benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos." pith.science (2026). https://pith.science/paper/CBOPKMGA

@misc{pith2026250610857,
  author       = {Pith},
  title        = {Pith review of: VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CBOPKMGA}},
  note         = {Machine review of arXiv:2506.10857}
}
read the original abstract

We present VRBench, the first long narrative video benchmark crafted for evaluating large models' multi-step reasoning capabilities, addressing limitations in existing evaluations that overlook temporal reasoning and procedural validity. It comprises 960 long videos (with an average duration of 1.6 hours), along with 8,243 human-labeled multi-step question-answering pairs and 25,106 reasoning steps with timestamps. These videos are curated via a multi-stage filtering process including expert inter-rater reviewing to prioritize plot coherence. We develop a human-AI collaborative framework that generates coherent reasoning chains, each requiring multiple temporally grounded steps, spanning seven types (e.g., event attribution, implicit inference). VRBench designs a multi-phase evaluation pipeline that assesses models at both the outcome and process levels. Apart from the MCQs for the final results, we propose a progress-level LLM-guided scoring metric to evaluate the quality of the reasoning chain from multiple dimensions comprehensively. Through extensive evaluations of 12 LLMs and 19 VLMs on VRBench, we undertake a thorough analysis and provide valuable insights that advance the field of multi-step reasoning.

Figures

Figures reproduced from arXiv: 2506.10857 by the authors.

Figure 1
Figure 1. Overview of VRBench. We present VRBench, a long narrative video benchmark for multi-step reasoning. VRBench includes 960 manual-filtered narrative videos, covering 8 languages and 7 video categories that are suitable for reasoning about temporal rela￾tions. We also provide high-quality stepwise annotations for reasoning, which are labeled and reviewed by human experts. Each video incorporates 8-10 complex question-a… view at source ↗
Figure 2
Figure 2. An example of annotation in VRBench. For each question, VRBench provides the question-answer pair, multi-step reasoning chain, question type, and the start-to-end timestamps of the entire question as well as each reasoning step. ing. System-2 optimization strategies yield disproportion￾ate gains in process metrics, and despite parameter parity, open-source VLMs lag behind proprietary counterparts by 12.30%, suggesti… view at source ↗
Figure 3
Figure 3. Statistics of VRBench. We provide the detailed distribution of videos and annotations of VRBench, including video languages [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Human preference alignment results. For each plot, we show the win ratios of three different tested LLMs evaluated by human experts and VRBench. We then fit them with a straight line and quantify the correlation by calculating the Spearman cor￾relation coefficient. LLM…
Figure 5
Figure 5. Figure 5: Test-Time Scaling Results. We report the average accu [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Temporal Compositional Reasoning in Long-Form Sports Videos

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    SportsTime plus Chain-of-Time Reasoning (temporal-reward GRPO and anchor-observe-infer) modestly lifts open-ended sports VideoQA and step-wise temporal grounding over 4B–8B MLLM baselines.

  2. Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting

    cs.CV 2026-03 accept novelty 6.5 of 10

    EC-Bench finds best MLLMs score 29.98% enumeration F1 and 23.74% counting accuracy on 152 hour-scale videos, with errors driven by instance identification and temporal grounding rather than arithmetic.

  3. VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

    cs.CV 2025-11 conditional novelty 6.0 of 10

    A 1,680-question video benchmark shows leading multimodal models lag humans by ~15 points on visual knowledge, and a See-Think-Answer RL-trained model narrows the gap.

Reference graph

Works this paper leans on

122 extracted references · 23 canonical work pages · cited by 3 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 6, 19

    arXiv 2024

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 6, 7, 8, 19

    arXiv 2025

  3. [3]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. 2, 6

    arXiv 2024

  4. [4]

    Cg- bench: Clue-grounded question answering benchmark for long video understanding

    Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. Cg- bench: Clue-grounded question answering benchmark for long video understanding. arXiv preprint arXiv:2412.12075,

    arXiv

  5. [5]

    Theo- remqa: A theorem-driven question answering dataset

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theo- remqa: A theorem-driven question answering dataset. arXiv preprint arXiv:2305.12524, 2023. 3

    arXiv 2023

  6. [6]

    Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering

    Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering. In European Conference on Computer Vision , pages 179–195. Springer, 2024. 3

    2024

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 2, 5, 6, 7, 8, 19

    arXiv 2024

  8. [8]

    Train- ing verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Train- ing verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 2, 3

    arXiv

Show all 122 references
  1. [9]

    Tvbench: Re- designing video-language evaluation

    Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. Tvbench: Re- designing video-language evaluation. arXiv preprint arXiv:2410.07752, 2024. 3

    2024 arXiv

  2. [10]

    Exams-v: A multi-discipline multilingual multi- modal exam benchmark for evaluating vision language mod- els

    Rocktim Jyoti Das, Simeon Emilov Hristov, Haonan Li, Dimitar Iliyanov Dimitrov, Ivan Koychev, and Preslav Nakov. Exams-v: A multi-discipline multilingual multi- modal exam benchmark for evaluating vision language mod- els. arXiv preprint arXiv:2403.10378, 2024. 3

    2024 arXiv

  3. [11]

    Deepl translate: The world’s most accurate trans- lator

    DeepL. Deepl translate: The world’s most accurate trans- lator. https://www.deepl.com/en/translator ,

  4. [12]

    Gemini 2.0 flash thinking

    Google Deepmind. Gemini 2.0 flash thinking. https: / / deepmind . google / technologies / gemini / flash-thinking/, 2025. Accessed: 2025-01-21. 2, 5

    2025

  5. [13]

    Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video under- standing. Advances in Neural Information Processing Sys- tems, 37:89098–89124, 2025. 3

    2025

  6. [14]

    Video-of-thought: Step-by-step video reasoning from perception to cognition

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. arXiv preprint arXiv:2501.03230, 2024. 2

    2024 arXiv

  7. [15]

    Sciknoweval: Evaluating multi- level scientific knowledge of large language models

    Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, and Huajun Chen. Sciknoweval: Evaluating multi- level scientific knowledge of large language models. arXiv preprint arXiv:2406.09098, 2024. 3

    2024

  8. [16]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 2, 3

    2024 arXiv

  9. [17]

    H2ovl-mississippi vision language models technical report

    Shaikat Galib, Shanshan Wang, Guanshuo Xu, Pascal Pfeif- fer, Ryan Chesler, Mark Landry, and Sri Satish Ambati. H2ovl-mississippi vision language models technical report. arXiv preprint arXiv:2410.13611, 2024. 6, 8, 19

    2024 arXiv

  10. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 2, 5, 6, 8

    2025 arXiv

  11. [19]

    Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection

    Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. arXiv preprint arXiv:2411.14794, 2024. 3

    2024 arXiv

  12. [20]

    Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos

    Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wan- rong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos. arXiv preprint arXiv:2406.08407, 2024. 3

    2024 arXiv

  13. [21]

    Mea- suring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mea- suring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. 3

    2009 arXiv

  14. [22]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline pro- fessional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline pro- fessional videos. arXiv preprint arXiv:2501.13826, 2025. 3

    2025 arXiv

  15. [23]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jing- han Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36...

  16. [24]

    Vbench: Comprehensive bench- mark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive bench- mark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  17. [25]

    Olympicarena: Benchmark- ing multi-discipline cognitive reasoning for superintelligent ai

    Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmark- ing multi-discipline cognitive reasoning for superintelligent ai. Advances in Neural Information Processing Systems, 37: ...

    2025

  18. [26]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. 3

    2024 arXiv

  19. [27]

    Vis- science: An extensive benchmark for evaluating k12 ed- ucational multi-modal scientific reasoning

    Zhihuan Jiang, Zhen Yang, Jinhao Chen, Zhengxiao Du, Weihan Wang, Bin Xu, Yuxiao Dong, and Jie Tang. Vis- science: An extensive benchmark for evaluating k12 ed- ucational multi-modal scientific reasoning. arXiv preprint arXiv:2409.13730, 2024. 3

    2024 arXiv

  20. [28]

    Tvqa: Localized, compositional video question answering

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696, 2018. 3

    2018 arXiv

  21. [29]

    Veu-bench: Towards comprehensive under- standing of video editing

    Bozheng Li, Yongliang Wu, Yi Lu, Jiashuo Yu, Licheng Tang, Jiawang Cao, Wenqing Zhu, Yuyang Sun, Jay Wu, and Wenbo Zhu. Veu-bench: Towards comprehensive under- standing of video editing. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 13671– 13...

    2025

  22. [30]

    Aria: An open multimodal native mixture-of- experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of- experts model. arXiv preprint arXiv:2410.05993, 2024. 2, 6, 19

    2024 arXiv

  23. [31]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  24. [32]

    Videochat-flash: Hierarchical com- pression for long-context video modeling

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical com- pression for long-context video modeling. arXiv preprint arXiv:2501.00574, 2024. 2, 6, 19

    2024 arXiv

  25. [33]

    Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958, 2025. 2

    2025 arXiv

  26. [34]

    Videovista: A versatile bench- mark for video understanding and reasoning

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile bench- mark for video understanding and reasoning. arXiv preprint arXiv:2406.11303, 2024. 3

    2024 arXiv

  27. [35]

    Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehen- sion

    Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehen- sion. In AI for Accelerated Materials Design-Vienna 2024 ,

    2024

  28. [36]

    Scemqa: A scientific col- lege entrance level multimodal question answering bench- mark

    Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, and Xiangliang Zhang. Scemqa: A scientific col- lege entrance level multimodal question answering bench- mark. arXiv preprint arXiv:2402.05138, 2024. 3

    2024 arXiv

  29. [37]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 26689–26699, 2024. 2

    2024

  30. [38]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 5, 6, 7

    2024 arXiv

  31. [39]

    Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024. 3

    2024 arXiv

  32. [40]

    Et bench: Towards open-ended event-level video-language understanding

    Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. Et bench: Towards open-ended event-level video-language understanding. arXiv preprint arXiv:2409.18111, 2024. 3

    2024 arXiv

  33. [41]

    Llama-3.3-70b-instruct

    Llama-3.3. Llama-3.3-70b-instruct. https : / / huggingface . co / meta - llama / Llama - 3 . 3 - 70B-Instruct, 2025. 6

    2025

  34. [42]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 3

    arXiv

  35. [43]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural In- formation Processing Systems, 36:46212–46244, 2023. 3

    2023

  36. [44]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022. 3

    2022 arXiv

  37. [45]

    Plotqa: Reasoning over scientific plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 1527–1536, 2020. 3

    2020

  38. [46]

    Video-bench: A com- prehensive benchmark and toolkit for evaluating video-based large language models

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A com- prehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103 ,

    arXiv

  39. [47]

    Hello gpt4-o

    OpenAI. Hello gpt4-o. https : / / openai . com / index/hello-gpt-4o/ , 2024. Accessed: 2024-05-13. 4, 5, 6, 7, 16, 17, 19

    2024

  40. [48]

    Introducing openai o1

    OpenAI. Introducing openai o1. https://openai. com/o1/, 2024. 2, 5, 6, 7

    2024

  41. [49]

    Openai o3-mini

    OpenAI. Openai o3-mini. https://openai.com/ index/openai-o3-mini/, 2025. 2

    2025

  42. [50]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing ...

    2023

  43. [51]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Interna- tional conference on machine learning, pages 28492–28518. PMLR, 2023. 4

    2023

  44. [52]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. 2, 7

    2023

  45. [53]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Miquel Farr ´e, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024. 2, 3

    2024 arXiv

  46. [54]

    Scienceqa: A novel resource for question answering on scholarly articles

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301, 2022. 3

    2022

  47. [55]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 2

    2017 arXiv

  48. [56]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  49. [57]

    Scieval: A multi-level large language model evaluation benchmark for scientific re- search

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. Scieval: A multi-level large language model evaluation benchmark for scientific re- search. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 19053–19061, 2024. 3

    2024

  50. [58]

    Movieqa: Understanding stories in movies through question- answering

    Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question- answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631–4640,

  51. [59]

    Claude Team

    Anthropic. Claude Team. Claude 3.7 sonnet. https:// www.anthropic.com/claude/sonnet , 2025. 2, 5, 6, 8, 19

    2025

  52. [60]

    Mimo-vl technical report

    Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, et al. Mimo-vl technical report. https://arxiv.org/abs/2506.03569, 2025. 6, 19

    2025 arXiv

  53. [61]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 5, 6, 8, 19

    2023 arXiv

  54. [62]

    Kimi-vl technical report

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chen- zhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025. 6, 8

    2025 arXiv

  55. [63]

    Kwai keye-vl technical report

    Kwai Keye Team, Biao Yang, Bin Wen, Changyi Liu, Chen- glong Chu, Chengru Song, Chongling Rao, Chuan Yi, Da Li, Dunju Zang, et al. Kwai keye-vl technical report. arXiv preprint arXiv:2507.01949, 2025. 6, 19

    2025 arXiv

  56. [64]

    Qwq: Reflect deeply on the boundaries of the unknown

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. https://qwenlm.github.io/blog/ qwq-32b-preview/, 2024. Accessed: 2024-11-28. 6

    2024

  57. [65]

    Qwq-32b: Embracing the power of reinforce- ment learning

    Qwen Team. Qwq-32b: Embracing the power of reinforce- ment learning. https://qwenlm.github.io/blog/ qwq-32b/, 2025. Accessed: 2025-3-6. 2, 5, 6, 7, 8

    2025

  58. [66]

    Llamav- o1: Rethinking step-by-step visual reasoning in llms

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav- o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint arXiv:2501.06186, 2025. 2

    2025 arXiv

  59. [67]

    Mea- suring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Sys- tems, 37:95095–95169, 2025. 3

    2025

  60. [68]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 5, 6, 19

    2024 arXiv

  61. [69]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024. 2

    2024 arXiv

  62. [70]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiao- han Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024. 2, 3

    2024 arXiv

  63. [71]

    Internvideo2: Scaling foundation models for mul- timodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for mul- timodal video understanding. In European Conference on Computer Vision, pages 396–416. Springer, 2024. 2

    2024

  64. [72]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Infor- mation ...

  65. [73]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xi- angyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386, 2025. 2, 5, 6, 19

    2025 arXiv

  66. [74]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sad- hika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural In- formation Processing Systems, 37:113569–1136...

    2025

  67. [75]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824–24837, 2022. 5

    2022

  68. [76]

    Star: A benchmark for situated reason- ing in real-world videos

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reason- ing in real-world videos. arXiv preprint arXiv:2405.09711,

    arXiv

  69. [77]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Informa- tion Processing Systems, 37:28828–28857, 2025. 2, 3

    2025

  70. [78]

    Chartinsights: Evaluating multimodal large language models for low-level chart question answer- ing

    Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. Chartinsights: Evaluating multimodal large language models for low-level chart question answer- ing. arXiv preprint arXiv:2405.07001, 2024. 3

    2024 arXiv

  71. [79]

    Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of- experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024. 6, 8, 19

    2024 arXiv

  72. [80]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 3

    2021

  73. [81]

    Llava-o1: Let vision language models reason step- by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step- by-step. arXiv preprint arXiv:2411.10440, 2024. 2

    2024 arXiv

  74. [82]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report, 2024. URL https://arxiv. org/abs/2407.10671, 2024. 7, 8

    2024 arXiv

  75. [83]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 5, 6, 7, 17

    2024 arXiv

  76. [84]

    Vript: A video is worth thousands of words

    Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. Advances in Neural Information Processing Systems, 37:57240–57261, 2025. 3

    2025

  77. [85]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  78. [86]

    Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024. 3

    2024 arXiv

  79. [87]

    Mr-gsm8k: A meta-reasoning bench- mark for large language model evaluation

    Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. Mr-gsm8k: A meta-reasoning bench- mark for large language model evaluation. arXiv preprint arXiv:2312.17080, 2023. 3

    2023 arXiv

  80. [88]

    Videollama 3: Frontier multi- modal foundation models for image and video understand- ing

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multi- modal foundation models for image and video understand- ing. arXiv preprint arXiv:2501.13106, 2025. 2

    2025 arXiv

  81. [89]

    Cmmmu: A chinese massive multi- discipline multimodal understanding benchmark

    Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, et al. Cmmmu: A chinese massive multi- discipline multimodal understanding benchmark. arXiv preprint arXiv:2401.11944, 2024. 3

    2024 arXiv

  82. [90]

    Movqa: A benchmark of versatile question-answering for long-form movie understanding

    Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understanding. arXiv preprint arXiv:2312.04817 ,

    arXiv

  83. [91]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 2, 6, 7, 19

    2024 arXiv

  84. [92]

    Mmvu: Measuring expert- level multi-discipline video understanding

    Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. Mmvu: Measuring expert- level multi-discipline video understanding. arXiv preprint arXiv:2501.12380, 2025. 2, 3

    2025 arXiv

  85. [93]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264,

    arXiv

  86. [94]

    Au- toshot: A short video dataset and state-of-the-art shot bound- ary detection

    Wentao Zhu, Yufang Huang, Xiufeng Xie, Wenxian Liu, Jin- can Deng, Debing Zhang, Zhangyang Wang, and Ji Liu. Au- toshot: A short video dataset and state-of-the-art shot bound- ary detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,...

    2023

  87. [95]

    • Multi-step process 1: A → B → C → D

    Event Prediction: Predict the next event based on event A that has already occurred in the video. • Multi-step process 1: A → B → C → D. • Multi-step process 2: B → A → C → D if A is at- tributed by B

  88. [96]

    • Multi-step process 1: A → B → C → D

    Hypothetical Reasoning: Given a hypothetical premise A, infer the corresponding development. • Multi-step process 1: A → B → C → D. • Multi-step process 2: B → A → C → D if A is at- tributed by B

  89. [97]

    • Multi-step process: D → C → B → A

    Event Attribution : Analyze the cause or purpose of event D in the video. • Multi-step process: D → C → B → A

  90. [98]

    • Multi-step process: D → C → B → A

    Implicit Reasoning : Infer the feelings/emotions of a specific character D, relationships between characters, or the situation of event development at the current point in time. • Multi-step process: D → C → B → A

  91. [99]

    • Multi-step process: A → C → D → B

    Logical Links: Analyze the correlation between two el- ements A and B in the video and explain their logical relationship. • Multi-step process: A → C → D → B

  92. [100]

    Information synopsis: Pose a summary question based on the video content and attempt to answer it (note: the question should not simply summarize the entire video but should be a synopsis question based on the video, requiring only single-step reasoning)

  93. [101]

    • Multi-step process: A ◦ B ◦ C ◦ D, where ◦ represents any logical/mathematical operation

    Counting Problems : Infer the transformation of ele- ment A under multiple conditions, possibly involving arithmetic or counting components such as numbers, dates, or specific points in time. • Multi-step process: A ◦ B ◦ C ◦ D, where ◦ represents any logical/mathematical oper...

  94. [103]

    [xx:xx:xx->xx:xx:xx]

  95. [104]

    Quality Assessment We adopt a full-scale quality assessment strategy, and the unqualified annotations should be modified until they meet the qualified criteria

    [xx:xx:xx->xx:xx:xx] Reasoning Type: xxxx I.1.4. Quality Assessment We adopt a full-scale quality assessment strategy, and the unqualified annotations should be modified until they meet the qualified criteria. We detail the quality assessment pro- cess in Section J. Figure A15...

  96. [105]

    Understanding and Analysis: Carefully read the provided text to ensure accurate comprehension of the plot development within the video

  97. [106]

    The summary should highlight the essential points while maintaining the completeness and logic of the narration

    Summarization and Distillation: Use fluent and coherent language to summarize the main content of the video. The summary should highlight the essential points while maintaining the completeness and logic of the narration. Instead of simply copying or repeating the text from th...

  98. [107]

    Length Control: The length of the summary should be approximately equal to that of the original provided text, neither too long nor too short

  99. [108]

    Provided Text: {merge_caption} Video Summary: [Write your summary here, adhering to the above guidelines.] Figure A17

    Formatting Standards: Organize your summary in a narrative style to ensure clarity and ease of understanding. Provided Text: {merge_caption} Video Summary: [Write your summary here, adhering to the above guidelines.] Figure A17. Prompts of generating the first video summary. T...

  100. [109]

    The narration of the preceding minutes is solely to provide necessary background for explaining relationships or events within the current scene

    Focus on the Present: Emphasize describing events occurring in the current video. The narration of the preceding minutes is solely to provide necessary background for explaining relationships or events within the current scene. Ensure the newly created content seamlessly conne...

  101. [110]

    While you can refer to the provided subtitles, avoid directly quoting them

    Detail and Original Expression: When describing, strive to rephrase using your own words to maintain originality and fluency, ensuring important information from the video is not lost. While you can refer to the provided subtitles, avoid directly quoting them. Instead, natural...

  102. [111]

    Employ vivid language and appropriate rhetorical devices to enhance textual expressiveness

    Fluency and Engagement: Ensure the narrative is coherent, engaging, logically clear, and flows smoothly, allowing readers to naturally follow the plot development. Employ vivid language and appropriate rhetorical devices to enhance textual expressiveness. Also, ensure accurate...

  103. [112]

    Use detailed depiction to make readers feel as if they are experiencing the events firsthand

    Formatting Standards: Organize your description in a narrative style, ensuring clarity, understandability, and appeal. Use detailed depiction to make readers feel as if they are experiencing the events firsthand. Provided Texts: Existing Video Content Description: {existing_su...

  104. [113]

    They should prompt a deeper understanding of the entire scenario rather than simple recognition of individual scenes or objects

    Multi-step inference questions: Questions should require logical reasoning to answer, not mere observation or perception. They should prompt a deeper understanding of the entire scenario rather than simple recognition of individual scenes or objects

  105. [114]

    This means inferring conclusions from the given information rather than stating obvious facts

    Reasoning process for multi-step inference: Beyond basic event descriptions, answers in the Q&A session should derive through multiple steps of logical thinking and information synthesis. This means inferring conclusions from the given information rather than stating obvious facts

  106. [115]

    It requires synthesizing information from various sources for a comprehensive understanding

    Integrating multiple sources of information: While questions and their answers can be addressed by visual content alone or combined with video and subtitles, judgments should not rely solely on subtitle-provided information or everyday common sense. It requires synthesizing in...

  107. [116]

    Give a unqualified/qualified binary video-level rating for the question distributions and provide reasons for the un- qualified annotations

  108. [117]

    HF” means “Hugging Face

    Give a four-choice question-level rating of unquali- fied/normal/good/excellent for each QA pair and provide the rationale. Validation Requirements. The quality assessors need to follow the following criteria to determine whether the given set of annotations is qualified or no...

    2024

  109. [118]

    Assessing correctness to determine compliance with quality control standards

  110. [119]

    Assessing quality scores for compliance with validation standards

  111. [120]

    Whether each manual annotation is scored, with or with- out reasons for vacant scoring. J.3. Validation Interface The video-level annotation interface is illustrated in Fig- ure A15, and we show the question-level annotation inter- face in Figure A16. K. Evaluation Setup K.1. ...

  112. [121]

    **Logical Consistency** (0-10): - Validate causal connections in the reasoning chain - Identify logical leaps or argument discontinuities - Assess congruence between assumptions and conclusions

  113. [122]

    **Factual Accuracy** (0-10): - Verify verifiability of all factual claims - Detect conflicts with established truths - Evaluate frequency and impact of factual errors

  114. [123]

    System prompt for a question having a unique answer L.2

    **Process Clarity** (0-10): - Analyze clarity and organization of step presentation - Check terminology accuracy and consistency - Assess effectiveness in explaining complex concepts Scoring Standards: For each dimension: 9-10: Exemplary performance with no flaws 7-8: Non-crit...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.