Pith. sign in

REVIEW 3 major objections 6 minor 80 references

VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current large video language models cannot reconstruct the causal order of shuffled procedural steps, and most score at or below random guessing on a new benchmark that humans solve at 96.4%.

desk verdict Useful new benchmark with a credible negative result, but the visual-causal interpretation is overstated because the goal name leaks and ordering can be done from language commonsense. read the letter →

arxiv 2505.08455 v1 pith:T3CGCQJV submitted 2025-05-13 cs.CV

Pritam Sarkar , Ali Etemad This is my paper · ORCID
classification cs.CV
keywords causalreasoninglargevideolanguagemodelsVCRBenchbenchmarkdesignproceduralunderstandingsequenceorderingrecognition-reasoningdecompositionpriors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that today's large video language models (LVLMs) cannot perform long-form causal reasoning in video: shown shuffled clips of an everyday procedure and asked to reconstruct the correct order, most open-source models score at or below random guessing, and the best model evaluated reaches 58.0% while humans reach 96.4%. To support this, the paper introduces VCRBench, 365 test videos across twelve everyday procedures, designed so that the correct order cannot be gleaned from option wording and is graded deterministically. It also proposes Recognition-Reasoning Decomposition (RRD), which first asks a model to describe each clip in one sentence and then orders those descriptions using common sense; on the strongest open model, Qwen2.5-VL-Instruct72B, this raises accuracy from 29.0% to 54.2%, surpassing Gemini-1.5-Pro. A sympathetic reader would care because the result separates action recognition from causal reasoning and identifies connecting visual events into a goal-directed chain as the capability that is missing.

What carries the argument

VCRBench is the measuring instrument: 365 test videos across twelve everyday procedures, built by cutting CrossTask instructional videos at human-annotated timestamps, discarding steps that lack causal dependencies, shuffling the surviving clips, and requiring the exact chronological order as the answer. Random-guess accuracy sits at 7.8% because a full permutation must match, while step accuracy gives partial credit per clip. RRD is the intervention: its video-recognition stage prompts the model for a one-sentence description of each clip with no ordering, and its causal-reasoning stage feeds those descriptions plus the named goal to the model's language knowledge to produce the order. Ablations show that recognizing clips one at a time helps, while replacing the all-at-once ordering with pairwise merge-sort comparisons hurts because long-range causal structure is lost.

What would settle it

Present the same shuffled-clip task using only the ground-truth text descriptions of each clip and no video, and compare accuracy with the video-condition scores; if text-only ordering matches or beats the roughly 29-58 percent video scores, the benchmark's signal is carried by language rather than visual causal inference. A second check is to remove the procedure name from the question template and see whether model accuracy drops toward the random-guess baseline.

Watch

Extended reading notes

Core claim

The central claim is that video-based long-form causal reasoning—deciding which visual events must come before which others to reach a stated goal—is a distinct skill that current LVLMs have not acquired, and that it can be measured and partly recovered by task decomposition. VCRBench presents a model with three to seven shuffled clips, each showing one key step of a familiar procedure, and demands the exact permutation as the answer; this format keeps deterministic grading while removing the linguistic cues of multiple-choice QA. On this benchmark, most open-source LVLMs perform at or below the 7.8% random-guess baseline, several simply emit consecutive numbers as their answer, and the best model, Gemini-2.0-Flash-Thinking, reaches only 58.0% against 96.4% for humans. The paper's own intervention, RRD, splits the task into a recognition stage that produces a one-sentence description per clip and a reasoning stage that orders those text descriptions; applied to Qwen2.5-VL-Instruct72B it raises accuracy from 29.0% to 54.2%, a gain of up to 25.2%. The paper also reports that feeding the video back into the reasoning stage lowers accuracy, which it interprets as evidence that current models lean on language knowledge rather than visual causal analysis.

Load-bearing premise

The benchmark's claim to measure visual causal reasoning rests on the assumption that the correct clip order cannot be worked out from the named goal plus commonsense language knowledge alone, without genuinely analyzing the video.

Editorial extensions

If this is right

  • If VCRBench measures what it claims, strong LVLM scores on multiple-choice video benchmarks overstate visual understanding, since those formats can be solved from option wording alone.
  • Scaling and reasoning-specialized training help only partially: even Gemini-2.0-Flash-Thinking leaves nearly forty points of accuracy on the table relative to humans.
  • Decomposing a complex multimodal task into recognition followed by text-based reasoning is a cheap way to gain up to 25.2 accuracy points, letting an open model surpass Gemini-1.5-Pro.
  • The accuracy drop when video is re-added at the reasoning stage implies that visual input can actively conflict with these models' language priors and hurt rather than help.
  • Because accuracy falls steeply as the number of causal steps grows from three to seven, the benchmark identifies the length of the causal chain as the main difficulty driver.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would be to hide or replace the procedure name in the prompt; if accuracy collapses, the benchmark is being solved from the named goal plus language priors, and the visual signal is not load-bearing.
  • Porting the shuffled-step format to domains whose causal order is not common knowledge—laboratory procedures, mechanical repairs, unfamiliar crafts—would yield a harder diagnostic that language priors cannot satisfy.
  • The success of the text-ordering stage suggests a concrete training target: teach LVLMs to extract order-relevant causal structure from pixels directly, using the recognition stage as a scaffold rather than a permanent crutch.
  • Since humans approach perfect accuracy, the videos themselves carry enough information to solve the task; a model forced to order events without a goal name would test whether the recorded pixels are sufficient.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. VCRBench is a new benchmark for long-form causal reasoning in video. The authors construct 365 test samples from CrossTask procedural videos and WikiHow step lists, shuffle 3-7 clips per task, and ask an LVLM to output the correct clip order. They evaluate 20+ open- and closed-source LVLMs, report that most open-source models perform at or below a random-guess baseline (e.g., Qwen2.5-VL-Instruct-72B: 29.0%) while Gemini-2.0-Flash-Thinking reaches 58.0% and humans reach 96.4%, and introduce Recognition-Reasoning Decomposition (RRD), a two-stage prompt (video recognition, then text-based ordering), which raises Qwen2.5-VL-72B to 54.2% under their best decomposition. They also report ablations showing that adding video to the reasoning stage degrades accuracy, and conclude that LVLMs mainly rely on language knowledge and cannot model long-range causal dependencies directly from visual observations.

Significance. VCRBench addresses a genuinely underexplored capability and its deterministic ordering format is a sensible middle ground between multiple-choice and open-ended QA. The paper's strengths are the breadth of the model suite, the high human accuracy (96.4%) indicating task clarity, the transparent construction from CrossTask/WikiHow, and the consistent direction of RRD gains across 7B-78B models. If the headline interpretation held, the finding that most LVLMs do not exceed random guessing would be an important diagnostic result. However, as discussed below, the goal-name leakage and the text-only reasoning stage mean the benchmark may be measuring event recognition plus language-level procedural ordering rather than causal reasoning over visual observations; the significance of the central claim therefore depends on the outcome of the proposed control experiments.

major comments (3)
  1. [§3.1 (Figure 4), §5.1, Table 3] The central claim that VCRBench isolates video-based long-form causal reasoning is not yet established. The question template in Figure 4 explicitly names the goal (e.g., 'Make Lemonade') and instructs the model to 'use your reasoning and common sense'; the RRD causal-reasoning stage receives only the procedure name and text clip descriptions; and Table 3 shows that reintroducing video at that stage lowers accuracy (49.9 to 46.6). A model that can visually recognize and verbalize each clip and then order the resulting event descriptions using procedural commonsense can therefore score high without performing causal inference over raw visual input. To support the headline interpretation, the authors should add control conditions that remove or perturb the goal name (e.g., asking for the 'most plausible sequence' without naming the procedure, or using a wrong/absent goal), and compare RRD's text-only reasoning stage with an oracle that receives ground-truth step labels instead of model-generated descriptions. If a text-only oracle succeeds at ordering, the benchmark should be described as measuring event recognition plus language-level ordering, not visual causal reasoning per se.
  2. [§3.2, step accuracy definition] The printed step-accuracy formula is missing the normalization by the number of steps s. As written, it sums per-step indicators over all videos and divides only by |D|, so the score would be an average number of correctly placed steps per video (up to s) rather than a percentage; the reported values (e.g., 24.1% for random, 98.3% for human) imply that the implementation also divides by s. Please correct the equation and state explicitly how step accuracy is averaged over steps and videos.
  3. [§4.2, Tables 1-5] All results are single-run point estimates with no error bars, confidence intervals, or significance tests. This matters because the paper uses 'significant' for RRD gains (e.g., §5.2) and because the evaluated closed-source models are stochastic; a gap such as 49.9 vs 46.6 in Table 3 may be within run-to-run noise. Please report mean and standard deviation over multiple generations or seeds for the key comparisons, and apply a paired significance test (e.g., McNemar or permutation) to the RRD gains. This is particularly important for the claim that adding video at the reasoning stage degrades performance.
minor comments (6)
  1. [§3.1, Step 3] The sentence 'The blank frames preceding the clips labeled chronologically' is unclear; please specify how clip numbers are displayed to the model (e.g., text overlay at the start of each clip) and confirm that the model can reliably perceive them.
  2. [§5.2, text after Table 2] The text reports gains of 15.3% for the 7B model and 20.8% for the 72B model, while Table 2 reports 15.4 and 20.9; please align the text and table.
  3. [Figure 5(D,E)] The top-50 action and object panels are too small to read in the preprint; please provide tabular versions in the appendix or increase the font size.
  4. [§3.2, random baseline] Please state explicitly how the random-guess baselines are computed, namely 1/k! for overall accuracy and the average per-position match for step accuracy, evaluated over the actual distribution of k in the benchmark rather than a single average value.
  5. [§4.2, GPT-4o discussion] The statement that GPT-4o performs worst among closed-source models 'likely due to its limited capacity for long visual inputs' is presented as fact; please rephrase it as a hypothesis or support it with measurements of input-length handling.
  6. [Appendix B.2, human evaluation] Human performance is based on eight volunteers and about 40% of the videos; please state whether each video was annotated by multiple evaluators and report inter-annotator agreement, since human accuracy is used to argue that the tasks are unambiguous.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VCRBench ground truth is externally sourced and RRD is a prompt-only method with no fitted inputs.

full rationale

The paper's central derivation chain is self-contained against external benchmarks and does not reduce to its own inputs. VCRBench ground-truth orders come from CrossTask human-annotated timestamps and WikiHow step lists, not from any of the evaluated LVLMs: the construction says videos are from CrossTask, segmenting by human-annotated timestamps, and core steps are identified using WikiHow with manual curation. No model output is used to define the correct order, so the benchmark scores are not built from the quantity they predict. RRD is a two-stage prompting method that adds no fitted parameters, learned constants, or tuned hyperparameters; its gain is measured empirically on the same fixed benchmark, so the improvement cannot be an artifact of fitting a parameter to the test set. The observation that adding video back at the causal reasoning stage hurts accuracy (Table 3) is an empirical result, not a conclusion forced by construction. The skeptical concern that the question template names the goal and asks the model to use common sense is a validity or leakage threat about what the benchmark measures, but it is not circularity: the paper does not define its target variable in terms of the model's output, and the language-knowledge dependence is reported as a finding. The only self-citation, reference [76] in the setup discussion of training methodologies, is a peripheral example and is not load-bearing for the benchmark or for RRD. Accordingly, no step of the derivation chain is equivalent to its input by definition or by fitted parameter reuse.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the quality of external human annotations and on task design choices. No numerical parameters are fitted, no new physical or theoretical entities are introduced, and no mathematical derivation is attempted.

assumptions (3)
  • domain assumption CrossTask human-annotated step timestamps and WikiHow step lists define the ground-truth causal order for each VCRBench sample.
    VCRBench construction in Section 3.1 relies on these external annotations to select, group, and order clips.
  • domain assumption Each everyday procedure used in VCRBench has a single correct chronological order for the stated goal.
    The evaluation assumes one ground-truth sequence; the 96.4% human accuracy supports this, but some procedures may admit multiple valid orders.
  • domain assumption LVLM-generated outputs can be parsed reliably into a clip ordering.
    The accuracy metric compares parsed predictions to the ground truth, and the paper does not report parsing failure rates or manual checks for malformed outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models." pith.science (2026). https://pith.science/paper/T3CGCQJV

@misc{pith2026250508455,
  author       = {Pith},
  title        = {Pith review of: VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3CGCQJV}},
  note         = {Machine review of arXiv:2505.08455}
}
read the original abstract

Despite recent advances in video understanding, the capabilities of Large Video Language Models (LVLMs) to perform video-based causal reasoning remains underexplored, largely due to the absence of relevant and dedicated benchmarks for evaluating causal reasoning in visually grounded and goal-driven settings. To fill this gap, we introduce a novel benchmark named Video-based long-form Causal Reasoning (VCRBench). We create VCRBench using procedural videos of simple everyday activities, where the steps are deliberately shuffled with each clip capturing a key causal event, to test whether LVLMs can identify, reason about, and correctly sequence the events needed to accomplish a specific goal. Moreover, the benchmark is carefully designed to prevent LVLMs from exploiting linguistic shortcuts, as seen in multiple-choice or binary QA formats, while also avoiding the challenges associated with evaluating open-ended QA. Our evaluation of state-of-the-art LVLMs on VCRBench suggests that these models struggle with video-based long-form causal reasoning, primarily due to their difficulty in modeling long-range causal dependencies directly from visual observations. As a simple step toward enabling such capabilities, we propose Recognition-Reasoning Decomposition (RRD), a modular approach that breaks video-based causal reasoning into two sub-tasks of video recognition and causal reasoning. Our experiments on VCRBench show that RRD significantly boosts accuracy on VCRBench, with gains of up to 25.2%. Finally, our thorough analysis reveals interesting insights, for instance, that LVLMs primarily rely on language knowledge for complex video-based long-form causal reasoning tasks.

Figures

Figures reproduced from arXiv: 2505.08455 by the authors.

Figure 1
Figure 1. Example question and video. We present an example of video-based long-form causal reasoning task from VCRBench. The correct order is: Clip 1: Cut lemon into slices, Clip 5: Squeeze lemon into the pitcher, Clip 4: Pour lemon juice and water into the pitcher, Clip 3: Stir the lemonade mixture, Clip 2: Pour lemonade into a glass. this gap by systematically evaluating the video-based causal reasoning capabilities of sta… view at source ↗
Figure 2
Figure 2. Impact of RRD. Qwen2.5- VL-Instruct72B with RRD outperforms Gemini-1.5-Pro and achieve compa￾rable performance to Gemini-2-Flash￾Thinking. To improve the long-form causal reasoning capabilities of LVLMs we introduce Recognition-Reasoning Decom￾position (RRD), a simple modular approach designed to enhance the video-based reasoning abilities of LVLMs. RRD breaks down video-based long-form causal reasoning into two int… view at source ↗
Figure 3
Figure 3. Overview of video construction. Step 1: Given a complete video, key procedural steps are identified based on human-annotated timestamps. Step 2: We keep the key events and discard those that do not depict visual events directly associated with the goal, such as talking or narrating in this example of grilling steak. Step 3: Each key event is shuffled across time and assigned a clip number. These clips are then merge… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The question template used in VCRBench. preparing pancakes (see Appendix B for the full list). For each procedure, we source instructional videos from the CrossTask dataset [63], which contains YouTube videos with human annotated timestamps of key events. Below, we out…
Figure 5
Figure 5. Figure 5: Key statistics of our VCRBench. these limitations by framing causal reasoning as a sequence ordering task. This setup avoids the use of linguistic cues in predefined options, yet yields deterministic ground truth answers. As a result, it enables accurate, objective eva…
Figure 6
Figure 6. Figure 6: Failure examples. Several open￾source LVLMs merely list consecutive numbers as the predicted order, exhibiting inability to make a meaningful attempt in VCRBench tasks. 0 20 40 60 80 100 Event Recognition Accuracy (%) 0 20 40 60 80 100 Causal Reasoning Accuracy (%) Ran…
Figure 7
Figure 7. Figure 7: Reasoning vs. Recognition. LVLMs can identify individual events but fail to connect them toward a specific goal in causal reasoning. 4.2 Results and Findings Here, we discuss our key observations regarding the performance of LVLMs on VCRBench, based on our detailed qua…
Figure 8
Figure 8. Figure 8: Failure examples. We observe that Gemini-2.0-Flash-Thinking generates a response with a detailed rationale explaining how it arrives at the final answer, unlike Gemini-1.5-Pro and Qwen2.5-VL-Instruct, which directly provide the final answer. Based on its detailed respo…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 27 canonical work pages

  1. [1]

    Robovqa: Multimodal long-horizon reasoning for robotics

    Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrish- nan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In ICRA, pages 645–652. IEEE, 2024. 1

    work page 2024

  2. [2]

    Mmro: Are multimodal llms eligible as the brain for in-home robotics? arXiv preprint arXiv:2406.19693, 2024

    Jinming Li, Yichen Zhu, Zhiyuan Xu, Jindong Gu, Minjie Zhu, Xin Liu, Ning Liu, Yaxin Peng, Feifei Feng, and Jian Tang. Mmro: Are multimodal llms eligible as the brain for in-home robotics? arXiv preprint arXiv:2406.19693, 2024. 1

    arXiv 2024

  3. [3]

    Egoplan-bench: Benchmarking multimodal large language models for human-level planning

    Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking multimodal large language models for human-level planning. arXiv preprint arXiv:2312.06722, 2023. 1, 2, 3, 4

    arXiv 2023

  4. [4]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16488–16498, 2024. 1

    work page 2024

  5. [5]

    Towards end-to-end embodied decision making via multi- modal large language model: Explorations with gpt4-vision and beyond

    Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang. Towards end-to-end embodied decision making via multi- modal large language model: Explorations with gpt4-vision and beyond. arXiv preprint arXiv:2310.02071, 2023. 1

    arXiv

  6. [6]

    A survey of large language model-powered spatial intelligence across scales: Advances in embodied agents, smart cities, and earth science

    Jie Feng, Jinwei Zeng, Qingyue Long, Hongyi Chen, Jie Zhao, Yanxin Xi, Zhilun Zhou, Yuan Yuan, Shengyuan Wang, Qingbin Zeng, et al. A survey of large language model-powered spatial intelligence across scales: Advances in embodied agents, smart cities, and earth science. arXiv preprint arXiv:2504.09848, 2025. 1

    arXiv 2025

  7. [7]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024. 1

    work page 2024

  8. [8]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023. 1

    arXiv 2023

Show all 80 references
  1. [9]

    A spectrum evaluation benchmark for medical multi-modal large language models

    Jie Liu, Wenxuan Wang, Yihang Su, Jingyuan Huan, Wenting Chen, Yudi Zhang, Cheng-Yi Li, Kao-Jung Chang, Xiaohan Xin, Linlin Shen, et al. A spectrum evaluation benchmark for medical multi-modal large language models. arXiv preprint arXiv:2402.11217, 2024. 1

    2024 arXiv

  2. [10]

    M3d: Advancing 3d medical image analysis with multi-modal large language models

    Fan Bai, Yuxin Du, Tiejun Huang, Max Q-H Meng, and Bo Zhao. M3d: Advancing 3d medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578,

    arXiv

  3. [11]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  4. [12]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 1, 5

    2023

  5. [13]

    Internvideo2: Scaling video foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024. 1, 3 10

    2024 arXiv

  6. [14]

    Longvu: Spa- tiotemporal adaptive compression for long video-language understanding

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spa- tiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 2024...

    arXiv

  7. [15]

    Slowfast-llava: A strong training-free baseline for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models. arXiv preprint arXiv:2407.15841, 2024. 1, 3

    2024 arXiv

  8. [16]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22195–2...

    2024

  9. [17]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 1, 3

    2023 arXiv

  10. [18]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 1, 3, 5

    2024 arXiv

  11. [19]

    Videollama 3: Frontier multimodal foundation models for image and video understanding

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025. 1, 3, 6, 19

    2025 arXiv

  12. [20]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313–14323, 2024. 1, 3

    2024

  13. [21]

    Longvila: Scaling long-context visual language models for long videos

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 1, 3, 6, 19

    2024 arXiv

  14. [22]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 1, 3, 6, 19

    2024 arXiv

  15. [23]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 3

    2023 arXiv

  16. [24]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv pr...

    2024 arXiv

  17. [25]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  18. [26]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,

    arXiv

  19. [27]

    Qwen-vl: A versatile vision-language model for understanding, localization

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization. Text Reading, and Beyond, 2, 2023. 1, 3

    2023

  20. [28]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923,

    arXiv

  21. [29]

    Intentqa: Context-aware video intent reasoning

    Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11963–11974, 2023. 2, 3, 4

    2023

  22. [30]

    Rex- time: A benchmark suite for reasoning-across-time in videos

    Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, and Frank Wang. Rex- time: A benchmark suite for reasoning-across-time in videos. Advances in Neural Information Processing Systems, 37:28662–28673, 2024. 2, 3 11

    2024

  23. [31]

    From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering

    Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21273–21282, 2022. 2, 3

    2022

  24. [32]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 3

    2024 arXiv

  25. [33]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  26. [34]

    Videochat-flash: Hierarchical compression for long-context video modeling

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574, 2024. 3

    2024 arXiv

  27. [35]

    video-salmonn: Speech-enhanced audio-visual large language models

    Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint arXiv:2406.15704, 2024. 3

    2024 arXiv

  28. [36]

    Videogpt+: Integrating image and video encoders for enhanced video understanding

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding. arXiv preprint arXiv:2406.09418,

    arXiv

  29. [37]

    Video understanding with large language models: A survey

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432, 2023. 3

    2023

  30. [38]

    Foundation models for video understanding: A survey

    Neelu Madan, Andreas Møgelmose, Rajat Modi, Yogesh S Rawat, and Thomas B Moeslund. Foundation models for video understanding: A survey. arXiv preprint arXiv:2405.03770, 2024. 3

    2024 arXiv

  31. [39]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134,

  32. [40]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017. 3, 4

    2017

  33. [41]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021. 3, 4

    2021

  34. [42]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758–2766, 2017. 3, 4

    2017

  35. [43]

    Tvbench: Redesigning video-language evaluation

    Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. Tvbench: Redesigning video-language evaluation. arXiv preprint arXiv:2410.07752, 2024. 3, 4

    2024 arXiv

  36. [44]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075,

    arXiv

  37. [45]

    Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476, 2024. 3, 4

    2024 arXiv

  38. [46]

    Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models

    Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, et al. Temporalbench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818, 2024. 3

    arXiv

  39. [47]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. 3 12

    2024 arXiv

  40. [48]

    Longvideobench: A benchmark for long- context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828–28857, 2025. 3, 4

    2025

  41. [49]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 3, 4

    2023

  42. [50]

    Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models

    Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint arXiv:2406.16338, 2024. 3, 4

    2024 arXiv

  43. [51]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    T Guan, F Liu, X Wu, R Xian, Z Li, X Liu, X Wang, L Chen, F Huang, Y Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. arxiv. 10.48550. arXiv preprint arXiv.2310.14566,

    arXiv

  44. [52]

    Sok-bench: A situated video reasoning benchmark with aligned open-world knowledge

    Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Li Erran Li, and Chuang Gan. Sok-bench: A situated video reasoning benchmark with aligned open-world knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2024

  45. [53]

    Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos

    Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos. arXiv preprint arXiv:2406.08407, 2024. 3

    2024 arXiv

  46. [54]

    Vilma: A zero-shot benchmark for linguistic and temporal grounding in video-language models

    Ilker Kesen, Andrea Pedrotti, Mustafa Dogan, Michele Cafagna, Emre Can Acikgoz, Letitia Parcalabescu, Iacer Calixto, Anette Frank, Albert Gatt, Aykut Erdem, et al. Vilma: A zero-shot benchmark for linguistic and temporal grounding in video-language models. arXiv preprint arXiv...

    2023 arXiv

  47. [55]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 3

    2022

  48. [56]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. NeurIPS, 30, 2017. 3

    2017

  49. [57]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 3

    2017 arXiv

  50. [58]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 3

    2025 arXiv

  51. [59]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. 3

    2022 arXiv

  52. [60]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020. 3

    2020

  53. [61]

    Webgpt: Browser-assisted question-answering with human feedback

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. 3

    2021 arXiv

  54. [62]

    Decomposed prompting: A modular approach for solving complex tasks

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022. 3

    2022 arXiv

  55. [63]

    Cross-task weakly supervised learning from instructional videos

    Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537–3545,

  56. [64]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 4 13

    2023 arXiv

  57. [65]

    Procedure planning in instructional videos

    Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. In European Conference on Computer Vision, pages 334–350. Springer, 2020. 5

    2020

  58. [66]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 5

    2023 arXiv

  59. [67]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 5

    2023 arXiv

  60. [68]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 5

    2024 arXiv

  61. [69]

    Identifying and mitigating vulnerabilities in llm-integrated applications

    Fengqing Jiang. Identifying and mitigating vulnerabilities in llm-integrated applications. Mas- ter’s thesis, University of Washington, 2024. 5

    2024

  62. [70]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 5

    2023 arXiv

  63. [71]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 5

    2024 arXiv

  64. [72]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023. 5

    2023

  65. [73]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 5

    2024

  66. [74]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 5

    2023 arXiv

  67. [75]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19948–19960, 2023. 5

    2023

  68. [76]

    Self-alignment of large video language models with refined regularized preference optimization

    Pritam Sarkar and Ali Etemad. Self-alignment of large video language models with refined regularized preference optimization. arXiv preprint arXiv:2504.12083, 2025. 5

    2025

  69. [77]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024. 5, 6, 19

    2024 arXiv

  70. [78]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 6, 19

    2024 arXiv

  71. [79]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 6, 15, 19 14 Appendix A Additional Results on VCRBench We present the det...

    2024 arXiv

  72. [2025]

    1, 3, 5, 6, 8, 9, 15, 19

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.