Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that current video QA benchmarks overestimate video understanding by rewarding language priors and static frame semantics, and that a Semantic-plus-Temporal core subset preserves model rankings with roughly half the…

desk verdict Useful decomposition, but the taxonomy floats on an unspecified frame shuffle and two proprietary labelers. read the letter →

arxiv 2505.14321 v1 pith:GIVJI7L7 submitted 2025-05-20 cs.CV

Bo Feng , Zhengfeng Lai , Shiyu Li , Zizhen Wang , Simon Wang , Ping Huang , Meng Cao This is my paper
classification cs.CV
keywords videounderstandingbenchmarkevaluationtemporalreasoningquestionansweringmultimodalLLManswerleakageframeshufflingVBenchComp
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that current video question-answering benchmarks reward language knowledge and static image understanding more than they reward true temporal reasoning. It introduces VBenchComp, an automated protocol that labels every question as LLM-Answerable, Semantic, Temporal, or Other based on two probes: whether models can answer without the video, and whether shuffling frames changes the answer. Applying the protocol to seven widely used benchmarks shows that LLM-Answerable and Semantic questions dominate, with Temporal questions in the minority in most datasets. The paper argues that a single overall score is therefore misleading, and that a VBenchComp score computed only on Semantic and Temporal questions preserves model rankings while cutting the question set roughly in half. If correct, this gives benchmark designers a concrete way to detect answer leakage and static-cue leakage.

What carries the argument

The load-bearing mechanism is the VBenchComp categorization protocol, a decision tree applied to each video question-answer pair. Two strong multimodal models first receive the question text without the video; if both answer correctly, the question is LLM-Answerable. For the remaining questions, the same two models answer with original frames and with frames shuffled twice; if both stay correct under shuffling, the question is Semantic; if at least one is correct originally but fails after shuffling, it is Temporal; everything else is Other. The protocol yields composition ratios and defines the VBenchComp score as accuracy restricted to Semantic and Temporal questions. The argument's force lies in the contrast between this breakdown and the monolithic overall score.

What would settle it

Run the same two-probe protocol on a random sample of questions labeled Temporal, but use several additional video-language models as judges: if most of those questions are still answered correctly after frame shuffling by the new judges, the Temporal label is an artifact of the original judge pair rather than an intrinsic property of the questions. A complementary check is to have human annotators mark which questions truly require knowing the order of events and measure agreement with the pipeline's labels.

Watch

Extended reading notes

Core claim

The central claim is that a single final score reported by current video benchmarks can overestimate a model's genuine video understanding. The paper demonstrates two failure modes: models answer a large share of questions correctly with no video input at all, with one closed model reaching roughly 50 percent accuracy on VideoMME and NExT-QA without video, and their accuracy stays stable or even improves when frames are shuffled, meaning those questions do not require temporal order. VBenchComp categorizes each question into one of four domains, and the resulting composition tables show that across seven benchmarks the Temporal share is often below twenty percent while LLM-Answerable and Semantic questions together dominate. Per-category scores also reveal that a model can lead overall on the strength of its language backbone while trailing on Temporal questions. The practical consequence is that evaluating only Semantic and Temporal questions, roughly half the original set, produces rankings consistent with the full benchmarks, making the trimmed core both cheaper and more diagnostic.

Load-bearing premise

The categorization assumes that the two proprietary judge models are reliable enough that 'both answer correctly without the video' and 'one model fails after shuffling' reveal intrinsic properties of the question rather than shared blind spots of the judges.

Editorial extensions

If this is right

  • Model rankings computed on Semantic plus Temporal questions match full-benchmark rankings, so roughly half the questions in current benchmarks are redundant for discriminating between video models.
  • Per-category scores expose cases where a model's overall lead comes from LLM-Answerable questions while it lags on Temporal ones, changing which model looks stronger.
  • Benchmark authors can use the protocol to measure the share of each question type and design sets with a higher Temporal share and lower answer leakage.
  • Cutting to the core subset reduces evaluation compute substantially, making repeated evaluation of large video models cheaper.
  • Shuffling-stable or shuffling-improved scores indicate that some benchmark questions reward static cues or noise rather than temporal understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's results, the same two-probe protocol could be applied to other multimodal benchmarks, such as image or audio QA, to separate world-knowledge priors from perception, provided the judge models are strong enough.
  • The paper's own Discussion (Section 6) concedes that the pipeline relies on two proprietary judge models and two shuffles; a more reliable variant would use more judges, more shuffle seeds, or human annotation before treating category counts as intrinsic properties of questions.
  • If Temporal questions are the ones that truly require video, then training models to maximize accuracy on a Temporal-only set, for example by adding shuffled-frame negatives, could produce video-specific gains that aggregate leaderboards currently mask.
  • The finding that Semantic questions dominate most benchmarks suggests that future leaderboards should weight or report Temporal accuracy separately, otherwise progress in video understanding will continue to be confused with progress in static perception.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that current video QA benchmarks overestimate video LLMs' temporal understanding because a large fraction of questions can be answered without video (language priors) or without temporal order (static semantic cues). To address this, it proposes VBenchComp, an automated pipeline that labels each question as LLM-Answerable, Semantic, Temporal, or Other based on the correctness of GPT-4o and Gemini-1.5-Pro under text-only and frame-shuffled settings. The pipeline is applied to seven benchmarks, producing composition statistics (Table 2), per-category model scores for several public video LLMs (Table 3), and a reduced 'VBenchComp Score' based only on Semantic and Temporal questions (Section 5.3, Figure 5), which is claimed to preserve full-benchmark rankings while cutting roughly half the questions. The paper concludes that overall scores conflate knowledge, static recognition, and true temporal reasoning, and recommends that future benchmarks be designed to isolate temporal understanding.

Significance. The diagnostic goal is valuable: separating language-prior, semantic, and temporal contributions is directly relevant to benchmark design and to interpreting model comparisons. The paper's strengths include an automated and scalable categorization procedure, analysis across seven widely used benchmarks, concrete per-category model comparisons, and a candid acknowledgment in the Discussion that the pipeline 'heavily relies on GPT-4o and Gemini, which may introduce biases.' If the central claims were fully supported, the proposed VBenchComp Score would offer a computationally cheaper and more interpretable evaluation protocol. However, the load-bearing parts of the analysis—the shuffle-based definition of Temporal questions and the claimed ranking consistency—are not currently specified or quantified sufficiently to establish the paper's headline conclusions.

major comments (4)
  1. [§4.2–§4.3, Figure 3] The shuffle operation is not specified. The paper states 'we apply random shuffles to the extracted frames twice' but never defines the shuffle distribution, block size, number of frames shuffled, or seed. Because the Temporal category is defined by a correct answer before shuffling followed by an incorrect answer after shuffling, the proportions γ in Table 2 and every conclusion built on them depend directly on the perturbation strength: a full permutation will classify many order-sensitive questions as Temporal, whereas a mild adjacent-pair swap will classify them as Semantic. The paper reports no robustness analysis over shuffle strength and no quantification of how γ changes across plausible shuffle protocols. This is load-bearing for the central claim that current benchmarks overestimate temporal understanding, so the protocol must be specified and the sensitivity of γ (and of the Table 3 category scores) to shuffle strength must be reported.
  2. [§4.1–§4.3, Table 2] The category labels are defined by the joint behavior of exactly two proprietary models, GPT-4o and Gemini-1.5-Pro, on two random shuffles, yet Table 2 and the surrounding text report them as intrinsic properties of benchmark questions. If the two labelers share blind spots—for example, both ignoring order while attending to object identity—then the proportions and the model conclusions in Table 3 are distorted. The Discussion acknowledges that the pipeline 'heavily relies on GPT-4o and Gemini, which may introduce biases,' but this limitation is central rather than peripheral. To support the benchmark-level claims, the paper should either provide labeler agreement or consistency evidence, include at least one additional independent labeler, or reframe the entire analysis explicitly as a model-relative diagnostic rather than a property of the benchmarks.
  3. [§5.3, Figure 5] The central claim that Semantic plus Temporal questions 'preserve the discriminative power' of each benchmark is asserted without any quantified ranking agreement. The text says model rankings 'remain highly consistent' and reports a 'strong correlation' with original scores, but no rank correlation coefficient, confidence interval, or per-benchmark statistic is given. Because the paper's headline conclusion and the proposed VBenchComp Score depend on this consistency, the authors should report Spearman or Kendall correlations with confidence intervals between original and VBenchComp rankings for each benchmark and across model sizes, and they should show that the result is stable under the shuffle-robustness analysis raised in the first major comment.
  4. [§4.2–§4.3, §5.2] The protocol classifies a question as Temporal if at least one labeler model succeeds before shuffling but fails after. This label is then used to compute 'Temporal' scores for other models such as Qwen2-VL and LLaVA-Video. A question that is order-dependent for the labelers may be order-independent for a stronger or differently biased evaluated model, so the per-model category scores do not cleanly measure the evaluated model's temporal reasoning; they measure performance on questions that are temporal for the labelers. This model-relative labeling should be stated as an explicit caveat in Section 5.2, and the analysis should show how many questions are category-consistent across the evaluated models.
minor comments (5)
  1. [Figure 3(a) caption] The caption contains the typo 'Close-sourced'; it should read 'Closed-source'.
  2. [Table 3] The column header 'LLM' is inconsistent with the term 'LLM-Answerable' used in Sections 4.1 and 5.1; unify the terminology.
  3. [§5.3] The claim of 'removing nearly 50% of the original questions' is imprecise: from Table 2, the combined Semantic and Temporal fractions range from 40.6% for LongVideoBench to 49.1% for PerceptionTest. Report the per-benchmark retained fractions.
  4. [§3.2] The shuffle experiment does not state whether the same random shuffle was applied to all models and both repetitions, nor how many shuffle trials were averaged; state this explicitly for reproducibility.
  5. [References] There are minor naming inconsistencies across the text and references, e.g., 'NeXT-QA' versus 'NextQA' and 'LLaV A' versus 'LLaVA'; normalize these names.

Circularity Check

1 steps flagged · score 4.0 of 10

Temporal/Semantic categories are defined by the two labeler models' shuffle behavior, making part of the benchmark-composition conclusion self-referential.

  1. self definitional [Section 4.3 (Temporal Questions); applied in Table 2 and Section 5.3]
    "if GPT-4o or Gemini-1.5-Pro answers the question correctly when provided with frames in their original order but fails to do so after the frames are shuffled, we classify the question as Temporal, indicating that the right sequential information is crucial for the answering process."

    The paper's central conclusion—that current video benchmarks overestimate temporal understanding—rests on the small proportion of Temporal questions reported in Table 2. But 'Temporal' is not an intrinsic property of a question; it is defined as a behavioral drop between original and shuffled frames for exactly the two labeler models (GPT-4o and Gemini-1.5-Pro). Section 3.2 already showed that both labelers are largely shuffle-insensitive, so the low gamma values are a restatement of that observation, not independent evidence about the benchmarks. The further assertion that these questions 'require understanding the correct temporal order' is the definition itself, not a separately verified property.

full rationale

The paper's initial observations (Sections 3.1 and 3.2) are not circular: they directly test several models with and without video and with shuffled frames, providing independent evidence that a single overall score can be inflated by language priors and static cues. However, VBenchComp formalizes these observations into categories defined by the behavior of just two proprietary models. The Temporal and Semantic labels are operational definitions tied to GPT-4o and Gemini-1.5-Pro, yet the paper reports the resulting proportions as properties of the benchmarks and uses them to conclude that benchmarks overestimate temporal understanding. This is a self-definitional step: the conclusion depends on the same model behavior used to construct the categories. The paper is transparent about the reliance on these two models in its Discussion, and no load-bearing self-citation or fitted-parameter renaming is present. The empirical results on other models still carry independent content, so the circularity is partial rather than total, warranting a score of 4 rather than higher.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The protocol has no fitted numerical constants. It does rely on several domain assumptions about the labeler ensemble and the shuffle procedure, plus the assumption that benchmark annotations are correct. No new physical or conceptual entities are introduced; VBenchComp and VBenchComp Score are metrics, not entities.

assumptions (4)
  • domain assumption The outputs of GPT-4o and Gemini-1.5-Pro can serve as ground truth for whether a question is answerable without video or is order-sensitive.
    Sections 4.1 to 4.3 assign every category from the correctness of these two closed models; if both share a blind spot, the labels are systematically wrong.
  • domain assumption A question answered correctly without video by both labelers is an intrinsic LLM-Answerable property of the benchmark, not merely of the two labelers.
    Section 4.1 defines the LLM-Answerable set this way, and Section 5.1 generalizes the result to all models.
  • domain assumption Two random shuffles are sufficient to test shuffle invariance.
    Sections 4.2 and 4.3 use two shuffles per question; no repeated-trial variance is reported, so a lucky or unlucky draw can change a label.
  • domain assumption The original benchmark ground-truth answers are correct and the answer-matching procedure used for open-ended benchmarks is adequate.
    All accuracy numbers in Tables 2 and 3 depend on these labels and on unspecified answer parsing for VCGBench and ActivityNet-QA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?." pith.science (2026). https://pith.science/paper/GIVJI7L7

@misc{pith2026250514321,
  author       = {Pith},
  title        = {Pith review of: Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GIVJI7L7}},
  note         = {Machine review of arXiv:2505.14321}
}
read the original abstract

Existing video understanding benchmarks often conflate knowledge-based and purely image-based questions, rather than clearly isolating a model's temporal reasoning ability, which is the key aspect that distinguishes video understanding from other modalities. We identify two major limitations that obscure whether higher scores truly indicate stronger understanding of the dynamic content in videos: (1) strong language priors, where models can answer questions without watching the video; and (2) shuffling invariance, where models maintain similar performance on certain questions even when video frames are temporally shuffled. To alleviate these issues, we propose VBenchComp, an automated pipeline that categorizes questions into different domains: LLM-Answerable, Semantic, and Temporal. Specifically, LLM-Answerable questions can be answered without viewing the video; Semantic questions remain answerable even when the video frames are shuffled; and Temporal questions require understanding the correct temporal order of frames. The rest of the questions are labeled as Others. This can enable fine-grained evaluation of different capabilities of a video LLM. Our analysis reveals nuanced model weaknesses that are hidden by traditional overall scores, and we offer insights and recommendations for designing future benchmarks that more accurately assess video LLMs.

Figures

Figures reproduced from arXiv: 2505.14321 by the authors.

Figure 1
Figure 1. Examples of LLM-Answerable, Semantic and Temporal questions in VideoMME [4]: (Top) The model uses LLM’s prior knowledge to answer correctly without the need of video; (Middle) The model relies on semantic understanding to answer without requiring temporal comprehension; (Bottom) The model relies on comprehensive temporal understanding to answer. model, raising serious concerns about the computational burden of bench… view at source ↗
Figure 2
Figure 2. Performance of different MLLMs without videos as the input on four benchmarks. Inspired by [25], we begin by evaluating whether questions can be answered without access to the corresponding videos. To do this, we input the questions into various MLLMs without providing the videos. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. After shuffling the extracted frames, the scores of each model remain unshaken across all benchmarks. *Frame settings: (a), (d) uses 128 frames for VideoMME-long, others use 64 frames; (b) uses 10slow + 50fast frames for all benchmarks; (c) uses 16 frames for all benchmarks. the models’ ability to comprehend and reason over temporal information, which is a core component of video understanding. 3.3 Potentially Misle… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: An overview of our standardized protocol: benchmark questions are categorized into four groups. Questions answerable by both GPT-4o and Gemini without video are classified as LLM-Answerable. For the remaining questions, we apply random shuffles to the extracted frames …
Figure 5
Figure 5. Figure 5: VBenchComp scores are aligned with the original scores but they can better evaluate the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...

  2. NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

    cs.CV 2026-01 conditional novelty 6.0 of 10

    NarrativeTrack shows that video AI models, including GPT-4o, falter at tracking a specific person across scene changes, outfit changes, and similar-looking characters in long videos.

Reference graph

Works this paper leans on

31 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    2024

  2. [2]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.NeurIPS, 2024

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.NeurIPS, 2024

    work page 2024

  3. [3]

    NExT-QA: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NExT-QA: Next phase of question-answering to explaining temporal actions. InCVPR, 2021

    work page 2021

  4. [4]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

  5. [5]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    arXiv 2024

  6. [6]

    Lvbench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024

    arXiv 2024

  7. [7]

    Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Systems, 36:42748– 42761, 2023

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Systems, 36:42748– 42761, 2023

    work page 2023

  8. [8]

    Palm: Scaling language modeling with pathways.JMLR, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.JMLR, 2023

    work page 2023

Show all 31 references
  1. [9]

    Llama 2: Open foundation and fine-tuned chat models.arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv:2307.09288, 2023

    2023 arXiv

  2. [10]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv:2303.08774, 2023

    2023 arXiv

  3. [11]

    Improved baselines with visual instruction tuning.arXiv:2310.03744, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning.arXiv:2310.03744, 2023

    2023 arXiv

  4. [12]

    LLaV A- NeXT: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A- NeXT: Improved reasoning, ocr, and world knowledge, 2024

    2024

  5. [13]

    Fewer tokens and fewer videos: Extending video understanding abilities in large vision-language models.arXiv:2406.08024, 2024

    Shimin Chen, Yitian Yuan, Shaoxiang Chen, Zequn Jie, and Lin Ma. Fewer tokens and fewer videos: Extending video understanding abilities in large vision-language models.arXiv:2406.08024, 2024

    2024 arXiv

  6. [14]

    Video-ChatGPT: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. InACL, 2024

    2024

  7. [15]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021

    2021

  8. [16]

    LLaV A-NeXT: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. LLaV A-NeXT: A strong zero-shot video understanding model, 2024

    2024

  9. [17]

    Videollamb: Long-context video understanding with recurrent memory bridges.arXiv:2409.01071, 2024

    Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long-context video understanding with recurrent memory bridges.arXiv:2409.01071, 2024. 10

    2024 arXiv

  10. [18]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling.arXiv:2501.12386, 2025

    2025 arXiv

  11. [19]

    Apollo: An exploration of video understanding in large multimodal models.arXiv:2412.10360, 2024

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models.arXiv:2412.10360, 2024

    2024 arXiv

  12. [20]

    Oryx MLLM: On-demand spatial-temporal understanding at arbitrary resolution.ICLR, 2025

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx MLLM: On-demand spatial-temporal understanding at arbitrary resolution.ICLR, 2025

    2025

  13. [21]

    VideoLLaMA 3: Frontier multimodal foundation models for image and video understanding.arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. VideoLLaMA 3: Frontier multimodal foundation models for image and video understanding.arXiv:2501.13106, 2025

    2025 arXiv

  14. [22]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. InACM Multimedia, 2017

    2017

  15. [23]

    ActivityNet-QA: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet-QA: A dataset for understanding complex web videos via question answering. InAAAI, 2019

    2019

  16. [24]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  17. [25]

    Lime: Less is more for mllm evaluation.arXiv preprint arXiv:2409.06851, 2024

    King Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li, Shawn Gavin, Tuney Zheng, Jiawei Guo, Bo Li, et al. Lime: Less is more for mllm evaluation.arXiv preprint arXiv:2409.06851, 2024

    2024 arXiv

  18. [26]

    SlowFast-LLaV A: A strong training-free baseline for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. SlowFast-LLaV A: A strong training-free baseline for video large language models. arXiv:2407.15841, 2024

    2024 arXiv

  19. [27]

    PLLaV A: Parameter-free llava extension from images to videos for video dense captioning.arXiv:2404.16994, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. PLLaV A: Parameter-free llava extension from images to videos for video dense captioning.arXiv:2404.16994, 2024

    2024 arXiv

  20. [28]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  21. [29]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  22. [30]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  23. [31]

    Video-LLaV A: Learning united visual representation by alignment before projection.arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual representation by alignment before projection.arXiv:2311.10122, 2023. 11

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.