VideoLLM Benchmarks and Evaluation: A Survey

Yogesh Kumar

REVIEW 4 major objections 6 minor 77 references

VideoLLM Benchmarks and Evaluation: A Survey

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey maps how video-language models are benchmarked and evaluated, and argues that open-set, LLM-judged scoring now dominates the field.

desk verdict A derivative but potentially useful survey whose main table—the one driving its central trend claim—has no per-row sources or protocol, making that claim uncheckable as written. read the letter →

arxiv 2505.03829 v1 pith:6O35T6UL submitted 2025-05-03 cs.CV cs.AI

Yogesh Kumar This is my paper · ORCID

classification cs.CVcs.AI

keywords VideoLLMunderstandingbenchmarksquestionansweringevaluationmethodologyopen-settemporalreasoningmultimodalintegration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper surveys how Video Large Language Models (VideoLLMs) are benchmarked and evaluated. It sets out to organise the field's benchmarks by size, duration, and task focus, and to classify evaluation into closed-set, open-set, and specialised temporal/spatiotemporal methods. The survey's main empirical observation, drawn from its own comparison table, is that newer VideoLLMs show a generational improvement in zero-shot question answering while still trailing a general-purpose multimodal model such as GPT-4V in generative dimensions like temporal understanding and context. A sympathetic reader comes away with a structured map of the terrain and a list of proposed next-generation benchmarks.

What carries the argument

The organising device is a pair of tables. Table I inventories more than twenty benchmarks by video count, clip count, average duration, and question-answer pairs; Table II compiles model scores on five generative dimensions (correctness, detail, context, temporal, consistency) plus zero-shot accuracy on MSVD-QA, MSRVTT-QA, and ActivityNet-QA, with rows sorted by temporal score. The analytical mechanism that produces the trend claims is the open-set evaluation protocol introduced by Video-ChatGPT, in which a GPT-based judge scores free-form model responses on those five dimensions. The survey also adopts an architectural taxonomy involving Video Analyzer × LLM, Video Embedder × LLM, and hybrid designs, using it to explain why some models excel at factual accuracy and others at temporal reasoning.

What would settle it

Take any row in Table II and compare it with the cited source paper; for instance, check whether Video-LLaMA's MSVD-QA score is 51.6 in [9] under zero-shot open-ended question answering. If the original paper reports a different number, split, or scoring method, the across-model ordering on which the survey's main trend rests cannot be reconstructed from the evidence it cites.

Watch

Extended reading notes

Core claim

The central claim is that VideoLLM evaluation has shifted from closed-set multiple-choice QA toward open-set protocols in which a large language model grades free-form answers, and that benchmark design is evolving in step: from short clips and factual questions to long videos, temporal reasoning, and multimodal integration. Examining the compiled scores, the paper asserts a clear generational improvement in model performance, citing the move from Video-LLaMA at 51.6 on MSVD-QA to IG-VLM at 76.7, and identifies a persistent gap between specialised VideoLLMs and GPT-4V on generative performance dimensions. It also reports that performance drops as video length grows, and that audio-visual models show more balanced scores. On the basis of these observations, the paper proposes six benchmark designs for future evaluation: hierarchical understanding, multimodal integration, long-form narrative, interactive evaluation, robustness and adversarial testing, and explainability.

Load-bearing premise

The trends in Section IV-A rest entirely on the scores compiled in Table II, but the paper does not disclose where those numbers come from, which prompts or judge models produced them, or whether every row was measured under the same protocol.

Editorial extensions

If this is right

If the reported trends hold, comparisons between VideoLLMs should concentrate on long-video, temporal, and multimodal tasks, since basic factual QA no longer separates the top models.
Open-set LLM-judged scoring becomes the de facto currency for reporting VideoLLM capability, making the choice of judge model a source of cross-paper variation.
The proposed hierarchical and long-form narrative benchmarks would allow each capability to be measured separately, which could change how model strengths and weaknesses are ranked.
The persistence of a gap with general-purpose multimodal models suggests that video-specific architectures should direct their next steps at temporal and contextual integration rather than factual recognition.
Because longer videos consistently produce lower scores, context retention across extended durations is the binding constraint for real-world VideoLLM deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the paper leaves implicit: the five-dimension scoring protocol likely carries judge-induced variance, so re-scoring one fixed set of model outputs with different judge models would quantify how much of Table II's ordering is an artifact of the evaluator.
A controlled extension the paper does not run: vary only the duration of the same narrative content across conditions; if accuracy declines smoothly with length, context retention is causal, whereas a step change would implicate specific architectural bottlenecks.
The survey aggregates rows from different source papers with unknown protocols; a natural editorial follow-up is to rebuild the comparison on a single execution harness before treating the generational trend as established.
The proposed explainability benchmark presupposes a shared standard for what counts as a good explanation; defining and validating that standard may be a prerequisite for the benchmark to be usable.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this survey does exactly what it says, organizing current VideoLLM benchmarks and evaluation methods, but the one empirical table behind its headline “generational improvement” claim has no sources and no stated evaluation protocol, so the trend is not checkable as written.

What is genuinely useful: Table II aggregates zero-shot QA and generative performance scores for many VideoLLMs in one place, and the prose around closed-set versus open-set evaluation is a reasonable orientation for newcomers. The benchmark evolution narrative—short clips to long-form, general to specialized—is accurate at a high level. The list of future directions is broad, though so broad that much of it reads as a checklist.

The soft spots are real and load-bearing. Table II sorts models by temporal score and feeds the paper's main trend claim, yet no row cites its source paper, and the text never states which prompt template, judge model, video subset, or decoding settings were used. Different rows may come from different papers with different protocols. This is not an accusation that the numbers are wrong; it is that the central empirical assertion is currently unverifiable from the manuscript. The stress-test note lands. Table I also has an apparent error: TGIF-QA is listed with 8,506 QA pairs, which is far below the original dataset's size, and the text repeats the same number. That looks like a copy error. Some citations are off-topic (the first reference, Frozen in Time, is an odd fit for the video-challenges point), and the paper leans heavily on Tang et al. [12], which is fine but leaves little original content.

The paper makes no falsifiable claim and introduces no new benchmark, metric, or measurement, so it is not a research contribution. As a survey, it would be useful if the tables were trustworthy, and they are not yet.

My recommendation: this deserves a peer-review slot rather than a desk reject, but with a clear instruction to add per-cell sources and protocol details to both tables, correct the TGIF-QA numbers, and soften any claim that depends on the unverified aggregation. The organizational skeleton is decent, and a corrected version would be a citable orientation for someone entering the field. As is, I would not cite it.

Referee Report

4 major / 6 minor

Summary. This manuscript is a survey of benchmarks and evaluation methodologies for Video Large Language Models (VideoLLMs). It inventories approximately 20 video QA and reasoning benchmarks in Table I (video counts, clip counts, durations, QA-pair counts), organizes evaluation practice into closed-set, open-set (LLM-judged), and specialized temporal/spatiotemporal categories in Section III, and aggregates zero-shot QA accuracies plus five generative-performance dimensions for roughly 28 models in Table II. Section IV reads performance trends and architectural observations off that aggregation, Sections V and VI propose challenges and future benchmark designs, and Section VII catalogues application domains. The paper explicitly positions itself as building on the prior survey of Tang et al. [12].

Significance. If the aggregation in Table II were properly documented and the factual entries of Table I corrected, this survey would be a useful reference for practitioners: it assembles a broad and recent benchmark inventory with duration statistics, articulates a workable three-way taxonomy of evaluation styles, and names concrete gaps (hallucination, long-form video, cross-modal integration, interactive evaluation). The paper makes no predictions and fits no parameters, so there is no fit-to-data circularity; its value is organizational. The contribution is modest relative to the acknowledged prior survey [12], and the claimed performance trends are currently not verifiable from the manuscript because Table II lacks per-row provenance and evaluation-protocol details, while at least one Table I entry (TGIF-QA) is internally inconsistent. If the provenance and factual issues are resolved, the benchmark inventory and taxonomy would be of genuine use to the VideoLLM community.

major comments (4)

[Table II / Section III-B / Section IV-A] The central empirical content of the survey is Table II, and the first trend asserted in Section IV-A ('there is a clear generational improvement in model performance, with newer models consistently outperforming their predecessors') is read directly off that table, yet no row in Table II cites its source and the text nowhere states the evaluation protocol used for the generative scores (which judge LLM, which prompt template, which video-sampling scheme, which decoding settings, whether all rows were produced under a common codebase). The paper itself acknowledges in Section III-B that 'different evaluator models may produce different scores for the same response,' which makes the absence of judge-model reporting in Table II a direct threat to cross-row comparability: rows for the Video-ChatGPT-era models originate from [11], while later rows (PLLaVA, IG-VLM, ST-LLM) necessarily come from their own papers and may have been graded by different judges or prompts. Missing cells (e.g., GPT4-V with no MSVD-QA or MSRVTT-QA, Video-LLaMA 2 with no MSRVTT-QA) are unexplained. Please add a per-row source column, state the exact protocol (or explicitly assert that every row was produced under the [11] protocol), and discuss the missing cells; otherwise the Section IV-A trends are not checkable.
[Section IV-A / Table II] The claim that newer models 'consistently outperform their predecessors' is contradicted by Table II itself. Within the same table, VideoGPT+, a later model, has the lowest temporal score (1.78, below Video-ChatGPT's 2.16); Video-LLaMA 2 scores 2.63 on temporal, exceeding the newer PLLaVA (2.33) and IG-VLM (2.34); and Chat-UniVi's temporal score (2.89) exceeds VideoChat2's (2.66). Monotonic improvement is at best visible in the MSVD-QA column (51.6 to 76.7), and the text should qualify the word 'consistently' accordingly. The same paragraph's closing claim that audio-visual models such as 'AV-LLM and AVicuna often show more balanced performance' is also not supported by the table: AVicuna (2.81/2.62/3.25/2.53/2.59) and AV-LLM (2.56/2.47/2.93/2.17/2.47) are neither the highest nor the most balanced rows, compared with, for example, LLaVA-NeXT-Video at 3.39/3.29/3.92/2.60/3.12.
[Table I / Section II-A] The TGIF-QA entry is internally inconsistent and factually wrong. Table I lists 8,506 QA pairs for 9,575 clips, and the prose repeats '9,575 short animated GIFs ... and 8,506 question-answer pairs'; the original TGIF-QA dataset contains on the order of 165k QA pairs across roughly 100k GIFs, and a QA count smaller than the clip count is implausible for a dataset with multiple questions per video. Additionally, several rows (MSVD-QA 504/13,157; MSRVTT-QA 2,990/72,821; NExT-QA 1,000/8,564; ActivityNet-QA 800/8,000) appear to be evaluation subsets or protocol-dependent splits rather than original dataset statistics, but Table I has no per-row source column and the text does not distinguish original-dataset statistics from evaluation subsets. Add per-row citations and a 'subset/protocol' flag where applicable.
[Section IV-B / Table II] The claims linking architecture class to evaluation results are asserted as if read off Table II, but the models invoked are largely absent from the table, and the table contradicts the stated patterns. VideoChat is classified as a Video Analyzer x LLM model in one paragraph and then as a hybrid (Analyzer + Embedder) model in the next ('Hybrid models that combine analyzer and embedder approaches, such as VideoChat and Vid2Seq'), an explicit internal contradiction. VTimeLLM is cited as exemplifying better temporal performance among Embedder x LLM models, yet its temporal score (2.49) is lower than its correctness (2.78), detail (3.10), and context (3.40) in Table II. ChatVideo, TimeChat, and Vid2Seq never appear in Table II. Either the claims should be derived strictly from the presented data, or each claim should be attributed to the specific analysis in [12] with the relevant section cited.

minor comments (6)

[Section VI-F] The itemized list is mis-numbered: it contains '(ii) Evidence grounding' followed by a second '(ii) Reasoning transparency' and a stray '(v).' with a period; renumber the list as (i)-(v).
[Table II caption / Section III-B] The phrase 'GPT-based metrics for MSVD-QA, MSRVTT-QA, and ActivityNet-QA' conflates the QA accuracy columns with the GPT-based generative scores; the QA columns are accuracy percentages, while GPT-based scoring applies to the five generative dimensions.
[Table II] The table is described as 'sorted by temporal understanding performance,' but the ordering is violated by the PLLaVA (2.33) / IG-VLM (2.34) pair, which appears in decreasing order after the higher 2.34 of RED-VILLM; either re-sort the table or drop the sorting claim.
[Throughout] Model names are typeset inconsistently: 'Video LLaMA 2' vs 'Video-LLaMA', 'AVicuna' (presumably A-Vicuna), 'RED-VILLM' (presumably Red-VILLM), and spacing artifacts such as 'LLaV A' and 'V ALL Y'; normalize all model names across text, tables, and captions.
[References] Reference [26] (AutoEval-Video) lacks an arXiv identifier, and several references mix venue names with bare 'arXiv preprint' descriptors without consistent identifiers; add the missing arXiv IDs or DOIs throughout.
[Table I] Average duration is listed as '-' for SOK-Bench and EditVid-QA with no explanation; either supply the values or state explicitly that duration is not reported for these benchmarks.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the survey aggregates external benchmark results and proposes future directions; its empirical claims are organizational, not derived from fitted inputs or self-citations.

full rationale

This manuscript is a survey, not a derivation: it contains no fitted parameters, no equations whose inputs are renamed as outputs, and no prediction that is constructed from the data it claims to explain. Its central claims are organizational—describing benchmarks, evaluation methodologies, performance trends, and future directions—and the performance discussion in Section IV-A reads trends directly from Table II, which aggregates numbers from external papers as reported. That practice raises checkability and provenance concerns (Table II has no per-row source citations and the evaluation protocol is not specified), but source dependence is not circularity: the table is consumed as external evidence rather than generated by the manuscript. There are no load-bearing self-citations; the author cites prior surveys such as Tang et al. only to position the survey and to adopt their model taxonomy, not to justify a result that is otherwise unverified. No uniqueness theorem, ansatz, or fitted quantity is imported to make the conclusions tautological. Accordingly, the appropriate circularity score is 0, with the caveat that the evidentiary reliability of the undocumented performance numbers is a separate correctness risk.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The paper contributes no new fitted parameters, axioms, or invented entities. Its claims rest on accepting the cited literature and reproduced tables at face value, which is a domain assumption rather than a derivation.

assumptions (1)

domain assumption All benchmark statistics and model performance scores cited from prior work are accurate and directly comparable.
The survey does not verify the numbers in Table I and Table II or standardize protocols across the sources, yet it draws performance-trend conclusions from them.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VideoLLM Benchmarks and Evaluation: A Survey." pith.science (2026). https://pith.science/paper/6O35T6UL

@misc{pith2026250503829,
  author       = {Pith},
  title        = {Pith review of: VideoLLM Benchmarks and Evaluation: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6O35T6UL}},
  note         = {Machine review of arXiv:2505.03829}
}

read the original abstract

The rapid development of Large Language Models (LLMs) has catalyzed significant advancements in video understanding technologies. This survey provides a comprehensive analysis of benchmarks and evaluation methodologies specifically designed or used for Video Large Language Models (VideoLLMs). We examine the current landscape of video understanding benchmarks, discussing their characteristics, evaluation protocols, and limitations. The paper analyzes various evaluation methodologies, including closed-set, open-set, and specialized evaluations for temporal and spatiotemporal understanding tasks. We highlight the performance trends of state-of-the-art VideoLLMs across these benchmarks and identify key challenges in current evaluation frameworks. Additionally, we propose future research directions to enhance benchmark design, evaluation metrics, and protocols, including the need for more diverse, multimodal, and interpretability-focused benchmarks. This survey aims to equip researchers with a structured understanding of how to effectively evaluate VideoLLMs and identify promising avenues for advancing the field of video understanding with large language models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 50 canonical work pages

[12]

Video understanding with large language models: A survey,

Y . Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An, J. Lin, R. Zhu et al. , “Video understanding with large language models: A survey,” arXiv preprint arXiv:2312.17432 , 2024

arXiv 2024
[11]

Video- chatgpt: Towards detailed video understanding via large vi sion and language models,

M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video- chatgpt: Towards detailed video understanding via large vi sion and language models,” arXiv preprint arXiv:2306.05424 , 2023

arXiv 2023
[1]

Frozen i n time: A joint video and image encoder for end-to-end retriev al,

M. Bain, A. Nagrani, G. V arol, and A. Zisserman, “Frozen i n time: A joint video and image encoder for end-to-end retriev al,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1728–1738

work page 2021
[2]

Scale invariant feature transform,

T. Lindeberg, “Scale invariant feature transform,” 201 2

work page
[3]

Speeded-u p ro- bust features (surf),

H. Bay, A. Ess, T. Tuytelaars, and L. V an Gool, “Speeded-u p ro- bust features (surf),” Computer vision and image understanding , vol. 110, no. 3, pp. 346–359, 2008

work page 2008
[4]

Histograms of oriented gradient s for human detection,

N. Dalal and B. Triggs, “Histograms of oriented gradient s for human detection,” in 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05) , vol. 1. Ieee, 2005, pp. 886–893

work page 2005
[5]

Large-scale video classiﬁcation with conv olu- tional neural networks,

A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthan kar, and L. Fei-Fei, “Large-scale video classiﬁcation with conv olu- tional neural networks,” in CVPR, 2014

work page 2014
[6]

Convoluti onal two-stream network fusion for video action recognition,

C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convoluti onal two-stream network fusion for video action recognition,” i n Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1933–1941

work page 2016

Show all 77 references

[7]

Videobert: A joint model for video and language representa - tion learning,

C. Sun, A. Myers, C. V ondrick, K. Murphy, and C. Schmid, “Videobert: A joint model for video and language representa - tion learning,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 7464–7473

2019
[8]

Videomae: Masked autoencoders are data-efﬁcient learners for self-supervi sed video pre-training,

Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoencoders are data-efﬁcient learners for self-supervi sed video pre-training,” Advances in neural information processing sys- tems, vol. 35, pp. 10 078–10 093, 2022

2022
[9]

Video-llama: An instructio n- tuned audio-visual language model for video understanding ,

H. Zhang, X. Li, and L. Bing, “Video-llama: An instructio n- tuned audio-visual language model for video understanding ,” arXiv preprint arXiv:2306.02858 , 2023

2023 arXiv
[10]

Videochat: Chat-centric video under- standing,

K. Li, Y . He, Y . Wang, Y . Li, W. Wang, P . Luo, Y . Wang, L. Wang, and Y . Qiao, “Videochat: Chat-centric video under- standing,” arXiv preprint arXiv:2305.06355 , 2023

2023 arXiv
[13]

Video question answering via gradually reﬁned attention o ver appearance and motion,

D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y . Zhuan g, “Video question answering via gradually reﬁned attention o ver appearance and motion,” in Proceedings of the 25th ACM international conference on Multimedia , 2017, pp. 1645–1653

2017
[14]

Tgif-qa: Toward spatio-temporal reasoning in visual question answering,

Y . Jang, Y . Song, Y . Y u, Y . Kim, and G. Kim, “Tgif-qa: Toward spatio-temporal reasoning in visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2758–2766

2017
[15]

Activitynet-qa: A dataset for understanding complex web videos via question answering,

Z. Y u, D. Xu, J. Y u, T. Y u, Z. Zhao, Y . Zhuang, and D. Tao, “Activitynet-qa: A dataset for understanding complex web videos via question answering,” in Proceedings of the AAAI Conference on Artiﬁcial Intelligence , vol. 33, no. 01, 2019, pp. 9127–9134

2019
[16]

Tvqa: Localized , compositional video question answering,

J. Lei, L. Y u, M. Bansal, and T. L. Berg, “Tvqa: Localized , compositional video question answering,” in EMNLP, 2018

2018
[17]

Mvbench: A comprehensive multi-modal video understanding benchmark,

K. Li, Y . Wang, Y . He, Y . Li, Y . Wang, Y . Liu, Z. Wang, J. Xu, G. Chen, P . Luo et al. , “Mvbench: A comprehensive multi-modal video understanding benchmark,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 195–22 206

2024
[18]

Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models,

M. Ning, B. Zhu, Y . Xie, B. Lin, J. Cui, L. Y uan, D. Chen, and L. Y uan, “Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models, ” arXiv preprint arXiv:2311.16103, 2023

2023 arXiv
[19]

Videovista: A versatile benchmark for video understandin g and reasoning,

Y . Li, X. Chen, B. Hu, L. Wang, H. Shi, and M. Zhang, “Videovista: A versatile benchmark for video understandin g and reasoning,” arXiv preprint arXiv:2406.11303 , 2024

2024 arXiv
[20]

Cinepile: A long video question answer- ing dataset and benchmark,

R. Rawal, K. Saifullah, R. Basri, D. Jacobs, G. Somepall i, and T. Goldstein, “Cinepile: A long video question answer- ing dataset and benchmark,” arXiv preprint arXiv:2405.08813 , 2024

2024 arXiv
[21]

Inﬁnibench: A comprehensive benchmark for large multimodal models in very long video understanding,

K. Ataallah, C. Gou, E. Abdelrahman, K. Pahwa, J. Ding, a nd M. Elhoseiny, “Inﬁnibench: A comprehensive benchmark for large multimodal models in very long video understanding,” arXiv preprint arXiv:2406.19875 , 2024

2024
[22]

Tempcompass: Do video llms really understand videos?

Y . Liu, S. Li, Y . Liu, Y . Wang, S. Ren, L. Li, S. Chen, X. Sun , and L. Hou, “Tempcompass: Do video llms really understand videos?” arXiv preprint arXiv:2403.00476 , 2024

2024 arXiv
[23]

Her o: Hierarchical encoder for video+ language omni-representa tion pre-training,

L. Li, Y .-C. Chen, Y . Cheng, Z. Gan, L. Y u, and J. Liu, “Her o: Hierarchical encoder for video+ language omni-representa tion pre-training,” in EMNLP, 2020

2020
[24]

Star: A benchmark for situated reasonin g in real-world videos,

B. Wu and S. Y u, “Star: A benchmark for situated reasonin g in real-world videos,” in NeurIPS, 2024

2024
[25]

Egoschema: A diagnostic benchmark for very long-form video language understanding,

K. Mangalam, R. Akshulakov, and J. Malik, “Egoschema: A diagnostic benchmark for very long-form video language understanding,” arXiv preprint arXiv:2308.09126 , 2023

2023 arXiv
[26]

Autoeval-video : An automatic benchmark for assessing large vision language mo d- els in open-ended video question answering,

X. Chen, Y . Lin, Y . Zhang, and W. Huang, “Autoeval-video : An automatic benchmark for assessing large vision language mo d- els in open-ended video question answering,” ArXiv preprint , 2023

2023
[27]

Video-mme: The ﬁrst- ever comprehensive evaluation benchmark of multi-modal ll ms in video analysis,

C. Fu, Y . Dai, Y . Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y . Shen, M. Zhang et al. , “Video-mme: The ﬁrst- ever comprehensive evaluation benchmark of multi-modal ll ms in video analysis,” arXiv preprint arXiv:2405.21075 , 2024

2024 arXiv
[28]

Sok-bench: A situated video reasoning bench - mark with aligned open-world knowledge,

A. Wang, B. Wu, S. Chen, Z. Chen, H. Guan, W.-N. Lee, L. E. Li, and C. Gan, “Sok-bench: A situated video reasoning bench - mark with aligned open-world knowledge,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 384–13 394

2024
[29]

Short ﬁlm dataset (sfd): A benchmark for story-level video understan ding,

R. Ghermi, X. Wang, V . Kalogeiton, and I. Laptev, “Short ﬁlm dataset (sfd): A benchmark for story-level video understan ding,” arXiv preprint arXiv:2406.10221 , 2024

2024 arXiv
[30]

Mlvu: A comprehensive benchmark for multi-task long video understanding,

J. Zhou, Y . Shu, B. Zhao, B. Wu, S. Xiao, X. Yang, Y . Xiong, B. Zhang, T. Huang, and Z. Liu, “Mlvu: A comprehensive benchmark for multi-task long video understanding,” arXiv preprint arXiv:2406.04264, 2024

2024 arXiv
[31]

Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos,

X. He, W. Feng, K. Zheng, Y . Lu, W. Zhu, J. Li, Y . Fan, J. Wang, L. Li, Z. Yang et al. , “Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos,” arXiv preprint arXiv:2406.08407, 2024

2024 arXiv
[32]

V elociti: Can video-language models bind sema n- tic concepts through time?

D. Saravanan, D. Singh, V . Gupta, Z. Khan, V . Gandhi, and M. Tapaswi, “V elociti: Can video-language models bind sema n- tic concepts through time?” arXiv preprint arXiv:2406.10889 , 2024

2024 arXiv
[33]

Next-qa: Next phase of question-answering to explaining temporal action s,

J. Xiao, X. Shang, A. Yao, and T.-S. Chua, “Next-qa: Next phase of question-answering to explaining temporal action s,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021, pp. 9777–9786

2021
[34]

Beyond raw videos: Understand- ing edited videos with large multimodal model,

L. Xu, S. Zhu, C. Li, C.-W. Kuo, F. Chen, X. Wang, G. Chen, D. Du, Y . Y uan, and L. Wen, “Beyond raw videos: Understand- ing edited videos with large multimodal model,” arXiv preprint arXiv:2406.10484, 2024. 12

2024 arXiv
[35]

Cider : Consensus-based image description evaluation,

R. V edantam, C. Lawrence Zitnick, and D. Parikh, “Cider : Consensus-based image description evaluation,” in Proceedings of the IEEE conference on computer vision and pattern recog- nition, 2015, pp. 4566–4575

2015
[36]

Meteor: An automatic metric f or mt evaluation with improved correlation with human judgments ,

S. Banerjee and A. Lavie, “Meteor: An automatic metric f or mt evaluation with improved correlation with human judgments ,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summar iza- tion, 2005, pp. 65–72

2005
[37]

Rouge: A package for automatic evaluation o f summaries,

C.-Y . Lin, “Rouge: A package for automatic evaluation o f summaries,” in Text summarization branches out , 2004, pp. 74– 81

2004
[38]

Spi ce: Semantic propositional image caption evaluation,

P . Anderson, B. Fernando, M. Johnson, and S. Gould, “Spi ce: Semantic propositional image caption evaluation,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 1 4. Springer, 2016, pp. 382–398

2016
[39]

Plla va: Parameter-free llava extension from images to videos for vi deo dense captioning,

L. Xu, Y . Zhao, D. Zhou, Z. Lin, S. K. Ng, and J. Feng, “Plla va: Parameter-free llava extension from images to videos for vi deo dense captioning,” arXiv preprint arXiv:2404.16994 , 2024

2024 arXiv
[40]

An image grid can be worth a video: Zero-shot video question answering using a vl m,

W. Kim, C. Choi, W. Lee, and W. Rhee, “An image grid can be worth a video: Zero-shot video question answering using a vl m,”
[41]

St-llm: Large language models are effective temporal learners,

R. Liu, C. Li, H. Tang, Y . Ge, Y . Shan, and G. Li, “St-llm: Large language models are effective temporal learners,” arXiv preprint arXiv:2404.00308, 2024

2024 arXiv
[42]

Large language models know your contextual search intent: A prompting framework for conversational search,

K. Mao, Z. Dou, H. Chen, F. Mo, and H. Qian, “Large language models know your contextual search intent: A prompting framework for conversational search,” arXiv preprint arXiv:2303.06573, 2023

2023 arXiv
[43]

Prompting visual-language models for efﬁcient video understanding,

C. Ju, T. Han, K. Zheng, Y . Zhang, and W. Xie, “Prompting visual-language models for efﬁcient video understanding, ” in European Conference on Computer Vision . Springer, 2022, pp. 105–124

2022
[44]

Vid2seq: Large-scale pr e- training of a visual language model for dense video captioni ng,

A. Yang, A. Nagrani, P . H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid, “Vid2seq: Large-scale pr e- training of a visual language model for dense video captioni ng,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 20...

2023
[45]

Lea rning video representations from large language models,

Y . Zhao, I. Misra, P . Kr¨ ahenb¨ uhl, and R. Girdhar, “Lea rning video representations from large language models,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6586–6597

2023
[46]

Text-video retrieval with disentangled conceptu aliza- tion and set-to-set alignment,

P . Jin, H. Li, Z. Cheng, J. Huang, Z. Wang, L. Y uan, C. Liu, and J. Chen, “Text-video retrieval with disentangled conceptu aliza- tion and set-to-set alignment,” arXiv preprint arXiv:2305.12218, 2023

2023 arXiv
[47]

Diffusionret: Generative text-video retrieval w ith diffusion model,

P . Jin, H. Li, Z. Cheng, K. Li, X. Ji, C. Liu, L. Y uan, and J. Chen, “Diffusionret: Generative text-video retrieval w ith diffusion model,” arXiv preprint arXiv:2303.09867 , 2023

2023 arXiv
[48]

Egovlpv2: Egocentric vid eo- language pre-training with fusion in the backbone,

S. Pramanick, Y . Song, S. Nag, K. Q. Lin, H. Shah, M. Z. Shou, R. Chellappa, and P . Zhang, “Egovlpv2: Egocentric vid eo- language pre-training with fusion in the backbone,” in Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp....

2023
[49]

Large language models in education: Vision and opportunities,

W. Gan, Z. Qi, J. Wu, and J. C.-W. Lin, “Large language models in education: Vision and opportunities,” arXiv preprint arXiv:2311.13160, 2023

2023 arXiv
[50]

A survey on deep multi-modal learning for body language recognition and generation,

L. Liu, L. Gao, W. Lei, F. Ma, X. Lin, and J. Wang, “A survey on deep multi-modal learning for body language recognition and generation,” arXiv preprint arXiv:2308.08849 , 2023

2023 arXiv
[51]

Machine translation from signed to spoken lan- guages: State of the art and challenges,

M. De Coster, D. Shterionov, M. V an Herreweghe, and J. Dambre, “Machine translation from signed to spoken lan- guages: State of the art and challenges,” Universal Access in the Information Society , pp. 1–27, 2023

2023
[52]

Generating video game quests from storie s,

M. K. Mishra, “Generating video game quests from storie s,” Master’s thesis, University of Twente, 2023

2023
[53]

Text generation for quests in multiplayer r ole- playing video games,

S. Koomen, “Text generation for quests in multiplayer r ole- playing video games,” Master’s thesis, University of Twent e, 2023

2023
[54]

The role of artiﬁcial intelligence and robotic solution technologies in metaverse design,

N. Gokce Narin, “The role of artiﬁcial intelligence and robotic solution technologies in metaverse design,” in Metaverse: Tech- nologies, Opportunities and Threats . Springer, 2023, pp. 45– 63

2023
[55]

Jung and M

T. Jung and M. C. tom Dieck, XR-Metaverse Cases: Business Application of AR, VR, XR and Metaverse . Springer Nature, 2023

2023
[56]

Promptﬁx: Y ou prompt and we ﬁx the photo,

Y . Y u, Z. Zeng, H. Hua, J. Fu, and J. Luo, “Promptﬁx: Y ou prompt and we ﬁx the photo,” arXiv preprint arXiv:2405.16785, 2024

2024 arXiv
[57]

Egocentric audio - visual object localization,

C. Huang, Y . Tian, A. Kumar, and C. Xu, “Egocentric audio - visual object localization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 22 910–22 921

2023
[58]

Misar: A multimodal instructional system with augmented reality,

J. Bi, N. M. Nguyen, A. V osoughi, and C. Xu, “Misar: A multimodal instructional system with augmented reality,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 1–5

2023
[59]

Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,

K. Rana, J. Haviland, S. Garg, J. Abou-Chakra, I. Reid, a nd N. Suenderhauf, “Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning,” in Conference on Robot Learning . PMLR, 2023, pp. 23–72

2023
[60]

The role of chatgpt, generative language models, and artiﬁcial intelligence in medical education: a con- versation with chatgpt and a call for papers,

G. Eysenbach et al. , “The role of chatgpt, generative language models, and artiﬁcial intelligence in medical education: a con- versation with chatgpt and a call for papers,” JMIR Medical Education, vol. 9, no. 1, p. e46885, 2023

2023
[61]

Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesi s,

H. Liu, Z. Zhu, N. Iwamoto, Y . Peng, Z. Li, Y . Zhou, E. Bozk urt, and B. Zheng, “Beat: A large-scale semantic and emotional multi-modal dataset for conversational gestures synthesi s,” in European conference on computer vision . Springer, 2022, pp. 612–630

2022
[62]

Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis,

H. Liu, N. Iwamoto, Z. Zhu, Z. Li, Y . Zhou, E. Bozkurt, and B. Zheng, “Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis,” in Proceed- ings of the 30th ACM International Conference on Multimedia , 2022, pp. 3764–3773

2022
[63]

Emage: Towards uni- ﬁed holistic co-speech gesture generation via expressive m asked audio gesture modeling,

H. Liu, Z. Zhu, G. Becherini, Y . Peng, M. Su, Y . Zhou, X. Zh e, N. Iwamoto, B. Zheng, and M. J. Black, “Emage: Towards uni- ﬁed holistic co-speech gesture generation via expressive m asked audio gesture modeling,” in Proceedings of the IEEE/CVF Conference on Computer Vision ...

2024
[64]

Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” arXiv preprint arXiv:2306.00890, 2023

2023 arXiv
[65]

Chatgpt for cybersecurity: practical applications, challenges, a nd future directions,

M. Al-Hawawreh, A. Aljuhani, and Y . Jararweh, “Chatgpt for cybersecurity: practical applications, challenges, a nd future directions,” Cluster Computing , vol. 26, no. 6, pp. 3421–3436, 2023

2023
[66]

Modelling language for cyber security incide nt handling for critical infrastructures,

H. Mouratidis, S. Islam, A. Santos-Olmo, L. E. Sanchez, and U. M. Ismail, “Modelling language for cyber security incide nt handling for critical infrastructures,” Computers & Security , vol. 128, p. 103139, 2023

2023
[67]

Socratic video un- derstanding on unmanned aerial vehicles,

I. de Zarz` a, J. de Curt` o, and C. T. Calafate, “Socratic video un- derstanding on unmanned aerial vehicles,” Procedia Computer Science, vol. 225, pp. 144–154, 2023

2023
[68]

Lanobert: System log anomal y detection based on bert masked language model,

Y . Lee, J. Kim, and P . Kang, “Lanobert: System log anomal y detection based on bert masked language model,” Applied Soft Computing, vol. 146, p. 110689, 2023

2023
[69]

Logﬁt : Log anomaly detection using ﬁne-tuned language models,

C. Almodovar, F. Sabrina, S. Karimi, and S. Azad, “Logﬁt : Log anomaly detection using ﬁne-tuned language models,” 2023

2023
[70]

Graphgpt: Graph instruction tuning for large language models,

J. Tang, Y . Yang, W. Wei, L. Shi, L. Su, S. Cheng, D. Yin, and C. Huang, “Graphgpt: Graph instruction tuning for large language models,” arXiv preprint arXiv:2310.13023 , 2023

2023 arXiv
[71]

Drive as you speak: Enabling human-like interaction with large lan- guage models in autonomous vehicles,

C. Cui, Y . Ma, X. Cao, W. Ye, and Z. Wang, “Drive as you speak: Enabling human-like interaction with large lan- guage models in autonomous vehicles,” in Proceedings of the 13 IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 902–909

2024
[72]

Ott er: A multi-modal model with in-context instruction tuning,

B. Li, Y . Zhang, L. Chen, J. Wang, J. Yang, and Z. Liu, “Ott er: A multi-modal model with in-context instruction tuning,” arXiv preprint arXiv:2305.03726, 2023

2023 arXiv
[73]

Lisa: Reasoning segmentation via large language model,

X. Lai, Z. Tian, Y . Chen, Y . Li, Y . Y uan, S. Liu, and J. Jia, “Lisa: Reasoning segmentation via large language model,” arXiv preprint arXiv:2308.00692, 2023

2023 arXiv
[74]

How retail video analytics enhances cust omer experience,

eInfochips, “How retail video analytics enhances cust omer experience,” eInfochips Blog , 2025. [Online]. Available: https://www.einfochips.com/blog/retail-video-analytics-enhances-customer-experience/

2025
[75]

How video analytics is transform ing the luxury retail experience,

Staqu Technologies, “How video analytics is transform ing the luxury retail experience,” Staqu Blog, 2025. [Online]. Available: https://www.staqu.com/how-video-analytics-is-transforming-the-luxury-retail-experience/

2025
[76]

Visual search and its growing inﬂuence on e-commerce,

Electrono Solutions, “Visual search and its growing inﬂuence on e-commerce,” Electrono Solutions Blog , July 2018. [Online]. Available: https://electronosolutions.com/visual-search-and-it s-growing-inﬂuence-on-e-commerce/

2018
[2024]

Available: https://arxiv.org/abs/2403

[Online]. Available: https://arxiv.org/abs/2403. 18406

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [12]

Video understanding with large language models: A survey,

Y . Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An, J. Lin, R. Zhu et al. , “Video understanding with large language models: A survey,” arXiv preprint arXiv:2312.17432 , 2024

arXiv 2024

[2] [11]

Video- chatgpt: Towards detailed video understanding via large vi sion and language models,

M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, “Video- chatgpt: Towards detailed video understanding via large vi sion and language models,” arXiv preprint arXiv:2306.05424 , 2023

arXiv 2023

[3] [1]

Frozen i n time: A joint video and image encoder for end-to-end retriev al,

M. Bain, A. Nagrani, G. V arol, and A. Zisserman, “Frozen i n time: A joint video and image encoder for end-to-end retriev al,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1728–1738

work page 2021

[4] [2]

Scale invariant feature transform,

T. Lindeberg, “Scale invariant feature transform,” 201 2

work page

[5] [3]

Speeded-u p ro- bust features (surf),

H. Bay, A. Ess, T. Tuytelaars, and L. V an Gool, “Speeded-u p ro- bust features (surf),” Computer vision and image understanding , vol. 110, no. 3, pp. 346–359, 2008

work page 2008

[6] [4]

Histograms of oriented gradient s for human detection,

N. Dalal and B. Triggs, “Histograms of oriented gradient s for human detection,” in 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05) , vol. 1. Ieee, 2005, pp. 886–893

work page 2005

[7] [5]

Large-scale video classiﬁcation with conv olu- tional neural networks,

A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthan kar, and L. Fei-Fei, “Large-scale video classiﬁcation with conv olu- tional neural networks,” in CVPR, 2014

work page 2014

[8] [6]

Convoluti onal two-stream network fusion for video action recognition,

C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convoluti onal two-stream network fusion for video action recognition,” i n Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1933–1941

work page 2016