Pith. sign in

REVIEW 4 major objections 4 minor 5 cited by

Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Video-MMLU, a benchmark of 1,065 lecture videos in mathematics, physics, and chemistry, reports that current video models score only 10-50%, with the best model at 69.3%, showing multimodal AI cannot yet reliably perceive and reason over…

desk verdict A large and useful captioning benchmark for lecture videos, but the quiz track does not establish a video-perception gap: the vision-blind baseline already scores 40.5%. read the letter →

arxiv 2504.14693 v2 pith:Q3QPYUON submitted 2025-04-20 cs.CV cs.AI

Enxin Song , Wenhao Chai , Weili Xu , Jianwen Xie , Yuxuan Liu , Gaoang Wang This is my paper · ORCID
classification cs.CVcs.AI
keywords videounderstandingbenchmarklecturecomprehensionlargemultimodalmodelsvisualtokencompressionquestionansweringdensecaptioningmulti-disciplinereasoningeducational
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-MMLU is a benchmark built from 1,065 short lecture videos in mathematics, physics, and chemistry that asks multimodal AI models (systems pairing a language model with visual input) to do two things: write detailed captions of what appears on screen, and answer open-ended quiz questions about the material. The paper's central claim is that current video-capable large multimodal models do poorly at both, scoring mostly between 10% and 50%, with the strongest proprietary model reaching 69.3%. The authors interpret this as a genuine capability gap: models can read text that appears in frames, but they struggle to follow formulas, theorem demonstrations, and animated visual arguments, particularly in mathematics. The stakes are practical - if the benchmark measures what it claims, then state-of-the-art multimodal systems cannot yet reliably serve as students of dense educational video, and lecture comprehension needs perception and reasoning methods of its own.

What carries the argument

The instrument is the benchmark itself: 1,065 lecture videos (average length 109 seconds, 90.3% mathematics, 3.6% physics, 6.1% chemistry) from ten YouTube channels, each paired with a ground-truth caption and 15 quiz questions. Two tracks separate perception from reasoning: the Notebook track scores generated captions against the ground truth using VDCscore, a divide-and-conquer metric that decomposes long captions into short question-answer pairs, while the Quiz track presents 15,746 open-ended reasoning questions with answers capped at 15 words. The ground truth is produced by a multi-stage pipeline in which Aria writes a global video caption, GPT-4o captions individual keyframes, and Claude-3.5-sonnet merges these and cross-checks every formula and number against the video's subtitles, so the entire benchmark rests on these AI-generated captions and answers.

What would settle it

Run the Quiz track with subtitles or transcripts only, feeding no frames to a strong text-only model. The paper's own text baseline already reaches 40.54%, so if a transcript-only model approaches Claude-3.5-sonnet's 72.7% quiz score, the benchmark would be measuring language reasoning rather than video understanding; a human audit answering quiz questions from transcripts alone would settle the same question.

Watch

Extended reading notes

Core claim

Evaluating 87 models on Video-MMLU, the paper reports that both proprietary and open-source large multimodal models mostly score between 10% and 50%, with Claude-3.5-sonnet the strongest at 69.3% overall (67.4% on captioning and 72.7% on quiz). Models do better in physics and chemistry, where lectures carry much of their content as on-screen text, and clearly worse in mathematics, where the meaning lives in formulas and changing visualizations; the paper takes this as evidence that current LMMs primarily extract textual content from frames rather than reason over dynamic abstract representations. The study further finds that model size correlates weakly with captioning quality (r = 0.18) but more strongly with quiz performance (r = 0.40), that scaling the language-model backbone helps with diminishing returns, and that video-specialized models do not consistently beat image-based models on this benchmark.

Load-bearing premise

The quiz questions and their gold answers were generated by an AI model from captions and subtitles and never verified by humans against the videos, so the benchmark's claim to measure video perception depends on those questions actually requiring the video rather than just language and general knowledge.

Editorial extensions

If this is right

  • Even the best model scores only 69.3%, so lecture-video understanding remains an open problem for both proprietary and open-source systems.
  • Models succeed where lectures put information in text and fail where meaning is carried by formulas and animated demonstrations, with mathematics the weakest discipline (mean 21.8 versus 27.0 for physics and 27.3 for chemistry).
  • Compressing visual tokens to roughly 16-300 per frame preserves most performance, and some compressed models surpass their full-token base, so efficiency and accuracy are not strictly opposed, while ultra-low token counts (2 per frame) collapse performance.
  • Scaling the language-model backbone improves quiz performance but with sharply diminishing returns, and architecture choice matters as much as scale.
  • Video-specialized models do not clearly beat image-based models here, suggesting OCR-heavy image training transfers better to lecture content than video temporal training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A control experiment follows directly from the paper's own numbers: a text-only model scores 40.54% on the Quiz track, so re-running the quiz with subtitles but no frames would separate the visual-perception gap from the language-reasoning gap.
  • A human audit of a random sample of quiz questions, answered from transcripts alone versus with the video, would test whether the benchmark measures video-specific understanding at all.
  • The mathematics deficit may be driven by dynamic OCR failure - formulas changing frame to frame - rather than missing reasoning; isolating static formulas from animated derivations would locate the true bottleneck.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. Video-MMLU is a new benchmark for multi-discipline lecture video understanding, comprising 1,065 short videos (10-240 seconds) from mathematics, physics, and chemistry, with two tasks: a detailed-captioning 'Notebook' track with 15 surface QA pairs per video and a reasoning 'Quiz' track with 15 open-ended QA pairs per video (15,746 pairs total). The authors describe a multi-stage annotation pipeline that combines Aria and GPT-4o captions with Claude-3.5-sonnet refinement against YouTube subtitles and manual review. They evaluate 87 models, including vision-blind LLM baselines, proprietary LMMs, and open-source image/video LMMs from 0.5B to 40B parameters, and report that proprietary models reach 43-69% overall while open-source models mostly fall in the 10-50% range. They also analyze the effect of model size, LLM backbone, and visual token compression (including AuroraCap ablations). The paper's central claim is that current LMMs perform poorly on lecture understanding, especially on tasks requiring both perception and reasoning.

Significance. If the benchmark is valid, it fills a real gap: existing video benchmarks often focus on short open-world clips or simple factual questions, while lecture videos require dense OCR, formula recognition, and domain reasoning; a large public benchmark with 87 evaluated models and open code/dataset would be a valuable community resource. Strengths include the scale of annotation and evaluation, the inclusion of vision-blind baselines, and the systematic study of visual-token compression. However, the construct validity of the Quiz track is not established: the vision-blind Qwen2.5-72B baseline already scores 40.54% (Table 2), and the QA generation uses subtitles as input, so a substantial fraction of quiz items may be answerable without video. Since the headline claim depends on quiz/overall performance, the current evidence does not support the conclusion that LMMs are poor at video lecture understanding specifically. The paper can become a useful resource after the authors add transcript-only baselines, human validation of question grounding and automated scoring, and a re-analysis of the performance gap.

major comments (4)
  1. [Section 4.2 / Table 2] The paper's central claim, stated in the Abstract and Section 4.2, that existing models 'generally perform poorly, with accuracy ranging from only 10% to 50%' is not supported for the Quiz track: Table 2 shows that the vision-blind Qwen2.5-72B baseline achieves 40.54% on the quiz with no video input. Since the quiz QA pairs are generated from both detailed captions and transcribed subtitles (Section 3.2), and the supplementary prompt examples in S3 include items such as 'What is the derivative of e^{2t}?' that require no visual evidence, a large fraction of quiz questions appears answerable from language knowledge alone. The authors should report a transcript-only baseline (subtitles without frames), report the share of quiz questions whose gold answer can be derived from the transcript or from general knowledge, and provide human validation that each quiz question requires visual information. Without these, the headline conclusion that LMMs are poor at video lecture understanding is overstated.
  2. [Section 3.2 / S3 / Section 4.1] The quiz generation pipeline uses Claude-3.5-sonnet to produce QA pairs from detailed captions plus transcribed subtitles, and the gold answers inherit any errors or hallucinated content from the captions. The paper reports no human validation of the quiz questions or gold answers, nor of the Qwen2.5-72B judge described in Section 4.1. Given that many evaluated models share Qwen2.5 backbones (Tables 2-4), the automated judge may be systematically biased toward models from the same family. The authors should provide a human-LLM agreement study on a stratified sample of quiz and notebook answers, and quantify judge agreement separately by model family and by OCR-heavy versus reasoning-heavy questions.
  3. [Section 4.2, Finding 2] The conclusion that 'lecture understanding in models relies more on textual content in frames than on animations' is based on discipline-level averages of quiz scores. Since the quiz is not established to be visually grounded (see above), the lower mathematics scores could reflect differences in language-only answerability or question difficulty rather than the presence of animations. The notebook track is more directly visual; the paper should either base this claim on notebook-track evidence with per-question visual-grounding labels, or temper the conclusion until such evidence is provided.
  4. [Tables 2-4] The large divergence between Notebook and Quiz scores for several models—for example, Aquila-VL-2B scores 13.78 on Notebook but 34.10 on Quiz, while Mini-InternVL-Chat-4B-V1.5 scores 25.76 on Notebook and 54.20 on Quiz—suggests the two tracks are not measuring a single 'lecture understanding' construct. The paper should discuss whether Quiz scores primarily reflect LLM knowledge and OCR-based text extraction rather than video perception, and should report a per-question or per-video correlation analysis between the two tracks to clarify what each track measures.
minor comments (4)
  1. [Section S5] The supplementary material lists the ten YouTube channels but does not report how many videos came from each channel; a table or figure with channel-level counts would help assess creator bias in the benchmark.
  2. [Section 2.1 / Section S10] There are spelling and formatting inconsistencies, including 'imrpove' in Section 2.1 and the alternating 'AuroaCap'/'AuroraCap' spellings in Section S10; these should be corrected.
  3. [Table 1] Several cells in Table 1 contain '×' placeholders or missing statistics (e.g., for MVBench, LongVideoBench, and others); please clarify whether these entries are not applicable or simply unavailable, and include the missing values where possible.
  4. [Figure 3] The caption reports a Jensen-Shannon distance of 0.668 between the two question-answer distributions; the paper should specify exactly which embedding vectors were used and how the distance was computed, since this number is presented as evidence of task distinctiveness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is empirical, and model-generated annotations or judging are validity risks rather than derivational loops.

full rationale

The paper's load-bearing claims are empirical measurements: captioning (Notebook) and quiz scores of 87 models on a newly constructed benchmark, plus correlation analyses between model size, visual tokens, and performance. No predicted quantity is derived from fitted inputs, no parameter is tuned on a subset and then reported as a prediction on a superset, and no central claim reduces by construction to the annotation pipeline. The quiz ground truths are generated by Claude-3.5-sonnet from captions and subtitles, and correctness is judged by Qwen2.5-72B; this can bias difficulty and favor LLM-like outputs, and the absence of human validation is a real construct-validity and fairness concern, but it is not a circular derivation. The self-citation to AuroraCap [21] for VDCscore is a methodological borrowing of an externally usable metric, and the dense evaluation of AuroraCap in the token-compression analysis is an evaluation of the authors' own method, not a smuggled premise that forces the paper's conclusions. The vision-blind Qwen2.5-72B baseline reaching 40.54% on the Quiz track weakens the strength of the 'video understanding' interpretation, but that is an external-falsifiability argument about what the quiz measures, not evidence that the scores are defined in terms of the inputs. Under the stated hard rules, no circular step can be exhibited, so the honest finding is no significant circularity with score 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three domain assumptions about annotation accuracy, judge fairness, and the visual grounding of the quiz. No new physical or theoretical entities are introduced; the benchmark is a dataset, not an entity postulate. The free parameters are evaluation design choices that influence scores but are not fitted to a target outcome.

free parameters (3)
  • Number of sampled frames for evaluation = 32 for video LMMs; 4 for image-based LMMs; 2 for LLaVA-NeXT-Vicuna and XComposer
    Frame counts are chosen per model capability (Section 4.1), which can affect scores and comparability.
  • Maximum answer length for quiz = 15 words
    Chosen to enable automatic evaluation (Section 3.2); affects question difficulty and answer extractability.
  • Keyframe sampling rate = 1 frame/second to 1 frame/5 seconds, set per creator
    Manually reviewed per creator to set optimal sampling rate (Section 3.1); subjective choice affects annotation coverage.
assumptions (3)
  • domain assumption LLM-generated captions and quiz QA pairs are accurate ground truth
    Captions are produced by Aria, GPT-4o, and Claude-3.5-sonnet and manually reviewed, but no inter-annotator agreement or human accuracy metrics are reported (Section 3.2).
  • domain assumption Qwen2.5-72B provides unbiased evaluation of model outputs
    The judge is a Qwen2.5 model used at temperature 0 with no validation against human judges or alternative LLM judges (Section 4.1, S7, S8).
  • domain assumption Quiz questions require visual perception of the video
    This is contradicted by the vision-blind Qwen2.5-72B baseline scoring 40.54% on the Quiz track (Table 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark." pith.science (2026). https://pith.science/paper/Q3QPYUON

@misc{pith2026250414693,
  author       = {Pith},
  title        = {Pith review of: Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3QPYUON}},
  note         = {Machine review of arXiv:2504.14693}
}
read the original abstract

Recent advancements in language multimodal models (LMMs) for video have demonstrated their potential for understanding video content, yet the task of comprehending multi-discipline lectures remains largely unexplored. We introduce Video-MMLU, a massive benchmark designed to evaluate the capabilities of LMMs in understanding Multi-Discipline Lectures. We evaluate over 90 open-source and proprietary models, ranging from 0.5B to 40B parameters. Our results highlight the limitations of current models in addressing the cognitive challenges presented by these lectures, especially in tasks requiring both perception and reasoning. Additionally, we explore how the number of visual tokens and the large language models influence performance, offering insights into the interplay between multimodal perception and reasoning in lecture comprehension.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Trace-grounded parametric profiling of three synthetic counting tasks shows current video-language models only count reliably at low event counts and low rates, and final-answer accuracy masks poor timestamp-level eve...

  2. MMHBench: A Multi-Perspective Benchmark for Mental Health Understanding in Long-Form Videos

    cs.AI 2026-07 conditional novelty 6.0 of 10

    MMHBench, a 268-video, 2,184-question benchmark, shows multimodal LLMs are much worse at first-person psychological perspective-taking than at third-person observation.

  3. HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.

  4. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  5. Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Models

    cs.CL 2025-05 conditional novelty 2.0 of 10

    A survey-style position paper claims that reinforcement fine-tuning powers reasoning in multimodal LLMs, summarizing over a hundred recent works and proposing five future research directions.

Reference graph

Works this paper leans on

181 extracted references · 6 canonical work pages · cited by 5 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Probing early modification of gravity with planck, act and spt

    Guillermo Franco Abell´an, Matteo Braglia, Mario Ballardini, Fabio Finelli, and Vivian Poulin. Probing early modification of gravity with planck, act and spt. Journal of Cosmology and Astropar- ticle Physics, 2023(12):017, 2023

    2023

  3. [3]

    Mvtamperbench: Evaluating robustness of vision-language models

    Amit Agarwal, Srikant Panda, Angeline Charles, Bhargava Kumar, Hitesh Patel, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, and Dong-Kyu Chae. Mvtamperbench: Evaluating robustness of vision-language models. arXiv preprint arXiv:2412.19794, 2024

    arXiv 2024

  4. [4]

    Fuyu-8b: A unified vision-language model, 2024

    Adept AI. Fuyu-8b: A unified vision-language model, 2024. URL https://www.adept.ai/ blog/fuyu-8b. Accessed: 2025-03-01

    2024

  5. [5]

    Ministral-8b-instruct-2410, 2024

    Mistral AI. Ministral-8b-instruct-2410, 2024. URL https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.2 . Accessed: 2025-03-01

    2024

  6. [6]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md

    2024

  7. [7]

    Smollm2: When smol goes big–data-centric training of a small language model

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart´ın Bl´azquez, Guilherme Penedo, Lewis Tunstall, Andr´es Marafioti, Hynek Kydl´ıˇcek, Agust´ın Piqueres Lajar´ın, Vaibhav Srivastav, et al. Smollm2: When smol goes big–data-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025

    arXiv 2025

  8. [8]

    Claude 3.5 sonnet announcement, 2024

    Anthropic. Claude 3.5 sonnet announcement, 2024. URL https://www.anthropic.com/news/ claude-3-5-sonnet . Accessed: 2025-03-01

    2024

Show all 181 references
  1. [9]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021

    2021

  2. [10]

    Mash- vlm: Mitigating action-scene hallucination in video-llms through disentangled spatial-temporal representations

    Kyungho Bae, Jinhyung Kim, Sihaeng Lee, Soonyoung Lee, Gunhee Lee, and Jinwoo Choi. Mash- vlm: Mitigating action-scene hallucination in video-llms through disentangled spatial-temporal representations. arXiv preprint arXiv:2503.15871, 2025

    2025 arXiv

  3. [11]

    A versatile vision-language model for understanding, localization, text reading, and beyond

    J Bai, S Bai, S Yang, S Wang, S Tan, P Wang, J Lin, C Zhou, and J Qwen-VL Zhou. A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  4. [13]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  5. [14]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  6. [15]

    Deepseek llm: Scaling open-source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024

    2024 arXiv

  7. [16]

    Everything can be described in words: A simple unified multi- modal framework with semantic and temporal alignment

    Xiaowei Bi and Zheyuan Xu. Everything can be described in words: A simple unified multi- modal framework with semantic and temporal alignment. arXiv preprint arXiv:2503.09081 , 2025

    2025 arXiv

  8. [17]

    Perception tokens enhance visual reasoning in multimodal language models

    Mahtab Bigverdi, Zelun Luo, Cheng-Yu Hsieh, Ethan Shen, Dongping Chen, Linda G Shapiro, and Ranjay Krishna. Perception tokens enhance visual reasoning in multimodal language models. arXiv preprint arXiv:2412.03548, 2024

    2024 arXiv

  9. [18]

    Mobile-mmlu: A mobile in- telligence language understanding benchmark

    Sondos Mahmoud Bsharat, Mukul Ranjan, Aidar Myrzakhan, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, and Zhiqiang Shen. Mobile-mmlu: A mobile in- telligence language understanding benchmark. arXiv preprint arXiv:2503.20786, 2025

    2025 arXiv

  10. [19]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024

    2024 arXiv

  11. [20]

    Video simpleqa: Towards factuality evaluation in large video language models

    Meng Cao, Pengfei Hu, Yingyao Wang, Jihao Gu, Haoran Tang, Haoze Zhao, Jiahua Dong, Wangbo Yu, Ge Zhang, Ian Reid, et al. Video simpleqa: Towards factuality evaluation in large video language models. arXiv preprint arXiv:2503.18923, 2025

    2025 arXiv

  12. [21]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024

    arXiv

  13. [22]

    Fastv: From-point visibility culling on complex models

    Anish Chandak, Lakulish Antani, Micah Taylor, and Dinesh Manocha. Fastv: From-point visibility culling on complex models. In Computer Graphics Forum, volume 28, pages 1237–1246. Wiley Online Library, 2009

    2009

  14. [23]

    Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105.14517, 2021

    2021 arXiv

  15. [24]

    Livecc: Learning video llm with streaming speech transcription at scale

    Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livecc: Learning video llm with streaming speech transcription at scale. arXiv preprint arXiv:2504.16030, 2025

    2025 arXiv

  16. [25]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023

    2023 arXiv

  17. [26]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024. 14 Video-MMLU: A Massive Multi-Discipline Lec...

    2024 arXiv

  18. [27]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024

    2024 arXiv

  19. [28]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems , 37:19472– 19495, 2025

    2025

  20. [29]

    Longvila: Scaling long-context visual language models for long videos

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024

    2024 arXiv

  21. [30]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 , 2024

    2024 arXiv

  22. [31]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  23. [32]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  24. [33]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  25. [34]

    Focuschat: Text-guided long video under- standing via spatiotemporal information filtering

    Zheng Cheng, Rendong Wang, and Zhicheng Wang. Focuschat: Text-guided long video under- standing via spatiotemporal information filtering. arXiv preprint arXiv:2412.12833, 2024

    2024 arXiv

  26. [35]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023...

    2023

  27. [36]

    Vidhal: Benchmarking temporal hallucinations in vision llms

    Wey Yeh Choong, Yangyang Guo, and Mohan Kankanhalli. Vidhal: Benchmarking temporal hallucinations in vision llms. arXiv preprint arXiv:2411.16771, 2024

    2024 arXiv

  28. [37]

    Fine-grained video captioning through scene graph consolidation

    Sanghyeok Chu, Seonguk Seo, and Bohyung Han. Fine-grained video captioning through scene graph consolidation. arXiv preprint arXiv:2502.16427, 2025

    2025 arXiv

  29. [38]

    Lmdeploy: A toolkit for compressing, deploying, and serving llm

    LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy, 2023

    2023

  30. [39]

    Tvbench: Redesigning video-language evaluation

    Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. Tvbench: Redesigning video-language evaluation. arXiv preprint arXiv:2410.07752, 2024. 15 Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    2024 arXiv

  31. [40]

    Xinyuan-vl-2b, 2024

    Cylingo. Xinyuan-vl-2b, 2024. URL https://huggingface.co/Cylingo/Xinyuan-VL-2B. Ac- cessed: 2025-03-01

    2024

  32. [41]

    Instructblip: Towards general-purpose vision- language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems , 36, 2024

    2024

  33. [42]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  34. [43]

    Scalable vision language model training via high quality data curation

    Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scalable vision language model training via high quality data curation. arXiv preprint arXiv:2501.05952, 2025

    2025 arXiv

  35. [44]

    Internlm-xcomposer2: Mastering free- form text-image composition and comprehension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free- form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16...

    2024 arXiv

  36. [45]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia...

    2024

  37. [46]

    Feather the throttle: Revisiting visual token pruning for vision-language model acceleration

    Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. arXiv preprint arXiv:2412.13180, 2024

    2024 arXiv

  38. [47]

    Smolvlm: A 1b vision-language model with moe, 2024

    Hugging Face. Smolvlm: A 1b vision-language model with moe, 2024. URL https: //huggingface.co/blog/smolvlm. Accessed: 2025-03-01

    2024

  39. [48]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. Advances in Neural Information Processing Systems , 37:89098–89124, 2024

    2024

  40. [49]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  41. [50]

    Vita: Towards open-source interactive omni multimodal llm

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211, 2024

    2024 arXiv

  42. [51]

    Vita-1.5: Towards gpt-4o level real-time vision and speech interaction

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957, 2025. 16 Video-MMLU: A Massive Multi-Discipline...

    2025 arXiv

  43. [52]

    Framefusion: Combining similarity and importance for video token reduction on large visual language models, 2024

    Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. Framefusion: Combining similarity and importance for video token reduction on large visual language models, 2024. URL https://arxiv.org/abs/2501.01986

    2024 arXiv

  44. [53]

    Exploring hallucination of large multimodal models in video understanding: Benchmark, analysis and mitigation

    Hongcheng Gao, Jiashu Qu, Jingyi Tang, Baolong Bi, Yue Liu, Hongyu Chen, Li Liang, Li Su, and Qingming Huang. Exploring hallucination of large multimodal models in video understanding: Benchmark, analysis and mitigation. arXiv preprint arXiv:2503.19622, 2025

    2025 arXiv

  45. [54]

    Long- vale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Long- vale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos. arXiv preprint arXiv:2411.19772, 2024

    2024 arXiv

  46. [55]

    Google gemini: Next-generation model (february 2024), 2024

    Google. Google gemini: Next-generation model (february 2024), 2024. URL https://blog. google/technology/ai/google-gemini-next-generation-model-february-2024/ . Ac- cessed: 2025-03-01

    2024

  47. [56]

    Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410.18558, 2024

    arXiv

  48. [57]

    Trace: Temporal grounding video llm via causal event modeling

    Yongxin Guo, Jingyu Liu, Mingda Li, Xiaoying Tang, Qingbin Liu, and Xi Chen. Trace: Temporal grounding video llm via causal event modeling. arXiv preprint arXiv:2410.05643, 2024

    2024 arXiv

  49. [58]

    Adafv: Accel- erating vlms with self-adaptive cross-modality attention mixture.arXiv preprint arXiv:2501.09532, 2025

    Jiayi Han, Liang Du, Yiwen Wu, Xiangguo Zhou, Hongwei Du, and Weibo Zheng. Adafv: Accel- erating vlms with self-adaptive cross-modality attention mixture.arXiv preprint arXiv:2501.09532, 2025

    2025 arXiv

  50. [59]

    Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection

    Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. arXiv preprint arXiv:2411.14794, 2024

    2024 arXiv

  51. [60]

    Rethinking token reduction in mllms: Towards a unified paradigm for training-free acceleration

    Yuhang Han, Xuyang Liu, Pengxiang Ding, Donglin Wang, Honggang Chen, Qingsen Yan, and Siteng Huang. Rethinking token reduction in mllms: Towards a unified paradigm for training-free acceleration. arXiv preprint arXiv:2411.17686, 2024

    2024

  52. [61]

    Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos

    Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos. arXiv preprint arXiv:2406.08407, 2024

    2024 arXiv

  53. [62]

    Story- teller: Improving long video description through global audio-visual character identification

    Yichen He, Yuan Lin, Jianchao Wu, Hanchong Zhang, Yuchen Zhang, and Ruicheng Le. Story- teller: Improving long video description through global audio-visual character identification. arXiv preprint arXiv:2411.07076, 2024

    2024 arXiv

  54. [63]

    Worldsense: Evaluating real-world omnimodal understanding for multimodal llms

    Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326 , 2025

    2025 arXiv

  55. [64]

    Cogvlm2: Visual language models for image and video understanding, 2024

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding, 2024. 17 Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    2024

  56. [65]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025

    2025 arXiv

  57. [66]

    Can lvlms describe videos like humans? a five-in-one video annotations benchmark for better human-machine comparison

    Shiyu Hu, Xuchen Li, Xuzhao Li, Jing Zhang, Yipei Wang, Xin Zhao, and Kang Hao Cheong. Can lvlms describe videos like humans? a five-in-one video annotations benchmark for better human-machine comparison. arXiv preprint arXiv:2410.15270, 2024

    arXiv

  58. [67]

    Efficient multi-modal large language models via visual token grouping

    Minbin Huang, Runhui Huang, Han Shi, Yimeng Chen, Chuanyang Zheng, Xiangguo Sun, Xin Jiang, Zhenguo Li, and Hong Cheng. Efficient multi-modal large language models via visual token grouping. arXiv preprint arXiv:2411.17773, 2024

    2024 arXiv

  59. [68]

    Prunevid: Visual token pruning for efficient video large language models

    Xiaohu Huang, Hao Zhou, and Kai Han. Prunevid: Visual token pruning for efficient video large language models. arXiv preprint arXiv:2412.16117, 2024

    2024 arXiv

  60. [69]

    Similarity-aware token pruning: Your vlm but faster

    Ahmadreza Jeddi, Negin Baghbanzadeh, Elham Dolatabadi, and Babak Taati. Similarity-aware token pruning: Your vlm but faster. arXiv preprint arXiv:2503.11549, 2025

    2025 arXiv

  61. [70]

    Ku, Qian Liu, and Wenhu Chen

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv2405.01483, 2024

    2024 arXiv

  62. [71]

    Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency.arXiv preprint arXiv:2502.09621, 2025

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency.arXiv preprint arXiv:2502.09621, 2025

    2025 arXiv

  63. [72]

    Token-efficient long video understanding for multimodal llms

    Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, et al. Token-efficient long video understanding for multimodal llms. arXiv preprint arXiv:2503.04130, 2025

    2025

  64. [73]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13700–...

    2024

  65. [74]

    Miradata: A large-scale video dataset with long durations and structured captions

    Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. arXiv preprint arXiv:2407.06358, 2024

    2024 arXiv

  66. [75]

    On the consistency of video large language models in temporal comprehension

    Minjoon Jung, Junbin Xiao, Byoung-Tak Zhang, and Angela Yao. On the consistency of video large language models in temporal comprehension. arXiv preprint arXiv:2411.12951, 2024

    2024 arXiv

  67. [76]

    Hicm 2: Hier- archical compact memory modeling for dense video captioning

    Minkuk Kim, Hyeon Bae Kim, Jinyoung Moon, Jinwoo Choi, and Seong Tae Kim. Hicm 2: Hier- archical compact memory modeling for dense video captioning. arXiv preprint arXiv:2412.14585, 2024

    2024 arXiv

  68. [77]

    Multimodal lecture presentations dataset: Understanding multimodality in educational slides

    Dong Won Lee, Chaitanya Ahuja, Paul Pu Liang, Sanika Natu, and Louis-Philippe Morency. Multimodal lecture presentations dataset: Understanding multimodality in educational slides. arXiv preprint arXiv:2208.08080, 2022. 18 Video-MMLU: A Massive Multi-Discipline Lecture Understa...

    2022 arXiv

  69. [78]

    LLaVA-OneVision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  70. [79]

    Aria: An open multimodal native mixture-of-experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024

    2024 arXiv

  71. [80]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  72. [81]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22195–2...

    2024

  73. [82]

    Videochat-flash: Hierarchical compression for long-context video modeling

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574, 2024

    2024 arXiv

  74. [84]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023

    2023 arXiv

  75. [85]

    Improving llm video understanding with 16 frames per second

    Yixuan Li, Changli Tang, Jimin Zhuang, Yudong Yang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. Improving llm video understanding with 16 frames per second. arXiv preprint arXiv:2503.13956, 2025

    2025 arXiv

  76. [86]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  77. [87]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533, 2023

    2023 arXiv

  78. [88]

    Streamingbench: Assessing the gap for mllms to achieve streaming video understanding

    Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. arXiv preprint arXiv:2411.03628, 2024

    2024 arXiv

  79. [89]

    Vlog: Video-language models by generative retrieval of narration vocabulary, 2025

    Kevin Qinghong Lin and Mike Zheng Shou. Vlog: Video-language models by generative retrieval of narration vocabulary, 2025. URL https://arxiv.org/abs/2503.09402

    2025 arXiv

  80. [90]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 19 Video-MMLU: A Massive Multi-Discipline Lecture Understand...

    2024

  81. [91]

    Streamchat: Chatting with streaming video

    Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, and Jose M Alvare. Streamchat: Chatting with streaming video. arXiv preprint arXiv:2412.08646 , 2024

    2024 arXiv

  82. [92]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision , pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision , pages 216–233. Springer, 2024

    2024

  83. [93]

    Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  84. [94]

    Keyframe-oriented vision token pruning: Enhancing efficiency of large vision language models on long-form video processing

    Yudong Liu, Jingwei Sun, Yueqian Lin, Jingyang Zhang, Ming Yin, Qinsi Wang, Jianyi Zhang, Hai Li, and Yiran Chen. Keyframe-oriented vision token pruning: Enhancing efficiency of large vision language models on long-form video processing. arXiv preprint arXiv:2503.10742, 2025

    2025 arXiv

  85. [95]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3202–3211, 2022

    2022

  86. [96]

    Deepseek-vl: towards real-world vision-language under- standing

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language under- standing. arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  87. [97]

    Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165, 2021

    2021 arXiv

  88. [98]

    Llava-mr: Large language-and-vision assistant for video moment retrieval

    Weiheng Lu, Jian Li, An Yu, Ming-Ching Chang, Shengpeng Ji, and Min Xia. Llava-mr: Large language-and-vision assistant for video moment retrieval. arXiv preprint arXiv:2411.14505, 2024

    2024 arXiv

  89. [99]

    B-vllm: A vision large language model with balanced spatio-temporal tokens

    Zhuqiang Lu, Zhenfei Yin, Mengwei He, Zhihui Wang, Zicheng Liu, Zhiyong Wang, and Kun Hu. B-vllm: A vision large language model with balanced spatio-temporal tokens. arXiv preprint arXiv:2412.09919, 2024

    2024 arXiv

  90. [100]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093, 2024

    2024

  91. [101]

    Quota: Query-oriented token assignment via cot query decouple for long video comprehension

    Yongdong Luo, Wang Chen, Xiawu Zheng, Weizhong Huang, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Jiebo Luo, et al. Quota: Query-oriented token assignment via cot query decouple for long video comprehension. arXiv preprint arXiv:2503.08689, 2025

    2025 arXiv

  92. [102]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  93. [103]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 20 Video-MMLU: A Massive Multi-Discipline Lecture Understan...

    2023

  94. [104]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36, 2024

    2024

  95. [105]

    Nous-hermes-2-yi-34b, 2023

    Nous Research. Nous-hermes-2-yi-34b, 2023. URL https://huggingface.co/NousResearch/ Nous-Hermes-2-Yi-34B . Accessed: 2024-08-29

    2023

  96. [106]

    Hello gpt-4o: Openai’s newest multimodal model, 2024

    OpenAI. Hello gpt-4o: Openai’s newest multimodal model, 2024. URL https://openai.com/ index/hello-gpt-4o/. Accessed: 2025-03-01

    2024

  97. [107]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. T ransactions on Machine Learning Research, 2023

    2023

  98. [108]

    Beyond semantics: Rediscovering spatial awareness in vision-language models

    Jianing Qi, Jiawei Liu, Hao Tang, and Zhigang Zhu. Beyond semantics: Rediscovering spatial awareness in vision-language models. arXiv preprint arXiv:2503.17349, 2025

    2025

  99. [109]

    Step: Enhancing video-llms’ compositional reasoning by spatio-temporal graph-guided self-training

    Haiyi Qiu, Minghe Gao, Long Qian, Kaihang Pan, Qifan Yu, Juncheng Li, Wenjie Wang, Siliang Tang, Yueting Zhuang, and Tat-Seng Chua. Step: Enhancing video-llms’ compositional reasoning by spatio-temporal graph-guided self-training. arXiv preprint arXiv:2412.00161, 2024

    2024 arXiv

  100. [110]

    Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models

    Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Ar- man Cohan. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. arXiv preprint arXiv:2410.23266, 2024

    arXiv

  101. [111]

    Growing a twig to accelerate large vision-language models

    Zhenwei Shao, Mingyang Wang, Zhou Yu, Wenwen Pan, Yan Yang, Tao Wei, Hongyuan Zhang, Ning Mao, Wei Chen, and Jun Yu. Growing a twig to accelerate large vision-language models. arXiv preprint arXiv:2503.14075, 2025

    2025 arXiv

  102. [112]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. arXiv preprint arXiv:2307.16449, 2023

    2023 arXiv

  103. [113]

    Moviechat+: Question-aware sparse memory for long video question answering

    Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering. arXiv preprint arXiv:2404.17176, 2024

    2024 arXiv

  104. [114]

    jina- embeddings-v3: Multilingual embeddings with task lora

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael G ¨unther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. jina- embeddings-v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409.10173 , 2024

    2024 arXiv

  105. [115]

    Dycoke: Dynamic compression of tokens for fast video large language models

    Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. arXiv preprint arXiv:2411.15024, 2024

    2024 arXiv

  106. [116]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/. 21 Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    2025

  107. [117]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024

    2024 arXiv

  108. [118]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  109. [119]

    Fastvlm: Efficient vision encoding for vision language models

    Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokul Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. Fastvlm: Efficient vision encoding for vision language models. arXiv preprint arXiv:2412.13303, 2024

    2024 arXiv

  110. [120]

    Folder: Accelerating multi-modal large language models with enhanced performance

    Haicheng Wang, Zhemeng Yu, Gabriele Spadaro, Chen Ju, Victor Qu´etu, and Enzo Tartaglione. Folder: Accelerating multi-modal large language models with enhanced performance. arXiv preprint arXiv:2501.02430, 2025

    2025 arXiv

  111. [121]

    Llava-zip: Adaptive visual token compression with intrinsic image information

    Ke Wang and Hong Xuan. Llava-zip: Adaptive visual token compression with intrinsic image information. arXiv preprint arXiv:2412.08771, 2024

    2024 arXiv

  112. [122]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  113. [123]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  114. [124]

    Retake: Reducing tem- poral and knowledge redundancy for long video understanding

    Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Retake: Reducing tem- poral and knowledge redundancy for long video understanding. arXiv preprint arXiv:2412.20504, 2024

    2024 arXiv

  115. [125]

    Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding

    Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding. arXiv preprint arXiv:2503.12559, 2025

    2025 arXiv

  116. [126]

    Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture. arXiv preprint arXiv:2409.02889, 2024

    2024

  117. [127]

    Vatex: A large-scale, high-quality multilingual dataset for video-and-language research

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4581–4591, 2019

    2019

  118. [128]

    Timerefine: Temporal grounding with time refining video llm

    Xizi Wang, Feng Cheng, Ziyang Wang, Huiyu Wang, Md Mohaiminul Islam, Lorenzo Torresani, Mohit Bansal, Gedas Bertasius, and David Crandall. Timerefine: Temporal grounding with time refining video llm. arXiv preprint arXiv:2412.09601, 2024

    2024 arXiv

  119. [129]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386, 2025. 22 Video-MMLU: A Massive Multi-D...

    2025 arXiv

  120. [130]

    Time: Temporal-sensitive multi-dimensional instruction tuning and bench- marking for video-llms

    Yunxiao Wang, Meng Liu, Rui Shao, Haoyu Zhang, Bin Wen, Fan Yang, Tingting Gao, Di Zhang, and Liqiang Nie. Time: Temporal-sensitive multi-dimensional instruction tuning and bench- marking for video-llms. arXiv preprint arXiv:2503.09994, 2025

    2025 arXiv

  121. [131]

    Longcaption- ing: Unlocking the power of long caption generation in large multimodal models

    Hongchen Wei, Zhihong Tan, Yaosi Hu, Changwen Chen, and Zhenzhong Chen. Longcaption- ing: Unlocking the power of long caption generation in large multimodal models. arXiv preprint arXiv:2502.15393, 2025

    2025 arXiv

  122. [132]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024

    2024 arXiv

  123. [133]

    Longvideobench: A benchmark for long- context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding. arXiv preprint arXiv:2407.15754, 2024

    2024 arXiv

  124. [134]

    Accelerating multimodal large language models via dynamic visual-token exit and the empirical findings

    Qiong Wu, Wenhao Lin, Weihao Ye, Yiyi Zhou, Xiaoshuai Sun, and Rongrong Ji. Accelerating multimodal large language models via dynamic visual-token exit and the empirical findings. arXiv preprint arXiv:2411.19628, 2024

    2024 arXiv

  125. [135]

    Longvitu: Instruction tuning for long-form video understanding

    Rujie Wu, Xiaojian Ma, Hai Ci, Yue Fan, Yuxuan Wang, Haozhe Zhao, Qing Li, and Yizhou Wang. Longvitu: Instruction tuning for long-form video understanding. arXiv preprint arXiv:2501.05037, 2025

    2025 arXiv

  126. [136]

    Valley2: Exploring multimodal models with scalable vision-language design

    Ziheng Wu, Zhenghao Chen, Ruipu Luo, Can Zhang, Yuan Gao, Zhentao He, Xian Wang, Haoran Lin, and Minghui Qiu. Valley2: Exploring multimodal models with scalable vision-language design. arXiv preprint arXiv:2501.05901, 2025

    2025 arXiv

  127. [137]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9777–9786, 2021

    2021

  128. [138]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia , pages 1645–1653, 2017

    2017

  129. [139]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5288–5296, 2016

    2016

  130. [140]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024

    2024 arXiv

  131. [141]

    Learning to inference adaptively for multimodal large language models

    Zhuoyan Xu, Khoi Duc Nguyen, Preeti Mukherjee, Saurabh Bagchi, Somali Chaterji, Yingyu Liang, and Yin Li. Learning to inference adaptively for multimodal large language models. arXiv preprint arXiv:2503.10905, 2025

    2025 arXiv

  132. [142]

    Iqvic: In-context, question adaptive vision compressor for long-term video understanding lmms

    Sosuke Yamao, Natsuki Miyahara, Yuki Harazono, and Shun Takeuchi. Iqvic: In-context, question adaptive vision compressor for long-term video understanding lmms. arXiv preprint arXiv:2412.09907, 2024. 23 Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    2024 arXiv

  133. [143]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  134. [144]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  135. [145]

    Pvc: Progressive visual token compression for unified image and video processing in large vision-language models

    Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, , and Jifeng Dai. Pvc: Progressive visual token compression for unified image and video processing in large vision-language models. arXiv preprint arXiv:2412.09613, 2024

    2024 arXiv

  136. [146]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467, 2024

    2024

  137. [147]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  138. [148]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. In The Thirteenth International Conference on Learning Representations , 2024

    2024

  139. [149]

    Llavaction: evaluat- ing and training multi-modal large language models for action recognition

    Shaokai Ye, Haozhe Qi, Alexander Mathis, and Mackenzie W Mathis. Llavaction: evaluat- ing and training multi-modal large language models for action recognition. arXiv preprint arXiv:2503.18712, 2025

    2025

  140. [150]

    Atp-llava: Adaptive token pruning for large vision language models

    Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adaptive token pruning for large vision language models. arXiv preprint arXiv:2412.00447, 2024

    2024 arXiv

  141. [151]

    T2vid: Translating long text into multi-image is the catalyst for video-llms

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Yunhang Shen, Chunjiang Ge, Yan Yang, Zuwei Long, Yuhan Dai, Tong Xu, Xing Sun, et al. T2vid: Translating long text into multi-image is the catalyst for video-llms. arXiv preprint arXiv:2411.19951, 2024

    2024 arXiv

  142. [152]

    Eve: Towards end-to-end video subtitle extraction with vision-language models

    Haiyang Yu, Jinghui Lu, Yanjie Wang, Yang Li, Han Wang, Can Huang, and Bin Li. Eve: Towards end-to-end video subtitle extraction with vision-language models. arXiv preprint arXiv:2503.04058, 2025

    2025

  143. [153]

    Espresso: High compression for rich extraction from videos for your vision-language model

    Keunwoo Peter Yu, Achal Dave, Rares Ambrus, and Jean Mercat. Espresso: High compression for rich extraction from videos for your vision-language model. arXiv preprint arXiv:2412.04729, 2024

    2024 arXiv

  144. [154]

    Shortv: Efficient multimodal large language models by freezing visual tokens in ineffective layers

    Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, and Le Sun. Shortv: Efficient multimodal large language models by freezing visual tokens in ineffective layers. arXiv preprint arXiv:2504.00502, 2025. 24 Video-MMLU: A Massive...

    2025

  145. [155]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023

    2023 arXiv

  146. [156]

    Movie101: A new movie understanding benchmark

    Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, and Qin Jin. Movie101: A new movie understanding benchmark. arXiv preprint arXiv:2305.12140, 2023

    2023 arXiv

  147. [157]

    Skip-vision: Efficient and scalable acceleration of vision-language models via adaptive token skipping

    Weili Zeng, Ziyuan Huang, Kaixiang Ji, and Yichao Yan. Skip-vision: Efficient and scalable acceleration of vision-language models via adaptive token skipping. arXiv e-prints , pages arXiv–2503, 2025

    2025

  148. [158]

    Sigmoid loss for language image pre-training, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023

    2023

  149. [159]

    Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 20...

    2025 arXiv

  150. [160]

    Memory helps, but confabulation misleads: Understanding streaming events in videos with mllms

    Gengyuan Zhang, Mingcong Ding, Tong Liu, Yao Zhang, and Volker Tresp. Memory helps, but confabulation misleads: Understanding streaming events in videos with mllms. arXiv preprint arXiv:2502.15457, 2025

    2025 arXiv

  151. [161]

    Token dynamics: Towards efficient and dynamic video token representation for video large language models

    Haichao Zhang, Zhuowei Li, Dimitris Metaxas, and Yun Fu. Token dynamics: Towards efficient and dynamic video token representation for video large language models. arXiv preprint arXiv:2503.16980, 2025

    2025

  152. [162]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  153. [163]

    Vinoground: Scrutinizing lmms over dense temporal reasoning with short videos

    Jianrui Zhang, Mu Cai, and Yong Jae Lee. Vinoground: Scrutinizing lmms over dense temporal reasoning with short videos. arXiv preprint arXiv:2410.02763, 2024

    2024 arXiv

  154. [164]

    p-mod: Building mixture-of-depths mllms via progressive ratio decay

    Jun Zhang, Desen Meng, Ji Qi, Zhenpeng Huang, Tao Wu, and Limin Wang. p-mod: Building mixture-of-depths mllms via progressive ratio decay. arXiv preprint arXiv:2412.04449, 2024

    2024 arXiv

  155. [165]

    Lmms-eval: Reality check on the evaluation of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772, 2024

    2024 arXiv

  156. [166]

    Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition

    Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112, 2023

    2023 arXiv

  157. [167]

    [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster

    Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, MinQi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818, 2024

    2024 arXiv

  158. [169]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. URL https://arxiv.org/abs/2410.02713

    2024 arXiv

  159. [170]

    Worldgui: Dynamic testing for comprehensive desktop gui automation

    Henry Hengyuan Zhao, Difei Gao, and Mike Zheng Shou. Worldgui: Dynamic testing for comprehensive desktop gui automation. arXiv preprint arXiv:2502.08047, 2025

    2025 arXiv

  160. [171]

    Accelerating multimodel large language models by searching optimal vision token reduction

    Shiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dimitris N Metaxas, and Licheng Yu. Accelerating multimodel large language models by searching optimal vision token reduction. arXiv preprint arXiv:2412.00556, 2024

    2024 arXiv

  161. [172]

    Omchat: A recipe to train multimodal language models with strong long context and video understanding

    Tiancheng Zhao, Qianqian Zhang, Kyusong Lee, Peng Liu, Lu Zhang, Chunxin Fang, Jiajia Liao, Kelei Jiang, Yibo Ma, and Ruochen Xu. Omchat: A recipe to train multimodal language models with strong long context and video understanding. arXiv preprint arXiv:2407.04923, 2024

    2024 arXiv

  162. [173]

    A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vlms

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, and Yang You. A stitch in time saves nine: Small vlm is a precise guidance for accelerating large vlms. arXiv preprint arXiv:2412.03324, 2024

    2024 arXiv

  163. [174]

    Aim: Adaptive inference of multi-modal llms via token merging and pruning

    Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. Aim: Adaptive inference of multi-modal llms via token merging and pruning. arXiv preprint arXiv:2412.03248, 2024

    2024 arXiv

  164. [175]

    Lyra: An efficient and speech-centric framework for omni-cognition

    Zhisheng Zhong, Chengyao Wang, Yuqi Liu, Senqiao Yang, Longxiang Tang, Yuechen Zhang, Jingyao Li, Tianyuan Qu, Yanwei Li, Yukang Chen, et al. Lyra: An efficient and speech-centric framework for omni-cognition. arXiv preprint arXiv:2412.09501, 2024

    2024 arXiv

  165. [176]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  166. [177]

    Humanvbench: Exploring human-centric video understanding capabilities of mllms with synthetic benchmark data, 2025

    Ting Zhou, Daoyuan Chen, Qirui Jiao, Bolin Ding, Yaliang Li, and Ying Shen. Humanvbench: Exploring human-centric video understanding capabilities of mllms with synthetic benchmark data, 2025. URL https://arxiv.org/abs/2412.17574

    2025 arXiv

  167. [178]

    St 3: Accelerat- ing multimodal large language model by spatial-temporal visual token trimming

    Jiedong Zhuang, Lu Lu, Ming Dai, Rui Hu, Jian Chen, Qiang Liu, and Haoji Hu. St 3: Accelerat- ing multimodal large language model by spatial-temporal visual token trimming. arXiv preprint arXiv:2412.20105, 2024

    2024 arXiv

  168. [179]

    Apollo: An exploration of video understanding in large multimodal models

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. arXiv preprint arXiv:2412.10360, 2024

    2024 arXiv

  169. [180]

    Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836, 2024. 26 Video-MMLU: A Massive Multi-Discipline Lecture Un...

    2024 arXiv

  170. [181]

    - Any difference in characters (including case, punctuation, or letter substitution) must result in ’no’

    For OCR-related questions: - Perform a strict letter-by-letter comparison. - Any difference in characters (including case, punctuation, or letter substitution) must result in ’no’. - Minor spelling errors or missing characters should not be accepted

  171. [182]

    - Synonyms or paraphrases can be considered valid matches

    For non-OCR-related questions: - Focus on the meaningful match between the predicted answer and the correct answer. - Synonyms or paraphrases can be considered valid matches. - Minor spelling differences or alternative expressions should not be penalized. User Please evaluate ...

  172. [183]

    MVPs” and “The Arithmetic Mean-Root Mean Square Inequality

    Evaluate Reasoning Tasks Strictly: - The predicted answer must capture all critical concepts and details mentioned in the correct answer. - If the correct answer mentions specific concepts or examples (e.g., ’odd numbers accumulate to form perfect squares’), the predicted answ...

  173. [184]

    Through the application of the Pythagorean theorem to this connecting triangle, the hypotenuse measures√ a2/2 + b2/2

    These measurements arise from the fact that these segments are hypotenuses of the isosceles right triangles formed within each square. Through the application of the Pythagorean theorem to this connecting triangle, the hypotenuse measures√ a2/2 + b2/2. The construction creates...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.