Pith. sign in

REVIEW 4 major objections 6 minor 63 references

VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VF-Eval benchmarks four feedback skills on AI-generated video and finds even the best model, GPT-4.1, scores 51.6% overall versus an 84.4% human baseline.

desk verdict Useful AIGC-video feedback benchmark, but the headline numbers are partly artifacts of an all-positive yes/no split and a text-only judge. read the letter →

arxiv 2505.23693 v1 pith:FR4OSDT5 submitted 2025-05-29 cs.CV cs.AIcs.CL

Tingyu Song , Tongyan Hu , Guo Gan , Yilun Zhao This is my paper
classification cs.CVcs.AIcs.CL
keywords AIGCvideomultimodallargelanguagemodelunderstandingbenchmarkfeedbackgenerationcoherencevalidationerrordetectionreasoningevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VF-Eval is a benchmark that tests whether multimodal large language models can give useful feedback on AI-generated (AIGC) videos, through four tasks: checking whether a video matches its generation prompt, saying whether an error is present, naming the error type, and answering reasoning questions about the video. The paper evaluates 13 models and reports that the best one, GPT-4.1, reaches 51.6% overall accuracy against an 84.4% human baseline. The authors argue that current models are therefore unreliable as automated critics for AI video generation, and that a further experiment, RePrompt, shows aligning model feedback with human preferences can improve regenerated videos. A sympathetic reader would take the central claim to be that the ability to critique AIGC videos is a distinct, underdeveloped capability worth benchmarking.

What carries the argument

The load-bearing object is the VF-Eval dataset itself: 9,740 question-answer pairs built from AIGC videos generated by proprietary models (Pika, Kling, Pixeldance, Gen-3) and open-source models (T2V-Turbo-V2, plus videos from LaVie and OpenSora). The four tasks are the mechanism that operationalizes 'feedback': Yes/No questions for error awareness, multiple-choice for error type detection, and open-ended questions for coherence validation and reasoning evaluation, the latter judged by GPT-4.1-mini against human-written reference answers. The RePrompt experiment, where humans revise the original generation prompt and the revised prompt is used to regenerate the video, is the mechanism connecting benchmark performance to the practical goal of improving AIGC video generation.

What would settle it

Re-score a random sample of the open-ended responses (coherence validation and reasoning evaluation) with independent human raters instead of GPT-4.1-mini, and compare the resulting model-human gap to the reported 51.6% versus 84.4%; if the gap narrows substantially, the headline result is an artifact of the automated judge rather than a true capability gap.

Watch

Extended reading notes

Core claim

On its own terms, VF-Eval claims that frontier multimodal LLMs cannot yet reliably interpret AI-generated videos well enough to serve as feedback providers. The benchmark's four tasks—coherence validation, error awareness, error type detection, and reasoning evaluation—each measure a component of that ability, and the reported results show a wide gap: the strongest model, GPT-4.1, scores 51.6% overall, while humans score 84.4%. The paper also reports that open-source models are competitive with proprietary ones, that models often fail by relying on textual cues or commonsense rather than actual video content, and that a prompt-revision experiment (RePrompt) yields modest win rates for human-revised prompts over original ones, suggesting that better alignment with human feedback could make MLLM feedback useful for video generation.

Load-bearing premise

The open-ended scores assume that GPT-4.1-mini, the LLM used to grade answers against human references, produces scores that faithfully reflect answer quality; if that judge is biased, noisy, or lenient, the reported gap between models and humans would be distorted.

Editorial extensions

If this is right

  • Directly using MLLM feedback in AIGC video quality assessment pipelines is likely to produce inaccurate results until models improve.
  • The benchmark provides a training signal: open-source models are close to proprietary ones, so fine-tuning on VF-Eval has room to close part of the gap.
  • Combining MLLMs with computer-vision auxiliary methods should improve feedback precision, as the paper suggests.
  • RePrompt indicates that human-aligned prompt revision can improve subject consistency and aesthetic quality in regenerated videos, though gains in image and background quality are small.
  • The four-task structure gives a reusable protocol for measuring whether a model can critique AI-generated video, not just answer questions about natural video.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that, because every Error Awareness question has 'Yes' as the correct answer, a trivial always-yes baseline would score 100% on that task; the reported sub-random performance means the task primarily measures a response bias toward declaring videos error-free, not necessarily the absence of error-detection skill.
  • We infer that if the GPT-4.1-mini judge is found unreliable, the open-ended scores should be re-estimated with human adjudication or a panel of judges, and the benchmark's conclusions about coherence validation and reasoning would then need revision.
  • We infer a natural extension, not explored in the paper: testing image-to-video and audio-video AIGC, where error types such as audio-visual mismatch would likely create new failure modes.
  • We infer from the RePrompt win rates that a model scoring well on VF-Eval could act as an automated critic inside a generation pipeline, reducing human involvement in iterative prompt refinement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces VF-Eval, a benchmark for evaluating multimodal large language models (MLLMs) on their ability to generate feedback on AI-generated videos (AIGC). The benchmark comprises four tasks: coherence validation (CV), error awareness (EA), error type detection (ED), and reasoning evaluation (RE), built over 9,740 question-answer pairs on videos from several text-to-video models. The authors evaluate 13 frontier MLLMs and report that even the best model, GPT-4.1, reaches only 51.6% overall accuracy against an 84.4% human baseline. They also present a RePrompt experiment intended to show that aligning MLLMs with human preferences can benefit video generation. The central claims are that VF-Eval is a valid benchmark for a previously underexplored capability and that current MLLMs are unreliable as feedback providers for AIGC videos.

Significance. If the benchmark is valid, it fills a genuine gap: existing video-understanding benchmarks focus on natural videos, while AIGC videos have distinct artifacts and are increasingly important. The dataset construction includes human annotation, expert validation, and public data/code releases, which are concrete strengths. The four-task design with fine-grained reasoning subcategories is a useful contribution to the community. However, the validity of the headline performance numbers depends on the evaluation protocol, particularly the use of an unvalidated LLM judge for open-ended responses and the all-positive design of the error-awareness task. These issues must be resolved before the benchmark can be considered a reliable measurement instrument.

major comments (4)
  1. [§3.1, Eq. (1) and Eq. (4); Appendix B.2 (Fig. 12)] The CV and RE scores are computed by GPT-4.1-mini, a judge that is prompted with only the question, the model response, and the gold answer, with no video access. This means the judge cannot verify visual correctness; it can only reward textual similarity to the gold answer. A model that gives a correct but differently-worded answer, or that identifies a valid error not listed in the gold answer, will be scored 0 or 1, making the reported CV/RE accuracy an indirect measure of textual overlap rather than a direct measure of feedback quality. Furthermore, GPT-4.1-mini is itself one of the rows in Table 3, so the rankings could partially reflect response-style preference within the GPT-4.1 family rather than AIGC feedback ability. The paper reports no inter-judge agreement, no human calibration of judge scores, and no per-model score distribution analysis. Because CV and RE account for 1,982 of 9,740 items and heavily influence the overall score, the headline gap between humans (84.4%) and GPT-4.1 (51.6%) is not fully established. Please either validate the judge (e.g., human agreement on a subset, comparison with another judge, or analysis of judge score distributions) or reformulate the open-ended evaluation to be less dependent on a single uncalibrated LLM.
  2. [§3.3; §4.2; Table 3, EA columns] The Error Awareness task is all-positive: the paper states that all Yes-Or-No questions are designed with 'Yes' as the correct answer. Under this design, accuracy equals the model's 'Yes' rate, and a model that always answers 'Yes' would achieve 100%. The comparison to a 50% 'random guess' baseline in Table 3 and the interpretation in §4.2 that performance 'worse than random guessing' indicates a bias toward perceiving videos as normal conflates label imbalance with perceptual bias. A model with a generic tendency to answer 'No' would score below 50% even if it had perfect video understanding of the presented content. The intentional setup may be defensible as a probe of bias, but the current reporting and interpretation are misleading. Please reframe the result by reporting the model's 'No' rate, comparing against an all-Yes baseline, or including a subset of negative examples to separate label bias from genuine perceptual error.
  3. [§4.1; Table 6] The evaluated models receive different numbers of input frames (ranging from 2 to 16), with the choice made based on the model's context window. For example, GPT-4.1 receives 16 frames while llama3-llava-next-8b receives 2 frames. This introduces a confound: a model's performance may partly reflect how much of the video it sees rather than its intrinsic capability. The paper does not analyze this effect, yet it uses the resulting scores to rank models and to claim that a scaling law applies in §4.2. Please report results with a matched frame count for a subset of models, or at least provide an analysis of how performance varies with frame count and discuss the limitation explicitly.
  4. [§5.3; Table 4] The RePrompt experiment compares videos generated from the original prompts with videos generated from prompts revised by human annotators. This is a human-in-the-loop study, not an evaluation of MLLM feedback. The conclusion that 'aligning MLLMs more closely with human preferences can benefit video generation' is not directly supported because the revised prompts were produced by humans, not by MLLMs. To support the claim, the experiment should include a condition where MLLM-generated feedback is used to revise prompts, or the claims should be scaled back to what the experiment actually demonstrates (that human-revised prompts can improve some aspects of generated videos).
minor comments (6)
  1. [§3.3] The paper states that '2,395 question-answer pairs are corrected' and then refers to 'the low percentage of revisions' in the same paragraph. The corrected fraction is approximately 24.6% of the 9,740 total pairs, which is not obviously low. Please clarify how this number should be interpreted or revise the wording.
  2. [§3.1 and §3.3; Eq. (3)] The description of Error Type Detection says it 'intends to identify all the errors present in the AIGC video' and gives example questions such as 'Select the choices that reflect...', which suggests multi-select answers. However, Eq. (3) defines the score as an exact match I(y_i = ŷ_i) and Table 3 reports a 25% random-guess baseline, which is consistent with single-answer multiple-choice. Please clarify the answer format (single-answer vs. multi-select) and specify the scoring accordingly.
  3. [§4.1; Table 6] The frame sampling procedure is not described: are frames uniformly sampled, randomly sampled, or keyframe-based? This information is needed for reproducibility, especially because the number of frames varies across models.
  4. [§5.3; Table 4] The win rates in Table 4 are all close to 50% (50.7% to 57.6%), but no confidence intervals or significance tests are reported. Please report statistical significance or at least the number of pairwise comparisons used for each aspect.
  5. [Appendix B.2 (Fig. 9 and Fig. 12)] The prompt for Reasoning Evaluation is labeled '[CV_SHOT_PROMPT]', which is the same label used for Coherence Validation. Please rename it to '[RE_SHOT_PROMPT]' to avoid confusion.
  6. [Throughout] The benchmark name is spelled inconsistently: 'VF-EVAL' in the abstract and some equations, 'VF-Eval' in the title and elsewhere. Please standardize the spelling. Also, in the introduction the sentence 'that significantly from those found in traditional video content' appears to be missing a verb (e.g., 'differ').

Circularity Check

1 steps flagged · score 2.0 of 10

No construction-forced circularity: benchmark results are empirical and the headline human-model gap survives on objective items alone; score 2 reflects one low-severity self-definitional near-miss (GPT-4.1-mini judges the open-ended CV/RE items, Eqs. 1 and 4, while being itself an evaluated model) and minor non-load-bearing self-citations.

  1. self definitional [Section 3.1, Eqs. (1) and (4); Figure 12 scoring prompt (Appendix B.2); Table 3]
    "Coherence Validation uses an LLM (i.e., GPT-4.1-mini) to rate the generated answer ŷ against the correct answer y. The final score is calculated as: Score_CV = 1/N Σ LLM(y_i, ŷ_i) ... Reasoning Evaluation used an LLM (i.e., GPT-4.1-mini) to evaluate the MLLM's response ŷ against the human-provided answer y. The final score ... is computed as: Score_RE = 1/N Σ LLM(y_i, ŷ_i)"

    For the 1,982 open-ended items, each model's reported CV/RE score is, by Eqs. (1) and (4), the number GPT-4.1-mini assigns when given only the question, the response, and the gold answer (Figure 12); the video is not provided. The measured quantity is therefore definitionally GPT-4.1-mini's text-agreement judgment, yet it is reported as feedback ability on AIGC videos. The judge is itself an evaluated model (Table 3) and a sibling of the top-ranked GPT-4.1, with no judge calibration or inter-judge agreement reported and no statement that the human 84.4% baseline used the same judge.

full rationale

VF-Eval is a benchmark paper, not a derivation: its headline findings are empirical accuracies on human-annotated items, and no parameter is fitted to a subset of results and then renamed as a prediction. The only quotable reduction is in the open-ended scoring channel: Eqs. (1) and (4) define CV and RE performance for every model as the score assigned by GPT-4.1-mini, which receives only the question, the model response, and the gold answer (Figure 12), not the video. Reported open-ended 'performance' is therefore, by construction, GPT-4.1-mini's text-agreement judgment; since GPT-4.1-mini is itself a row of Table 3 and a sibling of the top-ranked GPT-4.1, with no reported judge calibration or human-judge agreement, that channel is a low-severity self-definitional near-miss. It is not construction-forced: the judge could assign any scores, and the central human-vs-model gap also holds on the objective Yes/No and multiple-choice items (e.g., Error Awareness: human 84.3 vs GPT-4.1 39.7; Error Type Detection CP: human 86.9 vs GPT-4.1 56.0), which require no LLM judge. The all-'Yes' design of the Yes/No task is disclosed (Section 3.3), so the inference from sub-50% accuracy to a 'perceives video as normal' bias is an explicit design choice rather than a hidden fit, though it does conflate detection failure with judging normality (a validity, not circularity, concern). Section 4.2 openly acknowledges possible clue leakage in the EA/ED tasks. The Limitations section lists three unrelated limitations and omits the judge-validity gap, which the skeptical analysis flags. Citations of MMVU (Zhao et al. 2025b) and TOMATO (Shangguan et al. 2024) are by this paper's co-authors but are positioning references only; both benchmarks are externally available, and VF-Eval's conclusions rest on this paper's own experiments, so they are not load-bearing. Because the benchmark is self-contained and externally checkable (released data and code) and the headline gap survives without the judge-scored items, the circularity score is a 2, driven by the self-definitional judge channel and minor self-citations, not a 4+.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

VF-Eval is a dataset rather than a fitted model, so the main load-bearing items are the annotation pipeline, the LLM judge, and the video collection choices listed above. No invented physical entities are introduced.

free parameters (2)
  • Open-ended response score scale = 0, 1, 2
    The LLM judge assigns only 0/1/2 in EXTRACT_OPENEND_RESPONSE_PROMPT (Appendix B.2). This hand-set rubric determines every CV and RE score; no calibration against human ratings is reported.
  • Input frame count per model = 2, 4, 8, or 16 depending on model
    Section 4.1 and Table 6 choose frame counts based on context window; differences in frame count make cross-model comparison inexact.
assumptions (3)
  • domain assumption GPT-4.1-mini reliably grades open-ended responses against human answers.
    Eq. 1 and Eq. 4 define CV and RE scores via LLM(y_i, yhat_i); no validation of judge agreement with human graders is provided.
  • domain assumption Human annotations are correct ground truth for all QA pairs.
    Dataset labels and revised prompts come from paid annotators; the paper reports inter-annotator agreement but no independent audit.
  • domain assumption The selected text-to-video models are representative of AIGC videos.
    Videos come from Pika, Kling, Pixeldance, Gen-3, T2V-Turbo-v2, LaVie, and OpenSora; the Limitations section notes image-to-video and cross-modal videos are excluded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos." pith.science (2026). https://pith.science/paper/FR4OSDT5

@misc{pith2026250523693,
  author       = {Pith},
  title        = {Pith review of: VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FR4OSDT5}},
  note         = {Machine review of arXiv:2505.23693}
}
read the original abstract

MLLMs have been widely studied for video question answering recently. However, most existing assessments focus on natural videos, overlooking synthetic videos, such as AI-generated content (AIGC). Meanwhile, some works in video generation rely on MLLMs to evaluate the quality of generated videos, but the capabilities of MLLMs on interpreting AIGC videos remain largely underexplored. To address this, we propose a new benchmark, VF-Eval, which introduces four tasks-coherence validation, error awareness, error type detection, and reasoning evaluation-to comprehensively evaluate the abilities of MLLMs on AIGC videos. We evaluate 13 frontier MLLMs on VF-Eval and find that even the best-performing model, GPT-4.1, struggles to achieve consistently good performance across all tasks. This highlights the challenging nature of our benchmark. Additionally, to investigate the practical applications of VF-Eval in improving video generation, we conduct an experiment, RePrompt, demonstrating that aligning MLLMs more closely with human feedback can benefit video generation.

Figures

Figures reproduced from arXiv: 2505.23693 by the authors.

Figure 1
Figure 1. Overview of our research: (a) Collection of AIGC videos: We compile a diverse set of video generation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of four proposed tasks and the corresponding question types in the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance Comparison of InternVL3-38B. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Performance comparison within four models [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: UI of annotation [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Prompt for Yes-Or-No questions. [COT_PROMPT] Question: {question} Options: {optionized_str} Solve the given multiple-choice question step by step. Begin by explaining your reasoning process clearly and thoroughly. After completing your analysis, conclude by stating the…
Figure 7
Figure 7. Figure 7: Prompt for Multiple-choice questions. [CV_SHOT_PROMPT] Question: {question} Answer the given question. The last line of your response should be in the following format: "Answer: Your Answer" (without quotes), where Your Answer is the final answer to the question. Think…
Figure 8
Figure 8. Figure 8: , and the prompt for RE is shown in [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Prompt for RE(Reasoning Evaluation) questions. [EXTRACT_YESNO_RESPONSE_PROMPT]] Given a string, extract the reasoning process and the final answer from the string. Output a JSON object with the following structure: "reason": "The reasoning process derived from the stri…
Figure 10
Figure 10. Figure 10: Extraction prompt for Multiple-choice question. [EXTRACT_MULTICHOICE_RESPONSE_PROMPT] Given a string, extract the reasoning process and the final answer from the string. Output a JSON object with the following structure: "reason": "The reasoning process derived from t…
Figure 11
Figure 11. Figure 11: Extraction prompt for Multiple-choice question [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Extraction prompt for Open-Ended question [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Detailed examples of reasoning tasks [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Misconception of Video Creation [Question]: Identify any quality issues in the video, including low resolution, blurriness, or distortion of the images. [Choices]: A. The grass in the foreground is disappearing intermittently. B. The tree shadows are flickering incons…
Figure 15
Figure 15. Figure 15: Excessive Dependence on Textual Cues [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Neglect of Critical Details [Question] : Why does the balloon fall down after flying up in the video? [LLM Response]: The balloon falls down after flying up in the video because it loses its buoyancy. This can happen for several reasons: the helium inside the balloon …
Figure 17
Figure 17. Figure 17: Over-reliance on Commonsense Knowledge [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 18 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  5. [5]

    Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. 2024. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03520

    arXiv 2024

  6. [6]

    Davide Bucciarelli, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. http://arxiv.org/abs/2412.03665 Personalizing multimodal large language models for image captioning: An experimental analysis

    work page Pith review arXiv 2024

  7. [7]

    Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, and Yu-Chiang Frank Wang. 2024. Rextime: A benchmark suite for reasoning-across-time in videos. arXiv preprint arXiv:2406.19392

    arXiv 2024

  8. [8]

    Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. 2023. Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering. arXiv preprint arXiv:2311.14906

    arXiv 2023

Show all 63 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Fanda Fan, Chunjie Luo, Wanling Gao, and Jianfeng Zhan. 2024. http://arxiv.org/abs/2401.01651 Aigcbench: Comprehensive evaluation of image-to-video content generated by ai

    2024 arXiv

  3. [11]

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406.14515

    2024 arXiv

  4. [12]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2024. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075

    2024 arXiv

  5. [13]

    Qihang Ge, Wei Sun, Yu Zhang, Yunhao Li, Zhongpeng Ji, Fengyu Sun, Shangling Jui, Xiongkuo Min, and Guangtao Zhai. 2024. Lmm-vqa: Advancing video quality assessment with large multimodal models. arXiv preprint arXiv:2408.14008

    2024 arXiv

  6. [14]

    Bravo, and Thomas Brox

    Simon Ging, María A. Bravo, and Thomas Brox. 2024. http://arxiv.org/abs/2402.07270 Open-ended vqa benchmarking of vision-language models by exploiting classification datasets and their semantic hierarchy

    2024 arXiv

  7. [15]

    Team GLM. 2024. http://arxiv.org/abs/2406.12793 Chatglm: A family of large language models from glm-130b to glm-4 all tools

    2024 arXiv

  8. [16]

    Google. 2024 a . http://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    2024 arXiv

  9. [17]

    Google. 2024 b . https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/ Gemini-2.0

    2024

  10. [18]

    Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. 2025. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint arXiv:2501.13926

    2025 arXiv

  11. [19]

    Xuehai He, Weixi Feng, Kaizhi Zheng, Yujie Lu, Wanrong Zhu, Jiachen Li, Yue Fan, Jianfeng Wang, Linjie Li, Zhengyuan Yang, et al. 2024. Mmworld: Towards multi-discipline multi-faceted world model evaluation in videos. arXiv preprint arXiv:2406.08407

    2024 arXiv

  12. [20]

    Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang, Youliang Yuan, Xiaoyuan Liu, Wenxuan Wang, Wenxiang Jiao, Pinjia He, and Zhaopeng Tu. 2025. Visfactor: Benchmarking fundamental visual cognition in multimodal large language models. arXiv preprint arXiv:2502.16435

    2025 arXiv

  13. [21]

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. 2024. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2024

  14. [22]

    InternLM. 2025. http://arxiv.org/abs/2504.10479 Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    2025 arXiv

  15. [23]

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos \'e Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. 2023. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125

    2023 arXiv

  16. [24]

    Daeun Lee, Jaehong Yoon, Jaemin Cho, and Mohit Bansal. 2024. Videorepair: Improving text-to-video generation via misalignment evaluation and localized refinement. arXiv preprint arXiv:2411.15115

    2024 arXiv

  17. [25]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  18. [26]

    Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. 2024 b . T2v-turbo-v2: Enhancing video generation model post-training through data, reward, and conditional guidance design. arXiv preprint arXiv:2410.05677

    2024

  19. [27]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355

    2023 arXiv

  20. [28]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024 c . Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

    2024

  21. [29]

    Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. 2024 d . Vlrewardbench: A challenging benchmark for vision-language generative reward models. arXiv preprint arXiv:2411.17451

    2024 arXiv

  22. [30]

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. 2024 e . Videovista: A versatile benchmark for video understanding and reasoning. arXiv preprint arXiv:2406.11303

    2024 arXiv

  23. [31]

    Chia Xin Liang, Pu Tian, Caitlyn Heqi Yin, Yao Yua, Wei An-Hou, Li Ming, Tianyang Wang, Ziqian Bi, and Ming Liu. 2024. A comprehensive survey and guide to multimodal large language models in vision-language tasks. arXiv preprint arXiv:2411.06284

    2024

  24. [32]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  25. [33]

    Xiaohong Liu, Xiongkuo Min, Guangtao Zhai, Chunyi Li, Tengchuan Kou, Wei Sun, Haoning Wu, Yixuan Gao, Yuqin Cao, Zicheng Zhang, et al. 2024 b . Ntire 2024 quality assessment of ai-generated content challenge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat...

    2024

  26. [34]

    Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. 2023. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. Advances in Neural Information Processing Systems, 36:62352--62387

    2023

  27. [35]

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024 c . Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476

    2024 arXiv

  28. [36]

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093

    2023 arXiv

  29. [37]

    Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. 2024. http://arxiv.org/abs/2410.05363 Towards world simulator: Crafting physical commonsense-based benchmark for video generation

    2024 arXiv

  30. [38]

    MistraAI. 2025. https://mistral.ai/news/mistral-small-3-1 Mistral small 3.1

    2025

  31. [39]

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. 2023. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103

    2023 arXiv

  32. [40]

    OpenAI. 2023. https://api.semanticscholar.org/CorpusID:257532815 Gpt-4 technical report . ArXiv, abs/2303.08774

    2023 arXiv

  33. [41]

    Bowen Qu, Xiaoyu Liang, Shangkun Sun, and Wei Gao. 2024. Exploring aigc video quality: A focus on visual harmony, video-text consistency and domain distribution gap. arXiv preprint arXiv:2404.13573

    2024 arXiv

  34. [42]

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. 2024. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313--14323

    2024

  35. [43]

    Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. 2024. http://arxiv.org/abs/2410.23266 Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models

    2024 arXiv

  36. [44]

    2020-2024

    Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. 2020-2024. https://github.com/HumanSignal/label-studio Label Studio : Data labeling software . Open source software available from https://github.com/HumanSignal/label-studio

    2020

  37. [45]

    Zhengzhong Tu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, and Alan C. Bovik. 2021. https://doi.org/10.1109/tip.2021.3072221 Ugc-vqa: Benchmarking blind video quality assessment for user generated content . IEEE Transactions on Image Processing, 30:4449–4464

    2021

  38. [46]

    Andong Wang, Bo Wu, Sunli Chen, Zhenfang Chen, Haotian Guan, Wei-Ning Lee, Li Erran Li, and Chuang Gan. 2024 a . Sok-bench: A situated video reasoning benchmark with aligned open-world knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  39. [47]

    Kegang Wang, Yantao Wei, Mingwen Tong, Jie Gao, Yi Tian, YuJian Ma, and ZhongJin Zhao. 2023 a . Physbench: A benchmark framework for rppg with a new dataset and baseline. arXiv preprint arXiv:2305.04161

    2023 arXiv

  40. [48]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024 b . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  41. [49]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023 b . Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079

    2023 arXiv

  42. [50]

    Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. 2023 c . Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103

    2023 arXiv

  43. [51]

    Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. 2024 c . http://arxiv.org/abs/2311.16511 Gpt4video: A unified multimodal large language model for lnstruction-followed understanding and safety-awar...

    2024 arXiv

  44. [52]

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. 2024 a . http://arxiv.org/abs/2309.14181 Q-bench: A benchmark for general-purpose foundation models on low-level vision

    2024 arXiv

  45. [53]

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. 2023 a . Q-align: Teaching lmms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.17090

    2023 arXiv

  46. [54]

    Te-Lin Wu, Zi-Yi Dou, Qingyuan Hu, Yu Hou, Nischal Chandra, Marjorie Freedman, Ralph Weischedel, and Nanyun Peng. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.719 ACQUIRED : A dataset for answering counterfactual questions in real-life videos . In Proceedings of the 20...

    2023 doi

  47. [55]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. 2024 b . Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302

    2024 arXiv

  48. [56]

    Lu Xu, Sijie Zhu, Chunyuan Li, Chia-Wen Kuo, Fan Chen, Xinyao Wang, Guang Chen, Dawei Du, Ye Yuan, and Longyin Wen. 2024. Beyond raw videos: Understanding edited videos with large multimodal model. arXiv preprint arXiv:2406.10484

    2024 arXiv

  49. [57]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. 2024. http://arxiv.org/abs/2412.14171 Thinking in space: How multimodal large language models see, remember, and recall spaces

    2024 arXiv

  50. [58]

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. 2025. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106

    2025 arXiv

  51. [59]

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024. https://llava-vl.github.io/blog/2024-04-30-llava-next-video/ Llava-next: A strong zero-shot video understanding model

    2024

  52. [60]

    Yilun Zhao, Guo Gan, Chen Zhao, and Arman Cohan. 2025 a . https://aclanthology.org/2025.naacl-long.582/ Are multimodal LLM s robust against adversarial perturbations? R o MM ath: A systematic evaluation on multimodal math reasoning . In Proceedings of the 2025 Conference of th...

    2025

  53. [61]

    Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, Zhijian Xu, Chengye Wang, Weifeng Pan, Ziyao Shangguan, Xiangru Tang, Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan. 2025 b . http://arxiv.org/abs/2...

    2025 arXiv

  54. [62]

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. 2024. https://github.com/hpcaitech/Open-Sora Open-sora: Democratizing efficient video production for all

    2024

  55. [64]

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024 b . Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.