Pith. sign in

REVIEW 4 major objections 5 minor 47 references

SCBench: A Sports Commentary Benchmark for Video LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper builds SCBench, a sports-commentary benchmark with 5,775 clips and a six-dimensional score, and reports that every tested video LLM fails at the task, with the best model (InternVL-Chat-2) reaching only 5.44/10.

desk verdict The CommentarySet dataset is a real contribution, but the paper's leaderboard is not trustworthy yet—the judge is validated on only 15 samples and the winner's longer outputs raise a concrete bias concern. read the letter →

arxiv 2412.17637 v1 pith:KWFQ62W6 submitted 2024-12-23 cs.CV cs.AI

Kuangzhi Ge , Lingjun Chen , Kevin Zhang , Yulin Luo , Tianyu Shi , Liaoyuan Fan , Xiang Li , Guanqun Wang
show 1 more author
Shanghang Zhang
This is my paper · ORCID
classification cs.CVcs.AI
keywords sportsvideocommentarylargelanguagemodelsbenchmarksix-dimensionalmetricGPT-basedevaluationSetSCOREStemporalunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that sports commentary is a hard, realistic test for video language models, harder than captioning or QA benchmarks, and that current open-source models all fail it. To make the test, the authors assemble CommentarySet, 5,775 professionally commented clips across six sports, and SCORES, a six-dimensional rubric (key events, technical detail, background, tactics, match situation, emotion) scored by a GPT judge against ground truth. Their central result is that InternVL-Chat-2 performs best at 5.44 out of 10, beating the next model by 1.04, and that fine-tuning with chain-of-thought improves two small models by 8% and 6% but still leaves them far from human-level commentary. The paper's significance, if true, is that fluent video understanding is not enough: models need fine-grained temporal and contextual understanding plus professional narration, and current benchmarks do not measure this.

What carries the argument

The engine of the benchmark is CommentarySet, a dataset of 5,775 short video clips (about 14.5 seconds each) with real broadcast commentary as ground truth, plus GPT-generated labels that partition each commentary into one or more of six dimensions: key events, technical detail, background information, tactical analysis, match situation, and emotional expression. The evaluation mechanism is SCORES, a prompt-based protocol in which GPT-4o-mini is first given the task definition, then the six-dimensional metric definitions, and then asked to grade model commentary against the ground-truth commentary on a 0-10 scale. The three-tier prompt structure (task definition, metric definition, task requirement) is what turns a generic LLM judge into a domain-specific one; the same dimension definitions are also fed to video models during inference to channel generation.

What would settle it

Score the same model outputs with a different judge model or with outputs normalized by length; if InternVL-Chat-2 no longer leads, or if a larger human study drops agreement with SCORES well below 60 percent, the ranking and the 'all models fail' conclusion become an artifact of the judge.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that sports commentary generation is a valid and demanding benchmark for video LLMs, and that every open-source model tested fails at it. The paper constructs CommentarySet, 5,775 professionally commented video clips across athletics, basketball, soccer, gymnastics, table tennis, and tennis, with ground-truth commentary and six-dimensional labels. It then introduces SCORES, a metric that scores generated commentary on key events, technical detail, background information, tactical analysis, match situation, and emotional expression, administered through a GPT judge that compares model output with ground truth. On this benchmark, InternVL-Chat-2 scores 5.44 out of 10, 1.04 points above the runner-up Mini-InternVL-Chat-v1.5, and the authors interpret the uniformly mediocre scores as evidence that current models cannot perform practical sports commentary. They also report that fine-tuning Video-LLaVA and Chat-UniVi with chain-of-thought improves their scores by 8% and 6%, respectively, but leaves them below several zero-shot models, indicating that stylistic alignment alone is insufficient.

Load-bearing premise

The load-bearing premise is that GPT-4o-mini's six-dimensional scores agree with human judgment well enough to rank models, a check done on only 15 samples with 60 percent agreement.

Editorial extensions

If this is right

  • If the benchmark is accepted, future video LLM comparisons will need a generation task with temporal and emotional dimensions, not just QA, to expose real-world capability gaps.
  • The 8% and 6% gains from chain-of-thought fine-tuning quantify how much of the deficit is style and prompt alignment, leaving the rest attributable to visual understanding and knowledge.
  • The 1.04-point gap between InternVL-Chat-2 and the runner-up becomes a concrete target for other developers.
  • Current automatic captioning metrics (BLEU, CIDEr) are shown to be near-useless for this task, pushing evaluation toward LLM-based judges with structured rubrics.
  • Because the dataset includes per-dimension labels, future work can train or prompt models to emphasize specific commentary dimensions rather than all at once.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension not in the paper: re-score the 867 test clips with a different judge model or a length-controlled version of GPT-4o-mini; if the ranking changes, the headline result is partly an artifact of judge preference for longer outputs, which the paper itself notices InternVL-2 tends to produce.
  • The six-dimensional label distribution per sport is effectively a commentary-style profile; one could build a style-conditioned generator that takes a desired dimension mix as input, which the paper mentions only as a possibility.
  • Because the ground-truth labels and the judge are generated by the same model family, the benchmark may partially measure self-consistency; a cross-family judge study would clarify how much of the score reflects genuine commentary quality.
  • The dataset could be re-purposed for image LLMs by sampling frames, as the authors note, which would let the same benchmark compare video-native and frame-based models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SCBench, a benchmark for evaluating Video LLMs on sports commentary generation. It contributes CommentarySet, a dataset of 5,775 video clips across six sports with extracted broadcast commentary and GPT-4o-mini-generated six-dimensional labels, and SCORES, a six-dimensional evaluation rubric scored by a GPT-4o-mini judge. The authors evaluate eight open-source Video LLMs plus two fine-tuning strategies, report that InternVL-Chat-2 achieves the highest SCORES (5.44, 1.04 above the runner-up), and conclude that current Video LLMs are far from producing practical sports commentary. The benchmark addresses a genuinely underexplored task; however, the validity of the central ranking and the 'all models fail' conclusion depends on the adequacy of the GPT-based evaluation, which is not yet established.

Significance. If the evaluation is trustworthy, SCBench fills a real gap: it targets dense, temporally rich sports videos and open-ended commentary generation, departing from the QA-format benchmarks that dominate the field. The CommentarySet itself, with 5,775 clips across six sports and time-aligned broadcast commentary, is a potentially reusable resource, and the six-dimensional rubric is a reasonable attempt to capture aspects that BLEU/CIDEr miss. The authors should be credited for releasing the dataset plan and for explicitly attempting a human validation of the metric. However, the paper's headline claims all rest on a GPT-4o-mini judge whose agreement with human judgment is demonstrated on only 15 samples, and no statistical or robustness evidence is provided. The significance of the empirical conclusions is therefore conditional on substantial additional validation.

major comments (4)
  1. [§5.3.1, Fig. 7] The human validation of SCORES uses only 15 samples and reports 60% agreement with human majority decisions versus 40% for vanilla GPT. No confidence interval, kappa, or significance test is given; a 9-of-15 versus 6-of-15 split is not statistically significant under a binomial or McNemar test. Since the entire model ranking in Tab. 3 is produced by this judge, the claim that SCORES is a meaningfully better metric than Vanilla GPT—and hence the validity of the benchmark—is unsupported. Please report the raw per-sample agreements, an appropriate test, and ideally a larger human study.
  2. [§3.3, §4.1] The same GPT-4o-mini model family generates the six-dimensional labels (Sec. 4.1, step 4) and serves as the judge (Sec. 3.3). Moreover, these labels are shown to models at inference time in the three-tier prompt and are also given to the judge to define the scoring criteria. Any systematic bias in GPT-4o-mini's label assignments therefore propagates into both model generation and evaluation, making the pipeline self-referential. The paper provides no human agreement measure on the label assignments themselves. Please provide a label-level validation (e.g., human agreement on a sample of labels) or an ablation that varies the label source (e.g., human labels on a subset) to show that the ranking is not an artifact of the label generator.
  3. [Tab. 3, §5.2] All SCORES results are reported as point estimates with no error bars, confidence intervals, or significance tests. Because the judge runs at temperature 0.1, repeated runs with different seeds would be straightforward to report. The claim that InternVL-Chat-2 'surpasses the second-best by 1.04' is a single realization of a stochastic judge; without variance information, the reader cannot assess whether a 1.04 gap is meaningful, particularly given that the judge may be sensitive to output length. Please report mean±std over multiple judge runs and, if possible, per-clip standard errors.
  4. [§5.3.2] The paper itself observes that InternVL-2's outputs were 'notably longer' and interprets this as capturing more information. However, longer outputs are also exactly what a fluency- or verbosity-biased judge would favor. The 15-sample human validation is far too small to rule out this confound. A concrete test would be to compare human preference on a subset where outputs are matched for length, or to truncate InternVL-2's outputs to the length of the runner-up and re-score; without such an analysis, the top ranking and the broad 'all models fail' conclusion are not robust.
minor comments (5)
  1. [Abstract and §4.1] The abstract describes the dataset as 'professionally annotated sports video clips,' but the six-dimensional labels are generated by GPT-4o-mini (Sec. 4.1, step 4), not by professional annotators. Please align the wording with the actual annotation pipeline.
  2. [Tab. 3] The Chat-UniVi-CoT row contains '3,00' instead of '3.00' in the soccer column; please fix the typo and check all numeric formatting.
  3. [Supplementary, Figs. 10-16] Several figures are numbered or captioned incorrectly (e.g., Fig. 10 appears twice, Fig. 14 and Fig. 15 both describe table tennis examples). Please renumber and deduplicate the supplementary figures.
  4. [§5.1] The evaluation section states that a temperature of 0.1 is used for GPT-4o-mini but does not specify how many judge calls are made per clip or whether the reported scores are averages over multiple calls; this should be stated explicitly.
  5. [§1 and §5.2] The paper claims that fine-tuning with ICL and CoT improves Video-LLaVA and Chat-UniVi by 8% and 6%, but these percentages are not clearly tied to the SCORES scale or to the numbers in Tab. 3; please clarify the computation (e.g., relative versus absolute improvement).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SCBench's model rankings are empirical outputs of an external judge; the GPT-label/GPT-judge overlap is a validation weakness, not a by-construction reduction.

full rationale

The paper's derivation chain is: broadcast commentary is sliced into clips (Sec. 4.1), GPT-4o-mini assigns six-dimensional labels, models generate commentary from video and labels, and GPT-4o-mini scores model commentary against ground truth and labels (Sec. 3.3). The Tab. 3 scores are not computed from the labels by any equation; they are judgments of a separate model output conditioned on ground truth. The use of the same model family for label generation and judging is a legitimate measurement-validity concern, and the 15-sample, 60% human agreement in Sec. 5.3.1 is weak evidence, especially because Sec. 5.3.2 notes that the winning model's outputs were 'notably longer.' However, that is a bias/validity threat, not circularity: the judge's scores are not equal to the labels, no parameter is fitted so as to force the ranking, and the ground-truth commentary is real broadcast text rather than a re-statement of the labels. The fine-tuned models are trained on GPT-generated labels and scored by a GPT-based judge, which may reward alignment with the judge's preferences, but this is overfitting to the evaluation protocol, not an identity between training input and reported prediction. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation steps appear. The paper's own stated limitations (no closed-source MLLMs, small dataset, models still below human-level) are acknowledged and do not assert or conceal a circular step. Therefore, under the rule that circularity requires an exhibited reduction of a result to its inputs, no such step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central contributions depend on assumptions about the reliability of GPT-generated labels and scoring, alignment between commentary and video, and the completeness of the six-dimensional taxonomy. No free parameters are fitted to produce the dataset, but the merging threshold and judge temperature are ad hoc choices. The invented entities are the SCORES rubric and CommentarySet itself, which lack independent validation beyond a 15-sample human study.

free parameters (3)
  • Commentary merging cosine threshold = 0.7
    Ad hoc experimental choice in Sec. 4.1, Stage 1, for merging commentary sentences; not justified or validated.
  • GPT judge temperature = 0.1
    Chosen in Sec. 5.1 for GPT-4o-mini evaluation; no sensitivity analysis.
  • Maximum number of label selections = 3
    Prompt in Sec. C limits GPT label generation to at most three of six dimensions; arbitrary cap.
assumptions (4)
  • domain assumption Broadcast commentary timestamps faithfully align with visual events in the clipped video after merging and manual refinement.
    Sec. 4.1 steps 1-3 assume the extracted commentary corresponds to the video content; manual refinement is not quantified or quality-checked.
  • domain assumption GPT-4o-mini-generated six-dimensional labels are accurate enough for training and evaluation.
    Sec. 4.1 step 4 uses GPT-4o-mini to label all commentary without human verification; the label distribution in Fig. 4 is then treated as ground truth.
  • domain assumption GPT-4o-mini as judge produces scores that agree with human judgment.
    Sec. 3.3 and Sec. 5.3.1 rely on a 15-sample human study; no large-scale validation, no inter-rater agreement reported.
  • ad hoc to paper The six SCORES dimensions comprehensively cover sports commentary.
    Sec. 3.2.2 defines the taxonomy internally; no evidence it is complete or that dimensions are mutually exclusive.
invented entities (2)
  • SCORES six-dimensional evaluation rubric
    purpose: Defines what good sports commentary should contain, used for both model prompting and scoring.
    The six dimensions were defined by the authors and implemented via GPT labeling; only external validation is a 15-sample human study.
  • CommentarySet dataset
    purpose: Provides 5,775 video clips with ground-truth commentary and labels for benchmark training and testing.
    Dataset is not released and no external replication exists; the commentary is extracted from broadcasts and labels are GPT-generated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SCBench: A Sports Commentary Benchmark for Video LLMs." pith.science (2026). https://pith.science/paper/KWFQ62W6

@misc{pith2026241217637,
  author       = {Pith},
  title        = {Pith review of: SCBench: A Sports Commentary Benchmark for Video LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWFQ62W6}},
  note         = {Machine review of arXiv:2412.17637}
}
abstract

Recently, significant advances have been made in Video Large Language Models (Video LLMs) in both academia and industry. However, methods to evaluate and benchmark the performance of different Video LLMs, especially their fine-grained, temporal visual capabilities, remain very limited. On one hand, current benchmarks use relatively simple videos (e.g., subtitled movie clips) where the model can understand the entire video by processing just a few frames. On the other hand, their datasets lack diversity in task format, comprising only QA or multi-choice QA, which overlooks the models' capacity for generating in-depth and precise texts. Sports videos, which feature intricate visual information, sequential events, and emotionally charged commentary, present a critical challenge for Video LLMs, making sports commentary an ideal benchmarking task. Inspired by these challenges, we propose a novel task: sports video commentary generation, developed $\textbf{SCBench}$ for Video LLMs. To construct such a benchmark, we introduce (1) $\textbf{SCORES}$, a six-dimensional metric specifically designed for our task, upon which we propose a GPT-based evaluation method, and (2) $\textbf{CommentarySet}$, a dataset consisting of 5,775 annotated video clips and ground-truth labels tailored to our metric. Based on SCBench, we conduct comprehensive evaluations on multiple Video LLMs (e.g. VILA, Video-LLaVA, etc.) and chain-of-thought baseline methods. Our results found that InternVL-Chat-2 achieves the best performance with 5.44, surpassing the second-best by 1.04. Our work provides a fresh perspective for future research, aiming to enhance models' overall capabilities in complex visual understanding tasks. Our dataset will be released soon.

Figures

Figures reproduced from arXiv: 2412.17637 by the authors.

Figure 1
Figure 1. A sample from Six-Dimensional Metric and CommentarySet, including six dimensions’ description, frames from videos, and corresponding ground truth commentary. To holistically assess their margin of improvement, many benchmarks and evaluation methods have been proposed Ning et al. (2023a); Sanders and Van Durme (2024); Ning et al. (2023b); Fu et al. (2024). Existing benchmarks for Video-LLMs predominantly concentrate … view at source ↗
Figure 2
Figure 2. Distinctions between Caption and Commentary. Although captioning provides precise visual descriptions, commentary delivers dynamic, context-rich, and emotionally resonant narration. To be more specific, traditional metrics (e.g. BLEU Papineni et al. (2002), ROUGE_L Lin (2004), METEOR Banerjee and Lavie (2005), CIDEr Vedantam et al. (2014) and SPICE Anderson et al. (2016)) primarily evaluate the lexical or structural… view at source ↗
Figure 3
Figure 3. The overall framework of SCBench. SCBench consists of two parts: the construction of CommentarySet and our GPT-based evaluation [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Distribution of the labels The evaluation process could be divided into two stages: Model Inference and GPT Evaluation. In the first stage, we employ a three-tier prompt structure of task definition - corresponding metric definition - task requirement: For each video c…
Figure 5
Figure 5. Figure 5: Model Performances on Six Sports (left) and Six-Dimensions in SCORES (right) tasks [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: An example question and choices from the user study [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Results of user study (15 samples). Bars represent traditional metrics (BLEU, CIDEr, Vanilla LLM), while the line represents our SCORES and human evaluators (the dark red part is the overlap). specially designed to evaluate the task. Our experiments reveal significant …
Figure 8
Figure 8. Figure 8: Examples of the comparison of the generated commentaries by zero-shot and fine￾tuned models in soccer and tennis. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Samples of our Human Evaluation Questionnaire [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: An example of the comparison of the generated commentaries in basketball. The sections highlighted in different colors represent the content corresponding to each label. The same applies to subsequent cases. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 10
Figure 10. Figure 10: An example of the comparison of the generated commentaries in basketball [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: An example of the comparison of the generated commentaries in soccer. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 11
Figure 11. Figure 11: An example of the comparison of the generated commentaries in soccer [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: An example of the comparison of the generated commentaries in athletics. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 12
Figure 12. Figure 12: An example of the comparison of the generated commentaries in athletics [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: An example of the comparison of the generated commentaries in tennis. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 13
Figure 13. Figure 13: An example of the comparison of the generated commentaries in tennis. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: An example of the comparison of the generated commentaries in tabletennis. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 14
Figure 14. Figure 14: An example of the comparison of the generated commentaries [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: An example of the comparison of the generated commentaries in tabletennis. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: An example of the comparison of the generated commentaries in tabletennis. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 30 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Spice: Semantic propositional image caption evaluation, 2016

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation, 2016

    work page 2016

  3. [3]

    METEOR : An automatic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR : An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan, 2005. Association for Computational Linguistics

    work page 2005

  4. [4]

    P2anet: A dataset and benchmark for dense action detection from table tennis match broadcasting videos, 2024

    Jiang Bian, Xuhong Li, Tao Wang, Qingzhong Wang, Jun Huang, Chen Liu, Jun Zhao, Feixiang Lu, Dejing Dou, and Haoyi Xiong. P2anet: A dataset and benchmark for dense action detection from table tennis match broadcasting videos, 2024

    work page 2024

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    2020

  6. [6]

    Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering, 2024 a

    Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering, 2024 a

    work page 2024

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024 b

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024 b

    work page 2024

  8. [8]

    Sports re-id: Improving re-identification of players in broadcast videos of team sports, 2022

    Bharath Comandur. Sports re-id: Improving re-identification of players in broadcast videos of team sports, 2022

    work page 2022

Show all 47 references
  1. [9]

    Seikavandi, Jacob V

    Adrien Deliège, Anthony Cioppa, Silvio Giancola, Meisam J. Seikavandi, Jacob V. Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B. Moeslund, and Marc Van Droogenbroeck. Soccernet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer videos, 2021

    2021

  2. [10]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding, 2024

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding, 2024

    2024

  3. [11]

    Hayden Faulkner and Anthony R. Dick. Tenniset: A dataset for dense fine-grained event recognition, localisation and description. 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 1--8, 2017

    2017

  4. [12]

    Chatpose: Chatting about 3d human pose

    Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Chatpose: Chatting about 3d human pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2093--2103, 2024

    2024

  5. [13]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever compre...

    2024

  6. [14]

    Mini-internvl: A flexible-transfer pocket multimodal model with 5

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Jifeng Dai, and Wenhai Wang. Mini-internvl: A flexible-transfer pocket multimodal model with 5

  7. [15]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering, 2017

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering, 2017

    2017

  8. [16]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding, 2024

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding, 2024

    2024

  9. [17]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    2021

  10. [18]

    Mvbench: A comprehensive multi-modal video understanding benchmark, 2024

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark, 2024

    2024

  11. [19]

    Video-llava: Learning united visual representation by alignment before projection, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection, 2023

    2023

  12. [20]

    ROUGE : A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81, Barcelona, Spain, 2004. Association for Computational Linguistics

    2004

  13. [21]

    Vila: On pre-training for visual language models, 2024

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2024

    2024

  14. [22]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024 a

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024 a

    2024

  15. [23]

    Kangaroo: A powerful video-language model supporting long-context video input, 2024 b

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input, 2024 b

    2024

  16. [24]

    Fineaction: A fine-grained video dataset for temporal action localization

    Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal action localization. IEEE transactions on image processing, 31: 0 6937--6950, 2022

    2022

  17. [25]

    Tempcompass: Do video llms really understand videos?, 2024 c

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos?, 2024 c

    2024

  18. [26]

    Cross-block fine-grained semantic cascade for skeleton-based sports action recognition, 2024 d

    Zhendong Liu, Haifeng Xia, Tong Guo, Libo Sun, Ming Shao, and Siyu Xia. Cross-block fine-grained semantic cascade for skeleton-based sports action recognition, 2024 d

    2024

  19. [27]

    Egoschema: A diagnostic benchmark for very long-form video language understanding, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding, 2023

    2023

  20. [28]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages ...

    2019

  21. [29]

    Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103, 2023 a

    2023 arXiv

  22. [30]

    Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models, 2023 b

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models, 2023 b

    2023

  23. [31]

    B leu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. B leu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--318, Philadelphia, Pennsylvania, USA, 2002. Asso...

    2002

  24. [32]

    Perception test: A diagnostic benchmark for multimodal video models, 2023

    Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koste...

    2023

  25. [33]

    A survey of video datasets for grounded event understanding

    Kate Sanders and Benjamin Van Durme. A survey of video datasets for grounded event understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7314--7327, 2024

    2024

  26. [34]

    Finegym: A hierarchical video dataset for fine-grained action understanding, 2020

    Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine-grained action understanding, 2020

    2020

  27. [35]

    Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning, 2024

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning, 2024

    2024

  28. [36]

    Playertv: Advanced player tracking and identification for automatic soccer highlight clips, 2024

    Håkon Maric Solberg, Mehdi Houshmand Sarkhoosh, Sushant Gautam, Saeed Shafiee Sabet, Pål Halvorsen, and Cise Midoglu. Playertv: Advanced player tracking and identification for automatic soccer highlight clips, 2024

    2024

  29. [37]

    Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy

    OpenGVLab Team. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy. Accessed: 2024-11-14

    2024

  30. [38]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. CoRR, abs/1411.5726, 2014

    2014 arXiv

  31. [39]

    Lvbench: An extreme long video understanding benchmark, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024

    2024

  32. [40]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  33. [41]

    Sportshhi: A dataset for human-human interaction detection in sports videos, 2024

    Tao Wu, Runyu He, Gangshan Wu, and Limin Wang. Sportshhi: A dataset for human-human interaction detection in sports videos, 2024

    2024

  34. [42]

    Next-qa:next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat Seng Chua. Next-qa:next phase of question-answering to explaining temporal actions. 2021

    2021

  35. [43]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM, pages 1645--1653, 2017

    2017

  36. [44]

    Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks, 2023

    Haiyang Xu, Qinghao Ye, Xuan Wu, Ming Yan, Yuan Miao, Jiabo Ye, Guohai Xu, Anwen Hu, Yaya Shi, Guangwei Xu, Chenliang Li, Qi Qian, Maofei Que, Ji Zhang, Xiao Zeng, and Fei Huang. Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks, 2023

    2023

  37. [45]

    Activitynet-qa: A dataset for understanding complex web videos via question answering, 2019

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering, 2019

    2019

  38. [46]

    Long context transfer from language to vision, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision, 2024

    2024

  39. [47]

    A survey on generative ai and llm for video generation, understanding, and streaming

    Pengyuan Zhou, Lin Wang, Zhi Liu, Yanbin Hao, Pan Hui, Sasu Tarkoma, and Jussi Kangasharju. A survey on generative ai and llm for video generation, understanding, and streaming. arXiv preprint arXiv:2404.16038, 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.