Pith. sign in

REVIEW 4 major objections 6 minor 14 cited by

CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CG-Bench claims that multiple-choice-only benchmarks overstate long-video understanding, and supports this with clue-grounded evaluation: GPT-4o's 45.2% MCQ accuracy falls to 4.38% when the model must also point to the interval containing…

desk verdict Large, well-annotated long-video QA benchmark, but the credibility metrics are confounded by frame sampling and don't support the paper's central claim. read the letter →

arxiv 2412.12075 v1 pith:FLCHZZMC submitted 2024-12-16 cs.CV

Guo Chen , Yicheng Liu , Yifei Huang , Yuping He , Baoqi Pei , Jilan Xu , Yali Wang , Tong Lu
show 1 more author
Limin Wang
This is my paper · ORCID
classification cs.CV
keywords longvideounderstandingmultimodallargelanguagemodelsquestionansweringtemporalgroundingclue-groundedevaluationmultiple-choicebenchmarkcredibilityhuman-annotatedclues
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CG-Bench argues that existing multiple-choice benchmarks for long-video understanding can be passed without genuinely watching the video: models can combine short glimpses, captions, and elimination to pick the right option. To close this loophole, the paper builds a benchmark of 1,219 long videos with 12,129 human-annotated question-answer-clue triplets, where each question is tied to specific clue intervals. It then evaluates models two ways: whether the answer is correct on the full video, and whether the model can locate the clue interval or keep full-video accuracy above clip-only accuracy. On these credibility metrics, GPT-4o's long-video accuracy drops from 45.2% to 4.38% when it must also point to a clue, and no model reaches 100% Clue Recovery Rate, which the paper reads as evidence that current MLLMs are not yet trustworthy for long videos.

What carries the argument

The load-bearing object is the question-answer-clue (QAC) triplet, in which each multiple-choice question is annotated with one or more temporal intervals that contain the evidence. On top of it sit two evaluation mechanisms: white-box evaluation, which asks the model to output timestamp intervals and measures them with $\mathrm{tIoU}$, $\mathrm{mIoU}$, $\mathrm{recall@IoU}$, and $\mathrm{acc@IoU}$; and black-box evaluation, which compares full-video accuracy to clue-clip accuracy through the Clue Recovery Rate, $\mathrm{CRR} = \min(\mathrm{long\text{-}acc.}, \mathrm{clue\text{-}acc.})/\mathrm{clue\text{-}acc.}$. The identity doing the work is the inequality $\mathrm{long\text{-}acc.} \ge \mathrm{clue\text{-}acc.}$, which turns the benchmark into a self-check: if seeing the whole video is not better than seeing the clue, the model did not effectively retrieve the clue from the long context.

What would settle it

Re-annotate a random sample of CG-Bench questions with independent annotators and compute inter-annotator $\mathrm{tIoU}$ on the clue intervals; if many questions have low agreement, or if a model given the full video but with annotated clues masked still beats its clue-only accuracy, then hidden clues exist and the CRR inequality is not a valid test of clue retrieval.

Watch

Extended reading notes

Core claim

The paper's central claim is that multiple-choice accuracy overstates long-video understanding, and that clue-grounded evaluation exposes this. On its benchmark, the best commercial model GPT-4o scores 45.2% on long-video multiple choice, but when the same questions are scored with $\mathrm{acc@IoU}$—correct option plus a predicted interval with $\mathrm{tIoU} > 0$ against the annotated clue—the score collapses to 4.38%. The black-box Clue Recovery Rate, $\mathrm{CRR} = \min(\mathrm{long\text{-}acc.}, \mathrm{clue\text{-}acc.})/\mathrm{clue\text{-}acc.}$, is below 100% for every model, meaning full-video context dilutes rather than helps clue retrieval relative to watching only the clue clip. The paper concludes that current models can answer many questions without grounding them in the relevant evidence, and that CG-Bench's clue-based metrics provide a more credible measurement.

Load-bearing premise

The credibility metrics assume the human-annotated clue intervals are both necessary and sufficient evidence—if a question can be answered from other parts of the video, then a low full-video accuracy relative to clue-clip accuracy may reflect incomplete annotations rather than failed clue retrieval.

Editorial extensions

If this is right

  • MCQ-only long-video leaderboards systematically overstate model capability; adding a clue-interval requirement changes the ranking and widens the gaps between models.
  • All evaluated models score below 100% CRR, so current MLLMs are not reliably retrieving the relevant moments from long contexts; improving temporal grounding is a concrete training target.
  • Timestamp information from frames and subtitles is what lets models point at clues: adding both raises GPT-4o's $\mathrm{mIoU}$ from 3.39 to 9.68 and $\mathrm{acc@IoU}$ from 10.7 to 26.7.
  • Human performance on the same sparse 128-frame input is 59.85%, versus 90.3% with full video, indicating the benchmark is difficult and favors models that use long-context visual information rather than sparse frames.
  • Open-source models such as Qwen2-VL-72B approach GPT-4o's MCQ accuracy (41.3 vs. 45.2) but remain behind on clue grounding, so the open-source gap is mostly a grounding gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the clue-completeness assumption could be tested by asking a second annotation team to find all intervals that support each answer and recomputing CRR with the union of intervals; if CRR rises materially, the original annotations were incomplete.
  • As an editorial extension, the benchmark's black-box assumption also suggests a direct training signal: regularize models so full-context accuracy never falls below clip-level accuracy, which could improve long-video reliability without new annotations.
  • Because $\mathrm{acc@IoU}$ requires overlap with the annotated clue, a model that answers correctly using legitimate alternative evidence is penalized; a future variant could score with soft overlap against any sufficient interval, not just the human-annotated one.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CG-Bench, a long-video question-answering benchmark with 1,219 manually curated videos and 12,129 human-annotated question-answer-clue (QAC) triplets. It proposes two credibility evaluation protocols—white-box grounding (mIoU, rec@IoU, acc@IoU) and black-box clue recovery (CRR)—in addition to traditional MCQ and clue-aided open-ended evaluation, aiming to determine whether MLLMs base their answers on specific video evidence. Experiments across open- and closed-source MLLMs report moderate MCQ accuracy but much lower scores on credibility metrics (e.g., GPT-4o long-acc. 45.2 versus acc@IoU 4.38), which the paper interprets as evidence that current models do not genuinely understand long videos.

Significance. If the credibility metrics are valid, CG-Bench makes a valuable contribution: it is the largest long-video QA benchmark with question grounding annotations, uses a granular 14/171/638 category taxonomy, releases data and a leaderboard, and includes human baselines. The white-box/black-box distinction is a useful conceptual advance over pure MCQ evaluation, and the paper provides careful analyses of frame sampling, prompt/modality effects, and the open-ended evaluator, with human agreement used to validate the latter. However, the paper's central claim that models fail to ground answers in video evidence currently rests on metrics that are confounded by the evaluation protocol, so the quantitative findings and the benchmark's credibility contribution need strengthening before they can be fully accepted.

major comments (4)
  1. [Sec 4.1, Table 3] The frame-count asymmetry confounds the credibility metrics. Long-video MCQ uses 128 uniformly sampled frames over full videos (average 1624 s), while clue-based MCQ uses 32 frames over clue clips (average 19.24 s per Table 1). This yields roughly 1.5 sampled frames inside a typical clue interval for the long-video setting. Consequently, low long-acc is partly a temporal-resolution artifact, and the drop in CRR from 100 may reflect the model seeing fewer clue frames rather than a failure to retrieve clues. The paper's own human sparse-frame long-acc of 59.9 versus 90.3 full-video (Table 3) quantifies this bottleneck. To support the claim that low CRR indicates poor clue recovery, the authors should report human CRR under the same 128/32 frame protocol or match frame density across the two settings.
  2. [Sec 3.3.2, Eq. (2)] The premise underlying CRR, that long-acc should be greater than or equal to clue-acc, is violated by the design of the frame sampling. Because 32 frames over a 19-second clue are far denser than 128 frames over a 27-minute video, clue-acc has an inherent advantage unrelated to context dilution. The metric thus conflates temporal resolution with clue-retrieval ability. Additionally, the paper acknowledges in this same section that 'hidden clues' may exist, but it provides no coverage validation that the annotated intervals are either sufficient or necessary for answering each question. Without inter-annotator agreement or a coverage study, a low CRR could reflect incomplete annotation rather than a model deficit.
  3. [Sec 3.3.2, Table 3] The acc@IoU threshold is under-specified. The text states that the default threshold is τ=0 (so any overlap, tIoU>0, counts) but also says that acc@IoU is calculated at IoU thresholds of 0.1, 0.2, 0.3, 0.4, and 0.5. It is unclear which value is reported in Table 3 (e.g., GPT-4o's 4.38). If the table reports τ=0, the metric requires only a nonzero overlap; if it reports an average over the thresholds, the interpretation is different. The authors should state explicitly which threshold (or whether an average) is used, and report the full threshold sweep in a figure or table.
  4. [Sec 3.1] No inter-annotator agreement or coverage validation is reported for the clue intervals. The benchmark's credibility depends on the human-annotated clue intervals being both necessary and sufficient for each question, but the annotation process in Sec 3.1 describes only a qualitative 'review iteration' process. Provide quantitative agreement (e.g., temporal IoU between annotators, or kappa on interval selection) and a check that questions cannot be answered from outside the annotated clues. Without this, the interpretation of low acc@IoU and CRR as evidence of failure to ground answers is not fully supported.
minor comments (6)
  1. [Figure 2, Sec 1] There are minor typos: 'Electonic' should be 'Electronic' in Figure 2, and 'clue-grouded' should be 'clue-grounded' in the Introduction.
  2. [Table 2] In the CG-Bench-Clue row, the columns #Video and #Dur list 12,129 and 22.8, but these appear to be the number of QAC triplets and the average clue duration, not the number of videos and video duration. Please relabel the columns or correct the values.
  3. [Sec 4.1] The sentence 'use 32 frames as the for Clue-based MCQ' is grammatically incomplete; please revise it.
  4. [Eq. (1)] The tIoU formula sums pairwise intersections over all ground-truth and predicted intervals; if intervals within a set overlap, pairwise intersections can exceed the true union. Please clarify whether intervals are merged before computing the metric and whether the formula is the standard multi-interval IoU.
  5. [Sec 4.2] The human sparse-frame evaluation uses only 30 videos (296 questions), which is a small sample for a strong claim that 128 frames are insufficient. Please report confidence intervals or increase the sample size.
  6. [Figure 7, Table 4] The ablations in Figure 7 and Table 4 are computed on a 1000-QAC subset; please state explicitly whether these results were verified to be representative of the full benchmark.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CG-Bench is an external measurement instrument; the credibility metrics are defined from measured accuracies, and the frame-sampling asymmetry is a validity concern, not a circular reduction.

full rationale

CG-Bench does not derive its central claims from fitted parameters or from self-citations that carry the argument. The headline results (e.g., GPT-4o long-acc. 45.2 vs. acc.@IoU 4.38) are measured outcomes reported in Table 3, not quantities constructed to equal their inputs. The CRR metric, CRR = min(long-acc., clue-acc.) / clue-acc., is a definitional ratio of two measured accuracies; interpreting a low CRR as failed clue recovery relies on the assumption that hidden clues do not compensate for annotated clue intervals, but this is an annotation-completeness assumption rather than a circular derivation. The frame-sampling asymmetry (128 frames for long-video MCQ vs. 32 frames for clue-based MCQ, with average clue duration 19.24 s in videos averaging 1624.4 s) is a genuine potential confound for acc.@IoU and CRR, and the paper's own human sparse-frame long-acc. of 59.9 (Table 3) shows that sampling alone substantially reduces accuracy. This is a correctness/validity limitation to weigh in the benchmark's interpretation, not a case where the paper's equations reduce to their own inputs by construction. The use of GPT-4o as an open-ended evaluator and also as an evaluated model is a mild dependency, but the paper validates the evaluator against human scores (Figure 8, Table 5), so it does not force the main conclusions. Self-citations appear only in related-work listings (e.g., EgoExoLearn, MoVQA, InternVideo2) and are not load-bearing for the benchmark's evaluation methodology. No circular step meeting the quoted-evidence standard was found.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central evaluation rests primarily on two unproven domain assumptions about clue annotation completeness and pretraining overlap, not on fitted parameters. The threshold and frame-count choices are experimental settings that affect the headline numbers.

free parameters (2)
  • acc@IoU threshold tau = 0 (default for ablations; thresholds 0.1-0.5 also referenced)
    The combined accuracy-grounding metric is threshold-dependent; the paper sets tau=0 for 'more obvious comparison on ablation studies' but reports a single acc.@IoU in Table 3, so the effective threshold is ambiguous.
  • Frame counts for evaluation = 32 frames for clue-acc, 128 frames for long-acc (some models use 4-1200 frames)
    Frame sampling density affects long-acc and therefore CRR; the paper chooses these counts by hardware availability, not by a criterion that equalizes information density across short clue clips and long videos.
assumptions (3)
  • domain assumption Manually collected internet videos have not been seen in MLLM pretraining.
    Stated in Sec 3.1 ('To avoid using videos that have been used for pre-training... we manually collect videos from the internet'), but no deduplication or overlap check with training corpora is described.
  • domain assumption Annotated clue intervals are complete and sufficient evidence for each answer.
    Sec 3.1 instructs annotators to mark intervals covering relevant short moments; the credibility metrics in Sec 3.3.2 assume these intervals capture the evidence needed to answer, with no reported inter-annotator agreement.
  • domain assumption A model that truly understands the video should perform at least as well on the full video as on the clue-only clip.
    This monotonicity assumption underlies CRR in Sec 3.3.2; it ignores the possibility that long context dilutes attention even for a capable model, or that the full video contains misleading additional content.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding." pith.science (2026). https://pith.science/paper/FLCHZZMC

@misc{pith2026241212075,
  author       = {Pith},
  title        = {Pith review of: CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FLCHZZMC}},
  note         = {Machine review of arXiv:2412.12075}
}
read the original abstract

Most existing video understanding benchmarks for multimodal large language models (MLLMs) focus only on short videos. The limited number of benchmarks for long video understanding often rely solely on multiple-choice questions (MCQs). However, because of the inherent limitation of MCQ-based evaluation and the increasing reasoning ability of MLLMs, models can give the current answer purely by combining short video understanding with elimination, without genuinely understanding the video content. To address this gap, we introduce CG-Bench, a novel benchmark designed for clue-grounded question answering in long videos. CG-Bench emphasizes the model's ability to retrieve relevant clues for questions, enhancing evaluation credibility. It features 1,219 manually curated videos categorized by a granular system with 14 primary categories, 171 secondary categories, and 638 tertiary categories, making it the largest benchmark for long video analysis. The benchmark includes 12,129 QA pairs in three major question types: perception, reasoning, and hallucination. Compensating the drawbacks of pure MCQ-based evaluation, we design two novel clue-based evaluation methods: clue-grounded white box and black box evaluations, to assess whether the model generates answers based on the correct understanding of the video. We evaluate multiple closed-source and open-source MLLMs on CG-Bench. Results indicate that current models significantly underperform in understanding long videos compared to short ones, and a significant gap exists between open-source and commercial models. We hope CG-Bench can advance the development of more trustworthy and capable MLLMs for long video understanding. All annotations and video data are released at https://cg-bench.github.io/leaderboard/.

Figures

Figures reproduced from arXiv: 2412.12075 by the authors.

Figure 1
Figure 1. Left: examples of CG-Bench’s clue-grounded annotation. To correctly answer the questions, models need to ground their reasoning into the correct clue. Right: CG-Bench provides an evaluation suite with two novel credibility evaluation criteria while supporting both MCQ and open-ended evaluations. benchmark tries to address the problem of credible models by incorporating temporal grounding into MCQ. However, NExT-GQA … view at source ↗
Figure 2
Figure 2. Distribution of video root categories, dis [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Clue time coverage, il￾lustrating the frequency of clues across different time bins. Annotation Statistics #QAC Triplets 12129 #Avg/QAC per video 9.95 #Avg/Option per QAC 6.96 #Avg/Clue per QAC 1.18 #Avg/Words of Questions 20.07 #Avg/Words of Options 22.88 #Avg/Duration of Clues 19.24 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figures from the paper (4 more)
Figure 6
Figure 6. Figure 6: Two examples illustrating the ambiguity challenge of using LLMs for open-ended evaluation. While [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Impact of sampling frame numbers on different metrics for GPT-4o-0806 (OpenAI, 2024), Gemini [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Comparison of using different LLMs as open-ended evaluators for GPT-4o’s outputs. GT GT+Vis Vis Ours Bias(%)↓ 12.4 6.4 17.0 1.0 Time (s)↓ 741 20,040 19,640 3,600 Price ($)↓ 0.05 6.1 6 2 Trigger Rate (%)↓ 0 100 100 14 Trigger Recall Rate (%)↑ 0 100 100 88 [PITH_FULL_IM…
Figure 9
Figure 9. Figure 9: Long-Video-MCQ Accuracy grouped by video duration for GPT4o-0806 with 128 frames. #Frames Resolution Sampling Strategy long-acc. 128 Low Uniform 53.9 50 Low Uniform 46.7 50 Low Keyframe 45.7 50 High Uniform 51.0 [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Incentivizing Vision Language Models to Search for Long Video Question Answering

    cs.CV 2026-07 conditional novelty 7.0 of 10

    RL post-training of a VLM agent with neuro-symbolic temporal-logic rewards for evidence retrieval raises Pass@1 by up to 8% and Pass@4 by 15% on long-video QA.

  2. Towards Temporal Compositional Reasoning in Long-Form Sports Videos

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    SportsTime plus Chain-of-Time Reasoning (temporal-reward GRPO and anchor-observe-infer) modestly lifts open-ended sports VideoQA and step-wise temporal grounding over 4B–8B MLLM baselines.

  3. EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs

    cs.CV 2025-07 conditional novelty 7.0 of 10

    EgoExoBench introduces 7,330 multiple-choice questions across 11 tasks that measure how well multimodal AI models align, associate, and temporally reason across paired first-person and third-person videos; the best mo...

  4. Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding

    cs.CV 2025-06 conditional novelty 7.0 of 10

    MF2 evaluates long-movie understanding by asking models to classify fact/fib claim pairs; the best model trails humans by 23.5 points in pairwise accuracy.

  5. DisTime: Distribution-based Time Representation for Video Large Language Models

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A single learnable time token, decoded into a probability distribution over time bins, improves temporal grounding in Video-LLMs and is trained partly on a new 1.25M-event pseudo-labeled dataset.

  6. PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

    cs.CV 2025-04 conditional novelty 7.0 of 10

    PerceptionLM releases 2.8M human-labeled fine-grained video QA pairs and spatio-temporal captions, plus models and a new benchmark, arguing that human data, not just synthetic data, is needed for detailed video understanding.

  7. TimeThink: Reasoning with Time for Video LLMs

    cs.CV 2026-07 accept novelty 6.0 of 10

    TimeThink adds step-wise temporal process rewards (max IoU of referenced intervals) to GRPO for Video-LLMs, improving grounding and reasoning over outcome-only RL baselines.

  8. VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.

  9. AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs

    cs.CV 2025-06 reject novelty 6.0 of 10

    A clue-grounded audio-visual counting benchmark over 497 long videos and an RL-trained counting model, whose headline result is undermined by training on the DVD-Counting evaluation benchmark.

  10. SCOUT: Self-Checking and Recovery-Aware Tool-Thought Agents for Ultra-Long Egocentric Video Reasoning

    cs.AI 2026-08 conditional novelty 5.0 of 10

    SCOUT, a recovery-aware tool-use agent trained with uncertainty-prioritized RL, reports 47.6% on EgoLifeQA and 49.0% on Ego-R1 Bench, the best published accuracy on those ultra-long egocentric benchmarks.

  11. OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputs

    cs.CV 2025-06 conditional novelty 5.0 of 10

    OmniEval releases a Chinese-English, audio-visual-text benchmark with fine-grained temporal grounding questions, and reports that today's omni-modal models score low and depend mainly on textual cues.

  12. Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A real-time egocentric assistant combines a vision-language model, memory, video retrieval, and video generation to answer questions and show how-to guidance from live wearable camera streams.

  13. A Survey on Video Temporal Grounding with Multimodal Large Language Model

    cs.CV 2025-08 unverdicted novelty 3.0 of 10

    A taxonomized review of video temporal grounding with multimodal large language models, covering model roles, training paradigms, feature processing, benchmarks, and open problems.

  14. Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision

    cs.CV 2025-06 accept novelty 3.0 of 10

    A comprehensive review of cross-view video understanding that uses both first-person and third-person cameras, organized into a three-direction taxonomy with a dataset catalog and future research gaps.

Reference graph

Works this paper leans on

21 extracted references · 21 linked inside Pith · cited by 14 Pith papers

  1. [1]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Tim- othy P

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Tim- othy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michae...

    arXiv

  2. [3]

    Rextime: A benchmark suite for reasoning-across-time in videos

    Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, and Yu-Chiang Frank Wang. Rextime: A benchmark suite for reasoning-across-time in videos. arXiv preprint arXiv:2406.19392, 2024a. Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video und...

    arXiv

  3. [4]

    How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024d. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Da...

    arXiv

  4. [6]

    Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408.14023,

    arXiv

  5. [7]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    11 CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever co...

    arXiv

  6. [8]

    Tvqa: Localized, compositional video question answering

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696,

    arXiv

  7. [10]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023a. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video ...

    arXiv

  8. [11]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, abs/2310.03744,

    arXiv

Show all 21 references
  1. [12]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 36, 2024a. Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context vide...

    arXiv

  2. [13]

    Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103,

    arXiv

  3. [14]

    Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al

    URL https://openai.com/index/hello-gpt-4o/. Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocentric foundation model and downstream adaptation. arXiv preprint arXiv:2406.18070,

    arXiv

  4. [15]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Gold- stein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813,

    arXiv

  5. [16]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    arXiv

  6. [18]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188,

    arXiv

  7. [19]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800,

    arXiv

  8. [20]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    arXiv

  9. [21]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - System Demonstrations, Singapore, December 6-10, ...

    2023 arXiv

  10. [2021]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326,

    arXiv

  11. [2022]

    Internvideo2: Scaling video foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024c. Yu Wang, Zeyuan Zhang, Julian McAuley...

    arXiv

  12. [2023]

    Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308.12966,

    arXiv

  13. [2024]

    Mmbench-video: A long-form multi-shot benchmark for holistic video understanding

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. Mmbench-video: A long-form multi-shot benchmark for holistic video understanding. arXiv preprint arXiv:2406.14515,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.