Pith. sign in

REVIEW 6 cited by

AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.14906 v2 pith:KUJ4ILXM submitted 2023-11-25 cs.CV

Xiuyuan Chen , Yuan Lin , Yuchen Zhang , Weiran Huang This is my paper
classification cs.CV
keywords autoeval-videoaccuracymodelsopen-endedevaluationlargerulesanswering
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We propose a novel and challenging benchmark, AutoEval-Video, to comprehensively evaluate large vision-language models in open-ended video question answering. The comprehensiveness of AutoEval-Video is demonstrated in two aspects: 1) AutoEval-Video constructs open-ended video-questions across 9 skill dimensions, addressing capabilities of perception, comprehension, and generation. 2) AutoEval-Video contains newly collected videos that cover over 40 distinct themes. To efficiently evaluate responses to the open-ended questions, we employ an LLM-based evaluation approach, but instead of merely providing a reference answer, we annotate unique evaluation rules for every single instance (video-question pair). To maximize the robustness of these rules, we develop a novel adversarial annotation mechanism. By using instance-specific rules as prompt, GPT-4, as an automatic evaluator, can achieve a stable evaluation accuracy of around 97.0%, comparable to the 94.9% - 97.5% accuracy of a human evaluator. Furthermore, we assess the performance of eight large vision-language models on AutoEval-Video. Among them, GPT-4V(ision) significantly outperforms other models, achieving an accuracy of 32.2%. However, there is still substantial room for improvement compared to human accuracy of 72.8%. By conducting an extensive case study, we uncover several drawbacks of GPT-4V, such as limited temporal and dynamic comprehension, and overly general responses. Code is available at https://github.com/Xiuyuan-Chen/AutoEval-Video.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AdsQA: Towards Advertisement Video Understanding

    cs.CV 2025-09 conditional novelty 6.0 of 10

    AdsQA adds an ad-video question-answering benchmark and ReAd-R, a GRPO-trained model that beats 7B baselines but not larger closed models.

  2. AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning

    cs.CV 2025-08 reject novelty 6.0 of 10

    A new audio-visual reasoning benchmark and a two-part metric (factual consistency, core inference) claim to expose a gap between answer accuracy and reasoning quality in AV-LLMs.

  3. GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Introduces GLIMPSE, a video-QA benchmark whose questions cannot be answered from single frames; best model GPT-o3 scores 66.43% vs 94.82% human accuracy.

  4. Toward Annotation-Efficient Continuous Emotion Arousal Quantification via Group-Level EEG Dynamic Neural Synchrony

    cs.HC 2026-07 conditional novelty 5.0 of 10

    Group-level EEG dynamic neural synchrony (CorrCA) preferentially tracks the rate of change of continuous arousal and shows valence-dependent structure across four datasets.

  5. Iterative Zoom-In: Temporal Interval Exploration for Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A confidence-guided iterative zoom-in framework improves long-video question answering for multimodal LLMs without any training.

  6. VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A new benchmark, VF-Eval, measures how well multimodal LLMs check, detect, and reason about errors in AI-generated videos, and shows frontier models remain far below human performance.

Pith tools