Pith. sign in

REVIEW 5 major objections 4 minor 69 references

Video-LLM benchmark accuracy and visual grounding are dissociable: a model that wins on original video can be statistically indistinguishable from rivals once the video is replaced by a black screen.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:33 UTC pith:2RNBAQV3

load-bearing objection Worth engaging: VDG is a useful per-question audit and the accuracy/grounding dissociation on MVBench is convincing, but 'black screen = no vision' needs validation. the 5 major comments →

arxiv 2607.13305 v1 pith:2RNBAQV3 submitted 2026-07-14 cs.CV cs.AIcs.MM

Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

Jae Joong Lee This is my paper
classification cs.CV cs.AIcs.MM
keywords visual dependency gapvideo LLM evaluationblack-screen baselinelanguage priorsbenchmark validitytemporal reasoningframe diversityMcNemar test
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a model's score on a video QA benchmark tells you almost nothing about whether it actually uses the video. It introduces the Visual Dependency Gap (VDG), the per-question difference in correctness between seeing the original video and seeing a solid black screen. Across twenty models, the paper shows accuracy and visual dependency are statistically separable: one model can beat another on original video yet tie on black screens. It also finds that temporal order contributes near-zero accuracy—models gain almost nothing from restoring the order of shuffled frames—while seeing more distinct frames supplies most of the visual benefit. If correct, benchmark leaders should be re-read as language-prior exploiters until a VDG-style audit says otherwise.

Core claim

The paper's central claim is that benchmark accuracy and visual grounding are independent axes, and that the Visual Dependency Gap (VDG) — the per-question difference in correctness between original-video and black-screen conditions — exposes this dissociation. Concretely, on MVBench, InternVL2-8B reliably outperforms Qwen2-VL-7B on original video (p = 0.0003) yet the same pair is statistically indistinguishable on a black screen (p = 0.53). The task-type VDG spectrum is stable across all twenty models: Attribute Perception is the most visually grounded (≈0.41), while Temporal Reasoning is nearly indistinguishable from language-only performance (≈0.06). A four-condition diagnostic ladder — b

What carries the argument

The Visual Dependency Gap (VDG) is a per-question diagnostic defined as VDG(q) = 1[correct at original] − 1[correct at black screen], ranging over {−1, 0, +1}. Aggregated, it reduces to the accuracy difference between original and black-screen conditions, but the per-question form enables four-category decomposition, McNemar dissociation testing, and the four-condition ladder (black screen, single frame, shuffled frames, original video) that splits VDG into Δspatial, Δdiversity, and Δtemporal components. This ladder is the key instrument: it isolates how much of a model's visual benefit comes from static content, from seeing multiple distinct frames, and from temporal ordering.

Load-bearing premise

The black-screen condition is a faithful 'no-video' baseline: that a solid black frame behaves as absence of visual content rather than as a distinctive input with its own learned priors, so that VDG conflates language-prior exploitation with black-screen-specific artifacts only if that assumption fails.

What would settle it

Run the VDG protocol with a control condition that replaces video frames with static random noise or a fixed unrelated natural image. If a high-VDG model (e.g., Gemini 2.5 Flash Lite, VDG 0.315) shows a substantially different gap under the control than under black screen, the black-screen baseline is not a neutral no-video condition and the dissociation claim is weakened. More directly: find any question where a model answers correctly on original video but incorrectly when a single random frame is shown—this would violate the ladder's claim of near-zero temporal contribution; or construct an

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark ranks on MVBench and Video-MME should not be interpreted as visual grounding: accuracy and VDG are weakly correlated, with Nemotron Nano 12B (44% accuracy, 0.025 VDG) and Qwen2.5-VL-7B (63% accuracy, 0.297 VDG) differing by 12× in VDG despite only 1.4× in accuracy.
  • Temporal Reasoning questions on current benchmarks do not discriminate temporal understanding from language-prior exploitation: restoring temporal order from shuffled frames yields ≤ +0.078 accuracy across all sixteen models, so a benchmark that requires genuine temporal discrimination is needed.
  • Compression 'robustness' is largely a benchmark-composition artifact: flat CRF accuracy curves conceal bidirectional flips, and visually grounded questions are 3.76× more sensitive to compression, so robustness claims need to be stratified by VDG.
  • The VDG tier taxonomy transfers across benchmarks: semantic labels derived from Video-MME predicted 7/7 MVBench task-type tiers, and the pattern generalized to EgoSchema with zero video-source overlap, suggesting VDG is a property of question semantics rather than of a specific benchmark.
  • Generational progress is not monotonic: Qwen3-VL regresses in VDG relative to Qwen2.5-VL at every size class, due to a near-zero Δdiversity, a regression entirely invisible to accuracy scores.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The black-screen baseline may conflate 'no visual information' with 'a distinctive artificial input.' Models trained on black frames or with learned priors about uniform inputs could respond differently than they would to genuinely absent video; replacing black with static noise or a fixed unrelated image would strengthen the no-video interpretation.
  • The dissociation result suggests a practical audit protocol: report VDG alongside accuracy on every video benchmark, flag task types with VDG < 0.10 for review, and target VDG > 0.30 when writing new questions. The paper offers these as first-iteration heuristics, but they could be operationalized as acceptance criteria.
  • If temporal order contributes near-zero across current models, then 'video understanding' claims based on these benchmarks may actually measure static image understanding plus language priors; this could reorient evaluation toward tasks like TempCompass that require true temporal discrimination.
  • A direct falsifier of the near-zero temporal claim: construct videos where shuffling frames changes the correct answer (e.g., ordering questions), and test whether any current model scores above chance on them. The paper does not run this experiment, but its ladder logic implies such questions would be failed by all sixteen models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces the Visual Dependency Gap (VDG), the per-question difference in correctness between original-video and black-screen conditions, and applies it to 20 video LLMs across three benchmarks (Video-MME, MVBench, EgoSchema). The central claims are: (i) accuracy and visual grounding dissociate, demonstrated by McNemar tests on MVBench where InternVL2-8B outperforms Qwen2-VL-7B on original video (p=0.0003) but not on black screens (p=0.53); (ii) task-type VDG rankings are stable across models and transfer across benchmarks; (iii) a four-condition ladder (black, single-frame, shuffled, original) shows that frame diversity, not temporal order, drives visual benefit; (iv) H.264 robustness is an artifact of bidirectional answer flips. The paper includes extensive reproducibility material: code, per-question annotations, supplementary tables, and code listings.

Significance. If valid, this is a significant benchmarking contribution: it provides a simple, per-question audit statistic and demonstrates that leaderboard accuracy can be misleading about visual grounding. The paper's strengths include a parameter-free VDG definition, held-out tier predictions on MVBench and EgoSchema, detailed bootstrap CIs in several tables, reproducible code with exact inference configurations and core algorithm listings, and a diagnostic ladder that cleanly separates spatial, diversity, and temporal contributions. The main risk is that the black-screen baseline is not an established no-video control; if that assumption fails, the VDG interpretation and the headline dissociation require revision. The temporal-order-zero claim is also not yet supported beyond the low-frame-rate condition.

major comments (5)
  1. [Sec. 3.2, Eq. (1)] VDG treats Acc_black as the language-prior-only baseline. Solid black frames are a specific visual input that models may treat as 'missing/corrupted video' or exploit via learned black-frame priors; VDG would then conflate language-prior exploitation with black-frame-specific artifacts. The validity checks in Sec. 5.2 (architecture-independent floors, at-chance action_prediction) are suggestive but do not compare black-screen to a text-only (no visual tokens) condition. Please add a text-only baseline and, ideally, a non-black information-free visual control (e.g., static noise) for the three primary models. This is load-bearing: every VDG value and the McNemar dissociation inherit this assumption.
  2. [Sec. 4.2, Table 2] The headline dissociation rests on p_orig=0.0003 and p_black=0.53. A non-significant p does not establish equality; the paper provides a TOST only for the Qwen–LLaVA pair, not for the InternVL2–Qwen pair it highlights. Also, no direct test of the VDG difference between models (i.e., a test of (Acc_orig - Acc_black) equality) is reported for MVBench. Please add a TOST or confidence interval for the black-screen difference and a bootstrap/permutation test on per-question VDG for the main pair. Otherwise the claim that the model 'gains nothing over them' on black screens is not supported.
  3. [Sec. 5.5, Table 7; Sec. 5.3] The 'temporal ordering contributes near-zero' claim is computed at 0.25 FPS only. The FPS ablation (Sec. 5.3, Tables 5, S9–S10) measures original-vs-black VDG for Temporal Reasoning, not the shuffled-vs-original Δtemporal of the ladder; it therefore does not rule out frame sparsity for the temporal-order conclusion. Please run the ladder (shuffled vs original) at 8–24 FPS for at least the primary models, and report bootstrap CIs for Δtemporal. Without CIs, values between -0.008 and +0.078 cannot be interpreted as zero. This undermines the abstract's claim that the ablation 'rules out sparse sampling as the cause'.
  4. [Sec. 4.2, Table 2 and Table S11] The b01/b10 entries appear inconsistent with the text's accuracy directions. For Video-MME IV2-vs-Qwen, b01=34, b10=52 (with the table's convention A=IV2, B=Qwen) implies InternVL2 is more often correct when Qwen is wrong, yet the text says Qwen is more accurate (64.4% vs 60.3%). For MVBench IV2-vs-Qwen, b01=58, b10=28 implies Qwen is more accurate, yet the text says InternVL2 is more accurate. The Qwen–LLaVA row is marked 'No' dissociation despite p_orig=0.0007 and p_black=0.76, which by the paper's own definition is a dissociation, and the text says the pair 'differs on both' despite p_black=0.76. These need correction.
  5. [Sec. 4.2, Table 2] The reported p-values are too small relative to the displayed discordant counts. For b01=58, b10=28 (n=86), a two-sided exact McNemar/binomial test gives p≈0.0018 (chi-square with continuity correction also ≈0.0018), not 0.0003; similar discrepancies appear for the other rows (0.0006 vs ≈0.0019; 0.0007 vs ≈0.0042). Please provide the exact test statistic/formula and raw paired counts. Although the dissociation conclusion survives correction at 0.0083, the quantitative values need verification.
minor comments (4)
  1. [Abstract and Sec. 5.3] The phrase 'An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause' is too broad: the main FPS ablation covers only Temporal Reasoning, and only five models are tested at 4–24 FPS. Please qualify the scope.
  2. [Sec. 4.2] 'Paired McNemar testing proves' is too strong; the test provides statistical evidence in a sample. Suggest 'demonstrates' or 'provides evidence for'.
  3. [Table 5] Values such as Qwen2.5-VL-3B (+0.147/+0.212/+0.132) are described as 'essentially flat' without confidence intervals. Add CIs or a formal test of FPS invariance to support the 'flat' description.
  4. [Sec. 5.2] The architecture-independent black-screen floor is demonstrated for object_existence only; the text generalizes to 'specific task types' but should avoid implying all task types are equally validated.

Circularity Check

0 steps flagged

No significant circularity: VDG is an empirical accuracy difference; tier thresholds are calibrated on Video-MME and applied as held-out predictions, not fitted to the target.

full rationale

The paper's derivation chain is empirical throughout. VDG (Eq. 1) is a per-question correctness difference between original-video and black-screen conditions; no parameter is fitted to force the dissociation claim. The McNemar dissociation (Table 2) is a statistical test on measured accuracies, not a construction. Tier boundaries in Sec. 5.1 are defined from Video-MME data and then applied to MVBench/EgoSchema as held-out predictions; the paper explicitly acknowledges that the semantic mapping reflects standard VQA domain knowledge and that the contribution is the quantitative transfer, not the ranking direction. The four-condition ladder (Eqs. 2-4) is an algebraic decomposition VDG = Δspatial + Δdiversity + Δtemporal, but each component is separately measured and the near-zero Δtemporal claim is an empirical finding. The FPS ablation and CRF analyses are likewise empirical. The only self-citation ([24], Lee 2026) appears in Related Work and is not used to justify any load-bearing claim. The black-screen baseline is a construct-validity assumption, not a circular definition; the paper offers internal checks (architecture-independent floors, at-chance action_prediction) which are validity arguments, not circular reductions. No step of the derivation reduces by construction to its inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The paper's central claims rest mainly on the black-screen baseline assumption and the ladder decomposition assumption. The VDG metric itself is an empirical construct, not a fitted model. Two hand-chosen quantities (tier boundaries and sampling rate) shape the generality of the conclusions, though the FPS ablation partially mitigates the sampling-rate concern.

free parameters (2)
  • VDG tier boundaries = 0.10 and 0.30
    Section 5.1: thresholds >0.30, 0.10–0.30, <0.10 are defined from Video-MME VDG values, then applied to MVBench and EgoSchema as 'predictions'. These boundaries are hand-chosen from the calibration benchmark and are not derived from first principles.
  • Baseline sampling rate = 0.25 FPS, max 32 frames
    Section 3.2 / S14: default frame-sampling rate for all conditions. The temporal-order conclusion is conditional on this resolution; the paper partially addresses this with an FPS ablation to 24 FPS on Temporal Reasoning only.
axioms (6)
  • domain assumption Black-screen video is a valid no-visual-information baseline
    Section 3.2 defines VDG as correctness(original) minus correctness(black). If models treat black frames as an informative input (e.g., a learned 'no video' signal with its own priors), VDG conflates language-prior exploitation with black-screen artifacts. The paper validates with architecture-independent floors (Section 5.2) but cannot rule out encoder-specific black-image biases.
  • domain assumption The four-condition ladder isolates spatial, diversity, and temporal information additively
    Section 5.5 Eqs. (2)–(4) assume single-frame vs black measures only static spatial information, shuffled vs single only frame diversity, and original vs shuffled only temporal order. Shuffling may also disrupt position embeddings or continuity cues beyond temporal order, and single-frame selection is random.
  • domain assumption Semantic task-type mapping from Video-MME to MVBench and EgoSchema is valid
    Section 5.1 groups task types into perceptual_physical, action_comprehension, and temporal_linguistic by hand; the MVBench assignment is made 'based solely on semantic label'. If the mapping is wrong, the 7/7 'classification' is not meaningful.
  • standard math Greedy decoding with temperature 0 yields deterministic outputs
    S14: all open-weight models use greedy decoding; therefore reported flips under compression and ladder conditions are treated as systematic, not stochastic. API models use default parameters and are assumed deterministic, which may not hold.
  • standard math McNemar test assumptions (paired binary outcomes, discordant pairs) are satisfied
    Section 3.3 applies McNemar to per-question correctness; this is appropriate for paired matched samples, though the paper notes power limitations on the Video-MME original arm (Section 4.2).
  • domain assumption The 600-question Video-MME subset is representative of the full benchmark
    Section 3.1: 100 questions per task type are sampled uniformly, and this subset is used for all conditions and all twenty models. Claims about Video-MME's VDG spectrum assume this subset is unbiased.

pith-pipeline@v1.3.0-alltime-deepseek · 26726 in / 12050 out tokens · 120042 ms · 2026-08-02T05:33:02.348082+00:00 · methodology

0 comments
read the original abstract

Benchmark accuracy in video large language models (LLMs) is often treated as evidence of visual understanding. We audit this assumption across twenty models spanning 2-78B parameters and ten architecture families. We introduce the Visual Dependency Gap (VDG), the difference in per-question correctness between original-video and black-screen conditions. Paired McNemar tests on MVBench show that accuracy and visual dependency are separable: models differ on original video (p = 0.0003) but not on black screens (p = 0.53). Across models, task-type rankings are stable: Attribute Perception is strongly visual, whereas Temporal Reasoning approaches the language-only baseline. A diagnostic ladder from black screen to single frame, shuffled frames, and original video reveals that frame diversity supplies most of the visual benefit, while temporal order contributes near-zero accuracy across sixteen open-weight models. An ablation from 0.5 to 24 FPS rules out sparse sampling as the cause. H.264 experiments further show that stable aggregate accuracy conceals bidirectional question-level answer flips. The diagnostic also generalizes to four API-accessed models, whose VDG values range from 0.025 to 0.315. These results motivate VDG as a standard audit for whether video benchmarks measure visually grounded capability. Code is available at https://github.com/JaeLee18/accuracy-without-grounding.

Figures

Figures reproduced from arXiv: 2607.13305 by Jae Joong Lee.

Figure 1
Figure 1. Figure 1: Each question is answered twice: once with the original video and once with a black screen. The per-question Visual [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visual Dependency Gap (VDG) by task type on [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: VDG vs. model scale across four families. Solid [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 1 canonical work pages

  1. [1]

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering. InCVPR

    2018

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. InICCV

    2015

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2.5-VL Technical Report.arXiv preprint arXiv:2502.13923(2025). 8 Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks

    Pith/arXiv arXiv 2025

  4. [4]

    Bowman and George E

    Samuel R. Bowman and George E. Dahl. 2021. What Will it Take to Fix Bench- marking in Natural Language Understanding?. InNAACL-HLT

    2021

  5. [5]

    Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. 2022. Revisiting the “Video” in Video-Language Understand- ing. InCVPR

    2022

  6. [6]

    Remi Cadène, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. 2019. RUBi: Reducing Unimodal Biases for Visual Question Answering. InNeurIPS

    2019

  7. [7]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. 2024. Are We on the Right Way for Evaluating Large Vision-Language Models?. InNeurIPS

    2024

  8. [8]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jin- guo Zhu, Shenglong Ye, Hao Tian, et al. 2024. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. arXiv preprint arXiv:2412.05271(2024)

    Pith/arXiv arXiv 2024

  9. [9]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziqi Luo, Deli Zhao, and Lidong Bing. 2024. Vide- oLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs.arXiv preprint arXiv:2406.07476(2024)

    Pith/arXiv arXiv 2024

  10. [10]

    Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees G. M. Snoek, and Yuki M. Asano. 2024. TVBench: Redesigning Video-Language Evaluation.arXiv preprint arXiv:2410.07752(2024). BMVC 2025

    Pith/arXiv arXiv 2024

  11. [11]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In NeurIPS

    2023

  12. [12]

    Corentin Dancette, Remi Cadène, Damien Teney, and Matthieu Cord. 2021. Be- yond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering. InICCV

    2021

  13. [13]

    Samuel Dodge and Lina Karam. 2016. Understanding How Image Quality Affects Deep Neural Networks. InQoMEX

    2016

  14. [14]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. InICLR

    2021

  15. [15]

    Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding. InNeurIPS

    2024

  16. [16]

    Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021. Vision-and- Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers. InEMNLP

    2021

  17. [17]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models.arXiv preprint arXiv:2306.13394(2023)

    Pith/arXiv arXiv 2023

  18. [18]

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2025. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. InCVPR

    2025

  19. [19]

    Zemel, Wieland Brendel, Matthias Bethge, and Felix A

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut Learning in Deep Neural Networks.Nature Machine Intelligence2 (2020), 665– 673

    2020

  20. [20]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh

  21. [21]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. 2022. Ego4D: Around the World in 3,000 Hours of Egocentric Video. In CVPR

    2022

  22. [22]

    Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation Artifacts in Natural Language Inference Data. InNAACL-HLT

    2018

  23. [23]

    Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. InICLR

    2019

  24. [24]

    Jae Joong Lee. 2026. Language-guided invariance probing of vision–language models.Pattern Recognition Letters202 (2026), 108–113. doi:10.1016/j.patrec.2026. 02.012

    work page doi:10.1016/j.patrec.2026 2026

  25. [25]

    Jie Lei, Tamara L Berg, and Mohit Bansal. 2023. Revealing Single Frame Bias for Video-and-Language Learning. InACL

    2023

  26. [26]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. SEED-Bench: Benchmarking Multimodal Large Language Models. InCVPR

    2024

  27. [27]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Boot- strapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. InICML

    2023

  28. [28]

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. VideoChat: Chat-Centric Video Understanding. arXiv preprint arXiv:2305.06355(2023)

    Pith/arXiv arXiv 2023

  29. [29]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. 2024. MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. InCVPR

    2024

  30. [30]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. InEMNLP

    2023

  31. [31]

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2024. Video- LLaVA: Learning United Visual Representation by Alignment Before Projection. InEMNLP

    2024

  32. [32]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual In- struction Tuning. InNeurIPS

    2023

  33. [33]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin

  34. [34]

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024. TempCompass: Do Video LLMs Really Understand Videos?. InFindings of ACL

    2024

  35. [35]

    George Lydakis, Alexander Hermans, Ali Athar, Daan de Geus, and Bastian Leibe. 2025. How Important Are Videos for Training Video LLMs?arXiv preprint arXiv:2506.06928(2025)

    Pith/arXiv arXiv 2025

  36. [36]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan

  37. [37]

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. 2023. EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding. InNeurIPS

    2023

  38. [38]

    Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages.Psychometrika12, 2 (1947), 153–157

    1947

  39. [39]

    Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. InACL

  40. [40]

    Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji- Rong Wen. 2021. Counterfactual VQA: A Cause-Effect Look at Language Bias. InCVPR

    2021

  41. [41]

    OpenAI. 2023. GPT-4 Technical Report.arXiv preprint arXiv:2303.08774(2023)

    Pith/arXiv arXiv 2023

  42. [42]

    Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid

  43. [43]

    MoReVQA: Exploring Modular Reasoning Models for Video Question Answering. InCVPR

  44. [44]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InICML

    2021

  45. [45]

    Bender, Amandalynne Paullada, Emily Den- ton, and Alex Hanna

    Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Den- ton, and Alex Hanna. 2021. AI and the Everything in the Whole Wide World Benchmark. InNeurIPS Datasets and Benchmarks

    2021

  46. [46]

    Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens, Larisa Mar- keeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, et al. 2023. Perception Test: A Diagnostic Benchmark for Multimodal Video Models. InNeurIPS

    2023

  47. [47]

    Pouya Pezeshkpour and Estevam Hruschka. 2024. Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions. InFindings of NAACL

    2024

  48. [48]

    David Schlangen. 2021. Targeting the Benchmark: On Methodology in Current Natural Language Processing Research. InACL-IJCNLP

    2021

  49. [49]

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. 2023. Video Understanding with Large Language Models: A Survey.arXiv preprint arXiv:2312.17432(2023)

    arXiv 2023

  50. [50]

    Machel Reid, Nikolay Savinov, Denis Teber, et al. 2024. Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context.arXiv preprint arXiv:2403.05530(2024)

    Pith/arXiv arXiv 2024

  51. [51]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. InACL

    2020

  52. [52]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution.arXiv preprint arXiv:2409.12191(2024)

    Pith/arXiv arXiv 2024

  53. [53]

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. In NeurIPS

    2024

  54. [54]

    Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. 2020. Measuring Robustness to Natural Distribution Shifts in Image Classification. InNeurIPS

    2020

  55. [55]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. InCVPR

    2024

  56. [56]

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video Question Answering via Gradually Refined Atten- tion over Appearance and Motion. InACM Multimedia

    2017

  57. [57]

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng

  58. [58]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions. InCVPR

    2021

  59. [59]

    Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hes- sel, Jingkang Yang, and Ziwei Liu. 2024. FunQA: Towards Surprising Video Comprehension. InECCV. 9 Jae Joong Lee

    2024

  60. [60]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. InCVPR

    2024

  61. [61]

    Mert Yüksekgönül, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It?. InICLR

    2023

  62. [62]

    PLLaVA: Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning.arXiv preprint arXiv:2404.16994(2024)

    Pith/arXiv arXiv 2024

  63. [63]

    An Yang, Anfeng Zhang, Baosong Liu, Beichen Zhang, Binyuan Hui, Bowen Yu, et al. 2025. Qwen3-VL Technical Report.arXiv preprint arXiv:2511.21631(2025)

    Pith/arXiv arXiv 2025

  64. [64]

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering. InAAAI

    2019

  65. [67]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. InICCV

    2023

  66. [68]

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-Video: Video Instruction Tuning With Synthetic Data.arXiv preprint arXiv:2410.02713(2024)

    Pith/arXiv arXiv 2024

  67. [69]

    pure visual core

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large Language Models Are Not Robust Multiple Choice Selectors. InICLR. 10 Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks This supplementary material provides expanded tables, detailed statistical analyses, and methodological details that...

    2024

  68. [2017]

    Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. InCVPR

  69. [2024]

    MMBench: Is Your Multi-modal Model an All-around Player?. InECCV