Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Video-Oasis: Rethinking Evaluation of Video Understanding

T0 review · 2 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most video-understanding benchmarks are solvable without video; after filtering, models barely beat chance.

desk verdict Solid, reproducible audit: ~54% of 14 video benchmarks are shortcuts; residual SOTA is near chance, and the suite is usable. read the letter →

arxiv 2603.29616 v2 pith:5GT7OSKE submitted 2026-03-31 cs.CV

Geuntaek Lim , Sungjune Park , Jaeyun Lee , Inwoong Lee , Taeoh Kim , Dongyoon Wee , Minho Shim , Yukyung Choi This is my paper
classification cs.CV
keywords videounderstandingVideo-LLMbenchmarkauditingspatio-temporalreasoningshortcutdetectiontemporaldependencymultimodalevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-Oasis is a diagnostic suite that audits existing video-QA benchmarks by asking whether each sample truly needs visual input and temporal order. Across 14 benchmarks, roughly half the samples can be answered from language priors, audio transcripts, captions, a single middle frame, shuffled frames, or bag-of-frames matching. After those shortcuts are removed and ambiguous cases are checked by humans, the remaining “video-native” questions expose a large gap: leading multimodal models score only a little above random guessing. The same filtered set is then used as a testbed to compare design choices such as temporal grounding, adaptive thinking versus always-on reasoning, and supervised fine-tuning versus reinforcement learning with verifiable rewards. The paper’s practical claim is that rigorous evaluation of video understanding requires this kind of modality- and time-decoupling audit, and that future models and benchmarks should be judged by how they perform once shortcuts are stripped away.

What carries the argument

Video-Oasis: a multi-test diagnostic that decouples visual dependency (blind, audio-transcript, narrative-caption) and temporal dependency (center-frame, frame-shuffling, bag-of-frames CLIP matching), then applies cross-model consensus (default k=3) plus human verification of consistency, redundancy, and sensitivity to label samples as shortcuts or video-native challenges.

What would settle it

Re-run the same diagnostic ablations with a substantially different ensemble of models and independent human raters; if the shortcut fraction collapses far below ~50% or the filtered set no longer drops model accuracy near chance, the central claim fails.

Watch

Extended reading notes

Core claim

Across 14 major video-understanding benchmarks, about 54–55% of samples are solvable without visual input or temporal context under a cross-model consensus of diagnostic ablations. On the remaining video-native challenges, state-of-the-art models achieve overall accuracy only marginally above the random baseline of 25.63%, revealing that reported gains largely track shortcut prevalence rather than genuine spatio-temporal reasoning.

Load-bearing premise

A sample is treated as a non-video shortcut if and only if a consensus of models answers it correctly under at least one deliberately vision- or time-ablated condition, with human majority vote resolving remaining ambiguity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Video-Oasis, a diagnostic suite that audits 14 existing video-understanding benchmarks by decoupling visual and temporal cues (blind, audio, narrative, center-frame, frame-shuffling, bag-of-frames) and applying cross-model consensus plus human verification. It reports that roughly 54% of samples are solvable without full video dependency; after filtering, SOTA Video-LLMs and agentic methods score only marginally above the ~25.6% random baseline on the residual video-native set (Table 7). The filtered set is then used as a testbed to examine temporal grounding, adaptive thinking, and SFT vs RLVR design choices.

Significance. If the central empirical claim holds, the work is a high-value contribution to evaluation methodology in multimodal video understanding. It supplies a concrete, open-source, multi-model audit protocol rather than another leaderboard, quantifies a large shortcut prevalence across a broad suite of benchmarks, and shows that residual performance collapses near chance. Strengths include transparent multi-model consensus, human majority-vote refinement, correlation checks between shortcut and standard settings (Table 6), robustness ablations under alternate diagnostic models (Supplement A.2, ~90% overlap), and released code. The algorithmic ablations further convert the diagnosis into actionable design guidance (temporal grounding, adaptive reasoning depth, reward design).

major comments (2)
  1. The operational definition of a shortcut (a sample answered correctly by a consensus of k models under at least one vision- or time-ablated condition; default k=3 in §3.2–3.3) is free and directly sets the headline 54% figure. Table 3 already shows strong sensitivity (k≥1 ≈93%, k=3 ≈54%). While the paper provides useful supporting checks (unique-test contributions in Table 5, ~76% correlation with standard evaluation in Table 6, 90% overlap under alternate models in Supplement A.2), the manuscript should more explicitly treat k as a free parameter of the audit and report residual SOTA performance (Table 7 style) under at least one alternate consensus threshold so that the near-chance claim is not tied to a single operating point.
  2. Section 4.1 derives five video-native categories via LLM ensemble labeling of surviving QA pairs. The categories are useful for analysis, but the paper does not quantify inter-annotator or inter-LLM agreement beyond the 122 manual cases, nor does it show that the residual near-chance result is robust if the category taxonomy is coarsened or re-clustered. A short stability check would strengthen the claim that the distilled set isolates intrinsic spatio-temporal challenges rather than an artifact of the labeling pipeline.
minor comments (5)
  1. Abstract states 55% while the body and Table S2 consistently report ~54%/53.6%. Align the headline number.
  2. Figure 1 and several later figures are described but the manuscript text alone leaves some qualitative examples hard to parse without the actual images; ensure captions are self-contained.
  3. Table 2 reports diagnostic accuracies without always restating the random baseline per test; adding it in the table header would improve readability.
  4. In §5.1 the oracle grounding experiment is restricted to ImplicitQA and KFS-Bench; a sentence clarifying why these two sources are representative of the full residual set would help.
  5. Minor notation inconsistency: Video-Oasis / V-Oasis / Video Oasis appear interchangeably; pick one form after first definition.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical diagnostic audit of external public benchmarks under an explicit operational definition; no derivation reduces to its own inputs by construction.

full rationale

Video-Oasis is a diagnostic/empirical paper, not a theoretical derivation. The headline quantities (≈54% shortcut prevalence; residual SOTA near the 25.63% chance baseline) are measurements obtained by applying stated ablation tests (Blind, Audio, Narrative, Center-Frame, Frame Shuffling, Bag-of-Frames) plus human majority vote to 14 external public benchmarks, then evaluating independent models on the filtered remainder. Defining a shortcut as “solvable under at least one vision- or time-ablated condition with model consensus k=3” and then counting how many samples meet that definition is ordinary operational measurement, not a self-definitional loop that forces a claimed prediction. There are no fitted parameters re-labeled as predictions, no uniqueness theorems imported from the authors’ prior work, no ansatz smuggled via self-citation, and no renaming of a known closed-form result. Mild reuse of the same filtered set for later algorithmic ablations (temporal grounding, adaptive thinking, SFT vs RLVR) is the intended use of a diagnostic testbed and does not circularly define the headline audit figures. Robustness checks (alternate diagnostic ensembles with 90% overlap; correlation of shortcuts with standard evaluation ≈76%; sensitivity across consensus thresholds) further show the results are not forced by a single unexamined self-reference. Score 0 is therefore the correct finding.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The central claim rests on an operational definition of “video dependency” implemented by a finite set of ablation tests and a consensus rule, plus a human verification step. These are design choices rather than theorems; the free parameters (k, frame counts, model ensemble) directly control which samples survive. No new physical entities are postulated, but the five challenge categories and the Video-Oasis suite itself are invented constructs whose validity is internal to the paper’s procedure.

free parameters (3)
  • consensus threshold k = k=3
    Default k=3 (all three models must solve the ablated question) determines the reported 54% shortcut rate; Table 3 shows the ratio swings from 92.7% (k≥1) to ~55% (k=3).
  • frame sampling budget = 128 frames @ 1 fps
    Uniform 128 frames at 1 fps (and 16-frame chunks for redundancy/narrative tests) is fixed by hand and affects both diagnostic and evaluation scores.
  • BoF top-k = k=32
    k=32 frames retrieved by CLIP cosine similarity for the bag-of-frames test is an arbitrary cutoff that influences which samples are labeled temporal shortcuts.
assumptions (3)
  • ad hoc to paper A question that multiple models answer correctly under vision-ablated or time-ablated inputs does not require genuine video understanding.
    Core operational definition introduced in §3.1; not a standard mathematical fact and not independently proven outside the diagnostic suite.
  • domain assumption Human majority vote on consistency, redundancy, and sensitivity cases correctly resolves annotation ambiguity.
    §3.1 and Table 4; three annotators exclude 410 samples and restore 804; treated as ground truth without inter-annotator reliability statistics.
  • domain assumption Uniform frame sampling and fixed-interval captions are sufficient proxies for testing visual and temporal necessity.
    Used throughout the diagnostic tests; assumes that more sophisticated retrieval or summarization would not change the shortcut classification.
invented entities (2)
  • Video-Oasis diagnostic suite
    purpose: Reusable protocol that filters existing benchmarks into video-native challenges.
    The suite itself is the paper’s primary artifact; its validity is demonstrated only by internal consistency and correlation tables, not by an external gold standard of video understanding.
  • Five video-native challenge categories (Fine-Grained Perception, Spatial World, Temporal Dynamics, Causality, Global Narrative)
    purpose: Taxonomy of the filtered remainder used for model analysis.
    Derived bottom-up via LLM ensemble + author inspection (§4.1); useful organizing device but not independently validated against human cognitive categories.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video-Oasis: Rethinking Evaluation of Video Understanding." pith.science (2026). https://pith.science/paper/5GT7OSKE

@misc{pith2026260329616,
  author       = {Pith},
  title        = {Pith review of: Video-Oasis: Rethinking Evaluation of Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5GT7OSKE}},
  note         = {Machine review of arXiv:2603.29616}
}
read the original abstract

The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    Introduces V-RAGBench benchmark and CARVE method that selects per-chunk retrieval configurations via parallel retrievers and adaptive reranking, outperforming eight VideoRAG baselines.

Reference graph

Works this paper leans on

58 extracted references · 15 linked inside Pith · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2511.21631 (2025) 1, 3, 4, 6, 7, 10, 12, 13, 20, 23, 24, 25, 27, 28, 34

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025) 1, 3, 4, 6, 7, 10, 12, 13, 20, 23, 24, 25, 27, 28, 34

    arXiv 2025

  2. [2]

    arXiv preprint arXiv:2502.13923 (2025) 7, 11, 13, 14, 21, 23, 24, 34

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., et al.: Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) 7, 11, 13, 14, 21, 23, 24, 34

    arXiv 2025

  3. [3]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2026) 4

    Chen, B., Wang, Z., Yue, Z., Yan, K., Yu, C., Huang, Y., Liu, Z., Wen, Y., Chen, X., Liu, Y., et al.: Videochat-m1: Collaborative policy planning for video under- standing via multi-agent reinforcement learning. In: Proceedings of the Computer Vision and Pattern Recognition Conference (2026) 4

    2026

  4. [4]

    In: Proceedings of the International Conference on Computer Vision (2025) 4

    Chen, B., Yue, Z., Chen, S., Wang, Z., Liu, Y., Li, P., Wang, Y.: Lvagent: Long video understanding by multi-round dynamical collaboration of mllm agents. In: Proceedings of the International Conference on Computer Vision (2025) 4

    2025

  5. [5]

    In: Proceedings of the Neural Information Processing Systems (2025) 1, 4, 6, 7, 10, 12, 13, 14, 20, 23, 24, 25, 27, 28, 34

    Chen, G., Li, Z., Wang, S., Jiang, J., Liu, Y., Lu, L., Huang, D.A., Byeon, W., Le, M., Ehrlich, M., Lu, T., Wang, L., Catanzaro, B., Kautz, J., Tao, A., Yu, Z., Liu, G.: Eagle 2.5: Boosting long-context post-training for frontier vision-language models. In: Proceedings of the Neural Information Processing Systems (2025) 1, 4, 6, 7, 10, 12, 13, 14, 20, 23...

    2025

  6. [6]

    In: Proceedings of the Neural Information Processing Systems (2025) 3, 8, 11, 34

    Chen, Y., Huang, W., Shi, B., Hu, Q., Ye, H., Zhu, L., Liu, Z., Molchanov, P., Kautz, J., QI, X., Liu, S., Yin, H., Lu, Y., Han, S.: Scaling RL to long videos. In: Proceedings of the Neural Information Processing Systems (2025) 3, 8, 11, 34

    2025

  7. [7]

    Cheng, J., Ge, Y., Wang, T., Ge, Y., Liao, J., Shan, Y.: Video-holmes: Can mllm think like holmes for complex video reasoning? arXiv preprint arXiv:2505.21374 (2025) 2, 7, 22, 23, 24, 25, 29

    arXiv 2025

  8. [8]

    In: Proceedings of the International Conference on Machine Learning (2025) 14

    Chu, T., Zhai, Y., Yang, J., Tong, S., Xie, S., Schuurmans, D., Le, Q.V., Levine, S., Ma, Y.: SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In: Proceedings of the International Conference on Machine Learning (2025) 14

    2025

Show all 58 references
  1. [9]

    In: Proceedings of the British Machine Vision Conference (2025) 1, 2, 7, 22, 23, 24, 25, 29

    Cores, D., Dorkenwald, M., Mucientes, M., Snoek, C.G., Asano, Y.M.: Lost in time: A new temporal benchmark for videollms. In: Proceedings of the British Machine Vision Conference (2025) 1, 2, 7, 22, 23, 24, 25, 29

    2025

  2. [10]

    In: Proceedings of the Neural In- formation Processing Systems (2025) 4, 8, 10, 11, 34

    Fan, S., Cui, J., Guo, M.H., Yang, S.: Tool-augmented spatiotemporal reasoning for streamlining video question answering task. In: Proceedings of the Neural In- formation Processing Systems (2025) 4, 8, 10, 11, 34

    2025

  3. [11]

    In: Proceedings of the Neural Information Processing Systems (2025) 6, 10, 11, 14, 21, 28, 34 16 G

    Feng, K., Gong, K., Li, B., Guo, Z., Wang, Y., Peng, T., Wu, J., Zhang, X., Wang, B., Yue, X.: Video-r1: Reinforcing video reasoning in MLLMs. In: Proceedings of the Neural Information Processing Systems (2025) 6, 10, 11, 14, 21, 28, 34 16 G. Lim et al

    2025

  4. [12]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 1, 2, 4, 7, 22, 23, 24, 25, 29, 34

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al.: Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In: Proceedings of the Computer Vision and Pattern Recognition Conference ...

    2025

  5. [13]

    arXiv preprint arXiv:2507.06261 (2025) 9, 10, 11

    GoogleDeepMind,Comanici,G.,Bieber,E.,etal.:Gemini2.5:Pushingthefrontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025) 9, 10, 11

    2025 arXiv

  6. [14]

    arXiv preprint arXiv:2407.21783 (2024) 21

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Let- man, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024) 21

    2024 arXiv

  7. [15]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 4

    Han, S., Huang, W., Shi, H., Zhuo, L., Su, X., Zhang, S., Zhou, X., Qi, X., Liao, Y., Liu, S.: Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. In: Proceedings of the Computer Vision and Pattern Recognition Confer...

    2025

  8. [16]

    arXiv preprint arXiv:2310.06825 (2023) 21

    Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M.A., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7b. arXiv preprint arXiv:23...

    2023 arXiv

  9. [17]

    In: arXiv preprint arXiv:2305.06355 (2023) 4

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. In: arXiv preprint arXiv:2305.06355 (2023) 4

    2023 arXiv

  10. [18]

    In:ProceedingsoftheComputerVisionandPatternRecognitionConference(2024) 1, 2, 4, 7, 22, 23, 24, 25, 29

    Li,K.,Wang,Y., He,Y.,Li, Y.,Wang,Y., Liu,Y., Wang,Z., Xu,J., Chen,G., Luo, P.,etal.:Mvbench:Acomprehensivemulti-modalvideounderstandingbenchmark. In:ProceedingsoftheComputerVisionandPatternRecognitionConference(2024) 1, 2, 4, 7, 22, 23, 24, 25, 29

    2024

  11. [19]

    In: Proceedings of the Winter Conference on Applications of Computer Vision (2026) 12

    Li, Z., Ishida, K., Yamazaki, S., Ji, X., Liu, J.: Kfs-bench: Comprehensive evalu- ation of key frame sampling in long video understanding. In: Proceedings of the Winter Conference on Applications of Computer Vision (2026) 12

    2026

  12. [20]

    In: Proceedings of the Empirical Methods in Natural Language Processing (2024) 1, 4

    Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by alignment before projection. In: Proceedings of the Empirical Methods in Natural Language Processing (2024) 1, 4

    2024

  13. [21]

    arXiv preprint arXiv:2601.05175 (2026) 3, 4, 6, 7, 10, 13, 14, 20, 24, 25, 27, 28, 34

    Liu, S., Zhuge, M., Zhao, C., Chen, J., Wu, L., Liu, Z., Zhu, C., Cai, Z., Zhou, C., Liu, H., et al.: Videoauto-r1: Video auto reasoning via thinking once, answering twice. arXiv preprint arXiv:2601.05175 (2026) 3, 4, 6, 7, 10, 13, 14, 20, 24, 25, 27, 28, 34

    2026

  14. [22]

    arXiv preprint arXiv:2505.14640 (2025) 2

    Ma, W., Ren, W., Jia, Y., Li, Z., Nie, P., Zhang, G., Chen, W.: Videoeval- pro: Robust and realistic long video understanding evaluation. arXiv preprint arXiv:2505.14640 (2025) 2

    2025 arXiv

  15. [23]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 5

    Ma,Z.,Gou,C.,Shi,H.,Sun,B.,Li,S.,Rezatofighi,H.,Cai,J.:Drvideo:Document retrieval based long video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 5

    2025

  16. [24]

    In: Proceedings of the Neural Information Processing Systems (2023) 1, 2, 4, 7, 22, 23, 24, 25, 29

    Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A diagnostic benchmark for very long-form video language understanding. In: Proceedings of the Neural Information Processing Systems (2023) 1, 2, 4, 7, 22, 23, 24, 25, 29

    2023

  17. [25]

    In: Proceedings of the International Conference on Computer Vision (2025) 2, 7, 22, 23, 24, 25, 29

    Nagrani, A., Menon, S., Iscen, A., Buch, S., Mehran, R., Jha, N., Hauth, A., Zhu, Y., Vondrick, C., Sirotenko, M., et al.: Minerva: Evaluating complex video reasoning. In: Proceedings of the International Conference on Computer Vision (2025) 2, 7, 22, 23, 24, 25, 29

    2025

  18. [26]

    arXiv preprint arXiv:2410.21276 (2024) 9, 10 Video-Oasis: Rethinking Evaluation of Video Understanding 17

    OpenAI: GPT-4o system card. arXiv preprint arXiv:2410.21276 (2024) 9, 10 Video-Oasis: Rethinking Evaluation of Video Understanding 17

    2024 arXiv

  19. [27]

    OpenAI: OpenAI o3 and o4-mini system card.��������������������������� ������������������������������������������������������������������� (2025) 9

    2025

  20. [28]

    arXiv preprint arXiv:2601.03267 (2026) 9, 11, 34

    OpenAI: GPT-5 system card. arXiv preprint arXiv:2601.03267 (2026) 9, 11, 34

    2026 arXiv

  21. [29]

    In: Proceedings of the Computer Vision and Pattern Recognition Con- ference (2025) 2, 3, 4

    Plizzari, C., Tonioni, A., Xian, Y., Kulshrestha, A., Tombari, F.: Omnia de egotempo: Benchmarking temporal understanding of multi-modal llms in egocen- tric videos. In: Proceedings of the Computer Vision and Pattern Recognition Con- ference (2025) 2, 3, 4

    2025

  22. [30]

    arXiv preprint arXiv:2504.07956 (2025) 2, 7, 22, 23, 24, 25, 29

    Qi, Y., Zhao, Y., Zeng, Y., Bao, X., Huang, W., Chen, L., Chen, Z., Zhao, J., Qi, Z., Zhao, F.: Vcr-bench: A comprehensive evaluation framework for video chain- of-thought reasoning. arXiv preprint arXiv:2504.07956 (2025) 2, 7, 22, 23, 24, 25, 29

    2025 arXiv

  23. [31]

    In: Proceedings of the International Conference on Machine learning (2021) 6, 7, 25, 27

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Proceedings of the International Conference on Machine learning (2021) 6, 7, 25, 27

    2021

  24. [32]

    In: Proceedings of the 40th International Conference on Machine Learning (2023) 26

    Radford, A., Kim, J.W., Xu, T., Brockman, G., Mcleavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. In: Proceedings of the 40th International Conference on Machine Learning (2023) 26

    2023

  25. [33]

    Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., Liu, Z., Xu, H., Kim, H.J., Soran, B., Krishnamoorthi, R., Elhoseiny, M., Chandra, V.: LongVU: Spatiotemporal adaptive compression for longvideo-languageunderstanding.In:P...

    2025

  26. [34]

    In: Proceedings of the Neural Information Processing Systems (2025) 4

    Shen, X., Zhang, W., Chen, J., Elhoseiny, M.: Vgent: Graph-based retrieval- reasoning-augmented generation for long video understanding. In: Proceedings of the Neural Information Processing Systems (2025) 4

    2025

  27. [35]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 4

    Shu, Y., Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang, Z., Huang, T., Zhao, B.: Video-xl: Extra-long vision language model for hour-scale video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 4

    2025

  28. [36]

    arXiv preprint arXiv:2303.15389 (2023) 6, 7, 25, 27

    Sun, Q., Fang, Y., Wu, L., Wang, X., Cao, Y.: Eva-clip: Improved training tech- niques for clip at scale. arXiv preprint arXiv:2303.15389 (2023) 6, 7, 25, 27

    2023 arXiv

  29. [37]

    arXiv preprint arXiv:2506.21742 (2025) 2, 7, 12, 22, 23, 24, 25, 29

    Swetha, S., Gupta, R., Kulkarni, P.P., Shatwell, D.G., Santiago, J.A.C., Siddiqui, N., Fioresi, J., Shah, M.: Implicitqa: Going beyond frames towards implicit video reasoning. arXiv preprint arXiv:2506.21742 (2025) 2, 7, 12, 22, 23, 24, 25, 29

    2025

  30. [38]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 12

    Tang, X., Qiu, J., Xie, L., Tian, Y., Jiao, J., Ye, Q.: Adaptive keyframe sampling for long video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 12

    2025

  31. [39]

    In: Proceedings of the International Conference on Computer Vision (2025) 2, 7, 22, 23, 24, 25, 29

    Wang,W.,He,Z.,Hong,W.,Cheng,Y.,Zhang,X.,Qi,J.,Ding,M.,Gu,X.,Huang, S., Xu, B., et al.: Lvbench: An extreme long video understanding benchmark. In: Proceedings of the International Conference on Computer Vision (2025) 2, 7, 22, 23, 24, 25, 29

    2025

  32. [40]

    arXiv preprint arXiv:2508.18265 (2025) 3, 6, 8, 21, 28, 34

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., et al.: Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025) 3, 6, 8, 21, 28, 34

    2025 arXiv

  33. [41]

    In: Proceedings of the European Conference on Computer Vision

    Wang, Y., Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang, Z., Shi, Y., et al.: Internvideo2: Scaling foundation models for multimodal video understanding. In: Proceedings of the European Conference on Computer Vision. Springer (2024) 1, 4 18 G. Lim et al

    2024

  34. [42]

    In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence

    Wang, Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F., Bertasius, G., Bansal, M.: Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence. pp. 3272–3283 (2025) 4, 5, 10, 11, 34

    2025

  35. [43]

    In: Proceedings of the Neural Information Processing Systems (2024) 1, 4, 7, 22, 23, 24, 25, 29, 34

    Wu, H., Li, D., Chen, B., Li, J.: Longvideobench: A benchmark for long-context in- terleaved video-language understanding. In: Proceedings of the Neural Information Processing Systems (2024) 1, 4, 7, 22, 23, 24, 25, 29, 34

    2024

  36. [44]

    In: Proceedings of the International Conference on Learning Representations (2026) 5, 26

    Xu, Y., Li, X., Yang, Y., Meng, D., Huang, R., Wang, L.: Carebench: A fine-grained benchmark for video captioning and retrieval. In: Proceedings of the International Conference on Learning Representations (2026) 5, 26

    2026

  37. [45]

    In: Proceedings of the Neural Information Processing Systems (2025) 4

    Xue, Z., Zhang, J., Xie, X., yuxuan cai, Liu, Y., Li, X., Tao, D.: AdavideoRAG: Omni-contextual adaptive retrieval-augmented efficient long video understanding. In: Proceedings of the Neural Information Processing Systems (2025) 4

    2025

  38. [46]

    In: Proceedings of the Neural Information Processing Systems (2025) 2, 7, 22, 23, 24, 25, 29

    Xun, S., Tao, S., Li, J., Shi, Y., Lin, Z., Zhu, Z., Yan, Y., Li, H., Zhang, L., Wang, S., Liu, Y., Zhang, H., Ma, Y., Hu, X.: RTV-bench: Benchmarking MLLM continuous perception, understanding and reasoning through real-time video. In: Proceedings of the Neural Information Pro...

    2025

  39. [47]

    arXiv preprint arXiv:2505.09388 (2025) 21

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) 21

    2025 arXiv

  40. [48]

    In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference (2025) 2, 4, 7, 22, 23, 24, 25, 29

    Yang, J., Yang, S., Gupta, A.W., Han, R., Fei-Fei, L., Xie, S.: Thinking in space: How multimodal large language models see, remember, and recall spaces. In: Pro- ceedings of the Computer Vision and Pattern Recognition Conference (2025) 2, 4, 7, 22, 23, 24, 25, 29

    2025

  41. [49]

    In: Proceedings of the International Conference on Learning Representations (2026) 2, 3, 4

    Yang, S., Yang, J., Huang, P., II, E.L.B., Yang, Z., Yu, Y., Tong, S., Zheng, Z., Xu, Y., Wang, M., Fergus, R., LeCun, Y., Fei-Fei, L., Xie, S.: Towards spatial supersensing in video. In: Proceedings of the International Conference on Learning Representations (2026) 2, 3, 4

    2026

  42. [50]

    Proceedings of the Computer Vision and Pattern Recognition Conference (2026) 4

    Yeo, W., Kim, K., Yoon, J., Hwang, S.J.: Worldmm: Dynamic multimodal memory agent for long video reasoning. Proceedings of the Computer Vision and Pattern Recognition Conference (2026) 4

    2026

  43. [51]

    arXiv preprint arXiv:2601.13719 (2026) 4

    Yin, X., Peng, X., Li, X., Xiong, Z., Lu, Y.: Hierarchical long video under- standing with audiovisual entity cohesion and agentic search. arXiv preprint arXiv:2601.13719 (2026) 4

    2026

  44. [52]

    Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., Huang, G.: Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In: Proceedings of the Neural Information Processing Systems (2025) 14

    2025

  45. [53]

    In: Proceedings of the European Conference on Computer Vision (2024) 6, 7, 25, 27

    Zhang, B., Zhang, P., Dong, X., Zang, Y., Wang, J.: Long-clip: Unlocking the long- text capability of clip. In: Proceedings of the European Conference on Computer Vision (2024) 6, 7, 25, 27

    2024

  46. [54]

    In: Proceedings of the Neural Information Processing Systems (2025) 4

    Zhang, X., Jia, Z., Guo, Z., Li, J., Li, B., Li, H., Lu, Y.: Deep video discovery: Agentic search with tool use for long-form video understanding. In: Proceedings of the Neural Information Processing Systems (2025) 4

    2025

  47. [55]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 1, 2, 4, 7, 22, 23, 24, 25, 29

    Zhou, J., Shu, Y., Zhao, B., Wu, B., Liang, Z., Xiao, S., Qin, M., Yang, X., Xiong, Y., Zhang, B., et al.: Mlvu: Benchmarking multi-task long video understanding. In: Proceedings of the Computer Vision and Pattern Recognition Conference (2025) 1, 2, 4, 7, 22, 23, 24, 25, 29

    2025

  48. [56]

    arXiv preprint arXiv:2504.10479 (2025) 34 Video-Oasis: Rethinking Evaluation of Video Understanding 19

    Zhu, J., et al.: Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 (2025) 34 Video-Oasis: Rethinking Evaluation of Video Understanding 19

    2025 arXiv

  49. [57]

    In: Proceedings of the International Conference on Learning Representations (2026) 2, 7, 22, 23, 24, 25, 29

    Zhu, K., Jin, Z., Yuan, H., Li, J., Tu, S., Cao, P., Chen, Y., Liu, K., Zhao, J.: MMR- v: What’s left unsaid? a benchmark for multimodal deep reasoning in videos. In: Proceedings of the International Conference on Learning Representations (2026) 2, 7, 22, 23, 24, 25, 29

    2026

  50. [58]

    Zohar, O., Wang, X., Dubois, Y., Mehta, N., Xiao, T., Hansen-Estruch, P., Yu, L., Wang, X., Juefei-Xu, F., Zhang, N., et al.: Apollo: An exploration of video understanding in large multimodal models. In: Proceedings of the Computer Vision and Pattern Recognition Conference (20...

    2025

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.