Pith. sign in

REVIEW 4 major objections 4 minor 77 references

A new benchmark measures hour-long video AI on vision, speech, and ambient audio, and finds top systems answer barely half the rubric-scored questions correctly.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:22 UTC pith:CH7LJ3ZF

load-bearing objection The benchmark idea is genuinely useful, but the abstract's headline numbers directly contradict the main text, and that makes the current version unpublishable. the 4 major comments →

arxiv 2512.16978 v2 pith:CH7LJ3ZF submitted 2025-12-18 cs.CV

A Benchmark for Omni-Modal Reasoning in Long Videos

Mohammed Irfan Kurpath , Jaseel Muhammad Kaithakkodan , Jinxing Zhou , Sahal Shaji Mullappilly , Mohammad Almansoori , Noor Ahsan , Beknur Kalmakhanbet , Sambal Shikhar
show 7 more authors
Rishabh Lalla Jean Lahoud Mariette Awad Fahad Shahbaz Khan Salman Khan Rao Muhammad Anwer Hisham Cholakkal
This is my paper
classification cs.CV
keywords long-video understandingomni-modal benchmarkvideo question answeringrubric-based evaluationagentic tool useaudio-visual-speech reasoningLLM as judgetraining-free agent
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LongShOTBench, a benchmark of 157 long videos averaging 45 minutes, with 3,092 open-ended, intent-driven questions that require integrating vision, speech, and ambient audio. Each question comes with a human-verified reference answer and a weighted rubric, so a model's score can be traced to missed facts, temporal links, modality grounding, or reasoning steps. The paper also presents LongShOTAgent, a training-free agent that preprocesses the whole video, retrieves relevant segments, and refines evidence before answering. On the benchmark, the best evaluated closed-source system scores 52.95%, the proposed agent scores 44.66%, and open-source models stay below 30%. The paper's stated aim is to show that long-form omni-modal video reasoning is far from solved and that modular agentic coordination can meaningfully close part of the gap without additional training.

Core claim

The paper's central claim is that long-video omni-modal reasoning can be measured at 45-minute scale with diagnostic precision: questions are generated from realistic viewing scenarios, answers are grounded in machine-produced visual, speech, and audio captions, and each item is scored against criterion-level rubrics by an LLM judge. It reports that current models leave large gaps: the strongest evaluated system reaches 52.95% overall, LongShOTAgent reaches 44.66% with no task-specific training, and open-source models remain below 30%. The authors treat these results as evidence that jointly processing vision, speech, and ambient audio over extended temporal contexts remains unsolved, and th

What carries the argument

The load-bearing mechanisms are two. First, LongShOTBench's weighted criterion rubrics decompose each answer into verifiable facts, temporal and modality-grounding requirements, and tool-use steps, scored by an LLM judge checking each criterion separately; this converts a single accuracy number into a traceable diagnostic. Second, LongShOTAgent's search-refine-verify loop: an orchestrator LLM builds a multimodal index from the whole video, retrieves relevant segments via similarity search, refines them with higher-quality speech, vision, and audio specialists, and explicitly checks claims before producing an answer. The rubric gives interpretability; the agent turns evidence seeking from a m

Load-bearing premise

The scoring system depends on the machine-generated caption chain—visual captions, speech transcripts, audio tags, and cross-modal summaries—plus human verification being accurate enough to ground the reference answers and rubrics; if that chain misses or biases what actually happens in the video, every benchmark score inherits the error.

What would settle it

On a random sample of LongShOTBench items, have several independent human annotators score the same model answers under the published rubrics and compare their scores with the LLM judge's scores, and also check whether the reference answers contain facts not present in the original raw video; low agreement or frequent unsupported reference facts would overturn the benchmark's interpretability claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If LongShOTBench is sound, current state-of-the-art systems still fail roughly half of the factual and cross-modal criteria on hour-scale videos, leaving substantial headroom.
  • Open-source omni-modal models remain below 30%, indicating that audio and speech integration is a major bottleneck in long-video understanding.
  • The gap between the modular training-free agent and the strong closed-source system is smaller than the gap between open-source monolithic models and that system, supporting agentic orchestration as a viable research direction.
  • Rubric-level scoring makes specific failure modes—temporal localization, audio grounding, tool invocation, cross-modal verification—visible, enabling targeted model improvements rather than chasing a single number.
  • The native-video evaluation protocol allows future models to be compared without injecting evaluator-chosen frame-sampling heuristics, making scores more comparable across systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The abstract's headline figure of 66.64% for LongShOTAgent does not appear in the main results table, which reports 44.66% overall; until the number is reconciled, readers should treat the abstract claim as unsupported.
  • Because the benchmark's ground truth and rubrics are generated from machine-produced captions and then human-verified, the dataset inherits any blind spots of those caption models; a human-agreement study on a subset would quantify this.
  • A natural test would be to compare LLM-judge scores on the same answers with independent human rubric scores; if agreement is low or systematically favors certain phrasings, the interpretability claim weakens.
  • The 'ideal trajectory' multi-turn evaluation isolates each turn's quality but makes the benchmark easier than real conversation, where models must recover from their own prior mistakes; multi-turn scores should be read with this in mind.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. LongShOTBench is a new benchmark for omni-modal understanding of long videos (157 videos, ~45 minutes average, 3,092 QA instances) with open-ended, scenario-driven single- and multi-turn questions and weighted rubric-based scoring. The paper also introduces LongShOTAgent, a training-free agent that orchestrates frozen vision, speech, audio, and tool modules. The main experiments compare Gemini-2.5-Flash, several open-source MLLMs, and LongShOTAgent on the benchmark. The proposed resource and pipeline are potentially useful, but the manuscript contains a direct internal contradiction: the abstract reports LongShOTAgent at 66.64% overall and evaluation of 105 models, while Section 5.2 and Table 3 report LongShOTAgent at 44.66% overall and scores for only seven baselines, with Gemini-2.5-Flash highest. In addition, the LLM-as-judge evaluation chain is not validated against human judgments.

Significance. If the results were internally consistent and the evaluation were properly validated, LongShOTBench would be a valuable resource: it combines hour-scale video, vision/audio/speech modalities, intent-driven open-ended questions, multi-turn dialogues, and rubric-level scoring. The construction pipeline is detailed, the prompts are included in the appendix, and the native-video evaluation protocol plus the ideal-trajectory multi-turn evaluation are thoughtful design choices. LongShOTAgent is a reasonable modular, training-free baseline. However, the central empirical claims are not supported by the paper's own reported tables, and no human-judge agreement or inter-annotator statistics are provided for the rubric-based scoring. The significance of the contribution is therefore contingent on resolving these issues.

major comments (4)
  1. [Abstract vs. §5.2/Table 3] The abstract states that LongShOTAgent is the strongest training-free system, reaching 66.64% overall, and that 105 video-capable models were evaluated. Section 5.2 and Table 3 report LongShOTAgent at 44.66% overall and evaluate seven baselines plus LongShOTAgent; Gemini-2.5-Flash achieves 52.95% and is highest in every duration bucket in Table 4. No reported scoring variant produces 66.64: the category averages for LongShOTAgent are 36.28, 49.45, 45.30, and 47.59, and the agentic score is 38.25. This is a direct internal contradiction in the headline empirical claim, and the abstract cannot be reconciled with the manuscript's own leaderboard.
  2. [§3.1, §3.4, §5.1] The rubric-based ground truth and the evaluation judge are produced by the same model families used to construct the benchmark and to run LongShOTAgent. Visual captions come from Qwen2.5-VL-32B, speech from Whisper-large-v3, audio from Audio-Flamingo-3, and question/answer/rubric generation from Qwen3-30B-A3B; final scoring uses Qwen3-14B as an LLM judge. The paper reports no human-judge agreement, no inter-annotator agreement, and no calibration of the LLM judge against human scores. This is load-bearing because the benchmark's interpretability claim depends on the rubric judge tracking human judgments. The authors should report agreement statistics on a sampled subset and, ideally, score distributions from human judges.
  3. [§5.2] The paper claims to evaluate 105 video-capable models, but the reported experiments contain only seven baseline models plus LongShOTAgent. No appendix table, leaderboard excerpt, or supplementary results for the remaining models is provided. The scale claim in the abstract is therefore unsupported by the manuscript. Either the full results must be included or the claim must be corrected to match the evaluated set.
  4. [Table 3, Agentic Tasks row] Agentic tasks are reported only for Gemini-2.5-Flash and LongShOTAgent; all open-source entries are dash-marked. The text claims that LongShOTAgent 'significantly surpasses open-source models across all types of tasks,' but no open-source model is scored on the agentic task category. Without open-source baselines on agentic tasks, the tool-use advantage of LongShOTAgent is not actually demonstrated in the reported results.
minor comments (4)
  1. [Global] The arXiv title and the full-text title differ ('A Benchmark for Omni-Modal Reasoning in Long Videos' vs. 'A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos'). The abstract also mentions leaderboard URL longshot.cvmbzuai.com while the full text gives a GitHub URL; these should be unified.
  2. [§5.1] The statement that the main leaderboard is restricted to models with fewer than 10 billion parameters is inconsistent with the inclusion of Gemini-2.5-Flash, whose parameter count is not disclosed. The exclusion rationale should be clarified and applied consistently.
  3. [§5.1] Minor typo: 'each models deafult generation configuration' should be 'each model's default generation configuration.'
  4. [§3.5] The human validation section describes ten annotators and a training protocol but gives no quantitative outcome: no counts of discarded samples, no correction rates, and no agreement statistics. Reporting these numbers would strengthen confidence in the dataset.

Circularity Check

0 steps flagged

No circular derivation: benchmark construction is empirical and human-validated; the abstract/table inconsistency is a reporting issue, not a circular step.

full rationale

LongShOTBench is an empirically constructed benchmark and LongShOTAgent is a training-free agentic pipeline; neither makes a first-principles prediction that could reduce to its own inputs by construction. The benchmark's ground truth is generated through multimodal captioning (Qwen2.5-VL, Whisper, Audio-Flamingo) and LLM-based question/answer/rubric generation (Qwen3-30B), followed by human validation (Section 3.1-3.5). The evaluation uses an LLM-as-judge (Qwen3-14B; Section 5.1), and LongShOTAgent orchestrates the same model families (Qwen2.5-VL, Whisper-large-v3, Audio-Flamingo-3, Qwen3-4B; Section 4). This creates a genuine methodological concern about same-family bias: the judge and the agent share model ancestry with the data-generation pipeline, and no inter-annotator or judge-agreement statistics are reported. However, this is not circularity in the sense required by the review protocol: there is no equation or derived quantity that is identical to its inputs by construction, and the claimed human verification provides an independent, if unquantified, grounding step. The paper also contains a serious internal inconsistency: the abstract claims LongShOTAgent is the strongest training-free system at 66.64% overall across 105 models, while Table 3 reports LongShOTAgent at 44.66% and Gemini-2.5-Flash at 52.95% across seven baselines. That is an empirical/reporting contradiction, not a circular derivation. No load-bearing self-citations are present; references to Qwen, Gemini, Whisper, and Audio-Flamingo are to external systems. Therefore, no circular step can be exhibited with the required quote-and-reduction standard, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper's central claims rest on a chain of LLM-generated captions, human verification, LLM-generated rubrics, and LLM judging. None of these steps is externally validated or accompanied by agreement statistics. The only explicit hand-chosen numerical parameters are the rubric weights and the difficulty distribution targets. No new physical or mathematical entities are introduced.

free parameters (2)
  • rubric weights = high=5, medium=3, low=1, penalty=-5
    Hand-chosen in Section 3.4. All scores on LongShOTBench derive directly from these weights, so they are free parameters of the benchmark rather than quantities derived from data.
  • difficulty distribution targets = L1-2 <=15%, L3 25-30%, L4 35-40%, L5 25-30%
    Targeted mix of question difficulties chosen by the authors in Section 3.2. This composition affects the aggregate difficulty and therefore all reported model scores.
axioms (5)
  • domain assumption LLM-generated multimodal captions are accurate enough to serve as the basis for ground-truth QA pairs.
    Section 3.1 builds the benchmark on captions from Qwen2.5-VL, Whisper, and Audio-Flamingo. If these captions contain systematic errors or omissions, the benchmark's ground truth inherits them.
  • domain assumption Human validation sufficiently corrects errors and prevents hallucinated content in the machine-generated dataset.
    Section 3.5 describes a human review process but reports no inter-annotator agreement, no removal counts, and no error rates, so the effectiveness of this correction is unquantified.
  • domain assumption The LLM-as-judge (Qwen3-14B) reliably checks rubric criteria and produces valid scores.
    Section 5.1 uses Qwen3-14B as the evaluator, but no comparison to human judgments or alternative judges is reported. All leaderboard numbers depend on this assumption.
  • domain assumption The selected 157 videos (92 from Video-MME plus 65 YouTube videos) are a representative sample of omni-modal long-form video understanding tasks.
    Section 3.1 defines the corpus. No diversity analysis or selection criteria are given beyond category counts, so the benchmark's generalizability is assumed.
  • domain assumption The 16-tool agentic-task design captures meaningful real-world tool use in video understanding.
    Section A.2 introduces hypothetical tools with specified parameters, but there is no validation that these tools match real-world video-agent applications or that the task ground truth is unambiguous.

pith-pipeline@v1.3.0-alltime-deepseek · 24009 in / 10312 out tokens · 105322 ms · 2026-08-03T15:22:00.477406+00:00 · methodology

0 comments
read the original abstract

Long-form omni-modal video understanding requires integrating vision, speech, and ambient audio with coherent long-context reasoning. Existing video benchmarks often trade off temporal scale, modality coverage, open-ended interaction, and interpretable scoring. To address this gap, we introduce LongShOTBench, a long video understanding benchmark designed around three coupled goals: holistic omni-modal integration, intent-driven open-ended interaction, and rubric-level diagnosis. It builds single- and multi-turn questions from real viewing scenarios, with systematic tasks probing visual, speech, ambient-audio, temporal, and cross-modal reasoning. Each item includes a reference answer and a weighted criterion-level rubric, letting evaluation identify which perceptual facts, temporal links, modality-grounding requirements, and reasoning steps are satisfied or missed. All samples are manually verified to improve grounding, clarity, and rubric reliability. We also introduce LongShOTAgent, a training-free omni-modal evidence-seeking agent coupling full-video preprocessing with targeted retrieval, query-adaptive segment refinement, and explicit claim verification over visual, speech, and non-speech audio evidence. Its iterative search-refine-verify loop exposes intermediate evidence and lets modality-specific specialists re-analyze relevant moments before answering. We evaluate 105 video-capable models spanning open-source omni-modal models, vision-language systems, audio LLMs, agentic pipelines and closed-source APIs. Current MLLMs remain far from saturating LongShOTBench, while our LongShOTAgent is the strongest training-free system, reaching 66.64% overall. By releasing the benchmark, leaderboard, and method, we provide a shared, interpretable testbed for advancing long-form omni-modal video reasoning. Code, data, and the leaderboard are available at https://longshot.cvmbzuai.com/.

Figures

Figures reproduced from arXiv: 2512.16978 by Beknur Kalmakhanbet, Fahad Shahbaz Khan, Hisham Cholakkal, Jaseel Muhammad Kaithakkodan, Jean Lahoud, Jinxing Zhou, Mariette Awad, Mohammad Almansoori, Mohammed Irfan Kurpath, Noor Ahsan, Rao Muhammad Anwer, Rishabh Lalla, Sahal Shaji Mullappilly, Salman Khan, Sambal Shikhar.

Figure 1
Figure 1. Figure 1: Construction pipeline of LongShOTBench. The pipeline begins with raw video data where speech, visuals, and audio cues are extracted. These are passed into multimodal processing to generate segment-wise aligned and fused metadata. Only the distilled information flows to question design, where scenarios and question types are mapped, followed by the generation of questions and conversational answers. Next, v… view at source ↗
Figure 2
Figure 2. Figure 2: LongShOTAgent Pipeline. The orchestrator agent (Qwen3-4B) receives a user query and video input, then calls the Preprocessor to extract multimodal signals, including Whisper-small speech transcription, scene-based frame sampling, SigLIP embeddings, OCR, and audio analysis. These features populate a vector database, which the Search tool queries to retrieve top-k relevant segments via semantic similarity. F… view at source ↗
Figure 4
Figure 4. Figure 4: visualizes detailed video category distribution and [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Evaluation Examples of LongShOTBench. The data samples illustrate how we construct scenario context, model a users thought process, generate diverse questions (single- and multi-turn), and apply criterion-weighted evaluation rubrics for interpretable scoring. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 19 linked inside Pith

  1. [1]

    Ming-omni: A unified multimodal model for perception and generation.arXiv preprint arXiv:2506.09344, 2025

    Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang, Furong Xu, GuangMing Yao, Jun Zhou, Jingdong Chen, Jianxin Sun, Jiajia Liu, Jianjiang Zhu, Jun Peng, Kaixiang Ji, Kaiyou Song, Kaimeng Ren, Li- bin Wang, Lixiang Ru, Lele Xie, Longhua Tan, Lyuxin Xue, Lan Wang, Mochen B...

    Pith/arXiv arXiv 2025

  2. [2]

    Infinibench: A benchmark for large multi-modal models in long-form movies and tv shows.arXiv preprint arXiv:2406.19875, 2025

    Kirolos Ataallah, Eslam Abdelrahman, Mahmoud Ahmed, Chenhui Gou, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. Infinibench: A benchmark for large multi-modal models in long-form movies and tv shows.arXiv preprint arXiv:2406.19875, 2025. 1, 2, 3

    arXiv 2025

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 1, 4, 6, 7, 8

    Pith/arXiv arXiv 2025

  4. [4]

    Valor: Vision-audio- language omni-perception pretraining model and dataset

    Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Wein- ing Wang, Jinhui Tang, and Jing Liu. Valor: Vision-audio- language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345, 2023. 1, 3

    Pith/arXiv arXiv 2023

  5. [5]

    Vast: A vision-audio-subtitle- text omni-modality foundation model and dataset.Advances in Neural Information Processing Systems, 36:72842–72866,

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle- text omni-modality foundation model and dataset.Advances in Neural Information Processing Systems, 36:72842–72866,

  6. [6]

    Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answer- ing

    Xiuyuan Chen, Yuan Lin, Yuchen Zhang, and Weiran Huang. Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answer- ing. InEuropean Conference on Computer Vision, pages 179–195, 2024. 1

    2024

  7. [7]

    Video-holmes: Can mllm think like holmes for complex video reasoning?arXiv preprint arXiv:2505.21374, 2025

    Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can mllm think like holmes for complex video reasoning?arXiv preprint arXiv:2505.21374, 2025. 2

    Pith/arXiv arXiv 2025

  8. [8]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024. 1, 2, 3

    Pith/arXiv arXiv 2024

  9. [9]

    Why do large language models (llms) struggle to count letters?arXiv preprint arXiv:2412.18626,

    Tairan Fu, Raquel Ferrando, Javier Conde, Carlos Arriaga, and Pedro Reviriego. Why do large language models (llms) struggle to count letters?arXiv preprint arXiv:2412.18626,

    Pith/arXiv arXiv

  10. [10]

    Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline

    Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, and Feng Zheng. Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22942–22951, 2023. 1

    2023

  11. [11]

    Longvale: Vision-audio- language-event benchmark towards time-aware omni-modal perception of long videos

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio- language-event benchmark towards time-aware omni-modal perception of long videos. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18959– 18969, 2025. 1, 2, 3

    2025

  12. [12]

    Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities

    Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jae- hyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities. InForty-second International Conference on Ma- chine Learning, 2025. 4, 6

    2025

  13. [13]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.https:// storage.googleapis.com/deepmind- media/ gemini/gem- ini v1 5 report.pdf, 2024

    Google. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context.https:// storage.googleapis.com/deepmind- media/ gemini/gem- ini v1 5 report.pdf, 2024. 1, 2, 7, 8

    2024

  14. [14]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024. 7

    Pith/arXiv arXiv 2024

  15. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InPro- 9 ceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 7

    2023

  16. [16]

    Acav100m: Au- tomatic curation of large-scale datasets for audio-visual video representation learning

    Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song. Acav100m: Au- tomatic curation of large-scale datasets for audio-visual video representation learning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10274– 10284, 2021. 1, 3

    2021

  17. [17]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024. 7, 8

    Pith/arXiv arXiv 2024

  18. [18]

    Learning to answer questions in dynamic audio-visual scenarios.IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    Guangyao li, Yake Wei, Chenliang Xu Yapeng Tian, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios.IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1

    2022

  19. [19]

    Mvbench: A comprehensive multi-modal video understanding benchmark.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22195–22206, 2023

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22195–22206, 2023. 1

    2024

  20. [20]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195– 22206, 2024. 2

    2024

  21. [21]

    Watch and listen: Understanding audio- visual-speech moments with multimodal llm.arXiv preprint arXiv:2505.18110, 2025

    Zinuo Li, Xian Zhang, Yongxin Guo, Mohammed Ben- namoun, Farid Boussaid, Girish Dwivedi, Luqi Gong, and Qiuhong Ke. Watch and listen: Understanding audio- visual-speech moments with multimodal llm.arXiv preprint arXiv:2505.18110, 2025. 1, 2, 3

    arXiv 2025

  22. [22]

    Unleashing hour-scale video training for long video- language understanding.arXiv preprint arXiv:2506.05332,

    Jingyang Lin, Jialian Wu, Ximeng Sun, Ze Wang, Jiang Liu, Yusheng Su, Xiaodong Yu, Hao Chen, Jiebo Luo, Zicheng Liu, et al. Unleashing hour-scale video training for long video- language understanding.arXiv preprint arXiv:2506.05332,

    arXiv

  23. [23]

    Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcom- pass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024. 1

    Pith/arXiv arXiv 2024

  24. [24]

    Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 1, 2, 3

    2023

  25. [25]

    Gaia: a benchmark for general ai assistants

    Gr´egoire Mialon, Cl´ementine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Representations, 2023. 3

    2023

  26. [26]

    Per- ception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Systems, 36:42748–42761, 2023

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models.Advances in Neural Information Processing Systems, 36:42748–42761, 2023. 1

    2023

  27. [27]

    Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    Yujia Qin et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023. 3

    Pith/arXiv arXiv 2023

  28. [28]

    Robust speech recog- nition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recog- nition via large-scale weak supervision. InInternational con- ference on machine learning, pages 28492–28518. PMLR,

  29. [29]

    Toolformer: Language models can teach themselves to use tools.arXiv preprint arXiv:2302.04761,

    Timo Schick et al. Toolformer: Language models can teach themselves to use tools.arXiv preprint arXiv:2302.04761,

    Pith/arXiv arXiv

  30. [30]

    Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023. 2

    Pith/arXiv arXiv 2023

  31. [31]

    Allvb: All-in-one long video understanding benchmark

    Xichen Tan, Yuanjing Luo, Yunfan Ye, Fang Liu, and Zhiping Cai. Allvb: All-in-one long video understanding benchmark. InProceedings of the AAAI Conference on Artificial Intelli- gence, pages 7211–7219, 2025. 3

    2025

  32. [32]

    Gta: a benchmark for general tool agents.Advances in Neural Information Processing Systems, 37:75749–75790, 2024

    Jize Wang, Ma Zerun, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: a benchmark for general tool agents.Advances in Neural Information Processing Systems, 37:75749–75790, 2024. 1, 3

    2024

  33. [33]

    Lvbench: An extreme long video understanding benchmark.ArXiv, abs/2406.08035, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark.ArXiv, abs/2406.08035, 2024. 1, 2, 3

    Pith/arXiv arXiv 2024

  34. [34]

    Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 1, 7, 8

    Pith/arXiv arXiv 2025

  35. [35]

    Longvideobench: A benchmark for long-context inter- leaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024. 2

    Pith/arXiv arXiv 2024

  36. [36]

    Qwen2.5- omni technical report.arXiv preprint arXiv:2503.20215, 2025

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5- omni technical report.arXiv preprint arXiv:2503.20215, 2025. 1, 2, 7, 8

    Pith/arXiv arXiv 2025

  37. [37]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 1, 4, 6, 7, 8

    Pith/arXiv arXiv 2025

  38. [38]

    SVBench: A benchmark with temporal multi-turn dialogues for streaming video understanding

    Zhenyu Yang, Yuhang Hu, Zemin Du, Dizhan Xue, Sheng- sheng Qian, Jiahong Wu, Fan Yang, Weiming Dong, and Changsheng Xu. SVBench: A benchmark with temporal multi-turn dialogues for streaming video understanding. In The Thirteenth International Conference on Learning Repre- sentations, 2025. 2

    2025

  39. [39]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 6

    2023

  40. [40]

    Movqa: A 10 benchmark of versatile question-answering for long-form movie understanding.arXiv preprint arXiv:2312.04817, 2023

    Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A 10 benchmark of versatile question-answering for long-form movie understanding.arXiv preprint arXiv:2312.04817, 2023. 1, 3

    Pith/arXiv arXiv 2023

  41. [41]

    Lvbench: A benchmark for long- form video understanding with versatile multi-modal question answering.International Journal of Computer Vision, pages 1–22, 2025

    Hongjie Zhang, Lu Dong, Yi Liu, Yifei Huang, Yali Wang, Limin Wang, and Yu Qiao. Lvbench: A benchmark for long- form video understanding with versatile multi-modal question answering.International Journal of Computer Vision, pages 1–22, 2025. 1, 2, 3

    2025

  42. [42]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 7, 8

    2024

  43. [43]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024. 2

    Pith/arXiv arXiv 2024

  44. [44]

    Daily-omni: To- wards audio-visual reasoning with temporal alignment across modalities, 2025

    Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-omni: To- wards audio-visual reasoning with temporal alignment across modalities, 2025. 2 11 A Benchmark and Agentic Framework for Omni-Modal Reasoning and Tool Use in Long Videos Supplementary Material A. Details of LongShOTBench A.1. Dataset Statistics The LongShOTBench dataset is composed of 157 long- form vid...

    2025

  45. [45]

    **CAREFULLY** analyze the metadata to understand ONLY what is explicitly described in the video

  46. [46]

    Generate diverse, realistic scenarios representing different natural human perspectives for viewing this content

  47. [47]

    For each scenario, identify what kinds of questions a person in that context would naturally ask

  48. [48]

    Requirements

    Map those natural curiosities to evaluation task categories (to ensure comprehensive testing). Requirements

  49. [49]

    **Think like real people **: Consider realistic situations where someone would watch this video and be curious

  50. [50]

    **Natural multimodal interest **: People naturally connect what they see, hear, and observe - scenarios should reflect this

  51. [51]

    **Diverse perspectives **: Cover different reasons why people might watch (learning, entertainment, analysis, etc.)

  52. [52]

    **Distinct contexts **: Each scenario should represent a meaningfully different viewing situation

  53. [53]

    scenarios

    **STRICT METADATA GROUNDING **: Every scenario must be completely supportable by the provided metadata. Input: Structured or free-form metadata describing the visual, auditory, and speech content of a video. {video_metadata} Output format (strict JSON): { "scenarios": [ { "scenario": "<concise scenario description>", "tasks": ["<task_1>", "<task_2>", "......

  54. [54]

    **Different viewing contexts **: Each scenario should represent a distinct reason someone would watch

  55. [55]

    **Natural curiosity **: Focus on what people would genuinely want to know

  56. [56]

    **Realistic situations **: Avoid academic or artificial viewing contexts 16

  57. [57]

    **Diverse capabilities **: Ensure different scenarios test different multimodal capabilities

  58. [58]

    **Quality over quantity **: Better to have fewer, more realistic scenarios

  59. [59]

    **METADATA FIDELITY **: Every scenario must be completely derivable from the provided metadata **Task Selection Strategy: ** - Choose tasks based on what questions would naturally arise in each viewing context - Multimodal tasks should emerge naturally from the scenario, not be forced - Prioritize tasks that test capabilities people actually need in real ...

  60. [60]

    Is every detail in my scenario explicitly mentioned in the metadata?

  61. [61]

    Can someone verify each scenario element by checking the metadata?

  62. [62]

    Did I avoid adding any assumed context, locations, activities, or details?

  63. [63]

    Would my scenarios still make sense if the metadata were different? **Remember:** These scenarios drive the entire downstream pipeline. Natural, realistic scenarios that are strictly grounded in metadata lead to conversational questions that create a more useful and realistic video benchmark without hallucinated content. C.2. Question Type Mapping Prompt ...

  64. [64]

    Review the provided video metadata for context

  65. [65]

    Analyze the given scenario and ALL its assigned tasks

  66. [66]

    For each task, select the most relevant question type IDs from the comprehensive list below

  67. [67]

    Generate a MAXIMUM of 3 question types per task

  68. [68]

    Ensure diversity of question types across all tasks in the scenario

  69. [69]

    direct_fact

    Return only the question type IDs (e.g., "direct_fact", "causal_explanation") - no descriptions needed Input: Video Metadata (for context): {video_metadata} Scenario with Tasks: {scenario_with_tasks} COMPREHENSIVE QUESTION TYPE CATALOG Factual Retrieval Questions: - direct_fact: Extract specific information ("What did the coach tell them to focus on?") - ...

  70. [70]

    Match question types to task requirements: - Entity recognition tasks direct_fact, attribute_identification, visual_appearance - Temporal reasoning tasks timestamp_identification, duration_calculation, sequence_ordering - Spatial reasoning tasks location_identification, spatial_relationship, movement_tracking - Audio understanding tasks audio_description,...

  71. [71]

    scenario

    Selection criteria: - Choose exactly 1-3 most relevant question types per task - Focus on question types that directly test the task capability - Ensure diversity across all tasks in the scenario - avoid repeating question types - Include a mix of basic and complex types where appropriate - Ensure multimodal integration where applicable to the task Output...

  72. [72]

    NEVER invent or assume details not explicitly present

    **Be factually accurate ** Only say what you can actually see or hear in the video metadata. NEVER invent or assume details not explicitly present

  73. [73]

    **Sound conversational ** Use natural language that flows like normal speech

  74. [74]

    **Be helpful ** Fully answer what they asked, don’t leave them hanging

  75. [75]

    If something isn’t mentioned, DON’T make it up

    **Stay grounded ** Base everything on what’s actually in the video metadata. If something isn’t mentioned, DON’T make it up

  76. [76]

    **Be appropriately detailed ** Simple questions get simple answers, complex questions get more detail

  77. [77]

    they looked frustrated

    **Verify before stating ** If unsure about a detail, don’t include it. Better to be incomplete than wrong. Inputs - Video Metadata: {video_metadata} - Scenario: {scenario} - Task Context: {task_context} - Question Context: {question_context} - Conversation: {conversation} HOW TO ANSWER NATURALLY: **What to include: ** - What you actually see happening in ...