Pith. sign in

REVIEW 3 major objections 6 minor 10 cited by

This paper argues that long-video question answering is best framed as active, query-driven evidence seeking directly over pixels, and shows the approach outperforms caption-based agentic pipelines on five benchmarks while using a small fra

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:17 UTC pith:67CMYWP3

load-bearing objection AVP is a genuinely useful, well-ablated agentic long-video QA framework whose accuracy-over-captioning claim mostly holds up, but its efficiency numbers come from coarse sampling that can miss brief events, and the empirical support needs error bars and same-backbone rigor before the headline counts. the 3 major comments →

arxiv 2512.05774 v2 pith:67CMYWP3 submitted 2025-12-05 cs.CV cs.AIcs.CL

Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding

Ziyang Wang , Honglu Zhou , Shijie Wang , Junnan Li , Caiming Xiong , Silvio Savarese , Mohit Bansal , Michael S. Ryoo
show 1 more author
Juan Carlos Niebles
This is my paper
classification cs.CV cs.AIcs.CL
keywords active video perceptionlong video understandingagentic video reasoningplan-observe-reflectevidence seekingtemporal groundingvideo question answeringmultimodal LLM agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-video question answering usually suffers because relevant evidence is sparse while most footage is irrelevant. This paper claims that an agent should treat the video as an interactive environment and actively decide what to watch, where, and how densely, rather than first captioning everything. It proposes AVP, an iterative plan–observe–reflect process in which a planner specifies query-driven observations, an observer extracts compact time-stamped evidence directly from pixels, and a reflector decides whether enough evidence has been gathered. On five long-video benchmarks, AVP reports the highest accuracy among agentic methods, beating the strongest caption-based baseline by 5.7% average accuracy while using 18.4% of the inference time and 12.4% of the input tokens. If the result holds, query-driven perception is both more accurate and far cheaper than passive all-video captioning.

Core claim

AVP's central claim is that query-agnostic captioning is the wrong perceptual interface for long-video understanding: it wastes computation on irrelevant frames and blurs fine-grained temporal and spatial cues. The paper replaces it with goal-directed evidence seeking. Given a query, a planner decides what to look for, which temporal region to inspect, and at what fps and resolution; an observer reads pixels in that region and returns structured, time-stamped evidence; a reflector scores evidential sufficiency and either halts with an answer or provides a missing-information justification for another round. The loop keeps only query-relevant evidence, so the model can stop early with a confi

What carries the argument

The load-bearing component is an iterative plan–observe–reflect loop with three multimodal agents. The planner outputs an observation specification — what to inspect, where (a temporal region or uniform scan), and how (fps and spatial resolution). The observer executes that plan over raw pixels and emits structured, timestamped evidence entries, accumulated into an evidence list. The reflector computes a confidence score and justification, triggering re-planning until confidence exceeds a threshold (0.7) or a round limit (3) is reached. This closed loop is what lets the system perform coarse-to-fine localization and avoid dense full-video processing.

Load-bearing premise

The accuracy claim rests on the planner's sampling choices actually capturing query-relevant events; because the planner defaults to coarse scans (0.25–1 fps) when the query has no timestamp, very brief or subtle evidence can be skipped, as the paper's own failure case shows.

What would settle it

Run AVP on queries whose decisive evidence is a short event (under about one second) that occurs between frames of a coarse scan, or a subtle cue requiring high resolution in a region the planner deprioritizes; if the reflector outputs a confident wrong answer on such queries, the coarse-sampling assumption fails. A targeted benchmark of 'brief-evidence' questions would settle this.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, agentic long-video systems can omit the captioning stage and query pixels directly, which the paper finds is both more accurate and cheaper.
  • The sufficiency-checking reflector gives a principled stopping rule, so compute scales with query difficulty rather than video length.
  • Structured time-stamped evidence, rather than free-form captions, improves grounding; ablations show removing temporal structure hurts accuracy.
  • The framework is training-free and transferable to different backbone MLLMs, with performance scaling as the backbone strengthens.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: A natural next step is replacing hand-written planner heuristics with a learned sampling policy, because the paper's own failure case shows a coarse 0.5 fps scan missing a short, decisive event.
  • Editorial inference: The accumulated evidence list could be reused for follow-up questions on the same video, effectively serving as structured episodic memory for multi-turn dialogue.
  • Editorial inference: The same active-perception loop could be embedded in embodied agents that must decide where to look next under physical constraints, a direction the paper names but does not test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Active Video Perception (AVP), a training-free agentic framework for long-video QA. AVP replaces query-agnostic captioning with an iterative plan–observe–reflect loop: a planner specifies what/where/how to observe (temporal span, fps, spatial resolution), an observer MLLM extracts timestamped evidence directly from pixels, and a reflector estimates evidence sufficiency and either answers or triggers another round. Using Gemini-2.5-Pro, the authors report top accuracy on MINERVA, LVBench, MLVU, Video-MME, and LongVideoBench, and claim a 5.7% average accuracy gain over DVD with 18.4% inference time and 12.4% input tokens on LVBench. Ablations cover components, backbone choice, max rounds, confidence threshold, and evidence format; appendices add MiRA reasoning-trace scores, per-type LVBench results, a same-backbone comparison with DVD, and a failure case.

Significance. The manuscript addresses an important problem and puts forward a clean, reproducible idea: active, query-driven evidence seeking instead of passive full-video captioning. The prompt-level design is simple, the ablations are unusually thorough, and the efficiency data suggest large potential savings. If the accuracy claims could be established under controlled comparisons and with proper uncertainty quantification, AVP would be a valuable contribution to agentic long-video understanding. However, the current central claim is overstated: the headline comparison mixes backbones, the same-backbone comparison shows no accuracy advantage on LVBench, and per-type results reveal a large deficit on Key Information Retrieval. The paper is better framed as an efficiency-preserving accuracy method whose gains need to be demonstrated under matched backbones and question categories.

major comments (3)
  1. [§1, Tab. 1 vs. Appendix B.3, Tab. 8] The headline claim of a 5.7% average accuracy gain over DVD is not traceable from the reported tables. In Tab. 1, DVD has no entries for MINERVA, MLVU, or Video-MME Overall, so the basis of the 5.7% average is unclear. More importantly, the efficiency comparison in Tab. 2 uses Gemini-2.5-Pro for AVP and OpenAI-o3 for DVD, while the same-backbone comparison in Tab. 8 shows AVP at 73.1 on LVBench versus DVD at 74.2, i.e., slightly lower. Thus the accuracy advantage and the efficiency advantage are not established simultaneously. Please report same-backbone results on all benchmarks, or explicitly describe the cross-backbone numbers as a system-level comparison and qualify the accuracy claim.
  2. [§5.1, Tab. 7, App. C.2] The active-sampling design can miss brief events. The planner defaults to 0.25–1.0 fps uniform scans for untimestamped queries (Planner prompt, App. D.1), and the paper's own failure case (Fig. 5) shows a 0.5 fps scan missing a three-pointer at 00:20, producing a confident wrong answer. Per-type LVBench results in Tab. 7 are consistent with this risk: AVP is 12.7 points below DVD on Key Information Retrieval (67.7 vs. 80.4) and 1.5 points below on Entity Recognition. Since KIR directly tests detecting short, isolated key information, the broad claim that AVP 'achieves best performance' across LVU tasks needs to be qualified by question type, and the temporal-recall failure mode should be analyzed quantitatively (e.g., accuracy broken down by event duration/frequency).
  3. [§5.2, Tabs. 2, 5, 11] No error bars, repeated runs, or significance tests are reported. Several headline differences are small (LVBench +0.6, LongVideoBench Long +1.4), and API-based MLLM outputs can be stochastic; the phrase 'significant improvements' is not backed by statistics. Additionally, R_max and τ_conf appear to be tuned on the same evaluation benchmarks (Tabs. 5 and 11) without a reported validation split. Please provide variance estimates (at least on a subset) and either fix these hyperparameters a priori or validate them on a held-out split.
minor comments (6)
  1. [§4.1] The dataset list has two '(2)' entries (LV-Bench and MLVU) and the Video-MME description contains an incomplete sentence about average duration. Please fix numbering and wording.
  2. [§2] Typo: 'DeekSeek-R1' should be 'DeepSeek-R1'; also 'V AP' should be 'VAP'.
  3. [Tab. 1] Caption typo: 'Webold' should be 'We bold'.
  4. [Fig. 3] Options A and E are identical ('middle right midground'), which appears to be a typo. This makes the example harder to follow.
  5. [Abstract / §1] The abstract and introduction pair '5.7% average accuracy' with '18.4% inference time and 12.4% input tokens' as if measured on the same benchmarks, but the efficiency numbers are LVBench-specific. Please clarify the scope of each number.
  6. [Algorithm 1] The functions REFLECTOR.EXTRACTANSWER and REFLECTOR.FORCEANSWER are used in the algorithm but not described in the method text or prompts. Please add a brief description or prompt excerpt.

Circularity Check

0 steps flagged

No significant circularity; core claims are empirical and benchmark-based rather than derived by construction.

full rationale

AVP is an agentic pipeline with no formal derivation or fitted equation whose output is then relabeled as a prediction. The main claims—accuracy gains and token/time reductions versus DVD—are measured on external benchmarks (MINERVA, LVBench, MLVU, Video-MME, LongVideoBench) against published baselines, so the central evaluation is not equivalent to the method's inputs by construction. The paper's self-citations (e.g., VideoTree [63], SiLVR [76], Video-RTS [65]) appear in related work as background and do not serve as load-bearing justification for AVP's design or results; there is no imported uniqueness theorem and no ansatz smuggled in via citation. The mild self-referential aspects—the same MLLM acts as observer and reflector, and the confidence threshold / max rounds are tuned on MINERVA and LVBench before headline numbers are reported on those same benchmarks—are evaluation-design caveats rather than definitional circularity, and the gains on the other three benchmarks remain out-of-sample with respect to that tuning. The sampling-rate limitation identified in the failure case (Fig. 5) is a correctness and robustness risk, not a circularity of the derivation chain. Overall, the paper does not reduce its predictions to its inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

AVP introduces no new physical or mathematical entities; planner/observer/reflector are roles of an MLLM. The main auxiliary inputs are two stopping hyperparameters and a sampling heuristic, both tuned or chosen by hand.

free parameters (3)
  • confidence threshold tau_conf = 0.7
    Selected after ablation on benchmark accuracy (Table 11); 0.7 performs best on MINERVA and ties on LVBench, indicating tuning to the evaluation set.
  • maximum rounds R_max = 3
    Selected after ablation (Table 5); accuracy saturates at 3 rounds, so the stopping budget is benchmark-tuned.
  • coarse-scan fps range = 0.25-1.0 fps
    A heuristic in the planner prompt for queries with no temporal prior; the specific range controls recall and is not derived from a correctness bound.
axioms (3)
  • domain assumption Query-relevant evidence in long videos is sparse enough that low-cost coarse scans followed by targeted refinement can find it with high recall.
    The efficiency-accuracy claim depends on this; the failure case in App. C.2 shows it can be false for short, infrequent events.
  • domain assumption A proprietary MLLM (Gemini-2.5-Pro) can reliably act as planner, observer, and reflector when prompted, with no training.
    The framework's quality rests on the instruction-following and visual perception abilities of the chosen backbone (Sec. 4.3, Tab. 4).
  • domain assumption The published baseline numbers and ground-truth labels from the five benchmarks are accurate and were collected under comparable settings.
    The authors do not rerun all baselines under identical API/evaluation conditions; they rely on cited numbers (Tab. 1).

pith-pipeline@v1.3.0-alltime-deepseek · 20248 in / 10434 out tokens · 99379 ms · 2026-08-03T18:17:08.079462+00:00 · methodology

0 comments
read the original abstract

Long video understanding (LVU) is challenging because answering real-world queries often depends on sparse, temporally dispersed cues buried in hours of mostly redundant and irrelevant content. While agentic pipelines improve video reasoning capabilities, prevailing frameworks rely on a query-agnostic captioner to perceive video information, which wastes computation on irrelevant content and blurs fine-grained temporal and spatial information. Motivated by active perception theory, we argue that LVU agents should actively decide what, when, and where to observe, and continuously assess whether the current observation is sufficient to answer the query. We present Active Video Perception (AVP), an evidence-seeking framework that treats the video as an interactive environment and acquires compact, queryrelevant evidence directly from pixels. Concretely, AVP runs an iterative plan-observe-reflect process with MLLM agents. In each round, a planner proposes targeted video interactions, an observer executes them to extract time-stamped evidence, and a reflector evaluates the sufficiency of the evidence for the query, either halting with an answer or triggering further observation. Across five LVU benchmarks, AVP achieves highest overall accuracy with significant improvements. Notably, AVP outperforms the best agentic method by 5.7% in average overall accuracy while only requires 18.4% inference time and 12.4% input tokens.

Figures

Figures reproduced from arXiv: 2512.05774 by Caiming Xiong, Honglu Zhou, Juan Carlos Niebles, Junnan Li, Michael S. Ryoo, Mohit Bansal, Shijie Wang, Silvio Savarese, Ziyang Wang.

Figure 1
Figure 1. Figure 1: Motivation of Active Video Perception. Prior meth￾ods follow a passive perception paradigm which leverage query￾agonistic captioner to perceive the video information, leading to low efficiency and imprecise visual grounding. Instead, we actively perceive query-relevant content by treating the long video as an interactive environment to be explored in a goal-directed manner. integrate sparse, temporally dis… view at source ↗
Figure 2
Figure 2. Figure 2: Framework of Active Video Perception (AVP). AVP operates by an iterative plan-observe-reflect process with MLLM agents. At each round, the planner decide what/where/how to in￾teract with the video, the observer extract structured query-related evidence by executing the plan and the reflector evaluates the ex￾tracted evidence to decide whether an additional round is need. threshold, AVP outputs the answer a… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative example of AVP. Given a multiple-choice query about the Tombstone monument’s first on-screen appearance, Round 1 performs a coarse scan of the entire video (0.5 FPS, low resolution) and localizes a candidate interval [1:00, 1:10], but the REFLECTOR judges the evidence insufficient. Round 2 re-plans a targeted pass over this window (2 FPS, medium resolution), enabling the OBSERVER to localize th… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative example of multi-round active perception in AVP (MINERVA sample). Given the query, “After adding up all the millimeter totals on the sheet of paper illustrated at 09:58, and then adding the average length of Louisiana Pine Snake hatchlings according to the video, how many total millimeters are there?”, AVP first plans to focus on the local timestamped frame at 09:58 and extracts the seven milli… view at source ↗
Figure 5
Figure 5. Figure 5: Failure Case of AVP (MINERVA sample). Given a long broadcast basketball video, AVP must answer: “How many three-pointers are made before the second clip of Hawaii versus UCSB?” The planner chooses to scan the entire video at 0.5 FPS with low spatial resolution, the observer summarizes the retrieved segments into a structured evidence list, and the reflector produces a confident answer of two. However, the … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

    cs.CV 2026-05 unverdicted novelty 8.0

    EgoMemReason is a new benchmark showing that even the best multimodal models achieve only 39.6% accuracy on reasoning tasks that require integrating sparse evidence across days in egocentric video.

  2. Benchmarking Visual State Tracking in Multimodal Video Understanding

    cs.CV 2026-06 unverdicted novelty 7.0

    VSTAT benchmark shows state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines on visual state tracking, failing at visual perception despite correct textual reasoning.

  3. Decomposing Queries into Tool Calls for Long-Video Keyframe Retrieval

    cs.CV 2026-05 unverdicted novelty 7.0

    ToolMerge decomposes queries into LLM-planned tool calls merged by boolean operators for long-video keyframe retrieval and introduces the M2M benchmark, showing competitive results with 5% gains on caption retrieval.

  4. FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding

    cs.CV 2026-07 conditional novelty 6.0

    A query-conditioned, training-free frame selector that unifies relevance and diversity into a single volume-maximization objective improves keyframe recall and long-video question-answering accuracy.

  5. FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding

    cs.CV 2026-07 conditional novelty 6.0

    A training-free frame-selection method that weights frame embeddings by query relevance and maximizes the selected subspace's volume improves keyframe recall and VQA accuracy across eight MLLMs on Video-MME and LongVi...

  6. Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

    cs.CV 2026-07 conditional novelty 6.0

    An agent that searches a long video by navigating an adaptive temporal tree with zoom-in/zoom-out/shift actions improves grounded long-video QA, but the headline CG-Bench result is measured on a heavily filtered subset.

  7. Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

    cs.AI 2026-06 unverdicted novelty 6.0

    Introduces V-RAGBench benchmark and CARVE method that selects per-chunk retrieval configurations via parallel retrievers and adaptive reranking, outperforming eight VideoRAG baselines.

  8. Personal Visual Context Learning in Large Multimodal Models

    cs.CV 2026-05 unverdicted novelty 6.0

    Introduces Personal VCL formalization and benchmark revealing LMM context gaps, plus an Agentic Context Bank baseline that boosts personalized visual reasoning.

  9. Decomposing Queries into Tool Calls for Long-Video Keyframe Retrieval

    cs.CV 2026-05 conditional novelty 5.0

    An LLM planner decomposes long-video queries into tool calls and boolean merge rules, yielding competitive keyframe retrieval and a 5% gain on caption retrieval on the new M2M benchmark.

  10. Watch, Remember, Reason: Human-View Video Understanding with MLLMs

    cs.CV 2026-06 unverdicted novelty 4.0

    This is a survey that frames video MLLM research via a human-view formulation of perceptual representations, memory states, reasoning traces, and predictions, then reviews methods, datasets, benchmarks, and open problems.

Reference graph

Works this paper leans on

85 extracted references · 14 linked inside Pith · cited by 8 Pith papers

  1. [1]

    Psychology Press,

    Yiannis Aloimonos.Active perception. Psychology Press,

  2. [2]

    Temporal chain of thought: Long-video understanding by thinking in frames, 2025

    Anurag Arnab, Ahmet Iscen, Mathilde Caron, Alireza Fathi, and Cordelia Schmid. Temporal chain of thought: Long-video understanding by thinking in frames, 2025. 2

    2025

  3. [3]

    Qwen2.5- vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report.a...

    Pith/arXiv arXiv 2025

  4. [4]

    Active perception.Proceedings of the IEEE, 76(8):966–1005, 1988

    Ruzena Bajcsy. Active perception.Proceedings of the IEEE, 76(8):966–1005, 1988. 2, 3

    1988

  5. [5]

    Ruzena Bajcsy, Yiannis Aloimonos, and John K. Tsotsos. Revisiting active perception, 2016. 2, 3

    2016

  6. [6]

    Revisiting the “Video” in Video-Language Understanding

    Shyamal Buch, Cristobal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the “Video” in Video-Language Understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2

    2022

  7. [7]

    Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. Hourvideo: 1-hour video-language understanding. InNeurIPS Datasets and Benchmarks Track, 2024. 2

    2024

  8. [8]

    Lvagent: Long video under- standing by multi-round dynamical collaboration of mllm agents.arXiv preprint arXiv:2503.10200, 2025

    Boyu Chen, Zhengrong Yue, Siran Chen, Zikang Wang, Yang Liu, Peng Li, and Yali Wang. Lvagent: Long video under- standing by multi-round dynamical collaboration of mllm agents.arXiv preprint arXiv:2503.10200, 2025. 3, 5, 6

    arXiv 2025

  9. [9]

    Longvila: Scaling long-context visual language models for long videos,

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for long videos,

  10. [10]

    Deepseek-r1: Incentivizing reasoning capabil- ity in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capabil- ity in llms via reinforcement learning, 2025. 2, 6

    2025

  11. [11]

    See what you need: Query-aware visual intelligence through reasoning-perception loops, 2025

    Zixuan Dong, Baoyun Peng, Yufei Wang, Lin Liu, Xinxin Dong, Yunlong Cao, and Xiaodong Wang. See what you need: Query-aware visual intelligence through reasoning-perception loops, 2025. 3

    2025

  12. [12]

    Agentic keyframe search for video question answering, 2025

    Sunqi Fan, Meng-Hao Guo, and Shuojin Yang. Agentic keyframe search for video question answering, 2025. 3

    2025

  13. [13]

    Videoagent: A memory-augmented multi- modal agent for video understanding, 2024

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multi- modal agent for video understanding, 2024. 3

    2024

  14. [14]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yib- ing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025. 2

    Pith/arXiv arXiv 2025

  15. [15]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis, 2024. 2, 5

    2024

  16. [16]

    Love-r1: Advancing long video understanding with an adaptive zoom-in mechanism via multi- step reasoning, 2025

    Shenghao Fu, Qize Yang, Yuan-Ming Li, Xihan Wei, Xiaohua Xie, and Wei-Shi Zheng. Love-r1: Advancing long video understanding with an adaptive zoom-in mechanism via multi- step reasoning, 2025. 2

    2025

  17. [17]

    Framemind: Frame-interleaved video reasoning via reinforcement learning, 2025

    Haonan Ge, Yiwei Wang, Kai-Wei Chang, Hang Wu, and Yujun Cai. Framemind: Frame-interleaved video reasoning via reinforcement learning, 2025. 5, 6

    2025

  18. [18]

    Framethinker: Learning to think with long videos via multi-turn frame spotlighting, 2025

    Zefeng He, Xiaoye Qu, Yafu Li, Siyuan Huang, Daizong Liu, and Yu Cheng. Framethinker: Learning to think with long videos via multi-turn frame spotlighting, 2025. 2

    2025

  19. [19]

    Adaptive video understanding agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning, 2024

    Sullam Jeoung, Goeric Huybrechts, Bhavana Ganesh, Aram Galstyan, and Sravan Bodapati. Adaptive video understanding agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning, 2024. 3

    2024

  20. [20]

    Language repository for long video understanding

    Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. InFindings of the Association for Compu- tational Linguistics: ACL 2025, pages 5627–5646, Vienna, Austria, 2025. Association for Computational Linguistics. 3

    2025

  21. [21]

    Videomultiagents: A multi-agent framework for video question answering, 2025

    Noriyuki Kugo, Xiang Li, Zixin Li, Ashish Gupta, Arpandeep Khatua, Nidhish Jain, Chaitanya Patel, Yuta Kyuragi, Ya- sunori Ishii, Masamoto Tanabiki, Kazuki Kozuka, and Ehsan Adeli. Videomultiagents: A multi-agent framework for video question answering, 2025. 3

    2025

  22. [22]

    Aria: An open multimodal native mixture-of- experts model, 2025

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, Chongyan Zhu, Xiaoyi Ren, Chao Li, Yifan Ye, Peng Liu, Lihuan Zhang, Hanshu Yan, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of- experts model, 2025. 1

    2025

  23. [23]

    BLIP: Bootstrapping language-image pre-training for unified vision- language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision- language understanding and generation. InProceedings of the 39th International Conference on Machine Learning, pages 12888–12900. PMLR, 2022

    2022

  24. [24]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR,

  25. [25]

    Videochat-flash: Hierarchical compression for long-context video modeling

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, and Limin Wang. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574, 2024. 2

    Pith/arXiv arXiv 2024

  26. [26]

    Video-llava: Learning united visual represen- tation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023. 2

    Pith/arXiv arXiv 2023

  27. [27]

    Videomind: A chain-of-lora agent for long video reasoning.arXiv preprint arXiv:2503.13444, 2025

    Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou. Videomind: A chain-of-lora agent for long video reasoning.arXiv preprint arXiv:2503.13444, 2025. 3

    arXiv 2025

  28. [28]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension, 2024

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong 9 Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension, 2024. 3

    2024

  29. [29]

    Video active perception: Efficient inference-time long-form video understanding with vision-language models

    Martin Q Ma, Willis Guo, Aditya Agrawal, Ankit Gupta, Paul Pu Liang, Russ Salakhutdinov, and Louis-Philippe Morency. Video active perception: Efficient inference-time long-form video understanding with vision-language models

  30. [30]

    Drvideo: Document retrieval based long video understanding, 2024

    Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. Drvideo: Document retrieval based long video understanding, 2024. 3

    2024

  31. [31]

    Caviar: Critic- augmented video agentic reasoning, 2025

    Sachit Menon, Ahmet Iscen, Arsha Nagrani, Tobias Weyand, Carl V ondrick, and Cordelia Schmid. Caviar: Critic- augmented video agentic reasoning, 2025. 3

    2025

  32. [32]

    Morevqa: Exploring modular reasoning models for video question answering, 2025

    Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reasoning models for video question answering, 2025. 3

    2025

  33. [33]

    Minerva: Evaluating complex video reasoning, 2025

    Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl V ondrick, Mikhail Sirotenko, Cordelia Schmid, and Tobias Weyand. Minerva: Evaluating complex video reasoning, 2025. 2, 5, 12

    2025

  34. [34]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. 5, 6

    2024

  35. [35]

    Gpt-5 system card

    OpenAI. Gpt-5 system card. https://cdn.openai.com/ gpt-5-system-card.pdf, 2025. 1

    2025

  36. [36]

    Introducing gpt-4.1 in the api

    OpenAI. Introducing gpt-4.1 in the api. https://openai. com/index/gpt-4-1/, 2025. Accessed: 2025-11-10. 5, 6

    2025

  37. [37]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. Sys- tem Card v1, OpenAI, 2025. PDF available at: https://cdn.openai.com/pdf/2221c875-02dc-4789-800b- e7758f3722c1/o3-and-o4-mini-system-card.pdf. Accessed: 2025-11-10. 6

    2025

  38. [38]

    Conan: Progressive learning to reason like a detective over multi-scale visual evidence, 2025

    Kun Ouyang, Yuanxin Liu, Linli Yao, Yishuo Cai, Hao Zhou, Jie Zhou, Fandong Meng, and Xu Sun. Conan: Progressive learning to reason like a detective over multi-scale visual evidence, 2025. 2

    2025

  39. [39]

    mapreduce

    Ziqi Pang and Yu-Xiong Wang. Mr. video: "mapreduce" is the principle for long video understanding, 2025. 3

    2025

  40. [40]

    Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryu, Donghyun Kim, and Michael S. Ryoo. Too many frames, not all useful: Efficient strategies for long-form video qa, 2025. 3

    2025

  41. [41]

    Understanding long videos in one multimodal language model pass

    Kanchana Ranasinghe, Xiang Li, Kumara Kahatapitiya, and Michael Ryoo. Understanding long videos in one multimodal language model pass. InInternational Conference on Learn- ing Representations, 2025. 2

    2025

  42. [42]

    Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Jongwoo Park, Kanchana Ranasinghe, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles

    Michael S. Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Jongwoo Park, Kanchana Ranasinghe, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms, 2025. 2

    2025

  43. [43]

    Jinghuan Shang and Michael S. Ryoo. Active vision rein- forcement learning under limited visual observability, 2023. 2

    2023

  44. [44]

    Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Bal- akrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. Longvu: Spa- tiotemporal adaptive compression for long video-language understanding.arXiv prep...

    Pith/arXiv arXiv 2024

  45. [45]

    Vgent: Graph-based retrieval-reasoning-augmented generation for long video understanding, 2025

    Xiaoqian Shen, Wenxuan Zhang, Jun Chen, and Mohamed El- hoseiny. Vgent: Graph-based retrieval-reasoning-augmented generation for long video understanding, 2025. 2, 3, 5, 6

    2025

  46. [46]

    Enhanc- ing video-llm reasoning via agent-of-thoughts distillation,

    Yudi Shi, Shangzhe Di, Qirui Chen, and Weidi Xie. Enhanc- ing video-llm reasoning via agent-of-thoughts distillation,

  47. [47]

    Video- xl: Extra-long vision language model for hour-scale video understanding.arXiv preprint arXiv:2409.14485, 2024

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video- xl: Extra-long vision language model for hour-scale video understanding.arXiv preprint arXiv:2409.14485, 2024. 2

    Pith/arXiv arXiv 2024

  48. [48]

    Scene exploration by vision-language models,

    Venkatesh Sripada, Samuel Carter, Frank Guerin, and Amir Ghalamzan. Scene exploration by vision-language models,

  49. [49]

    Adaptive keyframe sampling for long video understanding, 2025

    Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding, 2025. 2

    2025

  50. [50]

    Moss-chatv: Reinforcement learning with process reasoning reward for video temporal reasoning, 2025

    Sicheng Tao, Jungang Li, Yibo Yan, Junyan Zhang, Yubo Gao, Hanqian Li, ShuHang Xun, Yuxuan Fan, Hong Chen, Jianxiang He, and Xuming Hu. Moss-chatv: Reinforcement learning with process reasoning reward for video temporal reasoning, 2025. 2

    2025

  51. [51]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025

    Gemini team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. 1, 5, 6, 7

    2025

  52. [52]

    Qwen3-vl: A general vision-language model

    Qwen Team. Qwen3-vl: A general vision-language model. https : / / qwenlm . github . io / blog / qwen3 - vl/, 2025. Model card and technical documentation. 6

    2025

  53. [53]

    Seed1.5-vl technical report, 2025

    Seed-VL team. Seed1.5-vl technical report, 2025. 1, 5, 6

    2025

  54. [54]

    Tarsier: Recipes for training and evaluating large video description models, 2024

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models, 2024. 2

    2024

  55. [55]

    Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning, 2025

    Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou. Videorft: Incentivizing video reasoning capability in mllms via reinforced fine-tuning, 2025. 2

    2025

  56. [56]

    Vamos: Versatile action models for video understanding, 2023

    Shijie Wang, Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwon- joon Lee, and Chen Sun. Vamos: Versatile action models for video understanding, 2023. 2

    2023

  57. [57]

    Alvarez, Lei Zhang, and Zhiding Yu

    Shihao Wang, Guo Chen, De an Huang, Zhiqi Li, Minghan Li, Guilin Li, Jose M. Alvarez, Lei Zhang, and Zhiding Yu. Videoitg: Multimodal video understanding with instructed temporal grounding, 2025. 2

    2025

  58. [58]

    thinking with videos

    Shijian Wang, Jiarui Jin, Xingjian Wang, Linxin Song, Run- hao Fu, Hecheng Wang, Zongyuan Ge, Yuan Lu, and Xuelian Cheng. Video-thinker: Sparking "thinking with videos" via reinforcement learning, 2025. 2

    2025

  59. [59]

    Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 1

    Pith/arXiv arXiv 2025

  60. [60]

    Videoagent: Long-form video understanding with large language model as agent, 2024

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent, 2024. 1, 5, 6 10

    2024

  61. [61]

    Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding, 2025

    Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding, 2025. 2, 5, 6

    2025

  62. [62]

    Timezero: Temporal video grounding with reasoning-guided lvlm.arXiv preprint arXiv:2503.13377, 2025

    Ye Wang, Boshen Xu, Zihao Yue, Zihan Xiao, Ziheng Wang, Liang Zhang, Dingyi Yang, Wenxuan Wang, and Qin Jin. Timezero: Temporal video grounding with reasoning-guided lvlm.arXiv preprint arXiv:2503.13377, 2025. 2

    Pith/arXiv arXiv 2025

  63. [63]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209,

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209,

    Pith/arXiv arXiv

  64. [64]

    Videochat-a1: Thinking with long videos by chain-of-shot reasoning, 2025

    Zikang Wang, Boyu Chen, Zhengrong Yue, Yi Wang, Yu Qiao, Limin Wang, and Yali Wang. Videochat-a1: Thinking with long videos by chain-of-shot reasoning, 2025. 3

    2025

  65. [65]

    Video-RTS: Re- thinking reinforcement learning and test-time scaling for ef- ficient and enhanced video reasoning

    Ziyang Wang, Jaehong Yoon, Shoubin Yu, Md Mohaiminul Islam, Gedas Bertasius, and Mohit Bansal. Video-RTS: Re- thinking reinforcement learning and test-time scaling for ef- ficient and enhanced video reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28114–28128, Suzhou, China, 2025. Asso- ciation for...

    2025

  66. [66]

    Longvideobench: A benchmark for long-context interleaved video-language understanding.Advances in Neural Informa- tion Processing Systems, 37:28828–28857, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding.Advances in Neural Informa- tion Processing Systems, 37:28828–28857, 2024. 2, 5

    2024

  67. [67]

    Vision in action: Learning active perception from human demonstrations, 2025

    Haoyu Xiong, Xiaomeng Xu, Jimmy Wu, Yifan Hou, Jean- nette Bohg, and Shuran Song. Vision in action: Learning active perception from human demonstrations, 2025. 2

    2025

  68. [68]

    Lvbench: An extreme long video under- standing benchmark.arXiv preprint arXiv:2406.08035, 2024

    Bowen Xu, Yifan Zhang, Yufei Zhao, Yizhou Wang, Yu Qiao, and Hongsheng Li. Lvbench: An extreme long video under- standing benchmark.arXiv preprint arXiv:2406.08035, 2024. 2, 5

    Pith/arXiv arXiv 2024

  69. [69]

    Ava: To- wards agentic video analytics with vision language models,

    Yuxuan Yan, Shiqi Jiang, Ting Cao, Yifan Yang, Qianqian Yang, Yuanchao Shu, Yuqing Yang, and Lili Qiu. Ava: To- wards agentic video analytics with vision language models,

  70. [70]

    Generative frame sampler for long video understanding.arXiv preprint arXiv:2503.09146, 2025

    Linli Yao, Haoning Wu, Kun Ouyang, Yuanxing Zhang, Caim- ing Xiong, Bei Chen, Xu Sun, and Junnan Li. Generative frame sampler for long video understanding.arXiv preprint arXiv:2503.09146, 2025. 2

    Pith/arXiv arXiv 2025

  71. [71]

    Re-thinking temporal search for long- form video understanding, 2025

    Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chan- drasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long- form video understanding, 2025. 2

    2025

  72. [72]

    End-to-end learning of action detection from frame glimpses in videos

    Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei- Fei. End-to-end learning of action detection from frame glimpses in videos. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1

    2016

  73. [73]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. InNeurIPS, 2023. 2

    2023

  74. [74]

    Videoex- plorer: Think with videos for agentic long-video understand- ing, 2025

    Huaying Yuan, Zheng Liu, Junjie Zhou, Hongjin Qian, Yan Shu, Nicu Sebe, Ji-Rong Wen, and Zhicheng Dou. Videoex- plorer: Think with videos for agentic long-video understand- ing, 2025. 2

    2025

  75. [75]

    A simple llm framework for long-range video question-answering, 2023

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering, 2023. 1, 3

    2023

  76. [76]

    Silvr: A simple language-based video rea- soning framework, 2025

    Ce Zhang, Yan-Bo Lin, Ziyang Wang, Mohit Bansal, and Gedas Bertasius. Silvr: A simple language-based video rea- soning framework, 2025. 3, 5, 6

    2025

  77. [77]

    Thinking with videos: Multimodal tool- augmented reinforcement learning for long video reasoning,

    Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma, Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang. Thinking with videos: Multimodal tool- augmented reinforcement learning for long video reasoning,

  78. [78]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024. 2

    Pith/arXiv arXiv 2024

  79. [79]

    Deep video discovery: Agentic search with tool use for long-form video understanding

    Xiaoyi Zhang, Zhaoyang Jia, Zongyu Guo, Jiahao Li, Bin Li, Houqiang Li, and Yan Lu. Deep video discovery: Agentic search with tool use for long-form video understanding. In Advances in Neural Information Processing Systems (NeurIPS 2025), 2025. 2, 3, 5, 6, 7, 12

    2025

  80. [80]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024. 2

    Pith/arXiv arXiv 2024

Showing first 80 references.