Pith. sign in

REVIEW 4 major objections 4 minor 148 references

MR. Video: "MapReduce" is the Principle for Long Video Understanding

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Treating long video understanding as MapReduce—caption every clip, then aggregate—outperforms prior models by 10 points on LVBench.

desk verdict A plausible and clearly written new recipe for long-video QA, but the headline >10% LVBench gain is not yet an apples-to-apples comparison. read the letter →

arxiv 2504.16082 v1 pith:6MUVKKY2 submitted 2025-04-22 cs.CV

Ziqi Pang , Yu-Xiong Wang This is my paper · ORCID
classification cs.CV
keywords longvideounderstandingMapReduceagentsvision-languagemodelsdensecaptioningLVBenchquestionintentionanalysisanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that long video understanding can be treated as large-scale data processing: break the video into short clips, perceive each clip independently and in detail (the Map step), then aggregate all clip-level results into a unified answer (the Reduce step). It instantiates this as MR. Video, a two-stage agent pipeline that first writes dense captions for every scene and then runs question-specific analysis over those captions. On LVBench, which contains hour-long videos with challenging questions, MR. Video scores 60.8% accuracy, more than 10 points above state-of-the-art VLMs and video agents. The authors argue that the MapReduce principle, not any single component, is what makes the improvement possible.

What carries the argument

The load-bearing mechanism is the MapReduce decomposition itself: independent, sequence-parallel perception of short clips (Map) followed by joint aggregation and reasoning over all clip-level outputs (Reduce). In MR. Video this takes concrete form as a Captioning stage—dense scene captions plus a Reduce step that merges repeated characters into standardized names—and an Analysis stage—question intention analysis over caption segments, goal-aware queries to a vision-language model on candidate scenes, and a final Reduce that composes an answer. This design sidesteps the context-length limit of sequence-to-sequence VLMs and replaces the sequential key-segment retrieval of earlier video agents with broader, parallel scene inspection and global reasoning.

What would settle it

For any LVBench question whose answer depends on a visual detail visible in only one short clip, edit the clip's caption to omit that detail and rerun MR. Video; if the model still answers correctly, the stated caption bottleneck is not load-bearing, whereas if accuracy drops, the bottleneck claim is confirmed. Equivalently, measure whether per-question accuracy is predicted by whether the correct object appears in the captions.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the MapReduce principle is the route to long video understanding: every short segment is perceived thoroughly and independently, so no detail is skipped, and the results are then combined globally, so no context is lost to a limited window. MR. Video implements this in two stages: captioning maps clips to dense scene captions and reduces repeated characters and objects to consistent names; analysis maps the user's question onto candidate scenes and reduces scene-level evidence into a final answer. The reported result is 60.8% accuracy on LVBench, an improvement of more than 10% over previous VLMs and video agents, with ablations showing that consistent character names, question intention analysis, and goal-aware perception each contribute. The authors frame the principle as applicable to both VLMs and video agents, with the LLM-agent implementation serving as a convenient validation.

Load-bearing premise

The load-bearing premise is that the dense captions and sparsely sampled frames produced in the Map stage preserve the visual details needed to answer each question; the authors state in Section 4.5.2 that information lost at captioning time, such as a missed object or face, cannot be recovered by the Reduce stage.

Editorial extensions

If this is right

  • A simple caption-then-analyze pipeline can outperform both context-limited VLMs and key-segment video agents on localization-heavy long-video benchmarks.
  • Because the Map steps are independent and sequence-parallel, the same recipe should scale to longer videos without hitting a context window, at the cost of more captioning calls.
  • Question intention analysis over captions localizes relevant scenes more reliably than embedding-based key-frame retrieval (68.8% vs. 34.4% recall on the paper's subset).
  • Character and object consistency in captions, and a final goal-aware perception step, each contribute measurable accuracy on the ablation subset, so the Reduce stages are doing real work, not just bookkeeping.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: the same MapReduce recipe could be trained end-to-end inside a single VLM using segment-level attention blocks, so dense local perception and global aggregation happen in one forward pass rather than through an agent loop; the paper notes this as future work.
  • Extension: because the authors identify caption quality as the upper bound, a closed-loop system that detects when a question needs details absent from captions and re-queries the VLM on those clips is a natural next step that could push accuracy further.
  • Extension: the principle should transfer to other long-form multimodal settings—audio, egocentric video, or multi-camera streams—where local details must be accumulated over a long horizon, though that transfer is not tested in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MR. Video, an agentic long-video understanding framework built on a 'MapReduce' principle: a Map stage independently and densely perceives short video clips (scene captioning, then per-question segment analysis), and a Reduce stage aggregates the resulting information into a consistent video-level representation and a final answer (character/object merging, global intention analysis, answer generation). The system is implemented with Gemini-2.0-Flash as the VLM and GPT-4o as the LLM, and it is evaluated on LVBench, LongVideoBench, Video-MME (long subset), and EgoSchema. The authors report 60.8% on LVBench and claim a >10% improvement over prior VLMs and video agents, with additional ablations on consistent characters, question intention analysis versus MM-Embed retrieval, and goal-aware analysis.

Significance. If the quantitative claims hold, the paper makes a strong conceptual and practical contribution: it proposes a simple and interpretable principle for long-video understanding, ships code and a complete prompt suite, and openly documents failure modes such as caption information loss. The reported results on LVBench and LongVideoBench are substantially higher than prior methods, and the MapReduce framing is attractive for scalable, sequence-parallel video agents. However, the current verification does not control for the much stronger LLM (GPT-4o), the much larger test-time compute budget, or the difference in evaluation question sets, so the central claim that the MapReduce structure itself is responsible for the improvement is not yet established. The paper's honest limitations section and its detailed failure analysis partially mitigate this, but the main comparison and ablation evidence need hardening.

major comments (4)
  1. [§4.3.2, Table 1] The headline comparison of MR. Video (60.8%) against prior VLMs (best 48.7%) and video agents (best 41.3%) is uncontrolled. MR. Video uses Gemini-2.0-Flash for visual perception and a separate GPT-4o LLM for reasoning, plus a dense captioning pass over the entire video and per-question VLM queries, whereas the listed baselines come from leaderboards or the VCA paper and use different backbones and much smaller test-time compute budgets. The only same-backbone baseline, Gemini-2.0-Flash with 256 uniformly sampled frames (48.6%), differs from MR. Video simultaneously in three variables: test-time compute, the presence of a separate LLM, and the MapReduce structure itself. The paper therefore cannot attribute the 12.2-point gap to the MapReduce principle on the basis of Table 1. The authors' own Fig. 6(a) shows that on the 98-question subset, replacing GPT-4o with Qwen-2.5-7B drops accuracy from 62.2% to 43.9% (18.3 points), a larger swing than any component ablation; this demonstrates that LLM choice is a major confound that must be controlled with a same-backbone non-MapReduce agent before the >10% claim can be accepted.
  2. [§4.1, §4.3.2, Appendix B.4] The main evaluation is not performed on the same question set as the baselines. Section 4.1 states that 4 of 103 LVBench videos are unavailable, leaving 1,492 questions, yet Section 4.3.2 refers to 'the complete set of LVBench' and Table 1 compares against leaderboard numbers presumably computed on the original full set. The manuscript is also internally inconsistent about the full size: Section 4.1 says LVBench curates 1,549 questions, while Appendix B.4 says 'we still have 1,492 out of 1,543 questions.' If the 4 missing videos contain 51–57 questions, the reported margin of >10% is measured against a different test set. The authors should either re-run available baselines on the same 1,492 questions or report how the margin changes if the removed videos are excluded from the baseline numbers; without this, the exact magnitude of the claimed improvement is not apples-to-apples.
  3. [§4.4, Fig. 6] The ablation study is based on only 98 questions from 6 videos, and no confidence intervals or significance tests are reported. The differences in Fig. 6(a) (62.2% vs. 59.2% for 'w/o Consistent Characters', 62.2% vs. 52.0% for 'w/o Goal-Aware Analysis') correspond to 3 and 10 questions respectively out of 98, which is within sampling noise for paired binary outcomes; a 3-question difference is effectively a tie. The key-frame recall comparison in Fig. 6(b) uses only 64 questions (as stated in Appendix B.5.2) and similarly lacks uncertainty quantification. More importantly, none of these ablations includes a same-backbone agent that uses sequential key-frame retrieval with the same VLM and LLM, so the experiments cannot isolate the MapReduce principle from the particular pipeline choices. The authors should either enlarge the ablation set, report binomial confidence intervals, or provide a direct controlled comparison with a comparable-compute non-MapReduce agent before claiming that the results validate the principle.
  4. [§3.4, §4.5.2, Conclusions] The paper itself acknowledges in Sec. 4.5.2 and in the Limitations section that 'the upper bound of MR. Video is limited by the quality of captions' and that information omitted by the VLM cannot be recovered by the Reduce stage. This is an honest caveat, but it directly qualifies the central claim that MapReduce is the principle for long video understanding. The conclusions currently state that the principle is validated without mentioning that it is only validated for questions whose required visual information survives the dense captioning step; for fine-grained detail questions such as the noodle example in Fig. 8, the method has a known failure mode. The paper should reword the conclusions to make this conditional explicit and should discuss the failure on detail-oriented questions in the main contribution rather than only in the supplementary failure analysis.
minor comments (4)
  1. [Fig. 6] The figure does not report the sample size; please add 'n=98' for (a) and 'n=64' for (b) in the captions or in the text, and ideally add error bars based on a binomial or paired estimate.
  2. [Appendix B.4 and Fig. 7] There are several typographical errors: 'Th six selected videos' should be 'The six selected videos' in Appendix B.4; 'shoty' should be 'shot' in Fig. 7; 'paritipating' should be 'participating' in Table C; 'previoous' should be 'previous' in Table J; and 'a captal letter' should be 'a capital letter' in Table J. These should be corrected.
  3. [§4.3.2, Table 2] The breadth comparison in Table 2 also relies on numbers from different sources for different backbones; for example, LongVideoBench accuracies for baselines are from the LongVideoBench paper rather than re-evaluated under the same conditions. Please state explicitly in the text that these are not controlled comparisons and consider adding the evaluation setup (e.g., number of frames, prompting) for each baseline.
  4. [§3.2.2 and Fig. 3] The captioning Reduce step merges repeated characters using VLM association, but the paper does not report the cost or failure rate of this association over the full LVBench set. Adding a brief quantitative statement (e.g., how often the merging step changes the final names) would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MR. Video reports empirical benchmark results, not a derivation that reduces to its own inputs.

full rationale

The paper's central claim is an empirical method comparison on LVBench and other benchmarks. MR. Video's accuracy of 60.8% is obtained by running an implemented pipeline (captioning, intention analysis, goal-aware analysis) on held-out benchmark videos and questions; it is not computed from a parameter fitted on the same data, and no quantity is renamed as a prediction. Hyperparameters such as the 32-scene segment size and the 32-frame context limit are stated as design choices, and the ablations are conducted on a separate 6-video, 98-question subset. The only overlapping-author citation ([60], used in related work on LLM agents) is not load-bearing: the MapReduce design and its validation do not depend on any theorem or result from that reference. Weaknesses such as uncontrolled baseline budgets and the acknowledged caption-quality upper bound are concerns about experimental control and information loss, not circularity. Therefore the derivation chain is self-contained with respect to the circularity criteria.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on benchmark validity, the fidelity of the text representation, and the comparability of leaderboard numbers. No new physical or mathematical entities are introduced. The hand-set hyperparameters listed above affect the results but are not reported as tuned on the target benchmark, and the pipeline depends on the assumption that captions and sampled frames do not lose information essential to the questions.

free parameters (4)
  • Intention-analysis segment size = 32 scenes per segment
    Chosen in Sec. 3.3.1 as 'each empirically containing 32 scenes'. This controls how many scene captions the LLM sees per Map call and affects the granularity and accuracy of the Reduce step.
  • VLM frame budget per query = fewer than 32 frames
    Set in Sec. 4.2 as a controlled implementation detail. It ensures MR. Video avoids context length limits, but it also caps the amount of visual detail the VLM can inspect for each goal-aware query.
  • Captioning sampling rates = 2 fps for 10 s clips; 0.25 fps for 2 min character-detection clips
    Footnotes in Sec. 3.2.1 specify 20 frames at 2 fps for scene splitting and 30 frames at 0.25 fps for character extraction. These rates determine caption density and character identification quality.
  • Baseline VLM frame counts = 256 frames for LVBench/LongVideoBench/Video-MME; 128 for EgoSchema
    Sec. B.5.1 states these counts follow the standard setting for Gemini-2.0-Flash. They define the comparison point for the 'over 10% improvement' claim.
assumptions (4)
  • domain assumption LVBench accuracy is a valid, sufficiently sensitive measure of long video understanding.
    The main claim is a 60.8 percent accuracy number on LVBench. The paper assumes the benchmark's question types and ground truth capture the capability it cares about, and that a ten-point gap on this benchmark is meaningful.
  • domain assumption The text captions and sparse frame samples produced by the VLM do not lose information needed to answer questions.
    The whole MapReduce pipeline reduces the video to captions and a few frames. The authors acknowledge in Sec. 4.5.2 that caption loss is an upper bound on performance, so this assumption is load-bearing.
  • domain assumption Published leaderboard accuracies are directly comparable to the paper's runs on a slightly different LVBench subset.
    MR. Video is evaluated on 1492 of 1543 LVBench questions (Sec. 4.1), while baseline numbers come from leaderboards computed on the full set. The comparison assumes the missing 4 videos do not change the ranking or the reported gaps.
  • domain assumption Zero-shot GPT-4o and Gemini-2.0-Flash are sufficient to instantiate the MapReduce principle.
    The paper explicitly validates the principle only with these off-the-shelf models and leaves VLM-native implementations to future work (Sec. 5 Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of MR. Video: "MapReduce" is the Principle for Long Video Understanding." pith.science (2026). https://pith.science/paper/6MUVKKY2

@misc{pith2026250416082,
  author       = {Pith},
  title        = {Pith review of: MR. Video: "MapReduce" is the Principle for Long Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6MUVKKY2}},
  note         = {Machine review of arXiv:2504.16082}
}
read the original abstract

We propose MR. Video, an agentic long video understanding framework that demonstrates the simple yet effective MapReduce principle for processing long videos: (1) Map: independently and densely perceiving short video clips, and (2) Reduce: jointly aggregating information from all clips. Compared with sequence-to-sequence vision-language models (VLMs), MR. Video performs detailed short video perception without being limited by context length. Compared with existing video agents that typically rely on sequential key segment selection, the Map operation enables simpler and more scalable sequence parallel perception of short video segments. Its Reduce step allows for more comprehensive context aggregation and reasoning, surpassing explicit key segment retrieval. This MapReduce principle is applicable to both VLMs and video agents, and we use LLM agents to validate its effectiveness. In practice, MR. Video employs two MapReduce stages: (A) Captioning: generating captions for short video clips (map), then standardizing repeated characters and objects into shared names (reduce); (B) Analysis: for each user question, analyzing relevant information from individual short videos (map), and integrating them into a final answer (reduce). MR. Video achieves over 10% accuracy improvement on the challenging LVBench compared to state-of-the-art VLMs and video agents. Code is available at: https://github.com/ziqipang/MR-Video

Figures

Figures reproduced from arXiv: 2504.16082 by the authors.

Figure 1
Figure 1. MapReduce Principle. Long video understanding re￾quires both global comprehension and detailed perception, as in the motivating example. For such needs, (a) sequence-to-sequence VLMs and (b) video agents are sub-optimal in terms of context lengths, sequential parallelization, and global context information. (c) Instead, we explore and develop a simple “MapReduce” prin￾ciple via MR. Video and (d) overcome such challe… view at source ↗
Figure 2
Figure 2. Overview. MR. Video validates the effectiveness of “MapReduce” principle with an LLM agent framework. We demonstrate two distinct types of questions for visual details and reasoning. (a) Captioning (Sec. 3.2) first generates detailed captions of individual scenes (Map) and then enhances consistency by merging repeated characters/objects for the scenes (Reduce). (b) Question Intention Analysis (Sec. 3.3) investigates… view at source ↗
Figure 3
Figure 3. Key Characters/Objects in Captioning. (a) The “map” step extracts the salient characters/objects along with a descrip￾tion, which is useful for frames with multiple characters (the 3rd frame). (b) Then, the “reduce” step uses VLM to associate the re￾peated characters, enhancing the consistency of captions. to their sequence-parallel generation, such as “one name be￾longing to two characters in different clips” or “o… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Customized Queries for Perception. With this ques￾tion requiring detailed visual perception, MR. Video proposes objective-aware queries for the VLMs, confirming the criteria. tially condenses or merges the candidate scenes listed in the “map” step according to both vid…
Figure 6
Figure 6. Figure 6: Analytical Experiments of MR. Video. (a) We inves￾tigate the benefits of MR. Video components and the influence of LLMs. (b) The comparison between our question intention anal￾ysis and key frame retrieval suggests the necessity of combining more video contexts for unde…
Figure 7
Figure 7. Figure 7: Referring back to the motivating example ( [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

148 extracted references · 50 canonical work pages

  1. [1]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Qwen2.5-VL technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. 6

    arXiv 2025

  3. [3]

    AuroraCap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Sain- ing Xie, and Christopher D Manning. AuroraCap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024. 2

    arXiv

  4. [4]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV, 2024. 2

    2024

  5. [5]

    Timemarker: A versatile video-llm for long and short video understanding with superior temporal localiza- tion ability

    Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-llm for long and short video understanding with superior temporal localiza- tion ability. arXiv preprint arXiv:2411.18211, 2024. 6

    arXiv 2024

  6. [6]

    InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024. 6

    2024

  7. [7]

    MapReduce: simplified data processing on large clusters

    Jeffrey Dean and Sanjay Ghemawat. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1):107–113, 2008. 1

    2008

  8. [8]

    VideoAgent: A memory-augmented mul- timodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. VideoAgent: A memory-augmented mul- timodal agent for video understanding. In ECCV, 2024. 1, 2, 4, 5, 6

    2024

Show all 148 references
  1. [9]

    Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. arXiv preprint arXiv:2405.21075, 2024. 5, 12, 13

    2024 arXiv

  2. [10]

    ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools. arXiv preprint arXiv:2406.12793, 2024. 6

    2024 arXiv

  3. [11]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In CVPR, 2023. 5

    2023

  4. [12]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In ICML, 2021. 2

    2021

  5. [13]

    Perceiver IO: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Kop- pula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver IO: A general architecture for structured inputs & outputs. In ICML, 2022. 2

    2022

  6. [14]

    SWE- Bench: Can language models resolve real-world github is- sues? In ICLR, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE- Bench: Can language models resolve real-world github is- sues? In ICLR, 2024. 2

    2024

  7. [15]

    Llava-OneVision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1

    2024 arXiv

  8. [16]

    LLaV A-NeXT- Interleave: Tackling multi-image, video, and 3d in large mul- timodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. LLaV A-NeXT- Interleave: Tackling multi-image, video, and 3d in large mul- timodal models. arXiv preprint arXiv:2407.07895, 2024. 1

    2024 arXiv

  9. [17]

    BLIP: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. InICML,

  10. [18]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  11. [19]

    VideoChat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. VideoChat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2

    2023 arXiv

  12. [20]

    Temporal preference optimization for long- form video understanding

    Rui Li, Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Ser- ena Yeung-Levy. Temporal preference optimization for long- form video understanding. arXiv preprint arXiv:2501.13919, 2025

    2025 arXiv

  13. [21]

    VideoChat-Flash: Hierarchical compression for long-context video modeling

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, and Limin Wang. VideoChat-Flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574, 2024. 2, 6

    2024 arXiv

  14. [22]

    Llama-VID: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-VID: An image is worth 2 tokens in large language models. In ECCV,

  15. [23]

    Video-LLaV A: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual rep- resentation by alignment before projection. In ACL, 2024. 1, 2

    2024

  16. [24]

    MM-Embed: Universal multimodal retrieval with multimodal LLMs

    Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-Embed: Universal multimodal retrieval with multimodal LLMs. In ICLR, 2025. 7, 14

    2025

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1

    2023 arXiv

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 2

    2023

  19. [27]

    LLaV A-NeXT: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-NeXT: Im- proved reasoning, ocr, and world knowledge, 2024. 1

    2024

  20. [28]

    NVILA: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yux- ian Gu, Dacheng Li, et al. NVILA: Efficient frontier visual language models. arXiv preprint arXiv:2412.04468, 2024. 2 9

    2024 arXiv

  21. [29]

    Oryx MLLM: On-demand spatial- temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx MLLM: On-demand spatial- temporal understanding at arbitrary resolution. In ICLR,

  22. [30]

    EgoSchema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A diagnostic benchmark for very long- form video language understanding. In NeurIPS, 2023. 5, 13

    2023

  23. [31]

    TimeChat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. TimeChat: A time-sensitive multimodal large language model for long video understanding. In CVPR, 2024. 2

    2024

  24. [32]

    LLaV A-PruMerge: Adaptive token reduc- tion for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaV A-PruMerge: Adaptive token reduc- tion for efficient large multimodal models. arXiv preprint arXiv:2403.15388, 2024

    2024

  25. [33]

    LongVU: Spa- tiotemporal adaptive compression for long video-language understanding

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Bal- akrishnan Varadarajan, Florian Bordes, et al. LongVU: Spa- tiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 2024. 2

    arXiv

  26. [34]

    MovieChat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. MovieChat: From dense token to sparse memory for long video understanding. In CVPR,

  27. [35]

    ViperGPT: Visual inference via python execution for reasoning

    D ´ıdac Sur´ıs, Sachit Menon, and Carl V ondrick. ViperGPT: Visual inference via python execution for reasoning. In ICCV, 2023. 5

    2023

  28. [36]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 6

    2023 arXiv

  29. [37]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 6

    2024 arXiv

  30. [38]

    LVBench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiao- han Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. LVBench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024. 2, 3, 5, 6, 12, 13

    2024 arXiv

  31. [39]

    ReTaKe: Reducing temporal and knowledge redundancy for long video understanding

    Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. ReTaKe: Reducing temporal and knowledge redundancy for long video understanding. arXiv preprint arXiv:2412.20504, 2024. 2, 6

    2024 arXiv

  32. [40]

    LongLLaV A: Scaling multi-modal LLMs to 1000 images efficiently via a hybrid architecture

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. LongLLaV A: Scaling multi-modal LLMs to 1000 images efficiently via a hybrid architecture. arXiv preprint arXiv:2409.02889, 2024. 2

    2024

  33. [41]

    VideoAgent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. VideoAgent: Long-form video understanding with large language model as agent. In ECCV, 2024. 1, 2, 4, 5, 6, 7, 14

    2024

  34. [42]

    InternVideo2.5: Empowering video MLLMs with long and rich context modeling.arXiv preprint arXiv:2501.12386, 2025

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xi- angyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. InternVideo2.5: Empowering video MLLMs with long and rich context modeling.arXiv preprint arXiv:2501.12386, 2025. 2

    2025 arXiv

  35. [43]

    VideoTree: Adaptive tree-based video represen- tation for LLM reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jae- hong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. VideoTree: Adaptive tree-based video represen- tation for LLM reasoning on long videos. arXiv preprint arXiv:2405.19209, 2024. 2, 6

    2024 arXiv

  36. [44]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. In NeurIPS, 2022. 2

    2022

  37. [45]

    LongVLM: Efficient long video understand- ing via large language models

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. LongVLM: Efficient long video understand- ing via large language models. In ECCV, 2024. 2

    2024

  38. [46]

    Longvideobench: A benchmark for long-context interleaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In NeurIPS, 2024. 5, 6, 12, 13

    2024

  39. [47]

    Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy re- duction

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy re- duction. In CVPR, 2025. 2

    2025

  40. [48]

    Slowfast-llava: A strong training-free base- line for video large language models

    Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free base- line for video large language models. arXiv preprint arXiv:2407.15841, 2024. 2

    2024 arXiv

  41. [49]

    LongVILA: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. LongVILA: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 2

    2024 arXiv

  42. [50]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. 2

    2024 arXiv

  43. [51]

    SWE- Agent: Agent-computer interfaces enable automated soft- ware engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE- Agent: Agent-computer interfaces enable automated soft- ware engineering. In NeurIPS, 2025. 2

    2025

  44. [52]

    HotpotQA: A dataset for diverse, ex- plainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. HotpotQA: A dataset for diverse, ex- plainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. 2

    2018 arXiv

  45. [53]

    VCA: Video curious agent for long video un- derstanding

    Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. VCA: Video curious agent for long video un- derstanding. arXiv preprint arXiv:2412.10471, 2024. 1, 2, 5, 6, 13

    2024 arXiv

  46. [54]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023. 2

    2023

  47. [55]

    mPLUG- OWL3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mPLUG- OWL3: Towards long image-sequence understanding in multi-modal large language models. In ICLR, 2024. 6 10

    2024

  48. [57]

    VideoLLaMA 3: Frontier mul- timodal foundation models for image and video understand- ing

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. VideoLLaMA 3: Frontier mul- timodal foundation models for image and video understand- ing. arXiv preprint arXiv:2501.13106, 2025. 6

    2025 arXiv

  49. [58]

    A simple LLM framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple LLM framework for long-range video question-answering. In EMNLP, 2024. 2

    2024

  50. [59]

    LLaV A-NeXT: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, Haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. LLaV A-NeXT: A strong zero-shot video understanding model, 2024. 1, 2

    2024

  51. [60]

    Language agent tree search unifies reasoning acting and planning in language models

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Hao- han Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. In ICML, 2024. 2

    2024

  52. [61]

    MLVU: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264,

    arXiv

  53. [62]

    MapReduce

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An explo- ration of video understanding in large multimodal models. arXiv preprint arXiv:2412.10360, 2024. 2 11 MR. Vi...

    2024 arXiv

  54. [63]

    Scene Merging

    We split each 10s video segment into individual scenes and check if the first scene of a segment can be merged with the last frame of the previous segment. Scene split- ting prompts are in Table A, and the “Scene Merging” prompts are in Table B

  55. [64]

    The prompts are in Table C

    We identify the salient characters and use them to gen- erate the dense captions in each video segment. The prompts are in Table C

  56. [65]

    Reduce: Consistent Characters and Objects

    With the selected characters, we generate the dense cap- tions of each scene with the prompts in Table D. Reduce: Consistent Characters and Objects. As in Sec. 3.2.1, our additional “Reduce” step enhances con- sistency by merging the repeated characters into unified names. It ...

  57. [66]

    We iteratively check if the characters from two video segments overlap with the prompts in Table E

  58. [67]

    After assigning new names to all the characters/objects, we modify the old names in the original dense captions with the prompts of Table F. B.2. Analysis I Prompts We describe the prompts for question intention analysis (Sec. 3.3). Map: Segment Intention Analysis. We let a st...

    2025

  59. [68]

    [1. Description]:

    Based on the provided frames of this video segment, please describe the contents of the video segment briefly and accurately. You should cover each action and event in the clip. The description should be detailed, faithful, and accurate. It should come with a header: "[1. Desc...

  60. [69]

    Is this video segment a single scene or a combination of multiple scenes?

    Based on your description, please answer the following question: "Is this video segment a single scene or a combination of multiple scenes?" The definition of a scene is a single, self-contained, and continuous event that could be easily summarized into one sentence by a human...

  61. [70]

    no", please provide the index of frame(s) separating the scenes from the given frame. Your answer should come with a header:

    If the answer to the previous question is "no", please provide the index of frame(s) separating the scenes from the given frame. Your answer should come with a header: "[3. Frames]:" and in the format of a list of integers. ## Example Your response should be in the following f...

  62. [71]

    Try to be rigorous and faithful to the video without making assumptions

    NAME should be a a general name, such as person a, person b, person c, object a, etc. Try to be rigorous and faithful to the video without making assumptions

  63. [72]

    DESCRIPTION should be a short description of the character’s and object’s appearance and properties, especially how to uniquely identify the character or object from the representative frame

  64. [73]

    person a

    FRAME should be the index of the frame that best represents the character or object in the scene, favorably the most salient frame showing the front face of the character. It should start from 0. ## 2.3 Example Output: [1. Appeared Characters]: ["person a", "person b", "dog a"...

  65. [74]

    Please imagine yourself as a human watching the video, trying to perceive the salient things from the video and understanding the deeper plots of the video

  66. [75]

    It could be a person in the movie, an animal in the documentary or cartoon, etc

    When you are selecting the characters for the visual memory, please be picky: (a) Only select the characters and objects that you believe are salient and could significantly influence the plot. It could be a person in the movie, an animal in the documentary or cartoon, etc. Ma...

  67. [76]

    Please keep the strings in identical formattings to ensure smooth post-processing

    Format is very important. Please keep the strings in identical formattings to ensure smooth post-processing

  68. [77]

    Character Selection

    Please make sure the [1. Appeared Characters] and [2. Character Details] are consistent. # 4. Your Job Now your job begins. Table C. “Character Selection” prompts for the Captioning stage (Sec. B.1) 15 # 1. Instructions You will be given a few continuous screenshots of a video...

  69. [78]

    Visual Memory: the names, representative video frames, and the identifiable properties of the characters and objects in the visual memory

  70. [79]

    Previous Caption: the caption of the previous scene

  71. [80]

    Video Frames: the few continuous screenshots of the current video clip. # 2. Guidelines When generating the caption, please follow the guidelines below and solve this problem step by step:

  72. [81]

    First, describe the main content of the current scene briefly

  73. [82]

    If so, please list their name out

    Second, use the visual memory to identify if any characters or objects from the visual memory appear in the current scene. If so, please list their name out

  74. [83]

    Some more detailed tips:

    Third, describe the scene in detail, including the characters, their actions, the objects, the properties of the characters and objects, the environment, and other types of contents, etc. Some more detailed tips:

  75. [84]

    The goal is that a human should read your captions and feel like watching a continuous video

    When generating the captions, please take the previous scene as contexts and pretend that you are watching a video continuously. The goal is that a human should read your captions and feel like watching a continuous video

  76. [85]

    When generating the captions, please be faithful to the video and make logical connections between the scenes

  77. [86]

    person a

    When you encounter characters, please utilize the information and name from the visual memory if what you see matches the visual memory. For instance, if the visual memory contains a character named "person a", you should use <person a> to refer to the character in your captio...

  78. [87]

    The quality of this step is very very very important

  79. [88]

    Dense Captioning

    I want you to be very detailed and faithful to the video. At least, you should go over the following aspects: 2.1 What are the characters, what are their appearances, what are there clothes, what are their actions, what are their emotions? 2.2 What are the objects, what are th...

  80. [89]

    Set 1: the names, representative video frames, and the identifiable properties of the characters and objects

  81. [90]

    Set 2: the names, representative video frames, and the identifiable properties of the characters and objects. # 2. Guidelines and Tips This is not an easy task, please make sure to use your advanced reasoning ability and check every item and step carefully. The following guide...

  82. [91]

    Please work on this problem via two steps: (a) check if any items from the first set is repeated with the second set; (b) if so, find the better frame representing the character or object

  83. [92]

    Please rely on both the video frame information and the identifiable properties to carefully understand the characters and objects

  84. [93]

    When you are selecting the better frame for an object, please consider the following factors: (a) the frame should be the most salient frame showing the front face of the character; (b) the frame should be the most representative frame showing the character or object

  85. [94]

    Sometimes the characters or objects are captured from different angles or distances, please make your best judgement to check if they are the same character or object. # 3. Output Format Please strictly follow the format below to ensure smooth post-processing: [Repeated Charac...

  86. [95]

    Character name in Set 1: the name of the character or object in the first set of frames

  87. [96]

    Character name in Set 2: the name of the character or object in the second set of frames

  88. [97]

    Character Merging

    Better character name: the name of the better character or object that represents the repeated character or object, must be consistent with the name in Character name in Set 1 or Character name in Set 2. An example output should be: [Repeated Characters and Objects]: (person a...

  89. [98]

    Old Description: the old description of the video clip, containing the fields of Brief Description, Appeared Characters, and Detailed Description

  90. [99]

    Output: Your output should be the modified description of the video clip strictly following the original format and contents, only with names changed

    Modified List: a list of characters to be modified in the format of OLD NAME -> NEW Name. Output: Your output should be the modified description of the video clip strictly following the original format and contents, only with names changed. # 2. Guidelines

  91. [100]

    Only change the names, do not change the format or any contents

  92. [101]

    Please remember to update all the Brief Description, Appeared Characters, and Detailed Description

  93. [102]

    Keep the names consistent

  94. [103]

    Caption Modification

    The format of the characters in Brief and Detailed Description is <NAME>, please follow the same format. # 3. Your Job Now your job begins. Table F. “Caption Modification” prompts for the Captioning stage (Sec. B.1) 17 # 1. Motivation You will conduct the first step of long vi...

  95. [104]

    The frames sampled from the video, each corresponding to a scene in the captions

  96. [105]

    Notably, we have marked the potentially key characters or objects using the format of <NAME>

    The captions of the video generated by a video captioning model, decomposed into short scenes representing different video actions. Notably, we have marked the potentially key characters or objects using the format of <NAME>. However, it is not entirely reliable (e.g., missing...

  97. [106]

    The question: raised by the user, the most important and reliable

  98. [107]

    Video frames: reliable, but only covers a small portion of the video

  99. [108]

    You should combine the information from the question and video frames when using the captions

    Captions: less reliable, but covering more details, especially the "NAME" representing character/object names. You should combine the information from the question and video frames when using the captions. ## Analysis Tips

  100. [111]

    If the question asks for certain characters in the plot/story, you should potentially localize its NAME in the captions, or clearly specify its appearance properties

  101. [112]

    Pay attention to the information reliability mentioned above

  102. [113]

    Imagine yourself watching a video using the sampled frames and the captions

  103. [114]

    When discussing your analysis, please provide the reasoning process and your confidence level between 1 (almost guessing, no clear evidence of being relevant to the question) to 5 (almost certain, clear evidence of being relevant to the question)

  104. [115]

    Relevant Segments

    If the question should be answered with contexts, for "Relevant Segments", you should include one more scene before and after the most possible scene to increase robustness. For example, if the most possible segment is (10, 20), and its previous and next scenes are (5, 10) and...

  105. [116]

    The question: a question coming with options

  106. [117]

    The frames: a list of frames sampled from the video

  107. [118]

    (t start, t end): caption

    The captions: a list of captions decomposed into short scenes representing different video actions. Each caption is the format of "(t start, t end): caption". Time is represented in seconds. # 4. Your Job Starts Table G. “Segment Intention Analysis” prompts for the Question In...

  108. [119]

    You will go through the analysis of each segment containing the following parts:

    Your analysis of short video segments: **is the video segment relevant to the question? ** Your analysis is the most important information in this step. You will go through the analysis of each segment containing the following parts:

  109. [120]

    (your explanation)

    Reasoning: ... (your explanation)

  110. [122]

    (Your confidence level.)

    Confidence Level: ... (Your confidence level.)

  111. [123]

    relevant segments

    Key Characters: [(character symnonym in question, identifiable properties or NAME in captions), ...]... (The key characters that are mentioned in the question and how to identify them. Could be unreliable.) # 2. Instructions and Guidelines ## Objectives Your goal is to merge t...

  112. [124]

    You should select the most relevant segments and characters based on a video-level understanding, and ignore the less relevant ones

    Segment-level analysis might guess some relevant segments or characters for the question. You should select the most relevant segments and characters based on a video-level understanding, and ignore the less relevant ones

  113. [125]

    You should carefully merge the information from different segments, and provide reliable information for the downstream analyses steps

    Segment-level analysis might contain contradicting information since they come from separate analyses. You should carefully merge the information from different segments, and provide reliable information for the downstream analyses steps

  114. [126]

    sum", "merge

    You should clarify how the results from segment-level can contribute to the long video understanding. For example, do we want to "sum", "merge", or "select" the information from individual segments. ## Output Formats [1. Reasoning]: ... (Your reasoning process. Please be preci...

  115. [128]

    If the user mentions a clear criteria, such as specific character of object, try to use it **precisely** and **rigorously** in your analysis

    Finding the key video segment is critical. If the user mentions a clear criteria, such as specific character of object, try to use it **precisely** and **rigorously** in your analysis

  116. [129]

    If the question asks for certain characters in the plot/story, you should potentially localize its <NAME> in the captions, or clearly specify its appearance properties

  117. [130]

    Figuring out the flow of the plots is critical

    Imagine yourself watching a video using the sampled analysis. Figuring out the flow of the plots is critical

  118. [131]

    If the question is not really about the **whole video **, do not specify more than 10 relevant segments

  119. [132]

    I have low confidence on the relevance of the segments

    You should propose **at least 1 relevant segment **. If you don’t think any segment is relevant, return a most likely segment and say "I have low confidence on the relevance of the segments". # 3. Your Job Starts Table H. “Global Intention Analysis” prompts for the Question In...

  120. [133]

    It contains the following information:

    A video-level analysis from yourself. It contains the following information:

  121. [134]

    (Your explanation about which parts of the video are relevant to the question.)

    Reasoning: ... (Your explanation about which parts of the video are relevant to the question.)

  122. [135]

    (The periods that are potentially relevant from your analysis

    Relevant Segments: [(t start, t end), ...]... (The periods that are potentially relevant from your analysis. Time is represented in seconds.)

  123. [136]

    (The key characters that are mentioned in the question and how to identify them

    Key Characters: [(character symnonym in question, identifiable properties or NAME in captions), ...]... (The key characters that are mentioned in the question and how to identify them. Keep the list empty if the question is not related to any characters.)

  124. [137]

    yes", then this is a global question. If

    Local or Global: ... (Whether the question requires combining contexts from different segments to answer. If "yes", then this is a global question. If "no", then this is a local question.) # 2. Instructions and Guidelines ## Objectives When thinking about the questions to ask,...

  125. [138]

    In this way, the vision-language models can use your question to check the details of each segment

    Local: Sample N video frames for each relevant segment, e.g., 32 frames. In this way, the vision-language models can use your question to check the details of each segment

  126. [139]

    In this way, the vision-language models can use your question to check the flow of the plots or conduct reasoning over a longer period of time

    Global: Sample 1 video frame for each segment, sequentially. In this way, the vision-language models can use your question to check the flow of the plots or conduct reasoning over a longer period of time. Therefore, you should propose two questions:

  127. [140]

    A local question: what kind of detailed information or evidence should the vision-language models find in each segment?

  128. [141]

    Reasoning]:

    A global question: what kind of reasoning should the vision-language models conduct on a longer time span? ## Output Formats Please strictly follow the output formats below to propose your questions, so that the downstream parts can easily extract the information: [1. Reasonin...

  129. [143]

    Do not enumerate or explicitly depending on any time information

    Keep your question concise, clear, and within a few sentences. Do not enumerate or explicitly depending on any time information

  130. [144]

    It is also important to include the original options as the context for the vision-language models

    Remember to use the options from the original questions, expressed with (A), (B), (C), (D), to think about the best way to distinguish the correct one. It is also important to include the original options as the context for the vision-language models

  131. [145]

    Use your knowledge of prompting large language models or vision-language models to improve your question

  132. [146]

    Describe ... (A) ..., (B) ..., (C) ..., (D) ..., (E)

    Your output questions should only contain a question and options. Do not include any analyses, speculations, or reasoning into the question. For example, the question should directly start as "Describe ... (A) ..., (B) ..., (C) ..., (D) ..., (E) ...", "What is ... (A) ..., (B)...

  133. [147]

    You also think about how each video segment could contribute to the long video understanding

    Video-level User Intention Analysis: You first analyze which parts of the video and what kind of characters are relevant to the user’s question. You also think about how each video segment could contribute to the long video understanding

  134. [148]

    local question

    Goal Proposal: To call vision-language models to analyze the video segments, you have proposed two questions for the VLMs to use. The first question is called "local question", used for detailed analysis for each segment, and the second question is called "global question", us...

  135. [149]

    Goal-aware Analysis: You will receive the results of the vision-language models’ perception for each video segment using the local question and across multiple segments using the global question. By understanding the previous steps, you will have a good understanding of the me...

  136. [150]

    Some examples are: - For question on visual details, you should check if the video segment **contains the scene that the user wants **

    Think carefully about how a short video segment could contribute to long video understanding by paying attention to the question and video segment contents. Some examples are: - For question on visual details, you should check if the video segment **contains the scene that the...

  137. [151]

    Carefully consider whether the analysis at local segments or across multiple segments is more important for answering the user’s question

  138. [152]

    Figuring out the flow of the plots is critical

    With the information provided to you, imagine youself as a human watching the video. Figuring out the flow of the plots is critical

  139. [153]

    You should utilize advanced reasoning skills to resolve the contradictions

    It is possible that some information is vague or contradicting each other. You should utilize advanced reasoning skills to resolve the contradictions. Some very useful principles are: - If the user has mentioned a specific criteria, try to use it **precisely** and **rigorously...

  140. [154]

    Answer Generation

    Pay attention to the time information. They might provide additional correspondence information across different segments and analyses. # 3. Output Format Please provide your answer in the following format: [1. Reasoning]: ... (Your advanced reasoning based on the information ...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.