Pith. sign in

REVIEW 3 major objections 4 minor 44 references

VideoTreeSearch claims that grounding evidence in long videos is best done as self-correcting navigation through a tree of scene segments, not continuous timestamp cropping.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:04 UTC pith:XCC2NAKL

load-bearing objection The tree-navigation design is genuinely new and the ablations support it, but the headline CG-Bench comparison is likely apples-to-oranges and must be fixed before the +12.5 mIoU claim is trusted. the 3 major comments →

arxiv 2607.16189 v1 pith:XCC2NAKL submitted 2026-07-17 cs.CV

Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

Ce Zhang , Ziyang Wang , Yulu Pan , Oluwatumininu Oguntola , Pranav Wagh , Qiyu Wu , Hiromi Wakaki , Mohit Bansal
show 1 more author
Gedas Bertasius
This is my paper
classification cs.CV
keywords grounded long-video QAtemporal groundingagentic video searchhierarchical video treeself-correctionbacktrackingreinforcement learningtrajectory synthesis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper takes aim at a structural flaw in agentic long-video QA: agents that repeatedly call a continuous crop_video action can narrow toward a region but have no built-in way to abandon a wrong region and search elsewhere. The authors propose VideoTreeSearch (VTS), which organizes the video into a tree of semantically coherent segments and trains an agent to navigate it with four discrete actions—zoom_in, zoom_out, shift, and answer—making backtracking an explicit, learnable move rather than an implicit re-cropping. To teach recovery, they synthesize training trajectories that deliberately wander into wrong branches and then escape, then use those trajectories for supervised fine-tuning followed by reinforcement learning. VTS outperforms prior agentic methods on three grounded long-video QA benchmarks, with the largest gains on the longest videos, and the policy also transfers to general long-video QA. If search over a structured tree is the right inductive bias, this points to a scalable way to answer evidence-seeking questions about hour-long videos without burning through frames.

Core claim

The central claim is that grounded long-video QA reduces to iterative self-correcting search over an adaptive temporal tree, and that the right search primitives are discrete and hierarchical. VTS builds the tree from CLIP-based scene boundaries so that children of a node partition a semantically coherent segment, then trains an agent to choose among zoom_in (descend to child), zoom_out (ascend to parent), shift (move to sibling), and answer (commit). The training signal is a trajectory synthesis pipeline that deliberately includes wrong-branch detours followed by recovery, first imitated via supervised fine-tuning and then refined by reinforcement learning with temporal-IoU and answer-accur

What carries the argument

The load-bearing object is the adaptive temporal tree: a non-uniform hierarchy built by recursively splitting video segments at CLIP cosine-distance peaks, so each node corresponds to a semantically coherent interval. Navigation is carried by four discrete, learnable actions—zoom_in, zoom_out, shift, answer—which separate exploration (zoom_in) from correction (zoom_out/shift) and turn precise localization into node selection. The trajectory synthesis pipeline generates training paths containing deliberate detours (wrong zoom_in followed by forced recovery), and the selective SFT loss teaches the recovery moves while masking the deliberate mistakes; GRPO then reinforces grounding and accuracy

Load-bearing premise

The reported gains depend on the filtered 1,176-question CG-Bench subset being a fair testbed—previous baselines must have been scored on the exact same filtered questions—and on CLIP scene boundaries being reliable enough to form meaningful tree nodes, which the paper itself notes is weak for visually homogeneous video.

What would settle it

Re-run every previous cropping agent on the exact same 1,176-question filtered CG-Bench subset and check whether the +12.5 mIoU gap persists; if the baseline numbers were taken from the unfiltered 3,000-question mini split, the comparison is invalid. Alternatively, train VTS on the full unfiltered mini split and see whether the gains hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If VTS is right, long-video QA agents should expose backtracking as a first-class action instead of approximating correction with repeated continuous crops.
  • The navigation policy transfers to general long-video QA (Video-MME, MLVU, LVBench) without grounding-specific training, suggesting tree search is a broadly useful inductive bias.
  • VTS achieves higher grounding accuracy while processing fewer frames than uniform sampling or captioner-LLM baselines, so hierarchical search is not bought with extra compute.
  • The benefit grows with search horizon: the largest gains occur on the longest benchmark (Haystack-Ego4D), implying that explicit recovery matters most when a wrong descent is costliest.
  • Synthetic timestamped QA from unlabeled YouTube videos alone beats prior agentic baselines, suggesting detour-and-recovery trajectories can be produced at scale without hand annotation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment: vary the detour length and forced-recovery rate in the trajectory synthesizer; the paper's hypothesis predicts that recovery frequency at test time tracks recovery exposure in training.
  • If the tree were built from learned or multimodal scene boundaries rather than CLIP embeddings, performance on visually homogeneous videos—which the authors flag as a limitation—might improve further.
  • The discrete action space makes human intervention possible mid-search: a user could push the agent out of a wrong branch with an explicit zoom_out/shift, something continuous-cropping agents make awkward.
  • The single-interval assumption suggests a path toward multi-clue questions: let the agent collect evidence from several branches before answering, an extension the authors explicitly leave for future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces VideoTreeSearch (VTS), an agentic framework for grounded long-video question answering. VTS first organizes a video into a non-uniform temporal tree using CLIP-based scene boundaries, then trains a Qwen3-VL-8B agent to navigate this tree with four discrete actions: zoom_in, zoom_out, shift, and answer. The training pipeline synthesizes trajectories that include deliberate wrong-branch detours followed by recovery, and combines supervised fine-tuning with GRPO-style reinforcement learning using answer-accuracy and grounding rewards. The paper reports strong gains over prior agentic baselines on CG-Bench, Haystack-LVBench, Haystack-Ego4D, and also reports transfer results on Video-MME, MLVU, and LVBench. The central claim is that explicit hierarchical search with backtracking primitives is a better inductive bias for long-video grounding than continuous-crop agentic methods.

Significance. If the results hold, the paper makes a useful contribution: it provides a concrete action space that separates exploration from correction, a trajectory-synthesis method that produces detour-and-recovery supervision, and controlled ablations showing that the tree structure, backtracking, and recovery trajectories each contribute to performance. The paper also reports a matched-backbone comparison (Table 6) and releases code, which strengthens reproducibility. However, the headline CG-Bench result is currently difficult to interpret because the evaluation uses a new 1,176-question filtered subset (Appendix B) without a clear statement that the Table 1 baselines were re-run on that same subset. In addition, the filtering pipeline is built with Qwen3-VL-8B-Instruct, the same model family as VTS, which introduces a potential selection bias. These issues are load-bearing for the external comparison, although the internal ablations in Tables 3–4 are informative and support the mechanism.

major comments (3)
  1. [§4 / §5.1 / Appendix B / Table 1] The CG-Bench comparison in Table 1 is not clearly like-for-like. The paper states in §4 that 'we apply a filtering procedure for CG-Bench' and Appendix B describes a four-stage filter that reduces the official 3,000-question mini split to 1,176 questions. However, the paper never states that the prior-agent baselines (LongVT, Video-o3, VideoZoomer, TimeSearch-R) were re-run on this filtered subset. The reported baseline values (e.g., LongVT 4.3 mIoU, VideoZoomer 3.0, Video-o3 2.9) appear to match published full-mini numbers. If that is the case, the abstract's claim of '+12.5 mIoU on CG-Bench' is not a comparison on the same test set. Please either run all baselines on the filtered subset, or report numbers on the official 3,000-question mini split, and indicate explicitly in Table 1 which subset each column refers to.
  2. [Appendix B / §3.4] The filtering procedure creates a selection bias that may favor VTS. Filters 2, 3, and 4 use Qwen3-VL-8B-Instruct to determine video dependency, clue sufficiency, and clue uniqueness, and the trajectory-synthesis controller in §3.4 uses Qwen3-VL-8B for relevance scoring. VTS uses Qwen3-VL-8B as its base model. Thus the 1,176-question subset may preferentially retain questions on which Qwen3-VL is already strong, while several baselines in Table 1 use Qwen2.5-VL backbones. The paper should quantify the filter's effect per backbone (e.g., survival rates by base model) and ideally report results on the unfiltered official mini split as a robustness check.
  3. [Appendix C.2] There is a potential train/evaluation overlap on CG-Bench. The trajectory-synthesis training data is described as 'the CG-Bench full set (excluding questions from the mini set).' If the mini split is a subset of the full set's questions drawn from the same videos, then the model may have seen the same videos during training even though the specific evaluation questions were excluded. This would give VTS an advantage on CG-Bench that is not attributable to the tree-search mechanism. Please clarify whether any video appearing in the CG-Bench mini split appears in the training set, or re-train with a video-disjoint split.
minor comments (4)
  1. [Table 3] In the row 'Tree-grounded, flat tree (no hierarchy)', the Haystack-Ego4D entry '18.962.9' is missing a separator; it should likely read '18.9 62.9'.
  2. [Appendix D / Figure 9 caption] Typo in the caption: 'hierachical' should be 'hierarchical'.
  3. [§6 / Appendix D] The sentence 'Figures 8–9 present two additional qualitative results' is duplicated at the beginning of Appendix D.
  4. [§5.2] The self-correction rate is defined as the fraction of trajectories containing any zoom_out or shift. It would be more informative to distinguish planned lateral moves from true corrections, e.g., by counting only actions taken after a wrong-branch state. As written, the 60% figure may overstate the role of recovery.

Circularity Check

1 steps flagged

CG-Bench comparison is partially self-referential: the evaluation subset is filtered by Qwen3-VL-8B-Instruct and VTS is instantiated with Qwen3-VL-8B, so the headline +12.5 mIoU gain over Qwen2.5-based baselines is not cleanly independent of the base model; the tree-search mechanism still has independent support from ablations and matched-backbone results.

specific steps
  1. other [Appendix B ('CG-Bench-mini (filtered)'); §3.6 Implementation Details]
    "All VLM-based filters use Qwen3-VL-8B-Instruct served via vLLM with temperature 0. The final filtered set contains 1,176 questions... // We instantiate VTS with Qwen3-VL-8B [1] as the base model."

    The CG-Bench evaluation set is not the official 3,000-question mini split but a 1,176-question subset selected by four filters, of which Filters 2-4 use Qwen3-VL-8B-Instruct to discard questions answerable without the clue or not answerable from the clue. VTS is built on Qwen3-VL-8B, the same model family. Therefore the benchmark used for the headline +12.5 mIoU claim is partly constructed from the base model's own clue-following behavior; the comparison against Qwen2.5-VL-based cropping agents (LongVT, VideoZoomer, TimeSearch-R) conflates base-model strength with the tree-search mechanism. The main text only says 'we apply a filtering procedure for CG-Bench' and never states that all baselines were re-scored on the exact filtered subset, so the reported gain is not shown to be independent

full rationale

The derivation of VTS itself is not circular: tree construction uses CLIP boundaries, navigation is trained on synthesized trajectories whose correct/wrong states are defined by ground-truth intervals, and RL rewards use external IoU and answer-accuracy targets. The central mechanism is supported by controlled ablations (Table 3: removing backtracking or hierarchy degrades; Table 4: detour-and-recovery trajectories beat optimal-path-only by +6.4 mIoU; Table 6: gains persist with Qwen2.5-VL-7B). The one partial circularity is evaluative rather than derivational: the CG-Bench subset is filtered by Qwen3-VL-8B-Instruct, the same model family used to instantiate VTS, and the paper does not document re-scoring of the prior-agent baselines on that exact subset. This is a real self-referential evaluation loop that should be disclosed and fixed (e.g., by re-running all baselines on the filtered set and by ablating the filter model), but it does not collapse the paper's central claim because the tree-search contribution has independent empirical support.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 1 invented entities

The central claim rests on a set of hand-chosen hyperparameters (tree construction, reward weights, filter thresholds) and on the assumption that CLIP-based segmentation and Qwen3-VL-based filtering/scoring are reliable. These are not derived from first principles, so the paper's contribution is the method and its controlled ablations, not a parameter-free derivation.

free parameters (6)
  • k (threshold coefficient) = 1.5
    Hand-chosen in the adaptive scene-boundary threshold tau = mean(delta) + k*std(delta); controls tree granularity (Section 3.1, Algorithm 1).
  • Nmin, Nmax (children count bounds) = 3, 8
    Hand-chosen constraints on number of children per node; shapes the tree and the search space.
  • Leaf duration threshold = 64 seconds
    Segments shorter than 64s become leaves; sets the minimum granularity of the search tree.
  • Max interaction turns = 10
    Inference budget and RL training budget; affects how many recovery moves are possible.
  • Reward weights (lambda_fmt, lambda_acc, lambda_IoU) = 0.5, 0.5, 1.0
    Weighted sum in Eq. (2); chosen without sensitivity analysis (Appendix Table 8).
  • CG-Bench coverage removal cutoff = 30% of video
    Hand-chosen threshold in Filter 1 that drops questions whose GT covers >30% of the video; changes the benchmark itself.
axioms (4)
  • domain assumption CLIP cosine distance between consecutive frame embeddings is a valid measure of semantic scene boundaries, and the adaptive threshold with k=1.5 yields semantically coherent segments.
    Used in Section 3.1 and Algorithm 1 to construct the tree; if the segmentation aligns poorly with query-relevant events, navigation cannot localize evidence.
  • domain assumption Ground-truth evidence is a single contiguous interval; multi-segment and high-coverage questions are removed.
    Explicitly assumed in Section 3 and Appendix B Filter 1; questions requiring distributed evidence are outside the method's scope and are filtered out of CG-Bench.
  • domain assumption Qwen3-VL-8B provides reliable question-relevance scores for choosing the next zoom_in child during trajectory synthesis.
    Section 3.4 uses Qwen3-VL-8B as the controller's scorer; if these scores are unreliable, the synthesized trajectories may not teach useful navigation.
  • domain assumption Qwen3-VL-8B-Instruct filter judgments correctly identify text-answerable questions, clue-sufficient questions, and clue-unique questions.
    Appendix B Filters 2-4 use this model to decide which CG-Bench questions survive; errors in the filter change the benchmark and bias the evaluation.
invented entities (1)
  • LongClueQA no independent evidence
    purpose: A synthetic training corpus of timestamped question-clue pairs generated from unlabeled YouTube videos, used for SFT and RL.
    The dataset is not released and has no external validation; its quality is bounded by the Qwen3-VL captions and Qwen3-Next question generation.

pith-pipeline@v1.3.0-alltime-deepseek · 19095 in / 14599 out tokens · 128523 ms · 2026-08-01T21:04:55.250680+00:00 · methodology

0 comments
read the original abstract

Grounded long-video question answering (Grounded LVQA) requires answering a question about a long video while localizing the short evidence interval that supports the answer. Recent agentic methods frame this task as multi-turn exploration with a single crop_video(start, end) action, which supports coarse-to-fine narrowing but provides no primitive for fine-to-coarse backtracking. As a result, these agents typically converge prematurely and cannot recover from an early mistake. We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree. VTS constructs a non-uniform tree from visual scene boundaries so that each node corresponds to a semantically coherent segment, and trains an agent to navigate the tree through four discrete operations: zoom_in, zoom_out, shift, and answer. These operations expose backtracking and recovery as explicit, learnable primitives rather than implicit behaviors. To train this navigation, we introduce a trajectory synthesis pipeline that produces multi-step paths through the tree, including deliberate detours into incorrect branches followed by recovery. We use these trajectories for supervised fine-tuning, followed by reinforcement learning with grounding and answer-accuracy rewards. On three Grounded LVQA benchmarks (CG-Bench, Haystack-LVBench, Haystack-Ego4D), VTS outperforms the strongest prior agentic methods by +12.5 mIoU on CG-Bench and +7.4 T-F1 on Haystack-Ego4D. The learned policy also transfers to general long-video QA, surpassing all prior agentic baselines on Video-MME, MLVU, and LVBench by up to +7.1 accuracy points. Ablations confirm that self-correcting hierarchical search is the central mechanism behind these gains: removing either adaptive descent or explicit backtracking substantially degrades performance. Code is available at https://github.com/CeeZh/VTS.

Figures

Figures reproduced from arXiv: 2607.16189 by Ce Zhang, Gedas Bertasius, Hiromi Wakaki, Mohit Bansal, Oluwatumininu Oguntola, Pranav Wagh, Qiyu Wu, Yulu Pan, Ziyang Wang.

Figure 1
Figure 1. Figure 1: Comparison between prior agentic methods and our VideoTreeSearch (VTS) on grounded LVQA. Prior agentic methods (bottom left) rely on a continuous crop_video action that conflates exploration with correction. As a result, they often converge prematurely and cannot recover from early mistakes. Our proposed VideoTreeSearch (bottom right) instead organizes the video as a tree of semantically coherent segments … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of VideoTreeSearch (VTS). Left: Given a long video, VTS recursively partitions it into a non-uniform tree using CLIP-based scene boundaries, so each node corresponds to a semantically coherent segment. Right: The agent navigates the tree through four discrete actions— zoom_in to descend into a child, zoom_out to ascend to the parent, shift to move laterally to a sibling, and answer to terminate wi… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative example: recovery via zoom_out. For a question about what is visible through a 22nd-floor window of the Fuji TV building, the agent first descends into a wrong branch showing the building’s ground-floor ticket booth (turns 1–3). It detects the error, issues zoom_out to return to the parent (turn 4), and explores a different child to reach the correct clue (turns 5–6). The example shows zoom_out… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy (left) and temporal-IoU (right) rewards over the first 150 GRPO updates of our [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overview of the LongClueQA construction pipeline. Collected YouTube videos are segmented into 10-second clips and captioned with Qwen3-VL. Multiple-choice QA pairs with temporal clues are then generated from the captions and filtered to retain only high-quality questions. Instruct to produce a free-form caption describing the visual content of that temporal segment. The caption is augmented with its start … view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of trajectory lengths across the three data sources. For each dataset we plot the number of trajectories that terminate after a given number of turns, where one turn corresponds to a single agent action (ZOOM_IN, ZOOM_OUT, SHIFT, or ANSWER). Trajectories with more than 15 turns are discarded as outliers, and lengths beyond 10 are aggregated into a single “> 10” bin. ZOOM_IN ZOOM_OUT SHIFT ANSW… view at source ↗
Figure 7
Figure 7. Figure 7: shows the breakdown of action types issued during trajectory generation. ZOOM_IN dominates on every dataset, reflecting the top-down nature of hierarchical search. ANSWER appears once per trajectory and therefore tracks dataset size. ZOOM_OUT and SHIFT act as corrective moves, and their relative frequency indicates how often the initial descent path needs to be revised before the agent commits to an answer… view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative example: recovery via zoom_out and shift. The question asks for the number of turkeys on the table behind the man. The agent first follows several zoom_in steps into an incorrect branch that shows the man but no turkeys (turns 1–6). It then recovers by using zoom_out to ascend to the parent node (turn 7), followed by shift to move laterally to an unvisited sibling (turn 8). This sibling contain… view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative example: hierachical zoom_in without backtracking. The question asks about the types of fruits placed on the cabinet. The agent directly reaches the relevant segment containing both the fruits and the cabinet after a single zoom_in step (turn 1), and then terminates with answer in the next turn (turn 2). 22 [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 18 linked inside Pith

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Revisiting the “video” in video-language understanding

    Shyamal Buch, Cristobal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the “video” in video-language understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  3. [3]

    VideoMiner: Iteratively grounding key frames of hour-long videos via tree-based group relative policy optimization.arXiv preprint arXiv:2510.06040, 2025

    Xinye Cao, Hongcan Guo, Jiawen Qian, Guoshun Nan, Chao Wang, Yuqi Pan, Tianhao Hou, Xiaojuan Wang, and Yutong Gao. VideoMiner: Iteratively grounding key frames of hour-long videos via tree-based group relative policy optimization.arXiv preprint arXiv:2510.06040, 2025

    arXiv 2025

  4. [4]

    CG-Bench: Clue-grounded question answering benchmark for long video understanding

    Guo Chen et al. CG-Bench: Clue-grounded question answering benchmark for long video understanding. InProceedings of the International Conference on Learning Representations (ICLR), 2025

    2025

  5. [5]

    ShareGPT4Video: Improving video understanding and generation with better captions.Advances in Neural Information Processing Systems (NeurIPS), 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. ShareGPT4Video: Improving video understanding and generation with better captions.Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  6. [6]

    LongVILA: Scaling long-context visual language models for long videos.arXiv preprint arXiv:2408.10188, 2024

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. LongVILA: Scaling long-context visual language models for long videos.arXiv preprint arXiv:2408.10188, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    VideoZoomer: Reinforcement-learned temporal focusing for long video reasoning.arXiv preprint arXiv:2512.22315, 2025

    Yang Ding, Yizhen Zhang, Xin Lai, Ruihang Chu, and Yujiu Yang. VideoZoomer: Reinforcement-learned temporal focusing for long video reasoning.arXiv preprint arXiv:2512.22315, 2025

    arXiv 2025

  8. [8]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  10. [10]

    ReVisionLLM: Recursive vision-language model for temporal grounding in hour-long videos

    Tanveer Hannan, Md Mohaiminul Islam, Jindong Gu, Thomas Seidl, and Gedas Bertasius. ReVisionLLM: Recursive vision-language model for temporal grounding in hour-long videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  11. [11]

    Long movie clip classification with state-space video models

    Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. InEuropean Conference on Computer Vision (ECCV), 2022

    2022

  12. [12]

    Efficient movie scene detection using state-space transformers

    Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsundar Athrey, Tony Braskich, and Gedas Bertasius. Efficient movie scene detection using state-space transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  13. [13]

    Video ReCap: Recursive captioning of hour-long videos

    Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius. Video ReCap: Recursive captioning of hour-long videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  14. [14]

    BIMBA: Selective-scan compression for long-range video question answering

    Md Mohaiminul Islam, Tushar Nagarajan, Huiyu Wang, Gedas Bertasius, and Lorenzo Torresani. BIMBA: Selective-scan compression for long-range video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  15. [15]

    Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S. Ryoo. Language repository for long video understanding. InFindings of the Association for Computational Linguistics (ACL Findings), 2025. 12

    2025

  16. [16]

    VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    LLaMA-VID: An image is worth 2 tokens in large language models.European Conference on Computer Vision (ECCV), 2024

    Yanwei Li, Chengyao Wang, and Jiaya Jia. LLaMA-VID: An image is worth 2 tokens in large language models.European Conference on Computer Vision (ECCV), 2024

    2024

  18. [18]

    TimeSearch-R: Adaptive temporal search for long-form video understanding via self-verification reinforcement learning.arXiv preprint arXiv:2511.05489, 2025

    Junwen Pan, Qizhe Zhang, Rui Zhang, Ming Lu, Xin Wan, Yuan Zhang, Chang Liu, and Qi She. TimeSearch-R: Adaptive temporal search for long-form video understanding via self-verification reinforcement learning.arXiv preprint arXiv:2511.05489, 2025

    arXiv 2025

  19. [19]

    Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryu, Donghyun Kim, and Michael S. Ryoo. Too many frames, not all useful: Efficient strategies for long-form video QA.arXiv preprint arXiv:2406.09396, 2025

    arXiv 2025

  20. [20]

    Learning transferable visual models from natural language supervision

    Alec Radford et al. Learning transferable visual models from natural language supervision. In ICML, 2021

    2021

  21. [21]

    Understanding long videos in one multimodal language model pass

    Kanchana Ranasinghe, Xiang Li, Kumara Kahatapitiya, and Michael Ryoo. Understanding long videos in one multimodal language model pass. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  22. [22]

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. TimeChat: A time-sensitive multimodal large language model for long video understanding.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  23. [23]

    Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Jongwoo Park, Kanchana Ranasinghe, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles

    Michael S. Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Jongwoo Park, Kanchana Ranasinghe, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-MM-vid (BLIP-3-video): You only need 32 tokens to represent a video even in VLMs. arXiv preprint arXiv:2410.16267, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    LongVU: Spa- tiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. LongVU: Spa- tiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    MovieChat: From dense token to sparse memory for long video understanding.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. MovieChat: From dense token to sparse memory for long video understanding.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  26. [26]

    Adaptive keyframe sampling for long video understanding

    Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  27. [27]

    Alvarez, Lei Zhang, and Zhiding Yu

    Shihao Wang, Guo Chen, De-an Huang, Zhiqi Li, Minghan Li, Guilin Li, Jose M. Alvarez, Lei Zhang, and Zhiding Yu. VideoITG: Multimodal video understanding with instructed temporal grounding.arXiv preprint arXiv:2507.13353, 2025

    arXiv 2025

  28. [28]

    LVBench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. LVBench: An extreme long video understanding benchmark.arXiv preprint arXiv:2406.08035, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    AdaReTaKe: Adaptive redundancy reduction to perceive longer for video-language understanding.arXiv preprint arXiv:2503.12559, 2025

    Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. AdaReTaKe: Adaptive redundancy reduction to perceive longer for video-language understanding.arXiv preprint arXiv:2503.12559, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    TimeZero: Temporal video grounding with reasoning-guided LVLM.arXiv preprint arXiv:2503.13377, 2025

    Ye Wang, Boshen Xu, Zihao Yue, Zihan Xiao, Ziheng Wang, Liang Zhang, Dingyi Yang, Wenxuan Wang, and Qin Jin. TimeZero: Temporal video grounding with reasoning-guided LVLM.arXiv preprint arXiv:2503.13377, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 13

    2025

  32. [32]

    Ryoo, and Juan Carlos Niebles

    Ziyang Wang, Honglu Zhou, Shijie Wang, Junnan Li, Caiming Xiong, Silvio Savarese, Mohit Bansal, Michael S. Ryoo, and Juan Carlos Niebles. Active video perception: Iterative evidence seeking for agentic long video understanding.arXiv preprint arXiv:2512.05774, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    LongVideoBench: A benchmark for long- context interleaved video-language understanding.Advances in Neural Information Processing Systems (NeurIPS), 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. LongVideoBench: A benchmark for long- context interleaved video-language understanding.Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  34. [34]

    thinking with long videos

    Zuhao Yang, Sudong Wang, Kaichen Zhang, Keming Wu, Sicong Leng, Yifan Zhang, Bo Li, Chengwei Qin, Shijian Lu, Xingxuan Li, and Lidong Bing. LongVT: Incentivizing “thinking with long videos” via native tool calling.arXiv preprint arXiv:2511.20785, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Generative frame sampler for long video understanding.arXiv preprint arXiv:2503.09146, 2025

    Linli Yao, Haoning Wu, Kun Ouyang, Yuanxing Zhang, Caiming Xiong, Bei Chen, Xu Sun, and Junnan Li. Generative frame sampler for long video understanding.arXiv preprint arXiv:2503.09146, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Re-thinking temporal search for long-form video understanding.arXiv preprint arXiv:2504.02259, 2025

    Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding.arXiv preprint arXiv:2504.02259, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    MomentSeeker: A benchmark for long-video moment retrieval.arXiv preprint arXiv:2502.12558, 2025

    Huaying Yuan, Jian Ni, Zheng Liu, Yueze Wang, Junjie Zhou, Zhengyang Liang, Bo Zhao, Zhao Cao, Zhicheng Dou, and Ji-Rong Wen. MomentSeeker: A benchmark for long-video moment retrieval.arXiv preprint arXiv:2502.12558, 2025

    arXiv 2025

  38. [38]

    Video-o3: Native interleaved clue seeking for long video multi-hop reasoning.arXiv preprint arXiv:2601.23224, 2026

    Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang, Changlian Ma, Qingyu Zhang, Zizheng Huang, Kun Ouyang, Tianxiang Jiang, et al. Video-o3: Native interleaved clue seeking for long video multi-hop reasoning.arXiv preprint arXiv:2601.23224, 2026

    Pith/arXiv arXiv 2026

  39. [39]

    A simple LLM framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple LLM framework for long-range video question-answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  40. [40]

    SiLVR: A simple language-based video reasoning framework.arXiv preprint arXiv:2505.24869, 2025

    Ce Zhang, Yan-Bo Lin, Ziyang Wang, Mohit Bansal, and Gedas Bertasius. SiLVR: A simple language-based video reasoning framework.arXiv preprint arXiv:2505.24869, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Thinking with videos: Multimodal tool-augmented reinforcement learning for long video reasoning.arXiv preprint arXiv:2508.04416, 2025

    Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma, Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang. Thinking with videos: Multimodal tool-augmented reinforcement learning for long video reasoning.arXiv preprint arXiv:2508.04416, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    MLVU: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024. 14 Appendix Our appendix consists of Additional Implementation Details (Section A), Evaluation Benchmarks (Section B), Data Const...

    Pith/arXiv arXiv 2024

  44. [44]

    User: Current Segment [t s-te]:

    answer <letter> <evidence_start> <evidence_end> : Provide the answer to the question along with the time interval (in seconds) that contains the supporting evidence. User: Current Segment [t s-te]: . . . Segmenti[t i s-ti e]: <frame_timestamp><image>. . . . . . Memory: Tree Structure: <materialized tree with [visited]/[current] flags and short node captio...

    2000