Pith. sign in

REVIEW 3 major objections 4 minor 8 cited by

Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A vision-language model that selects its own relevant frames before answering outperforms the same model given a much larger context window on hour-long videos.

desk verdict Good method, shaky headline number: the 61.7 LVBench result doesn't reproduce from the stated s=64, l=12 setup. read the letter →

arxiv 2507.02001 v1 pith:FKDMJBWW submitted 2025-07-01 cs.LG

Anurag Arnab , Ahmet Iscen , Mathilde Caron , Alireza Fathi , Cordelia Schmid This is my paper
classification cs.LG
keywords videoquestionansweringlong-videounderstandingframeselectioninference-timescalingcontextcurationchain-of-thoughtvision-languagemodelsLVBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a vision-language model can answer questions about long videos more accurately if, at inference time, it first selects the frames most relevant to the question and only then answers. The authors call this Temporal Chain of Thought, and they show it improves accuracy over standard long-context inference on four video QA benchmarks and with three different VLMs. The headline result is on LVBench, where videos average 68 minutes: using a 32K-token context window, the method scores 61.7%, beating the same model that processes 700K tokens (58.9%). If the claim holds, it means the practical limit on long-video QA is not the model's context window but how well the model can find the relevant frames.

What carries the argument

The Dynamic-Segment TCoT procedure is the load-bearing object. Given a video of N frames, it splits the video into l non-overlapping segments, uniformly samples s frames from each segment, and runs a VLM selection call per segment that returns a JSON list of relevant frame ids plus a justification. The selected frames are concatenated with u uniformly sampled context frames (added so the answerer does not lose spatial or floor context), subsampled to fit the context limit k, and passed to the same VLM as the answerer. This machinery decouples video length from the context window and turns relevance selection into an instruction-following task.

What would settle it

On LVBench, compute the recall of TCoT's selected frames against the human-annotated time-reference segments for the questions the method gets wrong; if those errors coincide with low recall (the reference frames were not sampled at all), then sampling density, not selection quality, is setting the ceiling. A sharper version: replace the VLM selector with random frame selection at the same 120-frame budget; if random selection matches TCoT's 61.7%, then curation is not the cause of the improvement.

Watch

Extended reading notes

Core claim

Temporal Chain of Thought decomposes video question answering into two stages performed by the same instruction-tuned VLM: a selection stage that partitions the video into segments, uniformly samples s frames from each, asks the model which frames are needed (returning frame ids and a justification), and concatenates the answers; and an answering stage that takes the selected frames plus a small number of uniformly sampled context frames and produces the answer. The paper's central claim is that this curation beats feeding the whole video to the model, because long contexts contain distractors. On LVBench, with l=12 segments and s=64 frames per segment, the method reaches 61.7% accuracy under a 32K-token budget, outperforming the same VLM using 700K tokens (58.9%), and it improves over baseline on Egoschema, NExT-QA, and OpenEQA as well.

Load-bearing premise

The method assumes that the frames needed to answer the question appear in the uniformly sampled s frames of at least one segment; if the decisive moment falls between samples, no amount of VLM selection can recover it.

Editorial extensions

If this is right

  • TCoT achieves state-of-the-art numbers on four video QA benchmarks: 75.2% (subset) and 69.1% (full) on Egoschema, 61.7% on LVBench, 81.0% on NExT-QA, and 69.2% on OpenEQA.
  • The gains hold across three different VLMs, and the improvement is largest for the model with the smallest native context window (GPT-4o-mini, +5.5 points on LVBench).
  • Accuracy rises smoothly as the number of segments l increases, while baseline inference saturates; TCoT's per-call context remains at 32K regardless of the total number of frames processed.
  • The number of frames the model selects adapts to question type and correlates with human-annotated time references, so the selection is not just a fixed subsample.
  • Pure language chain-of-thought variants (zero-shot CoT, two-stage prompting, self-consistency) give only marginal gains on LVBench, which is why the video-specific selection matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the limiting factor is recall of the sampler rather than the selector's judgement, then improving the initial sampling (for example, by motion or scene-change detection) should raise accuracy without changing the selection or answering stages.
  • The method's dependence on instruction-following suggests that fine-tuning or reinforcement-learning the selection call could close part of the gap to the oracle time-reference frames (67.4% on LVBench), a step the paper leaves for future work.
  • The same two-stage design—curate context first, then answer—may transfer to other long multimodal inputs such as audio streams or lengthy documents, where the distraction problem is known to be similar.
  • A direct test of the paper's logic: on LVBench, replace the VLM selector with random frames of the same budget; if random selection approaches 61.7%, then the gain comes from the segment decomposition rather than from semantic selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Temporal Chain of Thought (TCoT), an inference-time strategy for video question-answering in which the same VLM first selects relevant video frames from a long video, with a Dynamic-Segment variant that partitions the video into segments, samples frames within each segment, aggregates the selected frames, and then answers the question. The authors report consistent improvements over baseline inference across four datasets (Egoschema, LVBench, OpenEQA, NExT-QA) and three VLMs (Gemini 1.5 Flash, Qwen-2.5-VL, GPT-4o-mini). The headline claim is that on LVBench, TCoT with a 32K-token context window outperforms the same VLM using standard inference with a 700K-token context by 2.8 points at matched total tokens. The paper also provides ablations, an oracle comparison using human-annotated time references, and a failure-mode analysis.

Significance. If the empirical claims hold, the paper makes a useful contribution: it demonstrates a training-free, single-model inference-time scaling strategy for long-video QA, with adaptive frame selection that is question-dependent and interpretable justifications. The method's consistency across multiple VLMs and datasets, the transparent failure analysis, and the oracle upper-bound comparison are genuine strengths. However, the main quantitative claim—the 2.8-point improvement at matched 700K-token cost—is currently not reproducible from the published hyperparameters and token counts, so the central result needs to be verified with a corrected and consistent experimental report before the significance can be assessed.

major comments (3)
  1. [§4.2, Table 3, Fig. 4] The reported total-token counts are internally inconsistent with the stated hyperparameters. The main text fixes s=64 frames per segment and l=12 segments in §4.2. With Gemini 1.5 Flash using 258 tokens per frame (§4.1), the selection calls process 12 × 64 × 258 ≈ 198K visual tokens, and the answerer call processes at most 120 frames ≈ 31K tokens, for roughly 229K total visual tokens, not 672K as in Table 3 or 697K as in Fig. 4. Reaching 672K would require approximately l≈40 segments or a different per-call frame count. Since the central claim is that TCoT beats a 700K-token baseline at the same computational cost, the exact configuration (l, s, u, number of selection calls, and per-call frame count) must be pinned down and the token accounting corrected before the claim can be accepted.
  2. [Table 1 vs. App. A, Table 6] The accuracy numbers for the Dynamic-Segment TCoT configuration do not match across tables. Table 1 and Fig. 4 report 61.7 on LVBench for Dynamic-Segment TCoT, and Table 3 lists the same value with 672K total tokens. However, App. A Table 6a reports 57.8 for s=64 with m=120 and u=0, which is the setting described in §4.2; Table 6b's best uniform-context setting reaches only 59.3. No combination of the published hyperparameters reproduces 61.7. The authors need to clarify which configuration produced the headline number and ensure that the appendix and main text are consistent, or rerun the experiments under the stated settings.
  3. [Eq. (5) and App. B] The Dynamic-Segment method uniformly samples s frames from each segment before asking the VLM to select relevant frames. If the event needed to answer the question falls outside this uniform subsample, it is invisible to both the selector and the answerer. This is acknowledged in App. B, where the failure cases show exactly this low-recall mechanism, but the discussion in §1 and §4.3 describes the method as being able to 'effectively consider the whole video.' Since the main advantage over the long-context baseline depends on the total number of frames processed, the paper should quantify how sensitive the selection accuracy is to the pre-sampling density, and report, for example, the recall of human-annotated time-reference frames as a function of s. Without this, the method's strength on LVBench cannot be cleanly separated from the effect of simply processing more frames.
minor comments (4)
  1. [App. D, Hierarchical TCoT] The text says the method 'zooms in' on relevant regions as shown in 'Fig. 2b', but Fig. 2 has no panel labeled (b); the two panels are captioned 'Single-Stage TCoT' and 'Dynamic-Segment TCoT'. Please add the panel labels or fix the cross-reference.
  2. [§4.2, Fig. 4] The x-axis of Fig. 4 is labeled 'Visual tokens and (Frames)' but the TCoT point at 697K tokens is inconsistent with s=64, l=12 (see major comment). Once the configuration is corrected, the figure should be regenerated with the correct token counts.
  3. [App. A, Table 6a] The row s=120 in Table 6a appears to correspond to feeding an entire segment of 120 frames to the selection call, which is effectively Single-Step TCoT rather than a dynamic-segment variant; please state this explicitly in the table caption or main text to avoid confusion.
  4. [References] The reference to self-consistency [58] appears in Fig. 4 as 'Self-consistency CoT' but the bibliography entry is dated 2022; please ensure the citation style is consistent and that all in-text citations have corresponding entries.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: TCoT's reported gains are empirical benchmark outcomes, not encoded in the method's construction; the same-VLM selection step is independently validated by oracle time-reference comparisons.

full rationale

The paper's contribution is an inference algorithm rather than a theorem, so the claimed derivation chain is: define the selection function G and answerer H, run them on video QA benchmarks, and measure accuracy. The central claim (e.g., 61.7 vs 58.9 on LVBench) is an empirical outcome, not an input to the method. Using the same VLM for selection and answering could intuitively look self-referential, but the benchmark labels and the human-annotated time references are external to G and H; Table 2's oracle row (67.4) and Figure 8's per-question-type comparison independently confirm that selection quality, not a relabeled input, drives the improvement. No fitted parameter is renamed as a prediction: s, l, and u are reported hyperparameters with ablations in Appendix A, and the reported numbers are measured accuracies rather than quantities forced by the method's equations. No load-bearing self-citation is present: Gemini 1.5 Flash is cited as an external model with its own published benchmark results, and the authors' VideoAgent reimplementation is explicitly marked as their own (†). The internal inconsistency between the token counts implied by s=64, l=12 (roughly 230K visual tokens) and the 672K/700K values in Table 3 and Figure 4, and the discrepancy between the 57.8 in Table 6a and the 61.7 in Table 1, are serious reproducibility and correctness concerns, but they are not circularity: nothing in those inconsistencies shows that the accuracy number was derived from its own assumption. The paper also honestly documents failure cases (Appendix B) where relevant frames are not selected, which is inconsistent with a claim that the method's success is guaranteed by construction. Therefore, under the hard rules requiring a specific reduction to inputs or a self-citation chain, no circular step can be exhibited, and the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method's central effect depends primarily on the VLM's instruction-following ability for frame selection; no new physical or mathematical entities are introduced. The main 'free parameters' are the inference hyperparameters s, u, and l, which are tuned on the same benchmarks used for the headline claims.

free parameters (3)
  • s (segment sample size) = 64 (ablated over 4, 16, 32, 64, 120)
    Number of uniformly sampled frames per segment passed to the selection call; chosen on the same benchmark datasets and used for final results.
  • u (number of uniform context frames) = not clearly specified for main runs; ablation tests 0, 32, 56, 88, 120
    Number of uniformly sampled frames added to the selected frames for the final answer; the value used for the headline 61.7 result is not stated.
  • l (number of segments) = stated as 12, but total-token arithmetic implies roughly 41
    Number of video segments processed; controls inference-time compute. The stated value is inconsistent with reported total tokens.
assumptions (4)
  • domain assumption The VLM selection call S returns JSON with valid, relevant frame IDs.
    If the model outputs wrong IDs or fails to parse, the method falls back to all frames (Sec. 3.2), degrading to baseline. No parse-failure rate is reported.
  • domain assumption Uniform sampling of s frames per segment preserves the frames needed to answer the question.
    The dynamic-segment design (Eq. 5) assumes no critical event falls between sampled frames; the paper's own failure cases show this assumption fails.
  • domain assumption The answerer H can answer correctly given the curated context c = x_hat[m] union x[u].
    If the selected frames lack context (e.g., which floor the washing machine is on), accuracy drops; the paper adds uniform frames to compensate (Sec. 3.2, App. C).
  • domain assumption The evaluation benchmarks (Egoschema, LVBench, OpenEQA, NExT-QA) are valid measures of long-video understanding.
    The paper treats these benchmarks as ground truth for comparing methods. This is standard practice in the field and not specific to this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames." pith.science (2026). https://pith.science/paper/FKDMJBWW

@misc{pith2026250702001,
  author       = {Pith},
  title        = {Pith review of: Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FKDMJBWW}},
  note         = {Machine review of arXiv:2507.02001}
}
read the original abstract

Despite recent advances in Vision-Language Models (VLMs), long-video understanding remains a challenging problem. Although state-of-the-art long-context VLMs can process around 1000 input frames, they still struggle to effectively leverage this sequence length, and succumb to irrelevant distractors within the context window. We present Temporal Chain of Thought, an inference strategy for video question-answering that curates the model's input context. We use the VLM itself to iteratively identify and extract the most relevant frames from the video, which are then used for answering. We demonstrate how leveraging more computation at inference-time to select the most relevant context leads to improvements in accuracy, in agreement with recent work on inference-time scaling of LLMs. Moreover, we achieve state-of-the-art results on 4 diverse video question-answering datasets, showing consistent improvements with 3 different VLMs. In particular, our method shines on longer videos which would not otherwise fit within the model's context window: On longer videos of more than 1 hour on LVBench, our approach using a context window of 32K outperforms the same VLM using standard inference with a 700K context window by 2.8 points.

Figures

Figures reproduced from arXiv: 2507.02001 by the authors.

Figure 1
Figure 1. Temporal Chain of Thought. Motivated by the fact that long input contexts can have distractors which confuse the model, we use the VLM itself to first extract relevant context (blue box) before processing it. Our approach improves accuracy, and by iteratively processing parts of the video at a time, can also reduce the model’s required context window. fit within the model’s context limit. Moreover, our approach has … view at source ↗
Figure 2
Figure 2. Temporal Chain of Thought. We use Single-Step TCoT (left, Sec. 3.2) to construct our final approach (right). Namely, we use the VLM itself to extract relevant frames from an input video clip, conditioned on the input question. To scalably process longer videos, we perform this approach within l segments which span the video to extract the most relevant context. Finally, we use only the extracted context for answerin… view at source ↗
Figure 3
Figure 3. Prompt for our VLM selection call, S, (Eq. 4). where S denotes the VLM selection call (prompt in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Accuracy vs computation trade-off. We compare Temporal Chain of Thought to two alternatives: baseline in￾ference and self-consistency CoT [58]. We use the total num￾ber of visual tokens and frames (in parentheses) processed to measure computation, and vary l from 2 to …
Figure 5
Figure 5. Figure 5: Qualitative examples on LVBench. Note how our model focuses on different parts of the video to make its prediction. For clarity, we sample frames from the segments selected by TCoT. In the failure case, although TCoT finds various frames showing the fairy’s reactions, …
Figure 6
Figure 6. Figure 6: Percentage of selected frames by question type. Observe how the proportion of selected frames dynamically adjusts to the question type, aligning well with the human￾annotated time-reference frames on LVBench [57]. Dynamic Context Aggregation [PITH_FULL_IMAGE:figures/f…
Figure 7
Figure 7. Figure 7: Distribution of frames selected by our model. Our model selects the relevant context in an adaptive manner, choosing a greater proportion of input frames for Egoschema (mean of 74%) than for LVBench (mean of 15%) The results on Egoschema correlate with its “temporal ce…
Figure 8
Figure 8. Figure 8: Performance per question type on LVBench. We compare baseline inference, our TCoT method, and using the oracle of the human-annotated time references to select relevant frames. We achieve significant improvements on most question types, and often near the accuracy of t…
Figure 9
Figure 9. Figure 9: Accuracy according to the proportion of frames selected on Egoschema. Observe how our model’s accuracy remains consistent, regardless of the number of frames selected, suggesting that our TCoT method can effectively and adaptively select the relevant frames to answer t…
Figure 10
Figure 10. Figure 10: Failure modes of TCoT We show the precision and recall of frames that have been aggregated by our method, compared to the human-annotated time-reference frames of LVBench. This is performed for the instances in the dataset where TCoT fails, while the oracle which uses…
Figure 11
Figure 11. Figure 11: • Low precision and high recall (purple area in [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 11
Figure 11. Figure 11: Qualitative analysis of the failure modes of Temporal Chain of Thought. Failures due to low precision: Sometimes, our method selects too many frames. In the top left example, it is overly influenced by the question and selects frames at each window, even when they are…
Figure 12
Figure 12. Figure 12: Additional qualitative examples on LVBench. Note how our model focusses on different parts of the video to predict the correct answer. The top row shows an example of the model focussing on 3 diverse segments of the same video. The second row includes two such segment…
Figure 13
Figure 13. Figure 13: Importance of adding uniform context Our TCoT approach is able to select the relevant frames (103 to 115) for the question, by focussing on the washing machine (and provides the correct justification for doing so too). However, if we pass only these selected frames to…
Figure 14
Figure 14. Figure 14: Multiple choice question prompt for the Gemini and GPT-4o-mini answering call, H (Eq. 3). Frames: {frame1}, . . . , {frame N} Carefully watch the video and pay attention to the cause and sequence of events, the detail and movement of objects and the action and pose of…
Figure 15
Figure 15. Figure 15: Multiple choice question prompt for the Qwen 2.5-VL answering call, H (Eq. 3). We hierarchically extend Single-Step TCoT (Sec. 3.2) by iteratively sampling around the previously identified frames of interest. Intuitively, in a long video where our model’s context limi…
Figure 16
Figure 16. Figure 16: Open-ended question prompt for our Gemini answering call, H (Eq. 3). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MemLearner: Learning to Query Context memory for Video World Models

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    MemLearner introduces a learning-based adaptive context query method using query tokens in video world models to improve long-term scene consistency over rule-based retrieval.

  2. Internalized Reasoning for Long-Context Visual Document Understanding

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    A synthetic pipeline creates and internalizes reasoning traces in VLMs for long-context visual document understanding, with a 32B model surpassing a 235B model on MMLongBenchDoc and showing 12.4x fewer output tokens.

  3. Internalized Reasoning for Long-Context Visual Document Understanding

    cs.CV 2026-03 conditional novelty 6.5 of 10

    Synthetic page-ranked reasoning traces plus low-strength model merging give a 32B VLM 58.3 on MMLongBenchDoc, beating a 235B teacher while cutting output tokens ~12× versus explicit reasoning.

  4. Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

    eess.AS 2026-07 conditional novelty 6.0 of 10

    AV-Flamingo is an open audio-visual language model that outperforms similarly sized open models on many long-video understanding benchmarks, using a new 7M-question dataset and timestamped reasoning.

  5. Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    KeyVT improves zero-shot 3D question answering by hierarchically selecting semantically and geometrically relevant views and using optimal transport to extract representative tokens from them.

  6. Personal Visual Context Learning in Large Multimodal Models

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Introduces Personal VCL formalization and benchmark revealing LMM context gaps, plus an Agentic Context Bank baseline that boosts personalized visual reasoning.

  7. Event-Causal RAG: A Retrieval-Augmented Generation Framework for Long Video Reasoning in Complex Scenarios

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    Event-Causal RAG segments videos into events represented as SES graphs, merges them into a causal knowledge graph, and uses bidirectional retrieval to supply relevant event chains to a video foundation model for impro...

  8. Swift Sampling: Selecting Temporal Surprises via Taylor Series

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    Swift Sampling is a training-free frame selection method that uses Taylor expansions on video latent trajectories to pick temporally surprising frames, outperforming uniform sampling on long-video QA tasks.

Reference graph

Works this paper leans on

74 extracted references · 32 canonical work pages · cited by 7 Pith papers

  1. [1]

    Gpt-4o mini

    Open AI. Gpt-4o mini. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    work page

  2. [2]

    Gpt-4 technical report

    Open AI. Gpt-4 technical report. In arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Gpt-4v(ision) system card

    Open AI. Gpt-4v(ision) system card. 2024

    work page 2024

  4. [4]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024

    2024

  5. [5]

    Goldfish: Vision-language understanding of arbitrarily long videos

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Mingchen Zhuge, Jian Ding, Deyao Zhu, Jürgen Schmidhuber, and Mohamed Elhoseiny. Goldfish: Vision-language understanding of arbitrarily long videos. In ECCV, 2024

    work page 2024

  6. [6]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. In arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  7. [7]

    Memory consolidation enables long-context video understanding

    Ivana Balaževi´c, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. Memory consolidation enables long-context video understanding. In ICML, 2024

    work page 2024

  8. [8]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. InICLR, 2023

    2023

Show all 74 references
  1. [9]

    Large language monkeys: Scaling inference compute with repeated sampling

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. In arXiv preprint arXiv:2407.21787, 2024

    2024 arXiv

  2. [10]

    Revisiting the" video" in video-language understanding

    Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" video" in video-language understanding. In CVPR, 2022

    2022

  3. [11]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. In arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  4. [12]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017

    2017

  5. [13]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023

    2023

  6. [14]

    Structured information extraction from complex scientific text with fine-tuned large language models

    Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. Structured information extraction from complex scientific text with fine-tuned large language models. In arXiv preprint arXiv:2212.05238, 2022

    2022 arXiv

  7. [15]

    Videoagent: A memory- augmented multimodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. In ECCV, 2024

    2024

  8. [16]

    Video-of-thought: Step-by-step video reasoning from perception to cognition

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. In ICML, 2024

    2024

  9. [17]

    Vertex api

    Google. Vertex api. https://cloud.google.com/vertex-ai

  10. [18]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022

    2022

  11. [19]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. In arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  12. [20]

    Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection

    Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thought dataset for fine-grained video reasoning via core frame selection. arXiv preprint arXiv:2411.14794, 2024

    2024 arXiv

  13. [21]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In CVPR, 2024

    2024

  14. [22]

    Ruler: What’s the real context size of your long-context language models? In arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? In arXiv preprint arXiv:2404.06654, 2024. 10

    2024 arXiv

  15. [23]

    Unsupervised dense information retrieval with contrastive learning

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. TMLR, 2022

    2022

  16. [24]

    Perceiver IO: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver IO: A general architecture for structured inputs & outputs. In ICLR, 2022

    2022

  17. [25]

    Action genome: Actions as compositions of spatio-temporal scene graphs

    Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In CVPR, 2020

    2020

  18. [26]

    Billion-scale similarity search with gpus

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transac- tions on Big Data, 7(3):535–547, 2019

    2019

  19. [27]

    Language repository for long video understanding

    Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. In arXiv preprint arXiv:2403.14622, 2024

    2024 arXiv

  20. [28]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022

    2022

  21. [29]

    Text-conditioned resampler for long form video understanding

    Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zisserman, and Federico Tombari. Text-conditioned resampler for long form video understanding. In ECCV, 2024

    2024

  22. [30]

    Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models. In arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  23. [31]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023

    2023

  24. [32]

    Invariant grounding for video question answering

    Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In CVPR, 2022

    2022

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  26. [34]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  27. [35]

    Ring attention with blockwise transformers for near-infinite context

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. In ICLR, 2024

    2024

  28. [36]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  29. [37]

    Bolt: Boost large vision-language model without training for long-form video understanding

    Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. Bolt: Boost large vision-language model without training for long-form video understanding. In arXiv preprint arXiv:2503.21483, 2025

    2025 arXiv

  30. [38]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension. In arXiv preprint arXiv:2411.13093, 2024

    2024

  31. [39]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In CVPR, 2024

    2024

  32. [40]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In NeurIPS, 2023

    2023

  33. [41]

    Morevqa: Exploring modular reasoning models for video question answering

    Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reasoning models for video question answering. In CVPR, 2024

    2024

  34. [42]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. In arXiv preprint arXiv:2402.07872, 2024

    2024 arXiv

  35. [43]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022. 11

    2022

  36. [44]

    Too many frames, not all useful: Efficient strategies for long-form video qa

    Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryoo, Donghyun Kim, and Michael S Ryoo. Too many frames, not all useful: Efficient strategies for long-form video qa. In arXiv preprint arXiv:2406.09396, 2024

    2024

  37. [45]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, 2023

    2023

  38. [46]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai

    Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In arXiv prepr...

    2021 arXiv

  39. [47]

    Tokenlearner: What can 8 learned tokens do for images and videos? In NeurIPS, 2021

    Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can 8 learned tokens do for images and videos? In NeurIPS, 2021

    2021

  40. [48]

    Longvu: Spatiotemporal adaptive compression for long video-language understanding

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. In arXiv preprint arXiv:2410.17434, 2024

    arXiv

  41. [49]

    Aligning and prompting everything all at once for universal visual perception

    Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all at once for universal visual perception. In CVPR, 2024

    2024

  42. [50]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023

    2023

  43. [51]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. In arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  44. [52]

    Eva-clip-18b: Scaling clip to 18 billion parameters

    Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-18b: Scaling clip to 18 billion parameters. In arXiv preprint arXiv:2402.04252, 2024

    2024 arXiv

  45. [53]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. In ICCV, 2023

    2023

  46. [54]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. In arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  47. [55]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. In arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  48. [56]

    Vamos: Versatile action models for video understanding

    Shijie Wang, Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Vamos: Versatile action models for video understanding. In ECCV, 2024

    2024

  49. [57]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, et al. Lvbench: An extreme long video understanding benchmark. In arXiv preprint arXiv:2406.08035, 2024

    2024 arXiv

  50. [58]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  51. [59]

    Vila: Efficient video-language alignment for video question answering

    Xijun Wang, Junbang Liang, Chun-Kai Wang, Kenan Deng, Yu (Michael) Lou, Ming Lin, and Shan Yang. Vila: Efficient video-language alignment for video question answering. In ECCV, 2024

    2024

  52. [60]

    Videoagent: Long-form video under- standing with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video under- standing with large language model as agent. In arXiv preprint arXiv:2403.10517, 2024

    2024 arXiv

  53. [61]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In arXiv preprint arXiv:2405.19209, 2024

    2024 arXiv

  54. [62]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022

    2022

  55. [63]

    Visual haystacks: Answering harder questions about sets of images

    Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Visual haystacks: Answering harder questions about sets of images. In arXiv preprint arXiv:2407.13766, 2024. 12

    2024 arXiv

  56. [64]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. In arXiv preprint arXiv:2408.00724, 2024

    2024 arXiv

  57. [65]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, 2021

    2021

  58. [66]

    Retrieval-based video language model for efficient long video question answering

    Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Retrieval-based video language model for efficient long video question answering. In arXiv preprint arXiv:2312.04931, 2023

    2023

  59. [67]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023

    2023

  60. [68]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. In NeurIPS, 2024

    2024

  61. [69]

    Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k

    Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. In arXiv preprint arXiv:2402.05136, 2024

    2024

  62. [70]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023

    2023

  63. [71]

    A simple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. In EMNLP, 2024

    2024

  64. [72]

    Learning video representations from large language models

    Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In CVPR, 2023

    2023

  65. [73]

    Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms

    Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms. In arXiv preprint arXiv:2406.09367, 2024

    2024 arXiv

  66. [74]

    temporal certificate

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In CVPR, 2024. 13 Table 4: Independent segment answer aggregation : As an additional baseline to show that our method can reason...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.