Pith. sign in

REVIEW 5 major objections 6 minor 59 references

ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces a video-text interleaved chain-of-thought paradigm and shows that re-inserting key video frames into the reasoning chain boosts video understanding accuracy across multiple multimodal large language models.

desk verdict ViTCoT's 5.5% gain is an oracle ceiling; the only answer-agnostic control shows 1.7%, so the paradigm is plausible but the strong claim is not yet supported. read the letter →

arxiv 2507.09876 v1 pith:SNGQJFUG submitted 2025-07-14 cs.CV cs.AIcs.CL

Yongheng Zhang , Xu Liu , Ruihan Tao , Qiguang Chen , Hao Fei , Wanxiang Che , Libo Qin This is my paper
classification cs.CVcs.AIcs.CL
keywords videounderstandingchain-of-thoughtmultimodallargelanguagemodelkey-videoselectioninterleavedreasoningVideo-TextBenchmarkquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that letting multimodal language models re-examine key video frames while they reason, rather than reasoning only over text, makes video question answering more accurate. To test this, it builds a benchmark of 1,382 videos whose key frames were selected by a multimodal model and manually verified, then runs a two-stage prompting scheme that interleaves those frames into the chain-of-thought. On Qwen2.5-VL-7B, the interleaved paradigm beats text-only reasoning by an average of 5.5 percentage points when the key frames are oracle-quality, and by about 1.7 points when the frames are picked automatically by CLIP similarity. The paper also reports that interleaved reasoning activates more attention neurons than text-only reasoning.

What carries the argument

The central object is the key-video, a short clip of typically three to four frames extracted from the original video that contains the visual evidence needed to answer the question. In the ViTIB benchmark, an MLLM picks these frames with the question, reasoning, and answer in hand, and humans verify them. At inference, the key-video is spliced into the middle of the initial textual reasoning, expressed as the concatenation operator that embeds the key-video within the preliminary reasoning, and the model is prompted to produce a final reasoning with both the original video and the key-video visible. This interleaving is the mechanism claimed to carry the performance gain; the paper also tries to show the gain is not just extra video input by giving vanilla baselines both videos as input and by replacing oracle key-video with a CLIP-selected rough key-video.

What would settle it

Run ViTCoT on a dataset where the key-video is selected without access to the answer, for example by a model trained only on the question or by uniform random frame sampling, and compare accuracy against text-only CoT; if the gain drops to zero or reverses, the boost depends on oracle key frames rather than on the interleaving paradigm itself.

Watch

Extended reading notes

Core claim

Existing chain-of-thought methods for video feed the model the video at input time but then reason exclusively in text. ViTCoT changes that: it first produces a rough textual reasoning, then inserts a short key-video chunk of selected frames into that reasoning and asks the model to continue, producing a final reasoning that can visually re-check its claims. The authors claim this video-text interleaved paradigm consistently outperforms the corresponding text-only CoT method across five MLLMs and three CoT variants, with the largest average gain of 5.4-5.5% on Qwen2.5-VL-7B. They interpret the gain as evidence that the visual modality carries part of the reasoning load, and they support this with a neuron-activation comparison showing more active attention heads under the interleaved paradigm.

Load-bearing premise

The main experiments give the model a key-video chosen with knowledge of the correct answer and manually verified to support it, so the reported 5.5% gain measures an ideal hint rather than an automatically available one; the automatic CLIP-based selector recovers only about a third of that gain.

Editorial extensions

If this is right

  • The video-text interleaved paradigm applies to multiple CoT methods, including CoT, Desp-CoT, and Plan-and-Solve, and to multiple MLLMs, suggesting the benefit is not tied to one prompting style.
  • Even when vanilla baselines are given both the original video and the key-video as input, interleaved reasoning outperforms them by about 2.8%, indicating that the placement of visual evidence, not just the presence of extra pixels, drives part of the gain.
  • On CoT and Desp-CoT, using only the key-video as input outperforms using only the original video by 0.8% and 1.2% respectively, showing that the benchmark condenses task-relevant information.
  • The interleaved paradigm activates more attention heads across layers in Qwen2.5-VL-3B, which the paper interprets as deeper model engagement with the reasoning.
  • The paradigm is presented as generally applicable to any CoT method, so future reasoning frameworks could adopt the same splice-in-visual-evidence design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gap between the oracle key-video gain of 5.5% and the automatic CLIP-selected gain of 1.7% suggests that practical headroom depends heavily on the quality of automatic key-frame selection, a problem the paper leaves largely open.
  • The interleaving idea is not limited to video: the same two-stage splice could be applied to images, audio, or other modalities, but the paper does not test those settings.
  • The neuron-activation analysis is correlational rather than causal, so without an intervention it does not establish that extra activations are what produce the accuracy gain.
  • A testable extension would be to train the key-video selector solely on the question and initial reasoning, without the answer, and measure how much of the oracle gain such a selector can recover on unseen videos.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Video-Text Interleaved Chain-of-Thought (ViTCoT), a prompting paradigm in which key video frames are interleaved into the text-based reasoning chain of an MLLM. The authors construct a new benchmark, ViTIB, by selecting key frames with Gemini-2.0-Flash and manually verifying them, and they evaluate ViTCoT on five MLLMs with three CoT variants (CoT, Desp-CoT, Plan-and-Solve). The main experiments use an Oracle key-video selected with access to the gold answer; the paper reports average gains of about 5.5% for Qwen2.5-VL-7B and smaller gains for other models, plus a non-Oracle CLIP-based control with a 1.7% average gain and an analysis of attention-head activations. The central claim is that interleaving key visual frames into reasoning, rather than merely adding video input, drives the improvements.

Significance. If the central claim is established, ViTCoT is a simple and potentially useful paradigm for video reasoning, and the ViTIB benchmark with human-verified key-video annotations could be a useful resource for the community. The paper deserves credit for being transparent about the Oracle key-video in the main experiments, for including a non-Oracle CLIP control, and for attempting a control that gives vanilla reasoning both original and key video. However, the current evidence mainly supports an oracle-augmented upper bound rather than a deployable, answer-agnostic method, and the answer-agnostic control is too underspecified to carry the practical claim. The neuron-activation analysis is qualitative and does not yet substantiate the 'activates more neuron values' claim.

major comments (5)
  1. [§3.1.2 / §3.2 / Table 1] The main results in Table 1 are obtained with the Oracle key-video, whose construction in §3.1.2 gives Gemini-2.0-Flash the video, question, reasoning steps, and the gold answer, and whose human recheck in §3.2 retains only key-videos scored ≥80, where 80–90 means the key-video plus original video 'leads to the correct answer' and 90–100 means the correct answer can be derived from the key-video alone. By construction and manual filtering, the model therefore receives a visual hint that is guaranteed to be sufficient for the correct answer. This is the load-bearing evidence for the abstract's claim that ViTCoT 'significantly boosts performance,' but it is not an inference-time protocol, because the gold answer is not available when selecting key frames.
  2. [§5.5.2 / Figure 7] The only answer-agnostic evaluation, the CLIP rough-key-video experiment, reports an average improvement of 1.7 percent but does not specify the model used, the number of retrieved frames, per-category results, or any variability measure such as error bars or significance tests. With category sizes as small as 11–50 videos in Figure 3 and stochastic decoding, a 1.7-point average difference is within plausible noise. This experiment is too underspecified to establish the practical form of the central claim.
  3. [§5.4 / Table 1] The claim that ViT reasoning 'consistently' outperforms vanilla reasoning is not supported at the per-category level. For example, in Table 1, Qwen2.5-VL-7B ViT CoT is worse than Vanilla Plan-and-Solve in 5 of 14 categories, and VideoLLaMA3-7B ViT Desp-CoT is worse than Vanilla Desp-CoT in 6 of 14 categories. All reported differences are averaged over a single run with no error bars, significance tests, or multi-seed results; the paper should quantify variance, especially for the small categories, before making consistency claims.
  4. [§5.5.1 / Figure 6] The control that gives vanilla reasoning both the original video and the Oracle key-video is reported only for Gemini-2.0-Flash and only as an aggregate 2.8 percent margin. This does not rule out the possibility that part of the gain comes from the two-stage protocol itself, from additional visual tokens, or from a second inference pass. An additional control using a random or non-informative key-video in the same two-stage interleaved protocol would be needed to isolate the contribution of interleaving task-relevant frames.
  5. [§5.5.4 / Figure 9(a)] The claim that ViT reasoning 'activates significantly more neuron values' in MLLMs is supported only by the qualitative heatmap in Figure 9(a). No quantitative metric, normalization, layer-aggregation rule, or statistical comparison is provided, so this statement is not currently a measurable or verifiable finding.
minor comments (6)
  1. [§3.1.3] The heading 'Assemlble Key-Video' contains a typo; it should be 'Assemble Key-Video.'
  2. [Figure 3] Figure 3 appears to be a table but lacks clear column headers and axis labels, and the 'A VG' label is not explained in the figure itself; the text should clarify what each column represents.
  3. [Figure 7] The figure and its caption do not state which MLLM is evaluated; the text in §5.5.2 also omits this detail. The reader cannot interpret the 1.7 percent average without knowing the model and the experimental setup.
  4. [Figure 1(d) vs. Table 1] Figure 1(d) reports AVG values of 46.3 and 51.8 for Qwen2.5-VL-7B, whereas averaging the corresponding rows in Table 1 gives approximately 46.7 and 52.2; this discrepancy should be reconciled or the averaging procedure clarified.
  5. [§5.1] The 'Direct' baseline in Table 1 is used in the comparisons but is never defined in Section 5.1; the prompting protocol for Direct should be specified.
  6. [§4.2 / Eq. (3)] The notation for the final reasoning output is inconsistent: Eq. (3) uses R_{t→v}, Section 4.2 first writes R and then writes R', and Figure 5 labels the final output as 'Final Reasoning'; please unify the notation.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline gains are measured on answer-fitted Oracle key-video; the benchmark-coverage validation is self-fulfilling, though the non-Oracle CLIP control gives the paradigm some independent support.

  1. self definitional [Section 3.1.2 and Section 3.3.2 (Key-Frames Recognition; Key-Frame Coverage of Text Reasoning)]
    "Given the video, question, reasoning steps, and answer, the model selects frame indices that best support accurately answering the question. ... our analysis shows that key-frame features largely overlap with and encompass nearly the entire feature space of textual reasoning content."

    The coverage analysis is presented as evidence that key-frames capture the essence of the reasoning process, but the key-frames were selected by giving Gemini-2.0-Flash the video, question, reasoning steps, and answer and asking it to pick frames that best support answering. The later CLIP/UMAP overlap between key-frame features and text-reasoning features is therefore entailed by the selection criterion; it does not independently validate the benchmark. The validation reduces to the construction rule that produced the key-frames.

  2. fitted input called prediction [Sections 3.1.2, 3.2, 5.2 and Table 1 / Figure 1d]
    "To assess the effectiveness and generalizability of the Video-Text Interleaved CoT paradigm, we apply ViTCoT to current mainstream CoT methods using the Oracle key-video in the main experiments (the performance using non-Oracle key-video is presented in Section 5.5). ... Given the video, question, reasoning steps, and answer, the model selects frame indices that best support accurately answering the question. ... we only retain data where all three assign scores above 80."

    The headline improvement of 5.5% on Qwen2.5-VL-7B is measured with an Oracle key-video that is constructed using the correct answer and then manually filtered so that, by the stated guideline, the key-video alone (90-100) or the key-video with the original video (80-90) supports deriving the correct answer. Feeding this answer-fitted key-video into stage-2 reasoning makes the reported gain an oracle-conditioned upper bound rather than a property of the answer-agnostic ViTCoT pipeline. The paper's own non-Oracle control with a CLIP rough key-video (Figure 7) shows only a 1.7% average gain, indicating that much of the headline margin is built into the input construction rather than derived by the paradigm itself.

full rationale

The proposed prompting scheme itself is not circular: equations (1)-(3) define a standard two-stage interleaved-reasoning procedure, and no load-bearing self-citation or imported uniqueness theorem is used. The circularity burden is in the evaluation design. The ViTIB key-video is selected with access to the reasoning steps and the answer, then human-rechecked under criteria that require the key-video to support the correct answer; consequently, Table 1 and Figure 1d measure an oracle-injected condition, not an answer-agnostic method. The Section 3.3.2 coverage claim is also self-fulfilling because the selection target is exactly the reasoning content whose coverage is later demonstrated. The paper partially breaks this loop with the both-video vanilla control (Figure 6, +2.8%) and the CLIP rough key-video control (Figure 7, +1.7%), which give the central idea some independent empirical content, but the headline 'significantly boosts performance' claim rests on the oracle evaluation. This warrants a partial-circularity score of 6 rather than a higher score, because the non-oracle controls prevent the central claim from being entirely forced by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on a self-built benchmark and on the availability of high-quality key-video. The oracle selection is the main driver of the headline gains; the CLIP-based control is a partial de-biasing. No theoretical entities are introduced.

free parameters (3)
  • Oracle key-video selection = Selected by Gemini-2.0-Flash plus human review
    The key-video used in main experiments is chosen using the question, reasoning steps, and correct answer, making it an ideal (oracle) hint. This selection is not a fitted number, but it is a hand-designed input that strongly affects the reported gains.
  • Human recheck score threshold = 80
    Samples are retained only when all three reviewers score above 80; this threshold shapes the benchmark composition and was chosen without external validation.
  • Video frame rate for key-video assembly = 1 FPS
    Key-video frames are sampled at 1 FPS; this choice affects the number of frames in each key-video and is not justified against alternatives.
assumptions (4)
  • domain assumption Interleaving key video frames into the textual reasoning process improves accuracy by mimicking human re-examination of visual content.
    Central premise motivating ViTCoT; not proven independently of the experiments.
  • domain assumption The Gemini-2.0-Flash selected key frames and human review guarantee that the key-video supports the correct answer without containing the explicit answer text.
    Benchmark construction assumes the key-video is relevant yet not leaky; leakage is not measured.
  • domain assumption CLIP similarity between initial reasoning and frames is a valid proxy for oracle key-video selection.
    Used in Section 5.5 as a non-oracle control; the retrieval quality is not evaluated against the oracle.
  • domain assumption The 14 categories of ViTIB are representative of video understanding tasks.
    ViTIB is derived from VideoEspresso; generalization to other video QA datasets is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models." pith.science (2026). https://pith.science/paper/SNGQJFUG

@misc{pith2026250709876,
  author       = {Pith},
  title        = {Pith review of: ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SNGQJFUG}},
  note         = {Machine review of arXiv:2507.09876}
}
read the original abstract

Video understanding plays a vital role in bridging low-level visual signals with high-level cognitive reasoning, and is fundamental to applications such as autonomous driving, embodied AI, and the broader pursuit of AGI. The rapid development of large language models (LLMs), particularly those utilizing Chain-of-Thought (CoT) technology, has significantly advanced video reasoning capabilities. However, current approaches primarily depend on textual information for reasoning, overlooking the visual modality in the actual video reasoning process. In contrast, humans naturally re-examine visual content while reasoning. Motivated by this, we introduce a novel video reasoning paradigm: Video-Text Interleaved CoT (ViTCoT), which facilitates more intuitive and cognitively aligned reasoning. To the end, first, we construct the Video-Text Interleaved Benchmark (ViTIB), which is created using MLLMs for key-video selection and manually verified. Furthermore, we extensively explore the potential of the ViTCoT paradigm in the video understanding field. Extensive experiments demonstrate that ViTCoT significantly enhances performance compared to the traditional text-only CoT paradigm and effectively activates more neuron values in MLLMs.

Figures

Figures reproduced from arXiv: 2507.09876 by the authors.

Figure 1
Figure 1. (a) Vanilla Text CoT Reasoning; (b) Video-Text Interleaved CoT Reasoning; (c) Video-Text Interleaved Data Construction; [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The construction of the Video-Text Interleaved Benchmark includes: (a) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Utilizing CLIP [28] to visualize the semantic fea￾tures of key-frames and textual reasoning content, where the semantic features of the key-frames encompass the entire scope of the reasoning content. 3.3.2 Key-Frame Coverage of Text Reasoning. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Display of the Number of Videos and Key Frames [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: The overall workflow of the Video-Text Interleaved Chain-of-Thought (ViTCoT) consists of main stages: (a) [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Compare the Video-Text Interleaved (ViT) reasoning method and Vanilla reasoning approach (with both the original [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: The performance comparison of only the Key-video [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: (a) A comparison of the number of activated neuron values between Vanilla reasoning and ViT reasoning on [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 26 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M. Hadzic, Taran Kota, Jimming He, Cristobal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Li Fei-Fei. 2024. HourVideo: 1-Hour Video-Language Understand- ing. In Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vo...

    work page 2024

  3. [3]

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. 2024. Sharegpt4video: Im- proving video understanding and generation with better captions. Advances in Neural Information Processing Systems 37 (2024), 19472–19495

    work page 2024

  4. [4]

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. Towards reason- ing era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567 (2025)

    arXiv 2025

  5. [5]

    Qiguang Chen, Libo Qin, Jiaqi Wang, Jingxuan Zhou, and Wanxiang Che. 2024. Unlocking the capabilities of thought: A reasoning boundary framework to quan- tify and optimize chain-of-thought. Advances in Neural Information Processing Systems 37 (2024), 54872–54904

    work page 2024

  6. [6]

    Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, et al . 2025. AI4Research: A Survey of Artificial Intelligence for Scientific Research. arXiv preprint arXiv:2507.01903 (2025)

    arXiv 2025

  7. [7]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al . 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 24185–24198

    2024

  8. [8]

    Sijie Cheng, Kechen Fang, Yangyang Yu, Sicheng Zhou, Bohao Li, Ye Tian, Tingguang Li, Lei Han, and Yang Liu. 2024. Videgothink: Assessing egocentric video understanding capabilities for embodied ai. arXiv preprint arXiv:2410.11623 (2024)

    arXiv 2024

Show all 59 references
  1. [9]

    Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang, Alex Jinpeng Wang, Zhi Chen, Wanxiang Che, et al. 2025. Visual thoughts: A unified perspective of understanding multimodal chain-of-thought. arXiv preprint arXiv:2505.15510 (2025)

    2025

  2. [10]

    Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. 2025. Comt: A novel benchmark for chain of multi-modal thought on large vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 23678–23686

    2025

  3. [11]

    Joseph Cho, Fachrina Dewi Puspitasari, Sheng Zheng, Jingyao Zheng, Lik-Hang Lee, Tae-Ho Kim, Choong Seon Hong, and Chaoning Zhang. 2024. Sora as an agi world model? a complete survey on text-to-video generation. arXiv preprint arXiv:2403.05131 (2024)

    2024

  4. [12]

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, Tianren Gao, Erlong Li, Kun Tang, Zhipeng Cao, Tong Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao, Ziran Wang, and Chao Zheng. 2024. A Survey on Multimoda...

    2024

  5. [13]

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, and Tat-Seng Chua. 2024. Dysen-VDM: Empowering Dynamics-Aware Text-to-Video Diffusion with LLMs. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE Computer Society, 7641–7653

    2024

  6. [14]

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong Li Lee, and Wynne Hsu. 2024. Video-of-thought: step-by-step video reasoning from perception to cognition. In Proceedings of the 41st International Conference on Machine Learning. 13109–13125

    2024

  7. [15]

    Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. 2024. Enhancing Video-Language Representations With Struc- tural Spatio-Temporal Alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 12 (2024), 7701–7719

    2024

  8. [16]

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. 2025. Video- r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776 (2025)

    2025 arXiv

  9. [17]

    Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su, Shifeng Zhang, Xu Zhou, Xiaojuan Qi, Yue Liao, and Si Liu. 2025. VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection. 2025 IEEE/CVF Conference on Computer Vision a...

    2025

  10. [18]

    Vaishnavi Himakunthala, Andy Ouyang, Daniel Rose, Ryan He, Alex Mei, Yujie Lu, Chinmay Sonar, Michael Saxon, and William Yang Wang. 2023. Let’s think frame by frame with vip: A video infilling and prediction dataset for evaluating video chain-of-thought. arXiv preprint arXiv:2...

    2023 arXiv

  11. [19]

    Jian Hu, Zixu Cheng, Chenyang Si, Wei Li, and Shaogang Gong. 2025. CoS: Chain- of-Shot Prompting for Long Video Understanding.arXiv preprint arXiv:2502.06428 (2025)

    2025 arXiv

  12. [20]

    Wei Ji, Renjie Liang, Lizi Liao, Hao Fei, and Fuli Feng. 2023. Partial annotation- based video moment retrieval via iterative learning. In Proceedings of the 31st ACM International Conference on Multimedia . 4330–4339

    2023

  13. [21]

    Gal Lavee, Ehud Rivlin, and Michael Rudzsky. 2009. Understanding video events: A survey of methods for automatic interpretation of semantic occurrences in video. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 39, 5 (2009), 489–504

    2009

  14. [22]

    Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. 2024. A Survey of Multimodel Large Language Models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering. 405–409

    2024

  15. [23]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. 2024. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Lun...

    2024 doi

  16. [24]

    Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform man- ifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)

    2018 arXiv

  17. [25]

    Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. [n. d.]. What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

  18. [26]

    Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024. Large language models meet nlp: A survey. arXiv preprint arXiv:2405.12819 (2024)

    2024 arXiv

  19. [27]

    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2025. A survey of multilingual large language models. Patterns 6, 1 (2025)

    2025

  20. [28]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  21. [29]

    Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. 2023. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432 (2023)

    2023

  22. [30]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)

    2024 arXiv

  23. [31]

    Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. 2023. Towards Open-Vocabulary Video Instance Seg- mentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 4057–4066

    2023

  24. [32]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain- of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091 (2023)

    2023 arXiv

  25. [33]

    Peng Wang, Yongheng Zhang, Hao Fei, Qiguang Chen, Yukai Wang, Jiasheng Si, Wenpeng Lu, Min Li, and Libo Qin. 2024. S3 agent: Unlocking the power of VLLM for zero-shot multi-modal sarcasm detection. ACM Transactions on Multimedia Computing, Communications and Applications (2024)

    2024

  26. [34]

    Shuai Wang, Ivona Najdenkoska, Hongyi Zhu, Stevan Rudinac, Monika Kack- ovic, Nachoem Wijnberg, and Marcel Worring. 2025. ArtRAG: Retrieval- Augmented Generation with Structured Context for Visual Art Understanding. arXiv:2505.06020 [cs.AI] https://arxiv.org/abs/2505.06020

    2025 arXiv

  27. [35]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    2022 arXiv

  28. [36]

    Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, William Wang, Ziwei Liu, Jiebo Luo, and Hao Fei. 2025. Multimodal Chain-of-Thought Reasoning: A Compre- hensive Survey. arXiv preprint arXiv:2503.12605 (2025)

    2025 arXiv

  29. [37]

    Yan Wang, Yawen Zeng, Jingsheng Zheng, Xiaofen Xing, Jin Xu, and Xiangmin Xu. 2024. VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool. In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR). 92–101

    2024

  30. [38]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837. ACM MM ’25, October 27–...

    2022

  31. [39]

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang

  32. [40]

    Chao-Yuan Wu and Philipp Krahenbuhl. 2021. Towards long-form video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1884–1894

    2021

  33. [41]

    Hao Wu, Fan Xu, Chong Chen, Xian-Sheng Hua, Xiao Luo, and Haixin Wang

  34. [42]

    Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C Gee, and Yixin Nie. 2023. The role of chain-of-thought in complex vision-language reasoning task. arXiv preprint arXiv:2311.09193 (2023)

    2023 arXiv

  35. [43]

    In Proceedings of the 32nd ACM International Conference on Multimedia

    Pastnet: Introducing physical inductive biases for spatio-temporal video prediction. In Proceedings of the 32nd ACM International Conference on Multimedia. 2917–2926

  36. [44]

    Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. 2024. A survey on video diffusion models. Comput. Surveys 57, 2 (2024), 1–42

    2024

  37. [45]

    Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. 2024. Beyond chain-of-thought: A survey of chain-of-x paradigms for llms. arXiv preprint arXiv:2404.15676 (2024)

    2024 arXiv

  38. [46]

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont- Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer ...

    2023

  39. [47]

    Zhipeng Xu, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Chaojun Xiao, Zhiyuan Liu, Ge Yu, and Chenyan Xiong. 2024. ActiveRAG: Autonomously Knowledge Assimilation and Accommodation through Retrieval-Augmented Agents. arXiv preprint arXiv:2402.13547 (2024)

    2024

  40. [48]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems 36 (2023), 11809–11822

    2023

  41. [49]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  42. [50]

    Tong Yue, Mingrui Xiao, Dafeng Zhang, Xin Liu, Yali Li, and Shengjin Wang. 2025. Find Details in Long Videos: Tower-of-Thoughts and Self-Retrieval Augmented Generation for Video Understanding. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Pr...

    2025

  43. [51]

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. 2024. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006 (2024)

    2024 arXiv

  44. [52]

    Yongheng Zhang, Qiguang Chen, Min Li, Wanxiang Che, and Libo Qin. 2024. AutoCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thought. In Findings of the Association for Computational Linguistics ACL 2024. 9191–9200

    2024

  45. [53]

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. 2025. VideoLLaMA 3: Frontier Multi- modal Foundation Models for Image and Video Understan...

    2025 arXiv

  46. [54]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 1, 2 (2023)

    2023 arXiv

  47. [55]

    Yongheng Zhang, Xu Liu, Ruoxi Zhou, Qiguang Chen, Hao Fei, Wenpeng Lu, and Libo Qin. 2025. CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Models. arXiv preprint arXiv:2505.19108 (2025)

    2025 arXiv

  48. [56]

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024. Mlvu: A comprehensive bench- mark for multi-task long video understanding. arXiv preprint arXiv:2406.04264 (2024)

    2024 arXiv

  49. [57]

    Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. 2023. Learning video representations from large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6586–6597

    2023

  50. [59]

    Pengyuan Zhou, Lin Wang, Zhi Liu, Yanbin Hao, Pan Hui, Sasu Tarkoma, and Jussi Kangasharju. 2024. A survey on generative ai and llm for video generation, understanding, and streaming. arXiv preprint arXiv:2404.16038 (2024)

    2024 arXiv

  51. [2024]

    In European Conference on Computer Vision

    Longvlm: Efficient long video understanding via large language models. In European Conference on Computer Vision . Springer, 453–470

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.