Pith. sign in

REVIEW 3 major objections 3 minor 122 references

VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

T0 review · 3 major / 3 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VideoChat3 claims a 4B, fully open video model can beat larger open rivals by compressing visual tokens 16x before they reach the language model, while also handling streaming video.

desk verdict Serious open systems paper with a genuine architecture/data contribution, but the temporal-grounding and streaming comparisons need a fairness check before the headline numbers can be trusted. read the letter →

arxiv 2607.14935 v1 pith:IBJWLJI4 submitted 2026-07-16 cs.CV

Xinhao Li , Yuhan Zhu , Xiangyu Zeng , Yuhao Dong , Haoning Wu , Zhiqiu Zhang , Yuandong Yang , Changlian Ma
show 19 more authors
Qingyu Zhang Yansong Shi Xinyu Chen Haoran Chen Zizheng Huang Jun Zhang Kun Ouyang Lin Sui Ziang Yan Yicheng Xu Chenting Wang Yinan He Hongjie Zhang Yi Wang Yu Qiao Yali Wang Ziwei Liu Kai Chen Limin Wang
This is my paper · ORCID
classification cs.CV
keywords videomultimodallargelanguagemodelsefficienttokenization3Dvisiontransformertemporalgroundingstreamingunderstandinginstructiontuningopen-sourcedatasetslong
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a video multimodal language model can be both broadly capable and computationally cheap if video redundancy is removed inside the vision encoder rather than by sparse frame sampling. VideoChat3, at 4B parameters, uses an inflated 3D vision transformer that chunks frames, applies spatiotemporal attention, and temporally pools, achieving a 16x spatiotemporal compression before tokens reach the language model. The authors report that this is enough to surpass prior open models of equal or larger size across general, long-form, and streaming benchmarks: 18 of 19 direct comparisons over a leading 4B baseline, matching or beating another 4B open model on most benchmarks, with roughly half the visual tokens and substantially lower long-video latency. A second mechanism, adaptive frame resolution, lets the model spend more visual budget only after a 'standby' cue during streaming. The claim matters because it would mean efficiency and generality need not trade off, and because the full stack—weights, code, training strategy, and datasets—is released.

What carries the argument

The load-bearing object is the Inflated 3D Vision Transformer (I3D-ViT), a visual tokenizer created by inflating a pretrained image ViT's 2D self-attention into chunk-wise spatiotemporal attention over T-frame windows, adding learned temporal positional embeddings, and then applying chunk-wise temporal pooling. Combined with 2x2 pixel-shuffle spatial downsampling and T=4, it gives a 16x spatiotemporal compression ratio, halving the visual tokens a baseline would produce while preserving motion cues. The second mechanism is Adaptive Frame Resolution: at each streaming step the LLM emits a response-state token (Silence, Standby, Response), and a deterministic controller sets the next window's

What would settle it

Delete the TimeLens-100K (and any same-source grounding) samples from the Stage-2/3 mixtures, retrain, and re-measure the three TimeLens splits plus VUE-TR and MomentSeeker; if the 6-20 point grounding margins shrink to noise, the temporal-grounding claim is an artifact of train/test overlap. In parallel, re-run the streaming baselines under the same [0,1]-vs-[0,1000] bounding-box normalization and identical evaluation code; if the ODVBench and OVO-Timing gaps disappear, the streaming claim is protocol-driven.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that compressing video early—inside a vision tokenizer that models local space-time before the LLM—preserves motion evidence while cutting LLM context cost. I3D-ViT inflates a pretrained image ViT's 2D attention into chunk-wise 3D spatiotemporal attention with learned temporal positions, then pools over T frames; with T=4 and 2x2 spatial pixel-shuffle downsampling, the visual sequence shrinks 16x. This changes the compute profile: the vision encoder's cost grows roughly linearly in frames, while the LLM's quadratic context cost drops, so long-video inference becomes faster and lighter. The paper further shows the same tokenizer suppo

Load-bearing premise

The comparative claims stand or fall on the assumption that every model was evaluated under identical protocols with no training/test overlap; the paper's own tables show temporal-grounding training data derived from the same datasets as the temporal-grounding evaluation splits, and streaming baselines are mostly quoted while VideoChat3 uses a modified bounding-box output range.

Editorial extensions

If this is right

  • The efficiency claim is concrete: with 256 to 2048 input frames, VideoChat3 produces exactly half the visual tokens of the compared baseline under equal ViT patches per frame, and at 2048 frames the reported end-to-end latency drops from 44.4 s to 20.4 s while FLOPs fall by more than 60%.
  • Because the model is fully open (weights, code, training strategy, and all three datasets), any research group can replicate the training pipeline and extend it without reverse engineering.
  • The streaming training recipe—state tokens with a balanced transition mask and a deterministic low/high pixel-budget controller—is the paper's answer to 'when to answer' and can be applied to other video MLLMs.
  • The evidence-grounded annotation pipeline increases supervision density from sparse academic labels and is reported to improve temporal perception and grounding consistently, suggesting data quality rather than raw scale drives much of the gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 16x early-compression design is the real driver, it should transfer to other base LLMs and vision encoders; a controlled swap experiment would reveal how much of the gain is architectural rather than data-specific.
  • The adaptive frame-resolution controller could be lifted from streaming into offline long-video pipelines: a salience estimate, rather than a state token, would decide which segments get the high 448² pixel budget, potentially cutting cost without a real-time loop.
  • Given the TimeLens training/evaluation overlap noted in Sections 4.3 and 5.1, the most informative independent check is a zero-shot run on temporal-grounding benchmarks whose source datasets were never in the training mixture; the reported margins may not transfer.
  • A cheaper-to-reproduce variant—distilling the synthetic long-video and streaming supervision into a smaller open model—would test whether the state-transition masking and adaptive budget still yield proactive behavior at lower cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper presents VideoChat3, a 4B-parameter fully open video-centric MLLM, with two main contributions: an Inflated 3D Vision Transformer (I3D-ViT) that performs chunk-wise spatiotemporal attention and temporal pooling to reduce visual tokens, and an Adaptive Frame Resolution mechanism for streaming video that toggles per-frame pixel budgets based on predicted response-state tokens. The authors also release three training datasets (VideoChat3-Academic2M, VideoChat3-LV116K, VideoChat3-OL617K), a four-stage training curriculum, model weights, and code. Empirically, the paper claims strong generalization across temporal perception, long-video QA, temporal grounding, and streaming/proactive benchmarks, with particular gains over Qwen3-VL-4B and Molmo2-4B, plus efficiency improvements from reduced visual-token counts.

Significance. If the reported results are taken at face value, this is a substantial systems-and-data contribution: it demonstrates that early spatiotemporal compression in the visual tokenizer can preserve broad video understanding while cutting inference cost, and it provides fully released assets for reproducible research. The ablation isolating the state-transition mask and the dynamic perception-window policy is well designed and informative. However, the paper's strongest empirical claims—especially in temporal grounding and streaming—rest on benchmark comparisons whose train/eval disjointness and protocol symmetry are not established. The contribution is therefore potentially significant but currently conditional on resolving these evaluation concerns.

major comments (3)
  1. [Sections 4.3/4.4 vs 5.1, Table 2] TimeLens-100K [14] is included in both the Stage-2 and Stage-3 training mixtures, and temporal grounding is evaluated on the TimeLens suite [14] built from Charades-STA, ActivityNet Captions, and QVHighlights. No evidence is provided that the training and evaluation splits are disjoint at the video or annotation level. Because the largest differentials in Table 2 are on the three TimeLens columns (+9.7/+6.4/+8.3 over Qwen3-VL-4B, and +16.4/+29.8/+34.3 over VideoChat-Flash-7B), these gains are load-bearing for the claim of surpassing prior open models in temporal grounding. Please provide exact overlap statistics between TimeLens-100K training samples and each evaluation split, and re-report results after excluding overlapping items or on independent held-out grounding benchmarks.
  2. [Section 4.4 vs 5.2, Table 3] Stage-3 training explicitly includes StreamForest [19], while the streaming evaluation includes ODVBench [19], which is the benchmark introduced in the StreamForest paper. The paper does not establish that the StreamForest training data and the ODVBench evaluation clips are disjoint. Similarly, River [87] appears to be an author-created benchmark, and the Stage-3 mixture includes StreamForest/Streamo/Seeker data; no exclusion analysis is reported. Given that Table 3 claims best results on four of six streaming metrics, this potential overlap must be quantified and the results recomputed on non-overlapping data before the streaming claims can be accepted.
  3. [Appendix Table 11, Table 3] The evaluation protocol for VideoChat3 is not symmetric with the baselines. For ODVBench and OVBench, the paper modifies the bounding-box output range from [0,1] to [0,1000] when evaluating VideoChat3, while most baseline results are quoted from their original papers. Likewise, Table 12 applies custom low/high token budgets and high-budget frame counts to the proactive-response benchmarks. Since the streaming comparison is cross-paper, such protocol differences could explain part of the reported improvements. Please run at least the strongest open baselines (e.g., StreamForest, StreamingVLM, Qwen3-VL-4B) under the same evaluation harness and output-format conversions, and report all models with matched token/frame budgets.
minor comments (3)
  1. [Throughout] The PDF contains recurring ligature artifacts such as 'efficiency', 'sufficient', and 'official'. These should be corrected in the camera-ready version.
  2. [Table 9 / Section 5.2] Adding VideoChat3-OL617K substantially decreases the ProactiveVQA WEB score (39.0 to 28.4). The text says gains extend across streaming benchmarks without acknowledging this drop. Please add a sentence discussing this trade-off.
  3. [Section 3 / 7] The term 'fully open' should be qualified: the model weights and training assets are released, but a large fraction of the supervision is generated by proprietary models (Qwen3-VL-235B-A22B and Gemini). The assets are open, but the annotation pipeline is not fully reproducible without access to those closed APIs.

Circularity Check

3 steps flagged · score 4.0 of 10

Several headline metrics are evaluated on benchmarks that also appear in the training mixture (MotionBench, TimeLens/Charades/ActivityNet, StreamForest/ODVBench), so those columns are partly fitted rather than independent; the architecture and most external benchmarks remain independent.

  1. fitted input called prediction [Section 4.3 (Stage-2 data mixture) vs Section 5.1 / Table 2 (MotionBench val [56])]
    "Vript [28], PE-Video [37], TGIF [53], CLEVRER [54], Something-Something-v2 [43], EgoIT [55], MotionBench [56], ActivityNet [41], LaSOT/GOT-10k [57, 58], and related tracking/localization datasets broaden coverage over generic video captioning, action recognition, procedural/egocentric understanding, object tracking, physical/causal reasoning, and fine motion discrimination."

    MotionBench is explicitly placed in the Stage-2 training mixture, and Table 2 then reports 'MotionBench val' as an evaluation column. The model parameters are fitted on MotionBench data, so the reported MotionBench score is a fitted quantity rather than a prediction on an untouched distribution. No split-disjointness statement is provided, making the +3.1 advantage over Qwen3-VL-4B on this column non-independent.

  2. fitted input called prediction [Section 4.3/4.4 (Stage-2/3 data: 'TimeLens-100K [14]'; Stage-0/2 include Charades [42] and ActivityNet [41]) vs Section 5.1 / Table 2 (TimeLens suite)]
    "We further evaluate temporal grounding on the TimeLens suite [14], covering Charades-STA [72], ActivityNet Captions [73], and QVHighlights [74]"

    The Stage-2 and Stage-3 training mixtures explicitly include TimeLens-100K [14], and Stage-0/Stage-2 also include Charades [42] and ActivityNet [41] video sources. The temporal-grounding evaluation is then run on the TimeLens suite, which the paper itself defines as Charades-STA, ActivityNet Captions, and QVHighlights. Since TimeLens-100K shares citation [14] with the evaluation suite and no split-disjointness analysis is reported, the large TimeLens gains (+9.7/+6.4/+8.3 over Qwen3-VL-4B; +16.4/+29.8/+34.3 over VideoChat-Flash-7B) are construction-dependent unless the released splits are provably disjoint.

1 more flagged steps
  1. fitted input called prediction [Section 4.4 (Stage-3 data: StreamForest [19]) and Figure 8 vs Section 5.2 / Table 3 (ODVBench [19])]
    "Additional video datasets, including TimeLens-100K [14], Molmo2 [5], Vript [28], LLaVA-Video [26], CinePile [51], MotionBench [56], TGIF/TVQA-style QA [49, 53], AVA/EgoQA [59, 60], and StreamForest [19], provide complementary coverage of subtitle QA, dense captioning, motion understanding, tracking, spatial grounding, and driving-scene reasoning."

    Stage-3 trains on StreamForest [19], and Figure 8 shows that VideoChat3-OL617K contains 272,424 StreamForest General and 46,538 StreamForest Drive instances. The streaming evaluation then reports ODVBench [19] as a headline metric (72.3, +12.4 over StreamForest). ODVBench comes from the same cited work as the training data, so the ODVBench advantage is not independent of the fitted streaming data. This is compounded by Appendix Table 11, which modifies the bounding-box output range to [0,1000] for ODVBench/OVBench while most baselines are quoted from original papers.

full rationale

The core architecture and efficiency results are self-contained and not circular: I3D-ViT's 16x compression is a design property, and the efficiency table (Table 4) is a direct measurement independent of any benchmark fit. The general-video understanding claim is also substantially supported by external benchmarks not present in the training mixtures, e.g., Video-MME, LVBench, TempCompass, TOMATO, MMVU, VUE-TR, MomentSeeker, OVOBench, and StreamingBench. The circularity is partial and concentrated in the columns where a benchmark or its constituent source data is listed in the training recipe: MotionBench, the TimeLens suite / Charades / ActivityNet, and StreamForest / ODVBench. These contaminated columns happen to include some of the largest reported differentials, so the headline 'surpasses prior open-source models' is partly fitted rather than fully independent. Self-citation alone (TimeLens, StreamForest, River, etc.) is not treated as circular; it becomes a problem only because the same cited resources appear on both the training and evaluation sides without disclosed split disjointness. Score 4 reflects partial, not wholesale, circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The central claims rest on domain assumptions about transfer of pretrained visual features, trustworthiness of LLM-generated annotations, teacher-forced state supervision transferring to inference, and VLM-based clue verification. The comparative results also depend on several hand-set architecture and evaluation hyperparameters, plus unreported data-filtering thresholds.

free parameters (6)
  • Temporal chunk size T = 4
    Default temporal pooling factor; combined with 2x2 spatial merging it yields the 16x compression quoted throughout. No sensitivity sweep is reported. Section 2.2.
  • Streaming resolution quotas B_low and B_high = 224^2 / 448^2 pixels
    Hand-selected controller thresholds. The ablation compares fixed low, fixed high, and dynamic policies, but does not sweep the quota values. Section 2.3.
  • Standby high-budget frame count = 3 for OVO-Timing, 1 for ProactiveVQA
    Per-benchmark evaluation hyperparameter that directly affects proactive-response scores. Appendix Table 12.
  • State-supervision continuation ratio = equal number of continuation and transition positions (2:2:1 target)
    Balancing factor for the state-token loss. No alternative ratios are ablated. Section 4.4.
  • Benchmark token and pixel budgets = 80K/128K visual tokens; 448^2-768^2 pixels per frame
    Per-benchmark evaluation configurations chosen by the authors. They are applied to VideoChat3 and Qwen3-VL, but most other baselines are quoted from original papers. Appendix Tables 10-12.
  • Data-pipeline filtering thresholds = not reported
    Minimum/maximum segment durations, semantic-similarity filters, and generic-caption filters determine LV116K and OL617K quality, but their exact values are not specified. Sections 3.3-3.4.
assumptions (4)
  • domain assumption Inflating a pretrained image ViT to chunk-wise 3D attention preserves image-level perception while improving video perception.
    This is the central architectural bet. It is empirically supported by Table 5, but not derived; image benchmarks are close while video benchmarks improve.
  • domain assumption LLM-rewritten academic annotations, after judge-model verification, remain faithful to the original labels.
    The quality of the 2M-sample academic dataset rests on Qwen3-VL-235B rewriting and judging. No human evaluation of the rewritten annotations is reported. Section 3.2.
  • domain assumption Teacher-forced state-token supervision transfers to closed-loop inference, where predicted states drive the next window's resolution.
    At inference, the controller uses predicted s_t rather than ground truth. The end-to-end model is tested, but the transfer is not isolated. Sections 2.3 and 4.4.
  • domain assumption VLM clue localization and verification reliably identify the temporal intervals that contain the evidence needed for streaming answers.
    The OL617K pipeline assumes a VLM can localize and verify evidence intervals, then filters on whether the answer is inferable from the cropped interval. Sections 3.4.
invented entities (1)
  • Response-state tokens </Silence>, </Standby>, </Response>
    purpose: Special output vocabulary symbols that control streaming behavior, response timing, and the next window's visual resolution.
    These are internal trained tokens with no falsifiable handle outside the paper; they are teacher-forced during training and predicted at inference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding." pith.science (2026). https://pith.science/paper/IBJWLJI4

@misc{pith2026260714935,
  author       = {Pith},
  title        = {Pith review of: VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IBJWLJI4}},
  note         = {Machine review of arXiv:2607.14935}
}
read the original abstract

Recent advances in video understanding have spanned motion, long video, and streaming interaction, driving this field toward real-world applications. Despite this progress, current open-source models remain limited in several ways. They often struggle to generalize across diverse video types, making them effective only in specific domains. High computational demands further restrict their efficiency and scalability. Moreover, most models are only partially open, with key components such as training code, strategy, or datasets unavailable, which hinders reproducibility and slows community-driven development. To address these issues, we introduce VideoChat3, a fully open, efficient, and generalist video-centric MLLM. VideoChat3 advances video understanding through two complementary designs. For efficiency, we introduce Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for Streaming Video Perception, which enables efficient spatiotemporal representation and reduces the cost of processing video inputs during training and inference. For effectiveness, we develop a scalable video data synthesis pipeline that curates three diverse, high-quality training datasets: VideoChat3-Academic2M, VideoChat3-LV116K, and VideoChat3-OL617K, covering general, long-form, and streaming video scenarios, improving the model's generalization across domains. By integrating these designs, VideoChat3 achieves a rare balance of broad generalization and computational efficiency. Experiments across general, long-form, and streaming benchmarks demonstrate that VideoChat3 surpasses prior open-source models with equal or larger parameter counts with only 4B parameters and higher efficiency.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

122 extracted references · 57 linked inside Pith

  1. [14]

    Timelens: Rethinking video temporal grounding with multimodal llms

    Jun Zhang, Teng Wang, Yuying Ge, Yixiao Ge, Xinhao Li, Ying Shan, and Limin Wang. Timelens: Rethinking video temporal grounding with multimodal llms. arXiv preprint arXiv:2512.14698 , 2025. 22

    arXiv 2025

  2. [19]

    Streamforest: Efficient online video understanding with persistent event memory

    Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang, Jiaxin Li, Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, et al. Streamforest: Efficient online video understanding with persistent event memory. Advances in Neural Information Processing Systems , 38:75804–75835, 2026

    2026

  3. [87]

    River: A real-time interaction benchmark for video llms

    Yansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, and Limin Wang. River: A real-time interaction benchmark for video llms. arXiv preprint arXiv:2603.03985 , 2026

    arXiv 2026

  4. [1]

    Qwen3-vl technical report

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 , 2025

    arXiv 2025

  5. [2]

    Internvideo2.5: Empowering video mllms with long and rich context modeling

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2.5: Empowering video mllms with long and rich context modeling. arXiv preprint arXiv:2501.12386 , 2025

    arXiv 2025

  6. [3]

    Videochat-flash: Hierarchical compression for long-context video modeling

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574 , 2024

    arXiv 2024

  7. [4]

    Perceptionlm: Open-access data and models for detailed visual understanding

    Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, et al. Perceptionlm: Open-access data and models for detailed visual understanding. arXiv preprint arXiv:2504.13180 , 2025

    arXiv 2025

  8. [5]

    Molmo2: Open weights and data for vision-language models with video understanding and grounding

    Christopher Clark, Jieyu Zhang, Zixian Ma, Jae Sung Park, Mohammadreza Salehi, Rohun Tripathi, Sangho Lee, Zhongzheng Ren, Chris Dongjoo Kim, Yinuo Yang, et al. Molmo2: Open weights and data for vision-language models with video understanding and grounding. arXiv preprint arXiv:2601.10611 , 2026

    arXiv 2026

Show all 122 references
  1. [6]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  2. [7]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  3. [8]

    Videochat-r1

    Ziang Yan, Yinan He, Xinhao Li, Zhengrong Yue, Xiangyu Zeng, Yali Wang, Yu Qiao, Limin Wang, and Yi Wang. Videochat-r1. 5: Visual test-time scaling to reinforce multimodal reasoning by iterative perception. Advances in Neural Information Processing Systems , 38:119152–119184, 2026

    2026

  4. [9]

    Video-xl-2: Towards very long-video understanding through task-aware kv sparsification

    Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, and Zheng Liu. Video-xl-2: Towards very long-video understanding through task-aware kv sparsification. arXiv preprint arXiv:2506.19225 , 2025

    2025 arXiv

  5. [10]

    Video-o3: Native interleaved clue seeking for long video multi-hop reasoning

    Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang, Changlian Ma, Qingyu Zhang, Zizheng Huang, Kun Ouyang, Tianxiang Jiang, et al. Video-o3: Native interleaved clue seeking for long video multi-hop reasoning. arXiv preprint arXiv:2601.23224 , 2026

    2026 arXiv

  6. [11]

    Spacer: Reinforcing mllms in video spatial reasoning

    Kun Ouyang, Yuanxin Liu, Haoning Wu, Yi Liu, Hao Zhou, Jie Zhou, Fandong Meng, and Xu Sun. Spacer: Reinforcing mllms in video spatial reasoning. arXiv preprint arXiv:2504.01805 , 2025

    2025 arXiv

  7. [12]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14313–14323, 2024

    2024

  8. [13]

    Timesuite: Improving mllms for long video understanding via grounded tuning

    Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning. In International Conference on Learning Representations , volume...

    2025

  9. [15]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  10. [16]

    Flash-vstream: Memory-based real-time understanding for long video streams

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085 , 2024

    2024 arXiv

  11. [17]

    Online video understanding: Ovbench and videochat-online

    Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xiangyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: Ovbench and videochat-online. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 3328–3338, 2025

    2025

  12. [18]

    Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reac- tion

    Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time interaction via disentangled perception, decision, and reac- tion. In Proceedings of the Computer Vision and Pattern Recogn...

    2025

  13. [20]

    Streaming video instruction tuning

    Jiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun, and Kaiyang Zhou. Streaming video instruction tuning. arXiv preprint arXiv:2512.21334 , 2025

    2025 arXiv

  14. [21]

    Streambridge: Turning your offline video large language model into a proactive streaming assistant

    Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. Streambridge: Turning your offline video large language model into a proactive streaming assistant. Advances in Neural Information Processing Systems , 38:132332–132359, 2026

    2026

  15. [22]

    Gemini 3: News and announcements

    Google. Gemini 3: News and announcements. https://blog.google/products-and-platforms/products/ gemini/gemini-3-collection/ , 2025

    2025

  16. [23]

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276 , 2026

    2026 arXiv

  17. [24]

    Bytedance Seed. Seed1. 8 model card: Towards generalized real-world agency. arXiv preprint arXiv:2603.20633 , 2026

    2026 arXiv

  18. [25]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites,

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang,...

  19. [26]

    Llava-video: Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava-video: Video instruction tuning with synthetic data. Transactions on Machine Learning Research , 2025

    2025

  20. [27]

    Spoken moments: Learning joint audio-visual representations from video descriptions

    Mathew Monfort, SouYoung Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , p...

    2021

  21. [28]

    Vript: A video is worth thousands of words, 2024

    Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words, 2024. URL https://arxiv.org/abs/2406.06040

    2024 arXiv

  22. [29]

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yan...

    2025 arXiv

  23. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  24. [31]

    Caprl: Stimulating dense image caption capabilities via reinforcement learning

    Long Xing, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jianze Liang, Qidong Huang, Jiaqi Wang, Feng Wu, and Dahua Lin. Caprl: Stimulating dense image caption capabilities via reinforcement learning. arXiv preprint arXiv:2509.22647, 2025

    2025

  25. [32]

    Sharegpt4v: Improving large multi-modal models with better captions, 2023

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions, 2023. URL https://arxiv.org/abs/2311.12793

    2023 arXiv

  26. [33]

    Densefusion-1m: Merging vision experts for comprehensive multimodal perception, 2024

    Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang, Xinlong Wang, and Ling-Yu Duan. Densefusion-1m: Merging vision experts for comprehensive multimodal perception, 2024. URL https://arxiv.org/abs/2407. 08303

    2024

  27. [34]

    Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computa...

    2018 doi

  28. [35]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision , 2021

    2021

  29. [36]

    The kinetics human action video dataset, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017. URL https://arxiv.org/abs/1705.06950

    2017 arXiv

  30. [37]

    Perception encoder: The best visual embeddings are not at the output of the network, 2025

    Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, and Christoph Feichtenhofer. Perception encoder: ...

    2025 arXiv

  31. [38]

    Tarsier2: Advancing large vision- language models from detailed video description to comprehensive video understanding, 2025

    Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advancing large vision- language models from detailed video description to comprehensive video understanding, 2025. URL https: //arxiv.org/abs/2501.07888

    2025 arXiv

  32. [39]

    Movie description, 2016

    Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description, 2016. URL https://arxiv.org/abs/1605.03705

    2016 arXiv

  33. [40]

    Vatex: A large-scale, high-quality multilingual dataset for video-and-language research, 2020

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research, 2020. URL https://arxiv.org/abs/1904. 03493

    2020

  34. [41]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015

    2015

  35. [42]

    Hollywood in homes: Crowdsourcing data collection for activity understanding

    Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European conference on computer vision , pages 510–526. Springer, 2016

    2016

  36. [43]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In Pr...

    2017

  37. [44]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision ...

    2022

  38. [45]

    Sharegemini: Scaling up video caption data for multimodal large language models, June 2024

    Share. Sharegemini: Scaling up video caption data for multimodal large language models, June 2024. URL https://github.com/Share14/ShareGemini

    2024

  39. [46]

    Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024

    Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024. URL https://sharegpt4o.github.io/. 24

    2024

  40. [47]

    Bee: A high-quality corpus and full-stack suite to unlock advanced fully open mllms, 2026

    Yi Zhang, Bolin Ni, Xin-Sheng Chen, Heng-Rui Zhang, Yongming Rao, Houwen Peng, Qinglin Lu, Han Hu, Meng-Hao Guo, and Shi-Min Hu. Bee: A high-quality corpus and full-stack suite to unlock advanced fully open mllms, 2026. URL https://arxiv.org/abs/2510.13795

    2026

  41. [48]

    Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groen- eveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Sol- daini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pe...

  42. [49]

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. Tvqa: Localized, compositional video question answering, 2019. URL https://arxiv.org/abs/1809.01696

    2019 arXiv

  43. [50]

    Finevideo

    Miquel Farré, Andi Marafioti, Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Finevideo. https: //huggingface.co/datasets/HuggingFaceFV/finevideo, 2024

    2024

  44. [51]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813 , 2024

    2024 arXiv

  45. [52]

    Videogpt+: Integrating image and video encoders for enhanced video understanding, 2024

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding, 2024. URL https://arxiv.org/abs/2406.09418

    2024 arXiv

  46. [53]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering, 2017

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering, 2017. URL https://arxiv.org/abs/1704.04497

    2017 arXiv

  47. [54]

    Tenenbaum

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning, 2020. URL https://arxiv.org/abs/1910. 01442

    2020

  48. [55]

    Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding, 2025

    Haoyu Zhang, Qiaohui Chu, Meng Liu, Haoxiang Shi, Yaowei Wang, and Liqiang Nie. Exo2ego: Exocentric knowledge guided mllm for egocentric video understanding, 2025. URL https://arxiv.org/abs/2503.09143

    2025

  49. [56]

    Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models

    Wenyi Hong, Yean Cheng, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, and Jie Tang. Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models. In Proceedings of the IEEE/CVF Conference on Computer V...

    2025

  50. [57]

    Lasot: A high-quality benchmark for large-scale single object tracking

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages...

    2019

  51. [58]

    Got-10k: A large high-diversity benchmark for generic object tracking in the wild

    Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(5):1562–1577, May

  52. [59]

    Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijaya- narasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik

    Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijaya- narasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. A va: A video dataset of spatio-temporally localized atomic visual actions, ...

    2018

  53. [60]

    Egovqa-an egocentric video question answering benchmark dataset

    Chenyou Fan. Egovqa-an egocentric video question answering benchmark dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 0–0, 2019

    2019

  54. [61]

    Learning transferable temporal primitives for video reasoning via synthetic videos, 2026

    Songtao Jiang, Sibo Song, Chenyi Zhou, Yuan Wang, Ruizhe Chen, Tongkun Guan, Ruilin Luo, Yan Zhang, Zhihang Tang, Yuchong Sun, Hang Zhang, Zhibo Yang, Shuai Bai, Junyang Lin, and Zuozhu Liu. Learning transferable temporal primitives for video reasoning via synthetic videos, 20...

    2026

  55. [62]

    Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models

    Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: Assessing visual temporal reasoning capabilities in multimodal foundation models. In The Thirteenth International Conference on Learning Representations , 2025

    2025

  56. [63]

    Tvbench: Re- designing video-language evaluation

    Daniel Cores, Michael Dorkenwald, Manuel Mucientes, Cees GM Snoek, and Yuki M Asano. Tvbench: Re- designing video-language evaluation. arXiv preprint arXiv:2410.07752 , 2024

    2024 arXiv

  57. [64]

    Tempcompass: Do video llms really understand videos? In Findings of the Association for Computational Linguistics: ACL 2024 , pages 8731–8772

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? In Findings of the Association for Computational Linguistics: ACL 2024 , pages 8731–8772. Association for Computational Li...

    2024

  58. [65]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yun- hang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-...

    2025

  59. [66]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035 , 2024

    2024 arXiv

  60. [67]

    Videoeval-pro: Robust and realistic long video understanding evaluation

    Wentao Ma, Weiming Ren, Yiming Jia, Zhuofeng Li, Ping Nie, Ge Zhang, and Wenhu Chen. Videoeval-pro: Robust and realistic long video understanding evaluation. arXiv preprint arXiv:2505.14640 , 2025

    2025 arXiv

  61. [68]

    Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos

    Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826 , 2025

    2025 arXiv

  62. [69]

    Mmvu: Measuring expert-level multi-discipline video understanding

    Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. Mmvu: Measuring expert-level multi-discipline video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 84...

    2025

  63. [70]

    Minerva: Evaluating complex video reasoning

    Arsha Nagrani, Sachit Menon, Ahmet Iscen, Shyamal Buch, Ramin Mehran, Nilpa Jha, Anja Hauth, Yukun Zhu, Carl Vondrick, Mikhail Sirotenko, et al. Minerva: Evaluating complex video reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2396...

    2025

  64. [71]

    Video-mme-v2: Towards the next stage in benchmarks for comprehensive video understanding

    Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, Yunhang Shen, Xiaoxing Hu, Xueying Li, Jinsen Su, Chengwu Long, Xiaoyao Xie, et al. Video-mme-v2: Towards the next stage in benchmarks for comprehensive video understanding. arXiv preprint arXiv:2604.05015 , 2026

    2026 arXiv

  65. [72]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision , pages 5267–5275, 2017

    2017

  66. [73]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision , pages 706–715, 2017

    2017

  67. [74]

    Detecting moments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems , 34:11846–11858, 2021

    2021

  68. [75]

    Vidi: Large multimodal models for video understanding and editing

    Vidi Team, Celong Liu, Chia-Wen Kuo, Dawei Du, Fan Chen, Guang Chen, Jiamin Yuan, Lingxi Zhang, Lu Guo, Lusha Li, et al. Vidi: Large multimodal models for video understanding and editing. arXiv preprint arXiv:2504.15681, 2025

    2025 arXiv

  69. [76]

    Vidi2: Large multimodal models for video understanding and creation

    Vidi Team, Celong Liu, Chia-Wen Kuo, Chuang Huang, Dawei Du, Fan Chen, Guang Chen, Haoji Zhang, Haojun Zhao, Lingxi Zhang, et al. Vidi2: Large multimodal models for video understanding and creation. arXiv preprint arXiv:2511.19529 , 2025

    2025

  70. [77]

    Momentseeker: A task-oriented benchmark for long-video moment retrieval

    Huaying Yuan, Jian Ni, Zheng Liu, Yueze Wang, Junjie Zhou, Zhengyang Liang, Bo Zhao, Zhao Cao, Ji-Rong Wen, and Zhicheng Dou. Momentseeker: A task-oriented benchmark for long-video moment retrieval. Advances in Neural Information Processing Systems , 38, 2026

    2026

  71. [78]

    GPT-5 system card

    OpenAI. GPT-5 system card. https://openai.com/index/gpt-5-system-card/ , 2025

    2025

  72. [79]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil...

    2025 arXiv

  73. [80]

    Introducing claude sonnet 4.5

    Anthropic. Introducing claude sonnet 4.5. https://www.anthropic.com/news/claude-sonnet-4-5 , 2025. 26

    2025

  74. [81]

    Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe

    Tianyu Yu, Zefan Wang, Chongyi Wang, Fuwei Huang, Wenshuo Ma, Zhihui He, Tianchi Cai, Weize Chen, Yuxiang Huang, Yuanqian Zhao, et al. Minicpm-v 4.5: Cooking efficient mllms via architecture, data, and training recipe. arXiv preprint arXiv:2509.18154 , 2025

    2025 arXiv

  75. [82]

    Eagle 2.5: Boosting long-context post-training for frontier vision-language models

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Max Ehrlich, et al. Eagle 2.5: Boosting long-context post-training for frontier vision-language models. Advances in Neural Information Processing Systems , 38:91077–...

    2026

  76. [83]

    Videollama 3: Frontier multimodal foundation models for image and video understanding

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 , 2025

    2025 arXiv

  77. [84]

    Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, et al. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 , 2025

    2025 arXiv

  78. [85]

    Junbo Niu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench: How far is your video-llms from real-world online video under- standing? In Proceedings of the Computer Vision and Pattern Recognitio...

    2025

  79. [86]

    Streamingbench: Assessing the gap for mllms to achieve streaming video understanding

    Junming Lin, Zheng Fang, Chi Chen, Haoxuan Cheng, Zihao Wan, Fuwen Luo, Ziyue Wang, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech ...

    2026

  80. [88]

    Em-garde: A propose-match framework for proactive streaming video understanding

    Yikai Zheng, Xin Ding, Yifan Yang, Shiqi Jiang, Hao Wu, Qianxi Zhang, Weijun Wang, Ting Cao, and Yunxin Liu. Em-garde: A propose-match framework for proactive streaming video understanding. arXiv preprint arXiv:2603.19054, 2026

    2026 arXiv

  81. [89]

    Proactivevideoqa: A comprehensive benchmark evaluating proactive interactions in video large language models

    Yueqian Wang, Xiaojun Meng, Yifan Wang, Huishuai Zhang, and Dongyan Zhao. Proactivevideoqa: A comprehensive benchmark evaluating proactive interactions in video large language models. arXiv preprint arXiv:2507.09313, 2025

    2025 arXiv

  82. [90]

    Livecc: Learning video llm with streaming speech transcription at scale

    Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livecc: Learning video llm with streaming speech transcription at scale. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29083–29095, 2025

    2025

  83. [91]

    Timechat-online: 80% visual tokens are naturally redundant in streaming videos

    Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang, Shicheng Li, Sida Li, et al. Timechat-online: 80% visual tokens are naturally redundant in streaming videos. In Proceedings of the 33rd ACM International Conference on Multimedia , p...

    2025

  84. [92]

    Streamingvlm: Real-time understanding for infinite video streams

    Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Kelly Peng, Yao Lu, and Song Han. Streamingvlm: Real-time understanding for infinite video streams. arXiv preprint arXiv:2510.09608 , 2025

    2025 arXiv

  85. [93]

    Mm- duet2: Enhancing proactive interaction of video mllms with multi-turn reinforcement learning

    Yueqian Wang, Songxiang Liu, Disong Wang, Nuo Xu, Guanglu Wan, Huishuai Zhang, and Dongyan Zhao. Mm- duet2: Enhancing proactive interaction of video mllms with multi-turn reinforcement learning. arXiv preprint arXiv:2512.06810, 2025

    2025

  86. [94]

    Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024. URL https://arxiv.org/abs/2407.15754

    2024 arXiv

  87. [95]

    Videochat: Chat-centric video understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024. URL https://arxiv.org/abs/2305.06355

    2024 arXiv

  88. [96]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) , 2024

    2024

  89. [97]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122 , 2023

    2023 arXiv

  90. [98]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URL https://arxiv.org/ abs/2408.03326. 27

    2024 arXiv

  91. [99]

    Llava-onevision-2: Towards next- generation perceptual intelligence, 2026

    Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, Huajie Tan, Didi Zhu, Changrui Chen, Xiuwei Zhao, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Kaichen Zhang, Wenkang Zhang, Zheng Cheng, Nansen Zhang, Chunsheng Wu, Chunjiang Ge, Zimin Ran, Dehua Song, Chunyuan Li, Shikun Fen...

    2026 arXiv

  92. [100]

    Awt: Transferring vision-language models via augmentation, weighting, and transportation, 2024

    Yuhan Zhu, Yuyang Ji, Zhiyu Zhao, Gangshan Wu, and Limin Wang. Awt: Transferring vision-language models via augmentation, weighting, and transportation, 2024. URL https://arxiv.org/abs/2407.04603

    2024 arXiv

  93. [101]

    Efficient test-time prompt tuning for vision-language models, 2024

    Yuhan Zhu, Guozhen Zhang, Chen Xu, Haocheng Shen, Xiaoxin Chen, Gangshan Wu, and Limin Wang. Efficient test-time prompt tuning for vision-language models, 2024. URL https://arxiv.org/abs/2408.05775

    2024 arXiv

  94. [102]

    Freeret: Mllms as training-free retrievers, 2026

    Yuhan Zhu, Xiangyu Zeng, Chenting Wang, Xinhao Li, Chunxu Liu, Yicheng Xu, Ziang Yan, Yi Wang, and Limin Wang. Freeret: Mllms as training-free retrievers, 2026. URL https://arxiv.org/abs/2509.24621

    2026 arXiv

  95. [103]

    Motion-aware generative frame interpolation, 2025

    Guozhen Zhang, Yuhan Zhu, Yutao Cui, Xiaotong Zhao, Kai Ma, and Limin Wang. Motion-aware generative frame interpolation, 2025. URL https://arxiv.org/abs/2501.03699

    2025 arXiv

  96. [104]

    Kim, Bilge Soran, Raghura- man Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghura- man Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. Longvu: Spatio...

    2024 arXiv

  97. [105]

    Sharegpt4video: Improving video understanding and generation with better captions, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions, 2024. URL https://ar...

    2024 arXiv

  98. [106]

    Internvideo-next: Towards general video foundation models without video-text supervision, 2026

    Chenting Wang, Yuhan Zhu, Yicheng Xu, Jiange Yang, Lang Lin, Ziang Yan, Yali Wang, Yi Wang, and Limin Wang. Internvideo-next: Towards general video foundation models without video-text supervision, 2026. URL https://arxiv.org/abs/2512.01342

    2026

  99. [107]

    Vtimellm: Empower llm to grasp video moments, 2023

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments, 2023. URL https://arxiv.org/abs/2311.18445

    2023 arXiv

  100. [108]

    Lita: Language instructed temporal-localization assistant, 2024

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant, 2024. URL https://arxiv.org/abs/2403. 19046

    2024

  101. [109]

    Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding,

    Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xiaoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding,

  102. [110]

    Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models, 2025

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained temporal grounding in video large language models, 2025. URL https://arxiv.org/abs/2410.03290

    2025 arXiv

  103. [111]

    Dual detrs for multi-label temporal action detection, 2024

    Yuhan Zhu, Guozhen Zhang, Jing Tan, Gangshan Wu, and Limin Wang. Dual detrs for multi-label temporal action detection, 2024. URL https://arxiv.org/abs/2404.00653

    2024 arXiv

  104. [112]

    Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  105. [113]

    URL https://arxiv.org/abs/2405.13382

    arXiv

  106. [117]

    Sampling Rate

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms, 2025. URL https: //arxiv.org/abs/2503.21776. 28 A Appendix A.1 Implementation Details Offline...

    2025 arXiv

  107. [118]

    ① The woman in blue presses the button for the first floor. (0:18)

  108. [119]

    ② The man in green enters the elevator. (0:24)

  109. [120]

    ④ The elevator breaks down. (0:40)

  110. [121]

    ③ The woman in blue can't start the car. (2:27)

  111. [122]

    (3:03) Therefore, the correct sequence is ①, ②, ④, ③, ⑤

    ⑤ The man in green knocks on the car window to fix the car. (3:03) Therefore, the correct sequence is ①, ②, ④, ③, ⑤. Figure 12 Qualitative example of temporal video reasoning. VideoChat3 recognizes events distributed across the video and arranges them in chronological order. 3...

  112. [2021]

    doi: 10.1109/tpami.2019.2957464

    ISSN 1939-3539. doi: 10.1109/tpami.2019.2957464. URL http://dx.doi.org/10.1109/TPAMI.2019. 2957464

    1939

  113. [2024]

    URL https://arxiv.org/abs/2404.16821

    arXiv

  114. [2025]

    URL https://arxiv.org/abs/2512.13961

    arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.