Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Learning Streaming Video Representation via Multitask Training

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read StreamFormer turns an image-pretrained vision transformer into a causal streaming backbone via masked temporal attention and spatial low-rank adaptation; a multitask visual-language objective teaches global, per-frame, and per-pixel…

desk verdict A credible multitask streaming video backbone with a real gap in its long-context position-embedding and latency claims, plus a YouTubeVIS evaluation leak. read the letter →

arxiv 2504.20041 v2 pith:UMKOEFPY submitted 2025-04-28 cs.CV

Yibin Yan , Jilan Xu , Shangzhe Di , Yikun Liu , Yudi Shi , Qirui Chen , Zeqian Li , Yifei Huang
show 1 more author
Weidi Xie
This is my paper · ORCID
classification cs.CV
keywords streamingvideounderstandingcausaltemporalattentionmultitasklearningvisual-languagealignmentonlineactiondetectioninstancesegmentationquestionansweringlow-rankadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single video backbone can be trained to process streams frame by frame by adding causal temporal attention to a pre-trained image transformer, then teaching it with a multitask visual-language objective. The proposed StreamFormer attends only to current and past frames, so it can run with near-constant per-frame latency through cached attention keys and values, while learning global video semantics, per-frame temporal dynamics, and per-pixel spatial relationships together. On online action detection it reports 68.4 mAP without optical flow and 73.9 mAP with flow on THUMOS-14, beating a strong image-encoder baseline by roughly ten points without flow. The paper argues that diverse human-annotated video datasets provide enough supervision for this kind of backbone, making it an efficient alternative to massive video-text pre-training.

What carries the argument

The central mechanism is causal temporal attention computed at the same spatial patch position across frames, with a mask that sets attention to future frames to $-\infty$; this is combined with spatial attention whose pre-trained weights are modulated by low-rank adapters (LoRA), and a learnable zero-initialized tanh gate that lets temporal information enter gradually. Training ties every granularity to text embeddings through sigmoid visual-language alignment losses: action and caption supervision at the video level, frame-level supervision for temporal localization and grounding, and patch-level supervision for segmentation. The resulting features are read out at three resolutions, global (last frame), temporal (per frame), and spatial (per patch), so one backbone feeds several downstream heads.

What would settle it

Feed a video longer than 16 frames into the frozen StreamFormer frame by frame and measure online action detection mAP separately for early and late frames; if per-frame accuracy drops steadily after the 16th frame, the temporal position embedding extrapolation fails and the long-stream efficiency claims do not carry over.

Watch

Extended reading notes

Core claim

StreamFormer's central claim is that a causal mask on temporal attention, where every frame attends only to itself and earlier frames, turns a pre-trained image-language transformer into a streaming video backbone without sacrificing spatial accuracy. The authors train this backbone under a multitask visual-language alignment objective: global video embeddings align with action labels and captions, per-frame embeddings align with temporal localization and grounding labels, and patch embeddings align with segmentation masks. They report that this joint training outperforms training on ten times as much noisy web video-text data, and that the resulting frozen backbone transfers to online action detection, online video instance segmentation, and video question answering. On THUMOS-14 online action detection, it reaches 68.4 mAP without optical flow and 73.9 with flow, exceeding the image-encoder baseline by about ten points without flow while keeping near-constant per-frame latency via cached attention keys and values.

Load-bearing premise

The model was trained on 16-frame clips, but the streaming claims cover arbitrarily long videos; the paper does not explain how temporal positions are extended beyond the 16 trained positions, so long-stream performance depends on an unstated assumption.

Editorial extensions

If this is right

  • Online action detection can match or exceed flow-assisted pipelines using only RGB frames, removing optical-flow extraction from the inference path.
  • The same frozen backbone can be plugged into existing task-specific heads for detection, segmentation, and question answering by training only the head, because the causal features already carry temporal and spatial detail.
  • Because attention is causal, inference latency stays nearly flat as stream length grows when keys and values are cached, unlike bidirectional attention whose cost grows with length.
  • New streaming tasks can be added to the model by defining a new visual-language alignment objective; no changes to the video backbone are required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to relax the same-position constraint in temporal attention, for example with a causal window across neighboring patches, to sharpen motion cues; the paper does not explore this, but it is directly compatible with the architecture.
  • The data-efficiency comparison suggests annotation-rich multitask data may be a substitute for web-scale contrastive pre-training; an immediate test is whether the recipe continues to win as both data pools grow.
  • If the unresolved issue of temporal position embeddings beyond the 16 training frames is handled, for example by chunked or recycled position codes, StreamFormer becomes a practical frame encoder for robots and autonomous systems that run indefinitely.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes StreamFormer, a streaming video backbone built by adapting a SigLIP-initialized vision transformer with causal temporal attention, spatial low-rank adaptation, and a learnable tanh gate. The backbone is trained under a multitask visual-language alignment framework with global-level (action recognition, video-text retrieval), temporal-level (temporal action localization, temporal video grounding), and spatial-level (video object segmentation, referring video object segmentation) objectives, using about 1M publicly annotated video-text pairs. After training, the backbone is frozen and transferred to three downstream tasks: online action detection (THUMOS-14), online video instance segmentation (YouTubeVIS-2019), and video question answering (VideoMME, MLVU). The paper reports state-of-the-art OAD results (68.4 mAP without optical flow, 73.9 with flow), competitive VideoQA at 224 resolution, and improved data efficiency over WebVid-1M contrastive pretraining.

Significance. If the claims hold, the paper makes a useful architectural and data-efficiency contribution: it demonstrates that a causal video backbone trained with multitask video-language alignment on relatively small, richly annotated datasets can match or exceed prior online action detection methods, while supporting streaming inference with a KV-cache. The evaluation on THUMOS-14, VideoMME, and MLVU (none used in pretraining) is an important strength, as are the ablations in Tables 4-6 that isolate the contribution of each task group and of causal attention. The main caveats are the unspecified temporal-position-embedding mechanism for streams longer than the 16-frame training clips, the unexplained near-constant latency curve in Figure 3, and the YouTubeVIS-2019 overlap between pretraining and evaluation.

major comments (3)
  1. [Sec. 3.2, Sec. 4.2, Fig. 3] The temporal position embeddings are never defined. Section 3.2 states only that "spatial and temporal position embeddings are added separately," and Section 4.2 specifies that training uses 16 uniformly sampled frames, while Figure 3 reports inference up to 4096 frames. If the temporal embedding is a learned 16-entry table, a forward pass with T>16 is undefined; if it is interpolated or extrapolated, the long-stream evaluation operates in an untrained regime; if it is windowed or reset, the model cannot represent global temporal order. Because the streaming claim (per-frame processing plus long-term context) is the paper's central contribution, the manuscript must specify the encoding and validate it beyond the 16-frame training regime.
  2. [Eq. (4), Fig. 3] The claimed near-constant latency of causal attention with KV-cache is not implied by Eq. (4). Under the stated causal mask, each new frame's query attends to all cached keys and values of all previous frames, so per-frame temporal attention cost is O(T) per layer, growing linearly with stream length. The flat latency curve in Figure 3 therefore requires an additional mechanism (e.g., windowed attention, length extrapolation, or a different caching scheme) that is not described in the paper. Please state the exact inference-time procedure and its complexity, and reconcile the figure with the architecture.
  3. [Table 7, Sec. 4.3.2] YouTubeVIS-2019 appears in both the pretraining data (Table 7, spatial-level VOS) and the online video instance segmentation evaluation (Table 2). Section 4.3.2 says the model is "directly train[ed] on YoutubeVIS-2019 with COCO joint training" without specifying the split. If the validation split is included in pretraining, the AP/AR numbers in Table 2 are inflated by benchmark overlap; if only the training split is used, the paper should state that explicitly. This must be resolved before the VIS evidence can be taken at face value.
minor comments (5)
  1. [Fig. 3] Figure 3 has no axis labels or legend in the provided version; please clarify what is plotted (latency vs. number of frames, GPU memory, or both) and define each curve.
  2. [Table 4] The checkmark notation in Table 4 is not defined in the caption; indicate which checkmark corresponds to Global, Temporal, and Spatial task groups.
  3. [Multiple sections] The spelling of the dataset is inconsistent: the paper alternates between "YouTubeVIS-2019," "YoutubeVIS-2019," and "YouTubeVIS-19." Please standardize.
  4. [Sec. 4.2] The statement "By default, we uniformly sample 16 frames from the video as input" should specify whether the same 16-frame window is used for all training tasks and how the frame stride is chosen for datasets with very different frame rates.
  5. [Table 6] The entry "Ours-0.1M" in Table 6 is not defined in the text; clarify whether this is a 0.1M subset of the multitask data and how it was selected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the causal-temporal architecture and multitask losses are validated on external benchmarks, and no prediction reduces to a fitted input or load-bearing self-citation.

full rationale

StreamFormer's derivation chain is self-contained against circularity. The causal temporal attention in Eq. 4 is a fixed architectural constraint defined by the causal mask M, not a quantity fitted to any downstream benchmark; the spatial LoRA starts from pretrained SigLIP weights, and the global/temporal/spatial losses in Eqs. 6-8 are supervised objectives on public annotation datasets, not on the evaluation metrics they later predict. Downstream results on THUMOS-14, TVSeries, VideoMME, MLVU, and StreamingBench are external to the ~1M pretraining mixture. The appearance of YouTubeVIS-2019 in both pretraining and online-VIS evaluation is a data-contamination and soundness concern, not a circular reduction: the downstream protocol trains CTVIS on that benchmark with a frozen StreamFormer backbone, so no parameter of StreamFormer is fitted to the reported AP. MAT is used as a fixed detection head rather than as evidence for the streaming design, so the co-authored MAT citation is not load-bearing; the causal-temporal claim is instead tested by ablations that compare No TA, Bi-directional TA, and Causal TA architecture variants. The unstated behavior of temporal position embeddings beyond the 16-frame training regime is an unsupported extrapolation in the streaming claim, but it is not circular: no equation derives the long-stream behavior from the 16-frame assumption. Thus no circularity step can be exhibited under the required standard.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central contribution is an empirical training recipe, so the ledger lists the design choices and domain assumptions the recipe depends on: the SigLIP embedding space, the gating and LoRA adaptation, the compatibility of pooled annotations, and the unstated temporal-position extrapolation. No new physical entities are introduced.

free parameters (4)
  • LoRA rank = 32
    Chosen for spatial adaptation; no ablation is reported for alternative ranks.
  • Training clip length = 16 frames
    Uniformly sampled 16 frames per video during pretraining; the paper does not evaluate longer clips or specify how temporal position embeddings are extrapolated at test time.
  • Task oversampling ratios = unspecified
    The paper states over-sampling is used to balance task data scales (Sec. 4.2) but does not report the ratios; these affect the multitask objective balance.
  • Loss temperature and bias = learned
    The sigmoid losses (Eqs. 6 to 8) use a learnable temperature tau and bias b, following SigLIP; their learned values are not reported.
assumptions (4)
  • domain assumption SigLIP text encoder provides a shared visual-language embedding space suitable for labels, captions, and segmentation masks.
    All losses compute dot products between visual features and frozen SigLIP text embeddings (Sec. 3.3 and Sec. 3.4); if this space cannot separate fine-grained actions or object categories, multitask training would be ineffective.
  • domain assumption Causal temporal attention with a zero-initialized tanh gate preserves SigLIP spatial capabilities while learning temporal dependencies.
    Described in Sec. 3.2 and Sec. 4.2; the gating mechanism is intended to avoid disrupting the pretrained image encoder, but no direct evidence is provided beyond downstream results.
  • domain assumption Annotations across the diverse pretraining benchmarks are compatible enough for joint optimization.
    Sec. 4.1 pools AR, VTR, TAL, TVG, VOS, and RVOS datasets into one training mixture; label-space and annotation-style mismatches are assumed manageable via text prompts.
  • ad hoc to paper Temporal position embeddings trained on 16-frame clips generalize to long streams.
    Sec. 3.2 adds temporal position embeddings; Sec. 4.2 trains with 16 frames; no mechanism for longer sequences is described. This is the paper's weakest load-bearing premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Streaming Video Representation via Multitask Training." pith.science (2026). https://pith.science/paper/UMKOEFPY

@misc{pith2026250420041,
  author       = {Pith},
  title        = {Pith review of: Learning Streaming Video Representation via Multitask Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UMKOEFPY}},
  note         = {Machine review of arXiv:2504.20041}
}
read the original abstract

Understanding continuous video streams plays a fundamental role in real-time applications including embodied AI and autonomous driving. Unlike offline video understanding, streaming video understanding requires the ability to process video streams frame by frame, preserve historical information, and make low-latency decisions. To address these challenges, our main contributions are three-fold. (i) We develop a novel streaming video backbone, termed as StreamFormer, by incorporating causal temporal attention into a pre-trained vision transformer. This enables efficient streaming video processing while maintaining image representation capability. (ii) To train StreamFormer, we propose to unify diverse spatial-temporal video understanding tasks within a multitask visual-language alignment framework. Hence, StreamFormer learns global semantics, temporal dynamics, and fine-grained spatial relationships simultaneously. (iii) We conduct extensive experiments on online action detection, online video instance segmentation, and video question answering. StreamFormer achieves competitive results while maintaining efficiency, demonstrating its potential for real-time applications.

Figures

Figures reproduced from arXiv: 2504.20041 by the authors.

Figure 1
Figure 1. StreamFormer learns streaming video representations of various granularities through multitask training, making it applicable for diverse downstream tasks such as Online Action Detection, Online Video Instance Segmentation and Video Question Answering. Abstract Understanding continuous video streams plays a fundamen￾tal role in real-time applications, including embodied AI and autonomous driving. Unlike offline vide… view at source ↗
Figure 2
Figure 2. Overall framework of StreamFormer. Our StreamFormer is trained under a unified visual-language alignment framework, enabling simultaneous understanding of global semantics, temporal dynamics, and fine-grained spatial relationships. Each level utilizes features of different granularities: (i) last frame for the global level, (ii) per frame for the temporal level, (iii) and per frame per patch feature for the spatial … view at source ↗
Figure 3
Figure 3. Computational complexity. the streaming setting, particularly when supported by KV￾Cache. We compare the inference cost of the backbone (i.e., latency and GPU memory) in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative results of Online Video Instance Segmentation. Zoom in for a better view. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Diffractive electroproduction of light vector particles: leading Fock-state contribution in the presence of significant higher Fock-state effects

    hep-ph 2025-08 unverdicted novelty 5.0 of 10

    The paper claims the leading quark-antiquark approximation in the color dipole model only matches HERA data for rho/gamma above Q^2 of 20 GeV^2 and for phi above Q^2 of 10 GeV^2, unlike J/psi.

Reference graph

Works this paper leans on

137 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text

    Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In NeurIPS, 2021. 2

    2021

  2. [2]

    Localizing mo- ments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing mo- ments in video with natural language. In ICCV, 2017. 2, 6, 1 8

    2017

  3. [3]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021. 2, 4

    2021

  4. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 2

    arXiv 2023

  5. [5]

    Sequential modeling enables scalable learning for large vision models

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In CVPR, 2024. 2

    2024

  6. [6]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 2, 8

    2021

  7. [7]

    Is space-time attention all you need for video understanding? In ICML, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021. 1, 2, 3, 4

    2021

  8. [8]

    Rt-1: Robotics transformer for real-world con- trol at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world con- trol at scale. arXiv preprint arXiv:2212.06817, 2022. 1

    arXiv 2022

Show all 137 references
  1. [9]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. 3

    2020

  2. [10]

    Sst: Single-stream tem- poral action proposals

    Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. Sst: Single-stream tem- poral action proposals. In CVPR, 2017. 1

    2017

  3. [11]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR,

  4. [12]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. 2, 6, 1

    2017

  5. [13]

    Multitask learning

    Rich Caruana. Multitask learning. Machine learning, 1997. 3

    1997

  6. [14]

    Collecting highly paral- lel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly paral- lel data for paraphrase evaluation. In ACL, 2011. 6, 1

    2011

  7. [15]

    Videollm: Modeling video sequence with large language models

    Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305.13292, 2023. 3

    2023 arXiv

  8. [16]

    Gatehub: Gated history unit with background sup- pression for online action detection

    Junwen Chen, Gaurav Mittal, Ye Yu, Yu Kong, and Mei Chen. Gatehub: Gated history unit with background sup- pression for online action detection. In CVPR, 2022. 3, 6, 7

    2022

  9. [17]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongx- ing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In CVPR,

  10. [18]

    Generative pre- training from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee- woo Jun, David Luan, and Ilya Sutskever. Generative pre- training from pixels. In ICML, 2020. 2

    2020

  11. [19]

    Pix2seq: A language modeling framework for object detection

    Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Ge- offrey Hinton. Pix2seq: A language modeling framework for object detection. In ICLR, 2022. 3

    2022

  12. [20]

    Vision transformer adapter for dense predictions

    Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023. 6

    2023

  13. [21]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 7

    2024 arXiv

  14. [22]

    Transfuser: Imita- tion with transformer-based sensor fusion for autonomous driving

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imita- tion with transformer-based sensor fusion for autonomous driving. TPAMI, 2022. 1

    2022

  15. [23]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 2023. 3

    2023

  16. [24]

    Online action detection

    Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. Online action detection. In ECCV, 2016. 2

    2016

  17. [25]

    Streaming video question-answering with in- context video kv-cache retrieval

    Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, Hao Jiang, et al. Streaming video question-answering with in- context video kv-cache retrieval. In ICLR, 2025. 1, 3

    2025

  18. [26]

    Mevis: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. In ICCV,

  19. [27]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021. 2

    2021

  20. [28]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 2023. 1

    2023

  21. [29]

    Predicting depth, surface nor- mals and semantic labels with a common multi-scale con- volutional architecture

    David Eigen and Rob Fergus. Predicting depth, surface nor- mals and semantic labels with a common multi-scale con- volutional architecture. In ICCV, 2015. 3

    2015

  22. [30]

    Scal- able pre-training of large autoregressive image models

    Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scal- able pre-training of large autoregressive image models. In ICML, 2024. 2

    2024

  23. [31]

    Streamingbench: Assessing the gap for mllms to achieve streaming video understanding, 2024

    Junming Lin et al. Streamingbench: Assessing the gap for mllms to achieve streaming video understanding, 2024. 2

    2024

  24. [32]

    Trecvit: A recurrent video trans- former, 2024

    Viorica P ˘atr˘aucean et al. Trecvit: A recurrent video trans- former, 2024. 2

    2024

  25. [33]

    Multiscale vision transformers

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichten- hofer. Multiscale vision transformers. In ICCV, 2021. 2

    2021

  26. [34]

    Spatiotemporal multiplier networks for video action recog- nition

    Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal multiplier networks for video action recog- nition. In CVPR, 2017. 2 9

    2017

  27. [35]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In CVPR, 2019. 1, 2

    2019

  28. [36]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In CVPR, 2025. 7

    2025

  29. [37]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, 2017. 2, 6, 1

    2017

  30. [38]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ...

    2017

  31. [39]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In CVPR, 2024. 1

    2024

  32. [40]

    Learning from streaming video with orthogonal gradients

    Tengda Han, Dilara Gokay, Joseph Heyward, Chuhan Zhang, Daniel Zoran, Viorica Patraucean, Joao Carreira, Dima Damen, and Andrew Zisserman. Learning from streaming video with orthogonal gradients. In CVPR, 2025. 3

    2025

  33. [41]

    Masked autoencoders are scal- able vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scal- able vision learners. In CVPR, 2022. 2

    2022

  34. [42]

    Continual transformers: redundancy-free attention for online inference

    Lukas Hedegaard, Arian Bakhtiarnia, and Alexandros Iosi- fidis. Continual transformers: redundancy-free attention for online inference. In ICLR, 2023. 7, 1

    2023

  35. [43]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 4

    2022

  36. [44]

    Min- vis: A minimal video instance segmentation framework without video-based training

    De-An Huang, Zhiding Yu, and Anima Anandkumar. Min- vis: A minimal video instance segmentation framework without video-based training. In NeurIPS, 2022. 7

    2022

  37. [45]

    Vinci: A real-time embodied smart as- sistant based on egocentric vision-language model

    Yifei Huang, Jilan Xu, Baoqi Pei, Yuping He, Guo Chen, Lijin Yang, Xinyuan Chen, Yaohui Wang, Zheng Nie, Jinyao Liu, et al. Vinci: A real-time embodied smart as- sistant based on egocentric vision-language model. arXiv preprint arXiv:2412.21080, 2024. 1, 3

    2024 arXiv

  38. [46]

    Online video understanding: A compre- hensive benchmark and memory-augmented method

    Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xi- angyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: A compre- hensive benchmark and memory-augmented method. arXiv preprint arXiv:2501.00584, 2024. 1, 3

    2024 arXiv

  39. [47]

    Batch normalization: Accelerating deep network training by reducing internal co- variate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. In ICML, 2015. 6, 1

    2015

  40. [48]

    Jiang, J

    Y .-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. THUMOS challenge: Ac- tion recognition with a large number of classes. http: //crcv.ucf.edu/THUMOS14/, 2014. 6

    2014

  41. [49]

    Multi-task learning using uncertainty to weigh losses for scene geom- etry and semantics

    Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geom- etry and semantics. In CVPR, 2018. 3

    2018

  42. [50]

    Deep reinforcement learning for autonomous driv- ing: A survey

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Man- nion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick P´erez. Deep reinforcement learning for autonomous driv- ing: A survey. TPAMI, 2021. 1

    2021

  43. [51]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, 2017. 6, 1

    2017

  44. [52]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. In NeurIPS, 2021. 6, 1

    2021

  45. [53]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, 2023. 1, 2

    2023

  46. [54]

    Uniformer: Unifying convolution and self-attention for visual recogni- tion

    Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guan- glu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recogni- tion. TPAMI, 2023. 2

    2023

  47. [55]

    Mvbench: A comprehensive multi-modal video under- standing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video under- standing benchmark. In CVPR, 2024. 7

    2024

  48. [56]

    Zeroi2v: Zero- cost adaptation of pre-trained transformers from image to video

    Xinhao Li, Yuhan Zhu, and Limin Wang. Zeroi2v: Zero- cost adaptation of pre-trained transformers from image to video. In ECCV, 2024. 4

    2024

  49. [57]

    Mvitv2: Improved multiscale vision transformers for classification and detection

    Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Man- galam, Bo Xiong, Jitendra Malik, and Christoph Feichten- hofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022. 2

    2022

  50. [58]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InECCV,

  51. [59]

    Video-llava: Learning united visual repre- sentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual repre- sentation by alignment before projection. InEMNLP, 2023. 7

    2023

  52. [60]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6, 1

    2014

  53. [61]

    Frozen clip models are efficient video learners

    Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Frozen clip models are efficient video learners. In ECCV, 2022. 2, 8

    2022

  54. [62]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 2

    2024 arXiv

  55. [63]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 7

    2023

  56. [64]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024. 1, 2 10

    2024

  57. [65]

    Llava-next: Im- proved reasoning, ocr, and world knowledge

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge. https: //llava- vl.github.io/blog/2024- 01- 30- llava-next/, 2024. 2

    2024

  58. [66]

    Streamchat: Chatting with streaming video

    Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, and Jose M Alvare. Streamchat: Chatting with streaming video. arXiv preprint arXiv:2412.08646, 2024. 1

    2024 arXiv

  59. [67]

    Fineaction: A fine-grained video dataset for temporal ac- tion localization

    Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal ac- tion localization. TIP, 2022. 6, 1

    2022

  60. [68]

    Video swin transformer

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022. 2

    2022

  61. [69]

    Unified-io: A uni- fied model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A uni- fied model for vision, language, and multi-modal tasks. In ICLR, 2023. 3

    2023

  62. [70]

    Howto100m: Learning a text-video embedding by watch- ing hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watch- ing hundred million narrated video clips. In ICCV, 2019. 2, 8

    2019

  63. [71]

    Cross-stitch networks for multi-task learn- ing

    Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learn- ing. In CVPR, 2016. 3

    2016

  64. [72]

    Queryd: A video dataset with high-quality text and audio narrations

    Andreea-Maria Oncescu, Joao F Henriques, Yang Liu, An- drew Zisserman, and Samuel Albanie. Queryd: A video dataset with high-quality text and audio narrations. In ICASSP, 2021. 6, 1

    2021

  65. [73]

    Vlp: Vision language planning for autonomous driving

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Ab- hirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In CVPR, 2024. 1

    2024

  66. [74]

    St-adapter: Parameter-efficient image-to-video transfer learning

    Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hong- sheng Li. St-adapter: Parameter-efficient image-to-video transfer learning. In NeurIPS, 2022. 4

    2022

  67. [75]

    Im- age transformer

    Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Im- age transformer. In ICML, 2018. 2

    2018

  68. [76]

    Occluded video instance segmentation: A benchmark

    Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: A benchmark. In IJCV, 2022. 2

    2022

  69. [77]

    Streaming long video understanding with large language models

    Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuan- grui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. In NeurIPS, 2024. 1

    2024

  70. [78]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 2, 1

    2021

  71. [79]

    Real-world robot learn- ing with masked visual pre-training

    Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learn- ing with masked visual pre-training. In CoRL, 2023. 1

    2023

  72. [80]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020. 3

    2020

  73. [81]

    An empirical study of autoregressive pre- training from videos

    Jathushan Rajasegaran, Ilija Radosavovic, Rahul Ravis- hankar, Yossi Gandelsman, Christoph Feichtenhofer, and Jitendra Malik. An empirical study of autoregressive pre- training from videos. arXiv preprint arXiv:2501.05453 ,

    arXiv

  74. [82]

    Coherent multi-sentence video description with variable level of de- tail

    Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal, and Bernt Schiele. Coherent multi-sentence video description with variable level of de- tail. In GCPR, 2014. 6, 1

    2014

  75. [83]

    An overview of multi-task learning in deep neural networks

    Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098 ,

    arXiv

  76. [84]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In ECCV, 2020. 6, 1

    2020

  77. [85]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024. 2

    2024 arXiv

  78. [86]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 ,

    arXiv

  79. [87]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NeurIPS, 2024. 2

    2024

  80. [88]

    Learning language-visual embedding for movie understanding with natural-language

    Atousa Torabi, Niket Tandon, and Leonid Sigal. Learning language-visual embedding for movie understanding with natural-language. arXiv preprint arXiv:1609.08124, 2016. 6, 1

    2016 arXiv

  81. [89]

    Llama: Open and efficient foundation language mod- els

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  82. [90]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  83. [91]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torre- sani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015. 2

    2015

  84. [92]

    A closer look at spatiotem- poral convolutions for action recognition

    Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotem- poral convolutions for action recognition. In CVPR, 2018. 2

    2018

  85. [93]

    Video classification with channel-separated convolutional networks

    Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel-separated convolutional networks. In ICCV, 2019. 2

    2019

  86. [94]

    Conditional image gen- eration with pixelcnn decoders

    Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image gen- eration with pixelcnn decoders. In NeurIPS, 2016. 2 11

    2016

  87. [95]

    To- wards open-vocabulary video instance segmentation

    Haochen Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie, and Efstratios Gavves. To- wards open-vocabulary video instance segmentation. In ICCV, 2023. 6, 1

    2023

  88. [96]

    Memory-and-anticipation transformer for online action understanding

    Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang, and Tong Lu. Memory-and-anticipation transformer for online action understanding. In ICCV, 2023. 3, 6, 7, 1

    2023

  89. [97]

    Temporal segment networks: Towards good practices for deep action recogni- tion

    Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recogni- tion. In ECCV, 2016. 2, 6

    2016

  90. [98]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, 2022. 3

    2022

  91. [99]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2

    2024 arXiv

  92. [100]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NeurIPS,

  93. [101]

    Vatex: A large-scale, high-quality multilingual dataset for video-and-language research

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In ICCV, 2019. 6, 1

    2019

  94. [102]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022. 1, 2, 8

    2022 arXiv

  95. [103]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2024. 2, 8

    2024

  96. [104]

    Internvideo2: Scaling foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yan- song Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. In ECCV, 2024. 2, 8

    2024

  97. [105]

    In defense of online models for video in- stance segmentation

    Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. In defense of online models for video in- stance segmentation. In ECCV, 2022. 3, 7

    2022

  98. [106]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. In NeurIPS, 2024. 3

    2024

  99. [107]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. 2, 6, 1

    2016

  100. [108]

    Retrieval-augmented ego- centric video captioning

    Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Retrieval-augmented ego- centric video captioning. In CVPR, 2024. 2

    2024

  101. [109]

    Temporal recurrent networks for on- line action detection

    Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S Davis, and David J Crandall. Temporal recurrent networks for on- line action detection. In ICCV, 2019. 3, 7

    2019

  102. [110]

    Long short-term trans- former for online action detection

    Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, and Stefano Soatto. Long short-term trans- former for online action detection. In NeurIPS, 2021. 1, 3, 6, 7

    2021

  103. [111]

    Videococa: Video-text modeling with zero-shot transfer from con- trastive captioners

    Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video-text modeling with zero-shot transfer from con- trastive captioners. arXiv preprint arXiv:2212.04979, 2022. 2, 8

    2022 arXiv

  104. [112]

    Task preference optimization: Improving mul- timodal large language models with vision task alignment

    Ziang Yan, Zhilin Li, Yinan He, Chenting Wang, Kunchang Li, Xinhao Li, Xiangyu Zeng, Zilei Wang, Yali Wang, Yu Qiao, et al. Task preference optimization: Improving mul- timodal large language models with vision task alignment. arXiv preprint arXiv:2412.19326, 2024. 3

    2024 arXiv

  105. [113]

    Video instance segmentation

    Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019. 2, 6, 1

    2019

  106. [114]

    Colar: Ef- fective and efficient online action detection by consulting exemplars

    Le Yang, Junwei Han, and Dingwen Zhang. Colar: Ef- fective and efficient online action detection by consulting exemplars. In CVPR, 2022. 7

    2022

  107. [115]

    Crossover learning for fast online video instance segmentation

    Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Crossover learning for fast online video instance segmentation. In ICCV, 2021. 7

    2021

  108. [116]

    Ctvis: Consistent train- ing for online video instance segmentation

    Kaining Ying, Qing Zhong, Weian Mao, Zhenhua Wang, Hao Chen, Lin Yuanbo Wu, Yifan Liu, Chengxiang Fan, Yunzhi Zhuge, and Chunhua Shen. Ctvis: Consistent train- ing for online video instance segmentation. In ICCV, 2023. 3, 6, 7, 1

    2023

  109. [117]

    Taskonomy: Disentangling task transfer learning

    Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018. 3

    2018

  110. [118]

    Merlot reserve: Neural script knowledge through vision and language and sound

    Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yan- peng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In CVPR, 2022. 2

    2022

  111. [119]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 2, 4, 5, 6, 1

    2023

  112. [120]

    Anygpt: Unified multimodal llm with discrete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with discrete sequence modeling. In ACL, 2024. 3

    2024

  113. [121]

    Flash-vstream: Memory- based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory- based real-time understanding for long video streams.arXiv preprint arXiv:2406.08085, 2024. 1, 3

    2024 arXiv

  114. [122]

    Lmms-eval: Reality check on the evaluation of large multi- modal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuan- han Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multi- modal models, 2024. 2

    2024

  115. [123]

    Dvis++: Improved decoupled frame- work for universal video segmentation

    Tao Zhang, Xingye Tian, Yikang Zhou, Shunping Ji, Xuebo Wang, Xin Tao, Yuan Zhang, Pengfei Wan, Zhongyuan 12 Wang, and Yu Wu. Dvis++: Improved decoupled frame- work for universal video segmentation. arXiv preprint arXiv:2312.13305, 2023. 3, 6, 7

    2023 arXiv

  116. [124]

    Llava-next: A strong zero-shot video understanding model

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model. https://llava- vl.github.io/blog/2024- 04-30-llava-next-video/ , 2024. 2, 7, 1

    2024

  117. [125]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 7, 2

    2024 arXiv

  118. [126]

    Hacs: Human action clips and segments dataset for recognition and temporal localization

    Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In ICCV,

  119. [127]

    Videoprism: A founda- tional visual encoder for video understanding

    Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A founda- tional visual encoder for video understanding. In ICML,

  120. [128]

    Does video-text pretraining help open-vocabulary online action detection? NeurIPS, 2024

    Qingsong Zhao, Yi Wang, Jilan Xu, Yinan He, Zifan Song, Limin Wang, Yu Qiao, Cairong Zhao, et al. Does video-text pretraining help open-vocabulary online action detection? NeurIPS, 2024. 3

    2024

  121. [129]

    Real-time online video detection with temporal smoothing transformers

    Yue Zhao and Philipp Kr ¨ahenb¨uhl. Real-time online video detection with temporal smoothing transformers. In ECCV,

  122. [130]

    Learning video representations from large lan- guage models

    Yue Zhao, Ishan Misra, Philipp Kr ¨ahenb¨uhl, and Rohit Girdhar. Learning video representations from large lan- guage models. In CVPR, 2023. 2

    2023

  123. [131]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2023. 1

    2023

  124. [132]

    Mlvu: A comprehensive benchmark for multi- task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi- task long video understanding. In CVPR, 2025. 7

    2025

  125. [133]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In CVPR,

  126. [134]

    Uni-perceiver: Pre- training unified architecture for generic perception for zero- shot and few-shot tasks

    Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre- training unified architecture for generic perception for zero- shot and few-shot tasks. In CVPR, 2022. 3 13 Learning Streaming Video Representation via Multitask Trainin...

    2022

  127. [135]

    AR” for action recognition, “VTR

    Pre-training Datasets We list StreamFormer’s pre-training datasets in Table 7. Note that for our Video Object Segmentation datasets, we utilize the VIS datasets and VIS-style produced COCO-pseudo videos for their relatively well correspondence within the objects inside the vid...

  128. [136]

    Online Action Detection In Online Action Detection, we adopt MAT [96] as our baseline detection head due to its strong detection performance

    Downstream Implementation Details 2.1. Online Action Detection In Online Action Detection, we adopt MAT [96] as our baseline detection head due to its strong detection performance. Following prior works [42, 110, 129], we extract the video frames at 24 fps, using a video chunk...

  129. [137]

    OP” for Object Perception, “CR

    More Results We add more results of StreamFormer in this section. 3.1. Video Action Recognition In Table 9, we add more comparisons with recent causal and autoregressive models, including TRecViT [32] and Toto [81] on Video Action Recognition tasks of K400 and SSv2. While Stre...

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.