Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

VideoOrion: Tokenizing Object Dynamics in Videos

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that encoding each tracked object's spatial-temporal dynamics as a single object token gives video-language models a compact, disentangled visual representation that improves general question answering and makes…

desk verdict The object-token idea is worth serious attention, but the headline gains are confounded by extra training data and a weak controlled baseline. read the letter →

arxiv 2411.16156 v2 pith:YJXPG64O submitted 2024-11-25 cs.CV cs.LG

Yicheng Feng , Yijiang Li , Wanpeng Zhang , Hao Luo , Zihao Yue , Sipeng Zheng , Zongqing Lu This is my paper · ORCID
classification cs.CVcs.LG
keywords videolargelanguagemodelsobject-centricrepresentationdetect-segment-trackpipelineobjecttokensquestionansweringvideo-basedreferringspatial-temporaldynamicstokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that object-centric tokens—one token per tracked object, pooling its masked features across frames—are a better visual vocabulary for video-language models than downsampled patch tokens. It argues that this representation is more compact, more disentangled, and more aligned with text semantics, and it claims that adding such an Object-Centric Branch to a standard frame-based Video-LLM yields consistent gains on five video question-answering benchmarks and large gains on video-based referring. If right, this means a video-LLM does not have to choose between coarse frame sampling and expensive full-video tokenization; object dynamics can be carried in a handful of extra tokens.

What carries the argument

The central object is the object token: one vector in the LLM embedding space that summarizes one object's appearance and motion across the whole video. It is produced by a detect-segment-track pipeline—GroundingDINO proposes boxes, SAM segments masks, RAM++ selects key frames by object-tag overlap, and XMem tracks each mask across up to 128 frames—followed by an Object Projector that mask-pools CLIP features per frame and temporal-pools those pooled features before passing them through an MLP. The Object-Centric Branch thus compresses each object's spatial-temporal dynamics into one token while the Video-Centric Branch supplies context tokens.

What would settle it

A mask-corruption control would settle the claim: run VideoOrion with the same number and size of object tokens, but pool features from random or shuffled regions instead of the detected masks. If MVBench and VideoMME accuracy stay within a point or two, the object-token mechanism is not doing the claimed work; if accuracy drops sharply, the tracked masks are load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that explicitly tokenizing objects—their masks, trajectories, and feature dynamics—gives a Video-LLM a compact, disentangled semantic representation that downsampled frame patches cannot match. Concretely, adding the Object-Centric Branch to VideoLLaMA2 and VideoLLaMA2.1 makes the model consistently surpass those baselines by 5.1 to 15.6 points across MVBench, EgoSchema, Perception-Test, VideoMME, and ActivityNet-QA, and turns the same interface into a video-referring system that outperforms Artemis on VideoRef45K. The author's own framing is that the object tokens make video representation more like text tokenization, where each token already carries clear semantics.

Load-bearing premise

The object-token benefit rests on the upstream pipeline reliably producing the right mask for the right object in every frame; if detection, key-frame selection, or tracking fails, the added tokens carry noise rather than disentangled object semantics.

Editorial extensions

If this is right

  • General video QA improves on all five tested benchmarks, with same-backbone gains of 5.1 to 15.6 accuracy points, even though the video branch still sees only 8 to 16 frames.
  • Video-based referring becomes a natural capability: inserting the target object's token into the prompt outperforms dedicated referring models such as Artemis on VideoRef45K across all reported metrics.
  • Fine-grained temporal reasoning improves: VideoOrion+ scores 61.8 versus 59.5 multi-binary accuracy on TemporalBench compared with VideoLLaMA2.1.
  • Replacing the object branch with standard video encoders such as VideoMAE or UMT-L costs more than 2 average points, indicating the object representations, not just extra temporal features, drive the gain.
  • Object-branch pretraining on 700K video-text pairs is necessary; without it most gains disappear, and object-only encoding is not enough on its own, so both branches are needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to use the same object-token interface for tasks that require object identity rather than open-ended description—such as video grounding, editing, or embodied decision-making—since the representation already separates instances.
  • The design suggests that long-video understanding could be improved by scaling the tracking branch with more frames or faster trackers rather than by lengthening the LLM's context, a cheaper direction than long-context video transformers.
  • Because the object branch sees 64 to 128 frames while the video branch sees only 8 to 16, a control that gives the video branch the same extra frames would clarify whether the gains come from disentangled semantics or simply from richer temporal sampling.
  • The paper's failure analysis implies object tokens also make errors inspectable; a future system could add a per-token verification step that checks whether mask pooling is actually attending to the intended object.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. VideoOrion is a video-LLM with two encoding branches: a standard video-centric branch using an STC connector, and an object-centric branch that generates object masks via a detect-segment-track pipeline (GroundingDINO, SAM, RAM++, XMem), pools mask-aligned CLIP features, and maps them to object tokens. The model is trained in three stages (video projector pretraining, object projector pretraining on 700K filtered videos, and multi-modal instruction tuning including 1M ego-centric QA samples). Evaluations on MVBench, EgoSchema, Perception-Test, VideoMME, ActivityNet-QA, and video referring (VideoRef45K) show large gains over VideoLLaMA2/2.1 and competitive or superior performance on referring. Ablations examine the object branch, pipeline design choices, projector architecture, and token counts.

Significance. The idea of explicitly encoding object dynamics as compact, semantically meaningful tokens is a promising addition to video-LLM design. The paper contributes a concrete pipeline and extensive ablations, including design-choice analysis, failure cases, and data-scaling behavior. The controlled ablation in Table 4 shows consistent improvements over a same-data video-only baseline, albeit of modest size, supporting a weaker version of the central claim. However, the headline comparison in Table 2 is not controlled for training data or base-model strength, so the significance of the object-token contribution depends on whether the large margins reflect the object branch or simply a larger/stronger training setup.

major comments (3)
  1. [§4.2, Table 2] The claim that VideoOrion 'consistently surpasses' VideoLLaMA2 and VideoLLaMA2.1 by 5.1–15.6 points is not supported as a causal statement about the Object-Centric Branch, because the comparison is not matched on training data. Table 1 shows VideoOrion is trained with 700K object-pretraining videos, 1M ego-centric QA samples, and 2.7M MM instruction-tuning samples, whereas VideoLLaMA2/2.1 use their original training recipes. The only data-controlled ablation (Table 4) yields gains of about 2 points average (e.g., MVBench 44.2 vs 41.9), an order of magnitude smaller than the headline margins. Please provide a matched-data, matched-base comparison (e.g., train VideoLLaMA2 on the same mixture, or ablate object tokens on the full model) or substantially qualify the claim in §4.2.
  2. [Table 4] The data-controlled baseline in Table 4 is substantially weaker than the published VideoLLaMA2 (MVBench 41.9 vs 53.4), suggesting that the controlled comparison is performed on a limited-capacity training configuration. It is unclear whether the small object-token gains would persist with the full training scale. Please report a full-scale ablation or explain why the subset results are representative, and specify the exact composition of the video-only baseline (e.g., whether the 700K video data are indeed added to the video pretraining stage for the baseline).
  3. [§4.4, Tables 4 and 7] No error bars or repeated-trial variance are reported for any table. The differences in Table 4 are only 1–2 points on some benchmarks (e.g., ActivityNet-QA 43.3 vs 43.0), so it is difficult to assess whether the 'consistent' improvements are beyond evaluation noise. Please provide multiple seeds or an estimate of benchmark variability, and explain why Table 7 reports different VideoOrion numbers (MVBench 48.0) than Table 4 (MVBench 44.2) for what appears to be the same configuration.
minor comments (5)
  1. [Abstract and Appendix E] The abstract and introduction describe the method as having 'minimal computational cost,' but Appendix E reports a 38.5% increase in per-sample time; please soften the phrasing to 'moderate additional cost.'
  2. [§3.3] The sentence 'we sample to frames from the video' uses the variable 'to' before it is defined; introduce the symbol (e.g., to) with its meaning and value before this sentence.
  3. [§4.2] The phrase 'which has the same Video-Centric Branch' should be 'which have the same Video-Centric Branch' to agree with the plural subject 'VideoLLaMA2 and VideoLLaMA2.1.'
  4. [Table 2] The ActivityNet-QA column mixes formats such as '53.5 /3.2' and '57.7/3.7'; use consistent spacing and a clear 'Accuracy / Score' header.
  5. [§4.4, Table 8] The comparison with VideoMAE and UMT-L is useful, but the text says 'more than 2% on average' while the table shows 45.3 vs 43.0 average; please check the arithmetic and state the exact average gain.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's claims are evaluated on external benchmarks and the ablations are data-controlled rather than self-referential.

full rationale

VideoOrion's central claims are tested on external benchmarks (MVBench, EgoSchema, Perception-Test, VideoMME, ActivityNet-QA, TemporalBench, VideoRef45K), and the reported numbers are not generated by inverting the method's own fitted values. The object tokens are produced by a detect-segment-track pipeline followed by mask pooling and an MLP projector, and the LLM is trained with standard cross-entropy; no equation defines an evaluation quantity in terms of a training target. The thresholds θa=3 and θb=2 and the 64-token cap are hyperparameters selected through ablation experiments, not quantities the model is asked to predict, so they do not constitute a fitted-input-called-prediction step. The headline comparison with VideoLLaMA2/2.1 is not data-matched, and the controlled ablation in Table 4 uses a weaker baseline than the published VideoLLaMA2; this is a benchmark-validity confound (extra training data, base-strength imbalance), not a circular derivation. The referring experiments use the provided box prompt to construct the target object token, which is the task's prescribed input interface rather than a self-definitional prediction. No load-bearing argument depends on a self-citation chain: the reused STC connector is an external open-source component, and the paper explicitly contrasts VideoOrion with it in both the main comparison and the ablation. Therefore no circularity is identified, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central contribution is an architecture built from pretrained components; the load-bearing assumptions are that those components work on arbitrary videos and that mask-pooled CLIP features preserve object semantics. Main hand-chosen numbers are the pipeline thresholds and token caps.

free parameters (4)
  • theta_a (key frame threshold) = 3
    Frames with fewer than 3 object tags are filtered as noisy; value is 'empirically determined' (Sec 3.3, Sec 4.1).
  • theta_b (tag overlap threshold) = 2
    A frame becomes a new key frame when tag overlap with the current key frame falls below 2; 'empirically determined' (Sec 3.3, Sec 4.1).
  • object frame sample count to = 64 short videos, 128 for >1 min
    Number of frames used in tracking; chosen by hand for token/compute balance (Sec 3.3).
  • maximum object tokens Noi = 64
    Cap on object token count; ablation in Table 11 shows 64 performs best among 16, 32, 64, 80.
assumptions (3)
  • domain assumption The expert models (GroundingDINO, SAM, RAM++, XMem) generate sufficiently accurate object proposals, masks, tags, and tracks on diverse video content.
    The whole object branch depends on these off-the-shelf models; the paper itself notes mask errors on low-quality videos (Limitations, Appendix D.1).
  • domain assumption CLIP ViT-L/14 (or SigLIP) visual features, pooled by masks, retain the semantic and spatial information needed for QA.
    Object tokens are formed by mask-pooling the same vision encoder features used in the context branch (Sec 3.3); no independent evidence that mask-pooled features are sufficient.
  • domain assumption The STC-Connector from VideoLLaMA2 is a valid video projector whose pretraining transfers to this setting.
    Video-Centric Branch directly reuses VideoLLaMA2's pretrained connector (Sec 3.2, 3.4), importing its training assumptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VideoOrion: Tokenizing Object Dynamics in Videos." pith.science (2026). https://pith.science/paper/YJXPG64O

@misc{pith2026241116156,
  author       = {Pith},
  title        = {Pith review of: VideoOrion: Tokenizing Object Dynamics in Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YJXPG64O}},
  note         = {Machine review of arXiv:2411.16156}
}
read the original abstract

We present VideoOrion, a Video Large Language Model (Video-LLM) that explicitly captures the key semantic information in videos - the spatial-temporal dynamics of objects throughout the videos. VideoOrion employs expert vision models to extract object dynamics through a detect-segment-track pipeline, encoding them into a set of object tokens by aggregating spatial-temporal object features. Our method addresses the persistent challenge in Video-LLMs of efficiently compressing high-dimensional video data into semantic tokens that are comprehensible to LLMs. Compared to prior methods which resort to downsampling the original video or aggregating visual tokens using resamplers, leading to information loss and entangled semantics, VideoOrion not only offers a more natural and efficient way to derive compact, disentangled semantic representations but also enables explicit object modeling of video content with minimal computational cost. Moreover, the introduced object tokens naturally allow VideoOrion to accomplish video-based referring tasks. Experimental results show that VideoOrion can learn to make good use of the object tokens, and achieves competitive results on both general video question answering and video-based referring benchmarks.

Figures

Figures reproduced from arXiv: 2411.16156 by the authors.

Figure 1
Figure 1. With explicit modeling of object dynamics, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall architecture of VideoOrion. Two branches are employed to encode the video content into tokens: the Video-Centric Branch encodes the general information with context tokens, while the Object-Centric Branch encodes the dynamics of objects through the detect-segment-track pipelines in the video into a set of object tokens. All these tokens are fed together to the LLM for integrating information from both br… view at source ↗
Figure 3
Figure 3. Case studies showing how VideoOrion utilizes object tokens to generate responses based on different instructions [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Examples of the detect-segment-track pipeline. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 6
Figure 6. Figure 6: Failure of the detect-segment-track pipeline. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Failure case due to the object attention. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 5
Figure 5. Figure 5: Qualitative examples of VideoOrion+, VideoOrion-Ref and VideoOrion-Ref-FT+. can still correctly infer the action based on contextual cues. D.2. Failures on Object Attention Although VideoOrion successfully detects and tracks criti￾cal objects, it can still occasionally…
Figure 8
Figure 8. Figure 8: An illustration showcasting how VideoOrion benefits from increased training data. G. Hyperparameters We report in [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A dexterous VLA pretrained on a 2.5M-instance human hand motion dataset transfers skills to a real robot hand, outperforming baselines in manipulation tasks.

  2. Unified Multimodal Understanding via Byte-Pair Visual Encoding

    cs.CV 2025-06 conditional novelty 4.0 of 10

    Priority-guided byte-pair encoding of quantized image patches plus curriculum training yields an 8B discrete-token MLLM competitive with continuous-embedding models on VQA and multimodal benchmarks.

  3. Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI

    cs.CV 2025-04 reject novelty 4.0 of 10

    A modular vision-language pipeline for automatic surgical report generation from laparoscopic videos, evaluated on CholecT50 with high reported scores but a leakage-prone evaluation protocol.

Reference graph

Works this paper leans on

80 extracted references · 46 canonical work pages · cited by 3 Pith papers

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS,

  2. [2]

    Spice: Semantic propositional image cap- tion evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In ECCV, 2016. 6

    work page 2016

  3. [3]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Es- sam Sleiman, Deyao Zhu, Jian Ding, and Mohamed El- hoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413, 2024. 2

    arXiv 2024

  4. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1

    arXiv 2023

  5. [5]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, 2005. 6

    work page 2005

  6. [6]

    O’Reilly Media, Inc

    Steven Bird, Ewan Klein, and Edward Loper. Natural lan- guage processing with Python: analyzing text with the natu- ral language toolkit. " O’Reilly Media, Inc.", 2009. 5

    work page 2009

  7. [7]

    Temporalbench: Towards fine-grained temporal understanding for multimodal video models

    Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, and Jianwei Yang. Temporalbench: Towards fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410.10818, 2024. 8

    arXiv

  8. [8]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In ECCV, 2020. 3

    2020

Show all 80 references
  1. [9]

    Minigpt-v2: large language model as a unified interface 9 for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface 9 for vision-language multi-task learning. arXiv preprint arXiv:23...

    arXiv

  2. [10]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 7

    2024 arXiv

  3. [11]

    Weakly-supervised spatio-temporally grounding nat- ural sentence in video

    Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan-Yee K Wong. Weakly-supervised spatio-temporally grounding nat- ural sentence in video. arXiv preprint arXiv:1906.02549 ,

    1906 arXiv

  4. [12]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290–1299, 2022. 3, 7

    2022

  5. [13]

    Ho Kei Cheng and Alexander G. Schwing. Xmem: Long- term video object segmentation with an atkinson-shiffrin memory model. In ECCV, 2022. 3, 5

    2022

  6. [14]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 2, 4, 5, 7

    2024 arXiv

  7. [15]

    Study on density peaks clustering based on k-nearest neighbors and principal component analysis

    Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowl. Based Syst., 2016. 3

    2016

  8. [16]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  9. [17]

    Lasot: A high-quality benchmark for large-scale single ob- ject tracking

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single ob- ject tracking. In CVPR, 2019. 6

    2019

  10. [18]

    Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 6

    2024 arXiv

  11. [19]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. 5

    2022

  12. [20]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In CVPR, 2024. 5

    2024

  13. [21]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir- shick. Mask r-cnn. In ICCV, 2017. 3

    2017

  14. [22]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 8

    1997

  15. [23]

    Open-set image tagging with multi-grained text su- pervision

    Xinyu Huang, Yi-Jie Huang, Youcai Zhang, Weiwei Tian, Rui Feng, Yuejie Zhang, Yanchun Xie, Yaqian Li, and Lei Zhang. Open-set image tagging with multi-grained text su- pervision. arXiv e-prints, 2023. 5

    2023

  16. [24]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 4, 6

    arXiv

  17. [25]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. In CVPR, 2024. 3, 7

    2024

  18. [26]

    Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization

    Yang Jin, Zhicheng Sun, Kun Xu, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, Kun Gai, and Yadong Mu. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. In ICML, 2024. 2, 3

    2024

  19. [27]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross B

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In ICCV, 2023. 4

    2023

  20. [28]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 7

    2024 arXiv

  21. [29]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 2, 4

    2023

  22. [30]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2

    2023 arXiv

  23. [31]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, 2023. 5, 8

    2023

  24. [32]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In CVPR, 2024. 2, 5, 6, 7

    2024

  25. [33]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 2, 7

    2023 arXiv

  26. [34]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 5, 7

    2023 arXiv

  27. [35]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004. 6

    2004

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 4

    2023

  29. [37]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 5, 7

    2023

  30. [38]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 3, 4 10

    2023 arXiv

  31. [39]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207 ,

    arXiv

  32. [40]

    Video-chatgpt: Towards detailed video un- derstanding via large vision and language models

    Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video un- derstanding via large vision and language models. In ACL,

  33. [41]

    Trackflow: Multi-object tracking with normalizing flows

    Gianluca Mancusi, Aniello Panariello, Angelo Porrello, Matteo Fabbri, Simone Calderara, and Rita Cucchiara. Trackflow: Multi-object tracking with normalizing flows. In ICCV, 2023. 3

    2023

  34. [42]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. In NeurIPS, 2023. 6

    2023

  35. [43]

    Trackformer: Multi-object track- ing with transformers

    Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi-object track- ing with transformers. In CVPR, 2022. 3

    2022

  36. [44]

    Openvid-1m: A large-scale high-quality dataset for text-to- video generation

    Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhen- heng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to- video generation. arXiv preprint arXiv:2407.02371 , 2024. 5

    2024 arXiv

  37. [45]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1

    2023 arXiv

  38. [46]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 3

    2023 arXiv

  39. [47]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. 6

    2002

  40. [48]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adrià Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Do- ersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexandre Fréchette, Hanna Klimczak, Raphael Koster,...

    2023

  41. [49]

    Learning video object segmentation from static images

    Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, and Alexander Sorkine-Hornung. Learning video object segmentation from static images. In CVPR, 2017. 3

    2017

  42. [50]

    Towards generalizable multi-object tracking

    Zheng Qin, Le Wang, Sanping Zhou, Panpan Fu, Gang Hua, and Wei Tang. Towards generalizable multi-object tracking. In CVPR, 2024. 3

    2024

  43. [51]

    Artemis: Towards referential understanding in com- plex videos

    Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Yunjie Tian. Artemis: Towards referential understanding in com- plex videos. arXiv preprint arXiv:2406.00258, 2024. 2, 3, 6, 7

    2024 arXiv

  44. [52]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 2, 3, 4, 6

    2021

  45. [53]

    Girshick, Kaiming He, and Piotr Dollár

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020. 4

    2020

  46. [54]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024. 3, 4

    2024 arXiv

  47. [55]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2024. 7

    2024

  48. [56]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NIPS, 2022. 5

    2022

  49. [57]

    Audio-visual llm for video understanding

    Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual llm for video understanding. arXiv preprint arXiv:2312.06720, 2023. 1

    2023 arXiv

  50. [58]

    Human-centric spatio-temporal video grounding with visual transformers

    Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Tech- nology, 2021. 6

    2021

  51. [59]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  52. [60]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. NIPS, 2022. 8

    2022

  53. [61]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 2017. 8

    2017

  54. [62]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015. 6

    2015

  55. [63]

    Elysium: Exploring object-level perception in videos via mllm

    Han Wang, Yongjie Ye, Yanjie Wang, Yuxiang Nie, and Can Huang. Elysium: Exploring object-level perception in videos via mllm. In ECCV, 2024. 2

    2024

  56. [64]

    Fast online object tracking and segmentation: A unifying approach

    Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. In CVPR, 2019. 3

    2019

  57. [65]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2023. 5

    2023

  58. [66]

    Slot-vlm: Slowfast slots for video-language modeling

    Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Slot-vlm: Slowfast slots for video-language modeling. arXiv preprint arXiv:2402.13088, 2024. 2, 3 11

    2024 arXiv

  59. [67]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 4, 6

    2024 arXiv

  60. [68]

    Associating ob- jects with transformers for video object segmentation

    Zongxin Yang, Yunchao Wei, and Yi Yang. Associating ob- jects with transformers for video object segmentation. In NeurIPS, 2021. 3

    2021

  61. [69]

    Detclip: Dictionary-enriched visual-concept paralleled pre- training for open-world detection

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre- training for open-world detection. In NeurIPS, 2022. 3

    2022

  62. [70]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 4

    2023 arXiv

  63. [71]

    Merlin: Empowering multimodal llms with foresight minds

    En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge, Xi- angyu Zhang, et al. Merlin: Empowering multimodal llms with foresight minds. In ECCV, 2025. 2, 6, 7

    2025

  64. [72]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019. 6

    2019

  65. [73]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 6

    2023

  66. [74]

    Ni, and Heung-Yeung Shum

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2023. 3

    2023

  67. [75]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. In EMNLP, 2023. 1, 2, 4, 7

    2023

  68. [76]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 7

    2024 arXiv

  69. [77]

    Bytetrack: Multi-object tracking by associating every detection box

    Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In ECCV, 2022. 3

    2022

  70. [78]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 5, 7

    2024

  71. [79]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024. 2

    2024

  72. [80]

    Tracking anything in high quality

    Jiawen Zhu, Zhenyu Chen, Zeqi Hao, Shijie Chang, Lu Zhang, Dong Wang, Huchuan Lu, Bin Luo, Jun-Yan He, Jin- Peng Lan, et al. Tracking anything in high quality. arXiv preprint arXiv:2307.13974, 2023. 3 12 A. Additional Ablation Studies Due to space constraint, we provide the re...

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.