Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Task Preference Optimization improves multimodal LLMs by co-training differentiable vision-task heads, gaining 14.6% on benchmarks and zero-shot fine-grained skills.

desk verdict Useful recipe and one clean ablation; the headline 14.6% is confounded by extra conversation data, and Table 12 looks internally inconsistent. read the letter →

arxiv 2412.19326 v2 pith:2V53CWQC submitted 2024-12-26 cs.CV

Ziang Yan , Zhilin Li , Yinan He , Chenting Wang , Kunchang Li , Xinhao Li , Xiangyu Zeng , Zilei Wang
show 4 more authors
Yali Wang Yu Qiao Limin Wang Yi Wang
This is my paper · ORCID
classification cs.CV
keywords taskpreferenceoptimizationmultimodallargelanguagemodelsvisualalignmentfine-grainedperceptionmulti-taskco-trainingtemporalgroundingreferringsegmentationzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal large language models (MLLMs) describe scenes fluently but routinely miss precise visual facts such as which cup the candy is under, when an action starts, or where an object is in a frame. This paper proposes Task Preference Optimization (TPO), which attaches small task-specific heads to an MLLM and uses dense visual labels to backpropagate a “task preference” signal while the model continues learning dialogue. The authors claim that this improves general image and video understanding by 14.6% on average across multiple benchmarks, and simultaneously gives the MLLM zero-shot fine-grained skills such as tracking, moment retrieval, and referring segmentation that rival supervised expert models. If true, TPO offers a way to add perceptual precision to conversational models without sacrificing their general abilities.

What carries the argument

The machinery is a “task preference model”: a bank of learnable task tokens $\{v_i\}$ that the MLLM's language model transforms into task embeddings $e_i$, each feeding one of three task heads—a two-layer MLP region head for boxes, a CG-DETR-based temporal head for moment retrieval and highlights, and a SAM2-based mask head for segmentation and tracking. These tokens make task supervision differentiable into the LLM, because gradients from dense annotation losses update the MLLM's hidden representations through the tokens. TPO organizes training into three stages: instruction-tuned task assignment so the model emits the right special token, per-head training on task data, then joint co-training on task and conversation data, so the model first learns to recognize a task, then aligns each head, then lets the heads refine the shared visual backbone.

What would settle it

Run a matched control that trains VideoChat on the same task datasets and the same heads but without TPO's three-stage schedule (for instance, joint training from the start on all task and conversation data), and compare MVBench and Charades-STA. If that control matches or exceeds VideoChat-TPO, then the task-token routing and staged training, rather than the dense supervision itself, would not be the cause of the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that an MLLM can be improved for general dialogue by treating fine-grained visual annotations as differentiable task preferences rather than as text to be generated. TPO attaches several task heads—a region head, a temporal head, and a mask head—to hidden embeddings routed through learnable task tokens, and optimizes the MLLM with the sum of the dialogue loss, a task-assignment cross-entropy, and task-specific regression or classification losses. Using a three-stage local-to-global schedule (task assignment, head adaptation, joint co-training), the authors report that VideoChat-TPO and LLaVA-TPO outperform their baselines on image and video benchmarks while also acquiring zero-shot capabilities in spatial grounding, moment retrieval, highlight detection, tracking, and referring segmentation that are comparable to or better than supervised specialist models. The paper further claims that co-training multiple task heads together yields synergistic gains, with each individual task improving beyond its single-task training result.

Load-bearing premise

The gains are attributed to the TPO co-training recipe rather than to the extra volume of task data and the pre-trained expert heads; the synergy ablation in Table 12 adds data and heads together, so that separation is not directly tested.

Editorial extensions

If this is right

  • On MVBench, VideoChat-TPO scores 66.8 versus 60.4 for the VideoChat2 baseline, a 6.4-point gain with the same 16 input frames.
  • On VideoMME, the same model improves by 9.3 points overall and by 10 points on short videos without subtitles; on MLVU it gains 10.2 points in mean average.
  • Applying TPO to a second MLLM family, LLaVA-OneVision, lifts MVBench by 8.1 points and PerceptionTest by 6.9 points, suggesting the recipe transfers across models.
  • Zero-shot, VideoChat-TPO reaches 40.2 R@1 at IoU=0.5 on Charades-STA moment retrieval and 69.4 success on LaSOT tracking, beating specialist models that were fine-tuned on the target benchmarks.
  • Replacing textualized task data with TPO heads improves MVBench by 2.1 points while lifting moment retrieval R@0.5 from 18.6 to 40.2, indicating that dense supervision is preserved better through heads than through next-token prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A control that keeps data and heads fixed but replaces the three-stage recipe with one-stage joint training would isolate whether the schedule, rather than raw supervision, drives the gains; the paper's Table 12 cannot fully separate those factors because data and heads are added simultaneously.
  • If the mechanism is the routing of dense spatial-temporal gradients rather than the specific heads, then substituting depth, optical flow, or audio-visual sync heads should produce similar dialogue gains; this is a cheap empirical check of the “task preference” abstraction.
  • The reported synergy among heads suggests a design heuristic: choose a complementary set of heads that share underlying geometry (space, time, objects) but emit different output formats, and co-train them; this could turn head selection into a portfolio-optimization problem rather than a per-task choice.
  • The zero-shot tracking and segmentation results imply that MLLM-TPO could serve as a unified interface for interaction tasks such as pointing, following, and referring in robotic or embodied settings, an extension the authors do not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Task Preference Optimization (TPO), a three-stage training scheme that attaches differentiable visual task heads (region, temporal, and mask heads) to a multimodal large language model through learnable task tokens. Stage 1 trains the LLM to emit task-assignment tokens, Stage 2 trains the task heads and tokens on dense visual supervision, and Stage 3 co-trains the MLLM and heads on a mixture of task data and conversation data. The method is instantiated on VideoChat2 and LLaVA-OneVision. Experiments report gains on general video/image understanding benchmarks (MVBench, VideoMME, MLVU, MMIU, SEED-Bench2, NExT-GQA) and on spatial grounding, moment retrieval, highlight detection, tracking, and referring segmentation, with an overall claimed 14.6% average improvement over baselines and synergistic benefits from multi-task co-training.

Significance. If the attribution to the TPO training scheme is established, the paper would make a practical contribution: it offers a scalable way to inject dense visual supervision into MLLMs without the information loss of serializing masks, boxes, and timestamps into text. The strongest evidence in the paper is the matched control in Table 11, where TPO's decoupled heads outperform textualized task data at fixed data, and the data-scaling ablation in Table 15, which isolates the effect of task-data volume. The three-stage recipe is clearly described and the code release is promised. However, as detailed below, the headline 14.6% gain is not yet isolated from the large amount of additional conversation data, and the zero-shot task results are not separated from the pre-trained expert components (SAM2, CG-DETR, InternVideo2) embedded in the task heads. The central mechanism is defensible, but several load-bearing controls are missing.

major comments (4)
  1. [Section 4.1, Tables 2 and 4; Abstract] The reported '14.6% average improvement' is not a well-defined quantity. The underlying deltas are absolute point differences on benchmark-specific scales: MVBench +6.4, VideoMME +9.3, MLVU +10.2, MMIU +5.2, and SEED-Bench2 +40.8/+42.4. Averaging these point deltas gives about 14.6, but that number is neither a percentage improvement nor a meaningful pooled effect, and it is dominated by the SEED-Bench2 scale. In addition, the text in Section 4.1 says VideoChat-TPO achieves a '41.7% improvement on average performance' on SEED-Bench2, while Table 4 shows point gains of 40.8 and 42.4. Please report per-benchmark relative and absolute changes, define the aggregation protocol explicitly, or drop the pooled '14.6%' claim.
  2. [Section 4.3, Tables 18 and 19] The central comparison is VideoChat-TPO against the released VideoChat2 checkpoint, but TPO adds roughly 3M conversation samples (VideoChat2-IT, ShareGPT-4o, LLaVA-Hound-DPO, ShareGPT4V) plus about 0.77M task samples. The only same-data control, the 'textualized task data' row of Table 11, is reported only for MVBench and Charades-STA and retains the extra conversation data; it is not reported for VideoMME, MLVU, MMIU, or SEED-Bench2, which are the other benchmarks behind the 14.6% headline. Table 19 lists an 'only conversation data' run but gives no benchmark scores for it anywhere in the paper. Without those scores, the 14.6% improvement over VideoChat2 cannot be attributed to the TPO training signal rather than to the additional conversation data. Please evaluate the only-conversation run on all headline benchmarks and report it beside VideoChat2 and VideoChat-TPO.
  3. [Section 4.2 and Implementation Details] The zero-shot task results are confounded by pre-trained expert components. The temporal head is built on CG-DETR with pre-trained InternVideo2 features, and the mask head is initialized from SAM2; only the region head is randomly initialized. Tables 5, 7, 8, and 9 do not include head-only baselines that use the same heads and task data without MLLM co-training. For example, Table 8 reports tracking results but has no SAM2-only baseline row, so the 'zero-shot' tracking success may be largely inherited from SAM2 rather than produced by TPO. Please add head-only baselines (with the same task data, with and without fine-tuning) and report the increment specifically attributable to MLLM co-training.
  4. [Section 4.3, Table 12] The synergy experiment conflates adding tasks with adding data and parameters. Moving from row 4 to row 5 adds the mask head, mask training data, and mask token simultaneously, so the improvement in temporal grounding and referring segmentation could reflect more data or parameters rather than synergistic co-training; row 6 then adds conversation data, which is a separate data-scale effect already acknowledged in the text. An equal-data or total-parameter-matched control is needed before claiming that 'collaborative learning of visual tasks facilitates the transfer of knowledge' and produces synergy. The cleaner data-scaling comparison in Table 15 is a good start and should be extended to the co-training claim.
minor comments (5)
  1. [Section 4.1, 'Benchmarks' paragraph and Table 4] There is a typo 'SEED2-Bench2' in the benchmarks paragraph; the correct name is SEED-Bench2. Also, Table 4's column headers 'SEED2I' and 'SEED2M' are not defined in the text, and the text's '41.7% improvement' should be reconciled with the table's point gains.
  2. [Table 11 caption] The caption text 'replace by simple head' is a fragment; please specify which head was replaced and confirm that all other training data and hyperparameters were held fixed.
  3. [Equation (1) and surrounding text] The notation L_mllm is not defined; please state explicitly that it is the standard next-token cross-entropy loss over the dialogue and clarify how the three loss terms are combined when a training example contains both conversation and task supervision.
  4. [Figure 2 and Figure 3 captions] The figures are visually dense: the distinction between solid and dotted arrows, the meaning of flame symbols, and the P2S/P2E labels are hard to parse at print scale. The captions should define every symbol and state which modules are frozen and unfrozen at each stage.
  5. [Table 16 caption] The caption contains a typo, 'Perfermance', and the caption line is duplicated. Please also state whether the LLaVA-OV-TPO row uses the same stage-3 conversation data as VideoChat-TPO.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the TPO objective is an explicit sum of supervised losses, and the central claims are empirical comparisons rather than definitional identities.

full rationale

Equation (1) defines the TPO objective as L = L_mllm + L_assign(G(T_q), s) + sum_i L_task(A_i, H_i(G(v_i))), an explicit sum of standard cross-entropy, regression, and classification losses over conversation and visual-task labels. No term in this objective is defined in terms of the benchmarks being predicted (MVBench, VideoMME, MLVU, MMIU, SEED-Bench2), and no benchmark score is a training target. The claimed 14.6% improvement in Tables 2-4 is an empirical comparison of trained checkpoints against VideoChat2 and other models, not an identity that follows from the loss construction. The 'synergistic gains' claim in Table 12 is an ablation comparison; although some rows add heads and conversation data simultaneously, that is a confound in causal attribution, not a circular reduction. The use of pretrained expert components (SAM2, CG-DETR, InternVideo2) is explicitly disclosed in Section 3.1 and the implementation details, and the paper does not claim these components are derived from TPO, so the task-head results are inherited-capability attribution rather than circular reasoning. Self-citations to MVBench and VideoChat2 are benchmark and baseline references from the same group, but the evaluation also includes external benchmarks (VideoMME, MLVU, MMIU, SEED-Bench2, NExT-GQA, Charades-STA), so the central comparison is not forced by a self-citation chain. The missing benchmark scores for the 'only conversation data' run listed in Appendix Table 19 are a genuine experimental gap that weakens the claim that the 14.6% gain is caused specifically by the TPO training signal rather than by the extra 3M conversation samples; however, a missing control is a correctness-risk issue, not a logical circularity. No load-bearing step reduces to its own input by construction, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central mechanism relies on domain assumptions about MLLM training and benchmark validity. No physical constants or new forces are introduced. The learnable task tokens and task preference model are internal parameters with no independent evidence outside the paper's experiments.

free parameters (3)
  • Equal loss weights in Eq. (1) = 1.0 (implicit)
    The three loss terms are summed with unit weights; no sensitivity analysis is provided, so the results may depend on this unstated choice.
  • Stage-2 per-task epochs = region 1, temporal 25, mask 3
    Epoch counts differ by task and were selected by the authors; they affect the quality of the heads and thus the measured improvements.
  • LoRA rank and alpha = 16 and 32
    Standard hyperparameters, but part of the recipe; no ablation of rank is given.
assumptions (4)
  • domain assumption Backpropagating box, timestamp, and mask losses through the LLM improves the LLM's general visual representations
    The central mechanism of TPO (Section 3.2, Eq. 1). This is not proven; the paper's experiments are the evidence.
  • domain assumption The three-stage local-to-global curriculum prevents catastrophic forgetting of conversational ability
    Stage design in Section 3.2; no comparison against other schedules.
  • domain assumption The benchmark tasks are not contaminated by the training data
    Zero-shot claims in Section 4.2 assume no leakage from training datasets like QVHighlight or ActivityNet into Charades-STA, LaSOT, and GOT-10k.
  • domain assumption Frozen video features from InternVideo2 and text features from Chinese-Llama-Alpaca are compatible with the temporal head
    Implementation details, Section 4, temporal head design.
invented entities (2)
  • Learnable task tokens v_i
    purpose: Bridge between MLLM hidden states and task-specific heads; the model learns to activate them via instructions.
    These are trainable embedding vectors introduced by the paper; their only evidence is the internal experiments.
  • Task preference model (TPM)
    purpose: Abstraction encompassing the task tokens and heads; it converts visual annotations into training preferences.
    A conceptual container, not a physical entity; no external falsifiable handle beyond the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment." pith.science (2026). https://pith.science/paper/2V53CWQC

@misc{pith2026241219326,
  author       = {Pith},
  title        = {Pith review of: Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2V53CWQC}},
  note         = {Machine review of arXiv:2412.19326}
}
read the original abstract

Current multimodal large language models (MLLMs) struggle with fine-grained or precise understanding of visuals although they give comprehensive perception and reasoning in a spectrum of vision applications. Recent studies either develop tool-using or unify specific visual tasks into the autoregressive framework, often at the expense of overall multimodal performance. To address this issue and enhance MLLMs with visual tasks in a scalable fashion, we propose Task Preference Optimization (TPO), a novel method that utilizes differentiable task preferences derived from typical fine-grained visual tasks. TPO introduces learnable task tokens that establish connections between multiple task-specific heads and the MLLM. By leveraging rich visual labels during training, TPO significantly enhances the MLLM's multimodal capabilities and task-specific performance. Through multi-task co-training within TPO, we observe synergistic benefits that elevate individual task performance beyond what is achievable through single-task training methodologies. Our instantiation of this approach with VideoChat and LLaVA demonstrates an overall 14.6% improvement in multimodal performance compared to baseline models. Additionally, MLLM-TPO demonstrates robust zero-shot capabilities across various tasks, performing comparably to state-of-the-art supervised models. The code will be released at https://github.com/OpenGVLab/TPO

Figures

Figures reproduced from arXiv: 2412.19326 by the authors.

Figure 1
Figure 1. TPO uses differentiable task preferences from dense vi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Learning Method. A solid line indi￾cates data flow, and a dotted line represents feedback. Textual Data Visual Label Visual Label Reference Model DPO TPO P2S P2E Visual Label Conversation Data MLLM Task Head MLLM MLLM Task Head MLLM Maximum Likelihood Textual Data Visual Labels Visual Labels Reference Model DPO TPO P2S P2E Visual Labels Conversation Data MLLM Task Head MLLM MLLM Task Head MLLM Maximum … view at source ↗
Figure 3
Figure 3. Overall Pipeline of TPO. The architecture of Task Preference Optimization (TPO) consists of four main components: (1) a vision encoder, (2) a connector, (3) a large language model, and (4) a series of visual task heads. Differently colored flame symbols indicate which components are unfrozen at various stages of the training process. of handling a wide range of tasks across vision and lan￾guage domains, requiring mi… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative Results of Spatial Grounding [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Qualitative Results of Referring Segmentation [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Qualitative Results of Tracking [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Qualitative Results of Moment Retrieval and Highlight Detection. The orange curve represents saliency score, the blue interval represents the time interval predicted by the model, and the green interval represents the ground truth [PITH_FULL_IMAGE:figures/full_fig_p02…
Figure 8
Figure 8. Figure 8: Qualitative Results of Multimodal Video Understanding [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Survey on Video Temporal Grounding with Multimodal Large Language Model

    cs.CV 2025-08 unverdicted novelty 3.0 of 10

    A taxonomized review of video temporal grounding with multimodal large language models, covering model roles, training paradigms, feature processing, benchmarks, and open problems.

  2. MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answering

    cs.CV 2025-06

Reference graph

Works this paper leans on

145 extracted references · 39 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 6

    arXiv 2023

  2. [2]

    Openflamingo: An open- source framework for training large autoregressive vision- language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Worts- man, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision- language models. arXiv preprint...

    arXiv 2023

  3. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 14

    arXiv 2023

  4. [4]

    One token to seg them all: Language in- structed reasoning segmentation in videos

    Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language in- structed reasoning segmentation in videos. arXiv preprint arXiv:2409.19603, 2024. 3, 8

    arXiv 2024

  5. [5]

    Fully-convolutional siamese networks for object tracking

    Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully-convolutional siamese networks for object tracking. In ECCV, pages 850–865,

  6. [6]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961–970, 2015. 5, 8, 15

    2015

  7. [7]

    Fengwu: Pushing the skillful global medium- range weather forecast beyond 10 days lead

    Kang Chen, Tao Han, Junchao Gong, Lei Bai, Fenghua Ling, Jing-Jia Luo, Xi Chen, Leiming Ma, Tianning Zhang, Rui Su, et al. Fengwu: Pushing the skillful global medium- range weather forecast beyond 10 days lead. arXiv preprint arXiv:2304.02948, 2023. 1

    arXiv 2023

  8. [8]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 1, 3, 7

    arXiv 2023

Show all 145 references
  1. [9]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 15

    2023 arXiv

  2. [10]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, 2024. 5, 7, 14, 16

    2024

  3. [11]

    Sharegpt4video: Improving video under- standing and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video under- standing and generation with better captions.arXiv preprint arXi...

    2024 arXiv

  4. [12]

    A simple framework for contrastive learn- ing of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learn- ing of visual representations. In ICML, pages 1597–1607. PMLR, 2020. 8

    2020

  5. [13]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 5, 14, 15, 16

    2024

  6. [14]

    Efficient and ef- fective text encoding for chinese llama and alpaca

    Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and ef- fective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023. 5, 15

    2023 arXiv

  7. [15]

    Atom: Accurate tracking by overlap maximization

    Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Atom: Accurate tracking by overlap maximization. In CVPR, pages 4660–4669, 2019. 7, 8

    2019

  8. [16]

    A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching

    Pradipto Das, Chenliang Xu, Richard F Doell, and Jason J Corso. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In CVPR, pages 2634–2641, 2013. 5, 8, 15

    2013

  9. [17]

    MeViS: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. In ICCV,

  10. [18]

    Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model. arXiv preprint arXiv:2401.1...

    2024 arXiv

  11. [19]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. 1

  12. [20]

    Lasot: A high-quality benchmark for large-scale sin- gle object tracking

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Si- jia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale sin- gle object tracking. In CVPR, pages 5374–5383, 2019. 5, 7, 8

    2019

  13. [21]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 2, 5, 6, 14

    2024 arXiv

  14. [22]

    An empirical study of end-to-end video-language transformers with masked vi- sual modeling

    Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. An empirical study of end-to-end video-language transformers with masked vi- sual modeling. In CVPR, 2023. 7

    2023

  15. [23]

    Tall: Temporal activity localization via language query

    Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, pages 5267–5275, 2017. 5, 6, 7, 8, 14

    2017

  16. [24]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995–19012, 2022. 5, 15

    2022

  17. [25]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In CVPR, pages 9729–9738, 2020. 8

    2020

  18. [26]

    Localizing mo- ments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing mo- ments in video with natural language. In ICCV, 2017. 5, 15

    2017

  19. [27]

    Lora: Low- rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low- rank adaptation of large language models. In ICLR, 2022. 4, 14

    2022

  20. [28]

    Vtimellm: Empower llm to grasp video mo- ments

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video mo- ments. In CVPR, 2024. 7

    2024

  21. [29]

    Got-10k: A large high-diversity benchmark for generic object tracking in the wild

    Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. TPAMI, 43(5):1562–1577, 2019. 5, 7, 8

    2019

  22. [30]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 5

    arXiv

  23. [31]

    Sparse sharing relation network for panoptic driving perception

    Fan Jiang and Zilei Wang. Sparse sharing relation network for panoptic driving perception. In ACMMM, pages 800– 808, 2023. 1

    2023

  24. [32]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. In CVPR, pages 13700–13710, 2024. 6

    2024

  25. [33]

    Language repository for long video understanding

    Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. arXiv preprint arXiv:2403.14622 ,

    arXiv

  26. [34]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InnaacL-HLT, page 2. Minneapolis, Minnesota, 2019. 5

    2019

  27. [35]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV, pages 4015–4026, 2023. 3

    2023

  28. [36]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32–73, 2017. 5, 15

    2017

  29. [37]

    Lisa: Reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model. In CVPR, pages 9579–9589,

  30. [38]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. NeurIPS, 34:11846–11858, 2021. 5, 7, 13, 15

    2021

  31. [39]

    SiamRPN++: Evolution of siamese visual tracking with very deep networks

    Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. SiamRPN++: Evolution of siamese visual tracking with very deep networks. In CVPR, pages 4282– 4291, 2019. 8

    2019

  32. [40]

    Seed-bench: Bench- marking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Bench- marking multimodal large language models. In CVPR, pages 13299–13308, 2024. 2, 5, 7

    2024

  33. [42]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. CoRR, abs/2408.03326, 2024. 14

    2024 arXiv

  34. [43]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 7, 14

    2024 arXiv

  35. [44]

    Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks

    Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. In CVPR, pages 2691–2700, 2023. 2, 3

    2023

  36. [45]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742. PMLR, 2023. 3

    2023

  37. [46]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2, 3, 14

    2023 arXiv

  38. [47]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, pages 19948–19960, 2023. 2, 5

    2023

  39. [48]

    Mvbench: A comprehensive multi-modal video under- standing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video under- standing benchmark. In CVPR, pages 22195–22206, 2024. 2, 5, 6, 7, 14, 15, 16

    2024

  40. [49]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 6, 14

    2023 arXiv

  41. [50]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In ICCV, pages 2794–2804,

  42. [51]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2024. 2, 3, 7, 8, 14

    2024

  43. [52]

    St-llm: Large language models are effective temporal learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In ECCV, 2024. 5, 6, 14

    2024

  44. [53]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024. 6, 7

    2024

  45. [54]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 14, 15

    2017 arXiv

  46. [55]

    Unified-io: A uni- fied model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A uni- fied model for vision, language, and multi-modal tasks. In ICLR, 2022. 2

    2022

  47. [56]

    Videogpt+: Integrating image and video en- coders for enhanced video understanding

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video en- coders for enhanced video understanding. arXiv preprint arXiv:2406.09418, 2024. 14

    2024 arXiv

  48. [57]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In ACL, 2024. 3, 7, 14

    2024

  49. [58]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, pages 11–20, 2016. 5, 15

    2016

  50. [59]

    Mmiu: Multimodal multi-image understanding for evaluating large vision-language models.arXiv preprint arXiv:2408.02718, 2024

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models.arXiv preprint arXiv:2408.02718, 2024. 2, 5, 7, 14

    2024 arXiv

  51. [60]

    Correlation-guided query-dependency calibration in video representation learning for temporal grounding

    WonJun Moon, Sangeek Hyun, SuBeen Lee, and Jae- Pil Heo. Correlation-guided query-dependency calibration in video representation learning for temporal grounding. arXiv preprint arXiv:2311.08835, 2023. 5, 8, 15

    2023 arXiv

  52. [61]

    Query-dependent video representa- tion for moment retrieval and highlight detection

    WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representa- tion for moment retrieval and highlight detection. InCVPR, pages 23023–23033, 2023. 7

    2023

  53. [62]

    Queryd: A video dataset with high-quality text and audio narrations

    Andreea-Maria Oncescu, Joao F Henriques, Yang Liu, An- drew Zisserman, and Samuel Albanie. Queryd: A video dataset with high-quality text and audio narrations. In ICASSP, pages 2265–2269. IEEE, 2021. 5, 15

    2021

  54. [63]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Kop- pula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. 2024. 14

    2024

  55. [64]

    Streaming long video understanding with large language models

    Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuan- grui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. arXiv preprint arXiv:2405.16009, 2024. 7

    2024 arXiv

  56. [65]

    Chatvtg: Video temporal grounding via chat with video dialogue large language models

    Mengxue Qu, Xiaodong Chen, Wu Liu, Alicia Li, and Yao Zhao. Chatvtg: Video temporal grounding via chat with video dialogue large language models. In CVPR, pages 1847–1856, 2024. 6, 7

    2024

  57. [66]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 36, 2024. 2, 3

    2024

  58. [67]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion param- eters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion param- eters. In SIGKDD, pages 3505–3506, 2020. 15

    2020

  59. [68]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll ´ar, and Christoph Fe-...

    2024 arXiv

  60. [69]

    Ground- ing action descriptions in videos

    Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Ground- ing action descriptions in videos. Transactions of the As- sociation for Computational Linguistics, 1:25–36, 2013. 5, 15

    2013

  61. [70]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. CVPR, abs/2312.02051, 2024. 1, 3, 6, 7, 14

    2024 arXiv

  62. [71]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 3

    2017 arXiv

  63. [72]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In ECCV, pages 208–223. Springer,

  64. [73]

    Intern: A new learning paradigm towards general vision

    Jing Shao, Siyu Chen, Yangguang Li, Kun Wang, Zhenfei Yin, Yinan He, Jianing Teng, Qinghong Sun, Mengya Gao, Jihao Liu, et al. Intern: A new learning paradigm towards general vision. arXiv preprint arXiv:2111.08687, 2021. 1, 2

    2021 arXiv

  65. [74]

    Aligning large multi- modal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023. 3

    2023 arXiv

  66. [75]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 1

    2024 arXiv

  67. [76]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, pages 10078–10093, 2022. 2

    2022

  68. [77]

    Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models

    Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yu- fan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollm: Sharpening fine-grained tem- poral grounding in video large language models. arXiv preprint arXiv:2410.03290, 2024. 3

    arXiv

  69. [78]

    Temporal segment networks: Towards good practices for deep action recogni- tion

    Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recogni- tion. In ECCV, pages 20–36. Springer, 2016. 2

    2016

  70. [79]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yi- nan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023. 2

    2023

  71. [80]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, pages 23318–23340. PMLR, 2022. 7

    2022

  72. [81]

    Masked video distillation: Rethinking masked feature mod- eling for self-supervised video representation learning

    Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature mod- eling for self-supervised video representation learning. In CVPR, pages 6312–6322, 2023. 2

    2023

  73. [82]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NeurIPS,

  74. [83]

    The all-seeing project v2: Towards general relation comprehension of the open world

    Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-seeing project v2: Towards general relation comprehension of the open world. arXiv preprint arXiv:2402.19474, 2024. 5, 15, 16

    2024 arXiv

  75. [84]

    Seggpt: Towards seg- menting everything in context

    Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Towards seg- menting everything in context. In ICCV, pages 1130–1140,

  76. [85]

    Internvideo2: Scaling video foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. In ECCV, 2024. 2, 5, 15

    2024

  77. [86]

    Hawkeye: Training video-text llms for grounding text in videos

    Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan Zhao. Hawkeye: Training video-text llms for grounding text in videos. arXiv preprint arXiv:2403.10228, 2024. 3, 7

    2024 arXiv

  78. [87]

    Onlinerefer: A simple online baseline for referring video object segmentation

    Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen. Onlinerefer: A simple online baseline for referring video object segmentation. In ICCV, pages 2761–2770, 2023. 8

    2023

  79. [88]

    Language as queries for referring video object seg- mentation

    Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object seg- mentation. In CVPR, pages 4974–4984, 2022. 7, 8

    2022

  80. [89]

    Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks

    Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Wenhai Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. arXiv preprint arXiv:2406.08394,

    arXiv

  81. [90]

    Can i trust your answer? visually grounded video question answering

    Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In CVPR, 2024. 5, 6

    2024

  82. [91]

    Videoclip: Contrastive pre- training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre- training for zero-shot video-text understanding. InEMNLP, pages 6787–6800, 2021. 2

    2021

  83. [92]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 6, 14

    2024 arXiv

  84. [93]

    Siamfc++: Towards robust and accurate visual tracking with target estimation guidelines

    Yinda Xu et al. Siamfc++: Towards robust and accurate visual tracking with target estimation guidelines. In AAAI, pages 140–148, 2020. 8

    2020

  85. [94]

    Video-text mod- eling with zero-shot transfer from contrastive captioners

    Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, So- ham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text mod- eling with zero-shot transfer from contrastive captioners. ArXiv, abs/2212.04979, 2022. 2

    2022 arXiv

  86. [95]

    Zero-shot video question answering via frozen bidirectional language models

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. In NeurIPS, 2022. 7

    2022

  87. [96]

    mplug-owl: Modularization empowers large lan- guage models with multimodality, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large lan- guage models...

    2023

  88. [97]

    Merlin: Empowering multimodal llms with foresight minds

    En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wu, Lingyu Kong, Haoran Wei, Tiancai Wang, Zheng Ge, Xiangyu Zhang, et al. Merlin: Empowering multimodal llms with foresight minds. In ECCV, pages 425–443. Springer, 2025. 3, 7, 8

    2025

  89. [98]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. In ECCV, pages 69–85. Springer, 2016. 5, 6, 7, 15

    2016

  90. [99]

    Mattnet: Modular at- tention network for referring expression comprehension

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular at- tention network for referring expression comprehension. In CVPR, pages 1307–1315, 2018. 7

    2018

  91. [100]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. In NeurIPS, 2023. 7

    2023

  92. [101]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In CVPR, pages 13807–13816, 2024. 3

    2024

  93. [102]

    Hierarchi- cal video-moment retrieval and step-captioning

    Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Bar- las Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchi- cal video-moment retrieval and step-captioning. In CVPR, pages 23056–23065, 2023. 5, 15

    2023

  94. [103]

    Timesuite: Improving mllms for long video understanding via grounded tuning

    Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhen- grong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning. In ICLR. 1

  95. [104]

    Next-chat: An lmm for chat, detec- tion and segmentation

    Ao Zhang, Liming Zhao, Chen-Wei Xie, Yun Zheng, Wei Ji, and Tat-Seng Chua. Next-chat: An lmm for chat, detec- tion and segmentation. arXiv preprint arXiv:2311.04498 ,

    arXiv

  96. [105]

    A simple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235, 2023. 6, 7

    2023 arXiv

  97. [106]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023. 14

    2023 arXiv

  98. [107]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In EMNLP, pages 543–553, 2023. 3

    2023

  99. [108]

    Direct preference op- timization of video large multimodal models from language model reward

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexan- der Hauptmann, Yonatan Bisk, et al. Direct preference op- timization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258, 2024. 5, 15, 16

    2024 arXiv

  100. [109]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 3

    2023 arXiv

  101. [110]

    Llava-next: A strong zero-shot video understanding model,

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model,

  102. [111]

    Videoprism: A foundational visual encoder for video understanding

    Long Zhao, Nitesh Bharadwaj Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. In ICML, 2024. 2

    2024

  103. [112]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. 2, 5, 6

    2024 arXiv

  104. [113]

    Aligning modalities in vision large language models via preference fine-tuning

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. In ICLR Workshop on Reliable and Responsible Foundation Models,

  105. [114]

    Uni- perceiver-moe: Learning sparse generalist models with conditional moes

    Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni- perceiver-moe: Learning sparse generalist models with conditional moes. In NeurIPS, pages 2664–2678, 2022. 2, 3

    2022

  106. [115]

    Uni-perceiver: Pre- training unified architecture for generic perception for zero- shot and few-shot tasks

    Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre- training unified architecture for generic perception for zero- shot and few-shot tasks. In CVPR, pages 16804–16815,

  107. [116]

    We present the detailed performance of MVBench in Table 13, VideoChat-TPO achieves an av- erage score of 66.8, increasing by 6.4 points based on VideoChat2

    Experiment Details MVbench. We present the detailed performance of MVBench in Table 13, VideoChat-TPO achieves an av- erage score of 66.8, increasing by 6.4 points based on VideoChat2. It gets superior performance among MLLMs with the same number of input frames and LLMs of co...

  108. [117]

    In each stage, the model is parametrized from the weights from the previ- ous stage and continues training

    Training and Data Details Table 17 and 18 lists the detailed training configurations and data of VideoChat-TPO in different stages. In each stage, the model is parametrized from the weights from the previ- ous stage and continues training. Settings of Stage 1. The LLM is equip...

  109. [118]

    Localize the visual content described by the given textual query ⟨query ⟩ in the video, and output the start and end timestamps in seconds

  110. [119]

    Detect and report the start and end timestamps of the video segment that semantically matches the given textual query ⟨query ⟩

  111. [120]

    Locate and describe the visual content mentioned in the text query ⟨query ⟩ within the video, including timestamps

  112. [121]

    The given natural language query ⟨query ⟩ is semantically aligned with a video moment, please give the start time and end time of the video moment

  113. [122]

    Table 20

    Find the video segment that corresponds to the given textual query ⟨query ⟩ and determine its start and end seconds. Table 20. Instructions for Temporal Grounding

  114. [123]

    Track the object in the video using a box with initial coordinates ⟨track box⟩

  115. [124]

    Use a bounding box with coordinates ⟨track box⟩ to follow the movement of the moving object in the visual input

  116. [125]

    Given an initial bounding box with coordinates ⟨track box⟩, track the motion of the target object in the sequence of frames

  117. [126]

    Starting from the box defined by the coordinates ⟨track box⟩, monitor the movement of the object in the video

  118. [127]

    Utilizing the initial box specified by the coordinates ⟨track box⟩, continuously track and update the location of the object in the video stream

  119. [128]

    Given a video with an object of interest enclosed in a bounding box with coordinates ⟨track box⟩, generate a sequence of bounding boxes that track the object’s movement

  120. [129]

    With an initial box defined by ⟨track box⟩, trace the object’s trajectory by generating a sequence of bounding boxes that follow the object’s movement in the visual input

  121. [130]

    Apply an object tracking algorithm to a video, starting with a bounding box defined by ⟨track box⟩

  122. [131]

    Given a video and an initial bounding box defined by ⟨track box⟩, track the movement of the object within the video

  123. [132]

    Table 21

    Starting from an initial box defined by ⟨track box⟩, track the movement of the object in the visual input. Table 21. Instructions for Tracking

  124. [133]

    Can you find ⟨expr⟩?

  125. [134]

    Can you detect ⟨expr⟩?

  126. [135]

    Can you locate ⟨expr⟩?

  127. [136]

    Please detect ⟨expr⟩?

  128. [137]

    Please locate ⟨expr⟩?

  129. [138]

    Instructions for Spatial Grounding

    Locate ⟨expr⟩? Table 22. Instructions for Spatial Grounding

  130. [139]

    Please give the motion path of ⟨obj⟩ in the video over time

  131. [140]

    Show the tracking trajectory of ⟨obj⟩’s movement through the scene in the video

  132. [141]

    Please generate a motion path of ⟨obj⟩’s movement in the video, highlighting its tracking trajectory

  133. [142]

    Show the tracking trajectory of ⟨obj⟩

  134. [143]

    Generate ⟨obj⟩’s tracking trajectory

  135. [144]

    Visualize the tracking trajectory of ⟨obj⟩ in the video

  136. [145]

    Table 23

    Please generate a visual representation of ⟨obj⟩’s movement in the video, highlighting its tracking trajectory. Table 23. Instructions for Referring Segmentation

  137. [146]

    Dog playing with monkey

    Qualitative Results We evaluate VideoChat-TPO on various visual perception tasks and display the visualizations from Figure 4 to Figure Figure 7. In addition, we also show the results of multi- modal video understanding in Figure 8. Spatial Grounding. In Figure 4, we show the ...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.