Pith. sign in

REVIEW 4 major objections 5 minor 80 references

Object-centric Video Question Answering with Visual Grounding and Referring

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RGA3 unifies object referring and grounding in video, letting users point at any frame and get both a reasoned answer and a segmentation mask across 12 benchmarks.

desk verdict A genuinely useful unification of referring and grounding in video QA, but the headline VideoInfer comparison is unfair and the STOM ablation is inconsistent, so the arbitrary-prompt claim needs more evidence. read the letter →

arxiv 2507.19599 v1 pith:MPVEIVNM submitted 2025-07-25 cs.CV

Haochen Wang , Qirui Chen , Cilin Yan , Jiayin Cai , Xiaolong Jiang , Yao Hu , Weidi Xie , Stratis Gavves This is my paper
classification cs.CV
keywords object-centricvideounderstandingquestionansweringreferringobjectsegmentationvisualpromptingspatio-temporalpromptpropagationmultimodallargelanguagemodelsreasoninginstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to close a gap in VideoLLMs: existing models answer holistically with text only, so users cannot point at a specific object in a video and get a response about that object. RGA3 lets users provide arbitrary visual prompts, such as masks, boxes, arrows, points, or scribbles, at any timestamp, propagates that prompt across frames, and generates both a textual answer and, when asked, segmentation masks for the referred object. The authors claim this unified model outperforms prior generalist and specialist models on 12 benchmarks spanning 6 tasks, including open-ended VideoQA on their new VideoInfer dataset and reasoning video segmentation on ReVOS and ReasonVOS. If the claim holds, object-centric video dialogue becomes possible in one system rather than separate QA and segmentation pipelines.

What carries the argument

The Spatial-Temporal Overlay Module (STOM) is the load-bearing component: it converts a single-frame visual prompt, stored as an RGBA image, into per-frame overlays by tracking points around the prompt's centroid with CoTracker3 in both directions, then drawing a circle covering the visible propagated pixels and alpha-blending it onto each frame. A second piece is the [SEG] token, whose hidden state is projected into the SAM2 prompt-encoder space and fed to the SAM2 decoder together with dense features from the Hiera-L encoder, producing segmentation masks. Training couples autoregressive text loss with per-pixel BCE and DICE mask losses on a mix of image and video QA and segmentation data, with the visual encoder frozen and the LLM tuned by LoRA.

What would settle it

Take a video where the pointed object is fully occluded for several seconds or changes shape drastically after the prompt frame; if RGA3's answer and mask describe a different object once the propagated circle drifts onto the background, while the oracle-mask variant still answers correctly, the STOM tracking premise is falsified. A more direct check is to compare CoTracker3's propagated points with the ground-truth object centers on those frames and see whether drift size predicts the wrong answer.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the combination of a spatial-temporal overlay propagation and a single [SEG] token pipeline lets one model do object referring and grounding simultaneously in videos. Instead of pooling region features from a mask, STOM tracks the centroid of whatever prompt the user draws, redraws a translucent circle (or the original shape) on every frame, and alpha-blends it into the input, so the LLM sees the prompt's location end-to-end. The paper reports that RGA3-7B outperforms VideoRefer on VideoInfer GPT-4o accuracy by 6.3 points, beats VISA and VideoLISA on ReVOS by roughly 10 points in J&F, and achieves strong results on image benchmarks as well, with ablations showing STOM contributes the main referring gain and the decoder choice a smaller segmentation gain.

Load-bearing premise

STOM's load-bearing premise is that tracking the centroid of a visual prompt with a point tracker keeps the same object identified across occlusion and deformation, so the overlaid circle still refers to what the user pointed at.

Editorial extensions

If this is right

  • A user can point, box, or scribble on any single frame and then ask follow-up questions about that object in later frames; the model keeps the reference alive through its answer.
  • The same model can alternate text-only video QA with grounded segmentation in one conversation, so a single system covers general VideoQA, referring QA, and referring and reasoning segmentation.
  • Prompts that only partially overlap an object, such as arrows, scribbles, or outlines, are handled without feature-pooling noise; the ablation shows STOM raises VideoInfer accuracy from 43.6 to 46.7.
  • Because the visual encoder and SAM2 are frozen and only the LLM is LoRA-tuned, the framework is relatively cheap to train and could be extended to other backbones.
  • Joint training on QA and segmentation slightly hurts each individual task compared to separate training, so multi-stage training is an immediate next step that the authors note.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: since STOM reduces every prompt to a circle around the tracked centroid, prompt types that carry orientation or shape information, namely arrows and scribbles, may be under-served; testing each of the eight prompt types separately would reveal whether the reported gain is uniform or concentrated in point-like prompts.
  • Editorial inference: VideoInfer's open-ended reasoning questions could be answered partly from language priors or a single static frame; an image-only or text-only probe of RGA3 on VideoInfer would separate genuine temporal reasoning from shortcutting.
  • Editorial inference: the architecture separates tracking, reasoning, and masking; replacing CoTracker3 with a semantic object-aware tracker should directly improve long-occlusion cases, which is a testable extension beyond the paper.
  • Editorial inference: the 12-benchmark claim could be stress-tested on real user-drawn prompts, which tend to be messier than the randomly generated masks, boxes, and scribbles used in training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents RGA3, a VideoLLM that accepts arbitrary visual prompts (masks, boxes, arrows, points, scribbles) at any timestamp and can generate both textual answers and SAM2 segmentation masks. To propagate a single-frame visual prompt across time, it introduces STOM, which tracks points in a circle around the prompt centroid with CoTracker3 and overlays the propagated mark on every frame. The authors also contribute VideoInfer, a manually curated object-centric video QA dataset with 1,620 videos and 28,811 QA pairs. Evaluation is reported on VideoInfer, VideoRefer-BenchQ, ViP-Bench, and multiple referring/reasoning segmentation benchmarks, with claims of consistent state-of-the-art performance across six tasks.

Significance. If the claims are sustained, the contribution is significant: RGA3 would be one of the first VideoLLMs to unify arbitrary-prompt referring inputs with segmentation outputs, and VideoInfer is a more reasoning-heavy benchmark than prior automated pipelines. The paper has notable strengths: it evaluates on many external, established benchmarks (ReVOS, ReasonVOS, MeViS, Ref-DAVIS, RefCOCO family, ViP-Bench), reports both 3B and 7B results, provides an ablation of the SAM decoder versus SAM2, includes a failure-case analysis, and states an intent to release code and data. However, the central comparative claims are currently undermined by an in-distribution comparison on VideoInfer and by inconsistent STOM ablations; these are fixable but require additional experiments.

major comments (4)
  1. [Sec. 4.2, Table 1; Sec. 3.3] The headline comparison against VideoRefer on VideoInfer is not a fair state-of-the-art comparison. RGA3 is instruction-tuned on the VideoInfer training split (950 videos, 20,320 QA pairs), while VideoRefer-7B was never trained on this dataset; the VideoInfer test split (670 videos, 8,491 QA pairs) comes from the same annotation protocol and partly from the same source video pools (TAO, LVOS, VIPSeg, and others). The reported gain of 6.3 accuracy / 0.31 score therefore conflates model quality with an in-distribution training advantage. Please add a VideoRefer baseline fine-tuned on the VideoInfer instruction split, or otherwise equalize training data, and in the text separate "performance on our benchmark" from "outperforms existing models."
  2. [Table 6 and Supplementary Table 14] The oracle-propagation ablation is internally inconsistent and weakens the claim that STOM's tracking is the source of the gains. On VideoRefer-BenchQ the numbers order as expected (oracle 75.1 > STOM 74.0 > no STOM 72.2), but on VideoInfer STOM (46.7/2.60) exceeds the oracle (44.8/2.52), and Supplementary Table 14 shows that adding STOM gives no improvement on ReasonVOS (51.7 with and without STOM for the 3B model). If ground-truth mask propagation is not an upper bound, the measured effect of STOM is not robustly explained. Please report per-prompt-type results, variance across random seeds or test splits, and an oracle variant that preserves the original prompt type instead of replacing it with a mask.
  3. [Sec. 3.2, Visual Prompting; Tables 1-2] The central "arbitrary visual prompt" claim is not yet supported for non-mask prompts. STOM tracks points in a circle of radius r around the geometric centroid of the prompt; for an arrow the centroid lies on the shaft rather than at the tip, and for a partial scribble or point the circle can include background or drift to another object. The paper reports only aggregate accuracy over eight prompt types in Tables 1 and 2, so the aggregate gain could be driven by mask/box prompts while arrow/point/scribble propagation fails. Please report a per-prompt-type breakdown on VideoInfer and VideoRefer-BenchQ and provide a qualitative analysis of tracking under occlusion and viewpoint change.
  4. [Sec. 4.2 and Abstract] The abstract's claim of "consistently outperforms baselines in both video question answering and segmentation" is broader than the evidence. On general VideoQA in Supplementary Table 10, RGA3-7B scores 75.3 on NExT-QA, below LLaVA-Video-7B's 83.2, and on ViP-Bench (Table 3) RGA3 is comparable to, rather than clearly better than, ViP-LLaVA. The claim should be qualified to the object-centric referring/grounding settings, and the VideoInfer result should not be counted as a baseline comparison until the training-data imbalance is addressed.
minor comments (5)
  1. [Supplementary Tables 11-12] The method name is misspelled as "RAG3" in the RGA3 rows of Tables 11 and 12.
  2. [Sec. 3.2] The sentence "We usealpha blending" is missing a space, and the description "A new circle (or the original shape)" never states when the original shape is preserved versus replaced by a circle; this changes what the model sees in propagated frames.
  3. [Sec. 4.3, Table 7] The text says SAM versus SAM2 gives a 15.7 improvement on ReasonSeg; from Table 7 the gIoU improvement is 15.8 (68.5 - 52.7) and the cIoU improvement is 12.4 (70.2 - 57.8). Please reconcile the stated number.
  4. [Table 6] The notation "(16)" and "(8)" in the column headers should be defined in the caption as the number of input frames.
  5. [Conclusion] The paper says the dataset, code, and demo are publicly released, but the main text provides only the project page URL; include the actual links if available.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are evaluated on external benchmarks and the VideoInfer split is a held-out test set.

full rationale

The paper's central claims are that RGA3 supports arbitrary visual prompts in video and outperforms baselines on both video QA and segmentation. These claims are supported by evaluations on external benchmarks (VideoRefer-BenchQ, ViP-Bench, ReVOS, ReasonVOS, MeViS, Ref-Youtube-VOS, Ref-DAVIS, refCOCO, ReasonSeg, LongVideoBench) that were not constructed from RGA3's outputs or fitted values. STOM is an algorithmic module using an off-the-shelf point tracker (CoTracker3) and alpha-blending; it is not defined in terms of the evaluation metrics or the dataset answers, so there is no self-definitional reduction. The VideoInfer dataset has an explicit train/test split (950 training videos and 670 test videos), so reporting performance on the test split is standard held-out evaluation rather than circularity. Self-citations such as VISA [60] appear as baselines and as sources of training/evaluation data, but they are not used as load-bearing justifications of the model's design or as uniqueness theorems forcing the approach. The oracle-propagation result in Table 6 (where ground-truth mask propagation is lower than STOM on VideoInfer) is an empirical anomaly that may indicate a robustness or evaluation issue, but it is not a case of a prediction reducing by construction to its input. No fitted parameter is renamed as a prediction, and no known result is merely relabeled. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard deep learning training plus domain assumptions about point tracking reliability, alpha-blending interpretability, and GPT-4o as a judge. No new physical or theoretical entities are introduced; STOM is a proposed module, not an invented entity in the sense of a new particle or force.

free parameters (3)
  • loss weights lambda_txt, lambda_bce, lambda_dice = 1.0, 2.0, 0.5
    Hand-chosen to balance text generation and mask losses; the values are not derived and affect the trained model's behavior.
  • input frame count for inference = 16 frames
    The model is limited to 16 input frames per video (Sec. 7.2), and results depend on this sampling density, as the authors acknowledge in the failure analysis.
  • tracking radius r around prompt centroid = unspecified
    STOM tracks points in a circle of radius r around the prompt centroid; the chosen r is not reported and affects how much of the prompt is propagated.
assumptions (5)
  • domain assumption CoTracker3 provides accurate bi-directional point tracks in videos.
    STOM's propagation of visual prompts depends entirely on point tracking; failures under occlusion or drift would misplace prompts.
  • domain assumption Alpha-blended RGBA prompts overlaid on frames allow the LLM to learn geometric relationships between prompts and objects end-to-end.
    The paper argues this instead of mask pooling; if the LLM cannot interpret overlays, the referring mechanism fails.
  • domain assumption GPT-4o-based automatic scoring is a valid proxy for open-ended QA quality.
    VideoInfer results rely on GPT-4o accuracy and score; no human correlation or variance is reported.
  • domain assumption Training on a mixture of existing image and video QA and segmentation datasets transfers to object-centric referring and grounding.
    The model's general ability is claimed from co-training on 20+ datasets without a per-dataset transfer analysis.
  • standard math Standard backpropagation and deep learning training assumptions hold.
    All neural network training relies on these; not specific to the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Object-centric Video Question Answering with Visual Grounding and Referring." pith.science (2026). https://pith.science/paper/MPVEIVNM

@misc{pith2026250719599,
  author       = {Pith},
  title        = {Pith review of: Object-centric Video Question Answering with Visual Grounding and Referring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPVEIVNM}},
  note         = {Machine review of arXiv:2507.19599}
}
read the original abstract

Video Large Language Models (VideoLLMs) have recently demonstrated remarkable progress in general video understanding. However, existing models primarily focus on high-level comprehension and are limited to text-only responses, restricting the flexibility for object-centric, multiround interactions. In this paper, we make three contributions: (i) we address these limitations by introducing a VideoLLM model, capable of performing both object referring for input and grounding for output in video reasoning tasks, i.e., allowing users to interact with videos using both textual and visual prompts; (ii) we propose STOM (Spatial-Temporal Overlay Module), a novel approach that propagates arbitrary visual prompts input at any single timestamp to the remaining frames within a video; (iii) we present VideoInfer, a manually curated object-centric video instruction dataset featuring questionanswering pairs that require reasoning. We conduct comprehensive experiments on VideoInfer and other existing benchmarks across video question answering and referring object segmentation. The results on 12 benchmarks of 6 tasks show that our proposed model consistently outperforms baselines in both video question answering and segmentation, underscoring its robustness in multimodal, object-centric video and image understanding. Project page: https://qirui-chen.github.io/RGA3-release/.

Figures

Figures reproduced from arXiv: 2507.19599 by the authors.

Figure 1
Figure 1. RGA3 enables both referring and grounding capabili￾ties for object-centric video understanding. In contrast to previous VideoLLMs, which are limited to text-only responses with mask reference, RGA3 can process arbitrary visual prompts and gener￾ate both textual answers and segmentation masks. guage Models (VideoLLMs) [9, 13, 14, 30, 33, 61, 71] have primarily focused on holistic scene comprehension, such as answerin… view at source ↗
Figure 2
Figure 2. The proposed RGA3 architecture overview. (a) The Spatial-Temporal Overlay Module (STOM) is introduced to process arbi￾trary visual prompts (e.g., scribble, ellipse, arrow, etc.) at any timestamp and propagate to all frames, allowing for interactive object-centric reasoning and continual visual attention. (b) A visual encoder is employed to extract video representations of overlaid frames processed through STOM. (c) … view at source ↗
Figure 3
Figure 3. Visualization of samples in VideoInfer dataset. Some long answers have been truncated to accommodate width constraints. The embedding h are then passed to the SAM2 [49] decoder Ψdec(·) with dense visual feature maps vdense, to produce the final segmentation masks M ∈ R T ×H×W : \mathcal {M} = \Psi _{\text {dec}}(\mathbf {h}, v_{\text {dense}}) This integration allows the model to seamlessly combine language understa… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of RGA3 results on VideoInfer dataset (zoom in for a better view). 28,811 QA pairs, split into two subsets: an instruction tun￾ing set (950 videos with 2,555 objects and 20,320 QA pairs) and a challenging test set (670 videos with 1,572 objects and 8,491 …
Figure 5
Figure 5. Figure 5: Visualization of the RGA3 grounding results on [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Failure case on VideoInfer dataset. The frames with grey [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Visualization of statistics of the test split of VideoInfer. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 47 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 6

    arXiv 2023

  2. [2]

    Burst: A benchmark for unifying ob- ject recognition, segmentation and tracking in video

    Ali Athar, Jonathon Luiten, Paul V oigtlaender, Tarasha Khurana, Achal Dave, Bastian Leibe, and Deva Ramanan. Burst: A benchmark for unifying ob- ject recognition, segmentation and tracking in video. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1674– 1683, 2023. 4

    work page 2023

  3. [3]

    Qwen2.5- vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shi- jie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report...

    arXiv

  4. [4]

    One token to seg them all: Lan- guage instructed reasoning segmentation in videos

    Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Lan- guage instructed reasoning segmentation in videos. Advances in Neural Information Processing Systems , 37:6833–6859, 2024. 2, 7, 8, 1

    work page 2024

  5. [5]

    Xmem++: Production-level video segmentation from few annotated frames

    Maksym Bekuzarov, Ariana Bermudez, Joon-Young Lee, and Hao Li. Xmem++: Production-level video segmentation from few annotated frames. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 635–644, 2023. 8

    work page 2023

  6. [6]

    Coco-stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1209–1218,

    work page

  7. [7]

    Vip-llava: Making large multi- modal models understand arbitrary visual prompts

    Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Vip-llava: Making large multi- modal models understand arbitrary visual prompts. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , pages 12914–12923, 2024. 4, 6, 7, 1

    work page 2024

  8. [8]

    Shikra: Unleash- ing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleash- ing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023. 6

    arXiv 2023

Show all 80 references
  1. [9]

    Grounded multi-hop videoqa in long-form egocentric videos

    Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi-hop videoqa in long-form egocentric videos. arXiv preprint arXiv:2408.14469, 2024. 1

    2024 arXiv

  2. [10]

    Detect what you can: Detecting and representing objects using holistic models and body parts

    Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pa...

    1971

  3. [11]

    How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 6

    2024 arXiv

  4. [12]

    Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model

    Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Con- ference on Computer Vision (ECCV), pages 640–658. Springer, 2022. 8

    2022

  5. [13]

    Videol- lama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videol- lama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 1

    2024 arXiv

  6. [14]

    Grounded question- answering in long egocentric videos

    Shangzhe Di and Weidi Xie. Grounded question- answering in long egocentric videos. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) , pages 12934– 12943, 2024. 1, 6

    2024

  7. [15]

    Mevis: A large-scale bench- mark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale bench- mark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV) , pages 2694– 2703, 2023. 7, 8, 1

    2023

  8. [16]

    Mose: A new dataset for video object segmentation in complex scenes

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV) , pages 20224– 20234, 2023. 4

    2023

  9. [17]

    Lvos: A benchmark for large-scale long-term video object seg- mentation

    Lingyi Hong, Zhongying Liu, Wenchao Chen, Chen- zhi Tan, Yuang Feng, Xinyu Zhou, Pinxue Guo, Jinglun Li, Zhaoyu Chen, Shuyong Gao, et al. Lvos: A benchmark for large-scale long-term video object seg- mentation. arXiv preprint arXiv:2404.19326, 2024. 4

    2024 arXiv

  10. [18]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 ,

    arXiv

  11. [19]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Rad- ford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 6

    2024 arXiv

  12. [20]

    Cotracker3: Simpler and better point tracking by pseudo-labelling real videos

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Na- talia Neverova, Andrea Vedaldi, and Christian Rup- precht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), 2025. 3

    2025

  13. [21]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP) , pages 787–798,

    2014

  14. [22]

    Video object segmentation with language referring ex- pressions

    Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring ex- pressions. In Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Aus- tralia, December 2–6, 2018, Revised Selected Papers, Part IV 14, pages 123–141. ...

    2018

  15. [23]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 4015–4026, 2023. 8

    2023

  16. [24]

    Grounding language models to images for multi- modal inputs and outputs

    Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multi- modal inputs and outputs. InInternational Conference on Machine Learning , pages 17283–17300. PMLR,

  17. [25]

    Generating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems , 36, 2024. 2

    2024

  18. [26]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9579–9589,

  19. [27]

    Mimic-it: Multi-modal in-context instruction tun- ing

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tun- ing. arXiv preprint arXiv:2306.05425, 2023. 2

    2023 arXiv

  20. [28]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326,

    arXiv

  21. [29]

    Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 2

    2024 arXiv

  22. [30]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 22195...

    2024

  23. [31]

    Towards robust referring video object segmentation with cyclic relational consensus

    Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhik- sha Raj, and Yan Lu. Towards robust referring video object segmentation with cyclic relational consensus. arXiv preprint arXiv:2207.01203, 2022. 6

    2022 arXiv

  24. [32]

    Describe anything: Detailed localized image and video captioning

    Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, and Yin Cui. Describe anything: Detailed localized image and video captioning. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (I...

    2025

  25. [33]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 2

    2023 arXiv

  26. [34]

    Rouge: A package for automatic eval- uation of summaries

    Chin-Yew Lin. Rouge: A package for automatic eval- uation of summaries. In Text Summarization Branches Out, pages 74–81, 2004. 6

    2004

  27. [35]

    Glus: Global-local reasoning unified into a single large language model for video segmentation

    Lang Lin, Xueyang Yu, Ziqi Pang, and Yu-Xiong Wang. Glus: Global-local reasoning unified into a single large language model for video segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2

    2025

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems , 36, 2024. 2, 1

    2024

  29. [37]

    Lamra: Large multimodal model as your ad- vanced retrieval assistant

    Yikun Liu, Pingan Chen, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your ad- vanced retrieval assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2025. 2

    2025

  30. [38]

    Decou- pled weight decay eegularization

    Ilya Loshchilov and Frank Hutter. Decou- pled weight decay eegularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  31. [39]

    Video-chatgpt: Towards detailed video understanding via large vision and lan- guage models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and lan- guage models. arXiv preprint arXiv:2306.05424 ,

    arXiv

  32. [40]

    Generation and comprehension of unambiguous ob- ject descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous ob- ject descriptions. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 11–20, 2016. 7, 1, 2

    2016

  33. [41]

    Large-scale video panoptic segmentation in the wild: A benchmark

    Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) , pages 21033–21043, 2022. 4

    2022

  34. [42]

    V-net: Fully convolutional neural networks for volumetric medical image segmentation

    Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 Fourth International Conference on 3D Vision, pages 565–571. IEEE, 2016. 5

    2016

  35. [43]

    Bleu: A method for automatic eval- uation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computa- tional Linguistics, pages 311–318, 2002. 6

    2002

  36. [44]

    Perception test: A diagnostic bench- mark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic bench- mark for multimodal video models. Advances in Neu- ral Information Processing ...

  37. [45]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 6

    2023 arXiv

  38. [46]

    Occluded video in- stance segmentation: A benchmark

    Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xi- aoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video in- stance segmentation: A benchmark. International Journal of Computer Vision, 130(8):2022–2039, 2022. 4

    2022

  39. [47]

    Artemis: Towards referential un- derstanding in complex videos

    Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tian- ren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Yunjie Tian. Artemis: Towards referential un- derstanding in complex videos. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1

    2024

  40. [48]

    Paco: Parts and attributes of common objects

    Vignesh Ramanathan, Anmol Kalia, Vladan Petro- vic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recogn...

    2023

  41. [49]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R ¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 , 2024. 2, 3, 4, 6

    2024 arXiv

  42. [50]

    Hiera: A hierarchical vision transformer without the bells-and-whistles

    Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoff- man, et al. Hiera: A hierarchical vision transformer without the bells-and-whistles. In International Con- ference on Machine Lear...

    2023

  43. [51]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In European Conference on Computer Vision (ECCV), pages 208–

  44. [52]

    Gen- erative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Gen- erative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023. 2

    2023 arXiv

  45. [53]

    Cider: Consensus-based image descrip- tion evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image descrip- tion evaluation. In Proceedings of the IEEE con- ference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2015. 6

    2015

  46. [54]

    Ov-vis: Open-vocabulary video instance segmentation

    Haochen Wang, Cilin Yan, Keyan Chen, Xiaolong Jiang, Xu Tang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Ov-vis: Open-vocabulary video instance segmentation. International Journal of Computer Vision, pages 1–18, 2024. 4

    2024

  47. [55]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 ,

    arXiv

  48. [56]

    Unidentified video objects: A benchmark for dense, open-world segmentation

    Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10776–10785, 2021. 4

    2021

  49. [57]

    Internvideo: General video foundation models via generative and discrim- inative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discrim- inative learning. arXiv preprint arXiv:2212.03191 ,

    arXiv

  50. [58]

    Internvideo2: Scaling foun- dation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yi- nan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foun- dation models for multimodal video understanding. In European Conference on Computer Vision (ECCV) , pages 396–416. Springer, 2024. 2

    2024

  51. [59]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777–9786, 2021. 1

    2021

  52. [60]

    Visa: Reasoning video object segmen- tation via large language models

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efs- tratios Gavves. Visa: Reasoning video object segmen- tation via large language models. In European Con- ference on Computer Vision (ECCV) , pages 98–115. Springer, 2024. 2, 6, 7, 8, 1

    2024

  53. [61]

    Videogpt: Video generation using vq-vae and transformers

    Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021. 1, 2

    2021 arXiv

  54. [62]

    Set-of-mark prompt- ing unleashes extraordinary visual grounding in gpt- 4v

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompt- ing unleashes extraordinary visual grounding in gpt- 4v. arXiv preprint arXiv:2310.11441, 2023. 6

    2023 arXiv

  55. [63]

    Lisa++: An im- proved baseline for reasoning segmentation with large language model

    Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. Lisa++: An im- proved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240 ,

    arXiv

  56. [64]

    mplug-owl: Modular- ization empowers large language models with multi- modality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modular- ization empowers large language models with multi- modality. arXiv preprint arXiv:2304.14178, 2023. 2

    2023 arXiv

  57. [65]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 2, 6

    2023 arXiv

  58. [66]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Con- ference on Artificial Intelligence , pages 9127–9134,

  59. [67]

    Sa2va: Marrying sam2 with llava for dense grounded understanding of im- ages and videos

    Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of im- ages and videos. arXiv, 2025. 2

    2025

  60. [68]

    Os- prey: Pixel understanding with visual instruction tun- ing

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Os- prey: Pixel understanding with visual instruction tun- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28202–28211, 2024. 2, 6, 1

    2024

  61. [69]

    Videorefer suite: Advancing spatial-temporal object understanding with video llm

    Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wen- qiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video llm. In Proceedings of the IEEE/CVF Con- ference on Computer Vision...

    2025

  62. [70]

    Vide- ollama 3: Frontier multimodal foundation models for image and video understanding

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Vide- ollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025. 6

    2025 arXiv

  63. [71]

    Video- llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video- llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023. 1

    2023 arXiv

  64. [72]

    Lmms-eval: Reality check on the evalua- tion of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evalua- tion of large multimodal models. arXiv preprint arXiv:2407.12772, 2024. 1

    2024 arXiv

  65. [73]

    Gpt4roi: Instruction tuning large lan- guage model on region-of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large lan- guage model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023. 2, 6

    2023 arXiv

  66. [74]

    Video in- struction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Ze- jun Ma, Ziwei Liu, and Chunyuan Li. Video in- struction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 2, 1

    2024 arXiv

  67. [75]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages 633–641, 2017. 1

    2017

  68. [76]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2 Object-centric Video Question Answering with Visual Grounding and Referring Su...

    2023 arXiv

  69. [78]

    The complete list of used datasets in training is presented in Tab

    More Implementation Details Training Data Composition. The complete list of used datasets in training is presented in Tab. 9. For the datasets with overlapping, we select the disjoint samples during training, such as ViP-LLaV A-Instruct and LLaV A-150k. Task Datasets # Samples...

    2024

  70. [79]

    More Experimental Results 7.1. Quantitative Results Method Perception Test [44]MVBench[30]NExT-QA[59] Generalist Models LLaV A-OV-7B [28] - 56.7 79.4 VideoLLaMA2.1-7B [13] 54.9 57.3 75.6 LLaV A-Video-7B [74] 67.9 58.6 83.2 Specialist Models Artemis [47] 47.1 34.1 - VideoRefer-...

  71. [80]

    Discussion and Visualizations 8.1. Potential Information Loss The STOM module blends prompts onto original frames with transparency through alpha blending, so that the ob- jects will not be completely occluded, and the features can be reserved. [0, 5) [5, 10) [10, 15) [15, 20)...

    2000

  72. [223]

    Springer, 2020. 7, 8, 1

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.