REVIEW 4 major objections 4 minor 32 references
InterRVOS: Interaction-aware Referring Video Object Segmentation
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that referring video object segmentation should segment both the actor and the target of an interaction separately, and that the proposed ReVIOSa model with the InterRVOS-127K dataset achieves state-of-the-art results on…
desk verdict A new task and dataset for role-aware RVOS that deserve referee time; the method is incremental and the main risks are annotation validation and benchmark tuning, not the core idea. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core mechanism is a pair of interaction-aware special tokens, [SEG_ACT] and [SEG_TAR], generated by a multimodal large language model instead of a single [SEG] token. Each token's hidden state is projected into the prompt space of the SAM2 video segmentation decoder, producing separate masks. An attention mask loss supervises the self-attention maps of these tokens against the ground-truth actor and target masks, making the MLLM attend to the correct object regions; the paper selects the layer-head pairs with the strongest vision attention for this supervision. The supporting data machinery is a four-stage automatic annotation pipeline that uses SAM2 mask tracks, GPT-4o interaction and role detection, and LLaMA expression generation to produce actor-target paired expressions from VidOR videos.
What would settle it
Have human annotators verify a random sample of the automatically generated training pairs: for each expression, does the stored actor mask track the object that performs the action and does the target mask track the object acted upon? If a substantial fraction of samples are mislabeled, the reported gains on the evaluation set could come from learning annotation patterns rather than interaction semantics.
Extended reading notes
Core claim
The central claim is that interaction-aware referring video object segmentation — requiring distinct masks for the actor and the target — is a learnable and useful task, and that the proposed formulation improves fine-grained understanding of object relationships. Concretely, the paper claims ReVIOSa achieves state-of-the-art results on its own InterRVOS-127K evaluation set, with the 4B model reaching 74.5 J&F for actor segmentation and 68.3 J&F for target segmentation, and that explicitly modeling actor and target roles also improves actor localization itself. The paper further claims that its automatically constructed dataset generalizes to standard RVOS benchmarks, surpassing a strong baseline in zero-shot transfer despite using fewer training samples.
Load-bearing premise
The load-bearing premise is that the automatic annotation pipeline's actor/target role assignments and mask tracks are correct enough that the model's role-specific training signal reflects real interaction structure, even though the training set was not human-verified.
Editorial extensions
If this is right
- Models trained on InterRVOS-127K can be evaluated for actor and target segmentation separately, giving a more direct measure of whether a model understands role asymmetry in interactions.
- An MLLM that outputs two segmentation tokens in one forward pass can jointly localize subject and object for expressions like "person walking a dog," which standard RVOS cannot do.
- If the role-separated training signal is the cause of the reported gains, then adding interaction-aware annotations to existing RVOS training sets should improve actor-only segmentation on standard benchmarks as well.
- The proposed evaluation protocol could be applied to other interaction-centric video tasks to score actor and target independence rather than a single merged mask.
Reading between the lines
- Editorial inference: if the role-separated training signal is what drives the reported gains, then the same two-token recipe should improve actor-only segmentation even when trained on a dataset without target masks, because the model is forced to distinguish the actor from everything else.
- Editorial inference: the evaluation protocol could be reused by any model that outputs two masks, including non-MLLM baselines, making it a task definition rather than a method-specific benchmark; the paper does not explore that route.
- Editorial inference: a sharper test of interaction understanding would be to hold out interaction types (verb–object combinations) and measure whether the model can segment novel actor-target pairs, which the current random split does not guarantee.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new task, InterRVOS, in which a referring expression describing an interaction must be segmented into two masks: the actor and the target. To support this, the authors build InterRVOS-127K, a 127K-expression dataset generated automatically from VidOR with a four-stage pipeline (SAM2 tracking, GPT-4o interaction detection/role assignment, LLaMA expression generation), with a 5,048-expression evaluation set refined by human annotators. They also propose ReVIOSa, an MLLM-based architecture that emits [SEG_ACT] and [SEG_TAR] tokens and uses an attention mask loss (AML) to supervise selected attention heads with ground-truth masks. Experiments report state-of-the-art results on the proposed evaluation set and competitive results on standard RVOS benchmarks.
Significance. If the dataset and results are validated, the paper makes a useful contribution: it defines a role-aware RVOS task, provides the first large-scale actor-target interaction dataset, and shows consistent internal gains from both special tokens and AML (+2.6 and +1.5 J&F respectively in Table 4). The external zero-shot results in Appendix C.3 (Table A3) provide independent evidence that the automatically generated data transfers to standard benchmarks. However, the central claims are conditional on the correctness of the automatically assigned roles and on clean evaluation comparisons, both of which need additional support.
major comments (4)
- [Training and evaluation set; Appendix D.1] The paper states that the 5,048-expression evaluation set was human-refined, but gives no human verification, inter-annotator agreement, or error statistics for the 122,188 training expressions. Since every training and ablation result (Tables 2 and 4) depends on the correctness of the SAM2 tracks, GPT-4o interaction detection, and LLaMA role assignment, unverified noise in these labels could drive the reported gains. Please provide a human-validated sample with per-stage error rates, or an automated consistency check between the generated role labels and the masks.
- [Appendix D.2 (Figure A7)] The dataset examples contain clear role-assignment errors. For instance, 'Car being discussed by people' is labeled Actor [3] / Target [1],[2], although in the passive construction the people are the actors and the car is the target. This suggests that the pipeline can systematically misassign roles in passive expressions, and because the training set is not human-verified, the model may be trained on swapped actor-target masks. Please quantify how often such errors occur and how they affect the role-specific metrics.
- [Table 2] The InterRVOS-Target columns contain no baseline results; the text says prior methods are not applicable. The abstract and conclusion claim state-of-the-art performance on InterRVOS-127K, but for the target setting this claim is an absolute number on a self-created benchmark with no comparison point. Please adapt existing RVOS methods to the target setting (e.g., by prompting them with target-oriented expressions or providing an oracle upper bound), or consistently qualify the SOTA claim to the actor and standard RVOS columns.
- [Layer-head selection for AML (Table 3, Figure 6)] The top-4 heads for AML are selected by inspecting attention magnitudes and by the J&F numbers reported on what appears to be the InterRVOS-127K evaluation set. The paper does not state that this selection was made on a held-out validation split. If the evaluation set was used for hyperparameter selection, the reported +5.0 J&F gain of the full model (Table 4) may include test-set peeking. Please clarify the split used for head selection or re-select heads on a validation set and report the corresponding evaluation numbers.
minor comments (4)
- [Throughout] There are numerous typographical and formatting artifacts, including 'suchinteraction' in the introduction and inconsistent spacing in 'LLaV A', 'Sa2V A', and 'Ref-DA VIS' vs. 'Ref-DAVIS'; these should be cleaned before publication.
- [Appendix C.1 (Table A1)] The standard-RVOS results are obtained with a variant that uses only AML, because the interaction-aware tokens are said to be incompatible with the standard RVOS setting. The abstract's claim of strong performance on standard RVOS benchmarks should state this caveat explicitly, since the full ReVIOSa model is not the one evaluated there.
- [Tables A2 and A3] The dataset-comparison experiments train Sa2VA on different data sources; please state explicitly that the training recipe, number of iterations, and evaluation protocol were identical across datasets, so that the comparisons isolate the effect of the training data.
- [Table 4] The baseline row (i) is described only as 'baseline'; please specify whether it uses the same InternVL-2.5 backbone, LoRA configuration, and SAM2 decoder as ReVIOSa-1B, so that the ablation isolates the proposed components.
Circularity Check
No significant circularity: the claims are supervised empirical results, not derivations equivalent to their inputs.
full rationale
This paper is an empirical systems-and-benchmark contribution; there is no derivation chain in which a quantity is defined in terms of the quantity it is claimed to predict. The actor/target masks are produced by an automatic pipeline (SAM2 tracks, GPT-4o role assignment, LLaMA expression generation), and the evaluation set is human-refined; the model is then trained with standard segmentation losses plus an attention mask loss that supervises attention maps with the same ground-truth masks. This is direct supervision, not a fitted parameter renamed as a prediction. The [SEG_ACT]/[SEG_TAR] tokens, AML, and losses (Eqs. 6-8) do not define the evaluation metric in terms of themselves; the metric is J&F against held-out human-refined masks. The main benchmark is the authors' own InterRVOS-127K, and the InterRVOS-Target columns in Table 2 contain no baseline numbers because prior methods are inapplicable; that is a benchmark-validity limitation, not circularity. External benchmarks (MeViS, Ref-Youtube-VOS, Ref-DAVIS) provide independent grounding. The appendices' unvalidated auto-annotation is a data-quality risk but does not make any claim equivalent to its input by construction. No self-citation chain or imported uniqueness theorem is load-bearing; the paper cites SAM2, LLaVA/GPT-4o, and prior RVOS methods for components, and none of those citations carry the target claim. Consequently no circular step is identified.
Assumptions & free parameters
free parameters (2)
- lambda_AML and lambda_text loss weights
- AML layer-head selection =
Layer 22 (1B), Layer 33 (4B); top-4 heads
assumptions (4)
- domain assumption SAM2 mask tracks accurately delineate every object involved in interactions.
- domain assumption GPT-4o can reliably detect interactions and assign actor/target roles from highlighted video frames.
- domain assumption LLaMA-generated referring expressions preserve actor-target role assignment.
- domain assumption MLLM attention maps from selected layers/heads correlate with object masks and are a valid supervision target.
Cite this review
Pith. "Pith review of InterRVOS: Interaction-aware Referring Video Object Segmentation." pith.science (2026). https://pith.science/paper/2ZCSNOQ5
@misc{pith2026250602356,
author = {Pith},
title = {Pith review of: InterRVOS: Interaction-aware Referring Video Object Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/2ZCSNOQ5}},
note = {Machine review of arXiv:2506.02356}
}
read the original abstract
Referring video object segmentation (RVOS) aims to segment objects in a video described by a natural language expression. However, most existing approaches focus on segmenting only the referred object (typically the actor), even when the expression clearly describes an interaction involving multiple objects with distinct roles. For instance, "A throwing B" implies a directional interaction, but standard RVOS segments only the actor (A), neglecting other involved target objects (B). In this paper, we introduce Interaction-aware Referring Video Object Segmentation (InterRVOS), a novel task that focuses on the modeling of interactions. It requires the model to segment the actor and target objects separately, reflecting their asymmetric roles in an interaction. This task formulation enables fine-grained understanding of object relationships, as many video events are defined by such relationships rather than individual objects. To support this task, we propose a new evaluation protocol that separately evaluates actor and target segmentation, enabling more accurate assessment of the model's ability to distinguish and segment actor and target roles. We also present InterRVOS-127K, a large-scale dataset with over 127K automatically annotated expressions, including interaction expressions annotated with distinct masks for actor and target objects. Furthermore, we develop ReVIOSa, an MLLM-based architecture that introduces interaction-aware special tokens and leverages an attention mask loss to enhance role-specific segmentation. Extensive experiments show that ReVIOSa not only outperforms existing baselines on our proposed InterRVOS-127K evaluation set, but also achieves strong performance on standard RVOS benchmarks. Our project page is available at: https://cvlab-kaist.github.io/InterRVOS.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[3]
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024
arXiv 2024
-
[1]
One token to seg them all: Language instructed reasoning segmentation in videos
Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language instructed reasoning segmentation in videos. Advances in Neural Information Processing Systems, 37: 0 6833--6859, 2024
work page 2024
-
[2]
End-to-end referring video object segmentation with multimodal transformers
Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4985--4995, 2022
work page 2022
-
[4]
Vision-language transformer and query generation for referring segmentation
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 16321--16330, October 2021
work page 2021
-
[5]
Mevis: A large-scale benchmark for video segmentation with motion expressions
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 2694--2703, 2023
work page 2023
-
[6]
Moma: A multi-object multi-action dataset for understanding human activities
Qi Fan, Yanghao Liu, Wei Wang, Ning Xu, Dahua Lin, Alan Yuille, and Chen Change Loy. Moma: A multi-object multi-action dataset for understanding human activities. In Advances in Neural Information Processing Systems (NeurIPS), 2021
work page 2021
-
[7]
Actor and action video segmentation from a sentence
Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. Actor and action video segmentation from a sentence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 5958--5966, 2018
work page 2018
-
[8]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv e-prints, pp.\ arXiv--2407, 2024
work page 2024
Show all 32 references
-
[9]
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 0 (2): 0 3, 2022
2022
-
[10]
Gpt-4o system card
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024
2024 arXiv
-
[11]
Action genome: Actions as compositions of spatiotemporal scene graphs
Mengmeng Ji, Jingwei Wang, Xing Xu, Siyuan Qi, Yixin Zhu, and Song-Chun Zhu. Action genome: Actions as compositions of spatiotemporal scene graphs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 10236--10247, 2020
2020
-
[12]
Video object segmentation with language referring expressions
Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In Computer Vision--ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2--6, 2018, Revised Selected Papers, Part IV 14, pp.\ 123--141. Sp...
2018
-
[13]
Videoglamm: Video grounded language-image pretraining with masked modeling
Hao Liu, Jiapeng Yang, Mingyu Wang, and et al. Videoglamm: Video grounded language-image pretraining with masked modeling. arXiv preprint arXiv:2411.04923, 2024
2024 arXiv
-
[14]
Visual instruction tuning
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023
2023
-
[15]
Spectrum-guided multi-granularity referring video object segmentation
Bo Miao, Mohammed Bennamoun, Yongsheng Gao, and Ajmal Mian. Spectrum-guided multi-granularity referring video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 920--930, 2023
2023
-
[16]
Refer-youtube-vos: A dataset for video object segmentation with language referring expressions
Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Refer-youtube-vos: A dataset for video object segmentation with language referring expressions. In ECCV, 2020
2020
-
[17]
Sam 2: Segment anything in images and videos
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R \"a dle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024
2024 arXiv
-
[18]
Urvos: Unified referring video object segmentation network with a large-scale benchmark
Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XV 16, pp.\ 208--223. Springer, 2020
2020
-
[19]
Video relationship detection
Xindi Shang, Linchao Li, Yao Zhang, Tao Jiang, Xin Yang, and Zhiwu Chen. Video relationship detection. In ACM International Conference on Multimedia (ACM MM), pp.\ 1074--1082, 2017
2017
-
[20]
Annotating objects and relations in user-generated videos
Xindi Shang, Xiyang Ren, Linchao Li, Zhiwu Chen, Yu-Gang Liu, and Yu Zhou. Annotating objects and relations in user-generated videos. In ACM International Conference on Multimedia (ACM MM), pp.\ 1308--1316, 2019
2019
-
[21]
Yfcc100m: The new data in multimedia research
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59 0 (2): 0 64--73, 2016
2016
-
[22]
Soc: Segmenting objects by categories for referring video object segmentation
Jianhao Wang, Feng Gao, Jianke Zhu, and Qionghai Dai. Soc: Segmenting objects by categories for referring video object segmentation. arXiv preprint arXiv:2305.17011, 2023
2023 arXiv
-
[23]
Villa: Unifying vision-language segmentation tasks with large multi-modal models
Renrui Wang, Ze Ma, Xiang Li, and et al. Villa: Unifying vision-language segmentation tasks with large multi-modal models. arXiv preprint arXiv:2407.14500, 2024
2024 arXiv
-
[24]
Language as queries for referring video object segmentation
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 4964--4974. IEEE, 2022
2022
-
[25]
Star: Structured action understanding in instructional videos
Yifei Wu, Guojun Yu, Wenqi Jia, Qi Jin, Jie Zhou, and Yu Qiao. Star: Structured action understanding in instructional videos. arXiv preprint arXiv:2405.09711, 2024
2024 arXiv
-
[26]
Visa: Reasoning video object segmentation via large language models
Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. In European Conference on Computer Vision, pp.\ 98--115. Springer, 2024
2024
-
[27]
Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos
Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001, 2025
2025 arXiv
-
[28]
Dshmp: Densely supervised hierarchical mask propagation for referring video object segmentation
Jianyuan Zhou, Jing Gao, Hui Zeng, and Huchuan Lu. Dshmp: Densely supervised hierarchical mask propagation for referring video object segmentation. arXiv preprint arXiv:2404.03645, 2024
2024 arXiv
-
[29]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
-
[30]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[31]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[32]
A child helping another child with a backpack
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.