REVIEW 4 major objections 6 minor 29 references
R2SM: Referring and Reasoning for Selective Masks
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper introduces R2SM, a benchmark that pairs modal and amodal text queries with masks, and claims that fine-tuning on it lets vision-language models learn to choose the mask type a prompt asks for.
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the R2SM dataset, composed of roughly 74,000 amodal and 73,000 modal text queries paired with masks across three source datasets. The mechanism that carries the argument is the query-generation pipeline: structured instance annotations, including an occlusion flag, are fed to a large language model that is instructed to produce diverse referring prompts and, for each occluded instance, an amodal variant that explicitly mentions the occlusion and asks for the complete or inferred shape. That explicit pairing of cue to mask type is what makes it possible to fine-tune a model on the mask-selection decision.
What would settle it
An experiment would be to evaluate a fine-tuned model on paraphrased amodal queries that ask for the complete object without using any explicit occlusion keyword; if performance drops to the level of the unfine-tuned baseline, the dataset has taught keyword detection rather than intent-based mask selection.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a dataset pairing modal and amodal text queries with matching ground-truth masks lets existing vision-language models learn to select mask type from language, and that this learning transfers across real and synthetic occlusion scenes. The strongest evidence is Table 2, where fine-tuning on R2SM improves gIoU and cIoU for LISA, PSALM, ReLA, GLaMM, and PixelLM on every split; for example, ReLA goes from 30.65 to 83.18 gIoU on D2SA and from 24.88 to 71.78 on MUVA. The authors argue no prior benchmark offered this paired supervision, so models were limited to fixed mask types. They also report that although fused gIoU/cIoU rise, per-instance mask quality can decline under amodal fine-tuning, which they interpret as evidence that instance-level metrics are needed.
Load-bearing premise
The load-bearing premise is that each generated text query's intended mask type is faithfully captured by the prompt template, so the benchmark measures real user intent rather than a model's ability to spot formulaic phrases such as 'occluded' or 'hidden behind'.
Editorial extensions
If this is right
- Fine-tuning on R2SM raises both gIoU and cIoU for every benchmarked model and every split, so mask-type selection appears learnable from paired supervision rather than requiring a new architecture.
- The benchmark's All and Amodal evaluation settings allow the mask-selection decision and the amodal-completion skill to be measured separately.
- Larger gains on D2SA and MUVA than on COCOA-cls suggest that the semi-synthetic and synthetic splits offer denser or more consistent occlusion annotations for learning amodal shape completion.
- Because cIoU and gIoU diverge across splits, conclusions about model quality depend on which metric is used; reporting both is necessary.
- The AP/AR analysis indicates that future work on this task should evaluate per-instance predictions, not only fused masks.
Reading between the lines
- My editorial inference: the generation pipeline's instruction to mention occlusion explicitly means the amodal queries may be recognizable by keywords such as 'occluded' or 'hidden behind'; testing with paraphrased prompts that express the same intent without those words would show whether models learn true intent or phrase matching.
- My editorial inference: the same text-query generation recipe could be applied to other amodal datasets such as KINS, WALT, or MP3D-Amodal, which would test whether the reported gains generalize beyond the three chosen source datasets.
- My editorial inference: the observed trade-off between sharper modal boundaries and coarser amodal predictions suggests a promising next step in which a model predicts both masks and lets the prompt, or an uncertainty signal, arbitrate between them.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces R2SM (Referring and Reasoning for Selective Masks), a task and benchmark that extends text-guided segmentation by requiring a model to choose between a modal mask (visible region) and an amodal mask (complete region, including occluded parts) based solely on a natural language prompt. The dataset is built by augmenting three existing amodal segmentation datasets (COCOA-cls, D2SA, MUVA) with Claude-generated text queries, roughly half modal and half amodal, each paired with the corresponding ground-truth mask. The authors fine-tune five representative vision-language models (LISA, PSALM, ReLA, GLaMM, PixelLM) on each split and report consistent gIoU/cIoU improvements over the un-fine-tuned baselines. They also analyze a metric mismatch between fused gIoU/cIoU and instance-level AP/AR, and provide qualitative evidence of remaining challenges in modal/amodal boundary selection and occlusion reasoning.
Significance. If the central claim were fully established, R2SM would be a useful new benchmark for intent-aware, occlusion-sensitive segmentation, and the observation that gIoU/cIoU can mask degraded per-instance predictions is a valuable methodological point. The dataset release, the breadth of models benchmarked, and the careful reporting of fine-tuning setups are concrete strengths. However, the paper's core claim that the benchmark measures 'inferring user intent' or 'reasoning' is not yet supported by the presented evidence. The query-generation pipeline in Section 4.2 introduces a strong lexical confound: amodal queries are explicitly instructed to mention occlusion and request the complete outline, while modal queries are not. This makes mask-type selection solvable by keyword spotting, and the reported fine-tuning gains are consistent with learning that surface mapping. In addition, the absence of a control fine-tuning condition on class-labeled amodal data means the improvements in Table 2 could stem from additional segmentation fine-tuning data rather than text-conditioned mask-type selection.
major comments (4)
- [Section 4.2, Figures 9-14] The query-generation pipeline deterministically ties mask type to explicit lexical cues: for every instance marked is_occluded, Claude is instructed to generate an amodal query that 'explicitly mentioning that the instance is occluded and requesting the complete or inferred outline of the hidden parts,' while modal queries are generated without that instruction. This means the label 'amodal' is a consequence of prompt construction, not of intrinsic user intent. A model could select the mask type by detecting words such as 'occluded,' 'hidden,' 'complete,' 'whole,' or 'behind' without performing any visual or commonsense reasoning about occlusion, so the task as currently formulated does not establish the 'inferring user intent' claim made in Sections 1.3 and 3.3. I would need to see a deconfounded version of the benchmark: amodal queries without explicit occlusion wording, modal queries that mention occlusion but ask for the visible part only, and an analysis of model accuracy on the subset of prompts that lack the cue lexicon. A human-validated intent split would also strengthen the claim.
- [Section 5.3, Table 2] The headline result that 'fine-tuning on R2SM invariably improves performance across all models and dataset splits' is missing a critical control condition. There is no experiment that fine-tunes the same models on an equal number of text queries generated from the same underlying amodal annotations but without the modal/amodal text conditioning, for example using class-label prompts or generic template prompts. Without such a control, the large gIoU gains in Table 2 (e.g., ReLA from 30.65 to 71.78 on MUVA All) could reflect the benefit of additional segmentation fine-tuning data rather than learning to condition mask type on language. Please add control experiments, or explicitly weaken the claim to state that fine-tuning with additional paired text-mask data improves segmentation quality on these splits.
- [Section 6.3, Table 3] The text states that 'although gIoU and cIoU reflect improved performance after fine-tuning when the evaluation is restricted to amodal queries, AP and AR exhibit a decline under the same condition.' This is not what Table 3 shows: the table compares the All and Amodal rows for the R2SM-fine-tuned PSALM only, and no baseline AP/AR values are reported. The supported claim is that gIoU/cIoU and AP/AR rank these two evaluation subsets differently, not that fine-tuning causes an AP/AR decline. Additionally, the AP/AR protocol matches predictions to ground-truth masks by query ID, but the handling of queries that refer to multiple instances is not described; please clarify the matching procedure and, if feasible, report baseline AP/AR values for a proper before/after comparison.
- [Section 4.2 and Section 5.3] The training and evaluation queries are both generated by the same Claude pipeline with the same instruction templates, so the benchmark may measure a model's fit to a particular LLM prompt style rather than general user intent. No human evaluation, distribution-shift test, or cross-generation generalization experiment is reported. I would like to see either a small human study that validates the naturalness and intent clarity of the generated queries, or an out-of-distribution evaluation using prompts written by humans or generated with a different instruction set, to support the claim that the benchmark measures intent understanding rather than style overfitting.
minor comments (6)
- [Abstract] The abstract says 'whether to generate amodal(visible) or amodal(complete) segmentation mask'; the first occurrence should be 'modal (visible)'.
- [Section 4.3, Table 1] The D2SA split has 3,600 validation images but only 2,000 training images; please clarify whether this follows the original dataset's split or a deliberate choice, since an inverted train/val ratio is unusual.
- [Figure 6 caption] The caption refers to the 'COCOA split' while the paper consistently uses 'COCOA-cls'; please unify the nomenclature.
- [Table 3] The '-' entries for APs and APm should be explained (for example, 'no sufficiently small objects exist in this split'), and the object-size thresholds should be defined in a footnote.
- [Section 5.2] The training setup says 'one epoch corresponds to a full pass through all queries in the respective split,' but the Appendix also mentions that PSALM's best checkpoint is epoch 10 for COCOA-cls and epoch 9 for D2SA/MUVA; please clarify whether the epoch count is measured over queries or over images, and whether hyperparameters were tuned per model or fixed from the official repositories.
- [Section 4.2] The text says each image is annotated with 'up to 10 diverse text queries,' but no distribution of queries per image is reported; please add this statistic.
Circularity Check
No significant circularity: the benchmark results are empirical measurements on held-out splits, and the lexical-cue confound is a validity concern rather than a derivation-level circularity.
full rationale
The paper does not contain a derivation chain in the sense of equations or first-principles results being shown equivalent to their inputs. Its central claims are empirical: fine-tuning on R2SM improves gIoU/cIoU across models and splits (Section 5.3, Table 2). These measurements are made on held-out validation splits, so the improvements are not fitted parameters renamed as predictions. The only mild concern is the query-generation pipeline (Section 4.2): amodal queries are generated by instructing Claude to explicitly mention occlusion and request the complete outline of hidden parts, which could allow a model to solve mask-type selection via lexical cues. However, this is a benchmark construct-validity threat, not an internal circularity of the paper's argument; the paper is transparent about the generation protocol, and its own AP/AR analysis (Section 6.3) reports declines in instance-level quality despite gIoU gains, which would not be expected if the benchmark were trivially solvable. The self-citations (Refs 4 and 14) are not load-bearing: Amodal-LVIS is mentioned in related work but is not one of the three source datasets used to construct R2SM, and no uniqueness theorem or ansatz is imported from prior work. The empirical results are self-contained against the constructed benchmark and do not reduce by construction to the pipeline inputs.
Assumptions & free parameters
assumptions (3)
- ad hoc to paper Amodal text queries must explicitly mention occlusion and request the complete or inferred outline of hidden parts.
- domain assumption The amodal ground-truth masks from COCOA-cls, D2SA, and MUVA are correct and are inherited without further validation.
- domain assumption gIoU and cIoU computed over fused masks adequately measure the ability to select the requested mask type.
Cite this review
Pith. "Pith review of R2SM: Referring and Reasoning for Selective Masks." pith.science (2026). https://pith.science/paper/AEGHTH54
@misc{pith2026250601795,
author = {Pith},
title = {Pith review of: R2SM: Referring and Reasoning for Selective Masks},
year = {2026},
howpublished = {\url{https://pith.science/paper/AEGHTH54}},
note = {Machine review of arXiv:2506.01795}
}
read the original abstract
We introduce a new task, Referring and Reasoning for Selective Masks (R2SM), which extends text-guided segmentation by incorporating mask-type selection driven by user intent. This task challenges vision-language models to determine whether to generate a modal (visible) or amodal (complete) segmentation mask based solely on natural language prompts. To support the R2SM task, we present the R2SM dataset, constructed by augmenting annotations of COCOA-cls, D2SA, and MUVA. The R2SM dataset consists of both modal and amodal text queries, each paired with the corresponding ground-truth mask, enabling model finetuning and evaluation for the ability to segment images as per user intent. Specifically, the task requires the model to interpret whether a given prompt refers to only the visible part of an object or to its complete shape, including occluded regions, and then produce the appropriate segmentation. For example, if a prompt explicitly requests the whole shape of a partially hidden object, the model is expected to output an amodal mask that completes the occluded parts. In contrast, prompts without explicit mention of hidden regions should generate standard modal masks. The R2SM benchmark provides a challenging and insightful testbed for advancing research in multimodal reasoning and intent-aware segmentation.
Figures
Figures from the paper (11 more)
Reference graph
Works this paper leans on
-
[1]
Unveiling the Invisible: Reasoning Complex Occlusions Amodally with AURA
Zhixuan Li, Hyunse Yoon, Sanghoon Lee, and Weisi Lin. Unveiling the invisible: Reasoning complex occlusions amodally with AURA.CoRR, abs/2503.10225, 2025
work page Pith review arXiv 2025
-
[2]
Learning to see the invisible: End-to-end trainable amodal instance segmentation
Patrick Follmann, Rebecca König, Philipp Härtinger, Michael Klostermann, and Tobias Böttger. Learning to see the invisible: End-to-end trainable amodal instance segmentation. InIEEE Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, HI, USA, January 7-11, 2019, pages 1328–1336. IEEE, 2019
work page 2019
-
[3]
Zhixuan Li, Weining Ye, Juan Terven, Zachary Bennett, Ying Zheng, Tingting Jiang, and Tiejun Huang. MUV A: A new large-scale benchmark for multi-view amodal instance segmentation in the shopping scenario. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 23447–23456. IEEE, 2023
work page 2023
-
[4]
Segment Anything, Even Occluded
Wei-En Tai, Yu-Lin Shih, Cheng Sun, Yu-Chiang Frank Wang, and Hwann-Tzong Chen. Segment anything, even occluded.CoRR, abs/2503.06261, 2025
work page Pith review arXiv 2025
-
[5]
LISA: reasoning segmentation via large language model
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: reasoning segmentation via large language model. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 9579–9589. IEEE, 2024
work page 2024
-
[6]
PSALM: pixelwise segmentation with large multi-modal model
Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. PSALM: pixelwise segmentation with large multi-modal model. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors,Computer Vision - ECCV 2024 - 18th European Confer- ence, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXXIV, volume 15092 of...
work page 2024
-
[7]
Referring transformer: A one-step approach to multi-task visual grounding
Muchen Li and Leonid Sigal. Referring transformer: A one-step approach to multi-task visual grounding. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors,Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December ...
work page 2021
-
[8]
Seqtr: A simple yet universal network for visual grounding
Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors,Computer Vision - ECCV 2022 - 17th European Conference, Tel Avi...
work page 2022
Show all 29 references
-
[9]
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip H. S. Torr. LA VT: language-aware vision transformer for referring image segmentation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022...
2022
-
[10]
GRES: generalized referring expression segmen- tation
Chang Liu, Henghui Ding, and Xudong Jiang. GRES: generalized referring expression segmen- tation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, V ancouver , BC, Canada, June 17-24, 2023, pages 23592–23601. IEEE, 2023
2023
-
[11]
An im- proved baseline for reasoning segmentation with large language model.CoRR, abs/2312.17240, 2023
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An im- proved baseline for reasoning segmentation with large language model.CoRR, abs/2312.17240, 2023
2023 arXiv
-
[12]
Pixellm: Pixel reasoning with large multimodal model
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 26364–263...
2024
-
[13]
Shaker, Salman H
Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Eric P. Xing, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Glamm: Pixel grounding large multimodal model. InIEEE/CVF 10 Conference on Compute...
2024
-
[14]
SAM4MLLM: enhance multi-modal large language model for referring expression segmenta- tion
Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. SAM4MLLM: enhance multi-modal large language model for referring expression segmenta- tion. InComputer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proce...
2024
-
[15]
Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet,...
2024
-
[16]
Berg, and Tamara L
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, edi- tors,Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, Oct...
2016
-
[17]
Yuille, and Kevin Murphy
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las V egas, NV , USA, June 27-30, 2016, ...
2016
-
[18]
Phrasecut: Language- based image segmentation in the wild
Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language- based image segmentation in the wild. In2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10213–10222. Computer Vision ...
2020
-
[19]
Plummer, Liwei Wang, Chris M
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In2015 IEEE International Conference on Computer Vision, ICCV 2015,...
2015
-
[20]
PACO: parts and attributes of common objects
Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, and Dhruv Mahajan. PACO: parts and attributes of common objects. In IEEE/CVF Conference on Co...
2023
-
[21]
Metaxas, and Piotr Dollár
Yan Zhu, Yuandong Tian, Dimitris N. Metaxas, and Piotr Dollár. Semantic amodal segmentation. In2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3001–3009. IEEE Computer Society, 2017
2017
-
[22]
Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors,Computer Vision - ECCV 2014 ...
2014
-
[23]
Mvtec D2S: densely segmented supermarket dataset
Patrick Follmann, Tobias Böttger, Philipp Härtinger, Rebecca König, and Markus Ulrich. Mvtec D2S: densely segmented supermarket dataset. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors,Computer Vision - ECCV 2018 - 15th European Conference, ...
2018
-
[24]
Segan: Segmenting and generating the invisible
Kiana Ehsani, Roozbeh Mottaghi, and Ali Farhadi. Segan: Segmenting and generating the invisible. In2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 6144–6153. Computer Vision Foundation / IEEE Computer...
2018
-
[25]
Amodal instance segmentation with KINS dataset
Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Amodal instance segmentation with KINS dataset. InIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3014–3023. Computer Vision Foundation / IEEE, 2019
2019
-
[26]
Dinesh Reddy, Robert Tamburo, and Srinivasa G
N. Dinesh Reddy, Robert Tamburo, and Srinivasa G. Narasimhan. WALT: watch and learn 2d amodal representation from time-lapse imagery. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 9346–9356. IEEE, 2022
2022
-
[27]
Amodal ground truth and completion in the wild
Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. Amodal ground truth and completion in the wild. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 28003–28013. IEEE, 2024
2024
-
[28]
Chang, Angela Dai, Thomas A
Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from RGB-D data in indoor environments. In2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, Oct...
2017
-
[29]
pix2gestalt: Amodal segmentation by synthesizing wholes
Ege Ozguroglu, Ruoshi Liu, Dídac Surís, Dian Chen, Achal Dave, Pavel Tokmakov, and Carl V ondrick. pix2gestalt: Amodal segmentation by synthesizing wholes. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 39...
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.