REVIEW 4 major objections 5 minor 1 cited by
ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that ROSE eliminates predefined category inputs and performs dense segmentation by generating category names from patch-level embeddings.
desk verdict A plausible patch-wise dense segmentation LMM whose headline open-set claim and CSR gains are undercut by the evaluation design; worth refereeing, but needs a real open-set benchmark and an honest refinement protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the patch-wise perception process. The image is divided into non-overlapping patches; for each patch, three MLP heads read the language model's object-level embedding and output an objectness score, a mask embedding, and a category embedding. At inference, objectness scores filter patches to a top-N set; mask embeddings feed a frozen promptable mask decoder to create dense masks; category embeddings are re-inserted into the language model through a fixed instruction such as 'decode this linguistic embedding into a noun concept,' so category names are generated as words. A 3x3 super-patch grouping assigns detecting roles for small, medium, large, and stuff regions, improving instance-scale behavior. A second stage, conversation-based segmentation refinement, concatenates the image with the previous mask and a corrective instruction, then generates a revised mask and category.
What would settle it
Run the full ROSE plus refinement pipeline with the model itself choosing which masks to refine, using its own objectness scores or predicted-category confidence, and compare the resulting ADE-20k mIoU and COCO mAP with Table 2. If the gains disappear when ground-truth confusion and IoU are not used to pick refinement targets, the refinement claim as an autonomous capability is falsified.
Extended reading notes
Core claim
The central discovery, as the paper states it, is that patch-wise perception makes dense prediction stable in a segmentation large multimodal model. Where stacking many special segmentation tokens causes the model to lose local image detail, assigning each image patch its own objectness score, mask embedding, and category embedding keeps each prediction local and lets a frozen promptable mask decoder produce the final mask. The category embedding is then treated as a linguistic token inserted into a fixed instruction template, so the language model outputs the category name autoregressively without any candidate category list at inference. The paper also introduces a conversation-based refinement paradigm that concatenates the previous mask with the image and a textual correction prompt, reporting that this raises semantic segmentation from 43.2 to 51.6 mIoU on the evaluated protocol and improves referring segmentation by roughly 12.6 points on average.
Load-bearing premise
The performance gains attributed to conversation-based refinement assume the model can decide which predictions to refine, but in the experiments ground-truth confusion and IoU select those targets, so autonomous selection is never demonstrated.
Editorial extensions
If this is right
- A single model can produce dense and sparse masks from a task-level prompt without a predefined category list, so open-set segmentation no longer requires a separate vocabulary selector.
- Category names are generated as language, allowing the model to label objects it has not been explicitly trained to name, at least within the language model's vocabulary.
- A second conversational round that receives the previous mask and a correction prompt can improve boundaries and categories, making segmentation an interactive process.
- Patch-level objectness scores provide confidence values, letting the same framework output ranked instance predictions rather than only semantic classes.
- Because the model is trained on semantic, instance, and referring data together, the paper claims competitive performance in all three tasks within one set of weights.
Reading between the lines
- We infer that if ROSE's category generation works without a fixed vocabulary, the same category-embedding prompt could be reused for open-vocabulary detection and retrieval without retraining.
- We infer that the CSR gains in Table 2 are upper bounds for a self-refining system, since a deployed version would need a confidence-based trigger to select refinement targets on its own.
- We infer that granularity failures such as 'plant' versus 'tree' point to a missing evaluation axis: a proper open-set benchmark would need to count near-synonym and hierarchy errors, not just exact-label IoU.
- We infer that the fixed 3x3 super-patch role assignment is one design point and could be learned end-to-end rather than fixed by scale and type.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ROSE, a large multimodal model for dense open-set segmentation. The method treats each image patch as an independent region of interest and predicts an objectness score, a mask embedding, and a category embedding per patch. Dense masks are decoded via SAM, while category names are generated by the LLM through a custom instruction-response paradigm, with no explicit candidate category set. A conversation-based segmentation refinement (CSR) mechanism is added to iteratively improve masks and categories. Experiments are reported on ADE-20k semantic segmentation, COCO instance segmentation, and RefCOCO/+/g referring segmentation, together with ablations of the segmentation framework, super-patch design, LoRA parameters, and refinement mechanisms.
Significance. If the claims were fully validated, the patch-wise perception design could be a useful building block for dense segmentation with LMMs, and the idea of decoding category embeddings into noun concepts through the LLM is a distinctive alternative to similarity-based open-vocabulary classification. The paper provides a clear architecture description, a reasonable set of ablations, and an explicit limitation statement. However, the significance is substantially reduced by three issues: the open-set capability is never quantitatively evaluated on categories outside the training vocabulary; the reported +CSR gains are obtained with ground-truth-guided selection of refinement targets; and Section 4.2 contains direct numerical contradictions with Table 2. These issues leave the central claims unverified in their current form.
major comments (4)
- [§4.1, §4.2, §A.4, Limitation] The central claim that ROSE performs open-set, free-vocabulary segmentation is not evaluated. The only quantitative evaluations are on ADE-20k val, COCO val, and RefCOCO/+/g, and all of these datasets (with their label vocabularies) are used in training per §4.1. There is no held-out-category experiment: the model never has to segment or name a category that was not seen during training. The paper's own Limitation admits that 'lacking a comprehensive benchmark limits our ability to fully evaluate model performance across diverse open-set scenarios.' As a result, the abstract's claim of 'category prediction independent of closed-set constraints or predefined categories' is not supported by any measured evidence. I recommend adding a quantitative zero-shot protocol, e.g., training on a subset of categories and evaluating on disjoint held-out categories, with a metric that matches generated category names to ground-truth labels.
- [Supplementary A.1, §4.2, Table 2] The +CSR improvements for semantic and instance segmentation rely on ground-truth information to select which predictions to refine. For semantic segmentation, the five categories with the highest Union-minus-Intersection are chosen from a confusion matrix computed against the ground truth; for instance segmentation, the ten predictions with IoU below 50 are selected using an IoU matrix against ground truth (Supplementary Algorithm 1 and the accompanying text). In deployment the model must decide autonomously when and what to refine, so these numbers describe an oracle-guided upper bound, not the autonomous refinement capability claimed in §3.4. Please either evaluate with model-driven selection (e.g., based on predicted confidence) or clearly re-label the +CSR results as oracle-refined.
- [§4.2 vs Table 2] There are direct numerical contradictions between the prose and Table 2. The text reports 43.2 mIoU for ROSE and 51.6 mIoU with CSR on ADE-20k, while Table 2 lists 51.0 and 57.4; for COCO instance segmentation the text reports 34.4 mAP while Table 2 lists 36.3; and the text claims CSR 'achieves SOTA' on semantic segmentation, but Mask2former's 57.7 exceeds ROSE+CSR's 57.4. Since the text and table cannot both be correct, the reported results are internally inconsistent. The authors must reconcile these numbers and indicate which are authoritative.
- [§3.3, §A.4] The open-category generation mechanism is not quantitatively evaluated. The instruction-response paradigm decodes the category embedding E_cat into a noun phrase, but no metric measures whether a generated noun is semantically valid or correctly matches a novel category. The only quantitative semantic-segmentation evaluation stacks predictions into a fixed dataset-specific N-channel output (§A.4), which requires matching generated names to the known vocabulary. Thus the 'category prediction independent of closed-set constraints' claim rests entirely on qualitative examples (Fig. 3, Fig. 9). Please provide an evaluation that measures the accuracy and quality of generated category names on categories not seen during training.
minor comments (5)
- [§4.3, Table 5] The LoRA ablation is trained with 20% of the iterations of the main model, so the reported ADE-20k and COCO numbers are not directly comparable to the main results; please state whether the observed trend holds with full training or present the comparison on a normalized basis.
- [§4.1] The patch size p is never specified; only '482 predicting patches' is given. Since S = floor(L/p), the number of patches should be a perfect square or otherwise derivable; please clarify how 482 is obtained, especially in relation to the super-patch design.
- [§3.2] The super-patch role assignment is introduced but it is not clear how the role is incorporated into the patch analyzer; Fig. 5 in the supplement shows the spatial arrangement but not how the role labels are encoded or used in the loss.
- [§4.2] The claim of 'multiple SOTAs in refcoco/+/g datasets' is stronger than Table 2 supports: on RefCOCO testA, GLaMM-7B exceeds ROSE, and on RefCOCOg test, GSVA-13B exceeds ROSE. Please qualify the claim.
- [Eq. (1) and §3.3] The notation y_txt is used both for the overall text-generation output in Eq. (1) and for the counting task in §3.5, while y_cat denotes the category generation; this overlapping notation is confusing and should be separated.
Circularity Check
No significant circularity: ROSE is an empirical benchmark system; the central open-category claim is under-tested but not defined into existence, and the self-citations are not load-bearing.
full rationale
ROSE is an empirical systems paper rather than a formal derivation, so the main circularity patterns (definitional equivalence, fitted-parameter-as-prediction, imported uniqueness theorems) do not apply. The quantitative claims are measured against external benchmarks (ADE-20k, COCO, RefCOCO/+/g), and the reported numbers are not constructed from the method's own definitions. The core open-category mechanism is a trained LLM instruction-response mapping; it is not defined in terms of the benchmark outputs, and no equation in Sec. 3 reduces a reported result to a training target. The paper's own Limitation section states: 'lacking a comprehensive benchmark limits our ability to fully evaluate model performance across diverse open-set scenarios,' which is an honest admission of an evaluation gap rather than evidence of circularity. Similarly, the oracle-guided refinement selection in Supplementary A.1 (choosing the worst categories and instances using ground-truth confusion and IoU) means the +CSR gains should not be interpreted as autonomous refinement performance, but that is an experimental-protocol confound, not a constructional circularity. Minor self-citations exist: the super-patch is 'Inspired by PixelLM [63]' and GKC [26] is by the first author, but neither is load-bearing; the super-patch is ablated and the central claim does not rest on those citations. Overall, no Eq. X = Eq. Y reduction, no fitted input renamed as prediction, and no self-citation chain forcing the main result were found.
Assumptions & free parameters
free parameters (5)
- Top-k selected patches at inference =
100 (from 482 predicting patches)
- Super-patch role assignment =
4 small, 3 medium, 1 large, 1 stuff per 3x3 block
- Loss weights and objectness negative sampling ratio =
not specified
- LoRA alpha and target modules =
alpha 32, all projection layers
- Refinement selection counts =
semantic: 5 worst UmI categories; instance: 10 predictions below IoU 50; referring: all predictions
assumptions (5)
- domain assumption SAM's mask decoder can turn a learned mask embedding into a high-quality mask when prompted from LMM features
- domain assumption Category embeddings produced by the patch analyzer occupy a space the frozen Qwen LLM can decode into noun concepts
- domain assumption Patch center plus 8-neighbor supervision is sufficient to assign object instances to patches
- domain assumption SigLIP text embedding space is a sufficient target for category alignment
- domain assumption Standard ADE-20k, COCO, and RefCOCO benchmarks measure open-set category generation
Cite this review
Pith. "Pith review of ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model." pith.science (2026). https://pith.science/paper/Y2TQJVUJ
@misc{pith2026241200153,
author = {Pith},
title = {Pith review of: ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y2TQJVUJ}},
note = {Machine review of arXiv:2412.00153}
}
read the original abstract
Advances in CLIP and large multimodal models (LMMs) have enabled open-vocabulary and free-text segmentation, yet existing models still require predefined category prompts, limiting free-form category self-generation. Most segmentation LMMs also remain confined to sparse predictions, restricting their applicability in open-set environments. In contrast, we propose ROSE, a Revolutionary Open-set dense SEgmentation LMM, which enables dense mask prediction and open-category generation through patch-wise perception. Our method treats each image patch as an independent region of interest candidate, enabling the model to predict both dense and sparse masks simultaneously. Additionally, a newly designed instruction-response paradigm takes full advantage of the generation and generalization capabilities of LMMs, achieving category prediction independent of closed-set constraints or predefined categories. To further enhance mask detail and category precision, we introduce a conversation-based refinement paradigm, integrating the prediction result from previous step with textual prompt for revision. Extensive experiments demonstrate that ROSE achieves competitive performance across various segmentation tasks in a unified framework. Code will be released.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
Bias from small-scale leakage in Pulsar Timing Array maps
Unmodeled small-scale gravitational-wave power systematically inflates reconstructed large-scale angular power spectra in pulsar timing array anisotropy searches.
Reference graph
Works this paper leans on
-
[1]
Deep learning using rectified linear units (relu)
Abien Fred Agarap. Deep learning using rectified linear units (relu). CoRR, abs/1803.08375, 2018. 6
arXiv 2018
-
[2]
Barron, Fer- ran Marques, and Jitendra Malik
Pablo Arbelaez, Jordi Pont-Tuset, Jonathan T. Barron, Fer- ran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2014. 2
2014
-
[3]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, et al. Qwen tech- nical report. arXiv preprint arXiv:2309.16609, 2023. 6
arXiv 2023
-
[4]
Cores: Orchestrating the dance of reasoning and seg- mentation
Xiaoyi Bao, Siyang Sun, Shuailei Ma, Kecheng Zheng, Yuxin Guo, Guosheng Zhao, Yun Zheng, and Xingang Wang. Cores: Orchestrating the dance of reasoning and seg- mentation. arXiv preprint arXiv:2404.05673, 2024. 3
arXiv 2024
-
[5]
PaliGemma: A versatile 3B VLM for transfer
Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, et al. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726, 2024. 2
arXiv 2024
-
[6]
Coco- stuff: Thing and stuff classes in context
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In CVPR, 2018. 6
2018
-
[7]
Emerg- ing properties in self-supervised vision transformers
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 5
2021
-
[8]
Shikra: Unleashing multi- modal llm’s referential dialogue magic
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 3
arXiv 2023
Show all 94 references
-
[9]
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic im- age segmentation with deep convolutional nets, atrous con- volution, and fully connected crfs. TPAMI, 2018. 1, 2
2018
-
[10]
Rethinking atrous convolution for semantic image segmentation
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587, 2017. 2
2017 arXiv
-
[11]
Fleet, and Geoffrey Hinton
Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. arXiv preprint arXiv:2206.07669, 2022. 6
2022 arXiv
-
[12]
Schwing, and Alexander Kir- illov
Bowen Cheng, Alexander G. Schwing, and Alexander Kir- illov. Per-pixel classification is not all you need for semantic segmentation. 2021. 3
2021
-
[13]
Schwing, Alexan- der Kirillov, and Rohit Girdhar
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022. 3, 6
2022
-
[14]
CascadePSP: Toward class-agnostic and very high- resolution segmentation via global and local refinement
Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. CascadePSP: Toward class-agnostic and very high- resolution segmentation via global and local refinement. In CVPR, 2020. 3, 6
2020
-
[15]
Gonzalez, Ion Stoica, and Eric P
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2, 3
2023
-
[16]
Instance-aware se- mantic segmentation via multi-task network cascades
Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware se- mantic segmentation via multi-task network cascades. In CVPR, 2016. 2
2016
-
[17]
Instructblip: Towards general- purpose vision-language models with instruction tuning
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. arXiv:2305.06500, 2023. 3
2023 arXiv
-
[18]
Vision-language transformer and query generation for refer- ring segmentation
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for refer- ring segmentation. In ICCV, pages 16321–16330, 2021. 3
2021
-
[19]
De- coupling zero-shot semantic segmentation
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. In CVPR, pages 11583–11592, 2022. 3
2022
-
[20]
A discriminatively trained, multiscale, deformable part model
Pedro Felzenszwalb, David McAllester, and Deva Ra- manan. A discriminatively trained, multiscale, deformable part model. In 2008 IEEE conference on computer vision and pattern recognition, pages 1–8. Ieee, 2008. 2
2008
-
[21]
Dual attention network for scene segmentation
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 2
2019
-
[22]
Scal- ing open-vocabulary image segmentation with image-level labels
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceed- ings, Part XXXVI, pages 540–557. Springer, 2022. 3
2022
-
[23]
Fast r-cnn
Ross Girshick. Fast r-cnn. In ICCV, 2015. 2
2015
-
[24]
Efficient hierarchical graph-based video segmentation
Matthias Grundmann, Vivek Kwatra, Mei Han, and Irfan Essa. Efficient hierarchical graph-based video segmentation. In 2010 ieee computer society conference on computer vision and pattern recognition, pages 2141–2148. IEEE, 2010. 2
2010
-
[25]
Segnext: Rethinking convolutional attention design for semantic segmentation
Meng-Hao Guo, Chengze Lu, Qibin Hou, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. arXiv preprint arXiv:2209.08575, 2022. 1
2022 arXiv
-
[26]
Global knowledge calibration for fast open-vocabulary segmentation
Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Yun- chao Wei, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, et al. Global knowledge calibration for fast open-vocabulary segmentation. In ICCV, 2023. 3
2023
-
[27]
Multi-modal instruction tuned llms with fine-grained visual perception
Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. Multi-modal instruction tuned llms with fine-grained visual perception. In CVPR, pages 13980–13990, 2024. 3
2024
-
[28]
Lora: Low-rank adaptation of large language mod- els
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language mod- els. arXiv:2106.09685, 2021. 6
2021 arXiv
-
[29]
Bi-directional relationship inferring network for referring image segmentation
Zhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang, and Huchuan Lu. Bi-directional relationship inferring network for referring image segmentation. In CVPR, pages 4424– 4433, 2020. 3
2020
-
[30]
Referring im- age segmentation via cross-modal progressive comprehen- sion
Shaofei Huang, Tianrui Hui, Si Liu, Guanbin Li, Yunchao Wei, Jizhong Han, Luoqi Liu, and Bo Li. Referring im- age segmentation via cross-modal progressive comprehen- sion. In CVPR, pages 10488–10497, 2020. 3
2020
-
[31]
CCNet: Criss-cross attention for semantic segmentation
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CCNet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 2 9
2019
-
[32]
Scaling up visual and vision-language representation learning with noisy text supervision
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–
-
[33]
Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation
Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yun- chao Wei, and Shi Humphrey. Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation. In ECCV, 2024. 3
2024
-
[34]
Locate then segment: A strong pipeline for referring image segmentation
Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, and Tie- niu Tan. Locate then segment: A strong pipeline for referring image segmentation. In CVPR, pages 9858–9867, 2021. 3
2021
-
[35]
Referitgame: Referring to objects in pho- tographs of natural scenes
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In EMNLP, 2014. 6
2014
-
[36]
Segment any- thing
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, pages 4015–4026, 2023. 6
2023
-
[37]
Large language models are zero-shot reasoners
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, pages 22199–22213. Cur- ran Associates, Inc., 2022. 5
2022
-
[38]
Lisa: Reasoning segmenta- tion via large language model
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model. In CVPR, pages 9579–9589,
-
[39]
Llava-onevision: Easy visual task transfer
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 6
2024 arXiv
-
[40]
BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. 2023. 3
2023
-
[41]
Referring transformer: A one- step approach to multi-task visual grounding
Muchen Li and Leonid Sigal. Referring transformer: A one- step approach to multi-task visual grounding. Advances in Neural Information Processing Systems , 34:19652–19664,
-
[42]
Textbooks are all you need ii: phi-1.5 technical report, 2023
Yuanzhi Li, S ´ebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023. 3
2023
-
[43]
Open-vocabulary semantic segmentation with mask-adapted clip
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, pages 7061–7070, 2023. 3
2023
-
[44]
Belongie, Lubomir D
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312,
-
[45]
Visual instruction tuning, 2023
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 2, 3
2023
-
[46]
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. 1, 2
2015
-
[47]
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017. 6
2017 arXiv
-
[48]
Cascade grouped attention network for referring expression segmentation
Gen Luo, Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jinsong Su, Chia-Wen Lin, and Qi Tian. Cascade grouped attention network for referring expression segmentation. InACM MM, pages 1274–1282, 2020. 3
2020
-
[49]
Generation and comprehension of unambiguous object descriptions
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016. 6
2016
-
[50]
The mapillary vistas dataset for semantic understanding of street scenes
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017. 6
2017
-
[51]
Chatgpt: A language model for conversational ai
OpenAI. Chatgpt: A language model for conversational ai. Technical report, OpenAI, 2023. 2, 3
2023
-
[52]
Gpt-4 technical report, 2023
OpenAI. Gpt-4 technical report, 2023. 2, 3
2023
-
[53]
Kosmos-2: Ground- ing multimodal large language models to the world
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. ArXiv, abs/2306, 2023. 3
2023
-
[54]
Kosmos-2: Grounding multimodal large language models to the world
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shao- han Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv:2306.14824, 2023. 3
2023 arXiv
-
[55]
Pedro H. O. Pinheiro, Ronan Collobert, and Piotr Doll ´ar. Learning to segment object candidates. In NeurIPS, 2015. 2
2015
-
[56]
Pinheiro, Tsung-Yi Lin, Ronan Collobert, and Piotr Doll´ar
Pedro O. Pinheiro, Tsung-Yi Lin, Ronan Collobert, and Piotr Doll´ar. Learning to refine object segments. In ECCV, 2016. 2
2016
-
[57]
Multiscale combinatorial grouping for image segmentation and object proposal gener- ation
Jordi Pont-Tuset, Pablo Arbelaez, Jonathan T Barron, Fer- ran Marques, and Jitendra Malik. Multiscale combinatorial grouping for image segmentation and object proposal gener- ation. IEEE transactions on pattern analysis and machine intelligence, 39(1):128–140, 2016. 2
2016
-
[58]
Learning to segment every referring object point by point
Mengxue Qu, Yu Wu, Yunchao Wei, Wu Liu, Xiaodan Liang, and Yao Zhao. Learning to segment every referring object point by point. In CVPR, pages 3021–3030, 2023. 3
2023
-
[59]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1
2021
-
[60]
Learn- ing transferable visual models from natural language super- vision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 3
2021
-
[61]
Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model. CVPR,
-
[62]
Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters. In SIGKDD, 2020. 6 10
2020
-
[63]
Pixellm: Pixel reasoning with large multimodal model
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, pages 26374–26383, 2024. 3, 5, 7
2024
-
[64]
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 1
2015
-
[65]
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2, 3
2023 arXiv
-
[66]
Selective search for object recognition
Jasper RR Uijlings, Koen EA Van De Sande, Theo Gev- ers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision , 104 (2):154–171, 2013. 2
2013
-
[67]
Llm-seg: Bridging image segmen- tation and large language model reasoning
Junchi Wang and Lei Ke. Llm-seg: Bridging image segmen- tation and large language model reasoning. In CVPR, pages 1765–1774, 2024. 5
2024
-
[68]
SegRefiner: Towards model- agnostic segmentation refinement with discrete diffusion process
Mengyu Wang, Henghui Ding, Jun Hao Liew, Jiajun Liu, Yao Zhao, and Yunchao Wei. SegRefiner: Towards model- agnostic segmentation refinement with discrete diffusion process. In NeurIPS, 2023. 3
2023
-
[69]
Visionllm: Large language model is also an open-ended decoder for vision-centric tasks
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. https://arxiv.org/abs/2305.11175) , 2023. 3, 5, 6
2023 arXiv
-
[70]
The all-seeing project: Towards panop- tic visual recognition and understanding of the open world
Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhen- hang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panop- tic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907, 2023. 3
2023 arXiv
-
[71]
Non-local neural networks
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- ing He. Non-local neural networks. In CVPR, 2018. 2
2018
-
[72]
SOLO: Segmenting objects by locations
Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. SOLO: Segmenting objects by locations. In ECCV,
-
[73]
Solov2: Dynamic and fast instance segmentation
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chun- hua Shen. Solov2: Dynamic and fast instance segmentation. NeurIPS, 2020. 2
2020
-
[74]
Images speak in images: A general- ist painter for in-context visual learning
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A general- ist painter for in-context visual learning. arXiv preprint arXiv:2212.02499, 2022. 6
2022 arXiv
-
[75]
Seggpt: Segmenting ev- erything in context
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting ev- erything in context. arXiv preprint arXiv:2304.03284, 2023. 6
2023 arXiv
-
[76]
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, pages 24824–24837. Curran Associates, Inc., 2022. 5
2022
-
[77]
Gsva: Generalized segmentation via multimodal large language models
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In CVPR, pages 3858– 3869, 2024. 3, 5, 6
2024
-
[78]
Alvarez, and Ping Luo
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and ef- ficient design for semantic segmentation with transformers. In NIPS, 2021. 1
2021
-
[79]
Open-vocabulary panop- tic segmentation with text-to-image diffusion models
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In CVPR, pages 2955–2966, 2023. 3
2023
-
[80]
A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model. In ECCV, pages 736–753. Springer, 2022. 3
2022
-
[81]
Fine-grained visual prompting, 2023
Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting, 2023. 8
2023
-
[82]
Ferret: Refer and ground anything anywhere at any granularity
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 2, 3, 5, 6
2023 arXiv
-
[83]
Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. NeurIPS, 36, 2023. 3
2023
-
[84]
Osprey: Pixel un- derstanding with visual instruction tuning
Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel un- derstanding with visual instruction tuning. In CVPR, pages 28202–28211, 2024. 6
2024
-
[85]
Sigmoid loss for language image pre-training,
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training,
-
[86]
Gpt4roi: In- struction tuning large language model on region-of-interest
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: In- struction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023. 3
2023 arXiv
-
[87]
Controlvideo: Training-free controllable text-to-video generation
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023. 2
2023 arXiv
-
[88]
Groundhog: Grounding large language models to holistic segmentation
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In CVPR, 2024. 5
2024
-
[89]
Psalm: Pixelwise segmentation with large multi-modal model
Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. In ECCV, pages 74–91. Springer, 2025. 3, 5, 6
2025
-
[90]
Rethinking semantic segmen- tation from a sequence-to-sequence perspective with trans- formers
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmen- tation from a sequence-to-sequence perspective with trans- formers. In CVPR, 2021. 2
2021
-
[91]
Scene parsing through ade20k dataset
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 6
2017
-
[92]
Seqtr: A simple yet universal network for visual grounding
Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, 11 and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. ECCV, 2022. 3
2022
-
[93]
category
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language mod- els. arXiv:2304.10592, 2023. 3 12 ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multi...
2023 arXiv
-
[94]
User: <IMAGE,MASK> Please segment target region with mask and corre- sponding category
missed detections: “ User: <IMAGE,MASK> Please segment target region with mask and corre- sponding category. ”. B. Visualizations B.1. Failure cases Fig. 7 shows the typical flaw caused by granularity differ- ences (“plant” and “tree”). Due to the inherent limitations of exist...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.