Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read This paper claims that ROSE eliminates predefined category inputs and performs dense segmentation by generating category names from patch-level embeddings.

desk verdict A plausible patch-wise dense segmentation LMM whose headline open-set claim and CSR gains are undercut by the evaluation design; worth refereeing, but needs a real open-set benchmark and an honest refinement protocol. read the letter →

arxiv 2412.00153 v3 pith:Y2TQJVUJ submitted 2024-11-29 cs.CV cs.LG

classification cs.CVcs.LG
keywords ROSEopen-setsegmentationdensepredictionlargemultimodalmodelpatch-wiseperceptioncategorygenerationrefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ROSE's claim is that dense open-set segmentation can be freed from predefined category prompts. Instead of matching masks against a fixed list of class names, the paper proposes a large multimodal model that treats every image patch as a potential object, scores it for object presence, decodes a mask from a learned embedding, and lets the language model generate a category name in words. The same framework reports results on semantic, instance, and referring segmentation, with a conversation-based refinement step that improves masks and labels by revisiting earlier predictions. This matters because current open-vocabulary segmenters only select from categories someone has already supplied; ROSE claims to be a genuine generator of both masks and category names.

What carries the argument

The load-bearing mechanism is the patch-wise perception process. The image is divided into non-overlapping patches; for each patch, three MLP heads read the language model's object-level embedding and output an objectness score, a mask embedding, and a category embedding. At inference, objectness scores filter patches to a top-N set; mask embeddings feed a frozen promptable mask decoder to create dense masks; category embeddings are re-inserted into the language model through a fixed instruction such as 'decode this linguistic embedding into a noun concept,' so category names are generated as words. A 3x3 super-patch grouping assigns detecting roles for small, medium, large, and stuff regions, improving instance-scale behavior. A second stage, conversation-based segmentation refinement, concatenates the image with the previous mask and a corrective instruction, then generates a revised mask and category.

What would settle it

Run the full ROSE plus refinement pipeline with the model itself choosing which masks to refine, using its own objectness scores or predicted-category confidence, and compare the resulting ADE-20k mIoU and COCO mAP with Table 2. If the gains disappear when ground-truth confusion and IoU are not used to pick refinement targets, the refinement claim as an autonomous capability is falsified.

Watch

Extended reading notes

Core claim

The central discovery, as the paper states it, is that patch-wise perception makes dense prediction stable in a segmentation large multimodal model. Where stacking many special segmentation tokens causes the model to lose local image detail, assigning each image patch its own objectness score, mask embedding, and category embedding keeps each prediction local and lets a frozen promptable mask decoder produce the final mask. The category embedding is then treated as a linguistic token inserted into a fixed instruction template, so the language model outputs the category name autoregressively without any candidate category list at inference. The paper also introduces a conversation-based refinement paradigm that concatenates the previous mask with the image and a textual correction prompt, reporting that this raises semantic segmentation from 43.2 to 51.6 mIoU on the evaluated protocol and improves referring segmentation by roughly 12.6 points on average.

Load-bearing premise

The performance gains attributed to conversation-based refinement assume the model can decide which predictions to refine, but in the experiments ground-truth confusion and IoU select those targets, so autonomous selection is never demonstrated.

Editorial extensions

If this is right

  • A single model can produce dense and sparse masks from a task-level prompt without a predefined category list, so open-set segmentation no longer requires a separate vocabulary selector.
  • Category names are generated as language, allowing the model to label objects it has not been explicitly trained to name, at least within the language model's vocabulary.
  • A second conversational round that receives the previous mask and a correction prompt can improve boundaries and categories, making segmentation an interactive process.
  • Patch-level objectness scores provide confidence values, letting the same framework output ranked instance predictions rather than only semantic classes.
  • Because the model is trained on semantic, instance, and referring data together, the paper claims competitive performance in all three tasks within one set of weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that if ROSE's category generation works without a fixed vocabulary, the same category-embedding prompt could be reused for open-vocabulary detection and retrieval without retraining.
  • We infer that the CSR gains in Table 2 are upper bounds for a self-refining system, since a deployed version would need a confidence-based trigger to select refinement targets on its own.
  • We infer that granularity failures such as 'plant' versus 'tree' point to a missing evaluation axis: a proper open-set benchmark would need to count near-synonym and hierarchy errors, not just exact-label IoU.
  • We infer that the fixed 3x3 super-patch role assignment is one design point and could be learned end-to-end rather than fixed by scale and type.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ROSE, a large multimodal model for dense open-set segmentation. The method treats each image patch as an independent region of interest and predicts an objectness score, a mask embedding, and a category embedding per patch. Dense masks are decoded via SAM, while category names are generated by the LLM through a custom instruction-response paradigm, with no explicit candidate category set. A conversation-based segmentation refinement (CSR) mechanism is added to iteratively improve masks and categories. Experiments are reported on ADE-20k semantic segmentation, COCO instance segmentation, and RefCOCO/+/g referring segmentation, together with ablations of the segmentation framework, super-patch design, LoRA parameters, and refinement mechanisms.

Significance. If the claims were fully validated, the patch-wise perception design could be a useful building block for dense segmentation with LMMs, and the idea of decoding category embeddings into noun concepts through the LLM is a distinctive alternative to similarity-based open-vocabulary classification. The paper provides a clear architecture description, a reasonable set of ablations, and an explicit limitation statement. However, the significance is substantially reduced by three issues: the open-set capability is never quantitatively evaluated on categories outside the training vocabulary; the reported +CSR gains are obtained with ground-truth-guided selection of refinement targets; and Section 4.2 contains direct numerical contradictions with Table 2. These issues leave the central claims unverified in their current form.

major comments (4)
  1. [§4.1, §4.2, §A.4, Limitation] The central claim that ROSE performs open-set, free-vocabulary segmentation is not evaluated. The only quantitative evaluations are on ADE-20k val, COCO val, and RefCOCO/+/g, and all of these datasets (with their label vocabularies) are used in training per §4.1. There is no held-out-category experiment: the model never has to segment or name a category that was not seen during training. The paper's own Limitation admits that 'lacking a comprehensive benchmark limits our ability to fully evaluate model performance across diverse open-set scenarios.' As a result, the abstract's claim of 'category prediction independent of closed-set constraints or predefined categories' is not supported by any measured evidence. I recommend adding a quantitative zero-shot protocol, e.g., training on a subset of categories and evaluating on disjoint held-out categories, with a metric that matches generated category names to ground-truth labels.
  2. [Supplementary A.1, §4.2, Table 2] The +CSR improvements for semantic and instance segmentation rely on ground-truth information to select which predictions to refine. For semantic segmentation, the five categories with the highest Union-minus-Intersection are chosen from a confusion matrix computed against the ground truth; for instance segmentation, the ten predictions with IoU below 50 are selected using an IoU matrix against ground truth (Supplementary Algorithm 1 and the accompanying text). In deployment the model must decide autonomously when and what to refine, so these numbers describe an oracle-guided upper bound, not the autonomous refinement capability claimed in §3.4. Please either evaluate with model-driven selection (e.g., based on predicted confidence) or clearly re-label the +CSR results as oracle-refined.
  3. [§4.2 vs Table 2] There are direct numerical contradictions between the prose and Table 2. The text reports 43.2 mIoU for ROSE and 51.6 mIoU with CSR on ADE-20k, while Table 2 lists 51.0 and 57.4; for COCO instance segmentation the text reports 34.4 mAP while Table 2 lists 36.3; and the text claims CSR 'achieves SOTA' on semantic segmentation, but Mask2former's 57.7 exceeds ROSE+CSR's 57.4. Since the text and table cannot both be correct, the reported results are internally inconsistent. The authors must reconcile these numbers and indicate which are authoritative.
  4. [§3.3, §A.4] The open-category generation mechanism is not quantitatively evaluated. The instruction-response paradigm decodes the category embedding E_cat into a noun phrase, but no metric measures whether a generated noun is semantically valid or correctly matches a novel category. The only quantitative semantic-segmentation evaluation stacks predictions into a fixed dataset-specific N-channel output (§A.4), which requires matching generated names to the known vocabulary. Thus the 'category prediction independent of closed-set constraints' claim rests entirely on qualitative examples (Fig. 3, Fig. 9). Please provide an evaluation that measures the accuracy and quality of generated category names on categories not seen during training.
minor comments (5)
  1. [§4.3, Table 5] The LoRA ablation is trained with 20% of the iterations of the main model, so the reported ADE-20k and COCO numbers are not directly comparable to the main results; please state whether the observed trend holds with full training or present the comparison on a normalized basis.
  2. [§4.1] The patch size p is never specified; only '482 predicting patches' is given. Since S = floor(L/p), the number of patches should be a perfect square or otherwise derivable; please clarify how 482 is obtained, especially in relation to the super-patch design.
  3. [§3.2] The super-patch role assignment is introduced but it is not clear how the role is incorporated into the patch analyzer; Fig. 5 in the supplement shows the spatial arrangement but not how the role labels are encoded or used in the loss.
  4. [§4.2] The claim of 'multiple SOTAs in refcoco/+/g datasets' is stronger than Table 2 supports: on RefCOCO testA, GLaMM-7B exceeds ROSE, and on RefCOCOg test, GSVA-13B exceeds ROSE. Please qualify the claim.
  5. [Eq. (1) and §3.3] The notation y_txt is used both for the overall text-generation output in Eq. (1) and for the counting task in §3.5, while y_cat denotes the category generation; this overlapping notation is confusing and should be separated.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: ROSE is an empirical benchmark system; the central open-category claim is under-tested but not defined into existence, and the self-citations are not load-bearing.

full rationale

ROSE is an empirical systems paper rather than a formal derivation, so the main circularity patterns (definitional equivalence, fitted-parameter-as-prediction, imported uniqueness theorems) do not apply. The quantitative claims are measured against external benchmarks (ADE-20k, COCO, RefCOCO/+/g), and the reported numbers are not constructed from the method's own definitions. The core open-category mechanism is a trained LLM instruction-response mapping; it is not defined in terms of the benchmark outputs, and no equation in Sec. 3 reduces a reported result to a training target. The paper's own Limitation section states: 'lacking a comprehensive benchmark limits our ability to fully evaluate model performance across diverse open-set scenarios,' which is an honest admission of an evaluation gap rather than evidence of circularity. Similarly, the oracle-guided refinement selection in Supplementary A.1 (choosing the worst categories and instances using ground-truth confusion and IoU) means the +CSR gains should not be interpreted as autonomous refinement performance, but that is an experimental-protocol confound, not a constructional circularity. Minor self-citations exist: the super-patch is 'Inspired by PixelLM [63]' and GKC [26] is by the first author, but neither is load-bearing; the super-patch is ablated and the central claim does not rest on those citations. Overall, no Eq. X = Eq. Y reduction, no fitted input renamed as prediction, and no self-citation chain forcing the main result were found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on empirical design choices and evaluation protocols rather than mathematical derivations. The hand-set hyperparameters and the ground-truth-guided refinement selection are the main quantities that affect the headline numbers, and several are not fully specified.

free parameters (5)
  • Top-k selected patches at inference = 100 (from 482 predicting patches)
    Inference keeps patches with top-100 objectness scores; this choice directly controls the density and quality of mask predictions and is fixed by the authors in Section 4.1.
  • Super-patch role assignment = 4 small, 3 medium, 1 large, 1 stuff per 3x3 block
    Hand-assigned detecting roles based on object scale and type; the ablation in Table 4 shows this changes the ADE-20k/COCO trade-off and materially affects instance segmentation performance.
  • Loss weights and objectness negative sampling ratio = not specified
    Section 3.5 defines BCE, Dice, and objectness losses but does not report the weights lambda_bce, lambda_dice, or how many unsupervised patches are sampled as negatives, making replication harder.
  • LoRA alpha and target modules = alpha 32, all projection layers
    The ablation in Table 5 shows performance continues to improve with larger alpha and more target modules, so these hyperparameters materially affect the reported results.
  • Refinement selection counts = semantic: 5 worst UmI categories; instance: 10 predictions below IoU 50; referring: all predictions
    This ground-truth-guided protocol in Supplementary A.1 drives the reported CSR gains; it is an evaluation protocol choice, not a model parameter, but it strongly affects the headline numbers.
assumptions (5)
  • domain assumption SAM's mask decoder can turn a learned mask embedding into a high-quality mask when prompted from LMM features
    The dense mask prediction pipeline in Section 3.3 (Eq. 3) inherits LISA's assumption that the SAM decoder generalizes to embeddings produced by the LMM's patch analyzer.
  • domain assumption Category embeddings produced by the patch analyzer occupy a space the frozen Qwen LLM can decode into noun concepts
    Open-category generation in Section 3.3 (Eq. 4) requires the LLM to map the MLP-predicted E_cat to text; no analysis or proof of this embedding-to-language mapping is given.
  • domain assumption Patch center plus 8-neighbor supervision is sufficient to assign object instances to patches
    Section 3.2 assigns supervision to the patch containing the object's mass center and its 8 neighbors; this heuristic determines which patches can fire and thus bounds recall.
  • domain assumption SigLIP text embedding space is a sufficient target for category alignment
    The SigLIP embedding loss L_sig uses InfoNCE to align E_cat with SigLIP text embeddings, assuming this space carries category semantics usable by the LLM.
  • domain assumption Standard ADE-20k, COCO, and RefCOCO benchmarks measure open-set category generation
    The paper claims free-vocabulary open-set ability but evaluates only closed-set benchmarks; the Limitation section admits that a comprehensive open-set benchmark is missing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model." pith.science (2026). https://pith.science/paper/Y2TQJVUJ

@misc{pith2026241200153,
  author       = {Pith},
  title        = {Pith review of: ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multimodal Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2TQJVUJ}},
  note         = {Machine review of arXiv:2412.00153}
}
read the original abstract

Advances in CLIP and large multimodal models (LMMs) have enabled open-vocabulary and free-text segmentation, yet existing models still require predefined category prompts, limiting free-form category self-generation. Most segmentation LMMs also remain confined to sparse predictions, restricting their applicability in open-set environments. In contrast, we propose ROSE, a Revolutionary Open-set dense SEgmentation LMM, which enables dense mask prediction and open-category generation through patch-wise perception. Our method treats each image patch as an independent region of interest candidate, enabling the model to predict both dense and sparse masks simultaneously. Additionally, a newly designed instruction-response paradigm takes full advantage of the generation and generalization capabilities of LMMs, achieving category prediction independent of closed-set constraints or predefined categories. To further enhance mask detail and category precision, we introduce a conversation-based refinement paradigm, integrating the prediction result from previous step with textual prompt for revision. Extensive experiments demonstrate that ROSE achieves competitive performance across various segmentation tasks in a unified framework. Code will be released.

Figures

Figures reproduced from arXiv: 2412.00153 by the authors.

Figure 1
Figure 1. Comparison of existing open-set segmentation frame [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of ROSE. (a) In Patch-wise Perception Processes, the vision encoder first encodes the input image and gets patched features, the feature is then concatenated with text instruction and fed into the Large Language model. Then every patch is analyzed by the patch analyzer, generating a mask embedding, a category embedding, and an objectness score. (b) In Patch-wise Mask and Category Decoding Process, p… view at source ↗
Figure 3
Figure 3. Qualitative results. We show some predictions of ROSE in cross-domain and in-domain scenarios, with generated categories labeled near each target. Please zoom in to see the details. The first row shows the results of images from other domains, including crayon drawings and clip art. The second row shows some predictions of the COCO val set. GT Input Mask Concat. Pred. Mask Image Mask Pred. Mask+Box Image Mask+Box Pr… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visualization of different refinement mechanisms. The first two columns are ground truth and mask expected to be refined. Concat. denotes concatenate mask with image, and Pred. stands for prediction. Mask and Mask+Box are other methods we try. Refinement Mechanism. To …
Figure 5
Figure 5. Figure 5: 3×3 super-patch arrangement. A.3. Training convergence We compared the loss convergence of ROSE with LISA. ROSE requires a bit more trainable parameters (4.8%) than LISA (3.7%), but it affects convergence little in the training stage according to [PITH_FULL_IMAGE:figu…
Figure 6
Figure 6. Figure 6: Mask loss during training. A.4. Dataset and Task Semantic Segmentation In the training stage, we use instance-level supervision for thing categories, instead of semantic-like supervision. Because we want our model to distinguish different identities as the category may…
Figure 7
Figure 7. Figure 7: Failure cases. We show some samples of typical failure scenario in the ADE20k dataset. Wrong classification results are labeled. Instance Segmentation In inference, we first use a thresh￾old to filter predicted instances, and then NMS post-process is conducted to get t…
Figure 8
Figure 8. Figure 8: Qualitative Results on ADE-20k. Input image, GT, original prediction result, and result after refinement are shown. The [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Qualitative Results on cross-domain images and referring tasks. Prompt, input image, GT, original prediction result and the result [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bias from small-scale leakage in Pulsar Timing Array maps

    astro-ph.IM 2025-10 conditional novelty 6.0 of 10

    Unmodeled small-scale gravitational-wave power systematically inflates reconstructed large-scale angular power spectra in pulsar timing array anisotropy searches.

Reference graph

Works this paper leans on

94 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep learning using rectified linear units (relu)

    Abien Fred Agarap. Deep learning using rectified linear units (relu). CoRR, abs/1803.08375, 2018. 6

  2. [2]

    Barron, Fer- ran Marques, and Jitendra Malik

    Pablo Arbelaez, Jordi Pont-Tuset, Jonathan T. Barron, Fer- ran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2014. 2

  3. [3]

    Qwen tech- nical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, et al. Qwen tech- nical report. arXiv preprint arXiv:2309.16609, 2023. 6

  4. [4]

    Cores: Orchestrating the dance of reasoning and seg- mentation

    Xiaoyi Bao, Siyang Sun, Shuailei Ma, Kecheng Zheng, Yuxin Guo, Guosheng Zhao, Yun Zheng, and Xingang Wang. Cores: Orchestrating the dance of reasoning and seg- mentation. arXiv preprint arXiv:2404.05673, 2024. 3

  5. [5]

    PaliGemma: A versatile 3B VLM for transfer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, et al. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726, 2024. 2

  6. [6]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In CVPR, 2018. 6

  7. [7]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 5

  8. [8]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 3

Show all 94 references
  1. [9]

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic im- age segmentation with deep convolutional nets, atrous con- volution, and fully connected crfs. TPAMI, 2018. 1, 2

  2. [10]

    Rethinking atrous convolution for semantic image segmentation

    Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587, 2017. 2

  3. [11]

    Fleet, and Geoffrey Hinton

    Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J. Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. arXiv preprint arXiv:2206.07669, 2022. 6

  4. [12]

    Schwing, and Alexander Kir- illov

    Bowen Cheng, Alexander G. Schwing, and Alexander Kir- illov. Per-pixel classification is not all you need for semantic segmentation. 2021. 3

  5. [13]

    Schwing, Alexan- der Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022. 3, 6

  6. [14]

    CascadePSP: Toward class-agnostic and very high- resolution segmentation via global and local refinement

    Ho Kei Cheng, Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. CascadePSP: Toward class-agnostic and very high- resolution segmentation via global and local refinement. In CVPR, 2020. 3, 6

  7. [15]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2, 3

  8. [16]

    Instance-aware se- mantic segmentation via multi-task network cascades

    Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware se- mantic segmentation via multi-task network cascades. In CVPR, 2016. 2

  9. [17]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. arXiv:2305.06500, 2023. 3

  10. [18]

    Vision-language transformer and query generation for refer- ring segmentation

    Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for refer- ring segmentation. In ICCV, pages 16321–16330, 2021. 3

  11. [19]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. In CVPR, pages 11583–11592, 2022. 3

  12. [20]

    A discriminatively trained, multiscale, deformable part model

    Pedro Felzenszwalb, David McAllester, and Deva Ra- manan. A discriminatively trained, multiscale, deformable part model. In 2008 IEEE conference on computer vision and pattern recognition, pages 1–8. Ieee, 2008. 2

  13. [21]

    Dual attention network for scene segmentation

    Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 2

  14. [22]

    Scal- ing open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceed- ings, Part XXXVI, pages 540–557. Springer, 2022. 3

  15. [23]

    Fast r-cnn

    Ross Girshick. Fast r-cnn. In ICCV, 2015. 2

  16. [24]

    Efficient hierarchical graph-based video segmentation

    Matthias Grundmann, Vivek Kwatra, Mei Han, and Irfan Essa. Efficient hierarchical graph-based video segmentation. In 2010 ieee computer society conference on computer vision and pattern recognition, pages 2141–2148. IEEE, 2010. 2

  17. [25]

    Segnext: Rethinking convolutional attention design for semantic segmentation

    Meng-Hao Guo, Chengze Lu, Qibin Hou, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. arXiv preprint arXiv:2209.08575, 2022. 1

  18. [26]

    Global knowledge calibration for fast open-vocabulary segmentation

    Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Yun- chao Wei, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, et al. Global knowledge calibration for fast open-vocabulary segmentation. In ICCV, 2023. 3

  19. [27]

    Multi-modal instruction tuned llms with fine-grained visual perception

    Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. Multi-modal instruction tuned llms with fine-grained visual perception. In CVPR, pages 13980–13990, 2024. 3

  20. [28]

    Lora: Low-rank adaptation of large language mod- els

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language mod- els. arXiv:2106.09685, 2021. 6

  21. [29]

    Bi-directional relationship inferring network for referring image segmentation

    Zhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang, and Huchuan Lu. Bi-directional relationship inferring network for referring image segmentation. In CVPR, pages 4424– 4433, 2020. 3

  22. [30]

    Referring im- age segmentation via cross-modal progressive comprehen- sion

    Shaofei Huang, Tianrui Hui, Si Liu, Guanbin Li, Yunchao Wei, Jizhong Han, Luoqi Liu, and Bo Li. Referring im- age segmentation via cross-modal progressive comprehen- sion. In CVPR, pages 10488–10497, 2020. 3

  23. [31]

    CCNet: Criss-cross attention for semantic segmentation

    Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CCNet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 2 9

  24. [32]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–

  25. [33]

    Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation

    Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yun- chao Wei, and Shi Humphrey. Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation. In ECCV, 2024. 3

  26. [34]

    Locate then segment: A strong pipeline for referring image segmentation

    Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, and Tie- niu Tan. Locate then segment: A strong pipeline for referring image segmentation. In CVPR, pages 9858–9867, 2021. 3

  27. [35]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In EMNLP, 2014. 6

  28. [36]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, pages 4015–4026, 2023. 6

  29. [37]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, pages 22199–22213. Cur- ran Associates, Inc., 2022. 5

  30. [38]

    Lisa: Reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmenta- tion via large language model. In CVPR, pages 9579–9589,

  31. [39]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 6

  32. [40]

    BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. 2023. 3

  33. [41]

    Referring transformer: A one- step approach to multi-task visual grounding

    Muchen Li and Leonid Sigal. Referring transformer: A one- step approach to multi-task visual grounding. Advances in Neural Information Processing Systems , 34:19652–19664,

  34. [42]

    Textbooks are all you need ii: phi-1.5 technical report, 2023

    Yuanzhi Li, S ´ebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023. 3

  35. [43]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, pages 7061–7070, 2023. 3

  36. [44]

    Belongie, Lubomir D

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312,

  37. [45]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 2, 3

  38. [46]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. 1, 2

  39. [47]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101, 2017. 6

  40. [48]

    Cascade grouped attention network for referring expression segmentation

    Gen Luo, Yiyi Zhou, Rongrong Ji, Xiaoshuai Sun, Jinsong Su, Chia-Wen Lin, and Qi Tian. Cascade grouped attention network for referring expression segmentation. InACM MM, pages 1274–1282, 2020. 3

  41. [49]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016. 6

  42. [50]

    The mapillary vistas dataset for semantic understanding of street scenes

    Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017. 6

  43. [51]

    Chatgpt: A language model for conversational ai

    OpenAI. Chatgpt: A language model for conversational ai. Technical report, OpenAI, 2023. 2, 3

  44. [52]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 2, 3

  45. [53]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. ArXiv, abs/2306, 2023. 3

  46. [54]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shao- han Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv:2306.14824, 2023. 3

  47. [55]

    Pedro H. O. Pinheiro, Ronan Collobert, and Piotr Doll ´ar. Learning to segment object candidates. In NeurIPS, 2015. 2

  48. [56]

    Pinheiro, Tsung-Yi Lin, Ronan Collobert, and Piotr Doll´ar

    Pedro O. Pinheiro, Tsung-Yi Lin, Ronan Collobert, and Piotr Doll´ar. Learning to refine object segments. In ECCV, 2016. 2

  49. [57]

    Multiscale combinatorial grouping for image segmentation and object proposal gener- ation

    Jordi Pont-Tuset, Pablo Arbelaez, Jonathan T Barron, Fer- ran Marques, and Jitendra Malik. Multiscale combinatorial grouping for image segmentation and object proposal gener- ation. IEEE transactions on pattern analysis and machine intelligence, 39(1):128–140, 2016. 2

  50. [58]

    Learning to segment every referring object point by point

    Mengxue Qu, Yu Wu, Yunchao Wei, Wu Liu, Xiaodan Liang, and Yao Zhao. Learning to segment every referring object point by point. In CVPR, pages 3021–3030, 2023. 3

  51. [59]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1

  52. [60]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 3

  53. [61]

    Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model. CVPR,

  54. [62]

    Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable train- ing deep learning models with over 100 billion parameters. In SIGKDD, 2020. 6 10

  55. [63]

    Pixellm: Pixel reasoning with large multimodal model

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, pages 26374–26383, 2024. 3, 5, 7

  56. [64]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 1

  57. [65]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2, 3

  58. [66]

    Selective search for object recognition

    Jasper RR Uijlings, Koen EA Van De Sande, Theo Gev- ers, and Arnold WM Smeulders. Selective search for object recognition. International journal of computer vision , 104 (2):154–171, 2013. 2

  59. [67]

    Llm-seg: Bridging image segmen- tation and large language model reasoning

    Junchi Wang and Lei Ke. Llm-seg: Bridging image segmen- tation and large language model reasoning. In CVPR, pages 1765–1774, 2024. 5

  60. [68]

    SegRefiner: Towards model- agnostic segmentation refinement with discrete diffusion process

    Mengyu Wang, Henghui Ding, Jun Hao Liew, Jiajun Liu, Yao Zhao, and Yunchao Wei. SegRefiner: Towards model- agnostic segmentation refinement with discrete diffusion process. In NeurIPS, 2023. 3

  61. [69]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. https://arxiv.org/abs/2305.11175) , 2023. 3, 5, 6

  62. [70]

    The all-seeing project: Towards panop- tic visual recognition and understanding of the open world

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhen- hang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panop- tic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907, 2023. 3

  63. [71]

    Non-local neural networks

    Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- ing He. Non-local neural networks. In CVPR, 2018. 2

  64. [72]

    SOLO: Segmenting objects by locations

    Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. SOLO: Segmenting objects by locations. In ECCV,

  65. [73]

    Solov2: Dynamic and fast instance segmentation

    Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chun- hua Shen. Solov2: Dynamic and fast instance segmentation. NeurIPS, 2020. 2

  66. [74]

    Images speak in images: A general- ist painter for in-context visual learning

    Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A general- ist painter for in-context visual learning. arXiv preprint arXiv:2212.02499, 2022. 6

  67. [75]

    Seggpt: Segmenting ev- erything in context

    Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting ev- erything in context. arXiv preprint arXiv:2304.03284, 2023. 6

  68. [76]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, pages 24824–24837. Curran Associates, Inc., 2022. 5

  69. [77]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In CVPR, pages 3858– 3869, 2024. 3, 5, 6

  70. [78]

    Alvarez, and Ping Luo

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and ef- ficient design for semantic segmentation with transformers. In NIPS, 2021. 1

  71. [79]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In CVPR, pages 2955–2966, 2023. 3

  72. [80]

    A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model. In ECCV, pages 736–753. Springer, 2022. 3

  73. [81]

    Fine-grained visual prompting, 2023

    Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting, 2023. 8

  74. [82]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 2, 3, 5, 6

  75. [83]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. NeurIPS, 36, 2023. 3

  76. [84]

    Osprey: Pixel un- derstanding with visual instruction tuning

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel un- derstanding with visual instruction tuning. In CVPR, pages 28202–28211, 2024. 6

  77. [85]

    Sigmoid loss for language image pre-training,

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training,

  78. [86]

    Gpt4roi: In- struction tuning large language model on region-of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: In- struction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023. 3

  79. [87]

    Controlvideo: Training-free controllable text-to-video generation

    Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023. 2

  80. [88]

    Groundhog: Grounding large language models to holistic segmentation

    Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In CVPR, 2024. 5

  81. [89]

    Psalm: Pixelwise segmentation with large multi-modal model

    Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model. In ECCV, pages 74–91. Springer, 2025. 3, 5, 6

  82. [90]

    Rethinking semantic segmen- tation from a sequence-to-sequence perspective with trans- formers

    Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmen- tation from a sequence-to-sequence perspective with trans- formers. In CVPR, 2021. 2

  83. [91]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 6

  84. [92]

    Seqtr: A simple yet universal network for visual grounding

    Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, 11 and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. ECCV, 2022. 3

  85. [93]

    category

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language mod- els. arXiv:2304.10592, 2023. 3 12 ROSE: Revolutionizing Open-Set Dense Segmentation with Patch-Wise Perceptual Large Multi...

  86. [94]

    User: <IMAGE,MASK> Please segment target region with mask and corre- sponding category

    missed detections: “ User: <IMAGE,MASK> Please segment target region with mask and corre- sponding category. ”. B. Visualizations B.1. Failure cases Fig. 7 shows the typical flaw caused by granularity differ- ences (“plant” and “tree”). Due to the inherent limitations of exist...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.