Pith. sign in

REVIEW 3 major objections 4 minor 66 references

What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that open-vocabulary segmentation improves when a generative vision-language model names objects before locating them, reporting 27.2 PQ, 17.0 mAP, and 35.3 mIoU on ADE20K-150 together with vocabulary-free segmentation.

desk verdict A solid engineering paper with a clean architecture and plausible gains, but the headline 'conceive before perceive' story is not cleanly supported by its own ablation. read the letter →

arxiv 2505.19569 v1 pith:YBHA7WKJ submitted 2025-05-26 cs.CV

Jianghang Lin , Yue Hu , Jiangtao Shen , Yunhang Shen , Liujuan Cao , Shengchuan Zhang , Rongrong Ji This is my paper · ORCID
classification cs.CV
keywords open-vocabularysegmentationvocabulary-freegenerativevision-languagemodelpanopticsemanticconcept-awarevisualenhancementconfidencereweightingcognition-inspireddecoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that open-vocabulary image segmentation improves when the model first names the objects in an image (conceives) and only then locates them (perceives), mirroring a claimed human cognitive order. It proposes a framework that uses a generative vision-language model to produce a short list of object concepts with confidence scores, then feeds those concepts into two new modules: a Concept-Aware Visual Enhancer that fuses concept text with global image features, and a Cognition-Inspired Decoder that injects the concepts into learnable query masks via shared cross-attention. On standard benchmarks, the full system reports 27.2 PQ, 17.0 mAP, and 35.3 mIoU on ADE20K-150, 56.2 mIoU on Cityscapes, and 95.8 mIoU on PASCAL-20, also supporting vocabulary-free segmentation where no category list is given. If true, the central insight is that restricting category matching to a small, relevant set generated from the image itself is what unlocks the gains, rather than better pixel-level machinery alone.

What carries the argument

The machinery is the conceive-before-perceive pipeline. It has three interacting parts: (1) a generative vision-language model (G-VLM, e.g., Qwen2.5-VL) that, given the image and a prompt, outputs a small set of object concepts along with per-token averaged confidence scores; (2) a Concept-Aware Visual Enhancer (CAVE), a stack of six layers that alternately runs text-to-image cross-attention, image-to-text cross-attention, and deformable self-attention to let the concept embeddings and global visual features reshape each other, with a mask matrix to handle images that contain different concept subsets; and (3) a Cognition-Inspired Decoder (CID) whose learnable queries first attend to the concept embeddings and then to the enhanced visual features using the same shared cross-attention weights, forcing a common embedding space for semantics and pixels. The final classification is a softmax over the dot product of pooled mask embeddings with concept embeddings, reweighted by the exponentiated confidence of each generated concept. The load-bearing mechanism is the shared cross-attention and the confidence reweighting, which together let a small concept set steer both where masks are drawn and how they are named.

What would settle it

Re-run a strong open-vocabulary baseline (e.g., the strongest single-stage baseline that the paper retrains) with the same generative concept list, the same nearest-neighbor category mapping, and the same exponential confidence reweighting, but without the Concept-Aware Visual Enhancer and Cognition-Inspired Decoder. If that baseline matches or exceeds the reported 27.2 PQ and 35.3 mIoU on A-150, the proposed architecture contributes nothing beyond the generative model's external knowledge. Alternatively, replace the generated concepts with random or fixed categories; if the numbers stay high, the concepts themselves are not load-bearing.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the conventional two-step pipeline of segmenting regions first and then matching categories fails on novel categories because mask embeddings are trained only against a few seen categories, creating a semantic disconnect. The paper claims that reversing the order—first generating a handful of object concepts for the whole image with a generative vision-language model, then using those concepts to guide both feature enhancement and mask decoding—removes the disconnect and yields state-of-the-art open-vocabulary segmentation across distribution-shifted benchmarks. The authors attribute the improvement to the combination of three components: the G-VLM concept generation, the Concept-Aware Visual Enhancer with its text-to-image and image-to-text cross-attention and deformable self-attention, and the Cognition-Inspired Decoder with shared cross-attention over concept and visual features. They also report a confidence-reweighting step that scales category predictions by the exponentiated token-level confidence of each generated concept. The paper further shows vocabulary-free segmentation, where the G-VLM replaces any human-provided category list, and reports an Ideal upper bound with ground-truth categories that substantially exceeds the G-VLM-assisted results, indicating the remaining bottleneck is concept generation, not segmentation itself.

Load-bearing premise

The paper's superiority claim depends on the comparison in Table 2 being apples-to-apples, but no baseline is given the same Qwen2.5-VL concept list and confidence reweighting without the proposed enhancer and decoder; if most of the gain comes from the external knowledge of the generative vision-language model rather than from the proposed architecture, the framework-level conclusion is not established.

Editorial extensions

If this is right

  • The same trained model can perform vocabulary-free segmentation with no human-provided category list, achieving 32.7 mIoU on ADE20K-150 and 95.3 on PASCAL-20.
  • The gap between the G-VLM-assisted result (32.7 on ADE20K-150) and the ground-truth-category upper bound (52.0) implies that segmentation quality is currently limited by concept generation rather than by the mask decoder.
  • Because the framework trains only on COCO Panoptic and transfers to Cityscapes and Mapillary Vistas, the concept-first order appears to transfer across large domain shifts.
  • Reweighting with exponential confidence consistently helps across five benchmarks, with the choice of reweighting formula being otherwise robust.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the restrict-the-category-set mechanism is the real driver, gains should be largest on benchmarks where each image contains few categories relative to the total vocabulary, and smallest when images are category-dense; the reported A-847 (many categories) versus PAS-20 (few categories) numbers are consistent with this, but the paper does not test it directly.
  • A stronger G-VLM with higher recall (finding more of the true categories) should push the G-VLM-assisted numbers toward the reported Ideal upper bound; this suggests a concrete research direction of improving concept generation rather than segmentation.
  • The confidence reweighting uses only the average token probability; using the full token-level distribution or calibration might give a finer-grained correction and is a natural extension.
  • The paper's Ideal oracle experiment could be turned into a diagnostic: if ground-truth categories only close part of the gap, the residual error belongs to the segmentation modules, not to concept generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a cognition-inspired framework for open-vocabulary image segmentation that first uses a generative vision-language model (G-VLM, specifically Qwen2.5-VL) to generate a small set of object concepts for an image, then uses these concepts to guide a Concept-Aware Visual Enhancer (CAVE) and a Cognition-Inspired Decoder (CID), with an inference-time confidence reweighting that upweights categories predicted by the G-VLM. The framework is evaluated on panoptic, instance, and semantic segmentation benchmarks, reporting state-of-the-art or competitive numbers on A-150, Cityscapes, Mapillary Vistas, A-847, PC-59, PC-459, PAS-21, and PAS-20, and also supports vocabulary-free segmentation. The paper includes a three-mode evaluation (G-VLM-assisted, predefined, and ideal ground-truth categories), ablations of the main components, and a study of different reweighting formulas.

Significance. If the reported results are reliable, the paper introduces a plausible way to inject external semantic priors from modern generative VLMs into open-vocabulary segmentation, and the vocabulary-free setting with its ground-truth upper bound is a useful diagnostic. The evaluation is broad and the ablation structure is transparent. However, the central attribution is not established by the current experiments: the ablation in Table 3 shows that most of the improvement over the baseline comes from G-VLM concept generation plus confidence reweighting, while the proposed CAVE and CID modules add only small increments, and Table 2 does not compare against a same-backbone baseline that receives the same G-VLM concepts and reweighting. The idea is interesting and the results are plausible, but the paper's framing over-credits the trainable architecture components relative to the external G-VLM prior.

major comments (3)
  1. [§4.4 (Table 3)] In Table 3, row 2 shows that adding G-VLM concepts alone decreases PQ from 25.6 to 24.3 and mAP from 16.1 to 15.1; only after adding confidence reweighting (row 3) does performance rise to 26.8 PQ and 16.7 mAP. The text in §4.4 states that the G-VLM 'reduces the category matching from hundreds to a few, which greatly reduces the difficulty of identifying novel objects,' but row 2 contradicts this. The claimed benefit is therefore not separable from the inference-time reweighting of Eq. (6), and the wording should be corrected to describe the G-VLM-plus-reweighting combination as the source of the improvement. In addition, the increments from CAVE (row 3 to row 4) and CID (row 4 to row 5) are small: PQ +0.2/+0.2, mAP +0.1/+0.2, mIoU +0.4/+0.1. This does not support the abstract and introduction's emphasis on CAVE and CID as the core contributions; please quantify the per-component effect sizes and rephrase the attribution accordingly.
  2. [§4.3 (Table 2) vs. §4.4 (Table 3)] The comparison in Table 2 is not apples-to-apples because FC-CLIP, EOV-Seg, and other baselines do not receive the same Qwen2.5-VL concepts and confidence reweighting at inference time. Table 3 row 3 shows that a configuration without CAVE and CID already achieves 26.8 PQ and 34.6 mIoU on A-150, which is above FC-CLIP's 25.3 PQ and 32.9 mIoU (Table 2). Consequently, most of the reported improvement over prior methods could come from the external G-VLM prior and the reweighting formula rather than from the proposed enhancer and decoder. Please add a control configuration to Table 2, or to an equivalent table, that applies the same G-VLM concept generation and confidence reweighting to a standard Mask2Former/FC-CLIP-style segmentation head on every benchmark, and state explicitly which portion of the gain is attributable to CAVE and CID.
  3. [§4.2-4.3 and Tables 1-2] The reported numbers are inconsistent across locations: Section 4.2 states 35.1% mIoU on A-150 and 59.1% on PC-59 in the Predefined setting, while Table 1, the abstract, and Table 4 report 35.3 and 59.2; Section 4.3 reports A-847 mIoU of 15.3, while Table 2 and the abstract report 15.4. These discrepancies must be reconciled before the results can be considered reliable.
minor comments (4)
  1. [§3.2.2, Eq. (2)] In Eq. (2), the projection matrices are listed as 'W_q, W_k and W_q'; the last should be W_v.
  2. [§4.4, Figure 4] The text says 'six benchmarks' but then lists seven: A-847, A-150, COCO, PAS-20, PAS-21, PC-459, and PC-59; please correct the count or the list.
  3. [§4.1] The implementation details contain a typo: 'the shorted side' should be 'the shorter side'.
  4. [References] Several references are duplicated (e.g., MaskCLIP [11] and [12], OVSeg [30] and [31], CLIP [41] and [42], ODISE [52] and [53], ALIGN [19] and [20]), which makes the bibliography harder to follow; please consolidate them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning: the method's components are evaluated on external benchmarks and no claimed prediction reduces to fitted inputs or self-citations.

full rationale

The paper's framework is an empirical system built from a frozen G-VLM (Qwen2.5-VL), a frozen CLIP encoder, and trainable enhancer/decoder modules; the claimed results are measured on held-out benchmarks (A-150, Cityscapes, etc.) under standard protocols, so there is no sense in which a predicted quantity is defined in terms of the target quantity. The confidence reweighting in Eq. (6)-(7) uses token probabilities from the G-VLM as an external signal, not values fitted to the evaluation labels. The 'Ideal' setting with ground-truth categories is explicitly described as an upper-bound analysis, not a prediction. The ablation in Table 3 does show that G-VLM concepts alone (row 2) reduce PQ relative to the baseline, and the full gain appears only after adding confidence reweighting; however, this is an attribution/experimental-design concern about isolating component contributions, not circular reasoning. The paper itself flags G-VLM limitations (Sec. 3.2.1 and Sec. 4.4) and a PAS-21 drop (Sec. 4.3); these are acknowledged constraints and do not constitute circularity. No load-bearing step is justified by a self-citation whose authors overlap with the present paper; prior work is cited only for standard components (Mask2Former, OpenCLIP, CLIP, Qwen2.5-VL). Numeric inconsistencies in the text (e.g., 35.1 vs. 35.3 mIoU) are reporting errors, not circularity. Therefore the derivation chain is self-contained with respect to the benchmark measurements.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central performance claims rely on the quality of external G-VLM concepts, on CLIP text space as a mapping metric, and on the frozen backbone's alignment. The proposed modules add trainable components but introduce no new physical or conceptual entities with independent falsifiable handles. The main uncontrolled free choices are the loss weights, architecture depths, and reweighting form.

free parameters (4)
  • Loss weights lambda_1, lambda_2, lambda_3 = 2.0, 5.0, 5.0
    Used in Eq. 5 to balance classification, pixel, and dice losses; chosen without a reported sensitivity study.
  • Number of CAVE stacks N and decoder stacks M = N=6, M=9
    Architecture depths inherited from Mask2Former and Detrex settings; no ablation on these counts is reported.
  • Confidence reweighting formula = W_i = e^{C_i}
    Selected from four variants in Table 4; the table shows all variants perform similarly, so this parameter has low sensitivity.
  • Inference shorter-side resize = 800, with 1024 for Cityscapes and Mapillary Vistas
    Standard evaluation detail that can affect mIoU; no sensitivity analysis is reported.
assumptions (4)
  • domain assumption The G-VLM (Qwen2.5-VL) produces a small set of relevant, non-redundant object concepts whose token-level confidence scores are usable for reweighting.
    Invoked in Eq. 1 and Eq. 6; the whole framework assumes the generated concept list is short and accurate enough to help rather than mislead classification.
  • domain assumption CLIP text embedding space is a reliable metric space for mapping generated concepts to nearest test categories.
    Section 3.2.4: each G-VLM concept is mapped to its nearest neighbor in the CLIP text encoder's space; this mapping is not ablated.
  • domain assumption A frozen OpenCLIP ConvNeXt-L visual backbone plus CLIP text encoder provides sufficient vision-language alignment for mask classification in the proposed pipeline.
    Section 4.1: the backbone is frozen, so the method inherits all alignment quality from these pretrained models.
  • ad hoc to paper Sharing cross-attention parameters between text and visual modalities enforces modality-invariant representations and improves novel-category segmentation.
    Section 3.2.3 asserts this; no theoretical derivation or controlled comparison of shared versus separate weights is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation." pith.science (2026). https://pith.science/paper/YBHA7WKJ

@misc{pith2026250519569,
  author       = {Pith},
  title        = {Pith review of: What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YBHA7WKJ}},
  note         = {Machine review of arXiv:2505.19569}
}
abstract

Open vocabulary image segmentation tackles the challenge of recognizing dynamically adjustable, predefined novel categories at inference time by leveraging vision-language alignment. However, existing paradigms typically perform class-agnostic region segmentation followed by category matching, which deviates from the human visual system's process of recognizing objects based on semantic concepts, leading to poor alignment between region segmentation and target concepts. To bridge this gap, we propose a novel Cognition-Inspired Framework for open vocabulary image segmentation that emulates the human visual recognition process: first forming a conceptual understanding of an object, then perceiving its spatial extent. The framework consists of three core components: (1) A Generative Vision-Language Model (G-VLM) that mimics human cognition by generating object concepts to provide semantic guidance for region segmentation. (2) A Concept-Aware Visual Enhancer Module that fuses textual concept features with global visual representations, enabling adaptive visual perception based on target concepts. (3) A Cognition-Inspired Decoder that integrates local instance features with G-VLM-provided semantic cues, allowing selective classification over a subset of relevant categories. Extensive experiments demonstrate that our framework achieves significant improvements, reaching $27.2$ PQ, $17.0$ mAP, and $35.3$ mIoU on A-150. It further attains $56.2$, $28.2$, $15.4$, $59.2$, $18.7$, and $95.8$ mIoU on Cityscapes, Mapillary Vistas, A-847, PC-59, PC-459, and PAS-20, respectively. In addition, our framework supports vocabulary-free segmentation, offering enhanced flexibility in recognizing unseen categories. Code will be public.

Figures

Figures reproduced from arXiv: 2505.19569 by the authors.

Figure 1
Figure 1. (a) shows traditional open vocabulary segmentation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. In the traditional open vocabulary image segmen [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of our Cognition-Inspired Framework. This framework follows the process of human visual recognition, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of Precision and Recall across Different Datasets for Four Vision-Language Models (BLIP2 [ [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visualization of K-means clustering of V𝑔, V𝑠𝑎 without our Concept-Aware Visual Enhancer (replace it with Pixel Decoder [7]) and V𝑠𝑎 with our Concept-Aware Visual Enhancer. highlight Qwen2.5-VL’s superior generalization in vision-language alignment for both detection a…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 40 canonical work pages

  1. [1]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report.arXiv preprint arXiv:2309.16609(2023)

    arXiv 2023

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923(2025)

    arXiv 2025

  3. [3]

    Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. 2019. Zero- shot semantic segmentation.Advances in Neural Information Processing Systems 32 (2019)

    work page 2019

  4. [4]

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3558–3568

    work page 2021

  5. [5]

    Xi Chen, Shuang Li, Ser-Nam Lim, Antonio Torralba, and Hengshuang Zhao

    work page

  6. [6]

    Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. 2020. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12475–12485

    work page 2020

  7. [7]

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. Masked-attention mask transformer for universal image segmen- tation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1290–1299

    2022

  8. [8]

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. 2021. Per-pixel classifica- tion is not all you need for semantic segmentation.Advances in neural information processing systems34 (2021), 17864–17875

    work page 2021

Show all 66 references
  1. [9]

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. 2024. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4113–4123

    2024

  2. [10]

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus En- zweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The Cityscapes Dataset for Semantic Urban Scene Understanding. InCVPR

    2016

  3. [11]

    Zheng Ding, Jieke Wang, and Zhuowen Tu. 2022. Open-vocabulary universal image segmentation with maskclip.arXiv preprint arXiv:2208.08984(2022)

    2022 arXiv

  4. [12]

    Zheng Ding, Jieke Wang, and Zhuowen Tu. 2023. Open-Vocabulary Universal Image Segmentation with MaskCLIP. InInternational Conference on Machine Learning. PMLR, 8090–8102

    2023

  5. [13]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2010. The Pascal Visual Object Classes (VOC) Challenge.IJCV(2010)

    2010

  6. [14]

    Mark J Fenske, Elissa Aminoff, Nurit Gronau, and Moshe Bar. 2006. Top-down facilitation of visual object recognition: object-based and context-based contribu- tions.Progress in brain research155 (2006), 3–21

    2006

  7. [15]

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. InProceedings of the IEEE international conference on computer vision. 2961–2969

    2017

  8. [16]

    Jie Hu, Linyan Huang, Tianhe Ren, Shengchuan Zhang, Rongrong Ji, and Liujuan Cao. 2023. You only segment once: Towards real-time panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17819–17829

    2023

  9. [17]

    Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. 2016. Segmentation from natural language expressions. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I

    2016

  10. [18]

    2021.OpenCLIP

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021.OpenCLIP. doi:10.5281/zenodo.5143773 If you use this software...

    2021 doi

  11. [19]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. InICML

    2021

  12. [20]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- language representation learning with noisy text supervision. InInternational conference on machine learning. PMLR, 4904–4916

    2021

  13. [21]

    Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yunchao Wei, and Humphrey Shi. 2024. Collaborative vision-text representation optimizing for open-vocabulary segmentation. InEuropean Conference on Computer Vision. Springer, 399–416

    2024

  14. [22]

    Yasufumi Kawano and Yoshimitsu Aoki. 2024. Tag: Guidance-free open- vocabulary semantic segmentation.IEEE Access(2024)

    2024

  15. [23]

    Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. 2022. Restr: Convolution-free referring image segmentation using transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 18145–18154

    2022

  16. [24]

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dol- lár. 2019. Panoptic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9404–9413

    2019

  17. [25]

    Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data? https://llava-vl.github.io/blog/2024-05- 25-llava-next-ablations/

    2024

  18. [26]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning. PMLR, 19730–19742

    2023

  19. [27]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning. PMLR, 12888–12900

    2022

  20. [28]

    Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. 2024. Omg-seg: Is one model good enough for all segmentation?. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 27948–27959

    2024

  21. [29]

    Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. 2022. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1280–1289

    2022

  22. [30]

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023. Open-vocabulary semantic segmentation with mask-adapted clip. InCVPR

    2023

  23. [31]

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023. Open-vocabulary semantic segmentation with mask-adapted clip. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. ...

    2023

  24. [32]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, procee...

    2014

  25. [33]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning.Advances in neural information processing systems36 (2023), 34892–34916

    2023

  26. [34]

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11976–11986

    2022

  27. [35]

    Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101(2017)

    2017 arXiv

  28. [36]

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. 2014. The Role of Context for Object Detection and Semantic Segmentation in the Wild. InCVPR

    2014

  29. [37]

    Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder

  30. [38]

    Hongwei Niu, Jie Hu, Jianghang Lin, Guannan Jiang, and Shengchuan Zhang

  31. [39]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El- Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193(2023)

    2023 arXiv

  32. [40]

    Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al . 2023. FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation. InCVPR

    2023

  33. [41]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InICML

    2021

  34. [42]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learnin...

    2021

  35. [43]

    Klara Reichard, Giulia Rizzoli, Stefano Gasperini, Lukas Hoyer, Pietro Zanuttigh, Nassir Navab, and Federico Tombari. 2025. From Open-Vocabulary to Vocabulary- Free Semantic Segmentation.arXiv preprint arXiv:2502.11891(2025)

    2025 arXiv

  36. [44]

    Tianhe Ren, Shilong Liu, Feng Li, Hao Zhang, Ailing Zeng, Jie Yang, Xingyu Liao, Ding Jia, Hongyang Li, He Cao, Jianan Wang, Zhaoyang Zeng, Xianbiao Qi, Yuhui Yuan, Jianwei Yang, and Lei Zhang. 2023. detrex: Benchmarking Detection Transformers. arXiv:2306.07265 [cs.CV]

    2023 arXiv

  37. [45]

    Pitchaporn Rewatbowornwong, Nattanat Chatthee, Ekapol Chuangsuwanich, and Supasorn Suwajanakorn. 2023. Zero-guidance segmentation using zero segment labels. InProceedings of the IEEE/CVF International Conference on Computer Vision. 1162–1172

    2023

  38. [46]

    Xiangheng Shan, Dongyue Wu, Guilin Zhu, Yuanjie Shao, Nong Sang, and Changxin Gao. 2024. Open-vocabulary semantic segmentation with image em- bedding balancing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 28412–28421

    2024

  39. [47]

    Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. Yfcc100m: The new data in multimedia research.Commun. ACM59, 2 (2016), 64–73

    2016

  40. [48]

    Osman Ülger, Maksymilian Kulicki, Yuki Asano, and Martin R Oswald. 2023. Auto-vocabulary semantic segmentation.arXiv preprint arXiv:2312.04539(2023)

    2023 arXiv

  41. [49]

    Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen

  42. [50]

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. 2022. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11686–11695

    2022

  43. [51]

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. 2022. Groupvit: Semantic segmentation emerges from text supervision. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 18134–18144

    2022

  44. [52]

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. 2023. Open-vocabulary panoptic segmentation with text-to-image diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2955–2966

    2023

  45. [53]

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. 2023. Open-vocabulary panoptic segmentation with text-to-image diffusion models. InCVPR

    2023

  46. [54]

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. 2023. Side adapter network for open-vocabulary semantic segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2945–2954

    2023

  47. [55]

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. 2022. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. InECCV

    2022

  48. [56]

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. 2022. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. InEuropean Conference on Computer Vision. Springer, 736–753

    2022

  49. [57]

    Xin Xu, Tianyi Xiong, Zheng Ding, and Zhuowen Tu. 2023. MasQCLIP for Open-Vocabulary Universal Image Segmentation. InICCV

    2023

  50. [58]

    Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018. Mattnet: Modular attention network for referring ex- pression comprehension. InProceedings of the IEEE conference on computer vision and pattern recognition. 1307–1315

    2018

  51. [59]

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. 2023. Convolutions die hard: Open-vocabulary segmentation with single frozen con- volutional clip.Advances in Neural Information Processing Systems36 (2023), 32215–32234

    2023

  52. [60]

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. 2023. Recognize Anything: A Strong Image Tagging Model.arXiv preprint arXiv:2306.03514(2023)

    2023 arXiv

  53. [61]

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ade20k dataset. InCVPR

    2017

  54. [62]

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. 2023. Generalized decoding for pixel, image, and language. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15116–15127

    2023

  55. [2017]

    InProceedings of the IEEE international conference on computer vision

    The mapillary vistas dataset for semantic understanding of street scenes. InProceedings of the IEEE international conference on computer vision. 4990–4999

  56. [2021]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Max-deeplab: End-to-end panoptic segmentation with mask transform- ers. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5463–5474

  57. [2023]

    In Proceedings of the IEEE/CVF International Conference on Computer Vision

    Open-vocabulary panoptic segmentation with embedding modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1141– 1150

  58. [2024]

    Eov-seg: Efficient open-vocabulary panoptic segmentation.arXiv preprint arXiv:2412.08628(2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.