Pith. sign in

REVIEW 4 major objections 5 minor 66 references

What Holds Back Open-Vocabulary Segmentation?

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fixing mask choice alone adds 10 points to open-vocab segmentation

desk verdict A genuinely useful oracle-based decomposition of open-vocabulary segmentation bottlenecks—but the headline numbers rest on a single COCO→ADE20K transfer and a post-hoc inference tweak, so read the fix claims as hypotheses, not proven recipes. read the letter →

arxiv 2508.04211 v1 pith:WWFZWOZS submitted 2025-08-06 cs.CV

Josip Šarić , Ivan Martinović , Matej Kristan , Siniša Šegvić This is my paper · ORCID
classification cs.CV
keywords open-vocabularysegmentationpanopticvision-languagemodelsCLIPmasktransformersoracleexperimentsproposalgenerationno-objectclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that open-vocabulary segmentation methods built on mask transformers are not failing for one reason: they are held back by several independent bottlenecks, including vision-language models' region-level classification, mask proposal quality, and an inference-time mask selection step that discards valid masks. Using oracle components that replace parts of the pipeline with ground truth, the paper measures each bottleneck in isolation. With perfect masks, the best CLIP model reaches 41.8 PQ, nearly 8 points below an in-domain baseline. Fixing mask selection and dropping the no-object logit adds about 10 PQ points, and adding perfect classification raises the ceiling to 66.4 and 58.1 PQ for the two studied models. The paper also traces a large share of the remaining failures to annotation-policy conflicts between COCO and ADE20K, where models are trained to suppress exactly the masks the evaluation wants.

What carries the argument

The central mechanism is the oracle evaluation protocol: three ground-truth components are swapped into the standard mask-transformer pipeline one at a time. A segmentation oracle feeds ground-truth masks through MaskCLIP-extracted dense CLIP features with mask pooling to measure pure zero-shot region recognition; a classification oracle assigns the correct class to every predicted mask that overlaps a ground-truth segment with IoU > 0.5; and a mask selection oracle uses Hungarian matching to choose among the up to 250 candidate masks, optionally removing the learned no-object logit. The no-object embedding itself is a named central object: because it is trained only on the training taxonomy

What would settle it

Retrain or fine-tune a vision-language model with a dense, mask-level contrastive objective, then rerun the paper's segmentation oracle (perfect masks, mask-pooled features, CLIP text embeddings) on ADE20K validation; if PQ stays near the reported 41.8 ceiling rather than approaching the in-domain Mask2Former baseline, the region-level classification bottleneck is intrinsic to current VLMs, while a large jump would indicate the bottleneck was partly a feature-extraction artifact.

Watch

Extended reading notes

Core claim

The central claim is that the plateau in open-vocabulary segmentation comes from several independent, addressable bottlenecks rather than from a single architectural weakness. The oracle experiments show that vision-language models classify whole regions imperfectly even when given perfect masks; that mask proposal generators internally produce many valid masks which are then discarded because the learned no-object embedding, trained only on the closed training taxonomy, wins the classification; and that when oracle mask selection is paired with oracle classification, FC-CLIP and MAFT+ reach 66.4 and 58.1 PQ, far above their 26.8 and 26.9 baselines and above the in-domain Mask2Former model.

Load-bearing premise

The load-bearing premise is that MaskCLIP-style dense features extracted from frozen CLIP give a fair estimate of CLIP's region-level recognition ability; if those dense features are systematically worse than CLIP's true capacity, the finding that VLMs struggle with region-level classification may be an artifact of the extraction method.

Editorial extensions

If this is right

  • If the paper is right, open-vocabulary segmentation can improve substantially without new VLM pretraining: better mask selection and no-object calibration alone are worth about 10 PQ points.
  • Recognition is the binding constraint for unseen classes: oracle classification roughly doubles PQ_unseen for both studied models, so progress on region-level VLM features will directly raise open-vocabulary performance.
  • Even perfect classification leaves a gap to in-domain models, so mask proposal generation needs vocabulary-aware training rather than a fixed closed-set proposal head.
  • Annotation-policy conflicts mean part of the COCO-to-ADE20K gap is an artifact of label disagreement; unified label mappings or benchmark redesign would change reported numbers.
  • The oracle ceilings show current proposals already contain enough information to exceed in-domain baselines if selection and classification were solved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: the learned no-object logit acts as a training-set prior, so a test-time, vocabulary-conditioned proposal scorer could recover the discarded masks and capture the roughly 10-point gain the oracle identifies.
  • The segmentation-oracle result suggests a concrete pre-training objective: align CLIP features at the mask or region level rather than only at the image level; the oracle protocol could serve as a benchmark for measuring whether new VLMs close the 8-point region-classification gap.
  • Because the oracle replaces ground-truth masks without retraining, the measured ceilings likely depend on the specific backbone and proposal distribution; applying the same oracle battery to other mask-based open-vocabulary models would reveal whether these two bottlenecks are universal.
  • A testable extension: re-annotate a subset of ADE20K validation under COCO labeling rules and rerun the oracle experiments; to the extent PQ rises, the apparent open-vocabulary failure is partly benchmark misalignment rather than pure model failure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper analyzes why open-vocabulary segmentation methods built on mask transformers have plateaued on COCO→ADE20K. It introduces three oracle experiments: (i) a segmentation oracle that feeds ground-truth masks to MaskCLIP-extracted dense CLIP features (Table 1); (ii) a classification oracle that replaces the predicted class of any mask overlapping a ground-truth segment with IoU>0.5 by the correct label (Table 2); and (iii) a mask-selection oracle that selects candidate masks by Hungarian matching with the ground truth, with and without dropping the learned no-object logit (Tables 3–4). The main findings are that CLIP models still lag behind in-domain baselines even with perfect masks; oracle classification alone adds ~13 PQ; oracle mask selection alone hurts, but adding a no-object logit drop yields ~7–10 PQ gains; and oracle selection plus oracle classification reaches 66.4/58.1 PQ, well above in-domain baselines. The paper attributes the no-object bottleneck to labeling-policy conflicts between COCO and ADE20K and argues that mask proposal selection and region-level classification are the key limitations of current open-vocabulary models.

Significance. The oracle decomposition is a valuable and generally non-circular methodology: it bounds each component using ground truth and produces falsifiable predictions rather than fitting claims to data. The mask-selection/no-object result is striking and well supported by the qualitative examples in Figure 5. If it generalizes beyond COCO→ADE20K, it would refocus the field toward vocabulary-aware proposal generation and better no-object handling. However, the quantitative conclusions rest on a single transfer pair and on one particular dense-feature extraction strategy, so the paper currently establishes a strong hypothesis rather than a fully general diagnosis.

major comments (4)
  1. [§4.4, Tables 3–4, Findings 3–5] The central quantitative claims—the ~10 PQ gain from oracle mask selection plus no-object logit drop and the 66.4/58.1 PQ ceilings—are established only on COCO→ADE20K. Finding 5 itself attributes the no-object behavior to labeling-policy conflicts between COCO and ADE20K (paintings, sleeping pillows, non-stop traffic signs). If that attribution is correct, the no-object bottleneck is a property of this dataset pair, not an inherent limitation of mask-transformer open-vocabulary models; on a pair with more consistent taxonomies the gain could shrink substantially. Please add at least one additional transfer evaluation (e.g., COCO→Pascal VOC or COCO→Mapillary Vistas) or explicitly restrict the conclusions to the COCO→ADE20K setting. Without this, the claim that these are 'the key limitations in current open-vocabulary models' is overgeneralized.
  2. [§4.3, Table 3] The 'dropping no-object logit' row is a post-hoc inference change applied after oracle selection of candidate masks; it is not a trained component and cannot be deployed as such. The experiment demonstrates that valid masks are being discarded, but it does not by itself show that a real model with better selection/no-object handling would realize the full gain—removing the no-object logit without oracle selection would likely flood the panoptic output with false positives. The paper should clearly state that the gain is an upper bound and discuss what a learned selection/no-object module would need to achieve it. This is important because the abstract and Finding 4 phrase the result as a 'fix' rather than an oracle bound.
  3. [§4.1, Table 1] Finding 1 ('CLIP models struggle with region-level classification') is measured through MaskCLIP-extracted dense features, not through the VLMs' own region-level capacity directly. The paper itself notes in §4.1 that MaskCLIP gives unreliable dense features for ViT-L/14 and cites prior per-patch segmentation issues [28]. Thus the gap in Table 1 could be partly an artifact of the feature-extraction strategy rather than a fundamental VLM limitation. Please validate with at least one alternative dense-feature extraction method (e.g., attention refinement, decoder-based features, or the fine-tuned features used in MAFT+) or rephrase the finding to refer specifically to MaskCLIP-based features.
  4. [Tables 2 and 3] The baseline numbers for MAFT+ are inconsistent across tables: Table 2 lists MAFT+ baseline PQ_unseen = 19.5, while Table 3 lists the same baseline as PQ_unseen = 17.4 (PQ_all and PQ_seen are identical). If these are the same configuration and checkpoint, one of the entries is wrong; if they differ (e.g., due to different evaluation settings), the difference should be explained. The discrepancy undermines direct comparability of the two tables and should be resolved before publication.
minor comments (5)
  1. [§4.3] The Hungarian matching cost is only described as 'binary cross-entropy and Dice loss.' The exact cost weights, matching thresholds, and how the candidate set is formed (including the handling of the no-object class during matching) should be specified for reproducibility. These are free parameters that affect the oracle-selection results.
  2. [General] All oracle results are reported as single PQ values without error bars, seeds, or a statement about the number of runs. For an analysis paper with strong quantitative claims, please state whether the underlying checkpoints are fixed and deterministic and, if training is involved, report at least three runs or variance estimates.
  3. [§4.4, Figure 4] The '300 labeled images' comparison lacks detail on how the subset was selected (random seed, class balance, repeated sampling). Since this is a central argument about annotation cost, the sampling protocol should be described.
  4. [Table 5 caption] Typos: 'misssegmentation' and 'missclassification' should be 'mis-segmentation' and 'mis-classification'.
  5. [§3] The notation N' and the reused D dimension in Eq. (4) are slightly confusing; D denotes both feature dimension and number of candidates in P_CLIP ∈ R^{N'×D}. Please use distinct symbols for feature dimension and candidate count.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: oracle upper bounds are self-contained empirical measurements, not fitted predictions.

full rationale

The paper's central claims are established by direct oracle experiments: mask selection, mask classification, and segmentation are replaced by ground-truth-based operations, and the resulting panoptic quality is measured on a fixed benchmark. No parameter is fitted to a target and then reported as a prediction, and no derived quantity is equivalent to an input by construction. The only author-overlapping citation (Ref. [39]) is used as an example of mask pooling in related work and is not load-bearing. The COCO-to-ADE20K specificity and the potential unreliability of MaskCLIP dense features are validity and generality limitations that the paper itself acknowledges (Section 4.4, Finding 5), not circular reasoning. The no-object logit dropping is an explicit intervention whose effect is measured, not a fitted parameter renamed as a finding.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new free parameters fitted to support its claims; the oracle experiments use ground truth rather than fitted quantities. The main assumptions are domain-level: the oracle decomposition is valid, the benchmark is representative, MaskCLIP features are faithful, and two methods represent the paradigm. The unspecified Hungarian matching weights are the only hidden degree of freedom that could shift quantitative results.

free parameters (2)
  • mask binarization threshold = 0.5
    Used in mask pooling (Eq. 3) and in the standard FC-CLIP pipeline; chosen by hand, not fitted, but affects the masks used for CLIP pooling.
  • Hungarian matching cost weights
    Oracle mask selection uses a combination of binary cross-entropy and Dice loss (Section 4.3), but the weights are not specified; the oracle selection results may depend on this unspecified choice.
assumptions (4)
  • domain assumption Ground-truth oracle replacement isolates component-specific upper bounds
    The central decomposition assumes that perfecting one component (segmentation, classification, or mask selection) bounds the contribution of the other components without strong interaction effects. This is the methodological premise of the entire analysis.
  • domain assumption COCO-to-ADE20K is a representative open-vocabulary transfer evaluation
    All quantitative findings and the plateau claim are based on this single transfer pair. The paper argues it is standard, but the generality of the bottleneck conclusions is not tested on other benchmarks.
  • domain assumption MaskCLIP dense feature extraction faithfully reflects VLM dense perception
    The segmentation oracle in Section 4.1 relies on MaskCLIP features; the paper itself notes ViT-L/14 gives unreliable dense features, so the faithfulness of this extraction underlies Finding 1.
  • domain assumption FC-CLIP and MAFT+ represent the mask-transformer open-vocabulary paradigm
    The paper generalizes findings to the whole mask-transformer paradigm based on studying two methods. This is a reasonable but unproven sampling assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Holds Back Open-Vocabulary Segmentation?." pith.science (2026). https://pith.science/paper/WWFZWOZS

@misc{pith2026250804211,
  author       = {Pith},
  title        = {Pith review of: What Holds Back Open-Vocabulary Segmentation?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WWFZWOZS}},
  note         = {Machine review of arXiv:2508.04211}
}
read the original abstract

Standard segmentation setups are unable to deliver models that can recognize concepts outside the training taxonomy. Open-vocabulary approaches promise to close this gap through language-image pretraining on billions of image-caption pairs. Unfortunately, we observe that the promise is not delivered due to several bottlenecks that have caused the performance to plateau for almost two years. This paper proposes novel oracle components that identify and decouple these bottlenecks by taking advantage of the groundtruth information. The presented validation experiments deliver important empirical findings that provide a deeper insight into the failures of open-vocabulary models and suggest prominent approaches to unlock the future research.

Figures

Figures reproduced from arXiv: 2508.04211 by the authors.

Figure 1
Figure 1. Performance over time of open-vocabulary semantic (left) and panoptic (right) segmentation methods (COCO [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Standard mask-based pipeline for open-vocabulary segmentation, as proposed in FC-CLIP [ [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We estimate CLIP’s out-vocab recognition ceiling us [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance comparison of open-vocabulary mod [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Illustration of labeling policy conflicts between COCO and ADE20K that hinder open-vocabulary model performance. The rows [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 53 canonical work pages

  1. [28]

    Proxyclip: Proxy attention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. In European Conference on Computer Vision, pages 70–88. Springer, 2024. 5

    work page 2024

  2. [1]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11165–11174, 2023. 2

    work page 2023

  3. [2]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017. 2

    work page 2017

  4. [3]

    Encoder-decoder with atrous separable convolution for semantic image segmentation

    Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018. 2

    2018

  5. [4]

    Pali: A jointly- scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022. 2, 5

    arXiv 2022

  6. [5]

    Per- pixel classification is not all you need for semantic segmen- tation

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per- pixel classification is not all you need for semantic segmen- tation. Advances in neural information processing systems , 34:17864–17875, 2021. 2

    work page 2021

  7. [6]

    Schwing, Alexan- der Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1290–1299, 2022. 1, 2, 3, 5

    work page 2022

  8. [7]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023. 2, 4, 5

    work page 2023

Show all 66 references
  1. [8]

    Cat- seg: Cost aggregation for open-vocabulary semantic seg- mentation

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat- seg: Cost aggregation for open-vocabulary semantic seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4113– 4123, 2024. 2

    2024

  2. [9]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceed- ings of the IEEE conference on computer vision and pattern re...

    2016

  3. [10]

    Deepglobe 2018: A challenge to parse the earth through satellite images

    Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. Deepglobe 2018: A challenge to parse the earth through satellite images. In Proceedings of the IEEE conference on computer vision and pattern rec...

    2018

  4. [11]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11583–11592, 2022. 2

    2022

  5. [12]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  6. [13]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303–338, 2010. 2

    2010

  7. [14]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Representations, 2024. 2

    2024

  8. [15]

    Scal- ing open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. In European conference on computer vision , pages 540–557. Springer, 2022. 1

    2022

  9. [16]

    Fast r-cnn

    Ross Girshick. Fast r-cnn. In Proceedings of the IEEE inter- national conference on computer vision , pages 1440–1448,

  10. [17]

    Rich feature hierarchies for accurate object detection and semantic segmentation

    Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 580–587, 2014. 2

    2014

  11. [18]

    Open-vocabulary semantic segmentation with decou- pled one-pass network

    Cong Han, Yujie Zhong, Dengjie Li, Kai Han, and Lin Ma. Open-vocabulary semantic segmentation with decou- pled one-pass network. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 1086–1096, 2023. 2

    2023

  12. [19]

    Simultaneous detection and segmentation

    Bharath Hariharan, Pablo Arbel ´aez, Ross Girshick, and Ji- tendra Malik. Simultaneous detection and segmentation. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13, pages 297–312. Springer, 2014. 2

    2014

  13. [20]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 1, 2

    2017

  14. [21]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom 9 Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4...

  15. [22]

    Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation

    Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yun- chao Wei, and Humphrey Shi. Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation. In European Conference on Computer Vision, pages 399–416. Springer, 2024. 2, 3, 7, 8

    2024

  16. [23]

    Diffusion models for open-vocabulary segmen- tation

    Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for open-vocabulary segmen- tation. In European Conference on Computer Vision, pages 299–317. Springer, 2024. 2

    2024

  17. [24]

    Panoptic segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Doll ´ar. Panoptic segmentation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9404–9413, 2019. 1, 2

    2019

  18. [25]

    TIPS: Text-image pretraining with spatial awareness

    Kevis kokitsi Maninis, Kaifeng Chen, Soham Ghosh, Ar- jun Karpur, Koert Chen, Ye Xia, Bingyi Cao, Daniel Salz, Guangxing Han, Jan Dlabal, Dan Gnanapragasam, Mojtaba Seyedhosseini, Howard Zhou, and Andre Araujo. TIPS: Text-image pretraining with spatial awareness. In The Thir- ...

    2025

  19. [26]

    Ladder-style densenets for semantic segmentation of large natural im- ages

    Ivan Kreso, Sinisa Segvic, and Josip Krapac. Ladder-style densenets for semantic segmentation of large natural im- ages. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 238–245, 2017. 2

    2017

  20. [27]

    Clearclip: Decom- posing clip representations for dense vision-language infer- ence

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decom- posing clip representations for dense vision-language infer- ence. In European Conference on Computer Vision , pages 143–160. Springer, 2024. 2

    2024

  21. [29]

    Mask dino: Towards a unified transformer-based framework for object detection and segmentation

    Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Yeung Shum. Mask dino: Towards a unified transformer-based framework for object detection and segmentation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , p...

    2023

  22. [30]

    An inverse scal- ing law for clip training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scal- ing law for clip training. Advances in Neural Information Processing Systems, 36:49068–49087, 2023. 2

    2023

  23. [31]

    Scaling language-image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichten- hofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 23390– 23400, 2023. 2

    2023

  24. [32]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks. arXiv e-prints, pages arXiv–2304, 2023. 2

    2023

  25. [33]

    Mask-adapter: The devil is in the masks for open-vocabulary segmentation

    Yongkang Li, Tianheng Cheng, Bin Feng, Wenyu Liu, and Xinggang Wang. Mask-adapter: The devil is in the masks for open-vocabulary segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14998–15008, 2025. 2

    2025

  26. [34]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pa...

    2023

  27. [35]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pa...

    2023

  28. [36]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eu- ropean Conference on Computer Vision (ECCV), pages 740–

  29. [37]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 11976–11986,

  30. [38]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 3431–3440, 2015. 1, 2

    2015

  31. [39]

    Dearli: Decoupled enhancement of recog- nition and localization for semi-supervised panoptic segmen- tation

    Ivan Martinovi ´c, Josip ˇSari´c, Marin Or ˇsi´c, Matej Kristan, and Siniˇsa ˇSegvi´c. Dearli: Decoupled enhancement of recog- nition and localization for semi-supervised panoptic segmen- tation. arXiv preprint arXiv:2507.10118, 2025. 2

    2025 arXiv

  32. [40]

    Open vocabulary semantic segmentation with patch aligned con- trastive learning

    Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned con- trastive learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages ...

    2023

  33. [41]

    Silc: Improving vision language pretraining with self-distillation

    Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, and Federico Tombari. Silc: Improving vision language pretraining with self-distillation. In European Conference on Computer Vision, pages 38–55. Springer, 2024. 2

    2024

  34. [42]

    The mapillary vistas dataset for semantic understanding of street scenes

    Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision , pages 4990– 4999, 2017. 2

    2017

  35. [43]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  36. [44]

    Faster r-cnn: Towards real-time object detection with region 10 proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region 10 proposal networks. Advances in neural information process- ing systems, 28, 2015. 2

    2015

  37. [45]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In International Conference on Medical image com- puting and computer-assisted intervention (MICCAI), pages 234–241. Springer, 2015. 1, 2

    2015

  38. [46]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  39. [47]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 2

    2023 arXiv

  40. [48]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, ...

    2025 arXiv

  41. [49]

    Sclip: Rethink- ing self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethink- ing self-attention for dense vision-language inference. In European Conference on Computer Vision, pages 315–332. Springer, 2024. 2

    2024

  42. [50]

    Max-deeplab: End-to-end panoptic segmentation with mask transformers

    Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5463–5474, 2021. 2

    2021

  43. [51]

    Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions

    Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vi- si...

    2023

  44. [52]

    Clipself: Vision trans- former distills itself for open-vocabulary dense prediction

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision trans- former distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403, 2023. 2

    2023 arXiv

  45. [53]

    Sed: A simple encoder-decoder for open- vocabulary semantic segmentation

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3426–3436, 2024. 2

    2024

  46. [54]

    Demystify- ing clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystify- ing clip data. arXiv preprint arXiv:2309.16671, 2023. 2

    2023 arXiv

  47. [55]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 18134–18144, 2022. 2

    2022

  48. [56]

    Learning open-vocabulary seman- tic segmentation models from natural language supervision

    Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary seman- tic segmentation models from natural language supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2935–2944,

  49. [57]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 2955–2966, 2023. 2, 4

    2023

  50. [58]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2945– 2954, 2023. 2

    2023

  51. [59]

    k-means mask transformer

    Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means mask transformer. In European Conference on Computer Vision, pages 288–307. Springer, 2022. 2

    2022

  52. [60]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36:32215–32234,

  53. [61]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 18123–18133, 2022. 2

    2022

  54. [62]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 2

    2023

  55. [63]

    Pyramid scene parsing network

    Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017. 2

    2017

  56. [64]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122–5130, 2017. 1, 2

    2017

  57. [65]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Com- puter Vision, pages 696–712. Springer, 2022. 2, 4 11

    2022

  58. [755]

    Springer, 2014. 1, 2

    2014

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.