Pith. sign in

REVIEW 5 major objections 5 minor 40 references

Effective SAM Combination for Open-Vocabulary Semantic Segmentation

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ESC-Net claims that one-stage open-vocabulary semantic segmentation can reuse pretrained SAM decoder blocks, driven by pseudo prompts from CLIP correlation maps, to beat two-stage and prior one-stage methods at comparable cost.

desk verdict Convincing one-stage OVSS method with real gains from SAM decoder blocks; the unreported pseudo-prompt threshold and missing code are the only things keeping me from a clean yes. read the letter →

arxiv 2411.14723 v2 pith:2A55KLWG submitted 2024-11-22 cs.CV

Minhyeok Lee , Suhwan Cho , Jungho Lee , Sunghun Yang , Heeseung Choi , Ig-Jae Kim , Sangyoun Lee This is my paper · ORCID
classification cs.CV
keywords open-vocabularysemanticsegmentationsegmentanythingmodelCLIPpseudopromptsvision-languagefusionone-stagepixel-levelclassificationefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ESC-Net is a one-stage model for open-vocabulary semantic segmentation: it labels every pixel of an image using class names that were never seen during training. The paper's central claim is that the heavy mask-proposal stage of prior two-stage systems can be replaced by reusing the decoder blocks of SAM, the segment-anything model, and prompting them with pseudo points and masks read off from CLIP's image-text correlation map. This yields more spatially precise and denser masks than the previous one-stage correlation methods, and it does so at roughly the same inference cost as the CAT-Seg baseline rather than the much higher cost of running SAM's image encoder. On standard benchmarks the paper reports gains on every test set with both CLIP ViT-B/16 and CLIP ViT-L/14, for example A-847 mIoU rising from 16.0 to 18.1 and PC-459 from 23.8 to 27.0 against the prior best. If correct, this makes high-quality open-vocabulary segmentation practical at roughly one-stage cost.

What carries the argument

The load-bearing object is the SAM mask-decoder transformer block, reused with pretrained weights and wrapped together with a Pseudo Prompt Generator (PPG) into an 'ESC block'. PPG turns the CLIP correlation map for each class into a binarized, k-means-clustered set of candidate regions, then selects for each region the highest-probability point and the region mask as pseudo prompts; these enter through SAM's prompt encoder as sparse and dense prompt embeddings. The SAM block then applies prompt self-attention and bidirectional image-prompt cross-attention, updating CLIP's image features class-by-class in parallel batches. The paper's ablation attributes the performance jump to the pretrained SAM weights: with SAM blocks removed A-847 mIoU is 4.8, with randomly initialized SAM blocks it is 5.9, and with pretrained blocks it is 18.1.

What would settle it

A decisive experiment would take ESC-Net and swap in the decoder blocks of a different promptable segmentation model, or swap CLIP features for a different frozen vision-language encoder's features without retraining; if A-847 mIoU collapses to near the no-SAM baseline (4.8), then the gain is not generic promptable-decoder transfer but a CLIP-specific SAM adaptation. A cheaper check already in the paper: with randomly initialized SAM blocks the A-847 score is only 5.9, so any variant that keeps that score while removing the pretrained weights would refute the claim that pretrained SAM blocks carry the result.

Watch

Extended reading notes

Core claim

The paper's discovery is that the promptable segmentation machinery of SAM can be transplanted, with its pretrained weights, into a one-stage correlation pipeline built on CLIP. Instead of generating full candidate masks with SAM's image encoder and then classifying them, ESC-Net computes a per-class cosine-similarity map between CLIP image and text features, thresholds and clusters it into pseudo-object regions, and converts those regions into point and mask prompts. These prompts are embedded by SAM's prompt encoder and fed, together with CLIP's image features, into the pretrained SAM decoder blocks, which refine the CLIP features with class-agnostic spatial aggregation. A Vision-Language Fusion module then re-correlates the refined image features with text and produces the final masks. The authors claim this achieves state-of-the-art results on ADE20K, PASCAL-VOC, and PASCAL-Context with both CLIP ViT-B/16 and CLIP ViT-L/14, while keeping parameters (451.3M), inference time (0.76 s), and GFLOPs (2203.5) comparable to the much simpler CAT-Seg baseline rather than to the heavier two-stage pipelines.

Load-bearing premise

The entire performance gain rests on the assumption that SAM's pretrained decoder blocks, trained on SAM image features, still perform useful spatial aggregation when fed CLIP image features and pseudo prompts, even though they never saw CLIP features during pretraining.

Editorial extensions

If this is right

  • A dedicated mask proposal generator can be replaced by a pretrained SAM decoder fed with pseudo prompts, so open-vocabulary segmentation no longer has to pay for SAM's full image encoder at inference.
  • Because all prompt operations are batched per class, the model scales to hundreds of classes at roughly constant per-class marginal cost, limited mainly by the memory of the expanded correlation maps.
  • The method inherits CLIP's open vocabulary: new classes can be added by changing text prompts without retraining the segmentation head.
  • On the standard benchmarks, the reported gains hold across all six evaluation splits with both ViT-B/16 and ViT-L/14 backbones, so the benefit is not tied to one model scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the large gap between random-init (5.9) and pretrained (18.1) SAM blocks suggests the pretrained decoder is supplying a learned spatial prior about object boundaries, not just a trainable architecture; a further test would be to freeze the SAM blocks and see whether the prompt generator alone can adapt to CLIP features.
  • Editorial inference: because ESC-Net still refines low-resolution CLIP features, the paper's own limitation statement implies that pairing it with a lightweight boundary-refinement module could recover the boundary quality of two-stage methods without paying for their full cost.
  • Editorial inference: the recipe could transfer to other promptable segmentation decoders, and the transfer would be testable by swapping in a decoder pretrained on a different task or domain; if the performance jump over random init persists, the effect is generic promptable-decoder transfer, and if only SAM's decoder shows it, the effect is specific to SAM's training distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes ESC-Net, a one-stage open-vocabulary semantic segmentation model that combines a CLIP vision-language encoder with pre-trained SAM mask-decoder transformer blocks. Image-text correlation maps are binarized and clustered into pseudo points and pseudo masks, which are fed through SAM's prompt encoder and then into several pre-trained SAM blocks to refine CLIP image features. A Vision-Language Fusion (VLF) module further refines the correlation maps, and a U-Net-style decoder produces the final masks. The method is evaluated on ADE20K, PASCAL-VOC, and PASCAL-Context, reporting state-of-the-art mIoU with both CLIP ViT-B/16 and ViT-L/14 while maintaining inference time comparable to the one-stage baseline CAT-Seg. Ablations cover the effect of SAM blocks, prompt-type combinations, and model efficiency.

Significance. The core idea of reusing SAM decoder blocks with pseudo prompts generated from CLIP correlation maps is a plausible and potentially efficient alternative to two-stage mask-proposal pipelines. The random-init SAM control in Table 2 is an honest and informative baseline that strengthens the causal claim that the gain comes from pre-trained SAM weights rather than added parameters. If the reported results are reproducible, the method offers a meaningful improvement over CAT-Seg on several benchmarks. However, the manuscript currently lacks critical implementation details, particularly the binarization threshold alpha, and contains ambiguities in the SAM-block formulation, which prevent full validation of the central SOTA claim.

major comments (5)
  1. [Section 3.3, Pseudo Prompt Generator] The binarization threshold alpha is never reported. Section 3.3 states that the softmax probability mask is binarized 'using a predefined threshold value alpha' to produce pseudo masks and pseudo points, but no value or sensitivity analysis is given anywhere in Sections 4.3 or 4.5. Since Table 2 shows that the SAM blocks contribute the bulk of the accuracy (A-847 mIoU moves from 4.8 without SAM to 18.1 with pretrained SAM), the quality of the pseudo prompts is a load-bearing component of the reported SOTA numbers. Please report the value of alpha and include an ablation over a range of alpha values.
  2. [Section 3.4, Eq. (2)] Equation (2) appears inconsistent with the surrounding text. The text states that the SAM block applies prompt self-attention SA and bidirectional cross-attention BCA, but the equation writes (F_v^n)' = BCA(SA(F_l^n), F_v), where F_l^n is the n-th language feature vector. It does not show the embedded pseudo prompt features P_sparse^n and P_dense^n produced by the prompt encoder described in Section 3.3. This makes the exact computation of the SAM block ambiguous and unreproducible. Please clarify the notation and provide the precise computation that includes the prompt embeddings.
  3. [Abstract and Section 4.4, Table 4] The abstract claims ESC-Net 'outperforms prior methods in both efficiency and accuracy.' Table 4 shows that relative to CAT-Seg, ESC-Net has more parameters (451.3M vs 433.7M) and higher inference GFLOPs (2203.5 vs 2121.1); only inference time is marginally lower (0.76 s vs 0.78 s). The efficiency claim should be revised to 'comparable efficiency' or otherwise qualified, as the current wording is contradicted by the paper's own efficiency table.
  4. [Section 4.5 and Table 2] No error bars or multiple-seed results are reported in any table. Several reported improvements are small (e.g., +0.3 mIoU on PAS-20 with ViT-B/16 in Table 1), so without variance estimates it is unclear whether these gains, and the overall SOTA claim, are statistically meaningful. Please report means and standard deviations over at least three random seeds, or otherwise justify why a single run is sufficient.
  5. [Section 4.3, Implementation Details] The key hyperparameters N_o=5 pseudo prompts per class and N=4 ESC blocks are selected without any ablation. Since these directly control the prompt count, the number of SAM blocks, and the computational cost, the paper should include an ablation over N and N_o to justify the chosen configuration and to show the sensitivity of the reported results to these choices.
minor comments (5)
  1. [Table 3] The checkmark layout in Table 3 is garbled: row (e) is described in the text as using all three prompt types, but the table row shows only two checkmarks. Please fix the table so that each row clearly indicates which prompt types are used.
  2. [Table 2] The last column header in Table 2 reads 'PAS-20' but should likely be 'PAS-20b' for consistency with Table 1 and the dataset description in Section 4.1.
  3. [Section 4.1, Datasets] The sentence 'Unlike PC-59, PAS-20b is defined without including the "background" class' is confusing because the preceding sentences describe PASCAL-VOC and PASCAL-Context without establishing a contrast with PC-59. Please clarify the relationship between PAS-20b and PC-59.
  4. [Section 4.5, Figure 6] The qualitative activation maps in Figure 6 would be more convincing if accompanied by quantitative localization metrics or by the same visual comparison for the random-init SAM variant.
  5. [Limitations paragraph] The limitations paragraph is a useful addition, but the discussion of low-resolution feature levels could be tied more concretely to the mask-resolution behavior observed in the experiments, for example by reporting boundary metrics or high-resolution variants.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ESC-Net's results are empirical benchmark evaluations with no prediction derived from its own inputs.

full rationale

ESC-Net makes no formal derivation; its central claim is an empirical benchmark comparison (Table 1) after training on COCO-Stuff and evaluating on ADE20K, PASCAL-VOC, and PASCAL-Context. No equation in the paper defines the reported mIoU in terms of the method's own components; the pseudo-prompt generator, SAM blocks, and VLF module are architecture choices evaluated by controlled ablations (Tables 2 and 3), including a random-initialization control that isolates the effect of pretrained SAM parameters. The paper contains no load-bearing self-citations, and no uniqueness result or ansatz is imported from prior work by these authors. Concerns raised by the skeptic, such as the unreported binarization threshold alpha in Section 3.3 and the absence of released code, affect reproducibility and sensitivity rather than circularity: they do not make any reported result true by construction or equivalent to an input. The ablations do select hyperparameters (N=4, No=5, points and masks) on the same benchmarks used for the headline numbers, which is a real model-selection-on-test-data concern, but it is a correctness and reproducibility risk, not a derivation-level circularity. Under the stated criteria, no circular step can be quoted with a specific reduction, so the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are postulated. The PPG, SAM block, and VLF module are architectural components assembled from existing pretrained models and standard layers; none claims a falsifiable handle outside the benchmark scores themselves, so the invented-entity ledger is empty.

free parameters (6)
  • PPG binarization threshold alpha = not reported
    Binarizes softmaxed correlation maps into candidate object regions in Section 3.3; the value is called predefined but never stated, and it directly controls which regions become pseudo prompts.
  • Pseudo prompts per class No = 5
    Number of points and masks generated per class in the PPG, set in Section 4.3 and selected via ablation on the evaluation benchmarks.
  • Number of ESC blocks N = 4
    Number of consecutive pre-trained SAM decoder transformer blocks used, set in Section 4.3 and selected via ablation on the evaluation benchmarks.
  • CLIP and SAM learning rate = 2e-6
    Fine-tuning rate for CLIP encoders and SAM blocks, reported in Section 4.3.
  • Decoder and rest-of-model learning rate = 2e-4
    Learning rate for the VLF module and mask decoder, reported in Section 4.3.
  • Input image size = 336x336
    Resolution used for all training and evaluation, reported in Section 4.3.
assumptions (4)
  • domain assumption Pre-trained CLIP image and text encoders can be fine-tuned on COCO-Stuff without losing open-vocabulary discrimination.
    Adopted from CAT-Seg in Section 4.3; this underpins the entire one-stage training setup and is not re-validated in the paper.
  • domain assumption SAM decoder blocks pre-trained on SA-1B transfer spatial aggregation to CLIP's feature space when prompted with pseudo prompts.
    Central hypothesis of Section 3.4; Table 2 shows the entire gain over baseline comes from pretrained weights (18.1 vs 5.9 vs 4.8 on A-847), so this transfer assumption is load-bearing.
  • domain assumption Cosine similarity between CLIP image and text features is a usable proxy for pixel-class association.
    Basis of the correlation map C_v&l in Eq. 1 and of all downstream pseudo prompt generation; validated only indirectly through end-task accuracy.
  • standard math Standard components behave as published: softmax, k-means clustering, Swin transformer, linear attention, U-Net upsampling.
    Used without modification in Sections 3.3, 3.5, and 3.6; no new mathematical claims are made about them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Effective SAM Combination for Open-Vocabulary Semantic Segmentation." pith.science (2026). https://pith.science/paper/2A55KLWG

@misc{pith2026241114723,
  author       = {Pith},
  title        = {Pith review of: Effective SAM Combination for Open-Vocabulary Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2A55KLWG}},
  note         = {Machine review of arXiv:2411.14723}
}
read the original abstract

Open-vocabulary semantic segmentation aims to assign pixel-level labels to images across an unlimited range of classes. Traditional methods address this by sequentially connecting a powerful mask proposal generator, such as the Segment Anything Model (SAM), with a pre-trained vision-language model like CLIP. But these two-stage approaches often suffer from high computational costs, memory inefficiencies. In this paper, we propose ESC-Net, a novel one-stage open-vocabulary segmentation model that leverages the SAM decoder blocks for class-agnostic segmentation within an efficient inference framework. By embedding pseudo prompts generated from image-text correlations into SAM's promptable segmentation framework, ESC-Net achieves refined spatial aggregation for accurate mask predictions. ESC-Net achieves superior performance on standard benchmarks, including ADE20K, PASCAL-VOC, and PASCAL-Context, outperforming prior methods in both efficiency and accuracy. Comprehensive ablation studies further demonstrate its robustness across challenging conditions.

Figures

Figures reproduced from arXiv: 2411.14723 by the authors.

Figure 1
Figure 1. (a) A model structure that generates proposal masks using a mask generation model. (b) A model structure that refines the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The proposed ESC-Net consists of the CLIP vision and language encoders, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The process of the proposed Pseudo Prompt Generator [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Qualitative comparison of CAT-Seg and our ESC-Net across various datasets. Our model is capable of generating more accurate [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of image-text correlation maps with and [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 24 canonical work pages

  1. [1]

    Zero-shot semantic segmentation.Advances in Neural Information Processing Systems, 32, 2019

    Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick P´erez. Zero-shot semantic segmentation.Advances in Neural Information Processing Systems, 32, 2019. 2, 6

    work page 2019

  2. [2]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 1209–1218, 2018. 5

    2018

  3. [3]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 6

    arXiv 2015

  4. [4]

    Per- pixel classification is not all you need for semantic segmen- tation

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per- pixel classification is not all you need for semantic segmen- tation. Advances in neural information processing systems , 34:17864–17875, 2021. 1

    2021

  5. [5]

    Cat- seg: Cost aggregation for open-vocabulary semantic seg- mentation

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat- seg: Cost aggregation for open-vocabulary semantic seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4113– 4123, 2024. 2, 3, 5, 6, 8

    work page 2024

  6. [6]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583–11592, 2022. 3, 5, 6, 8

    work page 2022

  7. [7]

    Open- vocabulary universal image segmentation with maskclip

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open- vocabulary universal image segmentation with maskclip. arXiv preprint arXiv:2208.08984, 2022. 1, 3

    arXiv 2022

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3

    arXiv 2010

Show all 40 references
  1. [9]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303–338, 2010. 2, 5, 6

    2010

  2. [10]

    Scal- ing open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pages 540–557. Springer, 2022. 3, 5, 6

    2022

  3. [11]

    Open-vocabulary semantic segmentation with decou- pled one-pass network

    Cong Han, Yujie Zhong, Dengjie Li, Kai Han, and Lin Ma. Open-vocabulary semantic segmentation with decou- pled one-pass network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1086– 1096, 2023. 6

    2023

  4. [12]

    Algorithm as 136: A k-means clustering algorithm

    John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal sta- tistical society. series c (applied statistics) , 28(1):100–108,

  5. [13]

    Open-vocabulary instance segmentation via ro- bust cross-modal pseudo-labeling

    Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmentation via ro- bust cross-modal pseudo-labeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020–7031, 2022. 1

    2022

  6. [14]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  7. [15]

    Learning mask-aware clip representations for zero-shot segmentation

    Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. Learning mask-aware clip representations for zero-shot segmentation. Advances in Neural Information Processing Systems, 36:35631–35653, 2023. 6

    2023

  8. [16]

    Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation

    Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yun- chao Wei, and Humphrey Shi. Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation. arXiv preprint arXiv:2408.00744, 2024. 2, 3

    2024 arXiv

  9. [17]

    Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation

    Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yun- chao Wei, and Humphrey Shi. Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation. In European Conference on Computer Vision, pages 399–416. Springer, 2025. 6

    2025

  10. [18]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International confer- ence on machine learning, pages 5156–5165. PMLR, 2020. 5

    2020

  11. [19]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 202...

    2023

  12. [20]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Ren ´e Ranftl. Language-driven semantic seg- mentation. arXiv preprint arXiv:2201.03546, 2022. 3, 6

    2022 arXiv

  13. [21]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pag...

    2023

  14. [22]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3491– 3500, 2024. 6

    2024

  15. [23]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 5

    2021

  16. [24]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  17. [25]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recogni...

    2014

  18. [26]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...

    2019

  19. [27]

    Connecting vision and lan- guage with localized narratives

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and lan- guage with localized narratives. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part V 16, pages 647–66...

    2020

  20. [28]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  21. [29]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  22. [30]

    Open-vocabulary semantic segmentation with image embedding balancing

    Xiangheng Shan, Dongyue Wu, Guilin Zhu, Yuanjie Shao, Nong Sang, and Changxin Gao. Open-vocabulary semantic segmentation with image embedding balancing. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28412–28421, 2024. 6

    2024

  23. [31]

    Use: Universal segment embeddings for open-vocabulary image segmentation

    Xiaoqi Wang, Wenbin He, Xiwei Xuan, Clint Sebastian, Jorge Piazentin Ono, Xin Li, Sima Behpour, Thang Doan, Liang Gou, Han-Wei Shen, et al. Use: Universal segment embeddings for open-vocabulary image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion ...

    2024

  24. [32]

    Semantic projection network for zero-and few-label semantic segmentation

    Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8256–8265, 2019. 2, 6

    2019

  25. [33]

    Sed: A simple encoder-decoder for open- vocabulary semantic segmentation

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3426–3436, 2024. 2, 6

    2024

  26. [34]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 2955–2966, 2023. 3, 6

    2023

  27. [35]

    A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model. In European Conference on Computer Vi- sion, pages 736–753. Springer, 2022. 1, 3, 5, 6, 8

    2022

  28. [36]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2945– 2954, 2023. 3, 6

    2023

  29. [37]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36:32215–32234,

  30. [38]

    Open vocabulary scene parsing

    Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and An- tonio Torralba. Open vocabulary scene parsing. In Proceed- ings of the IEEE International Conference on Computer Vi- sion, pages 2002–2010, 2017. 2

    2002

  31. [39]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019. 2, 5, 6

    2019

  32. [40]

    Zegclip: Towards adapting clip for zero-shot se- mantic segmentation

    Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot se- mantic segmentation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 11175–11185, 2023. 3, 6

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.