Pith. sign in

REVIEW 4 major objections 5 minor 69 references

DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding Stable Diffusion-generated visual prompts to CLIP text embeddings closes the text-image gap and yields state-of-the-art open-vocabulary segmentation.

desk verdict DPSeg's dual-prompt idea is genuinely new, but the SOTA claim rests on template counts chosen on the test sets and a transductive second pass, so the headline numbers are not trustworthy as reported. read the letter →

arxiv 2505.11676 v1 pith:ZXWJQJCP submitted 2025-05-16 cs.CV

Ziyu Zhao , Xiaoguang Li , Linjia Shi , Nasrin Imanpour , Song Wang This is my paper · ORCID
classification cs.CV
keywords open-vocabularysemanticsegmentationdual-promptcostvolumevisualpromptingStableDiffusionCLIPvolume-guideddecodersemantic-guidedpromptrefinementmulti-scalefeaturefusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that open-vocabulary segmentation is held back by the text-image modality gap in CLIP's embedding space, and that this gap can be largely bridged by generating a visual prompt for each category with Stable Diffusion and averaging it with the text prompt before computing pixel-level cost volumes. The resulting DPSeg system reports improved mIoU over prior cost-volume methods on ADE20K-847, Pascal Context-459, ADE20K-150, Pascal Context-59, and Pascal VOC, with a second inference pass that crops detected objects and uses them as refined visual prompts adding further gains. A sympathetic reader would take away that synthetic visual prompts are a practical, cheap source of category prototypes that improves both seen and unseen class segmentation.

What carries the argument

The load-bearing mechanism is the dual-prompt cost volume: a pixel-level cosine-similarity map between image features and a unified prompt embedding formed by averaging CLIP text embeddings with Stable Diffusion visual embeddings for each category. Around it, DPSeg adds (1) multi-scale visual cost volumes $F_v^j$ that inject intra-modality, category-level similarity at 4x, 8x, and 16x downsampling without upsampling the initial cost volume, and (2) a two-pass semantic-guided refinement in which the first pass's segmentation masks crop detected objects and substitute them as the visual prompts for the second pass.

What would settle it

Construct a rare-category subset of a benchmark like ADE20K-847, score each Stable Diffusion visual prompt for visible fidelity, and check whether DPSeg's per-category IoU drops below a text-only cost volume baseline whenever the synthetic prompt is obviously misrendered; if it does not, the faithfulness premise is not the actual driver, and if it does, the premise is violated.

Watch

Extended reading notes

Core claim

The paper establishes that visual prompts live closer to image features than text prompts do in CLIP space, and that this intra-modality proximity translates into cost volumes with sharper spatial-semantic structure. On that basis, DPSeg defines the dual-prompt cost volume $F_c(x,y,k,m)$ as the cosine similarity between the image feature at each location and the averaged text-plus-visual prompt embedding $R = \mathrm{Avg}(V+T)$ (Eq. 1), and additionally derives multi-scale visual cost volumes $F_v^j$ from global-average-pooled visual prompt prototypes matched against intermediate image features (Eq. 2). A cost volume-guided decoder consumes these volumes stage by stage, and a semantic-guided refinement pass replaces generic visual prompts with cropped objects from the first segmentation output. In the paper's reported numbers, this combination outperforms the compared methods on all five benchmarks in both base and large backbone configurations.

Load-bearing premise

Stable Diffusion-generated images faithfully represent every queried category, including rare unseen ones, so averaging their embeddings with text embeddings improves rather than pollutes the prompt signal.

Editorial extensions

If this is right

  • Synthetic text-to-image generation becomes a direct source of category prototypes, reducing dependence on hand-collected reference images for unseen classes.
  • Averaging prompt embeddings before similarity, rather than fusing separately computed cost volumes, is the better fusion strategy for text and visual prompts.
  • A second inference pass that re-prompts with detected object crops yields consistent gains over a single pass, so test-time self-refinement is a viable accuracy lever.
  • Multi-scale visual cost volumes preserve small-object detail better than naive cost volume upsampling, giving a concrete decoder design for fine-grained open-vocabulary segmentation.
  • Even degraded visual prompts keep the method above a text-only baseline, suggesting the dual-prompt design is not brittle to prompt quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: per-category gains should track Stable Diffusion's fidelity for that category, so a confidence-weighted blend of text and visual prompts could rescue classes whose synthetic renderings are poor.
  • If the two-pass refinement works because the second pass sees instance-level crops, the same idea could be applied to open-vocabulary detection or panoptic segmentation, where detected masks already provide instance-level prompts; the paper only evaluates semantic segmentation.
  • The modality-gap diagnosis is independent of the decoder, so the same dual-prompt embeddings could be plugged into other cost-volume architectures, with the decoder contribution being additive rather than strictly necessary.
  • The paper's numbers are dataset averages; reporting per-category IoU broken down by generation quality would turn the faithfulness assumption into a testable design guide.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DPSeg, an open-vocabulary semantic segmentation method that generates visual prompts by running Stable Diffusion on text templates, averages the visual and text prompt embeddings in CLIP space, computes a pixel-level cost volume, and feeds it along with multi-scale visual cost volumes into a cost volume-guided decoder. A two-pass semantic-guided refinement replaces visual prompts with crops from first-pass predicted masks. The authors report state-of-the-art mIoU on A-847, PC-459, A-150, PC-59, and PAS-20 in both ConvNeXt-B and ConvNeXt-L configurations.

Significance. If the reported results are reproducible, the paper makes a useful contribution: it provides empirical evidence (Figs. 2 and 3) that visual prompts reduce the CLIP modality gap for OVSS, and the ablations on prompt strategies, cost volume fusion, and decoder guidance are well designed. The proposed architecture is coherent, and the reported gains are internally plausible for a method that adds intra-modal cues and multi-scale cost volumes. The value of the work is, however, currently contingent on fixing the test-set selection of the template count, providing a fair comparison for the two-pass inference, and correcting the numerical inconsistencies in the experimental section.

major comments (4)
  1. [§5.5, Table 4] The number of prompt templates M is ablated directly on the five test benchmarks and M=80 is selected because it gives the best test mIoU; no validation split or fixed template budget is described. This is load-bearing: at M=40 the ConvNeXt-B configuration scores 10.1/16.8/29.6/55.9/91.8 on A-847/PC-459/A-150/PC-59/PAS-20, below the SED baseline (11.4/18.6/31.6/57.3/94.4) in Table 1, while at M=80 it exceeds SED. The claimed advantage over the strongest baseline therefore appears to be an artifact of test-set ablation unless a validation-based protocol for choosing M is provided.
  2. [§4.3, Table 1] The headline 'Inference II' results are obtained from a two-pass procedure in which the first-pass predicted masks on the test image are used to crop and replace the visual prompts for the second pass. This makes the comparison with single-pass methods such as SED, CAT-Seg, and SAN not apples-to-apples, because the method is effectively using test-image content (its own predictions) to refine itself. The authors should report Inference I as the primary result in the abstract and conclusion, or add a comparison in which all methods are evaluated under a comparable single-pass protocol.
  3. [§5.3, Table 1] The quantitative reporting is internally inconsistent. Section 5.3 claims gains of +1.1%, +1.5%, +2.5%, and +1.1% over SED on 'A-187', PC-459, A-150, and PC-59, but Table 1 shows Inference I ConvNeXt-B deltas of +0.6, +0.9, +1.3, and +0.8 (plus +1.6 on PAS-20), and 'A-187' is not a dataset used in the paper. The claimed average Inference-II gain of +0.87% over Inference I is not supported by Table 1, whose per-dataset gains are 0.5/0.6/0.4/0.3/0.9 (mean 0.54); the large-backbone gain of +0.92% is also unsupported (mean 0.70). These discrepancies must be corrected before the results can be taken as reliable.
  4. [§5.1, §5.2, §5.5] The paper does not provide code, a release of the visual prompt bank, or multi-seed results with error bars. The visual prompt bank (M=80 Stable Diffusion images per category) is a core input to the method, and without it the exact reported numbers cannot be reproduced or independently verified. Please release the prompt bank and code, and report mean and standard deviation over at least three training runs for the main configurations.
minor comments (5)
  1. [§2.3, References] The citation for PFENet appears to be incorrect: in Sec. 2.3, PFENet is cited as [68], but reference [68] is 'Extract free dense labels from CLIP' by Zhou et al.; the PFENet paper is not in the reference list.
  2. [Table 1, References [26] and [27]] The SCAN paper is cited twice as [26] and [27] with different venues; please consolidate to a single reference.
  3. [§3.1, Fig. 2] The cosine similarity experiment is described only as using 'more than one hundred samples'; please specify the number of images, the number of categories, and whether any error bars or confidence intervals are shown, since Fig. 2 appears to show raw scatter points.
  4. [Fig. 1] The t-SNE figure reports distances without stating which embedding pairs are averaged or how the image feature is obtained, which makes the headline distance of 0.18 hard to interpret.
  5. [§5.5, Table 3] The text says 'an average increase of 1.8% mIoU' when all scales are incorporated, but the row deltas in Table 3 average 1.3; please reconcile this number with the table.

Circularity Check

0 steps flagged · score 1.0 of 10

No derivation-level circularity: DPSeg's cost volumes are defined similarities to externally generated prompts, and the headline gains are measured on external benchmarks; only minor non-load-bearing self-citations and a test-set template-selection concern appear.

full rationale

The derivation chain is self-contained at the equation level. Visual prompts are generated by a frozen Stable Diffusion model from text templates (Sec. 4.1), the dual-prompt representation is R = Avg(V + T), and the cost volume in Eq. 1 is a cosine similarity between the image embedding E and R. Eq. 2 defines the multi-scale visual cost volume using global-average-pooled prompt features and image features. These are definitions, not reductions of the output to the input: no term in Eq. 1 or Eq. 2 is itself the segmentation prediction. The semantic-guided refinement (Sec. 4.3) reuses the Inference I mask as a new visual prompt, but this is an iterative test-time procedure rather than a tautology; the final prediction is not equal to the initial prediction by construction. The reported SOTA comparisons are against external benchmarks (Table 1), with model weights trained on COCO-Stuff and frozen CLIP/SD encoders. The self-citations (refs 1, 60, 64, 65) appear only in related-work enumerations and are not load-bearing for the DPSeg design or its benchmark gains. The main circularity-adjacent issue is the template-count ablation in Table 4, which is run directly on the five evaluation benchmarks and selects M=80; that is a test-set model-selection/overfitting concern rather than an equation-level circularity, and it does not make the prediction equivalent to an input by construction. The numerical inconsistencies between Sec. 5.3 and Table 1 are correctness/verifiability problems, not circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper is empirical and contains no formal derivation. The central claim rests on benchmark numbers plus the domain assumptions about CLIP and Stable Diffusion listed above. The main hand-chosen design values are the template count, cost volume dimension, decoder widths, and visual prompt resolution.

free parameters (5)
  • Number of prompt templates per category M = 80
    Ablated in Table 4; mIoU rises monotonically from 3.6 to 12.0 on A-847 as M goes from 1 to 80, so the final value is selected from test-set ablations rather than specified a priori.
  • Cost volume embedding dimension dF = 128
    Set in Sec. 5.2; affects decoder capacity and is not ablated.
  • Decoder hidden dimensions = [62, 32, 16]
    Chosen per decoder stage in Sec. 5.2; no ablation or justification is provided.
  • Visual prompt resolution = 768x768
    Set to match input image resolution in Sec. 5.2; not ablated.
  • Number of decoder stages = 3
    Structural design choice in Sec. 4.2; not ablated.
assumptions (4)
  • domain assumption CLIP cosine similarity in the shared embedding space is a valid pixel-level semantic likelihood for arbitrary unseen categories.
    The entire cost volume mechanism treats CLIP scores as semantic similarity in Eq. 1 and Sec. 4.1; a failure for some categories would break the method.
  • domain assumption Stable Diffusion-generated visual prompts faithfully represent their text categories for all tested categories.
    Sec. 3.1 and Sec. 4.1 use SD outputs as visual prompts; if SD omits or misrenders a category, the averaged prompt embedding R is contaminated and accuracy drops.
  • domain assumption Global average pooling of a visual prompt feature map yields a category prototype aligned with the real image feature distribution.
    Eq. 2 in Sec. 4.2 pools Vj over the prompt image, assuming the synthetic prompt's average feature, not its spatial structure, is semantically discriminative.
  • domain assumption Training on COCO-Stuff only, with frozen text and visual encoders, transfers to the five evaluation datasets including 847-category ADE20K.
    Sec. 5.1 and Table 1 assume zero-shot generalization from COCO-Stuff classes to the other datasets without any cross-dataset adaptation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation." pith.science (2026). https://pith.science/paper/ZXWJQJCP

@misc{pith2026250511676,
  author       = {Pith},
  title        = {Pith review of: DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXWJQJCP}},
  note         = {Machine review of arXiv:2505.11676}
}
read the original abstract

Open-vocabulary semantic segmentation aims to segment images into distinct semantic regions for both seen and unseen categories at the pixel level. Current methods utilize text embeddings from pre-trained vision-language models like CLIP but struggle with the inherent domain gap between image and text embeddings, even after extensive alignment during training. Additionally, relying solely on deep text-aligned features limits shallow-level feature guidance, which is crucial for detecting small objects and fine details, ultimately reducing segmentation accuracy. To address these limitations, we propose a dual prompting framework, DPSeg, for this task. Our approach combines dual-prompt cost volume generation, a cost volume-guided decoder, and a semantic-guided prompt refinement strategy that leverages our dual prompting scheme to mitigate alignment issues in visual prompt generation. By incorporating visual embeddings from a visual prompt encoder, our approach reduces the domain gap between text and image embeddings while providing multi-level guidance through shallow features. Extensive experiments demonstrate that our method significantly outperforms existing state-of-the-art approaches on multiple public datasets.

Figures

Figures reproduced from arXiv: 2505.11676 by the authors.

Figure 1
Figure 1. T-SNE visualization of embeddings in CLIP feature space, showing text prompt, visual prompt, target image, and our dual-prompt for a street scene image. Numbers indicate dis￾tance to image feature, where our dual-prompt approach achieves closest proximity (0.18). emerged as a promising solution to this limitation. By lever￾aging large-scale vision-language models like CLIP [37], OVSS enables segmentation beyond fixe… view at source ↗
Figure 2
Figure 2. Visualization of cosine similarities comparing image [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of cost volume: (a) image with ‘mirror’ [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Architecture of our DPSeg network. We begin by generating visual prompts based on text prompt templates, which are then [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Structure of Semantic-Guided Prompt Refinement. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Visualization examples of our model on open vocabulary semantic segmentation. First four columns: We present results from our model alongside the corresponding ground truth. Fifth and sixth columns: SED [52] results are shown for comparison, highlighting our model’s im…
Figure 7
Figure 7. Figure 7: Comparison between inference I and II with our refine [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Comparison of the visualized cost volumes. Evaluat￾ing the effects of cost volume by comparing the (b) 8× upsampled cost volume with the (c) the cost volume derived from 4th inter￾mediate feature and visual prompt. across all datasets when all scales are incorporated (…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 46 canonical work pages

  1. [68]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Com- puter Vision, pages 696–712. Springer, 2022

    work page 2022

  2. [1]

    Cdul: Clip-driven unsupervised learning for multi-label image classification

    Rabab Abdelfattah, Qing Guo, Xiaoguang Li, Xiaofeng Wang, and Song Wang. Cdul: Clip-driven unsupervised learning for multi-label image classification. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 1348–1357, 2023

    work page 2023

  3. [2]

    Zero-shot semantic segmentation.Advances in Neural Information Processing Systems, 32, 2019

    Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick P´erez. Zero-shot semantic segmentation.Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  4. [3]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In Computer vision and pattern recognition (CVPR), 2018 IEEE conference on . IEEE, 2018

    work page 2018

  5. [4]

    Attention to scale: Scale-aware semantic im- age segmentation

    Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille. Attention to scale: Scale-aware semantic im- age segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3640–3649, 2016

    work page 2016

  6. [5]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017

    2017

  7. [6]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104–120. Springer, 2020

    work page 2020

  8. [7]

    Cat- seg: Cost aggregation for open-vocabulary semantic seg- mentation

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat- seg: Cost aggregation for open-vocabulary semantic seg- mentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4113– 4123, 2024

    2024

Show all 69 references
  1. [8]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583–11592, 2022

    2022

  2. [9]

    Open- vocabulary universal image segmentation with maskclip

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open- vocabulary universal image segmentation with maskclip. arXiv preprint arXiv:2208.08984, 2022

    2022 arXiv

  3. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first international conference on mach...

    2024

  4. [11]

    The pascal visual object classes challenge 2011 (voc2011) development kit

    Mark Everingham and John Winn. The pascal visual object classes challenge 2011 (voc2011) development kit. Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep, 8, 2011

    2011

  5. [12]

    Scal- ing open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pages 540–557. Springer, 2022

    2022

  6. [13]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 , 2021

    2021 arXiv

  7. [14]

    Global knowledge calibration for fast open-vocabulary segmentation

    Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Ji- ajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, Yao Zhao, et al. Global knowledge calibration for fast open-vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision ...

    2023

  8. [15]

    Clip-s4: Language-guided self-supervised semantic segmen- tation

    Wenbin He, Suphanut Jamonnak, Liang Gou, and Liu Ren. Clip-s4: Language-guided self-supervised semantic segmen- tation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 11207–11216, 2023

    2023

  9. [16]

    Visual prompting for generalized few- shot segmentation: A multi-scale approach

    Mir Rayat Imtiaz Hossain, Mennatullah Siam, Leonid Sigal, and James J Little. Visual prompting for generalized few- shot segmentation: A multi-scale approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 23470–23480, 2024

    2024

  10. [17]

    Open-vocabulary instance segmentation via ro- bust cross-modal pseudo-labeling

    Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmentation via ro- bust cross-modal pseudo-labeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020–7031, 2022

    2022

  11. [18]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–491...

    2021

  12. [19]

    Understanding and constructing latent modality structures in multi-modal representation learning

    Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modality structures in multi-modal representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Visio...

    2023

  13. [20]

    Panoptic segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Doll ´ar. Panoptic segmentation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9404–9413, 2019

    2019

  14. [21]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012

    2012

  15. [22]

    Attribute-based classification for zero-shot visual object categorization

    Christoph H Lampert, Hannes Nickisch, and Stefan Harmel- ing. Attribute-based classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intelligence, 36(3):453–465, 2013

    2013

  16. [23]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pag...

    2023

  17. [24]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Ser- ena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Sys- tems, 35:17612–17625, 2022. 9

    2022

  18. [25]

    Feature pyra- mid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyra- mid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2117–2125, 2017

    2017

  19. [26]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. arXiv preprint arXiv:2312.04089, 2023

    2023 arXiv

  20. [27]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3491– 3500, 2024

    2024

  21. [28]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 3431–3440, 2015

    2015

  22. [29]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019

    2019

  23. [30]

    Clip- cap: Clip prefix for image captioning

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021

    2021 arXiv

  24. [31]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recogni...

    2014

  25. [32]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. In European conference on computer vision, pages 529–544. Springer, 2022

    2022

  26. [33]

    Open vocabulary semantic segmentation with patch aligned con- trastive learning

    Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned con- trastive learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages ...

    2023

  27. [34]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems,...

    2019

  28. [35]

    Hierarchical dense cor- relation distillation for few-shot segmentation

    Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense cor- relation distillation for few-shot segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23641–23651, 2023

    2023

  29. [36]

    Connecting vision and lan- guage with localized narratives

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and lan- guage with localized narratives. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part V 16, pages 647–66...

    2020

  30. [37]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  31. [38]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  32. [39]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  33. [40]

    Open-vocabulary semantic segmentation with image embedding balancing

    Xiangheng Shan, Dongyue Wu, Guilin Zhu, Yuanjie Shao, Nong Sang, and Changxin Gao. Open-vocabulary semantic segmentation with image embedding balancing. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28412–28421, 2024

    2024

  34. [41]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  35. [42]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  36. [43]

    Lxmert: Learning cross- modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019

    1908 arXiv

  37. [44]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR, 2019

    2019

  38. [45]

    Yfcc100m: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016

    2016

  39. [46]

    Panet: Few-shot image semantic seg- mentation with prototype alignment

    Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic seg- mentation with prototype alignment. In proceedings of the IEEE/CVF international conference on computer vision, pages 9197–9206, 2019

    2019

  40. [47]

    USE: Univer- sal Segment Embeddings for Open-V ocabulary Image Seg- mentation

    Xiaoqi Wang, Wenbin He, Xiwei Xuan, Clint Sebastian, Jorge Piazentin Ono, Xin Li, Sima Behpour, Thang Doan, Liang Gou, Han-Wei Shen, and Liu Ren. USE: Univer- sal Segment Embeddings for Open-V ocabulary Image Seg- mentation . In 2024 IEEE/CVF Conference on Computer Vision and ...

    2024

  41. [48]

    Use: Universal segment embeddings for open-vocabulary image segmentation

    Xiaoqi Wang, Wenbin He, Xiwei Xuan, Clint Sebastian, Jorge Piazentin Ono, Xin Li, Sima Behpour, Thang Doan, Liang Gou, Han-Wei Shen, et al. Use: Universal segment embeddings for open-vocabulary image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion ...

    2024

  42. [49]

    Hierar- chical open-vocabulary universal image segmentation

    Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hierar- chical open-vocabulary universal image segmentation. Ad- vances in Neural Information Processing Systems, 36, 2024

    2024

  43. [50]

    Detectron2

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github. com/facebookresearch/detectron2, 2019

    2019

  44. [51]

    Semantic projection network for zero-and few-label semantic segmentation

    Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8256–8265, 2019

    2019

  45. [52]

    Sed: A simple encoder-decoder for open- vocabulary semantic segmentation

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3426–3436, 2024

    2024

  46. [53]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18134–18144, 2022

    2022

  47. [54]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 2955–2966, 2023

    2023

  48. [55]

    A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model. In European Conference on Computer Vi- sion, pages 736–753. Springer, 2022

    2022

  49. [56]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2945– 2954, 2023

    2023

  50. [57]

    Attentive mask clip

    Yifan Yang, Weiquan Huang, Yixuan Wei, Houwen Peng, Xinyang Jiang, Huiqiang Jiang, Fangyun Wei, Yin Wang, Han Hu, Lili Qiu, et al. Attentive mask clip. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 2771–2781, 2023

    2023

  51. [58]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021

    2021 arXiv

  52. [59]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36:32215–32234, 2023

    2023

  53. [60]

    Sair: Learning semantic-aware implicit representation

    Canyu Zhang, Xiaoguang Li, Qing Guo, and Song Wang. Sair: Learning semantic-aware implicit representation. In European Conference on Computer Vision, pages 319–335. Springer, 2024

    2024

  54. [61]

    Few-shot segmentation via cycle-consistent trans- former

    Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle-consistent trans- former. Advances in Neural Information Processing Systems, 34:21984–21996, 2021

    2021

  55. [62]

    Transparent image layer diffusion using latent transparency, 2024

    Lvmin Zhang and Maneesh Agrawala. Transparent image layer diffusion using latent transparency, 2024

    2024

  56. [63]

    Zero-shot learning via joint latent similarity embedding

    Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via joint latent similarity embedding. In proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 6034–6042, 2016

    2016

  57. [64]

    Crossmodal few-shot 3d point cloud semantic seg- mentation

    Ziyu Zhao, Zhenyao Wu, Xinyi Wu, Canyu Zhang, and Song Wang. Crossmodal few-shot 3d point cloud semantic seg- mentation. In Proceedings of the 30th ACM international conference on multimedia, pages 4760–4768, 2022

    2022

  58. [65]

    Crossmodal few-shot 3d point cloud semantic segmentation via view synthesis

    Ziyu Zhao, Pingping Cai, Canyu Zhang, Xiaoguang Li, and Song Wang. Crossmodal few-shot 3d point cloud semantic segmentation via view synthesis. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8777– 8785, 2024

    2024

  59. [66]

    Regionclip: Region- based language-image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2022

  60. [67]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633–641, 2017

    2017

  61. [69]

    Generalized decoding for pixel, image, and lan- guage

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and lan- guage. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.