Pith. sign in

REVIEW 5 major objections 4 minor 80 references

Show or Tell? A Benchmark To Evaluate Visual and Textual Prompts in Semantic Segmentation

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Visual prompts beat text prompts for semantic segmentation on average, with the edge concentrated in specialized domains.

desk verdict Useful benchmark and a mostly transparent protocol, but the central visual-vs-textual claim rests on an unvalidated merging rule and an unweighted average; referee it, don't desk-reject it. read the letter →

arxiv 2505.06280 v1 pith:XG3GXBVD submitted 2025-05-06 cs.CV

Gabriele Rosi , Fabio Cermelli This is my paper · ORCID
classification cs.CV
keywords semanticsegmentationvisualpromptingtextualopen-vocabularybenchmarkvisionfoundationmodelspromptsensitivitymulti-classadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to settle a simple question: when a segmentation model is told what to look for, is a picture better than a sentence? It introduces SoT, a benchmark that runs nine prompt-driven segmentation methods head-to-head on 14 datasets in seven domains. The central finding is that visual reference prompts outperform open-vocabulary text prompts on average, with the best visual method reaching 38.7 mIoU with five prompts. The advantage is not uniform: text wins for everyday objects, while visuals win by large margins for tools, waste, and X-ray scans. The paper's adaptation of single-class visual methods to multi-class segmentation is what makes the head-to-head possible, and that adaptation carries the main load-bearing assumption of the comparison.

What carries the argument

The central object is the Show or Tell (SoT) benchmark itself, together with a confidence-based mask merging strategy that adapts binary visual-reference methods to multi-class segmentation. Each class is handled by a separate run that produces a binary mask, and the binary masks are merged by taking an argmax over confidence scores; for training-free methods the confidence is the mean visual-backbone (DINOv2 or SAM) score inside the mask, and for trained methods it is the decoder probability. This merging rule is what allows single-class visual prompting methods to be compared fairly with open-vocabulary text methods on multi-class scenes.

What would settle it

Re-run the SoT comparison with confidence scores normalized per class, for example rank-based or z-scored within each class, before the argmax merge, and check whether GFSAM's average 38.7 mIoU and the visual-method lead over the best textual method shrink or disappear; if they do, the reported advantage is an artifact of the merging rule.

Watch

Extended reading notes

Core claim

The paper claims that, under identical benchmark conditions, visual reference prompt methods generally outperform open-vocabulary textual prompting for semantic segmentation. The best visual method, GFSAM, reaches 38.7 mIoU on average across the 14 datasets with five prompts, while the best open-vocabulary method trails on most specialized domains. Textual prompting remains competitive or superior in common scenes such as PASCAL VOC and Cityscapes, but visual prompting dominates waste, tools, and X-ray imagery, where verbal descriptions poorly capture the visual cues. The paper also shows that visual methods are highly sensitive to prompt selection, with standard deviations exceeding 14 mIoU in some datasets, and that they require many forward passes because each class is processed separately.

Load-bearing premise

The head-to-head comparison assumes that the confidence values used to merge per-class masks, mean DINOv2 or SAM scores for training-free methods and decoder probabilities for SINE, are comparable across classes and methods, so that taking the argmax does not systematically favor visual reference methods over textual ones.

Editorial extensions

If this is right

  • If the central result holds, practitioners working in specialized domains such as security X-ray, waste sorting, and tool identification should prefer visual reference prompts over text descriptions when accuracy is the priority.
  • Textual prompts remain the practical choice for common scenes and urban settings, where they match or beat visual methods at a fraction of the inference cost.
  • Visual prompt selection becomes a first-class concern: five randomly chosen support images shift results by more than 14 mIoU in some datasets, so deployment should include prompt ensembling or automatic support-set selection.
  • The multi-class adaptation means single-class visual reference methods can be applied to broad semantic segmentation without retraining, at a cost that grows linearly with the number of classes.
  • Training-free methods generalize better across domains than trained ones in this comparison, suggesting that foundation-model priors matter more than task-specific fine-tuning for prompt-based segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test of the paper's comparison is to replace raw confidence values with calibrated or per-class normalized scores before the argmax merge; if the visual methods' average lead shrinks, part of the reported advantage belongs to the merging rule rather than to visual prompting itself.
  • The domain split suggests a hybrid system: use text prompts as a cheap default, and switch to visual prompts only when text confidence is low or the domain is specialized; the benchmark provides the per-domain breakpoints such a controller would need.
  • Because the benchmark randomizes support-set selection, its standard deviations quantify prompt sensitivity; a follow-up could search for support-set selection heuristics, such as diversity of pose or background, that reduce the observed swings.
  • The same confidence-merging recipe could be applied to combined text-plus-visual prompting, letting a model fall back to textual descriptions for small objects in cluttered scenes, where the paper's qualitative results show visual prompts failing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces Show or Tell (SoT), a benchmark for directly comparing textual prompts (via open-vocabulary semantic segmentation methods) and visual reference prompts (via few-shot visual prompting methods) for semantic segmentation across 14 datasets in 7 domains. It evaluates 5 open-vocabulary methods (MaskCLIP, TCL, CLIP-DINoiser, NACLIP, ProxyCLIP) and 4 visual reference methods (SINE, PerSAM, Matcher, GFSAM), and adapts the latter to multi-class segmentation by generating per-class binary masks and merging them via a confidence-based argmax over mean backbone scores. The main reported finding is that visual prompting methods generally outperform textual approaches on average, with GFSAM achieving the highest average mIoU of 38.7, while textual methods remain competitive on common scenes. The paper also analyzes inference cost and prompt sensitivity, and releases code.

Significance. If the results are reliable, SoT would be a valuable community resource: it is the first benchmark to compare the two prompt modalities under a common evaluation protocol, spans a broad range of domains, samples support sets five times and reports standard deviations, and releases code. The paper also provides a useful decomposition of where each modality fails (e.g., specialized tools and waste favor visual prompts; common categories yield to text). However, the central quantitative comparison rests on the multi-class adaptation in Sec. 3.2, whose fairness is not established; the aggregation over datasets of very different sizes is not justified; and some reported numbers are internally inconsistent. The contribution is real, but the load-bearing claims need additional verification before the benchmark's headline conclusion can be accepted.

major comments (5)
  1. [Sec. 3.2, Table 3] The confidence-based mask merging rule is load-bearing: every visual-method result in Table 3 is produced by taking the argmax over per-class masks using the mean backbone confidence within each mask. The paper provides no evidence that these mean confidences are comparable across classes or across methods. Mean-over-mask statistics favor large, homogeneous classes (e.g., 'road' in Cityscapes, 'building' in LoveDA-Urban) over small or partially visible classes (e.g., 'traffic light', 'Allen key'), so overlapping predictions can be assigned to the wrong class even when the per-pixel mask is correct. The confidence scales also differ across DINOv2 similarity, SAM scores, and SINE's decoder probabilities. The paper should report a sensitivity analysis of the main claim to the merging rule, e.g., compare with per-pixel max without mean aggregation, with class-wise calibration, and with an oracle merging rule that uses ground-truth overlap to resolve conflicts. Without such a check, the headline 'visual generally outperforms textual' (Sec. 4.2) may be an artifact of this adaptation rather than a property of the prompt modality.
  2. [Sec. 4.2, Table 2, Table 3] The headline average mIoU is an unweighted mean over 14 datasets that differ by more than two orders of magnitude in validation-set size: Toolkits has 6 validation images, ADE20K has 2000, and Cityscapes has 500. Equal weighting lets the 88.9 mIoU on Toolkits count the same as the 23.4 mIoU on ADE20K for GFSAM. The paper should report per-domain averages, a size-weighted average, or otherwise justify the unweighted aggregation. It should also flag the statistical fragility on the smallest datasets; for example, GFSAM's ±14.1 standard deviation on Pizza indicates that its 62.2 mIoU there is not a stable estimate.
  3. [Table 3, 'AVG' column] For Matcher, two entries (ADE20K and UECFood) are marked '†' (excessive time), but the AVG is reported as 27.2 for 1 prompt and 33.5 for 5 prompts without stating the aggregation rule. The arithmetic mean of the twelve numeric entries shown for Matcher 1-prompt is 27.0 (not 27.2), and for 5-prompt it is 33.4 (not 33.5). The paper must state how averages are computed when dataset entries are missing (e.g., average over available datasets only), and should report the number of datasets included in each average. This is necessary for readers to interpret the central comparison.
  4. [Sec. 1 and Supplementary Sec. 7] The abstract and introduction state that the methods are evaluated under 'identical experimental conditions,' but the methods use different backbones: NACLIP and ProxyCLIP use ViT-L/14, the other open-vocabulary methods use ViT-B/16, and the visual methods use DINOv2 ViT-L/14 with SAM ViT-H. Since larger backbones generally improve segmentation quality, this confounds the prompt-modality comparison with model capacity. The authors should either match backbones across modalities for a cleaner head-to-head, or explicitly acknowledge this as a limitation and analyze how it might affect the conclusions.
  5. [Sec. 4.1] The visual reference methods (SINE, PerSAM, Matcher, GFSAM) were 'reimplemented within our codebase,' but the paper does not report any verification that these reimplementations reproduce the original methods' published behavior on a standard few-shot segmentation setting. For a benchmark whose main output is a set of scores, such a reproducibility check is important to rule out that the visual-method results (and hence the headline conclusion) are artifacts of the reimplementation rather than the original algorithms. The authors should compare their implementations against the official code on at least one representative dataset (e.g., PASCAL-5i or COCO-20i) and report the deviation.
minor comments (4)
  1. [Supplementary Sec. 8] The class list for Trash is internally inconsistent: the text states 'The Trash [42] dataset is made up of 12 classes,' but then enumerates 22 class names. Similarly, the House-Parts list claims 22 classes but enumerates 12 names. Please correct these counts and ensure the lists exactly match the dataset versions used.
  2. [Table 3 caption] The caption says PerSAM reports only 1-prompt results 'due to its limitation in handling multiple prompts,' but the table shows full rows of hyphens for the 5-prompt configuration. Please state explicitly what the hyphens mean (e.g., 'not applicable / not evaluated') and clarify the reason that multiple prompts are not supported.
  3. [Sec. 4.2] The sentence 'The performance gap between modalities dramatically varies across domains' is vague. Consider reporting a concrete measure, such as the per-domain difference between the best visual and best textual method, or a table of these gaps.
  4. [Throughout] The benchmark name is used inconsistently: 'Toolkit' (Table 2) vs. 'Toolkits' (Table 3, Fig. 3), and 'Toolkit [39]' in Sec. 3.3 vs. 'Toolkits [39]' elsewhere. Please standardize the spelling.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark conclusions are empirical measurements, not constructions from the paper's own inputs.

full rationale

Show or Tell is an evaluation benchmark rather than a derivation chain. The central claim in Sec. 4.2 that visual prompting methods generally outperform textual approaches is a measured outcome over 14 datasets and 9 externally developed methods; it is not obtained by fitting a parameter to that claim, and no equation defines the conclusion into existence. The confidence-based mask merging in Sec. 3.2 is a fixed protocol for adapting binary visual-reference methods to multi-class segmentation; even if it introduces a validity risk (uncalibrated confidence scores favoring certain classes), that risk does not make the result circular, because the reported mIoU values are not algebraic consequences of the merging rule alone and the rule was not chosen to force the headline outcome. The only self-citations ([7], [8], [45]) appear in the introduction and related work and are not load-bearing for the benchmark's conclusions; no uniqueness theorem or ansatz is imported from the authors' prior work. The evaluation is self-contained against external methods, public code, and standard datasets, so the appropriate finding is no significant circularity; the score of 1 reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim of the paper is empirical, so it rests on the validity of the benchmark protocol rather than on mathematical axioms or fitted parameters. The key assumptions are representativeness of the datasets, fairness of the multi-class adaptation for visual prompts, and the meaningfulness of the unweighted average. No free parameters or invented entities are introduced.

assumptions (3)
  • domain assumption The 14 datasets and their validation splits are representative of the 7 target domains and large enough for reliable per-domain conclusions.
    The benchmark's conclusions depend on dataset representativeness; for example, Toolkits has only 6 validation images, so its extreme results are statistically fragile.
  • ad hoc to paper The confidence-based mask merging strategy in Sec. 3.2 yields a fair multi-class comparison without systematically favoring visual reference methods.
    The strategy is introduced in this paper and uses confidence scores from different backbones (DINOv2, SAM, or SINE's decoder) that may not be calibrated; no validation is provided that this does not bias the comparison.
  • domain assumption The unweighted average mIoU across datasets (AVG column in Table 3) is a meaningful summary metric for overall performance.
    The AVG weights datasets equally regardless of size or difficulty, which can drive the headline 'visual methods generally outperform' conclusion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Show or Tell? A Benchmark To Evaluate Visual and Textual Prompts in Semantic Segmentation." pith.science (2026). https://pith.science/paper/XG3GXBVD

@misc{pith2026250506280,
  author       = {Pith},
  title        = {Pith review of: Show or Tell? A Benchmark To Evaluate Visual and Textual Prompts in Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XG3GXBVD}},
  note         = {Machine review of arXiv:2505.06280}
}
read the original abstract

Prompt engineering has shown remarkable success with large language models, yet its systematic exploration in computer vision remains limited. In semantic segmentation, both textual and visual prompts offer distinct advantages: textual prompts through open-vocabulary methods allow segmentation of arbitrary categories, while visual reference prompts provide intuitive reference examples. However, existing benchmarks evaluate these modalities in isolation, without direct comparison under identical conditions. We present Show or Tell (SoT), a novel benchmark specifically designed to evaluate both visual and textual prompts for semantic segmentation across 14 datasets spanning 7 diverse domains (common scenes, urban, food, waste, parts, tools, and land-cover). We evaluate 5 open-vocabulary methods and 4 visual reference prompt approaches, adapting the latter to handle multi-class segmentation through a confidence-based mask merging strategy. Our extensive experiments reveal that open-vocabulary methods excel with common concepts easily described by text but struggle with complex domains like tools, while visual reference prompt methods achieve good average results but exhibit high variability depending on the input prompt. Through comprehensive quantitative and qualitative analysis, we identify the strengths and weaknesses of both prompting modalities, providing valuable insights to guide future research in vision foundation models for segmentation tasks.

Figures

Figures reproduced from arXiv: 2505.06280 by the authors.

Figure 1
Figure 1. Our Show or Tell (SoT) benchmark evaluates the effec [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Sample images drawn from the datasets that compose [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Qualitative results of ProxyCLIP [27] (textual prompt) and GFSAM [70] (visual prompt) across all the dataset in our benchmark. in specialized domains like tools and parts, where textual descriptions inadequately capture complex visual character￾istics. Visual reference prompt methods demonstrated more consistent performance across domains, with GFSAM [70] showing strong results when provided with multiple visual pro… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 61 canonical work pages

  1. [1]

    Single-stage semantic segmentation from image labels

    Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. InCVPR, 2020. 12

    work page 2020

  2. [2]

    Enhancing open-vocabulary semantic seg- mentation with prototype retrieval

    Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, and Rita Cucchiara. Enhancing open-vocabulary semantic seg- mentation with prototype retrieval. InInternational Confer- ence on Image Analysis and Processing, 2023. 1, 3

    work page 2023

  3. [3]

    Zerowaste dataset: To- wards deformable object segmentation in cluttered scenes

    Dina Bashkirova, Mohamed Abdelfattah, Ziliang Zhu, James Akl, Fadi Alladkani, Ping Hu, Vitaly Ablavsky, Berk Calli, Sarah Adel Bargal, and Kate Saenko. Zerowaste dataset: To- wards deformable object segmentation in cluttered scenes. In CVPR, 2022. 4, 5, 13

    work page 2022

  4. [4]

    What a mess: Multi-domain evaluation of zero-shot semantic segmentation.Advances in Neural Infor- mation Processing Systems, 36:73299–73311, 2023

    Benedikt Blumenstiel, Johannes Jakubik, Hilde K ¨uhne, and Michael V¨ossing. What a mess: Multi-domain evaluation of zero-shot semantic segmentation.Advances in Neural Infor- mation Processing Systems, 36:73299–73311, 2023. 2, 4

    work page 2023

  5. [5]

    Zero-shot semantic segmentation.NeurIPS, 32, 2019

    Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick P´erez. Zero-shot semantic segmentation.NeurIPS, 32, 2019. 2

    work page 2019

  6. [6]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 2, 3, 4, 7

    work page 2021

  7. [7]

    Modeling the background for incremental learning in semantic segmentation

    Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Caputo. Modeling the background for incremental learning in semantic segmentation. InCVPR,

  8. [8]

    Prototype-based Incremental Few-Shot Semantic Segmentation

    Fabio Cermelli, Massimiliano Mancini, Yongqin Xian, Zeynep Akata, and Barbara Caputo. Prototype-based in- cremental few-shot semantic segmentation.arXiv preprint arXiv:2012.01415, 2020. 3, 4

    work page Pith review arXiv 2012

Show all 80 references
  1. [9]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. InCVPR, 2023. 1, 3, 6, 7

    2023

  2. [10]

    Unleashing the potential of prompt engineer- ing in large language models: a comprehensive review.arXiv preprint arXiv:2310.14735, 2023

    Banghao Chen, Zhaofeng Zhang, Nicolas Langren ´e, and Shengxin Zhu. Unleashing the potential of prompt engineer- ing in large language models: a comprehensive review.arXiv preprint arXiv:2310.14735, 2023. 1

    2023 arXiv

  3. [11]

    Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only

    Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Cu- latana, and Mohamed Elhoseiny. Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only. InICCV, 2023. 2, 3, 6

    2023

  4. [12]

    MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.https : / / github

    MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.https : / / github . com / open - mmlab/mmsegmentation, 2020. 5

    2020

  5. [13]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InCVPR,

  6. [14]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint ar...

    2010 arXiv

  7. [15]

    A new large- scale food image segmentation dataset and its application to food calorie estimation based on grains of rice

    Takumi Ege, Wataru Shimoda, and Keiji Yanai. A new large- scale food image segmentation dataset and its application to food calorie estimation based on grains of rice. InPro- ceedings of the 5th international workshop on multimedia assisted dietary management, 2019. 4, 5, 7, 13

    2019

  8. [16]

    The pascal visual object classes (voc) challenge.International journal of computer vision, 88:303–338, 2010

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge.International journal of computer vision, 88:303–338, 2010. 4, 5, 6, 7, 12

    2010

  9. [17]

    Model- agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model- agnostic meta-learning for fast adaptation of deep networks. InICML, 2017. 1

    2017

  10. [18]

    Scal- ing open-vocabulary image segmentation with image-level labels

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scal- ing open-vocabulary image segmentation with image-level labels. InECCV, 2022. 1, 3

    2022

  11. [19]

    Context-aware feature generation for zero- shot semantic segmentation

    Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero- shot semantic segmentation. InProceedings of the 28th ACM International Conference on Multimedia, 2020. 2

    2020

  12. [20]

    Mvp-seg: Multi-view prompt learning for open-vocabulary semantic segmentation

    Jie Guo, Qimeng Wang, Yan Gao, Xiaolong Jiang, Shaohui Lin, and Baochang Zhang. Mvp-seg: Multi-view prompt learning for open-vocabulary semantic segmentation. InChi- nese Conference on Pattern Recognition and Computer Vi- sion (PRCV), 2023. 2

    2023

  13. [21]

    Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation.arXiv preprint arXiv:2404.08181, 2024

    Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation.arXiv preprint arXiv:2404.08181, 2024. 2, 3, 6, 7, 12

    2024 arXiv

  14. [22]

    Cost aggregation with 4d convolutional swin transformer for few-shot segmentation

    Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. InECCV. Springer, 2022. 3

    2022

  15. [23]

    Open- clip.https : / / github

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip.https : / / github . com / mlfoundations / open_clip,...

    2021

  16. [24]

    Diffusion models for zero-shot open-vocabulary segmentation.arXiv e-prints, pages arXiv–2306, 2023

    Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation.arXiv e-prints, pages arXiv–2306, 2023. 3

    2023

  17. [25]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InICCV, 2023. 1, 2, 3, 4, 7, 12

    2023

  18. [26]

    Overcoming catastrophic forgetting in neu- ral networks.Proceedings of the national academy of sci- ences, 114(13):3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran 9 Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neu- ral networks.Proceedings of the national academy of s...

    2017

  19. [27]

    Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation. In ECCV, 2024. 2, 3, 6, 7, 8, 12

    2024

  20. [28]

    Adaptive prototype learning and allocation for few-shot segmentation

    Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive prototype learning and allocation for few-shot segmentation. InCVPR,

  21. [29]

    Multiple- human parsing in the wild.arXiv preprint arXiv:1705.07206,

    Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, Terence Sim, Shuicheng Yan, and Jiashi Feng. Multiple- human parsing in the wild.arXiv preprint arXiv:1705.07206,

    arXiv

  22. [30]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks.arXiv e-prints, pages arXiv–2304, 2023

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks.arXiv e-prints, pages arXiv–2304, 2023. 3

    2023

  23. [31]

    Learning without forgetting

    Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelli- gence, 40(12):2935–2947, 2017. 1

    2017

  24. [32]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. InCVPR, 2023. 1, 3

    2023

  25. [33]

    Learning non-target knowledge for few- shot semantic segmentation

    Yuanwei Liu, Nian Liu, Qinglong Cao, Xiwen Yao, Junwei Han, and Ling Shao. Learning non-target knowledge for few- shot semantic segmentation. InCVPR, 2022. 2

    2022

  26. [34]

    Matcher: Segment anything with one shot using all-purpose feature matching.arXiv preprint arXiv:2305.13310, 2023

    Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching.arXiv preprint arXiv:2305.13310, 2023. 2, 3, 4, 6, 7

    2023 arXiv

  27. [35]

    A simple im- age segmentation framework via in-context examples

    Yang Liu, Chenchen Jing, Hengtao Li, Muzhi Zhu, Hao Chen, Xinlong Wang, and Chunhua Shen. A simple im- age segmentation framework via in-context examples. In NeurIPS, 2024. 2, 3, 4, 6, 7

    2024

  28. [36]

    Image segmentation using text and image prompts

    Timo L ¨uddecke and Alexander Ecker. Image segmentation using text and image prompts. InCVPR, 2022. 2

    2022

  29. [37]

    Uavid: A semantic segmentation dataset for uav imagery.ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020

    Ye Lyu, George V osselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery.ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020. 4, 5, 6, 7, 12

    2020

  30. [38]

    Hypercorre- lation squeeze for few-shot segmentation

    Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorre- lation squeeze for few-shot segmentation. InICCV, 2021. 3

    2021

  31. [39]

    mask dataset.https://universe.roboflow

    mst. mask dataset.https://universe.roboflow. com/mst/mask-2ihnt, 2022. visited on 2025-03-06. 4, 5, 7, 13

    2022

  32. [40]

    Open vocabulary semantic segmentation with patch aligned con- trastive learning

    Jishnu Mukhoti, Tsung-Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned con- trastive learning. InCVPR, 2023. 1, 2, 3

    2023

  33. [41]

    Samic: Segment anything with in-context spa- tial prompt engineering.arXiv preprint arXiv:2412.11998,

    Savinay Nagendra, Kashif Rashid, Chaopeng Shen, and Daniel Kifer. Samic: Segment anything with in-context spa- tial prompt engineering.arXiv preprint arXiv:2412.11998,

    arXiv

  34. [42]

    Trash (v2).https : / / universe

    Sara Najafi. Trash (v2).https : / / universe . roboflow . com / sara - najafi / trash _ segmentation2/dataset/2, 2022. 4, 5, 7, 12

    2022

  35. [43]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2, 3, 4, 7, 12

    2023 arXiv

  36. [44]

    Continual lifelong learning with neural networks: A review.Neural networks, 113:54–71,

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review.Neural networks, 113:54–71,

  37. [45]

    A closer look at self-training for zero-label semantic segmentation

    Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimil- iano Mancini, Zeynep Akata, and Barbara Caputo. A closer look at self-training for zero-label semantic segmentation. In CVPR, 2021. 2

    2021

  38. [46]

    Freeseg: Unified, universal and open-vocabulary im- age segmentation

    Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xue- feng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open-vocabulary im- age segmentation. InCVPR, 2023. 1, 3

    2023

  39. [47]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, 2021. 1, 2, 3, 4

    2021

  40. [48]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. InICML, 2021. 2

    2021

  41. [49]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. 2, 3

    2024 arXiv

  42. [50]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InCVPR, 2022. 2

    2022

  43. [51]

    One-shot learning for semantic segmentation

    Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410, 2017. 3, 4

    2017 arXiv

  44. [52]

    Reco: Re- trieve and co-segment for zero-shot transfer.NeurIPS, 35: 33754–33767, 2022

    Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Re- trieve and co-segment for zero-shot transfer.NeurIPS, 35: 33754–33767, 2022. 3

    2022

  45. [53]

    What does clip know about a red circle? visual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. InICCV, 2023. 2

    2023

  46. [54]

    Vrp-sam: Sam with visual reference prompt

    Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang, and Zechao Li. Vrp-sam: Sam with visual reference prompt. In CVPR, 2024. 2, 3, 4

    2024

  47. [55]

    abc dataset.https : / / universe

    TestCoco. abc dataset.https : / / universe . roboflow.com/testcoco/abc- fqun0, 2022. vis- ited on 2025-03-06. 4, 5, 6, 13

    2022

  48. [56]

    Springer, 1998

    Sebastian Thrun and Lorien Pratt.Learning to Learn. Springer, 1998. 1

    1998

  49. [57]

    Sclip: Rethinking self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. InECCV,

  50. [58]

    Few-shot semantic seg- mentation with democratic attention networks

    Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen. Few-shot semantic seg- mentation with democratic attention networks. InECCV,

  51. [59]

    Sam-clip: Merging vision foundation models towards semantic and spatial understanding

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. InCVPR, 2024. 3

    2024

  52. [60]

    Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation

    Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zhong. Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation. InNeurIPS,

  53. [61]

    Review of large vision models and visual prompt engineering.Meta-Radiology, 1(3):100047,

    Jiaqi Wang, Zhengliang Liu, Lin Zhao, Zihao Wu, Chong Ma, Sigang Yu, Haixing Dai, Qiushi Yang, Yiheng Liu, Songyao Zhang, et al. Review of large vision models and visual prompt engineering.Meta-Radiology, 1(3):100047,

  54. [62]

    Panet: Few-shot image semantic segmenta- tion with prototype alignment

    Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmenta- tion with prototype alignment. InICCV, 2019. 3

    2019

  55. [63]

    Images speak in images: A generalist painter for in-context visual learning

    Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. InCVPR, 2023. 3

    2023

  56. [64]

    Seggpt: Towards seg- menting everything in context

    Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Towards seg- menting everything in context. InICCV, 2023. 3

    2023

  57. [65]

    Clip-dinoiser: Teaching clip a few dino tricks for open- vocabulary semantic segmentation

    Monika Wysocza ´nska, Oriane Sim´eoni, Micha¨el Ramamon- jisoa, Andrei Bursuc, Tomasz Trzci ´nski, and Patrick P ´erez. Clip-dinoiser: Teaching clip a few dino tricks for open- vocabulary semantic segmentation. InECCV, 2024. 2, 3, 6, 7

    2024

  58. [66]

    Semantic projection network for zero-and few-label semantic segmentation

    Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. InCVPR, 2019. 2, 4

    2019

  59. [67]

    Cat-sam: Con- ditional tuning for few-shot adaptation of segment anything model

    Aoran Xiao, Weihao Xuan, Heli Qi, Yun Xing, Ruijie Ren, Xiaoqin Zhang, Ling Shao, and Shijian Lu. Cat-sam: Con- ditional tuning for few-shot adaptation of segment anything model. InECCV, 2024. 3

    2024

  60. [68]

    piiz dataset.https://universe.roboflow.com/ y-rgb4q/piiz, 2023

    y. piiz dataset.https://universe.roboflow.com/ y-rgb4q/piiz, 2023. visited on 2025-03-06. 4, 5, 7, 13

    2023

  61. [69]

    Contin- ual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Contin- ual learning through synaptic intelligence. InICML, 2017. 1

    2017

  62. [70]

    Bridge the points: Graph-based few-shot segment anything semantically.NeurIPS, 37:33232–33261, 2024

    Anqi Zhang, Guangyu Gao, Jianbo Jiao, Chi Liu, and Yun- chao Wei. Bridge the points: Graph-based few-shot segment anything semantically.NeurIPS, 37:33232–33261, 2024. 2, 3, 4, 5, 6, 7, 8

    2024

  63. [71]

    Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation

    Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation. InICCV, 2019. 3

    2019

  64. [72]

    Few-shot segmentation via cycle-consistent trans- former.NeurIPS, 34:21984–21996, 2021

    Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle-consistent trans- former.NeurIPS, 34:21984–21996, 2021. 3

    2021

  65. [73]

    Pidray: A large-scale x-ray benchmark for real-world prohibited item detection.International Journal of Computer Vision, 131 (12):3170–3192, 2023

    Libo Zhang, Lutao Jiang, Ruyi Ji, and Heng Fan. Pidray: A large-scale x-ray benchmark for real-world prohibited item detection.International Journal of Computer Vision, 131 (12):3170–3192, 2023. 4, 5, 7, 13

    2023

  66. [74]

    Personalize segment anything model with one shot

    Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junt- ing Pan, Xianzheng Ma, Hao Dong, Peng Gao, and Hong- sheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03048, 2023. 2, 3, 4, 6, 7

    2023 arXiv

  67. [75]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. InCVPR, 2017. 4, 5, 6, 7, 12

    2017

  68. [76]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. InECCV, 2022. 3

    2022

  69. [77]

    Generalized decoding for pixel, image, and language

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. InCVPR, 2023. 2, 4 11 Show or Tell? A Benchmark To Evaluate Visual and Textual Prompts in Semant...

    2023

  70. [78]

    Semantic support set generation In Algorithm 1 we describe in detail how the semantic sup- port setS sem has been generated. Algorithm 1Generation of Semantic Support Set Input:Set of class IDsC, Set of training images names for each classD c, Number of visual promptsk Output:...

  71. [79]

    Open-vocabulary methods.To ensure a fair comparison, we report the results for open-vocabulary methods without applying any mask refinement step (e.g

    Additional implementation details Visual reference prompt methods.Following the origi- nal implementations, visual reference prompt methods are evaluated using DINOv2 [43] with ViT-L/14 [14] (when ap- plicable), and SAM [25] with ViT-H [14] (when applicable). Open-vocabulary m...

  72. [80]

    ADE20KThe ADE20K [75] dataset is made up of 150 classes

    Dataset classes For each dataset comprised in our SoT benchmark, we re- port the list of classes. ADE20KThe ADE20K [75] dataset is made up of 150 classes. The classes are:wall,building,sky, floor,tree,ceiling,road,bed,windowpane, grass,cabinet,sidewalk,person,earth, door,table...

    2012

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.