Pith. sign in

REVIEW 4 major objections 4 minor 51 references

CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A training-free CLIP pipeline that fuses early-layer maps with Stable Diffusion attention sets state-of-the-art on seven segmentation benchmarks.

desk verdict Solid two-stage training-free segmentation with a clean ablation, but the state-of-the-art claim leans on an unvalidated diffusion timestep; deserves review with requests for sensitivity analysis. read the letter →

arxiv 2411.13836 v1 pith:DDIUHV2X submitted 2024-11-21 cs.CV

Lin Sun , Jiale Cao , Jin Xie , Xiaoheng Jiang , Yanwei Pang This is my paper · ORCID
classification cs.CV
keywords open-vocabularysemanticsegmentationtraining-freeCLIPspatialrepresentationearly-layerattentionfusiondiffusionmodelself-attentionStablerefinementzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that CLIP's poor pixel-level spatial representation can be fixed without any training by drawing on two unused sources of spatial structure: early transformer layers inside CLIP, and the self-attention maps of a frozen Stable Diffusion model. The proposed CLIPer first builds a coarse segmentation map from early-layer CLIP embeddings and an averaged early-layer attention map, then refines it by multiplying with the fused multi-head self-attention map of Stable Diffusion. On seven open-vocabulary segmentation benchmarks it reports the highest mean Intersection over Union (mIoU) among training-free methods, including 69.8% on PASCAL VOC and 43.3% on COCO Object with a ViT-L backbone. If the results hold, open-vocabulary segmentation can be pushed forward by reusing frozen off-the-shelf models rather than training a segmentation head.

What carries the argument

The two load-bearing mechanisms are the early-layer fusion and the fine-grained compensation. Early-layer fusion computes an averaged attention map $A_{\text{avg}} = \frac{1}{N-1} \sum_{n=1}^{N-1} A_n$ from all CLIP transformer blocks before the last one, substitutes it for the last layer's self-attention, and passes early-layer embeddings through the modified last block (without FFN or residuals, following ClearCLIP). The resulting patch embeddings are cosine-compared to text embeddings and averaged across layers to give the coarse map. Fine-grained compensation extracts multi-head self-attention maps from Stable Diffusion at a single timestep with a null prompt, fuses them by matrix chain multiplication $A_f = A_m[0] \times \cdots \times A_m[H-1]$, and refines the upscaled coarse map as $S_f = A_f \times S_c$. The refinement operator is the central identity: a diffusion self-attention product acting on a CLIP similarity map transfers boundary-accurate spatial detail into the coarse semantic map.

What would settle it

Run the full CLIPer pipeline on PASCAL VOC while varying the Stable Diffusion timestep over the denoising schedule (e.g., steps 20, 30, 40, 45, 50) and switching the null prompt to the target category names; if the reported 69.8% mIoU shifts by more than about two points, the single-timestep null-prompt configuration is load-bearing and the method is not robust to that choice.

Watch

Extended reading notes

Core claim

The central claim is that the spatial information CLIP lacks for dense prediction is already present in its own early layers and in Stable Diffusion's self-attention, and that a two-stage fusion makes both usable. The first stage replaces the last-layer self-attention of CLIP with the average of attention maps from all earlier layers, feeds the early-layer patch embeddings through that modified last layer (skipping the FFN and residuals), and averages the cosine similarities of the resulting embeddings to text embeddings to form a coarse map. The second stage takes the multi-head self-attention maps of Stable Diffusion at its highest spatial resolution, fuses the heads by matrix chain multiplication, and multiplies this fused map into the upscaled coarse map to sharpen local details. On the paper's own evaluation, this hierarchy outperforms prior training-free methods on all seven datasets, with the largest gains coming from the early-layer fusion (roughly +10 mIoU on VOC) and further gains from the diffusion compensation (roughly +8.6 mIoU on VOC).

Load-bearing premise

The fine-grained compensation assumes that self-attention maps from Stable Diffusion, extracted at a single fixed timestep (45 of 50) with a null text prompt and fused by multiplying all heads, provide boundary-accurate spatial refinement that transfers across all seven datasets without tuning.

Editorial extensions

If this is right

  • A frozen CLIP plus a frozen Stable Diffusion can act as a strong open-vocabulary segmenter with no parameter updates, replacing trained segmentation heads for arbitrary category sets.
  • The early-layer averaged attention alone (without diffusion compensation) already outperforms several methods that modify only the last CLIP layer, indicating CLIP's early layers are an underused spatial resource.
  • The hierarchy also improves image-level classification metrics computed by max-pooling the segmentation maps, so better masks translate into better category recognition on all evaluated datasets.
  • The same pipeline produces pseudo-masks that outperform weakly supervised training-based methods on VOC and COCO training sets, suggesting it can serve as a label generator for downstream training.
  • Because the method is training-free, scaling to larger backbones (e.g., ViT-H) or higher-resolution inputs is a matter of compute, not re-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The diffusion compensation is tested at exactly one timestep (45 of 50) with a null prompt; a natural stress test is to sweep the timestep and prompt, since the reported gains could be sensitive to that single setting.
  • The matrix-product fusion of attention heads is a strong inductive choice; whether a single head or an average would transfer more stably across datasets is not addressed and could be tested directly.
  • The hierarchy suggests a general recipe for dense prediction from image-level models: use early-layer signals for coherence and a generative model's attention for boundaries; this may extend to depth estimation or edge detection with the same coarse-to-fine machinery.
  • The paper notes tiny objects remain hard; this may point to the diffusion feature resolution rather than the fusion scheme, and a high-resolution Stable Diffusion variant would be a direct fix to test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CLIPer, a training-free hierarchical framework for open-vocabulary semantic segmentation. It improves CLIP's spatial representations with two modules: an early-layer fusion module that averages self-attention maps from early CLIP layers and uses the result in place of the last-layer self-attention, and a fine-grained compensation module that extracts multi-head self-attention maps from Stable Diffusion V2.1 (with a null prompt at timestep 45/50), fuses them by matrix multiplication (Eq. 9), and applies the fused map to refine the coarse CLIP-based segmentation map. The method is evaluated on seven segmentation benchmarks with ViT-B and ViT-L backbones, reporting state-of-the-art mIoU (e.g., 69.8% on VOC and 43.3% on COCO Object with ViT-L). Ablations show monotonic gains: early-layer fusion adds ~10.0 mIoU on VOC, fine-grained compensation adds ~8.6 mIoU, and the combined pipeline adds 18.6 mIoU over the baseline.

Significance. If the reported results are reproducible, CLIPer is a meaningful advance in training-free open-vocabulary segmentation: the framework is simple, requires no training, is substantially faster than ProxyCLIP without the diffusion refinement, and the ablation pattern is clean and monotonic. The authors also state that source code will be released. The main risk to significance is that the state-of-the-art claim is carried by the fine-grained compensation module, whose key hyperparameters (diffusion timestep, null prompt, SD checkpoint, head-fusion rule) are not sensitivity-tested, and all comparisons are single-run numbers without variance. On Context and ADE, the no-refinement variant CLIPer* is actually below ProxyCLIP, so the decisive margin relies on choices that are currently under-validated.

major comments (4)
  1. [§4.2, Table 1; §4.3, Table 5] The abstract's state-of-the-art claim is carried by fine-grained compensation (FGC), but the paper reports one run per configuration without error bars or seed variance. Without FGC, CLIPer* reaches 34.3 on Context and 21.8 on ADE, below ProxyCLIP's 34.5 and 22.6; on VOC and Object the margin is only 0.6 and 0.4 mIoU. These gaps are small enough to fall within run-to-run or checkpoint variation, so the superiority over ProxyCLIP on those benchmarks is not established. I request at least three independent runs (or a variance estimate) for the main table and for the no-FGC variant, and a statement of how the reported number was selected.
  2. [§4.1, §4.3, Eq. (9)] The FGC module is specified by several unvalidated choices: SD V2.1, timestep 45 of 50, a null text prompt, and matrix-chain fusion of all attention heads. Table 7 varies only the head-fusion strategy at a single timestep, and the 'Single' row reports the best-performing head, which is itself a validation-set selection. No timestep sweep, prompt variation, SD-version check, or per-dataset analysis is reported. Since FGC is what lifts CLIPer above ProxyCLIP on Context and ADE, the paper should show that the results are robust to these choices or demonstrate that the choices were fixed on a separate validation split.
  3. [§3.3, Eq. (9)–(10)] The fine-grained compensation is underspecified as a spatial operation. The paper does not state the spatial resolution L at which SD self-attention is extracted, how the coarse map S_c is upsampled to L, or why the product of all H attention heads (Eq. 9) is a meaningful fused operator, given that the product is not invariant to head ordering. Please provide the exact shapes and resizing steps, and report at least a sanity check on head order or a comparison with a fixed head permutation.
  4. [§3.3, Fig. 4] The early-layer fusion module is described too loosely for reproduction: 'we feed all the embeddings to the last layer' and 'generate multiple output embeddings for different layers' do not specify which embeddings are fed, which layers' outputs are used, or how the multiple similarity maps are averaged. This is central to the method; please give a precise layer-by-layer pseudocode or formal equations.
minor comments (4)
  1. [Throughout] Typos and wording: Abstract 'sematic' should be 'semantic'; §3.3 'spatail' should be 'spatial', 'corse' should be 'coarse', and 'futher' should be 'further'; Table 5 'difference modules' should be 'different modules'; Fig. 2 caption 'path embeddings' should be 'patch embeddings'.
  2. [§4.1] The exact text prompt template for category descriptions is not given; please state it, since the text side is part of the method and affects reproducibility.
  3. [Table 1] The text says 'almost achieves the best' and the footnote excludes VOC20 for ViT-L, while the abstract claims state-of-the-art on seven datasets; reconcile these statements.
  4. [Fig. 2] The 3D visualization has no axis labels and the similarity curve lacks explicit layer indices, making the motivation hard to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CLIPer is a training-free pipeline evaluated on external benchmarks, with no fitted parameter reported as a prediction and no load-bearing self-citation.

full rationale

The paper derives its results from fixed, pre-trained components (CLIP and Stable Diffusion V2.1) and evaluates on standard external benchmarks. The claimed mIoU improvements come from an architectural pipeline that is fully specified before evaluation: early-layer fusion of CLIP embeddings and attention maps, followed by fine-grained compensation using SD self-attention maps via Equation 9. No parameter is fitted to the test labels and then renamed as a prediction. The hyperparameter choices mentioned in Section 4.1 (timestep 45 of 50, null text prompt) and the head-fusion strategy in Table 7 are standard model-selection decisions made before reporting final numbers; even the Table 7 'Single' row, which reports the best-performing head, is an ablation baseline rather than the final method and does not enter the headline claim. The paper cites prior work by its own authors (iSeg, Ref. [32]) only in related work and in a comparison table, not as the justification for any derivation step, so the self-citation is not load-bearing. No uniqueness theorem, ansatz-smuggling citation, or renaming of a known result is present. The central claim is therefore self-contained against external benchmarks and does not reduce by construction to its inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or model-level entities; it reuses CLIP and Stable Diffusion as frozen feature extractors. The free parameters are the SD timestep and the head-fusion strategy, both chosen empirically. The axioms are empirical observations and ad hoc fusion designs that the paper supports with visualizations and ablations, not with formal derivations.

free parameters (2)
  • Stable Diffusion timestep for attention extraction = 45 (of 50 steps)
    Set in Sec. 4.1 with no ablation across timesteps; the FGC module's gains (e.g., +8.6 mIoU on VOC) depend on this choice.
  • SD multi-head attention fusion strategy = matrix multiplication
    Chosen in Table 7 by comparing multiplication, mean, and single-best-head on the same validation benchmarks; no theoretical justification.
assumptions (5)
  • domain assumption Early-layer CLIP patch embeddings and attention maps preserve spatial structure and share the embedding space with the last layer.
    Stated as motivation in Sec. 3.2 with Fig. 2; used to justify averaging early-layer attention into the last layer.
  • ad hoc to paper Averaging attention maps from all early layers and substituting the result for the last layer attention map is a valid operation.
    Eq. 8 in Sec. 3.3; no theoretical justification, only empirical improvement in ablations.
  • domain assumption Stable Diffusion self-attention maps capture local details better than CLIP attention maps.
    Sec. 3.2, Fig. 3; the fine-grained compensation module relies on this property.
  • ad hoc to paper Matrix multiplication of all SD attention heads yields a meaningful fused attention operator.
    Eq. 9 in Sec. 3.3; selected empirically over mean and single-head strategies in Table 7.
  • domain assumption A null text prompt in Stable Diffusion still produces useful self-attention maps for spatial refinement.
    Sec. 3.3; SD is run with an empty prompt, so no text guidance is used for the refinement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation." pith.science (2026). https://pith.science/paper/DDIUHV2X

@misc{pith2026241113836,
  author       = {Pith},
  title        = {Pith review of: CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDIUHV2X}},
  note         = {Machine review of arXiv:2411.13836}
}
read the original abstract

Contrastive Language-Image Pre-training (CLIP) exhibits strong zero-shot classification ability on various image-level tasks, leading to the research to adapt CLIP for pixel-level open-vocabulary semantic segmentation without additional training. The key is to improve spatial representation of image-level CLIP, such as replacing self-attention map at last layer with self-self attention map or vision foundation model based attention map. In this paper, we present a novel hierarchical framework, named CLIPer, that hierarchically improves spatial representation of CLIP. The proposed CLIPer includes an early-layer fusion module and a fine-grained compensation module. We observe that, the embeddings and attention maps at early layers can preserve spatial structural information. Inspired by this, we design the early-layer fusion module to generate segmentation map with better spatial coherence. Afterwards, we employ a fine-grained compensation module to compensate the local details using the self-attention maps of diffusion model. We conduct the experiments on seven segmentation datasets. Our proposed CLIPer achieves the state-of-the-art performance on these datasets. For instance, using ViT-L, CLIPer has the mIoU of 69.8% and 43.3% on VOC and COCO Object, outperforming ProxyCLIP by 9.2% and 4.1% respectively.

Figures

Figures reproduced from arXiv: 2411.13836 by the authors.

Figure 1
Figure 1. Comparison with existing CLIP-based open-vocabulary [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Visualization of path embeddings in CLIP. In (a), we vi [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization of self-attention maps between CLIP and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Overall architecture of our proposed method CLIPer. Our CLIPer contains two components: early-layer fusion and fine-grained [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Visualization of fine-grained compensation. Given the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 44 canonical work pages

  1. [1]

    Weakly su- pervised learning of instance segmentation with inter-pixel relations

    Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly su- pervised learning of instance segmentation with inter-pixel relations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2209–2218, 2019. 7

    work page 2019

  2. [2]

    Zero-shot semantic segmentation

    Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick P´erez. Zero-shot semantic segmentation. In Advances in Neural Information Processing Systems , pages 468–479,

    work page

  3. [3]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1209–1218, 2018. 5

    work page 2018

  4. [4]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision , pages 9650–19660, 2021. 2

    work page 2021

  5. [5]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. InIEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 11165–11174, 2023. 2, 6

    work page 2023

  6. [6]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1290–1299, 2022. 2

    work page 2022

  7. [7]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023. 2

    work page 2023

  8. [8]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmentation

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint arXiv:2303.11797, 2023. 1, 2

    arXiv 2023

Show all 51 references
  1. [9]

    Open- vocabulary panoptic segmentation with maskclip

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open- vocabulary panoptic segmentation with maskclip. arXiv preprint arXiv:2208.08984, 2022. 1, 2, 3, 7

    2022 arXiv

  2. [10]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint a...

    2010 arXiv

  3. [11]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303–338, 2010. 5

    2010

  4. [12]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International Conference on Machine Learning, pages 4904–491...

    2021

  5. [13]

    Diffusion models for zero-shot open-vocabulary segmentation

    Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. In European Conference on Computer Vision,

  6. [14]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023. 2

    2023

  7. [15]

    ProxyCLIP: Proxy attention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. ProxyCLIP: Proxy attention improves clip for open-vocabulary segmentation. In European Conference on Computer Vision, 2024. 1, 2, 3, 5, 6, 7

    2024

  8. [16]

    ClearCLIP: Decom- posing clip representations for dense vision-language infer- ence

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. ClearCLIP: Decom- posing clip representations for dense vision-language infer- ence. In European Conference on Computer Vision, 2024. 1, 2, 3, 4, 5, 6, 7

    2024

  9. [17]

    Anti- adversarially manipulated attributions for weakly and semi- supervised semantic segmentation

    Jungbeom Lee, Eunji Kim, and Sungroh Yoon. Anti- adversarially manipulated attributions for weakly and semi- supervised semantic segmentation. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 4071–4080, 2021. 7

    2021

  10. [18]

    A closer look at the explainability of contrastive language-image pre-training

    Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, and Xiaomeng Li. A closer look at the explainability of contrastive language-image pre-training. arXiv preprint arXiv:2304.05653, 2023. 6

    2023 arXiv

  11. [19]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061–7070, 2023. 2

    2023

  12. [20]

    CLIP is also an efficient segmenter: A text-driven approach for weakly su- pervised semantic segmentation

    Yuqi Lin, Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofei He. CLIP is also an efficient segmenter: A text-driven approach for weakly su- pervised semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15...

    2023

  13. [21]

    TagCLIP: A local-to-global framework to enhance open-vocabulary multi-label classification of clip without training

    Yuqi Lin, Minghao Chen, Kaipeng Zhang, Hengjia Li, Ming- ming Li, Zheng Yang, Dongqin Lv, Binbin Lin, Haifeng 9 Liu, and Deng Cai. TagCLIP: A local-to-global framework to enhance open-vocabulary multi-label classification of clip without training. In AAAI Conference on Artific...

    2024

  14. [22]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 431–440, 2015. 2

    2015

  15. [23]

    SegCLIP: Patch aggregation with learnable centers for open-vocabulary semantic segmentation

    Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. SegCLIP: Patch aggregation with learnable centers for open-vocabulary semantic segmentation. In In- ternational Conference on Machine Learning, pages 23033– 23044, 2023. 2, 6

    2023

  16. [24]

    The role of context for object detection and semantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition , pages 891–898,

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  18. [26]

    Per- ceptual grouping in contrastive vision-language models

    Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, and Jonathon Shlens. Per- ceptual grouping in contrastive vision-language models. In IEEE/CVF International Conference on Computer Vision , pages 5548–5561, 2023. 2

    2023

  19. [27]

    ViewCo: Discovering text-supervised segmentation masks via multi-view semantic consistency

    Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. ViewCo: Discovering text-supervised segmentation masks via multi-view semantic consistency. In International Con- ference on Learning Representations, 2023. 2, 6

    2023

  20. [28]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10674– 10685, 2022. 3, 5

    2022

  21. [29]

    To- ken contrast for weakly-supervised semantic segmentation

    Lixiang Ru, Heliang Zheng, Yibing Zhan, and Bo Du. To- ken contrast for weakly-supervised semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093–3102, 2023. 7

    2023

  22. [30]

    Laion-5B: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Ross Wightman Cade Gordon, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laio...

    2022

  23. [31]

    ReCo: Re- trieve and co-segment for zero-shot transfer

    Gyungin Shin, Weidi Xie, and Samuel Albanie. ReCo: Re- trieve and co-segment for zero-shot transfer. In Interna- tional Conference on Neural Information Processing Sys- tems, pages 33754–33767, 2022. 2

    2022

  24. [32]

    iSeg: iseg: An iterative refinement-based framework for training-free segmentation

    Lin Sun, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. iSeg: iseg: An iterative refinement-based framework for training-free segmentation. arXiv preprint arXiv:2409.03209, 2024. 3, 7

    2024 arXiv

  25. [33]

    CLIP as RNN: Segment countless visual concepts with- out training endeavor

    Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. CLIP as RNN: Segment countless visual concepts with- out training endeavor. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 13171–13182,

  26. [34]

    Sclip: Rethinking self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In Euro- pean Conference on Computer Vision, pages 315–332, 2024. 1, 2, 3, 5, 6

    2024

  27. [35]

    Sam-clip: Merging vision foundation models towards semantic and spatial understanding

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. In IEEE/CVF Conferenc...

    2024

  28. [36]

    Diffusion model is secretly a training-free open vocabulary semantic segmenter

    Jinglong Wang, Xiawei Li, Jing Zhang, Qingyuan Xu, Qin Zhou, Qian Yu, Lu Sheng, and Dong Xu. Diffusion model is secretly a training-free open vocabulary semantic segmenter. arXiv preprint arXiv:2309.02773, 2023. 3, 7

    2023 arXiv

  29. [37]

    Clip-dinoiser: Teaching clip a few dino tricks for open- vocabulary semantic segmentation

    Monika Wysocza ´nska, Oriane Sim´eoni, Micha¨el Ramamon- jisoa, Andrei Bursuc, Tomasz Trzci ´nski, and Patrick P ´erez. Clip-dinoiser: Teaching clip a few dino tricks for open- vocabulary semantic segmentation. European Conference on Computer Vision, pages 1–18, 2024. 6

    2024

  30. [38]

    From text to mask: Localizing entities using the attention of text-to-image diffusion models

    Changming Xiao, Qi Yang, Zhou Feng, and Changshui Zhang. From text to mask: Localizing entities using the attention of text-to-image diffusion models. arXiv preprint arXiv:2309.04109, 2023. 7

    2023 arXiv

  31. [39]

    Sed: A simple encoder-decoder for open- vocabulary semantic segmentation

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- vocabulary semantic segmentation. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 3426–3436, 2024. 2

    2024

  32. [40]

    Alvarez, and Ping Luo

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. SegFormer: Simple and effi- cient design for semantic segmentation with transformers. In International Conference on Neural Information Processing Systems, pages 12077–12090, 2021. 2

    2021

  33. [41]

    Clims: Cross language image matching for weakly supervised se- mantic segmentation

    Jinheng Xie, Xianxu Hou, Kai Ye, and Linlin Shen. Clims: Cross language image matching for weakly supervised se- mantic segmentation. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 4483–4492,

  34. [42]

    Rewrite caption semantics: Bridging seman- tic gaps for language-supervised semantic segmentation

    Yun Xing, Jian Kang, Aoran Xiao, Jiahao Nie, Shao Ling, and Shijian Lu. Rewrite caption semantics: Bridging seman- tic gaps for language-supervised semantic segmentation. In Advances in Neural Information Processing Systems , pages 68798–68809, 2023. 6

    2023

  35. [43]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134–18144, 2022. 6 10

    2022

  36. [44]

    Learning open-vocabulary seman- tic segmentation models from natural language supervision

    Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary seman- tic segmentation models from natural language supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2935–2944, 2023. 2, 6

    2023

  37. [45]

    Open-vocabulary panoptic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xi- aolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955–2966, 2023. 2, 3

    2023

  38. [46]

    Multi-class token transformer for weakly supervised se- mantic segmentation

    Lian Xu, Wanli Ouyang, Mohammed Bennamoun, and Dan Xu. Multi-class token transformer for weakly supervised se- mantic segmentation. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 4310–4319,

  39. [47]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2945–2954, 2023. 1, 2

    2023

  40. [48]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. arXiv preprint arXiv:2308.02487, 2023. 1

    2023 arXiv

  41. [49]

    Open vocabulary scene parsing

    Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In IEEE/CVF International Conference on Computer Vision , pages 2002–2010, 2017. 1

    2002

  42. [50]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019. 5

    2019

  43. [51]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Com- puter Vision, pages 696–712, 2022. 1, 2, 5, 6 11

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.