Pith. sign in

REVIEW 4 major objections 5 minor 85 references

Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a frozen fragment of CLIP's visual encoder — the value/FFN sub-block of the final layer plus the fixed projection head — is enough to map segmentation features into CLIP's semantic space, and that Chimera-Seg…

desk verdict Solid incremental extension of CLIP2Seg with a genuinely interesting partial-CLIP head; the headline hIoU gains are small and partly ride on a tuned gamma bias, so the 'consistently best' claim overreaches. read the letter →

arxiv 2506.22032 v1 pith:VUHZBR7N submitted 2025-06-27 cs.CV

Jialei Chen , Xu Zheng , Danda Pani Paudel , Luc Van Gool , Hiroshi Murase , Daisuke Deguchi This is my paper
classification cs.CV
keywords zero-shotsemanticsegmentationCLIPknowledgedistillationfrozenvisualencoderselectivealignmenttransductivelearningopen-vocabulary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that transferring CLIP's vision-language alignment to a segmentation model does not require the full CLIP visual encoder or even its query-key attention. Chimera-Seg pairs a trainable segmentation backbone with a CLIP Semantic Head built from a frozen sub-block of CLIP's final transformer layer — value projection and feed-forward network with residuals — plus CLIP's frozen projection MLP, so only a small linear projector and a batch-normalization layer are trained. Selective Global Distillation aligns a Gumbel-softmax-selected, time-decayed subset of dense features with the CLIP class token, and a Semantic Alignment Module matches visual prototypes with frozen text embeddings. On COCO-Stuff and PASCAL Context the paper reports the best zero-shot harmonic IoU, improving the CLIP2Seg baseline by 0.9 and 1.2 points with SegFormer-B4 backbones in inductive and transductive settings. The paper's own limitation note concedes that the mechanism behind CSH's improvement remains underexplored and that SGD's selection occasionally activates irrelevant regions.

What carries the argument

The load-bearing object is the CLIP Semantic Head (CSH), built from a frozen fragment of the last transformer block of CLIP's visual encoder — first layer norm, value projection, second layer norm, feed-forward network, and residuals — followed by CLIP's frozen visual MLP projection head. Its job is to map dense segmentation features into CLIP's text-aligned semantic space with no trainable attention weights. The two auxiliary mechanisms are Selective Global Distillation (SGD), which uses Gumbel-softmax-decayed top-K sampling to aggregate features most similar to the CLS token into a global feature aligned by an InfoNCE-style loss, and Semantic Alignment Module (SAM), which aligns seen-class prototypes and text embeddings with the text-encoder CLS token via a KL divergence.

What would settle it

Run the COCO-Stuff inductive benchmark with the full final CLIP transformer block, including query-key attention, in place of the partial VEncoder while keeping everything else fixed: if hIoU does not drop below the reported 44.8, the claim that the query-key path is dispensable is falsified.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a frozen 'partial CLIP' can replace both full-encoder adaptation and joint optimization of segmentation plus vision-to-text projection. The CSH projects backbone features through a trainable MLP into CLIP's embedding dimension, pushes them through the frozen last-block sub-module (LN1 to value projection, LN2 to FFN, each with residual), applies a trainable batch-normalization with another residual, and finally maps through CLIP's frozen visual projection MLP, yielding features in the same semantic space as the text embeddings. The paper argues that the value branch, not the query-key attention, carries the semantic content needed for dense prediction, while the segmentation backbone supplies spatial precision. With SGD and SAM, the model is claimed to achieve the best hIoU on both benchmarks in both inductive and transductive settings, and ablations show that freezing the VEncoder and the projection head is better than making either learnable.

Load-bearing premise

The load-bearing premise is that the query-key attention path of CLIP's final visual transformer block is unnecessary for semantic alignment, and that the remaining frozen value/FFN sub-block plus CLIP's fixed projection head is enough to put dense features into CLIP's semantic space.

Editorial extensions

If this is right

  • A single frozen VEncoder sub-block is the effective bridge: using two or three blocks degrades hIoU, and making the value path or FFN learnable also hurts.
  • The decayed top-K strategy in SGD matters: decreasing the number of selected features as training progresses improves uIoU, while no decay or increasing decay does not.
  • Freezing CLIP's final projection head is essential: making it learnable, even from CLIP initialization, drops uIoU and hIoU, so the fixed projection preserves zero-shot transfer.
  • The framework is backbone-agnostic: with both SegFormer-B4 and ViT-B backbones it improves over the CLIP2Seg baseline, supporting the plug-and-play claim.
  • Batch normalization gives more balanced layer-wise similarity with deep CLIP layers than fixed or learnable LayerNorm, explaining the normalization choice in CSH.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If a value-only frozen sub-block is truly sufficient, the same recipe may transfer to other dense prediction tasks such as panoptic segmentation, depth estimation, or open-vocabulary detection, where a small frozen fragment of a vision-language model could serve as the alignment head.
  • Editorial inference: The decayed top-K distillation is a form of curriculum that could be generalized beyond a hand-decay schedule, for example by adapting the selection size to per-image confidence or semantic diversity; this is a testable extension the paper does not explore.
  • Editorial inference: The result suggests a structural hypothesis about CLIP's internal division of labor — query-key attention mainly serves global discrimination, while the value/FFN path carries transferable semantic content — which could be probed directly by analyzing CLIP features on dense tasks.
  • Editorial inference: Since the paper's own limitation statement notes SGD occasionally activates irrelevant regions, an uncertainty- or saliency-gated selection mechanism is a natural follow-up that might improve the distillation target.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Chimera-Seg for zero-shot semantic segmentation. The model couples a trainable segmentation backbone (SegFormer-B4 or ViT-B) with a CLIP Semantic Head (CSH) that projects backbone features into CLIP's joint embedding space using a frozen partial CLIP transformer block (value projection and FFN) and the frozen CLIP visual projection head, while only a linear projector and a BatchNorm layer are trained. Two auxiliary objectives are added: Selective Global Distillation (SGD), which selects a gumbel-softmax top-K subset of dense features, aggregates them, and aligns the aggregate with the CLIP CLS token under a decaying K schedule, and a Semantic Alignment Module (SAM), which aligns seen-class visual prototypes with CLIP text embeddings. Experiments on COCO-Stuff and PASCAL Context report inductive and transductive sIoU/uIoU/hIoU, with claimed gains of 0.9 and 1.2 hIoU points over the CLIP2Seg baseline.

Significance. The architectural idea is attractive, and the empirical study is substantial: the paper includes incremental ablations of CSH/SGD/SAM, component ablations of the VEncoder, normalization choices, the number of VEncoder blocks, the SGD decay schedule and loss weight, plus an ADE20K cross-dataset check. These ablations are internally consistent and provide useful evidence about which components matter. If the reported gains are robust, the method offers a lightweight recipe for transferring CLIP alignment to dense prediction. The main concerns are the small size of the headline gains relative to a tuned logit bias, the absence of variance estimates, and the mismatch between the paper's 'consistently best' wording and its own tables; these substantially temper the strength of the claim as written.

major comments (4)
  1. [Sec. IV-C, Table I] The sentence 'Our method consistently achieves the best performance across all metrics and benchmarks' is not supported by the paper's own results. With the ViT-B backbone on PASCAL Context, Chimera-Seg (50.1 sIoU / 62.5 uIoU / 55.6 hIoU) is worse than CLIP2Seg (52.6 / 64.5 / 58.0) on every metric; on COCO-Stuff with ViT-B, uIoU and hIoU are lower (44.1 / 43.7 vs. 44.7 / 43.9); and on COCO-Stuff with Segformer-B4, sIoU is marginally lower (43.1 vs. 43.2). The abstract's 0.9% and 1.2% hIoU improvements therefore hold only for the Segformer-B4 row. The claims in Sec. IV-C and the abstract should be reworded to state precisely which configuration and metric is being compared.
  2. [Sec. IV-B, Fig. 10] Because the claimed gains over CLIP2Seg are only 0.9 and 1.2 hIoU points, the unseen-class logit bias gamma=0.5 is a plausible alternative explanation for part of the improvement. The manuscript does not state whether CLIP2Seg was evaluated with the same gamma value, how gamma was chosen (validation set or test set), or how variable the results are across training runs. Fig. 10 shows that Chimera-Seg's uIoU/hIoU is sensitive to gamma while CLIP2Seg is more robust. Please report a matched-gamma comparison for both methods, describe the model selection procedure, and give standard deviations over at least three seeds; without this, the benefit of CSH/SGD/SAM over calibration is not clearly identified.
  3. [Sec. III-B, Table VI] The paper's central thesis, that discarding the query-key attention path from CLIP's transformer block is sufficient, is not directly tested. Table VI ablates only the value projection (V) and FFN within the chosen sub-block; there is no comparison against a complete CLIP transformer block that retains the QK path, nor against a block with only QK and no value projection. Since the title and Sec. III-B make 'partial CLIP is enough' a load-bearing claim, add an ablation that keeps the QK attention (or the full frozen block) to show that the omitted path is indeed unnecessary.
  4. [Sec. III-C, Eq. (3); Sec. IV-B] The SGD objective is mis-specified. Eq. (3) is written as a normalized probability (a softmax ratio) rather than as an InfoNCE loss, and the negative samples C_j are never defined; if the printed equation is minimized directly, the model would maximize a probability rather than minimize a contrastive loss. In addition, the K schedule is ambiguous: Sec. IV-B says K starts at 9000 and is 'decreased 0.1 as each iteration', while Table XI labels rows as 'num decay 3000/5000/9000' without stating whether these are initial K values. Please give the exact decay formula, define C_j, and confirm that K never exceeds the spatial size of the dense feature map used for selection.
minor comments (5)
  1. [General] The manuscript contains numerous typos and grammar errors, including 'boardly categoriezd', 'alignemnt', 'Segmenatation', 'Intersetcion-over-Union', 'diclaim', and 'konwledge'; a careful proofread is needed.
  2. [Sec. IV-A] The sentence 'This dataset is split into 15 seen and 5 unseen classes' appears immediately after introducing PASCAL Context but COCO-Stuff is described with 156/15 classes; please state the seen/unseen split explicitly for each dataset.
  3. [Sec. III-C, Eqs. (1)-(3)] Equation (2) introduces sampling weights W, then the text defines W' = softmax(...) and later uses W_k to compute F_g; W_k is never defined, and the relationship among W, W', and W_k is unclear.
  4. [Fig. 10] The horizontal axis is labeled gamma but the curves for uIoU and hIoU have no visible axis labels or legend in the figure, which makes the sensitivity comparison hard to read.
  5. [References] Reference [29] is cited as 'Anonymous'; if it refers to the authors' own concurrent manuscript, this must be disclosed and updated prior to publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: distillation pipeline is self-contained and externally evaluated; remaining concerns are calibration/fairness, not definitional.

full rationale

No load-bearing circular step was found. The method is a knowledge-distillation pipeline: trainable backbone features are projected through a frozen CLIP sub-block (value projection plus FFN) and the frozen CLIP MLP projection head, then aligned to CLS and text embeddings via SGD, SAM, and pseudo-mask segmentation losses. These alignment objectives are deliberately chosen training signals, not hidden restatements of the evaluation metric. Final predictions are made by computing similarity between projected dense features and text embeddings, and are evaluated against ground-truth labels using standard hIoU, so the result is not defined by the training objective. The premise that query-key attention can be discarded is supported by external references [22,23] and by the paper's own ablations (Tables IV, VI, VIII), not by a self-citation or an imported uniqueness theorem. Self-citations [30,31] appear only as related work and are not used to justify the model's correctness. The gamma bias applied to unseen-class logits (Sec. IV.B) and the small hIoU margins relative to CLIP2Seg raise legitimate evaluation-fairness and robustness questions (Fig. 10, Table I), but these are calibration/statistical concerns, not circularity by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard CLIP distillation assumptions: CLIP's frozen embeddings are a valid target space, pseudo-labels from CLIP's visual encoder are reliable enough, and the value/FFN path of the final CLIP block carries the semantic information needed. These are domain assumptions from the prior literature; the paper does not prove them.

free parameters (4)
  • Gamma logit bias for unseen classes = 0.5 (COCO-Stuff inductive)
    A per-dataset additive bias on unseen-class logits during inference, tuned by validation sweep in Fig. 10; directly boosts uIoU and hIoU.
  • Initial K and decay schedule in SGD = 9000, reduce by 0.1 per iteration
    Chosen by ablation in Table XI; as written it exceeds the spatial feature count for 512x512 inputs, so the exact schedule is ambiguous.
  • Loss weights and temperatures = tau=0.07, tau_c=0.01, tau_f=0.07, lambda_sam=0.1
    Hyperparameters, partly inherited from CLIP2Seg, selected by experiments rather than derived.
  • VEncoder block count = 1
    Number of frozen CLIP transformer blocks used in the semantic head, chosen by ablation in Table VIII.
assumptions (5)
  • standard math Gumbel-Softmax reparameterization, softmax, KL divergence, and InfoNCE are valid for the proposed losses.
    Used in Sec. III-C and Sec. III-D without proof; standard tools.
  • domain assumption CLIP's frozen visual and text encoders define a semantic space that supports zero-shot transfer to unseen classes.
    The whole method assumes CLIP's pretrained vision-language alignment is a reliable teacher; stated throughout Sec. I and III.
  • domain assumption The query-key attention path in CLIP's visual transformer is not needed for dense vision-language alignment.
    Invoked in Sec. III-B when the VEncoder removes attention and keeps only value projection and FFN, citing [22,23].
  • domain assumption Pseudo-masks generated by multi-scale K-Means and mask merging from CLIP are reliable enough to supervise training.
    Introduced in Sec. III-A and used in Eq. 5 and the segmentation loss; inherited from CLIP2Seg [29].
  • domain assumption The frozen final projection MLP of the CLIP visual encoder maps features into the same text-aligned space as semantic embeddings.
    Assumed in Sec. III-B and supported only by ablation Table V, not by an independent derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation." pith.science (2026). https://pith.science/paper/VUHZBR7N

@misc{pith2026250622032,
  author       = {Pith},
  title        = {Pith review of: Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VUHZBR7N}},
  note         = {Machine review of arXiv:2506.22032}
}
read the original abstract

Zero-shot Semantic Segmentation (ZSS) aims to segment both seen and unseen classes using supervision from only seen classes. Beyond adaptation-based methods, distillation-based approaches transfer vision-language alignment of vision-language model, e.g., CLIP, to segmentation models. However, such knowledge transfer remains challenging due to: (1) the difficulty of aligning vision-based features with the textual space, which requires combining spatial precision with vision-language alignment; and (2) the semantic gap between CLIP's global representations and the local, fine-grained features of segmentation models. To address challenge (1), we propose Chimera-Seg, which integrates a segmentation backbone as the body and a CLIP-based semantic head as the head, like the Chimera in Greek mythology, combining spatial precision with vision-language alignment. Specifically, Chimera-Seg comprises a trainable segmentation model and a CLIP Semantic Head (CSH), which maps dense features into the CLIP-aligned space. The CSH incorporates a frozen subnetwork and fixed projection layers from the CLIP visual encoder, along with lightweight trainable components. The partial module from CLIP visual encoder, paired with the segmentation model, retains segmentation capability while easing the mapping to CLIP's semantic space. To address challenge (2), we propose Selective Global Distillation (SGD), which distills knowledge from dense features exhibiting high similarity to the CLIP CLS token, while gradually reducing the number of features used for alignment as training progresses. Besides, we also use a Semantic Alignment Module (SAM) to further align dense visual features with semantic embeddings extracted from the frozen CLIP text encoder. Experiments on two benchmarks show improvements of 0.9% and 1.2% in hIoU.

Figures

Figures reproduced from arXiv: 2506.22032 by the authors.

Figure 1
Figure 1. Existing methods can be broadly categorized into (a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of our proposed method. Given an input image, we utilize the CLIP visual encoder to generate [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The overview of CLIP Semantic Head (CSH). Given a feature map F ∈ R H×W×C extracted from a trainable backbone (e.g., SegFormer [38]) where C indicates the channel number of the backbone features, we first apply a trainable MLP to project the features into CLIP’s embedding dimension D, resulting in Fp = MLP(F). While this projection matches the dimensionality of CLIP, it does not guarantee semantic or structural comp… view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: The overview of inference. decreases over time. In practice, we reduce K by 0.1 per iteration and round it to the nearest integer. This decay mechanism encourages the model to focus on fewer but more semantically reliable regions as training proceeds. It also balances …
Figure 7
Figure 7. Figure 7: Representation structure of ViT-CLIP vs. Chimera-Seg. We visualize four normalization types: no normalization, CLIP [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Similarity visualization between CLS token and dense features. Each row corresponds to the input image, results without [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: The visualization of our method. We compare our method with the CLIP2Seg [ [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Comparison on hIoU and uIoU between our method [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

85 extracted references · 67 canonical work pages

  1. [1]

    Fully convolu- tional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolu- tional networks for semantic segmentation,” in Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440. JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 11

    2015

  2. [2]

    Cpal: Cross- prompting adapter with loras for rgb+ x semantic seg- mentation,

    Y . Liu, P. Wu, M. Wang, and J. Liu, “Cpal: Cross- prompting adapter with loras for rgb+ x semantic seg- mentation,” IEEE Transactions on Circuits and Systems for Video Technology, 2025

    2025

  3. [3]

    Frozen is better than learning: A new design of prototype- based classifier for semantic segmentation,

    J. Chen, D. Deguchi, C. Zhang, X. Zheng, and H. Murase, “Frozen is better than learning: A new design of prototype- based classifier for semantic segmentation,” Pattern Recog- nition, vol. 152, p. 110431, 2024

    2024

  4. [4]

    Both style and distortion matter: Dual-path unsupervised domain adaptation for panoramic semantic segmentation,

    X. Zheng, J. Zhu, Y . Liu, Z. Cao, C. Fu, and L. Wang, “Both style and distortion matter: Dual-path unsupervised domain adaptation for panoramic semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 1285–1295

    2023

  5. [5]

    Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

    X. Zheng, H. Xue, J. Chen, Y . Yan, L. Jiang, Y . Lyu, K. Yang, L. Zhang, and X. Hu, “Learning robust anymodal segmentor with unimodal and cross-modal distillation,” arXiv preprint arXiv:2411.17141 , 2024

    work page Pith review arXiv 2024

  6. [6]

    Distilling efficient vision transformers from cnns for semantic segmentation,

    X. Zheng, Y . Luo, P. Zhou, and L. Wang, “Distilling efficient vision transformers from cnns for semantic segmentation,” Pattern Recognition, vol. 158, p. 111029, 2025

    2025

  7. [7]

    Deep residual learn- ing for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learn- ing for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 770–778

    work page 2016

  8. [8]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

Show all 85 references
  1. [9]

    Sign: Spatial-information incorporated generative network for generalized zero-shot semantic segmentation,

    J. Cheng, S. Nandi, P. Natarajan, and W. Abd-Almageed, “Sign: Spatial-information incorporated generative network for generalized zero-shot semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9556–9566

    2021

  2. [10]

    Zero-shot semantic segmentation,

    M. Bucher, T.-H. Vu, M. Cord, and P. P ´erez, “Zero-shot semantic segmentation,” Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  3. [11]

    Context- aware feature generation for zero-shot semantic segmen- tation,

    Z. Gu, S. Zhou, L. Niu, Z. Zhao, and L. Zhang, “Context- aware feature generation for zero-shot semantic segmen- tation,” in Proceedings of the 28th ACM International Conference on Multimedia , 2020, pp. 1921–1929

    2020

  4. [12]

    Semantic projection network for zero-and few-label semantic segmentation,

    Y . Xian, S. Choudhury, Y . He, B. Schiele, and Z. Akata, “Semantic projection network for zero-and few-label semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 8256–8265

    2019

  5. [13]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning . PMLR, 2021, pp. 8748–8763

    2021

  6. [14]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.- Y . Loet al. , “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4015–4026

    2023

  7. [15]

    Language-driven semantic segmentation,

    B. Li, K. Q. Weinberger, S. Belongie, V . Koltun, and R. Ranftl, “Language-driven semantic segmentation,” in International Conference on Learning Representations , 2022

    2022

  8. [16]

    Scaling open- vocabulary image segmentation with image-level labels,

    G. Ghiasi, X. Gu, Y . Cui, and T.-Y . Lin, “Scaling open- vocabulary image segmentation with image-level labels,” in European Conference on Computer Vision . Springer, 2022, pp. 540–557

    2022

  9. [17]

    Zegclip: Towards adapting clip for zero-shot semantic segmen- tation,

    Z. Zhou, Y . Lei, B. Zhang, L. Liu, and Y . Liu, “Zegclip: Towards adapting clip for zero-shot semantic segmen- tation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 11 175–11 185

    2023

  10. [18]

    Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,

    Q. Yu, J. He, X. Deng, X. Shen, and L.-C. Chen, “Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,” in NeurIPS, 2023

    2023

  11. [19]

    Global knowl- edge calibration for fast open-vocabulary segmentation,

    K. Han, Y . Liu, J. H. Liew, H. Ding, J. Liu, Y . Wang, Y . Tang, Y . Yang, J. Feng, Y . Zhaoet al. , “Global knowl- edge calibration for fast open-vocabulary segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 797–807

    2023

  12. [20]

    Open-vocabulary panoptic segmentation with text-to-image diffusion models,

    J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello, “Open-vocabulary panoptic segmentation with text-to-image diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2955–2966

    2023

  13. [21]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmentation,

    S. Cho, H. Shin, S. Hong, S. An, S. Lee, A. Arnab, P. H. Seo, and S. Kim, “Cat-seg: Cost aggregation for open-vocabulary semantic segmentation,” arXiv preprint arXiv:2303.11797, 2023

    2023 arXiv

  14. [22]

    Extract free dense labels from clip,

    C. Zhou, C. C. Loy, and B. Dai, “Extract free dense labels from clip,” in European Conference on Computer Vision . Springer, 2022, pp. 696–712

    2022

  15. [23]

    Sclip: Rethinking self- attention for dense vision-language inference,

    F. Wang, J. Mei, and A. Yuille, “Sclip: Rethinking self- attention for dense vision-language inference,” in ECCV, 2024

    2024

  16. [24]

    Open- vocabulary semantic segmentation with decoupled one- pass network,

    C. Han, Y . Zhong, D. Li, K. Han, and L. Ma, “Open- vocabulary semantic segmentation with decoupled one- pass network,” in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , 2023, pp. 1086– 1096

    2023

  17. [25]

    Learning mask-aware clip representations for zero-shot segmenta- tion,

    S. Jiao, Y . Wei, Y . Wang, Y . Zhao, and H. Shi, “Learning mask-aware clip representations for zero-shot segmenta- tion,” arXiv preprint arXiv:2310.00240 , 2023

    2023 arXiv

  18. [26]

    A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model,

    M. Xu, Z. Zhang, F. Wei, Y . Lin, Y . Cao, H. Hu, and X. Bai, “A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model,” in European Conference on Computer Vision . Springer, 2022, pp. 736–753

    2022

  19. [27]

    Masked-attention mask transformer for universal image segmentation,

    B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1290–1299

    2022

  20. [28]

    Per-pixel classi- fication is not all you need for semantic segmentation,

    B. Cheng, A. Schwing, and A. Kirillov, “Per-pixel classi- fication is not all you need for semantic segmentation,” Advances in Neural Information Processing Systems , vol. 34, pp. 17 864–17 875, 2021

    2021

  21. [29]

    CLIP-to-seg distillation for inductive zero- shot semantic segmentation,

    Anonymous, “CLIP-to-seg distillation for inductive zero- shot semantic segmentation,” 2025. [Online]. Available: JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 12 https://openreview.net/forum?id=IFaZZmPz5K

    2025

  22. [30]

    Gen- eralizable semantic vision query generation for zero- shot panoptic and semantic segmentation,

    J. Chen, D. Deguchi, C. Zhang, and H. Murase, “Gen- eralizable semantic vision query generation for zero- shot panoptic and semantic segmentation,” arXiv preprint arXiv:2402.13697, 2024

    2024 arXiv

  23. [31]

    Split matching for inductive zero-shot semantic segmentation,

    J. Chen, X. Zheng, D. Li, C. Yi, S. Ito, D. P. Paudel, L. Van Gool, H. Murase, and D. Deguchi, “Split matching for inductive zero-shot semantic segmentation,” arXiv preprint arXiv:2505.05023, 2025

    2025

  24. [32]

    Reducing unimodal bias in multi-modal semantic segmentation with multi-scale functional entropy regularization,

    X. Zheng, Y . Lyu, L. Jiang, D. P. Paudel, L. Van Gool, and X. Hu, “Reducing unimodal bias in multi-modal semantic segmentation with multi-scale functional entropy regularization,” arXiv preprint arXiv:2505.06635 , 2025

    2025 arXiv

  25. [33]

    360sfuda++: Towards source-free uda for panoramic segmentation by learning reliable category prototypes,

    X. Zheng, P. Y . Zhou, A. V . Vasilakos, and L. Wang, “360sfuda++: Towards source-free uda for panoramic segmentation by learning reliable category prototypes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  26. [34]

    Adversarial co-training for semantic segmen- tation over medical images,

    H. Xie, C. Fu, X. Zheng, Y . Zheng, C.-W. Sham, and X. Wang, “Adversarial co-training for semantic segmen- tation over medical images,” Computers in biology and medicine, vol. 157, p. 106736, 2023

    2023

  27. [35]

    Look at the neighbor: Distortion-aware unsupervised domain adaptation for panoramic semantic segmentation,

    X. Zheng, T. Pan, Y . Luo, and L. Wang, “Look at the neighbor: Distortion-aware unsupervised domain adaptation for panoramic semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 18 687–18 698

    2023

  28. [36]

    Encoder-decoder with atrous separable convo- lution for semantic image segmentation,

    L.-C. Chen, Y . Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convo- lution for semantic image segmentation,” in Proceedings of the European Conference on Computer Vision , 2018, pp. 801–818

    2018

  29. [37]

    Semantic matters: A constrained approach for zero-shot video action recognition,

    Z. Quan, J. Chen, D. Deguchi, J. Sun, C. Zhang, Y . Li, and H. Murase, “Semantic matters: A constrained approach for zero-shot video action recognition,” Pattern Recognition, p. 111402, 2025

    2025

  30. [38]

    Segformer: Simple and efficient design for semantic segmentation with transformers,

    E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Advances in Neural Information Processing Systems , vol. 34, 2021

    2021

  31. [39]

    Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,

    S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y . Wang, Y . Fu, J. Feng, T. Xiang, P. H. Torr et al. , “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2021

  32. [40]

    Unified perceptual parsing for scene understanding,

    T. Xiao, Y . Liu, B. Zhou, Y . Jiang, and J. Sun, “Unified perceptual parsing for scene understanding,” in Proceed- ings of the European Conference on Computer Vision , 2018, pp. 418–434

    2018

  33. [41]

    Uncertainty teacher with dense focal loss for semi- supervised medical image segmentation,

    J. Chen, C. Fu, H. Xie, X. Zheng, R. Geng, and C.-W. Sham, “Uncertainty teacher with dense focal loss for semi- supervised medical image segmentation,” Computers in Biology and Medicine , vol. 149, p. 106034, 2022

    2022

  34. [42]

    A good student is cooperative and reliable: Cnn-transformer collaborative learning for semantic segmentation,

    J. Zhu, Y . Luo, X. Zheng, H. Wang, and L. Wang, “A good student is cooperative and reliable: Cnn-transformer collaborative learning for semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 11 720–11 730

    2023

  35. [43]

    Semantics distortion and style matter: Towards source- free uda for panoramic segmentation,

    X. Zheng, P. Zhou, A. V . Vasilakos, and L. Wang, “Semantics distortion and style matter: Towards source- free uda for panoramic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 27 885–27 895

    2024

  36. [44]

    Transformer-cnn cohort: Semi-supervised semantic seg- mentation by the best of both students,

    X. Zheng, Y . Luo, C. Fu, K. Liu, and L. Wang, “Transformer-cnn cohort: Semi-supervised semantic seg- mentation by the best of both students,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 11 147–11 154

    2024

  37. [45]

    Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,

    X. Zheng, Y . Lyu, J. Zhou, and L. Wang, “Centering the value of every modality: Towards efficient and resilient modality-agnostic semantic segmentation,” in European Conference on Computer Vision . Springer, 2024, pp. 192–212

    2024

  38. [46]

    Goodsam: Bridging domain and capacity gaps via segment anything model for distortion-aware panoramic semantic segmen- tation,

    W. Zhang, Y . Liu, X. Zheng, and L. Wang, “Goodsam: Bridging domain and capacity gaps via segment anything model for distortion-aware panoramic semantic segmen- tation,” arXiv preprint arXiv:2403.16370 , 2024

    2024 arXiv

  39. [47]

    Cus- tomize segment anything model for multi-modal semantic segmentation with mixture of lora experts,

    C. Zhu, B. Xiao, L. Shi, S. Xu, and X. Zheng, “Cus- tomize segment anything model for multi-modal semantic segmentation with mixture of lora experts,” arXiv preprint arXiv:2412.04220, 2024

    2024 arXiv

  40. [48]

    Omnisam: Omnidirectional seg- ment anything model for uda in panoramic semantic segmentation,

    D. Zhong, X. Zheng, C. Liao, Y . Lyu, J. Chen, S. Wu, L. Zhang, and X. Hu, “Omnisam: Omnidirectional seg- ment anything model for uda in panoramic semantic segmentation,” arXiv preprint arXiv:2503.07098 , 2025

    2025

  41. [49]

    Magic++: Efficient and resilient modality-agnostic se- mantic segmentation via hierarchical modality selection,

    X. Zheng, Y . Lyu, L. Jiang, J. Zhou, L. Wang, and X. Hu, “Magic++: Efficient and resilient modality-agnostic se- mantic segmentation via hierarchical modality selection,” arXiv preprint arXiv:2412.16876 , 2024

    2024 arXiv

  42. [50]

    Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,

    C. Liao, K. Lei, X. Zheng, J. Moon, Z. Wang, Y . Wang, D. P. Paudel, L. Van Gool, and X. Hu, “Benchmarking multi-modal semantic segmentation under sensor failures: Missing and noisy modality robustness,” arXiv preprint arXiv:2503.18445, 2025

    2025 arXiv

  43. [51]

    Unveiling the potential of segment anything model 2 for rgb-thermal semantic segmentation with language guidance,

    J. Zhao, F. Teng, K. Luo, G. Zhao, Z. Li, X. Zheng, and K. Yang, “Unveiling the potential of segment anything model 2 for rgb-thermal semantic segmentation with language guidance,” arXiv preprint arXiv:2503.02581 , 2025

    2025 arXiv

  44. [52]

    Non-local neural networks,

    X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 7794–7803

    2018

  45. [53]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al. , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  46. [54]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 10 012–10 022

    2021

  47. [55]

    A good student is cooperative and reliable: Cnn-transformer JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 13 collaborative learning for semantic segmentation,

    J. Zhu, Y . Luo, X. Zheng, H. Wang, and L. Wang, “A good student is cooperative and reliable: Cnn-transformer JOURNAL OF LATEX CLASS FILES, VOL. 18, NO. 9, SEPTEMBER 2020 13 collaborative learning for semantic segmentation,” in Proceedings of the IEEE/CVF International Confere...

    2020

  48. [56]

    Language-driven visual consensus for zero-shot semantic segmentation,

    Z. Zhang, W. Ke, Y . Zhu, X. Liang, J. Liu, Q. Ye, and T. Zhang, “Language-driven visual consensus for zero-shot semantic segmentation,” IEEE Transactions on Circuits and Systems for Video Technology , 2024

    2024

  49. [57]

    Zero-shot learning with semantic output codes,

    M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell, “Zero-shot learning with semantic output codes,” Advances in neural information processing systems , vol. 22, 2009

    2009

  50. [58]

    Decoupling zero-shot semantic segmentation,

    J. Ding, N. Xue, G.-S. Xia, and D. Dai, “Decoupling zero-shot semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11 583–11 592

    2022

  51. [59]

    A closer look at self-training for zero- label semantic segmentation,

    G. Pastore, F. Cermelli, Y . Xian, M. Mancini, Z. Akata, and B. Caputo, “A closer look at self-training for zero- label semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2693–2702

    2021

  52. [60]

    Towards open vocabulary learning: A survey,

    J. Wu, X. Li, S. Xu, H. Yuan, H. Ding, Y . Yang, X. Li, J. Zhang, Y . Tong, X. Jiang et al. , “Towards open vocabulary learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 46, no. 7, pp. 5092–5113, 2024

    2024

  53. [61]

    Segment and recognize anything at any granularity,

    F. Li, H. Zhang, P. Sun, X. Zou, S. Liu, C. Li, J. Yang, L. Zhang, and J. Gao, “Segment and recognize anything at any granularity,” in European Conference on Computer Vision. Springer, 2024, pp. 467–484

    2024

  54. [62]

    Dst-det: Simple dynamic self-training for open- vocabulary object detection,

    S. Xu, X. Li, S. Wu, W. Zhang, Y . Tong, and C. C. Loy, “Dst-det: Simple dynamic self-training for open- vocabulary object detection,” IEEE Transactions on Circuits and Systems for Video Technology , 2024

    2024

  55. [63]

    Coco-stuff: Thing and stuff classes in context,

    H. Caesar, J. Uijlings, and V . Ferrari, “Coco-stuff: Thing and stuff classes in context,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 1209–1218

    2018

  56. [64]

    Scene parsing through ade20k dataset,

    B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 633–641

    2017

  57. [65]

    Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,

    L. Wang and K.-J. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 6, pp. 3048–3068, 2021

    2021

  58. [66]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  59. [67]

    Mawkdn: A multimodal fusion wavelet knowledge distillation approach based on cross-view attention for action recognition,

    Z. Quan, Q. Chen, M. Zhang, W. Hu, Q. Zhao, J. Hou, Y . Li, and Z. Liu, “Mawkdn: A multimodal fusion wavelet knowledge distillation approach based on cross-view attention for action recognition,” IEEE Transactions on Circuits and Systems for Video Technology , 2023

    2023

  60. [68]

    Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only,

    J. Chen, D. Zhu, G. Qian, B. Ghanem, Z. Yan, C. Zhu, F. Xiao, S. C. Culatana, and M. Elhoseiny, “Exploring open-vocabulary semantic segmentation from clip vision encoder distillation only,” inProceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 699–710

    2023

  61. [69]

    Distilling detr with visual-linguistic knowledge for open-vocabulary object detection,

    L. Li, J. Miao, D. Shi, W. Tan, Y . Ren, Y . Yang, and S. Pu, “Distilling detr with visual-linguistic knowledge for open-vocabulary object detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 6501–6510

    2023

  62. [70]

    Self pseudo entropy knowledge distillation for semi-supervised semantic segmentation,

    X. Lu, L. Jiao, L. Li, F. Liu, X. Liu, and S. Yang, “Self pseudo entropy knowledge distillation for semi-supervised semantic segmentation,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 34, no. 8, pp. 7359–7372, 2024

    2024

  63. [71]

    CLIPSelf: Vision transformer distills itself for open-vocabulary dense prediction,

    S. Wu, W. Zhang, L. Xu, S. Jin, X. Li, W. Liu, and C. C. Loy, “CLIPSelf: Vision transformer distills itself for open-vocabulary dense prediction,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  64. [72]

    Open-vocabulary object detection via vision and language knowledge distillation,

    X. Gu, T.-Y . Lin, W. Kuo, and Y . Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” in International Conference on Learning Representations, 2022

    2022

  65. [73]

    Primitive generation and semantic-related alignment for universal zero-shot seg- mentation,

    S. He, H. Ding, and W. Jiang, “Primitive generation and semantic-related alignment for universal zero-shot seg- mentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 11 238–11 247

    2023

  66. [74]

    Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation,

    Z. Ma, G. Luo, J. Gao, L. Li, Y . Chen, S. Wang, C. Zhang, and W. Hu, “Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 14 074–14 083

    2022

  67. [75]

    FROSTER: Frozen CLIP is a strong teacher for open-vocabulary ac- tion recognition,

    X. Huang, H. Zhou, K. Yao, and K. Han, “FROSTER: Frozen CLIP is a strong teacher for open-vocabulary ac- tion recognition,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  68. [76]

    Focal loss for dense object detection,

    T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll ´ar, “Focal loss for dense object detection,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 2980–2988

    2017

  69. [77]

    Exploiting a joint embedding space for generalized zero-shot semantic seg- mentation,

    D. Baek, Y . Oh, and B. Ham, “Exploiting a joint embedding space for generalized zero-shot semantic seg- mentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9536–9545

    2021

  70. [78]

    Ex- ploring regional clues in clip for zero-shot semantic seg- mentation,

    Y . Zhang, M.-H. Guo, M. Wang, and S.-M. Hu, “Ex- ploring regional clues in clip for zero-shot semantic seg- mentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 3270–3280

    2024

  71. [79]

    Otseg: Multi-prompt sinkhorn attention for zero-shot semantic segmentation,

    K. Kim, Y . Oh, and J. C. Ye, “Otseg: Multi-prompt sinkhorn attention for zero-shot semantic segmentation,” in Proceedings of the European Conference on Computer Vision, 2024

    2024

  72. [80]

    Freeseg: Unified, universal and open-vocabulary image segmentation,

    J. Qin, J. Wu, P. Yan, M. Li, R. Yuxi, X. Xiao, Y . Wang, R. Wang, S. Wen, X. Pan et al. , “Freeseg: Unified, universal and open-vocabulary image segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 446–19 455. JOUR...

    2023

  73. [81]

    The role of context for object detection and semantic segmentation in the wild,

    R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille, “The role of context for object detection and semantic segmentation in the wild,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2014, pp. 891–898

    2014

  74. [82]

    Mmsegmentation: Openmmlab seman- tic segmentation toolbox and benchmark,

    M. Contributors, “Mmsegmentation: Openmmlab seman- tic segmentation toolbox and benchmark,” 2020

    2020

  75. [83]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1026–1034

    2015

  76. [84]

    Do vision transformers see like convolu- tional neural networks?

    M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy, “Do vision transformers see like convolu- tional neural networks?” Advances in neural information processing systems, vol. 34, pp. 12 116–12 128, 2021

    2021

  77. [85]

    Similarity of neural network representations revisited,

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in International conference on machine learning . PMLR, 2019, pp. 3519–3529

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.