Pith. sign in

REVIEW 4 major objections 5 minor 69 references

Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read If X-Agent is right, fine-tuning CLIP for open-vocabulary segmentation need not erase knowledge of unseen classes: a small set of 'agent' tokens selected by optimal-transport affinity can amplify latent semantics and keep them discriminable

desk verdict Competent incremental OVSS paper with a plausible but under-supported latent-semantic story; deserves peer review with one crucial ablation demanded. read the letter →

arxiv 2509.01275 v2 pith:UAGX77RG submitted 2025-09-01 cs.CV

Jiahao Li , Yang Lu , Yachao Zhang , Fangyong Wang , Yuan Xie , Yanyun Qu This is my paper
classification cs.CV
keywords open-vocabularysemanticsegmentationlatentunseencategoryoptimaltransportcross-modalattentionagenttokensparameter-efficientfine-tuningCLIPzero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-vocabulary segmentation asks a model trained on known classes to label images of classes it has never seen. X-Agent begins from a probing observation: a pre-trained CLIP distinguishes unseen classes well initially, but standard supervised fine-tuning progressively erases that ability, until unseen-class activations collapse to random. The paper's claim is that this erosion can be prevented by interposing a small set of 'agent' tokens inside the visual encoder's cross-modal attention. The agents are chosen from the visual tokens whose optimal-transport-refined affinity to text is lowest, pooled with visual and textual context, then used as query and key in a cascaded differential attention that amplifies latent semantics. X-Agent reports state-of-the-art mIoU on six open-vocabulary benchmarks while fine-tuning only the query and value projections, so the method is also parameter-efficient.

What carries the argument

The load-bearing object is the 'agent token'—a small set of visual tokens selected as carriers of latent unseen semantics. Selection uses a semantic affinity matrix built by Sinkhorn optimal transport between CLIP text tokens and the visual attention key matrix; the k category rows with highest mean affinity are kept, and within them the q tokens with the lowest affinity (Eq. 9) become agents. Agent pooling fuses visual and textual global context, and agent attention runs two cascaded differential-attention blocks—attention computed as the difference of two softmax terms, which suppresses noise—with the agent acting first as query over text, then as key over visual tokens. This quadruplet (Q

What would settle it

Train X-Agent identically but select agent tokens with the highest affinity values instead of the lowest (swap largest=False in Eq. 9). If mIoU on A-150 and PC-459 stays the same or improves, the claim that the lowest-affinity tokens carry latent unseen semantics is falsified; a drop would support the claim.

Watch

Extended reading notes

Core claim

The central discovery is empirical and architectural. Probing CLIP under inductive learning, the authors find that latent unseen semantics live in category channels with the highest semantic-affinity scores, and that during training these activations first emerge and then fade, collapsing toward a random baseline. To counter this, X-Agent selects agent tokens from the visual value matrix using a top-k, then top-q, strategy on an optimal-transport-refined affinity matrix between text tokens and the visual attention key matrix. The agents are pooled over visual and textual context and then drive two cascaded differential-attention blocks: the agent first reads textual tokens as query, then gui

Load-bearing premise

The claim stands on the assumption that text-to-visual affinity scores refined by optimal transport mark the exact visual tokens carrying unseen-category meaning, and that picking the lowest-affinity tokens amplifies those categories rather than noise.

Editorial extensions

If this is right

  • If X-Agent is right, open-vocabulary segmentation can be improved without new data or extra backbones: only CLIP's query/value projections and a lightweight attention module are tuned.
  • The probing result gives a concrete failure diagnosis for VLM fine-tuning: unseen-category discriminability is present early and decays, so methods aiming for generalization should measure and counter that decay.
  • The reported numbers would make X-Agent the strongest CLIP-only adapter on these benchmarks, including top results on PC-459, A-150, PC-59, and PAS-21 with CLIP-B/16 and on all six benchmarks with CLIP-L/14.
  • Agent-mediated attention can be viewed as a parameter-efficient fine-tuning recipe that steers the optimization trajectory of a large model rather than overwriting it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The affinity-selection mechanism is not tied to segmentation; the same agent-selection and agent-attention pattern could be tried in other VLM fine-tuning tasks, such as open-vocabulary detection or retrieval, wherever unseen-class discriminability decays.
  • A direct testable extension is to check whether the chosen agent tokens actually align with ground-truth unseen-class regions pixel-wise; if they do, the same selection could provide pseudo-labels for unseen-class discovery.
  • The probing protocol—linear probing on CLIP with a seen/unseen split—could be reused as a general diagnostic for any VLM fine-tuning method, predicting which adapters will generalize to novel classes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses open-vocabulary semantic segmentation (OVSS) by proposing X-Agent, a method that inserts learned 'agent' tokens into CLIP's attention to preserve latent unseen-class semantics during inductive fine-tuning. The authors first run a probing experiment on CLIP, reporting that latent semantics are concentrated in high-affinity category channels and that their discriminability degrades during training. Building on this, the method selects agent tokens by computing an optimal-transport affinity between text tokens and visual key features, taking the top-k category channels and the lowest-affinity q tokens per channel from the value matrix; it then pools these tokens with visual and textual context and uses cascaded differential attention to modulate visual tokens. A contrastive alignment loss regularizes the text projector. Experiments on ADE20K, Pascal Context, and Pascal VOC report state-of-the-art or near-SOTA mIoU with CLIP-B/16 and CLIP-L/14, plus improved zero-shot seen/unseen hIoU over RPN. The paper also includes ablations of components, selection strategies, cost matrices, and attention-matrix choices.

Significance. If the central mechanism is confirmed, the paper makes a useful contribution: a parameter-efficient fine-tuning module that operates on intermediate CLIP attention and an empirical characterization of latent-semantic dynamics under inductive OVSS training. The code is released, and the ablation structure is thorough in some dimensions (component addition, selection prior, cost matrix, attention matrices). However, the headline SOTA claim currently rests on margins of 0.1–0.6 mIoU from single runs, and the key selection rule—choosing the lowest-affinity tokens—is not validated against the stated 'latent-related' interpretation. These issues weaken the empirical support for the novelty, although the framework is intuitively plausible and the reported gains are consistent across datasets.

major comments (4)
  1. [§3.3, Eq. (9)] The paper's central claim is that agent tokens localize latent unseen semantics. This claim is not established. Eq. (9) selects q tokens with the lowest affinity in each of the top-k channels (largest=False), while the text calls them 'latent-related top-q tokens'. The probing experiment in Fig. 1(a) only shows that high-scoring category channels carry latent semantics; it says nothing about whether low-affinity tokens within those channels are the novel-category tokens. Table 4 ablates random/learnable/cosine/OT selection but never reverses the direction to largest=True or samples randomly within the same top-k channels. Without such controls, the observed gains could come from any low-affinity token modulation and do not support the latent-semantic localization mechanism.
  2. [Tables 1-2 and §4.2] The state-of-the-art comparisons are based on single training runs. The reported improvements over the second-best method are 0.1–0.6 mIoU (e.g., CLIP-B/16 A-150: 32.1 vs 31.8; CLIP-L/14 PC-59: 63.7 vs 63.3), which are within typical training noise for segmentation models. No error bars, multiple seeds, or statistical significance tests are given. The zero-shot comparison in Table 2 reports a 0.8 hIoU gain. Given the margins, 'state-of-the-art' and even the relative ranking of the method are not statistically supported.
  3. [§1 and Fig. 1] The probing experiment is the motivation for the whole method, but it is only qualitatively described. There is no specification of which CLIP layer was probed, how the 5,000 images were chosen beyond 'predominant unseen category representation', or how the semantic affinity scores and activation magnitudes were computed. No error bars or quantitative summaries are provided. More importantly, it does not provide evidence for the token-level selection rule in Eq. (9), which is the load-bearing component of the method.
  4. [§4.2 and §4.3] The manuscript repeatedly refers to an appendix for extended zero-shot results and agent token visualization, but no appendix is present in the submitted version. This is not merely a formatting issue: the missing appendix is used as evidence for the method's interpretability and generalization. Additionally, the hyperparameters k, q, gamma_init, and the Sinkhorn epsilon are fixed without sensitivity analysis; since k and q jointly determine the agent token set (40 tokens), their influence on the selection mechanism should be reported.
minor comments (5)
  1. [Eq. (5)] The cost matrix definition uses '·' between the normalized text and key matrices; please clarify whether this is an outer product and specify the dimensions. Also, the notation δ𝐾⊤ in Eq. (3) is confusing—should be a Dirac at the j-th key vector.
  2. [Eqs. (7)-(10)] The symbol k is used both for the number of category channels and as the argument of top-k. In Eq. (9), top-k(𝐀*, q, largest=False) should be top-q to avoid confusion.
  3. [Table 1] The RPN baseline rows are missing PAS-21 entries for both CLIP-B/16 and CLIP-L/14; please add them or indicate why they are omitted so that all comparisons are complete.
  4. [§3.5] In Eq. (17) the differential attention uses two softmax terms with the same Q,K,V projected separately, but it is not clear whether the projections in Eq. (18) are shared across the two softmax paths or are per-path; please clarify.
  5. [General] There are template artifacts in the header ('Conference acronym ’XX, June 03–05, 2018, Woodstock, NY') and minor typos in the abstract ('latent unseen category' vs 'latent unseen categories'). Please clean these up.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: X-Agent's selection mechanism and losses do not reduce to their outputs; the only overlapping-author citation (RPN) is used as a baseline, not as load-bearing evidence.

full rationale

The paper's derivation chain is not circular. The probing experiment (Fig. 1) is an external empirical observation about CLIP affinity distributions; agent selection (Eqs. 7-10) operationalizes that observation through optimal-transport affinity and top-k token selection, but the selected tokens are not defined in terms of the final mIoU, and the benchmarks are independent of the selection rule. The textual alignment loss (Eqs. 22-24) is a consistency regularizer that anchors the learned text projector to the frozen text encoder; it is a constraint, not a prediction derived from the evaluation target. The agent pooling and agent attention modules (Eqs. 11-21) are architectural interventions borrowed and adapted from external prior work ([12, 53]), not from the authors' own prior claims. The only self-citation with overlapping authors is RPN [24], which appears as a comparison baseline in Tables 1-2 and in the related-work discussion; X-Agent's central mechanism does not depend on any RPN-specific result. The skeptic's concern about the largest=False direction in Eq. 9 is a potential validity/ablation gap—the paper does not ablate largest=True—but it is not circularity, because the selection rule is not fitted to the reported metric and no equation equates the predicted performance to the selection criterion. Likewise, the missing appendix and lack of error bars are reproducibility concerns, not circularity. Therefore, no circular step can be exhibited, and the paper warrants a low score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The method rests on hand-chosen hyperparameters (k, q, gamma_init, epsilon) and on domain assumptions about CLIP's feature geometry and the degradation of unseen-class discriminability. The 'agent token' is a new conceptual entity without independent evidence. The central claim does not rely on circular reasoning, but the selection of lowest-affinity tokens is an untested assumption.

free parameters (4)
  • k (number of category channels) = 10
    Chosen by hand; no sensitivity analysis reported. Used in Eq. 7-8.
  • q (tokens per channel) = 4
    Chosen by hand; no sensitivity analysis reported. Used in Eq. 9.
  • gamma_init (initial scalar in Eq. 14) = 0.1
    Chosen by hand; the re-parameterization with learnable gamma_v and gamma_t is claimed to help, but the initial value is ad hoc.
  • Sinkhorn regularization epsilon = not reported
    Equation 4 uses epsilon > 0 but the value is not given; it affects the affinity matrix and is a free choice.
assumptions (4)
  • domain assumption CLIP's pre-trained visual features encode discriminative information about unseen categories that can be localized in a subset of tokens.
    Stated in the probing experiment (Section 1 and Figure 1) and used to justify agent selection.
  • domain assumption Inductive fine-tuning on seen classes monotonically degrades unseen-class discriminability, and this degradation can be counteracted by the proposed attention modulation.
    The paper's central hypothesis; supported only by a qualitative probing experiment on 5,000 images.
  • domain assumption The optimal transport plan between text embeddings and visual key features produces a semantically meaningful affinity matrix (Eq. 3-6).
    No theoretical guarantee; the ablation in Table 4 shows OT alone is worse than cosine, and only the combination helps.
  • domain assumption The reported benchmark scores are representative and reproducible; standard datasets are used without additional training data.
    Assumes the standard evaluation protocol is followed and the code link is functional.
invented entities (1)
  • Agent tokens
    purpose: Selected visual tokens that act as mediators between text and visual features to amplify latent semantic saliency.
    The 'agent' is a new architectural construct; its existence is only evidenced by benchmark performance within this paper, with no external falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentation." pith.science (2026). https://pith.science/paper/UAGX77RG

@misc{pith2026250901275,
  author       = {Pith},
  title        = {Pith review of: Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UAGX77RG}},
  note         = {Machine review of arXiv:2509.01275}
}
read the original abstract

Open-vocabulary semantic segmentation (OVSS) conducts pixel-level classification via text-driven alignment, where the domain discrepancy between base category training and open-vocabulary inference poses challenges in discriminative modeling of latent unseen category. To address this challenge, existing vision-language model (VLM)-based approaches demonstrate commendable performance through pre-trained multi-modal representations. However, the fundamental mechanisms of latent semantic comprehension remain underexplored, making the bottleneck for OVSS. In this work, we initiate a probing experiment to explore distribution patterns and dynamics of latent semantics in VLMs under inductive learning paradigms. Building on these insights, we propose X-Agent, an innovative OVSS framework employing latent semantic-aware ``agent'' to orchestrate cross-modal attention mechanisms, simultaneously optimizing latent semantic dynamic and amplifying its perceptibility. Extensive benchmark evaluations demonstrate that X-Agent achieves state-of-the-art performance while effectively enhancing the latent semantic saliency.

Figures

Figures reproduced from arXiv: 2509.01275 by the authors.

Figure 1
Figure 1. Our probing experiment results reveal that (a) la [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of X-Agent. Our X-Agent framework operates directly on the cross-layer visual embeddings of VLM, e.g., [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. (a) Agent Pooling. The agent tokens are fed into [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Qualitative comparison to RPN [24]. Visualization results for the A-150 and PC-459 benchmarks are positioned to the left- and right-hand sides of the dashed line, respectively [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparative analysis of attention heatmaps. Our [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Mean Attention Distance (MAD) analysis of each [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 59 canonical work pages

  1. [1]

    Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. 2019. Zero- shot semantic segmentation. Advances in Neural Information Processing Systems 32 (2019)

    work page 2019

  2. [2]

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. 2018. COCO-Stuff: Thing and stuff classes in context. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE conference on . IEEE

    work page 2018

  3. [3]

    Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. 2020. Learning open set network with dis- criminative reciprocal points. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16 . Springer, 507–522

    work page 2020

  4. [4]

    Jiaxin Cheng, Soumyaroop Nandi, Prem Natarajan, and Wael Abd-Almageed

    work page

  5. [5]

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. 2023. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint arXiv:2303.11797 (2023)

    arXiv 2023

  6. [6]

    Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems 26 (2013)

    work page 2013

  7. [7]

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. 2022. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11583–11592

    work page 2022

  8. [8]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. [n. d.]. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- network.org/challenges/VOC/voc2012/workshop/index.html

    work page 2012

Show all 69 references
  1. [9]

    Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. 2021. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 9284–9292

    2021

  2. [10]

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. 2022. Scaling open- vocabulary image segmentation with image-level labels. In European Conference on Computer Vision. Springer, 540–557

    2022

  3. [11]

    Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. 2020. Context-aware feature generation for zero-shot semantic segmentation. In Pro- ceedings of the 28th ACM International Conference on Multimedia . 1921–1929

    2020

  4. [12]

    Dongchen Han, Tianzhu Ye, Yizeng Han, Zhuofan Xia, Siyuan Pan, Pengfei Wan, Shiji Song, and Gao Huang. 2024. Agent attention: On the integration of softmax and linear attention. In European Conference on Computer Vision . Springer, 124– 140

    2024

  5. [13]

    Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and An- drew Zisserman. 2020. Automatically Discovering and Learning New Visual Categories with Ranking Statistics

    2020

  6. [14]

    Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and An- drew Zisserman. 2021. AutoNovel: Automatically Discovering and Learning Novel Visual Categories. (2021)

    2021

  7. [15]

    Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2019. Learning to Discover Novel Visual Categories via Deep Transfer Clustering

    2019

  8. [16]

    Shuting He, Henghui Ding, and Wei Jiang. 2023. Primitive generation and semantic-related alignment for universal zero-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11238– 11247

    2023

  9. [17]

    Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. 2018. Learning to cluster in order to transfer across domains and tasks

    2018

  10. [18]

    Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. 2019. Multi-class classification without multi-class labels

    2019

  11. [19]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision- language representation learning with noisy text supervision. In International conference on machine learning . PMLR, 4904–4916

    2021

  12. [20]

    Xuhui Jia, Kai Han, Yukun Zhu, and Bradley Green. 2021. Joint representation learning and novel category discovery on single-and multi-modal data. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision . 610–619

    2021

  13. [21]

    Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. 2023. Learning mask-aware clip representations for zero-shot segmentation. Advances in Neural Information Processing Systems 36 (2023), 35631–35653

    2023

  14. [22]

    Hyeongjun Kwon, Taeyong Song, Somi Jeong, Jin Kim, Jinhyun Jang, and Kwanghoon Sohn. 2023. Probabilistic Prompt Learning for Dense Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. 6768–6777

    2023

  15. [23]

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl

  16. [24]

    Jiahao Li, Yang Lu, Yuan Xie, and Yanyun Qu. 2024. Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation. Advances in Neural Information Processing Systems 37 (2024), 74298–74324

    2024

  17. [25]

    Peike Li, Yunchao Wei, and Yi Yang. 2020. Consistent structural relation learning for zero-shot segmentation. Advances in Neural Information Processing Systems 33 (2020), 10317–10327

    2020

  18. [26]

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2022. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint arXiv:2210.04150 (2022)

    2022 arXiv

  19. [27]

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2023

  20. [28]

    Bo Liu, Hao Kang, Haoxiang Li, Gang Hua, and Nuno Vasconcelos. 2020. Few- shot open-set recognition using meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 8798–8807

    2020

  21. [29]

    Juncheng Liu, Zhouhui Lian, Yi Wang, and Jianguo Xiao. 2017. Incremental kernel null space discriminant analysis for novelty detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 792–800

    2017

  22. [30]

    Xinyu Liu, Beiwen Tian, Zhen Wang, Rui Wang, Kehua Sheng, Bo Zhang, Hao Zhao, and Guyue Zhou. 2023. Delving into Shape-aware Zero-shot Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2999–3009

    2023

  23. [31]

    Marc Masana, Idoia Ruiz, Joan Serrat, Joost van de Weijer, and Antonio M Lopez. 2018. Metric learning for novelty and anomaly detection. arXiv preprint arXiv:1808.05492 (2018)

    2018 arXiv

  24. [32]

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. 2014. The Role of Context for Object Detection and Semantic Segmentation in the Wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2014

  25. [33]

    Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo. 2021. A closer look at self-training for zero-label semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2693–2702

    2021

  26. [34]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing system...

    2019

  27. [35]

    Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al . 2023. FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . ...

    2023

  28. [36]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on Machine Learni...

    2021

  29. [37]

    Scheirer, Anderson Rocha, Archana Sapkota, and Terrance E

    Walter J. Scheirer, Anderson Rocha, Archana Sapkota, and Terrance E. Boult

  30. [38]

    Rui Shao, Pramuditha Perera, Pong C Yuen, and Vishal M Patel. 2020. Open- set adversarial defense. In European Conference on Computer Vision . Springer, 682–698

    2020

  31. [39]

    Feihong Shen, Jun Liu, and Ping Hu. 2021. Conterfactual generative zero-shot semantic segmentation. arXiv preprint arXiv:2106.06360 (2021)

    2021 arXiv

  32. [40]

    Hengcan Shi, Son Duy Dao, and Jianfei Cai. 2025. LLMFormer: Large language model for open-vocabulary semantic segmentation. International Journal of Computer Vision 133, 2 (2025), 742–759

    2025

  33. [41]

    Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, and Yonghong Tian. 2020. P-odn: Prototype-based open deep network for open set recognition. Scientific reports 10, 1 (2020), 7146

    2020

  34. [42]

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Woj- ciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 15638...

    2022

  35. [43]

    Xin Sun, Zhenning Yang, Chi Zhang, Keck-Voon Ling, and Guohao Peng. 2020. Conditional gaussian distribution learning for open set recognition. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition . 13480– 13489

    2020

  36. [44]

    Jinglong Wang, Xiawei Li, Jing Zhang, Qingyuan Xu, Qin Zhou, Qian Yu, Lu Sheng, and Dong Xu. 2025. Diffusion model is secretly a training-free open vocabulary semantic segmenter. IEEE Transactions on Image Processing (2025)

    2025

  37. [45]

    Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen

  38. [46]

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick

  39. [47]

    Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata

  40. [48]

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. 2023. SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation. arXiv preprint arXiv:2311.15537 (2023)

    2023 arXiv

  41. [49]

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. 2023. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2955–2966

    2023

  42. [50]

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. 2023. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2945–2954

    2023

  43. [51]

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. 2022. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision. Springer, 736–753

    2022

  44. [52]

    Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. 2024. Generalized out- of-distribution detection: A survey. International Journal of Computer Vision 132, 12 (2024), 5635–5662

    2024

  45. [53]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Semantic projection network for zero-and few-label semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8256–8265

  46. [54]

    Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. 2023. Zero-shot Referring Image Segmentation with Global-Local Context Features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 19456–19465

    2023

  47. [55]

    He Zhang and Vishal M Patel. 2016. Sparse representation-based open set recog- nition. IEEE transactions on pattern analysis and machine intelligence 39, 8 (2016), 1690–1696

    2016

  48. [56]

    Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. 2017. Open vocabulary scene parsing. InProceedings of the IEEE International Conference on Computer Vision. 2002–2010

    2017

  49. [57]

    Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe

  50. [58]

    Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. 2021. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9462–9470

    2021

  51. [59]

    Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei

  52. [60]

    Chong Zhou, Chen Change Loy, and Bo Dai. 2022. Extract free dense labels from clip. In European Conference on Computer Vision . Springer, 696–712

    2022

  53. [61]

    Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. 2023. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 11175–11185

    2023

  54. [65]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Neighborhood contrastive learning for novel class discovery. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10867– 10875

  55. [67]

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2019. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127 (2019), 302–321

    2019

  56. [2013]

    Towards Open Set Recognition. (2013)

    2013

  57. [2019]

    https://github.com/facebookresearch/detectron2

    Detectron2. https://github.com/facebookresearch/detectron2

  58. [2021]

    In Proceedings of the IEEE/CVF International Conference on Computer Vision

    Sign: Spatial-information incorporated generative network for generalized zero-shot semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 9556–9566

  59. [2022]

    arXiv preprint arXiv:2201.03546 (2022)

    Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546 (2022)

    2022 arXiv

  60. [2023]

    arXiv preprint arXiv:2303.11681 (2023)

    Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. arXiv preprint arXiv:2303.11681 (2023)

    2023 arXiv

  61. [2024]

    arXiv preprint arXiv:2410.05258 (2024)

    Differential transformer. arXiv preprint arXiv:2410.05258 (2024)

    2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.