Pith. sign in

REVIEW 6 major objections 4 minor 66 references

Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation

T0 review · 6 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-vocabulary segmentation benchmarks are semantically too close to the training space, so standard scores overstate generalization; the new OpenBench reverses the ranking and shows fine-tuning CLIP hurts on novel categories.

desk verdict OpenBench flips the usual OVS leaderboard and that is worth taking seriously, but the domain shift in the benchmark makes the headline claim about fine-tuning CLIP unproven; the method half is solid, incremental work. read the letter →

arxiv 2506.16058 v2 pith:JTOTL54W submitted 2025-06-19 cs.CV

Yong Liu , SongLi Wu , Sule Bai , Jiahao Wang , Yitong Wang , Yansong Tang This is my paper
classification cs.CV
keywords open-vocabularysegmentationbenchmarkevaluationCLIPsemanticsimilaritygeneralizationProxyCalibrationGradient-FreeAggregationOpenBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current open-vocabulary segmentation benchmarks rank models within a semantic space that is almost the same as the training space, so high scores on them do not show that a model can handle genuinely novel concepts. It introduces OpenBench, a test set whose category names sit much farther from the training categories of COCO, so that every model must rely on retained vision-language generalization rather than memorized alignments. When the paper evaluates existing methods on OpenBench, the ranking reverses: fine-tuning CLIP, which reliably helps on standard benchmarks, hurts on OpenBench. The paper also proposes OVSNet, which preserves frozen-CLIP generalization through a gradient-free feature fusion and a proxy calibration training trick, and reports that this model improves performance both on standard sets and on OpenBench.

What carries the argument

OpenBench is the central measuring object: candidate images from Food103, ImageNet-S, and CamVid are scored by the minimum CLIP text-similarity of their annotated categories to any COCO category, images above a threshold are dropped, and remaining near-training categories are relabeled 'others'. The method side has two named mechanisms. Proxy Calibration (PC) forms random convex combinations of class embeddings, sampled from a Beta distribution with gamma equal to 2, and applies distance supervision to those synthetic embeddings, widening the training space without new labels. Gradient-Free Aggregation (GFA) treats the decoder's query embeddings and mask-pooled CLIP features as two feature sets and alternately updates them through an affinity matrix, replacing the infinite iteration with a closed-form Neumann-series inverse so that no learned fusion layer can overfit to seen semantics.

What would settle it

An experiment could use the same OpenBench images with an alternative label set whose class names are equally fine-grained but similar to COCO; if fine-tuned CLIP drops on that version too, the divergence is not about semantic distance, and if it drops only on the original labels, the benchmark's premise is supported.

Watch

Extended reading notes

Core claim

The paper claims that the field's standard evaluation of open-vocabulary segmentation is misleading because the test sets used everywhere live inside the training set's semantic neighborhood. On VOC, Pascal Context-59, and ADE20K-150, the average maximum CLIP similarity between test categories and COCO training categories is 0.97, 0.95, and 0.81 respectively; even ADE-847 and PC-459 score 0.79 and 0.84. OpenBench is built from Food103, ImageNet-S, and CamVid by filtering to images whose categories are far from COCO, giving a mean similarity of 0.61, and it includes an 'others' class so models cannot always guess from the candidate list. On this benchmark, methods that fine-tune CLIP, which gain steadily on standard sets, lose several mIoU points, while frozen-CLIP methods maintain their performance, so the standard ranking reverses. The proposed OVSNet, using proxy-calibrated training and gradient-free aggregation of CLIP and decoder features, achieves the highest average mIoU on both the standard sets and OpenBench.

Load-bearing premise

OpenBench's conclusions stand or fall on the assumption that CLIP similarity to COCO isolates semantic novelty; if the food, object-centric, and driving image domains themselves explain the performance drops, existing benchmarks may not overstate generalization.

Editorial extensions

If this is right

  • If OpenBench is a fair test, rankings on standard OVS benchmarks should not be read as generalization rankings; a model that wins on VOC, Pascal Context, and ADE20K can still be among the worst on semantically novel categories.
  • Fine-tuning CLIP on the training distribution is a liability for truly open use: the same recipe that adds mIoU points on near-training benchmarks removes points on OpenBench, so preserving the pretrained vision-language alignment is the safer design choice.
  • The two proposed mechanisms are complementary: Proxy Calibration contributes +0.8/+2.9/+2.0 mIoU and Gradient-Free Aggregation +1.6/+1.7/+1.4 on ADE-150/PC-459/OpenBench, and together they add +2.7/+4.8/+2.6 over the baseline.
  • The number of candidate categories during inference changes scores substantially, so benchmark comparisons should control candidate-set size before attributing differences to semantic novelty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: OpenBench's three source datasets bring their own visual domains, so the reported divergence could partially reflect food close-ups, object-centric images, and driving scenes rather than semantic distance alone; a single-domain control set would separate these factors.
  • Editorial inference: the filtering thresholds sigma-1 and sigma-2 are named but not reported, so an independent regeneration of OpenBench is not currently possible; publishing them would let the community audit the benchmark's construction.
  • Editorial inference: a more continuous evaluation would bin OpenBench categories by their COCO similarity and plot mIoU per bin, turning the benchmark into a stress curve for how gracefully a model degrades as semantic distance grows.
  • Editorial inference: the same similarity-filtering recipe could be used to build open-distribution test splits for other vision-language tasks such as retrieval or detection, not just segmentation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. Open-vocabulary segmentation models are typically trained on COCO and evaluated on VOC, Pascal Context, ADE20K, and Cityscapes. The paper argues that these test vocabularies lie close to the COCO semantic space in CLIP text-embedding space, so high scores on them do not demonstrate ability to segment genuinely novel categories. To remedy this, the authors construct OpenBench from Food103, ImageNet-S, and CamVid by keeping images and categories with low CLIP similarity to COCO, yielding 286 classes with an average maximum similarity of 0.61. They report that a fine-tuned-CLIP method (CATSeg) gains on existing benchmarks but falls below a frozen-CLIP method (SAN) on OpenBench, suggesting that fine-tuning harms open-domain generalization. They also propose OVSNet, which uses a gradient-free aggregation of CLIP and decoder features and a Proxy Calibration training strategy, and report state-of-the-art average mIoU on both existing sets and OpenBench.

Significance. The paper makes a useful and falsifiable observation: existing OVS benchmarks are semantically close to the training vocabulary, and Table 1 quantifies this in a transparent way. If OpenBench is validated and released, it can serve as a complementary stress test for OVS. The authors deserve credit for explicitly connecting this observation to the fine-tuning-versus-frozen CLIP behavior, and the method ablations in Table 3 show consistent gains from both proposed components. However, the benchmark's validity currently rests on unreported thresholds and an uncontrolled domain mix, and the fine-tuning conclusion is drawn from a cross-method comparison rather than a controlled ablation; these issues need to be addressed before the central claims can be fully accepted.

major comments (6)
  1. [Section 3.2] The thresholds σ1 and σ2 are never reported, the category list and manual filtering procedure are not described, and no URL or release plan for OpenBench is provided. Since the benchmark is defined by applying these thresholds, the reader cannot audit whether OpenBench truly isolates semantic novelty from other factors. The authors should report the threshold values, provide a sensitivity analysis, publish the complete category list, and release the construction code and annotations.
  2. [Section 3.2] The image-level filtering rule does not guarantee category-level novelty. An image passes the filter if it contains at least one category far from COCO, and all other categories with similarity above σ2 are relabeled 'others'. Because 'others' is not a real semantic concept and does not appear in any evaluated model's training vocabulary, a model that correctly recognizes a familiar COCO category in a novel image is penalized, while a model that merely guesses 'others' can be rewarded. This confounds open-vocabulary comprehension with the ability to abstain, so the mIoU gap in Figure 1(b) cannot be attributed solely to semantic novelty.
  3. [Section 3.2] The three source datasets (Food103, ImageNet-S, CamVid) introduce systematic visual domain shifts—close-up food photography, object-centric ImageNet crops, and driving scenes—that are entangled with semantic distance. A model fine-tuned on COCO may fail on OpenBench because of unfamiliar object scale, background, and viewpoint rather than because the categories are semantically new. The authors should provide a per-source performance breakdown, evaluate familiar COCO categories in shifted image domains, and ideally construct an image-domain-matched control set that varies only semantic distance.
  4. [Figure 1(b), Table 2] The central claim that fine-tuning CLIP hurts on OpenBench is inferred from a comparison of CATSeg (fine-tuned CLIP) with SAN (frozen CLIP). These methods differ in decoder architecture, feature aggregation, training loss, and other implementation choices, so the observed −3.5 gap on OpenBench is not a controlled test of fine-tuning. A within-architecture ablation that toggles only CLIP fine-tuning, reporting both existing and OpenBench results, is needed to support the paper's stated conclusion.
  5. [Section 3.2, Table 1] The similarity statistics are computed against 'COCO categories', but the evaluated models are trained on different vocabularies (COCO-Stuff-171 vs COCO-Panoptic-133). The exact vocabulary and text templates used for the CLIP-L/14 similarity computation should be specified, and statistics should be reported for both candidate training vocabularies; otherwise the claim that OpenBench is novel relative to 'the training space' is ambiguous.
  6. [Section 4.2, Section 5.2] The fusion weight ω in Eqs. (1)–(4) is a core hyperparameter of the Gradient-Free Aggregation, but its default value is not reported, and no conditions are given for the convergence of the Neumann series in Eq. (4). The authors should report ω, specify the exact normalization used in Norm(Z), and verify or justify that the spectral radius of ω²A is less than one.
minor comments (4)
  1. [Table 4(b)] The caption states that γ=1 is equivalent to not using PC, but Beta(1,1) is a uniform distribution and would still generate random convex combinations in Eq. (5) unless explicitly disabled; please clarify the implementation or correct the claim.
  2. [Table 2] The 'Average Score' column is a simple unweighted mean over six datasets with very different class counts and difficulty levels; please report a defined aggregation or present per-dataset numbers so that the average is interpretable.
  3. [Abstract and Section 5.3] The abstract states 'state-of-the-art results on both existing datasets and our proposed OpenBench', but Table 2 shows that OVSNet is below SCAN on VOC; the claim should be qualified to match the reported results.
  4. [Section 5.4] The claim in Figure 5 that model performance degrades consistently as the number of inference categories increases is based on two methods without error bars; please add standard deviations across runs or clarify that the trend is qualitative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the OpenBench divergence is an empirical measurement, not a construction artifact.

full rationale

The paper's central claims are that existing OVS test sets are semantically close to the COCO training space, that the new OpenBench, selected to be far from that space, yields different method rankings, and that OVSNet's proxy calibration and gradient-free aggregation improve open-world performance. None of these is derived by construction from the paper's own equations or from a self-citation chain. OpenBench is constructed by thresholding CLIP-L/14 text similarities, but the reported performance gaps in Table 2 and Figure 1(b) are measured from actual model outputs; the benchmark selection does not algebraically force those gaps. The method's fusion formula (Eq. 4) and proxy loss (Eqs. 5-7) are trainable augmentations whose gains are established by ablations, not by definition. The citations to the authors' prior SCAN/GKC work are contextual and not load-bearing: the semantic-duplication observation is independently restated, and no uniqueness theorem is imported. The self-consistency that OpenBench uses CLIP and OVSNet preserves CLIP is a design alignment, not a logical reduction; a finetuned method could in principle excel on OpenBench, and the paper reports that existing finetuned methods do not. Therefore no circular step meets the evidence bar.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The benchmark construction depends on unreported thresholds (σ1, σ2) and a CLIP-similarity-based definition of openness. The method adds two hyperparameters (λ, γ) plus an unspecified ω. Key assumptions include the validity of CLIP similarity as a proxy for semantic distance, the exclusion of source datasets from training data, and the convergence of the Neumann series approximation.

free parameters (5)
  • σ1
    Threshold for filtering images whose minimum category similarity to COCO exceeds σ1; value not reported in the paper, so the benchmark cannot be reconstructed exactly.
  • σ2
    Threshold above which a category in a retained image is relabeled to 'others'; value not reported.
  • ω
    Fusion factor in GFA equations (1)-(4); defined as ω∈(0,1) but the default value is not given in Section 5.2, which only lists λ=0.2 and γ=2.
  • λ = 0.2
    Scaling factor in the affinity Z of GFA; set by hand in Section 5.2.
  • γ = 2
    Shape parameter of Beta distribution in Proxy Calibration; chosen by hand in Section 5.2, with ablation showing γ=2 best.
assumptions (4)
  • domain assumption CLIP text embedding cosine similarity between a test category name and the closest COCO class name measures how 'open' that category is relative to training.
    Used throughout Section 3 to define OpenBench and to argue existing benchmarks are too similar to training. If this measure does not reflect the actual semantic gap experienced by a segmentation model, the benchmark's validity is undermined.
  • domain assumption Food103, ImageNet-S, and CamVid images are not part of the training data of the compared OVS models, so performance on OpenBench is a fair out-of-distribution test.
    Assumed in Section 3.2; if any model was trained on these sources or their categories, the comparison would be biased.
  • standard math The Neumann series approximation for the GFA fixed point converges, i.e. the spectral radius of ω^2 A is less than 1.
    Equation (4) uses the closed-form (I - ω^2 A)^{-1}; the paper does not verify the convergence condition for the affinity matrix A defined after Equation (3).
  • domain assumption The random walk fixed point of GFA produces a meaningful joint representation for mask classification.
    The fusion method in Section 4.2 rests on the assumption that mixing query embeddings and pooled CLIP features by diffusion preserves discriminative information; no theoretical guarantee is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation." pith.science (2026). https://pith.science/paper/JTOTL54W

@misc{pith2026250616058,
  author       = {Pith},
  title        = {Pith review of: Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTOTL54W}},
  note         = {Machine review of arXiv:2506.16058}
}
read the original abstract

Open-vocabulary segmentation aims to achieve segmentation of arbitrary categories given unlimited text inputs as guidance. To achieve this, recent works have focused on developing various technical routes to exploit the potential of large-scale pre-trained vision-language models and have made significant progress on existing benchmarks. However, we find that existing test sets are limited in measuring the models' comprehension of ``open-vocabulary" concepts, as their semantic space closely resembles the training space, even with many overlapping categories. To this end, we present a new benchmark named OpenBench that differs significantly from the training semantics. It is designed to better assess the model's ability to understand and segment a wide range of real-world concepts. When testing existing methods on OpenBench, we find that their performance diverges from the conclusions drawn on existing test sets. In addition, we propose a method named OVSNet to improve the segmentation performance for diverse and open scenarios. Through elaborate fusion of heterogeneous features and cost-free expansion of the training space, OVSNet achieves state-of-the-art results on both existing datasets and our proposed OpenBench. Corresponding analysis demonstrate the soundness and effectiveness of our proposed benchmark and method.

Figures

Figures reproduced from arXiv: 2506.16058 by the authors.

Figure 1
Figure 1. (a) shows the similarity distribution of the existing test [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the samples in the proposed OpenBench. Compared to existing test sets, our OpenBench has semantic categories [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pipeline of our OVSNet. The input image and corresponding categories are first encoded by pretrained CLIP [ [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visualization of category embeddings across the training [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Impact of the number of inference categories on model [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of segmentation results. (a) - (d) demonstrate the successful segmentation cases of our method for seen and unseen [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 48 canonical work pages

  1. [1]

    Self-calibrated clip for training-free open-vocabulary segmentation

    Sule Bai, Yong Liu, Yifei Han, Haoji Zhang, and Yansong Tang. Self-calibrated clip for training-free open-vocabulary segmentation. arXiv preprint arXiv:2411.15869, 2024. 2

    arXiv 2024

  2. [2]

    Univg-r1: Reasoning guided universal visual grounding with reinforce- ment learning

    Sule Bai, Mingxing Li, Yong Liu, Jing Tang, Haoji Zhang, Lei Sun, Xiangxiang Chu, and Yansong Tang. Univg-r1: Reasoning guided universal visual grounding with reinforce- ment learning. arXiv preprint arXiv:2505.14231, 2025. 1

    arXiv 2025

  3. [3]

    Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla

    Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using struc- ture from motion point clouds. InECCV, pages 44–57, 2008. 4

    work page 2008

  4. [4]

    Zero-shot semantic segmentation

    Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick P´erez. Zero-shot semantic segmentation. In NeurIPS, 2019. 2

    work page 2019

  5. [5]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In ECCV, 2020. 6

    work page 2020

  6. [6]

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic im- age segmentation with deep convolutional nets, atrous con- volution, and fully connected crfs. TPAMI, 2018. 1

    work page 2018

  7. [7]

    UNITER: learning universal image-text representations

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: learning universal image-text representations. arXiv preprint arXiv:1909.11740, 2019. 3

    arXiv 1909

  8. [8]

    Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexan- der Kirillov, Rohit Girdhar, and Alexander G. Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764, 2021. 6

    arXiv 2021

Show all 66 references
  1. [9]

    Schwing, Alexan- der Kirillov, and Rohit Girdhar

    Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexan- der Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR,

  2. [10]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion. In CVPR, pages 4113–4123, 2024. 1, 2, 3, 6

    2024

  3. [11]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR,

  4. [12]

    De- coupling zero-shot semantic segmentation

    Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. De- coupling zero-shot semantic segmentation. In CVPR, 2022. 1, 2, 3, 6

    2022

  5. [13]

    The pascal visual object classes challenge: A retrospective.IJCV, 111:98–136, 2015

    Mark Everingham, SM Ali Eslami, Luc Van Gool, Christo- pher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective.IJCV, 111:98–136, 2015. 2, 3, 6

    2015

  6. [14]

    Large-scale unsu- pervised semantic segmentation

    Shanghua Gao, Zhong-Yu Li, Ming-Hsuan Yang, Ming- Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsu- pervised semantic segmentation. TPAMI, 45(6):7457–7476,

  7. [15]

    Open-vocabulary image segmentation

    Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:2112.12143, 2021. 1, 2, 6

    2021 arXiv

  8. [16]

    Random walks for image segmentation

    Leo Grady. Random walks for image segmentation. IEEE TPAMI, 28(11):1768–1783, 2006. 2, 5

    2006

  9. [17]

    Global knowledge calibration for fast open-vocabulary segmentation

    Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Ji- ajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, Yao Zhao, et al. Global knowledge calibration for fast open-vocabulary segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision ...

    2023

  10. [18]

    Primitive gener- ation and semantic-related alignment for universal zero-shot segmentation

    Shuting He, Henghui Ding, and Wei Jiang. Primitive gener- ation and semantic-related alignment for universal zero-shot segmentation. In CVPR, 2023. 1

    2023

  11. [19]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853–17862, 2023. 1

    2023

  12. [20]

    Densely connected parameter- efficient tuning for referring image segmentation

    Jiaqi Huang, Zunnan Xu, Ting Liu, Yong Liu, Haonan Han, Kehong Yuan, and Xiu Li. Densely connected parameter- efficient tuning for referring image segmentation. arXiv preprint arXiv:2501.08580, 2025. 1

    2025 arXiv

  13. [21]

    Segment and caption anything

    Xiaoke Huang, Jianfeng Wang, Yansong Tang, Zheng Zhang, Han Hu, Jiwen Lu, Lijuan Wang, and Zicheng Liu. Segment and caption anything. arXiv preprint arXiv:2312.00869, 2023. 1

    2023 arXiv

  14. [22]

    Proxydet: Synthesizing proxy novel classes via classwise mixup for open-vocabulary object detection

    Joonhyun Jeong, Geondo Park, Jayeon Yoo, Hyungsik Jung, and Heesu Kim. Proxydet: Synthesizing proxy novel classes via classwise mixup for open-vocabulary object detection. In AAAI, pages 2462–2470, 2024. 3

    2024

  15. [23]

    Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 3

    2021

  16. [24]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904– 4916, 2021. 1

    2021

  17. [25]

    Learning mask-aware clip repre- sentations for zero-shot segmentation

    Siyu Jiao, Yunchao Wei, Yaowei Wang, Yao Zhao, and Humphrey Shi. Learning mask-aware clip repre- sentations for zero-shot segmentation. arXiv preprint arXiv:2310.00240, 2023. 1, 6

    2023 arXiv

  18. [26]

    Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation

    Siyu Jiao, Hongguang Zhu, Jiannan Huang, Yao Zhao, Yun- chao Wei, and Humphrey Shi. Collaborative vision-text rep- resentation optimizing for open-vocabulary segmentation. In ECCV, pages 399–416, 2025. 6, 7

    2025

  19. [27]

    Weinberger, Serge J

    Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, Vladlen Koltun, and Ren ´e Ranftl. Language-driven semantic seg- mentation. In ICLR, 2022. 1, 3

    2022

  20. [28]

    Unicoder-vl: A universal encoder for vision and lan- guage by cross-modal pre-training

    Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and lan- guage by cross-modal pre-training. In AAAI, 2020. 3 9

    2020

  21. [29]

    Ordinalclip: Learning rank prompts for language-guided ordinal regression

    Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu. Ordinalclip: Learning rank prompts for language-guided ordinal regression. NeurIPS, pages 35313–35325, 2022. 3

    2022

  22. [30]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV,

  23. [31]

    Open-vocabulary semantic segmentation with mask-adapted CLIP

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yi- nan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted CLIP. arXiv preprint arXiv:2210.04150,

    arXiv

  24. [32]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014. 4

    2014

  25. [33]

    Quality- aware and selective prior enhancement memory network for video object segmentation

    Yong Liu, Ran Yu, Xinyuan Zhao, and Yujiu Yang. Quality- aware and selective prior enhancement memory network for video object segmentation. In CVPR Workshop, 2021. 1

    2021

  26. [34]

    Global spectral filter memory network for video object segmentation

    Yong Liu, Ran Yu, Jiahao Wang, Xinyuan Zhao, Yitong Wang, Yansong Tang, and Yujiu Yang. Global spectral filter memory network for video object segmentation. In ECCV, pages 648–665, 2022

    2022

  27. [35]

    Learning quality-aware dynamic memory for video object segmentation

    Yong Liu, Ran Yu, Fei Yin, Xinyuan Zhao, Wei Zhao, Wei- hao Xia, and Yujiu Yang. Learning quality-aware dynamic memory for video object segmentation. In ECCV, pages 468–486, 2022

    2022

  28. [36]

    Universal segmentation at arbi- trary granularity with language instruction

    Yong Liu, Cairong Zhang, Yitong Wang, Jiahao Wang, Yujiu Yang, and Yansong Tang. Universal segmentation at arbi- trary granularity with language instruction. arXiv preprint arXiv:2312.01623, 2023. 1

    2023 arXiv

  29. [37]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. In CVPR, pages 3491–3500, 2024. 1, 4, 6, 7

    2024

  30. [38]

    Learning high-quality dynamic memory for video object segmentation

    Yong Liu, Ran Yu, Fei Yin, Xinyuan Zhao, Wei Zhao, Wei- hao Xia, Jiahao Wang, Yitong Wang, Yansong Tang, and Yu- jiu Yang. Learning high-quality dynamic memory for video object segmentation. IEEE TPAMI, 2025. 1

    2025

  31. [39]

    Thinkbot: Embodied instruction fol- lowing with thought chain reasoning

    Guanxing Lu, Ziwei Wang, Changliu Liu, Jiwen Lu, and Yansong Tang. Thinkbot: Embodied instruction fol- lowing with thought chain reasoning. arXiv preprint arXiv:2312.07062, 2023. 1

    2023 arXiv

  32. [40]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, 2019. 3

    2019

  33. [41]

    Soc: Semantic-assisted object cluster for referring video object segmentation

    Zhuoyan Luo, Yicheng Xiao, Yong Liu, Shuyan Li, Yi- tong Wang, Yansong Tang, Xiu Li, and Yujiu Yang. Soc: Semantic-assisted object cluster for referring video object segmentation. arXiv preprint arXiv:2305.17011, 2023. 2

    2023 arXiv

  34. [42]

    Hdc: Hierarchical semantic decoding with counting assistance for generalized referring expression segmentation

    Zhuoyan Luo, Yinghao Wu, Yong Liu, Yicheng Xiao, Xiao- Ping Zhang, and Yujiu Yang. Hdc: Hierarchical semantic decoding with counting assistance for generalized referring expression segmentation. arXiv preprint arXiv:2405.15658,

    arXiv

  35. [43]

    Matrix analysis and applied linear algebra

    Carl D Meyer and Ian Stewart. Matrix analysis and applied linear algebra. SIAM, 2023. 5

    2023

  36. [44]

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan L. Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014. 2, 3, 4, 6

    2014

  37. [45]

    Siri: A simple selective retraining mechanism for transformer-based visual grounding

    Mengxue Qu, Yu Wu, Wu Liu, Qiqi Gong, Xiaodan Liang, Olga Russakovsky, Yao Zhao, and Yunchao Wei. Siri: A simple selective retraining mechanism for transformer-based visual grounding. In ECCV, 2022. 3

    2022

  38. [46]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 2,...

    2021

  39. [47]

    Hierarchical memory for long video qa

    Yiqin Wang, Haoji Zhang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Hierarchical memory for long video qa. arXiv preprint arXiv:2407.00603, 2024. 3

    2024 arXiv

  40. [48]

    Uni-adafocus: Spatial- temporal dynamic computation for video recognition

    Yulin Wang, Haoji Zhang, Yang Yue, Shiji Song, Chao Deng, Junlan Feng, and Gao Huang. Uni-adafocus: Spatial- temporal dynamic computation for video recognition. IEEE TPAMI, 2024. 3

    2024

  41. [49]

    Iterprime: Zero-shot referring image segmen- tation with iterative grad-cam refinement and primary word emphasis

    Yuji Wang, Jingchen Ni, Yong Liu, Chun Yuan, and Yan- song Tang. Iterprime: Zero-shot referring image segmen- tation with iterative grad-cam refinement and primary word emphasis. In AAAI, pages 8159–8168, 2025. 1

    2025

  42. [50]

    Sam2-love: Segment anything model 2 in language- aided audio-visual scenes

    Yuji Wang, Haoran Xu, Yong Liu, Jiaze Li, and Yansong Tang. Sam2-love: Segment anything model 2 in language- aided audio-visual scenes. In CVPR, pages 28932–28941, 2025

    2025

  43. [51]

    Hyperseg: Towards univer- sal visual segmentation with large language model

    Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyperseg: Towards univer- sal visual segmentation with large language model. arXiv preprint arXiv:2411.17606, 2024

    2024 arXiv

  44. [52]

    Instructseg: Unifying instructed visual segmentation with multi-modal large lan- guage models

    Cong Wei, Yujie Zhong, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, and Yujiu Yang. Instructseg: Unifying instructed visual segmentation with multi-modal large lan- guage models. arXiv preprint arXiv:2412.14006, 2024. 1

    2024 arXiv

  45. [53]

    A large-scale benchmark for food im- age segmentation

    Xiongwei Wu, Xin Fu, Ying Liu, Ee-Peng Lim, Steven CH Hoi, and Qianru Sun. A large-scale benchmark for food im- age segmentation. In ACM MM, 2021. 4

    2021

  46. [54]

    Detectron2

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github. com/facebookresearch/detectron2, 2019. 6

    2019

  47. [55]

    Semantic projection network for zero- and few-label semantic segmentation

    Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero- and few-label semantic segmentation. In CVPR, 2019. 1, 2

    2019

  48. [56]

    Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection

    Yicheng Xiao, Zhuoyan Luo, Yong Liu, Yue Ma, Heng- wei Bian, Yatai Ji, Yujiu Yang, and Xiu Li. Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection. In CVPR, pages 18709– 18719, 2024. 5

    2024

  49. [57]

    Sed: A simple encoder-decoder for open- vocabulary semantic segmentation

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- vocabulary semantic segmentation. In CVPR, pages 3426– 3436, 2024. 1, 2, 3, 6 10

    2024

  50. [58]

    Alvarez, and Ping Luo

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and ef- ficient design for semantic segmentation with transformers. In NIPS, 2021. 1

    2021

  51. [59]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In CVPR, pages 2955–2966, 2023. 2

    2023

  52. [60]

    A simple baseline for zero- shot semantic segmentation with pre-trained vision-language model

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero- shot semantic segmentation with pre-trained vision-language model. arXiv preprint arXiv:2112.14757, 2021. 1, 2, 3, 6

    2021 arXiv

  53. [61]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR, pages 2945–2954, 2023. 1, 2, 6, 7

    2023

  54. [62]

    Masq- clip for open-vocabulary universal image segmentation

    Xin Xu, Tianyi Xiong, Zheng Ding, and Zhuowen Tu. Masq- clip for open-vocabulary universal image segmentation. In ICCV, pages 887–898, 2023. 1

    2023

  55. [63]

    Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang- Chieh Chen. Convolutions die hard: Open-vocabulary seg- mentation with single frozen convolutional clip. NeurIPS, pages 32215–32234, 2023. 1, 2, 6

    2023

  56. [64]

    Prototypical matching and open set rejection for zero-shot semantic segmentation

    Hui Zhang and Henghui Ding. Prototypical matching and open set rejection for zero-shot semantic segmentation. In ICCV, 2021. 1

    2021

  57. [65]

    Flash-vstream: Memory- based real-time understanding for long video streams

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory- based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085, 2024. 3

    2024 arXiv

  58. [66]

    Scene parsing through ADE20K dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017. 2, 3, 4, 6 11

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.