Pith. sign in

REVIEW 2 major objections 4 minor 51 references

CLIP-based open-vocabulary detectors fail when labels are swapped for synonyms; SynCLIP trains attention maps to stay consistent across them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 07:41 UTC pith:H5YBRXY3

load-bearing objection Solid engineering fix for a real CLIP dense-perception failure mode; gains are real, modest, and well-supported by ablations and a synonym-stress test. the 2 major comments →

arxiv 2607.11008 v1 pith:H5YBRXY3 submitted 2026-07-13 cs.CV cs.AI

SynCLIP: Synonym-Coherent Language-Image Pretraining for Robust Open-Vocabulary Dense Perception

Mingjie Xie , Guangjun He , Dongli Xu , Youtian Lin , Hongjue Li , Pengming Feng , Jian Guan , Yue Deng This is my paper
classification cs.CV cs.AI
keywords open-vocabulary dense perceptionCLIPsynonym robustnessspatial attention alignmentvision-language pretrainingobject detectionsemantic segmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Open-vocabulary dense perception tries to find objects never seen in training by matching image regions to free-form text. Current CLIP-style models produce different spatial attention maps for synonyms of the same object, so detection and segmentation scores fall when users rephrase a label. The paper claims this synonym-induced grounding inconsistency is a core robustness failure. It introduces SynCLIP, a pretraining recipe that forces the student model’s attention for ordinary category names to match the attention produced by synonym-and-definition expansions, then refines those maps by selecting the strongest tokens and mixing in spatial context from a vision foundation model. A new corpus (SEViC) supplies the expanded phrases for every COCO/LVIS category. On standard open-vocabulary detection benchmarks the method raises novel-class accuracy and, crucially, loses far less accuracy when the evaluation labels themselves are replaced by synonyms.

Core claim

Synonym-induced grounding inconsistency is a measurable failure mode of CLIP-based open-vocabulary dense perception: semantically equivalent expressions produce divergent spatial attention maps and therefore lower localization accuracy. Aligning student attention maps to synonym-enriched teacher maps (SSA) and then refining them via top-k semantic-token selection plus vision-foundation-model context (SAR) measurably reduces that inconsistency and yields higher novel-class performance among CLIP-based detectors.

What carries the argument

Semantic-consistent Spatial Attention alignment (SSA) plus Spatial Attention Refinement (SAR): SSA minimizes the L2 distance between attention maps of original labels and of synonym/definition expansions; SAR keeps only the top-k tokens of the enriched map, builds spatial-correlation maps from a frozen vision foundation model, and fuses them to produce a refined supervision target.

Load-bearing premise

The LLM-generated synonyms and definitions used as teacher targets are faithful enough, and visually equivalent enough, that forcing student attention to match them produces real synonym robustness rather than overfitting to those particular expansions.

What would settle it

Replace every evaluation label on OV-COCO with a fresh set of human-written synonyms never seen in SEViC; if the novel-class AP drop for SynCLIP becomes as large as the drops reported for CLIPSelf and DeCLIP, the claimed robustness fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Open-vocabulary detectors can be made substantially less sensitive to how a user phrases a category name.
  • Pretraining objectives that explicitly regularize spatial attention across lexical variants become a practical design lever for CLIP-based dense perception.
  • A synonym-enriched image-text corpus built from COCO/LVIS categories can serve as reusable supervision for other region-level vision-language models.
  • Novel-class accuracy on OV-COCO and OV-LVIS rises without any increase in inference parameters or FLOPs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same SSA/SAR recipe could be applied to open-vocabulary video or 3-D perception where synonym variation is equally common.
  • If the LLM expansions are the main source of the signal, cheaper synonym lists (thesaurus + definition lookup) might recover most of the gain without large language models.
  • Attention-consistency losses of this form may also reduce brittleness when prompts contain adjectives, plurals, or multi-word descriptions rather than pure synonyms.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper identifies synonym-induced grounding inconsistency in CLIP-based open-vocabulary dense perception (OVDP), where synonymous or definitional expressions produce inconsistent spatial attention maps and degrade localization (Figs. 1–2). It proposes SynCLIP, built on a DeCLIP-style dual-encoder setup, with two modules: Semantic-consistent Spatial Attention alignment (SSA) that minimizes the L2 discrepancy between student attention maps of original labels and teacher maps of synonym-enriched expressions (Eqs. 5–7), and Spatial Attention Refinement (SAR) that selects top-k tokens from the enriched maps, computes VFM (DINOv2) spatial correlations, and fuses them via weighted aggregation (Eqs. 8–11). A Synonym-Enriched Visual Corpus (SEViC) is constructed from COCO/LVIS images by LLM-augmenting 1,234 categories with synonyms and definitions. Downstream evaluation with F-ViT on OV-COCO and OV-LVIS reports improved novel-class metrics (43.6/49.8 AP_novel_50 on OV-COCO) and a smaller drop under synonym substitution (Table 3) relative to CLIPSelf and DeCLIP, supported by component and hyper-parameter ablations.

Significance. If the empirical claims hold, the work supplies a concrete, reproducible pretraining recipe that measurably reduces linguistic brittleness in region-level CLIP adaptations—an issue of practical importance for robotics and autonomous driving. The synonym-substitution protocol (Table 3), component ablations (Table 4), k-sweep (Fig. 6) and aggregation-weight study (Supp. Table 6) constitute a solid experimental package; the public code and SEViC construction pipeline further raise the contribution’s utility. The identification of the inconsistency phenomenon itself is a useful diagnostic for the OVDP community.

major comments (2)
  1. [Table 2b / §4.3.1] Table 2b (OV-LVIS, ViT-L/14): SynCLIP reports 37.2 mAP_mask_r, identical to the DeCLIP baseline. The abstract and §4.3.1 claim “state-of-the-art performance among CLIP-based OVDP methods”; the tie on the more challenging long-tailed benchmark should be stated explicitly rather than subsumed under a blanket SOTA assertion, and the text should clarify whether the gain is backbone-dependent.
  2. [§4.1.1 / Supp. §6] §4.1.1 and Supp. §6: SEViC relies on DeepSeek-generated synonyms/definitions filtered by ChatGPT. While the synonym-substitution evaluation (Table 3) is independent of the training expansions, the paper never quantifies residual semantic drift or visual non-equivalence after filtering. A small human-rated sample or an ablation that replaces LLM expansions with purely LVIS-provided synonyms would strengthen the claim that the observed robustness is not an artifact of the particular lexical set chosen by the LLMs.
minor comments (4)
  1. [§4.3] No error bars or multi-seed statistics are reported for any table. Even a single additional seed for the key OV-COCO numbers would help readers gauge stability of the 2–3-point gains.
  2. [§3.1] Eq. (4) re-uses the correlative self-attention of SCLIP without restating the temperature or projection details; a one-sentence reminder would aid readers who have not memorized the cited work.
  3. [Fig. 5] Figure 5 captions use “Top/Bottom” without panel labels; adding (a)/(b) markers would improve readability.
  4. [§4.2] The default semantic-alignment weight 0.05 and α=β=0.5 are stated but not justified beyond the later ablations; a short sentence linking them to the Supp. sweeps would tighten the narrative.

Circularity Check

0 steps flagged

No significant circularity: empirical pretraining method with independent held-out evaluation; losses and modules do not reduce reported metrics to their own inputs by construction.

full rationale

SynCLIP is an engineering paper that defines two training modules (SSA via L_semantic / L+_semantic MSE alignment of attention maps in Eqs. 5–7 and 11; SAR via top-k token selection + VFM spatial correlation aggregation in Eqs. 8–10) and a synonym-enriched corpus SEViC, then measures downstream AP / mAP and synonym-substitution robustness on the standard OV-COCO and OV-LVIS splits. The reported numbers (e.g., 43.6 / 49.8 AP_novel_50, Table-3 drop of –4.4) are ordinary empirical outcomes of training + evaluation; none of the equations algebraically force those metrics, nor is any free parameter fitted on the test quantities and then re-presented as a prediction. SEViC construction (LLM generation + consistency filter) is an input design choice, not a self-referential derivation of the claims. Citations to DeCLIP / CLIPSelf / DINOv2 are ordinary baselines and components; they are not load-bearing uniqueness theorems or self-citations that close a circular chain. Component ablations (Table 4, Supp. Tables 5–6, Fig. 6) further treat the modules as independent variables rather than tautologies. Consequently the derivation chain contains no self-definitional step, no fitted-input-as-prediction, and no uniqueness imported from the authors’ prior work. Score 0 is the correct, non-manufactured finding.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 3 invented entities

The central empirical claim rests on a small set of hand-chosen hyperparameters, standard CLIP/DINOv2 domain assumptions, and three invented modules/corpus whose only evidence is the paper’s own ablations.

free parameters (3)
  • semantic alignment loss weight = 0.05
    Default 0.05; controls strength of L_semantic / L+_semantic; chosen by hand, not derived.
  • attention aggregation coefficients α, β = 0.5 / 0.5
    Default both 0.5; trade-off between spatial and semantic attention in Eq. 10; ablated but still free.
  • number of semantic tokens k = 7
    Default 7; TopK selection in SAR; performance peaks at 7 then drops, so fitted to validation.
axioms (3)
  • domain assumption CLIP’s correlative self-attention (Eq. 4) yields usable dense features after discarding the CLS token
    Inherited from SCLIP / prior dense-CLIP literature; never re-proved.
  • domain assumption DINOv2 spatial correlations are a reliable external teacher for refining attention maps
    Assumed in SAR (Section 3.3); success of the method depends on this transfer.
  • ad hoc to paper LLM-generated synonyms and definitions (DeepSeek + ChatGPT filter) are visually equivalent to the original category names
    Core premise of SEViC construction (Supp. Section 6); no human verification study is reported.
invented entities (3)
  • Semantic-consistent Spatial Attention alignment (SSA) module no independent evidence
    purpose: Minimize L2 discrepancy between attention maps of original labels and synonym-enriched expressions
    New loss and dual-encoder construction introduced in Section 3.2; no independent prior evidence.
  • Spatial Attention Refinement (SAR) module no independent evidence
    purpose: Select top-k tokens and fuse VFM spatial correlations for cleaner grounding
    New two-stage procedure in Section 3.3; evidence is only the paper’s own ablations.
  • Synonym-Enriched Visual Corpus (SEViC) no independent evidence
    purpose: Provide multi-synonym + definition supervision for each of 1,234 categories
    New dataset constructed for this work; quality rests on LLM generation + filter.

pith-pipeline@v1.1.0-grok45 · 20646 in / 2615 out tokens · 30541 ms · 2026-07-14T07:41:01.026392+00:00 · methodology

0 comments
read the original abstract

Open-vocabulary dense perception (OVDP) aims to localize objects unseen during training by leveraging textual knowledge. Despite the remarkable progress of recent CLIP-based approaches, we identify a critical limitation: synonym-induced grounding inconsistency, where semantically equivalent expressions yield disparate spatial attention patterns. This inconsistency undermines the robustness and performance of existing methods in real-world OVDP applications. To address this issue, we propose SynCLIP, a Synonym-Coherent Language-Image Pretraining framework that enhances synonym-robust grounding for OVDP. SynCLIP introduces a Semantic-consistent Spatial Attention alignment (SSA) module to enhance spatial attention consistency by minimizing discrepancies between attention maps of original and synonymous expressions. Furthermore, a Spatial Attention Refinement (SAR) module selectively strengthens the most semantically relevant spatial regions within aligned maps for more precise and stable grounding. To support synonym-coherent pretraining, we also construct a Synonym-Enriched Visual Corpus (SEViC), which augments each category with multiple synonyms and textual definitions. Extensive experiments on multiple benchmarks demonstrate that SynCLIP substantially improves grounding consistency under diverse linguistic variants and achieves state-of-the-art performance among CLIP-based OVDP methods. Code is available at https://github.com/Justlovesmile/SynCLIP.

Figures

Figures reproduced from arXiv: 2607.11008 by Dongli Xu, Guangjun He, Hongjue Li, Jian Guan, Mingjie Xie, Pengming Feng, Youtian Lin, Yue Deng.

Figure 1
Figure 1. Figure 1: Illustration of synonym-induced grounding inconsis [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of spatial attention distributions generated [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the SynCLIP framework, which consists of two key components, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: visualizes this process, showing how semantic at￾tention, spatial correlation, and aggregated attention jointly contribute to SAR. Finally, to balance semantic relevance and spatial preci￾sion, SAR fuses the semantic attention Asem and the spatial correlation attention Aspa as: \mathbf {A}_{\text {agg}} = \alpha \mathbf {A}_{\text {spa}} + \beta \mathbf {A}_{\text {sem}}, (10) where α and β control the tra… view at source ↗
Figure 6
Figure 6. Figure 6: Effect of the number of semantic tokens, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of attention maps under synony [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Overview of the SEViC construction pipeline, consisting of (1) Data Collection from COCO and LVIS datasets, which gathers the [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of five examples from the SAR module with [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative results on (a) OV-COCO and (b) OV-LVIS datasets. Each example includes both ground-truth annotations and [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 5 linked inside Pith

  1. [1]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    CAT-Seg: Cost aggregation for open-vocabulary semantic segmenta- tion

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. CAT-Seg: Cost aggregation for open-vocabulary semantic segmenta- tion. InCVPR, pages 4113–4123, 2024. 3

    2024

  3. [3]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InICLR, 2021. 3

    2021

  4. [4]

    A hierarchical semantic distilla- tion framework for open-vocabulary object detection.IEEE TMM, 27:8835–8846, 2025

    Shenghao Fu, Junkai Yan, Qize Yang, Xihan Wei, Xiaohua Xie, and Wei-Shi Zheng. A hierarchical semantic distilla- tion framework for open-vocabulary object detection.IEEE TMM, 27:8835–8846, 2025. 1

    2025

  5. [5]

    Vision-language pre-training: Basics, re- cent advances, and future trends.Found

    Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao. Vision-language pre-training: Basics, re- cent advances, and future trends.Found. Trends Comput. Graph. Vis., 14:163–352, 2022. 3

    2022

  6. [6]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. InICLR, 2022. 3, 6, 7

    2022

  7. [7]

    EARL: An elliptical distribution aided adaptive rotation label assignment for oriented object detec- tion in remote sensing images.IEEE Trans

    Jian Guan, Mingjie Xie, Youtian Lin, Guangjun He, and Pengming Feng. EARL: An elliptical distribution aided adaptive rotation label assignment for oriented object detec- tion in remote sensing images.IEEE Trans. Geosci. Remote Sens., 61:1–15, 2023. 1

    2023

  8. [8]

    LVIS: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, Ross Girshick, et al. LVIS: A dataset for large vocabulary instance segmentation. InCVPR, pages 5356–5364, 2019. 5, 6, 1

    2019

  9. [9]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InCVPR, pages 770–778, 2016. 3, 7

    2016

  10. [10]

    Mask R-CNN

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask R-CNN. InCVPR, pages 2961–2969, 2017. 1

    2017

  11. [11]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InICML, pages 4904–

  12. [12]

    Con- trastive feature masking open-vocabulary vision transformer

    Dahun Kim, Anelia Angelova, Weicheng Kuo, et al. Con- trastive feature masking open-vocabulary vision transformer. InICCV, pages 15602–15612, 2023. 7

    2023

  13. [13]

    Region- aware pretraining for open-vocabulary object detection with vision transformers

    Dahun Kim, Anelia Angelova, Weicheng Kuo, et al. Region- aware pretraining for open-vocabulary object detection with vision transformers. InCVPR, pages 11144–11154, 2023. 7

    2023

  14. [14]

    F-VLM: Open-vocabulary object detec- tion upon frozen vision and language models

    Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-VLM: Open-vocabulary object detec- tion upon frozen vision and language models. InICLR, 2023. 7

    2023

  15. [15]

    Band prompting aided SAR and multi-spectral data fusion framework for local climate zone classification

    Haiyan Lan, Shujun Li, Mingjie Xie, Xuanjia Zhao, Hongn- ing Liu, Pengming Feng, Dongli Xu, Guangjun He, and Jian Guan. Band prompting aided SAR and multi-spectral data fusion framework for local climate zone classification. In ICASSP, pages 1–5, 2025. 3

    2025

  16. [16]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jian- wei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. InCVPR, pages 10965–10975,

  17. [17]

    Scaling language-image pre-training via masking

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichten- hofer, and Kaiming He. Scaling language-image pre-training via masking. InCVPR, pages 23390–23400, 2023. 1

    2023

  18. [18]

    Open-vocabulary semantic segmentation with mask-adapted CLIP

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted CLIP. InCVPR, pages 7061–7070, 2023. 3

    2023

  19. [19]

    Learning object-language alignments for open-vocabulary object de- tection

    Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gho- lamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object de- tection. InICLR, 2023. 3

    2023

  20. [20]

    Microsoft COCO: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 2, 5, 6, 1

    2014

  21. [21]

    DeepSeek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 6, 1

    Pith/arXiv arXiv 2024

  22. [22]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019. 6

    2019

  23. [23]

    DINOv2: Learning robust visual features without supervi- sion.Trans

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervi- sion.Trans. Mach. Learn. Res., 2024. 2, 5, 6

    2024

  24. [24]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InICML, pages 8748–8763. PMLR, 2021. 1, 2

    2021

  25. [25]

    Prompt-guided transform- ers for end-to-end open-vocabulary object detection.arXiv preprint arXiv:2303.14386, 2023

    Hwanjun Song and Jihwan Bang. Prompt-guided transform- ers for end-to-end open-vocabulary object detection.arXiv preprint arXiv:2303.14386, 2023. 7

    Pith/arXiv arXiv 2023

  26. [26]

    EV A-CLIP: Improved training techniques for CLIP at scale.arXiv preprint arXiv:2303.15389, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EV A-CLIP: Improved training techniques for CLIP at scale.arXiv preprint arXiv:2303.15389, 2023. 1, 2

    Pith/arXiv arXiv 2023

  27. [27]

    EfficientNet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. InICML, pages 6105–6114. PMLR, 2019. 7

    2019

  28. [28]

    SCLIP: Rethinking self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. SCLIP: Rethinking self-attention for dense vision-language inference. InECCV, pages 315–332. Springer, 2024. 3

    2024

  29. [29]

    OV-DQUO: Open-vocabulary detr with denoising text query training and open-world un- known objects supervision

    Junjie Wang, Bin Chen, Bin Kang, Yulin Li, Weizhi Xian, Yichi Chen, and Yong Xu. OV-DQUO: Open-vocabulary detr with denoising text query training and open-world un- known objects supervision. InAAAI, pages 7762–7770,

  30. [30]

    DeCLIP: Decoupled learning for open- vocabulary dense perception

    Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. DeCLIP: Decoupled learning for open- vocabulary dense perception. InCVPR, pages 14824–14834,

  31. [31]

    Towards open vocabulary learning: A survey.IEEE TPAMI, 46(7):5092–5113, 2024

    Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, et al. Towards open vocabulary learning: A survey.IEEE TPAMI, 46(7):5092–5113, 2024. 1

    2024

  32. [32]

    Aligning bag of regions for open- vocabulary object detection

    Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open- vocabulary object detection. InCVPR, pages 15254–15264,

  33. [33]

    CLIPSelf: Vision trans- former distills itself for open-vocabulary dense prediction

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. CLIPSelf: Vision trans- former distills itself for open-vocabulary dense prediction. In ICLR, 2024. 1, 2, 3, 6, 7

    2024

  34. [34]

    CLIM: Contrastive language-image mosaic for region representation

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, and Chen Change Loy. CLIM: Contrastive language-image mosaic for region representation. InAAAI, pages 6117–6125,

  35. [35]

    CORA: Adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching

    Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. CORA: Adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching. InCVPR, pages 7031–7040, 2023. 1, 7

    2023

  36. [36]

    A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model

    Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open- vocabulary semantic segmentation with pre-trained vision- language model. InECCV, pages 736–753. Springer, 2022. 3

    2022

  37. [37]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. InCVPR, pages 2945–2954, 2023. 3

    2023

  38. [38]

    ResCLIP: Residual attention for training-free dense vision- language inference

    Yuhang Yang, Jinhong Deng, Wen Li, and Lixin Duan. ResCLIP: Residual attention for training-free dense vision- language inference. InCVPR, pages 29968–29978, 2025. 1

    2025

  39. [39]

    Open-vocabulary SAM: Segment and recognize twenty-thousand classes interactively

    Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open-vocabulary SAM: Segment and recognize twenty-thousand classes interactively. InECCV, pages 419–437. Springer, 2024. 7

    2024

  40. [40]

    Florence: A new foundation model for computer vision.arXiv preprint arXiv:2111.11432, 2021

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision.arXiv preprint arXiv:2111.11432, 2021. 2

    Pith/arXiv arXiv 2021

  41. [41]

    Open-vocabulary detr with conditional matching

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. InECCV, pages 106–122. Springer, 2022. 7

    2022

  42. [42]

    GaussianGrasper: 3d lan- guage gaussian splatting for open-vocabulary robotic grasp- ing.IEEE Robot

    Yuhang Zheng, Xiangyu Chen, Yupeng Zheng, Songen Gu, Runyi Yang, Bu Jin, Pengfei Li, Chengliang Zhong, Zeng- mao Wang, Lina Liu, et al. GaussianGrasper: 3d lan- guage gaussian splatting for open-vocabulary robotic grasp- ing.IEEE Robot. Autom. Lett., 9(9):7827–7834, 2024. 1

    2024

  43. [43]

    RegionCLIP: Region-based language-image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. RegionCLIP: Region-based language-image pretraining. InCVPR, pages 16793–16803,

  44. [44]

    Extract free dense labels from CLIP

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from CLIP. InECCV, pages 696–712. Springer,

  45. [45]

    ALGPT: Multi-agent cooperative framework for open- vocabulary multi-modal auto-annotating in autonomous driving.IEEE Trans

    Yijie Zhou, Xianhui Cheng, Qiming Zhang, Lei Wang, Wenchao Ding, Xiangyang Xue, Chunbo Luo, and Jian Pu. ALGPT: Multi-agent cooperative framework for open- vocabulary multi-modal auto-annotating in autonomous driving.IEEE Trans. Intell. Veh., pages 1–15, 2024. 1

    2024

  46. [46]

    A survey on open- vocabulary detection and segmentation: Past, present, and future.IEEE TPAMI, 46(12):8954–8975, 2024

    Chaoyang Zhu and Long Chen. A survey on open- vocabulary detection and segmentation: Past, present, and future.IEEE TPAMI, 46(12):8954–8975, 2024. 1, 2 SynCLIP: Synonym-Coherent Language-Image Pretraining for Robust Open-Vocabulary Dense Perception Supplementary Material Overview This supplementary material provides additional details and results that com...

    2024

  47. [47]

    Details of SEViC Construction This section provides a detailed description of the con- struction pipeline of our synonym-enriched visual corpus (SEViC), complementing the overview in the main paper. As illustrated in Figure 7, the pipeline consists of three major stages,i.e., Data Collection, which gathers the full category vocabulary and initial textual ...

  48. [48]

    ob- ject

    and LVIS [8], which share the same image set but differ in granularity and coverage. This provides a unified vocabu- lary of 1,232 unique object category names and 118,287 im- ages. When available, we also extract their accompanying LVIS-provided definitions and synonyms. These human- curated textual descriptions serve as the initial reference for later L...

  49. [49]

    These studies further validate the design choices introduced in the main paper and clarify how each component contributes to robust synonym-coherent dense perception

    Additional Ablation Studies This section provides additional analyses on two key com- ponents of SynCLIP,i.e., the types of semantically enriched textual variants used in SSA for semantic–consistent atten- tion alignment, and the aggregation weights in SAR that balance semantic relevance and spatial precision. These studies further validate the design cho...

  50. [50]

    Additional Qualitative Analysis This section presents additional qualitative analyses, includ- ing visualizations of attention maps from the SAR mod- ule and prediction results on two standard dense perception benchmarks, offering a more comprehensive illustration of the effectiveness and superiority of our method. α β APnovel 50 APbase 50 APall 50 0.9 0....

  51. [51]

    All measurements are conducted under the same hardware setup using four NVIDIA A100 GPUs with 40GB memory and an input resolution of 560

    Efficiency Analysis To provide a comprehensive view of the computational characteristics of SynCLIP, we report the training time, model parameters, FLOPs and APnovel 50 on OV-COCO in Ta- ble 7. All measurements are conducted under the same hardware setup using four NVIDIA A100 GPUs with 40GB memory and an input resolution of 560. Compared with Method Back...