Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A training-free pipeline called Trident raises average open-vocabulary segmentation accuracy on eight benchmarks from 44.4 to 48.6 by splicing CLIP and DINO features first and then aggregating with SAM's correlations.

desk verdict The core splice-then-segment idea is new, decently ablated, and likely real; the +4.2 mIoU SOTA claim is not yet secure because the headline comparison mixes evaluation protocols. read the letter →

arxiv 2411.09219 v1 pith:JMVO3B7O submitted 2024-11-14 cs.CV

Yuheng Shi , Minjing Dong , Chang Xu This is my paper · ORCID
classification cs.CV
keywords open-vocabularysemanticsegmentationtraining-freeCLIPDINOSAMsplice-then-segmentaffinitymatrixhigh-resolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-vocabulary semantic segmentation usually hits a resolution ceiling: CLIP works on small inputs, so high-resolution images are chopped into windows, segmented, and stitched back together. This paper argues that the chopping-then-stitching order is the problem, because each window sees too little context and classification fails on large objects. It proposes reversing the pipeline: extract CLIP and DINO features from the windows, splice them into one map, then aggregate that map with a global correlation matrix built from SAM's high-resolution encoder. This training-free system, called Trident, is reported to raise average mean intersection-over-union (mIoU) across eight benchmarks from 44.4 to 48.6, beating all training-free baselines and some trained ones. The claim matters because it suggests fused frozen foundation models can substantially improve dense prediction without any learned parameters.

What carries the argument

The load-bearing object is the affinity matrix $\mathbf{A}$. SAM encodes the full high-resolution source image into features $\mathbf{F}$; cosine similarity $\mathbf{C}$ between normalized $\mathbf{F}$ entries captures low-level visual agreement, while the attention weights $\mathbf{W}$ from SAM's last transformer layer carry more semantic grouping but attend to background. Trident forms $\mathbf{A}$ as $(\mathbf{W} + \mathbf{M}) / \|\mathbf{W} + \mathbf{M}\|$, where $M_{ij}=0$ if $C_{ij} \ge \epsilon$ and $M_{ij}=-W_{ij}$ otherwise, so attention to tokens with weak low-level agreement is suppressed. Multiplying the spliced CLIP/DINO feature map by $\mathbf{A}$ performs one global aggregation whose receptive field is the entire source image rather than a single sliding window. DINO supplies object-level spatially covariant correlations inside each sub-image before splicing, and SAM's decoder refines the output using point, box, and scaled mask prompts derived from Trident's own confidence map.

What would settle it

Take a high-resolution image containing repeated similar objects at different window positions. Run Trident once with the full global affinity matrix $\mathbf{A}$ and once with $\mathbf{A}$ replaced by a block-diagonal matrix that only mixes tokens within the same window; if the full matrix does not beat the block-diagonal version by a clear margin, the claimed cross-window aggregation is not doing the work. A direct alignment check is to register the spliced CLIP/DINO feature map against SAM's feature grid using known correspondences and measure whether the average misalignment exceeds one feature-grid cell.

Watch

Extended reading notes

Core claim

The paper's central claim is that the resolution failure of training-free CLIP segmentation is caused by isolated per-window processing, and that reversing the order removes the failure. In the old segment-then-splice pipeline, each sub-image's features are correlated, classified, and spliced only at the end; as source resolution rises from 336 to 688 on PASCAL VOC, ProxyCLIP's mIoU falls by up to 9.7 points because no window sees a whole object. Trident instead splices the per-window CLIP and DINO features into a single map, then applies a global affinity matrix built from SAM. The affinity matrix takes SAM's last-layer attention weights and zeros out attention to token pairs whose SAM-feature cosine similarity falls below a threshold, suppressing background links while keeping semantic ones. After one global aggregation, text embeddings classify the map, and the coarse result is converted into point, box, and mask prompts for SAM's decoder to refine. With OpenCLIP-H/14, the paper reports 48.6 average mIoU against 44.4 for the previous best training-free method.

Load-bearing premise

The load-bearing premise is that stitching together per-window CLIP/DINO feature maps, each computed with its own position coding, still lines up with SAM's full-image feature grid closely enough that SAM's correlation matrix can mix features across window boundaries without scrambling which pixels correspond to which locations.

Editorial extensions

If this is right

  • Higher source-image resolution generally becomes an asset rather than a liability: the reported ablations show Trident's mIoU improving with resolution on most benchmarks, with VOC20 the noted exception because the underlying baseline weakens.
  • Window-boundary 'panel' artifacts disappear from activation and segmentation maps because the global affinity matrix lets every output token attend across the whole image.
  • The reported gains over ProxyCLIP are about 3.5 mIoU on average with CLIP ViT-B/16 and 4.2 with OpenCLIP ViT-H/14, split between the splice-then-segment aggregation and the SAM prompt refinement.
  • SAM refinement contributes roughly 1.5 mIoU on top of the aggregation and beats the PAMR post-processing baseline on Trident's cleaner masks.
  • All gains come without training, at the cost of lower throughput: about 10 images per second in the base configuration and 5 in the huge configuration on one RTX 4090.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same splice-then-segment pattern could transfer to other frozen dense-prediction models, such as open-vocabulary detection or monocular depth estimation, whenever a high-resolution encoder supplies a global correlation prior.
  • Editorial extension: a cheaper variant would compute the correlation matrix from a single high-resolution self-supervised encoder and drop per-window DINO guidance, isolating whether the global aggregation alone carries the gain.
  • Editorial extension: window size, stride, and per-dataset resolution are manual choices; a systematic sweep over them would show whether Trident's advantage persists without per-benchmark tuning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Trident, a training-free framework for open-vocabulary semantic segmentation that replaces the segment-then-splice sliding-window pipeline with a splice-then-segment paradigm. CLIP and DINO features extracted from overlapping sub-images are spliced into a global feature map, which is then aggregated through a correlation/affinity matrix derived from SAM's encoder features and attention weights, with a thresholded masking mechanism. The coarse segmentation is further refined by converting it into point, box, and mask prompts for SAM. Experiments are reported on eight benchmarks (VOC20/21, Context59/60, COCO Object/Stuff, Cityscapes, ADE20k), claiming an average mIoU of 48.6 with OpenCLIP-H/14 compared with 44.4 for the previous training-free SOTA ProxyCLIP. Component ablations in Tables 3 through 7 support the contribution of global aggregation, the affinity matrix, and SAM refinement.

Significance. If the reported gains survive a properly controlled comparison, Trident is a meaningful advance: it is training-free, uses no learned parameters, and shows consistent improvements across all eight benchmarks. The manuscript's main strengths are the component-level ablation tables, which support the internal design choices, and the efficiency analysis in Table 8. The central SOTA claim, however, currently rests on cross-paper baseline numbers obtained under different input resolutions and strides, so the quantitative headline is not yet secure. The proposed paradigm is clearly scoped and likely to be of interest to the dense-prediction and vision-foundation-model communities.

major comments (4)
  1. [§4.1, Tables 2 and 7] The headline comparison is not performed under a common evaluation protocol. Section 4.1 assigns Trident a different short-side resolution and stride for each benchmark (336 for VOC20, 448 for VOC21/Object/Stuff, 576 for Context/ADE, 688 for Cityscapes) and states that for other competitors 'we report their performance as described in their respective publications.' Since Table 7 shows ProxyCLIP's own mIoU varies with resolution (e.g., VOC20 drops from 79.7 at 336-336-112 to 73.4 at 576-336-224), and since the ProxyCLIP numbers in Table 2 (e.g., Context59 39.1, Context60 35.3, ADE 20.2) do not match any ProxyCLIP row in Table 7 (e.g., Context59 34.4-34.8, Context60 37.0-38.3, ADE 19.0-19.4), the asserted improvement from 44.4 to 48.6 may be partly a result of different resolutions and strides. Please rerun all training-free competitors under Trident's protocol, or under a single common protocol, and report those numbers in the main table.
  2. [§3.3, Eq. (3)] The load-bearing spatial alignment in the splice-then-segment step is not described. The sentence 'For brevity, the interpolation for Ifeat to align its size with A is omitted here' leaves open how sub-image features with crop-local position embeddings are placed into the global grid, how overlapping windows are combined, and what interpolation is used to match Ifeat with SAM's feature grid. This is precisely the mechanism that makes global aggregation across windows meaningful, and the paper's quantitative claims cannot be reproduced without it. Please provide the full mapping, including tensor reshaping and interpolation details, or pseudocode.
  3. [§3.3, Eqs. (4)-(5)] The masking threshold epsilon is a free parameter that controls which token pairs contribute to the affinity matrix, yet its value is never reported in the paper or in the ablation tables. Without this value and without a sensitivity analysis, the reported gains cannot be reproduced or judged for stability. Please report epsilon for each configuration and include an ablation over epsilon.
  4. [§4.1, Table 7] The per-dataset resolutions and strides appear to be chosen to optimize Trident's performance (for example, stride 112 is used only for VOC20), and Table 7 shows that these choices have a large effect on results. The paper should state whether these settings were selected on the validation splits, and should report results under a fixed resolution/stride protocol as a robustness check, alongside the tuned numbers.
minor comments (5)
  1. [Abstract and §1] There are typos: 'spatial invariance semantic' should likely be 'spatial-invariance semantics', and 'migrate' in Section 3.2 should be 'mitigate'.
  2. [§3.4] The phrase 'morphology methods [19, 63]' is vague; please specify the connected-component labeling and morphological operations used to decompose the binary mask.
  3. [Table 2] The 'Avg.' column is not defined; please clarify how missing entries (e.g., SAM-CLIP has no VOC20/Context59 values) are handled when computing the average.
  4. [Table 4] The ProxyCLIP baseline in Table 4 (e.g., V21 59.2, ADE 19.0) differs from the ProxyCLIP row in Table 2 (V21 61.3, ADE 20.2); the text acknowledges resolution differences, but it would help to state explicitly that Table 4 uses the authors' reimplementation under Trident's protocol.
  5. [Introduction, Figure 1] Figure 1 is referenced in the introduction but is not visible in the manuscript text; please ensure the final version includes it with labeled axes.

Circularity Check

2 steps flagged · score 4.0 of 10

Reported SOTA margin is partly fitted to the evaluation benchmarks via per-dataset resolution/stride and alpha selection; no derivationally circular self-citation chain.

  1. fitted input called prediction [Section 3.4 (Eq. 7) and Section 4.3, Table 6]
    "Performance improves substantially as α decreases logarithmically, reaching optimal results at α = 0.005, which we adopt as our default configuration for SAM refinement."

    Table 6 sweeps α on VOC20 and Cityscapes and reports mIoU; the final Table 2 numbers for those datasets (VOC20 84.5, Cityscapes 42.9) are exactly the values obtained with the α selected by that sweep. The SAM-refinement gain is therefore a fitted maximum on the reported benchmarks rather than an independent prediction of the method.

  2. fitted input called prediction [Section 4.1 and Table 7]
    "To accommodate different dataset configurations, we resize images accordingly: the shorter side is set to 336 pixels for VOC20, 448 pixels for VOC21, Object, and Stuff, 576 pixels for Context59, Context60, and ADE, and 688 pixels for Cityscapes. All benchmarks use a sliding window of 336×336 pixels and the stride is 224 for most, except for VOC20, which uses a stride of 112."

    Table 7 is a per-dataset resolution/stride sweep on the same benchmarks whose final mIoU is reported, and the selected settings are the ones with the best Trident scores (e.g., 576 for Context59 gives 37.2 vs 35.8 at 336; 448 for Stuff gives 27.6). Baselines in Table 2 are quoted from their publications rather than rerun under these settings, so part of the claimed 44.4→48.6 improvement is an artifact of choosing evaluation protocol on the reported data, not a free prediction.

full rationale

The paper contains no self-citation chain and no unique-theorem or ansatz-by-citation circularity: CLIP, DINO, and SAM are external frozen models; ProxyCLIP's masking mechanism is cited externally and ablated; the splice-then-segment operation is a fixed, parameter-free computation on those features. The core claim therefore has independent empirical content. The circularity-adjacent issue is confined to evaluation: the mask-prompt coefficient α, per-dataset input resolutions and strides, and the unreported threshold ε are free hyperparameters, and the paper selects the reported ones by optimizing mIoU on the same validation benchmarks used in the final tables. In addition, baselines are taken from their original publications rather than run under Trident's selected protocols, so the headline margin is not a controlled comparison. These are fitted-evaluation and comparability problems, not a derivation that reduces to its own inputs; I therefore rate the circularity as partial and mild-to-moderate (4/10).

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are postulated; Trident composes existing frozen models and standard matrices. The main ledger entries are tuned hyperparameters and domain assumptions about feature alignment and semantic transfer between CLIP, DINO, and SAM.

free parameters (3)
  • mask prompt scaling coefficient alpha = 0.005
    Chosen from ablation on VOC20 and Cityscapes validation (Table 6) and applied to all benchmarks. This is a validation-set-tuned hyperparameter, not derived from first principles.
  • affinity threshold epsilon = not reported
    Threshold in Eq. 4 and Eq. 5 that prunes SAM correlation and attention weights. No value or sensitivity analysis is given, yet it controls the affinity matrix used for all reported results.
  • per-benchmark input resolution and stride = e.g., VOC20 336-336-112; VOC21/COCO 448-336-224; Context/ADE 576-336-224; Cityscapes 688-336-224
    Section 4.1 and Table 7 use different shorter-side resolutions and strides per dataset, and the choices appear to follow the best or near-best configurations from the ablation, making them effectively tuned per benchmark.
assumptions (5)
  • domain assumption CLIP text embeddings from a fixed template 'a photo of a {}' are an adequate class-conditioning feature space for open-vocabulary segmentation.
    Used throughout Eq. 1 and all experiments; inherited from the CLIP literature and not derived in this paper.
  • domain assumption SAM's encoder features and attention weights at high resolution encode semantic grouping that can be transferred to CLIP features through a linear aggregation matrix A.
    Core of the Splice-then-Segment paradigm in Eq. 3 through Eq. 5. The paper provides empirical ablations but no formal argument that SAM's correlation structure matches CLIP's semantic needs.
  • domain assumption Feature maps extracted from independent sliding-window crops can be spliced into a global feature map aligned with SAM's full-image feature grid.
    Eq. 3 splices local CLIP/DINO features and applies SAM's correlation matrix. The interpolation is explicitly omitted 'for brevity', and the impact of crop-local position embeddings is not analyzed.
  • domain assumption DINO provides object-level spatially covariant semantic correlation for sub-image CLIP features, following ProxyCLIP.
    Section 3.3 relies on ProxyCLIP's DINO-based correlation during sub-image processing; if DINO features are not spatially aligned with CLIP tokens, sub-image features degrade.
  • domain assumption Connected components of the predicted class mask correspond to meaningful SAM prompt regions, and the highest-confidence pixel in each component is a good point prompt.
    Section 3.4 and Eq. 6-7 define the prompt generation; this is validated only indirectly through the refinement ablation in Table 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation." pith.science (2026). https://pith.science/paper/JMVO3B7O

@misc{pith2026241109219,
  author       = {Pith},
  title        = {Pith review of: Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JMVO3B7O}},
  note         = {Machine review of arXiv:2411.09219}
}
read the original abstract

While Contrastive Language-Image Pre-training (CLIP) has advanced open-vocabulary predictions, its performance on semantic segmentation remains suboptimal. This shortfall primarily stems from its spatial-invariant semantic features and constrained resolution. While previous adaptations addressed spatial invariance semantic by modifying the self-attention in CLIP's image encoder, the issue of limited resolution remains unexplored. Different from previous segment-then-splice methods that segment sub-images via a sliding window and splice the results, we introduce a splice-then-segment paradigm that incorporates Segment-Anything Model (SAM) to tackle the resolution issue since SAM excels at extracting fine-grained semantic correlations from high-resolution images. Specifically, we introduce Trident, a training-free framework that first splices features extracted by CLIP and DINO from sub-images, then leverages SAM's encoder to create a correlation matrix for global aggregation, enabling a broadened receptive field for effective segmentation. Besides, we propose a refinement strategy for CLIP's coarse segmentation outputs by transforming them into prompts for SAM, further enhancing the segmentation performance. Trident achieves a significant improvement in the mIoU across eight benchmarks compared with the current SOTA, increasing from 44.4 to 48.6.Code is available at https://github.com/YuHengsss/Trident.

Figures

Figures reproduced from arXiv: 2411.09219 by the authors.

Figure 1
Figure 1. Comparison with previous SOTA performance of open [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the segmentation results of CLIP and ProxyCLIP. Figures (a) and (e) show the results of CLIP and ProxyCLIP [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Segmentation results using our Splice-then-Segment [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Framework of the proposed Trident model. Foundation models are first used to introduce correlations for sub-image’s features. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison with previous training-free open vocabulary segmentation methods. sub-image’s feature maps, to exclude the impact of other factors. Note SAM refinement is not included here. The de￾tailed ablation results for the effects of different paradigms an…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Training-free CorrCLIP reconstructs patch correlations in CLIP with SAM masks and DINO similarity, raising averaged mIoU across eight benchmarks from 48.6 to 53.6.

  2. Low-Frequency Stochastic Gravitational-Wave Background in Gaia DR3 catalog

    astro-ph.CO 2026-03 unverdicted novelty 5.0 of 10

    Gaia DR3 quasar proper-motion noise and sky coverage imply a detectable stochastic GW strain floor of order 10^{-11} below ~5.6 nHz, with VSH more robust than Hellings-Downs to uneven sampling.

Reference graph

Works this paper leans on

74 extracted references · 43 canonical work pages · cited by 2 Pith papers

  1. [1]

    Single-stage semantic segmentation from image labels

    Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In CVPR, 2020. 6, 8

    work page 2020

  2. [2]

    Segnet: A deep convolutional encoder-decoder architecture for image segmentation

    Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern anal- ysis and machine intelligence, 2017. 1

    work page 2017

  3. [3]

    BEiT: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 2

    arXiv 2021

  4. [4]

    Coco- stuff: Thing and stuff classes in context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco- stuff: Thing and stuff classes in context. In CVPR, 2018. 6

    work page 2018

  5. [5]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, 2021. 1, 2, 5

    work page 2021

  6. [6]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In CVPR, 2023. 2

    work page 2023

  7. [7]

    Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learn- ing to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In CVPR, 2023. 2, 6

    work page 2023

  8. [8]

    Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In CVPR,

Show all 74 references
  1. [9]

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs

    Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 2017. 1

    2017

  2. [10]

    Vision transformer adapter for dense predictions

    Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534, 2022. 2

    2022 arXiv

  3. [11]

    Per- pixel classification is not all you need for semantic segmen- tation

    Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per- pixel classification is not all you need for semantic segmen- tation. NeurIPS, 2021. 1

    2021

  4. [12]

    Reproducible scal- 9 ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- 9 ing laws for contrastive language-image learning. In CVPR,

  5. [13]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion. In CVPR, 2024. 1, 2

    2024

  6. [14]

    MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark

    MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https : / / github . com / open - mmlab/mmsegmentation, 2020. 6

    2020

  7. [15]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR,

  8. [16]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Pi- otr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023. 2

    2023 arXiv

  9. [17]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2020. 3

    2020

  10. [18]

    The pascal visual object classes challenge: A retrospective.IJCV,

    Mark Everingham, SM Ali Eslami, Luc Van Gool, Christo- pher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective.IJCV,

  11. [19]

    Two linear time union- find strategies for image processing

    Christophe Fiorio and Jens Gustedt. Two linear time union- find strategies for image processing. Theoretical Computer Science, 1996. 5

    1996

  12. [20]

    Bootstrap your own latent a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new ...

    2020

  13. [21]

    Calip: Zero-shot en- hancement of clip with parameter-free attention

    Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xu- peng Miao, Xuming He, and Bin Cui. Calip: Zero-shot en- hancement of clip with parameter-free attention. In AAAI,

  14. [22]

    Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation

    Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. In WACV, 2025. 2, 3, 6

    2025

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  16. [24]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 1

    2022

  17. [25]

    Ccnet: Criss-cross attention for semantic segmentation

    Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 1

    2019

  18. [26]

    Ttd: Text-tag self-distillation enhanc- ing image-text alignment in clip to alleviate single tag bias

    Sanghyun Jo, Soohyun Ryu, Sungyub Kim, Eunho Yang, and Kyungsu Kim. Ttd: Text-tag self-distillation enhanc- ing image-text alignment in clip to alleviate single tag bias. arXiv preprint arXiv:2404.00384, 2024. 2, 6

    2024 arXiv

  19. [27]

    In defense of lazy vi- sual grounding for open-vocabulary semantic segmentation

    Dahyun Kang and Minsu Cho. In defense of lazy vi- sual grounding for open-vocabulary semantic segmentation. arXiv preprint arXiv:2408.04961, 2024. 6

    2024 arXiv

  20. [28]

    Diffusion models for zero-shot open-vocabulary segmentation

    Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. arXiv preprint arXiv:2306.09316, 2023. 1, 2, 6

    2023 arXiv

  21. [29]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, 2023. 1, 2, 4, 5

    2023

  22. [30]

    Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation. In ECCV, 2024. 2, 3, 4, 6, 7

    2024

  23. [31]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML,

  24. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  25. [33]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In ECCV, 2022. 2

    2022

  26. [34]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks. arXiv preprint arXiv:2304.05653, 2023. 2, 3

    2023 arXiv

  27. [35]

    Open-vocabulary object segmentation with diffusion models

    Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Open-vocabulary object segmentation with diffusion models. In ICCV, 2023. 2

    2023

  28. [36]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, 2023. 1

    2023

  29. [37]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6

    2014

  30. [38]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023. 1, 2

    2023

  31. [39]

    Open-world semantic seg- mentation via contrasting and clustering vision-language embedding

    Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open-world semantic seg- mentation via contrasting and clustering vision-language embedding. In ECCV, 2022. 2

    2022

  32. [40]

    Matcher: Segment anything with one shot using all-purpose feature matching

    Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching. arXiv preprint arXiv:2305.13310, 2023. 2

    2023 arXiv

  33. [41]

    Open-vocabulary segmentation with semantic-assisted calibration

    Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary segmentation with semantic-assisted calibration. In CVPR, 2024. 2

    2024

  34. [42]

    Fully convolutional networks for semantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. 1 10

    2015

  35. [43]

    Image segmentation using text and image prompts

    Timo L ¨uddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR, 2022. 1, 2

    2022

  36. [44]

    Segclip: Patch aggregation with learn- able centers for open-vocabulary semantic segmentation

    Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segclip: Patch aggregation with learn- able centers for open-vocabulary semantic segmentation. In ICML, 2023. 2

    2023

  37. [45]

    Visual classification via description from large language models

    Sachit Menon and Carl V ondrick. Visual classification via description from large language models. In ICLR, 2023. 2

    2023

  38. [46]

    Clip- cap: Clip prefix for image captioning

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021. 2

    2021 arXiv

  39. [47]

    The role of context for object detection and se- mantic segmentation in the wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and se- mantic segmentation in the wild. In CVPR, 2014. 6

    2014

  40. [48]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2

    2023 arXiv

  41. [49]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1, 2, 3, 6

    2021

  42. [50]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML. PMLR, 2021. 6

    2021

  43. [51]

    Denseclip: Language-guided dense prediction with context- aware prompting

    Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context- aware prompting. In CVPR, 2022. 1, 2

    2022

  44. [52]

    Imagenet-21k pretraining for the masses

    Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972, 2021. 6

    2021 arXiv

  45. [53]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI. Springer, 2015. 1

    2015

  46. [54]

    Ex- plore the potential of clip for training-free open vocabulary semantic segmentation

    Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Ex- plore the potential of clip for training-free open vocabulary semantic segmentation. arXiv preprint arXiv:2407.08268 ,

    arXiv

  47. [55]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In ACL,

  48. [56]

    Reco: Re- trieve and co-segment for zero-shot transfer

    Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: Re- trieve and co-segment for zero-shot transfer. NeurIPS, 2022. 2, 6

    2022

  49. [57]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 1, 2

    2023 arXiv

  50. [58]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 2

    2017

  51. [59]

    Sclip: Rethinking self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In ECCV,

  52. [60]

    Sam-clip: Merging vision foundation models towards semantic and spatial understanding

    Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. In CVPR, 2024. 1, 2, 4, 6

    2024

  53. [61]

    Cut and learn for unsupervised object detection and instance segmentation

    Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In ICCV, 2023. 2

    2023

  54. [62]

    Image-text co- decomposition for text-supervised semantic segmentation

    Ji-Jia Wu, Andy Chia-Hao Chang, Chieh-Yu Chuang, Chun-Pei Chen, Yu-Lun Liu, Min-Hung Chen, Hou-Ning Hu, Yung-Yu Chuang, and Yen-Yu Lin. Image-text co- decomposition for text-supervised semantic segmentation. In CVPR, 2024. 6

    2024

  55. [63]

    Optimizing connected component labeling algorithms

    Kesheng Wu, Ekow Otoo, and Arie Shoshani. Optimizing connected component labeling algorithms. In Medical Imag- ing 2005: Image Processing, 2005. 5

    2005

  56. [64]

    Clipself: Vision trans- former distills itself for open-vocabulary dense prediction

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipself: Vision trans- former distills itself for open-vocabulary dense prediction. arXiv preprint arXiv:2310.01403, 2023. 2, 4

    2023 arXiv

  57. [65]

    Clip-dinoiser: Teaching clip a few dino tricks.arXiv preprint arXiv:2312.12359, 2023

    Monika Wysocza ´nska, Oriane Sim´eoni, Micha¨el Ramamon- jisoa, Andrei Bursuc, Tomasz Trzci ´nski, and Patrick P ´erez. Clip-dinoiser: Teaching clip a few dino tricks.arXiv preprint arXiv:2312.12359, 2023. 1, 2, 6

    2023 arXiv

  58. [66]

    Segformer: Simple and ef- ficient design for semantic segmentation with transformers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and ef- ficient design for semantic segmentation with transformers. NeurIPS, pages 12077–12090, 2021. 1

    2021

  59. [67]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, 2022. 2

    2022

  60. [69]

    Learning open-vocabulary semantic segmentation models from natural language supervision

    Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Yi Wang, Yu Qiao, and Weidi Xie. Learning open-vocabulary semantic segmentation models from natural language supervision. In CVPR, 2023. 2

    2023

  61. [70]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR, 2023. 1, 2

    2023

  62. [71]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 2

    2022 arXiv

  63. [72]

    Open-vocabulary detr with conditional matching

    Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In ECCV, 2022. 2

    2022

  64. [73]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. IJCV, 2019. 6 11

    2019

  65. [74]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022. 1, 2, 3, 6

    2022

  66. [75]

    Zegclip: Towards adapting clip for zero-shot se- mantic segmentation

    Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot se- mantic segmentation. In CVPR, 2023. 1, 2 12

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.