Pith. sign in

REVIEW 4 major objections 4 minor 68 references

Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A training-free method called CASS distills the low-rank spectral structure of a vision foundation model's attention graph into CLIP's attention and refines text embeddings with an object-presence prior, raising average mIoU by 3.0 points…

desk verdict A genuinely new training-free OVSS method with solid ablations and SOTA numbers, but the core equation is underspecified: it adds attention graphs that live on different token grids (DINO B/8 vs CLIP B/16), and no alignment is described. read the letter →

arxiv 2411.17150 v3 pith:MTQT6YY3 submitted 2024-11-26 cs.CV

Chanyoung Kim , Dayun Ju , Woojung Han , Ming-Hsuan Yang , Seong Jae Hwang This is my paper · ORCID
classification cs.CV
keywords open-vocabularysemanticsegmentationtraining-freespectralgraphtheoryattentiondistillationCLIPDINOobject-levelcontextmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that training-free open-vocabulary semantic segmentation fails mainly because CLIP lacks object-level context: it sees patches but does not group parts of the same object into one entity. It proposes CASS, which pulls the object-grouping structure out of a vision foundation model's attention graph, keeping only its low-rank, energy-dominant components, and injects that structure into CLIP's attention during inference. It also uses CLIP's own zero-shot object-presence score to reshape text embeddings toward objects actually in the image. The paper reports state-of-the-art training-free results, averaging 3.0 mIoU points above the previous best across eight segmentation datasets.

What carries the argument

The central object is the VFM attention graph $A_{\mathrm{VFM}} = K_{\mathrm{VFM}}K_{\mathrm{VFM}}^{\top}$, treated as an adjacency matrix whose low-rank eigendecomposition is supposed to carry object-level grouping structure. The argument runs through complementary spectral graph matching: eigenvalues of each attention head of $A_{\mathrm{VFM}}$ and of CLIP's $A_{\mathrm{CLIP}}$ are normalized, compared by Wasserstein distance, and paired by Hungarian matching so that structurally contrasting heads are aggregated. The distilled head is $A^{j}_{\psi} = (w_{ij}\ddot{A}^{i}_{\mathrm{VFM}} + A^{j}_{\mathrm{CLIP}})/(w_{ij}+1)$, where $\ddot{A}^{i}_{\mathrm{VFM}}$ comes from an energy-based low-rank approximation with dynamic eigenscaling and $w_{ij}$ is the spectral distance. A second mechanism computes an object-presence prior $P(i) = t^{i}_{\mathrm{CLIP}} \cdot v_{\mathrm{CLIP}}$ from the CLIP [CLS] token, then uses it to cluster and shift text embeddings toward image-specific semantics and to blend patch-text similarities.

What would settle it

Run the full CASS pipeline on PASCAL VOC V21 with the complementary matching replaced by random head pairing or by maximum-similarity pairing, keeping low-rank eigenscaling and the object-presence modules unchanged; if the 65.8 mIoU result persists, complementary spectral matching is not the driver.

Watch

Extended reading notes

Core claim

The central claim is that object-level context can be transferred between two frozen models at inference time by treating attention maps as graphs and distilling their spectral backbone. Concretely, after matching VFM and CLIP attention heads by Wasserstein distance between their eigenvalue distributions, the method applies energy-based low-rank approximation and dynamic eigenscaling to the VFM graph, then aggregates it into CLIP's attention with a weight proportional to spectral distance. The resulting attention groups wheels, cargo, and body of a truck into one mask. Text embeddings are then nudged toward the average of the image patches most similar to the presence-likely class, and patch-text similarities are blended with the global object-presence prior. With only frozen CLIP ViT-B/16 and DINO ViT-B/8, the paper reports 65.8 mIoU on PASCAL VOC 21 and an average 44.4 mIoU and 68.9 pAcc across eight datasets.

Load-bearing premise

The matching step assumes that two attention heads whose eigenvalue patterns are most different are the ones that best complete each other's object-level picture; if that pairing is arbitrary, the distilled attention could be injecting noise and the reported gains would not come from the proposed mechanism.

Editorial extensions

If this is right

  • CASS implies that training-free open-vocabulary segmentation can be improved by reweighting attention with spectral structure, without new training data or fine-tuning.
  • Object-level context distilled from the VFM should make segmentation maps group parts of the same object into one class, which is visible in the 5.3 mIoU gain over the second-best model on PASCAL VOC 20.
  • The object-presence prior refines arbitrary text prompts toward objects actually present, so user-defined classes such as 'space needle' align with the right image regions.
  • Because the method is training-free and backbone-agnostic, the same distillation recipe transfers to CLIP ViT-B/32 and ViT-L/14, with average mIoU rising on both.
  • Cleaner object masks should improve downstream applications such as image inpainting and object removal, since the whole object is masked rather than only fragments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same spectral graph-matching recipe could be applied to other dense prediction tasks where a frozen CLIP-like encoder lacks object-level context, such as referring segmentation or video object segmentation, provided the head-pairing assumption holds.
  • A testable extension would be to learn the head pairing from a few labeled images rather than relying on Wasserstein distance; if learned pairings beat spectral pairings, the complementarity criterion is not the full story.
  • The reported 5.6 FPS suggests that faster low-rank or eigensolver approximations could determine whether the object-context gain is worth the cost in real-time settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes CASS, a training-free open-vocabulary semantic segmentation method that injects object-level context into CLIP. It has two components: (1) spectral distillation of low-rank VFM attention graphs into CLIP attention, using a complementary head-matching scheme based on Wasserstein distance between eigenvalue spectra, and (2) object-presence-driven text embedding adjustment and patch-text similarity refinement. The paper reports an average mIoU of 44.4 across eight datasets, outperforming the previous state of the art by 3.0 mIoU, and provides extensive ablations.

Significance. If the method is correct and reproducible, this is a solid contribution to training-free OVSS: it achieves strong results without additional training data, directly addresses object-level coherence (a known CLIP weakness), and includes thorough component analyses and honest discussion of computational limitations. The paper's central novelty—distilling low-rank spectral structure from VFM attention into CLIP—is conceptually interesting. However, the core equation defining the distillation is underspecified for the actual backbone pair, and several hyperparameters are tuned per dataset, so the reported results cannot currently be verified from the manuscript.

major comments (4)
  1. [Section 3.2.2, Eq. (7)] The aggregation A_psi^j = (w_ij * A_VFM_tilde^i + A_CLIP^j) / (w_ij + 1) requires A_VFM and A_CLIP to have the same spatial dimensions. With Sec. 4.1's backbone pair (DINO ViT-B/8, CLIP ViT-B/16) and a 224x224 input window, A_VFM per head is 784x784 while A_CLIP per head is 196x196. The paper never describes a resizing, interpolation, or positional alignment step that makes the addition well-defined. This is not a minor detail: the distilled attention matrix A_psi is the mechanism by which VFM context is transferred, and the choice of alignment (e.g., bilinear upsampling of CLIP attention vs. patch-correspondence pooling of DINO attention) changes which VFM structure is injected. The main quantitative results in Table 1 therefore cannot be reproduced from the text as it stands. Please specify the alignment, justify it, and confirm that the results are robust to reasonable alignment choices.
  2. [Section 4.3 / Table 11 (supplementary)] The paper claims 'strong generalizability across diverse datasets,' but the main experiments use different alpha and gamma for every dataset (Table 11). The selection procedure (validation split vs. test-set tuning) is not reported. Given that the individual component gains in Table 3 are often around 0.1-1.0 mIoU, per-dataset test-set tuning could materially change the reported 3.0 mIoU average improvement. Report how these hyperparameters were chosen and provide results with fixed values across all datasets, or with a principled validation strategy.
  3. [Sections 3.2.1, 3.2.2, and supplementary] Several parameters that determine the method's behavior are not specified: the top-n number of object-specific vectors in Eq. (8), the energy threshold eta and initial rank q0/step delta_q in Algorithm 1, the hierarchical clustering threshold h_threshold, and the number m of eigenvalues used in Eq. (5). Without these values, the method cannot be reimplemented, and some of them (e.g., n, eta) directly control how much object-level context is injected. Please report all hyperparameter values in the main text or supplementary.
  4. [Section 3.3, Eq. (8)] The object-guided text embedding adjustment selects the top-n patch vectors by cosine similarity to the original text embedding, then moves that text embedding toward their mean mu_n. This is self-referential: the most similar patches may simply be the ones that already match the text, so the refinement can reinforce the original bias rather than adding object-level information. A concrete control is to compare this selection against one based on the object-presence prior P (or against iterative refinement), and to show how results vary with n. Without such a control, the gains attributed to OTA in Table 3 are not clearly attributable to the proposed mechanism.
minor comments (4)
  1. [Section 3.2.1, Eq. (5) and following text] The Wasserstein distance D_W is written as a sum over i=1..h (the number of heads), but it should sum over the m selected eigenvalues; the notation is inconsistent.
  2. [Table 1] The meaning of the 'Fair' column is not clear from the caption alone; please clarify which methods are directly comparable to CASS and how the checks were assigned.
  3. [Figure 3] The symbol 'A_psi w/o LR' is used in the caption but not defined in the main text; please define all abbreviations in the caption or text.
  4. [Section 11.1] The limitations on computational efficiency and small-object performance are honestly reported, but they should also be mentioned in the abstract or introduction to avoid overstating the 'strong generalizability' claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the spectral distillation and text-refinement modules are inference-time heuristics validated on external benchmarks; minor self-citations are not load-bearing, though Eq. (7) has an unstated spatial-alignment gap.

full rationale

The paper's derivation chain is not circular in a load-bearing way. The two proposed mechanisms are inference-time heuristics whose contributions are established by ablation against external ground-truth labels (Table 3), not by construction. The Object-Guided Text Embedding Adjustment (Eq. (8)) does select the top-n patch vectors by cosine similarity to the original text embedding and then recomputes patch-text similarity with the adjusted embedding in Eq. (4); however, the final logit for the adjusted class is the convex combination (1-alpha)*F_CLIP*t + alpha*F_CLIP*mu_n, so the original text term is retained and the added term is an image self-similarity prior. This is self-referential in an algorithmic sense but not a circular derivation: no quantity claimed as a prediction is identical to a fitted input, and the module's value is measured by whether it improves mIoU on held-out datasets. Similarly, the complementary spectral graph matching (Eqs. (5)-(7)) pairs heads by Wasserstein distance and weights the aggregation by that same distance; this is definitional, but the effectiveness of the strategy is tested empirically (Table 8, Fig. 8) rather than assumed. The paper contains self-citations ([23,24,25]) but none is load-bearing: the spectral-to-object-level premise is also supported by the independent Deep Spectral Methods paper [37], and the main evidence is external benchmarks. One non-circular verifiability concern should be flagged per the reviewing rule: Eq. (7) adds A_VFM and A_CLIP both written as R^{N x N}, while Sec. 4.1 uses DINO ViT-B/8 (28x28 patch grid) and CLIP ViT-B/16 (14x14 patch grid), and no spatial alignment or resizing is specified; this makes the central equation underspecified as written, but it is a correctness or implementation gap, not a case of the result reducing to its input. Overall, the central claim has independent content, so the circularity score is low.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about DINO's attention, CLIP's zero-shot scores, and complementary head matching. Free parameters include per-dataset alpha and gamma plus several unstated thresholds, so the reported gains are partly a product of tuning rather than a purely parameter-free derivation.

free parameters (6)
  • alpha (per dataset) = 0.01 to 0.05 (Table 11)
    Balances original text embedding against object-specific vector in Eq. (8); tuned separately for every dataset, so reported gains are after dataset-specific tuning.
  • gamma (per dataset) = 0.10 to 0.40 (Table 11)
    Blends patch-text similarity with the object presence prior in Eq. (9); tuned per dataset.
  • epsilon (dynamic eigenscaling) = 1.5
    Fixed constant in Eq. (13) controlling the scaling range of eigenvalues; chosen by hand, no sensitivity analysis.
  • energy threshold eta (low-rank approximation) = unspecified
    Threshold for selecting the rank k in Algorithm 1; not reported, so the low-rank approximation cannot be exactly reproduced.
  • top-n object-specific vectors = unspecified
    Number of most similar patch features averaged into mu_n in Eq. (8); not reported.
  • hierarchical clustering threshold = unspecified
    Threshold for grouping text embeddings in Algorithm 2; not reported.
assumptions (4)
  • domain assumption DINO attention keys encode object-level semantic structure that can be transferred to CLIP via low-rank eigendecomposition.
    Central premise of Sec. 3.2; if the VFM graph does not contain object-level structure, the distilled attention A_psi has no object-level signal.
  • domain assumption Wasserstein distance between eigenvalue spectra identifies complementary attention heads.
    Sec. 3.2.1 Eq. (5); no evidence is given that contrasting eigenvalue distributions imply complementary rather than irrelevant attention patterns.
  • domain assumption CLIP's zero-shot classification score on the full image is a reliable object presence prior.
    Sec. 3.3; if CLIP misclassifies the whole image, the text adjustment in Eq. (8) and the similarity refinement in Eq. (9) are biased toward wrong classes.
  • standard math Eigendecomposition, Hungarian matching, and Wasserstein distance are applied correctly as background mathematical tools.
    These are standard algorithms invoked in Sec. 3.2 and are assumed to be implemented correctly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation." pith.science (2026). https://pith.science/paper/MTQT6YY3

@misc{pith2026241117150,
  author       = {Pith},
  title        = {Pith review of: Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MTQT6YY3}},
  note         = {Machine review of arXiv:2411.17150}
}
read the original abstract

Open-Vocabulary Semantic Segmentation (OVSS) has advanced with recent vision-language models (VLMs), enabling segmentation beyond predefined categories through various learning schemes. Notably, training-free methods offer scalable, easily deployable solutions for handling unseen data, a key goal of OVSS. Yet, a critical issue persists: lack of object-level context consideration when segmenting complex objects in the challenging environment of OVSS based on arbitrary query prompts. This oversight limits models' ability to group semantically consistent elements within object and map them precisely to user-defined arbitrary classes. In this work, we introduce a novel approach that overcomes this limitation by incorporating object-level contextual knowledge within images. Specifically, our model enhances intra-object consistency by distilling spectral-driven features from vision foundation models into the attention mechanism of the visual encoder, enabling semantically coherent components to form a single object mask. Additionally, we refine the text embeddings with zero-shot object presence likelihood to ensure accurate alignment with the specific objects represented in the images. By leveraging object-level contextual knowledge, our proposed approach achieves state-of-the-art performance with strong generalizability across diverse datasets.

Figures

Figures reproduced from arXiv: 2411.17150 by the authors.

Figure 1
Figure 1. We present CASS, object-level Context-Aware training-free open-vocabulary Semantic Segmentation model. (a) Overview: Our method distills the vision foundation model’s (VFM) object-level contextual spectral graph into CLIP’s attention and refines query text embeddings towards object-specific seman￾tics. (b) Object-Level Context: This result illustrates how incor￾porating object-level context improves segmentation acc… view at source ↗
Figure 2
Figure 2. Detailed illustration of our proposed training-free spectral object-level context distillation mechanism in Sec. 3.2. By matching the attention graphs of VFM and CLIP head-by-head to establish complementary relationships, and distilling the fundamental object-level context of the VFM graph to CLIP, we enhance CLIP’s ability to capture intra-object contextual coherence. feature and attention levels. Among various dow… view at source ↗
Figure 3
Figure 3. Attention score visualization for various query points. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Visualization of the matched complementary graph [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison across the Pascal VOC [ [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Performance evaluation of different distance metrics [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Visualization of image inpainting and object removal using our predicted mask. For image inpainting, we use “ [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Visualization of limitation of CASS. spectral-based graph matching, our approach narrows the gap between pixel-level predictions and object-level under￾standing, contributing a significant advancement towards the primary objectives of OVSS. However, CASS has limitatio…
Figure 11
Figure 11. Figure 11: Additional qualitative comparison between recent state-of-the-art methods SCLIP [ [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Additional qualitative comparison between recent state-of-the-art methods SCLIP [ [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Additional quantitative comparison between recent state-of-the-art methods SCLIP [ [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Additional qualitative comparison between recent state-of-the-art methods SCLIP [ [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Real-world open-vocabulary semantic segmentation results of our model after applying mask refinement [ [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 59 canonical work pages

  1. [1]

    Single-Stage Semantic Segmentation from Image Labels

    Nikita Araslanov and Stefan Roth. Single-Stage Semantic Segmentation from Image Labels. In CVPR, 2020. 2, 11

    work page 2020

  2. [2]

    Single-stage semantic segmentation from image labels

    Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In CVPR, 2020. 7

    work page 2020

  3. [3]

    Optimal shrinkage of sin- gular values under random data contamination

    Danny Barash and Matan Gavish. Optimal shrinkage of sin- gular values under random data contamination. NeurIPS,

    work page

  4. [4]

    Fossil: Free open-vocabulary semantic seg- mentation through synthetic references retrieval

    Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, and Rita Cucchiara. Fossil: Free open-vocabulary semantic seg- mentation through synthetic references retrieval. In WACV,

    work page

  5. [5]

    Training-free open- vocabulary segmentation with offline diffusion-augmented prototype generation

    Luca Barsellotti, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Training-free open- vocabulary segmentation with offline diffusion-augmented prototype generation. In CVPR, 2024. 2, 6

    work page 2024

  6. [6]

    Shrinkage Function And Its Applications In Matrix Approximation

    Toby Boas, Aritra Dutta, Xin Li, Kathryn P Mercier, and Eric Niderman. Shrinkage function and its applications in matrix approximation. arXiv preprint arXiv:1601.07600, 2016. 5

    work page Pith review arXiv 2016

  7. [7]

    Grounding everything: Emerging localiza- tion properties in vision-language transformers

    Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localiza- tion properties in vision-language transformers. In CVPR,

    work page

  8. [8]

    COCO- Stuff: Thing and Stuff Classes in Context

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO- Stuff: Thing and Stuff Classes in Context. In CVPR, 2018. 6, 7, 8, 2

    work page 2018

Show all 68 references
  1. [9]

    Emerg- ing Properties in Self-Supervised Vision Transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing Properties in Self-Supervised Vision Transformers. In ICCV, 2021. 2, 4, 6

    2021

  2. [10]

    Learning To Generate Text-Grounded Mask for Open-World Semantic Segmentation From Only Image-Text Pairs

    Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning To Generate Text-Grounded Mask for Open-World Semantic Segmentation From Only Image-Text Pairs. In CVPR, 2023. 1, 6

    2023

  3. [11]

    Encoder-decoder with atrous separable convolution for semantic image segmentation

    Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 1, 3

    2018

  4. [12]

    Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion

    Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmenta- tion. In CVPR, 2024. 1

    2024

  5. [13]

    Spectral graph theory

    Fan RK Chung. Spectral graph theory . American Mathe- matical Soc., 1997. 5

    1997

  6. [14]

    The Cityscapes Dataset for Semantic Urban Scene Understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Understanding. In CVPR,

  7. [15]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition ...

    2021

  8. [16]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Chal- lenge 2012 (VOC2012) Results, 2012. 6, 7, 8, 2

    2012

  9. [17]

    Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation

    Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. In WACV, 2025. 1, 2, 3, 6, 7, 8, 4, 9, 10

    2025

  10. [18]

    Unsupervised semantic segmentation by distilling feature correspondences

    Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.08414, 2022. 3

    2022 arXiv

  11. [19]

    Complementary branch fusing class and semantic knowledge for robust weakly supervised semantic segmen- tation

    Woojung Han, Seil Kang, Kyobin Choo, and Seong Jae Hwang. Complementary branch fusing class and semantic knowledge for robust weakly supervised semantic segmen- tation. Pattern Recognition, 2025. 1

    2025

  12. [20]

    Unetr: Transformers for 3d med- ical image segmentation

    Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d med- ical image segmentation. In CVPR, 2022. 1, 3

    2022

  13. [21]

    Clip- count: Towards text-guided zero-shot object counting

    Ruixiang Jiang, Lingbo Liu, and Changwen Chen. Clip- count: Towards text-guided zero-shot object counting. In ACM Multimedia, 2023. 2, 5

    2023

  14. [22]

    In defense of lazy visual grounding for open-vocabulary semantic segmentation

    Dahyun Kang and Minsu Cho. In defense of lazy visual grounding for open-vocabulary semantic segmentation. In ECCV, 2024. 2, 6, 7, 8, 3, 4

    2024

  15. [23]

    See what you are told: Visual attention sink in large multimodal models

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. In ICLR, 2025. 4

    2025

  16. [24]

    Your large vision-language model only needs a few attention heads for visual grounding

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. Your large vision-language model only needs a few attention heads for visual grounding. In CVPR, 2025. 2

    2025

  17. [25]

    Eagle: Eigen aggregation learning for object-centric unsupervised semantic segmentation

    Chanyoung Kim, Woojung Han, Dayun Ju, and Seong Jae Hwang. Eagle: Eigen aggregation learning for object-centric unsupervised semantic segmentation. In CVPR, 2024. 2, 3

    2024

  18. [26]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, 2023. 3

    2023

  19. [27]

    Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

    Philipp Kr ¨ahenb¨uhl and Vladlen Koltun. Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. In NeurIPS, 2011. 7, 2

    2011

  20. [28]

    The hungarian method for the assignment problem

    Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955. 4

    1955

  21. [29]

    Clearclip: Decom- posing clip representations for dense vision-language infer- ence

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decom- posing clip representations for dense vision-language infer- ence. In ECCV, 2024. 1, 6

    2024

  22. [30]

    Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy at- tention improves clip for open-vocabulary segmentation. In ECCV, 2024. 1, 2, 3, 6, 7, 8, 5, 9, 10

    2024

  23. [31]

    Clip surgery for better explainability with enhancement in open- vocabulary tasks

    Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li. Clip surgery for better explainability with enhancement in open- vocabulary tasks. arXiv preprint arXiv:2304.05653 , 2023. 2

    2023 arXiv

  24. [32]

    How does attention work in vision transformers? a vi- sual analytics attempt

    Yiran Li, Junpeng Wang, Xin Dai, Liang Wang, Chin- Chia Michael Yeh, Yan Zheng, Wei Zhang, and Kwan-Liu 9 Ma. How does attention work in vision transformers? a vi- sual analytics attempt. IEEE Transactions on Visualization and Computer Graphics, 2023. 4

    2023

  25. [33]

    Open-vocabulary semantic segmentation with mask-adapted clip

    Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, 2023. 1

    2023

  26. [34]

    Emergent open-vocabulary semantic segmen- tation from off-the-shelf vision-language models

    Jiayun Luo, Siddhesh Khandelwal, Leonid Sigal, and Boyang Li. Emergent open-vocabulary semantic segmen- tation from off-the-shelf vision-language models. In CVPR,

  27. [35]

    Clip4hoi: towards adapting clip for prac- tical zero-shot hoi detection

    Yunyao Mao, Jiajun Deng, Wengang Zhou, Li Li, Yao Fang, and Houqiang Li. Clip4hoi: towards adapting clip for prac- tical zero-shot hoi detection. NeurIPS, 2023. 2, 5

    2023

  28. [36]

    Im- proving semantic correspondence with viewpoint-guided spherical maps

    Octave Mariotti, Oisin Mac Aodha, and Hakan Bilen. Im- proving semantic correspondence with viewpoint-guided spherical maps. In CVPR, 2024. 3

    2024

  29. [37]

    Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization

    Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022. 2, 3

    2022

  30. [38]

    The Role of Context for Object Detection and Semantic Segmentation in the Wild

    Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The Role of Context for Object Detection and Semantic Segmentation in the Wild. In CVPR, 2014. 6, 7, 8, 2, 9

    2014

  31. [39]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 2

    2023 arXiv

  32. [40]

    Learning Transferable Visual Models From Natural Language Super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Super- vision. In ICML, 2021. 1, 2, 3, 5, 6, 7

    2021

  33. [41]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 1, 3

    2015

  34. [42]

    Bridging the gap to real-world object-centric learning

    Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Do- minik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Sch¨olkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In ICLR, 2023. 3

    2023

  35. [43]

    Leveraging hidden positives for unsupervised semantic segmentation

    Hyun Seok Seong, WonJun Moon, SuBeen Lee, and Jae-Pil Heo. Leveraging hidden positives for unsupervised semantic segmentation. In CVPR, 2023. 3

    2023

  36. [44]

    Open-vocabulary semantic segmentation with image embedding balancing

    Xiangheng Shan, Dongyue Wu, Guilin Zhu, Yuanjie Shao, Nong Sang, and Changxin Gao. Open-vocabulary semantic segmentation with image embedding balancing. In CVPR,

  37. [45]

    Ex- plore the potential of clip for training-free open vocabulary semantic segmentation

    Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Ex- plore the potential of clip for training-free open vocabulary semantic segmentation. In ECCV, 2024. 3, 6, 7

    2024

  38. [46]

    ReCo: Re- trieve and Co-segment for Zero-shot Transfer

    Gyungin Shin, Weidi Xie, and Samuel Albanie. ReCo: Re- trieve and Co-segment for Zero-shot Transfer. In NeurIPS,

  39. [47]

    Localizing objects with self-supervised transformers and no labels

    Oriane Sim ´eoni, Gilles Puy, Huy V V o, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick P ´erez, Renaud Mar- let, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC, 2021. 3

    2021

  40. [48]

    Clip as rnn: Segment countless visual concepts without training endeavor

    Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, and Siyang Li. Clip as rnn: Segment countless visual concepts without training endeavor. In CVPR, 2024. 2, 6

    2024

  41. [49]

    Resolution-robust large mask inpainting with fourier convolutions

    Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In WACV, 2022. 4

    2022

  42. [50]

    Sclip: Rethinking self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In ECCV,

  43. [51]

    Cut and Learn for Unsupervised Object Detection and In- stance Segmentation

    Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and Learn for Unsupervised Object Detection and In- stance Segmentation. In CVPR, 2023. 3

    2023

  44. [52]

    Self-supervised visual representation learning with semantic grouping

    Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. NeurIPS, 2022. 3

    2022

  45. [53]

    Image-text co- decomposition for text-supervised semantic segmentation

    Ji-Jia Wu, Andy Chia-Hao Chang, Chieh-Yu Chuang, Chun-Pei Chen, Yu-Lun Liu, Min-Hung Chen, Hou-Ning Hu, Yung-Yu Chuang, and Yen-Yu Lin. Image-text co- decomposition for text-supervised semantic segmentation. In CVPR, 2024. 6

    2024

  46. [54]

    Clip-dinoiser: Teaching clip a few dino tricks.arXiv preprint arXiv:2312.12359, 2023

    Monika Wysocza ´nska, Oriane Sim´eoni, Micha¨el Ramamon- jisoa, Andrei Bursuc, Tomasz Trzci ´nski, and Patrick P ´erez. Clip-dinoiser: Teaching clip a few dino tricks.arXiv preprint arXiv:2312.12359, 2023. 6, 7

    2023 arXiv

  47. [55]

    Sed: A simple encoder-decoder for open- vocabulary semantic segmentation

    Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. Sed: A simple encoder-decoder for open- vocabulary semantic segmentation. In CVPR, 2024. 1

    2024

  48. [56]

    GroupViT: Semantic Segmentation Emerges From Text Supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. GroupViT: Semantic Segmentation Emerges From Text Supervision. In CVPR, 2022. 1, 6

    2022

  49. [57]

    Open-vocabulary panop- tic segmentation with text-to-image diffusion models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiao- long Wang, and Shalini De Mello. Open-vocabulary panop- tic segmentation with text-to-image diffusion models. In CVPR, 2023. 1

    2023

  50. [58]

    Side adapter network for open-vocabulary semantic segmentation

    Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xi- ang Bai. Side adapter network for open-vocabulary semantic segmentation. In CVPR, 2023. 1

    2023

  51. [59]

    Semantic Un- derstanding of Scenes Through the ADE20K Dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic Un- derstanding of Scenes Through the ADE20K Dataset. IJCV,

  52. [60]

    Extract Free Dense Labels from CLIP

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract Free Dense Labels from CLIP. In ECCV, 2022. 1, 2, 6

    2022

  53. [61]

    Weaktr: Exploring plain vision transformer for weakly-supervised semantic segmen- tation

    Lianghui Zhu, Yingyue Li, Jiemin Fang, Yan Liu, Hao Xin, Wenyu Liu, and Xinggang Wang. Weaktr: Exploring plain vision transformer for weakly-supervised semantic segmen- tation. arXiv preprint arXiv:2304.01184, 2023. 4 10 Distilling Spectral Graph for Object-Context Aware Open-...

    2023

  54. [63]

    The link to the project page is as follows: https://micv-yonsei.github.io/cass/

    Additional Material: Project Page & Pre- sentation Video We have described our results in an easily accessible man- ner on our project page, where a brief presentation video is also available. The link to the project page is as follows: https://micv-yonsei.github.io/cass/

  55. [64]

    Detailed Method 7.1. Energy-based Low-rank Approximation Algorithm 1 Optimal Rank k Selection with Low-Rank Eigendecomposition Input: Adjacency matrix A ∈ Rn×n, energy threshold η, initial rank q0, step size ∆q Output: Optimal rank k, eigenvectors U, eigenvalues Σ Set qmax ← n...

  56. [65]

    Sequence

    Additional Experiments 8.1. Additional Evaluation Results In this section, we provide additional evaluation results including qualitative evaluation (8.1.1), real-world open- vocabulary semantic segmentation result (10.2), and scale- up version of CASS (8.1.2). 8.1.1. Addition...

  57. [66]

    a photo of {prompt}

    Implementation Details 9.1. Prompt Templates Following recent works [17, 22, 50], we employ a compre- hensive set of prompt templates to enhance the diversity of text embeddings (e.g., “a photo of {prompt}”). Specifi- Table 10. Examples of the prompt templates used for text em...

  58. [67]

    Toyota car

    Application 10.1. Image Inpainting and Object Removal While our CASS can naturally be used for semantic segmen- tation directly based on user-provided prompts in real-world scenarios (see Sec. 10.2 for diverse examples), its ability to be object-level context-aware allows it t...

  59. [68]

    Limitations Our CASS introduces object-level contextual knowledge into training-free OVSS, enabling the effective grouping of object components into coherent semantic entities

    Discussion 11.1. Limitations Our CASS introduces object-level contextual knowledge into training-free OVSS, enabling the effective grouping of object components into coherent semantic entities. By combining Energy-based Low-rank Approximation with ProxyCLIP Ours Ground Truth F...

  60. [2024]

    1, 2, 3, 6, 7, 8, 4, 9, 10

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.