REVIEW 6 cited by
ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference
read the original abstract
Despite the success of large-scale pretrained Vision-Language Models (VLMs) especially CLIP in various open-vocabulary tasks, their application to semantic segmentation remains challenging, producing noisy segmentation maps with mis-segmented regions. In this paper, we carefully re-investigate the architecture of CLIP, and identify residual connections as the primary source of noise that degrades segmentation quality. With a comparative analysis of statistical properties in the residual connection and the attention output across different pretrained models, we discover that CLIP's image-text contrastive training paradigm emphasizes global features at the expense of local discriminability, leading to noisy segmentation results. In response, we propose ClearCLIP, a novel approach that decomposes CLIP's representations to enhance open-vocabulary semantic segmentation. We introduce three simple modifications to the final layer: removing the residual connection, implementing the self-self attention, and discarding the feed-forward network. ClearCLIP consistently generates clearer and more accurate segmentation maps and outperforms existing approaches across multiple benchmarks, affirming the significance of our discoveries.
Forward citations
Cited by 6 Pith papers
-
Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation
A feedback self-adaptive attention module uses CLIP's own output predictions as a spatial coherence prior to reweight intermediate attention, improving training-free open-vocabulary segmentation across 8 benchmarks.
-
Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images
SegEarth-OV performs annotation-free open-vocabulary segmentation of remote-sensing images by upsampling CLIP features, removing global bias, and distilling optical knowledge into a SAR encoder.
-
The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model
CLIP embeddings are modeled as a mixture of von Mises-Fisher distributions on the unit sphere, improving out-of-distribution detection and semantic decomposition over single-Gaussian baselines.
-
SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images
SAM 3 can be applied training-free to remote sensing open-vocabulary segmentation and change detection by fusing its semantic and instance heads and filtering with presence scores.
-
ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation
ReCLIP++ rectifies class and space biases in CLIP via separate reference and positional features, logit subtraction, and a mask decoder with contrastive loss to improve unsupervised semantic segmentation on PASCAL VOC...
-
Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation
GCLIP improves TF-OVSS by reshaping last-block attention via fusion of global-token block attention with Query-Query attention and applying channel suppression to Value embeddings, outperforming prior methods on five ...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.