REVIEW 12 cited by
A Closer Look at the Explainability of Contrastive Language-Image Pre-training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Contrastive language-image pre-training (CLIP) is a powerful vision-language model that has shown great benefits for various tasks. However, we have identified some issues with its explainability, which undermine its credibility and limit the capacity for related tasks. Specifically, we find that CLIP tends to focus on background regions rather than foregrounds, with noisy activations at irrelevant positions on the visualization results. These phenomena conflict with conventional explainability methods based on the class attention map (CAM), where the raw model can highlight the local foreground regions using global supervision without alignment. To address these problems, we take a closer look at its architecture and features. Based on thorough analyses, we find the raw self-attentions link to inconsistent semantic regions, resulting in the opposite visualization. Besides, the noisy activations are owing to redundant features among categories. Building on these insights, we propose the CLIP Surgery for reliable CAM, a method that allows surgery-like modifications to the inference architecture and features, without further fine-tuning as classical CAM methods. This approach significantly improves the explainability of CLIP, surpassing existing methods by large margins. Besides, it enables multimodal visualization and extends the capacity of raw CLIP on open-vocabulary tasks without extra alignment. The code is available at https://github.com/xmed-lab/CLIP_Surgery.
Forward citations
Cited by 12 Pith papers
-
SoccerLens: Grounded Soccer Video Understanding Beyond Accuracy
SoccerLens benchmark shows state-of-the-art soccer VLMs achieve high classification accuracy yet fail to exceed 50% visual grounding performance and underutilize temporal information.
-
SoccerLens: Grounded Soccer Video Understanding Beyond Accuracy
SoccerLens benchmark shows state-of-the-art soccer VLMs achieve strong classification accuracy yet fail to exceed 50% grounding performance on annotated visual cues and underutilize temporal information.
-
From Measurement to Mitigation: Quantifying and Reducing Identity Leakage in Image Representation Encoders with Linear Subspace Removal
Visual encoders leak identity information; a one-shot linear subspace removal method (ISP) reduces leakage to near-chance levels while retaining high non-biometric utility across datasets.
-
Advancing Metallic Surface Defect Detection via Anomaly-Guided Pretraining on a Large Industrial Dataset
AGSSP pretrains object detectors by distilling anomaly maps from a CLIP-based teacher into the backbone and using derived pseudo-boxes to pretrain the detector, improving defect detection over ImageNet pretraining.
-
Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation
A feedback self-adaptive attention module uses CLIP's own output predictions as a spatial coherence prior to reweight intermediate attention, improving training-free open-vocabulary segmentation across 8 benchmarks.
-
Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images
SegEarth-OV performs annotation-free open-vocabulary segmentation of remote-sensing images by upsampling CLIP features, removing global bias, and distilling optical knowledge into a SAR encoder.
-
Bridge Feature Matching and Cross-Modal Alignment with Mutual-filtering for Zero-shot Anomaly Detection
FiSeCLIP achieves state-of-the-art zero-shot anomaly detection by using a batch of test images as mutual references and filtering noisy features with text-guided masks, without any training.
-
Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP
Grad-ECLIP produces gradient-based visual and textual explanation heatmaps for CLIP by applying channel and spatial weights to token features instead of relying on sparse self-attention maps.
-
Open-Attribute Recognition for Person Retrieval: Finding People Through Distinctive and Novel Attributes
A task, a model, and four rebuilt benchmarks for retrieving people from attribute words unseen during training, presented as Open-Attribute Recognition for Person Retrieval.
-
Training-Free Class Purification for Open-Vocabulary Semantic Segmentation
Comparing class activation maps before and after affinity refinement identifies redundant classes, while inter-class overlap plus LLM descriptions resolve ambiguous ones, improving training-free open-vocabulary segmentation.
-
AF-CLIP: Zero-Shot Anomaly Detection via Anomaly-Focused CLIP Adaptation
AF-CLIP improves zero-shot visual anomaly detection by adding a trainable attention adapter and multi-scale spatial aggregation that redirect CLIP features toward local defects, with a patch alignment loss.
-
Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation
GCLIP improves TF-OVSS by reshaping last-block attention via fusion of global-token block attention with Query-Query attention and applying channel suppression to Value embeddings, outperforming prior methods on five ...
Discussion (0). Sign in to comment.