Pith. sign in

REVIEW 16 cited by

A Closer Look at the Explainability of Contrastive Language-Image Pre-training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.05653 v2 pith:K2ZTHHAQ submitted 2023-04-12 cs.CV

Yi Li , Hualiang Wang , Yiqun Duan , Jiheng Zhang , Xiaomeng Li This is my paper
classification cs.CV
keywords clipexplainabilityfeaturesmethodsregionstasksvisualizationwithout
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Contrastive language-image pre-training (CLIP) is a powerful vision-language model that has shown great benefits for various tasks. However, we have identified some issues with its explainability, which undermine its credibility and limit the capacity for related tasks. Specifically, we find that CLIP tends to focus on background regions rather than foregrounds, with noisy activations at irrelevant positions on the visualization results. These phenomena conflict with conventional explainability methods based on the class attention map (CAM), where the raw model can highlight the local foreground regions using global supervision without alignment. To address these problems, we take a closer look at its architecture and features. Based on thorough analyses, we find the raw self-attentions link to inconsistent semantic regions, resulting in the opposite visualization. Besides, the noisy activations are owing to redundant features among categories. Building on these insights, we propose the CLIP Surgery for reliable CAM, a method that allows surgery-like modifications to the inference architecture and features, without further fine-tuning as classical CAM methods. This approach significantly improves the explainability of CLIP, surpassing existing methods by large margins. Besides, it enables multimodal visualization and extends the capacity of raw CLIP on open-vocabulary tasks without extra alignment. The code is available at https://github.com/xmed-lab/CLIP_Surgery.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SoccerLens: Grounded Soccer Video Understanding Beyond Accuracy

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SoccerLens benchmark shows state-of-the-art soccer VLMs achieve high classification accuracy yet fail to exceed 50% visual grounding performance and underutilize temporal information.

  2. SoccerLens: Grounded Soccer Video Understanding Beyond Accuracy

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SoccerLens benchmark shows state-of-the-art soccer VLMs achieve strong classification accuracy yet fail to exceed 50% grounding performance on annotated visual cues and underutilize temporal information.

  3. From Measurement to Mitigation: Quantifying and Reducing Identity Leakage in Image Representation Encoders with Linear Subspace Removal

    cs.CV 2026-04 conditional novelty 7.0 of 10

    Visual encoders leak identity information; a one-shot linear subspace removal method (ISP) reduces leakage to near-chance levels while retaining high non-biometric utility across datasets.

  4. Advancing Metallic Surface Defect Detection via Anomaly-Guided Pretraining on a Large Industrial Dataset

    cs.CV 2025-09 conditional novelty 7.0 of 10

    AGSSP pretrains object detectors by distilling anomaly maps from a CLIP-based teacher into the backbone and using derived pseudo-boxes to pretrain the detector, improving defect detection over ImageNet pretraining.

  5. Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A feedback self-adaptive attention module uses CLIP's own output predictions as a spatial coherence prior to reweight intermediate attention, improving training-free open-vocabulary segmentation across 8 benchmarks.

  6. Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images

    cs.CV 2025-08 conditional novelty 6.0 of 10

    SegEarth-OV performs annotation-free open-vocabulary segmentation of remote-sensing images by upsampling CLIP features, removing global bias, and distilling optical knowledge into a SAR encoder.

  7. Bridge Feature Matching and Cross-Modal Alignment with Mutual-filtering for Zero-shot Anomaly Detection

    cs.CV 2025-07 conditional novelty 6.0 of 10

    FiSeCLIP achieves state-of-the-art zero-shot anomaly detection by using a batch of test images as mutual references and filtering noisy features with text-guided masks, without any training.

  8. ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    ReME builds a cleaned, synonym-enriched segment-text reference set from real images and shows that simple similarity retrieval on it beats 14 prior training-free open-vocabulary segmentation methods across ten benchmarks.

  9. Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Grad-ECLIP produces gradient-based visual and textual explanation heatmaps for CLIP by applying channel and spatial weights to token features instead of relying on sparse self-attention maps.

  10. Open-Attribute Recognition for Person Retrieval: Finding People Through Distinctive and Novel Attributes

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A task, a model, and four rebuilt benchmarks for retrieving people from attribute words unseen during training, presented as Open-Attribute Recognition for Person Retrieval.

  11. Training-Free Class Purification for Open-Vocabulary Semantic Segmentation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Comparing class activation maps before and after affinity refinement identifies redundant classes, while inter-class overlap plus LLM descriptions resolve ambiguous ones, improving training-free open-vocabulary segmentation.

  12. AF-CLIP: Zero-Shot Anomaly Detection via Anomaly-Focused CLIP Adaptation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    AF-CLIP improves zero-shot visual anomaly detection by adding a trainable attention adapter and multi-scale spatial aggregation that redirect CLIP features toward local defects, with a patch alignment loss.

  13. VISTA: A Visual Analytics Framework to Enhance Foundation Model-Generated Data Labels

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A human-in-the-loop visual analytics system that detects, summarizes, and corrects label and alignment errors in foundation-model-generated image segmentation data, improving downstream open-vocabulary segmentation pe...

  14. StackCLIP: Clustering-Driven Stacked Prompt in Zero-Shot Industrial Anomaly Detection

    cs.CV 2025-06 conditional novelty 4.0 of 10

    Stacking multiple category names in a CLIP text prompt, along with cluster-specific alignment layers, improves zero-shot industrial defect detection and localization.

  15. Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation

    cs.LG 2025-02 unverdicted novelty 4.0 of 10

    GCLIP improves TF-OVSS by reshaping last-block attention via fusion of global-token block attention with Query-Query attention and applying channel suppression to Value embeddings, outperforming prior methods on five ...

  16. Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges

    cs.CV 2025-07 conditional novelty 2.0 of 10

    A structured survey of prompt engineering methods for the Segment Anything Model, covering geometric, textual, and multimodal prompts and their applications.

Pith tools