Pith. sign in

REVIEW 10 cited by

A Closer Look at the Explainability of Contrastive Language-Image Pre-training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.05653 v2 pith:K2ZTHHAQ submitted 2023-04-12 cs.CV

Yi Li , Hualiang Wang , Yiqun Duan , Jiheng Zhang , Xiaomeng Li This is my paper
classification cs.CV
keywords clipexplainabilityfeaturesmethodsregionstasksvisualizationwithout
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Contrastive language-image pre-training (CLIP) is a powerful vision-language model that has shown great benefits for various tasks. However, we have identified some issues with its explainability, which undermine its credibility and limit the capacity for related tasks. Specifically, we find that CLIP tends to focus on background regions rather than foregrounds, with noisy activations at irrelevant positions on the visualization results. These phenomena conflict with conventional explainability methods based on the class attention map (CAM), where the raw model can highlight the local foreground regions using global supervision without alignment. To address these problems, we take a closer look at its architecture and features. Based on thorough analyses, we find the raw self-attentions link to inconsistent semantic regions, resulting in the opposite visualization. Besides, the noisy activations are owing to redundant features among categories. Building on these insights, we propose the CLIP Surgery for reliable CAM, a method that allows surgery-like modifications to the inference architecture and features, without further fine-tuning as classical CAM methods. This approach significantly improves the explainability of CLIP, surpassing existing methods by large margins. Besides, it enables multimodal visualization and extends the capacity of raw CLIP on open-vocabulary tasks without extra alignment. The code is available at https://github.com/xmed-lab/CLIP_Surgery.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SoccerLens: Grounded Soccer Video Understanding Beyond Accuracy

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SoccerLens benchmark shows state-of-the-art soccer VLMs achieve high classification accuracy yet fail to exceed 50% visual grounding performance and underutilize temporal information.

  2. SoccerLens: Grounded Soccer Video Understanding Beyond Accuracy

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    SoccerLens benchmark shows state-of-the-art soccer VLMs achieve strong classification accuracy yet fail to exceed 50% grounding performance on annotated visual cues and underutilize temporal information.

  3. From Measurement to Mitigation: Quantifying and Reducing Identity Leakage in Image Representation Encoders with Linear Subspace Removal

    cs.CV 2026-04 conditional novelty 7.0 of 10

    Visual encoders leak identity information; a one-shot linear subspace removal method (ISP) reduces leakage to near-chance levels while retaining high non-biometric utility across datasets.

  4. Advancing Metallic Surface Defect Detection via Anomaly-Guided Pretraining on a Large Industrial Dataset

    cs.CV 2025-09 conditional novelty 7.0 of 10

    AGSSP pretrains object detectors by distilling anomaly maps from a CLIP-based teacher into the backbone and using derived pseudo-boxes to pretrain the detector, improving defect detection over ImageNet pretraining.

  5. Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A feedback self-adaptive attention module uses CLIP's own output predictions as a spatial coherence prior to reweight intermediate attention, improving training-free open-vocabulary segmentation across 8 benchmarks.

  6. Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images

    cs.CV 2025-08 conditional novelty 6.0 of 10

    SegEarth-OV performs annotation-free open-vocabulary segmentation of remote-sensing images by upsampling CLIP features, removing global bias, and distilling optical knowledge into a SAR encoder.

  7. Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Grad-ECLIP produces gradient-based visual and textual explanation heatmaps for CLIP by applying channel and spatial weights to token features instead of relying on sparse self-attention maps.

  8. Open-Attribute Recognition for Person Retrieval: Finding People Through Distinctive and Novel Attributes

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A task, a model, and four rebuilt benchmarks for retrieving people from attribute words unseen during training, presented as Open-Attribute Recognition for Person Retrieval.

  9. Training-Free Class Purification for Open-Vocabulary Semantic Segmentation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Comparing class activation maps before and after affinity refinement identifies redundant classes, while inter-class overlap plus LLM descriptions resolve ambiguous ones, improving training-free open-vocabulary segmentation.

  10. Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation

    cs.LG 2025-02 unverdicted novelty 4.0 of 10

    GCLIP improves TF-OVSS by reshaping last-block attention via fusion of global-token block attention with Query-Query attention and applying channel suppression to Value embeddings, outperforming prior methods on five ...

Pith tools