Pith. sign in

REVIEW 16 cited by

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01403 v2 pith:JGBGL2UZ submitted 2023-10-02 cs.CV

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

Size Wu , Wenwei Zhang , Lumin Xu , Sheng Jin , Xiangtai Li , Wentao Liu , Chen Change Loy This is my paper
classification cs.CV
keywords clipimagedenseopen-vocabularyclipselfpredictionrepresentationvits
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Open-vocabulary dense prediction tasks including object detection and image segmentation have been advanced by the success of Contrastive Language-Image Pre-training (CLIP). CLIP models, particularly those incorporating vision transformers (ViTs), have exhibited remarkable generalization ability in zero-shot image classification. However, when transferring the vision-language alignment of CLIP from global image representation to local region representation for the open-vocabulary dense prediction tasks, CLIP ViTs suffer from the domain shift from full images to local image regions. In this paper, we embark on an in-depth analysis of the region-language alignment in CLIP models, which is essential for downstream open-vocabulary dense prediction tasks. Subsequently, we propose an approach named CLIPSelf, which adapts the image-level recognition ability of CLIP ViT to local image regions without needing any region-text pairs. CLIPSelf empowers ViTs to distill itself by aligning a region representation extracted from its dense feature map with the image-level representation of the corresponding image crop. With the enhanced CLIP ViTs, we achieve new state-of-the-art performance on open-vocabulary object detection, semantic segmentation, and panoptic segmentation across various benchmarks. Models and code are released at https://github.com/wusize/CLIPSelf.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP

    cs.CV 2026-06 unverdicted novelty 7.0

    A^4D detects adversarial attacks in an attack- and classifier-agnostic way by measuring non-arbitrary shifts in CLIP embedding space from prompt-based similarity scores.

  2. A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP

    cs.CV 2026-06 unverdicted novelty 7.0

    A^4D is a classifier- and attack-agnostic zero-shot adversarial attack detector based on CLIP embedding shifts that claims SOTA performance.

  3. COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injection

    cs.CV 2026-05 unverdicted novelty 7.0

    Introduces COVD task and Novel-114 benchmark plus NoIn-Det, a parameter-free method that freezes the visual encoder and updates limited text parameters to inject novel concepts while preserving old knowledge.

  4. SHED: Style-Homogenized Embedding Alignment for Domain Generalization

    cs.CV 2026-05 conditional novelty 7.0

    SHED improves domain generalization in CLIP by aligning style-homogenized embeddings instead of raw ones, achieving state-of-the-art results on five benchmarks including a 4% gain on DomainNet.

  5. OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance

    cs.CV 2026-04 unverdicted novelty 7.0

    OVS-DINO structurally aligns DINO with SAM to revitalize attenuated boundary features, achieving SOTA gains of 2.1% average and 6.3% on Cityscapes in weakly-supervised open-vocabulary segmentation.

  6. SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval

    cs.IR 2026-04 unverdicted novelty 7.0

    SubSearch improves LLM reasoning traces on QA and multi-hop QA tasks by rewarding intermediate steps with intrinsic process rewards instead of only final outcomes.

  7. ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification

    cs.CV 2026-06 unverdicted novelty 6.0

    ExDet proposes a lightweight framework using text-guided extrapolation, detector-compatible rectification, and recalibrated proposals to achieve state-of-the-art open-domain open-vocabulary detection on multiple benchmarks.

  8. LV-OSD: Language-Vision-Complementary Open-Set Object Detection

    cs.CV 2026-05 unverdicted novelty 6.0

    The authors define the LV-OSD problem and introduce the LVDor dual-branch framework with TPDW dynamic weighting and PRM masking to align multimodal prompts for open-set detection.

  9. WOW-Seg: A Word-free Open World Segmentation Model

    cs.CV 2026-05 conditional novelty 6.0

    WOW-Seg proposes a word-free open-world segmentation model using Mask2Token and Cascade Attention Mask modules, reporting 89.7 semantic similarity and 82.4 semantic IoU on LVIS with one-eighth the parameters of prior ...

  10. Pi-HOC: Pairwise 3D Human-Object Contact Estimation

    cs.CV 2026-04 unverdicted novelty 6.0

    Pi-HOC predicts dense 3D semantic contacts for all human-object pairs in an image via instance-aware tokens and an InteractionFormer, achieving higher accuracy and 20x throughput than prior methods.

  11. Pi-HOC: Pairwise 3D Human-Object Contact Estimation

    cs.CV 2026-04 unverdicted novelty 6.0

    Pi-HOC is a new instance-aware framework that predicts dense 3D semantic contacts between all human-object pairs in an image via dedicated HO tokens, InteractionFormer refinement, and a SAM decoder, achieving higher a...

  12. Vision Transformers Need More Than Registers

    cs.CV 2026-02 unverdicted novelty 6.0

    ViTs exhibit lazy aggregation by relying on irrelevant background patches for global semantics, and selectively integrating patch features into the CLS token reduces this effect and improves results across label-, tex...

  13. Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP

    cs.CV 2025-02 conditional novelty 6.0

    Grad-ECLIP produces gradient-based visual and textual explanation heatmaps for CLIP by applying channel and spatial weights to token features instead of relying on sparse self-attention maps.

  14. Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation

    cs.CV 2026-04 unverdicted novelty 5.0

    The approach uses the analytic solution of distribution discrepancy consistency within categories as semantic maps, eliminating training and model-specific modulation while claiming state-of-the-art results on eight b...

  15. Sparse Attention for Dense Open-Vocabulary Prediction in CLIP

    cs.CV 2026-07 accept novelty 4.0

    Replacing softmax with α-entmax in frozen CLIP's final attention layers denoises dense predictions by zeroing irrelevant token interactions, with gains proportional to baseline attention diffuseness.

  16. Generalization Under Scrutiny: Cross-Domain Detection Progresses, Pitfalls, and Persistent Challenges

    cs.CV 2026-04 unverdicted novelty 3.0

    A survey that organizes methods for cross-domain object detection into a taxonomy, analyzes domain shift across detection stages, and outlines persistent challenges.