Pith. sign in

REVIEW 13 cited by

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01403 v2 pith:JGBGL2UZ submitted 2023-10-02 cs.CV

Size Wu , Wenwei Zhang , Lumin Xu , Sheng Jin , Xiangtai Li , Wentao Liu , Chen Change Loy This is my paper · ORCID
classification cs.CV
keywords clipimagedenseopen-vocabularyclipselfpredictionrepresentationvits
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Open-vocabulary dense prediction tasks including object detection and image segmentation have been advanced by the success of Contrastive Language-Image Pre-training (CLIP). CLIP models, particularly those incorporating vision transformers (ViTs), have exhibited remarkable generalization ability in zero-shot image classification. However, when transferring the vision-language alignment of CLIP from global image representation to local region representation for the open-vocabulary dense prediction tasks, CLIP ViTs suffer from the domain shift from full images to local image regions. In this paper, we embark on an in-depth analysis of the region-language alignment in CLIP models, which is essential for downstream open-vocabulary dense prediction tasks. Subsequently, we propose an approach named CLIPSelf, which adapts the image-level recognition ability of CLIP ViT to local image regions without needing any region-text pairs. CLIPSelf empowers ViTs to distill itself by aligning a region representation extracted from its dense feature map with the image-level representation of the corresponding image crop. With the enhanced CLIP ViTs, we achieve new state-of-the-art performance on open-vocabulary object detection, semantic segmentation, and panoptic segmentation across various benchmarks. Models and code are released at https://github.com/wusize/CLIPSelf.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What Holds Back Open-Vocabulary Segmentation?

    cs.CV 2025-08 conditional novelty 7.0 of 10

    Open-vocabulary segmentation is bottlenecked by CLIP's weak region-level classification, mask proposer quality, the no-object mask filter, and COCO-ADE20K annotation conflicts, each quantified with oracle experiments.

  2. Perceptual Anchoring: Prototype-Guided Text Calibration for Training-free Open-Vocabulary Semantic Segmentation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    PTC recalibrates CLIP text embeddings using per-image visual prototypes, improving training-free open-vocabulary segmentation.

  3. Fine-grained CLIP fine-tuning with self-annotated region alignment

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Fine-tuning CLIP with self-generated region-phrase pairs from caption heatmaps improves dense region understanding without any region annotations, while preserving global retrieval.

  4. Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A feedback self-adaptive attention module uses CLIP's own output predictions as a spatial coherence prior to reweight intermediate attention, improving training-free open-vocabulary segmentation across 8 benchmarks.

  5. Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Fine-tuning CLIP's visual encoder to match DINOv2's kernel-based similarity structure improves its fine-grained visual perception while preserving its alignment to text.

  6. Locality-Aware Zero-Shot Human-Object Interaction Detection

    cs.CV 2025-05 conditional novelty 6.0 of 10

    LAIN adds locality and interaction adapters to CLIP, reaching state-of-the-art zero-shot human-object interaction detection on HICO-DET and V-COCO.

  7. DynAlign: Unsupervised Dynamic Taxonomy Alignment for Cross-Domain Segmentation

    cs.CV 2025-01 conditional novelty 6.0 of 10

    DynAlign aligns source and target taxonomies by using GPT-4 to map labels, SAM for mask proposals, and CLIP to reassign fine-grained target labels, improving unsupervised cross-domain segmentation under label-space shift.

  8. DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Training a text encoder to align with frozen DINOv2 features, plus two small adaptation blocks and balanced data curation, yields strong zero-shot classification and open-vocabulary segmentation at a fraction of CLIP'...

  9. Incorporating Feature Pyramid Tokenization and Open Vocabulary Semantic Segmentation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    PAT clusters multi-resolution VLM features into semantic codebook tokens and jointly trains reconstruction and segmentation, improving open-vocabulary segmentation over the SAN baseline.

  10. Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Trident, a training-free framework combining CLIP, DINO, and SAM, raises state-of-the-art open-vocabulary segmentation mIoU from 44.4 to 48.6 by splicing sub-image features and aggregating them with a SAM affinity matrix.

  11. Sparse Attention for Dense Open-Vocabulary Prediction in CLIP

    cs.CV 2026-07 conditional novelty 5.5 of 10

    Replacing softmax with α-entmax in frozen CLIP's final attention layers denoises dense predictions by zeroing irrelevant token interactions, with gains proportional to baseline attention diffuseness.

  12. YOLO-UniOW: Efficient Universal Open-World Object Detection

    cs.CV 2024-12 conditional novelty 5.0 of 10

    YOLO-UniOW unifies open-vocabulary and open-world detection by removing cross-modal fusion and adding a self-supervised wildcard embedding, reaching 34.6 AP on LVIS at 69.6 FPS.

  13. Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A dynamic pruning method scores each sample by combining task loss with CLIP image-text similarity and selects samples near the median score each epoch.

Pith tools