A^4D detects adversarial attacks in an attack- and classifier-agnostic way by measuring non-arbitrary shifts in CLIP embedding space from prompt-based similarity scores.
hub
Clipself: Vision transformer distills itself for open-vocabulary dense prediction
14 Pith papers cite this work. Polarity classification is still indexing.
14
Pith papers citing it
abstract
Open-vocabulary dense prediction tasks including object detection and image segmentation have been advanced by the success of Contrastive Language-Image Pre-training (CLIP). CLIP models, particularly those incorporating vision transformers (ViTs), have exhibited remarkable generalization ability in zero-shot image classification. However, when transferring the vision-language alignment of CLIP from global image representation to local region representation for the open-vocabulary dense prediction tasks, CLIP ViTs suffer from the domain shift from full images to local image regions. In this paper, we embark on an in-depth analysis of the region-language alignment in CLIP models, which is essential for downstream open-vocabulary dense prediction tasks. Subsequently, we propose an approach named CLIPSelf, which adapts the image-level recognition ability of CLIP ViT to local image regions without needing any region-text pairs. CLIPSelf empowers ViTs to distill itself by aligning a region representation extracted from its dense feature map with the image-level representation of the corresponding image crop. With the enhanced CLIP ViTs, we achieve new state-of-the-art performance on open-vocabulary object detection, semantic segmentation, and panoptic segmentation across various benchmarks. Models and code are released at https://github.com/wusize/CLIPSelf.
hub tools
citation-role summary
background 2
citation-polarity summary
roles
background 2polarities
background 2representative citing papers
Introduces COVD task and Novel-114 benchmark plus NoIn-Det, a parameter-free method that freezes the visual encoder and updates limited text parameters to inject novel concepts while preserving old knowledge.
SHED improves domain generalization in CLIP by aligning style-homogenized embeddings instead of raw ones, achieving state-of-the-art results on five benchmarks including a 4% gain on DomainNet.
OVS-DINO structurally aligns DINO with SAM to revitalize attenuated boundary features, achieving SOTA gains of 2.1% average and 6.3% on Cityscapes in weakly-supervised open-vocabulary segmentation.
SubSearch improves LLM reasoning traces on QA and multi-hop QA tasks by rewarding intermediate steps with intrinsic process rewards instead of only final outcomes.
ExDet proposes a lightweight framework using text-guided extrapolation, detector-compatible rectification, and recalibrated proposals to achieve state-of-the-art open-domain open-vocabulary detection on multiple benchmarks.
The authors define the LV-OSD problem and introduce the LVDor dual-branch framework with TPDW dynamic weighting and PRM masking to align multimodal prompts for open-set detection.
WOW-Seg proposes a word-free open-world segmentation model using Mask2Token and Cascade Attention Mask modules, reporting 89.7 semantic similarity and 82.4 semantic IoU on LVIS with one-eighth the parameters of prior SOTA plus a new 7,662-class benchmark.
Pi-HOC predicts dense 3D semantic contacts for all human-object pairs in an image via instance-aware tokens and an InteractionFormer, achieving higher accuracy and 20x throughput than prior methods.
ViTs exhibit lazy aggregation by relying on irrelevant background patches for global semantics, and selectively integrating patch features into the CLS token reduces this effect and improves results across label-, text-, and self-supervision.
Inference-time α-entmax sparsification of CLIP’s final self-attention denoises diffuse mass and lifts dense open-vocabulary segmentation and region retrieval in proportion to baseline off-class spread.
The approach uses the analytic solution of distribution discrepancy consistency within categories as semantic maps, eliminating training and model-specific modulation while claiming state-of-the-art results on eight benchmarks.
A survey that organizes methods for cross-domain object detection into a taxonomy, analyzes domain shift across detection stages, and outlines persistent challenges.
citing papers explorer
-
A Classifier-Agnostic Zero-Shot Adversarial Attack Detection via CLIP
A^4D detects adversarial attacks in an attack- and classifier-agnostic way by measuring non-arbitrary shifts in CLIP embedding space from prompt-based similarity scores.
-
COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injection
Introduces COVD task and Novel-114 benchmark plus NoIn-Det, a parameter-free method that freezes the visual encoder and updates limited text parameters to inject novel concepts while preserving old knowledge.
-
SHED: Style-Homogenized Embedding Alignment for Domain Generalization
SHED improves domain generalization in CLIP by aligning style-homogenized embeddings instead of raw ones, achieving state-of-the-art results on five benchmarks including a 4% gain on DomainNet.
-
OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance
OVS-DINO structurally aligns DINO with SAM to revitalize attenuated boundary features, achieving SOTA gains of 2.1% average and 6.3% on Cityscapes in weakly-supervised open-vocabulary segmentation.
-
SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval
SubSearch improves LLM reasoning traces on QA and multi-hop QA tasks by rewarding intermediate steps with intrinsic process rewards instead of only final outcomes.
-
ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification
ExDet proposes a lightweight framework using text-guided extrapolation, detector-compatible rectification, and recalibrated proposals to achieve state-of-the-art open-domain open-vocabulary detection on multiple benchmarks.
-
LV-OSD: Language-Vision-Complementary Open-Set Object Detection
The authors define the LV-OSD problem and introduce the LVDor dual-branch framework with TPDW dynamic weighting and PRM masking to align multimodal prompts for open-set detection.
-
WOW-Seg: A Word-free Open World Segmentation Model
WOW-Seg proposes a word-free open-world segmentation model using Mask2Token and Cascade Attention Mask modules, reporting 89.7 semantic similarity and 82.4 semantic IoU on LVIS with one-eighth the parameters of prior SOTA plus a new 7,662-class benchmark.
-
Pi-HOC: Pairwise 3D Human-Object Contact Estimation
Pi-HOC predicts dense 3D semantic contacts for all human-object pairs in an image via instance-aware tokens and an InteractionFormer, achieving higher accuracy and 20x throughput than prior methods.
-
Vision Transformers Need More Than Registers
ViTs exhibit lazy aggregation by relying on irrelevant background patches for global semantics, and selectively integrating patch features into the CLS token reduces this effect and improves results across label-, text-, and self-supervision.
-
Sparse Attention for Dense Open-Vocabulary Prediction in CLIP
Inference-time α-entmax sparsification of CLIP’s final self-attention denoises diffuse mass and lifts dense open-vocabulary segmentation and region retrieval in proportion to baseline off-class spread.
-
Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation
The approach uses the analytic solution of distribution discrepancy consistency within categories as semantic maps, eliminating training and model-specific modulation while claiming state-of-the-art results on eight benchmarks.
-
Generalization Under Scrutiny: Cross-Domain Detection Progresses, Pitfalls, and Persistent Challenges
A survey that organizes methods for cross-domain object detection into a taxonomy, analyzes domain shift across detection stages, and outlines persistent challenges.
- Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP