REVIEW 9 cited by
CLIP-GCD: Simple Language Guided Generalized Category Discovery
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
CLIP-GCD: Simple Language Guided Generalized Category Discovery
read the original abstract
Generalized Category Discovery (GCD) requires a model to both classify known categories and cluster unknown categories in unlabeled data. Prior methods leveraged self-supervised pre-training combined with supervised fine-tuning on the labeled data, followed by simple clustering methods. In this paper, we posit that such methods are still prone to poor performance on out-of-distribution categories, and do not leverage a key ingredient: Semantic relationships between object categories. We therefore propose to leverage multi-modal (vision and language) models, in two complementary ways. First, we establish a strong baseline by replacing uni-modal features with CLIP, inspired by its zero-shot performance. Second, we propose a novel retrieval-based mechanism that leverages CLIP's aligned vision-language representations by mining text descriptions from a text corpus for the labeled and unlabeled set. We specifically use the alignment between CLIP's visual encoding of the image and textual encoding of the corpus to retrieve top-k relevant pieces of text and incorporate their embeddings to perform joint image+text semi-supervised clustering. We perform rigorous experimentation and ablations (including on where to retrieve from, how much to retrieve, and how to combine information), and validate our results on several datasets including out-of-distribution domains, demonstrating state-of-art results.
Forward citations
Cited by 9 Pith papers
-
Identifying Latent Concepts and Structures for Generalized Category Discovery
CPF-GCD enforces low-rank compositional structure on vision backbone features via spatial primitive fields so that novel categories emerge as new activation patterns over a shared vocabulary of reusable visual primitives.
-
PACO: Proxy-Task Alignment and Online Calibration for On-the-Fly Category Discovery
PACO provides a hierarchical online decision system with proxy-simulated initial thresholds and adaptive updates from mature prototypes to enable consistent category discovery in streaming sequences.
-
DP-BOA: Dirichlet-Process Birth-or-Assign for On-the-Fly Category Discovery
DP-BOA replaces fixed match thresholds in on-the-fly category discovery with an online Dirichlet-process Gaussian mixture that compares posterior-predictive evidence for assigning a sample to an existing category vers...
-
SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learning
SECOS enables direct semantic label prediction in open-world semi-supervised learning by aligning representations with external knowledge for novel classes, outperforming prior methods by up to 5.4% even without post-...
-
Generalized Category Discovery under Domain Shifts: From Vision to Vision-Language Models
Three frameworks adapt foundation models for generalized category discovery under domain shifts via disentanglement and prompt tuning, showing gains on synthetic and real multi-domain data.
-
The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery
EAGC mitigates gradient entanglement in GCD by anchoring supervised gradients and adaptively projecting unlabeled ones, boosting existing methods to new state-of-the-art performance.
-
SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
SpectralGCD represents images as mixtures over CLIP word-concept similarities, filters the concept dictionary by eigendecomposition of a teacher covariance matrix, and trains a student with forward/reverse distillatio...
-
Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery
SDBA enhances dual-branch multi-modal GCD with a cross-modal synergistic adapter that injects visual info into text encoders and a neighborhood mutual learning module using bidirectional KL divergence, claiming SOTA r...
-
Video-based Generalized Category Discovery via Memory-Guided Consistency-Aware Contrastive Learning
Video-GCD: a new benchmark and a consistency-aware contrastive learning method for discovering known and novel categories in videos.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.