REVIEW 5 cited by
A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation
read the original abstract
Contrastive Language-Image Pretraining (CLIP) has gained popularity for its remarkable zero-shot capacity. Recent research has focused on developing efficient fine-tuning methods, such as prompt learning and adapter, to enhance CLIP's performance in downstream tasks. However, these methods still require additional training time and computational resources, which is undesirable for devices with limited resources. In this paper, we revisit a classical algorithm, Gaussian Discriminant Analysis (GDA), and apply it to the downstream classification of CLIP. Typically, GDA assumes that features of each class follow Gaussian distributions with identical covariance. By leveraging Bayes' formula, the classifier can be expressed in terms of the class means and covariance, which can be estimated from the data without the need for training. To integrate knowledge from both visual and textual modalities, we ensemble it with the original zero-shot classifier within CLIP. Extensive results on 17 datasets validate that our method surpasses or achieves comparable results with state-of-the-art methods on few-shot classification, imbalanced learning, and out-of-distribution generalization. In addition, we extend our method to base-to-new generalization and unsupervised learning, once again demonstrating its superiority over competing approaches. Our code is publicly available at \url{https://github.com/mrflogs/ICLR24}.
Forward citations
Cited by 5 Pith papers
-
Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses in Vision-Language Models
High-noise feature drift distinguishes adversarial from clean inputs in CLIP, allowing a plug-in gating mechanism to selectively trigger existing test-time defenses and raise mean clean+adversarial accuracy across 13 ...
-
Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction
KL-anchored penalized likelihood with class- and instance-dependent shrinkage, implemented with von Mises-Fisher mixtures, improves CLIP test-time transduction under class imbalance.
-
[CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation
PIAA improves multi-label recognition by enhancing patch-wise predictions via semantic disentanglement and an unsupervised visual classifier, then adaptively aggregating them, yielding over 6% mAP gain on NUS-WIDE in ...
-
Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification
MVSL improves low-resource biomedical image classification via decoupled vision-language adaptation, multi-granularity contrastive learning, and LLM-based semantic regularization.
-
CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values
CLIP-SVD performs parameter-efficient adaptation of CLIP by fine-tuning singular values from SVD of weight matrices, reporting SOTA few-shot accuracy on 21 datasets plus a language-based interpretability analysis.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.