REVIEW 6 cited by
C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In deep learning, test-time adaptation has gained attention as a method for model fine-tuning without the need for labeled data. A prime exemplification is the recently proposed test-time prompt tuning for large-scale vision-language models such as CLIP. Unfortunately, these prompts have been mainly developed to improve accuracy, overlooking the importance of calibration, which is a crucial aspect for quantifying prediction uncertainty. However, traditional calibration methods rely on substantial amounts of labeled data, making them impractical for test-time scenarios. To this end, this paper explores calibration during test-time prompt tuning by leveraging the inherent properties of CLIP. Through a series of observations, we find that the prompt choice significantly affects the calibration in CLIP, where the prompts leading to higher text feature dispersion result in better-calibrated predictions. Introducing the Average Text Feature Dispersion (ATFD), we establish its relationship with calibration error and present a novel method, Calibrated Test-time Prompt Tuning (C-TPT), for optimizing prompts during test-time with enhanced calibration. Through extensive experiments on different CLIP architectures and datasets, we show that C-TPT can effectively improve the calibration of test-time prompt tuning without needing labeled data. The code is publicly accessible at https://github.com/hee-suk-yoon/C-TPT.
Forward citations
Cited by 6 Pith papers
-
ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Training a policy to prefer responses by optimizing only its own low-confidence (high-surprisal) tokens improves alignment over uniform token optimization in SimPO and DPO.
-
Measurement Plasticity: Sensor-Level Adaptation for Vision-Language Models
Selecting physical camera settings by feature-space affinity to the source domain and voting over multiple exposures outperforms digital-only test-time adaptation for vision-language models on sensor-shifted benchmarks.
-
Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models
The submitted full text does not match the abstract, so the manuscript cannot be assessed as a coherent preprint.
-
Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EM
An online EM algorithm fits class-conditional Gaussians to the test stream from CLIP text-embedding initializations, improving test-time adaptation accuracy over prior methods on 15 benchmarks.
-
Generalizing vision-language models to novel domains: A comprehensive survey
A survey of VLM generalization literature organized by transferred module, with benchmark tables and a review of multimodal LLMs.
-
CLIPTime: Time-Aware Multimodal Representation Learning from Images and Text
CLIPTime adds a classification head and a transformer-style regression head to CLIP embeddings, hitting 98.7% accuracy on synthetic fungi but with weak timestamp predictions, especially for spores.
Discussion (0). Sign in to comment.