Pith. sign in

REVIEW 4 major objections 8 minor 67 references

Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score

T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-grained unsupervised adaptation of CLIP gains 2.78% average accuracy over prior methods by learning a crop-to-anchor alignment score during self-training.

desk verdict Solid, well-specified method for unsupervised CLIP adaptation with a plausible but unverified 2.78% gain; lack of code, seeds, and error bars keeps me from trusting the exact number. read the letter →

arxiv 2507.09615 v1 pith:AKDJ3NL6 submitted 2025-07-13 cs.CV

Eman Ali , Sathira Silva , Chetan Arora , Muhammad Haris Khan This is my paper
classification cs.CV
keywords fine-grainedclassificationunsupervisedadaptationvision-languagemodelsCLIPpseudo-labelingcross-modalalignmentself-trainingclassdescriptionanchors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that fine-grained unsupervised adaptation of CLIP works best when the alignment score itself adapts during training, rather than staying fixed or being tied to handcrafted prompts. It introduces FAIR, which aligns top-k local image crops with class-description anchors built from LLM-generated text and uses a margin-weighted pseudo-label objective for self-training. Across 13 fine-grained datasets with no target labels, FAIR reports 76.40% average top-1 accuracy, beating the previous best method at 73.62%. This matters because fine-grained classes are separated by small visual differences, and the paper argues that dynamic cross-modal alignment captures those differences and produces cleaner pseudo-labels.

What carries the argument

The Class Description Anchor (CDA) is a per-class embedding initialized as the mean of LLM-generated description embeddings, after which the text encoder is discarded and the anchors are updated alongside the image encoder. The Learned Alignment Score (Eq. 14) computes the similarity between selected top-k image crops and CDA rows, weighting each crop by its normalized CLS-token similarity to the global view, and serves as the adaptive classifier for pseudo-labeling. The adaptive weighting term gamma (Eq. 16) equals the top alignment score times the margin between the top and second scores, which lowers the self-training loss when classes are ambiguous. The overall objective combines a gamma-weighted cross-entropy loss on strongly augmented images with a fairness regularization that encourages balanced predictions across classes.

What would settle it

Take a subset of StanfordCars where two classes share the same manufacturer and model year but differ only in small parts, run FAIR with the reported (N,k)=(16,4) configuration and 15 epochs, and compare pseudo-label accuracy against true labels on a held-out annotated set; the claim fails if those classes do not improve over zero-shot CLIP or if the cosine similarity between the two CDA rows increases during training.

Watch

Extended reading notes

Core claim

FAIR establishes that modeling fine-grained cross-modal interactions during self-training produces more class-discriminative pseudo-labels than fixed prompt ensembles or a frozen LLM-tuned classifier. The central discovery is a Learned Alignment Score that combines selective local crops, computed with CLS-token-based relevance weights without softmax normalization, with Class Description Anchors that act as an adaptive classifier co-trained with the image encoder. A margin-based weighting term, gamma = S_i times (S_i minus S_j), down-weights ambiguous pseudo-labels where the top two alignment scores are close. With only layer-normalization weights and the anchors trainable, FAIR reports 76.40% average top-1 accuracy on 13 datasets, a 2.78% gain over the strongest prior method under the reported protocols.

Load-bearing premise

The whole pipeline assumes the LLM-generated descriptions already separate the classes well enough that the initial anchors give largely correct pseudo-labels; if the text is generic or overlapping, self-training has no external signal to recover.

Editorial extensions

If this is right

  • If the central claim is correct, fine-grained unsupervised adaptation of CLIP can use 16 crops and keep only 4 per image, avoiding the computational cost of the roughly 60-crop pseudo-labeling used by prior crop-based methods.
  • The method transfers to a different vision-language model (MetaCLIP ViT-B/32) and a different backbone (ViT-B/16), so the learned-alignment mechanism is not tied to a single pretrained encoder.
  • In the transductive setting, FAIR reaches 75.57% average accuracy, above DPA (72.86%) and ReCLIP (69.27%), indicating the same score function works when adapting directly on the test split.
  • The ablation results attribute a large share of the gain to the Learned Alignment Score itself (removing it drops accuracy by 7.04% on the six ablation datasets), with the margin weighting adding a smaller but consistent improvement.
  • Only layer-normalization weights and anchors are optimized (about 45.1K parameters on EuroSAT), so FAIR is parameter-efficient compared with methods that backpropagate through both encoders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the text encoder is discarded after initialization, the ceiling of FAIR is set by how well the LLM-generated descriptions separate the classes; a natural extension is to feed crop clusters with low top-2 margins back into the LLM to generate more specific descriptions.
  • Editorial inference: the margin statistic S_i minus S_j could be used as an online detector of anchor overlap, flagging classes whose descriptions are too generic before self-training compounds the error.
  • Editorial inference: replacing gamma with an entropy-based or temperature-calibrated confidence measure might improve robustness on imbalanced fine-grained datasets, where the current cosine margin may not be well calibrated.
  • Editorial inference: combining FAIR with candidate pseudo-label strategies that retain multiple possible labels per image could mitigate confirmation bias specifically in the low-margin cases that gamma down-weights.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This manuscript proposes FAIR, an unsupervised adaptation method for CLIP-based fine-grained classification. Using only unlabeled target images and class names, FAIR initializes one Class Description Anchor (CDA) per class as the averaged embedding of LLM-generated (CuPL) descriptions (Algorithm 1, Eqs. 11-13), then defines a Learned Alignment Score (LAS) as a weighted top-k-aggregated cosine similarity between random image crops and CDA (Eq. 14). Pseudo-labels from LAS supervise a strongly augmented view through a cross-entropy loss weighted by gamma_x = S_i,x * |S_i,x - S_j,x| (Eqs. 16-17), plus a fairness regularization. Only LayerNorm parameters of the image encoder and the CDA are updated. The main empirical claim is a 76.40% mean top-1 accuracy over 13 datasets (ViT-B/32), a 2.78-point average improvement over DPA (73.62%), with ablations (Table 3) attributing most of the gain to LAS.

Significance. If the reported results are reproducible, FAIR is a competitive addition to the unsupervised-adaptation literature for VLMs: it outperforms several recent baselines across 13 datasets, transfers to ViT-B/16 and MetaCLIP backbones (Tables 2 and 8), and improves in both inductive and transductive protocols (Table 4). The paper deserves credit for a clearly specified and coherent formulation, internally consistent arithmetic (e.g., the 13-dataset average of Table 1 checks out), and a thorough ablation study that isolates LAS as the dominant component (7.04-point drop without it; Table 3). The authors also document the StanfordCars failure mode explicitly (Section 8, Figs. 11-12), which is more candid than common practice. The central empirical claim is nevertheless not yet established with sufficient rigor: all numbers are single-run, no code, seeds, or CuPL description files are provided, and key hyperparameters are selected on test-set accuracy. The headline gain is also highly uneven, being driven primarily by EuroSAT and UCF101. For these reasons the significance of the contribution is conditional on the additional evidence requested below.

major comments (4)
  1. [4 (Tables 1-4) and Supplementary Sec. 6.1] All numbers in Tables 1-4, 7, 8, and 10 come from single runs; no error bars, seeds, or significance tests are reported, and the code is only promised 'upon acceptance' (footnote 1), with the stochastic CuPL description draws not released. The pipeline is stochastic in several coupled places: crop sampling lambda_i in Eq. (6), GPT-3 sampling in CuPL that determines the CDA initialization, and RandAugment. The reported 2.78% average gain over DPA is also uneven: it is carried by EuroSAT (+11.98) and UCF101 (+6.85), while FAIR loses to DPA on Caltech101 (-1.82) and CUB (-1.45) and gains less than half a point on Food101, Flowers102, OxfordPets, and SUN397. The EuroSAT improvement over zero-shot CLIP (48.90 -> 91.92) is large enough that it, and the headline average, need to be verified across independent runs before the claim is established. Please report mean and standard deviation over at least three runs per dataset, publish the exact CuPL description files and seeds, and state per-dataset best/worst.
  2. [3.4 (Eqs. 11-17), Algorithm 1, Section 8] The self-training loop has no external anchor that ties the updated CDA Z* to the original class identities. The pseudo-label is the argmax of a similarity computed against the very Z* that is being optimized (Eqs. 14 and 17), and the only safeguard, gamma_x, down-weights individually ambiguous examples but cannot detect systematic mislabeling of an entire class or a class permutation. This is not merely speculative: Section 8 documents that on StanfordCars the CuPL descriptions for two Lamborghini classes are generic sports-car text (Fig. 11c/d), the corresponding anchors overlap in embedding space (Fig. 12), and accuracy degrades; Section 8 also states that 'pseudo-labeling becomes unreliable... potentially causing cascading errors during training.' Given that the abstract's mechanism claim is that the learned alignment score produces 'more accurate, class-discriminative pseudo-labels,' the paper should provide direct evidence that drift is bounded: e.g., pseudo-label accuracy as a function of training epoch on the six ablation datasets, the cosine distance between Z*(t) and the initial text-derived anchors, and a check that the final anchor assignment has not permuted the initial class order. Either such measurements or a modification that anchors Z* (e.g., a small regularization toward the initial text embeddings) would address this load-bearing risk.
  3. [4 (Implementation Details) and Fig. 10] The hyperparameter choices appear to be made on test-set accuracy in a setting that is supposed to be label-free. The manuscript states that (N,k)=(16,4) is used 'according to the analysis illustrated in Fig. 10,' which evaluates accuracy on the Flowers dataset; the per-dataset learning rate is 10^-4 except for Food101 and SUN397, set to 10^-6, with no stated selection criterion; and the six datasets in Table 3 are used for design iterations. Since no labeled target data is available by the protocol of Section 3, selecting hyperparameters on the test split of the benchmarks (or on the same six datasets whose results are then reported in the ablations) can inflate the reported gains relative to baselines whose hyperparameters were not tuned on these splits. This interacts with the single-run issue raised above. Please state explicitly the selection criterion and whether test labels or a held-out unlabeled set were used, add a sensitivity analysis of the headline 13-dataset average to reasonable alternatives (e.g., k=2 or k=8; learning rate 10^-5 vs 10^-4 on several datasets), and describe the fixed protocol used for FAIR and for each baseline in one place.
  4. [3.3-3.4 (Eqs. 7-14) and Algorithm 1] There is a gap between the claimed mechanism and the design: the abstract and introduction attribute the gains to 'modeling fine-grained cross-modal interactions,' but CDA initialization averages all M LLM descriptions per class into a single vector (Algorithm 1, lines 2-6) and the text encoder is then discarded. The resulting eTheta in Eq. (11) is an N x C crop-by-class matrix, so the per-description granularity that WCA exploits through the description weights v_j (Eq. 9) is entirely collapsed before training. The 'fine-grained' component that remains is crop-level selection plus per-class anchor adaptation. To support the mechanism claim, please report an ablation that keeps M anchors per class versus the collapsed single anchor, and show anchor trajectories that illustrate what LAS learns beyond re-weighting crops. If the collapsed version is retained, the paper should be rephrased so that the claimed contribution is 'learned per-class anchors with crop-level selection,' which is a defensible but narrower claim.
minor comments (8)
  1. [4, Table 1] The footnote for WCA* says results are 'reproduced using the same number of crops as FAIR,' but it is unclear whether this means N=16 with WCA's original weighting or the FAIR (N,k)=(16,4) protocol; since Fig. 2 shows WCA accuracy is highly sensitive to N, specify the exact setting.
  2. [3.4, Eqs. (12)-(14)] The crop weights fw_i in Eq. (12) are normalized by the sum over all N crops, yet Eq. (14) restricts the sum to the top-k crops via the indicator I{i in Ik}; state whether the selected weights are renormalized after selection, since the interpretation of LAS as a weighted combination depends on this.
  3. [Algorithm 1, line 18; Sec. 6.1] Line 18 of Algorithm 1 samples crops from the weakly augmented view alpha(x), while Sec. 3.4 defines p(x) on the raw image; reconcile the notation by defining alpha as CenterCrop in the main text.
  4. [4, Eq. (15)] The fairness regularization Lreg is taken from [32] with no ablation isolating its contribution; add a 'FAIR w/o Lreg' row to Table 3 or Table 5 and state the relative weight of Lst and Lreg in the total loss.
  5. [Table 6] CIFAR10 appears in the dataset statistics table but is not among the 13 datasets in Table 1; either remove it or explain its role.
  6. [Figs. 8, 10, 11] Several figure panels contain garbled text (e.g., the y-axis labels of Fig. 8(b) and the inner labels of Fig. 10); regenerate these figures with embeddable fonts.
  7. [Abstract and Sec. 1] The 13-dataset suite is characterized as 'fine-grained' although it includes scene and object datasets such as EuroSAT, SUN397, Caltech101, CIFAR100, and UCF101; qualify the terminology, especially since EuroSAT alone contributes most of the reported average gain.
  8. [3.4] The sentence 'we set vj in Eq. (9) to 1, since it represents the self-similarity of each CDA' is unclear: vj weights text descriptions in WCA, whereas FAIR has one anchor per class; clarify the intended analogy.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the reported 2.78% gain is a measured benchmark comparison against external baselines, not a quantity derived from fitted parameters; the only self-citation is a baseline, and the self-training loop is disclosed.

full rationale

The paper's central claim is an empirical accuracy comparison: Table 1 reports top-1 accuracy on 13 datasets using ViT-B/32, and Table 2 reports ViT-B/16 results, with baselines reproduced from official codebases as stated in Section 6.1. No reported accuracy is obtained by fitting parameters to the headline number. The CDA anchors are initialized from CuPL LLM-generated descriptions (Eqs. 11-14 and Algorithm 1, lines 13, 20-24), which is an external, non-fitted source; they are then updated by a self-training objective (Eqs. 15-17). Using pseudo-labels generated by the same model receiving the loss is standard self-training and is fully disclosed in the method and Algorithm 1, so it is not a disguised 'prediction' from fitted inputs. The manuscript's own Section 8 documents a genuine failure mode on StanfordCars caused by insufficiently discriminative LLM descriptions; this is a real limitation and a robustness concern, but it is not a circular step. The only self-citation is the authors' prior work DPA [3], used as a baseline rather than as load-bearing justification. Accordingly, no claimed result reduces by construction to its inputs or to a self-citation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The method pulls its text descriptions from CuPL, its crop-augmentation protocol from WCA, and its fairness regularizer from MUST (Li et al.), all cited. The genuinely new parts are the learnable CDA, the top-k crop selection, and the margin weighting, each carrying hand-chosen hyperparameters. Axioms 1-3 are standard domain assumptions for this literature; axioms 4-5 are ad hoc to this paper because the design choices are justified only by test-set experiments. The CDA is an invented entity whose only support is the benchmark average.

free parameters (6)
  • N (number of random crops per image) = 16
    Chosen from the Flowers test-set sweep (Fig. 10), not derived; the paper states N and k 'require tuning' (Section 3.4).
  • k (number of top crops kept) = 4
    Chosen from the same Flowers test-set sweep (Fig. 10); affects pseudo-label quality and compute, with no analytical justification.
  • Crop scale bounds (alpha, beta) = alpha=0.5, beta=0.9
    Inherited unchanged from WCA (Section 6.1), not re-justified for the adaptation setting.
  • Per-dataset learning rate = 1e-4, with 1e-6 for Food101 and SUN397
    Hand-set per dataset with no stated selection rule (Sections 4 and 6.1).
  • Training epochs and batch size = 15 epochs, batch size 32
    Fixed choices with no sensitivity analysis; convergence claims (15 vs 50 epochs for ReCLIP) are not backed by curves.
  • Number of LLM descriptions per class = 20 to 90 per class depending on dataset (Table 6)
    Depends on stochastic CuPL generation; the seed, prompt settings, and cleaned description set are not reported or shipped.
assumptions (5)
  • domain assumption Cosine similarity in CLIP's shared embedding space is a valid score between image crop features and class anchors built from averaged text embeddings.
    Eqs. (1), (11)-(14) compute pseudo-labels and the training loss from cosines between crop features and averaged description embeddings; this assumes the VLM space is informative at crop level, inherited from WCA without re-validation.
  • domain assumption LLM-generated descriptions from CuPL are discriminative enough that averaged per-class anchors give a usable pseudo-labeling signal.
    Section 3.4 initializes CDA as the mean of LLM descriptions and then discards the text encoder. Section 8 shows this fails for StanfordCars, where descriptions for visually similar Lamborghini classes are generic; the entire pipeline depends on this initialization.
  • domain assumption Self-training on pseudo-labels improves accuracy when the pseudo-labeler is reasonably accurate, with confirmation bias adequately controlled by the fairness regularizer.
    Eqs. (15)-(17) plus L_reg from Li et al. [32]. No mechanism detects or corrects systematic pseudo-label errors beyond the scalar weight gamma_x, so the central assumption is that LAS pseudo-labels are accurate enough.
  • ad hoc to paper Top-k crop selection with [CLS]-token relevance weights preserves the discriminative signal of the full crop set.
    Eqs. (12)-(13); k=4 of N=16 is validated only empirically on the Flowers test set (Fig. 10), and the softmax-removal choice is justified by the illustrative Fig. 7 rather than by theory or prior work.
  • ad hoc to paper Test-set accuracy on the reported benchmarks is an acceptable objective for choosing hyperparameters and the six ablation datasets.
    Fig. 10 selects (N,k) from Flowers test accuracy versus cost, and Section 6.1 selects the six ablation datasets following ReCLIP's procedure. In a label-free pipeline there is no validation set, so test-informed tuning is unaddressed.
invented entities (1)
  • Class Description Anchors (CDA)
    purpose: Learnable per-class vectors replacing text embeddings; used as the pseudo-label classifier (Eq. 14) and as the inference classifier (Section 3.4).
    CDA is the only genuinely new model component. Its evidence is the end-task accuracy of the whole pipeline; there is no falsifiable handle outside the reported datasets, no analysis that the anchors retain semantic content after training, and no released anchor files.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score." pith.science (2026). https://pith.science/paper/AKDJ3NL6

@misc{pith2026250709615,
  author       = {Pith},
  title        = {Pith review of: Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKDJ3NL6}},
  note         = {Machine review of arXiv:2507.09615}
}
read the original abstract

Vision-language models (VLMs) like CLIP excel in zero-shot learning by aligning image and text representations through contrastive pretraining. Existing approaches to unsupervised adaptation (UA) for fine-grained classification with VLMs either rely on fixed alignment scores that cannot capture evolving, subtle class distinctions or use computationally expensive pseudo-labeling strategies that limit scalability. In contrast, we show that modeling fine-grained cross-modal interactions during adaptation produces more accurate, class-discriminative pseudo-labels and substantially improves performance over state-of-the-art (SOTA) methods. We introduce Fine-grained Alignment and Interaction Refinement (FAIR), an innovative approach that dynamically aligns localized image features with descriptive language embeddings through a set of Class Description Anchors (CDA). This enables the definition of a Learned Alignment Score (LAS), which incorporates CDA as an adaptive classifier, facilitating cross-modal interactions to improve self-training in unsupervised adaptation. Furthermore, we propose a self-training weighting mechanism designed to refine pseudo-labels in the presence of inter-class ambiguities. Our approach, FAIR, delivers a substantial performance boost in fine-grained unsupervised adaptation, achieving a notable overall gain of 2.78% across 13 fine-grained datasets compared to SOTA methods.

Figures

Figures reproduced from arXiv: 2507.09615 by the authors.

Figure 1
Figure 1. Class-wise classification accuracy (%) comparison be [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Efficiency comparison of FAIR vs. WCA [31] as a pseudo-labeling method using the EuroSAT dataset. The radius of each circle corresponds to the amount of GPU memory allocated. sion–language correspondences, enabling zero-shot predic￾tions across diverse inputs. However, CLIP’s performance is highly susceptible to prompting, making optimal prompt selection complex and time-consuming [64]. Recent re￾search [9, 26, 34, … view at source ↗
Figure 3
Figure 3. Overall framework of FAIR. (a) shows the initialization of Class Description Anchors (CDA) using LLM-generated prompts, which are aggregated into representative features for each class. (b) illustrates the FAIR pseudo-labeling and self-supervision pipeline, where cross-modal fine-grained interaction and alignment ( circled in the figure) between random image crops and CDAs generate pseudo￾labels via a refined alignm… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: t-SNE plots of image embeddings (circle ) for LaFTer [38], WCA [31] as a pseudo-labeling method, and FAIR, along with their corresponding classifiers (star ⋆) on the EuroSAT dataset. LaFTer uses fixed alignment-based pseudo-labels, caus￾ing class confusion, while FAIR …
Figure 5
Figure 5. Figure 5: Comparative confusion matrices illustrating model per [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 8
Figure 8. Figure 8: Visualization of performance improvements from our [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Comparison of accuracy (%) and number of parameters between FAIR and different parameter-efficient fine-tuning meth￾ods. Accuracy is averaged across the six ablation datasets. in FAIR: the number of sampled crops (N) and the num￾ber of top-weighted crops (k). We evalua…
Figure 10
Figure 10. Figure 10: Analysis of accuracy, training time, and GPU memory [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Comparison of images between two visually similar car models from the same manufacturer: (a) Lamborghini Aventador Coupe [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: t-SNE visualization of embeddings from five similar car classes in the StanfordCars [ [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 51 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Dpa: Dual prototypes alignment for unsupervised adaptation of vision-language models

    Eman Ali, Sathira Silva, and Muhammad Haris Khan. Dpa: Dual prototypes alignment for unsupervised adaptation of vision-language models. In Proceedings of the Winter Con- ference on Applications of Computer Vision (WACV), pages 6083–6093, February 2025. 1, 2, 6, 7

    work page 2025

  4. [4]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. Layer normalization. arXiv preprint arXiv:1607.06450,

    arXiv

  5. [5]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pages 446–461. Springer, 2014. 7

    work page 2014

  6. [6]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 2

    1901

  7. [7]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 5

    work page 2021

  8. [8]

    Remote sens- ing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sens- ing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017. 7

    work page 2017

Show all 67 references
  1. [9]

    Distribution-aware prompt tuning for vision-language mod- els

    Eulrang Cho, Jooyeon Kim, and Hyunwoo J Kim. Distribution-aware prompt tuning for vision-language mod- els. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 22004–22013, 2023. 2

    2023

  2. [10]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014. 6

    2014

  3. [11]

    Randaugment: Practical automated data augmen- tation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmen- tation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020. 7, 1

    2020

  4. [12]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Sys- tems, 36:35544–35575, 2023. 3

    2023

  5. [13]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004. 6

    2004

  6. [14]

    Gpt-3: Its nature, scope, limits, and consequences

    Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines , 30:681–694, 2020. 2

    2020

  7. [15]

    Perceptron-based learning algo- rithms

    Stephen I Gallant et al. Perceptron-based learning algo- rithms. IEEE Transactions on neural networks , 1(2):179– 191, 1990. 2

    1990

  8. [16]

    Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl. https://github.com/ facebookresearch/vissl, 2021. 1

    2021

  9. [17]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2021. 1

    2021

  10. [18]

    Parameter-efficient model adaptation for vision transformers

    Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-efficient model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 817–825, 2023. 8, 2

    2023

  11. [19]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 1, 7, 2

    2019

  12. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In- ternational Conference on Learning Representations, 1(2):3,

  13. [21]

    Reclip: Refine contrastive language image pre-training with source free domain adaptation

    Xuefeng Hu, Ke Zhang, Lu Xia, Albert Chen, Jiajia Luo, Yuyin Sun, Ken Wang, Nan Qiao, Xiao Zeng, Min Sun, et al. Reclip: Refine contrastive language image pre-training with source free domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Compu...

    2024

  14. [22]

    · · ·sim(f1, Z∗ C) ... ... ... sim(fN , Z∗

  15. [23]

    · · ·sim(fN , Z∗ C)   21: fwi ← sim(f [CLS],f [CLS] i )PN l=1 sim(f [CLS],f [CLS] l ) 22: Ik ← argsort( fW)[: k] ▷ Select the top-k crop indices based on fW 23: ψFAIR (x, y|p, Ev, Z∗, C) ← PN i=1 fwi eΘij|j=C(y)I{i∈Ik} ▷ Evaluate FAIR similarity function 24: ˆy ← arg maxy∈Y...

  16. [24]

    Adapting visual-language models for generalizable anomaly detection in medical im- ages

    Chaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang, Xin- chao Wang, and Yanfeng Wang. Adapting visual-language models for generalizable anomaly detection in medical im- ages. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 11375–11385,

  17. [25]

    Unsupervised prompt learning for vision-language models

    Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03649, 2022. 1, 2, 6, 7

    2022 arXiv

  18. [26]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, 5 Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4...

  19. [27]

    Learning to prompt with text only supervision for vision- language models

    Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision- language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 4230–4238, 2025. 3

    2025

  20. [28]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muham- mad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19113–19122, 2023. 1, 2

    2023

  21. [29]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, pages 554–561, 2013. 7, 2, 3

    2013

  22. [30]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7

    2009

  23. [31]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic seg- mentation. International Conference on Learning Represen- tations, 2022. 1

    2022

  24. [32]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2

    2022

  25. [33]

    Visual-text cross alignment: Refining the similarity score in vision-language models

    Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng, James Bai- ley, and Feng Liu. Visual-text cross alignment: Refining the similarity score in vision-language models. In International Conference on Machine Learning, 2024. 1, 2, 3, 4, 5, 6, 7

    2024

  26. [34]

    Masked unsupervised self-training for label-free image classifica- tion

    Junnan Li, Silvio Savarese, and Steven CH Hoi. Masked unsupervised self-training for label-free image classifica- tion. International Conference on Learning Representations,

  27. [35]

    Align before fuse: Vision and language representation learn- ing with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 2

    2021

  28. [36]

    Promptkd: Unsupervised prompt distillation for vision-language models

    Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26617–26626, 2024. 2

    2024

  29. [37]

    Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation

    Yuqi Lin, Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofei He. Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  30. [38]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. 1

    2017

  31. [39]

    Prompt distribution learning

    Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5206–5215, 2022. 2

    2022

  32. [40]

    Lafter: Label-free tuning of zero-shot clas- sifier using language and unlabeled image collections

    Muhammad Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Possegger, Mateusz Kozinski, Rogerio Feris, and Horst Bischof. Lafter: Label-free tuning of zero-shot clas- sifier using language and unlabeled image collections. Ad- vances in Neural Information Processing Systems, 36:...

    2023

  33. [41]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008. 7

    2008

  34. [42]

    Valse: A task- independent benchmark for vision and language models cen- tered on linguistic phenomena

    Letitia Parcalabescu, Michele Cafagna, Lilitta Muradjan, Anette Frank, Iacer Calixto, and Albert Gatt. Valse: A task- independent benchmark for vision and language models cen- tered on linguistic phenomena. In Proceedings of the 60th Annual Meeting of the Association for Compu...

    2022

  35. [43]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 7

    2012

  36. [44]

    What does a platypus look like? generating customized prompts for zero-shot image classification

    Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15691–15701, 2023. 1, 3, 6, 7, 2, 4

    2023

  37. [45]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  38. [46]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guil- laume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 156...

    2022

  39. [47]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. Center for Research in Computer Vision, 2012. 7

    2012

  40. [48]

    Pouf: Prompt-oriented unsupervised fine-tuning for large pre-trained models

    Korawat Tanwisuth, Shujian Zhang, Huangjie Zheng, Pengcheng He, and Mingyuan Zhou. Pouf: Prompt-oriented unsupervised fine-tuning for large pre-trained models. In In- ternational Conference on Machine Learning, pages 33816– 33832. PMLR, 2023. 2, 6, 7

    2023

  41. [49]

    Clip the gap: A single domain generalization approach for object detection

    Vidit Vidit, Martin Engilberge, and Mathieu Salzmann. Clip the gap: A single domain generalization approach for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3219–3229,

  42. [50]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 7

    2011

  43. [51]

    Tent: Fully test-time adaptation by entropy minimization

    Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Ol- shausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. 7, 1 6

    2021

  44. [52]

    Debiased learning from naturally imbalanced pseudo-labels

    Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14647–14657, 2022. 6

    2022

  45. [53]

    Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching

    Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7031–7040, 2023. 1

    2023

  46. [54]

    Aid: A benchmark data set for performance evaluation of aerial scene classification

    Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7):3965–3981, 2017. 7

    2017

  47. [55]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer so- ciety conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010. 7

    2010

  48. [56]

    Demystify- ing clip data

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystify- ing clip data. arXiv preprint arXiv:2309.16671, 2023. 2

    2023 arXiv

  49. [57]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 18134–18144, 2022. 1

    2022

  50. [58]

    Lever- aging cross-modal neighbor representation for improved clip classification

    Chao Yi, Lu Ren, De-Chuan Zhan, and Han-Jia Ye. Lever- aging cross-modal neighbor representation for improved clip classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27402– 27411, 2024. 1, 3

    2024

  51. [59]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 2

    2021 arXiv

  52. [60]

    When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023. 1, 2

    2023

  53. [61]

    Boost- ing vision-language models with transduction

    Maxime Zanella, Benoît Gérin, and Ismail Ben Ayed. Boost- ing vision-language models with transduction. Advances in Neural Information Processing Systems , 37:62223–62256,

  54. [62]

    Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data.International Conference on Machine Learning, 2024

    Jiahan Zhang, Qi Wei, Feng Liu, and Lei Feng. Candidate pseudolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data.International Conference on Machine Learning, 2024. 2

    2024

  55. [63]

    Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners

    Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Han- qiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2023

  56. [64]

    Mediclip: Adapting clip for few-shot medical image anomaly detection

    Ximiao Zhang, Min Xu, Dehui Qiu, Ruixin Yan, Ning Lang, and Xiuzhuang Zhou. Mediclip: Adapting clip for few-shot medical image anomaly detection. In International Confer- ence on Medical Image Computing and Computer-Assisted Intervention, pages 458–468. Springer, 2024. 1, 2

    2024

  57. [65]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 16816–16825,

  58. [66]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  59. [67]

    Not all features mat- ter: Enhancing few-shot clip with adaptive prior refinement

    Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features mat- ter: Enhancing few-shot clip with adaptive prior refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2605–2615, 2023. 3 7

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.