Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensembling

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPE, a group-wise prompt ensemble, preserves CLIP's zero-shot ability while adding domain-specific knowledge, improving novel-class accuracy by 1.7% and cross-dataset transfer.

desk verdict A credible incremental method and a useful new cross-dataset benchmark, but the zero-shot retention claim is statistically fragile and the paper overstates its findings. read the letter →

arxiv 2412.07077 v1 pith:RWRJDAW5 submitted 2024-12-10 cs.CV

Donggeun Kim , Yujin Jo , Myungjoo Lee , Taesup Kim This is my paper · ORCID
classification cs.CV
keywords group-wisepromptensembleCLIPlearningmaskedattentionzero-shotgeneralizationcross-datasettransfercovarianceregularizationvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a prompt-ensemble method, Group-wise Prompt Ensemble (GPE), that fine-tunes CLIP on a downstream dataset without destroying the zero-shot knowledge CLIP already has. GPE divides learnable prompts into two main groups plus auxiliary prompts, uses masked attention so the groups cannot interfere with CLIP's frozen special tokens, and trains each group with its own classification loss plus a covariance penalty that decorrelates the prompts. At inference, GPE averages predictions from all prompt pairs and the original CLIP special tokens. The paper reports that GPE achieves a base-to-new harmonic mean of 79.24 across 11 datasets, outperforming prior prompt-learning baselines, and that it loses very little zero-shot accuracy in cross-dataset transfer even after fine-tuning on fine-grained datasets.

What carries the argument

The central mechanism is the Group-wise Prompt Ensemble (GPE): two groups of learnable prompt tokens, each of length K, plus auxiliary prompts, inserted into CLIP's text and image encoders with hand-designed attention masks. The first group attends only to input tokens (read-only), the second group attends to input and auxiliary prompts, and auxiliary prompts attend only to themselves. This isolates the learnable prompts from CLIP's frozen special tokens, keeping the original representation intact while letting each group evolve independently. During training, a group-wise cross-entropy loss is applied per group and a covariance regularization term decorrelates prompt embeddings; during inference, the softmax probabilities of all (2K+1)^2 prompt-plus-special-token pairs are averaged. The special tokens are the carriers of zero-shot knowledge and are the reason the ensemble retains CLIP's original capabilities.

What would settle it

Measure the actual drift of CLIP's special-token embeddings (the [CLS] and [EOS] vectors) before and after GPE fine-tuning on a fine-grained dataset such as Flowers102; if the cosine similarity between pre- and post-training special tokens is near zero, the zero-shot retention claim fails. Alternatively, evaluate GPE on a held-out dataset not in the 11-dataset suite and check whether novel-class accuracy still exceeds zero-shot CLIP.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that prompt grouping, combined with masked attention and group-wise ensemble training, lets a frozen CLIP model absorb new domain knowledge while keeping its original zero-shot classifier intact. By restricting the first prompt group to read-only attention on input tokens and giving the second group access to auxiliary prompts, the two groups learn complementary features rather than redundant ones; a covariance regularization term, adapted from Barlow Twins, further decorrelates prompt embeddings. The full ensemble at inference includes CLIP's original special tokens, which is what allows the model to retain zero-shot ability. The paper reports a harmonic mean of 79.24 for base-to-new generalization, 1.7% higher than zero-shot CLIP on novel classes, and the best cross-dataset transfer average (63.17) among compared methods, with minimal zero-shot degradation after fine-tuning on fine-grained sources.

Load-bearing premise

The method works only if the attention masks genuinely keep the learnable prompts from disturbing CLIP's frozen representation, so that the original special tokens still encode zero-shot knowledge.

Editorial extensions

If this is right

  • GPE can be used as a drop-in prompt-learning head for CLIP-style models; it needs only a few learnable tokens per encoder and no change to the frozen backbone.
  • Fine-tuning on fine-grained or niche datasets no longer forces a trade-off against zero-shot performance, so models can be adapted to specialized domains and still transferred to other tasks.
  • The group-wise training recipe (masked attention plus covariance loss) provides a template for other prompt-based adapters that want to preserve pre-trained knowledge.
  • Cross-dataset transfer results suggest GPE could serve as a building block for continual learning of vision-language models, where the model must accumulate knowledge without forgetting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's explanation for zero-shot retention is architectural (masked attention isolates prompts from special tokens), but it never measures the representation shift directly; a direct check would be to compare CLIP special-token embeddings before and after GPE training on datasets like Flowers102 and FGVCAircraft.
  • The covariance regularization is adapted from self-supervised learning (Barlow Twins); GPE could be combined with other diversity-promoting losses, such as contrastive decorrelation, to see whether further diversity gains translate to further novel-class accuracy.
  • Because GPE uses a fixed prompt length (Ktotal=24), extending it to variable-length prompts or prompt pruning might be needed for very large label sets; the paper does not test that regime.
  • The claim that GPE redefines the adaptability benchmark depends on the 11-dataset suite; testing on a broader set of fine-grained datasets or scaling to larger backbones would reveal how robust the improvement is.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes Group-wise Prompt Ensemble (GPE), a prompt-tuning method for frozen CLIP that combines two groups of learnable prompts with masked attention (each group reads input tokens; the second group also attends to auxiliary prompts), a covariance regularization that decorrelates prompt embeddings, and a full ensemble at inference that averages classifiers formed by all main prompts together with the original CLIP special tokens. The method is evaluated on the 11-dataset benchmark for base-to-new generalization, on an extended cross-dataset transfer protocol with all 11 sources, and on ImageNet domain generalization, reporting the best average harmonic mean (79.24) over prior prompt methods and claiming improved novel-class and zero-shot retention.

Significance. If its empirical claims withstand scrutiny, GPE is a useful contribution to parameter-efficient CLIP adaptation: the idea of training prompt groups with isolated attention and then ensembling them with the frozen special-token classifier is simple and well ablated (Tables 4-5), and the extended cross-dataset protocol with multiple source datasets is a more informative test than the standard ImageNet-only transfer evaluation. The paper also makes a falsifiable prediction that prompt diversity, quantified by the Vendi score, tracks harmonic-mean performance (Table 4). However, the headline claims of 'surpassing zero-shot CLIP on novel classes' and 'near zero-shot' cross-dataset retention are stronger than the reported numbers support, and the absence of variance or significance information makes the main comparisons hard to evaluate. With corrected claims and additional statistical evidence, the method would be a solid, incremental advance in prompt ensembling.

major comments (4)
  1. [Section 4.2, Table 1] The claim that 'GPE surpasses zero-shot CLIP in novel class evaluations, achieving a 1.7% improvement' is not robust: GPE's novel accuracy is below zero-shot CLIP on 6 of the 11 datasets (Caltech101 -0.10, StanfordCars -0.39, Flowers102 -1.50, Food101 -0.02, FGVCAircraft -1.89, UCF101 -1.13), and the average gain is driven largely by EuroSAT (+12.22) and DTD (+4.07). The paper reports only 3-run averages without standard deviations or significance tests, so the central superiority claim is statistically fragile. Please report per-dataset variance and paired significance tests, or revise the claim to state that GPE's average novel accuracy is comparable to, and slightly above, zero-shot CLIP only on select datasets.
  2. [Section 4.3, Table 2] The statement that GPE 'shows minimal loss in zero-shot capabilities after fine-tuning' and achieves 'near zero-shot performance' is contradicted by the table: GPE's average target accuracy is 63.17 versus 65.25 for zero-shot CLIP, a 2.08-point gap, and after training on Flowers102 the gap is 5.0 points (59.64 vs 64.64), after OxfordPets 2.25 points, and after FGVCAircraft 2.35 points. The proper claim is that GPE degrades less than other prompt-tuning baselines, not that it approaches the zero-shot upper bound. Please quantify the gaps and adjust the wording in Section 4.3 and the abstract.
  3. [Section 4.1, Table 5] The main hyperparameters—number of groups, K=9, K'=6, Ktotal=24, λ=500, and the choice of pre-softmax group-wise training—are selected by experiments on the same 11-dataset benchmark used for the headline results, and Table 5 shows that the harmonic mean ranges from 75.32 to 79.24 across these design choices. No validation split or selection protocol is described, so the comparison to baselines may reflect tuning on the benchmark rather than a robust advantage. Please specify how the configuration was chosen (e.g., on a held-out split), or provide a sensitivity analysis that reports results for a small grid of hyperparameters with the same protocol.
  4. [Section 3.2, Figures 2-3] The mechanism claim that masked attention 'safeguards zero-shot capabilities' by preventing disruption of CLIP's original representation is supported only by downstream accuracy, not by any direct measurement of representation shift. Because the final ensemble in Eq. (3) always includes the original special-token classifier, the observed novel-class retention could also arise from averaging a stable zero-shot classifier with shifted prompt classifiers rather than from true isolation of the representation. Please report a direct analysis of the learned prompts' effect on the special-token embeddings (e.g., cosine similarity to the untouched CLIP embeddings), or at minimum ablate the special-token contribution across all settings in Table 2, not only in the base-to-new setting of Table 5.
minor comments (5)
  1. [Table 2] In the MaPLe row, the entry '62.3354.32' appears to be a missing space between two numbers; please fix the formatting.
  2. [Section 4.1] The dataset list contains the typo 'StandfordCars'; it should be 'StanfordCars'.
  3. [Section 2.2] The phrase 'for a various applications' is ungrammatical and should be revised.
  4. [Abstract] The phrase 'redefines the benchmarks' is promotional; a concrete statement of the reported gains would be more appropriate.
  5. [Section 6] The paper does not include a limitations or failure-case discussion; a short section acknowledging the dataset-dependence of the gains would improve calibration and reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are benchmark accuracies produced by an explicit, externally evaluated method; no result is forced by construction or by self-citation.

full rationale

The paper is an empirical prompt-ensembling method for CLIP. Its central claims—base-to-novel harmonic mean of 79.24, novel-class accuracy 75.92 vs. 74.22 for zero-shot CLIP, and cross-dataset average of 63.17—are reported measurements, not quantities derived from fitted parameters by definition. The main mechanism, masked-attention prompt grouping, is a concrete architectural choice (Eq. 1 and the attention masks in Section 3.2), and the ensemble inference (Eq. 3) explicitly averages over learned prompt embeddings and the frozen CLIP special tokens. Including the special token is a deliberate design decision, not a hidden fit: it makes zero-shot knowledge a component of the ensemble, but the reported improvement over zero-shot CLIP is an empirical outcome, not an identity. The covariance regularization and Vendi Score analysis provide independent, measurable evidence of prompt diversity rather than renaming a known result. There are no self-citations: the prior work most closely relied on, RPO [21] and CLIP [26], is authored by different research groups, and the paper does not import any uniqueness theorem or ansatz from the present authors' earlier work. Hyperparameters (e.g., K=9, K'=6, lambda=500) and design choices such as pre-softmax training were selected using the same 11-dataset benchmark, and the paper does not report error bars or significance tests; this is a legitimate statistical-robustness concern and a potential source of benchmark-selection bias, but it is not circularity under the definition used here because the reported accuracies are not forced to equal any fitted quantity or to follow from the paper's equations. The paper therefore deserves a low circularity score; the robustness concerns belong to correctness risk rather than circular reasoning.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard benchmark assumptions and on method-specific design choices (prompt lengths, covariance weight, number of groups, ensembling mode) that were tuned on the same evaluation sets. No new physical entities are introduced; auxiliary prompts are learnable parameters.

free parameters (4)
  • Prompt lengths per group and auxiliary prompts = K=9, K'=6, Ktotal=24
    Chosen as optimal after 'extensive experiments with various prompt lengths' (Section 5.2); affects ensemble size and performance.
  • Covariance loss weight lambda = 500
    Set in Section 4.1 'we set lambda=500 to weight the covariance loss'; no sensitivity analysis is reported.
  • Training and inference ensembling mode = pre-softmax for training, post-softmax for inference
    Selected because 'experiments demonstrated that it provides better performance' (Section 3.3), a post hoc choice on the evaluation benchmark.
  • Number of prompt groups = 2
    Expanding to three groups did not improve performance (Table 5), so two groups were kept.
assumptions (5)
  • domain assumption CLIP's frozen weights, including the special token embeddings, provide a stable zero-shot representation that prompt tuning should preserve.
    Section 3.1 treats CLIP as fixed and builds prompts on top; if CLIP's representation is not stable under the added tokens, the zero-shot retention claim fails.
  • domain assumption The attention masks in Section 3.2 ensure that the first group reads only the input and the second group reads the input plus auxiliary prompts, with no back-coupling that would shift the original features.
    Figures 2-3 and the mask equations assume this masking is sufficient to prevent representation shift; no direct measurement of shift is given.
  • ad hoc to paper Covariance regularization on prompt embeddings decorrelates classifiers and improves ensemble accuracy.
    Borrowed from Barlow Twins [40] and applied to prompt embeddings; the transfer is assumed rather than derived.
  • domain assumption The 11 dataset, 16-shot base-to-new protocol and the extended cross-dataset protocol are valid proxies for zero-shot capability and generalization.
    Taken from CoOp [43] and extended by the authors; if these benchmarks do not reflect real distribution shifts, the conclusions are limited.
  • domain assumption The last-epoch model with 3-run averaging is a reliable estimator of performance.
    Section 4.1 reports no variance or model selection; small differences may be noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensembling." pith.science (2026). https://pith.science/paper/RWRJDAW5

@misc{pith2026241207077,
  author       = {Pith},
  title        = {Pith review of: Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensembling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RWRJDAW5}},
  note         = {Machine review of arXiv:2412.07077}
}
read the original abstract

The advancement of vision-language models, particularly the Contrastive Language-Image Pre-training (CLIP) model, has revolutionized the field of machine learning by enabling robust zero-shot learning capabilities. These capabilities allow models to understand and respond to previously unseen data without task-specific training. However, adapting CLIP to integrate specialized knowledge from various domains while retaining its zero-shot capabilities remains a significant challenge. To address this, we introduce a novel prompt ensemble learning approach called Group-wise Prompt Ensemble (GPE). This method aims to enhance CLIP's zero-shot capabilities by incorporating new domain knowledge while improving its adaptability and robustness against data distribution shifts. Our approach hinges on three main strategies: prompt grouping with masked attention to optimize CLIP's adaptability while safeguarding its zero-shot capabilities; the incorporation of auxiliary prompts for the seamless integration of new domain insights without disrupting the original model's representation; and an ensemble learning strategy that effectively merges original and new knowledge. Through rigorous experimentation, including more challenging cross-dataset transfer evaluations, our GPE method redefines the benchmarks for the adaptability and efficiency of vision-language models, surpassing existing models across various scenarios.

Figures

Figures reproduced from arXiv: 2412.07077 by the authors.

Figure 1
Figure 1. Cross-Dataset Evaluation across various source datasets. This evaluation measures how well models trained on a specific source dataset (e.g., ImageNet, Flowers102, FGVCAir￾craft) generalize when tested on 10 other target datasets, relative to CLIP’s zero-shot performance. When trained on a general dataset like (a), most models maintain or even exceed CLIP’s zero-shot performance. However, when fine-tuned on speciali… view at source ↗
Figure 2
Figure 2. Overview of our framework. The framework consists of a Text Encoder with Grouped Prompts(Pt) and an Image Encoder with Grouped Prompts(Pv). The first group of the main prompts(P 1 t , P 1 v ) is shown in blue, the second group(P 2 t , P 2 v ) in red, and the auxiliary prompts(P ′ t , P ′ v) in gray. During training, we utilize a Group-wise Ensemble approach, while for inference, we employ a Full Ensemble strategy. T… view at source ↗
Figure 3
Figure 3. Attention masks of GPE. In transformer models, attention masks determine which input parts can interact by allowing or blocking connections between them. The colored boxes indicate areas where attention occurs, while the white boxes indicate masked regions. The first group prompts are restricted to reading the input only, without modifying it. In the second group prompts, masking allows attention to both the input a… view at source ↗
Figures from the paper (1 more)
Figure 2
Figure 2. Figure 2: p(yk | x) = 1 (2K+1)2 P2K i=0 P2K j=0 P exp(simi,j (x,yk)/τ) C k′=1 exp(simi,j (x,yk′ )/τ) (3) where simi,j (x, yk) = vi·tj |vi||tj | . Special Tokens for Zero-shot Performance As shown in Eq 3, our ensemble approach utilizes both groups’ prompts and special tokens dur…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 27 canonical work pages

  1. [1]

    A simple zero-shot prompt weight- ing technique to improve prompt ensembling in text-image models

    James Urquhart Allingham, Jie Ren, Michael W Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Bal- aji Lakshminarayanan. A simple zero-shot prompt weight- ing technique to improve prompt ensembling in text-image models. In International Conference on Machine Learning, pages 547–568. PMLR, 2023

    work page 2023

  2. [2]

    Vi- creg: Variance-invariance-covariance regularization for self- supervised learning

    Adrien Bardes, Jean Ponce, and Yann LeCun. Vi- creg: Variance-invariance-covariance regularization for self- supervised learning. arXiv preprint arXiv:2105.04906, 2021

    arXiv 2021

  3. [3]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461. Springer, 2014

    2014

  4. [4]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020

    1901

  5. [5]

    Apollo: Unified adapter and prompt learning for vision lan- guage models

    Sanjoy Chowdhury, Sayan Nag, and Dinesh Manocha. Apollo: Unified adapter and prompt learning for vision lan- guage models. In The 2023 Conference on Empirical Meth- ods in Natural Language Processing, 2023

    work page 2023

  6. [6]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014

    2014

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  8. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

Show all 45 references
  1. [9]

    Ensemble methods in machine learn- ing

    Thomas G Dietterich. Ensemble methods in machine learn- ing. In International workshop on multiple classifier systems, pages 1–15. Springer, 2000

    2000

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  3. [11]

    Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories

    Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning gener- ative visual models from few training examples: An incre- mental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004

    2004

  4. [12]

    Deep ensembles: A loss landscape perspective

    Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019

    1912 arXiv

  5. [13]

    The vendi score: A diversity evaluation metric for machine learning, 2023

    Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning, 2023

    2023

  6. [14]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019

    2019

  7. [15]

    The many faces of robust- ness: A critical analysis of out-of-distribution generalization

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robust- ness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF Internation...

    2021

  8. [16]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 15262–15271, 2021

    2021

  9. [17]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–491...

    2021

  10. [18]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113–19122, 2023

    2023

  11. [19]

    Self-regulating prompts: Foundational model adaptation without forgetting

    Muhammad Uzair Khattak, Syed Talal Wasim, Muzam- mal Naseer, Salman Khan, Ming-Hsuan Yang, and Fa- had Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15...

    2023

  12. [20]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, pages 554–561, 2013

    2013

  13. [21]

    Read-only prompt optimization for vision-language few-shot learning

    Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1401–1411, 2023

    2023

  14. [22]

    Prompt distribution learning

    Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5206–5215, 2022

    2022

  15. [23]

    Fine-grained visual classi- fication of aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classi- fication of aircraft. arXiv preprint arXiv:1306.5151, 2013

    2013 arXiv

  16. [24]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008

    2008

  17. [25]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 3498–

    2012

  18. [26]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  19. [27]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  20. [28]

    Do imagenet classifiers generalize to im- agenet? In International conference on machine learning , pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In International conference on machine learning , pages 5389–5400. PMLR, 2019

    2019

  21. [29]

    Test- time prompt tuning for zero-shot generalization in vision- language models

    Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test- time prompt tuning for zero-shot generalization in vision- language models. Advances in Neural Information Process- ing Systems, 35:14274–14289, 2022

    2022

  22. [30]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  23. [31]

    Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models.arXiv preprint arXiv:2407.05342, 2024

    Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Han- tao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models.arXiv preprint arXiv:2407.05342, 2024

    2024 arXiv

  24. [32]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  25. [33]

    Learning robust global representations by penalizing local predictive power

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. volume 32, 2019

    2019

  26. [34]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gon- tijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Visi...

    2022

  27. [35]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer so- ciety conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010

    2010

  28. [36]

    Visual- language prompt tuning with knowledge-guided context op- timization

    Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- language prompt tuning with knowledge-guided context op- timization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6757–6767, June 2023

    2023

  29. [37]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021

    2021 arXiv

  30. [38]

    Boosting continual learning of vision-language models via mixture-of-experts adapters

    Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 23219–23230, 2024

    2024

  31. [39]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021

    2021 arXiv

  32. [40]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Ma- chine Learning, pages 12310–12320. PMLR, 2021

    2021

  33. [41]

    Lit: Zero-shot transfer with locked-image text tuning

    Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18123–18133, 2022

    2022

  34. [42]

    Preventing zero-shot transfer degradation in continual learning of vision-language models

    Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xi- angyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19125–19136, 2023

    2023

  35. [43]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 16816–16825, 2022

    2022

  36. [44]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348, 2022

    2022

  37. [45]

    Prompt-aligned gradient for prompt tuning

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Han- wang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15659–15669, 2023

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.