Pith. sign in

REVIEW 4 major objections 6 minor 86 references

Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MuGCP claims that a frozen multimodal large language model can act as a conditional prompt learner whose key-value memory, read by a short learnable query, produces instance-specific semantic prompts that improve few-shot vision-language…

desk verdict Solid few-shot prompt-learning paper with strong numbers, but the SCP mechanism is under-evidenced and comparison fairness is shaky. read the letter →

arxiv 2507.08410 v1 pith:XF27YRKV submitted 2025-07-11 cs.CV

Shijun Yang , Xiang Zhang , Wanqing Zhao , Hangzai Luo , Sheng Zhong , Jinye Peng , Jianping Fan This is my paper
classification cs.CV
keywords promptlearningvision-languagemodelsmultimodallargelanguagefew-shotclassificationbase-to-newgeneralizationconditionalpromptsKV-cachemutual-guidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MuGCP is a prompt-learning method that uses a frozen multimodal large language model (MLLM) as a conditional prompt generator for a vision-language model. The paper's central claim is that the MLLM's final-decoder key-value cache, read by a short learnable query, yields instance-specific semantic prompts that model class-embedding distributions better than image features or class descriptions alone. Those semantic prompts are converted into visual conditional prompts by an Attention Mutual-Guidance module, and a Multi-Prompt Fusion mechanism inserts both into the frozen transformer layers of the vision-language model. On 11 datasets in the 16-shot base-to-new setting, MuGCP reports a harmonic mean of 82.03, compared with 80.65 for the previous LLamP method, and it claims higher average accuracy across 14 datasets on few-shot, cross-dataset, and domain generalization. If correct, the method shows that expensive MLLM knowledge can be extracted cheaply and routed into a vision-language model without fine-tuning either model.

What carries the argument

The central mechanism is the KV-cache readout of the MLLM's final decoder layer: for each image, the visual representation $Z_x$ is pushed through the frozen MLLM, the keys $K_x$ and values $V_x$ of the last decoder block are cached, and a learnable query $P_Q$ computes $P_D = D_{\text{last}}^N([K_x, V_x, P_Q])$ (Eq. 2), which is then projected into the vision-language model's text-embedding space as the Semantic Conditional Prompt. The Attention Mutual-Guidance (AMG) module combines self-attention with cross-attention whose keys and values are the vision-language model's image features, producing Visual Conditional Prompts that are tightly coupled to the semantic prompts. Multi-Prompt Fusion (MPF) then inserts SCP and VCP into the frozen transformer layers of both encoders alongside contextual prompts and word/patch embeddings. These three components together let the vision-language model read MLLM knowledge layer by layer, and they are what the ablations credit for the gap between MuGCP and the random-initialization baseline.

What would settle it

Swap in a corrupted cache at test time: build the Semantic Conditional Prompt from the keys and values of a different image, a mismatched caption, or random values while keeping everything else fixed. If the 82.03 harmonic mean is unchanged, instance-specific cache content is not doing the work; if accuracy falls toward the 79.85 of the no-MLLM ablation, the KV-cache retrieval is load-bearing.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the classification-relevant knowledge in a frozen MLLM can be decoded with a few-shot-trained query instead of large-scale fine-tuning. Concretely, the image is passed through the MLLM once; the keys and values of the final decoder layer are cached offline, and a learnable query of length 16 attends to that cache to produce Semantic Conditional Prompts (SCP). The Attention Mutual-Guidance (AMG) module then uses image features as keys and values in cross-attention, turning SCPs into Visual Conditional Prompts (VCP) and allowing semantic and visual prompts to refine each other. The Multi-Prompt Fusion (MPF) mechanism concatenates and adds these prompts with contextual prompts, word embeddings, and patch embeddings inside the frozen CLIP transformer layers. The result, the paper argues, is that the vision-language model's text and image embeddings stay closer to its pre-trained multi-modal space while gaining instance-level semantic detail, which is why the reported 16-shot base-to-new harmonic mean rises to 82.03 and why new-class accuracy improves most on fine-grained datasets.

Load-bearing premise

The load-bearing premise is that the key-value memory left in the MLLM's final decoder layer contains the fine-grained, class-relevant visual semantics of each image, and that a 16-token learned query can pull that information out; if the cache is mostly generic language or the query cannot isolate the useful part, the semantic prompts degrade to noise and the reported gains disappear.

Editorial extensions

If this is right

  • MuGCP reports gains across 1-, 2-, 4-, 8-, and 16-shot settings, with the 8-shot harmonic mean already close to the previous best method's 16-shot result.
  • The largest reported improvements are on fine-grained datasets such as Stanford Cars and Aircraft, where instance-level semantic prompts add discriminating information that class names alone do not carry.
  • Swapping the MLLM between MiniGPT-4 and BLIP-2 yields competitive results, indicating the conditional-prompt readout is not bound to one specific generative model.
  • Trained on 16-shot ImageNet, MuGCP improves over prior prompt methods on most cross-dataset targets and on all three ImageNet-variant domain-shift sets, suggesting the instance-specific prompts transfer rather than overfit.
  • Compared with fine-tuning an MLLM projection layer on all training data, MuGCP approaches comparable numbers on some datasets while using 16 shots and fewer parameters, suggesting the cost of tapping MLLM knowledge can be much lower.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same KV-cache readout could be tested on retrieval or open-vocabulary detection; if instance-specific semantic prompts help there too, the mechanism generalizes beyond classification, which the paper does not examine.
  • The paper does not quantify the cost of running the MLLM on every image to build the cache, so end-to-end inference is likely more expensive than the reported prompt-training time alone suggests.
  • A direct test of the mechanism's story would be to corrupt the cached keys and values at test time; if accuracy is unchanged, the gain comes from something other than the extracted MLLM knowledge.
  • If the mechanism works as claimed, it implies that a frozen generative model can act as a cheap reusable memory for a discriminative model, which would open a route beyond prompt learning toward other few-shot adaptation schemes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. MuGCP is a prompt-learning method for frozen Vision-Language Models (CLIP). For each image, a frozen Multi-modal Large Language Model (MLLM) is run to produce a final-decoder KV-cache (Eq. 1); a learnable query attends to this cache to produce Semantic Conditional Prompts (SCP) via Eq. (2), which are mapped into CLIP's text space through bottleneck layers. An Attention Mutual-Guidance (AMG) module combines SCP with CLIP image features to generate Visual Conditional Prompts (VCP), and a Multi-Prompt Fusion (MPF) mechanism merges SCP, VCP, and contextual prompts inside frozen CLIP transformer layers. Training uses cross-entropy and consistency losses with MLLM/GPT-generated text augmentations. Experiments cover base-to-new generalization, few-shot classification, cross-dataset generalization, and domain generalization on 14 datasets, reporting state-of-the-art harmonic means (82.03 HM vs. 80.65 for LLamP at 16 shots, and 85.82 average 16-shot accuracy).

Significance. If the central mechanism is verified, the idea of decoding frozen MLLM internal knowledge into VLM prompt space is a useful and timely contribution. The paper has clear strengths: extensive evaluation with three seeds and standard deviations, ablations of AMG/MPF/text augmentation, compatibility with two MLLMs, and lower training cost than fine-tuning an MLLM. However, the KV-cache content is not isolated as the active ingredient, the comparison with CLIP-only baselines is not compute-fair because a 7B MLLM is run at test time, and hyperparameters are selected on the same benchmark averages used for the final claims. These issues are load-bearing for the paper's central novelty and for its SOTA claims, so they need to be addressed before the contribution is fully established.

major comments (4)
  1. [III-A, Eq. (2) and Table VI] The ablation does not isolate the KV-cache content. The row 'Ours ✗' (w/o MLLMs, only initialize PD with a normal distribution) removes not only the cache but the entire MLLM pathway, while 'CoCoOp‡' adds the learnable query/projection layers (Lq, Lk, Lv, Lo), the bottleneck F_t, and the MLLM visual encoder relative to CoCoOp. Consequently the +2.18 HM in Table VI could be explained by the extra trainable parameters or by MLLM visual features rather than by the information in Kx, Vx. Please add an ablation that keeps the same query/bottleneck architecture and varies only the cache (e.g., random keys/values, permuted keys/values, or a frozen task-agnostic cache), and provide a quantitative diagnostic (linear probe accuracy on Kx/Vx, or attention-weight analysis) showing that class-discriminative information is actually retrieved by Eq. (2). Without this, the paper's central novelty — decoding MLLM latent knowledge into SCP — is not established.
  2. [IV-A, Table X, Table XII] The comparison with CLIP-only baselines is not compute-fair. MuGCP must run a frozen MiniGPT-4 (7B) forward pass on every test image to compute Kx, Vx for Eq. (1)-(2), whereas all prompt-learning baselines use only CLIP at test time. Table XII reports only training time and parameter counts, not test-time inference cost, so the SOTA comparisons in Tables I-III are not like-for-like. The VLMClassifier comparison in Table X is also overstated: on ImageNet, MuGCP obtains 77.40 vs. VLMClassifier's 88.00 (all-shots), a 10.6-point gap, which is not 'comparable' as claimed in §IV-C. Please report test-time FLOPs/latency, include a CLIP-only baseline augmented with the same MLLM-generated captions or features, and revise the claim accordingly.
  3. [III-A, Eq. (2)] The displayed formula for PD is not a complete transformer decoder operation. A standard attention layer computes softmax(QK^T / sqrt(d)) V with Q = Lq(PQ), K = [Kx; Lk(PQ)], V = [Vx; Lv(PQ)], followed by an output projection Lo. The printed formula omits the softmax, uses ambiguous concatenation/product notation, and does not state how n, dmllms, and the number of attention heads interact with the cached Kx/Vx. Please give exact tensor shapes, normalization, and masking used in the implementation, or replace Eq. (2) with the precise self-attention expression. As written, a reader cannot reproduce the core mechanism.
  4. [IV-C, Fig. 5, Table XI] Hyperparameters (conditional prompt length n=16, contextual prompt length m=4, insertion layers l=12, loss balance factor λ=8, and initialization template 'X X X X {}') are selected by maximizing the final 11-dataset harmonic mean on the same evaluation sets reported in Tables I-III. There is no held-out validation split, and no sensitivity analysis showing that the reported gains are stable across a reasonable range of these choices. This makes the SOTA gaps (e.g., +1.38 HM over LLamP in Table I) difficult to interpret. Please either select hyperparameters on a validation split or provide a stability analysis demonstrating that the main conclusions are unchanged over a range of n, m, l, and λ.
minor comments (6)
  1. [Abstract and Section I] The abstract claims 'outperforms existing state-of-the-art methods on 14 different datasets', but the main base-to-new and few-shot tables cover 11 datasets; clarify exactly which claim applies to which 14 datasets, including the ImageNet variants and the cross-dataset/domain settings.
  2. [Table X vs. Tables V and XVIII] Table X reports ImageNet accuracy 77.40 for Ours, while Table V and Table XVIII report 74.80 for the same 16-shot setting; specify the MLLM and protocol used in each table to resolve the discrepancy.
  3. [Equation (6)] Equation (6) uses 'j=0' and sums to n while n already denotes the conditional prompt length; use j=1..n to avoid off-by-one confusion.
  4. [Section III-B] The paper cites GPT-3.5 usage with reference [49], which is the GPT-4 technical report; please add the correct GPT-3.5 reference or adjust the citation.
  5. [Various] Minor typos and formatting issues: 'Staford Cars' in Section IV-B, 'Methodolgy' in the Section III heading, and the Table VII header for shared vs. non-shared AMG is visually ambiguous; please fix these.
  6. [Table IV] The statement 'outperforms TCP on 6 datasets' is clear, but the boldface in Table IV makes it hard to see which entries are being counted; align the boldface with the claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: MuGCP is a standard learned pipeline validated against external benchmarks; the only self-citation is a non-load-bearing baseline reference.

full rationale

MuGCP's derivation chain is a conventional learned system: frozen CLIP encoders and a frozen MLLM provide cached keys and values, while trainable components (query PQ, bottleneck F_t, AMG, MPF, adapters) are optimized with cross-entropy and consistency losses (Eq. 9). The Semantic Conditional Prompt is not a fitted parameter renamed as a prediction; Eq. 2 defines a conditional computation from the instance's cached Kx and Vx, and all reported numbers are held-out base-to-new, few-shot, cross-dataset, and domain-generalization benchmarks against external methods. No load-bearing step reduces to its own input by construction. The paper's only notable self-citation is [52] (TGP-T-F), which appears as a baseline and in related work but is not used to justify the core mechanism, so it is not load-bearing. The skeptic's concern that Table VI does not isolate whether KV-cache content or the extra parameters drive the gains is an evidential or correctness limitation rather than circular reasoning. Similarly, tuning hyperparameters (n, m, l, lambda, initialization template) on the same benchmark averages is selection bias, which may inflate results, but it is not a circular derivation. The central claim therefore has independent empirical content.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical ML method; it does not derive constants from theory. The load-bearing assumptions are the informativeness of the MLLM KV-cache for instance-level prompts, the non-contamination of MLLM pretraining, and the standard VLM backbone assumptions. No new physical entities are introduced.

free parameters (5)
  • conditional prompt length n = 16
    Ablation in Fig. 5(a) shows new-class accuracy peaks at n=16; chosen by tuning on the 11-dataset benchmark average.
  • contextual prompt length m = 4
    Fig. 5(b): peak average HM at m=4; selected via grid search on the benchmark.
  • prompt insertion layers l = 12 (all transformer layers)
    Fig. 5(d): deeper insertion improves performance; l=12 chosen as optimal.
  • loss balance factor lambda = 8
    Fig. 5(c): lambda=8 gives the best average performance across the 11 datasets.
  • initialization template = X X X X {}
    Table XI: chosen over 'a photo of a {}' because it improves harmonic mean by 0.70 points on the benchmark.
assumptions (4)
  • domain assumption The frozen MLLM decoder's cached keys and values encode task-relevant fine-grained visual semantics that can be probed by a short learned query.
    Central to SCP generation (Eq. 2); supported only by ablations, not by an analysis of the cache content.
  • domain assumption CLIP ViT-B/16 pretrained representations are suitable for prompt-based adaptation.
    Standard in the field; the paper uses CLIP as the frozen VLM backbone.
  • domain assumption MLLM pretraining data is sufficiently disjoint from downstream task data so that instance-level descriptions do not leak test labels.
    The paper asserts MLLMs provide only coarse category info (Sec. I), but does not audit contamination on the 14 benchmarks.
  • domain assumption GPT-3.5-generated text augmentations used during training do not introduce test-set information.
    The prompts include class names and MLLM descriptions from training images only; non-determinism of GPT-3.5 outputs is not discussed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models." pith.science (2026). https://pith.science/paper/XF27YRKV

@misc{pith2026250708410,
  author       = {Pith},
  title        = {Pith review of: Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XF27YRKV}},
  note         = {Machine review of arXiv:2507.08410}
}
read the original abstract

Prompt learning facilitates the efficient adaptation of Vision-Language Models (VLMs) to various downstream tasks. However, it faces two significant challenges: (1) inadequate modeling of class embedding distributions for unseen instances, leading to suboptimal generalization on novel classes; (2) prevailing methodologies predominantly confine cross-modal alignment to the final output layer of vision and text encoders, which fundamentally limits their capacity to preserve topological consistency with pre-trained multi-modal embedding spaces. To this end, we introduce MuGCP (Multi-modal Mutual-Guidance Conditional Prompt Learning), a novel paradigm designed for conditional prompt generation. MuGCP leverages Multi-modal Large Language Models (MLLMs) as conditional prompt learners to adaptively generate Semantic Conditional Prompts (SCP) that incorporate rich, fine-grained high-level semantic knowledge for image instances. To ensure effective alignment and interaction across the multi-modal space of Vision-Language Models (VLMs), we introduce the Attention Mutual-Guidance (AMG) module, which facilitates interactions between visual and semantic information. Through mutual guidance, the AMG module generates Visual Conditional Prompts (VCP), enhancing the model's performance in multi-modal tasks. Additionally, we present a Multi-Prompt Fusion (MPF) mechanism that integrates SCP and VCP with contextual prompts, ensuring seamless coordination among the different prompts and enhancing the modeling of class embeddings and instance-specific knowledge. Our MuGCP outperforms existing state-of-the-art methods on 14 different datasets. The code will be made available after publication.

Figures

Figures reproduced from arXiv: 2507.08410 by the authors.

Figure 1
Figure 1. (a) Comparison of text from LLMs and MLLMs: LLMs generate class [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of Multi-modal Mutual-Guidance Conditional Prompt Learning (MuGCP). MuGCP leverages Multi-modal Large Language Models [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The Generation Process Diagram of Semantic Conditional Prompts. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The performance comparison of our MuGCP with the state-of-the-art methods on few-shot learning, including 1/2/4/8/16-shots on 11 datasets. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: (a) Ablation study of different condition prompt lengths [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: T-SNE plots of image and text embeddings in conditional prompting [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: T-SNE plots of image embeddings and text embeddings in conditional prompting method CoCoOp (top) and Our MuGCP (bottom) on [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Grad-CAM heatmaps for new class generalization on 11 different image recognition datasets for [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

86 extracted references · 47 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  2. [2]

    Why are visually-grounded language models bad at image classification?

    Y . Zhang, A. Unell, X. Wang, D. Ghosh, Y . Su, L. Schmidt, and S. Yeung-Levy, “Why are visually-grounded language models bad at image classification?” arXiv preprint arXiv:2405.18415 , 2024

    arXiv 2024

  3. [3]

    Eva: Exploring the limits of masked visual representation learning at scale,

    Y . Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y . Cao, “Eva: Exploring the limits of masked visual representation learning at scale,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 358–19 369

    2023

  4. [4]

    Conditional prompt learning for vision-language models,

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Conditional prompt learning for vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 16 816–16 825

    2022

  5. [5]

    Learning to prompt for vision-language models,

    ——, “Learning to prompt for vision-language models,” International Journal of Computer Vision , vol. 130, no. 9, pp. 2337–2348, 2022

    2022

  6. [6]

    Maple: Multi-modal prompt learning,

    M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan, “Maple: Multi-modal prompt learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 113–19 122

    2023

  7. [7]

    Tcp: Textual-based class-aware prompt tuning for visual-language model,

    H. Yao, R. Zhang, and C. Xu, “Tcp: Textual-based class-aware prompt tuning for visual-language model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 23 438–23 448

    2024

  8. [8]

    Dept: Decoupled prompt tuning,

    J. Zhang, S. Wu, L. Gao, H. T. Shen, and J. Song, “Dept: Decoupled prompt tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 12 924–12 933

    work page 2024

Show all 86 references
  1. [9]

    Promptkd: Unsupervised prompt distillation for vision-language models,

    Z. Li, X. Li, X. Fu, X. Zhang, W. Wang, and J. Yang, “Promptkd: Unsupervised prompt distillation for vision-language models,” arXiv preprint arXiv:2403.02781, 2024

    2024 arXiv

  2. [10]

    Self-regulating prompts: Foundational model adaptation without forgetting,

    M. U. Khattak, S. T. Wasim, M. Naseer, S. Khan, M.-H. Yang, and F. S. Khan, “Self-regulating prompts: Foundational model adaptation without forgetting,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 15 190–15 200

    2023

  3. [11]

    Consistency-guided prompt learning for vision- language models,

    S. Roy and A. Etemad, “Consistency-guided prompt learning for vision- language models,” arXiv preprint arXiv:2306.01195 , 2024

    2024 arXiv

  4. [12]

    Large language models are good prompt learners for low-shot image classification,

    Z. Zheng, J. Wei, X. Hu, H. Zhu, and R. Nevatia, “Large language models are good prompt learners for low-shot image classification,” arXiv preprint arXiv:2312.04076 , 2024

    2024 arXiv

  5. [13]

    Improved zero-shot classification by adapting vlms with text descriptions,

    O. Saha, G. Van Horn, and S. Maji, “Improved zero-shot classification by adapting vlms with text descriptions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2024, pp. 17 542–17 552

    2024

  6. [14]

    Bilateral adaptive cross-modal fusion prompt learning for clip,

    Q. Wang, K. Yan, and S. Ding, “Bilateral adaptive cross-modal fusion prompt learning for clip,” in ACM Multimedia 2024 , 2024

    2024

  7. [15]

    Unified vision and language prompt learning,

    Y . Zang, W. Li, K. Zhou, C. Huang, and C. C. Loy, “Unified vision and language prompt learning,” arXiv preprint arXiv:2210.07225 , 2022

    2022 arXiv

  8. [16]

    Visual prompt tuning,

    M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in European Conference on Computer Vision. Springer, 2022, pp. 709–727

    2022

  9. [17]

    Enhancing clip with gpt-4: Harnessing visual descriptions as prompts,

    M. Maniparambil, C. V orster, D. Molloy, N. Murphy, K. McGuinness, and N. E. O’Connor, “Enhancing clip with gpt-4: Harnessing visual descriptions as prompts,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 262–271

    2023

  10. [18]

    Prompt distribution learning,

    Y . Lu, J. Liu, Y . Zhang, Y . Liu, and X. Tian, “Prompt distribution learning,” in CVPR, 2022, pp. 5206–5215

    2022

  11. [19]

    Prompt-aligned gradient for prompt tuning,

    B. Zhu, Y . Niu, Y . Han, Y . Wu, and H. Zhang, “Prompt-aligned gradient for prompt tuning,” in CVPR, 2023, pp. 15 659–15 669

    2023

  12. [20]

    Visual-language prompt tuning with knowledge-guided context optimization,

    H. Yao, R. Zhang, and C. Xu, “Visual-language prompt tuning with knowledge-guided context optimization,” in CVPR, 2023, pp. 6757–6767

    2023

  13. [21]

    Gradient-regulated meta-prompt learning for generalizable vision-language models,

    J. Li, M. Gao, L. Wei, S. Tang, W. Zhang, M. Li, W. Ji, Q. Tian, T.-S. Chua, and Y . Zhuang, “Gradient-regulated meta-prompt learning for generalizable vision-language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 2551–2562

    2023

  14. [22]

    Overcoming the pitfalls of vision-language model finetuning for ood generalization,

    Y . Zang, H. Goh, J. Susskind, and C. Huang, “Overcoming the pitfalls of vision-language model finetuning for ood generalization,” arXiv preprint arXiv:2401.15914, 2024

    2024 arXiv

  15. [23]

    Prompt learning via meta-regularization,

    J. Park, J. Ko, and H. J. Kim, “Prompt learning via meta-regularization,” arXiv preprint arXiv:2404.00851 , 2024

    2024 arXiv

  16. [24]

    Extract free dense labels from clip,

    C. Zhou, C. C. Loy, and B. Dai, “Extract free dense labels from clip,” in European Conference on Computer Vision. Springer, 2022, pp. 696–712

    2022

  17. [25]

    Maskclip: Masked self-distillation advances contrastive language-image pretraining,

    X. Dong, J. Bao, Y . Zheng, T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chen et al., “Maskclip: Masked self-distillation advances contrastive language-image pretraining,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023...

    2023

  18. [26]

    Scaling open-vocabulary image segmentation with image-level labels,

    G. Ghiasi, X. Gu, Y . Cui, and T.-Y . Lin, “Scaling open-vocabulary image segmentation with image-level labels,” in European Conference on Computer Vision . Springer, 2022, pp. 540–557

    2022

  19. [27]

    Clip-actor: Text-driven recom- mendation and stylization for animating human meshes,

    K. Youwang, K. Ji-Yeon, and T.-H. Oh, “Clip-actor: Text-driven recom- mendation and stylization for animating human meshes,” in European Conference on Computer Vision . Springer, 2022, pp. 173–191

    2022

  20. [28]

    Motionclip: Exposing human motion generation to clip space,

    G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and D. Cohen-Or, “Motionclip: Exposing human motion generation to clip space,” in European Conference on Computer Vision. Springer, 2022, pp. 358–374

    2022

  21. [29]

    Lerf: Language embedded radiance fields,

    J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik, “Lerf: Language embedded radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 19 729–19 739

    2023

  22. [30]

    Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,

    Y . Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly,” IEEE transactions on pattern analysis and machine intelligence , vol. 41, no. 9, pp. 2251–2265, 2018

    2018

  23. [31]

    Zero-shot text-to-image generation,

    A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. V oss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in International Conference on Machine Learning . PMLR, 2021, pp. 8821–8831

    2021

  24. [32]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 10 684–10 695

    2022

  25. [33]

    Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,

    Q. Yu, J. He, X. Deng, X. Shen, and L.-C. Chen, “Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  26. [34]

    Learning multiple visual do- mains with residual adapters,

    S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual do- mains with residual adapters,” Advances in neural information processing systems, vol. 30, 2017

    2017

  27. [35]

    Task residual for tuning vision-language models,

    T. Yu, Z. Lu, X. Jin, Z. Chen, and X. Wang, “Task residual for tuning vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 10 899–10 909

    2023

  28. [36]

    Graphadapter: Tuning vision-language models with dual knowledge graph,

    X. Li, D. Lian, Z. Lu, J. Bai, Z. Chen, and X. Wang, “Graphadapter: Tuning vision-language models with dual knowledge graph,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  29. [37]

    Tip-adapter: Training-free adaption of clip for few-shot classification,

    R. Zhang, W. Zhang, R. Fang, P. Gao, K. Li, J. Dai, Y . Qiao, and H. Li, “Tip-adapter: Training-free adaption of clip for few-shot classification,” in European Conference on Computer Vision. Springer, 2022, pp. 493–510

    2022

  30. [38]

    Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners,

    R. Zhang, X. Hu, B. Li, S. Huang, H. Deng, Y . Qiao, P. Gao, and H. Li, “Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 15 211–15 222

    2023

  31. [39]

    Scaling up visual and vision-language representation learning with noisy text supervision,

    C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision-language representation learning with noisy text supervision,” in International conference on machine learning . PMLR, 2021, pp. 4904–4916. JOURN...

    2021

  32. [40]

    Clip-adapter: Better vision-language models with feature adapters,

    P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y . Zhang, H. Li, and Y . Qiao, “Clip-adapter: Better vision-language models with feature adapters,” International Journal of Computer Vision , vol. 132, no. 2, pp. 581–595, 2024

    2024

  33. [41]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”Advances in neural information processing systems , vol. 36, 2024

    2024

  34. [42]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning . PMLR, 2022, pp. 12 888–12 900

    2022

  35. [43]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023

    2023 arXiv

  36. [44]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592 , 2023

    2023 arXiv

  37. [45]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning,

    J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V . Chandra, Y . Xiong, and M. Elhoseiny, “Minigpt-v2: large language model as a unified interface for vision-language multi-task learning,” arXiv preprint arXiv:2310.09478 , 2023

    arXiv

  38. [46]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  39. [47]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  40. [48]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” pp. 1877–1901, 2020

    1901

  41. [49]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  42. [50]

    Lit: Zero-shot transfer with locked-image text tuning,

    X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer, “Lit: Zero-shot transfer with locked-image text tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 123–18 133

    2022

  43. [51]

    Coca: Contrastive captioners are image-text foundation models,

    J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: Contrastive captioners are image-text foundation models,” arXiv preprint arXiv:2205.01917, 2022

    2022 arXiv

  44. [52]

    Compound text- guided prompt tuning via image-adaptive cues,

    H. Tan, J. Li, Y . Zhou, J. Wan, Z. Lei, and X. Zhang, “Compound text- guided prompt tuning via image-adaptive cues,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, 2024, pp. 5061–5069

    2024

  45. [53]

    What does a platypus look like? generating customized prompts for zero-shot image classification,

    S. Pratt, I. Covert, R. Liu, and A. Farhadi, “What does a platypus look like? generating customized prompts for zero-shot image classification,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 691–15 701

    2023

  46. [54]

    Knowledge- aware prompt tuning for generalizable vision-language models,

    B. Kan, T. Wang, W. Lu, X. Zhen, W. Guan, and F. Zheng, “Knowledge- aware prompt tuning for generalizable vision-language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 670–15 680

    2023

  47. [55]

    Visual classification via description from large language models,

    S. Menon and C. V ondrick, “Visual classification via description from large language models,” arXiv preprint arXiv:2210.07183 , 2022

    2022 arXiv

  48. [56]

    Learning concise and descriptive attributes for visual recognition,

    A. Yan, Y . Wang, Y . Zhong, C. Dong, Z. He, Y . Lu, W. Y . Wang, J. Shang, and J. McAuley, “Learning concise and descriptive attributes for visual recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3090–3100

    2023

  49. [57]

    Language in a bottle: Language model guided concept bottlenecks for interpretable image classification,

    Y . Yang, A. Panagopoulou, S. Zhou, D. Jin, C. Callison-Burch, and M. Yatskar, “Language in a bottle: Language model guided concept bottlenecks for interpretable image classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023,...

    2023

  50. [58]

    Commoncanvas: An open diffusion model trained with creative-commons images,

    A. Gokaslan, A. F. Cooper, J. Collins, L. Seguin, A. Jacobson, M. Patel, J. Frankle, C. Stephenson, and V . Kuleshov, “Commoncanvas: An open diffusion model trained with creative-commons images,” arXiv preprint arXiv:2310.16825, 2023

    arXiv

  51. [59]

    Enhancing clip with a third modality,

    E. Tsaprazlis, G. Smyrnis, A. G. Dimakis, and P. Maragos, “Enhancing clip with a third modality,” 2024

    2024

  52. [60]

    Democratizing fine-grained visual recognition with large language models,

    M. Liu, S. Roy, W. Li, Z. Zhong, N. Sebe, and E. Ricci, “Democratizing fine-grained visual recognition with large language models,” arXiv preprint arXiv:2401.13837, 2024

    2024 arXiv

  53. [61]

    Fine- tuned clip models are efficient video learners,

    H. Rasheed, M. U. Khattak, M. Maaz, S. Khan, and F. S. Khan, “Fine- tuned clip models are efficient video learners,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 6545–6554

    2023

  54. [62]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

    2009

  55. [63]

    3d object representations for fine-grained categorization,

    J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine-grained categorization,” in Proceedings of the IEEE international conference on computer vision workshops , 2013, pp. 554–561

    2013

  56. [64]

    Ucf101: A dataset of 101 human actions classes from videos in the wild,

    K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  57. [65]

    Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,

    L. Fei-Fei, R. Fergus, and P. Perona, “Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” in 2004 conference on computer vision and pattern recognition workshop . IEEE, 2004, pp. 178–178

    2004

  58. [66]

    Automated flower classification over a large number of classes,

    M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing . IEEE, 2008, pp. 722–729

    2008

  59. [67]

    Sun database: Large-scale scene recognition from abbey to zoo,

    J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database: Large-scale scene recognition from abbey to zoo,” in 2010 IEEE computer society conference on computer vision and pattern recognition . IEEE, 2010, pp. 3485–3492

    2010

  60. [68]

    Describing textures in the wild,

    M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 3606–3613

    2014

  61. [69]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,

    P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 7, pp. 2217–2226, 2019

    2019

  62. [70]

    Fine- grained visual classification of aircraft,

    S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine- grained visual classification of aircraft,” arXiv preprint arXiv:1306.5151 , 2013

    2013 arXiv

  63. [71]

    Cats and dogs,

    O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, “Cats and dogs,” in 2012 IEEE conference on computer vision and pattern recognition . IEEE, 2012, pp. 3498–3505

    2012

  64. [72]

    Food-101–mining discriminative components with random forests,

    L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101–mining discriminative components with random forests,” in Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 . Springer, 2014, pp. 446–461

    2014

  65. [73]

    Natural adversarial examples,

    D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural adversarial examples,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 15 262–15 271

    2021

  66. [74]

    Do imagenet classifiers generalize to imagenet?

    B. Recht, R. Roelofs, L. Schmidt, and V . Shankar, “Do imagenet classifiers generalize to imagenet?” in International conference on machine learning. PMLR, 2019, pp. 5389–5400

    2019

  67. [75]

    Learning robust global representations by penalizing local predictive power,

    H. Wang, S. Ge, Z. Lipton, and E. P. Xing, “Learning robust global representations by penalizing local predictive power,” Advances in Neural Information Processing Systems , vol. 32, 2019

    2019

  68. [76]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101 , 2017

    2017 arXiv

  69. [77]

    Vl-mamba: Exploring state space models for multimodal learning,

    Y . Qiao, Z. Yu, L. Guo, S. Chen, Z. Zhao, M. Sun, Q. Wu, and J. Liu, “Vl-mamba: Exploring state space models for multimodal learning,” arXiv preprint arXiv:2403.13600, 2024

    2024 arXiv

  70. [78]

    Grad-cam: Visual explanations from deep networks via gradient-based localization,

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 618–626

    2017

  71. [79]

    Efficiently scaling transformer inference,

    R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean, “Efficiently scaling transformer inference,” Proceedings of Machine Learning and Systems , vol. 5, pp. 606–624, 2023

    2023

  72. [80]

    Transformers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 conference on empirical methods in natural language processing: system dem...

    2020

  73. [81]

    Modality- consistent prompt tuning with optimal transport,

    H. Ren, F. Tang, H. Zheng, H. Zhao, D. Guo, and Y . Chang, “Modality- consistent prompt tuning with optimal transport,” IEEE Transactions on Circuits and Systems for Video Technology , 2024

    2024

  74. [82]

    Hierarchy- aware interactive prompt learning for few-shot classification,

    X. Yin, J. Wu, W. Yang, X. Zhou, S. Zhang, and T. Zhang, “Hierarchy- aware interactive prompt learning for few-shot classification,” IEEE Transactions on Circuits and Systems for Video Technology , 2024

    2024

  75. [83]

    Language-driven visual consensus for zero-shot semantic segmentation,

    Z. Zhang, W. Ke, Y . Zhu, X. Liang, J. Liu, Q. Ye, and T. Zhang, “Language-driven visual consensus for zero-shot semantic segmentation,” IEEE Transactions on Circuits and Systems for Video Technology , 2024

    2024

  76. [84]

    Pedestrian attribute recognition via clip based prompt vision-language fusion,

    X. Wang, J. Jin, C. Li, J. Tang, C. Zhang, and W. Wang, “Pedestrian attribute recognition via clip based prompt vision-language fusion,” IEEE Transactions on Circuits and Systems for Video Technology , 2024. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

    2024

  77. [85]

    Understanding and mitigating overfitting in prompt tuning for vision-language models,

    C. Ma, Y . Liu, J. Deng, L. Xie, W. Dong, and C. Xu, “Understanding and mitigating overfitting in prompt tuning for vision-language models,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 9, pp. 4616–4629, 2023

    2023

  78. [86]

    Clipood: Generalizing clip to out-of-distributions,

    Y . Shu, X. Guo, J. Wu, X. Wang, J. Wang, and M. Long, “Clipood: Generalizing clip to out-of-distributions,” in International Conference on Machine Learning . PMLR, 2023, pp. 31 716–31 731. APPENDIX Here, we provide additional discussions and detailed results that could not be...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.