Pith. sign in

REVIEW 3 major objections 6 minor 58 references

Integrated Structural Prompt Learning for Vision-Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Integrated Structural Prompt (ISP) claims that wiring learnable prompts to frozen tokens, within and across modalities, plus a sample-difficulty loss weight, lifts CLIP's average base-to-new harmonic mean to 80.70% on 11 datasets, above…

desk verdict The cross-structural prompt module is defined with invalid matrix subtraction (non-square matrices transposed), so the method cannot be run as written; otherwise this is a competent, incremental prompt-tuning paper with a likely fixable flaw. read the letter →

arxiv 2507.05677 v2 pith:LPHBMSJE submitted 2025-07-08 cs.CV

Jiahui Wang , Qin Xu , Bo Jiang , Bin Luo This is my paper · ORCID
classification cs.CV
keywords promptlearningvision-languagemodelsCLIPfew-shottransfercross-modalinteractionsampledifficultyweightingstructure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that CLIP's few-shot transfer improves when learnable prompts are treated as part of a structure rather than as isolated vectors. It introduces self-structural prompt modules, in which prompts attend to the most responsive frozen tokens of the same modality, and cross-structural prompt modules, in which prompts from one modality exchange information with tokens of the other through an affinity graph. A third component, sample probing, reweights each sample's loss by how much the prompted model's confidence differs from zero-shot CLIP's confidence, down-weighting easy samples. The paper reports that this combination raises the average harmonic mean of base and new class accuracy to 80.70% across 11 datasets, with 66.75% average accuracy in cross-dataset evaluation, and it attributes the gains to better prompt–token structure and to reduced overfitting on simple samples.

What carries the argument

The load-bearing objects are three modules inserted after frozen CLIP encoder layers. The self-structural prompt is cross-attention between $L$ learnable prompt vectors and the top-response frozen tokens of the same modality, with a two-layer MLP after attention. The cross-structural prompt computes cosine affinity matrices $A_{vt} = \cos(P^v_l, W_l)$ and $A_{tv} = \cos(P^t_l, X_l)$, defines prompt-to-prompt similarities $A_{vv} = \exp(-\beta\|A_{vt} - A_{vt}^\top\|^2)$ and $A_{tt} = \exp(-\beta\|A_{tv} - A_{tv}^\top\|^2)$, and runs one graph-convolution layer over the prompt set to refine $P^v_l$ and $P^t_l$. The sample-probing module computes a per-sample loss weight from the relative gap between the zero-shot CLIP probability and the prompted-model probability, and applies it to the cross-entropy term while leaving the regularization term unchanged. Only the prompts, attention projections, MLPs, and graph-convolution weights are trained; the CLIP backbone stays frozen.

What would settle it

Run the released code or reimplement Eqs. (13)-(18) literally with $L_v=4$, $L_t=6$, $N=77$, and $M=50$: the subtraction in Eq. (15) will fail because $A_{vt}$ is $4 \times 77$ and $A_{vt}^\top$ is $77 \times 4$. That failure settles that the method as written cannot be executed; if a modified operation is used instead, the reported 80.70 average harmonic mean must be reproduced from that modified operation to support the paper's claim.

Watch

Extended reading notes

Core claim

The central claim is that structural interaction between learnable prompts and frozen tokens, within and across modalities, is a usable and beneficial resource for prompt tuning of CLIP. Within each modality, the self-structural prompt selects the tokens with the largest squared channel responses and applies cross-attention so the learnable prompts absorb class-specific token information while retaining domain knowledge. Across modalities, the cross-structural prompt builds affinity matrices between visual prompts and text tokens and between text prompts and visual tokens, uses those matrices to define prompt-to-prompt similarities $A_{vv}$ and $A_{tt}$, and applies a single graph-convolution layer to refine both prompt sets. The sample-probing module then sets a per-sample loss weight $\alpha = (2|p(y|x') - p(y|x)| / |p(y|x') + p(y|x)|)^\gamma$, clipped to be at least 1, so samples that zero-shot CLIP already handles confidently receive less training pressure. The paper reports an 80.70 average harmonic mean on the 11-dataset base-to-new benchmark, 66.75% cross-dataset accuracy, and competitive domain generalization, with ablations indicating each module contributes.

Load-bearing premise

The method's cross-structural module assumes the affinity matrices $A_{vt}$ and $A_{tv}$ are square enough that $A_{vt}$ minus its transpose and $A_{tv}$ minus its transpose are valid subtractions; with 4 visual prompts, 6 text prompts, roughly 77 image tokens, and roughly 50 text tokens, $A_{vt}$ is $4 \times 77$ and $A_{tv}$ is $6 \times 50$, so Equations (15)-(16) are undefined as written.

Editorial extensions

If this is right

  • The sample-probing weight is a drop-in reweighting scheme: it modifies only the loss, and inference continues to use the prompted model's probability $p(y|x)$.
  • If the reported results reproduce, prompt tuning does not need adapters or new backbone parameters to benefit from cross-modal structure; the gains come from the prompt and token organization.
  • The cross-dataset result (66.75% average) implies that prompts tuned with structural modules on ImageNet transfer to unseen datasets better than text-only prompt methods such as CoOp and CoCoOp.
  • The ablation table places the largest single-component gain on the self-structural prompt, which alone raises harmonic mean from 78.31 to 80.20, suggesting token–prompt attention carries most of the benefit.
  • The FGVC Aircraft base accuracy jump (from 40.57 with MMA to 48.40 with ISP) suggests the structural modules are particularly helpful on fine-grained tasks with difficult classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The cross-structural equations as printed are not executable at the stated sizes: with $L_v=4$ and $L_t=6$, $A_{vt}$ is $4 \times 77$ and $A_{tv}$ is $6 \times 50$, so $A_{vt} - A_{vt}^\top$ and $A_{tv} - A_{tv}^\top$ are undefined; a reproduction attempt must first resolve whether this is a typo, a padding or slicing step, or a different intended operation.
  • One testable extension is to apply sample probing alone, without the structural modules, on top of CoOp or MaPLe; if it transfers, the difficulty-based reweighting is an independent mechanism, not a byproduct of prompt structure.
  • The top-K token selection is a sparsity claim; comparing it against random token selection of the same size would show whether the choice of discriminative tokens, rather than the extra parameters, drives the self-structural gain.
  • The cross-modal affinity construction could be replaced with a dimensionally correct bilinear similarity between prompts and tokens; if that change preserves the reported numbers, the published form of Eqs. (15)-(16) is not the operative mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Integrated Structural Prompt (ISP) for adapting CLIP to few-shot image classification. ISP consists of three components: a self-structural prompt module that uses cross-attention between learnable prompts and selected frozen tokens, a cross-structural prompt module that builds affinity matrices between prompts and cross-modal tokens and then applies graph convolution to refine prompts, and a sample probing module that assigns per-sample loss weights based on the disagreement between prompted predictions and zero-shot CLIP predictions. The method is evaluated on base-to-new generalization, cross-dataset evaluation, and domain generalization using CLIP ViT-B/16, reporting state-of-the-art harmonic means (e.g., 80.70 on 11 base-to-new datasets) and competitive cross-dataset and domain-generalization accuracies. The paper also includes ablations on prompt lengths, the sample-probing scaling factor, and the number of layers where the modules are applied.

Significance. If the method works as described, the paper would make a useful contribution to prompt learning for vision-language models by explicitly modeling prompt-token and cross-modal structural relationships and by proposing a difficulty-aware loss weighting scheme. The benchmark coverage is broad, and the arithmetic in Tables 1-3 appears internally consistent. However, the current manuscript has a load-bearing technical flaw: the core cross-structural equations, Eqs. (15)-(16), are dimensionally invalid under the paper's own hyperparameters, and no code is supplied to disambiguate the intended computation. Because the cross-structural module is one of the two named technical contributions, the empirical claims cannot currently be verified as written. The self-structural module also contains a shape mismatch in Eq. (10). These issues must be resolved before the results can be accepted.

major comments (3)
  1. [§3.4, Eqs. (13)-(16)] Eq. (13) defines Avt ∈ R^{Lv×N} and Eq. (15) computes exp(−β∥Avt − A_vt^T∥_2^2). Under the implementation choices in §4.1.2 (Lv=4, Lt=6) and standard CLIP ViT-B/16 token counts, Avt is not square: for example, Lv=4 while N is the text-token count (roughly 77 for a default template), so A_vt^T ∈ R^{N×Lv} and the subtraction Avt − A_vt^T is undefined. The same problem affects Atv in Eq. (16), where Lt=6 and M is the visual token count. Since Eqs. (17)-(18) use Avv and Att as graph adjacency matrices, the entire cross-structural module has no well-defined input as written. The paper provides no code or alternative definition that would disambiguate the intended formula, so the ablation claim in Table 4 that cross-structural prompting contributes gains cannot be checked. Please supply corrected equations and, ideally, a runnable implementation.
  2. [§3.3, Eqs. (9)-(10)] Eq. (10) writes fP_l^v' = CrossAttn(LN(P_l^v), LN(x̃), LN(x̃)) + x̃. If CrossAttn is the standard cross-attention operation with P_l^v as queries and x̃ as keys/values, its output has shape Lv×d_v, whereas x̃ ∈ R^{Lt×d_v}; with Lv=4 and Lt=6, the addition is dimensionally invalid. The surrounding text also first says to select Lv visual tokens and then says to select Lt tokens. This needs to be fixed for the self-structural module to be executable as described.
  3. [§3.5, Eq. (21)] Eq. (21) defines α = (2|p(y|x')−p(y|x)| / |p(y|x')+p(y|x)|)^γ. For easy samples, where the zero-shot and prompted probabilities on the ground-truth class are both high and close, α is below 1. Applying max(α,1) then sets the weight to 1, so the promised down-weighting of simple samples does not occur; only hard samples with α>1 are affected. If the intended operation is min(α,1), or clipping to a bounded interval, please state it explicitly and re-run the corresponding ablation.
minor comments (6)
  1. [Abstract and §3.5] The phrase 'preventing the mode from overfitting' should read 'preventing the model from overfitting.'
  2. [§2 and Table 1] The name 'CoOpOp' appears to be a typo for 'CoCoOp'; please check all occurrences.
  3. [§3.3, Eq. (10)] The reference to 'Eq 5' after Eq. (10) should be to Eq. (9), the cross-attention formula.
  4. [§3.4, Eq. (12)] Under the ViT-B/16 backbone used in the experiments, the visual and text feature dimensions are both 768, so the DCT-based dimensionality reduction in Eq. (12) is vacuous; the motivation and the equation should be clarified or adjusted.
  5. [Table 4 and §4.3.1] The table headers SP, SS, and CS are not defined in the table; in §3.3 SP is used for self-structural prompt, but in the ablation it appears to denote sample probing, which is confusing.
  6. [§4.1.2] The paper reports averages over three seeds but gives no error bars or per-seed results; given the small differences in several comparisons (e.g., Table 3 on ImageNet-V2 and ImageNet-Sketch), some measure of variance would strengthen the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ISP is an empirical prompt-learning method whose losses and reweighting depend on fixed CLIP features and labels, not on its own outputs.

full rationale

The paper's derivation chain is self-contained against external benchmarks. The central output (base/new accuracies, harmonic means, and ablation deltas in Tables 1-4) is produced by training prompt vectors with a cross-entropy loss and a regularization term (Eq. 22), where the reported target metrics are never used as inputs. The self-structural prompt (Eqs. 8-11) is a cross-attention aggregation of prompt and token features; the cross-structural prompt (Eqs. 13-18) is a graph-convolution refinement; neither defines its output in terms of the reported metric. The sample-probing weight alpha (Eq. 21) compares the prompted prediction p(y|x) with the fixed zero-shot CLIP prediction p(y|x'), a pretrained reference, and the weight only scales the cross-entropy term in Eq. 23; this is a reweighting heuristic, not a fitted prediction renamed as a result. The regularization toward frozen CLIP features (Eq. 22) is a standard feature-stability term and does not encode the target benchmark numbers. No load-bearing claim relies on a self-citation: reference [45] is external prior work for token selection, and reference [38] is an unrelated prior paper by overlapping authors. No uniqueness theorem or ansatz is imported via the authors' own citations. The dimension mismatch in Eqs. (15)-(16), where Avt is Lv by N and A_vt^T is N by Lv so the subtraction is undefined under the stated Lv=4, Lt=6 and CLIP token counts, is a potential correctness or executability defect, not a circularity, and does not raise the circularity score. Therefore no circular step is identified.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or ontological entities. The central claim rests on several domain assumptions and six fitted hyperparameters, none of which is justified by theory. The most serious issue is the implicit assumption that the affinity matrices in Eqs (15)-(16) are conformable for subtraction, which is false under the paper's own settings.

free parameters (6)
  • Text prompt length L_t = 6
    Selected in Table 5 by sweeping L_t = 2..10 on the base-to-new benchmark; performance peaks at 6.
  • Visual prompt length L_v = 4
    Selected in Table 5 by sweeping L_v = 2..10 with L_t fixed at 6; peak HM 80.70 at 4.
  • Sample probing scaling gamma = 0.3
    Selected in Table 6 by sweeping gamma in [0.1, 2.0]; 0.3 gives the best HM. The formula in Eq (21) determines per-sample loss weights.
  • Affinity graph scaling beta = 10
    Set in Eqs (15)-(16) as 'usually fixed to 10'; no ablation or sensitivity analysis is provided.
  • Regularization weights omega_v and omega_t = not reported
    Used in Eq (22) for feature regularization, but their values are never stated, so the final objective is not fully specified.
  • Number of encoder layers for SSP/CSP = 1-12 for base-new, 10-12 for cross-dataset and domain generalization
    Chosen based on Figure 3 ablation, which shows new-class accuracy decreases when layers 2-6 are added and improves from layer 10 onward; final settings differ by benchmark.
assumptions (5)
  • domain assumption Only a few tokens are discriminative for classification, justifying top-k selection in Eq (8)
    Borrowed from SEP [45]; if discarded tokens carry class-relevant information, the self-structural prompt will ignore it. Stated in Section 3.3.
  • domain assumption Frozen CLIP features are a valid regularizer for preserving general knowledge (Eq (22))
    The regularization loss pulls prompted features toward zero-shot CLIP features; this assumes CLIP's original features are the correct target and that reducing cosine distance does not harm task-specific adaptation. Section 3.6.
  • domain assumption Zero-shot CLIP confidence is a valid measure of sample difficulty (Eq (21))
    The sample probing downweights samples where zero-shot and prompted probabilities agree; this assumes agreement implies 'easy' and that the ground-truth label probability is the right signal. Section 3.5.
  • domain assumption DCT low-frequency truncation preserves discriminative visual information across modalities
    Visual tokens are reduced to text dimension d_t using DCT/IDCT with no ablation or error analysis; if channel-dropping removes class-discriminative signal, cross-structural transfer is degraded. Section 3.4, Eq (12).
  • ad hoc to paper Eqs (15)-(16) define a valid affinity graph
    The graph-convolution step in Eqs (17)-(18) assumes Avv and Att are well-defined square matrices; as written the expressions contain dimension mismatches, so this axiom is not met.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Integrated Structural Prompt Learning for Vision-Language Models." pith.science (2026). https://pith.science/paper/LPHBMSJE

@misc{pith2026250705677,
  author       = {Pith},
  title        = {Pith review of: Integrated Structural Prompt Learning for Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LPHBMSJE}},
  note         = {Machine review of arXiv:2507.05677}
}
read the original abstract

Prompt learning methods have significantly extended the transferability of pre-trained Vision-Language Models (VLMs) like CLIP for various downstream tasks. These methods adopt handcraft templates or learnable vectors to provide text or image instructions in fine-tuning VLMs. However, most existing works ignore the structural relationships between learnable prompts and tokens within and between modalities. Moreover, balancing the performance of base and new classes remains a significant challenge. In this paper, we propose an Integrated Structural Prompt (ISP) for VLMs to enhance the interaction of information representations between the text and image branches. ISP introduces self-structural and cross-structural prompt modules to model the structural relationships between learnable prompts and frozen tokens within and across modalities. This enables efficient information transfer while preserving feature stability. Additionally, we propose a sample probing module that dynamically adjusts loss coefficients based on sample difficulty, preventing the mode from overfitting to simple samples and improving generalization ability to new classes. Extensive experiments on three widely used settings: base-to-new generalization, cross-dataset evaluation, and domain generalization demonstrate that the proposed ISP achieves competitive performance against state-of-the-art methods.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 52 canonical work pages

  1. [1]

    Advances in neural information processing systems 35, 23716--23736 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., others : Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, 23716--23736 (2022)

    work page 2022

  2. [2]

    In: Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13

    Bossard, L., Guillaumin, M., Van Gool, L.: Food-101--mining discriminative components with random forests. In: Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13. pp. 446--461. Springer (2014)

    work page 2014

  3. [3]

    ICLR (2023)

    Chen, G., Yao, W., Song, X., Li, X., Rao, Y., Zhang, K.: Plot: Prompt Learning with Optimal Transport for Vision - Language Models . ICLR (2023)

    work page 2023

  4. [4]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3606--3613 (2014)

    work page 2014

  5. [5]

    In: 2009 IEEE Conference on Computer Vision and Pattern Recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition . pp. 248--255 (2009)

    work page 2009

  6. [6]

    In: International Conference on Learning Representations

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An Image is Worth 16x16 Words : Transformers for Image Recognition at Scale . In: International Conference on Learning Representations . vol. abs/2010.11929 (2020)

    arXiv 2020

  7. [7]

    In: 2004 conference on computer vision and pattern recognition workshop

    Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In: 2004 conference on computer vision and pattern recognition workshop. pp. 178--178. IEEE (2004)

    work page 2004

  8. [8]

    In: European Conference on Computer Vision

    Feng, C., Zhong, Y., Jie, Z., Chu, X., Ren, H., Wei, X., Xie, W., Ma, L.: Promptdet: Towards open-vocabulary detection using uncurated images. In: European Conference on Computer Vision. pp. 701--717. Springer (2022)

    work page 2022

Show all 58 references
  1. [9]

    Proceedings of the AAAI Conference on Artificial Intelligence 38(3), 1815--1823 (2024)

    Gao, J., Ruan, J., Xiang, S., Yu, Z., Ji, K., Xie, M., Liu, T., Fu, Y.: Lamm: Label Alignment for Multi - Modal Prompt Learning . Proceedings of the AAAI Conference on Artificial Intelligence 38(3), 1815--1823 (2024)

    2024

  2. [10]

    International Journal of Computer Vision 132(2), 581--595 (2024)

    Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., Qiao, Y.: Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision 132(2), 581--595 (2024)

    2024

  3. [11]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) (2014)

    Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation . In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) (2014)

    2014

  4. [12]

    arXiv preprint arXiv:2104.13921 (2021)

    Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 (2021)

    2021 arXiv

  5. [13]

    In: Computer Vision and Pattern Recognition ( CVPR )

    He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition . In: Computer Vision and Pattern Recognition ( CVPR ). pp. 770--778 (2016)

    2016

  6. [14]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    He, W., Jamonnak, S., Gou, L., Ren, L.: Clip-s4: Language-guided self-supervised semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11207--11216 (2023)

    2023

  7. [15]

    IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(7), 2217--2226 (2019)

    Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(7), 2217--2226 (2019)

    2019

  8. [16]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al.: The many faces of robustness: A critical analysis of out-of-distribution generalization. In: Proceedings of the IEEE/CVF international conference on compu...

    2021

  9. [17]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural adversarial examples. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15262--15271 (2021)

    2021

  10. [18]

    In: International conference on machine learning

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: International conference on machine learning. pp. 2790--2799. PMLR (2019)

    2019

  11. [19]

    arXiv preprint arXiv:2106.09685 (2021)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  12. [20]

    In: AAAI Conference on Artificial Intelligence ( AAAI )

    Hu, L., Gao, L., Liu, Z., Pun, C.M., Feng, W.: Comma: Co-articulated Multi - Modal Learning . In: AAAI Conference on Artificial Intelligence ( AAAI ). pp. 2238--2246 (2024)

    2024

  13. [21]

    Advances in Neural Information Processing Systems 36, 72096--72109 (2023)

    Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O.K., Patra, B., others : Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems 36, 72096--72109 (2023)

    2023

  14. [22]

    arXiv preprint arXiv:2110.08484 (2021)

    Jin, W., Cheng, Y., Shen, Y., Chen, W., Ren, X.: A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models. arXiv preprint arXiv:2110.08484 (2021)

    2021 arXiv

  15. [23]

    In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR )

    Khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: Maple: Multi- Modal Prompt Learning . In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR ). pp. 19113--19122 (2023)

    2023

  16. [24]

    In: IEEE International Conference on Computer Vision ( ICCV )

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Doll \' a r, P., Girshick, R.B.: Segment Anything . In: IEEE International Conference on Computer Vision ( ICCV ). pp. 3992--4003 (2023)

    2023

  17. [25]

    In: International Conference on Machine Learning ( ICML )

    Li, J., Li, D., Xiong, C., Hoi, S.C.H.: Blip: Bootstrapping Language - Image Pre -training for Unified Vision - Language Understanding and Generation . In: International Conference on Machine Learning ( ICML ). pp. 12888--12900 (2022)

    2022

  18. [26]

    ACM Computing Surveys 55(9), 1--35 (2023)

    Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55(9), 1--35 (2023)

    2023

  19. [27]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Lu, Y., Liu, J., Zhang, Y., Liu, Y., Tian, X.: Prompt distribution learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5206--5215 (2022)

    2022

  20. [28]

    arXiv preprint arXiv:1306.5151 (2013)

    Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)

    2013 arXiv

  21. [29]

    In: 2008 Sixth Indian conference on computer vision, graphics & image processing

    Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: 2008 Sixth Indian conference on computer vision, graphics & image processing. pp. 722--729. IEEE (2008)

    2008

  22. [30]

    In: 2012 IEEE conference on computer vision and pattern recognition

    Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.: Cats and dogs. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 3498--3505. IEEE (2012)

    2012

  23. [31]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., others : Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748--8763. PMLR (2021)

    2021

  24. [32]

    Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do imagenet classifiers generalize to imagenet? In: International conference on machine learning. pp. 5389--5400. PMLR (2019)

    2019

  25. [33]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)

    Redmon, J.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)

    2016

  26. [34]

    In: Conference on Neural Information Processing Systems ( NeurIPS ) (2022)

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., Jitsev, J.: Laion-5b: An open large-scale dataset for training next genera...

    2022

  27. [35]

    arXiv preprint arXiv:1212.0402 (2012)

    Soomro, K.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)

    2012 arXiv

  28. [36]

    In: IEEE/ CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (2024)

    Tang, Y., Lin, Z., Wang, Q., Zhu, P., Hu, Q.: Amu- Tuning : Effective Logit Bias for CLIP -based Few -shot Learning . In: IEEE/ CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (2024)

    2024

  29. [37]

    Advances in Neural Information Processing Systems 32 (2019)

    Wang, H., Ge, S., Lipton, Z., Xing, E.P.: Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems 32 (2019)

    2019

  30. [38]

    IEEE Transactions on Image Processing 33, 4529--4542 (2024)

    Wang, J., Xu, Q., Jiang, B., Luo, B., Tang, J.: Multi- Granularity Part Sampling Attention for Fine - Grained Visual Classification . IEEE Transactions on Image Processing 33, 4529--4542 (2024)

    2024

  31. [39]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Wang, Y., Jiang, X., Cheng, D., Li, D., Zhao, C.: Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision - Language Models . In: Proceedings of the AAAI Conference on Artificial Intelligence . vol. 38, pp. 5749--5757 (2024)

    2024

  32. [40]

    In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (2020)

    Wang, Y., Zhang, J., Kan, M., Shan, S., Chen, X.: Self- Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation . In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (2020)

    2020

  33. [41]

    In: 2010 IEEE computer society conference on computer vision and pattern recognition

    Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: 2010 IEEE computer society conference on computer vision and pattern recognition. pp. 3485--3492. IEEE (2010)

    2010

  34. [42]

    In: IEEE/ CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (2024)

    Yang, L., Zhang, R.Y., Wang, Y., Xie, X.: Mma: Multi- Modal Adapter for Vision - Language Models . In: IEEE/ CVF Conference on Computer Vision and Pattern Recognition ( CVPR ) (2024)

    2024

  35. [43]

    In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR )

    Yao, H., Zhang, R., Xu, C.: Visual- Language Prompt Tuning With Knowledge - Guided Context Optimization . In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR ). pp. 6757--6767 (2023)

    2023

  36. [44]

    In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR )

    Yao, H., Zhang, R., Xu, C.: Tcp:textual-based Class -aware Prompt tuning for Visual - Language Model . In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR ). pp. 23438--23448 (2024)

    2024

  37. [45]

    arXiv abs/2405.15549 (2024)

    Yao, H., Zhang, R., Yu, L., Xu, C.: Sep: Self- Enhanced Prompt Tuning for Visual - Language Model . arXiv abs/2405.15549 (2024)

    2024 arXiv

  38. [46]

    In: International Conference on Learning Representations ( ICLR ) (2022)

    Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., Xu, C.: Filip: Fine-grained Interactive Language - Image Pre - Training . In: International Conference on Learning Representations ( ICLR ) (2022)

    2022

  39. [47]

    In: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition

    Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: Lit: Zero-shot transfer with locked-image text tuning. In: Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. pp. 18123--18133 (2022)

    2022

  40. [48]

    In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR )

    Zhang, J., Wu, S., Gao, L., Shen, H.T., Song, J.: Dept: Decoupled Prompt Tuning . In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition ( CVPR ). pp. 12924--12933 (2024)

    2024

  41. [49]

    In: European Conference on Computer Vision ( ECCV )

    Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., Li, H.: Tip- Adapter : Training- Free Adaption of CLIP for Few - Shot Classification . In: European Conference on Computer Vision ( ECCV ). pp. 493--510 (2022)

    2022

  42. [50]

    IEEE Transactions on Image Processing 33, 1348--1360 (2024)

    Zhao, C., Wang, Y., Jiang, X., Shen, Y., Song, K., Li, D., Miao, D.: Learning Domain Invariant Prompt for Vision - Language Models . IEEE Transactions on Image Processing 33, 1348--1360 (2024)

    2024

  43. [51]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al.: Regionclip: Region-based language-image pretraining. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16793--16803 (2022)

    2022

  44. [52]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 40(6), 1452--1464 (2018)

    Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 Million Image Database for Scene Recognition . IEEE Transactions on Pattern Analysis and Machine Intelligence 40(6), 1452--1464 (2018)

    2018

  45. [53]

    In: European Conference on Computer Vision

    Zhou, C., Loy, C.C., Dai, B.: Extract free dense labels from clip. In: European Conference on Computer Vision. pp. 696--712. Springer (2022)

    2022

  46. [54]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision-language models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16816--16825 (2022)

    2022

  47. [55]

    International Journal of Computer Vision 130(9), 2337--2348 (2022)

    Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to Prompt for Vision - Language Models . International Journal of Computer Vision 130(9), 2337--2348 (2022)

    2022

  48. [56]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhu, B., Niu, Y., Han, Y., Wu, Y., Zhang, H.: Prompt-aligned gradient for prompt tuning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15659--15669 (2023)

    2023

  49. [57]

    Reference title

    Author A, Author B, Author C. Reference title. Journal, Year, Vol: Number or pages

  50. [58]

    Reference title

    Author A, Author B, Author C, et al. Reference title. In: Proceedings of Conference, Place, Year. Number or pages

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.