Pith. sign in

REVIEW 2 major objections 6 minor 56 references

LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matching

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that CLIP's image-level alignment misses actions, and that LLM-generated action triplets and state descriptions, injected through two-stage multi-modal prompt tuning, recover that fine-grained action perception and…

desk verdict Solid prompt-tuning paper with a real confound: inference-time text-conditioned image features, not LLM action knowledge, may explain much of the reported gain. read the letter →

arxiv 2506.23502 v2 pith:XYHV26NL submitted 2025-06-30 cs.CV

Mengxiao Tian , Xinxiao Wu , Shuo Yang This is my paper
classification cs.CV
keywords action-awareprompttuningimage-textmatchingCLIPlargelanguagemodelknowledgefine-grainedvisualunderstandingvision-languageretrievalactiontripletsmulti-modal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLIP aligns an image with a whole caption, so it often misses what is happening in the scene: who is doing what to whom, and in what state. This paper asks whether that fine-grained action information can be recovered without retraining CLIP, and answers that it can be injected through prompts generated by a large language model. For each caption, an LLM first extracts action triplets of the form ⟨subject, action, object⟩ and then writes a sentence describing the physical state of each action; these become prompts fed into CLIP's image encoder. On Flickr30K and COCO, the resulting model retrieves correct matches far more often than frozen CLIP and is competitive with recent CLIP-based methods. The authors position this as the first action-aware, LLM-enhanced prompt-tuning method for image-text matching.

What carries the argument

The load-bearing machinery is a pair of action-aware prompts built from LLM output. An action-triplet prompt is a continuous embedding produced by a small transformer over the word embeddings of each ⟨subject, action, object⟩ triple, capturing compositional relations between entities; an action-state prompt is produced by feeding a hand-crafted template around the LLM's state description through the frozen CLIP text encoder. A prompt adapter (two MLP layers with ReLU and layer-norm) converts both into visual prompts, and an action-aware adaptive interaction module runs cross-attention from these prompts to image patches, self-attention on the results, then an attention-weighted merge (with mixing weight 0.7) to suppress irrelevant prompt content. These action-enhanced visual features are then processed by the remaining frozen transformer layers of CLIP.

What would settle it

On the COCO 5K test set, run the full two-stage pipeline but use state descriptions that are grammatically correct and image-unrelated (or shuffle triplets across captions). If Rsum does not fall measurably below the reported 421.1 for ViT-B/16, the action knowledge itself is not the cause of the gain. A second check: take the backhand example in Figure 5c, replace the LLM's description with a correct one, and see whether the retrieved result changes from wrong to right.

Watch

Extended reading notes

Core claim

The paper's central claim is that CLIP's weakness in image-text matching is largely an action-perception deficit, and that this deficit can be corrected by modifying only the input side of the image encoder, with no pretraining. The authors show that on Flickr30K, action mismatches account for roughly a quarter of CLIP's top-1 image-to-text decisions, and that a caption like 'A man wearing a blue shirt is jumping in the air' decomposes into '<man, jumping, air>', which CLIP fails to weigh. Their remedy is a two-stage prompt-tuning method: in the first stage, contrastive loss updates new action-aware prompts, a triplet encoder, and an adaptive interaction module while CLIP stays frozen; in the second stage, only the interaction module is trained with a triplet loss. With ViT-B/16 on COCO, the method raises Rsum from 374.9 for frozen CLIP to 421.1, with gains in both image-to-text and text-to-image directions.

Load-bearing premise

The method assumes that LLM-generated action triplets and state descriptions are accurate enough about what is happening in the image that the prompts they produce help CLIP align rather than mislead it; the paper's own qualitative results show cases where wrong state descriptions cause wrong retrieval.

Editorial extensions

If this is right

  • On COCO and Flickr30K, action-aware prompt tuning improves retrieval over frozen CLIP across ViT-B/32, ViT-B/16, and ViT-L/14-336 backbones, so the gain is not tied to one architecture.
  • Both prompt types earn their keep: removing the action triplet prompt or the action state prompt lowers Rsum, and replacing LLM-generated triplets with a hand-crafted template also hurts.
  • The adaptive interaction module contributes beyond concatenation: swapping it for plain concatenation of prompts degrades performance, showing that filtering prompt noise matters.
  • The method is robust to the choice of LLM: GPT-3.5, Llama-2-7B, and Llama-2-13B give nearly identical results, so open-source models can reproduce the gains.
  • The two-stage schedule outperforms one-stage and joint training, indicating that learning prompts before refining the interaction module avoids interference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's mechanism would be to replace LLM-generated action states with random or unrelated sentences through the same pipeline; if Rsum stays at the same level, the benefit is coming from extra prompt capacity rather than from action semantics.
  • The paper's own failure cases suggest that stronger LLMs should remove the remaining errors; one can A/B test by manually correcting a wrong state description (for example 'backhand') and checking whether retrieval flips to the correct match.
  • The same action-knowledge injection should transfer to video-text matching and action recognition, as the authors plan, since those tasks make action states even more central than still-image retrieval.
  • Because triplets also carry attribute knowledge (e.g. 'girl, is, small'), an ablation that removes only the relation part would clarify whether the gain is due to actions or to simply describing entities more fully.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper proposes an LLM-enhanced, action-aware prompt-tuning method for CLIP-based image-text matching. The authors use GPT-3.5 to decompose captions into action triplets and to generate fine-grained action-state descriptions, then insert these as prompts into the visual encoder via a prompt adapter and an adaptive interaction module. A two-stage training procedure (contrastive loss followed by triplet loss) is used, and an inference-time re-ranking over a CLIP top-k shortlist is applied. On Flickr30K and COCO, the method reports large gains over the frozen CLIP baseline and competitive or state-of-the-art results across several ViT backbones.

Significance. The idea of injecting structured action knowledge from LLMs into CLIP-like models is timely and potentially useful, and the paper includes a reasonable set of ablations (prompt types, interaction module, LLM choice, training schedule). The quantitative results are strong relative to the CLIP baseline. However, the current experimental design does not isolate the effect of LLM action knowledge: the inference procedure makes the image representation depend on the candidate text, and the key ablation removes both the knowledge and that conditional dependence. The comparison protocol with prior methods is also not fully specified. If the missing controls confirm that action knowledge is responsible for the gains, the paper would be a solid contribution; as it stands, the central attribution claim is not yet supported.

major comments (2)
  1. [§3.5, Eqns. (7)–(9), Table 3] The image-to-text inference in §3.5 first generates LLM action prompts from each pre-selected candidate caption and then extracts an image feature 'enhanced by the enriched texts.' This makes the visual representation a function of the candidate text at inference. The 'w/o action knowledge' row in Table 3 replaces the action prompts with learnable visual prompts, which also removes this text-conditioning; therefore, the performance gap between 'w/o action knowledge' (55.3 i2t R@1 on COCO) and 'Ours' (58.4) could be due to late-interaction text-conditioned image features rather than to LLM-generated action semantics. The 'Hand-Craft Triplet' row in Table 2 (55.5) is the only condition that retains text-conditioning while changing only the prompt wording, and its gap to 'Ours' is much smaller. To support the central claim, the authors must add an ablation that keeps the text-conditioned, cross-attention architecture unchanged but builds the prompts directly from the original caption tokens or from a non-LLM paraphrase, so that the only difference is the presence of LLM action knowledge. They should also run the same ablation for text-to-image retrieval.
  2. [§3.5, Table 1] The paper does not state whether the comparison methods in Table 1 use the same top-k pre-selection and re-ranking procedure described for the proposed method. If the baselines rank the entire test gallery with fixed image/text embeddings while the proposed method scores only a CLIP top-20 shortlist with a text-conditioned image encoder, the comparison is unfair and the 'significant performance improvements' claim is not established. The authors should perform all evaluations under a common protocol (either full-gallery scoring for every method, or the same shortlist re-ranking for every method) and specify the protocol in the paper.
minor comments (6)
  1. [§3.4, Eq. (11)] The triplet loss margin α is never assigned a value, although it is a hyperparameter of the second training stage that Table 5 shows to be important. Please report it and its sensitivity.
  2. [General experimental reporting] All results appear to be from single runs; no standard deviations or significance tests are reported. Given that several ablations differ by less than one R@1 point (e.g., rows in Table 2), the authors should add error bars or at least multiple seeds.
  3. [Table 1] Some rows (PyramidCLIP, SaCo, CLIPDualDIS) have missing R@10 values, yet Rsum is listed for some rows; state how Rsum is computed when entries are absent and avoid comparing incomplete aggregates.
  4. [§4.3] The phrase 'without pre-training' is ambiguous because CLIP itself is pre-trained. Rephrase to 'without additional pre-training on external training datasets.'
  5. [Figure 5c, Figure 8b] The paper should provide a quantitative estimate of the fraction of LLM-generated state descriptions that are semantically incorrect, since the qualitative examples show that such errors directly cause retrieval mistakes.
  6. [Introduction, §2.1] The claim of being 'the first to enhance CLIP with fine-grained action-aware perception' is too strong given the related work AME [20], which already enhances action-aware representations for this task; consider narrowing the claim to 'the first to use LLM-generated action knowledge in prompt tuning for CLIP-based image-text matching.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; LLM action knowledge is an external input, not a fitted prediction, and the paper's gains are not forced by definition.

full rationale

The paper's derivation chain is: LLM-generated action triplets and state descriptions are produced from captions (Eqs. 1 and 3), encoded into prompts (Eqs. 4-6), injected into the image encoder via cross-attention and adaptive interaction (Eqs. 7-9), and optimized with standard contrastive and triplet losses (Eqs. 10-11). No parameter is fitted to the reported retrieval metric, no test labels are used in prompt generation, and the LLM outputs are fixed inputs rather than learned predictions. The claimed improvement therefore does not reduce by construction to a fitted quantity or to a self-citation chain. The paper contains no load-bearing self-citations and no invoked uniqueness theorem. The reviewer-identified concern that inference-time conditioning of the image feature on each candidate enriched text may confound the attribution of gains to LLM action knowledge is a legitimate experimental-design and correctness issue, not a circularity: the similarity score is not defined as the prompt content, and the 'w/o action knowledge' ablation removes more than one variable. The paper's own qualitative failures (e.g., incorrect LLM state descriptions in Figures 5c and 8b) are limitations, not evidence that the method is circular. The ablation results, including the large gain over frozen CLIP, may be partly explained by late-interaction text-conditioned image features, but that does not make the derivation equivalent to its inputs. Honest finding: no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two external information sources: frozen CLIP and LLM-generated action knowledge. The latter is generated from the ground-truth captions, so it is not independent evidence. The method introduces no new physical entities.

free parameters (3)
  • Attention re-weighting coefficient lambda = 0.7
    Hand-tuned over {0.1, 0.3, 0.5, 0.7, 0.9} on both datasets; reported in Section C and Figure 6. Performance varies with lambda.
  • Triplet loss margin alpha = not reported
    Required in Eq. 11 for stage-2 training, but its value is not given anywhere in the main text or appendices.
  • Top-k pre-selection size k = 20
    Inference uses CLIP to pre-select top-k (k=20) before re-ranking, which can affect final R@K.
assumptions (4)
  • domain assumption CLIP's pre-trained visual and textual encoders provide a good base representation that prompt tuning can adapt.
    The entire method sits on top of frozen CLIP; if CLIP embeddings are poorly aligned, the prompt tuning may not help.
  • domain assumption LLM-generated action triplets and state descriptions accurately reflect the actions present in the images.
    Section 3.2 uses GPT-3.5 to generate action knowledge; the paper's own qualitative results (Figure 5c, 8b) show failures when this is wrong.
  • domain assumption The two-stage training procedure (contrastive then triplet loss) improves alignment without catastrophic forgetting.
    Section 3.4 and ablations in Table 5 show two-stage training beats one-stage and combined training, but this is an empirical claim without theoretical guarantee.
  • standard math Cross-attention and self-attention operations can select relevant action cues from the prompts.
    The adaptive interaction module in Section 3.3 uses standard attention mechanisms; no exotic math is required.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matching." pith.science (2026). https://pith.science/paper/XYHV26NL

@misc{pith2026250623502,
  author       = {Pith},
  title        = {Pith review of: LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XYHV26NL}},
  note         = {Machine review of arXiv:2506.23502}
}
read the original abstract

Driven by large-scale contrastive vision-language pre-trained models such as CLIP, recent advancements in the image-text matching task have achieved remarkable success in representation learning. Due to image-level visual-language alignment, CLIP falls short in understanding fine-grained details such as object attributes and spatial relationships between objects. Recent efforts have attempted to compel CLIP to acquire structured visual representations by introducing prompt learning to achieve object-level alignment. While achieving promising results, they still lack the capability to perceive actions, which are crucial for describing the states or relationships between objects. Therefore, we propose to endow CLIP with fine-grained action-level understanding by introducing an LLM-enhanced action-aware multi-modal prompt-tuning method, incorporating the action-related external knowledge generated by large language models (LLMs). Specifically, we design an action triplet prompt and an action state prompt to exploit compositional semantic knowledge and state-related causal knowledge implicitly stored in LLMs. Subsequently, we propose an adaptive interaction module to aggregate attentive visual features conditioned on action-aware prompted knowledge for establishing discriminative and action-aware visual representations, which further improves the performance. Comprehensive experimental results on two benchmark datasets demonstrate the effectiveness of our method.

Figures

Figures reproduced from arXiv: 2506.23502 by the authors.

Figure 1
Figure 1. Failure cases of image-text matching (image-to-text and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The statistical analysis of inconsistent actions between [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed method. exploit the powerful in-context learning capability of LLMs to generate fine-grained action knowledge instead of relying solely on fixed original textual descriptions, thus enabling the model to achieve more fine-grained action-aware visual understanding. 2.2. Prompt Learning for CLIP Various methods for prompt learning [18, 39, 40] have emerged as an effective and parameter-efficien… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Action knowledge generation using LLM. Action State Prompts. Action state prompts can be con￾structed by plugging the generated action state descriptions from actions in a predefined prompt template like “Fine￾grained state description of action triplet <subject, actio…
Figure 5
Figure 5. Figure 5: The qualitative results on the COCO 5K test set, where [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: An example from the ablation study in text-to-image retrieval, comparing various variants ( [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Visual comparisons of text-to-image retrieval examples between our method and baseline CLIP on Flickr30K and COCO datasets. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 52 canonical work pages

  1. [1]

    Incorporating geo-diverse knowledge into prompting for increased geographical robustness in object recognition

    Kyle Buettner, Sina Malakouti, Xiang Lorraine Li, and Adri- ana Kovashka. Incorporating geo-diverse knowledge into prompting for increased geographical robustness in object recognition. InCVPR, pages 13515–13524, 2024. 3

    work page 2024

  2. [2]

    Learning the best pooling strategy for visual semantic embedding

    Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. InCVPR, pages 15789–15798, 2021. 11

    work page 2021

  3. [3]

    Large language models are visual reasoning coordinators.NeurIPS, pages 10–16, 2024

    Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chun- yuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators.NeurIPS, pages 10–16, 2024. 1

    work page 2024

  4. [4]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. 6, 11

    arXiv 2015

  5. [5]

    Cross-modal graph matching network for image- text retrieval.TOMM, 18(4):1–23, 2022

    Yuhao Cheng, Xiaoguang Zhu, Jiuchao Qian, Fei Wen, and Peilin Liu. Cross-modal graph matching network for image- text retrieval.TOMM, 18(4):1–23, 2022. 2

    work page 2022

  6. [6]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. InCVPR, pages 2818–2829, 2023. 7

    work page 2023

  7. [7]

    Sim- ilarity reasoning and filtration for image-text matching

    Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Sim- ilarity reasoning and filtration for image-text matching. In AAAI, pages 1218–1226, 2021. 2

    work page 2021

  8. [8]

    Fleet, Jamie Ryan Kiros, and Sanja Fidler

    Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: improving visual-semantic embeddings with hard negatives. InBMVC, page 12, 2018. 2

    work page 2018

Show all 56 references
  1. [9]

    Learning semantic relationship among instances for image- text matching

    Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang. Learning semantic relationship among instances for image- text matching. InCVPR, pages 15159–15168, 2023. 2

    2023

  2. [10]

    Pyramidclip: Hierarchi- cal feature alignment for vision-language model pretraining

    Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidclip: Hierarchi- cal feature alignment for vision-language model pretraining. NeurIPS, pages 35959–35970, 2022. 7

    2022

  3. [11]

    Cross-modal semantic enhanced interaction for image-sentence retrieval

    Xuri Ge, Fuhai Chen, Songpei Xu, Fuxiang Tao, and Joe- mon M Jose. Cross-modal semantic enhanced interaction for image-sentence retrieval. InWACV, pages 1022–1031, 2023. 1, 2

    2023

  4. [12]

    Hiclip: Contrastive language-image pre- training with hierarchy-aware attention

    Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. Hiclip: Contrastive language-image pre- training with hierarchy-aware attention. InICLR, 2023. 1

    2023

  5. [13]

    From im- ages to textual prompts: Zero-shot visual question answering with frozen large language models

    Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From im- ages to textual prompts: Zero-shot visual question answering with frozen large language models. InCVPR, pages 10867– 10877, 2023. 1

    2023

  6. [14]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In CVPR, pages 14953–14962, 2023. 1

    2023

  7. [15]

    Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representa- tions

    Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xin- feng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, et al. Structure-clip: Towards scene graph knowledge to enhance multi-modal structured representa- tions. InAAAI, number 3, pages 2417–2425, 2024. 1

    2024

  8. [16]

    Fineclip: Self-distilled region-based clip for better fine-grained under- standing

    Dong Jing, Xiaolong He, Yutian Luo, Nanyi Fei, Guoxing Yang, Wei Wei, Huiwen Zhao, and Zhiwu Lu. Fineclip: Self-distilled region-based clip for better fine-grained under- standing. InNeurIPS, pages 2301–2310, 2024. 7

    2024

  9. [17]

    Knowledge-aware prompt tun- ing for generalizable vision-language models

    Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tun- ing for generalizable vision-language models. InICCV, pages 15670–15680, 2023. 1, 3

    2023

  10. [18]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. InCVPR, pages 19113–19122,

  11. [19]

    Stacked cross attention for image-text matching

    Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xi- aodong He. Stacked cross attention for image-text matching. InECCV, pages 201–216, 2018. 2

    2018

  12. [20]

    Action-aware em- bedding enhancement for image-text retrieval

    Jiangtong Li, Li Niu, and Liqing Zhang. Action-aware em- bedding enhancement for image-text retrieval. InAAAI, num- ber 2, pages 1323–1331, 2022. 2

    2022

  13. [21]

    Learning background prompts to dis- cover implicit knowledge for open vocabulary object detec- tion

    Jiaming Li, Jiacheng Zhang, Jichang Li, Ge Li, Si Liu, Liang Lin, and Guanbin Li. Learning background prompts to dis- cover implicit knowledge for open vocabulary object detec- tion. InCVPR, pages 16678–16687, 2024. 3

    2024

  14. [22]

    Cross- modal alternating learning with task-aware representations for continual learning.TMM, 2023

    Wujin Li, Bin-Bin Gao, Bizhong Xia, Jinbao Wang, Jun Liu, Yong Liu, Chengjie Wang, and Feng Zheng. Cross- modal alternating learning with task-aware representations for continual learning.TMM, 2023. 3

    2023

  15. [23]

    Image-text bidirectional learning network based cross-modal retrieval

    Zhuoyi Li, Huibin Lu, Hao Fu, and Guanghua Gu. Image-text bidirectional learning network based cross-modal retrieval. Neurocomputing, 483:148–159, 2022. 2

    2022

  16. [24]

    Learning customized visual models with retrieval-augmented knowledge

    Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li. Learning customized visual models with retrieval-augmented knowledge. InCVPR, pages 15148–15158, 2023. 7

    2023

  17. [25]

    Multi-modal attribute prompting for vision-language models.TCSVT, 2024

    Xin Liu, Jiamin Wu, Wenfei Yang, Xu Zhou, and Tianzhu Zhang. Multi-modal attribute prompting for vision-language models.TCSVT, 2024. 1

    2024

  18. [26]

    Fine-grained visual– text prompt-driven self-training for open-vocabulary object detection.TNNLS, pages 1–11, 2023

    Yanxin Long, Jianhua Han, Runhui Huang, Hang Xu, Yi Zhu, Chunjing Xu, and Xiaodan Liang. Fine-grained visual– text prompt-driven self-training for open-vocabulary object detection.TNNLS, pages 1–11, 2023. 3

    2023

  19. [27]

    Visual classification via description from large language models

    Sachit Menon and Carl Vondrick. Visual classification via description from large language models. InICLR, 2023. 3

    2023

  20. [28]

    SCHEMA: state changes matter for procedure planning in instructional videos

    Yulei Niu, Wenliang Guo, Long Chen, Xudong Lin, and Shih- Fu Chang. SCHEMA: state changes matter for procedure planning in instructional videos. InICLR, 2024. 3

    2024

  21. [29]

    Teaching clip to count to ten

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. InICCV, pages 3170–3180, 2023. 1

    2023

  22. [30]

    Fine-grained image-text matching by cross-modal hard aligning network

    Zhengxin Pan, Fangyu Wu, and Bailing Zhang. Fine-grained image-text matching by cross-modal hard aligning network. InCVPR, pages 19275–19284, 2023. 1, 2

    2023

  23. [31]

    Dynamic modality interaction modeling for image-text retrieval

    Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. Dynamic modality interaction modeling for image-text retrieval. InSIGIR, pages 1104–1113, 2021. 2

    2021

  24. [32]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, pages 8748–8763, 2021. 1, 7

    2021

  25. [33]

    Vlc-bert: Visual question answering with contextualized commonsense knowledge

    Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and Vered Shwartz. Vlc-bert: Visual question answering with contextualized commonsense knowledge. InWACV, pages 1155–1165, 2023. 1

    2023

  26. [34]

    Language models are causal knowledge ex- tractors for zero-shot video question answering

    Hung-Ting Su, Yulei Niu, Xudong Lin, Winston H Hsu, and Shih-Fu Chang. Language models are causal knowledge ex- tractors for zero-shot video question answering. InCVPR, pages 4951–4960, 2023. 3

    2023

  27. [35]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 7

    2023 arXiv

  28. [36]

    Compound text-guided prompt tuning via image-adaptive cues

    Hao Tan, Jun Li, Yizhuang Zhou, Jun Wan, Zhen Lei, and Xiangyu Zhang. Compound text-guided prompt tuning via image-adaptive cues. InAAAI, pages 5061–5069, 2024. 3

    2024

  29. [37]

    Multi-granularity cross-modal align- ment for generalized medical visual representation learning

    Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanab- huti, and Lequan Yu. Multi-granularity cross-modal align- ment for generalized medical visual representation learning. NeurIPS, pages 33536–33549, 2022. 7

    2022

  30. [38]

    Consensus-aware visual-semantic embedding for image- text matching

    Haoran Wang, Ying Zhang, Zhong Ji, Yanwei Pang, and Lin Ma. Consensus-aware visual-semantic embedding for image- text matching. InECCV, pages 18–34. Springer, 2020. 11

    2020

  31. [39]

    Vilt- clip: Video and language tuning clip with multimodal prompt learning and scenario-guided optimization

    Hao Wang, Fang Liu, Licheng Jiao, Jiahao Wang, Zehua Hao, Shuo Li, Lingling Li, Puhua Chen, and Xu Liu. Vilt- clip: Video and language tuning clip with multimodal prompt learning and scenario-guided optimization. InAAAI, num- ber 6, pages 5390–5400, 2024. 3

    2024

  32. [40]

    Position-guided text prompt for vision-language pre- training

    Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. Position-guided text prompt for vision-language pre- training. InCVPR, pages 23242–23251, 2023. 1, 3

    2023

  33. [41]

    Actionclip: A new paradigm for video action recognition.arXiv preprint arXiv:2109.08472, 2021

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition.arXiv preprint arXiv:2109.08472, 2021. 1, 3

    2021 arXiv

  34. [42]

    Cross-modal scene graph matching for relationship-aware image-text retrieval

    Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, and Xilin Chen. Cross-modal scene graph matching for relationship-aware image-text retrieval. InWACV, pages 1508–1517, 2020. 2

    2020

  35. [43]

    Diffusion feedback helps clip see better

    Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. InICLR, 2024. 7

    2024

  36. [44]

    Balance act: Mitigat- ing hubness in cross-modal retrieval with query and gallery banks

    Yimu Wang, Xiangru Jian, and Bo Xue. Balance act: Mitigat- ing hubness in cross-modal retrieval with query and gallery banks. InEMNLP, pages 10542–10567, 2023. 7

    2023

  37. [45]

    Learning hierarchical prompt with structured linguistic knowledge for vision-language models

    Yubin Wang, Xinyang Jiang, De Cheng, Dongsheng Li, and Cairong Zhao. Learning hierarchical prompt with structured linguistic knowledge for vision-language models. InAAAI, pages 5749–5757, 2024. 3

    2024

  38. [46]

    Multi-view inter-modality representation with progressive fusion for image-text matching.Neurocomputing, 535:1–12, 2023

    Jie Wu, Leiquan Wang, Chenglizhao Chen, Jing Lu, and Chunlei Wu. Multi-view inter-modality representation with progressive fusion for image-text matching.Neurocomputing, 535:1–12, 2023. 2

    2023

  39. [47]

    Saco loss: Sample-wise affinity con- sistency for vision-language pre-training

    Sitong Wu, Haoru Tan, Zhuotao Tian, Yukang Chen, Xiao- juan Qi, and Jiaya Jia. Saco loss: Sample-wise affinity con- sistency for vision-language pre-training. InCVPR, pages 27358–27369, 2024. 7

    2024

  40. [48]

    Visual- language prompt tuning with knowledge-guided context op- timization

    Hantao Yao, Rui Zhang, and Changsheng Xu. Visual- language prompt tuning with knowledge-guided context op- timization. InCVPR, pages 6757–6767, 2023. 3

    2023

  41. [49]

    FILIP: fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: fine-grained interactive language-image pre-training. InICLR, 2022. 7

    2022

  42. [50]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions.TACL, 2:67–78, 2014

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions.TACL, 2:67–78, 2014. 6, 11

    2014

  43. [51]

    Dual-path convolutional image-text embeddings with instance loss.TOMM, 16(2): 1–23, 2020

    Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Mingliang Xu, and Yi-Dong Shen. Dual-path convolutional image-text embeddings with instance loss.TOMM, 16(2): 1–23, 2020. 2

    2020

  44. [52]

    Large language models are good prompt learners for low-shot image classification.arXiv preprint arXiv:2312.04076, 2023

    Zhaoheng Zheng, Jingmin Wei, Xuefeng Hu, Haidong Zhu, and Ram Nevatia. Large language models are good prompt learners for low-shot image classification.arXiv preprint arXiv:2312.04076, 2023. 3

    2023 arXiv

  45. [53]

    Regionclip: Region-based language- image pretraining

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language- image pretraining. InCVPR, pages 16793–16803, 2022. 1

    2022

  46. [54]

    Relclip: Adapting language- image pretraining for visual relationship detection via rela- tional contrastive learning

    Yi Zhu, Zhaoqing Zhu, Bingqian Lin, Xiaodan Liang, Feng Zhao, and Jianzhuang Liu. Relclip: Adapting language- image pretraining for visual relationship detection via rela- tional contrastive learning. InEMNLP, pages 4800–4810,

  47. [336]

    w/o ac- tion knowledge

    During training, we use SGD optimization with an initial learning rate of 1e-5, a maximum of 4 epochs, and a batch size of 128. C. More Ablation Studies Effect of action-aware multi-modal prompting.To evalu- ate the effectiveness of action-aware multi-modal prompting, we desig...

  48. [2022]

    Datasets Details Flickr30K[50] dataset contains 31,000 images collected from the Flickr website

    1 A. Datasets Details Flickr30K[50] dataset contains 31,000 images collected from the Flickr website. These images mostly depict humans performing various activities. Each image is described by five different sentences, and there are 155,000 sentences. Following the settings [...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.