Pith. sign in

REVIEW 3 major objections 5 minor 50 references

ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieved captions become visual prompts, lifting CIDEr to 122.9

desk verdict A solid plug-and-play module that turns retrieved text into visual prompts for lightweight captioning; gains are real but modest and the paper overstates significance via test-set tuning and missing error bars. read the letter →

arxiv 2412.19289 v3 pith:Y42BSAOU submitted 2024-12-26 cs.CV cs.AIcs.CLcs.LG

Taewhan Kim , Soeun Lee , Si-Woo Kim , Dong-Jin Kim This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.LG
keywords imagecaptioninglightweightmodelsretrieval-augmentedgenerationvisualpromptsCLIPGaussiansamplingpatchretrievalcross-modalfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that lightweight image captioning can be improved by turning retrieved text captions into visual prompts instead of only feeding them to the decoder as text. The proposed ViPCap method encodes retrieved captions with CLIP, models the text embedding as a multivariate Gaussian, samples many candidate features, matches each image patch to the closest sample, and fuses the matches into the image features before decoding. On COCO this raises CIDEr from 119.7 (SmallCap baseline) to 122.9, and on NoCaps from 77.9 to 81.3, using only 14M trainable parameters. The paper argues that retrieved text carries visual information the vision encoder alone misses, so converting it into visual prompts yields consistent gains across several captioning frameworks.

What carries the argument

The central mechanism is the ViP module, which converts retrieved text embeddings into visual prompts in three steps. First, the CLIP text embedding is mapped by two MLPs to a mean \(\mu\) and standard deviation \(\$\sigma$\), plus a learnable offset \(\$\alpha$\cdot \omega_{\text{add}}\), and M samples \(g_i \sim \mathcal{N}(\mu, \$sigma^{2}$ I)\) are drawn via the reparameterization trick. Second, a patch retrieval module computes cosine similarity between each image patch feature \(v_j\) and each sample, keeping the most similar sample for each patch to form R. Third, a feature fusion network with one self-attention and one cross-attention layer takes the image features V as query and R as key to produce the visual prompt Z, and the model feeds \(V' = V + Z\) to the decoder. The paper's claim is that sampling from a learnable Gaussian distribution bridges the modality gap between text-derived semantic features and visual features, so that a single fusion layer and a simple summation are enough.

What would settle it

A controlled ablation that replaces the Gaussian sampling with the raw CLIP text embedding (no sampling) but keeps the same patch retrieval and fusion would settle the role of the distribution assumption; if CIDEr on COCO does not drop, the Gaussian mechanism is not load-bearing. Conversely, if training without the learned mean and standard deviation maps (using a fixed Gaussian) gives the same results, the learned distribution is unnecessary.

Watch

Extended reading notes

Core claim

ViPCap establishes that retrieved text captions, which previous models treat purely as text prompts, can be converted into visual prompts that enrich the image features. The method models each retrieved caption's CLIP text embedding as a Gaussian distribution, samples M semantic vectors, and for each image patch selects the sampled vector with highest cosine similarity. These patch-matched vectors are passed through a single-layer fusion network to produce a visual prompt Z, and the refined image feature V' = V + Z is fed to a frozen GPT-2 decoder through cross-attention. The paper's empirical claim is that this recipe improves CIDEr and related metrics over the SmallCap baseline and also boosts CapDec, ViECap, EVCap, OPT, and XGLM when the ViP module is inserted, with only 14M trainable parameters.

Load-bearing premise

The load-bearing premise is that the CLIP text embedding of a retrieved caption can be modeled as a multivariate Gaussian, and that samples from this distribution, after cosine-similarity patch retrieval and a single-layer fusion, land close enough to the true image feature distribution that a simple summation V' = V + Z improves cross-attention.

Editorial extensions

If this is right

  • Lightweight captioning models can gain several CIDEr points without adding large trainable modules, by exploiting the retrieved text they already use as prompts.
  • The ViP module works as a plug-in across decoders (GPT-2, OPT, XGLM) and even large retrieval-augmented models like EVCap, suggesting the visual-prompt conversion is broadly useful.
  • With 14M trainable parameters, ViPCap surpasses OSCAR Large on NoCaps in-domain CIDEr, narrowing the gap between lightweight and large models.
  • Because the module improves both retrieval-based and simple-prompt settings, it can be applied to captioning pipelines regardless of the type of text prompt used.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Gaussian sampling might act as a stochastic regularizer; varying M and measuring variance across training seeds could reveal whether the benefit comes from diversity or from the patch-retrieval selection.
  • The same text-to-visual-prompt conversion could be applied to other vision-language tasks, such as visual question answering, where retrieved textual knowledge could be injected into visual features.
  • Since the method only requires a frozen CLIP and a small trainable fusion layer, it may be effective in low-resource or domain-adaptation scenarios where paired image-text data is scarce but text datastores are available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ViPCap, a lightweight retrieval-augmented image captioning method that turns retrieved text into a visual prompt. The ViP module encodes retrieved captions with the CLIP text encoder, estimates a multivariate Gaussian with learnable mean and standard-deviation MLPs, samples M semantic features, retrieves for each image patch the most cosine-similar semantic feature, fuses the retrieved features with image features via a single-layer transformer, and adds the fused output to the image features before cross-attention in the frozen GPT-2 decoder. Experiments on COCO, Flickr30k, and NoCaps report consistent CIDEr and BLEU improvements over SmallCap, and the module is also tested on CapDec, ViECap, EVCap, OPT, XGLM, and a MAPL-style setup. Ablations cover the sampling distribution, the learnable offset vector, the patch-retrieval module, the fusion network, and the number of samples M.

Significance. If the proposed visual-prompt mechanism is real, ViPCap is a useful plug-and-play contribution: it adds only about 7M trainable parameters over SmallCap, works with several frozen encoders/decoders and prompt styles, and the authors provide the source code. The breadth of the plug-and-play experiments is the paper's main strength. However, the evidence that the gains come from genuinely visual alignment is indirect, and the evaluation protocol has weaknesses (test-set hyperparameter selection, no variance or significance tests), so the current version does not fully support the strong 'visually aligned' interpretation placed on the results.

major comments (3)
  1. [Experiments, Tables 7 and 9] Hyperparameters α=5 and M=200 are selected using COCO test-set results: Table 7 reports the ablation of the learnable vector and scale factor on COCO test, and Table 9 sweeps M on COCO test, while the same COCO test set is used for the headline comparison in Table 1 (CIDEr 122.9 vs. SmallCap 119.7). This conflates model selection with final evaluation and makes the reported gains potentially optimistic. Please move hyperparameter selection to a validation split, describe the selection protocol explicitly, and report run-to-run variance or significance tests for the key comparisons.
  2. [Proposed Method, Eqs. (1)-(3)] The central claim is that samples g_i from N(μ, σ²I) and the retrieved set R lie close to the image patch features V, so that V' = V + Z functions as a visual prompt. However, the training loss in Eq. (3) supervises only the caption tokens; there is no auxiliary objective or evaluation measuring the closeness of G or R to V. Given the acknowledged CLIP modality gap, the cosine-similarity retrieval in Eq. (2) may select only the least-dissimilar text samples, and the CIDEr-only ablations in Tables 3 and 7 cannot rule out that the gains reflect a textual bias or the added capacity of the fusion network. Please add a direct diagnostic, for example comparing the feature distributions of R and V before and after training, or an ablation in which the retrieved text is replaced by non-visual or shuffled text, to substantiate the visual-alignment mechanism.
  3. [Abstract and Main Results] The abstract and the main-results discussion use 'significantly' without statistical support, and several reported deltas are small (e.g., Table 4: OPT 122.0 to 122.5 CIDEr; Table 2: ViECap+ViP on COCO +0.7 CIDEr). No confidence intervals or significance tests are reported. In addition, Table 2 contains negative deltas (e.g., CapDec COCO→Flickr30k B@4 from 17.3 to 15.7), so blanket statements about consistent improvement are too strong. Please provide per-seed variability or significance tests, and temper the language accordingly.
minor comments (5)
  1. [Experimental Setup] The sentence 'the ViP module uses a patch size of M=200' should read 'the number of samples M=200', since M is not a patch size.
  2. [Eq. (1)] The notation N(μ, σ²I; φ(T)) is nonstandard; the conditioning on φ(T) should appear outside the distribution, and the paper should clarify that σ is a vector used for an elementwise multiplication in the reparameterization g = μ + σ·ε.
  3. [Table 3] The legend 'Unif(0,1)×N(0,1) represents the method proposed by LinCIR' is hard to parse; please describe each sampling scheme in a separate row or in a clearer caption.
  4. [Figure 2] The caption contains grammatical errors ('peaking' should be 'peeking') and does not clearly identify which line is the ViPCap output; please clarify.
  5. [Table 9] The M sweep is non-monotonic (CIDEr 121.4 at M=100, 122.9 at M=200, 121.5 at M=300), so the explanation that 'G becomes scattered' for M>200 is not directly supported; please discuss or provide additional evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ViPCap's reported gains are empirical, end-to-end trained, and benchmarked against external baselines; no claimed prediction reduces to a fitted input or self-citation.

full rationale

ViPCap is an empirical captioning architecture built on SmallCap. The central claim is that adding the ViP module improves CIDEr, and this is verified by end-to-end training on COCO and evaluation on COCO, Flickr30k, and NoCaps against external baselines. The Gaussian sampling in Eq. (1), patch retrieval in Eq. (2), and feature fusion producing V' are all learned or fixed operations inside a system whose only training objective is the caption likelihood in Eq. (3). No quantity called a prediction is fitted from a subset of data and then reported as a prediction of the same data; the reported CIDEr numbers are test-set scores after training. The citations to CapDec and LinCIR motivate the Gaussian modeling choice but do not function as a uniqueness theorem or as a substitute for experiments; the paper states the assumption explicitly and supports it with ablations in Tables 3, 7, 8, and 9. The skeptic's concern that sampled text features may not truly align with image features is a mechanism and validity concern about whether the module works as described, not a circularity: the paper does not define 'visual prompt' as whatever improves CIDEr, nor does it derive the improvement from the Gaussian assumption alone. The method is self-contained against external benchmarks, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumption that retrieved text captions, when converted to visual-like features via Gaussian sampling and patch retrieval, improve visual representations for captioning. This is empirically motivated but not theoretically grounded.

free parameters (3)
  • alpha (scale factor) = 5
    Chosen based on COCO test set experiments (Table 7); controls scale of learnable vector added to Gaussian mean.
  • M (number of Gaussian samples) = 200
    Selected via COCO test set sweep (Table 9); number of Gaussian samples per text prompt.
  • k (number of retrieved captions) = 3
    Number of retrieved captions per image, limited by CLIP's 77-token context length.
assumptions (4)
  • domain assumption Retrieved captions carry image-relevant semantic information.
    The entire method depends on retrieved texts being informative about the image content, as in SmallCap.
  • domain assumption The CLIP text embedding follows a multivariate Gaussian distribution.
    Eq. (1) models the text feature as N(mu, sigma^2 I); this is a modeling choice, not a proven property.
  • domain assumption Frozen CLIP and GPT-2 provide adequate representations.
    Both the image encoder and language decoder are kept frozen; success relies on their pretrained quality.
  • domain assumption A single-layer fusion network is sufficient to bridge the modality gap.
    The Feature Fusion Network uses one transformer layer; experiments show it works, but there is no theoretical justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning." pith.science (2026). https://pith.science/paper/Y42BSAOU

@misc{pith2026241219289,
  author       = {Pith},
  title        = {Pith review of: ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y42BSAOU}},
  note         = {Machine review of arXiv:2412.19289}
}
read the original abstract

Recent lightweight image captioning models using retrieved data mainly focus on text prompts. However, previous works only utilize the retrieved text as text prompts, and the visual information relies only on the CLIP visual embedding. Because of this issue, there is a limitation that the image descriptions inherent in the prompt are not sufficiently reflected in the visual embedding space. To tackle this issue, we propose ViPCap, a novel retrieval text-based visual prompt for lightweight image captioning. ViPCap leverages the retrieved text with image information as visual prompts to enhance the ability of the model to capture relevant visual information. By mapping text prompts into the CLIP space and generating multiple randomized Gaussian distributions, our method leverages sampling to explore randomly augmented distributions and effectively retrieves the semantic features that contain image information. These retrieved features are integrated into the image and designated as the visual prompt, leading to performance improvements on the datasets such as COCO, Flickr30k, and NoCaps. Experimental results demonstrate that ViPCap significantly outperforms prior lightweight captioning models in efficiency and effectiveness, demonstrating the potential for a plug-and-play solution. The source code is available at https://github.com/taewhankim/VIPCAP.

Figures

Figures reproduced from arXiv: 2412.19289 by the authors.

Figure 1
Figure 1. (a) ViPCap shows the best efficiency among the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. SmallCap (Ramos et al. 2023) fails to accurately [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. ViPCap leverages the CLIP text encoder to extract retrieved text features for visual prompts generation. The ViP [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Calculating similarity between input feature [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 15 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Agrawal, H.; Desai, K.; Wang, Y.; Chen, X.; Jain, R.; Johnson, M.; Batra, D.; Parikh, D.; Lee, S.; and Anderson, P. 2019. nocaps: novel object captioning at scale. In Proceedings of the IEEE International Conference on Computer Vision, 8948--8957

    2019

  4. [4]

    Anderson, P.; Fernando, B.; Johnson, M.; and Gould, S. 2016. SPICE: Semantic Propositional Image Caption Evaluation. ArXiv, abs/1607.08822

    arXiv 2016

  5. [5]

    Bahng, H.; Jahanian, A.; Sankaranarayanan, S.; and Isola, P. 2022. Exploring Visual Prompts for Adapting Large-Scale Models. arXiv:2203.17274

    arXiv 2022

  6. [6]

    Barraco, M.; Stefanini, M.; Cornia, M.; Cascianelli, S.; Baraldi, L.; and Cucchiara, R. 2022. CaMEL: Mean Teacher Learning for Image Captioning. arXiv:2202.10492

    work page Pith review arXiv 2022

  7. [7]

    K.; Riquelme, C.; Steiner, A.; Angelova, A.; Zhai, X.; Houlsby, N.; and Soricut, R

    Chen, X.; Wang, X.; Changpinyo, S.; Piergiovanni, A.; Padlewski, P.; Salz, D.; Goodman, S.; Grycner, A.; Mustafa, B.; Beyer, L.; Kolesnikov, A.; Puigcerver, J.; Ding, N.; Rong, K.; Akbari, H.; Mishra, G.; Xue, L.; Thapliyal, A.; Bradbury, J.; Kuo, W.; Seyedhosseini, M.; Jia, C.; Ayan, B. K.; Riquelme, C.; Steiner, A.; Angelova, A.; Zhai, X.; Houlsby, N.; ...

    arXiv 2023

  8. [8]

    E.; Stoica, I.; and Xing, E

    Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\

    2023

Show all 50 references
  1. [9]

    J.; and Lavie, A

    Denkowski, M. J.; and Lavie, A. 2014. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In WMT@ACL

    2014

  2. [10]

    Fei, J.; Wang, T.; Zhang, J.; He, Z.; Wang, C.; and Zheng, F. 2023. Transferable Decoding with Visual Entities for Zero-Shot Image Captioning. arXiv:2307.16525

    2023 arXiv

  3. [11]

    Gao, T.; Fisch, A.; and Chen, D. 2021. Making Pre-trained Language Models Better Few-shot Learners. arXiv:2012.15723

    2021 arXiv

  4. [12]

    Gu, G.; Chun, S.; Kim, W.; Kang, Y.; and Yun, S. 2024. Language-only Efficient Training of Zero-shot Composed Image Retrieval. arXiv:2312.01998

    2024 arXiv

  5. [13]

    Hu, X.; Gan, Z.; Wang, J.; Yang, Z.; Liu, Z.; Lu, Y.; and Wang, L. 2022. Scaling Up Vision-Language Pre-training for Image Captioning. arXiv:2111.12233

    2022 arXiv

  6. [14]

    A.; and Fathi, A

    Hu, Z.; Iscen, A.; Sun, C.; Wang, Z.; Chang, K.-W.; Sun, Y.; Schmid, C.; Ross, D. A.; and Fathi, A. 2023. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory. arXiv:2212.05221

    2023 arXiv

  7. [15]

    V.; Sung, Y.; Li, Z.; and Duerig, T

    Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.-T.; Parekh, Z.; Pham, H.; Le, Q. V.; Sung, Y.; Li, Z.; and Duerig, T. 2021. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. arXiv:2102.05918

    2021 arXiv

  8. [16]

    Jia, M.; Tang, L.; Chen, B.-C.; Cardie, C.; Belongie, S.; Hariharan, B.; and Lim, S.-N. 2022. Visual Prompt Tuning. arXiv:2203.12119

    2022 arXiv

  9. [17]

    Johnson, J.; Douze, M.; and Jégou, H. 2017. Billion-scale similarity search with GPUs. arXiv:1702.08734

    2017 arXiv

  10. [18]

    Karpathy, A.; and Fei-Fei, L. 2015. Deep Visual-Semantic Alignments for Generating Image Descriptions. arXiv:1412.2306

    2015 arXiv

  11. [19]

    P.; and Welling, M

    Kingma, D. P.; and Welling, M. 2022. Auto-Encoding Variational Bayes. arXiv:1312.6114

    2022 arXiv

  12. [20]

    Lester, B.; Al-Rfou, R.; and Constant, N. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691

    2021 arXiv

  13. [21]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 a . BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597

    2023 arXiv

  14. [22]

    M.; Sugimoto, A.; and Nakayama, H

    Li, J.; Vo, D. M.; Sugimoto, A.; and Nakayama, H. 2024. EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehension. arXiv:2311.15879

    2024 arXiv

  15. [23]

    Li, W.; Zhu, L.; Wen, L.; and Yang, Y. 2023 b . DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training. In The Eleventh International Conference on Learning Representations

    2023

  16. [24]

    Li, X.; Yin, X.; Li, C.; Zhang, P.; Hu, X.; Zhang, L.; Wang, L.; Hu, H.; Dong, L.; Wei, F.; Choi, Y.; and Gao, J. 2020. Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. arXiv:2004.06165

    2020 arXiv

  17. [25]

    L.; and Liang, P

    Li, X. L.; and Liang, P. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190

    2021 arXiv

  18. [26]

    L.; and Dollár, P

    Lin, T.-Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C. L.; and Dollár, P. 2015. Microsoft COCO: Common Objects in Context. arXiv:1405.0312

    2015 arXiv

  19. [27]

    V.; Mihaylov, T.; Artetxe, M.; Wang, T.; Chen, S.; Simig, D.; Ott, M.; Goyal, N.; Bhosale, S.; Du, J.; Pasunuru, R.; Shleifer, S.; Koura, P

    Lin, X. V.; Mihaylov, T.; Artetxe, M.; Wang, T.; Chen, S.; Simig, D.; Ott, M.; Goyal, N.; Bhosale, S.; Du, J.; Pasunuru, R.; Shleifer, S.; Koura, P. S.; Chaudhary, V.; O'Horo, B.; Wang, J.; Zettlemoyer, L.; Kozareva, Z.; Diab, M.; Stoyanov, V.; and Li, X. 2022. Few-shot Learni...

    2022 arXiv

  20. [28]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual Instruction Tuning. arXiv:2304.08485

    2023 arXiv

  21. [29]

    Luo, Z.; Hu, Z.; Xi, Y.; Zhang, R.; and Ma, J. 2023. I-Tuning: Tuning Frozen Language Models with Image for Lightweight Image Captioning. arXiv:2202.06574

    2023 arXiv

  22. [30]

    Mañas, O.; Rodriguez, P.; Ahmadi, S.; Nematzadeh, A.; Goyal, Y.; and Agrawal, A. 2023. MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting. arXiv:2210.07179

    2023 arXiv

  23. [31]

    Mokady, R.; Hertz, A.; and Bermano, A. H. 2021. ClipCap: CLIP Prefix for Image Captioning. arXiv:2111.09734

    2021 arXiv

  24. [32]

    Nukrai, D.; Mokady, R.; and Globerson, A. 2022. Text-Only Training for Image Captioning using Noise-Injected CLIP

    2022

  25. [33]

    Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Annual Meeting of the Association for Computational Linguistics

    2002

  26. [34]

    A.; Wang, L.; Cervantes, C

    Plummer, B. A.; Wang, L.; Cervantes, C. M.; Caicedo, J. C.; Hockenmaier, J.; and Lazebnik, S. 2016. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. arXiv:1505.04870

    2016 arXiv

  27. [35]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020

    2021 arXiv

  28. [36]

    Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2018. Language Models are Unsupervised Multitask Learners

    2018

  29. [37]

    Ramos, R.; Martins, B.; and Elliott, D. 2023. LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting. arXiv:2305.19821

    2023 arXiv

  30. [38]

    Ramos, R.; Martins, B.; Elliott, D.; and Kementchedjhieva, Y. 2023. SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation. arXiv:2209.15323

    2023 arXiv

  31. [39]

    P.; Elliott, D.; and Martins, B

    Ramos, R. P.; Elliott, D.; and Martins, B. 2023. Retrieval-augmented Image Captioning. In Conference of the European Chapter of the Association for Computational Linguistics

    2023

  32. [40]

    Sun, Q.; Fang, Y.; Wu, L.; Wang, X.; and Cao, Y. 2023. EVA-CLIP: Improved Training Techniques for CLIP at Scale. arXiv:2303.15389

    2023 arXiv

  33. [41]

    L.; and Parikh, D

    Vedantam, R.; Zitnick, C. L.; and Parikh, D. 2014. CIDEr: Consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4566--4575

    2014

  34. [42]

    Wang, B.; and Komatsuzaki, A. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model

    2021

  35. [43]

    Wang, J.; Yan, M.; Zhang, Y.; and Sang, J. 2023. From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping. In Elkind, E., ed., Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 , 4326--4334. Int...

    2023

  36. [44]

    Wang, W.; Lv, Q.; Yu, W.; Hong, W.; Qi, J.; Wang, Y.; Ji, J.; Yang, Z.; Zhao, L.; Song, X.; Xu, J.; Xu, B.; Li, J.; Dong, Y.; Ding, M.; and Tang, J. 2024. CogVLM: Visual Expert for Pretrained Language Models. arXiv:2311.03079

    2024 arXiv

  37. [45]

    W.; Dai, Z.; Tsvetkov, Y.; and Cao, Y

    Wang, Z.; Yu, J.; Yu, A. W.; Dai, Z.; Tsvetkov, Y.; and Cao, Y. 2022 a . SimVLM: Simple Visual Language Model Pretraining with Weak Supervision. arXiv:2108.10904

    2022 arXiv

  38. [46]

    Wang, Z.; Zhang, Z.; Ebrahimi, S.; Sun, R.; Zhang, H.; Lee, C.-Y.; Ren, X.; Su, G.; Perot, V.; Dy, J.; and Pfister, T. 2022 b . DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning. arXiv:2204.04799

    2022 arXiv

  39. [47]

    Yang, Z.; Ping, W.; Liu, Z.; Korthikanti, V.; Nie, W.; Huang, D.-A.; Fan, L.; Yu, Z.; Lan, S.; Li, B.; Liu, M.-Y.; Zhu, Y.; Shoeybi, M.; Catanzaro, B.; Xiao, C.; and Anandkumar, A. 2023. Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning....

    2023 arXiv

  40. [48]

    Zeng, Z.; Xie, Y.; Zhang, H.; Chen, C.; Wang, Z.; and Chen, B. 2024. MeaCap: Memory-Augmented Zero-shot Image Captioning. arXiv:2403.03715

    2024 arXiv

  41. [49]

    V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P

    Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022. OPT: Open Pre-trained Transformer Language Mo...

    2022 arXiv

  42. [50]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.