Pith. sign in

REVIEW 3 major objections 3 minor 49 references

CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CONCAP shows that retrieved captions plus image-specific concepts can lift multilingual captioning past models many times its size.

desk verdict Data-efficient multilingual captioning with genuinely additive RAG signals, but the evaluation datastore is undisclosed and the tables disagree. read the letter →

arxiv 2507.20411 v1 pith:MKKKV5YS submitted 2025-07-27 cs.CL

George Ibrahim , Rita Ramos , Yova Kementchedjhieva This is my paper · ORCID
classification cs.CL
keywords multilingualimagecaptioningretrieval-augmentedgenerationconceptretrievallow-resourcelanguagesXM3600benchmarkCIDErvision-languagemodelsdata-efficienttraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a multilingual image captioning model can reach strong results without massive multilingual training by conditioning generation on two retrieved signals: full captions of similar images and short image-specific concepts in the target language. CONCAP, built on the lightweight mBLIP architecture, is trained on 566K image-caption pairs with 111M trainable parameters, yet scores 34.2 on the CIDEr caption-quality metric averaged over 36 languages of the XM3600 benchmark, ahead of Pangea (31.8 with 7B parameters and 6M examples) and mBLIP (25.9 with 2.71M examples). The result matters because it points to a cheaper route toward closing the English-centric gap: retrieval supplies target-language phrasing and broad visual coverage that would otherwise require expensive scaled-up multilingual training.

What carries the argument

The load-bearing mechanism is a retrieval-augmented prompt assembled from two datastores indexed by a multilingual vision-language retriever (mSigLIP). For captions, the image encoder retrieves the top-4 nearest captions from a per-language datastore; for concepts, single tokens extracted from the training captions are wrapped in language-specific templates such as "a photo of a dog", the top-10 are retrieved per image, and both sets are inserted into a fixed English prompt before the decoder: "Similar images show: ..., This image might contain: ..., Caption in {language}:". The decoder is the mBLIP stack, with frozen vision encoder and language model and trainable Q-Former, projection, and LoRA layers. Concept retrieval is what makes the method robust: it does not depend on translated English caption pivots, and it supplies terms that retrieved full captions miss.

What would settle it

Decisive check: replace the mSigLIP retrieval with random caption and concept sampling and measure CIDEr on XM3600. If CONCAP retains most of its 34.2 score, retrieval is not the cause; if it falls toward the 26.9 no-retrieval level, retrieval is load-bearing. A second check is to identify the evaluation-time caption corpus: if it is the same translated COCO-35L data used for training, compare with a native-language datastore.

Watch

Extended reading notes

Core claim

The central discovery is that caption retrieval and concept retrieval are complementary and additively beneficial in multilingual captioning. In an otherwise identical setup, the no-retrieval model scores 26.9 CIDEr; caption-only retrieval gives 31.4; concept-only retrieval gives 30.3; combining both gives 34.2. The paper further finds that concepts alone are a surprisingly effective retrieval signal, that concept retrieval remains useful even when caption retrieval fails (direct target-language caption retrieval at training time drops to 20.4), and that enriching the concept list with external cultural and web-sourced terms hurts rather than helps, whereas oracle cultural concepts improve an Arabic cross-cultural test at early training but the benefit fades with continued finetuning.

Load-bearing premise

The claim stands on the assumption that retrieving single-token concepts from noisy, translation-derived wordlists with a multilingual image-text retriever yields visual grounding that stays useful across languages; if the retriever or the wordlists are too noisy in a target language, concept augmentation adds little and the reported gains would not transfer.

Editorial extensions

If this is right

  • If CONCAP's results hold, a 111M-parameter model trained on 566K pairs can beat a 7B model trained on 6M examples on average across 36 languages, shifting the cost-benefit balance of multilingual captioning toward retrieval memory rather than scale.
  • The additive gain of captions and concepts means future retrieval-augmented captioning systems should treat sentence-level style guides and lexical content cues as separate channels rather than as redundant evidence.
  • Because concept-only augmentation survives a weak caption retriever, target-language concept retrieval offers a fallback for languages where translated caption parallels are unavailable.
  • The finding that broader concept lexicons hurt performance implies that datastore quality, not size, governs concept retrieval, so curation efforts should focus on relevance filtering rather than raw coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never names the caption datastore used during XM3600 evaluation. If it is the same translated, Western-centric COCO-35L data used for training, then CONCAP's margin partly reflects retrieving near neighbors of its own training distribution, and the claim of 'seeing beyond English' would be overstated for target-language grounding.
  • A direct two-way test follows: index a datastore of native, human-written captions per XM3600 language and re-run evaluation; if CIDEr does not rise or hold, the retrieval memory is doing the work, not the target-language concepts.
  • The oracle result on the 200 Arabic cultural images suggests a training dynamic worth probing: early finetuning can absorb new lexical items, but later epochs lock in the decoder's vocabulary; curriculum schedules that introduce cultural concepts late could extend the oracle gain beyond one epoch.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The manuscript presents CONCAP, a retrieval-augmented multilingual image captioning model built on mBLIP. The model augments the decoder prompt with (i) top-n captions retrieved from a caption datastore and (ii) top-m image-specific concepts retrieved from language-specific wordlists. Training uses a 566K-pair subsample of COCO-35L, with frozen vision encoder and language decoder plus trainable Q-Former, projection, and LoRA layers. Evaluation on XM3600 reports L36 CIDEr 34.2, above Pangea (31.8) and mBLIP (25.9) despite far fewer parameters and training examples. Ablations show NoRAG 26.9, CapRAG 31.4, ConRAG 30.3, and CONCAP 34.2, which the authors interpret as additive gains; a Text-only control in Appendix A.8 supports the visual-grounding interpretation. Additional analyses cover retriever choice, number of concepts, concept-list enrichment, and per-language performance.

Significance. If the reported results are reproducible, CONCAP is a valuable data-efficient alternative to scaling in multilingual captioning. The evaluation is on an external human-annotated benchmark, hyperparameters are tuned on a dev split, and the appendix provides per-language tables and a Text-only control. The oracle experiment on JEEM and the enrichment analysis are honest probes of the method's limitations. The main caveats are the unspecified evaluation caption datastore and internal inconsistencies in the ablation tables, which need to be resolved before the headline comparison can be fully trusted.

major comments (3)
  1. [Section 4.1 (Caption Retrieval)] The evaluation-time caption retrieval datastore is not identified. The text says that retrieval is performed "using the mSigLIP model directly in the target language" and motivates this by the geographic diversity of XM3600, but it never states which caption corpus is indexed at inference. If the datastore is the XM3600 reference captions and the query image's own captions are not excluded, then the reported L36 advantage over Pangea could be an artifact of test-set leakage rather than of target-language retrieval. Please specify the datastore and the exclusion policy for the query image, and note that no analogous anti-leakage statement is made for caption retrieval even though Section 5 excludes XM100 words and images from enriched concept lexicons.
  2. [Table 2 vs Table 10] The CapRAG and CapRAGM rows are inconsistent. In Table 2, CapRAG lists en/es/zh/hi = 66.2/53.3/23.9/20.2, which are exactly the CapRAGM values in Table 10, while CapRAGM in Table 2 lists 38.3/30.4/16.4/13.9, values that do not appear in Table 10. Because Section 4.4's conclusion that target-language caption retrieval "actively hurts performance" rests on CapRAGM, the correct per-language values must be confirmed and the table corrected.
  3. [Section 4.4] The CapRAGM comparison changes two variables at once: the retriever (English CLIP vs mSigLIP) and the retrieval language (English pivot vs target language). The conclusion that "the quality of the retriever is of utmost importance" is therefore not uniquely identified; the observed drop could also reflect noisier target-language captions or the interaction of both factors. Please disentangle these variables or soften the causal claim.
minor comments (3)
  1. [Appendix A.4] The text cites "Table 9" for concept list sizes, but Table 9 in the appendix is the prompt-template table; there are two items numbered Table 9. Renumber the appendix tables.
  2. [Appendix A.8] The Text-only control replaces the image with a solid-color mean-RGB image, but it is not stated whether retrieval for this condition uses the original image or the solid image; this determines exactly what the control isolates.
  3. [Section 5] The oracle result is reported after one epoch of training and on 200 images, and the claim that gains fade with continued finetuning is presented without supporting learning curves. Adding the epoch progression would make this claim substantiated.

Circularity Check

0 steps flagged · score 2.0 of 10

No demonstrated circularity: empirical RAG captioning method evaluated on an external benchmark; the main caveat (unnamed eval-time caption datastore) is a data-integrity risk, not an exhibited derivation collapse.

full rationale

CONCAP is an empirical retrieval-augmented captioning model; there is no formal derivation chain to collapse. The headline result is measured on XM3600, an external benchmark, and the ablations (NoRAG, CapRAG, ConRAG) are trained and selected on COCO-35L dev splits, so hyperparameters such as m=10 are standard dev-set tuning rather than fitted predictions. The paper cites prior work by two co-authors (SmallCap, PAELLA, JEEM), but these citations supply baselines, retrieval conventions, and an oracle dataset; they do not function as a self-citation chain that forces the main result. The concept lexicons are derived from the same COCO-35L training captions used for training, which is the intended retrieval memory and does not make the XM3600 score a rename of the training fit. The one serious caveat is Section 4.1: the evaluation-time caption retrieval datastore is never named or excluded from XM3600. If that datastore contains XM3600 reference captions, the CIDEr gain would be inflated by test-set conditioning; but the paper does not state this, so under the no-speculation standard this is a correctness/leakage risk to resolve, not a circularity that can be exhibited from the text.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

CONCAP introduces no new entities or formal derivations. It relies on benchmark validity, the quality of the mSigLIP retriever, and the usefulness of noisy, translation-derived concept lists. These are empirical assumptions rather than mathematical axioms.

free parameters (5)
  • Number of retrieved concepts m = 10
    Tuned over m=4, 10, 20 on the COCO-35L development set (Appendix A.3); m=10 selected for all main results.
  • Retriever model for captions and concepts = mSigLIP
    Selected empirically over mCLIP and Jina-CLIP using CIDEr on the COCO development set (Section 4.5). The central results depend on this choice.
  • Language-specific concept prompt templates = Per-language prefixes generated by GPT-4o and verified with Google Translate
    These templates determine what the retriever matches against (Section 3.3); they are constructed by hand and not benchmarked for retrieval quality independently.
  • Number of retrieved captions n = 4
    Set to 4 following prior work (Ramos et al., 2023b), not tuned in this paper.
  • Training data subsample size = 566K image-caption pairs
    Subsampled from COCO-35L with equal representation across languages (Section 4.1); the exact selection procedure is not specified but is central to the data-efficiency claim.
assumptions (6)
  • domain assumption XM3600 human captions are a valid ground truth for multilingual caption quality.
    All conclusions rely on this benchmark and CIDEr scores computed against it.
  • domain assumption CIDEr adequately reflects caption quality across languages.
    Used as the sole optimization and evaluation metric; known to favor n-gram overlap with references.
  • domain assumption mSigLIP embedding space supports cross-lingual retrieval of single-concept templates.
    The entire concept augmentation mechanism assumes isolated tokens wrapped in templates can be matched to images across languages; retriever chosen only on COCO dev.
  • domain assumption Token-derived concept lists from translated COCO-35L contain useful visual signal despite subword tokens and stopwords.
    Section 4.1 states lists are not curated and retrieval is trusted to select meaningful tokens.
  • domain assumption Google Translate outputs for COCO-35L are adequate training data.
    The model is trained solely on machine-translated captions; translationese may bias retrieval and generation.
  • domain assumption The 566K subsample with equal language representation is representative enough for training.
    Subsampling procedure is not fully described but underpins the data-efficiency claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning." pith.science (2026). https://pith.science/paper/MKKKV5YS

@misc{pith2026250720411,
  author       = {Pith},
  title        = {Pith review of: CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKKKV5YS}},
  note         = {Machine review of arXiv:2507.20411}
}
read the original abstract

Multilingual vision-language models have made significant strides in image captioning, yet they still lag behind their English counterparts due to limited multilingual training data and costly large-scale model parameterization. Retrieval-augmented generation (RAG) offers a promising alternative by conditioning caption generation on retrieved examples in the target language, reducing the need for extensive multilingual training. However, multilingual RAG captioning models often depend on retrieved captions translated from English, which can introduce mismatches and linguistic biases relative to the source language. We introduce CONCAP, a multilingual image captioning model that integrates retrieved captions with image-specific concepts, enhancing the contextualization of the input image and grounding the captioning process across different languages. Experiments on the XM3600 dataset indicate that CONCAP enables strong performance on low- and mid-resource languages, with highly reduced data requirements. Our findings highlight the effectiveness of concept-aware retrieval augmentation in bridging multilingual performance gaps.

Figures

Figures reproduced from arXiv: 2507.20411 by the authors.

Figure 1
Figure 1. (Left) CONCAP architecture combining visual features with retrieved captions and concepts via Q-Former and LLM. (Right) Multilingual image-text retrieval using cosine similarity with prefixed concept prompts and cross-lingual caption mapping. names) instead of full captions to avoid redundancy and mitigate misleading information in the retrieved text. Zeng et al. (2024) also explores augmenting image captioning with… view at source ↗
Figure 2
Figure 2. Retrieval performance on the COCO development set across two settings: (a) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Per-language performance between CONCAP and Pangea on XM3600. 4.6 Per-language Performance In [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance difference (∆ CIDEr) on the XM100 test set when changing the wordlists: COCO-35L+XM3600* (CX), CX+Pangea (CXP), CXP+Wikipedia (CXPW). The * indicates that XM3600 has been filtered to exclude any words and images related to the test subset (XM100) to avoid c…
Figure 5
Figure 5. Figure 5: Oracle-augmented ConRAG correctly identifies the tuk-tuk in the image. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Comparison between the file sizes of the caption and the concepts datastores per [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: details per-language performance for CONCAP, ConRAG, and CapRAG. We see that CONCAP consistently outperforms the other two variants on all languages except for Maori (mi), which seems to benefit more from retrieved concepts than captions. enesnl svpt nofr daviitidfath …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 21 canonical work pages

  1. [1]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkow...

    work page 2022

  2. [2]

    mCLIP : Multilingual clip via cross-lingual transfer

    Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, and Wenping Wang. mCLIP : Multilingual clip via cross-lingual transfer. In Proceedings of ACL, Toronto, Canada, 2023

    work page 2023

  3. [3]

    Microsoft COCO captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. URL https://arxiv.org/abs/1504.00325

    arXiv 2015

  4. [4]

    Vicuna: An open-source chatbot impressing GPT-4 with 90\ See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing GPT-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2 0 (3): 0 6, 2023

    work page 2023

  5. [5]

    The Faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The Faiss library . 2024

    work page 2024

  6. [6]

    A survey on rag meeting llms: Towards retrieval-augmented large language models

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting llms: Towards retrieval-augmented large language models . In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024

    work page 2024

  7. [7]

    m BLIP : Efficient bootstrapping of multilingual vision- LLM s

    Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glava s . m BLIP : Efficient bootstrapping of multilingual vision- LLM s. In Jing Gu, Tsu-Jui (Ray) Fu, Drew Hudson, Asli Celikyilmaz, and William Wang (eds.), Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), pp.\ 7--25, Bangkok, Thailand, August 2024. Association for Co...

    doi:10.18653/v1/2024.alvr-1.2 2024

  8. [8]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models , 2021. URL https://arxiv.org/abs/2106.09685

    arXiv 2021

Show all 49 references
  1. [9]

    A survey on retrieval-augmented text generation for large language models

    Yizheng Huang and Jimmy Huang. A survey on retrieval-augmented text generation for large language models . arXiv preprint arXiv:2404.10981, 2024

    2024 arXiv

  2. [10]

    Jieba Chinese text segmentation

    Sun Junyi. Jieba Chinese text segmentation . https://github.com/fxsjy/jieba, 2012. Accessed: 2025-03-27

    2012

  3. [11]

    JEEM: Vision-Language Understanding in Four Arabic Dialects , 2025

    Karima Kadaoui, Hanin Atwany, Hamdan Al-Ali, Abdelrahman Mohamed, Ali Mekky, Sergei Tilga, Natalia Fedorova, Ekaterina Artemova, Hanan Aldarmaki, and Yova Kementchedjhieva. JEEM: Vision-Language Understanding in Four Arabic Dialects , 2025. URL https://arxiv.org/abs/2503.21910

    2025 arXiv

  4. [12]

    Jina CLIP: Your CLIP Model Is Also Your Text Retriever , 2024

    Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martínez, Saahil Ognawala, Susana Guzman, Maximilian Werk, Nan Wang, and Han Xiao. Jina CLIP: Your CLIP Model Is Also Your Text Retr...

    2024 arXiv

  5. [13]

    MeCab: Yet Another Part-of-Speech and Morphological Analyzer

    Taku Kudo. MeCab: Yet Another Part-of-Speech and Morphological Analyzer . https://taku910.github.io/mecab/, 2006. Accessed: 2025-03-27

    2006

  6. [14]

    The IndicNLP Library

    Anoop Kunchukuttan. The IndicNLP Library . https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf, 2020

    2020

  7. [15]

    What matters when building vision-language models? Advances in Neural Information Processing Systems, 37: 0 87874--87907, 2024

    Hugo Lauren c on, L \'e o Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37: 0 87874--87907, 2024

    2024

  8. [16]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP tasks. In Advances i...

    2020

  9. [17]

    Evcap: Retrieval-augmented image captioning with external visual-name memory for open-world comprehension

    Jiaxuan Li, Duc Minh Vo, Akihiro Sugimoto, and Hideki Nakayama. Evcap: Retrieval-augmented image captioning with external visual-name memory for open-world comprehension . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp.\ 13733--13742, 2024 a

    2024

  10. [18]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models , 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models , 2023. URL https://arxiv.org/abs/2301.12597

    2023 arXiv

  11. [19]

    Understanding retrieval robustness for retrieval-augmented image captioning

    Wenyan Li, Jiaang Li, Rita Ramos, Raphael Tang, and Desmond Elliott. Understanding retrieval robustness for retrieval-augmented image captioning . arXiv preprint arXiv:2406.02265, 2024 b

    2024 arXiv

  12. [20]

    Few-shot learning with multilingual language models

    Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. Few-shot learning with multilingual language models . arXiv preprint arXiv:2112.10668, 2021

    2021 arXiv

  13. [21]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning . Advances in neural information processing systems, 36: 0 34892--34916, 2023

    2023

  14. [22]

    LLaVA-NeXT: Improved reasoning, OCR, and world knowledge , January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge , January 2024. URL https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  15. [23]

    fugashi, a tool for tokenizing J apanese in python

    Paul McCann. fugashi, a tool for tokenizing J apanese in python. In Eunjeong L. Park, Masato Hagiwara, Dmitrijs Milajevs, Nelson F. Liu, Geeticka Chauhan, and Liling Tan (eds.), Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pp.\ 44--51, Online, Novembe...

    2020 doi

  16. [24]

    Ron Mokady, Amir Hertz, and Amit H. Bermano. ClipCap : Clip prefix for image captioning, 2021. URL https://arxiv.org/abs/2111.09734

    2021 arXiv

  17. [25]

    Crosslingual Generalization through Multitask Finetuning , 2023

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward ...

    2023 arXiv

  18. [26]

    GPT-4o System Card , 2024

    OpenAI. GPT-4o System Card , 2024. URL https://arxiv.org/abs/2410.21276

    2024 arXiv

  19. [27]

    X-InstructBLIP: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-InstructBLIP: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning . arXiv prep...

    2023 arXiv

  20. [28]

    BLEU: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ACL '02, pp.\ 311–318, USA, 2002. Association for Computationa...

    2002

  21. [29]

    P y T hai NLP : T hai natural language processing in P ython, June 2024

    Wannaphong Phatthiyaphaibun, Korakot Chaovavanich, Charin Polpanumas, Arthit Suriyawongkul, Lalita Lowphansirikul, and Pattarawat Chormai. P y T hai NLP : T hai natural language processing in P ython, June 2024. URL https://github.com/PyThaiNLP/pythainlp/

    2024

  22. [30]

    Learning Transferable Visual Models From Natural Language Supervision , 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision , 2021. URL https://a...

    2021 arXiv

  23. [31]

    LMC ap: Few-shot multilingual image captioning by retrieval augmented language model prompting

    Rita Ramos, Bruno Martins, and Desmond Elliott. LMC ap: Few-shot multilingual image captioning by retrieval augmented language model prompting. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, p...

    2023 doi

  24. [32]

    SmallCap : Lightweight image captioning prompted with retrieval augmentation

    Rita Ramos, Bruno Martins, Desmond Elliott, and Yova Kementchedjhieva. SmallCap : Lightweight image captioning prompted with retrieval augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023 b

    2023

  25. [33]

    PAELLA : Parameter-efficient lightweight language-agnostic captioning model

    Rita Ramos, Emanuele Bugliarello, Bruno Martins, and Desmond Elliott. PAELLA : Parameter-efficient lightweight language-agnostic captioning model. In Findings of the Association for Computational Linguistics: NAACL 2024, June 2024

    2024

  26. [34]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning . In Proceedings of the Annual Meeting of the Association for Computational Linguistics , 2018

    2018

  27. [35]

    Stanford Alpaca: An instruction-following LLaMA model , 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford Alpaca: An instruction-following LLaMA model , 2023

    2023

  28. [36]

    Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut

    Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2022...

    2022 doi

  29. [37]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models . arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  30. [38]

    SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features , 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual Vision-Language Enc...

    2025 arXiv

  31. [39]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based Image Description Evaluation , 2015. URL https://arxiv.org/abs/1411.5726

    2015 arXiv

  32. [40]

    mT5 : A massively multilingual pre-trained text-to-text transformer

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5 : A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020

    2010 arXiv

  33. [41]

    Retrieval-augmented multimodal language modeling

    Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmented multimodal language modeling . arXiv preprint arXiv:2211.12561, 2022

    2022 arXiv

  34. [42]

    Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages , 2025

    Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages , 2025. URL https://arxiv.org/abs/2410.16153

    2025 arXiv

  35. [43]

    MeaCap: Memory-augmented zero-shot image captioning

    Zequn Zeng, Yan Xie, Hao Zhang, Chiyu Chen, Bo Chen, and Zhengjue Wang. MeaCap: Memory-augmented zero-shot image captioning . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp.\ 14100--14110, 2024

    2024

  36. [44]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp.\ 12104--12113, 2022

    2022

  37. [45]

    Sigmoid Loss for Language Image Pre-Training , 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre-Training , 2023. URL https://arxiv.org/abs/2303.15343

    2023 arXiv

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  39. [47]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  40. [48]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  41. [49]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.