Pith. sign in

REVIEW 4 major objections 5 minor 99 references

UniCoRN: Unified Commented Retrieval Network with LMMs

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read UniCoRN claims that a frozen LMM, augmented only by trainable adapters, can both retrieve relevant real images for compositional queries and generate grounded comments, outperforming prompt-based RAG.

desk verdict Retrieval gains are credible and the CoR task is a useful addition, but the commenting evaluation is confounded by shared train/test generation, so treat those numbers with caution. read the letter →

arxiv 2502.08254 v1 pith:2ORITE5U submitted 2025-02-12 cs.CV

Maximilian Jaritz , Matthieu Guillaumin , Sabine Sternig , Loris Bazzani This is my paper · ORCID
classification cs.CV
keywords CommentedretrievalComposedmultimodalLargemodelsRetrieval-augmentedgenerationEntityadapterFrozenLMMContrastivelearningCIRR-CoR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a large multimodal model can be extended to both retrieve real images and write explanatory comments while its weights stay frozen. UniCoRN adds two small trainable adapters: one projects the LMM's hidden state into a CLIP-style retrieval space, and another injects the retrieved image and its text back into the LMM as a new input modality. On the new Commented Retrieval (CoR) task, built from CIRR and WikiWeb2M with human-edited golden answers, the method reports +4.5% recall over the UniIR baseline for multimodal retrieval and +14.9% METEOR and +18.4% BEM over RAG baselines for commenting. The point of the design is that a frozen LMM can keep all its original skills while gaining linked retrieval and generation, something prompt-based retrieval-augmented generation does not achieve.

What carries the argument

The load-bearing pieces are two small trainable adapters wrapped around a frozen LMM. The Hidden State Adapter $\psi_{\mathrm{LMM}}(h)=\mathrm{FC}(\mathrm{GeLU}(\mathrm{FC}(h)))$ projects the LMM's last-layer hidden state into the CLIP embedding space, producing the query representation $\psi(q)=\beta\,\psi_{\mathrm{LMM}}(H_{\mathrm{LMM}}(q))+(1-\beta)\,\psi_{\mathrm{MM}}(q)$ with a learnable mixing weight $\beta$. The Entity Adapter $\varphi^{\langle r\rangle}_\xi$ maps the retrieved image's ViT features into the LMM's input-embedding space via $\mathrm{FC}(\mathrm{GeLU}(\mathrm{FC}(\mathrm{LayerNorm}(x_m))))$, so the frozen LMM attends to those embeddings as it generates the comment. A retrieval token emitted mid-generation triggers retrieval, making the retrieved entity part of the model's own output. The retriever is trained with a contrastive loss on targets whose text includes complex comments, and the entity adapter is trained only with cross-entropy next-token prediction of ground-truth comments.

What would settle it

Take the released Wiki-CoR test set and replace the gold comments with comments written independently by human annotators who have not seen either system's output; if UniCoRN's METEOR and BEM advantage over the RAG baseline drops to near zero on this rewritten set, the reported gains mostly reflect matching the statistical style of the auto-generated training comments rather than genuinely better commenting.

Watch

Extended reading notes

Core claim

UniCoRN unifies discriminative cross-modal retrieval with generative commenting by treating retrieval as a step inside the LMM's own generation process. The retriever scores a document $d$ by $ψ(q)^T \cdot \psi_{\mathrm{MM}}(d)$, where the query embedding $ψ(q)=\beta\,\psi_{\mathrm{LMM}}(H_{\mathrm{LMM}}(q))+(1-\beta)\,\psi_{\mathrm{MM}}(q)$ combines the LMM's last hidden state with a CLIP-style multimodal encoder; documents are embedded with the cheap CLIP encoder only, keeping indexing lightweight. During generation, a special retrieval token triggers the retriever, and the Entity Adapter converts the retrieved entity's image features into the LMM's input embedding space so the frozen model can attend to them while predicting the comment. The retriever is trained with contrastive loss on query–target pairs whose text includes complex comments rather than only captions, and the Entity Adapter is trained solely by next-token prediction on ground-truth comments. The paper also introduces CIRR-CoR and Wiki-CoR datasets, whose evaluation gold sets are produced by generating question–answer pairs with an automated assistant and then filtering and editing them with human annotators, leaving 695 of 1768 Wiki-CoR pairs as the golden test set.

Load-bearing premise

The commenting result rests on assuming that the auto-generated, human-edited golden comments are a fair measure of comment quality, because if those gold comments still carry systematic patterns from the generator that produced them, the measured gains may partly reflect matching those patterns.

Editorial extensions

If this is right

  • Composed retrieval improves on the first ranks across Fashion-IQ, CIRR, OVEN, and Wiki-CoR, with the largest relative gain on Wiki-CoR (recall@1 from 21.73 to 40.14), showing that LMM hidden states carry relational information CLIP-style embeddings miss.
  • Retrieval-aware generation beats prompt-based RAG on both CoR datasets across METEOR, BLEU, ROUGE-1/2, and BEM, indicating that feeding the adapted retrieved entity into the LMM input stream conditions generation more effectively than adding it to the prompt.
  • Ablations show that the trained entity adapter, not the better retriever, is the main driver of commenting gains: swapping in the same retriever within a RAG prompt yields little improvement, while replacing the trained adapter with the pretrained visual encoder drops performance back to RAG level.
  • All components of the retrieved Wikipedia entity matter: removing the image, the caption, or the Wikipedia title/description each lowers METEOR by roughly 12–19% relative, so the entity representation should stay multimodal.
  • Because the base LMM is frozen, the design preserves its original captioning, VQA, and grounding abilities as a structural guarantee rather than as a property that must be re-verified after fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: since the entity adapter injects embeddings rather than a fixed prompt, the same architecture should support multi-entity commenting, where several retrieved images are shown and each is discussed; the paper evaluates only the single best entity per query.
  • Beyond the paper: because only two small adapters are trained, the recipe is LMM-agnostic; a direct test would swap the base InternVL2-4B for another open LMM with accessible hidden states and visual features and rerun the CoR experiments to see whether the gains transfer.
  • Beyond the paper: the CoR datasets are generated by pairing images from the same Wikipedia article and asking an automated assistant for question–answer pairs; one could test whether these AI-flavored gold comments bias the metrics by re-scoring both systems against independently authored human comments on the same image pairs.
  • Beyond the paper: the underlying motivation cites evidence that humans learn better when they see text alongside visuals, but the paper itself does not measure learning or user preference; a human study comparing UniCoRN-style multimodal answers with text-only or image-only answers would test that motivation directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces UniCoRN, a unified framework that augments a frozen Large Multimodal Model (InternVL2-4B) with two trainable modules: a comment-aware retrieval adapter that maps LMM hidden states into the space of a CLIP-based retriever, and an entity adapter that injects retrieved multimodal documents back into the frozen LMM for conditioning the generation of textual comments. The authors also define the Commented Retrieval (CoR) task and construct two datasets, CIRR-CoR and Wiki-CoR, by pairing images, generating questions/answers with Claude 3.5 Sonnet, and applying human filtering/editing. Experiments report an average +4.5% recall improvement over UniIR across five composed-retrieval benchmarks, and +14.9 METEOR / +18.4 BEM improvements over a RAG baseline for commenting on the two CoR datasets.

Significance. If the claims hold, the paper makes a useful contribution: it demonstrates that a frozen LMM can be augmented with lightweight adapters to perform both composed retrieval and image-grounded commenting, preserving the base model's capabilities by design. The retrieval results on established benchmarks (Fashion-IQ, CIRR, OVEN, InfoSeek) use external ground truth and thus offer a sound, independent basis for the retrieval contribution. The CoR task and datasets fill a real gap, as existing datasets lack rich, complementary textual comments. The main weakness is that the commenting evaluation is currently confounded: the training labels and the gold evaluation comments are produced by the same Claude-plus-human-editing pipeline, so the reported gains over RAG may partly reflect distribution matching rather than genuinely better commenting, and no human preference study is provided.

major comments (4)
  1. [Sec. 3.2.3, Sec. 4, Table 3] The commenting evaluation conflates the effect of making the retrieved image available to the LMM with the effect of training the entity adapter to imitate the comment distribution. The entity adapter is trained by next-token prediction on comments generated by Claude 3.5 Sonnet and human-edited (Sec. 3.2.3), and the golden test sets are built through exactly the same Claude-plus-editing pipeline (Sec. 4). The RAG baselines are never trained on this distribution. Consequently, the reported +14.9 METEOR and +18.4 BEM improvements over RAG may largely reflect matching the reference-generation distribution rather than producing comments that are factually better or more image-grounded. The paper provides no human preference study and no evaluation against independently written human references; the SigLIP-based metric is also reference-based. To make the commenting claim load-bearing, please add (i) a human evaluation comparing UniCoRN and RAG comments, or (ii) an evaluation on gold comments written by independent human annotators without LLM pre-generation, or (iii) an analysis that shows the adapter's gains persist when the gold distribution is replaced by another source.
  2. [Sec. 1, Sec. 3.2] The paper states that because the base LMM is frozen, UniCoRN 'guarantees the preservation of all its original capabilities' (e.g., captioning, VQA, grounding). But no experiment verifies this. Since the entity adapter injects new learned embeddings into the LMM's input, it is possible that the adapter interferes with the original input distribution, and the guarantee is by construction only for the frozen weights, not for the actual behavior. Please report standard benchmarks (e.g., VQAv2, COCO captioning, or an instruction-following suite) with the original InternVL2 and with UniCoRN's adapters attached, with the retrieval path ablated or not triggered, to demonstrate no degradation.
  3. [Abstract, Table 3] The headline numbers are ambiguous. The +14.9% METEOR is the average of the absolute METEOR increases on CIRR-CoR (0.444 vs 0.313, +13.1) and Wiki-CoR (0.379 vs 0.212, +16.7), while the +18.4% BEM matches only the CIRR-CoR increase (50.4 vs 32.0) and is not the average (the Wiki-CoR increase is +17.9). Please state in the abstract and intro exactly which comparison is being made (per-dataset or average) and report the underlying per-dataset numbers, to avoid misleading relative-percentage interpretations.
  4. [Abstract, Table 1] The '+4.5% recall over the state of the art' is the arithmetic mean of differences across heterogeneous recall metrics (R@1/5/10 or R@10/20/50) and across five datasets, and it hides a consistent regression on InfoSeek (R@1 26.41 vs 27.08, R@5 48.01 vs 49.04, R@10 57.24 vs 58.24) and a similar R@50 on Fashion-IQ (43.33 vs 43.66). Because the average is dominated by Wiki-CoR (+18.4 R@1) and CIRR R@1 (+6.95), the claim that UniCoRN improves composed retrieval over the state of the art should be reported with per-dataset confidence intervals or paired significance tests, and the InfoSeek regression should be explicitly acknowledged in the abstract-level summary.
minor comments (5)
  1. [Table 2] The ablation row 'w/o Wiki-CoR' yields CIRR R@1 of 17.29 vs. 16.45 for the full joint model, so removing Wiki-CoR improves CIRR, which is at odds with the text's claim that omitting Wiki-CoR causes a 'big drop in performance' without qualification. Please discuss this per-dataset trade-off.
  2. [Table 1] The reported 'Average' column is an arithmetic mean over different recall@k metrics across datasets; this should be defined explicitly or replaced with a consistent metric (e.g., a single fixed k for all datasets).
  3. [Sec. 5] The paper reports no error bars or multiple-seed runs. Given the small golden set sizes (695 Wiki-CoR test comments), please report variance or at least bootstrap confidence intervals for the main retrieval and commenting tables.
  4. [Sec. 4] The text says 'we subsample a set of 50K samples for training and 2K for testing' for Wiki-CoR, but later reports that only 695 of 1768 test pairs survive the manual review; please clarify the exact sizes of the CIRR-CoR and Wiki-CoR train/validation/test splits used for retrieval and commenting.
  5. [References] Reference [10] and [11] appear to be the same paper (Burns et al., 2023); please deduplicate. Also, the abstract uses '+' signs inconsistently with percentage-point units; use 'pp' or 'absolute' where appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: retrieval results are benchmarked against external ground truth and the commenting results are standard supervised evaluation on held-out references.

full rationale

UniCoRN's derivation chain is not circular. The retrieval contribution (Sec. 3.1, Eq. 3) is a learned combination of CLIP embeddings and LMM hidden states trained with contrastive loss, and Table 1 reports recall on Fashion-IQ, CIRR, OVEN, InfoSeek, and Wiki-CoR with external or independently constructed ground-truth image targets; no fitted parameter is renamed as a prediction, and the comparison against UniIR is a fair external baseline. The commenting contribution (Sec. 3.2.3, Eq. 8) trains the entity adapter with next-token prediction on (query, document, comment) triples and evaluates on held-out golden comments in Table 3. It is true that both the training comments and the golden evaluation comments originate from the same Claude 3.5 Sonnet plus human-editing pipeline described in Sec. 4, but this is a concern about benchmark validity and reference quality, not a circular derivation: the model is not defined in terms of the test metric, the RAG baseline is not trained on that distribution, and the evaluation uses held-out samples. The paper also reports human auditing of CIRR-CoR comments (97% marked high quality), which, even if imperfect, does not make the measured improvement an identity. The only self-citations (Refs. [16] and [37], involving co-author Bazzani) are related-work attributions for interactive fashion retrieval and are not load-bearing for any central claim; no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The core retrieval claim is therefore self-contained against external benchmarks, and the commenting claim is standard supervised evaluation rather than a circular step.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces new datasets and a task, which are evaluation artifacts rather than postulated physical or theoretical entities. No new particles, forces, dimensions, or conserved quantities are introduced. The free parameters are the learned adapter weights and the scalar blend weight that the method depends on.

free parameters (3)
  • beta (retrieval blend weight) = learned, value not reported
    Eq. 3: controls the interpolation between the LMM hidden state projection and the CLIP embedding; fitted on retrieval data.
  • Hidden state adapter weights (psi_LMM) = trained MLP, values not reported
    Eq. 2: projects the LMM hidden state into the CLIP embedding space; fitted via contrastive loss.
  • Entity adapter weights (phi_r_xi) = trained per dataset, values not reported
    Eq. 7: adapts retrieved image features into the LMM input space; trained by next-token prediction on generated comments.
assumptions (4)
  • domain assumption CLIP embedding space and LMM hidden state space can be aligned by a two-layer MLP.
    Invoked in Sec. 3.1, Eq. 2; the validity of the projection is assumed and tested only indirectly via retrieval accuracy.
  • domain assumption The hidden state at the last input token before generation captures the full compositional reasoning about the query image and text.
    Used in Sec. 3.1 to define h = H_LMM(q); if this token is not a good summary, the retrieval adapter has no useful signal.
  • ad hoc to paper Comments generated by Claude 3.5 Sonnet and then human-edited are a valid ground truth for useful commenting.
    This underpins the CoR task in Sec. 4; the model is trained and evaluated on the same style of generated comments, so the benchmark's validity depends on this assumption.
  • standard math Contrastive loss with in-batch negatives is sufficient to learn universal multimodal retrieval.
    Used in Sec. 3.1 and 5.1; a standard training choice, not new to this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniCoRN: Unified Commented Retrieval Network with LMMs." pith.science (2026). https://pith.science/paper/2ORITE5U

@misc{pith2026250208254,
  author       = {Pith},
  title        = {Pith review of: UniCoRN: Unified Commented Retrieval Network with LMMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ORITE5U}},
  note         = {Machine review of arXiv:2502.08254}
}
read the original abstract

Multimodal retrieval methods have limitations in handling complex, compositional queries that require reasoning about the visual content of both the query and the retrieved entities. On the other hand, Large Multimodal Models (LMMs) can answer with language to more complex visual questions, but without the inherent ability to retrieve relevant entities to support their answers. We aim to address these limitations with UniCoRN, a Unified Commented Retrieval Network that combines the strengths of composed multimodal retrieval methods and generative language approaches, going beyond Retrieval-Augmented Generation (RAG). We introduce an entity adapter module to inject the retrieved multimodal entities back into the LMM, so it can attend to them while generating answers and comments. By keeping the base LMM frozen, UniCoRN preserves its original capabilities while being able to perform both retrieval and text generation tasks under a single integrated framework. To assess these new abilities, we introduce the Commented Retrieval task (CoR) and a corresponding dataset, with the goal of retrieving an image that accurately answers a given question and generate an additional textual response that provides further clarification and details about the visual information. We demonstrate the effectiveness of UniCoRN on several datasets showing improvements of +4.5% recall over the state of the art for composed multimodal retrieval and of +14.9% METEOR / +18.4% BEM over RAG for commenting in CoR.

Figures

Figures reproduced from arXiv: 2502.08254 by the authors.

Figure 1
Figure 1. Commented retrieval [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comment-aware Retrieval. The query is inputed to both a CLIP-trained image-text encoder and an LMM. The LMM rep￾resentation is projected to the space of the image-text encoder. Alignment of query and targets is done using contrastive loss. where FCs are fully connected layers and GeLU is the ac￾tivation function. The final embedding for q combines the CLIP-based embeddings and the adapter output: ψ(q) = β · ψLMM(HLM… view at source ↗
Figure 3
Figure 3. Retrieval-aware Generation. The query image and text are fed to the LMM, which asks the retriever for relevant entities. The best entity is provided to the user and adapted into the LMM, so it can attend to it for generating a useful comment. p(τn|τ0, . . . , τn−1). The first layer of the LLM projects the tokens into an input embedding space, mapping each of the V possible text tokens to a vector φ ⟨t⟩ (τk) = ϕ ⟨t⟩ … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results retrieval. We show retrieved images for UniIR and UniCoRN on three datasets. Captions are not displayed because of space limits. CIRR-CoR Wiki-CoR METEOR BLEU ROUGE-1 ROUGE-2 BEM SigLIP-R@1 METEOR BLEU ROUGE-1 ROUGE-2 BEM SigLIP-R@1 InternVL2 [19] 0…
Figure 5
Figure 5. Figure 5: Qualitative results. We show retrieved images and comments for UniIR with RAG and UniCoRN on two different datasets. Comments highlighted in red indicate responses that either do not answer the original question or are not related to the retrieved image. a plausible co…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

99 extracted references · 40 canonical work pages

  1. [1]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Am ´elie H ´eliou, Paul Ja- cob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073 ,

    arXiv

  2. [2]

    Learning attribute representations with local- ization for flexible fashion search

    Kenan E Ak, Ashraf A Kassim, Joo Hwee Lim, and Jo Yew Tham. Learning attribute representations with local- ization for flexible fashion search. In CVPR, 2018. 1

    2018

  3. [3]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 6077–6086, 2018. 1

    2018

  4. [4]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024. 1, 2

    2024

  5. [5]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 1

    2015

  6. [6]

    Effective conditioned and composed im- age retrieval combining clip-based features

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Al- berto Del Bimbo. Effective conditioned and composed im- age retrieval combining clip-based features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 21466–21474, 2022. 1, 2

    2022

  7. [7]

    Zero-shot composed image retrieval with textual inversion

    Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. Zero-shot composed image retrieval with textual inversion. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 15338–15347, 2023. 2

    2023

  8. [8]

    Vlmo: Unified vision-language pre-training with mixture-of-modality-experts

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems , 35:32897–32912,

Show all 99 references
  1. [9]

    Tomayto, tomahto

    Jannis Bulian, Christian Carl Friedrich Buck, Woj- ciech Paweł Gajewski, Benjamin Boerschinger, and Tal Schuster. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. In EMNLP 2022, 2022. 7

    2022

  2. [10]

    A suite of generative tasks for multi- level multimodal webpage understanding

    Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Ge- off Brown, Bryan A Plummer, Kate Saenko, Jianmo Ni, and Mandy Guo. A suite of generative tasks for multi- level multimodal webpage understanding. arXiv preprint arXiv:2305.03668, 2023. 5

    2023 arXiv

  3. [11]

    Plummer, Kate Saenko, Jianmo Ni, and Mandy Guo

    Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, and Mandy Guo. A suite of generative tasks for multi-level multimodal webpage understanding. In The 2023 Confer- ence on Empirical Methods in Natural Language Processing (E...

    2023

  4. [12]

    Wiki-llava: Hierarchical retrieval-augmented gener- ation for multimodal llms

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cuc- chiara. Wiki-llava: Hierarchical retrieval-augmented gener- ation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  5. [13]

    Murag: Multimodal retrieval-augmented generator for open question answering over images and text

    Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. arXiv preprint arXiv:2210.02928, 2022. 2

    2022 arXiv

  6. [14]

    Pali-3 vision language models: Smaller, faster, stronger

    Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul V oigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023. 1, 2

    arXiv

  7. [15]

    Image search with text feedback by visiolinguistic attention learn- ing

    Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learn- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2020. 2

    2020

  8. [16]

    Image search with text feedback by visiolinguistic attention learn- ing

    Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolinguistic attention learn- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 3001–3011,

  9. [17]

    Can pre-trained vision and language models answer vi- sual information-seeking questions? arXiv preprint arXiv:2302.11713, 2023

    Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, So- ravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can pre-trained vision and language models answer vi- sual information-seeking questions? arXiv preprint arXiv:2302.11713, 2023. 2, 5, 6

    2023 arXiv

  10. [18]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open- source suites, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhan...

    2024

  11. [19]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  12. [20]

    Meteor universal: Lan- guage specific translation evaluation for any target language

    Michael Denkowski and Alon Lavie. Meteor universal: Lan- guage specific translation evaluation for any target language. 9 In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation, 2014. 6

    2014

  13. [21]

    Hyper- bolic image-text representations

    Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedantam. Hyper- bolic image-text representations. In International Confer- ence on Machine Learning, pages 7694–7731. PMLR, 2023. 1

    2023

  14. [22]

    Toutanova

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina N. Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. 2018. 7

    2018

  15. [23]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  16. [24]

    Entities as ex- perts: Sparse memory access with entity supervision

    Thibault F ´evry, Livio Baldini Soares, Nicholas FitzGer- ald, Eunsol Choi, and Tom Kwiatkowski. Entities as ex- perts: Sparse memory access with entity supervision. arXiv preprint arXiv:2004.07202, 2020. 2

    2004 arXiv

  17. [25]

    Pyramidclip: Hierarchi- cal feature alignment for vision-language model pretrain- ing

    Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramidclip: Hierarchi- cal feature alignment for vision-language model pretrain- ing. Advances in Neural Information Processing Systems , 35:35959–35970, 2022. 1, 2, 3

    2022

  18. [26]

    Softclip: Softer cross- modal alignment makes clip stronger

    Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Wei Liu, Jie Yang, Ke Li, and Xing Sun. Softclip: Softer cross- modal alignment makes clip stronger. arXiv preprint arXiv:2303.17561, 2023. 1, 3

    2023 arXiv

  19. [27]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218, 2023. 2

    2023 arXiv

  20. [28]

    Cyclip: Cyclic contrastive language-image pretraining

    Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Informa- tion Processing Systems, 35:6704–6719, 2022. 1

    2022

  21. [29]

    Fashionvlp: Vision language transformer for fashion re- trieval with feedback

    Sonam Goenka, Zhaoheng Zheng, Ayush Jaiswal, Rakesh Chada, Yue Wu, Varsha Hedau, and Pradeep Natarajan. Fashionvlp: Vision language transformer for fashion re- trieval with feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p...

    2022

  22. [30]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  23. [31]

    Language-only training of zero-shot com- posed image retrieval

    Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun. Language-only training of zero-shot com- posed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13225–13234, 2024. 2

    2024

  24. [32]

    Dialog-based interactive image retrieval

    Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogerio Feris. Dialog-based interactive image retrieval. In Advances in Neural Information Processing Sys- tems, 2018. 1

    2018

  25. [33]

    The fashion iq dataset: Retrieving images by combining side information and relative natural language feedback

    Xiaoxiao Guo, Hui Wu, Yupeng Gao, Steven Rennie, and Rogerio Feris. The fashion iq dataset: Retrieving images by combining side information and relative natural language feedback. arXiv preprint arXiv:1905.12794, 2019. 1, 5, 6

    1905 arXiv

  26. [34]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  27. [35]

    Retrieval augmented language model pre- training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre- training. In International conference on machine learning , pages 3929–3938. PMLR, 2020. 2

    2020

  28. [36]

    Au- tomatic spatially-aware fashion concept discovery

    Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Au- tomatic spatially-aware fashion concept discovery. In ICCV,

  29. [37]

    Learning attribute-driven disentangled represen- tations for interactive fashion retrieval

    Yuxin Hou, Eleonora Vig, Michael Donoser, and Loris Bazzani. Learning attribute-driven disentangled represen- tations for interactive fashion retrieval. In Proceedings of the IEEE/CVF International conference on computer vision, pages 12147–12157, 2021. 1

    2021

  30. [38]

    Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities

    Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming- Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer...

    2023

  31. [39]

    Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge mem- ory

    Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge mem- ory. In Proceedings of the IEEE/CVF conference on com-...

  32. [40]

    Openclip, 2021

    Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. 1, 2, 3, 5

    2021

  33. [41]

    Mantis: Interleaved multi-image instruction tuning, 2024

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning, 2024. 1, 2

    2024

  34. [42]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160, 2024. 2, 3

    2024 arXiv

  35. [43]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas O ˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen- tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020. 2

    2004 arXiv

  36. [44]

    Vision-by-language for training- free compositional image retrieval

    Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. Vision-by-language for training- free compositional image retrieval. arXiv preprint arXiv:2310.09291, 2023. 2

    2023 arXiv

  37. [45]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014. 2 10

    2014

  38. [46]

    Grounding language models to images for multimodal in- puts and outputs

    Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal in- puts and outputs. In International Conference on Machine Learning, pages 17283–17300. PMLR, 2023. 2

    2023

  39. [47]

    Uni- clip: Unified framework for contrastive language-image pre- training

    Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uni- clip: Unified framework for contrastive language-image pre- training. arXiv preprint arXiv:2209.13430, 2022. 1, 2

    2022 arXiv

  40. [48]

    La- tent retrieval for weakly supervised open domain question answering

    Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. La- tent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00300, 2019. 2

    1906 arXiv

  41. [49]

    Chatting makes perfect: Chat-based image retrieval

    Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischin- ski. Chatting makes perfect: Chat-based image retrieval. Ad- vances in Neural Information Processing Systems, 36, 2024. 2

    2024

  42. [50]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Syst...

    2020

  43. [51]

    Retrieval-augmented genera- tion for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen tau Yih, Tim Rockt¨aschel, Sebas- tian Riedel, and Douwe Kiela. Retrieval-augmented genera- tion for knowledge-intensive nlp tasks, 2021. 2

    2021

  44. [52]

    Textbind: Multi-turn interleaved multimodal instruction- following in the wild

    Huayang Li, Siheng Li, Deng Cai, Longyue Wang, Lemao Liu, Taro Watanabe, Yujiu Yang, and Shuming Shi. Textbind: Multi-turn interleaved multimodal instruction- following in the wild. 2023. 2

    2023

  45. [53]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 1, 2

    2022

  46. [54]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 6

    2004

  47. [55]

    Mm-embed: Universal multimodal retrieval with multimodal llms

    Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms. arXiv preprint arXiv:2411.02571, 2024. 2, 3

    2024 arXiv

  48. [56]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023. 1, 4

    2023

  49. [57]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 1, 2

    2023

  50. [58]

    Image retrieval on real-life images with pre-trained vision-and-language models

    Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2125–2134, 2021. 2, 5, 6

    2021

  51. [59]

    Image retrieval on real-life images with pre- trained vision-and-language models

    Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. Image retrieval on real-life images with pre- trained vision-and-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2125–2134, 2021. 2

    2021

  52. [60]

    Bi-directional training for composed im- age retrieval via text prompt learning

    Zheyuan Liu, Weixuan Sun, Yicong Hong, Damien Teney, and Stephen Gould. Bi-directional training for composed im- age retrieval via text prompt learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5753–5762, 2024. 2

    2024

  53. [61]

    Three facets of visual and verbal learners: Cognitive ability, cognitive style, and learning preference

    Richard E Mayer and Laura J Massa. Three facets of visual and verbal learners: Cognitive ability, cognitive style, and learning preference. Journal of educational psychology, 95 (4):833, 2003. 2

    2003

  54. [62]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 1, 2

    2024 arXiv

  55. [63]

    Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories

    Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, Andr ´e Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Confer...

    2023

  56. [64]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. 1, 2

    2024

  57. [65]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , page 311–318, USA, 2002. Association for Computational Linguistics. 6

    2002

  58. [66]

    Rora-vlm: Robust retrieval-augmented vision language models, 2024

    Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jing Di, Yu Cheng, Qifan Wang, and Lifu Huang. Rora-vlm: Robust retrieval-augmented vision language models, 2024. 2

    2024

  59. [67]

    Alleviating hallucination in large vision-language models with active retrieval augmentation

    Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, and Jianfeng Dong. Alleviating hallucination in large vision-language models with active retrieval augmentation. arXiv preprint arXiv:2408.00555, 2024. 2

    2024 arXiv

  60. [68]

    Learning transferable vi- sual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable vi- sual models from natural language supervision. CoRR, abs/2103.000...

    2021 arXiv

  61. [69]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 1, 2

    2021 arXiv

  62. [70]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In European conference on computer vision , pages 146–162. Springer, 2022. 1

    2022

  63. [71]

    Kvqa: Knowledge-aware visual question answering

    Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, pages 8876–8884, 2019

    2019

  64. [72]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 1 11

    2019

  65. [73]

    Knowledge-enhanced dual-stream zero-shot composed im- age retrieval

    Yucheng Suo, Fan Ma, Linchao Zhu, and Yi Yang. Knowledge-enhanced dual-stream zero-shot composed im- age retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26951–26962, 2024. 2

    2024

  66. [74]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 2

    2024 arXiv

  67. [75]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context, 2024. 1, 2

    2024

  68. [76]

    Mm-interleaved: Interleaved image-text generative model- ing via multi-modal feature synchronizer

    Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, Hongsheng Li, Yu Qiao, and Jifeng Dai. Mm-interleaved: Interleaved image-text generative model- ing via multi-modal feature synchronizer. arXiv preprint arXiv:...

    2024 arXiv

  69. [77]

    Genecis: A benchmark for general conditional image similarity

    Sagar Vaze, Nicolas Carion, and Ishan Misra. Genecis: A benchmark for general conditional image similarity. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6862–6872, 2023. 2

    2023

  70. [78]

    Composing text and image for image retrieval - an empirical odyssey

    Nam V o, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval - an empirical odyssey. In CVPR, 2019. 1

    2019

  71. [79]

    Composing text and image for image retrieval-an empirical odyssey

    Nam V o, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6439–6448, 2019. 2

    2019

  72. [80]

    Cross-modal feature alignment and fusion for com- posed image retrieval

    Yongquan Wan, Wenhai Wang, Guobing Zou, and Bofeng Zhang. Cross-modal feature alignment and fusion for com- posed image retrieval. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 8384–8388, 2024. 2

    2024

  73. [81]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 2

    2024 arXiv

  74. [82]

    Image as a foreign language: Beit pretraining for vision and vision- language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision- language tasks. In Proceedings of the IEEE/CVF Conferen...

    2023

  75. [83]

    Uniir: Training and benchmarking universal multimodal information retriev- ers

    Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retriev- ers. arXiv preprint arXiv:2311.17136, 2023. 2, 3, 5, 6, 7, 8

    2023 arXiv

  76. [84]

    Fashion iq: A new dataset towards retrieving images by natural language feedback

    Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , p...

    2021

  77. [85]

    Fashion iq: A new dataset towards retrieving images by natural language feedback

    Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. Fashion iq: A new dataset towards retrieving images by natural language feedback. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 11...

    2021

  78. [86]

    Visual question answer- ing: A survey of methods and datasets

    Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Visual question answer- ing: A survey of methods and datasets. Computer Vision and Image Understanding, 163:21–40, 2017. 1

    2017

  79. [87]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023. 1, 2

    2023 arXiv

  80. [88]

    EchoSight: Advancing visual- language models with Wiki knowledge

    Yibin Yan and Weidi Xie. EchoSight: Advancing visual- language models with Wiki knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 1538–1551, Miami, Florida, USA, 2024. Association for Computational Linguistics. 1, 2, 7, 8

    2024

  81. [89]

    Filip: fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. 1, 2

    2021 arXiv

  82. [90]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 1, 2

    2024 arXiv

  83. [91]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 2

    2023 arXiv

  84. [92]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 1, 2

    2022 arXiv

  85. [93]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 2

    arXiv

  86. [94]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594, 2024. 2

    arXiv

  87. [95]

    A large-scale study of representation learning with the visual task adaptation benchmark

    Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djo- longa, Andre Susano Pinto, Maxim Neumann, Alexey Doso- vitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv ...

    1910 arXiv

  88. [96]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 1, 2, 7

    2023

  89. [97]

    Magi- clens: Self-supervised image retrieval with open-ended in- structions

    Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wei Chang. Magi- clens: Self-supervised image retrieval with open-ended in- structions. arXiv preprint arXiv:2403.19651, 2024. 2, 6 12

    2024 arXiv

  90. [98]

    Non-contrastive learning meets language-image pre- training

    Jinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang, and Furu Wei. Non-contrastive learning meets language-image pre- training. arXiv preprint arXiv:2210.09304, 2022. 1, 2

    2022 arXiv

  91. [99]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024. 1, 2 13

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.