Pith. sign in

REVIEW 2 major objections 6 minor 7 cited by

Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey claims that multimodal retrieval-augmented generation has become a distinct, mappable field, and it organizes over one hundred recent systems into a pipeline taxonomy that separates retrieval, fusion, augmentation, generation…

desk verdict Useful survey of multimodal RAG with a sensible taxonomy, but the 'comprehensive' claim needs a reproducible search protocol before it can be trusted. read the letter →

arxiv 2502.08826 v3 pith:5TDKIG7B submitted 2025-02-12 cs.CL cs.AIcs.IR

Mohammad Mahdi Abootorabi , Amirhosein Zobeiri , Mahdi Dehghani , Mohammadali Mohammadkhani , Bardia Mohammadi , Omid Ghahroodi , Mahdieh Soleymani Baghshah , Ehsaneddin Asgari This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords multimodalretrieval-augmentedgenerationtaxonomycross-modalretrievalfusionevaluationbenchmarksagenticRAGknowledgegrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal retrieval-augmented generation (RAG) is the attempt to ground AI generation in external knowledge that is not just text but also images, audio, video, tables, and documents. The paper establishes that this is a distinct design space with problems ordinary text RAG does not have: how to encode different modalities into one comparable space, how to retrieve across them, how to fuse them into a coherent context, and how to make a generator cite and reason over mixed evidence. Its central contribution is a taxonomy of over one hundred recent systems, organized by where their innovation sits in the pipeline, together with a formal query-retrieval-generation formulation, a catalogue of datasets and benchmarks, and an account of open problems such as modality bias, coarse attribution, and adversarial knowledge poisoning. The paper claims the field has matured enough to be mapped, and that the taxonomy's categories are the right ones for comparing and building future systems.

What carries the argument

The carrying object is the taxonomy itself, backed by a formal pipeline: a multimodal corpus $D$, modality-specific encoders producing $z_i = \mathrm{Enc}_{M_{d_i}}(d_i)$, a retrieval model $R$ scoring $s(e_q, z_i)$ against a modality-specific threshold $\tau$, and a generator $r = G(q, X)$ over the retrieved context $X$. This formulation makes every surveyed method comparable by locating its contribution in one pipeline stage. The taxonomy's categories, covering retrieval strategies, fusion mechanisms, augmentation techniques, generation techniques, and training strategies, with agentic systems inside generation, are what allow the survey to present a scattered literature as a structured design space.

What would settle it

A reader could test the comprehensiveness claim by running a documented, reproducible search of the same venues for multimodal RAG systems and checking whether all found systems fit one of the taxonomy's families; if a substantial share falls outside or spans families in a way the taxonomy cannot express, the organizing claim fails. A weaker test: count published multimodal RAG benchmarks and datasets against the survey's inventory and look for missing ones that change the open-problems list.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that multimodal RAG systems can be understood as instantiations of one pipeline: encode each document in its modality, score query-document relevance in a shared space, select documents above a threshold, and condition a generator on the selected context. It then reads more than one hundred recent systems as innovations in one of six places in that pipeline: retrieval strategy, fusion mechanism, augmentation, generation technique, training strategy, or agentic interaction. The taxonomy is innovation-driven rather than application-driven, which distinguishes it from the one earlier survey it identifies. The paper also inventories the evaluation apparatus, including about sixty metrics and dozens of datasets and benchmarks, and uses that apparatus to expose open problems: systems over-rely on text, attributions are coarse, and a few adversarial knowledge injections can hijack cross-modal retrieval.

Load-bearing premise

The load-bearing premise is that the papers the authors selected, mainly from major NLP and machine-learning venues, are representative of the field and that their taxonomy can absorb every genuine innovation, a premise the paper does not make verifiable because it provides no search strategy, inclusion criteria, or exclusion rules.

Editorial extensions

If this is right

  • If the taxonomy is right, a new multimodal RAG system can be described by where it intervenes in the pipeline, and gaps in the map indicate genuinely under-explored territory.
  • The formal formulation implies that every multimodal RAG system must solve a cross-modal relevance threshold problem; the survey's examples show the field has not settled on a universal threshold or a universal embedding space.
  • The evaluation catalogue implies that current practice splits retrieval quality and generation quality, so systems optimized on one may be misjudged on the other.
  • The robustness findings imply that multimodal RAG inherits text RAG's poisoning risk and adds a new one: misleading images or audio can steer retrieval, so defenses must be cross-modal.
  • The agentic trend implies that the next generation of systems will treat retrieval not as a one-shot lookup but as a plan that can branch, iterate, and cite.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the survey's taxonomy could be used as a classification instrument: a reader could take the next year's multimodal RAG papers, assign each to one of the six families, and measure whether the field's center of gravity is shifting from retrieval quality toward agentic planning.
  • Editorial inference: the paper's emphasis on unified embedding spaces suggests a testable research programme: if a single encoder could embed text, image, audio, and video in one metric space, the modality-specific thresholds in the formalization would collapse into one global threshold, likely changing retrieval evaluation.
  • Editorial inference: the poisoning examples imply a standardized red-team benchmark is missing: a suite of cross-modal adversarial injections that measures how many poisoned documents are needed to flip a generated answer would make the robustness claims comparable across systems.
  • Editorial inference: the survey does not compare systems head-to-head, so an implied sequel is an empirical benchmark that fixes datasets, metrics, and compute budget across the six families; until then, the taxonomy orders ideas, not measured performance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper presents a survey of multimodal Retrieval-Augmented Generation (RAG), proposing a taxonomy that organizes recent work into retrieval strategies, fusion mechanisms, augmentation techniques, generation techniques, and training strategies. It also compiles datasets, benchmarks, evaluation metrics, applications, and open problems, and it provides a public GitHub repository with additional resources. The survey covers over 100 recent papers, primarily from ACL, EMNLP, NeurIPS, CVPR, ICLR, ICML, ACM Multimedia, and arXiv, and it distinguishes itself from an earlier survey by adopting an innovation-driven perspective.

Significance. If the coverage is reliable, the survey fills a genuine gap: the only prior survey on multimodal RAG (Zhao et al., 2023a) is organized by application and modality, whereas this paper offers a more method-centric taxonomy and includes recent audio- and video-centric approaches, agent-based systems, and robustness considerations. The public repository is a practical contribution that can help researchers navigate the field. However, the survey's value as an analysis is limited by the absence of a reproducible literature search protocol and by the very brief, non-comparative descriptions of individual methods. These issues do not negate the paper's usefulness as an entry point, but they do prevent the 'comprehensive analysis' claim from being fully substantiated.

major comments (2)
  1. [Section 6 (Limitations) and Section 1 (Related Works)] The claim that the survey is 'comprehensive' is not verifiable because the paper does not report its literature search methodology. It states that studies were curated from major venues and arXiv, but it does not provide the search queries, databases, date range, inclusion/exclusion criteria, screening process, or a list of excluded papers. As a result, the taxonomy in Figure 2 and the coverage claims cannot be independently reproduced or assessed, and any selection bias is invisible. Please add a methodology subsection describing the search and selection process, and include the full list of included works with their provenance.
  2. [Sections 3.1–3.5 and Appendix C] The abstract and conclusion promise a 'comprehensive analysis,' but the body of the survey is largely a catalog: most methods are described in a single sentence and no comparative synthesis is provided. For example, the fusion mechanisms in Section 3.2 are presented as separate techniques without any discussion of when one should be preferred over another, what their computational costs are, or what empirical evidence supports them. Similarly, Appendix C lists retrieval and generation metrics but does not analyze their suitability or limitations for multimodal RAG evaluation. To support the 'analysis' claim, the authors should either add comparative discussion and synthesis (e.g., tables contrasting methods by task, modality, and reported results) or revise the claims to describe the work as a structured taxonomy rather than an analysis.
minor comments (6)
  1. [Appendix C, Eq. (4)] The ROUGE-L formula is given as LCS(X,Y)/|Y|, which is only the recall component; the standard ROUGE-L F-measure should be used.
  2. [Table 1, MIMIC-IT row] The citation (Li et al., 2025a) points to the Otter paper in the reference list, not to the MIMIC-IT dataset; please correct the citation.
  3. [Table 1, ELI5 row] The entry lists Text as the only modality, but the ELI5 dataset includes web pages and images; please correct the modality assignment.
  4. [Figure 2] Several entries are duplicated (e.g., Xue et al. (2024b) appears under both Vision-Centric and Context Enrichment, and the name 'VideoRAG' is used for two distinct papers), which can confuse readers; consider numbering entries or using unique labels.
  5. [Section 1, Multimodal RAG Formulation] The use of a modality-specific threshold tau_{M_di} is introduced but never discussed; please clarify how such thresholds are set in practice or remove the formalism.
  6. [Section 6 (Limitations)] The phrase 'we have made our maximum effort; however, some limits may persist' is ungrammatical; please rephrase.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's claims rest on external literature, not on the authors' own outputs.

full rationale

This paper is a survey, not a derivation or empirical study. It presents a mathematical formulation of multimodal RAG (Section 1), a taxonomy (Figure 2), and summaries of external papers; none of these steps reduces to its own inputs. The central claim is that the survey offers a 'structured and comprehensive analysis' of multimodal RAG. That claim is supported by the cited literature and by the paper's own organization, not by any fitted parameter or by a self-referential derivation. The only self-citation is the authors' SpeechBrown dataset, listed in Table 1 among many audio-text datasets. This entry is not load-bearing: the taxonomy, the discussion of retrieval/fusion/augmentation/generation techniques, and the open challenges do not depend on SpeechBrown, and removing it would not change any conclusion. The Limitations section concedes that selection 'may inadvertently overlook emerging or domain-specific research' and that no comparative performance evaluation is included. These are transparency and completeness limitations, which concern verifiability and correctness rather than circularity. Accordingly, the paper does not exhibit any of the enumerated circularity patterns: no self-definitional reduction, no fitted input presented as prediction, no load-bearing self-citation chain, no imported uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result as a new derivation. The appropriate score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The survey is descriptive and does not fit parameters or propose entities. The only assumptions are standard domain beliefs about RAG benefits and multimodal alignment, which the survey takes from the literature.

assumptions (2)
  • domain assumption Retrieval-augmented generation improves factual accuracy and reduces hallucination compared to non-retrieval models.
    Invoked in Section 1 (Introduction) citing Lewis et al. 2020 and Shuster et al. 2021; the survey frames the field around this premise.
  • domain assumption Cross-modal alignment and fusion are necessary for effective multimodal RAG.
    Implicit throughout Section 3.2 and the formulation in Section 1, which assumes a shared embedding space with modality-specific encoders.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/5TDKIG7B

@misc{pith2026250208826,
  author       = {Pith},
  title        = {Pith review of: Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5TDKIG7B}},
  note         = {Machine review of arXiv:2502.08826}
}
read the original abstract

Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges beyond those in unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We review training strategies, robustness enhancements, loss functions, and agent-based approaches, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. All resources are publicly available at https://github.com/llm-lab-org/Multimodal-RAG-Survey.

Figures

Figures reproduced from arXiv: 2502.08826 by the authors.

Figure 1
Figure 1. Overview of the multimodal RAG pipeline, illustrating key techniques and recent advancements. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Taxonomy of recent advances in Multimodal RAG. Refer to Appendix (§ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Taxonomy of application domains for Multimodal Retrieval-Augmented Generation systems. [PITH_FULL_IMAGE:figures/full_fig_p030_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Post-hoc selective modality escalation—draft from text+tables, verify the missing modality, then escalate only when predicted accuracy gain exceeds visual cost—matches always-on VLM accuracy on MultiModalQA with far f...

  2. BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions

    cs.CL 2025-06 conditional novelty 6.0 of 10

    BioMol-MQA is a new multimodal QA dataset for polypharmacy in which LLMs perform poorly zero-shot but much better when given gold context.

  3. mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A systematic empirical study finds that for multimodal RAG, EVA-CLIP retrieval, listwise LVLM reranking, and feeding only the top-ranked document works best, with a self-reflection agent adding further gains.

  4. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  5. Agent-based Condition Monitoring Assistance with Multimodal Industrial Database Retrieval Augmented Generation

    cs.LG 2025-06 conditional novelty 5.0 of 10

    MindRAG retrieves similar historical vibration recordings and maintenance annotations, then uses LLM agents to generate fault predictions and alarm recommendations for industrial condition monitoring.

  6. SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    SceneRAG uses LLM-driven scene segmentation and a scene-level knowledge graph to retrieve and answer questions about long videos, reporting higher LLM-judged win-rates than chunk-based RAG baselines on the LongerVideo...

  7. Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Multimodal RAG models show a U-shaped position bias in evidence ordering, and the bias grows logarithmically with the number of retrieved items, according to a new metric.

Reference graph

Works this paper leans on

34 extracted references · 16 canonical work pages · cited by 7 Pith papers

  1. [3]

    Preprint, arXiv:2407.01449

    Colpali: Efficient document retrieval with vision language models. Preprint, arXiv:2407.01449. Chun-Mei Feng, Yang Bai, Tao Luo, Zhen Li, Salman Khan, Wangmeng Zuo, Xinxing Xu, Rick Siow Mong Goh, and Yong Liu. 2023. Vqa4cir: Boosting com- posed image retrieval with visual question answering. Preprint, arXiv:2312.12273. Yunfan Gao, Yun Xiong, Xinyu Gao, K...

    arXiv 2023

  2. [4]

    ACM Trans

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst. Just Accepted. Liting Huang, Zhihao Zhang, Yiran Zhang, Xiyue Zhou, and Shoujin Wang. 2025. Ru-ai: A large multimodal dataset for machine-generated content detection. In Companion Proceedings of the ACM on Web Confer- ence 2025, ...

    arXiv 2025

  3. [5]

    In Proceedings of the Seventh Fact Extraction and VERification Work- shop (FEVER), pages 280–296, Miami, Florida, USA

    RAGAR, your falsehood radar: RAG- augmented reasoning for political fact-checking using multimodal large language models. In Proceedings of the Seventh Fact Extraction and VERification Work- shop (FEVER), pages 280–296, Miami, Florida, USA. Association for Computational Linguistics. Faizan Farooq Khan, Jun Chen, Youssef Mohamed, Chun- Mei Feng, and Mohame...

    work page arXiv 2025

  4. [8]

    In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5

    Speech retrieval-augmented generation without automatic speech recognition. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. Matin Mortaheb, Mohammad A. Amir Khojastepour, Sri- mat T. Chakradhar, and Sennur Ulukus. 2025a. Rag- check: Evaluating multimodal retrieval augmented generation per...

    arXiv 2025

  5. [9]

    arXiv preprint arXiv:2410.18104

    Enwar: A rag-empowered multi-modal llm framework for wireless environment perception. arXiv preprint arXiv:2410.18104. Thong Nguyen, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. 2024. Multimodal learned sparse retrieval with probabilistic expansion control. In Advances in Information Retrieval, pages 448–464, Cham. Springer Nature Switzerland. Y...

    arXiv 2024

  6. [10]

    Weihua Ou, Yingjie Chen, Linqing Liang, Jianping Gou, Jiahao Xiong, Jiacheng Zhang, Lingge Lai, and Lei Zhang

    Dinov2: Learning robust visual features without supervision. Weihua Ou, Yingjie Chen, Linqing Liang, Jianping Gou, Jiahao Xiong, Jiacheng Zhang, Lingge Lai, and Lei Zhang. 2025. Cross-modal retrieval of chest x-ray images and diagnostic reports based on report entity graph and dual attention: Cross-modal retrieval of chest x-ray images and diagnostic repo...

    arXiv 2025

  7. [12]

    In EMNLP-Findings

    Retrieval augmented code generation and summarization. In EMNLP-Findings. John Pavlopoulos, Vasiliki Kougia, and Ion Androut- sopoulos. 2019. A survey on biomedical image captioning. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 26–36, Minneapolis, Minnesota. Association for Computa- tional Linguistics. Abhirama Subra...

    work page 2019

  8. [13]

    V olta: Vision-language transformer with weakly-supervised local-feature alignment. TMLR. Tyler Thomas Procko and Omar Ochoa. 2024. Graph retrieval-augmented generation for large language models: A survey. In 2024 Conference on AI, Science, Engineering, and Technology (AIxSET), pages 166– 169. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho,...

    arXiv 2024

Show all 34 references
  1. [14]

    In International conference on machine learning, pages 8748–8763

    Learning transferable visual models from natu- ral language supervision. In International conference on machine learning, pages 8748–8763. PMLR. Hamed Rahimi, Jeanne Cattoni, Meriem Beghili, Mouad Abrini, Mahdi Khoramshahi, Maribel Pino, and Mo- hamed Chetouani. 2025. Reasonin...

    2025 arXiv

  2. [15]

    Preprint, arXiv:2406.19150

    Raven: Multitask retrieval augmented vision- language learning. Preprint, arXiv:2406.19150. David Rau, Shuai Wang, Hervé Déjean, and Stéphane Clinchant. 2024. Context embeddings for efficient answer generation in rag. Preprint, arXiv:2407.09252. Xubin Ren, Lingrui Xu, Long Xia...

    2024 arXiv

  3. [16]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19305–19314

    Pic2word: Mapping pictures to words for zero-shot composed image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19305–19314. Fulvio Sanguigni, Davide Morelli, Marcella Cornia, and Rita Cucchiara. 2025. Fashion-rag: Multim...

    2025 arXiv

  4. [17]

    In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5520–5530, Abu Dhabi, United Arab Emirates

    RACE: Retrieval-augmented commit message generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5520–5530, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Faisal Tareque Shohan, Mir Tafseer Nayee...

    2022 arXiv

  5. [18]

    Preprint, arXiv:2405.20834

    Retrieval meets reasoning: Even high-school textbook knowledge benefits multimodal reasoning. Preprint, arXiv:2405.20834. Yansong Tang, Xiaohan Wang, Jingdong Wang, et al

    arXiv

  6. [20]

    In Advances in Neural Information Processing Sys- tems

    VideoMAE: Masked autoencoders are data- efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Sys- tems. SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of h...

    2024 arXiv

  7. [21]

    The fashion iq dataset: Retrieving images by combining side information and relative natural language feedback. CVPR. Ian Wu, Sravan Jayanthi, Vijay Viswanathan, Simon Rosenberg, Sina Khoshfetrat Pakazad, Tongshuang Wu, and Graham Neubig. 2024a. Synthetic mul- timodal question...

    2024 arXiv

  8. [22]

    Yibin Yan and Weidi Xie

    Corrective retrieval augmented generation. Yibin Yan and Weidi Xie. 2024. Echosight: Advancing visual-language models with wiki knowledge. In Findings of the Association for Computational Linguis- tics: EMNLP 2024, pages 1538–1551, Miami, Florida, USA. Association for Computat...

    2024 arXiv

  9. [23]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10502–10511

    Cross-modal self-attention network for re- ferring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10502–10511. Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jin- heon Baek, and Sung Ju Hwang. 2025. Universalrag: Re...

    2025 arXiv

  10. [24]

    Preprint, arXiv:2402.10828

    Rag-driver: Generalisable driving explana- tions with retrieval-augmented in-context learning in multi-modal large language model. Preprint, arXiv:2402.10828. Zheng Yuan, Qiao Jin, Chuanqi Tan, Zhengyun Zhao, Hongyi Yuan, Fei Huang, and Songfang Huang. 2023. Ramm: Retrieval-au...

    2023

  11. [25]

    In 2019 International conference on document analysis and recognition (ICDAR), pages 1015–1022

    Publaynet: largest dataset ever for document layout analysis. In 2019 International conference on document analysis and recognition (ICDAR), pages 1015–1022. IEEE. Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong...

    2019 arXiv

  12. [26]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22010–22019

    Ufinebench: Towards text-based person re- trieval with ultra-fine granularity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22010–22019. A Taxonomy In this section, we provide more details regarding the taxonomy of multimodal RAG ...

    2014

  13. [28]

    are essential. As discussed in the benchmarks overview above, unified evaluation frameworks like M 2RAG (Ma et al., 2024d) provide a comprehen- sive assessment across multiple tasks, including image captioning, visual question answering, and fact verification. Evaluating compl...

    2021

  14. [29]

    Any-to-Any

    and Conceptual Captions (Sharma et al., 2018) 27 provide substantial scale essential for pre-training, enabling models to learn generalizable represen- tations across diverse domains. However, their reliance on web-crawled data introduces inherent noise that can compromise tra...

    2024

  15. [30]

    measures caption quality using TF-IDF and cosine similarity (Yasunaga et al., 2023; Zhao et al., 2024; Luo et al., 2024a; Yuan et al., 2024; Shar- ifymoghaddam et al., 2024; Hu et al., 2023; Rao et al., 2024; Xu et al., 2024a; Kim et al., 2024; Zhang et al., 2024c), while SPIC...

    2023

  16. [34]

    enables style-aware retrieval by jointly em- bedding garment images and textual descriptors, while Dang (2024) reduces search friction through multimodal query expansion. For fashion image editing, Fashion-RAG (Sanguigni et al., 2025) em- ploys a retrieval-augmented approach, ...

    2024

  17. [2015]

    In the med- ical domain, MIMIC-CXR (Johnson et al., 2019) and CheXpert (Irvin et al., 2019) facilitate tasks such as medical report generation

    and YouCook2 (Zhou et al., 2018). In the med- ical domain, MIMIC-CXR (Johnson et al., 2019) and CheXpert (Irvin et al., 2019) facilitate tasks such as medical report generation. It should be noted that a number of these datasets are unimodal (e.g., solely text-based or image-b...

    2018

  18. [2016]

    SPIDEr (Liu et al., 2017), used in (Zhang et al., 2024c), combines both metrics

    focuses on semantics. SPIDEr (Liu et al., 2017), used in (Zhang et al., 2024c), combines both metrics. For semantic alignment, BERTScore (Zhang et al.,

    2017

  19. [2018]

    The formula for FID is: FID = ∥µr − µg∥2 + tr(Σr + Σg − 2 p ΣrΣg) (8) where µr and Σr are the mean and covariance of real images’ feature representations, respectively

    provides an unbiased alternative. The formula for FID is: FID = ∥µr − µg∥2 + tr(Σr + Σg − 2 p ΣrΣg) (8) where µr and Σr are the mean and covariance of real images’ feature representations, respectively. µg and Σg are the mean and covariance of generated images’ feature represe...

    2024

  20. [2019]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–10

    Coin: A large-scale dataset for comprehensive instructional video analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–10. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Sch...

    arXiv

  21. [2020]

    CLIP Score (Hessel et al., 2021), used in (Shari- fymoghaddam et al., 2024; Zhang et al., 2024c), measures image-text similarity using CLIP (Radford et al., 2021)

    compares BERT embeddings (Sun et al., 2024b; Shohan et al., 2024), and evaluates fluency (Chen et al., 2022a; Zhi Lim et al., 2024; Ma et al., 2024d). CLIP Score (Hessel et al., 2021), used in (Shari- fymoghaddam et al., 2024; Zhang et al., 2024c), measures image-text similari...

    2024

  22. [2021]

    In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209

    Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209. Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. 2024. Wav...

    2024

  23. [2022]

    In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 3360–3362, New York, NY , USA

    Clueweb22: 10 billion web documents with rich information. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 3360–3362, New York, NY , USA. Association for Computing Machinery. Vassil Panayotov, ...

    2015

  24. [2023]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–10

    Zero-shot composed image retrieval with textual inversion. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–10. Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2019. Multimodal machine learning: A survey and taxonomy. IEE...

    2019 arXiv

  25. [2024]

    Preprint, arXiv:2412.13071

    Clasp: Contrastive language-speech pretrain- ing for multilingual multimodal information retrieval. Preprint, arXiv:2412.13071. Omar Adjali, Olivier Ferret, Sahar Ghannay, and Hervé Le Borgne. 2024. Multi-level information retrieval augmented generation for knowledge-based vis...

    2024 arXiv

  26. [2025]

    arXiv preprint arXiv:2501.01120

    Retrieval-augmented dynamic prompt tuning for incomplete multimodal learning. arXiv preprint arXiv:2501.01120. Myeonghwa Lee, Seonho An, and Min-Soo Kim. 2024. PlanRAG: A plan-then-retrieval augmented genera- tion for generative large language models as decision makers. In Pro...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.