Pith. sign in

REVIEW 7 cited by

mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2505.24073 v2 pith:PZPCGIUD submitted 2025-05-29 cs.AI cs.CLcs.CV

mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

Chan-Wei Hu , Yueqi Wang , Shuo Xing , Chia-Ju Chen , Suofei Feng , Ryan Rossi , Zhengzhong Tu This is my paper
classification cs.AI cs.CLcs.CV
keywords generationlvlmsevidenceretrievalgroundingmitigatemultimodalperformance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Large Vision-Language Models (LVLMs) have made remarkable strides in multimodal tasks such as visual question answering, visual grounding, and complex reasoning. However, they remain limited by static training data, susceptibility to hallucinations, and inability to verify claims against up-to-date, external evidence, compromising their performance in dynamic real-world applications. Retrieval-Augmented Generation (RAG) offers a practical solution to mitigate these challenges by allowing the LVLMs to access large-scale knowledge databases via retrieval mechanisms, thereby grounding model outputs in factual, contextually relevant information. Here in this paper, we conduct the first systematic dissection of the multimodal RAG pipeline for LVLMs, explicitly investigating (1) the retrieval phase: on the modality configurations and retrieval strategies, (2) the re-ranking stage: on strategies to mitigate positional biases and improve the relevance of retrieved evidence, and (3) the generation phase: we further investigate how to best integrate retrieved candidates into the final generation process. Finally, we extend to explore a unified agentic framework that integrates re-ranking and generation through self-reflection, enabling LVLMs to select relevant evidence and suppress irrelevant context dynamically. Our full-stack exploration of RAG for LVLMs yields substantial insights, resulting in an average performance boost of 5% without any fine-tuning.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

    cs.CL 2026-06 unverdicted novelty 7.0

    Multimodal KB-VQA exhibits a primacy bias where gold passages at prompt start outperform those at the end by 16-26 points, flipping the text-only lost-in-the-middle pattern.

  2. Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

    cs.CL 2026-06 unverdicted novelty 6.5

    In multimodal KB-VQA, gold evidence at the first prompt slot beats gold at the last by 16–26 points, flipping the classic U-shaped lost-in-the-middle pattern into primacy bias.

  3. Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG

    cs.IR 2026-07 conditional novelty 6.0

    Post-hoc selective modality escalation—draft from text+tables, verify the missing modality, then escalate only when predicted accuracy gain exceeds visual cost—matches always-on VLM accuracy on MultiModalQA with far f...

  4. DramaDirector: Geometry-Guided Short Drama Generation

    cs.CV 2026-06 conditional novelty 6.0

    Geometry-indexed depth–pose retrieval plus schema SFT and GRPO planning improves faithfulness, consistency, and controllability of plot-to-short-drama video generation over multi-agent and text-only baselines.

  5. MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

    cs.IR 2026-04 conditional novelty 6.0

    A lightweight hierarchical multimodal graph RAG that fuses entity-grounded visual objects with text nodes and propagates relevance via multi-granularity PPR, delivering SOTA multimodal task performance at far lower co...

  6. MG$^2$-RAG: Multi-Granularity Graph for Multimodal Retrieval-Augmented Generation

    cs.IR 2026-04 unverdicted novelty 6.0

    MG²-RAG proposes a multi-granularity graph RAG framework that constructs hierarchical multimodal nodes via entity-driven visual grounding and performs structured retrieval, delivering SOTA results on four multimodal t...

  7. DramaDirector: Geometry-Guided Short Drama Generation

    cs.CV 2026-06 unverdicted novelty 5.0

    DramaDirector retrieves depth-pose references from real drama shots to guide first-frame and image-to-video synthesis for plot-driven short dramas, paired with the DramaBoard benchmark.