REVIEW 8 cited by
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
While language Models store a massive amount of world knowledge implicitly in their parameters, even very large models often fail to encode information about rare entities and events, while incurring huge computational costs. Recently, retrieval-augmented models, such as REALM, RAG, and RETRO, have incorporated world knowledge into language generation by leveraging an external non-parametric index and have demonstrated impressive performance with constrained model sizes. However, these methods are restricted to retrieving only textual knowledge, neglecting the ubiquitous amount of knowledge in other modalities like images -- much of which contains information not covered by any text. To address this limitation, we propose the first Multimodal Retrieval-Augmented Transformer (MuRAG), which accesses an external non-parametric multimodal memory to augment language generation. MuRAG is pre-trained with a mixture of large-scale image-text and text-only corpora using a joint contrastive and generative loss. We perform experiments on two different datasets that require retrieving and reasoning over both images and text to answer a given query: WebQA, and MultimodalQA. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20\% absolute on both datasets and under both distractor and full-wiki settings.
Forward citations
Cited by 8 Pith papers
-
Region-Level Context-Aware Multimodal Understanding
Region-level context-aware instruction tuning with a large synthetic dataset improves MLLMs' ability to connect objects in images to their textual descriptions.
-
Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs
Multi-modal RAG (text plus UI screenshots) with reward-based polishing generates acceptance criteria from user stories that three industry experts rated near 4/5 on relevance, correctness, and understandability.
-
M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generation
M2IO-R1 uses GRPO reinforcement learning to train a 3B model that selects and places retrieved images into generated text answers, improving MRAMG quality on several benchmarks.
-
Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval
UNITE combines curated multimodal training data and a modality-masked contrastive loss to achieve strong retrieval performance across text, image, and video tasks.
-
SEFRQO: A Self-Evolving Fine-Tuned RAG-Based Query Optimizer
SEFRQO claims a self-evolving fine-tuned LLM with retrieval and execution feedback reduces query latency versus PostgreSQL, but the provided body is a different paper, blocking verification.
-
Benchmarking Poisoning Attacks against Retrieval-Augmented Generation
A unified benchmark evaluation finds that existing RAG poisoning attacks remain effective on standard QA datasets, drop on expanded knowledge bases, and are only partially mitigated by current defenses.
-
Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
Four-axis fusion (dense, BM25, triples, images) over 43 Japanese disaster PDFs claims 0.99 recall and a 71.6% multi-hop gain, but the benchmark labels and fusion weights come from inside the same pipeline.
-
MIRA: A Novel Framework for Fusing Modalities in Medical RAG
A medical multimodal RAG pipeline with rethink-and-rearrange and online search; the claimed SOTA is contradicted by the paper's own PMC-VQA numbers.
Discussion (0). Sign in to comment.