Pith. sign in

REVIEW 16 cited by

Retrieval-Augmented Multimodal Language Modeling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2211.12561 v2 pith:7ZMX4MQX submitted 2022-11-22 cs.CV cs.CLcs.LG

Retrieval-Augmented Multimodal Language Modeling

Michihiro Yasunaga , Armen Aghajanyan , Weijia Shi , Rich James , Jure Leskovec , Percy Liang , Mike Lewis , Luke Zettlemoyer
show 1 more author
Wen-tau Yih
This is my paper
classification cs.CV cs.CLcs.LG
keywords multimodalmodelgenerationmodelsdall-eimageknowledgera-cm3
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in text-to-image and image-to-text generation. However, these models store all learned knowledge (e.g., the appearance of the Eiffel Tower) in the model parameters, requiring increasingly larger models and training data to capture more knowledge. To integrate knowledge in a more scalable and modular way, we propose a retrieval-augmented multimodal model, which enables a base multimodal model (generator) to refer to relevant text and images fetched by a retriever from external memory (e.g., documents on the web). Specifically, for the retriever, we use a pretrained CLIP, and for the generator, we train a CM3 Transformer on the LAION dataset. Our resulting model, named Retrieval-Augmented CM3 (RA-CM3), is the first multimodal model that can retrieve and generate both text and images. We show that RA-CM3 significantly outperforms baseline multimodal models such as DALL-E and CM3 on both image and caption generation tasks (12 FID and 17 CIDEr improvements on MS-COCO), while requiring much less compute for training (<30% of DALL-E). Moreover, we show that RA-CM3 exhibits novel capabilities, such as faithful image generation and multimodal in-context learning (e.g., image generation from demonstrations).

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systems

    cs.MA 2025-06 accept novelty 7.0

    A survey that defines Compound AI Systems, proposes a multi-dimensional taxonomy based on component roles and orchestration strategies, reviews four foundational paradigms, and identifies key challenges for future research.

  2. Language Is Not All You Need: Aligning Perception with Language Models

    cs.CL 2023-02 conditional novelty 7.0

    Kosmos-1 shows strong zero-shot and few-shot results on language tasks, image captioning, visual QA, OCR-free document understanding, and image recognition guided by text instructions.

  3. Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.5

    Pre-generation visual tokens and prompt hidden states can diagnose whether a VLM will fail from recognition, visual evidence, or factual knowledge, enabling targeted interventions.

  4. Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0

    Pre-generation visual tokens and prompt hidden states can predict whether a VLM will fail from recognition bottlenecks or from post-recognition factual gaps, enabling targeted interventions.

  5. Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity

    cs.CV 2026-06 unverdicted novelty 6.0

    Presents Invoice Haystack benchmark for homogeneous document retrieval and VL-RAG hybrid framework achieving 60% Recall@1 and up to 13.5 point gains over prior methods.

  6. Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity

    cs.CV 2026-06 unverdicted novelty 6.0

    New Invoice Haystack benchmark exposes embedding collapse under high visual homogeneity (0.73 cosine similarity) and VL-RAG hybrid retrieval reaches 60% Recall@1, beating prior methods by 13.5 points on the new set an...

  7. Differentiable Optimization Layers for Guaranteed Fairness in Deep Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    Introduces a fairness layer for deep learning models that guarantees output parity and an online primal-dual algorithm for aggregate fairness guarantees in streaming predictions with small batch sizes.

  8. Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs

    cs.CV 2026-05 unverdicted novelty 6.0

    PVM adds a parallel branch to LVLMs that directly supplies visual embeddings to prevent attention decay over long generated sequences, yielding accuracy gains on reasoning tasks with minimal overhead.

  9. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    cs.CL 2023-10 unverdicted novelty 6.0

    Self-RAG trains LLMs to adaptively retrieve passages on demand and self-critique using reflection tokens, outperforming ChatGPT and retrieval-augmented Llama2 on QA, reasoning, and fact verification.

  10. REPLUG: Retrieval-Augmented Black-Box Language Models

    cs.CL 2023-01 conditional novelty 6.0

    REPLUG improves frozen black-box LMs by prepending LM-supervised retrieved documents, delivering 6.3% better language modeling on GPT-3 and 5.1% better five-shot MMLU on Codex.

  11. PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation

    cs.CV 2026-06 unverdicted novelty 5.0

    PhysRAG curates 7K videos from WISA-80K, builds a physical video database, and injects knowledge via learnable queries into a diffusion model to reach SOTA visual quality and physical compliance on PhyGenBench and VBench.

  12. Qiskit Code Migration with LLMs

    cs.SE 2026-06 unverdicted novelty 5.0

    A taxonomy-guided RAG system with LLMs reduces hallucinations and improves migration suggestions for Qiskit code compared to unconstrained retrieval.

  13. Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs

    cs.CV 2026-05 unverdicted novelty 5.0

    PVM adds a parallel learnable branch to LVLMs that supplies visual embeddings on demand to structurally prevent attention decay and visual signal dilution during deep autoregressive generation.

  14. Attention Grounded Enhancement for Visual Document Retrieval

    cs.IR 2025-11 unverdicted novelty 5.0

    AGREE boosts visual document retrieval by adding local relevance signals from MLLM attention maps to global document labels during retriever training.

  15. mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models

    cs.CV 2024-08 unverdicted novelty 5.0

    mPLUG-Owl3 introduces hyper attention blocks to integrate vision and language for long image-sequence understanding and reports SOTA results on single-image, multi-image, and video benchmarks.

  16. Retrieval-Augmented Generation for Large Language Models: A Survey

    cs.CL 2023-12 unverdicted novelty 3.0

    A survey of RAG paradigms, components, benchmarks, and challenges for improving LLMs on knowledge-intensive tasks.