Pith. sign in

REVIEW 10 cited by

Grounding Language Models to Images for Multimodal Inputs and Outputs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2301.13823 v4 pith:LGOLKZKO submitted 2023-01-31 cs.CL cs.AIcs.CVcs.LG

Jing Yu Koh , Ruslan Salakhutdinov , Daniel Fried This is my paper
classification cs.CLcs.AIcs.CVcs.LG
keywords languageinterleavedmodelsimagesmodeltextabilitiesarbitrarily
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images. Our method leverages the abilities of language models learnt from large scale text-only pretraining, such as in-context learning and free-form text generation. We keep the language model frozen, and finetune input and output linear layers to enable cross-modality interactions. This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images. We achieve strong zero-shot performance on grounded tasks such as contextual image retrieval and multimodal dialogue, and showcase compelling interactive abilities. Our approach works with any off-the-shelf language model and paves the way towards an effective, general solution for leveraging pretrained language models in visually grounded settings.

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

    cs.CV 2023-10 unverdicted novelty 7.0 of 10

    A new shared video-image tokenizer enables large language models to surpass diffusion models on standard visual generation benchmarks.

  2. Visual Instruction Tuning

    cs.CV 2023-04 unverdicted novelty 7.0 of 10

    LLaVA is trained on GPT-4 generated visual instruction data to achieve 85.1% relative performance to GPT-4 on synthetic multimodal tasks and 92.53% accuracy on Science QA.

  3. Language Is Not All You Need: Aligning Perception with Language Models

    cs.CL 2023-02 conditional novelty 7.0 of 10

    Kosmos-1 shows strong zero-shot and few-shot results on language tasks, image captioning, visual QA, OCR-free document understanding, and image recognition guided by text instructions.

  4. Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

    cs.AI 2026-06 conditional novelty 6.0 of 10

    A cue-coverage reward plus a modality-token KL penalty makes multimodal emotion-reasoning models cite more real visual/audio evidence and hallucinate less, with reported SoTA on emotion benchmarks.

  5. Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    OPPO applies RL with an Omni-Perception Reward and masked-input KL loss to boost cue utilization and suppress hallucinations in emotion reasoning MLLMs, claiming SOTA results on MER-UniBench, MME-Emotion, and MEP-Bench.

  6. MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

    cs.CV 2024-01 conditional novelty 6.0 of 10

    MoE-LLaVA applies mixture-of-experts sparsity to LVLMs via MoE-Tuning, delivering LLaVA-1.5-7B level visual understanding and better hallucination resistance with only ~3B active parameters.

  7. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

    cs.CL 2023-06 conditional novelty 6.0 of 10

    AWQ quantizes LLM weights to low bits by scaling salient channels based on activation statistics, outperforming prior methods on language, coding, math, and multi-modal benchmarks.

  8. Visual Language Models as Zero-Shot Deepfake Detectors

    cs.CV 2025-07 conditional novelty 4.0 of 10

    Zero-shot VLMs scored by normalized yes/no token probabilities beat most trained deepfake detectors on a new SimSwap dataset, and a lightly fine-tuned InstructBLIP is near-perfect on DFDC-P.

  9. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model

    cs.CV 2024-02 unverdicted novelty 4.0 of 10

    MobileVLM V2 shows that 1.7B and 3B parameter vision-language models can reach or exceed the performance of 3B and 7B+ models on common VLM benchmarks via targeted design and data improvements.

  10. OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

    cs.CV 2023-08 unverdicted novelty 4.0 of 10

    OpenFlamingo provides open-source autoregressive vision-language models that achieve 80-89% of Flamingo performance on seven vision-language datasets.

Pith tools