Pith. sign in

REVIEW 17 cited by

Generative Multimodal Models are In-Context Learners

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.13286 v2 pith:MYYHQB2S submitted 2023-12-20 cs.CV

Quan Sun , Yufeng Cui , Xiaosong Zhang , Fan Zhang , Qiying Yu , Zhengxiong Luo , Yueze Wang , Yongming Rao
show 3 more authors
Jingjing Liu Tiejun Huang Xinlong Wang
This is my paper · ORCID
classification cs.CV
keywords multimodaltasksemu2modelsin-contextmodeldemonstrategeneration
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A 3,068-prompt benchmark with per-instance Q&A scoring shows that current text-to-image models, including reasoning-enhanced ones, handle reasoning-driven prompts poorly, with mathematical reasoning near zero.

  2. When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICL

    cs.CV 2026-08 conditional novelty 6.0 of 10

    The Selection-Realization Hypothesis holds that static task vectors suffice when demonstration-induced activation changes are largely shared across queries; query-conditioned, multi-site, or routing interventions are ...

  3. ReMoT: Reinforcement Learning with Motion Contrast Triplets

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Training a 4B vision-language model on rule-generated motion-contrast triplets with GRPO lifts spatio-temporal QA accuracy by about 17 points on the authors' own benchmark and by smaller margins on standard benchmarks.

  4. AgroBench: Vision-Language Model Benchmark in Agriculture

    cs.CV 2025-07 conditional novelty 6.0 of 10

    AgroBench is a new expert-annotated benchmark showing that current vision-language models, especially open-source ones, struggle with fine-grained agricultural identification such as weed species.

  5. T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Reinforcement learning that jointly optimizes a text planning step and patch-by-patch image generation in one autoregressive model improves compositional text-to-image benchmarks by double-digit absolute percentage po...

  6. UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths

    cs.CV 2025-02 conditional novelty 6.0 of 10

    UniMoD prunes tokens with task-specific routers in unified multimodal transformers, cutting training FLOPs by 15-40% while roughly maintaining benchmark performance.

  7. CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    CHIRP, a 104-question pairwise benchmark, exposes scaling trends in vision-language models that existing benchmarks miss, and its evaluations correlate better with training loss.

  8. Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Feeding large multimodal models a prompt in multiple languages, not just English, improves text-to-image alignment and human-preference scores across three benchmarks.

  9. FashionComposer: Compositional Fashion Image Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A single diffusion framework composes multiple garment and face references into one fashion image using an asset library and subject-binding attention.

  10. Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Diffusion-conditioned denoising trains a video tokenizer whose features support both video question answering and text-to-video generation in a single LLM.

  11. Teaching VLMs to Localize Specific Objects from In-context Examples

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Fine-tuning VLMs on video-tracking conversations with made-up object names teaches them to localize a specific object in a new image from only a few in-context examples.

  12. Qwen-Audio-VAE Technical Report

    eess.AS 2026-07 conditional novelty 5.0 of 10

    A 12.5 Hz continuous audio VAE reconstructs speech, music, and sound well while encoding 64×30s clips in 541 ms after latency-aware encoder pruning.

  13. Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Images are encoded into discrete tokens projected from LLM embeddings, so a single autoregressive model does visual understanding and generation with matched or improved benchmark scores.

  14. ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding

    cs.AI 2025-05 conditional novelty 5.0 of 10

    ArtRAG improves painting-explanation generation by retrieving relevant subgraphs from a knowledge graph built from Wikipedia and using them to guide a multimodal LLM, outperforming trained baselines on SemArt and Artp...

  15. EVEv2: Improved Baselines for Encoder-Free Vision-Language Models

    cs.CV 2025-02 conditional novelty 5.0 of 10

    An encoder-free vision-language model using separate attention, normalization, and feed-forward weights for image versus text tokens outperforms earlier encoder-free models and narrows the gap to encoder-based VLMs wi...

  16. SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

    cs.CV 2024-12 conditional novelty 5.0 of 10

    SynerGen-VL introduces token folding and vision expert FFNs to train a 2.4B encoder-free MLLM that matches larger unified models like Emu3 on multiple image understanding and generation benchmarks.

  17. On the Comprehensibility of Multi-structured Financial Documents using LLMs and Pre-processing Tools

    cs.IR 2025-06 conditional novelty 4.0 of 10

    Preprocessing financial PDFs into text, tables, and chart data with existing tools improves LLM question-answering accuracy over direct GPT-4o image input in a small private evaluation.

Pith tools