Pith. sign in

REVIEW 22 cited by

Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.02591 v1 pith:57AIJRR2 submitted 2023-09-05 cs.LG cs.CLcs.CV

Lili Yu , Bowen Shi , Ramakanth Pasunuru , Benjamin Muller , Olga Golovneva , Tianlu Wang , Arun Babu , Binh Tang
show 19 more authors
Brian Karrer Shelly Sheynin Candace Ross Adam Polyak Russell Howes Vasu Sharma Puxin Xu Hovhannes Tamoyan Oron Ashual Uriel Singer Shang-Wen Li Susan Zhang Richard James Gargi Ghosh Yaniv Taigman Maryam Fazel-Zarandi Asli Celikyilmaz Luke Zettlemoyer Armen Aghajanyan
This is my paper
classification cs.LGcs.CLcs.CV
keywords multi-modalcm3leongenerationmodelmodelsdemonstratelanguagemethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented, token-based, decoder-only multi-modal language model capable of generating and infilling both text and images. CM3Leon uses the CM3 multi-modal architecture but additionally shows the extreme benefits of scaling up and tuning on more diverse instruction-style data. It is the first multi-modal model trained with a recipe adapted from text-only language models, including a large-scale retrieval-augmented pre-training stage and a second multi-task supervised fine-tuning (SFT) stage. It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs. Extensive experiments demonstrate that this recipe is highly effective for multi-modal models. CM3Leon achieves state-of-the-art performance in text-to-image generation with 5x less training compute than comparable methods (zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate unprecedented levels of controllability in tasks ranging from language-guided image editing to image-controlled generation and segmentation.

Discussion (0). Sign in to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Obliviate: Erasing Concepts from Autoregressive Image Generation Models

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Obliviate erases targeted concepts from autoregressive image generators via KL supervision on visual tokens over full trajectories, cutting nudity rates sharply on benchmarks while keeping general performance.

  2. Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models

    cs.CR 2026-05 conditional novelty 7.0 of 10

    ToBAC is the first backdoor attack on unified autoregressive models, using data or model poisoning to make triggers elicit cross-modal malicious behavior in text and image generation.

  3. Beyond Patches: Global-aware Autoregressive Model for Multimodal Few-Shot Font Generation

    cs.CV 2026-01 unverdicted novelty 7.0 of 10

    GAR-Font is a global-aware autoregressive framework for multimodal few-shot font generation that adds global tokenization, a language-style adapter, and post-refinement to improve style coherence over patch-based methods.

  4. DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies

    cs.CV 2025-03 unverdicted novelty 7.0 of 10

    DualToken disentangles semantics and appearance via separate codebooks in one tokenizer, reporting 0.25 rFID, 82% ImageNet zero-shot accuracy, and gains over VILA-U on understanding and generation benchmarks.

  5. Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

    cs.CV 2023-10 unverdicted novelty 7.0 of 10

    A new shared video-image tokenizer enables large language models to surpass diffusion models on standard visual generation benchmarks.

  6. HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A human-efficient robot post-training pipeline with two-operator, twelve-robot supervision and an automatic rollout-segmentation critic raises success rates to 80–95% on four real manipulation tasks.

  7. SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    SPAR introduces semantic-pixel self-alignment via asymmetric tokenizer and adaptive routing for unified MLLMs that achieve SOTA generation and reconstruction while retaining understanding.

  8. FullFlow: Upgrading Text-to-Image Flow Matching Models for Bidirectional Vision--Language Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    FullFlow adds LoRA adapters and discrete text insertion to pretrained rectified-flow text-to-image models, achieving bidirectional generation with major gains in FID, CIDEr, VRAM, and throughput over Dual Diffusion baselines.

  9. Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thought

    cs.CV 2026-01 conditional novelty 6.0 of 10

    A visual chain-of-thought model that interleaves text plans and rendered 2D states improves compositional object assembly, hitting 88.4% component numeracy and 84.8% structural topology on a new benchmark.

  10. Mirai: Autoregressive Visual Generation Needs Foresight

    cs.CV 2026-01 conditional novelty 6.0 of 10

    Mirai injects future-token foresight into autoregressive visual generators, accelerating convergence up to 10x and cutting ImageNet FID from 5.34 to 4.34.

  11. Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation

    cs.CV 2025-05 unverdicted novelty 6.0 of 10

    Mogao presents a causal unified model with deep fusion, dual encoders, and interleaved position embeddings that achieves strong performance on multi-modal understanding, text-to-image generation, and coherent interlea...

  12. Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    MoT decouples non-embedding parameters by modality in transformers to match dense multi-modal performance with roughly one-third to one-half the FLOPs.

  13. Emu3: Next-Token Prediction is All You Need

    cs.CV 2024-09 unverdicted novelty 6.0 of 10

    Emu3 shows that next-token prediction on a unified discrete token space for text, images, and video lets a single transformer outperform task-specific models such as SDXL and LLaVA-1.6 in multimodal generation and perception.

  14. VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

    cs.CV 2024-09 unverdicted novelty 6.0 of 10

    VILA-U unifies visual understanding and generation inside one autoregressive next-token prediction model, removing separate diffusion components while claiming near state-of-the-art results.

  15. Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

    cs.AI 2024-08 unverdicted novelty 6.0 of 10

    A single transformer combines language modeling loss and diffusion loss on mixed-modality data, scaling to 7B parameters and 2T tokens while matching specialized language and diffusion models.

  16. Chameleon: Mixed-Modal Early-Fusion Foundation Models

    cs.CL 2024-05 unverdicted novelty 6.0 of 10

    Chameleon is an early-fusion token model that handles mixed image-text sequences for understanding and generation, achieving competitive or superior performance to larger models like Llama-2, Mixtral, and Gemini-Pro o...

  17. SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation

    cs.CV 2024-04 unverdicted novelty 6.0 of 10

    SEED-X is a unified multimodal foundation model that handles multi-granularity visual semantics for both comprehension and generation across arbitrary image sizes and ratios.

  18. HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation

    cs.RO 2026-07 conditional novelty 5.5 of 10

    VLAC-Cut-guided multi-robot HITL post-training reaches 80–95% success and 1.7–4.2× throughput over the base VLA, outperforming HITL-only under the same human budget.

  19. SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    SPAR introduces a semantic-pixel self-alignment tokenizer and dynamic token routing to create a unified multimodal model that performs both understanding and generation at claimed state-of-the-art levels.

  20. Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications

    cs.CV 2026-07 conditional novelty 4.0 of 10

    A survey of instruction-based image editing plus a new 21-task benchmark, CDD-IIE, on which ten open models are scored by human experts.

  21. InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning

    cs.CV 2026-06 unverdicted novelty 4.0 of 10

    InternVideo3 introduces Multimodal Contextual Reasoning and M^2LA attention to enable closed-loop evidence accumulation in long-video understanding and agentic tool use, reporting strong benchmark results.

  22. PaliGemma: A versatile 3B VLM for transfer

    cs.CV 2024-07 unverdicted novelty 4.0 of 10

    PaliGemma is an open 3B VLM based on SigLIP and Gemma that achieves strong performance on nearly 40 diverse open-world tasks including benchmarks, remote-sensing, and segmentation.

Pith tools