Pith. sign in

REVIEW 33 cited by

Making LLaMA SEE and Draw with SEED Tokenizer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01218 v1 pith:GK4TEVBU submitted 2023-10-02 cs.CV

Yuying Ge , Sijie Zhao , Ziyun Zeng , Yixiao Ge , Chen Li , Xintao Wang , Ying Shan This is my paper · ORCID
classification cs.CV
keywords llmsmultimodalabilitiesemergentgenerationimageseedshould
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The great success of Large Language Models (LLMs) has expanded the potential of multimodality, contributing to the gradual evolution of General Artificial Intelligence (AGI). A true AGI agent should not only possess the capability to perform predefined multi-tasks but also exhibit emergent abilities in an open-world context. However, despite the considerable advancements made by recent multimodal LLMs, they still fall short in effectively unifying comprehension and generation tasks, let alone open-world emergent abilities. We contend that the key to overcoming the present impasse lies in enabling text and images to be represented and processed interchangeably within a unified autoregressive Transformer. To this end, we introduce SEED, an elaborate image tokenizer that empowers LLMs with the ability to SEE and Draw at the same time. We identify two crucial design principles: (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a 1D causal dependency, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens should capture high-level semantics consistent with the degree of semantic abstraction in words, and be optimized for both discriminativeness and reconstruction during the tokenizer training phase. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by large-scale pretraining and instruction tuning on the interleaved textual and visual data, demonstrating impressive performance on a broad range of multimodal comprehension and generation tasks. More importantly, SEED-LLaMA has exhibited compositional emergent abilities such as multi-turn in-context multimodal generation, acting like your AI assistant.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 33 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  2. Sample-efficient Integration of New Modalities into Large Language Models

    cs.CL 2025-09 conditional novelty 7.0 of 10

    A hypernetwork trained on image, audio, and video adapts a shared projector to new, low-resource modalities from as few as 32 examples.

  3. FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL

    cs.CV 2025-06 conditional novelty 7.0 of 10

    FocusDiff improves autoregressive text-to-image generation by training on paired similar prompts with a modified GRPO objective, achieving state-of-the-art alignment on PairComp and gains on GenEval and T2I-CompBench.

  4. Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens

    cs.CV 2025-04 conditional novelty 7.0 of 10

    Diffusion timestep tokens give large language models a recursive visual language that improves unified multimodal comprehension and generation relative to spatial patch tokens.

  5. IDEA-Bench: How Far are Generative Models from Professional Designing?

    cs.CV 2024-12 conditional novelty 7.0 of 10

    IDEA-Bench measures generative models on 100 professional design tasks and finds the best tested system scores only 22.48 out of 100.

  6. Twins: Learn to Predict Unified Representations with Focal Loss

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Channel-wise concatenation of SigLIP2 and Flux VAE features into one token, trained with a focal-style flow-matching loss, yields a unified representation with 1.59 gFID on ImageNet 256 and VAE-level reconstruction.

  7. ChatUMM: Robust Context Tracking for Conversational Interleaved Generation

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A unified multimodal model trained on synthetic multi-turn dialogues that can interleave image generation with text across a conversation.

  8. BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A self-supervision method makes multimodal LLMs align their input image embeddings with the model's own refined internal representations, improving visual QA scores over LLaVA baselines.

  9. IGD: Instructional Graphic Design with Multimodal Layer Generation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    IGD generates editable multi-layer graphic designs (posters, slides, stickers) from text instructions using an MLLM for layout and a diffusion model for image assets.

  10. VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VideoREPA adds a token-relation distillation loss that aligns a text-to-video diffusion model's internal features with VideoMAEv2, boosting physical commonsense scores on VideoPhy and VideoPhy2.

  11. FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A 1.5B unified multimodal model trained with discrete flow matching and metric-induced probability paths matches autoregressive baselines of similar size on generation and understanding benchmarks.

  12. TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    TokLIP semanticizes VQ image tokens with a causal CLIP-style encoder, improving multimodal comprehension while preserving autoregressive image generation.

  13. X-Fusion: Introducing New Modality to Frozen Large Language Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    X-Fusion adds a trainable vision tower to a frozen LLM, enabling both image captioning and text-to-image generation while keeping the LLM's language ability intact.

  14. UniCoRN: Unified Commented Retrieval Network with LMMs

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A frozen multimodal LLM is extended with a retrieval adapter and an entity adapter to retrieve a relevant image and generate a supportive textual comment.

  15. Next Patch Prediction for Autoregressive Visual Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Averaging neighboring image tokens into patches during training lets autoregressive image models train faster and generate higher-quality images, with inference unchanged.

  16. ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers

    cs.CV 2024-12 conditional novelty 6.0 of 10

    An LLM-agent planner plus an unmodified FLUX.1-dev diffusion transformer achieves the top aggregate score on IDEA-Bench for zero-shot, freely described visual generation tasks, with no fine-tuning.

  17. EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios

    cs.AI 2024-12 conditional novelty 6.0 of 10

    EgoPlan-Bench2 evaluates multimodal LLMs on next-action planning across 24 egocentric real-world scenarios, finding most models near random chance, and a prompt-based method raises GPT-4V's accuracy to 43 percent on a...

  18. Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Diffusion-conditioned denoising trains a video tokenizer whose features support both video question answering and text-to-video generation in a single LLM.

  19. OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A large new benchmark and an offline judge model for open-ended interleaved image-text generation, with IntJudge matching human agreement better than GPT-4o.

  20. MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding

    cs.CV 2024-11 conditional novelty 6.0 of 10

    MUSE-VL adds a semantic-aware discrete image tokenizer to a pretrained LLM so that a single autoregressive model can understand and generate images with benchmark scores near dedicated VLM understanding models.

  21. ProLaViT: Learning Progressive Latent Visual Thoughts in Structured Latent Space

    cs.CV 2026-07 conditional novelty 5.5 of 10

    Progressive multi-step latent visual thoughts, endogenously distilled from a model's own encoder on synthetic trajectories and regularized by distance-weighted diversity, improve MLLM visual reasoning accuracy and efficiency.

  22. UniECG: Understanding and Generating ECG in One Unified Model

    cs.CL 2025-09 conditional novelty 5.0 of 10

    UniECG combines ECG interpretation and text-to-ECG generation in one model by fine-tuning a language model and aligning its output tokens with a pretrained ECG diffusion generator.

  23. TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuning

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A 3B model that injects intermediate MLLM query states into matching diffusion transformer layers reports competitive text-to-image and image-editing benchmark scores.

  24. Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Images are encoded into discrete tokens projected from LLM embeddings, so a single autoregressive model does visual understanding and generation with matched or improved benchmark scores.

  25. CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models

    cs.CV 2024-12 conditional novelty 5.0 of 10

    CoF improves multimodal LLM benchmark scores by having the model locate an answer region, then reweighting attention toward that region during inference.

  26. SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A soft-categorical image tokenizer compresses images to 32-64 tokens and reaches competitive ImageNet generation FID with large speedups.

  27. ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

    cs.CV 2024-12 conditional novelty 5.0 of 10

    ILLUME unifies visual understanding and generation in one LLM with a semantic vision tokenizer and a self-enhancing alignment scheme, reaching competitive benchmarks with only 15M pretraining pairs.

  28. X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models

    cs.CV 2024-12 conditional novelty 5.0 of 10

    X-Prompt compresses in-context image examples into a few learned tokens and adds text-description tasks, enabling a Chameleon-style autoregressive model to handle multiple image generation tasks in one framework.

  29. Tree-of-Thoughts Reasoning for Text-to-Image In-Context Learning

    cs.CV 2026-07 conditional novelty 4.0 of 10

    A four-stage Tree-of-Thoughts reasoning pipeline improves text-to-image in-context learning on the CoBSAT benchmark by generating and selecting among multiple candidate prompt hypotheses.

  30. Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.

  31. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

  32. MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models

    cs.SD 2024-12 reject novelty 3.0 of 10

    MuMu-LLaMA uses LLaMA with pretrained encoders and music decoders to understand and generate music from text, images, and videos, trained on a 167.69 hour machine-annotated dataset.

  33. Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM era

    cs.CV 2024-11 unverdicted novelty 3.0 of 10

    A survey that organizes over 100 instruction-guided image and multimedia editing papers into a process-based taxonomy, with an emphasis on LLM and MLLM empowered methods.

Pith tools