REVIEW 42 cited by
Muse: Text-To-Image Generation via Masked Generative Transformers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io
Forward citations
Cited by 42 Pith papers
-
Dimension-Free Convergence of Discrete Diffusion Models: Adjoint Equations Induce the Right Space
Adjoint-equation framework yields dimension-free convergence bounds in any IPM for discrete diffusion models under masked or uniform priors using one rate-matrix regularity assumption.
-
Large Language Diffusion Models
LLaDA is a scalable diffusion-based language model that matches autoregressive LLMs like LLaMA3 8B on tasks and surpasses GPT-4o on reversal poem completion.
-
MentalThink: Shaping Thoughts in Mental SVG World
MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.
-
FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model
FlexiSLM is the first spoken language model supporting dynamic and controllable frame rates on speech input and output, outperforming fixed-rate 7B models at high quality and enabling faster inference at lower rates l...
-
Expected Free Energy-based Planning as Variational Inference
EFE-based planning is formulated as variational free energy minimization with epistemic priors, decomposing into expected plan costs plus a complexity term.
-
What Type of Inference is Active Inference?
Proper EFE-based planning is VFE plus planning and epistemic entropy corrections, realized by channel-reparameterized message passing that captures novelty.
-
What Type of Inference is Active Inference?
EFE-based active inference planning is characterized as VFE on an augmented model plus entropy and planning corrections, with a derived message-passing implementation and grid-world validation.
-
Diffusing in the Right Space: A Systematic Study of Latent Diffusability
A large-scale empirical study across tokenizers and diffusion backbones identifies Velocity Irreducible Variance (VIV) as one of the most stable predictors of latent diffusion generation quality.
-
Dimension-Free Convergence of Discrete Diffusion Models: Adjoint Equations Induce the Right Space
Introduces adjoint-equation framework establishing dimension-free convergence bounds in any IPM for discrete diffusion models under masked and uniform priors.
-
Dimension-Free Convergence of Discrete Diffusion Models: Adjoint Equations Induce the Right Space
Using adjoint equations, the paper derives S-independent IPM convergence bounds for masked and uniform discrete diffusion, controlled by score-matching losses.
-
UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
UDM-GRPO is the first RL integration for uniform discrete diffusion models, using final clean samples as actions and forward-process trajectory reconstruction to raise GenEval accuracy from 69% to 96% and OCR accuracy...
-
Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation
DREAM introduces Masking Warmup and Semantically Aligned Decoding to let a single encoder handle both contrastive alignment and masked generation, yielding gains over CLIP and FLUID on understanding and generation benchmarks.
-
Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewards
SOLACE improves text-to-image generation by using intrinsic self-confidence rewards from noise reconstruction accuracy during reinforcement learning post-training without external supervision.
-
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Janus decouples visual encoding into task-specific pathways inside a single autoregressive transformer to unify multimodal understanding and generation while outperforming earlier unified models.
-
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation
Scaled vanilla autoregressive models based on Llama achieve 2.18 FID on ImageNet 256x256 image generation, beating popular diffusion models without visual inductive biases.
-
Learning Interactive Real-World Simulators
UniSim learns a universal real-world simulator from orchestrated diverse datasets, enabling zero-shot deployment of policies trained purely in simulation.
-
Finite Scalar Quantization: VQ-VAE Made Simple
Finite scalar quantization simplifies VQ-VAE latents by independently rounding a few dimensions to fixed levels, producing an equivalent-sized implicit codebook with competitive performance and no collapse.
-
SplitAvatar: One-shot Head Avatar with Autoregressive Gaussian Splitting
SplitAvatar applies an autoregressive graph splitting network with mesh topology extension and gated density control to generate detailed one-shot head avatars via 3D Gaussian Splatting.
-
Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South
A participatory red-teaming project in the Global South created the PLACES dataset of 26k T2I failure examples that reveal unique cultural and linguistic harms missed by existing safety frameworks.
-
VAGS: Velocity Adaptive Guidance Scale for Image Editing and Generation
VAGS adapts the CFG scale at each ODE step using velocity alignment signals to raise structural fidelity in editing and sample quality in generation over fixed-scale baselines.
-
Coupling Models for One-Step Discrete Generation
Coupling Models enable single-step discrete sequence generation via learned couplings to Gaussian latents and outperform prior one-step baselines on text perplexity, biological FBD, and image FID metrics.
-
UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
UDM-GRPO integrates reinforcement learning with Uniform Discrete Diffusion Models by treating the final clean sample as the action and reconstructing trajectories via the forward process, achieving state-of-the-art te...
-
Controllable Image Generation with Composed Parallel Token Prediction
A new formulation for composing discrete generative processes enables precise control over novel condition combinations in image generation, cutting error rates by 63% and speeding up inference.
-
Sparse-LaViDa: Sparse Multimodal Discrete Diffusion Language Models
Sparse-LaViDa speeds up masked diffusion LLMs by truncating masked tokens, using register tokens and a step-causal attention mask, reaching ~2-2.8x speedup over LaViDa-O with comparable generation quality.
-
RubricRL: Simple Generalizable Rewards for Text-to-Image Generation
Using an LLM to generate prompt-specific visual rubrics and grade each criterion independently gives a more interpretable reward that improves text-to-image model alignment beyond composite and learned scalar rewards.
-
IAR2: Improving Autoregressive Visual Generation with Semantic-Detail Associated Token Prediction
IAR2 achieves state-of-the-art ImageNet 256×256 image generation (FID 1.50 with rejection sampling) by splitting visual tokens into semantic and detail codes and predicting them hierarchically with a local-context-awa...
-
Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation
Lavida-O introduces an elastic mixture-of-transformers architecture that brings high-resolution text-to-image generation, object grounding, and image editing into a single masked diffusion model, using planning and se...
-
Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking
SMARLI achieves strong layout control in autoregressive text-to-image generation via structured attention masks and GRPO post-training with a CLIP-based layout reward.
-
MMaDA: Multimodal Large Diffusion Language Models
MMaDA is a unified multimodal diffusion model using mixed chain-of-thought fine-tuning and a new UniGRPO reinforcement learning algorithm that outperforms specialized models in reasoning, understanding, and text-to-im...
-
Autoregressive Video Generation without Vector Quantization
NOVA reformulates video generation as non-quantized autoregressive frame-by-frame temporal prediction combined with set-by-set spatial prediction, outperforming prior AR video models and some diffusion models in effic...
-
Controllable Image Generation with Composed Parallel Token Prediction
A derived formulation for composing discrete probabilistic generative processes enables novel condition combinations in image generation, yielding 63.4% relative error reduction and FID gains on CLEVR and FFHQ datasets.
-
VideoPoet: A Large Language Model for Zero-Shot Video Generation
VideoPoet is a large language model that performs zero-shot video generation with audio from diverse multimodal conditioning signals.
-
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
Open-source text-to-video and image-to-video diffusion models generate high-quality 1024x576 videos, with the I2V variant claimed as the first to strictly preserve reference image content.
-
DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory
DragNUWA integrates text, image, and trajectory controls into a diffusion video model using a Trajectory Sampler, Multiscale Fusion, and Adaptive Training to enable fine-grained open-domain video generation.
-
Scaling Robot Learning with Semantically Imagined Experience
Augmenting robot datasets via diffusion-based semantic inpainting enables manipulation policies to solve unseen tasks with new objects and improves robustness to novel distractors.
-
Co-occurring associated retained concepts in Diffusion Unlearning
Defines CARE score and proposes ReCARE framework to preserve co-occurring benign concepts during targeted unlearning in diffusion models.
-
ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations
ARM is a 7B autoregressive multimodal model with a unified discrete visual tokenizer and RL that performs image understanding, generation, and editing while showing cross-task synergy from preference optimization.
-
The Script is All You Need: An Agentic Framework for Long-Horizon Dialogue-to-Cinematic Video Generation
An agentic dialogue-to-video pipeline (ScripterAgent, DirectorAgent, CriticAgent) claims to improve long-horizon cinematic coherence, but its supporting evaluation is partly self-referential.
-
MSDformer: Multi-scale Discrete Transformer For Time Series Generation
MSDformer introduces a multi-scale discrete transformer that tokenizes time series at multiple scales and models them autoregressively in discrete space, claiming superior performance over prior DTM methods with rate-...
-
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
Show-o unifies autoregressive and discrete diffusion modeling inside one transformer to support multimodal understanding and generation tasks with competitive benchmark performance.
-
Zero-Residual Concept Erasure via Progressive Alignment in Text-to-Image Model
A closed-form concept erasure method that enforces zero alignment residual in the optimization objective and applies updates progressively across layers to better preserve generation quality.
-
FBI: Learning Dexterous In-hand Manipulation with Dynamic Visuotactile Shortcut Policy
The abstract claims a new visuotactile robot manipulation policy (FBI) that outperforms baselines, but the manuscript body is an unrelated paper on text-to-image synthesis, so the claimed result is absent.
Discussion (0). Sign in to comment.