Pith. sign in

REVIEW 10 cited by

MonoFormer: One Transformer for Both Diffusion and Autoregression

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.16280 v1 pith:A4NLSGG3 submitted 2024-09-24 cs.CV

Chuyang Zhao , Yuxing Song , Wenhao Wang , Haocheng Feng , Errui Ding , Yifan Sun , Xinyan Xiao , Jingdong Wang This is my paper · ORCID
classification cs.CV
keywords generationautoregressiondiffusiontransformervisualtextattentionmask
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Most existing multimodality methods use separate backbones for autoregression-based discrete text generation and diffusion-based continuous visual generation, or the same backbone by discretizing the visual data to use autoregression for both text and visual generation. In this paper, we propose to study a simple idea: share one transformer for both autoregression and diffusion. The feasibility comes from two main aspects: (i) Transformer is successfully applied to diffusion for visual generation, and (ii) transformer training for autoregression and diffusion is very similar, and the difference merely lies in that diffusion uses bidirectional attention mask and autoregression uses causal attention mask. Experimental results show that our approach achieves comparable image generation performance to current state-of-the-art methods as well as maintains the text generation capability. The project is publicly available at https://monoformer.github.io/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. XYZFlow:Scaling Multi dimensional Shortcut Flows for Efficient Generative Modeling

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Conditioning each patch's denoising on the full trajectories of earlier patches lets XYZFlow generate ImageNet images with FID 1.22 to 1.63 in only 2 to 5 steps per patch, at 7.2 to 8.5x teacher speedups.

  2. UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding

    cs.CV 2025-02 conditional novelty 6.0 of 10

    UniCMs applies consistency distillation to a unified multimodal transformer, treating image mask-diffusion steps and text parallel-decoding steps as one shared denoising trajectory, enabling 2 to 8 step generation and...

  3. MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants

    cs.AI 2024-12 conditional novelty 6.0 of 10

    A new 1.47M-instance biomedical instruction-tuning dataset improves a mixed-modal 7B model's medical VQA accuracy by 18 to 26 percentage points over GPT-4o and Chameleon.

  4. Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Diffusion-conditioned denoising trains a video tokenizer whose features support both video question answering and text-to-video generation in a single LLM.

  5. Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoning

    cs.RO 2024-12 conditional novelty 6.0 of 10

    A robot policy generates its own language reasoning before acting and injects it into a diffusion action decoder, outperforming several VLA baselines on real-robot manipulation.

  6. Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads

    cs.CV 2024-11 conditional novelty 6.0 of 10

    Orthus combines an LM head and a diffusion head in one transformer to generate interleaved text and images, claiming better GenEval and MME-P scores than Chameleon and Show-o.

  7. SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A unified multimodal model can improve its own text-to-image generation by using its understanding module as a rewarder in a global-plus-local reward-weighted training loop.

  8. ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback

    cs.AI 2025-05 conditional novelty 5.0 of 10

    A ComfyUI-based multi-agent system with semantic workflow modules and tree-based local-feedback planning reports near-perfect pass rates on ComfyBench and competitive scores on GenEval and Reason-Edit.

  9. Causal Diffusion Transformers for Generative Modeling

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A decoder-only transformer that factors generation over both token order and noise level, coupling autoregressive and diffusion training, achieves competitive ImageNet generation and in-context editing.

  10. MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation

    cs.CV 2025-06 conditional novelty 4.0 of 10

    An empirical testbed shows that allocating more transformer layers to autoregressive conditioning improves FID under low inference budgets, while block size should scale with image resolution.

Pith tools