Pith. sign in

REVIEW 2 major objections 4 minor 5 cited by

A single mask-based discrete diffusion model unifies text, speech, and image understanding and generation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 13:42 UTC pith:UE75FE3V

load-bearing objection Solid first pure-MDM any-to-any system with competitive numbers; the joint-alignment claim is only weakly isolated from TTS leakage. the 2 major comments →

arxiv 2603.06577 v2 pith:UE75FE3V submitted 2026-03-06 cs.CV

Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion

Lijiang Li , Zuwei Long , Yunhang Shen , Heting Gao , Haoyu Cao , Xing Sun , Caifeng Shan , Ran He
show 1 more author
Chaoyou Fu
This is my paper
classification cs.CV
keywords mask-based discrete diffusionany-to-any multimodaljoint token distributiontext-speech-imageparallel decodingspeech-driven visual interactionattenuated tail-pad masking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most multimodal systems still rest on autoregressive language models and bolt on separate heads or decoders for other modalities. This paper argues that a pure mask-based discrete diffusion model can replace that stack. Omni-Diffusion tokenizes text, speech, and images into a shared discrete vocabulary, then trains one diffusion network to predict the joint distribution of those tokens by recovering randomly masked positions. Progressive three-stage training plus a small speech-driven visual interaction set is used to align the modalities. The resulting 7B model handles ASR, TTS, VQA, text-to-image, speech-to-image, and spoken visual dialogue, matching or beating existing any-to-any systems on the reported benchmarks while also supporting parallel decoding and free-form inpainting without extra training.

Core claim

A single mask-token diffusion model that directly models the joint distribution over discrete text, speech, and image tokens is sufficient to produce an any-to-any multimodal system whose understanding and generation quality is competitive with specialized or autoregressive multimodal models.

What carries the argument

Unified mask-token prediction: every modality is converted to discrete tokens, wrapped by modality markers, randomly masked, and recovered by one cross-entropy diffusion objective; no modality-specific loss or decoder is used during training.

Load-bearing premise

That three-stage fine-tuning of a pre-trained diffusion language model plus fixed off-the-shelf tokenizers and roughly 60k synthetic speech-visual examples produces a truly shared multimodal semantic space rather than loosely coupled behaviors that only look aligned on the chosen tests.

What would settle it

A controlled speech-to-image versus text-to-image evaluation on a held-out set whose spoken captions are deliberately noisy or accented; if CLIP scores or human preference collapse relative to clean text conditioning, the claimed joint alignment fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. Omni-Diffusion is presented as the first any-to-any multimodal model whose backbone is entirely a mask-based discrete diffusion language model (Dream-7B). Discrete tokens from text, MAGVIT-v2 images, and GLM-4-Voice speech are concatenated into a single sequence; the model is trained with the standard masked cross-entropy objective (Eq. 1) to capture their joint distribution. A three-stage progressive fine-tuning schedule, an attenuated tail-pad masking strategy, a synthetic Speech-Driven Visual Interaction (SDVI) set, and several inference heuristics (position penalty, special-token pre-infilling, adaptive length) are introduced. Empirical results on LibriSpeech/LibriTTS ASR/TTS, POPE/MME/Seed-2-Plus VQA, MSCOCO CLIP text-to-image, and a speech-to-image transfer of the same captions are reported as competitive with or better than prior any-to-any systems (AnyGPT, NExT-GPT) and comparable to some specialist models.

Significance. If the joint-distribution claim holds, the work supplies a concrete alternative backbone for multimodal foundation models and demonstrates that discrete diffusion can support simultaneous understanding and generation across three modalities without modality-specific output heads. The parallel-decoding efficiency results (Table 3, Fig. 7) and the zero-shot inpainting capability are genuine practical advantages over pure autoregressive designs. The paper is therefore of clear interest to the multimodal and generative-modeling communities even if some of the alignment evidence remains incomplete.

major comments (2)
  1. The central claim of an intrinsically aligned joint distribution over speech and image tokens rests almost exclusively on Table 3: speech-to-image CLIP scores (0.225/0.645) nearly match text-to-image (0.235/0.667) when MSCOCO captions are first converted by CosyVoice2. Because CosyVoice2 is high-fidelity TTS, the speech tokens are near-deterministic encodings of the original text; the model can succeed by recovering text-like content (or via the special-token pre-infilling path of §3.4 that explicitly injects text semantics) rather than by learning a shared speech–image manifold. The same CosyVoice2 cloning is used to construct the entire SDVI training set (§3.3). Without a control that breaks the text–speech isomorphism (real spontaneous, accented, or noisy speech; or an ablation that disables pre-infilling), the numbers do not distinguish genuine joint modeling from loosely coupled mod
  2. ASR performance (Table 1, WER 7.05 on LibriSpeech) is better than AnyGPT but substantially worse than the speech LLM baseline GLM-4-Voice (2.82). The paper still claims “outperforms or performs on par with existing multimodal systems that process two or more modalities.” The claim is true for the any-to-any subset but is overstated relative to the broader multimodal literature; the abstract and §4.1 should be tightened so that the comparison class is unambiguous and the remaining gap to specialist speech models is acknowledged.
minor comments (4)
  1. Table 3 is reused for both the speech–vision alignment experiment and the sampling-efficiency experiment; the two uses should be separated into distinct tables for clarity.
  2. Several free parameters (γ for attenuated tail-pad, γp and Nt for position penalty, the 0.25L pre-infill index, the 3.5×/0.2× adaptive length multipliers) are introduced without ablation. Even a short sensitivity study would strengthen confidence that the reported numbers are not brittle to these choices.
  3. Typographical inconsistencies appear throughout (e.g., “V oice”, “tipically”, “regidly”, “CosyV oice”). A careful proof-reading pass is needed.
  4. Figure 1 and the qualitative spoken-visual examples (Fig. 4) are useful but lack quantitative metrics (e.g., ASR WER on the spoken questions or human preference scores) that would make the multi-modality claim more rigorous.

Circularity Check

0 steps flagged

Standard train-then-evaluate ML paper; no derivation reduces a reported metric to a fitted constant or self-citation by construction.

full rationale

Omni-Diffusion is an empirical systems paper: it initializes from Dream-7B, expands the vocabulary for MAGVIT-v2 image tokens and GLM-4-Voice speech tokens, trains with the standard mask-token cross-entropy (Eq. 1) under a three-stage progressive schedule, and reports WER / POPE / MME / Seed-2-Plus / CLIP-T/I against external public benchmarks (LibriSpeech, LibriTTS, MSCOCO, etc.). The joint-distribution claim is an architectural modeling choice, not a uniqueness theorem or closed-form derivation. SDVI is synthetic training data constructed from LLaVA-OneVision and JourneyDB via CosyVoice2; it is not used as the sole evaluation of the main claim. Self-citations (VITA series, VITA-Audio, Dream-7B) supply tooling and the base model but do not force the benchmark numbers. The speech-to-image ≈ text-to-image CLIP scores (Table 3) may overstate cross-modal alignment because CosyVoice2 speech is near-isomorphic to text, but that is an evaluation-design / correctness concern, not circularity: the metrics are still computed on held-out MSCOCO captions and are not algebraically identical to any fitted training parameter. No self-definitional loop, no fitted-input-called-prediction, and no load-bearing uniqueness imported from the authors. Score 1 for ordinary self-use of related prior tooling that is not load-bearing for the reported results.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 4 invented entities

The central empirical claim rests on standard discrete-diffusion training assumptions, frozen or lightly adapted external tokenizers/decoders, a pretrained Dream-7B backbone, hand-chosen inference hyperparameters, and a synthetic spoken-visual dataset. No new physical entities; the free knobs and domain assumptions below are what a re-implementer must accept or retune for the reported quality to appear.

free parameters (5)
  • attenuated tail-pad mask scale γ = 0.6
    Set to 0.6 to reduce pad-token overfitting; chosen by authors, not derived (§3.3, App. A).
  • position penalty γp and Nt = γp=0.5, Nt=L-100
    Logits of last Nt tokens scaled by γp early in sampling to reduce image repetition; γp=0.5, Nt=L−100 (App. A).
  • adaptive mask-length multipliers = 3.5 (TTS), 0.2 (ASR)
    TTS initial length 3.5× text tokens; ASR 0.2× speech tokens—heuristic ratios tied to speech rate (§3.4).
  • special-token pre-infill index = 0.25L
    [begin-of-speech] placed at 0.25L of the mask sequence to force text-then-speech layout (§3.4).
  • stage-wise learning rates = 1e-4 / 1e-5
    1e-4 for stages 1–2, 1e-5 for stage 3; standard but claim-sensitive training knobs (App. A).
axioms (5)
  • domain assumption Mask-based discrete diffusion with cross-entropy on masked positions (Eq. 1) is a valid joint model for mixed-modality discrete token sequences.
    Imported from Dream/MDM literature; training treats text, speech, and image tokens identically under this loss (§3.1).
  • domain assumption MAGVIT-v2 (f=16, codebook 8192), SenseVoiceSmall+MLP, and GLM-4-Voice (12.5 Hz, codebook 16384) provide adequate discrete interfaces for understanding and generation.
    Tokenizers/decoders are frozen external components; quality ceiling of Omni-Diffusion is bounded by them (§3.2).
  • domain assumption Dream-7B-Instruct is a suitable pretrained backbone whose text capability transfers under progressive multimodal fine-tuning.
    All multimodal ability is obtained by extending this checkpoint’s vocab and continuing training (§3.2–3.3).
  • ad hoc to paper Synthetic SDVI speech (CosyVoice2 voice-clone over filtered LLaVA-OneVision and JourneyDB captions) is distributionally close enough to real spoken visual interaction for stage-3 alignment.
    Dataset is constructed by the authors with filtering rules (no math/code, open-ended rewrite, ≤100-word answers) and fixed/cloned voices (§3.3).
  • domain assumption Entropy-based parallel unmasking with repetition penalty and CFG yields fair comparison to AR multimodal systems on the chosen metrics.
    Inference protocol follows Dream-Instruct; step-count reductions are reported but not standardized against AR latency (§3.4, §4.3).
invented entities (4)
  • Speech-Driven Visual Interaction (SDVI) dataset no independent evidence
    purpose: Stage-3 fine-tuning for spoken VQA and speech-to-image to force joint speech–vision alignment.
    ≈30k spoken VQA + 30k speech–image pairs synthesized by the authors; no external independent release described in the paper.
  • Attenuated Tail-Pad Masking no independent evidence
    purpose: Reduce pad-token overfitting while still allowing variable-length generation in MDMs.
    Paper-specific training modification (scale γ on pad masks); validated only by authors’ qualitative observation of pad overfitting (§3.3).
  • Position Penalty (image decoding) no independent evidence
    purpose: Discourage simultaneous decoding from both ends of the image token sequence to reduce repetitive patterns.
    Soft order constraint introduced here; distinct from semi-AR block decoding they cite (§3.4).
  • Special Token Pre-Infilling for spoken dialogue no independent evidence
    purpose: Force interleaved text-then-speech layout so speech generation can attend to text semantics.
    Uses MDM’s editable initial mask; layout fraction 0.25L is author-chosen (§3.4).

pith-pipeline@v1.1.0-grok45 · 19585 in / 4334 out tokens · 39670 ms · 2026-07-15T13:42:59.426420+00:00 · methodology

0 comments
read the original abstract

While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving significant room to explore effective and efficient alternatives in architectural design. Concurrently, recent studies have successfully applied discrete diffusion models to various domains, such as visual understanding and image generation, revealing their considerable potential as a promising backbone for multimodal systems. Drawing inspiration from these pioneering studies, we introduce Omni-Diffusion, the first any-to-any multimodal language model built entirely on mask-based discrete diffusion models, which unifies understanding and generation across text, speech, and images. Omni-Diffusion employs a unified mask-based discrete diffusion model to directly capture the joint distribution over discrete multimodal tokens. This approach supports not only bimodal tasks but also more complex scenarios involving multiple modalities. On a diverse set of benchmarks, our method outperforms or performs on par with existing multimodal systems that process two or more modalities, highlighting the significant promise of diffusion models in powering the next generation of multimodal foundation models. Project webpage: https://omni-diffusion.github.io.

Figures

Figures reproduced from arXiv: 2603.06577 by Caifeng Shan, Chaoyou Fu, Haoyu Cao, Heting Gao, Lijiang Li, Ran He, Xing Sun, Yunhang Shen, Zuwei Long.

Figure 1
Figure 1. Figure 1: Overview of Omni-Diffusion. Our model takes multimodal tokens as input, processes them using a unified mask-based discrete [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture overview. Omni-Diffusion is an any-to-any [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training pipeline of Omni-Diffusion. The first stage [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Generated samples of Omni-Diffusion on spoken interac [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Generated samples of Omni-Diffusion on text-to-image and speech-to-image tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Generated samples of Omni-Diffusion under various [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Generated samples of Omni-Diffusion on text-to-image task. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Generated samples of Omni-Diffusion on speech-to-image task. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ELF: Embedded Language Flows

    cs.CL 2026-05 unverdicted novelty 6.0

    ELF is a continuous embedding-space flow matching model for language that stays continuous until the last step and outperforms prior discrete and continuous diffusion language models with fewer sampling steps.

  2. ELF: Embedded Language Flows

    cs.CL 2026-05 unverdicted novelty 6.0

    ELF applies continuous-time flow matching in embedding space for language generation and reports outperforming prior discrete and continuous diffusion language models with fewer steps.

  3. Omnimodal Dataset Distillation via High-order Proxy Alignment

    cs.CV 2026-04 unverdicted novelty 6.0

    HoPA captures high-order cross-modal alignments via a shared proxy to enable scalable omnimodal dataset distillation with better performance-compression trade-offs.

  4. Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator

    cs.CV 2026-04 unverdicted novelty 6.0

    Uni-ViGU unifies video generation and understanding by extending a diffusion video generator with unified continuous-discrete flow matching, modality-driven MoE layers, and bidirectional training stages that repurpose...

  5. MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset

    cs.CV 2026-05 unverdicted novelty 5.0

    MONET is an open 104.9M image-text pair dataset created via safety filtering, deduplication, and multi-VLM recaptioning from 2.9B raw pairs, validated by training a competitive 4B-parameter latent diffusion model.

Reference graph

Works this paper leans on

55 extracted references · 23 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms.arXiv preprint arXiv:2407.04051, 2024

    Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, et al. Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms.arXiv preprint arXiv:2407.04051, 2024. 4, 12

    Pith/arXiv arXiv 2024

  2. [2]

    Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and V olodymyr Kuleshov

    Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and V olodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. InICLR, 2025. 3

    2025

  3. [3]

    Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan

    Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jos´e Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. InICML, 2023. 3

    2023

  4. [4]

    GigaSpeech: An evolving, multi-domain ASR corpus with 10, 000 hours of transcribed audio

    Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei- Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watan- abe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. GigaSpeech: An evolving, multi-domain ASR corpus with 10, 000 hours of transcribed audi...

    2021

  5. [5]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025. 1, 5

    Pith/arXiv arXiv 2025

  6. [6]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. InNeurIPS,

  7. [7]

    Dreamllm: Synergistic multimodal comprehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal comprehension and creation. InICLR, 2024. 6

    2024

  8. [8]

    Cosyvoice: A scalable multilingual zero-shot text- to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text- to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024. 6

    Pith/arXiv arXiv 2024

  9. [9]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117,

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117,

    Pith/arXiv arXiv

  10. [10]

    Vita: Towards open-source interactive omni multimodal llm.arXiv preprint arXiv:2408.05211, 2024

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, et al. Vita: Towards open-source interactive omni multimodal llm.arXiv preprint arXiv:2408.05211, 2024. 1, 2

    Pith/arXiv arXiv 2024

  11. [11]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. InNeurIPS, 2025. 6

    2025

  12. [12]

    Vita-1.5: Towards gpt-4o level real-time vi- sion and speech interaction.arXiv preprint arXiv:2501.01957,

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yun- hang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vi- sion and speech interaction.arXiv preprint arXiv:2501.01957,

    Pith/arXiv arXiv

  13. [13]

    The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage

    Daniel Galvez, Greg Diamos, Juan Torres, Keith Achorn, Juan Felipe Cer ´on, Anjali Gopi, David Kanter, Max Lam, Mark Mazumder, and Vijay Janapa Reddi. The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage. InNeurIPS, 2021. 12

    2021

  14. [14]

    Lucy: Linguistic understanding and control yielding early stage of her.arXiv preprint arXiv:2501.16327,

    Heting Gao, Hang Shao, Xiong Wang, Chaofan Qiu, Yunhang Shen, Siqi Cai, Yuchen Shi, Zihan Xu, Zuwei Long, Yike Zhang, et al. Lucy: Linguistic understanding and control yielding early stage of her.arXiv preprint arXiv:2501.16327,

    Pith/arXiv arXiv

  15. [15]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. InCVPR, 2024. 2

    2024

  16. [16]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In IEEE SLT, 2024. 12

    2024

  17. [17]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. InNeurIPS, 2020. 3

    2020

  18. [18]

    Pc-sampler: Position-aware calibration of decoding bias in masked diffu- sion models.arXiv preprint arXiv:2508.13021, 2025

    Pengcheng Huang, Shuhao Liu, Zhenghao Liu, Yukun Yan, Shuo Wang, Zulong Chen, and Tong Xiao. Pc-sampler: Position-aware calibration of decoding bias in masked diffu- sion models.arXiv preprint arXiv:2508.13021, 2025. 5

    arXiv 2025

  19. [19]

    Tulu 3: Push- ing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Mi- randa, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Push- ing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024. 12

    Pith/arXiv arXiv 2024

  20. [20]

    Seed-bench-2-plus: Benchmarking multi- modal large language models with text-rich visual compre- hension.arXiv preprint arXiv:2404.16790, 2024

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multi- modal large language models with text-rich visual compre- hension.arXiv preprint arXiv:2404.16790, 2024. 6

    Pith/arXiv arXiv 2024

  21. [21]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 4, 12

    Pith/arXiv arXiv 2024

  22. [22]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InEMNLP, 2023. 6

    2023

  23. [23]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014. 6

    2014

  24. [24]

    Visual instruction tuning.NeurIPS, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NeurIPS, 2023. 6

    2023

  25. [25]

    VITA-Audio: Fast interleaved cross- modal token generation for efficient large speech-language model.arXiv preprint arXiv:2505.03739, 2025

    Zuwei Long, Yunhang Shen, Chaoyou Fu, Heting Gao, Li- jiang Li, Peixian Chen, Mengdan Zhang, Hang Shao, Jian Li, Jinlong Peng, et al. VITA-Audio: Fast interleaved cross- modal token generation for efficient large speech-language model.arXiv preprint arXiv:2505.03739, 2025. 4 9

    arXiv 2025

  26. [26]

    Discrete diffusion modeling by estimating the ratios of the data distri- bution

    Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distri- bution. InICML, 2024. 3

    2024

  27. [27]

    Next-omni: Towards any-to-any omnimodal foundation models with discrete flow matching.arXiv preprint arXiv:2510.13721, 2025

    Run Luo, Xiaobo Xia, Lu Wang, Longze Chen, Renke Shan, Jing Luo, Min Yang, and Tat-Seng Chua. Next-omni: Towards any-to-any omnimodal foundation models with discrete flow matching.arXiv preprint arXiv:2510.13721, 2025. 1, 3

    arXiv 2025

  28. [28]

    Diffusion language models are super data learners.arXiv preprint arXiv:2511.03276, 2025

    Jinjie Ni, Qian Liu, Longxu Dou, Chao Du, Zili Wang, Hang Yan, Tianyu Pang, and Michael Qizhe Shieh. Diffusion language models are super data learners.arXiv preprint arXiv:2511.03276, 2025. 1

    arXiv 2025

  29. [29]

    Librispeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. InICASSP, 2015. 6, 12

    2015

  30. [30]

    Robust speech recog- nition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recog- nition via large-scale weak supervision. InICML, 2023. 6

    2023

  31. [31]

    LAION- 5B: an open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Lud- wig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION- 5B: an open large-scale dataset for training next generation image-text mode...

    2022

  32. [32]

    video-salmonn: Speech-enhanced audio-visual large language models

    Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. InICML, 2024. 2

    2024

  33. [33]

    Journeydb: A benchmark for generative image under- standing

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. Journeydb: A benchmark for generative image under- standing. InNeurIPS, 2023. 12

    2023

  34. [34]

    Generative multimodal models are in- context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in- context learners. InCVPR, 2024. 6

    2024

  35. [35]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. InICLR, 2024. 6

    2024

  36. [36]

    V oxPopuli: A large-scale multilingual speech corpus for representation learning, semi- supervised learning and interpretation

    Changhan Wang, Morgane Rivi`ere, Ann Lee, Anne Wu, Chai- tanya Talnikar, Daniel Haziza, Mary Williamson, Juan Miguel Pino, and Emmanuel Dupoux. V oxPopuli: A large-scale multilingual speech corpus for representation learning, semi- supervised learning and interpretation. InACL/IJCNLP, 2021. 12

    2021

  37. [37]

    Globe: A high-quality english corpus with global accents for zero-shot speaker adaptive text-to-speech.arXiv preprint arXiv:2406.14875, 2024

    Wenbin Wang, Yang Song, and Sanjay Jha. Globe: A high-quality english corpus with global accents for zero-shot speaker adaptive text-to-speech.arXiv preprint arXiv:2406.14875, 2024. 12

    Pith/arXiv arXiv 2024

  38. [38]

    Fast-dllm v2: Efficient block-diffusion llm

    Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yong- gan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. Fast-dllm v2: Efficient block-diffusion llm. arXiv preprint arXiv:2509.26328, 2025. 1

    arXiv 2025

  39. [39]

    Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618, 2025

    Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.arXiv preprint arXiv:2505.22618, 2025. 1

    Pith/arXiv arXiv 2025

  40. [40]

    Next-gpt: Any-to-any multimodal LLM

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal LLM. InICML,

  41. [41]

    Show-o: One single transformer to unify multimodal understanding and genera- tion.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and genera- tion.arXiv preprint arXiv:2408.12528, 2024. 4

    Pith/arXiv arXiv 2024

  42. [42]

    Mini-omni: Language models can hear, talk while thinking in streaming.arXiv preprint arXiv:2408.16725, 2024

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming.arXiv preprint arXiv:2408.16725, 2024. 12

    Pith/arXiv arXiv 2024

  43. [43]

    Lumina-dimoo: An omni diffusion large language model for multi-modal generation and understanding.arXiv preprint arXiv:2510.06308, 2025

    Yi Xin, Qi Qin, Siqi Luo, Kaiwen Zhu, Juncheng Yan, Yan Tai, Jiayi Lei, Yuewen Cao, Keqi Wang, Yibin Wang, et al. Lumina-dimoo: An omni diffusion large language model for multi-modal generation and understanding.arXiv preprint arXiv:2510.06308, 2025. 1, 5

    arXiv 2025

  44. [44]

    Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo ...

    Pith/arXiv arXiv 2025

  45. [45]

    Mmada: Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809,

    Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models.arXiv preprint arXiv:2505.15809,

    Pith/arXiv arXiv

  46. [46]

    A survey of unified multimodal understanding and generation: Advances and challenges.Authorea Preprints,

    Yan Yang, Haochen Tian, Yang Shi, Wulin Xie, Yi-Fan Zhang, Yuhao Dong, Yibo Hu, Liang Wang, Ran He, Caifeng Shan, et al. A survey of unified multimodal understanding and generation: Advances and challenges.Authorea Preprints,

  47. [47]

    Dream 7b: Diffu- sion large language models.arXiv preprint arXiv:2508.15487,

    Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffu- sion large language models.arXiv preprint arXiv:2508.15487,

    Pith/arXiv arXiv

  48. [48]

    mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023. 6

    Pith/arXiv arXiv 2023

  49. [49]

    Llada-v: Large language diffusion models with visual instruction tuning

    Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.16933, 2025. 3

    Pith/arXiv arXiv 2025

  50. [50]

    Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A

    Lijun Yu, Jos´e Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to visual generation. InICLR, 2024. 3, 12 10

    2024

  51. [51]

    Dimple: Dis- crete diffusion multimodal large language model with parallel decoding.arXiv preprint arXiv:2505.16990, 2025

    Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Dis- crete diffusion multimodal large language model with parallel decoding.arXiv preprint arXiv:2505.16990, 2025. 1, 3, 5

    Pith/arXiv arXiv 2025

  52. [52]

    Libritts: A corpus derived from librispeech for text-to-speech.arXiv preprint arXiv:1904.02882, 2019

    Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech.arXiv preprint arXiv:1904.02882, 2019. 6, 12

    Pith/arXiv arXiv 1904

  53. [53]

    Glm- 4-voice: Towards intelligent and human-like end-to-end spo- ken chatbot.arXiv preprint arXiv:2412.02612, 2024

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm- 4-voice: Towards intelligent and human-like end-to-end spo- ken chatbot.arXiv preprint arXiv:2412.02612, 2024. 4, 6, 12

    Pith/arXiv arXiv 2024

  54. [54]

    Anygpt: Unified multimodal LLM with discrete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal LLM with discrete sequence modeling. InACL, 2024. 1, 3, 6

    2024

  55. [55]

    Llada 1.5: Variance-reduced preference opti- mization for large language diffusion models.arXiv preprint arXiv:2505.19223, 2025

    Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chun- wei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference opti- mization for large language diffusion models.arXiv preprint arXiv:2505.19223, 2025. 1, 3 11 A. Implementation Details Our model is initialized with the weights of the pre-trained Drea...

    Pith/arXiv arXiv 2025