Pith. sign in

REVIEW 35 references

Block discrete diffusion over X-Codec2 tokens yields competitive zero-shot TTS with 0.6B parameters, 20K training hours, and a 0.15 real-time factor.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:45 UTC pith:LGXTNJ7J

arxiv 2608.00011 v1 pith:LGXTNJ7J submitted 2026-06-19 cs.CL cs.AI

DLLM-TTS: Block Discrete Diffusion Language Model for Text-to-Speech Synthesis

Wasim Madha , Nityanand Mathur , Hamees Sayed , Apoorv Singh , Sameer Khurana , Akshat Mandloi , Sudarshan Kamath This is my paper
classification cs.CL cs.AI
keywords blockblocksdiffusiondiscretelanguagemodelmodelscodec
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Speech is encoded by X-Codec2 into a stream of token IDs (6561 possible tokens, 50 per second). DLLM-TTS splits that stream into blocks of 32 tokens. During generation, it starts with a block of [MASK] placeholders; a transformer predicts which tokens are likely, unmasking tokens whose predicted probability passes a threshold, and repeats. Since tokens inside a block are predicted in parallel, generation is faster than autoregressive TTS.

To keep text-speech alignment, blocks are processed left to right: earlier confirmed blocks are visible to later noisy blocks, while inside the current block attention is free in both directions. The transformer is pre-initialized from Qwen2 and trained twice: 16K hours of Emilia audio, then 4K hours of synthetic high-quality speech. On the Seed-TTS-eval benchmark the authors report WER 2.25%, speaker similarity 0.750, MOS 4.25; ablations show B=32 and 32 denoising steps are best.

No code, model weights, audio demos, or error bars are included, and the paper mixes a T=32 configuration for quality numbers with a T=16 configuration for the 0.15 RTF claim.

Core claim

DLLM-TTS "formulates TTS as conditional block discrete diffusion over X-Codec2 neural audio codec tokens" and reports that "a 0.6B-parameter model trained on 20K hours achieves competitive performance on the Seed-TTS-eval benchmark" (Abstract; Section 5). If correct, block diffusion yields near-AR intelligibility with in-block parallel decoding and a 3-12x data reduction relative to AR codec LMs.

Load-bearing premise

X-Codec2's single-stream FSQ tokenization at 50 Hz (vocabulary 6561) preserves enough semantic and acoustic structure that masked token reconstruction alone can reach WER 2.25/SIM 0.75. The paper gives no codec resynthesis ceiling; this assumption enters at Section 3.1 and underlies every training/eval result. If codec reconstruction already caps these metrics, the improvements credited to block diffusion would shrink.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method is a composition of existing components: X-Codec2, masked diffusion, BD3-LM, and Qwen2 initialization. No new physical or mathematical entity is introduced. The main free parameters are inference hyperparameters tuned on the benchmark, and the data-efficiency claim rests on an untested assumption about masked training as augmentation.

free parameters (4)
  • block_size B = 32 (ablated over 8/16/32; B=32 chosen as best WER/CER)
    Selected empirically on the Seed-TTS-eval benchmark (Section 4.3); part of the headline configuration.
  • denoising_steps T = T=32 for reported quality; T=16 for RTF=0.15 (inconsistent)
    Ablated in Table 3; the paper conflates the quality and speed configurations.
  • confidence_threshold tau = 0.6
    Set by hand (Eq. 7); no sensitivity analysis is reported.
  • linear masking schedule alpha_t = 1 - t = alpha_t = 1 - t
    Chosen diffusion schedule; alternatives are not explored.
axioms (4)
  • domain assumption X-Codec2 single-stream FSQ tokens at 50 Hz preserve sufficient semantic and acoustic information for TTS reconstruction.
    Section 3.1 uses this as the representation; no codec resynthesis ceiling on the evaluation set is measured.
  • domain assumption Staircase attention (Eq. 4) provides correct text-speech alignment without explicit duration modeling.
    Section 3.2 assumes the mask pattern enforces both local coherence and global alignment, but no alignment visualization or attention analysis is shown.
  • ad hoc to paper Masked diffusion training provides implicit data augmentation, causing the claimed 3-12x data efficiency.
    Section 4.2 states this as an explanation, but no controlled experiment compares diffusion vs. autoregressive training on the same 20K-hour dataset.
  • domain assumption Published baseline numbers in Table 2 are directly comparable to DLLM-TTS results.
    The paper does not reproduce baselines under identical evaluation conditions; MOS from 25 listeners is a single point value.

pith-pipeline@v1.3.0-alltime-deepseek · 8191 in / 12393 out tokens · 113150 ms · 2026-08-04T01:45:58.244200+00:00 · methodology

0 comments
read the original abstract

Current text-to-speech systems face a trade-off: autoregres- sive codec language models produce highly intelligible speech but require large-scale models and training data and decode tokens sequentially, while non-autoregressive approaches im- prove speed at the cost of linguistic accuracy. We present DLLM-TTS, a framework that formulates TTS as conditional block discrete diffusion over X-Codec2 neural audio codec to- kens. The model decomposes sequences into blocks and applies masked diffusion within each block while processing blocks se- quentially, learning both local acoustic coherence and global text-speech alignment. During inference, parallel token pre- diction within blocks enables efficient generation with a real- time factor (RTF) of 0.15. A 0.6B-parameter model trained on 20K hours achieves competitive performance on the Seed- TTS-eval benchmark, demonstrating that block discrete diffu- sion language models enable practical and data-efficient speech synthesis with parallel generation.

Figures

Figures reproduced from arXiv: 2608.00011 by Akshat Mandloi, Apoorv Singh, Hamees Sayed, Nityanand Mathur, Sameer Khurana, Sudarshan Kamath, Wasim Madha.

Figure 1
Figure 1. Figure 1: Overview of DLLM-TTS. Text and reference codec tokens are concatenated with generation text and [MASK]L targets, then processed by the Block Discrete Diffusion Trans￾former using staircase attention (bidirectional/causal). Each block is iteratively denoised and decoded by X-Codec2 into speech. trained from scratch. MegaTTS 3 [16] proposes sparse align￾ment with a latent diffusion transformer for robust zer… view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of the block-wise denoising process. (B = 32, T = 32) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 19 linked inside Pith

  1. [1]

    Autoregressive codec language models [1, 2, 3] achieve high-quality zero-shot synthesis but require 60K–250K hours of data and generate tokens sequentially, incurring high latency

    Introduction Text-to-speech synthesis faces a fundamental efficiency-quality trade-off. Autoregressive codec language models [1, 2, 3] achieve high-quality zero-shot synthesis but require 60K–250K hours of data and generate tokens sequentially, incurring high latency. Non-autoregressive approaches based on flow match- ing [4, 5] and diffusion [6, 7] enabl...

  2. [2]

    V ALL-E [1] pioneered this approach by training on 60K hours of speech, achieving strong zero-shot synthesis from a 3-second prompt

    Related Work Autoregressive Codec Language Models.A dominant paradigm in TTS formulates speech synthesis as language mod- eling over discrete neural codec tokens. V ALL-E [1] pioneered this approach by training on 60K hours of speech, achieving strong zero-shot synthesis from a 3-second prompt. V ALL- E 2 [2] introduced repetition-aware sampling and group...

    Pith/arXiv arXiv 2026

  3. [3]

    Background Neural Audio Codecs.Neural audio codecs compress contin- uous audio waveforms into discrete token sequences through learned quantization

    Methodology 3.1. Background Neural Audio Codecs.Neural audio codecs compress contin- uous audio waveforms into discrete token sequences through learned quantization. Traditional residual vector quantization (RVQ) approaches like EnCodec [17] employ multiple code- book layers to progressively refine representations, producing parallel token streams. In con...

    2048

  4. [4]

    Experiments 4.1. Experimental Setup Training Data.We train DLLM-TTS through a two-stage curriculum:Stage 1trains the model on 16K hours sampled from the Emilia dataset [20] for 20 epochs, establishing co- herent codec token generation conditioned on text and speaker prompts. Training uses a batch size of 16 per GPU with 8- step gradient accumulation (effe...

  5. [5]

    Conclusion We presented DLLM-TTS, a TTS framework based on block discrete diffusion language modeling over neural audio codec tokens. By decomposing codec token sequences into blocks and applying masked diffusion within each block while pro- cessing them sequentially, the model learns both local acoustic coherence and global text–speech alignment without ...

  6. [6]

    All research contributions, including the methodology, experimen- tal design, results, and scientific claims, are the authors’ own

    Use of Generative AI Disclosure In preparing this manuscript, the authors used generative AI tools for language refinement (rephrasing and improving the clarity of author-written text) and as a coding assistant (help- ing write and debug software for experiments and analysis). All research contributions, including the methodology, experimen- tal design, r...

  7. [7]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Neural codec language mod- els are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    V ALL-E 2: Neural codec language models are human parity zero-shot text to speech syn- thesizers,

    S. Chen, S. Yu, L. Zhou, Y . Wuet al., “V ALL-E 2: Neural codec language models are human parity zero-shot text to speech syn- thesizers,”arXiv preprint arXiv:2406.05370, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    LLASA: Scaling train-time and inference-time compute for LLaMA-based speech synthesis,

    Z. Ye, P. Ai, J. Sunet al., “LLASA: Scaling train-time and inference-time compute for LLaMA-based speech synthesis,” arXiv preprint arXiv:2502.04128, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    V oice- box: Text-guided multilingual universal speech generation at scale,

    M. Le, A. Vyas, B. Shi, B. Karrer, L. Sager, X. Adel, M. Williamson, V . Manohar, N. Moritz, W.-N. Hsuet al., “V oice- box: Text-guided multilingual universal speech generation at scale,” inProc. NeurIPS, 2023

    2023

  11. [11]

    F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Wu, Z. Zhanget al., “F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching,”arXiv preprint arXiv:2410.06885, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y . Liu, Y . Leng, K. Song, S. Tanget al., “NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” inProc. ICML, 2024

    2024

  13. [13]

    StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,

    Y . A. Li, C. Han, V . S. Raber, and N. Mesgarani, “StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,” inProc. NeurIPS, 2023

    2023

  14. [14]

    Simple and effective masked diffusion language models,

    S. S. Sahoo, M. Arriola, Y . Schiff, A. Gokaslan, E. Marroquin, J. T. Chiu, A. Rush, and V . Kuleshov, “Simple and effective masked diffusion language models,” inProc. NeurIPS, 2024

    2024

  15. [15]

    Large language diffusion models,

    S. Nie, F. Zhu, C. You, X. Zhang, and J. Gong, “Large language diffusion models,”arXiv preprint arXiv:2502.09992, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Block diffusion: Interpolating between autoregressive and diffu- sion language models,

    M. Arriola, A. Gokaslan, S. S. Sahoo, L. Hsu, and V . Kuleshov, “Block diffusion: Interpolating between autoregressive and diffu- sion language models,” inProc. ICLR, 2025

    2025

  17. [17]

    Codec does matter: Exploring the semantic shortcoming of codec for audio language model,

    Z. Ye, P. Ai, J. Sunet al., “Codec does matter: Exploring the semantic shortcoming of codec for audio language model,”arXiv preprint arXiv:2408.17175, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Seed-TTS: A family of high-quality versatile speech generation models,

    P. Anastassiou, J. Cheng, D. Lenget al., “Seed-TTS: A family of high-quality versatile speech generation models,”arXiv preprint arXiv:2406.02430, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    DiTAR: Diffusion transformer autoregressive modeling for speech generation,

    D. Jia, Z. Chen, Y . Wanget al., “DiTAR: Diffusion transformer autoregressive modeling for speech generation,”arXiv preprint arXiv:2502.03930, 2025

    arXiv 2025

  20. [20]

    CosyV oice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Du, Q. Chen, S. Shiet al., “CosyV oice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    SoundStorm: Efficient parallel audio gen- eration,

    Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, “SoundStorm: Efficient parallel audio gen- eration,”arXiv preprint arXiv:2305.09636, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    MegaTTS 3: Sparse alignment enhanced latent diffusion transformer for zero-shot speech syn- thesis,

    Z. Jiang, Y . Ren, R. Liet al., “MegaTTS 3: Sparse alignment enhanced latent diffusion transformer for zero-shot speech syn- thesis,”arXiv preprint arXiv:2502.18924, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”arXiv preprint arXiv:2210.13438, 2022

    Pith/arXiv arXiv 2022

  24. [24]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Huiet al., “Qwen2 technical report,”arXiv preprint arXiv:2407.10671, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    RoFormer: Enhanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “RoFormer: Enhanced transformer with rotary position embedding,”Neuro- computing, vol. 568, p. 127063, 2024

    2024

  26. [26]

    Emilia: An extensive, multi- lingual, and diverse speech dataset for large-scale speech genera- tion,

    H. He, Z. Shang, C. Wanget al., “Emilia: An extensive, multi- lingual, and diverse speech dataset for large-scale speech genera- tion,”arXiv preprint arXiv:2407.05361, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Decoupled weight decay regulariza- tion,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” inProc. ICLR, 2019

    2019

  28. [28]

    Robust speech recognition via large-scale weak su- pervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” inProc. ICML, 2023

    2023

  29. [29]

    WavLM: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “WavLM: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  30. [30]

    CodecMOS-Accent: A MOS benchmark of resynthesized and TTS speech from neural codecs across English accents,

    W.-C. Huang, N. Sanders, and E. Cooper, “CodecMOS-Accent: A MOS benchmark of resynthesized and TTS speech from neural codecs across English accents,”arXiv preprint arXiv:2603.14328, 2026

    arXiv 2026

  31. [31]

    IndexTTS2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech,

    S. Zhou, Y . Zhou, Y . He, X. Zhou, J. Wang, W. Deng, and J. Shu, “IndexTTS2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech,” arXiv preprint arXiv:2506.21619, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Qwen2.5-Omni technical report,

    J. Xu, Z. Guo, J. Heet al., “Qwen2.5-Omni technical report,” arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    MaskGCT: Zero-shot text-to- speech with masked generative codec transformer,

    Y . Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “MaskGCT: Zero-shot text-to- speech with masked generative codec transformer,”arXiv preprint arXiv:2409.00750, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    CosyV oice 3: Towards in-the- wild speech generation via scaling-up and post-training,

    Z. Du, C. Gao, Y . Wanget al., “CosyV oice 3: Towards in-the- wild speech generation via scaling-up and post-training,”arXiv preprint arXiv:2505.17589, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Fish-Speech: Leveraging large language models for advanced multilingual text-to-speech synthesis,

    S. Liao, Y . Wang, T. Li, Y . Cheng, R. Zhang, R. Zhou, and Y . Xing, “Fish-Speech: Leveraging large language models for advanced multilingual text-to-speech synthesis,”arXiv preprint arXiv:2411.01156, 2024

    Pith/arXiv arXiv 2024