Pith. sign in

REVIEW 3 major objections 8 minor 1 cited by

DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Duration-predictor RL cuts WER from 3.75 to 1.75 in zero-shot TTS

desk verdict A genuine extension—RL on the duration predictor—with good internal controls, but the headline gains are compromised by hyperparameter selection on the evaluation set itself. read the letter →

arxiv 2507.14988 v1 pith:RO4IVPUN submitted 2025-07-20 eess.AS

Yinghao Aaron Li , Xilin Jiang , Fei Tao , Cheng Niu , Kaifeng Xu , Juntong Song , Nima Mesgarani This is my paper
classification eess.AS
keywords text-to-speechdurationpredictionreinforcementlearningGRPOspeakersimilarityworderrorrateflowmatchingmetricoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that in diffusion-based zero-shot text-to-speech, the component that decides how long the speech will be is a major quality bottleneck and can be optimized directly. It treats the duration predictor as a stochastic policy and trains it with group-relative policy optimization, rewarding generated speech for low CTC word error and high speaker-embedding similarity. The result is a four-step student model that reaches a WER of 1.752 and SIM of 0.698 on the English test set, beating both the flow-matching teacher (1.947, 0.662) and the prior system without RL (3.750, 0.672) while running about five times faster than the teacher. The paper also introduces teacher-guided sampling, which uses the teacher for early denoising steps and the student for later ones, recovering most of the teacher's prosodic diversity that distillation had removed. If these claims hold, targeted RL on a single non-differentiable component is enough to close most of the gap to a best-of-8 oracle (WER 1.723, SIM 0.724) without end-to-end gradients.

What carries the argument

The engine of the method is a GRPO loop around a transformer duration policy $P_\phi(L_t|x,p_t)$ that outputs a distribution over 300 remaining-length classes of 100 ms each. For each input, sixteen durations are sampled with Gumbel-softmax temperature 0.7, speech is generated from each by the four-step student, and the reward $r_k = \log p(x|C(y_k)) + \lambda_{\mathrm{SIM}} \cdot \cos(e_p, e_{y_k})$ is normalized into group advantages; a clipped importance-ratio loss with KL regularization toward a frozen reference policy updates the predictor. This machinery matters because duration prediction has no differentiable path to the generator, so the RL loop is what connects duration choices to measurable WER and speaker similarity. The second piece of machinery is teacher-guided sampling: the teacher denoises from noise down to a switching time $t_{\mathrm{switch}}$, then the student completes the remaining steps, which restores early-step prosodic diversity that distillation compressed.

What would settle it

Run the identical GRPO duration-predictor training on a held-out language or recording domain not represented in the training data and report WER and SIM before and after RL; if the gains vanish, the reward proxy, not duration optimization, is doing the work. A second check is to extend training past 2K steps; if performance does not degrade, the early-stopping story shown in Figure 4 is not reproducible.

Watch

Extended reading notes

Core claim

The paper's central claim is that the duration predictor, not just the speech generator, can be aligned to perceptual metrics via reinforcement learning. The authors model remaining-length prediction as a policy and apply GRPO with groups of sixteen sampled durations, each rendered by the four-step distilled student and scored by the sum of a CTC ASR log-likelihood and a speaker-embedding cosine similarity term. This procedure lowers English WER from 3.750 to 1.752 and raises SIM from 0.672 to 0.698 compared with the same student before RL, and it beats the flow-matching teacher baseline (WER 1.947, SIM 0.662) while approaching the best-of-8 oracle (WER 1.723, SIM 0.724). The same pattern appears in Chinese CER and SIM. The paper further claims that teacher-guided sampling restores pitch variation from 30.3% below the teacher to 89.1% of the teacher's diversity while keeping a 1.8x speed advantage over full teacher inference.

Load-bearing premise

The load-bearing premise is that the CTC ASR likelihood used as the intelligibility reward is a faithful, non-gameable proxy for real intelligibility, and that the reported gains survive the early stopping at 1.5K steps before the reward overfitting the paper itself shows around 2K steps.

Editorial extensions

If this is right

  • If correct, non-autoregressive TTS systems can improve intelligibility and speaker similarity without backpropagating through the generator, because the RL signal comes from cheap four-step samples.
  • A 0.3B-parameter model with RL-optimized duration prediction can beat larger zero-shot baselines in WER and CER while running about five times faster than its own teacher, so scaling model size is not the only route to quality.
  • The near-oracle result (WER 1.752 versus best-of-8 1.723) implies duration prediction is close to saturated for this pipeline, leaving generator-side diversity and reward robustness as the next bottlenecks.
  • The teacher-guided variant shows that diversity and metric quality are not in conflict: mixing teacher early steps with student later steps recovers pitch variation while keeping the WER and SIM gains.
  • The same targeted-RL recipe could be applied to other generative components that are hard to differentiate, such as the teacher model itself, as the paper's conclusion suggests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reward's CTC term is trained on the Emilia corpus, so the reported gains likely reflect the proxy's own distribution; on unseen accents or recording conditions the WER advantage may shrink or reverse.
  • Editorial inference: Figure 4's sharp degradation after roughly 2K steps suggests the 1.5K checkpoint is an early-stopped optimum, so a production system would need a validation-based stopping rule rather than a fixed step count.
  • Editorial inference: because rewards are computed per full utterance, the method inherits any bias in the ASR and speaker-verification models; an alternative reward that penalizes duration outliers or uses reference-free intelligibility could trade some WER gain for robustness.
  • Editorial inference: the 100 ms duration bins impose a quantization floor on prosody; finer bins or continuous regression might unlock the remaining gap to the best-of-8 oracle.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper introduces DMOSpeech 2, an extension of the DMOSpeech framework that applies group relative policy optimization (GRPO) to the duration predictor of a flow-matching zero-shot TTS system, using speaker similarity and CTC-based word error rate as rewards. It also proposes teacher-guided sampling, a hybrid inference procedure that starts denoising with the teacher model and switches to the distilled student for the final steps, aiming to restore prosodic diversity lost during distillation. Experiments on Seed-TTS-en and Seed-TTS-zh compare the proposed system with the F5-TTS teacher, the student without duration-predictor RL, and several public baselines, reporting improved WER/CER and SIM at low real-time factors, plus CMOS listening test results.

Significance. If the reported gains hold under unbiased evaluation, the paper makes a solid contribution: it demonstrates that a previously untrained component of a non-autoregressive TTS pipeline (the duration predictor) can be optimized with a lightweight RL procedure, and it provides a simple sampling strategy that partially restores diversity in distilled diffusion models. The paper includes useful internal controls—a no-RL ablation, ground-truth durations, a best-of-8 oracle, and a teacher baseline—which strengthen the causal interpretation of the duration-predictor optimization. The promise of public code and models is commendable. However, the evaluation protocol contains a selection-bias issue that directly affects the headline quantitative claims, and the objective metrics lack uncertainty estimates, so the current version overstates the reliability of the improvements.

major comments (3)
  1. [Appendix A.2.1 and Table 1] Hyperparameters and the early-stopping point (1.5K steps, group size 16) were selected using 'a small subset of the Seed-TTS-en evaluation set' (Appendix A.2.1, Figure 4), and the final results in Table 1 are then reported on the full Seed-TTS-en set, which includes that selection subset. The RL-vs-no-RL comparison (WER 3.750 vs 1.752; SIM 0.672 vs 0.698) is therefore optimistically biased, because the no-RL baseline did not receive the same selection process and the checkpoint was chosen on the evaluation distribution. This is especially consequential because Figure 4 shows sharp degradation after roughly 2K steps, making the result hinge on the selected early-stopped checkpoint. Please re-evaluate on a genuinely held-out test set that is disjoint from any hyperparameter or checkpoint selection (for example, split Seed-TTS-en into a validation subset and a disjoint test subset, or add an independent cross-sentence test set such as LibriTTS or VCTK), and report the no-RL baseline on the same held-out set. Also report bootstrap confidence intervals or significance tests for the objective metrics in Table 1, particularly for the small WER gap between DMOSpeech 2 (1.752) and Teacher-Guided (1.738).
  2. [Eq. (4), Section 3.2.2] The reward in Eq. (4) is computed with a CTC-based ASR model trained on Emilia, whereas the evaluation in Table 1 uses Whisper-large-v3 and Paraformer-zh on waveforms. The paper claims improved intelligibility but does not show that optimizing the CTC proxy transfers to the evaluation ASR models; the paper's own Figure 4 indicates that the proxy overfits after about 2K steps, so the final result depends on early stopping. Please provide evidence that the selected checkpoint's improvements are not an artifact of the CTC reward, for example by reporting reward and evaluation-metric trajectories on a validation set that is disjoint from the evaluation set, and by verifying the gains with a held-out ASR model that was not used in the reward.
  3. [Table 3, Section A.1] Table 3 reports that Ground Truth Audio has a WER of 2.143 on Seed-TTS-en, which is higher than DMOSpeech 2's WER of 1.752 and also higher than Ground Truth Duration's WER of 1.821. This is counterintuitive: the reference audio used for evaluation should have low ASR error, and a higher WER for ground truth than for synthesized speech suggests either a transcription or evaluation artifact. Please clarify why the ground-truth recordings have such a high WER in this evaluation protocol, and discuss how this affects the interpretation of the WER differences among systems.
minor comments (8)
  1. [Section 4.1] Typo: 'objecetive' should be 'objective' in the Baselines paragraph.
  2. [Appendix B.3] Typo: 'Kullback-Liebler' should be 'Kullback-Leibler'.
  3. [Eq. (4)] The notation log p(x|C(y_k)) is ambiguous because C(y_k) is the ASR model's output sequence; please define C(·) precisely, e.g., as a function returning the log-likelihood of transcript x under the CTC model.
  4. [Figure 4] The figure would be clearer with a shared legend describing the three group sizes; the current legends inside each panel are redundant, and the vertical dashed line at 1.5K should be explicitly labeled in the caption.
  5. [Section 4.1 / Appendix A.2.2] The paper states that the student model uses 'half the batch size of the teacher model training' but does not give the actual batch size; please state the number of audio frames per batch. Also, Appendix A.2.2 refers to a 'validation set' for calibrating λ_SIM, but it does not specify whether that set is disjoint from the evaluation set; please clarify.
  6. [Abstract and Section 3.3.2] The abstract says 'reducing sampling steps by half without quality degradation,' but the comparison in Table 1 is between 32 teacher steps and 16 teacher-guided steps, while the student-only model uses 4 steps. Please clarify that the 'half' refers to the teacher-guided variant relative to the teacher, not to the student-only model.
  7. [Appendix C] The text first says 'a total of 320 samples in both English and Chinese' and later says '320 English and 320 Chinese samples'; please disambiguate the total number of evaluations.
  8. [Algorithm 1] The use of the queue Q and the update timing of π_old is somewhat unclear; please specify how many groups are accumulated before a policy update and when the old policy snapshot is refreshed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the RL duration-policy claim rests on a genuine optimization with reward proxies distinct from the reported evaluation metrics; no equation-level reduction to inputs exists.

full rationale

The derivation chain is self-contained. The duration policy is defined in Eq. (1) as a transformer length predictor, and the GRPO objective in Eqs. (5)-(8) uses standard importance ratios, clipping, and KL regularization against a frozen supervised reference; none of these definitions presuppose the headline WER or SIM numbers. The reward in Eq. (4) is a CTC log-likelihood plus a cosine speaker similarity from models trained on Emilia, whereas the reported evaluations use Whisper-large-v3, Paraformer-zh, and a WavLM-based speaker verification model, so the reported metrics are not the optimized objective by construction. The best-of-8 row in Table 3 is explicitly labeled an oracle using privileged information, not a predicted result. Teacher-guided sampling is supported by a separate CVf0 diversity measurement and F0/UTMOS distribution analyses, independent of the RL reward. The paper's self-citations to DMOSpeech [28] supply the base generator and direct metric optimization framework, but the new contribution—RL on the duration predictor—is evaluated against a no-RL baseline, the teacher, and external SOTA systems, so no load-bearing claim reduces to a self-citation. The only caveat worth flagging is Appendix A.2.1, where the 1.5K-step checkpoint and group size were selected on 'a small subset of the Seed-TTS-en evaluation set,' making Table 1's Seed-TTS-en numbers partly in-sample for hyperparameter choice; this is a test-set contamination and generalization risk, not a constructional circularity, and it does not make any equation equal to its input. Therefore no circular step is identified.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The central empirical result relies on a small set of hand-set hyperparameters (lambda_SIM, temperature, group size, early stopping, switching time) and on domain assumptions about reward validity and diffusion noise-level semantics. No new particles, forces, or entities are introduced.

free parameters (7)
  • lambda_SIM = 3
    Calibrated in Appendix A.2.2 to equalize the magnitudes of the CTC log-likelihood and speaker-similarity reward terms in Eq. (4).
  • Gumbel-softmax temperature tau = 0.7
    Hand-chosen in Section 3.2.2 to encourage exploration; no systematic sweep is reported.
  • GRPO KL weight beta = 0.04
    Standard GRPO value, stated in Eq. (7); not tuned specifically for this task.
  • reward diversity threshold = 0.01
    Batches with max-min reward below 0.01 are skipped in Algorithm 1; hand-set threshold.
  • RL training steps = 1500
    Selected based on validation performance (Figure 4) on a subset of the Seed-TTS-en evaluation set; degradation occurs after about 2K steps.
  • group size K = 16
    Selected from {8, 16, 32} using the validation subset (Appendix A.2.1).
  • teacher-guided t_switch, teacher steps, student steps = 0.25, 14, 2
    Chosen to balance diversity and speed; authors state these offer a flexible trade-off (Section 3.3.2).
assumptions (3)
  • domain assumption The pre-trained CTC ASR model's log-likelihood log p(x|C(y_k)) is a valid reward for intelligibility.
    Used in Eq. (4); the paper relies on this proxy to optimize WER without a direct gradient to the generator, and acknowledges the risk of overfitting.
  • domain assumption Early denoising steps establish prosodic structure while later steps refine acoustic detail.
    Justifies teacher-guided sampling in Section 3.3.1 and Figure 2; asserted from empirical observation, not proven.
  • standard math Flow-matching, distribution matching distillation, and GRPO update equations are correct as borrowed from prior work.
    Invoked throughout Section 3 and Appendix B; no formalization or derivation is provided in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis." pith.science (2026). https://pith.science/paper/RO4IVPUN

@misc{pith2026250714988,
  author       = {Pith},
  title        = {Pith review of: DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RO4IVPUN}},
  note         = {Machine review of arXiv:2507.14988}
}
read the original abstract

Diffusion-based text-to-speech (TTS) systems have made remarkable progress in zero-shot speech synthesis, yet optimizing all components for perceptual metrics remains challenging. Prior work with DMOSpeech demonstrated direct metric optimization for speech generation components, but duration prediction remained unoptimized. This paper presents DMOSpeech 2, which extends metric optimization to the duration predictor through a reinforcement learning approach. The proposed system implements a novel duration policy framework using group relative preference optimization (GRPO) with speaker similarity and word error rate as reward signals. By optimizing this previously unoptimized component, DMOSpeech 2 creates a more complete metric-optimized synthesis pipeline. Additionally, this paper introduces teacher-guided sampling, a hybrid approach leveraging a teacher model for initial denoising steps before transitioning to the student model, significantly improving output diversity while maintaining efficiency. Comprehensive evaluations demonstrate superior performance across all metrics compared to previous systems, while reducing sampling steps by half without quality degradation. These advances represent a significant step toward speech synthesis systems with metric optimization across multiple components. The audio samples, code and pre-trained models are available at https://dmospeech2.github.io/.

Figures

Figures reproduced from arXiv: 2507.14988 by the authors.

Figure 1
Figure 1. Overview of the DMOSpeech 2 framework. (a) Left: The original DMOSpeech architecture, where the duration predictor (Pϕ) is trained self-supervisedly and separate from the TTS component, creating a disconnection that prevents end-to-end optimization. (b) Right: Our proposed DMOSpeech 2 framework, which employs Group Relative Policy Optimization (GRPO) to train the duration predictor with reinforcement learning (Algor… view at source ↗
Figure 2
Figure 2. Illustration of teacher-guided sampling (Algorithm 2). The process begins with noise and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of diversity across sampling methods. (a) F0 value distributions (shown as [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance dynamics during RL training of the duration predictor with various group [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Screenshot of the comparative subjective evaluation interface. The interface presents three [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Landscape of Agentic Reinforcement Learning for LLMs: A Survey

    cs.AI 2025-09 accept novelty 6.0 of 10

    Survey that defines agentic RL for LLMs via POMDPs, introduces a taxonomy of planning/tool-use/memory/reasoning capabilities and domains, and compiles open environments from over 500 papers.

Reference graph

Works this paper leans on

58 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Naturalspeech: End-to-end text-to-speech synthesis with human-level quality

    Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    work page 2024

  2. [2]

    Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models

    Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. Advances in Neural Information Processing Systems , 36, 2024

    work page 2024

  3. [3]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100, 2024

    arXiv 2024

  4. [4]

    Enhancing zero-shot text-to-speech synthesis with human feedback

    Chen Chen, Yuchen Hu, Wen Wu, Helin Wang, Eng Siong Chng, and Chao Zhang. Enhancing zero-shot text-to-speech synthesis with human feedback. arXiv preprint arXiv:2406.00654 , 2024

    arXiv 2024

  5. [5]

    Speechalign: Aligning speech generation to human preferences

    Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechalign: Aligning speech generation to human preferences. arXiv preprint arXiv:2404.05600, 2024

    arXiv 2024

  6. [6]

    Emo-dpo: Con- trollable emotional speech synthesis through direct preference optimization

    Xiaoxue Gao, Chen Zhang, Yiming Chen, Huayun Zhang, and Nancy F Chen. Emo-dpo: Con- trollable emotional speech synthesis through direct preference optimization. In ICASSP 2025- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2025

    work page 2025

  7. [7]

    Preference alignment improves language model-based tts

    Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, and Dong Yu. Preference alignment improves language model-based tts. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2025. 12

    work page 2025

  8. [8]

    Koel-tts: Enhancing llm based speech generation with preference alignment and classifier free guidance

    Shehzeen Hussain, Paarth Neekhara, Xuesong Yang, Edresson Casanova, Subhankar Ghosh, Mikyas T Desta, Roy Fejgin, Rafael Valle, and Jason Li. Koel-tts: Enhancing llm based speech generation with preference alignment and classifier free guidance. arXiv preprint arXiv:2502.05236, 2025

    arXiv 2025

Show all 58 references
  1. [9]

    Evaluation of best-of-n sampling strategies for language model alignment

    Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe, Mitsuki Sakamoto, and Eiji Uchibe. Evaluation of best-of-n sampling strategies for language model alignment. arXiv preprint arXiv:2502.12668, 2025

    2025 arXiv

  2. [10]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  3. [11]

    V oice- craft: Zero-shot speech editing and text-to-speech in the wild

    Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath. V oice- craft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint arXiv:2403.16973, 2024

    2024 arXiv

  4. [12]

    Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370, 2024

    2024 arXiv

  5. [13]

    Maskgct: Zero-shot text-to-speech with masked generative codec transformer

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskgct: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750, 2024

    2024 arXiv

  6. [14]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  7. [15]

    Vall-t: Decoder-only generative transducer for robust and decoding- controllable text-to-speech

    Chenpeng Du, Yiwei Guo, Hankun Wang, Yifan Yang, Zhikang Niu, Shuai Wang, Hui Zhang, Xie Chen, and Kai Yu. Vall-t: Decoder-only generative transducer for robust and decoding- controllable text-to-speech. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech an...

    2025

  8. [16]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117, 2024

    2024 arXiv

  9. [17]

    Autoregressive speech synthesis with next-distribution prediction

    Xinfa Zhu, Wenjie Tian, and Lei Xie. Autoregressive speech synthesis with next-distribution prediction. arXiv preprint arXiv:2412.16846, 2024

    2024

  10. [18]

    Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens

    Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710, 2025

    2025 arXiv

  11. [19]

    Touchtts: An embarrassingly simple tts framework that everyone can touch

    Xingchen Song, Mengtao Xing, Changwei Ma, Shengqiang Li, Di Wu, Binbin Zhang, Fuping Pan, Dinghao Zhou, Yuekai Zhang, Shun Lei, et al. Touchtts: An embarrassingly simple tts framework that everyone can touch. arXiv preprint arXiv:2412.08237, 2024

    2024 arXiv

  12. [20]

    Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis

    Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi DAI, et al. Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis. arXiv preprint arXiv:2502.04128, 2025

    2025 arXiv

  13. [21]

    V oicebox: Text-guided multilin- gual universal speech generation at scale

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. V oicebox: Text-guided multilin- gual universal speech generation at scale. Advances in neural information processing systems , 36, 2024

    2024

  14. [22]

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

    Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116, 2023. 13

    2023 arXiv

  15. [23]

    Styletts-zs: Efficient high- quality zero-shot text-to-speech synthesis with distilled time-varying style diffusion

    Yinghao Aaron Li, Xilin Jiang, Cong Han, and Nima Mesgarani. Styletts-zs: Efficient high- quality zero-shot text-to-speech synthesis with distilled time-varying style diffusion. arXiv preprint arXiv:2409.10058, 2024

    2024 arXiv

  16. [24]

    E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts

    Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. arXiv preprint arXiv:2406.18009, 2024

    2024 arXiv

  17. [25]

    Simplespeech 2: Towards simple and efficient text-to-speech with flow-based scalar latent transformer diffusion models

    Dongchao Yang, Rongjie Huang, Yuanyuan Wang, Haohan Guo, Dading Chong, Songxiang Liu, Xixin Wu, and Helen Meng. Simplespeech 2: Towards simple and efficient text-to-speech with flow-based scalar latent transformer diffusion models. arXiv preprint arXiv:2408.13893, 2024

    2024 arXiv

  18. [26]

    Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer

    Keon Lee, Dong Won Kim, Jaehyeon Kim, and Jaewoong Cho. Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer. arXiv preprint arXiv:2406.11427, 2024

    2024 arXiv

  19. [27]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024

    2024 arXiv

  20. [28]

    Dmospeech: Direct metric optimization via distilled diffusion model in zero-shot speech synthesis

    Yingahao Aaron Li, Rithesh Kumar, and Zeyu Jin. Dmospeech: Direct metric optimization via distilled diffusion model in zero-shot speech synthesis. arXiv preprint arXiv:2410.11097, 2024

    2024 arXiv

  21. [29]

    One-step diffusion with distribution matching distillation

    Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6613–6623, 2024

    2024

  22. [30]

    Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone

    Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning , pages 2709–2720. PMLR, 2022

    2022

  23. [31]

    Sc-glowtts: An efficient zero-shot multi-speaker text-to-speech model

    Edresson Casanova, Christopher Shulby, Eren Gölge, Nicolas Michael Müller, Frederico Santos De Oliveira, Arnaldo Candido Junior, Anderson da Silva Soares, Sandra Maria Aluisio, and Moacir Antonelli Ponti. Sc-glowtts: An efficient zero-shot multi-speaker text-to-speech model. a...

    2021 arXiv

  24. [32]

    Adaspeech 4: Adaptive text to speech in zero-shot scenarios

    Yihan Wu, Xu Tan, Bohan Li, Lei He, Sheng Zhao, Ruihua Song, Tao Qin, and Tie-Yan Liu. Adaspeech 4: Adaptive text to speech in zero-shot scenarios. arXiv preprint arXiv:2204.00436, 2022

    2022 arXiv

  25. [33]

    Hierspeech: Bridging the gap between text and speech by hierarchical variational inference using self-supervised representations for speech synthesis

    Sang-Hoon Lee, Seung-Bin Kim, Ji-Hyun Lee, Eunwoo Song, Min-Jae Hwang, and Seong- Whan Lee. Hierspeech: Bridging the gap between text and speech by hierarchical variational inference using self-supervised representations for speech synthesis. Advances in Neural Information Pro...

    2022

  26. [34]

    Meta-stylespeech: Multi- speaker adaptive text-to-speech generation

    Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. Meta-stylespeech: Multi- speaker adaptive text-to-speech generation. In International Conference on Machine Learning , pages 7748–7759. PMLR, 2021

    2021

  27. [35]

    Styletts: A style-based generative model for natural and diverse text-to-speech synthesis

    Yinghao Aaron Li, Cong Han, and Nima Mesgarani. Styletts: A style-based generative model for natural and diverse text-to-speech synthesis. arXiv preprint arXiv:2205.15439, 2022

    2022 arXiv

  28. [36]

    Nansy++: Unified voice synthesis with neural analysis and synthesis

    Hyeong-Seok Choi, Jinhyeok Yang, Juheon Lee, and Hyeongju Kim. Nansy++: Unified voice synthesis with neural analysis and synthesis. arXiv preprint arXiv:2211.09407, 2022

    2022 arXiv

  29. [37]

    Reinforcement learning for fine-tuning text-to-speech diffusion models

    Jingyi Chen, Ju-Seung Byun, Micha Elsner, and Andrew Perrault. Reinforcement learning for fine-tuning text-to-speech diffusion models. arXiv preprint arXiv:2405.14632, 2024

    2024 arXiv

  30. [38]

    F5r-tts: Improving flow matching based text-to-speech with group relative policy optimization

    Xiaohui Sun, Ruitong Xiao, Jianye Mo, Bowen Wu, Qun Yu, and Baoxun Wang. F5r-tts: Improving flow matching based text-to-speech with group relative policy optimization. arXiv preprint arXiv:2504.02407, 2025. 14

    2025 arXiv

  31. [39]

    V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis

    Hubert Siuzdak. V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.00814, 2023

    2023 arXiv

  32. [40]

    Improved distribution matching distillation for fast image synthesis

    Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405.14867, 2024

    2024 arXiv

  33. [41]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  34. [42]

    Utmos: Utokyo-sarulab system for voicemos challenge 2022

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02152, 2022

    2022 arXiv

  35. [43]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. 2...

    2024

  36. [44]

    Seed-tts: A family of high-quality versatile speech generation models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li...

    2024 arXiv

  37. [45]

    Common voice: A massively-multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019

    1912 arXiv

  38. [46]

    Didispeech: A large scale mandarin speech corpus

    Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. Didispeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)...

    2021

  39. [47]

    Fixing Weight Decay Regularization in Adam, 2018

    Ilya Loshchilov and Frank Hutter. Fixing Weight Decay Regularization in Adam, 2018

    2018

  40. [48]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  41. [49]

    Funasr: A fundamental end-to-end speech recognition toolkit

    Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al. Funasr: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305.11013, 2023

    2023 arXiv

  42. [50]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing , 16(6...

    2022

  43. [51]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840–6851, 2020

    2020

  44. [52]

    Murphy, and Tim Salimans

    Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin P. Murphy, and Tim Salimans. Diffusion meets flow matching: Two sides of the same coin. 2024

    2024

  45. [53]

    Autoregressive diffusion transformer for text-to-speech synthesis

    Zhijun Liu, Shuai Wang, Sho Inoue, Qibing Bai, and Haizhou Li. Autoregressive diffusion transformer for text-to-speech synthesis. arXiv preprint arXiv:2406.05551, 2024

    2024 arXiv

  46. [54]

    Wespeaker: A research and production oriented speaker embedding learning toolkit

    Hongji Wang, Chengdong Liang, Shuai Wang, Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. Wespeaker: A research and production oriented speaker embedding learning toolkit. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal P...

    2023

  47. [55]

    Audio 1" and

    SimAMResNet34 model on the Emilia dataset following [28]. 20 C Subjective Evaluation In addition to the absolute rating evaluation described previously, we conducted comparative mean opinion score (CMOS) tests to directly assess the relative performance of our proposed models ...

  48. [56]

    Listen to all three audio samples

  49. [57]

    Select which of the two synthesized samples sounds more natural (left question)

  50. [58]

    Select which of the two synthesized samples sounds more similar to the reference voice (right question) The DMOSpeech 2 model (with 4 sampling steps) served as the anchor system for all comparisons, with participants unaware of which sample corresponded to which system. The dr...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.