Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Experts

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MoE-TTS keeps a pre-trained textual LLM frozen and adds speech-expert weights to transformer layers, reporting that the model follows out-of-domain descriptions more accurately than leading commercial TTS systems.

desk verdict Solid MoE-for-TTS application with an unproven central claim: no ablation against full fine-tuning, and the OOD evaluation is too small and self-made to support 'significantly outperforms'. read the letter →

arxiv 2508.11326 v1 pith:5NDYR6CP submitted 2025-08-15 eess.AS cs.SD

Heyang Xue , Xuchen Song , Yu Tang , Jianyu Chen , Yanru Chen , Yang Li , Yahui Zhou This is my paper
classification eess.AScs.SD
keywords description-basedTTSmixture-of-expertsout-of-domaingeneralizationtext-to-speechlargelanguagemodelmodality-specificroutingspeechsynthesiscatastrophicforgetting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Description-based text-to-speech systems struggle when users phrase voice and style requirements in ways never seen in training data. MoE-TTS attacks this by building the TTS model around a frozen pre-trained textual LLM (Qwen3-4B) and inserting a separate set of speech-expert weights into the transformer layers; during training only those speech weights change, so the LLM keeps its general language understanding. The paper reports that on a 40-sample out-of-domain test set built from metaphors, analogies, implications, and paraphrases, MoE-TTS scores higher than ElevenLabs and MiniMax on description-alignment dimensions, despite training only on open datasets. If true, this suggests frozen textual LLMs plus modality-specific experts are a practical route to robust description-following speech.

What carries the argument

Modality-based mixture-of-experts applied to every major transformer component: each layer's attention, feed-forward network, and layer norm have a text-expert version (the original frozen LLM weights) and a speech-expert version (newly initialized, updated weights), selected by whether the current token is text or speech. This token-modality routing preserves the LLM's pre-trained language knowledge for description understanding while giving speech its own parameter path, which is the mechanism that is supposed to prevent catastrophic forgetting and enable generalization to unseen phrasings.

What would settle it

Collect a few hundred naturally written user descriptions from an independent source, synthesize with MoE-TTS, ElevenLabs, and MiniMax, and run the same SEA/OA rating protocol. If MoE-TTS does not lead on this larger set, its out-of-domain advantage is specific to the 40 constructed prompts rather than a general text-understanding improvement.

Watch

Extended reading notes

Core claim

MoE-TTS claims that a description-based TTS model can gain out-of-domain text understanding without paying the usual catastrophic-forgetting cost: initialize the whole system from a textual LLM, freeze its weights, and route text tokens through the original layers while routing speech tokens through newly added speech-expert copies of attention, feed-forward, and layer-norm. The speech experts are initialized from the text experts, so the model starts with the LLM's representational structure and only adapts the speech path. In the paper's evaluations, the resulting 8B-parameter model outperforms closed-source commercial TTS systems on stylistic and overall alignment with descriptions while

Load-bearing premise

The claim rests on treating 40 hand-authored out-of-domain descriptions as representative of real user-generated descriptions and on treating 21 professional raters' alignment scores as a measure of text understanding; if those prompts are unrepresentative, the reported out-of-domain advantage may not appear with naturally written descriptions.

Editorial extensions

If this is right

  • The largest claimed consequence is that open-source-trained models can surpass closed commercial systems on description alignment, at least on the paper's test sets.
  • The frozen-LLM design means language understanding does not need to be re-learned for TTS, so additional speech data can be added cheaply without degrading text competence.
  • Because only speech-expert weights receive gradients, training requires far fewer updated parameters than full fine-tuning of an 8B model.
  • The out-of-domain degradation observed in all tested systems indicates that description-based TTS has a measurable generalization gap that MoE-TTS narrows but does not close.
  • The architecture is agnostic to the discrete speech tokenizer, so it can adopt better semantic or acoustic tokens as those improve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 40-prompt evaluation generalizes, the same frozen-backbone-plus-modality-experts recipe should transfer to multilingual descriptions simply by swapping the base LLM and speech tokenizer; the paper explicitly leaves multilingual extension untested.
  • The design invites a controlled ablation the paper does not run: varying the size or number of speech experts should trade off speech quality against retained text understanding, and would identify which components carry the out-of-domain gain.
  • A stronger test would be an external, naturally collected set of user descriptions; the current evidence rests on 40 hand-authored prompts, so the out-of-domain claim is only as strong as that set's representativeness.
  • The same parameter-decoupling idea could be applied to other generation tasks where a frozen pretrained model must learn a new output modality with limited paired data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MoE-TTS, a description-based TTS system built on a frozen Qwen3-4B textual LLM. Speech-modality expert copies of attention, feed-forward, and layer-norm parameters are inserted and trained, with token-modality routing (text tokens use the frozen original parameters, speech tokens use the newly initialized speech experts). Speech tokens come from the CosyVoice2 tokenizer, and waveforms are reconstructed with a Stable Audio diffusion/VQGAN stack. Training uses only open datasets (VoxBox, Emilia-YODAS, TextrolSpeech, SpeechCraft, Parler-TTS, LibriTTS-P, ParaspeechCaps). The evaluation is a subjective MOS study with 21 raters on 20 in-domain and 40 self-authored out-of-domain descriptions, comparing against ElevenLabs and MiniMax. The paper claims MoE-TTS significantly outperforms both commercial systems on description-alignment dimensions (SEA and OA), especially for out-of-domain descriptions.

Significance. If the central claim were fully supported, the paper would report a practically useful result: a frozen textual LLM augmented with speech experts can follow out-of-domain description prompts better than commercial systems while training only on open data. The architecture is clearly described, the training recipe is reproducible in principle, and the choice to freeze the text LLM is well motivated by the catastrophic-forgetting literature. However, the evidence currently supplied does not isolate the effect of the Mixture-of-Experts mechanism, and the evaluation rests on a small, self-authored, unreleased OOD test set without inferential statistics. The significance of the contribution therefore depends on additional experiments that the manuscript does not yet contain.

major comments (3)
  1. [Section 3.2 and Section 4.2] The title and abstract attribute the OOD improvement to the MoE mechanism, but Section 4.2 contains no ablation that isolates this component. The only comparisons are the complete MoE-TTS system versus two commercial APIs. There is no full-fine-tuned version of the same Qwen3-4B backbone, no frozen-LLM LoRA/adapter baseline, and no open LLM-based TTS baseline such as CosyVoice2, Parler-TTS, or Llasa. Because Eq. (4) routes text tokens through the frozen original parameters and speech tokens through newly initialized copies of those parameters, the improved SEA/OA scores in Table 2 could be explained by the pretrained Qwen3 model itself, by the added parameter capacity, or by the CosyVoice2 tokenizer plus Stable Audio decoder, rather than by modality-based routing. A same-backbone ablation is necessary for the paper's central attribution.
  2. [Section 4.1] The out-of-domain evaluation is built on 40 self-authored descriptions, with no external OOD benchmark, no release of the prompt set, and no quantitative evidence that these descriptions are actually outside the training distribution of the open datasets. The examples in Table 1 are deliberately metaphorical and adversarial (e.g., 'chipmunk-on-jet-fuel energy'), not sampled from natural user logs. The claim that MoE-TTS generalizes to 'real-world user-generated' OOD prompts is therefore not supported by the current evidence. The authors should release the full OOD and in-domain test sets, provide a distributional overlap analysis, and ideally validate on an independently constructed OOD set.
  3. [Section 4.2, Table 2] The text states that MoE-TTS 'significantly outperforms' both commercial systems on SEA and OA, but no significance test is reported and the table does not define whether the ± values are standard errors or confidence intervals. Under the usual interval reading, several key differences are not clearly separated: in-domain OA MoE-TTS is 3.61±0.132 versus MiniMax 3.46±0.132, and OOD SEA MoE-TTS is 4.02±0.063 versus ElevenLabs 3.89±0.080, which overlap or nearly overlap. With 21 raters and 60 test items, per-item paired significance tests (e.g., Wilcoxon signed-rank), adjusted for multiple comparisons, are needed. Inter-rater agreement should also be reported. Without this, the core empirical claim is not established.
minor comments (4)
  1. [Table 1] There are typographical artifacts in the table headers and example text: 'V oice' and 'V olume' should be 'Voice' and 'Volume'.
  2. [Section 4.2, Figure 2] The caption and accompanying text say that 'all systems exhibited significant accuracy degradation' on OOD descriptions, but the attribute-accuracy results in Figure 2 are not accompanied by confidence intervals or significance tests. The wording should be softened or supported.
  3. [Section 3.2, Eq. (3)] Notation such as MoE_Q, MoE_K, MoE_V, and MoE_O is not explicitly defined in the text; the reader has to infer that these are the modality-conditioned projections described by Eq. (4). A short clarification would improve readability.
  4. [Section 4.1] The paper says MoE-TTS was 'scaled to 8 billion parameters.' Since the Qwen3-4B text parameters are frozen and the speech experts are initialized as copies, it would be useful to state explicitly the number of trainable parameters and how the 8B figure is computed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central OOD claim rests on an internal test set and a missing full-fine-tuning ablation, but no step reduces to its own inputs by construction.

full rationale

The paper's derivation chain is: initialize from a frozen Qwen3-4B LLM, insert speech-modality expert parameters into attention/FFN/LN with modality-based routing, train only those speech experts on open TTS and description-based datasets, then evaluate on hand-authored in-domain and out-of-domain description sets against commercial APIs. At no step is a parameter fitted to the evaluation set and then reported as a prediction: the 40 out-of-domain descriptions are not used during training, and the reported SEA/OA MOS values are measured, not derived from the model's own definitions. The MoE design is adopted from external prior work (Mono-InternVL [18], EVEv2 [19]) with no author overlap, so there is no load-bearing self-citation chain and no imported uniqueness theorem. The claim that freezing the text experts preserves pre-trained text understanding is an architectural hypothesis tested by evaluation, not an algebraic identity. The genuine weaknesses are experimental-attribution concerns rather than circularity: Table 2 lacks a full-fine-tuned LLM baseline on the same backbone and data, so the OOD margins cannot be uniquely attributed to the MoE mechanism, and the self-constructed 40-prompt test set raises external-validity concerns. These are validity gaps, not reductions of the conclusion to its inputs. Accordingly, the paper shows no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The quantitative claims rest on a small, author-constructed subjective test set and on priors about LLMs and MoE behavior. No code, checkpoints, test-set files, or external OOD benchmark are provided, so the only falsifiable handle is the hosted demo page.

free parameters (3)
  • OOD test set composition
    40 hand-authored descriptions generated with metaphors, analogies, implications, and paraphrases; the central OOD claim is measured only on these prompts.
  • In-domain test set composition
    20 author-selected descriptions assumed to stay within the training tag distribution.
  • Training recipe = lr=3e-4, warmup=0.08, one epoch
    Hand-chosen hyperparameters with no sensitivity analysis; they are needed to reproduce the model but are not derived from any principle.
assumptions (5)
  • domain assumption Qwen3-4B contains sufficient pre-trained text understanding for description following.
    The model is initialized from Qwen3 and the text parameters are frozen; no analysis isolates which specific language capabilities drive the reported gains.
  • ad hoc to paper Modality-based routing with frozen text experts prevents catastrophic forgetting and retains text understanding.
    This is the core design claim in Section 3.2, but the paper provides no ablation against full fine-tuning or other parameter-preserving methods.
  • ad hoc to paper The 40 OOD descriptions are representative of real user-generated out-of-domain inputs.
    Assumed in Section 4.1; no external OOD corpus or user study validates this representativeness.
  • domain assumption Subjective MOS from 21 professional evaluators, with 7 per attribute check, reliably measures description alignment.
    Standard practice in TTS evaluation, but no inter-rater agreement or statistical significance testing is reported.
  • domain assumption CosyVoice2 speech tokens plus the EDM and VAEGAN components preserve speaker and style attributes.
    The acoustic pipeline is taken from prior work, not validated inside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Experts." pith.science (2026). https://pith.science/paper/5NDYR6CP

@misc{pith2026250811326,
  author       = {Pith},
  title        = {Pith review of: MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NDYR6CP}},
  note         = {Machine review of arXiv:2508.11326}
}
read the original abstract

Description-based text-to-speech (TTS) models exhibit strong performance on in-domain text descriptions, i.e., those encountered during training. However, in real-world applications, the diverse range of user-generated descriptions inevitably introduces numerous out-of-domain inputs that challenge the text understanding capabilities of these systems. To address this issue, we propose MoE-TTS, a description-based TTS model designed to enhance the understanding of out-of-domain text descriptions. MoE-TTS employs a modality-based mixture-of-experts (MoE) approach to augment a pre-trained textual large language model (LLM) with a set of specialized weights adapted to the speech modality while maintaining the original LLM frozen during training. This approach allows MoE-TTS to effectively leverage the pre-trained knowledge and text understanding abilities of textual LLMs. Our experimental results indicate that: first, even the most advanced closed-source commercial products can be challenged by carefully designed out-of-domain description test sets; second, MoE-TTS achieves superior performance in generating speech that more accurately reflects the descriptions. We encourage readers to listen to the demos at https://welkinyang.github.io/MoE-TTS/.

Figures

Figures reproduced from arXiv: 2508.11326 by the authors.

Figure 1
Figure 1. Overview of MoE-TTS. MoE-TTS is initialized from a pre-trained textual LLM and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. We compare the accuracy of MoE-TTS, ElevenLabs, and Minimax across four funda [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts

    cs.SD 2026-05 unverdicted novelty 7.0 of 10

    PlanAudio introduces a unified autoregressive LLM framework with semantic latent chain-of-thought for generating composite speech and sound audio from free-form text, plus a new benchmark.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL https://elevenlabs.io/

    Elevenlabs. URL https://elevenlabs.io/

    work page

  2. [2]

    URL https://www.minimaxi.com/

    Minimax. URL https://www.minimaxi.com/

    work page

  3. [3]

    Prompttts: Controllable text-to-speech with text descriptions

    Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. Prompttts: Controllable text-to-speech with text descriptions. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023 , pages 1–5. IEEE,

    work page 2023

  4. [5]

    Textrolspeech: A text style control speech corpus with codec language text-to-speech models

    Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao. Textrolspeech: A text style control speech corpus with codec language text-to-speech models. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024 , pages 10301–10305. ...

    arXiv 2024

  5. [6]

    Prompttts 2: Describing and generating voices with text prompt

    Yichong Leng, Zhifang Guo, Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, Lei He, Xiangyang Li, Sheng Zhao, Tao Qin, and Jiang Bian. Prompttts 2: Describing and generating voices with text prompt. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 ....

    work page 2024

  6. [7]

    Libritts-p: A corpus with speaking style and speaker identity prompts for text- to-speech and style captioning

    Masaya Kawamura, Ryuichi Yamamoto, Yuma Shirahata, Takuya Hasumi, and Kentaro Tachibana. Libritts-p: A corpus with speaking style and speaker identity prompts for text- to-speech and style captioning. In Itshak Lapidot and Sharon Gannot, editors, 25th Annual Conference of the International Speech Communication Association, Interspeech 2024, Kos, Greece, S...

    doi:10.21437/interspeech.2024-692 2024

  7. [8]

    Speechcraft: A fine-grained expressive speech dataset with natural language description

    Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, and Zhiyong Wu. Speechcraft: A fine-grained expressive speech dataset with natural language description. In Jianfei Cai, Mohan S. Kankanhalli, Balakrishnan Prabhakaran, Susanne Boll, Ramanathan Subramanian, Liang Zheng, Vivek K. Singh, Pablo César, Lexing Xie, and Dong Xu, edi...

    arXiv 2024

  8. [9]

    Natural language guidance of high-fidelity text-to-speech with synthetic annotations

    Daniel Lyth and Simon King. Natural language guidance of high-fidelity text-to-speech with synthetic annotations. CoRR, abs/2402.01912, 2024. doi: 10.48550/ARXIV .2402.01912. URL https://doi.org/10.48550/arXiv.2402.01912. 8

    arXiv doi:10.48550/arxiv.2402.01912 2024

Show all 46 references
  1. [10]

    Scaling rich style-prompted text-to-speech datasets

    Anuj Diwan, Zhisheng Zheng, David Harwath, and Eunsol Choi. Scaling rich style-prompted text-to-speech datasets. CoRR, abs/2503.04713, 2025. doi: 10.48550/ARXIV .2503.04713. URL https://doi.org/10.48550/arXiv.2503.04713

    2025 doi

  2. [11]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV .2303. 08774. URL https://doi.org/10.48550/arXiv.2303.08774

    2023 doi arXiv

  3. [12]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 doi arXiv

  4. [13]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020. URL https://jmlr. org/p...

    2020

  5. [14]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. Cosyvoice 2: Scalable streaming speech synthesis ...

    2024 doi arXiv

  6. [15]

    Emovoice: Llm-based emotional text-to-speech model with freestyle text prompting

    Guanrou Yang, Chen Yang, Qian Chen, Ziyang Ma, Wenxi Chen, Wen Wang, Tianrui Wang, Yifan Yang, Zhikang Niu, Wenrui Liu, Fan Yu, Zhihao Du, Zhifu Gao, Shiliang Zhang, and Xie Chen. Emovoice: Llm-based emotional text-to-speech model with freestyle text prompting. CoRR, abs/2504....

    2025 doi arXiv

  7. [16]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi ...

    2025 doi arXiv

  8. [17]

    Investi- gating the catastrophic forgetting in multimodal large language models

    Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investi- gating the catastrophic forgetting in multimodal large language models. CoRR, abs/2309.10313,

    arXiv

  9. [18]

    Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. In IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2025

  10. [19]

    URL https://doi.org/10.48550/arXiv.2309

    doi: 10.48550/ARXIV .2309.10313. URL https://doi.org/10.48550/arXiv.2309. 10313

    doi arXiv

  11. [20]

    Vlmo: Unified vision- language pre-training with mixture-of-modality-experts

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision- language pre-training with mixture-of-modality-experts. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. ...

    2022

  12. [21]

    Evev2: Improved baselines for encoder-free vision- language models

    Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free vision- language models. CoRR, abs/2502.06788, 2025. doi: 10.48550/ARXIV .2502.06788. URL https://doi.org/10.48550/arX...

    2025 doi arXiv

  13. [22]

    Simbert: Integrating retrieval and generation into bert

    Jianlin Su. Simbert: Integrating retrieval and generation into bert. Technical report, 2020. URL https://github.com/ZhuiyiTechnology/simbert

    2020

  14. [23]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. CoRR, abs/2208.10442,

    arXiv

  15. [24]

    Fastspeech: Fast, robust and controllable text to speech

    Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural ...

    2019

  16. [25]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. CoRR, abs/2301.02111, 2023. doi: 10.48550/ARXIV .2...

    2023 doi arXiv

  17. [26]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of...

    2019

  18. [27]

    Minimax-speech: Intrinsic zero-shot text-to-speech with a learnable speaker encoder

    Bowen Zhang, Congchao Guo, Geng Yang, Hang Yu, Haozhe Zhang, Heidi Lei, Jialong Mai, Junjie Yan, Kaiyue Yang, Mingqi Yang, Peikai Huang, Ruiyang Jin, Sitan Jiang, Weihua Cheng, Yawei Li, Yichen Xiao, Yiying Zhou, Yongmao Zhang, Yuan Lu, and Yucen He. Minimax-speech: Intrinsic ...

    2025 doi arXiv

  19. [28]

    Scaling vision-language models with sparse mixture of experts

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singa...

    2023 doi

  20. [29]

    Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis

    Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi Dai, Hongzhan Lin, Jianyi Chen, Xingjian Du, Liumeng Xue, Yunlin Chen, Zhifei Li, Lei Xie, Qiuqiang Kong, Yike Guo, and Wei Xue. Llasa: Scaling train-time and inference-tim...

    2025 doi arXiv

  21. [30]

    Seed-tts: A family of high-quality versatile speech generation models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li...

    2024 doi arXiv

  22. [31]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 doi arXiv

  23. [32]

    Orpheus tts: Towards human-sounding tts

    The Canopy Labs Team. Orpheus tts: Towards human-sounding tts. Technical report, 2025. URL https://canopylabs.ai/model-releases

    2025

  24. [33]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process. , 29:3451–3460,

  25. [34]

    Qwen2.5-1m technical report

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang R...

    2025 doi arXiv

  26. [35]

    Neural discrete representation learning

    Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors,Advances in Neural Information Processing Sys...

    2017

  27. [36]

    Soundstream: An end-to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE ACM Trans. Audio Speech Lang. Process., 30:495–507, 2022. doi: 10.1109/TASLP.2021.3129994. URL https://doi.org/ 10.1109/TASLP.2021.3129994

    2022

  28. [37]

    High-fidelity audio compression with improved RVQGAN

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Ku- mar. High-fidelity audio compression with improved RVQGAN. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Ad- vances in Neural Information Proce...

    2023

  29. [38]

    Spark-tts: An efficient llm-based text-to- speech model with single-stream decoupled speech tokens

    Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, Weizhen Bian, Zhen Ye, Sitong Cheng, Ruibin Yuan, Zhixian Zhao, Xinfa Zhu, Jiahao Pan, Liumeng Xue, Pengcheng Zhu, Yunlin Chen, Zhifei Li, Xie Chen, L...

    2025 doi arXiv

  30. [39]

    Self-supervised learning with random-projection quantizer for speech recognition

    Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors,International Conference ...

    2022

  31. [40]

    Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons

    Zach Evans, Julian D. Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Sta- ble audio open. In 2025 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2025, Hyderabad, India, April 6-11, 2025 , pages 1–5. IEEE, 2025. doi: 10.1109/ICA...

    2025

  32. [41]

    Albergo, Nicholas M

    Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattl...

    2024

  33. [42]

    Emilia: A large-scale, extensive, multilingual, and diverse dataset for speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: A large-scale, extensive, multilingual, and diverse dataset for speech generation. CoRR, ab...

    2025 doi

  34. [44]

    Elucidating the de- sign space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the de- sign space of diffusion-based generative models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural In- formation Processing Systems 35: Annual ...

    2022

  35. [2021]

    URL https://doi.org/10.1109/TASLP.2021

    doi: 10.1109/TASLP.2021.3122291. URL https://doi.org/10.1109/TASLP.2021. 3122291

    2021

  36. [2022]

    URL https://doi.org/10.48550/arXiv.2208

    doi: 10.48550/ARXIV .2208.10442. URL https://doi.org/10.48550/arXiv.2208. 10442

    doi arXiv

  37. [2023]

    URL https://doi.org/10.1109/ ICASSP49357.2023.10096285

    doi: 10.1109/ICASSP49357.2023.10096285. URL https://doi.org/10.1109/ ICASSP49357.2023.10096285

    2023

  38. [4186]

    doi: 10.18653/V1/N19-1423

    Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1423. URL https://doi.org/10.18653/v1/n19-1423

    2019 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.