Pith. sign in

REVIEW 4 major objections 6 minor 37 references

Muyan-TTS: A Trainable Text-to-Speech Model Optimized for Podcast Scenarios with a $50K Budget

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Under a $50K budget, Muyan-TTS claims competitive podcast TTS quality and the fastest inference among the open-source baselines it compares against.

desk verdict A genuinely useful open-source engineering report: the $50K budget and speed claims are concrete and credible, but the naturalness evidence is weakened by using NISQA both to filter training data and to score results, so the quality advantages are provisional until human listening tests confirm them. read the letter →

arxiv 2504.19146 v1 pith:MIUMJLGT submitted 2025-04-27 cs.SD eess.AS

Xin Li , Kaikai Jia , Hao Sun , Jun Dai , Ziyang Jiang This is my paper · ORCID
classification cs.SDeess.AS
keywords text-to-speechlargelanguagemodelspodcastaudiozero-shotvoicecloningspeakeradaptationinferenceaccelerationMOSfilteringspeechsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Muyan-TTS is a trainable text-to-speech system that the authors say can be built from scratch for about $50,000 and still match the quality of much larger open-source TTS models on podcast-style speech. The paper's target is podcast and voice-interaction use, where long-form natural speech and fast synthesis matter. It argues that continuing pre-training a 3B-parameter LLM on over 100,000 hours of podcast audio, then decoding with a VITS-based model, gives a rare combination of zero-shot voice cloning, speaker adaptation from dozens of minutes of target speech, and the fastest inference in its comparison group. If the claim holds, small teams can reproduce and customize a competitive TTS model rather than relying on closed services.

What carries the argument

The load-bearing mechanism is a parallel text-and-audio-token corpus. Podcast audio is transcribed to text, while Hubert embeddings are quantized into 1,024 discrete audio tokens that are added to the LLM's vocabulary; the LLM is continued-pretrained to generate the audio tokens that follow a transcript. At inference the model can be prompted with reference text plus reference audio tokens to clone a voice zero-shot, or after supervised fine-tuning on one speaker with text alone. A VITS-based decoder, fine-tuned on a high-quality podcast subset, then converts the predicted tokens, phonemes, and a speaker embedding into a waveform, giving the grapheme-to-phoneme grounding that controls hallucination.

What would settle it

Run a blind human listening test on a fixed sample of LibriSpeech test-clean and SEED utterances, comparing Muyan-TTS against its strongest baseline with the same text and reference voice; if human naturalness ratings do not reproduce the reported automated MOS ordering, with listeners clearly preferring the baseline, the competitive-quality claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a hybrid cascade—an LLM for text-to-audio-token generation and a VITS-based decoder for tokens-to-waveform—delivers open-source TTS quality competitive with the best compared models while training for about $50K. On LibriSpeech test-clean the base model achieves the second-lowest WER (3.44%) among the six systems; on SEED it attains the highest MOS among models that use both the prompt text and prompt speech; supervised fine-tuning raises MOS from 4.58 to 4.97 and speaker similarity from 0.37 to 0.46; and the optimized inference ratio is 0.33 seconds of compute per second of audio, the fastest reported. The authors attribute the stability to the decoder's grapheme-to-phoneme grounding, which they say reduces the hallucinations common in LLM-based TTS.

Load-bearing premise

The quality claims hold only if the automated speech-quality scores used both to clean the training data and to rank the outputs agree with how human listeners perceive the audio.

Editorial extensions

If this is right

  • A small team can reproduce a competitive TTS model from scratch for about $50K, including data collection, cleaning, and training, rather than needing industrial-scale compute.
  • Zero-shot synthesis from a reference clip works without fine-tuning, and dozens of minutes of target speech are enough to push perceived quality and speaker similarity higher.
  • The 0.33 seconds-per-second-of-audio ratio places the non-streaming model in range of latency-sensitive voice applications.
  • The open training code and data pipeline make podcast-style TTS adaptable to other voices and domains without waiting for a vendor to release a fine-tune.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The data-cleaning and formatting pipeline is described as reusable, so the same budget recipe should transfer to audiobooks or meeting recordings; the paper only demonstrates podcast audio.
  • Because the speed benchmark disables streaming and parallelization, the real-world latency profile will depend on the application, and the listed dependency on complete phoneme sequences blocks true streaming in the current design.
  • The same extension-by-quantization recipe could be tried with other LLM sizes and tokenizers to see whether the quality-speed balance scales; the paper reports one configuration.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents Muyan-TTS, an open-source, trainable text-to-speech system intended for podcast scenarios. The system continues pre-training Llama-3.2-3B on over 100,000 hours of processed podcast audio, converts text into quantized HuBERT-style audio tokens using a GPT-SoVITS quantizer, and uses a VITS-based (SoVITS) decoder. The manuscript describes a large-scale data collection, cleaning, and formatting pipeline, LLM pre-training and supervised fine-tuning for speaker adaptation, decoder fine-tuning on high-MOS subsets, and an inference acceleration framework. Experiments on LibriSpeech and SEED compare word error rate, NISQA MOS, and speaker similarity against several open-source TTS models, and a speed benchmark reports a synthesis ratio of 0.33 seconds per second of audio, the fastest among compared systems. The paper releases code, model checkpoints, and the full training procedure.

Significance. If the reported results hold, the main contribution is practical reproducibility: the paper provides training code, model checkpoints, a full data-processing pipeline, and a cost breakdown of roughly $50K, which is uncommon for LLM-based TTS. The inference-acceleration result is concrete and falsifiable, and the ablation in Table 6 on decoder data quality is useful. However, the evidence for the naturalness and intelligibility claims is weakened by the overlap between the predictors used to construct the training data and those used to evaluate the model. The significance is therefore contingent on independent validation of the perceptual metrics.

major comments (4)
  1. [§3.2, §3.5, §4.1] The same NISQA predictor is used to filter the training corpus (MOS > 3.8), to select the decoder subset (MOS > 4.5), and then as the primary naturalness metric in Tables 3, 4, and 6. This creates a selection/evaluation loop: the model is optimized, through data filtering and decoder selection, to satisfy the very predictor that later grades it. The reported MOS advantages, including the MOS > 4.5 improvement in Table 6, may therefore reflect NISQA's preferences rather than perceived speech quality. No confidence intervals or significance tests are provided, and several Table 3 gaps (e.g., 4.58 vs. 4.81 vs. 5.00) are small relative to typical NISQA utterance-level variance. I ask for human listening MOS with confidence intervals, or at minimum an independent objective metric that was not used in any training-data selection step.
  2. [§3.2, §4.1] Whisper-large-v3 is used both to generate the training transcripts (Section 3.2) and to compute the WER metric (Section 4.1). Because the LLM is trained on Whisper's transcriptions, the WER evaluation is partially circular and may overstate intelligibility relative to human perception. Please evaluate with a different ASR model, or with human transcription, and report confidence intervals for WER. This is load-bearing for the intelligibility component of the central claim that Muyan-TTS is competitive with state-of-the-art open-source models.
  3. [§4.2.1, Table 3] Step-Audio is evaluated without a reference prompt and its SIM is not reported, yet it is included in the MOS comparison and later excluded via the sentence 'Among all models that use both text and speech prompts...'. This is a post-hoc comparison group. Please either evaluate Step-Audio under the same prompting protocol, or clearly separate the comparison groups in the table and text so that the reader can see exactly which models are being compared on equal footing.
  4. [§4.2.1, Table 3, Table 4] Muyan-TTS has the lowest or second-lowest SIM among all baselines with reported SIM (0.37 on LibriSpeech, 0.41 on SEED), and the SFT model in Table 4 reaches only 0.46, still below CosyVoice's zero-shot SIM of 0.66-0.70. This does not support the conclusion that the system is competitive in voice adaptation or speaker similarity. Please temper the conclusion, or provide additional evidence that SIM is not the relevant criterion for podcast-oriented TTS. The current framing in the conclusion ('while maintaining flexibility for voice adaptation') is not supported by the reported SIM values.
minor comments (6)
  1. [Table 1, Abstract] Table 1 reports a total of $50.54K, which conflicts with the repeated phrase 'within a $50,000 budget'. Please use 'approximately $50K' or explain the rounding.
  2. [§3.2, §4.1] Section 3.2 refers to 'NISQA [25]' while Section 4.1 specifies 'NISQA v2 model'. Please state explicitly whether the same version is used in data filtering and evaluation; this is important for assessing the circularity concern.
  3. [§4.2.1] The sentence 'Among all models that use both text and speech prompts from the SEED dataset, Muyan-TTS achieves the highest MOS' appears as an aside after the results; consider introducing this comparison group before presenting Table 3.
  4. [§4.2.3] The speed ratio r would benefit from reporting the exact hardware and batch size used for each model; the current table lists only the ratio, making the comparison difficult to reproduce.
  5. [§3.3] The example prompt 'Hey, great to have you in Chatpods. <|audio token 520|>...' does not show the reference audio tokens used in zero-shot inference; please include the full input format with reference text and reference audio tokens.
  6. [§3.2] The pipeline description would benefit from a data-retention table, showing how many hours remain after each cleaning and filtering step; the current text jumps from 'over 150,000 hours' to 'over 100,000 hours' without a per-stage breakdown.

Circularity Check

1 steps flagged · score 4.0 of 10

Naturalness claims partly circular: NISQA MOS both selects training/decoder data and evaluates the model.

  1. fitted input called prediction [Section 3.2 (Data Formatting), Section 3.5 (Decoder Training), Section 4.1 (Evaluation Metrics); Tables 3, 4, 6]
    "Each sentence is then scored using the MOS estimate provided by NISQA. We retain segments with MOS scores above 3.8 ... selecting only clips with a MOS value greater than 4.5, as evaluated using NISQA. ... MOS: The perceptual quality of the generated audio is estimated using the NISQA v2 model"

    NISQA MOS serves triple duty: it filters the training corpus (MOS > 3.8), selects the 10,000-hour decoder subset (MOS > 4.5), and then is the reported MOS evaluation metric. The 'naturalness' advantage is therefore not independent evidence: the model was trained only on audio that NISQA already rated highly, and Table 6's conclusion that MOS>4.5 decoder data improves quality largely restates the selection criterion. External WER and SIM partially mitigate this, but the MOS-based naturalness claims (Tables 3, 4, 6) are self-referential and may reflect NISQA's preferences rather than validated human perception.

full rationale

Muyan-TTS is a trained system, not a deductive derivation, so most of the paper (architecture, data pipeline, inference speed, and budget accounting) is self-contained and externally checkable. The central naturalness result, however, has a partial selection/evaluation loop: NISQA MOS is used both to curate the training corpus (Section 3.2, MOS > 3.8) and the decoder subset (Section 3.5, MOS > 4.5), and then the same NISQA v2 model is the reported MOS metric (Section 4.1). Thus Tables 3, 4, and 6 partly report whether the system scores well on the same learned preference function used to filter its training data; the Table 6 improvement from the MOS>4.5 subset is in particular a re-statement of the selection criterion rather than an independent perceptual validation. This is not a full derivation-from-itself: WER and SIM are independent external metrics, and no load-bearing self-citation or uniqueness import occurs. The Whisper-large-v3 overlap (training transcripts vs. WER backend) is a related confound but does not by itself make the WER claim definitionally circular, since the benchmark text is independent. Score 4 reflects a partially self-referential central naturalness claim, not full circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest mainly on hand-chosen data-quality thresholds, ASR transcription quality, compatibility of reused GPT-SoVITS components, and the validity of NISQA as a human-quality proxy. No entirely new theoretical entities are introduced.

free parameters (3)
  • NISQA MOS threshold for data retention = > 3.8
    Used in Section 3.2 to keep about 70% of transcribed sentences; chosen by hand, not justified by an independent validation set.
  • NISQA MOS threshold for decoder subset = > 4.5
    Section 3.5 selects 10,000 hours for SoVITS fine-tuning, and Table 6 shows it improves the same NISQA-based MOS metric.
  • Training epochs = 15 (LLM pretrain), 10 (SFT), 8 (decoder)
    Epoch counts are reported as fixed choices; no ablation shows that these values are optimal, and Table 6 shows epoch effects depend on data quality.
assumptions (4)
  • domain assumption NISQA v2 MOS estimates correlate with human perceptual quality.
    NISQA is used both to filter training data and as the naturalness metric, so the reported MOS results inherit any bias or error in NISQA.
  • domain assumption Whisper-large-v3 and FunASR transcriptions are accurate enough to serve as training text labels.
    Section 3.2 relies on ASR transcripts as the text side of the parallel corpus; transcription errors propagate into the text-to-token mapping.
  • domain assumption The GPT-SoVITS quantizer and SoVITS decoder remain compatible with Llama-3.2-3B after vocabulary extension.
    Section 3.3 injects 1,024 new audio tokens into Llama's vocabulary but provides no analysis showing that the resulting embeddings and decoder alignment are optimal.
  • domain assumption The collected podcast audio can be legally and ethically used for training and release.
    Section 3.2 states a proprietary podcast collection is used, but the preprint gives no licensing or consent details, which matters for a release-claimed open-source model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Muyan-TTS: A Trainable Text-to-Speech Model Optimized for Podcast Scenarios with a $50K Budget." pith.science (2026). https://pith.science/paper/MIUMJLGT

@misc{pith2026250419146,
  author       = {Pith},
  title        = {Pith review of: Muyan-TTS: A Trainable Text-to-Speech Model Optimized for Podcast Scenarios with a $50K Budget},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MIUMJLGT}},
  note         = {Machine review of arXiv:2504.19146}
}
abstract

Recent advancements in text-to-speech (TTS) models have been driven by the integration of large language models (LLMs), enhancing semantic comprehension and improving speech naturalness. However, existing LLM-based TTS models often lack open-source training code and efficient inference acceleration frameworks, limiting their accessibility and adaptability. Additionally, there is no publicly available TTS model specifically optimized for podcast scenarios, which are in high demand for voice interaction applications. To address these limitations, we introduce Muyan-TTS, an open-source trainable TTS model designed for podcast applications within a $50,000 budget. Our model is pre-trained on over 100,000 hours of podcast audio data, enabling zero-shot TTS synthesis with high-quality voice generation. Furthermore, Muyan-TTS supports speaker adaptation with dozens of minutes of target speech, making it highly customizable for individual voices. In addition to open-sourcing the model, we provide a comprehensive data collection and processing pipeline, a full training procedure, and an optimized inference framework that accelerates LLM-based TTS synthesis. Our code and models are available at https://github.com/MYZY-AI/Muyan-TTS.

Figures

Figures reproduced from arXiv: 2504.19146 by the authors.

Figure 1
Figure 1. Framework of Muyan-TTS. Left is an LLM that models the parallel corpus of text (in blue) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Data processing pipeline. Data Collection. To ensure speaker diversity and high-quality speech samples, we collect audio data from both open-source datasets and a proprietary podcast collection. We aim to maximize speaker coverage and linguistic variability, with a focus on English and partial inclusion of Chinese content for increased robustness. Each audio source is uniformly sampled and evaluated using NISQA [25]… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages

  1. [1]

    Conditional variational autoencoder with adver- sarial learning for end-to-end text-to-speech

    Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adver- sarial learning for end-to-end text-to-speech. In International Conference on Machine Learn- ing, pages 5530–5540. PMLR, 2021

    work page 2021

  2. [2]

    Vits2: Improving quality and efficiency of single-stage text-to-speech with adversarial learning and architecture design

    Jungil Kong, Jihoon Park, Beomjeong Kim, Jeongmin Kim, Dohee Kong, and Sangjin Kim. Vits2: Improving quality and efficiency of single-stage text-to-speech with adversarial learning and architecture design. arXiv preprint arXiv:2307.16430, 2023. 10

    arXiv 2023

  3. [3]

    Wave-tacotron: Spectrogram-free end-to-end text-to-speech synthesis

    Ron J Weiss, RJ Skerry-Ryan, Eric Battenberg, Soroosh Mariooryad, and Diederik P Kingma. Wave-tacotron: Spectrogram-free end-to-end text-to-speech synthesis. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5679–5683. IEEE, 2021

    work page 2021

  4. [4]

    End- to-end adversarial text-to-speech

    Jeff Donahue, Sander Dieleman, Mikołaj Bi ´nkowski, Erich Elsen, and Karen Simonyan. End- to-end adversarial text-to-speech. arXiv preprint arXiv:2006.03575, 2020

    arXiv 2006

  5. [5]

    Clarinet: Parallel wave generation in end-to-end text-to-speech

    Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to-speech. arXiv preprint arXiv:1807.07281, 2018

    arXiv 2018

  6. [6]

    Gpt-sovits

    RVC-Boss. Gpt-sovits. https://github.com/RVC-Boss/GPT-SoVITS, 2024

    work page 2024

  7. [7]

    2Noise. Chattts. https://github.com/2noise/ChatTTS, 2024

    work page 2024

  8. [8]

    Neural speech synthesis with transformer network

    Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 6706–6713, 2019

    work page 2019

Show all 37 references
  1. [9]

    Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions

    Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustic...

    2018

  2. [10]

    Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis

    Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis. arXiv preprint arXiv:2005.05957, 2020

    2005 arXiv

  3. [11]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  4. [12]

    Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370, 2024

    2024 arXiv

  5. [13]

    Lauragpt: Listen, attend, understand, and regenerate audio with gpt

    Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023

    2023 arXiv

  6. [14]

    Fish-speech: Leveraging large language models for advanced multilingual text-to- speech synthesis

    Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yi- jin Xing. Fish-speech: Leveraging large language models for advanced multilingual text-to- speech synthesis. arXiv preprint arXiv:2411.01156, 2024

    2024 arXiv

  7. [15]

    Seed-tts: A family of high-quality versatile speech generation models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024

    2024 arXiv

  8. [16]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  9. [17]

    Improving robustness of llm-based speech synthesis by learning monotonic alignment

    Paarth Neekhara, Shehzeen Hussain, Subhankar Ghosh, Jason Li, Rafael Valle, Rohan Bad- lani, and Boris Ginsburg. Improving robustness of llm-based speech synthesis by learning monotonic alignment. arXiv preprint arXiv:2406.17957, 2024

    2024 arXiv

  10. [18]

    Improving audio codec- based zero-shot text-to-speech synthesis with multi-modal context and large language model

    Jinlong Xue, Yayue Deng, Yicheng Han, Yingming Gao, and Ya Li. Improving audio codec- based zero-shot text-to-speech synthesis with multi-modal context and large language model. arXiv preprint arXiv:2406.03706, 2024

    2024 arXiv

  11. [19]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117, 2024

    2024 arXiv

  12. [20]

    Step-audio: Unified understanding and generation in intelligent speech interaction

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502.11946, 2025

    2025 arXiv

  13. [21]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 11

    2024 arXiv

  14. [22]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  15. [23]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International con- ference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  16. [24]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi- nov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 2...

    2021

  17. [25]

    Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets

    Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian M ¨oller. Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets. arXiv preprint arXiv:2104.09494, 2021

    2021 arXiv

  18. [26]

    Music source separa- tion with band-split rope transformer

    Wei-Tsung Lu, Ju-Chiang Wang, Qiuqiang Kong, and Yun-Ning Hung. Music source separa- tion with band-split rope transformer. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 481–485. IEEE, 2024

    2024

  19. [27]

    V oicefixer: Toward general speech restoration with neural vocoder

    Haohe Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang. V oicefixer: Toward general speech restoration with neural vocoder. arXiv preprint arXiv:2109.13731, 2021

    2021 arXiv

  20. [28]

    Low-complexity acoustic echo cancellation with neural kalman filtering

    Dong Yang, Fei Jiang, Wei Wu, Xuefei Fang, and Muyong Cao. Low-complexity acoustic echo cancellation with neural kalman filtering. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  21. [29]

    Frcrn: Boosting feature representation using frequency recurrence for monaural speech enhancement

    Shengkui Zhao, Bin Ma, Karn N Watcharasupat, and Woon-Seng Gan. Frcrn: Boosting feature representation using frequency recurrence for monaural speech enhancement. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal process- ing (ICASSP), pages 928...

    2022

  22. [30]

    NVIDIA. Nemo. https://github.com/NVIDIA/NeMo, 2024

    2024

  23. [31]

    Funasr: A fundamental end-to-end speech recognition toolkit

    Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al. Funasr: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305.11013, 2023

    2023 arXiv

  24. [32]

    Alpaca: A strong, replica- ble instruction-following model

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replica- ble instruction-following model. Stanford Center for Research on Foundation Models. https://crfm.stanford.edu/2023/03/13/al...

    2023

  25. [33]

    Efficient memory management for large lan- guage model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, ...

    2023

  26. [34]

    Spark-tts: An efficient llm-based text-to- speech model with single-stream decoupled speech tokens

    Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-tts: An efficient llm-based text-to- speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710, 2025

    2025 arXiv

  27. [35]

    Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications

    Hao-Han Guo, Kun Liu, Fei-Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03283, 2024

    2024 arXiv

  28. [36]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015

    2015

  29. [37]

    Large-scale self-supervised speech representation learning for automatic speaker verification

    Zhengyang Chen, Sanyuan Chen, Yu Wu, Yao Qian, Chengyi Wang, Shujie Liu, Yanmin Qian, and Michael Zeng. Large-scale self-supervised speech representation learning for automatic speaker verification. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Sig...

    2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.