{"id":"57f2dd04-2220-4633-9f6d-4aa636f7511a","arxiv_id":"2504.19146","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Muyan-TTS, a 3B-parameter LLM-based TTS model trained on 100,000+ hours of podcast audio, produces competitive zero-shot speech and runs at 0.33 seconds of inference per second of speech.","lead":"This paper describes Muyan-TTS, an open-source text-to-speech model trained on over 100,000 hours of podcast audio using a $50,000 compute budget. It combines a Llama 3.2 language model with a VITS-style speech decoder to synthesize voices from text, and it reports competitive quality and faster-than-real-time inference.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Naturalness and intelligibility claims rest on automatic metrics (NISQA MOS, Whisper WER) that were also used to filter and transcribe the training data, so reported advantages may not reflect human judgment.","rationale":"The paper's central conclusion is conditional on the validity of its evaluation. The strongest evidence for competitiveness in naturalness and intelligibility is Table 3, with WER, NISQA MOS, and SIM. The MOS metric is not independent of the training pipeline: the same NISQA v2 model is used to filter data at MOS > 3.8, to choose the decoder subset at MOS > 4.5, and then to grade the final models. This introduces selection bias even if NISQA is a useful predictor of human opinion: the model is tuned to that predictor, and the evaluation cannot distinguish improved perceived quality from optimization of NISQA. The same double-duty pattern applies to Whisper-large-v3, which generates the training transcripts and is then used to compute WER. A human listening study with naturalness ratings and transcriptions is the direct check. The paper does provide some independent support: the WER and speed numbers are concrete, and the budget breakdown is coherent, so the concern is not fatal but requires conditional acceptance. The reader's NISQA concern is valid and captures the core issue; I extend it to the WER/ASR loop, hence partial agreement. The verdict remains CONDITIONAL, so no change is needed.","tokens_in":11314,"tokens_out":8747,"duration_ms":92396,"concrete_test":"Run a preregistered listening study on about 60 utterances sampled from LibriSpeech test-clean and the SEED test set, covering Muyan-TTS, CosyVoice2, FireRedTTS, and GPT-SoVITS v3. Recruit at least 20 native English listeners; collect MUSHRA or five-point naturalness ratings and, separately, verbatim transcriptions. Compare the human MOS ordering with the NISQA v2 ordering from Table 3, and compare human word error rates with Whisper-large-v3 WER. If Muyan-TTS is not in the statistically tied top group for human naturalness, or if human transcription error is materially worse than the reported WER, the headline claim is not supported. A secondary check is to recompute WER with a different ASR architecture, such as a Conformer-based model, on the same synthesized audio.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on automatic metrics that do double duty as training-data constructors and evaluators. Section 3.2 uses NISQA v2 to score and filter the corpus (MOS > 3.8), and Section 3.5 uses the same NISQA to select the 10,000-hour decoder subset (MOS > 4.5); Section 4.1 then reports NISQA MOS as the primary naturalness evidence (Tables 3, 4, and 6). Likewise, Section 3.2 uses Whisper-large-v3 to create training transcripts, and Section 4.1 uses the same ASR model for WER. The model is therefore optimized, through data filtering and label generation, to satisfy the very predictors that later grade it. This selection/evaluation loop means the reported MOS gains, including the MOS > 4.5 improvement in Table 6, may reflect NISQA's preferences rather than perceived quality. No human listening data, confidence intervals, or significance tests are provided, and several Table 3 MOS gaps (e.g., 4.58 vs. 4.81 vs. 5.00) are small relative to typical NISQA utterance-level variance. The speed and budget claims are less affected, but the naturalness and intelligibility conclusions are not established until the metrics are validated independently.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Muyan-TTS, an open-source, trainable text-to-speech system intended for podcast scenarios. The system continues pre-training Llama-3.2-3B on over 100,000 hours of processed podcast audio, converts text into quantized HuBERT-style audio tokens using a GPT-SoVITS quantizer, and uses a VITS-based (SoVITS) decoder. The manuscript describes a large-scale data collection, cleaning, and formatting pipeline, LLM pre-training and supervised fine-tuning for speaker adaptation, decoder fine-tuning on high-MOS subsets, and an inference acceleration framework. Experiments on LibriSpeech and SEED compare word error rate, NISQA MOS, and speaker similarity against several open-source TTS models, and a speed benchmark reports a synthesis ratio of 0.33 seconds per second of audio, the fastest among compared systems. The paper releases code, model checkpoints, and the full training procedure.","tokens_in":11571,"tokens_out":5607,"duration_ms":54114,"significance":"If the reported results hold, the main contribution is practical reproducibility: the paper provides training code, model checkpoints, a full data-processing pipeline, and a cost breakdown of roughly $50K, which is uncommon for LLM-based TTS. The inference-acceleration result is concrete and falsifiable, and the ablation in Table 6 on decoder data quality is useful. However, the evidence for the naturalness and intelligibility claims is weakened by the overlap between the predictors used to construct the training data and those used to evaluate the model. The significance is therefore contingent on independent validation of the perceptual metrics.","major_comments":[{"comment":"The same NISQA predictor is used to filter the training corpus (MOS > 3.8), to select the decoder subset (MOS > 4.5), and then as the primary naturalness metric in Tables 3, 4, and 6. This creates a selection/evaluation loop: the model is optimized, through data filtering and decoder selection, to satisfy the very predictor that later grades it. The reported MOS advantages, including the MOS > 4.5 improvement in Table 6, may therefore reflect NISQA's preferences rather than perceived speech quality. No confidence intervals or significance tests are provided, and several Table 3 gaps (e.g., 4.58 vs. 4.81 vs. 5.00) are small relative to typical NISQA utterance-level variance. I ask for human listening MOS with confidence intervals, or at minimum an independent objective metric that was not used in any training-data selection step.","section":"§3.2, §3.5, §4.1"},{"comment":"Whisper-large-v3 is used both to generate the training transcripts (Section 3.2) and to compute the WER metric (Section 4.1). Because the LLM is trained on Whisper's transcriptions, the WER evaluation is partially circular and may overstate intelligibility relative to human perception. Please evaluate with a different ASR model, or with human transcription, and report confidence intervals for WER. This is load-bearing for the intelligibility component of the central claim that Muyan-TTS is competitive with state-of-the-art open-source models.","section":"§3.2, §4.1"},{"comment":"Step-Audio is evaluated without a reference prompt and its SIM is not reported, yet it is included in the MOS comparison and later excluded via the sentence 'Among all models that use both text and speech prompts...'. This is a post-hoc comparison group. Please either evaluate Step-Audio under the same prompting protocol, or clearly separate the comparison groups in the table and text so that the reader can see exactly which models are being compared on equal footing.","section":"§4.2.1, Table 3"},{"comment":"Muyan-TTS has the lowest or second-lowest SIM among all baselines with reported SIM (0.37 on LibriSpeech, 0.41 on SEED), and the SFT model in Table 4 reaches only 0.46, still below CosyVoice's zero-shot SIM of 0.66-0.70. This does not support the conclusion that the system is competitive in voice adaptation or speaker similarity. Please temper the conclusion, or provide additional evidence that SIM is not the relevant criterion for podcast-oriented TTS. The current framing in the conclusion ('while maintaining flexibility for voice adaptation') is not supported by the reported SIM values.","section":"§4.2.1, Table 3, Table 4"}],"minor_comments":[{"comment":"Table 1 reports a total of $50.54K, which conflicts with the repeated phrase 'within a $50,000 budget'. Please use 'approximately $50K' or explain the rounding.","section":"Table 1, Abstract"},{"comment":"Section 3.2 refers to 'NISQA [25]' while Section 4.1 specifies 'NISQA v2 model'. Please state explicitly whether the same version is used in data filtering and evaluation; this is important for assessing the circularity concern.","section":"§3.2, §4.1"},{"comment":"The sentence 'Among all models that use both text and speech prompts from the SEED dataset, Muyan-TTS achieves the highest MOS' appears as an aside after the results; consider introducing this comparison group before presenting Table 3.","section":"§4.2.1"},{"comment":"The speed ratio r would benefit from reporting the exact hardware and batch size used for each model; the current table lists only the ratio, making the comparison difficult to reproduce.","section":"§4.2.3"},{"comment":"The example prompt 'Hey, great to have you in Chatpods. <|audio token 520|>...' does not show the reference audio tokens used in zero-shot inference; please include the full input format with reference text and reference audio tokens.","section":"§3.3"},{"comment":"The pipeline description would benefit from a data-retention table, showing how many hours remain after each cleaning and filtering step; the current text jumps from 'over 150,000 hours' to 'over 100,000 hours' without a per-stage breakdown.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuine practical contribution through open-sourcing code, models, and a cost breakdown, but the evaluation is not yet at the level needed to support the naturalness and intelligibility claims. The NISQA and Whisper circularity is substantial and should be addressed before publication. The title's '50K budget' is also slightly exceeded in Table 1, which may matter to readers. For the editor: I see this as a major-revision case rather than rejection, because the issues are fixable with additional evaluation work and more careful claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful open-source engineering report, not a research breakthrough. It delivers exactly what the title promises—a trainable TTS pipeline for podcast audio at roughly $50K—and the cost breakdown is refreshingly concrete. But the quality claims rest on automatic metrics that also shaped the training data, so the reported MOS advantages should be treated as provisional until human listening confirms them.\n\nWhat's new: the paper's contribution is the assembled package—a 100k-hour podcast corpus with a documented cleaning pipeline, continued pre-training of Llama-3.2-3B on quantized GPT-SoVITS tokens, a VITS-style decoder fine-tuned on a high-MOS subset, and an inference-optimized framework that hits 0.33 seconds per second of speech, the fastest in Table 5. That speed result plus the $50K cost estimate are the strongest, most falsifiable parts. The decoder training study (Table 6) is a nice empirical point: data quality matters more than epoch count.\n\nSoft spots: the evaluation is the weak link. NISQA v2 is used to filter the corpus (MOS > 3.8), select the decoder subset (MOS > 4.5), and then score the final models. Whisper-large-v3 generates training transcripts and measures WER. So the model is tuned to satisfy the same predictors that grade it. That doesn't make the results fraudulent, but it means the MOS gaps—e.g., 4.58 vs 4.81 vs 5.00 on LibriSpeech—could reflect NISQA's preferences more than human perception. No confidence intervals, no significance tests, and SIM is clearly below several baselines. The authors acknowledge the SIM weakness, which is honest. The speed and budget claims are insulated from most of this concern because they don't depend on the quality metrics.\n\nWho it's for: practitioners building TTS on a budget, and also a good case study for an evaluation-loop discussion in a methods class. The paper deserves a serious referee—the engineering content is real and the open-source promise (code, models, pipeline) would make verification possible. I'd send it out, but with a clear request: human listening tests for naturalness, error bars, and confirmation that the repo actually contains training code and checkpoints. If those come through, the core claims become credible. If not, the paper remains a useful technical report rather than a solid benchmark entry.","headline":"A genuinely useful open-source engineering report: the $50K budget and speed claims are concrete and credible, but the naturalness evidence is weakened by using NISQA both to filter training data and to score results, so the quality advantages are provisional until human listening tests confirm them.","tokens_in":12138,"tokens_out":1815,"would_cite":true,"duration_ms":17060,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under a $50K budget, Muyan-TTS claims competitive podcast TTS quality and the fastest inference among the open-source baselines it compares against.","keywords":["text-to-speech","large language models","podcast audio","zero-shot voice cloning","speaker adaptation","inference acceleration","MOS filtering","speech synthesis"],"falsifier":"Run a blind human listening test on a fixed sample of LibriSpeech test-clean and SEED utterances, comparing Muyan-TTS against its strongest baseline with the same text and reference voice; if human naturalness ratings do not reproduce the reported automated MOS ordering, with listeners clearly preferring the baseline, the competitive-quality claim is falsified.","tokens_in":11097,"feed_emoji":"🎙️","tokens_out":10279,"duration_ms":91246,"temperature":0.7,"pith_summary":"Muyan-TTS is a trainable text-to-speech system that the authors say can be built from scratch for about $50,000 and still match the quality of much larger open-source TTS models on podcast-style speech. The paper's target is podcast and voice-interaction use, where long-form natural speech and fast synthesis matter. It argues that continuing pre-training a 3B-parameter LLM on over 100,000 hours of podcast audio, then decoding with a VITS-based model, gives a rare combination of zero-shot voice cloning, speaker adaptation from dozens of minutes of target speech, and the fastest inference in its comparison group. If the claim holds, small teams can reproduce and customize a competitive TTS model rather than relying on closed services.","feed_headline":"Trained for $50K, TTS model beats open rivals on speed","feed_subtitle":"Muyan-TTS turns text into speech in 0.33 seconds per second of audio, near the top open models on quality.","key_machinery":"The load-bearing mechanism is a parallel text-and-audio-token corpus. Podcast audio is transcribed to text, while Hubert embeddings are quantized into 1,024 discrete audio tokens that are added to the LLM's vocabulary; the LLM is continued-pretrained to generate the audio tokens that follow a transcript. At inference the model can be prompted with reference text plus reference audio tokens to clone a voice zero-shot, or after supervised fine-tuning on one speaker with text alone. A VITS-based decoder, fine-tuned on a high-quality podcast subset, then converts the predicted tokens, phonemes, and a speaker embedding into a waveform, giving the grapheme-to-phoneme grounding that controls hallucination.","core_discovery":"The central claim is that a hybrid cascade—an LLM for text-to-audio-token generation and a VITS-based decoder for tokens-to-waveform—delivers open-source TTS quality competitive with the best compared models while training for about $50K. On LibriSpeech test-clean the base model achieves the second-lowest WER (3.44%) among the six systems; on SEED it attains the highest MOS among models that use both the prompt text and prompt speech; supervised fine-tuning raises MOS from 4.58 to 4.97 and speaker similarity from 0.37 to 0.46; and the optimized inference ratio is 0.33 seconds of compute per second of audio, the fastest reported. The authors attribute the stability to the decoder's grapheme-to-phoneme grounding, which they say reduces the hallucinations common in LLM-based TTS.","pith_inferences":["The data-cleaning and formatting pipeline is described as reusable, so the same budget recipe should transfer to audiobooks or meeting recordings; the paper only demonstrates podcast audio.","Because the speed benchmark disables streaming and parallelization, the real-world latency profile will depend on the application, and the listed dependency on complete phoneme sequences blocks true streaming in the current design.","The same extension-by-quantization recipe could be tried with other LLM sizes and tokenizers to see whether the quality-speed balance scales; the paper reports one configuration."],"forward_implications":["A small team can reproduce a competitive TTS model from scratch for about $50K, including data collection, cleaning, and training, rather than needing industrial-scale compute.","Zero-shot synthesis from a reference clip works without fine-tuning, and dozens of minutes of target speech are enough to push perceived quality and speaker similarity higher.","The 0.33 seconds-per-second-of-audio ratio places the non-streaming model in range of latency-sensitive voice applications.","The open training code and data pipeline make podcast-style TTS adaptable to other voices and domains without waiting for a vendor to release a fine-tune."],"supporting_citations":[{"why":"Supplies the cascade design, the pre-trained quantizer that produces audio tokens, and the decoder that the paper continues training.","marker":"[6]"},{"why":"Provides the VITS end-to-end architecture whose grapheme-to-phoneme grounding is used to control LLM hallucinations.","marker":"[1]"},{"why":"The principal LLM-based baseline on which the quality comparison must land.","marker":"[19]"},{"why":"The 3B-parameter language model that is continued pre-trained on the podcast corpus.","marker":"[21]"},{"why":"Transcribes the podcast audio to build the parallel text/audio corpus and also computes the WER metric.","marker":"[23]"},{"why":"Produces the self-supervised speech embeddings that are quantized into the discrete audio tokens.","marker":"[24]"},{"why":"Scores every utterance, selecting the training data and providing the MOS values used as the naturalness metric.","marker":"[25]"},{"why":"The memory-management scheme used to accelerate LLM decoding in the reported inference speed.","marker":"[33]"},{"why":"The audiobook test corpus that carries the main zero-shot WER/MOS/SIM comparison.","marker":"[36]"},{"why":"The prompt-based benchmark used to compare personalized zero-shot synthesis across models.","marker":"[15]"}],"fun_headline_variants":["$50K TTS hits 0.33s/s audio, rivals top open models","Muyan-TTS: $50K training, 0.33s/s, open-source","Podcast-specialized TTS trains for $50K, matches open leaders","Open TTS for podcasts: $50K training, 3x real-time","Budget TTS for podcasts: $50K, fastest inference ratio"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality claims hold only if the automated speech-quality scores used both to clean the training data and to rank the outputs agree with how human listeners perceive the audio.","fun_headline_variants_meta":{"raw":{"variants":["$50K TTS hits 0.33s/s audio, rivals top open models","Muyan-TTS: $50K training, 0.33s/s, open-source","Podcast-specialized TTS trains for $50K, matches open leaders","Open TTS for podcasts: $50K training, 3x real-time","Budget TTS for podcasts: $50K, fastest inference ratio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001425,"raw_usage":{"total_tokens":5754,"prompt_tokens":953,"completion_tokens":4801,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":4693}},"tokens_in":569,"tokens_out":4801,"duration_ms":29504,"temperature":1.0,"reasoning_tokens":4693,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:59:49.048271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a blind human listening test on a fixed sample of LibriSpeech test-clean and SEED utterances, comparing Muyan-TTS against its strongest baseline with the same text and reference voice; if human naturalness ratings do not reproduce the reported automated MOS ordering, with listeners clearly preferring the baseline, the competitive-quality claim is falsified.","supporting_citations":[{"cited_title":"Gpt-sovits","cited_arxiv_id":null,"evidence_quote":"Supplies the cascade design, the pre-trained quantizer that produces audio tokens, and the decoder that the paper continues training."},{"cited_title":"Conditional variational autoencoder with adver- sarial learning for end-to-end text-to-speech","cited_arxiv_id":null,"evidence_quote":"Provides the VITS end-to-end architecture whose grapheme-to-phoneme grounding is used to control LLM hallucinations."}],"review_version":1}