Pith. sign in

REVIEW 3 major objections 6 minor 33 references

The Codec Language Model-based Zero-Shot Spontaneous Style TTS System for CoVoC Challenge 2024

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a LLaMA-based codec language model with a delay pattern and classifier-free guidance achieves the best speech naturalness MOS of 3.80 in the CoVoC 2024 constrained track, enabling zero-shot spontaneous-style voice…

desk verdict Solid CoVoC system paper with a credible externally evaluated naturalness win, but the spontaneous-style claim is only partially supported by its own scores. read the letter →

arxiv 2412.01100 v2 pith:CWTVEGEJ submitted 2024-12-02 cs.SD eess.AS

Shuoyi Zhou , Yixuan Zhou , Weiqin Li , Jun Chen , Runchuan Ye , Weihao Wu , Zijian Lin , Shun Lei
show 1 more author
Zhiyong Wu
This is my paper · ORCID
classification cs.SDeess.AS
keywords voicecloningzero-shotTTSspontaneousspeechsynthesiscodeclanguagemodeldelaypatternclassifier-freeguidanceCoVoCChallenge2024Mandarinconversational
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that zero-shot voice cloning in a spontaneous conversational style can be done by a codec language model without explicit labels for laughter, pauses, or hesitation. The proposed system turns speech into discrete tokens and generates them with a LLaMA-style autoregressive transformer: text first predicts semantic tokens, then semantic tokens plus a short speech prompt predict acoustic tokens through a delay pattern in which each residual code layer is offset by one time step. Classifier-free guidance is used at inference to keep the output intelligible. In the official CoVoC 2024 constrained-track evaluation, the system reports the best naturalness MOS of 3.80 among submitted systems, with a lower spontaneous-style score of 3.33. A sympathetic reader would take this as evidence that large-scale pretraining plus fine-tuning on high-quality conversational data is a viable route to expressive, spontaneous voice cloning.

What carries the argument

The load-bearing mechanism is the two-stage codec language model with a delay pattern. In the first stage, an MT5 text encoder with LoRA adapters conditions autoregressive prediction of HuBERT/K-means semantic tokens at 50 Hz, with consecutive duplicate tokens removed. In the second stage, a LLaMA-style transformer predicts all 12 Descript Audio Codec residual tokens per frame in parallel, but the layers are offset by one time step each, so the token for layer $k$ at time $t$ is predicted after layer $k-1$ at the same time; this is the delay pattern the paper adopts from music and guided-speech generation and argues is suited to spontaneous prosody. Classifier-free guidance is implemented by masking the text or semantic-token condition with probability 0.1 during training and then interpolating unconditional and conditional log-probabilities at inference, with strengths set to $\gamma=1.5$, $\alpha=1.3$, and $\beta=1.5$. The mechanism's job is to let a single autoregressive LM keep strong text conditioning without sacrificing the prosodic variability that makes speech sound spontaneous.

What would settle it

A blind listening test comparing the submitted model against the same model fine-tuned on MAGICDATA-RAMC with laughter, overlaps, and unintelligible noises preserved (or transcribed as special tokens) would settle the preprocessing question: if the unfiltered variant scores higher on spontaneous style at comparable quality and intelligibility, the filtering is the limiting factor; if it scores lower, the filtering is not the bottleneck.

Watch

Extended reading notes

Core claim

The central claim is that a LLaMA-based codec language model with a delay pattern and classifier-free guidance achieves the best speech naturalness among CoVoC 2024 constrained-track systems while performing zero-shot spontaneous-style voice cloning. The authors describe the model in two autoregressive stages: text to 50 Hz HuBERT-based semantic tokens, then semantic tokens plus a prompt's acoustic tokens to DAC acoustic tokens. The delay pattern offsets each of the 12 residual codebook layers by one time step so layer $k$ is predicted conditioned on layer $k-1$ at the same frame; they argue this suits the rich prosodic variation of spontaneous speech. Classifier-free guidance, applied by masking conditions during training and interpolating log probabilities with strengths $\gamma=1.5$, $\alpha=1.3$, and $\beta=1.5$ at inference, is presented as the mechanism that keeps intelligibility high. The paper's reported official results are naturalness MOS 3.80 (rank 1), quality 3.84 (rank 2), similarity 3.49 (rank 2), spontaneous style 3.33 (rank 3), and average 3.61 (rank 3). The authors attribute the moderate spontaneous and similarity scores to a compromise: they blended HQ-Conversations with the Premium subset of WenetSpeech4TTS during fine-tuning and did not use explicit spontaneous-behavior labels.

Load-bearing premise

The load-bearing premise is that deleting overlapping speech, unintelligible noises, musical sounds, and laughter from the conversational training data removes only unwanted noise, not cues the model needs for spontaneous style; if those paralinguistic events are what spontaneous-style listeners reward, the preprocessing suppresses the behavior the system is meant to synthesize.

Editorial extensions

If this is right

  • If the reported results hold, a purely data-driven codec language model can reach the top naturalness score in a zero-shot spontaneous voice-cloning challenge without explicit spontaneous-behavior labels.
  • The delay pattern plus classifier-free guidance is a transferable configuration: other codec language models can adopt the one-step residual offset and two-sided CFG interpolation without changing their tokenizers.
  • Long test sentences can be handled by segmenting at punctuation into clips of at least 30 characters and concatenating them with 100 ms silence, a decoding strategy the paper uses to stay within the training length.
  • Fine-tuning on high-quality spontaneous conversation data improves spontaneity, but mixing in a large premium read-speech corpus dilutes spontaneity and speaker similarity; the paper directly states this compromise lowered those scores.
  • The two-stage semantic-then-acoustic token design means prosodic content is generated in the acoustic stage, so a speech prompt can steer speaker identity while the text controls content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper leaves implicit that its preprocessing may be the ceiling on spontaneity; deleting laughter, overlaps, and unintelligible noises could remove exactly the paralinguistic cues a spontaneous-style MOS rewards.
  • Editorial inference: because the paper does not sweep CFG strengths, varying $\gamma$, $\alpha$, and $\beta$ could give a controllability axis between intelligibility and spontaneity; stronger text conditioning tends to suppress disfluencies.
  • Editorial inference: the same data-driven recipe should transfer to other codec language models and languages whenever a high-quality spontaneous corpus exists, since no explicit behavior labels are used in training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper describes a zero-shot spontaneous-style TTS system submitted to the CoVoC 2024 constrained track. The system uses a LLaMA-based codec language model that first predicts HuBERT-derived semantic tokens and then predicts DAC acoustic tokens with a delay pattern, supported by classifier-free guidance and LoRA adaptation of an MT5 text encoder. The authors report data preprocessing of the challenge corpora, a three-stage training schedule (pre-training, continued pre-training, fine-tuning on premium/spontaneous data), and official evaluation results: naturalness MOS 3.80 (1st), quality MOS 3.84 (2nd), similarity MOS 3.49 (2nd), spontaneous MOS 3.33 (3rd), average MOS 3.61 (3rd), CER 10.29%, and SECS 0.797.

Significance. If the reported results are taken at face value, the paper demonstrates a competitive codec-language-model TTS system for zero-shot voice cloning in a conversational setting, with the externally judged official challenge evaluation being a clear strength. The paper does not ship code, machine-checked proofs, or component-level ablations; its contribution is an empirical system description. The best-naturalness result is independent and not circular, but the paper's broader claim of 'spontaneous-style voice cloning' is only weakly supported by the presented evidence, because the data preprocessing removes several canonical spontaneous behaviors and the spontaneous MOS is the weakest score (3.33, 3rd).

major comments (3)
  1. [Section 2 and Section 4.3.2] The manuscript's central claim is zero-shot spontaneous-style voice cloning, but Section 2 states that segments containing 'overlapping speech, unintelligible noises, musical sounds, and laughter' were eliminated from MAGICDATA-RAMC, and the abstract generalizes this as 'denoise by removing non-speech sounds.' The introduction explicitly lists laughter, pauses, and hesitations as the phenomena that spontaneous TTS must model. Removing laughter and overlapping speech from the training data removes exactly the paralinguistic behaviors the system is claimed to synthesize, and Section 4.4 demonstrates only elongated vowels and filled pauses, not laughter or overlap. The spontaneous MOS of 3.33 (3rd) and the admission in Section 4.3.2 that the fine-tuning compromise 'still sacrifices some of the spontaneity' are consistent with this gap. Please either provide evidence that the model can generate the removed phenomena (for example, through targeted listening tests or behavior-label analysis) or revise the claim to specify that the system achieves high naturalness on clean speech while exhibiting only limited spontaneity.
  2. [Section 3.2 and Section 3.3] The delay pattern is asserted to be 'more suitable for modeling the rich variations inherent in spontaneous style speech,' but no experiment compares the proposed delay pattern with a non-delayed parallel prediction scheme. Likewise, Section 3.3 attributes intelligibility gains to classifier-free guidance, but Tables 2 and 3 report only whole-system results, with no CFG-off or no-fine-tuning ablation. These component-level causal attributions are therefore not established by the presented evidence. Please add ablations or explicitly reframe the delay pattern and CFG as design choices rather than validated contributions.
  3. [Section 4.1 and Section 4.2] The loss weights α_k for the twelve DAC layers and the CFG strengths γ=1.5, α=1.3, and β=1.5 are reported as fixed values, but the paper does not describe how they were selected or how sensitive the official results are to them. Since the reported naturalness and quality scores are the main positive results, a sensitivity analysis or a description of the selection procedure would materially strengthen reproducibility and the credibility of the reported metrics.
minor comments (6)
  1. [Section 1] There is a typo: 'high-qualtiy' should be 'high-quality'.
  2. [Section 3.2] There is a typo: 'bacbone' should be 'backbone'.
  3. [Section 3.1] The model name 'Hubert' should be written as 'HuBERT' for consistency with the cited reference.
  4. [Equation (2)] The notation A(t−k+1,k) is not defined; please clarify that the first index is the frame position and the second index is the codebook layer, and state the boundary conditions at the start of the sequence.
  5. [Table 3] Table 3 lists only the proposed system's CER and SECS values, so the claimed 'ranking 2nd' for CER and 'ranking 4th' for SECS cannot be verified from the table; please include the official comparison values or cite the challenge report where they appear.
  6. [Section 4.4] The case study presents two hand-picked spectrograms, but the selection criteria are not stated; please indicate whether these samples are representative or chosen as best-case demonstrations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: official CoVoC scores are external, and the paper's self-citations are contextual rather than load-bearing.

full rationale

The paper's central claims are grounded in the official CoVoC 2024 evaluation results, which are external to the model and to the authors' prior work: naturalness MOS 3.80, quality 3.84, similarity 3.49, spontaneous style 3.33, CER 10.29%, and SECS 0.797 are all measured by the challenge organizers on held-out material. No quantity presented as a result is constructed from a parameter fitted to that same target; the LLaMA-based codec language model, delay pattern, CFG strengths, and fine-tuning recipe are fixed design choices whose merit is judged by the official scores. The two self-references (VoxInstruct [26] for CFG and SponLMTTS [14] for spontaneous-style LM TTS) are contextual: the former is cited only as inspiration for a standard CFG technique, and the latter is used as a contrast to the paper's data-driven approach without explicit spontaneous-behavior labels. Neither supplies a load-bearing uniqueness theorem, a fitted parameter, or a definitional identity, so they do not constitute circularity. The data-preprocessing decision to remove laughter, overlapping speech, and musical or unintelligible sounds from MAGICDATA-RAMC could plausibly weaken the spontaneous-style modeling claim, and the paper itself concedes that supplementing HQ-Conversations with the Premium subset 'sacrifices some of the spontaneity,' but that is a limitation of evidence rather than a derivation that reduces to its inputs. The honestly reported lower spontaneous MOS of 3.33 is consistent with that limitation. Overall, the system is benchmarked externally and its derivation chain does not collapse into its assumptions.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The system's performance claim rests on several untested modeling and data assumptions; most components are adopted from prior work, and the paper does not isolate their contributions.

free parameters (5)
  • Loss weights alpha_k for 12 DAC layers = {5, 2, 1, 0.5, 0.5, 0.2, 0.2, 0.2, 0.1, 0.1, 0.1, 0.1}
    Chosen by hand; weight lower acoustic layers more heavily in the NLL loss (Eq. 3), affecting all generated audio.
  • CFG guidance strengths gamma, alpha, beta = gamma=1.5, alpha=1.3, beta=1.5
    Set at inference to amplify conditional guidance; no grid search or validation details reported.
  • LoRA rank r and alpha = r=16, alpha=16
    Rank and scaling for the text encoder adapters; standard choices, no tuning described.
  • Maximum sequence length = 2560
    Context window split as 512 text and 2048 token positions; truncates longer inputs and requires sentence segmentation at inference.
  • Training iterations = 770k (450k + 250k + 70k)
    Training schedule durations, chosen by the authors without reported early-stopping criteria.
assumptions (5)
  • domain assumption HuBERT semantic tokens and DAC acoustic tokens retain the prosodic and paralinguistic information needed for spontaneous speech synthesis.
    The whole pipeline depends on these discrete representations (Section 3.1); the paper does not verify whether spontaneous cues such as pitch variation and fillers survive discretization.
  • domain assumption MT5 text encoder with LoRA provides adequate conditioning for conversational Chinese text including disfluencies.
    Claimed in Section 4.3.2, but no comparison with phoneme-based or other text encoders is made.
  • ad hoc to paper The delay pattern is more suitable than non-delayed parallel prediction for modeling spontaneous speech prosody.
    Stated in Sections 1 and 3.2 with no experiment; the delay pattern was originally developed for music generation.
  • domain assumption Fine-tuning on high-quality spontaneous speech data improves spontaneous style without destroying zero-shot voice cloning.
    The paper acknowledges a trade-off between spontaneity and similarity (Section 4.3.2) but does not ablate the fine-tuning stage.
  • ad hoc to paper The preprocessing that removes laughter, noise, and overlapping speech does not remove the spontaneous behaviors the system is meant to synthesize.
    Section 2 explicitly removes laughter, which is one of the spontaneous paralinguistic phenomena commonly associated with natural conversational audio; this may suppress the target style.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Codec Language Model-based Zero-Shot Spontaneous Style TTS System for CoVoC Challenge 2024." pith.science (2026). https://pith.science/paper/CWTVEGEJ

@misc{pith2026241201100,
  author       = {Pith},
  title        = {Pith review of: The Codec Language Model-based Zero-Shot Spontaneous Style TTS System for CoVoC Challenge 2024},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CWTVEGEJ}},
  note         = {Machine review of arXiv:2412.01100}
}
read the original abstract

This paper describes the zero-shot spontaneous style TTS system for the ISCSLP 2024 Conversational Voice Clone Challenge (CoVoC). We propose a LLaMA-based codec language model with a delay pattern to achieve spontaneous style voice cloning. To improve speech intelligibility, we introduce the Classifier-Free Guidance (CFG) strategy in the language model to strengthen conditional guidance on token prediction. To generate high-quality utterances, we adopt effective data preprocessing operations and fine-tune our model with selected high-quality spontaneous speech data. The official evaluations in the CoVoC constrained track show that our system achieves the best speech naturalness MOS of 3.80 and obtains considerable speech quality and speaker similarity results.

Figures

Figures reproduced from arXiv: 2412.01100 by the authors.

Figure 1
Figure 1. The model architecture in training procedure. quently, we will provide a comprehensive description of these datasets and the data processing steps we have undertaken. WenetSpeech4TTS1 [15], a multi-domain Mandarin corpus, is derived from the open-sourced WenetSpeech dataset [16] and tailored for TTS tasks. The dataset is then divided into several subsets including Premium, Standard, Basic, and Rest based on data qua… view at source ↗
Figure 2
Figure 2. The mel-spectrograms of speeches synthesized by pro￾posed model. The texts of these two samples are “嗯没有诶， 如果你爬到过的话可以和我介绍一下” (English translation: “Um, no, I haven’t. If you’ve climbed there before, you can tell me about it.”) and “然后类似于啊这样的，嗯，不太满意的 体验，啊还有很多。” (English translation: “And then, there were experiences like, uh, this, um, not very satisfying, uh, and there were many more.”) respectively. 5. Conclusions … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages

  1. [1]

    On the one hand, TTS systems need to accurately replicate the target speakers’ voice, including their timbre, pitch, and prosody, using voice cloning techniques

    Introduction Text-to-speech (TTS) aims to create natural and human-like speech for given input text. On the one hand, TTS systems need to accurately replicate the target speakers’ voice, including their timbre, pitch, and prosody, using voice cloning techniques. On the other hand, these systems must be able to model sponta- neous speech elements like paus...

    work page 2024

  2. [2]

    The Codec Language Model-based Zero-Shot Spontaneous Style TTS System for CoVoC Challenge 2024

    Data Preparation The system we have submitted adheres to the constrained track, meaning our model was trained exclusively using the Wenet- Speech4TTS, MAGICDATA-RAMC, and HQ-Conversations datasets provided for this Challenge. The duration and basic information of these datasets are presented in Table 1. Subse- arXiv:2412.01100v2 [cs.SD] 4 Feb 2025 AT head...

    work page Pith review arXiv 2025

  3. [3]

    Firstly, we will overview the text represen- tations and the discrete speech tokens, and then introduce the LLaMA-based codec language model with a delay pattern

    System In this part we will describe our Conversational V oice Clone System in detail. Firstly, we will overview the text represen- tations and the discrete speech tokens, and then introduce the LLaMA-based codec language model with a delay pattern. Fi- nally, we will introduce the classifier-free guidance for codec language model. 3.1. Feature Representa...

    work page

  4. [4]

    Model Configurations We use the open-source models and parameters of MT5-base, DAC and HuBERT, with both DAC and HuBERT configured for a sampling rate of 16 kHz

    Experiments 4.1. Model Configurations We use the open-source models and parameters of MT5-base, DAC and HuBERT, with both DAC and HuBERT configured for a sampling rate of 16 kHz. The MT5-base text encoder comprises 12 transformer blocks with a hidden size of 768. We insert trainable LoRA adapters with parameters α = 16 and r = 16into query and value layer...

    work page

  5. [5]

    嗯” (En- glish translation: “um

    For the first sample in Figure 2(a), the duration of “ 嗯” (En- glish translation: “um”), as denoted by the red box, is distinctly longer than other words, indicating a spontaneous phenomenon where the speaker is thinking before speaking. For the second one in Figure 2(b), there is a noticeable hesitation between “ 然 后类似于” (English translation: “And then, ...

    work page

  6. [6]

    We propose a LLaMA-based codec language model with a delay pattern for spontaneous style voice cloning

    Conclusions This paper introduces our zero-shot spontaneous style speech synthesis system for the CoV oC Challenge 2024. We propose a LLaMA-based codec language model with a delay pattern for spontaneous style voice cloning. We use CFG to enhance condi- tional guidance on token prediction, increasing the intelligibil- ity of the speech content. We also ad...

    work page 2024

  7. [7]

    Acknowledgements This work is supported by National Natural Science Foun- dation of China (62076144), Shenzhen Key Laboratory of next generation interactive media innovative technol- ogy (ZDSYS20210623092001004) and Shenzhen Science and Technology Program (WDZC20220816140515001)

    work page

  8. [8]

    Deep voice 3: Scaling text- to-speech with convolutional sequence learning,

    W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller, “Deep voice 3: Scaling text- to-speech with convolutional sequence learning,” in International Conference on Learning Representations, 2018

    work page 2018

Show all 33 references
  1. [9]

    Neural voice cloning with a few samples,

    S. Arik, J. Chen, K. Peng, W. Ping, and Y . Zhou, “Neural voice cloning with a few samples,”Advances in neural information pro- cessing systems, vol. 31, 2018

    2018

  2. [10]

    Adaspeech: Adaptive text to speech for custom voice,

    M. Chen, X. Tan, B. Li, Y . Liu, T. Qin, T.-Y . Liu et al. , “Adaspeech: Adaptive text to speech for custom voice,” in In- ternational Conference on Learning Representations

  3. [11]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Neural codec language mod- els are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  4. [12]

    Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,

    Z. Zhang, L. Zhou, C. Wang, S. Chen, Y . Wu, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,”arXiv preprint arXiv:2303.03926, 2023

    2023 arXiv

  5. [13]

    Speechx: Neural codec language model as a versatile speech transformer,

    X. Wang, M. Thakker, Z. Chen, N. Kanda, S. E. Eskimez, S. Chen, M. Tang, S. Liu, J. Li, and T. Yoshioka, “Speechx: Neural codec language model as a versatile speech transformer,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

    2024

  6. [14]

    Ella-v: Stable neural codec language modeling with alignment-guided sequence reordering,

    Y . Song, Z. Chen, X. Wang, Z. Ma, and X. Chen, “Ella-v: Stable neural codec language modeling with alignment-guided sequence reordering,” arXiv preprint arXiv:2401.07333, 2024

    2024 arXiv

  7. [15]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438 , 2022

    2022 arXiv

  8. [16]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, vol. 30, pp. 495–507, 2021

    2021

  9. [17]

    Conversational end-to-end tts for voice agents,

    H. Guo, S. Zhang, F. K. Soong, L. He, and L. Xie, “Conversational end-to-end tts for voice agents,” in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 403–409

    2021

  10. [18]

    End-to-end text-to-speech based on latent representa- tion of speaking styles using spontaneous dialogue,

    K. Mitsui, T. Zhao, K. Sawada, Y . Hono, Y . Nankaku, and K. Tokuda, “End-to-end text-to-speech based on latent representa- tion of speaking styles using spontaneous dialogue,” Interspeech 2022, 2022

    2022

  11. [19]

    Spontts: modeling and transferring spontaneous style for tts,

    H. Li, X. Zhu, L. Xue, Y . Song, Y . Chen, and L. Xie, “Spontts: modeling and transferring spontaneous style for tts,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 12 171– 12 175

    2024

  12. [20]

    Controllable context-aware conversational speech synthesis,

    J. Cong, S. Yang, N. Hu, G. Li, L. Xie, and D. Su, “Controllable context-aware conversational speech synthesis,” arXiv preprint arXiv:2106.10828, 2021

    2021 arXiv

  13. [21]

    Spontaneous style text-to-speech synthesis with con- trollable spontaneous behaviors based on language models,

    W. Li, P. Yang, Y . Zhong, Y . Zhou, Z. Wang, Z. Wu, X. Wu, and H. Meng, “Spontaneous style text-to-speech synthesis with con- trollable spontaneous behaviors based on language models,”arXiv preprint arXiv:2407.13509, 2024

    2024 arXiv

  14. [22]

    Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model bench- mark,

    L. Ma, D. Guo, K. Song, Y . Jiang, S. Wang, L. Xue, W. Xu, H. Zhao, B. Zhang, and L. Xie, “Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model bench- mark,” arXiv preprint arXiv:2406.05763, 2024

    2024 arXiv

  15. [23]

    Wenetspeech: A 10000+ hours multi- domain mandarin corpus for speech recognition,

    B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng et al. , “Wenetspeech: A 10000+ hours multi- domain mandarin corpus for speech recognition,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)....

    2022

  16. [24]

    mt5: A massively multilingual pre-trained text-to-text transformer,

    L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Sid- dhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguist...

    2021

  17. [25]

    HuBERT: Self-Supervised Speech Rep- resentation Learning by Masked Prediction of Hidden Units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “HuBERT: Self-Supervised Speech Rep- resentation Learning by Masked Prediction of Hidden Units,” IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, vol. 29, pp. 3451–3460, 2021

    2021

  18. [26]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” Ad- vances in Neural Information Processing Systems , vol. 36, 2024

    2024

  19. [27]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  20. [28]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Ad- vances in Neural Information Processing Systems , vol. 35, pp. 16 344–16 359, 2022

    2022

  21. [29]

    Speak, read and prompt: High-fidelity text-to-speech with min- imal supervision,

    E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, “Speak, read and prompt: High-fidelity text-to-speech with min- imal supervision,” Transactions of the Association for Computa- tional Linguistics, vol. 11...

    2023

  22. [30]

    Simple and controllable music gen- eration,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music gen- eration,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  23. [31]

    Natural language guidance of high- fidelity text-to-speech with synthetic annotations,

    D. Lyth and S. King, “Natural language guidance of high- fidelity text-to-speech with synthetic annotations,” arXiv preprint arXiv:2402.01912, 2024

    2024 arXiv

  24. [32]

    Stay on topic with classifier-free guidance,

    G. Sanchez, H. Fan, A. Spangher, E. Levi, P. S. Ammanamanchi, and S. Biderman, “Stay on topic with classifier-free guidance,” arXiv preprint arXiv:2306.17806, 2023

    2023 arXiv

  25. [33]

    V oxinstruct: Expressive human instruction-to-speech generation with unified multilingual codec language modelling,

    Y . Zhou, X. Qin, Z. Jin, S. Zhou, S. Lei, S. Zhou, Z. Wu, and J. Jia, “V oxinstruct: Expressive human instruction-to-speech generation with unified multilingual codec language modelling,” in Proceed- ings of the 32st ACM International Conference on Multimedia , 2024

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.