Pith. sign in

REVIEW 5 major objections 4 minor 41 references

Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization

T0 review · 5 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Phoenix TTS jointly trains its speech tokenizer with a flow-matching decoder, reporting word error rates below ground truth and zero-shot voice conversion without fine-tuning.

desk verdict Genuinely new joint-training recipe with a real data-efficiency result, but the headline mechanism is under-supported by the ablations and needs a revision round. read the letter →

arxiv 2608.11737 v1 pith:VDHPDHEJ submitted 2026-08-12 cs.SD

Peijie Chen , Zhuanling Zha , Zhipeng Nie , Weijie Wu , Yiming Liu , Daiyu Huang , Junbo Li , Jun Fang
show 3 more authors
Naiqiang Tan Hua Chai Qingyang Hong
This is my paper · ORCID
classification cs.SD
keywords zero-shottext-to-speechspeechtokenizerflowmatchingvoiceconversionvectorquantizationself-supervisedfeaturesspeakersimilarityworderrorrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the discrete speech tokens used in zero-shot text-to-speech should be trained together with the acoustic model that consumes them rather than in isolation and then frozen. It proposes the UniSpeechTokenizer, which reconstructs self-supervised speech features to keep linguistic content while simultaneously receiving gradient supervision from a flow-matching decoder that maps tokens to continuous waveform latents. The intended payoff is that semantic content and acoustic detail coexist in the same tokens, closing the feature gap that usually caps speaker similarity. Trained on 110K hours, the system reports word error rates below those of ground-truth recordings on three benchmarks, speaker-similarity scores competitive with several larger baselines, and zero-shot voice conversion without any task-specific fine-tuning. A reader should care because this identifies a specific bottleneck and a single training signal aimed at it, with the voice-conversion result acting as a direct test of whether the tokens really separate meaning from timbre.

What carries the argument

The load-bearing mechanism is the joint reconstruction path through the UniSpeechTokenizer. Semantic features $x$ from a pre-trained self-supervised model are quantized into token embeddings $e_{\text{sem}}$; a Conformer decoder reconstructs $\hat{x}$ under the $L_1$ loss $L_{\text{feat}} = \| x - \hat{x} \|_1$. In parallel, a pre-trained Waveform VAE maps raw speech to continuous latents $z_{\text{vae}}$, and a binary mask yields $z_{\text{mask}} = M \odot z_{\text{vae}}$. The condition vector $c = (e_{\text{sem}}, z_{\text{mask}})$ is fed to the flow-matching decoder, which learns the vector field along the interpolation path $y_t = t y_1 + (1-t) y_0$ with $y_1 = z_{\text{vae}}$. The central design choice is an exact frame-rate match: 25 Hz discrete tokens and 25 Hz VAE latents, so the flow-matching gradient reaches every token without temporal resampling. This one-to-one alignment is what lets acoustic supervision reshape the codebook.

What would settle it

Train the same pipeline twice with identical data and losses, but block the flow-matching gradient from reaching the tokenizer in one run; if the reported WER and SIM on SeedTTS-EN do not degrade when the gradient is blocked, the claim that acoustic supervision reshapes the discrete tokens is directly refuted.

Watch

Extended reading notes

Core claim

The central claim is that the flow-matching loss is not just a decoder-side training objective but an active shaper of the tokenizer itself. In Phoenix TTS, the tokenizer's semantic path—frozen SSL features passed through a Conformer encoder, an 8192-codebook vector quantizer, and a Conformer decoder—is optimized to reconstruct the SSL features, while the flow-matching decoder is trained along a straight interpolation path toward the waveform-VAE latents $y_1 = z_{\text{vae}}$ starting from noise $y_0 \sim \mathcal{N}(0,I)$, with condition $c = (e_{\text{sem}}, z_{\text{mask}})$. The total loss $L_{\text{total}} = \lambda_{\text{FM}} L_{\text{FM}} + \lambda_{\text{VQ}} L_{\text{VQ}} + \lambda_{\text{feat}} L_{\text{feat}}$ back-propagates acoustic reconstruction error into the Conformer encoder and the vector quantizer. Because the pre-trained waveform VAE compresses 24 kHz audio to 25 Hz latents, matching the 25 Hz token rate one-to-one, every discrete token receives direct, temporally aligned acoustic supervision. On this basis the paper claims the reported WER and SIM scores, and the tokenizer's ability to perform zero-shot voice conversion without task-specific fine-tuning.

Load-bearing premise

The load-bearing premise is that the pre-trained Waveform VAE's 25 Hz latents correspond one-to-one to the 25 Hz semantic tokens, so the flow-matching gradient reaches each discrete token without temporal mismatch; if that frame-rate alignment fails, or the VAE discards speaker-relevant detail, the claimed acoustic alignment has not been demonstrated.

Editorial extensions

If this is right

  • Speech tokenizers trained with a generative decoder's gradient should generalize to unseen speakers better than frozen semantic tokenizers, because the tokens themselves learn to carry timbre and prosody.
  • Zero-shot voice conversion becomes a byproduct of the TTS pipeline: source content and a target voice prompt flow through the same tokenizer and flow-matching decoder without task-specific fine-tuning.
  • Word error rates below ground-truth recordings suggest the synthesis path can act as a regularizer, producing more transcription-stable output than the original audio.
  • The joint-training recipe should transfer to other continuous decoders, including diffusion or score-based generators, whenever the token and latent frame rates can be aligned.
  • Frame-rate alignment becomes a first-class design constraint: any codec with a different downsampling factor would need learned resampling or interpolation before it can deliver the same direct supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If the mechanism is real, the semantic-versus-acoustic trade-off in speech tokens is not fixed; the supervisory loss can steer how much timbre survives quantization, so a single tokenizer could be tuned for both ASR-style tasks and synthesis.
  • Inference: A testable extension would swap the flow-matching loss for a perceptual or adversarial loss with the same frame-aligned latents; comparable gains would indicate the benefit comes from decoder supervision in general, not from flow matching specifically.
  • Inference: The 25 Hz frame-rate requirement suggests a scaling path: higher-rate tokens would need correspondingly higher-rate waveform latents, and whether the approach extends to full-bandwidth or music codecs is an open question.
  • Inference: Because the same tokenizer performs zero-shot voice conversion without fine-tuning, it may also support related prompt-conditioned edits, such as emotion transfer or accent conversion, under the same unified training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes Phoenix TTS, a zero-shot text-to-speech system whose discrete speech tokenizer is jointly optimized with a continuous flow-matching decoder. The tokenizer quantizes W2v-BERT 2.0 layer-16 features via a Conformer encoder and a single-codebook VQ, reconstructs the SSL features with an L1 loss, and is simultaneously supervised by a flow-matching loss defined on a frame-rate-matched Waveform VAE latent space (Eqs. 3-4). An autoregressive Qwen2.5-0.5B model generates semantic tokens from text and a learned speaker embedding, and the flow-matching DiT maps the generated tokens to VAE latents for audio synthesis. The authors claim state-of-the-art WER and SIM on LibriSpeech-PC, SeedTTS-EN, and SeedTTS-ZH, and demonstrate zero-shot voice conversion without task-specific fine-tuning, all with a 110K-hour training corpus.

Significance. The central idea is attractive and timely: instead of freezing a semantic tokenizer in a cascaded hybrid TTS system, the paper couples the tokenizer to the downstream acoustic generation space by backpropagating a flow-matching loss into the quantizer. If the causal mechanism is established, the design principle could generalize to other hybrid TTS/VC systems. The manuscript has notable strengths: it evaluates on external benchmarks, compares with several strong baselines, includes ablation studies in Tables 4 and 6, and reports subjective scores with confidence intervals in Table 3. The frame-rate alignment premise is internally consistent with Table 2 (24 kHz audio downsampled by factors [2,4,4,5,6] yields 25 Hz latents, matching the Conformer's 2x downsample of 50 Hz W2v-BERT features). However, the load-bearing evidence for the 'native alignment' claim is incomplete: the direct tokenizer-decoder VC test shows only small gains from joint training, and the objective metrics lack significance testing. The headline result that the system exceeds ground-truth WER and SIM also needs a measurement-level explanation.

major comments (5)
  1. [Section 4.5, Table 6; Section 4.4, Table 4] The joint-training ablation is the primary evidence for the central mechanism, but it is confounded. In the full TTS pipeline, removing joint training changes the token sequence that the LLM must predict; the WER increase from 1.56 to 2.36 in Table 6 could be caused by the LLM finding the decoupled tokens harder to model, independent of any acoustic misalignment. The cleaner test, the tokenizer-decoder VC experiment in Table 4, shows much smaller differences: WER 2.54 vs. 2.70, SIM 0.697 vs. 0.682, UTMOS 4.081 vs. 4.036. These deltas do not strongly corroborate the claim that flow-matching gradients force the discrete tokens to 'natively align with the Flow Matching generation space.' Please report LLM perplexity on the token sequences and add missing ablation cells that isolate the tokenizer contribution (e.g., fixed decoder with and without joint tokenizer training, and fixed tokenizer with and without joint decoder training), or otherwise show that the full-pipeline gain is attributable to acoustic alignment rather than to token-sequence predictability.
  2. [Section 3.1, Eqs. (3)-(4)] The paper does not specify how the flow-matching loss is backpropagated through the non-differentiable vector quantization operation. Since the central claim is precisely that gradients from Eq. (3) shape the discrete tokens, the gradient path is load-bearing: if straight-through estimation or a soft assignment is used, it should be stated explicitly, along with any codebook commitment or EMA update details. Without this, the mechanism described in Section 3.1 is under-specified and not reproducible.
  3. [Tables 1, 4, 5, 6] No error bars, confidence intervals, or significance tests are reported for any objective metric. Many of the conclusions rest on small differences, such as Table 4 SIM 0.697 vs. 0.690 for ground truth, Table 6 SIM 0.720 vs. 0.706 for the joint-training ablation, and Table 1 WER deltas below 0.2 for several baselines. Without paired tests or repeated-evaluation statistics, these differences cannot be distinguished from evaluation noise. Please provide CIs or significance tests, at least for the key comparisons that support the paper's claims.
  4. [Tables 1 and 4] The claim that the model 'consistently' outperforms ground-truth recordings on WER and SIM is surprising and likely reflects an artifact of the evaluation metric rather than true perceptual superiority. For example, in Table 1 Phoenix TTS achieves WER 1.94 on LibriSpeech-PC vs. 2.06 for GT, and in Table 4 the VC SIM of 0.697 exceeds the GT SIM of 0.690. The paper should provide a measurement-level explanation (e.g., Whisper transcriptions being more favorable for synthesized audio due to speaker-normalized pronunciation, or WavLM-TDNN being biased by clean synthetic signals) or temper the claim. Without such analysis, these over-GT results are not credible evidence for the central hypothesis.
  5. [Section 4.4, Table 4] The 'w Mel Prompt' ablation changes two variables simultaneously: the conditioning feature type (VAE latents vs. mel-spectrogram) and the frame rate (25 Hz vs. 93.75 Hz). The sharp degradation in SIM (0.597) and UTMOS (2.806) is therefore not attributable solely to frame-rate mismatch, and the text 'proving that matching the tokenizer's frame rate with the latent space is essential' overstates the evidence. Please add an ablation using frame-rate-mismatched VAE latents (or mel features at 25 Hz) to isolate the temporal-alignment factor.
minor comments (4)
  1. [Section 4.3, Table 3] The subjective evaluation includes only three baselines (CosyVoice2, IndexTTS2, VoxCPM), while Table 1 compares many more systems; please state which baselines were included in the listening test and why.
  2. [Section 4.2] Please specify the exact evaluation protocol for WER and SIM, including whether multiple reference utterances are averaged, how the 95% confidence intervals in Table 3 were computed, and whether the same prompts are used across all baselines to ensure a fair comparison.
  3. [Section 4.1, Table 2] The Waveform VAE is described as following MegaTTS3, but the downsampling factors [2,4,4,5,6] are given only in Table 2; please clarify whether the VAE is pre-trained and frozen or fine-tuned during the joint training, since this affects the interpretation of the gradient feedback path.
  4. [Section 4.3] The statement that SIM 0.718 on LibriSpeech-PC is 'a new state-of-the-art' should be qualified, since VoxCPM in Table 1 reports 0.710 and the difference is within the range of unexplained evaluation noise; a direct statistical comparison would strengthen the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: joint-training claim is supported by external-benchmark evaluations and ablations; no fitted parameter is relabeled as a prediction.

full rationale

The paper's derivation chain is self-contained. The UniSpeechTokenizer is optimized with the semantic reconstruction loss Eq. (2), the flow-matching objective Eq. (3), and the weighted combination Eq. (4). The claim that gradients from the flow-matching decoder 'force the discrete tokens to natively align with the Flow Matching generation space' is a stated mechanism of the architecture, not a definition: the tokenizer outputs enter the flow-matching condition vector c=(e_sem, z_mask), so the objective explicitly couples them. The reported WER, SIM, and UTMOS values are external evaluations on LibriSpeech-PC, SeedTTS-EN/ZH, and Seed-TTS-Eval, not quantities derived from or fitted to the method's own parameters. The zero-shot voice conversion result is a direct application of the jointly trained tokenizer and flow decoder, and it is tested against a 'w/o joint training' ablation (Table 4), so it is not an input renamed as a prediction. The choice of a Waveform VAE whose downsampling factor 'perfectly matches the frame rate of the semantic tokens' is an architectural assumption about temporal alignment, but it is not circular: the paper does not derive this assumption from the claim it supports. The self-citations present (e.g., DS-Codec in a codec list and SARA in the reference list) are not load-bearing: the pre-trained VAE is attributed to MegaTTS3, and the central joint-training claims are evaluated independently against external benchmarks. Accordingly, no circular step can be exhibited by quoting equations or fitted values, and the score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The claims rest on the quality of the W2v-BERT layer-16 features, the MegaTTS3 waveform VAE's fidelity and frame alignment, the validity of automatic WER/SIM metrics, and the representativeness of the training corpus. The visible hand-set values are the loss weights in Eq. (4), the VQ codebook size, and the chosen SSL layer. No new physical entities are introduced.

free parameters (3)
  • Loss weighting coefficients lambda_FM, lambda_VQ, lambda_feat
    Equation (4) combines three losses; the weights are not reported and their balance is hand-chosen, affecting the joint-training result.
  • VQ codebook size = 8192
    Table 2 sets a single codebook of size 8192; this capacity decision is chosen by hand and is not swept.
  • W2v-BERT layer index for semantic features = 16
    Section 3.1 uses hidden states from layer 16; the layer choice is a modeling decision and is not justified by a comparison.
assumptions (5)
  • domain assumption Hidden states from layer 16 of W2v-BERT 2.0 are a sufficient semantic target for tokenizer reconstruction.
    Section 3.1 selects this layer without comparison; L_feat assumes these features carry the linguistic content needed for low WER.
  • domain assumption The pre-trained Waveform VAE from MegaTTS3 provides high-fidelity 25 Hz latents whose frame rate exactly matches the tokenizer's 25 Hz tokens.
    Section 3.1 and Table 2 rely on this one-to-one alignment; if the VAE is lossy or misaligned, the flow-matching gradient may not align tokens with the acoustic space.
  • domain assumption WER measured by Whisper-large-v3 and Paraformer, and SIM by WavLM-TDNN, are valid proxies for intelligibility and speaker similarity.
    Section 4.2 uses these automatic metrics as primary evidence; no calibration against human perception is shown for the objective tables.
  • standard math Flow matching with linear interpolation y_t = t y_1 + (1-t) y_0 is a valid generative objective.
    Equation (3) uses a standard flow-matching target; this is background methodology, not contested.
  • domain assumption The 110K-hour corpus, public plus proprietary audiobooks, is representative enough for zero-shot generalization to SeedTTS and LibriSpeech-PC test sets.
    Section 4.1 describes the corpus; proprietary data are not detailed, so domain shift cannot be fully assessed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization." pith.science (2026). https://pith.science/paper/VDHPDHEJ

@misc{pith2026260811737,
  author       = {Pith},
  title        = {Pith review of: Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VDHPDHEJ}},
  note         = {Machine review of arXiv:2608.11737}
}
read the original abstract

In current zero-shot text-to-speech systems, conventional semantic tokenizers are typically optimized using supervised automatic speech recognition or self-supervised learning objectives. However, due to the inherent nature of speech, semantic and acoustic information cannot be completely decoupled, and ASR-based tokenizers discard acoustic details to focus on linguistic content; models relying on them usually struggle to achieve optimal speaker similarity. Furthermore, these tokenizers are optimized independently and lack direct supervision from downstream acoustic generation tasks. This isolated training creates a feature gap between the extracted discrete tokens and the continuous space required by acoustic models, fundamentally bottlenecking the upper bound of synthesis quality. To bridge this gap, we propose Phoenix TTS, a unified framework that tightly couples representation learning with generative acoustic modeling. Specifically, our speech tokenizer is optimized to reconstruct self-supervised features to maintain semantic richness, while simultaneously receiving direct supervision from a Flow Matching training loss. Through this joint training paradigm, the extracted discrete tokens successfully preserve essential semantic information and natively align with the feature space of the downstream Flow Matching model. Comprehensive evaluations highlight the efficiency and effectiveness of Phoenix TTS. Trained on 110K hours of data, the system achieves excellent speech intelligibility, yielding WER that consistently falls below that of ground-truth recordings. Simultaneously, it maintains robust zero-shot speaker similarity, rivaling or outperforming several prominent large-scale baselines. Furthermore, as an advantageous byproduct of this unified training, the learned tokenizer can be seamlessly adapted to zero-shot voice conversion tasks without requiring task-specific fine-tuning.

Figures

Figures reproduced from arXiv: 2608.11737 by the authors.

Figure 1
Figure 1. UnispeechTokenizer optimized via L1 loss: Lf eat = ∥x − ˆx∥1 (2) Crucially, to effectively map these discrete semantic embeddings back into the high-fidelity acoustic generation space, we introduce a novel reconstruction paradigm utilizing a Flow Matching decoder. Parallel to the semantic extraction path, a pre-trained Waveform VAE, which follows from MegaTTS3 [22], encodes the raw speech s into continuous acoustic … view at source ↗
Figure 2
Figure 2. A schematic diagram of Phoenix TTS’s autoregressive language modeling part. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 5 canonical work pages

  1. [5]

    Sara: A dual-stream vae for high-fidelity speech generation via integrating semantic and acoustic representations.arXiv preprint arXiv:2606.11611, 2026

    Peijie Chen, Wenhao Guan, Weijie Wu, Kaidi Wang, Daiyu Huang, Zhuanling Zha, Junbo Li, Jun Fang, Qingyang Hong, and Lin Li. Sara: A dual-stream vae for high-fidelity speech generation via integrating semantic and acoustic representations.arXiv preprint arXiv:2606.11611, 2026

    arXiv 2026

  2. [1]

    Seed-tts: A family of high-quality versatile speech generation models.arXiv preprint arXiv:2406.02430, 2024

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models.arXiv preprint arXiv:2406.02430, 2024

    arXiv 2024

  3. [2]

    Xtts: a massively multilingual zero-shot text-to-speech model.arXiv preprint arXiv:2406.04904, 2024

    Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. Xtts: a massively multilingual zero-shot text-to-speech model.arXiv preprint arXiv:2406.04904, 2024

    arXiv 2024

  4. [3]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio.arXiv preprint arXiv:2106.06909, 2021

    Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio.arXiv preprint arXiv:2106.06909, 2021

    arXiv 2021

  5. [4]

    Ds-codec: Dual-stage training with mirror-to-nonmirror architecture switching for speech codec

    Peijie Chen, Wenhao Guan, Kaidi Wang, Weijie Wu, Hukai Huang, Qingyang Hong, and Lin Li. Ds-codec: Dual-stage training with mirror-to-nonmirror architecture switching for speech codec. InProc. Interspeech 2025, pages 4908–4912, 2025

    work page 2025

  6. [6]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre- training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022. 11

    work page 2022

  7. [7]

    Neural codec language models are zero-shot text to speech synthesizers.IEEE Transactions on Audio, Speech and Language Processing, 33:705–718, 2025

    Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers.IEEE Transactions on Audio, Speech and Language Processing, 33:705–718, 2025

    2025

  8. [8]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, JianZhao JianZhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6255–6271, 2025

    2025

Show all 41 references
  1. [9]

    W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training

    Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU),...

    2021

  2. [10]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  3. [11]

    Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    2025 arXiv

  4. [12]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117, 2024

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117, 2024

    2024 arXiv

  5. [13]

    High fidelity neural audio compression.arXiv preprint arXiv:2210.13438, 2022

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression.arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  6. [14]

    E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts

    Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. In2024 IEEE spoken language technology workshop (SLT), pages 682–689. ...

    2024

  7. [15]

    Funasr: A fundamental end-to-end speech recognition toolkit.arXiv preprint arXiv:2305.11013, 2023

    Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al. Funasr: A fundamental end-to-end speech recognition toolkit.arXiv preprint arXiv:2305.11013, 2023

    2023 arXiv

  8. [16]

    Conformer: Convolution-augmented transformer for speech recognition.arXiv preprint arXiv:2005.08100, 2020

    Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmented transformer for speech recognition.arXiv preprint arXiv:2005.08100, 2020

    2005 arXiv

  9. [17]

    Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024

    Hao-Han Guo, Yao Hu, Kun Liu, Fei-Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024

    2024 arXiv

  10. [18]

    Didispeech: A large scale mandarin speech corpus

    Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. Didispeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)...

    2021

  11. [19]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In2024 IEEE Spoken Language Technology Workshop (SLT)...

    2024

  12. [20]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing, 29:3...

    2021

  13. [21]

    Ditar: Diffusion transformer autoregressive modeling for speech generation.arXiv preprint arXiv:2502.03930, 2025

    Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, et al. Ditar: Diffusion transformer autoregressive modeling for speech generation.arXiv preprint arXiv:2502.03930, 2025

    2025

  14. [22]

    Megatts 3: Sparse alignment enhanced latent diffusion transformer for zero-shot speech synthesis.arXiv preprint arXiv:2502.18924, 2025

    Ziyue Jiang, Yi Ren, Ruiqi Li, Shengpeng Ji, Boyang Zhang, Zhenhui Ye, Chen Zhang, Bai Jionghao, Xiaoda Yang, Jialong Zuo, et al. Megatts 3: Sparse alignment enhanced latent diffusion transformer for zero-shot speech synthesis.arXiv preprint arXiv:2502.18924, 2025

    2025 arXiv

  15. [23]

    Libriheavy: A 50,000 hours asr corpus with punctuation casing and context

    Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: A 50,000 hours asr corpus with punctuation casing and context. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASS...

    2024

  16. [24]

    High- fidelity audio compression with improved rvqgan.Advances in Neural Information Processing Systems, 36:27980–27993, 2023

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan.Advances in Neural Information Processing Systems, 36:27980–27993, 2023

    2023

  17. [25]

    Ditto-tts: Diffusion transformers for scalable text-to-speech without domain-specific factors.arXiv preprint arXiv:2406.11427, 2024

    Keon Lee, Dong Won Kim, Jaehyeon Kim, Seungjun Chung, and Jaewoong Cho. Ditto-tts: Diffusion transformers for scalable text-to-speech without domain-specific factors.arXiv preprint arXiv:2406.11427, 2024

    2024 arXiv

  18. [26]

    Zero-shot voice conversion with diffusion transformers.arXiv preprint arXiv:2411.09943, 2024

    Songting Liu. Zero-shot voice conversion with diffusion transformers.arXiv preprint arXiv:2411.09943, 2024

    2024 arXiv

  19. [27]

    Autoregressive diffusion transformer for text-to-speech synthesis.arXiv preprint arXiv:2406.05551, 2024

    Zhijun Liu, Shuai Wang, Sho Inoue, Qibing Bai, and Haizhou Li. Autoregressive diffusion transformer for text-to-speech synthesis.arXiv preprint arXiv:2406.05551, 2024

    2024 arXiv

  20. [28]

    Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model benchmark.arXiv preprint arXiv:2406.05763, 2024

    Linhan Ma, Dake Guo, Kun Song, Yuepeng Jiang, Shuai Wang, Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, and Lei Xie. Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model benchmark.arXiv preprint arXiv:2406.05763, 2024

    2024 arXiv

  21. [29]

    Librispeech-pc: Benchmark for evaluation of punctuation and capitaliza- tion capabilities of end-to-end asr models

    Aleksandr Meister, Matvei Novikov, Nikolay Karpov, Evelina Bakhturina, Vitaly Lavrukhin, and Boris Ginsburg. Librispeech-pc: Benchmark for evaluation of punctuation and capitaliza- tion capabilities of end-to-end asr models. In2023 IEEE automatic speech recognition and underst...

    2023

  22. [30]

    Autoregressive speech synthesis without vector quantization

    Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. Autoregressive speech synthesis without vector quantization. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V...

    2025

  23. [31]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015

  24. [32]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  25. [33]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  26. [34]

    V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis.arXiv preprint arXiv:2306.00814, 2023

    Hubert Siuzdak. V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis.arXiv preprint arXiv:2306.00814, 2023

    2023 arXiv

  27. [35]

    Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens.arXiv preprint arXiv:2503.01710, 2025

    Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens.arXiv preprint arXiv:2503.01710, 2025

    2025 arXiv

  28. [36]

    Maskgct: Zero-shot text-to-speech with masked generative codec transformer.arXiv preprint arXiv:2409.00750, 2024

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskgct: Zero-shot text-to-speech with masked generative codec transformer.arXiv preprint arXiv:2409.00750, 2024. 13

    2024 arXiv

  29. [37]

    Bigcodec: Pushing the limits of low-bitrate neural speech codec.arXiv preprint arXiv:2409.05377, 2024

    Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low-bitrate neural speech codec.arXiv preprint arXiv:2409.05377, 2024

    2024 arXiv

  30. [38]

    Speechtokenizer: Unified speech tokenizer for speech large language models.arXiv preprint arXiv:2308.16692, 2023

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models.arXiv preprint arXiv:2308.16692, 2023

    2023 arXiv

  31. [39]

    X-vc: Zero-shot streaming voice conversion in codec space

    Qixi Zheng, Yuxiang Zhao, Tianrui Wang, Wenxi Chen, Kele Xu, Yikang Li, Qinyuan Chen, Xipeng Qiu, Kai Yu, and Xie Chen. X-vc: Zero-shot streaming voice conversion in codec space. arXiv preprint arXiv:2604.12456, 2026

    2026 arXiv

  32. [40]

    Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech

    Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, and Jingchen Shu. Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, ...

    2026

  33. [41]

    V oxcpm: Tokenizer-free tts for context-aware speech generation and true-to-life voice cloning.arXiv preprint arXiv:2509.24650, 2025

    Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Ziyang Wang, Runchuan Ye, Weiyue Sun, Jiancheng Gui, Kehan Li, et al. V oxcpm: Tokenizer-free tts for context-aware speech generation and true-to-life voice cloning.arXiv preprint arXiv:2509.24650, 2025. 14

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.