Pith. sign in

REVIEW 3 major objections 4 minor 70 references

Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning zero-shot TTS on a 250K-pair preference dataset cuts word errors on tongue twisters, code-switching, and cross-lingual speech, and transfers to stronger models that never saw the data.

desk verdict Useful dataset and DPO recipe for TTS intelligibility, but the code-switching win is probably inflated by a Mandarin-only ASR scoring both the labels and final results. read the letter →

arxiv 2505.04113 v2 pith:AU7NTGP5 submitted 2025-05-07 cs.SD eess.AS

Xueyao Zhang , Yuancheng Wang , Chaoren Wang , Ziniu Li , Zhuo Chen , Zhizheng Wu This is my paper · ORCID
classification cs.SDeess.AS
keywords zero-shottext-to-speechintelligibilitypreferencealignmentdirectoptimizationspeechdatasetcode-switchingcross-lingualsynthesisweak-to-stronggeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern zero-shot text-to-speech systems read ordinary sentences fluently but frequently garble tongue twisters, repeated words, code-switched text, and cross-lingual speech. This paper's claim is that those failures can be treated as a preference-learning problem: build a large dataset whose "good" and "bad" speech clips are ranked by word error rate plus deliberately crafted human-guided errors, then fine-tune a TTS model with Direct Preference Optimization adapted to its architecture. The authors report that after this alignment, five different TTS models — including two stronger models that never contributed data to the dataset — improve on intelligibility, naturalness, and overall audio quality across all four difficult domains. They also report that a second round of alignment on data generated by the aligned model yields further, smaller gains, and that the preference dataset doubles as a tool for ranking competing TTS models. If correct, the approach gives a cheap, data-only recipe for fixing failure modes that current large-scale pre-training leaves unresolved.

What carries the argument

The load-bearing object is INTP, a preference dataset of roughly 251K pairs constructed by three complementary routes: intra-pairs (several samples of the same model on the same prompt, ranked by word error rate), inter-pairs (the same prompt synthesized by different models, so their complementary strengths can be pooled and, as a side effect, an "intelligibility arena" that ranks models), and perturbed pairs, in which the target text itself is deliberately corrupted — misspelled into near-homophones or re-punctuated to shift meaning — to create human-guided negative samples. The optimization machinery is Direct Preference Optimization (DPO), which converts preference pairs into a policy-gradient-free loss using the closed form of the RLHF objective; the paper derives two new variants, DPO-FM for flow-matching models, expressed as a velocity-field difference on optimal-transport interpolation paths, and DPO-MGM for masked generative models, expressed over the log-probabilities of unmasked tokens. Word error rate from Whisper-large-v3 (English) and Paraformer-zh (Chinese and code-switching) is the ranking signal that turns raw audio into preferences.

What would settle it

Take a stratified sample of the roughly 251K INTP pairs — especially code-switching and cross-lingual ones — and have native listeners (or a second, independent recognizer) transcribe both members of each pair; if a sizable share of pairs reverse their word-error ranking, the preference labels are partly recognizer noise and the intelligibility claim is weakened. A second check: rebuild the preference pairs using only CosyVoice 2's own outputs and realign it; if the gains vanish, the transfer depends on cross-model diversity rather than on the preference signal itself.

Watch

Extended reading notes

Core claim

The paper's central claim is that the intelligibility failures of zero-shot TTS in hard domains are an out-of-distribution problem that preference alignment can repair without new pre-training data. Concretely, it claims that Direct Preference Optimization fine-tuned on INTP — about 251,000 triples of (prompt, preferred speech, dispreferred speech) spanning regular, repeated, code-switching, pronunciation-perturbed, and punctuation-perturbed Chinese and English texts — lowers word error rate while also improving naturalness (N-CMOS) and overall quality (UTMOS) for autoregressive, flow-matching, and masked-generative models, both for the three models that generated the data (ARS, F5-TTS, MaskGCT) and for two stronger held-out models (CosyVoice 2, Ints). The improvements on the held-out models are presented as weak-to-strong generalization: preference data harvested from weaker models transfers to stronger ones. A final claim is that iterative alignment — letting the aligned Ints model generate a fresh round of preference pairs — produces further gains, with a smaller second-round effect that the authors attribute to the base model's inherent ceiling.

Load-bearing premise

The whole method rests on assuming that two automatic speech recognizers' word-error counts correctly say which of two clips is more intelligible — an assumption checked by human listeners on only 300 of about 251,000 pairs, so if the recognizers mis-hear accented or code-switched speech in a patterned way, the dataset's preferred and dispreferred labels silently inherit that error.

Editorial extensions

If this is right

  • Any zero-shot TTS model — autoregressive, flow-matching, or masked-generative — can be made more intelligible on hard domains by fine-tuning on a preference dataset, with no change to its architecture.
  • Preference datasets do not need to be built with the strongest available model; data from weaker models transfers to stronger ones, lowering the cost of dataset construction.
  • Iterative alignment provides a data-and-model flywheel: the aligned model generates the next round of preference pairs, yielding further gains until the base model's own capability becomes the bottleneck.
  • Failures are domain-specific — repeated-word data helps articulatory cases, pronunciation-perturbed data helps cross-lingual cases — so post-training data can be targeted instead of generic, and combining all data types gives the best overall result.
  • Because alignment also raised naturalness and kept speaker similarity roughly constant, intelligibility preference data does not appear to trade off against other quality dimensions.
  • The inter-pair construction also yields an intelligibility ranking of models that agrees with human reading-accuracy judgments, so the dataset doubles as an evaluation tool.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same pipeline applies to other failure modes — rare proper nouns, technical jargon, low-resource language pairs — but the WER-based ranking signal would need re-validation there, since a recognizer that cannot spell a word will mislabel both sides of a pair.
  • A consequence of the design worth testing: because WER ranks both members of every pair, the ceiling on dataset quality is set by the recognizers' own blind spots; an audit that re-transcribes a stratified sample with human listeners would quantify how much of the measured gain is preference signal versus recognizer bias.
  • The inter-pair arena could double as a continuously updated public benchmark for TTS intelligibility rather than a one-off training set, since its model rankings already align with human perception.
  • Reading the diminishing second-round gains as a capacity ceiling suggests the next bottleneck is the base model itself; combining INTP-style alignment with pre-training that actually includes the hard domains would likely outperform either alone — a combination this paper does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes preference alignment to improve zero-shot text-to-speech (TTS) intelligibility across regular, articulatory, code-switching, and cross-lingual domains. It introduces INTP, a synthetic preference dataset of roughly 250K pairs built from ARS, F5-TTS, and MaskGCT, with preference pairs constructed via intra-model sampling, inter-model comparisons, and human-guided prompt perturbations, all labeled using WER. The authors extend DPO to autoregressive, flow-matching, and masked-generative TTS architectures and report WER, SIM, UTMOS, N-CMOS, and subjective reading-accuracy improvements for the three INTP models and for two stronger models (CosyVoice 2 and Ints) not used in INTP construction, including an iterative alignment round for Ints. The central empirical claims are summarized in Table 4 and Figure 2.

Significance. If the results hold, the contribution is significant: the paper provides a large public preference dataset for TTS intelligibility, demonstrates DPO variants for non-autoregressive TTS architectures, and presents evidence of weak-to-strong generalization and iterative alignment. The release of the dataset, code, models, and evaluation sets, together with machine-checkable preference data, is a clear strength. The human reading-accuracy check on regular texts (Table 3) supports the validity of WER-based preferences in that domain, and the consistent objective and subjective trends for regular, articulatory, and cross-lingual cases are encouraging. The main unverified risk is the code-switching WER instrument issue, which directly affects one of the paper's headline domains.

major comments (3)
  1. [Section 5, Table 4, Appendix F.2] The code-switching WER numbers are produced by Paraformer-zh, a Mandarin-centric ASR, on English-Chinese mixed text, and the same instrument is used both to label INTP preferences and to evaluate the aligned models. On mixed-language text, this ASR systematically misrecognizes or deletes English tokens, so the high baseline WERs (e.g., ARS 54.15, F5-TTS 33.99 in Table 4) include a large fixed instrument-error component, and alignment can reduce WER by making English segments more Paraformer-friendly without improving human intelligibility. The claimed code-switching gains (e.g., F5-TTS 33.99 to 15.98, ARS 54.15 to 36.91, CosyVoice 2 33.36 to 28.31) are therefore not established as genuine intelligibility improvements. Because the human verification in Section 3.4 is explicitly on INTP regular texts only, this domain currently has no human ground truth. Please add a human reading-accuracy evaluation on a code-switching subset, or re-measure with an ASR that handles mixed-language input, and report the preference-labeling and evaluation ASRs separately.
  2. [Sections 3.3-3.4] WER-based preference labels are human-verified only for regular texts (Table 3), while the same ASR models used to rank positive and negative samples in INTP are also used to score the final models in Table 4 for every domain. For articulatory, code-switching, and cross-lingual data, there is no per-domain human verification and no analysis of how ASR label noise propagates into the DPO training signal. This is load-bearing for the paper's diverse-domains claim because it leaves open the possibility that alignment optimizes ASR-specific artifacts rather than intelligibility in these non-regular domains. A human verification on a few hundred pairs in each non-regular domain, even if smaller than the regular-text verification, would directly address this risk.
  3. [Section 5.1 and Appendix F.3] The subjective evaluation reporting is internally inconsistent: Section 5.1 states that we randomly sample 300 samples for subjective evaluation, while Appendix F.3 describes 10 samples per language per setting, yielding 80 pairs per model and 400 pairs across the five systems. This mismatch prevents the reader from knowing how many listening judgments actually underlie Figure 2. With roughly 80 samples per model pooled across four domains, the reading-accuracy results in Figure 2a are also too coarse to confirm domain-specific claims, especially the code-switching claim. Please reconcile the sample counts and, if possible, report per-domain reading-accuracy values with uncertainty estimates.
minor comments (4)
  1. [Table 4 caption] The N-CMOS column shows values only for the aligned models and '-' for the base models, which is ambiguous; please state explicitly that N-CMOS is a paired comparison of the aligned model against its base model and describe the sign convention.
  2. [Appendix A.1] The text says the scripts for repeated-text and punctuation-perturbation construction will be included in the future open-source repository, which conflicts with Section 1's claim that all resources are already open-sourced; if the scripts are available, please provide the link.
  3. [Table 2] The TTS Intelligibility Arena table is hard to read because the Win Rate column sums row percentages but the diagonal entries are omitted; please add a short description of how the win rates are computed from the 10K inter-pairs and what the percentages in each cell mean (row versus column win rates).
  4. [Figure 2] Figure 2a reports reading accuracy without error bars or confidence intervals; given the small sample size, please add uncertainty estimates or state the number of judgments per bar.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DPO/INTP derivation is self-contained and the key intelligibility claims are checked on held-out SeedTTS-based evaluation sets, with independent human and unseen-language verification.

full rationale

The paper's derivation chain is not circular under the stated standards. Preference pairs in INTP are constructed using WER-based ranking plus human-guided perturbed negatives, and the DPO/DPO-FM/DPO-MGM losses are standard extensions of Rafailov et al. (2023) and Wallace et al. (2024) with no equation reducing a claimed output to its input. The evaluation WER uses Whisper-large-v3 and Paraformer-zh, which are also used to label WER-based pairs, creating a mild risk of optimizing ASR-specific errors, especially for Paraformer-zh on code-switching texts. However, this is not a constructional circularity: DPO optimizes preference likelihood rather than WER directly, the evaluation prompts are held-out (SeedTTS-based) rather than INTP prompts (Emilia-based), and the paper supplies independent human reading-accuracy verification (Table 3 and Figure 2a) and unseen-language results (Table 10) that do not reduce to the INTP labels. No load-bearing premise is justified solely by a self-citation, and no uniqueness or ansatz is imported from prior work by the same authors. Therefore no circular step is established.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central claim rests on a hand-chosen WER threshold for pair filtering, per-model DPO beta values, and tuning of sampling hyperparameters. The main domain assumption is that ASR WER is a faithful proxy for human intelligibility across code-switching and cross-lingual speech, an assumption the paper verifies only on a small subset. The DPO math is standard, and no new physical entities or force-like entities are introduced.

free parameters (4)
  • WER threshold = 6.0
    Pairs with WER gap below 6.0 are discarded in intra and inter pair construction (Appendix A.3.1, A.3.2). The threshold is a hand-chosen value that controls the difficulty and size of the dataset.
  • Sampling temperatures and durations = temperatures 0.4, 0.6, 0.8, 1.0, 1.2; durations 0.8d, 0.9d, 1.0d, 1.1d, 1.2d
    Used to create diverse samples within intra pairs (Appendix A.3.1). These hyperparameters are chosen manually to enlarge the WER gap and are not derived from theory.
  • DPO beta per architecture = 0.1 (ARS, CosyVoice 2, Ints), 1000 (F5-TTS), 10 (MaskGCT)
    The KL regularization coefficient in DPO is tuned per model in Appendix D. Different beta values reflect different likelihood scales across architectures, but they are free hyperparameters.
  • One epoch = 1
    Training time is set to one epoch with a chosen learning rate; this is a hyperparameter choice that affects the final performance.
assumptions (4)
  • domain assumption WER from ASR is a valid reward proxy for intelligibility
    The preference label in intra/inter pairs is determined solely by ASR WER. If ASR errors are correlated with speaker or acoustic conditions rather than with true articulation errors, the preference signal is biased.
  • domain assumption Human perception agrees with WER-based ranking on the full dataset
    Human verification is only performed on a 300-pair subset for reading accuracy and 1000 pairs for naturalness. The paper assumes the same agreement extends to all 250K pairs.
  • standard math The closed-form DPO derivation applies verbatim to flow-matching and MGM
    The paper adapts the DPO derivation from Rafailov et al. (2023) and Wallace et al. (2024). It assumes that the partition function Z(x) cancels in the ratio, which is standard but requires that the policy and reference are in the same family with matching normalization.
  • domain assumption Emilia-Large text is a representative distribution for real-world TTS prompts
    Prompt texts are sampled from Emilia-Large (Appendix A.1), which is a large but specific corpus. The paper assumes this covers the 'diverse domains' claimed, though rare languages and specialized jargon are outside its scope.
invented entities (2)
  • INTP dataset independent evidence
    purpose: A preference dataset for intelligibility post-training
    The dataset is a new artifact that the paper releases; it is not a theoretical entity. Its independent evidence is the evaluation results and the public hosting on Hugging Face.
  • Ints independent evidence
    purpose: An intelligibility-enhanced speech language model used as a strong base for weak-to-strong and iterative alignment
    Ints is a model contributed by this group (Appendix C). Its independent evidence is the released model and the evaluations on unseen languages, which are outside the training data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment." pith.science (2026). https://pith.science/paper/AU7NTGP5

@misc{pith2026250504113,
  author       = {Pith},
  title        = {Pith review of: Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AU7NTGP5}},
  note         = {Machine review of arXiv:2505.04113}
}
read the original abstract

Modern zero-shot text-to-speech (TTS) systems, despite using extensive pre-training, often struggle in challenging scenarios such as tongue twisters, repeated words, code-switching, and cross-lingual synthesis, leading to intelligibility issues. To address these limitations, this paper leverages preference alignment techniques, which enable targeted construction of out-of-pretraining-distribution data to enhance performance. We introduce a new dataset, named the Intelligibility Preference Speech Dataset (INTP), and extend the Direct Preference Optimization (DPO) framework to accommodate diverse TTS architectures. After INTP alignment, in addition to intelligibility, we observe overall improvements including naturalness, similarity, and audio quality for multiple TTS models across diverse domains. Based on that, we also verify the weak-to-strong generalization ability of INTP for more intelligible models such as CosyVoice 2 and Ints. Moreover, we showcase the potential for further improvements through iterative alignment based on Ints. Audio samples are available at https://intalign.github.io/.

Figures

Figures reproduced from arXiv: 2505.04113 by the authors.

Figure 1
Figure 1. Three kinds of preference pairs in INTP. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Subjective evaluation of intelligibility and speaker similarity for models before and after INTP alignment. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. User interface for intelligibility and natural [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: User interface for speaker similarity evalua [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 33 canonical work pages

  1. [1]

    Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, S \' e bastien Bubeck, Martin Cai, Caio C \' e sar Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dix...

    arXiv 2024

  2. [2]

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, J...

    arXiv 2024

  3. [3]

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670

    arXiv 2019

  4. [4]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson...

    arXiv 2022

  5. [5]

    Zal \' a n Borsos, Rapha \" e l Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2023 a . Audiolm: A language modeling approach to audio generation. IEEE ACM Trans. Audio Speech Lang. Process. , 31:2523--2533

    work page 2023

  6. [6]

    Zal \' a n Borsos, Matthew Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. 2023 b . Soundstorm: Efficient parallel audio generation. arXiv preprint, abs/2305.09636

    arXiv 2023

  7. [7]

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeffrey Wu. 2024. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In ICML . OpenReview.net

    work page 2024

  8. [8]

    Chen Chen, Yuchen Hu, Wen Wu, Helin Wang, Eng Siong Chng, and Chao Zhang. 2024 a . Enhancing zero-shot text-to-speech synthesis with human feedback. arXiv preprint, abs/2406.00654

    arXiv 2024

Show all 70 references
  1. [9]

    Jingyi Chen, Ju-Seung Byun, Micha Elsner, and Andrew Perrault. 2024 b . Dlpo: Diffusion model loss-guided reinforcement learning for fine-tuning text-to-speech diffusion models. arXiv preprint, abs/2405.14632

    2024 arXiv

  2. [10]

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, ...

    2022

  3. [11]

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. 2024 c . F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint, abs/2410.06885

    2024 arXiv

  4. [12]

    Geoffrey Cideron, Sertan Girgin, Mauro Verzetti, Damien Vincent, Matej Kastelic, Zal \' a n Borsos, Brian McWilliams, Victor Ungureanu, Olivier Bachem, Olivier Pietquin, Matthieu Geist, L \' e onard Hussenot, Neil Zeghidour, and Andrea Agostinelli. 2024. Musicrl: Aligning musi...

    2024

  5. [13]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344--16359

    2022

  6. [14]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025

  7. [15]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek - AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei...

    2024 arXiv

  8. [16]

    Alexandre D \' e fossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2023. High fidelity neural audio compression. Trans. Mach. Learn. Res., 2023

    2023

  9. [17]

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. 2024 a . Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint, abs/2407.05407

    2024 arXiv

  10. [18]

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. 2024 b . Cosyvoice 2: Scalable streaming speech s...

    2024 arXiv

  11. [19]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 arXiv

  12. [20]

    Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung - Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, Yanqing Liu, Sheng Zhao, and Naoyuki Kanda. 2024. E2 TTS: embarrassingly easy fully non-autoregressive zero-shot TTS . In SLT . IEEE

    2024

  13. [21]

    Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. 2024. TLDR: token-level detective reward model for large vision language models. arXiv preprint, abs/2410.04734

    2024 arXiv

  14. [22]

    Xiaoxue Gao, Chen Zhang, Yiming Chen, Huayun Zhang, and Nancy F. Chen. 2024. Emo-dpo: Controllable emotional speech synthesis through direct preference optimization. arXiv preprint, abs/2409.10157

    2024 arXiv

  15. [23]

    Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, and Shiliang Zhang. 2023. Funasr: A fundamental end-to-end speech recognition toolkit. In INTERSPEECH , pages 1593--1597. ISCA

    2023

  16. [24]

    Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. 2022. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. In INTERSPEECH , pages 2063--2067. ISCA

    2022

  17. [25]

    Hao-Han Guo, Kun Liu, Fei-Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. 2024. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint, abs/2409.03283

    2024 arXiv

  18. [26]

    Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. 2021. Didispeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (I...

    2021

  19. [27]

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. 2024. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generat...

    2024

  20. [28]

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. 2025. Emilia: A large-scale, extensive, multilingual, and diverse dataset for speech generation. arXiv preprint, 2501.15907

    2025

  21. [29]

    Yuchen Hu, Chen Chen, Siyin Wang, Eng Siong Chng, and Chao Zhang. 2024. Robust zero-shot text-to-speech synthesis with reverse inference optimization. arXiv preprint, abs/2407.02243

    2024 arXiv

  22. [30]

    Desta, Roy Fejgin, Rafael Valle, and Jason Li

    Shehzeen Hussain, Paarth Neekhara, Xuesong Yang, Edresson Casanova, Subhankar Ghosh, Mikyas T. Desta, Roy Fejgin, Rafael Valle, and Jason Li. 2025. Koel-tts: Enhancing llm based speech generation with preference alignment and classifier free guidance. arXiv preprint, abs/2502.05236

    2025 arXiv

  23. [31]

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. 2024. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. In Forty-first International Conference on Machine Learning

    2024

  24. [32]

    Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazar \'e , Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. 2020. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE...

    2020

  25. [33]

    Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei - Ning Hsu. 2023. Voicebox: Text-guided multilingual universal speech generation at scale. In NeurIPS

    2023

  26. [34]

    Jiaqi Li, Xiaolong Lin, Zhekai Li, Shixi Huang, Yuancheng Wang, Chaoren Wang, Zhenpeng Zhan, and Zhizheng Wu. 2025 a . Dualcodec: A low-frame-rate, semantically-enhanced neural audio codec for speech generation. In INTERSPEECH . ISCA

    2025

  27. [35]

    Jiaqi Li, Xueyao Zhang, Yuancheng Wang, Haorui He, Chaoren Wang, Li Wang, Huan Liao, Junyi Ao, Zeyu Xie, Yiqiao Huang, Junan Zhang, and Zhizheng Wu. 2025 b . Overview of the amphion toolkit (v0.2). arXiv preprint arXiv:2501.15442

    2025 arXiv

  28. [36]

    Ziniu Li, Tian Xu, and Yang Yu. 2024 a . When is rl better than dpo in rlhf? a representation and optimization perspective. In The Second Tiny Papers Track at ICLR 2024

    2024

  29. [37]

    Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. 2024 b . Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning

    2024

  30. [38]

    Huan Liao, Haonan Han, Kai Yang, Tianjiao Du, Rui Yang, Qinmei Xu, Zunnan Xu, Jingquan Liu, Jiasheng Lu, and Xiu Li. 2024. BATON: aligning text-to-audio model using human preference feedback. In IJCAI , pages 4542--4550. ijcai.org

    2024

  31. [39]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben - Hamu, Maximilian Nickel, and Matthew Le. 2023. Flow matching for generative modeling. In ICLR . OpenReview.net

    2023

  32. [40]

    Navonil Majumder, Chia - Yu Hung, Deepanway Ghosal, Wei - Ning Hsu, Rada Mihalcea, and Soujanya Poria. 2024. Tango 2: Aligning diffusion-based text-to-audio generations through direct preference optimization. In ACM Multimedia , pages 564--572. ACM

    2024

  33. [41]

    Paarth Neekhara, Shehzeen Hussain, Subhankar Ghosh, Jason Li, Rafael Valle, Rohan Badlani, and Boris Ginsburg. 2024. Improving robustness of llm-based speech synthesis by learning monotonic alignment. In INTERSPEECH . ISCA

    2024

  34. [42]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  35. [43]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206--5210. IEEE

    2015

  36. [44]

    Puyuan Peng, Po-Yao Huang, Shang-Wen Li, Abdelrahman Mohamed, and David Harwath. 2024. Voicecraft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint arXiv:2403.16973

    2024 arXiv

  37. [45]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492--28518. PMLR

    2023

  38. [46]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS

    2023

  39. [47]

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. UTMOS: utokyo-sarulab system for voicemos challenge 2022. In INTERSPEECH , pages 4521--4525. ISCA

    2022

  40. [48]

    Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. A comprehensive survey of hallucination in large language, image, video and audio foundation models. In EMNLP (Findings) , pages 11709--11724. Association for Computational Linguistics

    2024

  41. [49]

    Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2024. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In ICLR . OpenReview.net

    2024

  42. [50]

    Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. In NeurIPS, volume 35, pages 9460--9471

    2022

  43. [51]

    Xu Tan. 2023. Neural Text-to-Speech Synthesis. Springer

    2023

  44. [52]

    Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, and Dong Yu. 2024. Preference alignment improves language model-based TTS . arXiv preprint, abs/2409.12403

    2024 arXiv

  45. [53]

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. 2024. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Visi...

    2024

  46. [54]

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint, abs/2301.02111

    2023 arXiv

  47. [55]

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. 2025 a . Maskgct: Zero-shot text-to-speech with masked generative codec transformer. In ICLR . OpenReview.net

    2025

  48. [56]

    Yuancheng Wang, Jiachen Zheng, Junan Zhang, Xueyao Zhang, Huan Liao, and Zhizheng Wu. 2025 b . Metis: A foundation speech generation model with masked generative pre-training. arXiv preprint arXiv:2502.03128

    2025 arXiv

  49. [57]

    Lilian Weng. 2024. https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ Reward hacking in reinforcement learning. lilianweng.github.io

    2024

  50. [58]

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2024. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under kl-constraint. In ICML . OpenReview.net

    2024

  51. [59]

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023. Imagereward: Learning and evaluating human preferences for text-to-image generation. In NeurIPS

    2023

  52. [60]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  53. [61]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  54. [62]

    Jixun Yao, Yuguang Yang, Yu Pan, Yuan Feng, Ziqian Ning, Jianhao Ye, Hongbin Zhou, and Lei Xie. 2025. Fine-grained preference optimization improves zero-shot text-to-speech. arXiv preprint, abs/2502.02950

    2025

  55. [63]

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495--507

    2021

  56. [64]

    Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2024 a . Speechalign: Aligning speech generation to human preferences. In NeurIPS

    2024

  57. [65]

    Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024 b . Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868

    2024 arXiv

  58. [66]

    Xueyao Zhang, Liumeng Xue, Yicheng Gu, Yuancheng Wang, Jiaqi Li, Haorui He, Chaoren Wang, Ting Song, Xi Chen, Zihao Fang, Haopeng Chen, Junan Zhang, Tze Ying Tang, Lexiao Zou, Mingxuan Wang, Jun Han, Kai Chen, Haizhou Li, and Zhizheng Wu. 2024 c . Amphion: An open-source audio...

    2024

  59. [67]

    Xueyao Zhang, Xiaohui Zhang, Kainan Peng, Zhenyu Tang, Vimal Manohar, Yingru Liu, Jeff Hwang, Dangna Li, Yuhao Wang, Julian Chan, Yuan Huang, Zhizheng Wu, and Mingbo Ma. 2025. Vevo: Controllable zero-shot voice imitation with self-supervised disentanglement. In ICLR . OpenReview.net

    2025

  60. [68]

    Yu Zhang, Changhao Pan, Wenxiang Guo, Ruiqi Li, Zhiyuan Zhu, Jialei Wang, Wenhao Xu, Jingyu Lu, Zhiqing Hong, Chuxin Wang, Lichao Zhang, Jinzheng He, Ziyue Jiang, Yuxin Chen, Chen Yang, Jiecheng Zhou, Xinyu Cheng, and Zhou Zhao. 2024 d . Gtsinger: A global multi-technique sing...

    2024

  61. [69]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.