Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

MPO aligns language-model TTS with human preferences across intelligibility, speaker similarity, and prosody using a preference set plus CE regularization.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

MPO improves TTS alignment by constructing multi-dimensional preference pairs and adding cross-entropy regularization to DPO, yielding better intelligibility, speaker similarity, and prosody.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A genuinely new preference-set construction for multi-dim TTS alignment, but the evaluation leans on the same metrics used to build the preference data and the human test is under-reported; worth a referee, not a desk reject. the 5 major comments →

arxiv 2509.00685 v1 pith:DTALJXGL submitted 2025-08-31 eess.AS cs.SD

MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speech

Kangxiang Xia , Xinfa Zhu , Jixun Yao , Lei Xie This is my paper
classification eess.AS cs.SD
keywords text-to-speechdirect preference optimizationmultidimensional preference optimizationpreference setspeech language modelintelligibilityspeaker similarityprosody
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a text-to-speech language model can be aligned to human preferences on several perceptual dimensions at once, without the usual trade-offs. It proposes MPO, built from two ideas: a preference set that records the best and worst generated sample for each evaluation metric, and a regularization term that keeps the original speech-generation loss during DPO training. If the paper is right, TTS systems trained this way become more intelligible, more similar to the target speaker, and more prosodically accurate than the supervised fine-tuning baseline, while avoiding the quality collapse that plain DPO often causes. The reported effects are a drop in character error rate from 4.72 to 3.90 on the test set and a 52.3% listener preference over the baseline. This matters because combining multiple preference dimensions without hand-tuning composite rewards is an open practical problem for speech alignment.

Core claim

The paper introduces Multidimensional Preference Optimization (MPO), a post-training alignment method for decoder-only language-model text-to-speech. For each input text, MPO generates multiple candidate speech samples and evaluates them with three automatic metrics—ASR character error rate for intelligibility, speaker-embedding cosine similarity for speaker similarity, and DTW-aligned log-F0 RMSE for prosody. From these scores it builds a 'preference set' containing the best and worst sample per metric, resolves overlaps by replacing with second-best or second-worst samples, and randomly pairs one winner and one loser for DPO training. During this training it keeps the original cross-entrop

What carries the argument

The preference set: for each prompt, the model generates several candidates; the best and worst per evaluation dimension go into a winner set and a loser set; conflicts are resolved with second-best/worst choices; training randomly pairs one winner with one loser and optimizes the DPO objective plus a retained cross-entropy loss, L = λL_DPO + L_CE. The set removes the need to collapse several metrics into a single reward, while the CE term prevents the DPO global optimum (which drives losing-token probability to zero) from destroying speech quality.

Load-bearing premise

Everything rests on the three automatic scores—character error rate, speaker-embedding similarity, and aligned F0 error—faithfully encoding what human listeners prefer, and on the selection thresholds (perfect CER, gaps of at least 0.1) not skewing the preference distribution.

What would settle it

Have human annotators label the winner/loser pairs that the automatic metrics selected; if the 'preferred' sample wins less than half the time on any dimension, or if listeners rate MPO's output as less intelligible than the baseline despite its lower CER, the central claim stops holding.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Following MPO for 20k preference-optimization steps keeps the model's speech ability intact, whereas DPO without the CE term degrades CER from 4.72 to 14.52.
  • Optimizing with the preference set gives better combined results than ranking all metrics and summing ranks, so practitioners can avoid designing a composite ranking.
  • Applying DPO on a single metric chiefly improves that metric, while the preference set improves CER and speaker similarity simultaneously and keeps prosody competitive.
  • The method extends naturally to any number of evaluation dimensions: each dimension contributes one winner and one loser to the preference set, and random pairing handles conflicts.
  • Regularized DPO training for TTS can be run on a single GPU, making the preference-alignment step accessible without large-scale compute.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The preference-set recipe—per-metric extrema plus overlap resolution—is generic enough to serve as a data-construction module for any multi-objective alignment problem, not only speech; that is a natural next test.
  • Because the preference data come entirely from automatic proxies, MPO's measured gains are upper-bounded by how well those proxies track human perception; a human re-labeling study of the same preference pairs would place that bound.
  • The CE regularization term effectively keeps the policy anchored to the SFT model; one testable prediction is that the balance parameter λ controls a preservation-versus-alignment trade-off, with an optimum that depends on data quality and size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript proposes MPO, a DPO-based post-training method for LM-based TTS. For each text prompt, ten candidate utterances are generated by an SFT model; the 'preference set' is built by retaining the best/worst candidates under three automatic metrics (Paraformer CER, WavLM speaker cosine similarity, log-F0 RMSE) subject to thresholds, and pairs are sampled between the preferred and dispreferred sets during training. To avoid DPO collapse, the CE loss is added with weight λ. Experiments on a Mandarin corpus show that MPO improves CER, speaker similarity, and prosody relative to SFT and single-dimension/ranking baselines, and an ABX test is reported as preferring MPO in 52.3% of comparisons versus the baseline. The main claims are that the preference set improves multidimensional alignment and that CE regularization prevents degradation.

Significance. If the empirical claims held, the preference-set construction would be a simple and potentially useful recipe for multi-objective DPO in TTS, and the CE-regularization experiment is a valuable cautionary demonstration of DPO collapse. The method is easy to implement and the paper compares against several sensible baselines. However, the evidence currently is not sufficient: objective gains are measured on the same metrics used for data selection; the subjective test lacks any sample-size or statistical support; and the ground-truth CER anomaly suggests metric miscalibration. The core idea is plausible but needs independent evaluation.

major comments (5)
  1. [§4.3–4.5, Table 2] The objective evaluation is circular: the preference set is constructed by ranking candidates on CER, SPKSIM, and Prosody, and Table 2 then reports improvements on exactly these three metrics. This shows that MPO can optimize the chosen proxies, not that it aligns with human perception. The anomaly that ground-truth CER (7.246) is worse than every synthesized system indicates the Paraformer CER is not a reliable measure of human intelligibility for this domain; optimizing it may exploit ASR-specific artifacts. Provide independent evidence (e.g., human transcriptions/ratings, or metrics not used in selection) before claiming better alignment.
  2. [Fig. 3, §4.5] The ABX preference test is not interpretable as reported: no number of listeners, utterances, trials, confidence intervals, or significance test is given. '52.3% preferred' against the baseline and '35.3% vs 30.0% with 34.7% ties' against ground truth cannot be distinguished from noise; the word 'significantly' is unsupported. Report exact N, the unit of analysis (listener, utterance, or trial), and a statistical test (e.g., mixed-effects logistic regression or at least a binomial test). Also specify what 'ABX' means here—which reference was used.
  3. [Tables 1–2] All objective numbers are single runs without variance or significance. Differences such as CER 3.90 vs 4.24 and SPKSIM 0.577 vs 0.564 may be within run-to-run noise. Report multiple seeds or bootstrap confidence intervals and paired significance tests for the main comparisons. This is needed to support the 'significant improvements' statement in the abstract.
  4. [§4.2, Eq. (6)] The DPO KL coefficient β is never specified, and no sensitivity analysis for λ=10 is provided. Since the combined loss L = λ L_dpo + L_ce depends on the relative scale of the two terms, and DPO loss magnitude depends on β, the method is not reproducible without these values. At minimum report β, optimizer details, and a small λ sweep.
  5. [§4.3] The filtering thresholds (CER=0, metric gap ≥0.1) and the candidate generation temperature are not analyzed. If these thresholds discard most of the 10 candidates, conclusions may rest on a small, selected subset; if not, they need justification. Report the number of prompts/pairs that survive filtering and the distribution of gaps.
minor comments (6)
  1. [Eq. (1)] Typo: 'excepted reward' should be 'expected reward'.
  2. [§3.1] The formal definition of the preference set could be clearer: define ymaxA, yminA, the conflict-resolution rule, and the random pairing procedure explicitly.
  3. [§3.2] The cross-entropy loss L_ce is referenced but not defined mathematically; please provide the acoustic-token cross-entropy objective.
  4. [§4.2] Candidate generation hyperparameters (temperature, top-p/top-k, etc.) are not reported. These matter for the diversity of the preference set.
  5. [Table 2] The ground-truth row (CER 7.246) is striking and needs explanation; if this is due to ASR mismatch, it further supports concern about CER calibration.
  6. [Fig. 3] The term 'ABX' is unusual for pairwise preference percentages; clarify the task design and whether a reference sample was presented.

Circularity Check

1 steps flagged

Objective gains reduce to the preference-set selection metrics; only the underreported ABX test is independent.

specific steps
  1. fitted input called prediction [Section 4.3 (Preference Set Preparation) and Section 4.5 (Table 2)]
    "Using the transcripts from the 100-hour high-quality TTS dataset, we generate 10 batches of speech data with the supervised fine-tuned model. We then construct the preference set based on CER, speaker similarity, and prosody metrics. ... Table 2: Objective evaluation results between baseline systems and our proposed MPO. Model CER ↓ SPKSIM↑ Prosody↓"

    The preference set is labeled with the same three automatic metrics that Table 2 reports as headline objective improvements. MPO's DPO loss trains the policy to assign higher probability to samples selected as CER=0 / higher SPKSIM / better prosody; the objective evaluation then measures exactly those same three quantities. Thus the CER/SPKSIM/prosody gains are measurements of the optimization target, not independent evidence of human alignment. The paper does not describe a held-out split for the 100-hour corpus used both to construct the preference set and to compute Table 2, so the reported improvements are at least partly a fit to the selection objective. The only independent human signal, the ABX test, is reported in Figure 3 without sample size or significance.

full rationale

The core DPO+CE derivation (Eqs. 4–6) is not circular: MPO is a standard preference-optimization loss with cross-entropy regularization, and the 'preference set' is a data-selection rule. The circularity burden is in the evaluation: the preference set is constructed by ranking candidates on CER, WavLM speaker similarity, and log-F0 prosody, and the paper's objective evaluation (Table 2) reports exactly those three metrics. Unless the test set was strictly disjoint from the preference-construction pool—and no such split is described—the objective improvements are forced by the selection procedure rather than independently confirmed. The ABX preference test against ground truth and the base model is an independent, human-based outcome, which reduces the circularity; however, it is reported only as percentages without listener/trial counts, confidence intervals, or significance tests, so it is weak evidence. Self-citations ([11], [23], [25]) appear as data/training sources and are not load-bearing for MPO's method or claims. Overall, the central 'objective' claim partially reduces to its inputs, while the 'subjective' claim has independent but underreported content; score 5 reflects partial circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central method introduces no new physical or model entities. The preference set is a data structure, and the main unstated weight is the reliance on automatic metrics both as preference signals and as evaluation criteria, plus several hand-set thresholds that are not ablated.

free parameters (6)
  • Lambda (CE regularization weight) = 10
    Set by hand to balance L_dpo and cross-entropy loss; no ablation is shown.
  • DPO beta (KL coefficient) = not reported
    Controls KL distance to reference policy in Eq. 5; value not given.
  • CER filter threshold = 0 character error
    Preferred audio must have zero CER, discarding many candidates and biasing data.
  • Speaker similarity gap threshold = 0.1 cosine
    Minimum difference between preferred and dispreferred samples; ad hoc.
  • Prosody gap threshold = 0.1 log-F0 RMSE
    Minimum difference used to construct preference pairs; ad hoc.
  • Candidate batch size = 10
    Number of audio samples generated per text input for preference set construction; chosen without justification.
axioms (5)
  • standard math Bradley-Terry model of preferences
    Used in Eq. 4 and Section 3.2 to define the probability that one response is preferred over another.
  • standard math DPO closed-form reparameterization (Rafailov et al., eq. 3)
    Relies on the known DPO result to express the reward implicitly through policy ratios; cited as [16].
  • domain assumption Automatic metrics track human perception
    Sections 4.3 and 4.5 assume Paraformer CER, WavLM cosine similarity, and log-F0 RMSE are valid proxies for human judgments of intelligibility, speaker similarity, and prosody.
  • domain assumption Single-quantizer codec and LLaMA-like LM are an adequate base
    Section 4.2 builds the TTS system on a single-quantizer neural codec and LLaMA-style decoder without comparing architectural alternatives.
  • domain assumption 100-hour Mandarin corpus is representative for preference alignment
    Section 4.1 and 4.3 use a single 100-hour high-quality Mandarin corpus for all preference optimization and evaluation.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speech." pith.science (2026). https://pith.science/paper/DTALJXGL

@misc{pith2026250900685,
  author       = {Pith},
  title        = {Pith review of: MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DTALJXGL}},
  note         = {Machine review of arXiv:2509.00685}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In recent years, text-to-speech (TTS) has seen impressive advancements through large-scale language models, achieving human-level speech quality. Integrating human feedback has proven effective for enhancing robustness in these systems. However, current approaches face challenges in optimizing TTS with preference data across multiple dimensions and often suffer from performance degradation due to overconfidence in rewards. We propose Multidimensional Preference Optimization (MPO) to better align TTS systems with human preferences. MPO introduces a preference set that streamlines the construction of data for multidimensional preference optimization, enabling alignment with multiple dimensions. Additionally, we incorporate regularization during training to address the typical degradation issues in DPO-based approaches. Our experiments demonstrate MPO's effectiveness, showing significant improvements in intelligibility, speaker similarity, and prosody compared to baseline systems.

Figures

Figures reproduced from arXiv: 2509.00685 by Jixun Yao, Kangxiang Xia, Lei Xie, Xinfa Zhu.

Figure 1
Figure 1. Figure 1: The overall architecture of the proposed MPO method. ness of our proposed MPO, showing significant improve￾ments in intelligibility, speaker similarity, and prosody of the generated speech compared to baseline systems. 2. Preliminaries 2.1. Preference Alignment Preference alignment is often formatted as a reinforcement learning problem. Let x be the input prompts, and let y be the language model’s response… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison results of training loss over different training steps. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results of ABX preference test. To further verify the overall effectiveness of our proposed training method, we conducted a subjective ABX preference test. The results, illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. DDPO-VC: Speaker De-Identification via Diffusion Denoising Policy Optimization

    eess.AS 2026-06 unverdicted novelty 6.0

    DDPO-VC applies diffusion denoising policy optimization with dual-teacher rewards to improve speaker de-identification while preserving cognitive utility on dementia speech benchmarks.

Reference graph

Works this paper leans on

37 extracted references · 23 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    LM-based TTS systems convert speech waveforms into sequences of discrete tokens using neural audio codecs [1, 2, 3, 4, 5] and operate in a discrete space [6, 7]

    Introduction Recent advancements in text-to-speech (TTS) technology have been impressive, particularly with the development of decoder- only language models (LMs) that generate diverse speech through next-token prediction manner, conditioned on text in- put. LM-based TTS systems convert speech waveforms into sequences of discrete tokens using neural audio...

    work page

  2. [2]

    Preference Alignment Preference alignment is often formatted as a reinforcement learning problem

    Preliminaries 2.1. Preference Alignment Preference alignment is often formatted as a reinforcement learning problem. Let x be the input prompts, and let y be the language model’s response to x. Given reward function r(x, y) and reference policy πref, the goal of alignment is to solve for the ”aligned” policy πθ that maximizes the excepted reward: max πθ E...

    work page

  3. [3]

    MPO involves constructing a multidi- mensional preference dataset and incorporating additional reg- ularization during training to prevent model degradation

    MPO Our proposed MPO improves the original DPO approach for TTS tasks by addressing the challenges of multidimensional preference alignment. MPO involves constructing a multidi- mensional preference dataset and incorporating additional reg- ularization during training to prevent model degradation. The overall architecture of MPO is illustrated in Figure 1...

    work page

  4. [4]

    Experimental Results 4.1. Dataset We train the base language model from scratch using multiple datasets: WenetSpeech4TTS [23], LibriHeavy [24], and an in- ternal dataset, totaling 160,000 hours of speech data. The in- ternal dataset is created from web-crawled audio and processed according to the data preparation pipeline described in Wenet- Speech4TTS. T...

    work page 2000

  5. [5]

    Conclusion In this study, we proposed a novel approach, MPO, to enhance the alignment of TTS systems with human preferences. Our method introduces the concept of a preference set, which facil- itates the construction of data for multidimensional direct pref- erence optimization, allowing TTS systems to consider mul- tiple evaluation dimensions simultaneou...

    work page

  6. [6]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural codec language models are zero-shot text to speech synthesizers,”CoRR, vol. abs/2301.02111, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    The Interspeech 2024 Challenge on Speech Processing Using Discrete Units

    X. Chang, J. Shi, J. Tian, Y . Wu, Y . Tang, Y . Wu, S. Watan- abe, Y . Adi, X. Chen, and Q. Jin, “The interspeech 2024 chal- lenge on speech processing using discrete units,” arXiv preprint arXiv:2406.07725, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    BASE TTS: lessons from building a billion-parameter text-to-speech model on 100k hours of data,

    M. Lajszczak, G. C ´ambara, Y . Li, F. Beyhan, A. van Korlaar, F. Yang, A. Joly, ´A. Mart´ın-Cortinas, A. Abbas, A. Michalski, A. Moinet, S. Karlapati, E. Muszynska, H. Guo, B. Putrycz, S. L. Gambino, K. Yoo, E. Sokolova, and T. Drugman, “BASE TTS: lessons from building a billion-parameter text-to-speech model on 100k hours of data,” CoRR, vol. abs/2402.0...

    Pith/arXiv arXiv 2024

  9. [9]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE ACM Trans. Audio Speech Lang. Process. , vol. 30, pp. 495–507, 2022

    work page 2022

  10. [10]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Trans. Mach. Learn. Res. , vol. 2023, 2023

    work page 2023

  11. [11]

    Funcodec: A funda- mental, reproducible and integrable open-source toolkit for neural speech codec,

    Z. Du, S. Zhang, K. Hu, and S. Zheng, “Funcodec: A funda- mental, reproducible and integrable open-source toolkit for neural speech codec,” in ICASSP. IEEE, 2024, pp. 591–595

    work page 2024

  12. [12]

    Speechtok- enizer: Unified speech tokenizer for speech language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech language models,” in The Twelfth International Conference on Learning Representa- tions, 2024

    work page 2024

  13. [13]

    Single-codec: Single-codebook speech codec towards high-performance speech generation,

    H. Li, L. Xue, H. Guo, X. Zhu, Y . Lv, L. Xie, Y . Chen, H. Yin, and Z. Li, “Single-codec: Single-codebook speech codec towards high-performance speech generation,” in Interspeech 2024, 2024, pp. 3390–3394

    work page 2024

  14. [14]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,” CoRR, vol. abs/2303.08774, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Simpo: Simple preference opti- mization with a reference-free reward,

    Y . Meng, M. Xia, and D. Chen, “Simpo: Simple preference opti- mization with a reference-free reward,” in NeurIPS, 2024

    work page 2024

  16. [16]

    Several works in the speech community have explored in- tegrating human evaluation into LM-based TTS optimization

    (DPO), which significantly simplifies and stabilizes train- ing. Several works in the speech community have explored in- tegrating human evaluation into LM-based TTS optimization. For example, SpeechAlign [17] presents the first method based on DPO that regards ground truth as preferred samples while the generated results as dispreferred samples. UNO [18]...

    Pith/arXiv arXiv 2025

  17. [17]

    V oice- craft: Zero-shot speech editing and text-to-speech in the wild,

    P. Peng, P. Huang, S. Li, A. Mohamed, and D. Harwath, “V oice- craft: Zero-shot speech editing and text-to-speech in the wild,” in ACL (1). Association for Computational Linguistics, 2024, pp. 12 442–12 462

    work page 2024

  18. [18]

    Seed-tts: A family of high-quality versatile speech generation models,

    P. Anastassiou, J. Chen, J. Chen, Y . Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao, M. Gong, P. Huang, Q. Huang, Z. Huang, Y . Huo, D. Jia, C. Li, F. Li, H. Li, J. Li, X. Li, X. Li, L. Liu, S. Liu, S. Liu, X. Liu, Y . Liu, Z. Liu, L. Lu, J. Pan, X. Wang, Y . Wang, Y . Wang, Z. Wei, J. Wu, C. Yao, Y . Yang, Y . Yi, J. Zhang, Q. Zhang, S. Zhang...

    Pith/arXiv arXiv 2024

  19. [19]

    Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis,

    Z. Ye, X. Zhu, C.-M. Chan, X. Wang, X. Tan, J. Lei, Y . Peng, H. Liu, Y . Jin, Z. DAI et al. , “Llasa: Scaling train-time and inference-time compute for llama-based speech synthesis,” arXiv preprint arXiv:2502.04128, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” in NeurIPS, 2022

    work page 2022

  21. [21]

    Model alignment as prospect theoretic optimization,

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Model alignment as prospect theoretic optimization,” in ICML. OpenReview.net, 2024

    work page 2024

  22. [22]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,” Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  23. [23]

    Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model bench- mark,

    L. Ma, D. Guo, K. Song, Y . Jiang, S. Wang, L. Xue, W. Xu, H. Zhao, B. Zhang, and L. Xie, “Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model bench- mark,” in Interspeech 2024, 2024, pp. 1840–1844

    work page 2024

  24. [24]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in NeurIPS, 2023

    work page 2023

  25. [25]

    This fine-tuned model serves as the baseline for our experiments

    with more accurate text transcriptions. This fine-tuned model serves as the baseline for our experiments. Preference optimization is conducted on a 100-hour high-quality Mandarin TTS corpus. 4.2. Configuration The base language model follows the similar architecture of LLaMA [26], predicting acoustic tokens conditioned text input in an autoregressive mann...

    work page

  26. [26]

    Speechalign: Aligning speech generation to human preferences,

    D. Zhang, Z. Li, S. Li, X. Zhang, P. Wang, Y . Zhou, and X. Qiu, “Speechalign: Aligning speech generation to human preferences,” in NeurIPS, 2024

    2024

  27. [27]

    En- hancing zero-shot text-to-speech synthesis with human feedback,

    C. Chen, Y . Hu, W. Wu, H. Wang, E. S. Chng, and C. Zhang, “En- hancing zero-shot text-to-speech synthesis with human feedback,” CoRR, vol. abs/2406.00654, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Robust zero- shot text-to-speech synthesis with reverse inference optimization,

    Y . Hu, C. Chen, S. Wang, E. S. Chng, and C. Zhang, “Robust zero- shot text-to-speech synthesis with reverse inference optimization,” CoRR, vol. abs/2407.02243, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Dynamic time warping is employed to align the generated and reference speech features of different sequential lengths, following the evaluation script in ESPnet [29]

    to calculate the difference in the log F0 sequences be- tween the generated and reference speech. Dynamic time warping is employed to align the generated and reference speech features of different sequential lengths, following the evaluation script in ESPnet [29]. Using the transcripts from the 100-hour high-quality TTS dataset, we generate 10 batches of ...

    work page

  30. [30]

    Preference alignment improves language model-based TTS,

    J. Tian, C. Zhang, J. Shi, H. Zhang, J. Yu, S. Watanabe, and D. Yu, “Preference alignment improves language model-based TTS,” CoRR, vol. abs/2409.12403, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Robust preference op- timization through reward model distillation,

    A. Fisch, J. Eisenstein, V . Zayats, A. Agarwal, A. Beirami, C. Nagpal, P. Shaw, and J. Berant, “Robust preference op- timization through reward model distillation,” CoRR, vol. abs/2405.19316, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Libriheavy: A 50, 000 hours ASR corpus with punctuation casing and context,

    W. Kang, X. Yang, Z. Yao, F. Kuang, Y . Yang, L. Guo, L. Lin, and D. Povey, “Libriheavy: A 50, 000 hours ASR corpus with punctuation casing and context,” in ICASSP. IEEE, 2024, pp. 10 991–10 995

    work page 2024

  33. [33]

    The ISCSLP 2024 con- versational voice clone (covoc) challenge: Tasks, results and find- ings,

    K. Xia, D. Guo, J. Yao, L. Xue, H. Li, S. Wang, Z. Guo, L. Xie, Q. Zhang, L. Luo, M. Dong, and P. Sun, “The ISCSLP 2024 con- versational voice clone (covoc) challenge: Tasks, results and find- ings,” in Proc. ISCSLP, 2024

    work page 2024

  34. [34]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” CoRR, vol. abs/2302.13971, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to- end speech recognition,

    Z. Gao, S. Zhang, I. McLoughlin, and Z. Yan, “Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to- end speech recognition,” in INTERSPEECH. ISCA, 2022, pp. 2063–2067

    work page 2022

  36. [36]

    Large-scale self-supervised speech representation learning for automatic speaker verification,

    Z. Chen, S. Chen, Y . Wu, Y . Qian, C. Wang, S. Liu, Y . Qian, and M. Zeng, “Large-scale self-supervised speech representation learning for automatic speaker verification,” in ICASSP. IEEE, 2022, pp. 6147–6151

    work page 2022

  37. [37]

    Espnet2-tts: Extending the edge of TTS research,

    T. Hayashi, R. Yamamoto, T. Yoshimura, P. Wu, J. Shi, T. Saeki, Y . Ju, Y . Yasuda, S. Takamichi, and S. Watanabe, “Espnet2-tts: Extending the edge of TTS research,”CoRR, vol. abs/2110.07840, 2021

    Pith/arXiv arXiv 2021

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.