Pith. sign in

REVIEW 4 major objections 5 minor 39 references

MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MPE-TTS: a zero-shot TTS system that takes emotion prompts as text, image, or speech and reports higher naturalness and similarity than prior single-prompt baselines.

desk verdict Solid extension of multi-modal zero-shot TTS with emotion control, but the evidence would be stronger with open code and a direct validation of the low-band prosody assumption. read the letter →

arxiv 2505.18453 v1 pith:RSYUOVWX submitted 2025-05-24 cs.SD cs.AIeess.AS

Zhichao Wu , Yueteng Kang , Songjun Cao , Long Ma , Qiulin Li , Qun Yang This is my paper
classification cs.SDcs.AIeess.AS
keywords zero-shottext-to-speechmulti-modalemotionpromptdisentanglementprosodypredictionconsistencylossdiffusionacousticmodelEmotion2Veccross-modalalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MPE-TTS, a zero-shot text-to-speech system whose emotion can be specified by a text description, an image, or an audio clip, while the speaker's timbre is taken from a separate reference utterance. The central claim is that speech can be disentangled into content, timbre, emotion, and prosody, and that an LLM-style prosody predictor with an emotion consistency loss can generate natural, emotionally matching speech for unseen speakers. Evaluations against MM-TTS, Meta-StyleSpeech, and GenerSpeech report higher naturalness and similarity, with emotion accuracy closer to ground-truth mel-spectrograms than the baselines. If true, this would give users flexible, fine-grained emotional control in zero-shot synthesis without needing a perfect reference recording or a carefully worded style description.

What carries the argument

The central mechanism is a hierarchical disentangling based on bottlenecks and band-limited prosody rather than on a single style embedding. Timbre is extracted as a global speaker vector from a random same-speaker sentence via an ECAPA-TDNN-like encoder; emotion is a global vector from the multi-modal prompt emotion encoder (MPEE), which aligns a fixed CLIP encoder's text/image embeddings to fixed Emotion2Vec speech emotion embeddings with an MSE loss; content is encoded frame-level by a conformer; and prosody is a frame-level VQ code read from the low 20 mel-spectrogram bins, which the paper assumes contain nearly all prosody and little timbre or content. An LLM-like autoregressive transformer predicts these prosody codes from content, timbre, and emotion, and an emotion classifier on the predicted codes provides the emotion consistency loss. A diffusion U-Net decoder then generates the full-band mel-spectrogram, converted to audio by HiFi-GAN.

What would settle it

Take a set of utterances from many speakers, keep only the lowest 20 mel bins, and train a speaker-identification classifier and a speech-transcription model on those bands. If speaker accuracy is well above chance or words can be transcribed from those bins, then the disentangling premise is violated, and the prosody predictor would be trained on features that mix timbre and content into the prosody code.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a multi-modal prompt can drive emotion in zero-shot synthesis as effectively as, or better than, a reference speech clip. The authors build a three-stage pipeline: a multi-modal prompt emotion encoder (MPEE) maps text and image prompts into the same emotion space as Emotion2Vec speech embeddings; a diffusion-based acoustic model generates the mel-spectrogram conditioned on content, timbre, and prosody codes; and an autoregressive, LLM-like prosody predictor fits the distribution of prosody and uses an emotion consistency loss to keep the predicted prosody emotionally aligned. The key structural choice is hierarchical: timbre and emotion are global vectors, while content and prosody are frame-level quantities. The paper reports that this outperforms MM-TTS on text and image prompts, and matches or exceeds Meta-StyleSpeech and GenerSpeech on speech prompts, with ablations showing both MPEE and the emotion consistency loss contribute.

Load-bearing premise

The load-bearing premise is that the bottom 20 mel-frequency bins of a spectrogram contain almost all of a person's prosody and almost none of their timbre or the words they say; if that is false, the prosody codes used for training and prediction are not clean, and the emotion consistency loss is fitted to contaminated targets.

Editorial extensions

If this is right

  • Users can independently swap the emotion prompt (text, image, or speech) and the timbre reference, so the same spoken content can be rendered in any speaker's voice with any of the supported emotions.
  • Since prosody is predicted from content, timbre, and emotion rather than copied from a reference, the framework supports generating emotional speech when no matching audio prompt exists.
  • The emotion consistency loss raises the ceiling for emotion transfer in zero-shot TTS; the ablations show that removing it drops emotion accuracy by 2-13 percentage points depending on prompt modality.
  • Reported word-error rates remain within a few points of ground-truth mel output (23-24% vs 18.8%), suggesting the multi-modal conditioning does not seriously degrade content intelligibility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The low-20-bin assumption could be tested directly: if a speaker-verification model can identify speakers from those bins alone, timbre leaks into the prosody codes, which would weaken the disentanglement claim.
  • The same MPEE alignment scheme—anchoring text and image embeddings to a fixed speech-emotion model via MSE—could generalize to other emotion-conditioned audio tasks such as expressive singing synthesis or emotional voice conversion.
  • The paper's eight-emotion, three-intensity evaluation leaves open whether the system handles mixed or subtle emotions; an extension to continuous emotion labels or a larger emotion corpus would clarify the customization ceiling.
  • Because the LLM-like prosody predictor is trained teacher-forcing on VQ codes, its autoregressive errors at inference are never corrected; curriculum or noised-training modifications might improve prosody robustness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. MPE-TTS proposes a zero-shot text-to-speech system that accepts emotion prompts in text, image, or speech modalities. The system combines a multi-modal prompt emotion encoder (MPEE) built on Emotion2Vec and CLIP adapters, a VQ prosody encoder that extracts prosody codes from the low 20 mel bins of ground-truth spectrograms, an LLM-like autoregressive prosody predictor trained with an emotion consistency loss (ECL), and a diffusion-based acoustic model. The paper reports MOS, ESMOS, SSMOS, WER, and emotion accuracy (ACC) on MEAD-TTS, comparing against MM-TTS, Meta-StyleSpeech, and GenerSpeech, with ablation studies that remove MPEE and ECL. The central claim is that MPE-TTS outperforms existing systems in naturalness and similarity for zero-shot customized emotion TTS.

Significance. If the results hold, the work is a useful step toward flexible fine-grained control in zero-shot TTS: it is among the first to support text, image, and speech emotion prompts in a single system, and the proposed MPEE and ECL components are simple and potentially transferable. The paper provides objective and subjective comparisons, a public demo page, and clear ablations. The use of a separately trained wav2vec2.0 SER model for ACC evaluation is structurally separate from the training objectives, so I see no circularity in the evaluation. However, the load-bearing low-20-bin prosody assumption is not validated in this pipeline, and the comparative evaluation against a self-reproduced MM-TTS baseline without statistical intervals on objective metrics limits the strength of the empirical claims. These issues are addressable and do not appear to require a fundamentally different method.

major comments (4)
  1. [§2.1 and §2.3] The claim that the low 20 bins of each ground-truth mel-spectrogram 'contain almost complete prosody and significantly less timbre and content information compared to the full band' is cited to [17] but is not verified for MEAD-TTS or for the 16 kHz setup used here. This assumption is load-bearing: it defines the ground-truth target for the VQ prosody encoder, the conditioning signal for the acoustic model, and the training target for the prosody predictor and ECL. If the low band carries speaker timbre or phonetic content, the prosody codes are contaminated and the ECL classifier can exploit non-prosodic cues; if the low band omits emotion-relevant pitch or energy variation, the prosody predictor is trained against an impoverished target. Please add a direct validation, for example by training speaker-ID and content classifiers on the low-band features or prosody codes to show they perform at chance, showing that emotion classification from the prosody codes is high, and comparing emotion transfer against using full-band prosody codes. This is needed before the disentangling and customization claims can be accepted.
  2. [§3.4, Tables 1–3] The MM-TTS baseline is reproduced by the authors because no official code is available, but the manuscript does not state the reproduction configuration, how closely its outputs match the published system, or whether hyperparameters were tuned. In addition, the objective metrics WER and ACC are reported as point estimates without confidence intervals, significance tests, or the number of evaluation utterances; several differences are small (for example, Table 1 WER 23.4 vs. 22.8 for GenerSpeech, and Table 3 ACC 47% vs. 41%). Please provide variance or confidence intervals, per-utterance paired tests, evaluation set size, and a description of the MM-TTS reproduction effort so that the comparative claim is not dependent on a single favorable configuration.
  3. [§2.2, Eq. (1)] The MPEE is trained only with MSE losses aligning text and image emotion codes to Emotion2Vec speech codes on MEAD-TTS. The paper does not evaluate how well this alignment generalizes to unseen emotion labels or conditions, nor does it report inter-modal consistency, such as the similarity of text and image codes for the same target emotion. Since the central capability is that any prompt modality should produce the same emotion, please add a quantitative alignment check, for example nearest-neighbor retrieval or classification accuracy of text and image codes in the speech emotion space, or a user study comparing emotion rendering across modalities for matched prompts.
  4. [§2.3, Tables 1–3] The ECL classifier is trained on the predicted prosody codes, and the reported ACC gains in Tables 1–3 are the main evidence that ECL improves emotion consistency. Because the prosody codes are extracted from the low-20-bin band, the classifier may be learning to identify emotion from timbre or content leakage rather than from prosody itself. Please report ECL accuracy when the classifier is evaluated on prosody codes from unseen speakers in a leave-one-speaker-out setting, and show that the improvements persist when speaker identity is controlled for, for instance through speaker-adversarial training or conditioning the classifier on probe speaker embeddings.
minor comments (5)
  1. [Abstract] The phrase 'an prosody predictor' should be corrected to 'a prosody predictor'.
  2. [Figure 1 caption] The caption uses 'multi-model' in two places; the intended term is 'multi-modal'.
  3. [§3.1] The construction of the text prompts with an LLM is described only briefly; please provide a few example prompts and describe how speech and image prompts are sampled during training, in particular whether they come from the same utterance as the target speech.
  4. [§3.3] The GT(mel) row in Tables 1–3 has no ESMOS or SSMOS values; please either report these values or explain why they are omitted.
  5. [§3.4] The text states that 'our method significantly outperforms the baseline' but no statistical significance tests are reported; the term 'significantly' should be reserved for cases with a supporting test.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: training losses and evaluation metrics are structurally independent.

full rationale

MPE-TTS reports three training objectives: the MPEE MSE alignment to Emotion2Vec speech embeddings, the VQ/cross-entropy prosody-predictor loss, and the ECL classifier loss. The claimed results (Tables 1–3) are measured with a separately trained wav2vec2.0 SER model for ACC, Whisper for WER, and human MOS/ESMOS/SSMOS. None of these evaluation quantities is the same function as a training loss, and no reported number is obtained by re-inserting a fitted parameter into the definition of the metric. The low-20-bin prosody assumption, cited to the external ProsoSpeech paper [17], is a design premise rather than a circular derivation: the paper does not define prosody codes as the evaluation target, and the assumption is not justified by the present authors' own prior work. The only cited works that could conceivably be self-citations (e.g., [4]) are related-work examples, not load-bearing premises; the reference list contains no same-author result used to force the proposed choices. The ablation 'Ours w/o ECL' and 'Ours w/o MPEE' also show that the reported gains are empirical effects of the added modules rather than algebraic identities.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The system relies on several borrowed domain assumptions (prosody in low mel bins, Emotion2Vec as emotion anchor, AutoVC timbre extraction) and a few unstated hyperparameters. No fundamentally new physical entities are postulated.

free parameters (3)
  • low_20_bins_cutoff = 20 (frequency bins)
    Chosen based on reference [17] as the prosody-dominant band; not tuned or validated against alternative cutoffs in this paper.
  • emotion_consistency_loss_weight
    Weight of the emotion consistency loss is not reported; it is a hand-tuned design choice that affects prosody predictor training.
  • prosody_VQ_codebook_size
    Codebook size for the VQ-based prosody encoder is not stated; it controls prosody granularity and is a free design choice.
assumptions (4)
  • domain assumption The low 20 mel bins contain almost complete prosody and significantly less timbre and content information than the full band.
    Invoked in Sections 2.1 and 2.3 to justify extracting ground-truth prosody codes and to train the prosody predictor. If false, prosody codes leak timbre or content, disrupting disentangling.
  • domain assumption Emotion2Vec provides a robust and generalizable emotion feature representation for speech.
    Used in Section 2.2 as the anchor modality for MPEE; text and image adapters are trained to match Emotion2Vec features, so the whole emotion space inherits this assumption.
  • ad hoc to paper MSE loss between text/image emotion codes and speech emotion codes produces a unified emotion latent space.
    Equation (1) directly imposes this alignment. The paper provides no evidence that simple MSE alignment yields semantic emotion generalization across text, image, and speech.
  • domain assumption Using a random sentence from the same speaker as timbre reference disentangles timbre from content.
    Borrowed from AutoVC-style speaker encoders (reference [15]); introduced in Section 2.1. Assumes the extracted global speaker vector is content-independent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt." pith.science (2026). https://pith.science/paper/RSYUOVWX

@misc{pith2026250518453,
  author       = {Pith},
  title        = {Pith review of: MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RSYUOVWX}},
  note         = {Machine review of arXiv:2505.18453}
}
read the original abstract

Most existing Zero-Shot Text-To-Speech(ZS-TTS) systems generate the unseen speech based on single prompt, such as reference speech or text descriptions, which limits their flexibility. We propose a customized emotion ZS-TTS system based on multi-modal prompt. The system disentangles speech into the content, timbre, emotion and prosody, allowing emotion prompts to be provided as text, image or speech. To extract emotion information from different prompts, we propose a multi-modal prompt emotion encoder. Additionally, we introduce an prosody predictor to fit the distribution of prosody and propose an emotion consistency loss to preserve emotion information in the predicted prosody. A diffusion-based acoustic model is employed to generate the target mel-spectrogram. Both objective and subjective experiments demonstrate that our system outperforms existing systems in terms of naturalness and similarity. The samples are available at https://mpetts-demo.github.io/mpetts_demo/.

Figures

Figures reproduced from arXiv: 2505.18453 by the authors.

Figure 1
Figure 1. (a) shows the overview of the proposed method. Our framework includes three training stages, in sequence: 1) Emotion Training. 2) Acoustic Model Training. 3) Prosody Training. (b) shows the inference phase of the proposed system 3) We introduce an LLM-like prosody predictor to predict the target prosody which bring natural prosody performance. Besides, we propose the emotion consistency loss(ECL) to en￾hance the emo… view at source ↗
Figure 2
Figure 2. the multi-model prompt emotion encoder 2.3. LLM-like Prosody Predictor The Auto-regressive(AR) Transformer[19], which plays a cru￾cial role in large language models, has also been shown to be effective in speech modeling[4, 20]. Given its excellent capa￾bilities in capturing both local and long-range dependency, we introduce an LLM-like model as the prosody predictor with the aim of modeling the target prosody based… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 17 canonical work pages

  1. [17]

    Meta-stylespeech: Multi-speaker adaptive text-to-speech generation,

    D. Min, D. B. Lee, E. Yang, and S. J. Hwang, “Meta-stylespeech: Multi-speaker adaptive text-to-speech generation,” in Interna- tional Conference on Machine Learning . PMLR, 2021, pp. 7748–7759

    work page 2021

  2. [1]

    In order to meet practical application scenarios, Zero-Shot TTS(ZS-TTS) which aims to generate speech with unseen style has been introduced

    Introduction The conventional TTS systems[1, 2, 3] are limited because they only generate the speech of seen speakers. In order to meet practical application scenarios, Zero-Shot TTS(ZS-TTS) which aims to generate speech with unseen style has been introduced. Generally, ZS-TTS systems face challenges in generalization, fine-grained customization and model...

    work page

  3. [2]

    We propose a hierarchical disentangling strategy, model- ing speech features at different granularity levels, and achieve effective disentangling

    work page

  4. [3]

    MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt

    We propose a multi-modal prompt emotion encoder based on Emotion2Vec[14] to extract emotion information from text, image or speech prompt. It enables users to flexibly choose one of the three prompt modals. arXiv:2505.18453v1 [cs.SD] 24 May 2025 (a) (b) Figure 1: (a) shows the overview of the proposed method. Our framework includes three training stages, ...

    work page Pith review arXiv 2025

  5. [4]

    Besides, we propose the emotion consistency loss(ECL) to en- hance the emotion information in the predicted prosody

    We introduce an LLM-like prosody predictor to predict the target prosody which bring natural prosody performance. Besides, we propose the emotion consistency loss(ECL) to en- hance the emotion information in the predicted prosody

    work page

  6. [5]

    Subjective and objective experiments have shown that our system has the ability of unseen generation and achieves flexible customization

    work page

  7. [6]

    These components meticulously model different speech attributes, achieving the disentangle- ment of speech attributes

    Method As depicted in Figure 1(a), MPE-TTS is composed of three main components: the multi-modal prompt emotion en- coder(MPEE), the diffusion-based acoustic model and the LLM-like prosody predictor. These components meticulously model different speech attributes, achieving the disentangle- ment of speech attributes. Firstly, MPEE extracts the emo- tion c...

    work page

  8. [7]

    Modeling these features as global vectors is more efficient and practical

    Coarse-grained features: we define the timbre and emo- tion features as coarse-grained features because, in general, a speaker’s timbre and emotion do not change significantly within a segment of speech. Modeling these features as global vectors is more efficient and practical

    work page

Show all 39 references
  1. [8]

    Content information is frame- related and temporal, requiring fine-grained modeling to ensure semantic accuracy

    Fine-grained features: we define the content and prosody features as fine-grained features. Content information is frame- related and temporal, requiring fine-grained modeling to ensure semantic accuracy. Since prosody information has a high dy- namic range, it is also modeled...

  2. [9]

    Construct the Multi-modal Emotion Dataset During the training, we utilized the LibriTTS[22] and MEAD- TTS[12] datasets

    Dataset and Experiments 3.1. Construct the Multi-modal Emotion Dataset During the training, we utilized the LibriTTS[22] and MEAD- TTS[12] datasets. The LibriTTS dataset, a multi-speaker En- glish corpus of approximately 585 hours of read English speech at 24kHz sampling rate,...

  3. [10]

    We in- troduce an LLM-like prosody predictor to predict emotion- wise prosody, enhancing the emotional alignment between the prompt input and the generated speech

    Discussion Aiming at improving the Zero-Shot generation capabilities of ZS-TTS systems, we propose a multi-modal prompt emotion encoder based on pre-trained Emotion2Vec, which enables our TTS system to support multi-modal prompt input. We in- troduce an LLM-like prosody predic...

  4. [11]

    Fastspeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” arXiv preprint arXiv:2006.04558, 2020

    2006 arXiv

  5. [12]

    Grad-tts: A diffusion probabilistic model for text-to-speech,

    V . Popov, I. V ovk, V . Gogoryan, T. Sadekova, and M. Kudinov, “Grad-tts: A diffusion probabilistic model for text-to-speech,” in International Conference on Machine Learning . PMLR, 2021, pp. 8599–8608

    2021

  6. [13]

    Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions,

    J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerrv-Ryan et al. , “Natural tts synthesis by conditioning wavenet on mel spectrogram pre- dictions,” in 2018 IEEE international conference on acoustics, speech and signal process...

    2018

  7. [14]

    Mega-tts: Zero-shot text- to-speech at scale with intrinsic inductive bias,

    Z. Jiang, Y . Ren, Z. Ye, J. Liu, C. Zhang, Q. Yang, S. Ji, R. Huang, C. Wang, X. Yin et al. , “Mega-tts: Zero-shot text- to-speech at scale with intrinsic inductive bias,” arXiv preprint arXiv:2306.03509, 2023

    2023 arXiv

  8. [15]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Neural codec language mod- els are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  9. [16]

    Style tokens: Un- supervised style modeling, control and transfer in end-to-end speech synthesis,

    Y . Wang, D. Stanton, Y . Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y . Xiao, Y . Jia, F. Ren, and R. A. Saurous, “Style tokens: Un- supervised style modeling, control and transfer in end-to-end speech synthesis,” in International conference on machine learn- ing. PMLR, 2018,...

    2018

  10. [18]

    Generspeech: Towards style transfer for generalizable out-of-domain text-to- speech,

    R. Huang, Y . Ren, J. Liu, C. Cui, and Z. Zhao, “Generspeech: Towards style transfer for generalizable out-of-domain text-to- speech,” Advances in Neural Information Processing Systems , vol. 35, pp. 10 970–10 983, 2022

    2022

  11. [19]

    Prompttts: Control- lable text-to-speech with text descriptions,

    Z. Guo, Y . Leng, Y . Wu, S. Zhao, and X. Tan, “Prompttts: Control- lable text-to-speech with text descriptions,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  12. [20]

    Instructtts: Modelling expressive tts in discrete latent space with natural lan- guage style prompt,

    D. Yang, S. Liu, R. Huang, C. Weng, and H. Meng, “Instructtts: Modelling expressive tts in discrete latent space with natural lan- guage style prompt,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  13. [21]

    Prompttts 2: Describing and generating voices with text prompt,

    Y . Leng, Z. Guo, K. Shen, X. Tan, Z. Ju, Y . Liu, Y . Liu, D. Yang, L. Zhang, K. Song et al., “Prompttts 2: Describing and generating voices with text prompt,”arXiv preprint arXiv:2309.02285, 2023

    2023 arXiv

  14. [22]

    Mm-tts: Multi-modal prompt based style trans- fer for expressive text-to-speech synthesis,

    W. Guan, Y . Li, T. Li, H. Huang, F. Wang, J. Lin, L. Huang, L. Li, and Q. Hong, “Mm-tts: Multi-modal prompt based style trans- fer for expressive text-to-speech synthesis,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 18 117–18 125

    2024

  15. [23]

    Flow straight and fast: Learning to generate and transfer data with rectified flow,

    X. Liu, C. Gong, and Q. Liu, “Flow straight and fast: Learning to generate and transfer data with rectified flow,” arXiv preprint arXiv:2209.03003, 2022

    2022 arXiv

  16. [24]

    emotion2vec: Self-supervised pre-training for speech emotion representation,

    Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, “emotion2vec: Self-supervised pre-training for speech emotion representation,” arXiv preprint arXiv:2312.15185, 2023

    2023 arXiv

  17. [25]

    Autovc: Zero-shot voice style transfer with only au- toencoder loss,

    K. Qian, Y . Zhang, S. Chang, X. Yang, and M. Hasegawa- Johnson, “Autovc: Zero-shot voice style transfer with only au- toencoder loss,” in International Conference on Machine Learn- ing. PMLR, 2019, pp. 5210–5219

    2019

  18. [26]

    Global prosody style transfer with- out text transcriptions,

    K. Qian, Y . Zhang, S. Chang, J. Xiong, C. Gan, D. Cox, and M. Hasegawa-Johnson, “Global prosody style transfer with- out text transcriptions,” in International Conference on Machine Learning. PMLR, 2021, pp. 8650–8660

    2021

  19. [27]

    Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech,

    Y . Ren, M. Lei, Z. Huang, S. Zhang, Q. Chen, Z. Yan, and Z. Zhao, “Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech,” inICASSP 2022-2022 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7577–7581

    2022

  20. [28]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  21. [29]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural In- formation Processing Systems, 2017

    2017

  22. [30]

    Better speech synthesis through scaling,

    J. Betker, “Better speech synthesis through scaling,” arXiv preprint arXiv:2305.07243, 2023

    2023 arXiv

  23. [31]

    Taming transform- ers for high-resolution image synthesis,

    P. Esser, R. Rombach, and B. Ommer, “Taming transform- ers for high-resolution image synthesis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2021, pp. 12 873–12 883

    2021

  24. [32]

    Libritts: A corpus derived from librispeech for text- to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text- to-speech,” arXiv preprint arXiv:1904.02882, 2019

    1904 arXiv

  25. [33]

    Tango 2: Aligning diffusion-based text-to- audio generations through direct preference optimization,

    N. Majumder, C.-Y . Hung, D. Ghosal, W.-N. Hsu, R. Mihal- cea, and S. Poria, “Tango 2: Aligning diffusion-based text-to- audio generations through direct preference optimization,” arXiv preprint arXiv:2404.09956, 2024

    2024 arXiv

  26. [34]

    Conformer: Convolution- augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu et al. , “Conformer: Convolution- augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020

    2005 arXiv

  27. [35]

    Ecapa- tdnn: Emphasized channel attention, propagation and ag- gregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa- tdnn: Emphasized channel attention, propagation and ag- gregation in tdnn based speaker verification,” arXiv preprint arXiv:2005.07143, 2020

    2005 arXiv

  28. [36]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. S...

    2015

  29. [37]

    Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,” Advances in neural information processing systems , vol. 33, pp. 17 022– 17 033, 2020

    2020

  30. [38]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,” Advances in neural information processing systems , vol. 33, pp. 12 449–12 460, 2020

    2020

  31. [39]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.