GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech

Chenye Cui; Jinglin Liu; Rongjie Huang; Yi Ren; Zhou Zhao

RECORD

GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech

Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, Zhou Zhao

Audio and Speech Processing · Submitted 15 May 2022

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2205.07211 (version 2)

Pith IDPBPQPYM5, Pith's permanent record of this paper and its review

CitationChenye Cui, Jinglin Liu, Rongjie Huang, Yi Ren, Zhou Zhao. "GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech." arXiv:2205.07211 (2022). https://pith.science/paper/PBPQPYM5

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,105 input tokens.

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.10 · human participants 0.09 · controlled comparison 0.60 · randomized assignment 0.01 · states quantitative result 0.05 · compares with prior results 0.94 · mentions code or data availability 0.59.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e.g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data. This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice. GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components: 1) a multi-level style adaptor to efficiently model a large range of style conditions, including global speaker and emotion characteristics, and the local (utterance, phoneme, and word-level) fine-grained prosodic representations; and 2) a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation and thus improve model generalization. Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity. The extension studies to adaptive style transfer further show that GenerSpeech performs robustly in the few-shot data setting. Audio samples are available at https://GenerSpeech.github.io/

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e.g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data.	background 0.97	0.07	0.02	asserted 0.99
s1	This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice.	contribution 0.99	0.78	0.11	asserted 0.98
s2	GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components: 1) a multi-level style adaptor to efficiently model a large range of style conditions, including global speaker and emotion characteristics, and the local (utterance, phoneme, and word-level) fine-grained prosodic representations; and 2) a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation and thus improve model generalization.	method 0.98	0.85	0.05	asserted 0.99
s3	Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity.	result 1.00	0.95	0.73	asserted 1.00
s4	The extension studies to adaptive style transfer further show that GenerSpeech performs robustly in the few-shot data setting.	result 0.99	0.94	0.09	asserted 1.00
s5	Audio samples are available at https://GenerSpeech.github.io/	contribution 0.48	0.21	0.02	asserted 0.72

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e.g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data. This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice. GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components: 1) a multi-level style adaptor to efficiently model a large range of style conditions, including global speaker and emotion characteristics, and the local (utterance, phoneme, and word-level) fine-grained prosodic representations; and 2) a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation and thus improve model generalization. Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity. The extension studies to adaptive style transfer further show that GenerSpeech performs robustly in the few-shot data setting. Audio samples are available at https://GenerSpeech.github.io/

Human review

How Pith checks papers → · Full audit of this paper →

References

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
eess.AS 2023-06 conditional novelty 6.0 of 10

Decomposing speech into content, timbre, prosody, and phase, and giving each its own module, yields a 20,000-hour zero-shot TTS system that beats YourTTS on naturalness and speaker similarity and shows no word repeats...
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
cs.SD 2023-01 conditional novelty 6.0 of 10

A latent diffusion model conditioned on CLAP text embeddings, trained on roughly 1M pseudo-captioned audio clips, beats Diffsound on the AudioCaps text-to-audio benchmark and is reused for inpainting and image/video-to-audio.
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
cs.SD 2023-03 conditional novelty 3.0 of 10

The paper organizes roughly forty diffusion-based speech systems into a three-way taxonomy for text-to-speech and a two-way taxonomy for speech enhancement, with side-by-side tables of numbers copied from the original papers.

How the literature uses it

19 arXiv papers cite this in 41 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 17 sentences without a confident label

Builds on it 1 paper

Inspired by recent works [ref] , we extend DSLN to mix-DSLN (M-DSLN).

CrossSpeech: Speaker-independent Acoustic Representation for Cross-lingual Speech Synthesis · Speaker Generalization · 0.96

Uses its method or tool 2 papers

Style teacher models In this paper, we explore two types of style teacher: (1) Supervised learning (SL) based expressive TTS model, GenerSpeech [ref] , which can effectively extract style features from clean speech.

NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS · Conditional diffusion model · 0.94
To further evaluate the style transfer performance, an AXY test used in [ref] is conducted, the range of 7-point score is from -3 to 3, 0 represents “Both are about the same distance".

Interpretable Style Transfer for Text-to-Speech with ControlVAE and Diffusion Bridge · Style Transfer Quality and Similarity · 0.97

Compares with it 3 papers

We compare the quality and similarity of generated audios by our IST-TTS with public baseline model GenerSpeech [ref] and our proposed baseline models, VAEFS(+1s/2s).

Interpretable Style Transfer for Text-to-Speech with ControlVAE and Diffusion Bridge · Baseline models · 1.00
Baselines For comparison, we set the following systems as baselines: 1) Ref, the reference audio. 2) MetaStyleSpeech [ref] , multi-speaker adaptive TTS with meta-learning. 3) YourTTS [ref] , VITS-based zero-shot multi-speaker TTS with the pre-trained speaker encoder. 4) GenerSpeech [ref] , style transfer method for out-of-domain TTS. 5) StyleTTS [ref] , style-based TTS with transferable aligner and AdaIN.

DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling on Time Variability · Experiment Setup · 0.99
Baseline models We compare the quality and similarity of generated audio samples of our NoreSpeech with other systems, including 1) Reference, the reference audio; 2) Reference (voc.), which means we convert the reference audio into mel-spectrograms and then convert them back to audio using HiFi-GAN; 3) FastSpeech 2 [ref] , which uses the speaker encoder to extract speaker embedding; 4) Styler [ref] , which uses adversarial training and information bottleneck to eliminate noise; 5) GenerSpeech [ref] , `clean', `nosiy', and `denoise' denote the types of reference audio.

NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTS · Dataset, training setting and baseline models · 0.97

Cites it as background 12 papers

GenerSpeech [ref] leverages multi-level style adaptors for the global and local stylization of the custom utterance.

Make-A-Voice: Unified Voice Synthesis With Discrete Representation · Speech/Singing Synthesis · 0.87
GenerSpeech [ref] proposes a multi-level style adaptor to efficiently model a large range of style conditions.

MSceneSpeech: A Multi-Scene Speech Dataset For Expressive Speech Synthesis · Style Transfer in Text-to-Speech · 0.92
Besides directly performing adversarial training on the mel-spectrograms, some works [ref] use GAN to improve the capability of the model for generating from the mel-spectrograms to the waveform (i.e., vocoder).

VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement · Generative Adversarial Network · 0.90

What it cites · 4 sentences without a confident label

Contests 1 paper

While current TTS literature [ref] has considered improving model capability towards OOD text-to-speech, they fail to address the above challenges fully.

Towards Multi-Scale Style Control for Expressive Speech Synthesis · Introduction · contest 0.81

Uses the method or tool of 1 paper

We utilize HiFi-GAN [ref] (V1) as the vocoder to synthesize waveform from the generated mel-spectrogram in all our experiments.

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis · Training and Evaluation · 1.00

Uses data or values from 2 papers

In the training stages, we utilize the LibriTTS [ref] dataset, which is a multi-speaker corpus (2456 speakers) derived from LibriSpeech [ref] and contains 586 hours of speech data.

LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech · Dataset · 0.98
In the pre-training stage, we adopt the multi-emotion dataset IEMOCAP [ref] which contains about 12 hours of emotional speech, and the multi-speaker dataset VoxCeleb1 [ref] which contains over 100,000 utterances from 1,251 celebrities.

VoxCeleb: a large-scale speaker identification dataset · Dataset · 0.94

Compares with 4 papers

We compare the quality and similarity of generated audio samples of our GenerSpeech with other systems, including 1) Reference, the reference audio; 2) Reference (voc.), where we first convert the reference audio into mel-spectrograms and then convert them back to audio using HiFi-GAN; 3) Mellotron [ref] : The auto-regressive multi-speaker TTS model based on the Tacotron using global style token (GST). 4) FG-TransformerTTS [ref] : The fine-grained style control on auto-regressive model Transformer-TTS. 5) Expressive FS2 [ref] : The combination of both multi-speaker [ref] and muli-emotion [ref] FastSpeech 2, which adds the speaker and emotion d-vectors extracted by the pre-trained discriminative models to the backbone. 6) Meta-StyleSpeech [ref] : The finetuned multi-speaker text-to-speech model with meta-learning. 7) Styler [ref] : The expressive text-to-speech model that model style fact

STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speech · Baseline models · 0.99
We compare the quality and similarity of generated audio samples of our GenerSpeech with other systems, including 1) Reference, the reference audio; 2) Reference (voc.), where we first convert the reference audio into mel-spectrograms and then convert them back to audio using HiFi-GAN; 3) Mellotron [ref] : The auto-regressive multi-speaker TTS model based on the Tacotron using global style token (GST). 4) FG-TransformerTTS [ref] : The fine-grained style control on auto-regressive model Transformer-TTS. 5) Expressive FS2 [ref] : The combination of both multi-speaker [ref] and muli-emotion [ref] FastSpeech 2, which adds the speaker and emotion d-vectors extracted by the pre-trained discriminative models to the backbone. 6) Meta-StyleSpeech [ref] : The finetuned multi-speaker text-to-speech model with meta-learning. 7) Styler [ref] : The expressive text-to-speech model that model style fact

EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model · Baseline models · 0.99
We compare the quality and similarity of generated audio samples of our GenerSpeech with other systems, including 1) Reference, the reference audio; 2) Reference (voc.), where we first convert the reference audio into mel-spectrograms and then convert them back to audio using HiFi-GAN; 3) Mellotron [ref] : The auto-regressive multi-speaker TTS model based on the Tacotron using global style token (GST). 4) FG-TransformerTTS [ref] : The fine-grained style control on auto-regressive model Transformer-TTS. 5) Expressive FS2 [ref] : The combination of both multi-speaker [ref] and muli-emotion [ref] FastSpeech 2, which adds the speaker and emotion d-vectors extracted by the pre-trained discriminative models to the backbone. 6) Meta-StyleSpeech [ref] : The finetuned multi-speaker text-to-speech model with meta-learning. 7) Styler [ref] : The expressive text-to-speech model that model style fact

Fine-grained style control in Transformer-based Text-to-speech Synthesis · Baseline models · 0.99

Cites as background 11 papers

[ref] utilizes both the global utterance-level and the local quasi-phoneme-level style features of the target speech.

Towards Multi-Scale Style Control for Expressive Speech Synthesis · Introduction · 0.90
SC-GlowTTS [ref] proposes a speaker-conditional architecture that explores a flow-based decoder in a zero-shot scenario.

SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model · Style Modeling and Transferring in Text-to-Speech · 0.96
Several works [ref] adopt meta-learning to adapt to new speakers that have not been seen during training.

Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech · Introduction · 0.89

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.