REVIEW 38 cited by
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.
Forward citations
Cited by 38 Pith papers
-
SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation
SemBridge supervises autoregressive states with discrete semantic tokens during training, improving content fidelity of continuous-latent speech generation without changing inference.
-
AST: Adaptive, Seamless, and Training-Free Precise Speech Editing
AST performs training-free text-based speech editing by stitching inverted source latents with synthesized targets and adaptively guiding the flow-matching decoder, achieving state-of-the-art temporal fidelity and spe...
-
A Non-autoregressive Model for Joint STT and TTS
A joint non-autoregressive model handles both STT and TTS in one framework, beating its own STT baseline and matching its TTS baseline with extra unpaired data and iterative refinement.
-
Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis
AU-supervised single-token face encoding plus dual visual–speech DPO on a large real-conversation dataset improves empathetic conversational TTS over text/speech-only and prior visual CSS systems.
-
Designed Vocalizations Dataset: Sound-Designed Human and Animal Voices for Non-human Voice Conversion
A new public dataset of 231,800 sound-designed vocalizations with seen/unseen preset-style and source-timbre splits, plus a baseline non-human voice-conversion evaluation.
-
StellarTTS: Sparse Temporal Embedding for Low-Latency and Robust Speech Synthesis
A mobile-oriented 83M-parameter masked transformer with sparse phone-anchored temporal embeddings achieves RTF 0.08 and lower WER than MaskGCT/F5-TTS on Seed-TTS test sets.
-
BlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching Speech
A Taiwan-specific tokenizer, language model, and bridge to a reused acoustic stack cut code-switching TTS CER from 11.45% to 4.81%, with 65.6% listener preference.
-
OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model
A zero-shot model that generates a video of a reference face speaking user-chosen text with a reference voice timbre.
-
SwiftF0: Fast and Accurate Monophonic Pitch Detection
SwiftF0 estimates monophonic pitch from a compact STFT-CNN, reporting better accuracy than CREPE under 10 dB noise at 42x lower CPU cost, alongside a new synthetic speech dataset and a six-component evaluation metric.
-
XEmoRAG: Cross-Lingual Emotion Transfer with Controllable Intensity Using Retrieval-Augmented Generation
XEmoRAG synthesizes Thai speech with emotions cloned from Chinese reference audio by retrieving matching Thai prompts and aligning prosody with flow matching, outperforming baseline TTS in emotion similarity and intel...
-
RepeaTTS: Towards Feature Discovery through Repeated Fine-Tuning
PCA of repeated synthesized utterances with fixed inputs can reveal controllable prosodic features that can be enrolled as new prompts via fine-tuning.
-
VS-Singer: Vision-Guided Stereo Singing Voice Synthesis with Consistency Schr\"odinger Bridge
A unified model synthesizes binaural singing from scene images using a consistency Schrödinger bridge, enabling one-step generation.
-
Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models
Counterfactual gradient edits to a pretrained TTS model's encoder activations can control prosody and correct mispronunciations at inference time, at least on Tacotron 2.
-
SpeakStream: Streaming Text-to-Speech with Interleaved Data
A decoder-only TTS trained on force-aligned interleaved text-speech chunks generates audio after a few words, achieving ~30ms TTS latency and WER comparable to non-streaming.
-
LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Context
An LLM-based ASR error corrector trained on synthetic rare-word speech and given simplified phonetic context lowers WER/CER and raises rare-word recall on English and Japanese benchmarks.
-
Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis
Continuous autoregressive text-to-speech with a Gaussian-mixture codec matches or beats a discrete-codec VALL-E baseline with a fraction of the language model parameters.
-
FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles
FaceSpeak generates speech from arbitrary-style portraits by learning separate identity and emotion embeddings from face images, with a new generated multi-style TTS dataset.
-
Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction
M2CI-Dubber improves dubbing prosody by extracting global sentence-level and local phoneme-level features from multimodal context and fusing them with the current text through attention and graph interaction.
-
Beyond One-Size-Fits-All: Personalized and Culturally Adaptive Emotional TTS via Interactive Optimization of Individual Emotion Perception Spaces
An interactive genetic algorithm tunes arousal-valence coordinates per listener in emotional TTS, and personalized or culture-specific coordinates beat a generic U.S.-average baseline in small A/B tests.
-
AutoSIFT: Automatic Style Sifting for Controllable Speech Generation with Arbitrary Style Infilling
AutoSIFT disentangles text-describable style categories from residual speech styles and selectively infills only the categories the user specifies.
-
Evolution Strategy-Based Calibration for Low-Bit Quantization of Speech Models
Using CMA-ES to jointly optimize activation quantization scales keeps speech-model accuracy near full precision under full INT8 and INT4 quantization.
-
WhisperVC: Decoupled Cross-Domain Alignment and Speech Generation for Low-Resource Whisper-to-Normal Conversion
WhisperVC converts whispered Mandarin to natural speech by decoupling domain alignment from synthesis, reporting DNSMOS 3.11, UTMOS 2.52, CER 18.67%, and speaker cosine 0.76.
-
MixedG2P-T5: G2P-free Speech Synthesis for Mixed-script texts using Speech Self-Supervised Learning and Language Model
A T5 model predicts SSL-derived discrete speech tokens directly from mixed-script Japanese text, letting a FastSpeech 2 synthesizer produce speech without a grapheme-to-phoneme module.
-
Adaptive Duration Model for Text Speech Alignment
DurFormer, an adaptive duration prediction model with speed, scene, and semantic conditioning, improves phoneme-level duration accuracy and lowers word error rate in Mandarin text-to-speech.
-
Quantize More, Lose Less: Autoregressive Generation from Residually Quantized Speech Representations
QTTS models speech as sequences from a multi-codebook RVQ audio codec whose first codebook is trained with ASR supervision, aiming for higher-fidelity TTS than single-codebook systems.
-
Collecting, Curating, and Annotating Good Quality Speech deepfake dataset for Famous Figures: Process and Challenges
A speech-deepfake dataset for ten public figures built with transcription-based segmentation reports high synthetic naturalness (NISQA 3.69) and a human misclassification rate of 61.9%.
-
SmoothSinger: A Conditional Diffusion Model for Singing Voice Synthesis with Multi-Resolution Architecture
A reference-guided diffusion model with a low-frequency upsampling module achieves marginal quality improvements over prior SVS baselines on Opencpop, with significant caveats about statistical significance and reprod...
-
UmbraTTS: Adapting Text-to-Speech to Environmental Contexts with Flow Matching
UmbraTTS jointly synthesizes speech and environmental audio via conditional flow matching, conditioned on text and acoustic context, with controllable background volume.
-
CloneShield: A Framework for Universal Perturbation Against Zero-Shot Voice Cloning
A universal adversarial perturbation framework claiming to protect speech against zero-shot voice cloning by degrading cloned outputs while preserving input naturalness.
-
MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt
A multi-modal emotion prompt encoder and prosody predictor let MPE-TTS control emotion from speech, text, or image while preserving speaker timbre.
-
Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
A masked generative model pre-trained on unlabeled speech then fine-tuned per task matches or beats task-specific systems across TTS, voice conversion, speaker extraction, enhancement, and lip-to-speech.
-
VisualSpeech: Enhancing Prosody Modeling in TTS Using Video
Adding video-derived visual features to FastSpeech2 improves pitch, energy, and duration prediction on a 33-hour movie TTS dataset, with UTMOS rising from 2.91 to 3.13.
-
PROEMO: Prompt-Driven Text-to-Speech Synthesis Based on Emotion and Intensity Control
PROEMO extends FastSpeech 2 with HuBERT-based emotion and intensity encoders plus GPT-4 prompt scaling to generate multi-speaker emotional speech with controllable intensity.
-
Inference-time Scaling for Diffusion-based Audio Super-resolution
Generating 120 candidate super-resolved audios and choosing the best by task-specific verifiers improves speech, music, and sound effects over single-sample diffusion output, at 120x compute.
-
A Survey on Generative Model Unlearning: Fundamentals, Taxonomy, Evaluation, and Future Direction
A survey and framework that categorizes generative model unlearning by point-wise versus concept-wise objectives, parameter-based versus non-parametric methods, and completeness/utility/efficiency evaluation.
-
Technical report: Impact of Duration Prediction on Speaker-specific TTS for Indian Languages
In a five-language zero-shot TTS study, no single duration prediction strategy dominates: speaker-prompted durations help some languages, infilling durations help others, and results vary by metric.
-
RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modeling
A lip-to-speech model that predicts prosody from an audio prompt and content from lip-reading, then fuses them to synthesize speech, achieving strong benchmark results on LRS2 and LRS3.
-
VocalCrypt: Novel Active Defense Against Deepfake Voice Based on Masking Effect
VocalCrypt embeds masked pseudo-timbre signals into audio to disrupt AI voice cloning, but its experiments lack a no-defense baseline and do not show a clear advantage over prior defenses.
Discussion (0). Continue with ORCID to comment.