Pith. sign in

REVIEW 10 cited by

HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.12454 v2 pith:ODAWM5ZW submitted 2023-11-21 cs.SD cs.AIcs.MMeess.AS

Sang-Hoon Lee , Ha-Yeong Choi , Seung-Bin Kim , Seong-Whan Lee This is my paper · ORCID
classification cs.SDcs.AIcs.MMeess.AS
keywords speechsynthesiszero-shothierarchicalhierspeechmodelsrepresentationframework
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLM)-based speech synthesis has been widely adopted in zero-shot speech synthesis. However, they require a large-scale data and possess the same limitations as previous autoregressive speech models, including slow inference speed and lack of robustness. This paper proposes HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech (TTS) and voice conversion (VC). We verified that hierarchical speech synthesis frameworks could significantly improve the robustness and expressiveness of the synthetic speech. Furthermore, we significantly improve the naturalness and speaker similarity of synthetic speech even in zero-shot speech synthesis scenarios. For text-to-speech, we adopt the text-to-vec framework, which generates a self-supervised speech representation and an F0 representation based on text representations and prosody prompts. Then, HierSpeech++ generates speech from the generated vector, F0, and voice prompt. We further introduce a high-efficient speech super-resolution framework from 16 kHz to 48 kHz. The experimental results demonstrated that the hierarchical variational autoencoder could be a strong zero-shot speech synthesizer given that it outperforms LLM-based and diffusion-based models. Moreover, we achieved the first human-level quality zero-shot speech synthesis. Audio samples and source code are available at https://github.com/sh-lee-prml/HierSpeechpp.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ClaritySpeech: Dementia Obfuscation in Speech

    cs.CL 2025-07 conditional novelty 6.0 of 10

    An ASR, text-obfuscation, and zero-shot TTS pipeline lowers automatic dementia detection in speech by 10 to 16 percent F1 while improving intelligibility, with only moderate speaker similarity.

  2. SemAlignVC: Enhancing zero-shot timbre conversion using semantic alignment

    eess.AS 2025-07 conditional novelty 6.0 of 10

    SemAlignVC strips source-speaker timbre by aligning a speech semantic encoder to BERT text embeddings, then resynthesizes the content conditioned only on a target voice reference.

  3. Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement

    cs.SD 2025-02 conditional novelty 6.0 of 10

    Vevo achieves zero-shot timbre, accent, and emotion imitation by using VQ-VAE codebook size on HuBERT features to create content and content-style tokens.

  4. Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Continuous autoregressive text-to-speech with a Gaussian-mixture codec matches or beats a discrete-codec VALL-E baseline with a fraction of the language model parameters.

  5. Towards Better Disentanglement in Non-Autoregressive Zero-Shot Expressive Voice Conversion

    cs.SD 2025-06 conditional novelty 5.0 of 10

    A FreeVC-style conditional VAE with mHuBERT-147 discrete units, mixed-style layer normalization, an augmentation similarity loss, and F0 cross-attention reports better emotion transfer and less source leakage than thr...

  6. Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching

    eess.AS 2025-06 conditional novelty 5.0 of 10

    R-VC performs zero-shot voice conversion in two sampling steps while transferring the target speaker's rhythm, matching or exceeding prior systems in naturalness and intelligibility.

  7. Metis: A Foundation Speech Generation Model with Masked Generative Pre-training

    cs.SD 2025-02 conditional novelty 5.0 of 10

    A masked generative model pre-trained on unlabeled speech then fine-tuned per task matches or beats task-specific systems across TTS, voice conversion, speaker extraction, enhancement, and lip-to-speech.

  8. Overview of the Amphion Toolkit (v0.2)

    cs.SD 2025-01 conditional novelty 4.0 of 10

    Amphion v0.2 is an open-source toolkit for audio, music, and speech generation, adding a 101K-hour multilingual dataset, processing pipelines, and pretrained models.

  9. Low-Resource Text-to-Speech Synthesis Using Noise-Augmented Training of ForwardTacotron

    eess.AS 2025-01 conditional novelty 4.0 of 10

    Adding white-noise copies of five minutes of a target speaker's data and rebalancing sampling lets a small ForwardTacotron model beat zero-shot baselines on speaker similarity with only four high-resource speakers.

  10. Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey

    cs.CL 2024-12 conditional novelty 3.0 of 10

    A survey of controllable text-to-speech methods with a small pilot study using Gemini to automatically rate instruction following, naturalness, and expressiveness.

Pith tools