Pith. sign in

REVIEW 9 cited by

Autoregressive Speech Synthesis without Vector Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.08551 v2 pith:P5LFSZZ3 submitted 2024-07-11 cs.CL cs.SDeess.AS

Lingwei Meng , Long Zhou , Shujie Liu , Sanyuan Chen , Bing Han , Shujie Hu , Yanqing Liu , Jinyu Li
show 4 more authors
Sheng Zhao Xixin Wu Helen Meng Furu Wei
This is my paper
classification cs.CLcs.SDeess.AS
keywords mellelossmodelcomparedcontinuouscontinuous-valuedlanguagequantization
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS). MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition, bypassing the need for vector quantization, which is typically designed for audio compression and sacrifices fidelity compared to continuous representations. Specifically, (i) instead of cross-entropy loss, we apply regression loss with a proposed spectrogram flux loss function to model the probability distribution of the continuous-valued tokens; (ii) we have incorporated variational inference into MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity and model robustness. Experiments demonstrate that, compared to the two-stage codec language model VALL-E and its variants, the single-stage MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes, achieves superior performance across multiple metrics, and, most importantly, offers a more streamlined paradigm. The demos of our work are provided at https://aka.ms/melle.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

    eess.AS 2025-10 conditional novelty 6.0 of 10

    A single LLM can do ASR and zero-shot TTS on continuous speech features by switching between causal and bidirectional attention, reaching competitive but not state-of-the-art results.

  2. Next Tokens Denoising for Speech Synthesis

    cs.SD 2025-07 conditional novelty 6.0 of 10

    Dragon-FM generates speech autoregressively over two-second chunks while using flow matching inside each chunk, achieving fast synthesis at 12.5 discrete audio tokens per second.

  3. DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis

    cs.MM 2025-07 conditional novelty 6.0 of 10

    A single multimodal language model can generate intelligible speech and synchronized background audio jointly from a silent video, transcript, and reference voice, outperforming separately concatenated speech and audi...

  4. Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction

    eess.AS 2025-07 conditional novelty 6.0 of 10

    A causal audio language model with continuous-valued tokens and masked next-token prediction matches diffusion-based text-to-audio quality with smaller, streamable models.

  5. Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling

    cs.LG 2025-05 conditional novelty 6.0 of 10

    SMLLE generates speech frame-by-frame using a Transducer for streaming semantic tokens plus a fully autoregressive mel-spectrogram model, reaching quality close to sentence-level zero-shot TTS.

  6. CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis

    eess.AS 2025-08 conditional novelty 5.0 of 10

    CLEAR is a zero-shot TTS model that autoregressively predicts compact continuous audio latents with a per-token rectified flow head, reaching 1.88% WER on LibriSpeech Subset-B with an RTF of 0.29 and a 96 ms streaming delay.

  7. Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling

    eess.AS 2025-05 conditional novelty 5.0 of 10

    A hand-designed, training-free step schedule prunes later sampling steps in flow-matching TTS, cutting F5-TTS inference cost by about 4x while keeping quality roughly unchanged.

  8. UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information

    cs.SD 2025-05 conditional novelty 5.0 of 10

    The authors propose DistilCodec, a 32,768-code single-codebook audio codec, and UniTTS, a Qwen2.5-7B TTS model trained with audio, text, and cross-modal autoregressive tasks on interleaved prompts.

  9. Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment

    eess.AS 2025-05 conditional novelty 4.0 of 10

    A-DMA adds text and speech alignment losses to F5-TTS training, cutting the number of steps to a target quality while also improving WER and speaker similarity.

Pith tools