Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SMLLE shows that streaming, frame-by-frame text-to-speech can reach the quality of sentence-level systems without future-text chunks.

desk verdict A promising streaming TTS architecture, but the single-pass claim is undercut by a missing training detail: whether the AR stage is trained on oracle or Transducer-generated semantic tokens and alignments. read the letter →

arxiv 2505.19669 v2 pith:2ZTRMEXM submitted 2025-05-26 cs.LG

Haiyang Sun , Shujie Hu , Shujie Liu , Lingwei Meng , Hui Wang , Bing Han , Yifan Yang , Yanqing Liu
show 3 more authors
Sheng Zhao Yan Lu Yanmin Qian
This is my paper
classification cs.LG
keywords streamingtext-to-speechzero-shottransducerautoregressivemodelmel-spectrogramsemantictokensframe-by-framegenerationlow-latencyspeechsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SMLLE claims that streaming, frame-by-frame text-to-speech can match sentence-level quality without reading chunks of future text. The framework uses a Transducer to emit semantic tokens on the fly and, in the same pass, to reveal which text position each token aligns to; a fully autoregressive model then converts those tokens together with the aligned text into mel-spectrograms one frame at a time. On LibriSpeech test-clean, single-pass SMLLE reaches a 6.37% word error rate and 0.516 speaker similarity, close to the sentence-level baseline's 5.9% and 0.580, while sampling the Transducer five times improves it to 1.67% WER and 0.578 similarity. If the claim holds, low-latency real-time speech synthesis no longer has to sacrifice the voice-cloning quality that sentence-level models achieve.

What carries the argument

The load-bearing mechanism is the Transducer's monotonic alignment path: at each grid node the vertical emission $p(i,j)$ advances the semantic token while replicating the current text token, and the horizontal blank $\emptyset(i,j)$ advances the text, so the same path yields both the semantic token stream $Y$ and the duration-aligned text $X'$. The Delete $\langle\mathrm{Bos}\rangle$ Mechanism (DBM) then removes the non-phonetic initial token and pads with an equal number of $\langle\mathrm{eos}\rangle$ tokens, letting the autoregressive model access the next real text token earlier. The AR stage applies Eq. (4), factorizing the mel-spectrogram as $p(m_t \mid m_{<t}; x^D_{\le t}; y_{\le t}; \theta)$, and uses latent sampling with a mean-and-variance linear head followed by a residual MLP to produce each frame. This chain carries the streaming property because nothing in either stage requires future text beyond the current token.

What would settle it

Force-align the Transducer's emitted semantic tokens to the text with an independent aligner on the test set and measure per-token insertion, deletion, and time-skew rates; if misalignment is high or correlates with WER and listening artifacts, the streaming claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a zero-shot voice can be generated frame by frame in real time by splitting synthesis into two streaming stages. The first stage is a Transducer that generates semantic tokens monotonically from text, producing both the token stream and a duration-aligned copy of the text. The second stage is an autoregressive model that predicts each mel-spectrogram frame conditioned on all previous mel frames, the current semantic token, and the current aligned text token, using latent sampling to stabilize generation. The empirical case rests on Table 1: single-pass SMLLE achieves 6.37% WER-H and 0.516 SIM, versus 5.9% and 0.580 for the sentence-level baseline, and the five-sample variant reaches 1.67% and 0.578. In the direct streaming comparison, SMLLE's 6.66% WER beats the frame-by-frame baseline's 40.7% and matches the first-token latency of one text step plus one model step.

Load-bearing premise

The whole pipeline assumes that the Transducer's monotonic text-to-semantic-token alignment is accurate enough that skipped or mistimed tokens do not accumulate into audible errors, but the paper reports no direct measurement of that alignment accuracy.

Editorial extensions

If this is right

  • Frame-by-frame streaming generation can reach word error rates near sentence-level systems: 6.37% WER-H in single pass and 1.67% after five transducer samplings, against 5.9% for the sentence-level baseline.
  • First-token latency can be reduced to roughly one text token plus one model step, instead of waiting for several future text chunks.
  • Improving the Transducer's semantic-token prediction and alignment is the main lever for better streaming quality, since repeated sampling yields steady WER and SIM gains.
  • The Delete $\langle\mathrm{Bos}\rangle$ Mechanism stabilizes generation when applied to the generated segment, but applying it to the prompt hurts performance, so prompt conditioning should use a different pattern.
  • A stronger streaming token predictor could close the remaining gap to the best sentence-level systems without changing the autoregressive acoustic stage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to replace the paper's five-sampling rerun with beam search or alignment-confidence rescoring; the gap between 6.37% and 1.67% WER suggests such a search could recover most of the gain in a single pass.
  • The two-stage design may transfer to real-time spoken dialogue, where text arrives incrementally and may be corrected on the fly, but that setting would stress-test the monotonic alignment assumption much harder than read audiobook speech.
  • Because SMLLE is trained only on LibriSpeech, its zero-shot claim is tested in a narrow read-speech register; conversational or overlapping speech would reveal whether the streaming alignment and the autoregressive reconstruction generalize beyond that register.
  • A direct measurement of Transducer alignment error would tell whether the AR stage needs a recovery mechanism; the paper does not report such a measurement, leaving that the main open engineering question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes SMLLE, a two-stage streaming text-to-speech system. Stage 1 uses a Transducer to convert phoneme text into semantic tokens (first-layer SpeechTokenizer codes) while simultaneously producing a duration-aligned text sequence. Stage 2 uses a fully autoregressive mel-spectrogram model, based on MELLE, that conditions on the semantic tokens and duration-aligned text to generate mel frames frame-by-frame. A Delete-BOS mechanism (DBM) removes the initial <bos> text token so that the AR model can access future text with minimal latency. Experiments on LibriSpeech and LibriTTS report word error rates, speaker similarity, MOS/CMOS, and first-token latency, comparing against sentence-level TTS systems (VALL-E, MELLE, Voicebox, etc.) and one streaming baseline (LiveSpeech2). The paper claims that SMLLE outperforms existing streaming TTS methods and achieves performance comparable to sentence-level zero-shot TTS systems.

Significance. If the single-pass results are reproducible, SMLLE is a meaningful contribution: it is among the first systems to combine Transducer-based streaming semantic-token generation with a fully autoregressive streaming mel decoder, and it demonstrates plausible zero-shot quality with low latency on standard benchmarks. The paper includes a demo page, uses several widely adopted objective metrics, and provides subjective evaluations. However, the load-bearing evidence currently has important gaps: the AR model's training-data source is unspecified, the headline R5 results rely on oracle selection of the best sample, and the comparison against streaming baselines is limited to a single prior system with numbers taken from that system's paper. These gaps prevent the paper from fully supporting its central claims as written.

major comments (3)
  1. [§2.2, Eq. (4); §3.2] The paper never states whether the AR model in Eq. (4) is trained with oracle semantic tokens and alignment (e.g., from SpeechTokenizer and a forced aligner) or with Transducer-generated outputs. At inference, the AR model consumes top-k sampled Transducer outputs, which may contain alignment and token errors. If training used oracle inputs, the Table 1 single-pass results are not a valid test of the integrated streaming system; if training used Transducer outputs, the paper should report token/alignment error rates or some other measure of conditioning fidelity. This gap is load-bearing because Figure 4 and Table 4 demonstrate strong sensitivity to the conditioning distribution.
  2. [§4.1, Table 1; Figure 4] The SMLLE-R5 results are obtained by selecting the best of five Transducer samples, and Figure 4 explicitly refers to the 'optimal' results. This requires oracle knowledge of the evaluation metric at inference time, and the paper does not provide a practical selection criterion that a streaming system could use. Consequently, the R5 numbers are not achievable by a deployable system. The abstract and conclusion are based on single-pass numbers, but §4.1 uses R5 to claim that SMLLE surpasses MELLE and Voicebox, which overstates the practical result. Please report single-pass as the primary result and clearly label R5 as an oracle upper bound.
  3. [§4.1, Table 2; §1] The comparison with streaming TTS baseline is limited to LiveSpeech2 (two variants), with results taken from [29]; none of the Transducer-based streaming TTS systems cited in §1 (Speech-T, Transduce and Speak, Token Transducer + GMLM, TTS-Transducer) is directly evaluated or compared. Thus the abstract's claim that SMLLE 'outperforms current streaming TTS methods' is not supported by the experiments as presented. In addition, no error bars or significance tests are reported for any objective or subjective metric, and the Table 1 baselines are copied from [8]; the 'comparable to VALL-E' claim is therefore not statistically grounded.
minor comments (6)
  1. [Eq. (2)] In the forward recursion, the first term should multiply by the blank probability ∅(i−1,j) rather than the emission probability p(i−1,j); as written, Eq. (2) is inconsistent with the text's definitions of p and ∅.
  2. [§1, §2.1, Table 2] There are several typos and formatting inconsistencies: 'sementic' should be 'semantic' (§2.1), 'LibeSpeech2' should be 'LiveSpeech2' (Table 2), 'necesssary' should be 'necessary' (§1), and the spacing in 'V ALL-E' is inconsistent throughout the manuscript.
  3. [§3.3] The paper should specify how the 'best' sample is chosen in the repeated-sampling experiment; if it is oracle selection using ground-truth WER or SIM, state that explicitly because it materially affects the interpretation of Figure 4 and the R5 results.
  4. [§2.2.2] It is not stated whether the latent variable z_t is sampled from the learned distribution or set to its mean at inference; please clarify this for reproducibility, as the latent sampling scheme affects output diversity and quality.
  5. [Table 3] MOS and CMOS are reported without confidence intervals or significance tests; with only 12 and 6 listeners respectively, the observed differences between systems may not be statistically reliable.
  6. [§2.2.1, Table 4] The DBM ablation only toggles whether DBM is applied to the prompt; it does not isolate the effect of DBM on the generated speech. A condition without DBM in both prompt and generation would more directly support the claim that DBM improves performance.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SMLLE is an empirical system whose central claims are validated against external ASR and speaker-similarity benchmarks; reuse of prior MELLE and SpeechTokenizer components is prior-art adoption, not a circular derivation.

full rationale

The paper's derivation chain is constructional, not circular. The Transducer stage (Eq. 1) minimizes the negative log-probability of semantic token sequences marginalized over monotonic alignments, and the AR stage (Eq. 4) models mel-spectrogram frames conditioned on previous mel frames, duration-aligned text, and semantic tokens. These are architectural definitions; mel-spectrograms are not defined in terms of the text or semantic tokens, and no predicted quantity is equal by construction to a fitted input. The headline metrics (WER-C, WER-H, SIM, MOS, CMOS) are computed with external systems (Conformer-Transducer, HuBERT-Large, WavLM-TDNN, human raters), not derived from the model's own training objectives. The reliance on MELLE [8] is a reuse of a prior architecture with overlapping authors, but MELLE is an independently published system and the present contribution is the streaming integration with a Transducer; this does not make the central claim load-bearing on an unverified self-citation. The 'SMLLE-R5' best-of-five selection is a sampling protocol and a soundness concern (oracle-style selection), not an equation-level reduction or a fitted-parameter-renamed-as-prediction. The under-specified question of whether AR training uses oracle or Transducer-generated semantic tokens and alignments could affect whether the single-pass results test the integrated system, but that is an experimental-validity concern, not circularity: nothing in the paper's equations makes the reported WER or SIM scores equivalent to the training inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The listed free parameters are hand-chosen loss weights and inference knobs with no ablation; the axioms are the main domain assumptions behind the zero-shot streaming claim.

free parameters (4)
  • KL loss weight lambda = 5e-2, activated after 10k steps
    Weight in Eq. (7) for the AR loss; chosen by hand, no ablation reported.
  • spectrogram flux loss weight beta = 0.5
    Weight in Eq. (7); chosen by hand, no ablation reported.
  • Transducer inference top-k = 15
    Used for semantic token sampling in Section 3.3; no sweep reported.
  • AR input mel dropout rate = 0.5
    Applied to prompt mel-spectrograms during inference in Section 3.3; stated but not ablated.
assumptions (4)
  • domain assumption SpeechTokenizer first-layer semantic tokens are a sufficient text-related conditioning signal for mel-spectrogram reconstruction.
    Section 2.1 says semantic tokens are 'text-related representations that are decoupled from speech' and the AR model relies on them in Eq. (4); no experiment shows residual-code information is unnecessary.
  • domain assumption Monotonic alignment between text and semantic tokens is correct enough for frame-level conditioning.
    Transducer training marginalizes over monotonic paths in Eq. (2), and inference uses the path to build duration-aligned text; no alignment-error analysis is provided.
  • domain assumption ASR WER and WavLM-TDNN speaker similarity are valid proxies for TTS quality.
    Section 3.3 uses WER-C, WER-H, and SIM as primary objective metrics; these measure intelligibility and speaker identity, not prosody or naturalness directly.
  • domain assumption LibriSpeech test-clean speakers are unseen and representative of zero-shot conditions.
    Section 3.3 ensures test speakers are not in training data, but all data come from LibriSpeech and LibriTTS read audiobooks; generalization to arbitrary speakers and domains is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling." pith.science (2026). https://pith.science/paper/2ZTRMEXM

@misc{pith2026250519669,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZTRMEXM}},
  note         = {Machine review of arXiv:2505.19669}
}
read the original abstract

Zero-shot streaming text-to-speech is an important research topic in human-computer interaction. Existing methods primarily use a lookahead mechanism, relying on future text to achieve natural streaming speech synthesis, which introduces high processing latency. To address this issue, we propose SMLLE, a streaming framework for generating high-quality speech frame-by-frame. SMLLE employs a Transducer to convert text into semantic tokens in real time while simultaneously obtaining duration alignment information. The combined outputs are then fed into a fully autoregressive (AR) streaming model to reconstruct mel-spectrograms. To further stabilize the generation process, we design a Delete < Bos > Mechanism that allows the AR model to access future text introducing as minimal delay as possible. Experimental results suggest that the SMLLE outperforms current streaming TTS methods and achieves comparable performance over sentence-level TTS systems. Samples are available on shy-98.github.io/SMLLE_demo_page/.

Figures

Figures reproduced from arXiv: 2505.19669 by the authors.

Figure 1
Figure 1. The overview of SMLLE. tency. For instance, Speech-T [16] employs a Transducer to di￾rectly model text-to-speech alignment, improving performance by constraining the alignment path during training. However, while it performs well in single-speaker scenarios, it lacks zero￾shot capability for generating speech from unseen speakers. Other approaches [17, 18] use a Transducer to generate se￾mantic tokens, text-related … view at source ↗
Figure 3
Figure 3. The AR model utilizes both duration-aligned text and semantic tokens to provide essential information for mel￾spectrogram reconstruction. The latent sampling module is in￾troduced to enhance the model’s generalization capability. The relationship between x D and x is indicated in the brackets. of the probabilities of all paths is calculated using an efficient forward algorithm. The probability of reaching the node α… view at source ↗
Figure 4
Figure 4. The repeated sampling ablation experiment, with the horizontal axis representing the number of repeated samplings. The SIM score is scaled by a factor of 10. 4.2. Analysis on Repeated Sampling To investigate how the Transducer model’s output influences the SMLLE model and explore its full potential, the optimal re￾sults achieved at varying sampling frequencies during the five￾sampling experiment, are illustrated in … view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Next Tokens Denoising for Speech Synthesis

    cs.SD 2025-07 conditional novelty 6.0 of 10

    Dragon-FM generates speech autoregressively over two-second chunks while using flow matching inside each chunk, achieving fast synthesis at 12.5 discrete audio tokens per second.

  2. StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling

    cs.SD 2025-06 conditional novelty 6.0 of 10

    StreamMel interleaves text tokens with continuous mel frames in one autoregressive model, reaching state-of-the-art streaming latency with quality comparable to offline zero-shot TTS on LibriSpeech.

Reference graph

Works this paper leans on

36 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [29]

    ELLA-V: Sta- ble neural codec language modeling with alignment-guided se- quence reordering,

    Y . Song, Z. Chen, X. Wang, Z. Ma, and X. Chen, “ELLA-V: Sta- ble neural codec language modeling with alignment-guided se- quence reordering,” arXiv preprint arXiv:2401.07333, 2024

    arXiv 2024

  2. [8]

    It uses a Transducer model to convert text into semantic tokens in real time and reconstructs them into mel-spectrograms frame by frame using an AR model

    Conclusion In this paper, we propose a novel zero-shot streaming TTS framework, SMLLE. It uses a Transducer model to convert text into semantic tokens in real time and reconstructs them into mel-spectrograms frame by frame using an AR model. We further design a DBM mechanism that allows SMLLE to ac- cess future text earlier, introducing as minimal delay a...

    work page

  3. [1]

    <eos> <bos> 𝑦! <eos> 𝑦#<bos> 𝑦

    Introduction In recent years, large language models (LLMs) have made remarkable advancements across various domains, including natural language processing (NLP) [1, 2] and computer vi- sion (CV) [3, 4]. Similarly, the field of speech synthesis has seen significant progress in autoregressive (AR) language mod- eling, with both discrete codec-based [5, 6, 7...

    work page

  4. [2]

    Specifically, it uses a Transducer to convert the text into a sequence of se- mantic token in real time

    We propose the first work of zero-shot streaming TTS model in frame-by-frame mode, SMLLE . Specifically, it uses a Transducer to convert the text into a sequence of se- mantic token in real time. At the same time, a fully AR model converts these semantic tokens and texts into mel- spectrograms, frame by frame

    work page

  5. [3]

    Delete ⟨Bos⟩ Mechanism

    We introduce a novel “Delete ⟨Bos⟩ Mechanism” (DBM) for SMLLE. This mechanism allows the model to access necesssary future text introducing as minimal delay as possi- ble, thereby improving the quality of speech reconstruction

    work page

  6. [4]

    𝑥#𝑥$y! 𝑦

    The experimental results show that SMLLE achieves perfor- arXiv:2505.19669v2 [cs.LG] 2 Jun 2025 𝑥!𝑥"𝑥#𝑥$y! 𝑦" 𝑦# 𝑦$ 𝑦% 𝑦& 𝑦' 𝑝(0,0) 𝑝(0,1)∅(0,2)𝑝(1,2)∅(1,3)𝑝(2,3) 𝑝(2,4)∅(2,5)𝑝(3,5)∅(3,6) Figure 2: The probabilistic path graph of the Transducer, where the red paths represent the possible alignment paths. mance on par with zero-shot non-streaming TTS model...

    arXiv 2025

  7. [5]

    𝑚# 𝑚$ 𝑚% 𝑚& 𝑚'…… 𝑚! 𝑚% 𝑚

    SMLLE SMLLE generates high-quality speech using a two-stage mod- eling. In the Transducer Stage, SMLLE converts the text se- quence into a semantic tokens sequence in a streaming man- ner. In the Autoregressive Stage, it uses the semantic tokens and texts to reconstruct mel-spectrograms frame by frame. 2.1. Transducer Stage The Transducer is proposed to m...

    work page

  8. [6]

    Training Datasets We train SMLLE on the LibriSpeech dataset [27]

    Experimental Setup 3.1. Training Datasets We train SMLLE on the LibriSpeech dataset [27]. It contains approximately 960 hours of English speech. For text, we use eSpeak for phoneme extraction. For the AR model, we extract 80-dimensional log-magnitude mel-spectrograms as the target. 3.2. Experimental Settings Model Configurations For the Transducer model, ...

    work page

Show all 36 references
  1. [7]

    SMLLE-R5

    Results and Discussion 4.1. Main Results Objective Evaluation. As shown in Table 1, the proposed frame-by-frame streaming TTS model, SMLLE achieves per- formance comparable to existing sentence-level TTS systems, while operating in real time. Specifically, in terms of speaker ...

  2. [9]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Ale- man, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  3. [10]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Let- man, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [11]

    Zero-shot text-to-image generation,

    A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. V oss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in International conference on machine learning . Pmlr, 2021, pp. 8821–8831

    2021

  5. [12]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  6. [13]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Neural codec language mod- els are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  7. [14]

    Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,

    Z. Zhang, L. Zhou, C. Wang, S. Chen, Y . Wu, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,”arXiv preprint arXiv:2303.03926, 2023

    2023 arXiv

  8. [16]

    Autoregressive speech synthesis without vector quantization,

    L. Meng, L. Zhou, S. Liu, S. Chen, B. Han, S. Hu, Y . Liu, J. Li, S. Zhao, X. Wu et al., “Autoregressive speech synthesis without vector quantization,” arXiv preprint arXiv:2407.08551, 2024

    2024 arXiv

  9. [17]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar´e, M. Orsini, A. Royer, P. P´erez, H. J´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  10. [18]

    Mini-omni: Language models can hear, talk while thinking in streaming,

    Z. Xie and C. Wu, “Mini-omni: Language models can hear, talk while thinking in streaming,” arXiv preprint arXiv:2408.16725 , 2024

    2024 arXiv

  11. [19]

    Llama- omni: Seamless speech interaction with large language models,

    Q. Fang, S. Guo, Y . Zhou, Z. Ma, S. Zhang, and Y . Feng, “Llama- omni: Seamless speech interaction with large language models,” arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  12. [20]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023

    2023 arXiv

  13. [21]

    Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  14. [22]

    Wavllm: Towards ro- bust and adaptive speech large language model,

    S. Hu, L. Zhou, S. Liu, S. Chen, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran, L. Liu et al. , “Wavllm: Towards ro- bust and adaptive speech large language model,” arXiv preprint arXiv:2404.00656, 2024

    2024 arXiv

  15. [23]

    Livespeech: Low- latency zero-shot text-to-speech via autoregressive modeling of audio discrete codes,

    T. Dang, D. Aponte, D. Tran, and K. Koishida, “Livespeech: Low- latency zero-shot text-to-speech via autoregressive modeling of audio discrete codes,” arXiv preprint arXiv:2406.02897, 2024

    2024 arXiv

  16. [24]

    Speech-t: Transducer for text to speech and beyond,

    J. Chen, X. Tan, Y . Leng, J. Xu, G. Wen, T. Qin, and T.-Y . Liu, “Speech-t: Transducer for text to speech and beyond,” Advances in Neural Information Processing Systems , vol. 34, pp. 6621– 6633, 2021

    2021

  17. [25]

    Transduce and speak: Neural transducer for text-to-speech with semantic to- ken prediction,

    M. Kim, M. Jeong, B. J. Choi, D. Lee, and N. S. Kim, “Transduce and speak: Neural transducer for text-to-speech with semantic to- ken prediction,” in2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–7

    2023

  18. [26]

    High fidelity text-to-speech via discrete tokens using to- ken transducer and group masked language model,

    J. Y . Lee, M. Jeong, M. Kim, J.-H. Lee, H.-Y . Cho, and N. S. Kim, “High fidelity text-to-speech via discrete tokens using to- ken transducer and group masked language model,”arXiv preprint arXiv:2406.17310, 2024

    2024 arXiv

  19. [27]

    Tts-transducer: End-to-end speech synthesis with neural transducer,

    V . Bataev, S. Ghosh, V . Lavrukhin, and J. Li, “Tts-transducer: End-to-end speech synthesis with neural transducer,” arXiv preprint arXiv:2501.06320, 2025

    2025 arXiv

  20. [28]

    Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtok- enizer: Unified speech tokenizer for speech large language mod- els,” arXiv preprint arXiv:2308.16692, 2023

    2023 arXiv

  21. [30]

    V ALL-E R: Robust and efficient zero- shot text-to-speech synthesis via monotonic alignment,

    B. Han, L. Zhou, S. Liu, S. Chen, L. Meng, Y . Qian, Y . Liu, S. Zhao, J. Li, and F. Wei, “V ALL-E R: Robust and efficient zero- shot text-to-speech synthesis via monotonic alignment,” arXiv preprint arXiv:2406.07855, 2024

    2024 arXiv

  22. [31]

    RALL-E: Robust codec language modeling with chain-of-thought prompting for text-to- speech synthesis,

    D. Xin, X. Tan, K. Shen, Z. Ju, D. Yang, Y . Wang, S. Takamichi, H. Saruwatari, S. Liu, J. Li et al. , “RALL-E: Robust codec language modeling with chain-of-thought prompting for text-to- speech synthesis,” arXiv preprint arXiv:2404.03204, 2024

    2024 arXiv

  23. [32]

    CLaM-TTS: Improving neural codec language model for zero-shot text-to-speech,

    J. Kim, K. Lee, S. Chung, and J. Cho, “CLaM-TTS: Improving neural codec language model for zero-shot text-to-speech,” inThe Twelfth International Conference on Learning Representations , 2024

    2024

  24. [33]

    V ALL-E 2: Neural codec language models are hu- man parity zero-shot text to speech synthesizers,

    S. Chen, S. Liu, L. Zhou, Y . Liu, X. Tan, J. Li, S. Zhao, Y . Qian, and F. Wei, “V ALL-E 2: Neural codec language models are hu- man parity zero-shot text to speech synthesizers,” arXiv preprint arXiv:2406.05370, 2024

    2024 arXiv

  25. [34]

    V oicebox: Text-guided multilingual universal speech gen- eration at scale,

    M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V . Manohar, Y . Adi, J. Mahadeokar, and W.-N. Hsu, “V oicebox: Text-guided multilingual universal speech gen- eration at scale,” in Thirty-seventh Conference on Neural Infor- mation Processing Systems, 2023

    2023

  26. [35]

    Lib- rispeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An ASR corpus based on public domain audio books,” in ICASSP, 2015, pp. 5206–5210

    2015

  27. [36]

    LibriTTS: A corpus derived from librispeech for text- to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A corpus derived from librispeech for text- to-speech,” arXiv preprint arXiv:1904.02882, 2019

    1904 arXiv

  28. [37]

    Zero- shot text-to-speech from continuous text streams,

    T. Dang, D. Aponte, D. Tran, T. Chen, and K. Koishida, “Zero- shot text-to-speech from continuous text streams,” arXiv preprint arXiv:2410.00767, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.