Pith. sign in

REVIEW 2 major objections 5 minor 55 references

A Variational Framework for Improving Naturalness in Generative Spoken Language Models

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a variational autoencoder can learn the continuous prosodic and paralinguistic attributes that semantic speech tokens omit, and that augmenting a token-based speech language model with these learned features…

desk verdict A well-executed, honestly reported paper whose headline N-MOS gain looks credible, but the central mechanism claim is under-evidenced because the supporting analyses use posterior features while generation uses prior-sampled features. read the letter →

arxiv 2506.14767 v1 pith:7CWSHXCZ submitted 2025-06-17 cs.CL cs.AIcs.LGcs.SDeess.AS

Li-Wei Chen , Takuya Higuchi , Zakaria Aldeneh , Ahmed Hussen Abdelaziz , Alexander Rudnicky This is my paper
classification cs.CLcs.AIcs.LGcs.SDeess.AS
keywords generativespokenlanguagemodelingvariationalautoencoderprosodysemantictokensparalinguisticfeaturesnaturalnessspeechcontinuationnormalizingflow
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the naturalness shortfall of token-based spoken language models is not a decoder problem but an input representation problem: discrete semantic tokens carry linguistic content but drop the continuous, paralinguistic side of speech. The authors propose training a variational autoencoder jointly with the autoregressive model so that a low-dimensional set of continuous latent features is learned to reconstruct the mel-spectrogram and to make next-token prediction easier. The learned features replace hand-engineered pitch features, and human raters prefer the resulting continuations over those of pitch-augmented and acoustic-token baselines. If the claim holds, a straightforward architectural addition of under one percent more parameters could make spoken language models sound more natural without sacrificing language ability.

What carries the argument

The load-bearing object is the variational posterior $q_\phi(z^c_t | X) = \mathcal{N}(z^c_t; \mu_\phi(X,t), \sigma_\phi(X,t))$ with the factorization $q_\phi(Z^c|X) = \prod_t q_\phi(z^c_t|X)$, paired with an autoregressive prior $p_\psi(Z^c)=\prod_t p_\psi(z^c_t|Z^c_{1:t-1})$ whose expressiveness is increased by a time-wise normalizing flow shared across frames. This pair converts the standard ELBO into the training objective $O_{ELBO} = O_{rec} - \beta (L^c_{kl} + \gamma L^d_{kl})$: $O_{rec}$ reconstructs the mel-spectrogram from $Z^d$ and $Z^c$ together, $L^c_{kl}$ makes the prior predict the next variational feature, and $L^d_{kl}$ makes the model predict the next semantic token. The scalars $\beta$ and $\gamma$ trade reconstruction fidelity against how predictable the continuous features are, and the flow lets the prior fit non-Gaussian feature distributions.

What would settle it

Run the identical framework on a tonal language such as Mandarin or Cantonese and compare human N-MOS against the pitch-augmented baseline; a statistically significant shrinkage of the naturalness margin relative to English would indicate that the frame-wise independence assumption loses long-range prosodic structure.

Watch

Extended reading notes

Core claim

The central claim is that continuous latent variables learned end-to-end can supply exactly the information that semantic tokens omit. Formally, the paper introduces a joint latent variable $Z = (Z^c, Z^d)$, where $Z^d$ are discrete HuBERT-derived semantic tokens and $Z^c$ are continuous variational features, and optimizes an ELBO with three terms: a reconstruction objective that forces $Z^c$ to complement $Z^d$ when decoding the mel-spectrogram, a KL term that makes an autoregressive prior predict $Z^c$ from past context, and a token prediction loss that keeps $Z^d$ predictable from the joint history. With a diagonal-Gaussian posterior, an autoregressive prior enhanced by a time-wise normalizing flow, and a diffusion decoder held fixed across methods, the learned $Z^c$ capture paralinguistic information: emotion classification reaches 91.57 percent accuracy from the variational features alone versus 57.46 percent from tokens. In human evaluation, continuations generated with the proposed model score N-MOS 3.60 versus 3.19 for Token-LM and 3.08 for Token-LM plus pitch, while meaningfulness stays comparable or higher.

Load-bearing premise

The variational posterior is assumed to factorize across time frames, so each frame's continuous feature is drawn independently given the whole utterance; prosody, however, lives in long-range pitch and rhythm contours that span many frames.

Editorial extensions

If this is right

  • Speech language models can drop hand-engineered paralinguistic feature extraction, such as pitch trackers and manual feature selection, and learn the needed continuous attributes end-to-end from the mel-spectrogram.
  • The learned variational features encode paralinguistic content beyond pitch: they nearly double emotion-recognition accuracy over semantic tokens alone and sharply improve speaker identification.
  • The framework is not tied to one tokenizer: swapping HuBERT tokens for SpeechTokenizer first-level RVQ tokens still yields higher naturalness and meaningfulness than the token-only baseline.
  • Increasing the weight on semantic-token prediction shifts the variational features toward phonetic content, while lowering it favors pitch-related content, giving a practical knob for what the continuous features store.
  • Reconstruction and continuation are separable: acoustic-token augmentation reconstructs best but continues worst, so the naturalness gain comes from keeping the variational features predictable, not from maximizing reconstruction fidelity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step would be to replace the factorized posterior with a temporally correlated posterior, such as an autoregressive or flow-based posterior, so that long-range prosodic contours are represented explicitly; the current model leaves this door open by construction.
  • The variational features could serve as a control interface for expressive generation: interpolating or swapping $Z^c$ between utterances might transplant speaking style, emotion, or speaker mannerisms while keeping linguistic content in $Z^d$.
  • If the framework transfers across languages, it would be a particularly cheap upgrade for tonal languages, where pitch is lexically meaningful and hand-picked pitch features are especially inadequate; this is explicitly untested in the paper.
  • Combining this variational input augmentation with text-pretrained speech language models may recover the sWUGGY and sBLIMP gap while keeping the naturalness gain, since the two mechanisms address different weaknesses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a variational framework for generative spoken language modeling in which a continuous latent sequence Zc is learned end-to-end to complement discrete semantic tokens Zd. The model couples a VAE with an autoregressive prior to a token-based speech language model; the objective in Eq. (5) combines reconstruction, a KL term that makes Zc predictable from the prior, and semantic-token prediction. Experiments on LibriSpeech and Libri-light compare Token-LM, Token-LM + Pitch, Token-LM + Acoustic, and the proposed method using reconstruction metrics (F0-RMSE, MCD, CER), ZeroSpeech metrics (sWUGGY, sBLIMP), and human ratings (M-MOS, N-MOS). The headline result is that the proposed method reaches N-MOS 3.60 (±0.10) vs. 3.19 (±0.11) for Token-LM in Table 2, with comparable M-MOS. Additional experiments examine loss-balancing hyperparameters β and γ, removal of semantic tokens, generalization to SpeechTokenizer tokens, and side analyses of emotion and speaker information in the learned features.

Significance. The ELBO derivation is standard and the experimental protocol is mostly solid: human ratings use 7 raters per item with 95% confidence intervals, and the main N-MOS difference is non-overlapping with the strongest baseline. Code, models, and samples are released, and the paper is candid about limitations (hyperparameter sensitivity, English-only evaluation, small model scale). If the naturalness gain is truly attributable to learned variational features, the method would be a useful step toward removing hand-engineered paralinguistic feature extraction in speech LMs. However, the evidence for the mechanism is incomplete, because the paralinguistic-content analyses rely on posterior features while generation uses prior-sampled features. This gap is fixable with additional analysis, so the contribution is promising but not yet fully established.

major comments (2)
  1. [§4.2, §4.3.1, Appendix H] The central mechanism claim is under-evidenced by a distribution mismatch. Reconstruction in §4.3.1 uses the posterior mean µφ(X,t), and the emotion/speaker classifiers in Appendix H are evaluated on features extracted from observed utterances via the encoder, i.e., posterior features. Speech continuation in Table 2, however, samples Zc from the autoregressive prior pψ using Eq. (6) with temperature 0.85 (§4.2). The paper therefore demonstrates that posterior Zc carries paralinguistic information, but the generation-time Zc comes from a different distribution. If prior-sampled Zc collapses toward generic low-information features, the N-MOS gain could be due to the extra continuous conditioning channel or the utterance encoder rather than to the learned paralinguistic attributes, and the abstract's claim that the model 'automatically learns to encode these continuous speech attributes' would not be supported. Please add an analysis of prior-sampled features (e.g., emotion/speaker classification or distributional statistics of prior vs. posterior samples) or an ablation that replaces Zc with a fixed or random continuous input.
  2. [§4.2, §5.1] The claim that manual extraction and selection of paralinguistic features is unnecessary is tested only against a single hand-crafted feature, log pitch from CREPE. The introduction argues that pitch alone is insufficient and that energy- and spectral-related features are also important, but no baseline using such a feature set is included. The experiments therefore support 'learned features are competitive with pitch augmentation' but not the stronger claim that learned features eliminate the need for feature engineering. Adding a baseline with a small hand-crafted set (e.g., F0 + energy + MFCCs, or a standard paralinguistic feature set) would directly test this motivation.
minor comments (5)
  1. [§3.1, Eq. (2)] The factorized posterior qφ(Zc|X)=∏_t qφ(z_t^c|X) assumes each frame's latent is conditionally independent of other frames given the full utterance. Prosodic phenomena such as intonation contours and rhythm are temporally correlated, and the posterior's inability to represent these correlations likely shapes what Zc learns during training, even though the autoregressive prior can generate correlated samples at inference. A diagnostic such as autocorrelation of posterior features or a prosody-transfer experiment would clarify whether this assumption is benign for the claimed naturalness gains.
  2. [§5.1] The claim that 'we analyze the ASR transcriptions from the compared methods' and that human raters needed multiple passes to comprehend the Token-LM output is qualitative and not supported by reported data; please include the transcriptions or a quantitative measure such as listening time or a second ASR pass.
  3. [§4.2, Appendix C] The diffusion decoder is trained jointly with the proposed method but separately (to 500k steps) for Token-LM and Token-LM + Pitch; this training discrepancy should be stated in the main text or empirically justified, as it could affect the reconstruction comparison in Table 1.
  4. [Appendix G] For methods with additional continuous or auxiliary inputs, sWUGGY and sBLIMP scores are computed using only the log-likelihood of the discrete tokens, while for Proposed (−tokens) the Zc likelihood is used; this asymmetry should be discussed because it makes these metrics not fully comparable across rows.
  5. [Abstract] The term 'end-to-end' may overstate the setup because the semantic tokenizer is fixed and not trainable; consider saying the variational feature extractor and autoregressive prior are trained end-to-end.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the variational objective is ELBO-derived and the naturalness/paralinguistic evidence is externally evaluated.

full rationale

The paper's central claim is that a jointly trained variational encoder and autoregressive prior improve the naturalness of speech continuations. This claim is not circular: the training objective in Eq. 5 is a standard ELBO decomposition, the headline N-MOS improvement in Table 2 (3.60 vs. 3.19 for Token-LM) is produced by external human raters on generated continuations, and the supporting paralinguistic-capacity evidence in Appendix H comes from training emotion and speaker classifiers on held-out datasets rather than reusing the model's own training targets. The only self-citation (Chen et al., 2023) appears in related work on vector-quantized TTS and is not load-bearing. The skeptical concern that paralinguistic evidence is collected from posterior features while generation samples from the autoregressive prior (Eq. 6 with temperature 0.85) is an evidentiary gap about distribution shift, not a definitional or fitted-input reduction; the human-rated naturalness score remains an externally falsifiable outcome. Model assumptions such as the factorized posterior in Eq. 2 are correctness risks, not circularity. No equation in the paper reduces a prediction to its own input, and no self-citation chain forces the conclusion.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard VAE/flow machinery plus a temporal factorization assumption and the tuning of two loss-weight scalars. No new physical entities are introduced; the variational features are learned representations, validated on external emotion and speaker-ID benchmarks.

free parameters (4)
  • β (loss weight for KL terms) = 0.04 (main Libri-light), 0.05 (LibriSpeech sweep)
    Balances reconstruction versus autoregressive predictability; swept over {0.03, 0.04, 0.05}.
  • γ (relative weight of semantic token loss) = 0.5 (main), swept {0.5, 1.0, 2.0}
    Controls trade-off between predicting discrete tokens and continuous features; lower values favored for MOS.
  • Latent dimension d_z^c = 4
    Chosen from {4, 16, 64} to keep parameter count comparable; larger dimensions improve metrics but increase parameters.
  • Semantic vocabulary size k = 200
    Selected from {50, 200, 1000} based on LM performance; consistent with Maiti et al. (2024).
assumptions (4)
  • domain assumption The variational posterior factorizes across time: q(z^c_t|X) are conditionally independent given X.
    Eq. 2 and Section 3.1; this is a strong simplification for temporally correlated speech features.
  • domain assumption The discrete semantic token z^d_t and continuous feature z^c_t are conditionally independent given past context.
    Appendix I; required for factoring the ELBO into separate cross-entropy and KL terms.
  • domain assumption The pretrained HuBERT tokenizer (with k-means) provides a faithful, fixed distribution p(Z^d|X) that captures linguistic content.
    Section 4.1; the entire approach assumes semantic tokens are an adequate linguistic representation.
  • standard math Standard VAE and normalizing flow derivations hold (reparameterization, invertibility, change of variables).
    Sections 3.4 and Appendix A; these are foundational statistical techniques.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Variational Framework for Improving Naturalness in Generative Spoken Language Models." pith.science (2026). https://pith.science/paper/7CWSHXCZ

@misc{pith2026250614767,
  author       = {Pith},
  title        = {Pith review of: A Variational Framework for Improving Naturalness in Generative Spoken Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CWSHXCZ}},
  note         = {Machine review of arXiv:2506.14767}
}
read the original abstract

The success of large language models in text processing has inspired their adaptation to speech modeling. However, since speech is continuous and complex, it is often discretized for autoregressive modeling. Speech tokens derived from self-supervised models (known as semantic tokens) typically focus on the linguistic aspects of speech but neglect prosodic information. As a result, models trained on these tokens can generate speech with reduced naturalness. Existing approaches try to fix this by adding pitch features to the semantic tokens. However, pitch alone cannot fully represent the range of paralinguistic attributes, and selecting the right features requires careful hand-engineering. To overcome this, we propose an end-to-end variational approach that automatically learns to encode these continuous speech attributes to enhance the semantic tokens. Our approach eliminates the need for manual extraction and selection of paralinguistic features. Moreover, it produces preferred speech continuations according to human raters. Code, samples and models are available at https://github.com/b04901014/vae-gslm.

Figures

Figures reproduced from arXiv: 2506.14767 by the authors.

Figure 1
Figure 1. Overview of our proposed approach. Our method integrates the token-based speech language model (outlined in Section 2, represented by the lower shaded region) with a variational autoencoder (VAE with autoregressive prior, shown in the upper shaded region). This setup allows the model to learn variational features Z c that complement the pre-extracted semantic speech tokens Z d . In our proposed joint setup, the vari… view at source ↗
Figure 2
Figure 2. (a) Residual block architecture or the encoder ϕ. (b) Model architecture for the time-wise normalization flow introduced in Section 3.4. A.2. Equation 4 Proof. Since Orec is straightforward to derive from Equation 1 (decompose Z into Z c and Z d ), here we show how L c kl and L d kl are derived from the DKL(qϕ(Z | X)||pψ(Z)) in Equation 1. With qϕ(Z | X) = qϕ(Z c | X)p(Z d | X) and pψ(zt | Z1:t−1) = pψ(z d t | Z1:t−… view at source ↗
Figure 3
Figure 3. A screenshot of the Meaningfulness (M-MOS) assessment task, as the crowd-sourced rater sees it [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A screenshot of the Naturalness (N-MOS) assessment task, as the crowd-sourced rater sees it [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

55 extracted references · 24 canonical work pages

  1. [1]

    E., Ostadabbas, S., and Dutoit, T

    Adigwe, A., Tits, N., Haddad, K. E., Ostadabbas, S., and Dutoit, T. The emotional voices database: Towards controlling the emotion dimension in voice generation systems, 2018. URL https://arxiv.org/abs/1806.09514

    arXiv 2018

  2. [2]

    Audiolm: A language modeling approach to audio generation

    Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N. Audiolm: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31: 0 2523--2533, 2023. doi:10.1109/TASLP.2023.3288409

    arXiv 2023

  3. [3]

    R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S

    Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S. Generating sentences from a continuous space. In Riezler, S. and Goldberg, Y. (eds.), Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning , pp.\ 10--21, Berlin, Germany, August 2016. Association for Computational Linguistics. doi:10.18653/v1/K1...

    doi:10.18653/v1/k16-1002 2016

  4. [4]

    A vector quantized approach for text to speech synthesis on real-world spontaneous speech

    Chen, L.-W., Watanabe, S., and Rudnicky, A. A vector quantized approach for text to speech synthesis on real-world spontaneous speech. Proceedings of the AAAI Conference on Artificial Intelligence, 37 0 (11): 0 12644--12652, Jun. 2023. doi:10.1609/aaai.v37i11.26488. URL https://ojs.aaai.org/index.php/AAAI/article/view/26488

    work page doi:10.1609/aaai.v37i11.26488 2023

  5. [5]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing

    Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16 0 (6): 0 1505--1518, 2022. doi:10...

    arXiv 2022

  6. [6]

    VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers, 2024

    Chen, S., Liu, S., Zhou, L., Liu, Y., Tan, X., Li, J., Zhao, S., Qian, Y., and Wei, F. VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers, 2024. URL https://arxiv.org/abs/2406.05370

    arXiv 2024

  7. [7]

    Neural codec language models are zero-shot text to speech synthesizers

    Chen, S., Wang, C., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L., Zhao, S., and Wei, F. Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing, 33: 0 705--718, 2025. doi:10.1109/TASLPRO.2025.3530270

    arXiv 2025

  8. [8]

    High fidelity neural audio compression

    D \'e fossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=ivCd8z8zR2. Featured Certification, Reproducibility Certification

    work page 2023

Show all 55 references
  1. [9]

    Density estimation using real NVP

    Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using real NVP . In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HkpbnH9lx

    2017

  2. [10]

    LauraGPT : Listen, attend, understand, and regenerate audio with GPT , 2024

    Du, Z., Wang, J., Chen, Q., Chu, Y., Gao, Z., Li, Z., Hu, K., Zhou, X., Xu, J., Ma, Z., Wang, W., Zheng, S., Zhou, C., Yan, Z., and Zhang, S. LauraGPT : Listen, attend, understand, and regenerate audio with GPT , 2024. URL https://arxiv.org/abs/2310.04673

    2024 arXiv

  3. [11]

    Moshi: a speech-text foundation model for real-time dialogue, 2024

    Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., and Zeghidour, N. Moshi: a speech-text foundation model for real-time dialogue, 2024. URL https://arxiv.org/abs/2410.00037

    2024 arXiv

  4. [12]

    R., Schuller, B

    Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andr \'e , E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. The G eneva minimalistic acoustic parameter set ( GeMAPS ) for voice research and affective computing. IEEE transactions on affe...

    2015

  5. [13]

    Cyclical annealing schedule: A simple approach to mitigating KL vanishing

    Fu, H., Li, C., Liu, X., Gao, J., Celikyilmaz, A., and Carin, L. Cyclical annealing schedule: A simple approach to mitigating KL vanishing. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association ...

    2019 doi

  6. [14]

    A., Gat, I., Conneau, A., Kreuk, F., Copet, J., Defossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y

    Hassid, M., Remez, T., Nguyen, T. A., Gat, I., Conneau, A., Kreuk, F., Copet, J., Defossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y. Textually pretrained speech language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.),...

    2023

  7. [15]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. Gaussian error linear units ( GELU s). In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Bk0MRI5lg

    2017

  8. [16]

    beta- VAE : Learning basic visual concepts with a constrained variational framework

    Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta- VAE : Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017. URL https://openreview.n...

    2017

  9. [17]

    Denoising diffusion probabilistic models

    Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 6840--6851. Curran Associates, Inc., 2020. URL https://pro...

    2020

  10. [18]

    H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A

    Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. HuBERT : Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29: 0 3451--3460, 2021. doi:10...

    2021

  11. [19]

    Libri-light: A benchmark for ASR with limited or no supervision

    Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazar \'e , P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-light: A benchmark for ASR with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech...

    2020

  12. [20]

    A., Riviere, M., Mohamed, A., Dupoux, E., and Hsu, W.-N

    Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T. A., Riviere, M., Mohamed, A., Dupoux, E., and Hsu, W.-N. Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computationa...

    2022 doi

  13. [21]

    Glow-tts: A generative flow for text-to-speech via monotonic alignment search

    Kim, J., Kim, S., Kong, J., and Yoon, S. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 8067--8077. C...

    2020

  14. [22]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech

    Kim, J., Kong, J., and Son, J. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Re...

    2021

  15. [23]

    W., Salamon, J., Li, P., and Bello, J

    Kim, J. W., Salamon, J., Li, P., and Bello, J. P. Crepe: A convolutional representation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 161--165, 2018. doi:10.1109/ICASSP.2018.8461329

    2018

  16. [24]

    Kingma, D. P. and Welling, M. An introduction to variational autoencoders. Foundations and Trends® in Machine Learning, 12 0 (4): 0 307–392, 2019. ISSN 1935-8245. doi:10.1561/2200000056. URL http://dx.doi.org/10.1561/2200000056

    2019 doi

  17. [25]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

    Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 17022--170...

    2020

  18. [26]

    On generative spoken language modeling from raw audio

    Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., and Dupoux, E. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9: 0 1336--1354, 2...

    2021 doi

  19. [27]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  20. [28]

    Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks

    Maiti, S., Peng, Y., Choi, S., Jung, J.-W., Chang, X., and Watanabe, S. Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal ...

    2024

  21. [29]

    A., de Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., Baevski, A., Dunbar, E., and Dupoux, E

    Nguyen, T. A., de Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., Baevski, A., Dunbar, E., and Dupoux, E. The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling, 2020. URL https://arxiv.org/abs/2011.11588

    2021 arXiv

  22. [30]

    GPT -4 technical report, 2024

    OpenAI. GPT -4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  23. [31]

    Librispeech: An ASR corpus based on public domain audio books

    Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 5206--5210, 2015. doi:10.1109/ICASSP.2015.7178964

    2015

  24. [32]

    FiLM : Visual reasoning with a general conditioning layer

    Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A. FiLM : Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, 32 0 (1), Apr. 2018. doi:10.1609/aaai.v32i1.11671. URL https://ojs.aaai.org/index.php/A...

    2018 doi

  25. [33]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Press, O., Smith, N., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=R8sQPpGCv0

    2022

  26. [34]

    W., Xu, T., Brockman, G., Mcleavey, C., and Sutskever, I

    Radford, A., Kim, J. W., Xu, T., Brockman, G., Mcleavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conferen...

    2023

  27. [35]

    and Torre, R

    Reyes-González, H. and Torre, R. Testing the boundaries: Normalizing flows for higher dimensional data sets. Journal of Physics: Conference Series, 2438 0 (1): 0 012155, 02 2023. Copyright - Published under licence by IOP Publishing Ltd. This work is published under http://cre...

    2023

  28. [36]

    and Mohamed, S

    Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp.\ 1530--1538, Lille, France, 07--09 Jul 20...

    2015

  29. [37]

    U-Net : Convolutional networks for biomedical image segmentation

    Ronneberger, O., Fischer, P., and Brox, T. U-Net : Convolutional networks for biomedical image segmentation. In Navab, N., Hornegger, J., Wells, W. M., and Frangi, A. F. (eds.), Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015, pp.\ 234--241, Cham, 201...

    2015

  30. [38]

    K., Asawaroengchai, C., Nguyen, D

    Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry, F., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., Muckenhirn, H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T., Schalkwyk, J., Sharifi, M., Ramanovich, M. T., Tagliasacchi...

    2023 arXiv

  31. [39]

    The interspeech 2009 emotion challenge

    Schuller, B., Steidl, S., and Batliner, A. The interspeech 2009 emotion challenge. In Interspeech 2009, pp.\ 312--315, 2009. doi:10.21437/Interspeech.2009-103

    2009 doi

  32. [40]

    The interspeech 2013 computational paralinguistics challenge: social signals, conflict, emotion, autism

    Schuller, B., Steidl, S., Batliner, A., Vinciarelli, A., Scherer, K., Ringeval, F., Chetouani, M., Weninger, F., Eyben, F., Marchi, E., Mortillaro, M., Salamin, H., Polychroniou, A., Valente, F., and Kim, S. The interspeech 2013 computational paralinguistics challenge: social ...

    2013 doi

  33. [41]

    Denoising diffusion implicit models

    Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=St1giarCHLP

    2021

  34. [42]

    J., Cao, Y., Zen, H., Rosenberg, A., Ramabhadran, B., and Wu, Y

    Sun, G., Zhang, Y., Weiss, R. J., Cao, Y., Zen, H., Rosenberg, A., Ramabhadran, B., and Wu, Y. Generating diverse and natural text-to-speech samples using a quantized fine-grained vae and autoregressive prosody prior. In ICASSP 2020 - 2020 IEEE International Conference on Acou...

    2020

  35. [43]

    Instance normalization: The missing ingredient for fast stylization, 2017

    Ulyanov, D., Vedaldi, A., and Lempitsky, V. Instance normalization: The missing ingredient for fast stylization, 2017. URL https://arxiv.org/abs/1607.08022

    2017 arXiv

  36. [44]

    and Kautz, J

    Vahdat, A. and Kautz, J. Nvae: A deep hierarchical variational autoencoder. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 19667--19679. Curran Associates, Inc., 2020

    2020

  37. [45]

    Learning de-identified representations of prosody from raw audio

    Weston, J., Lenain, R., Meepegama, U., and Fristed, E. Learning de-identified representations of prosody from raw audio. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Resea...

    2021

  38. [46]

    On layer normalization in the transformer architecture

    Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 o...

    2020

  39. [47]

    CSTR VCTK Corpus : English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92), 2019

    Yamagishi, J., Veaux, C., and MacDonald, K. CSTR VCTK Corpus : English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92), 2019. URL https://doi.org/10.7488/ds/2645

    2019 doi

  40. [48]

    Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Guo, H., Chang, X., Shi, J., Zhao, S., Bian, J., Zhao, Z., Wu, X., and Meng, H. M. U ni A udio: Towards universal audio generation with large language models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N.,...

    2024

  41. [49]

    Soundstream: An end-to-end neural audio codec

    Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 495--507, 2022. doi:10.1109/TASLP.2021.3129994

    2022

  42. [50]

    and Sennrich, R

    Zhang, B. and Sennrich, R. Root mean square layer normalization. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  43. [51]

    SpeechTokenizer : Unified speech tokenizer for speech language models

    Zhang, X., Zhang, D., Li, S., Zhou, Y., and Qiu, X. SpeechTokenizer : Unified speech tokenizer for speech language models. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AF9Q8Vip84

    2024

  44. [52]

    R., Kadav, A., and Graf, H

    Zhu, Y., Min, M. R., Kadav, A., and Graf, H. P. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  45. [53]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  46. [54]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  47. [55]

    The evaluation metrics are detailed in Section ssec:eval-all

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.