Pith. sign in

REVIEW 3 major objections 5 minor 128 references

Autoregressive speech generation stays stable at 50 seconds when an 8-Hz, 768-dim spherical token space is shaped around a low-dimensional core and decoded by multi-path, time-scheduled residual guidance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 08:29 UTC pith:2G45ZBSA

load-bearing objection A credible tokenizer-side contribution, but the load-bearing generator claim (multi-path routing) lacks a standard-ELD control and no artifacts are released. the 3 major comments →

arxiv 2607.29363 v1 pith:2G45ZBSA submitted 2026-07-31 eess.AS cs.AIcs.LGcs.SD

Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens

classification eess.AS cs.AIcs.LGcs.SD
keywords autoregressive speech generationcontinuous tokensflow matchingtokenizer designlatent geometryclassifier-free guidancelong-form stabilityspeech codec
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Autoregressive speech generation has to choose between long token sequences (expensive to generate, prone to drift) and compressed representations (which lose acoustic detail). This paper claims the trade-off can be broken by co-designing the tokenizer and the generator: an 8-Hz, 768-dimensional continuous spherical token whose space is organized around a lower-dimensional core manifold and given a prefix-heavy energy hierarchy, plus a flow-matching generator that routes local-continuity, self-consistency, and text-alignment information through separate residual guidance paths. The token space shaping preserves reconstruction quality while making single tokens easier to predict, and the multi-path guidance with a time schedule prevents the acoustic drift that normally accumulates in long generation. On a standard bilingual TTS benchmark the system reaches 0.95% word error rate for Chinese and 1.87% for English without external ASR/SSL models, pretrained text language models, or post-training stages, and it holds speaker similarity roughly steady across 50-second utterances. Why care: if this is right, high-bandwidth continuous tokens are viable AR targets, and the key to stability is representation geometry plus structured guidance, not external semantic supervision.

Core claim

The paper's central claim is that a low-frame-rate, high-dimensional continuous token can be a stable autoregressive target if the representation geometry and the generation framework are designed jointly. Locodec produces 8-Hz, 768-dimensional spherical tokens centered on a lower-dimensional interpolatable core manifold, and uses postfix dimension dropout so that availability bias becomes a coordinate-wise energy hierarchy, making the most reconstruction-critical dimensions the most identifiable under noise. MP-ELD predicts these tokens with a flow-matching decoder that separates local-continuity, self-consistency, and alignment-consistency conditioning into orthogonalized paths and combine

What carries the argument

The central object is the 8-Hz, 768-dimensional spherical continuous token. Three mechanisms carry the argument: (1) the spherical-cap area scale K_area(N,θ)=1/µ_N(θ)≈cosθ√(2πN)(sinθ)^(-(N-1)), an identity showing that dense interpolatability in the full high-dimensional sphere is exponentially impossible, which motivates constraining the token space around a lower-dimensional core; (2) postfix dimension dropout (PDD), which turns an availability bias into an energy hierarchy and thereby improves per-token identifiability under noise; and (3) multi-path residual classifier-free guidance, in which local-continuity, self-consistency, and alignment-consistency condition vectors are Gram-Schmidt

Load-bearing premise

The design assumes the main cause of long-horizon drift is that partially overlapping acoustic cues get amplified inconsistently when different guidance paths are combined; if the drift actually comes from exposure bias, high target-space entropy, or collapsed representations, the multi-path machinery is fixing a secondary cause.

What would settle it

Train the same Locodec tokens and the same training budget with a single-path ELD generator using standard full-condition CFG with a null path; if that model maintains last-segment speaker similarity above roughly 0.65 at 50 seconds while matching the reported WER, the pathway-conflict hypothesis is falsified and the multi-path orthogonalization is not the cause of stability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • High-bandwidth continuous tokens are a practical AR target: the 8-Hz, 768-dim design reaches 0.95% WER on Chinese and 1.87% on English TTS evaluation without external ASR/SSL models, pretrained text LMs, or post-training.
  • Long-horizon stability becomes a guidance-design problem: with a delayed self-consistency schedule, last-segment speaker similarity stays near 0.67 at 50 seconds, while constant guidance collapses to about 0.40, so short-utterance evaluations miss the drift.
  • Token-space shaping changes learnability: adding the low-dimensional core and PDD lowers the generator's training loss and improves rollout stability; a moderate core dimension (32) gives the best WER/stability balance.
  • Residual CFG gives interpretable controls: alignment guidance mainly moves WER, self-consistency guidance mainly moves speaker similarity, and the bridge-time schedule trades the two off.
  • Low frame rate itself contributes a semantic inductive bias: each 8-Hz token spans about 125 ms, close to phoneme timing, helping competitive WER without semantic supervision.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same shaped low-rate continuous-token recipe could transfer to other sequence modalities (image patches, video, long-context audio), since the reconstruction-capacity versus AR-stability tension is general.
  • Beyond the paper: the remaining speaker-similarity gap suggests a concrete next test — add a light semantic–acoustic factorization inside the continuous token space, still without SSL/ASR supervision, which could recover fine-grained similarity while keeping WER and stability gains.
  • Beyond the paper: if the pathway-conflict hypothesis is right, strong constant classifier-free guidance in autoregressive flow models is itself a principal cause of long-form drift, which would redirect exposure-bias fixes toward scheduled residual guidance rather than larger contexts or extra semantic tokens.
  • Beyond the paper: a dynamic CFG schedule that varies with AR step index, in addition to bridge time, should extend stability beyond 50 seconds; the paper leaves this untested.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper asks whether a low-frame-rate (8 Hz), high-dimensional (768-D) continuous spherical token can be a stable autoregressive speech-generation target. The authors propose Locodec, a locally encoded codec with rotation-based spherical corruption, postfix dimension dropout (PDD), and a learnable low-dimensional core manifold aligned via orthogonal projection/lifting and a bidirectional commitment loss. They propose MP-ELD, an encoder–LM–decoder flow-matching generator with three deliberately separated conditioning paths (local continuity, self-consistency, alignment), Gram–Schmidt orthogonalized residual conditioning, and residual classifier-free guidance with a bridge-time-dependent self-consistency schedule. Experiments cover eight tokenizer configurations and controlled MP-ELD training; results include reconstruction metrics, training-loss predictability curves, CFG WER/SIM sweeps, long-form segment-SIM curves, and Seed-TTS-eval comparisons. The headline results are competitive WER (0.95% ZH, 1.87% EN for the best configuration) without external SSL/ASR models or pretrained text LMs, and improved long-horizon stability with scheduled residual CFG.

Significance. The paper's analytical pieces are sound and largely self-contained: the spherical-cap covering argument (Eqs. 1–6), the chi-squared concentration motivating spherical latents, the under-stepping algebra (Eqs. 28–29), and the high-dimensional near-equivalence of SFM and VP paths are all correct. The experimental spine is also a strength: eight tokenizer configurations are compared with a fixed generator architecture and budget, all CFG settings are reported rather than cherry-picked, and the benchmark numbers use the public Seed-TTS-eval test set. If the central hypothesis is correct, the work demonstrates that geometric shaping of a continuous token space plus guided multi-path routing can substitute for external semantic supervision in AR TTS, which is an important and falsifiable claim. The main weakness is that the mechanism claimed to deliver the stability gain—multi-path information routing—is never isolated from the guidance schedule or model-capacity increase.

major comments (3)
  1. [§4.2; Eq. (63); Tables 4–5; Fig. 6] The central claim that multi-path routing is what mitigates error accumulation is not isolated experimentally. All reported MP-ELD results use the full three-path model with CFG extrapolation; the manuscript never compares against a standard ELD (one token encoder, one LM, one conditioning vector) under the same tokenizer, training budget, and residual-CFG schedule. Fig. 6 shows that the no-guidance full-condition setting (λmax_sc=λac=1) has WER>10% on the internal selection set, while every successful setting uses λac>1 or λmax_sc>1. Since the proposed residual CFG could in principle be applied to any two-condition ELD (e.g., full vs. local-only), the observed gains may come from the CFG schedule or from the extra parameter count (three encoders plus two LMs), not from pathway separation. A parameter-matched single-path baseline with the same residual schedule is required; without it, t
  2. [§3.3; Fig. 3; Table 3; Table 4] The paper claims that the PDD availability bias is converted through training into an energy bias, and that this energy hierarchy is what improves identifiability and predictability. The evidence shows a correlational pattern (energy profile and prefix-K reconstruction improve with K), but no ablation isolates the energy-bias mechanism from the more mundane effect of PDD as input dropout/regularization. For example, a uniform-random dimension-dropout variant (same expected retention but no prefix ordering) or a PDD variant trained without rotation corruption would test whether the ordered availability and the fixed spherical energy budget are necessary for the reported gains. This matters because PDD is one of the two main tokenizer contributions, and Table 4 attributes large long-form improvements to it (e.g., 32/× vs 32/✓ under CFG-S: fifth-segment SIM 0.419 vs 0.578).
  3. [§5.1; §5.3; Fig. 8; Table 4] The long-form stability claim rests entirely on an internal Chinese test set whose size, speaker composition, and recording conditions are not reported, and no public long-form benchmark is used. Since the headline 'long-horizon stability' is central and the Seed-TTS-eval utterances are only ~5 s, the paper should report the number of long-form utterances, the number of speakers, and ideally release the test set or an evaluation protocol. Fig. 8 shows 95% confidence intervals but no sample size; Table 4 reports long-form metrics without any uncertainty estimates.
minor comments (5)
  1. [§3.5] Typographical spacing issue: 'inthepresentsettings' appears as 'inthepresentsettings' in the text; please fix word spacing throughout.
  2. [Table 4] Several entries lack separators, e.g., '2.050.630', '9.630.748', '0.950.687'. These are hard to parse and should be formatted with commas or spaces.
  3. [Fig. 6] The caption says 'Darker points indicate configurations closer to the preferred low-WER, high-SIM region,' but the figure has no colorbar or explicit gray-scale legend. Please add one.
  4. [§4.2; Eq. (64)] When γ=0, s(τ)=τ^0 is undefined at τ=0 in the usual convention. Since the paper states s(0)=0, this corner case should be clarified (e.g., define s(τ)=1 for τ>0 and s(0)=0, or use a different schedule).
  5. [Fig. 5 caption] The notation 'd/PDD' is ambiguous: the caption says 'The notation d/PDD denotes the core dimension and whether PDD is used.' Please define the slash convention (e.g., 32/× vs 32/✓) explicitly, and use the same convention in Table 4.

Circularity Check

0 steps flagged

No significant circularity: the representation-shaping and stability claims are supported by external benchmarks and internal evaluations, and the paper's self-citations are methodological and non-load-bearing.

full rationale

I walked the claimed derivation chain. The analytical results (spherical-cap covering, Gaussian concentration, SFM/VP-path equivalence, MSE-optimal velocity magnitude, direction-only supervision) are derived from standard mathematics and do not assume the paper's conclusions. The tokenizer's energy hierarchy is induced by PDD and then verified empirically through coordinate-energy profiles and reconstruction from restricted representations, rather than being asserted by construction. Predictability is measured by MP-ELD training loss and by generation WER/SIM on the public Seed-TTS-eval benchmark, which was not used for fitting; CFG hyperparameters were tuned on internal medium/long-form sets and then evaluated on the public set, so the headline numbers are not fitted to their own test data. The few self-citations (e.g., [46, 70, 71, 72, 88]) are background or methodology references (long-sequence models, context-codec concept, STFT/GAN loss recipe) and do not carry the central argument. The paper explicitly labels the pathway-conflict mechanism as a hypothesis in Sec 4.2, and the absence of a standard-ELD ablation is a missing-control concern, not a circularity. No equation reduces to its own input by construction, and no predicted quantity is a renamed fitted parameter.

Axiom & Free-Parameter Ledger

6 free parameters · 7 axioms · 0 invented entities

No new physical entities (forces, particles, dimensions) are postulated. Identifiability, information routing, and the core manifold are properties or constructs of the proposed models, not freestanding entities. The central claims rest mainly on hand-set tokenizer hyperparameters (p=0.5, Beta(1,2), loss weights) and validation-selected design choices (d_core=32, CFG schedule), plus four domain assumptions that are plausible but not fully derived: the manifold hypothesis, the reconstruction-stable-basin capacity argument, the CFG-pathway-conflict cause of drift, and the availability-to-energy training dynamic.

free parameters (6)
  • PDD dropout keep-probability p = 0.5
    Sec 3.3: probability of keeping all 768 dimensions; hand-set; drives the prefix-to-postfix availability ordering that produces the energy hierarchy.
  • Spherical rotation corruption prior = Beta(1,2) scaled to [0, pi/2]
    Sec 3.2: angle theta=(pi/2)s with s~Beta(1,2); hand-chosen to bias toward small perturbations while allowing up to 90 degrees.
  • Tokenizer loss weights lambda_low, lambda_commit, lambda_lmag = 0.2 / 0.1 / 0.2
    Sec 4.1 defaults; hand-set; balance the low-dimensional path, commitment loss, and log-magnitude loss.
  • Core manifold dimension d_core = 32 (selected from 768, 256, 64, 32, 16)
    Sec 5.3: chosen by final MP-ELD training loss and generation checks; no bottleneck (768) or very small core (16) is worse.
  • CFG hyperparameters (lambda_sc_max, lambda_ac, gamma) = (2,2,1) long-form config; (2.5,2.5,1) best on CFG-selection set
    Sec 5.3: grid-searched on the internal medium-length set (40 configs); headline Table 5 numbers depend on the tuned configuration.
  • Inference velocity magnitude pi*R/2 and sphere radius R = R never reported
    Sec 3.5, Eq 34: the fixed magnitude replaces velocity-norm prediction; its value depends on the token sphere radius R, which the paper never specifies.
axioms (7)
  • standard math Spherical cap covering asymptotics (Eqs 1-4): mu_N(theta) ~ (sin theta)^(N-1) / (cos theta sqrt(2 pi N))
    Laplace end-point approximation for the cap measure on S^(N-1); used to argue dense interpolatability of the 768-dim token sphere is impractical.
  • standard math High-dimensional concentration: chi-squared_N lies on a thin shell; uniform random vectors on S^(D-1) are nearly orthogonal with inner-product std O(D^-1/2)
    Sec 3.1 and 3.5: motivates spherical tokens and the SFM approximately equal to VP equivalence; accurate at D=768.
  • domain assumption Manifold hypothesis: real-world signals lie near a lower-dimensional manifold, and a well-shaped high-dim token space inherits this structure
    Sec 2.4: motivates the low-dim core manifold; no proof that Locodec's token space satisfies it, only indirect empirical support.
  • domain assumption Reconstruction-stable basins: each token has a decodable cap of radius theta, and K_area(d,rho) bounds the number of distinguishable signal states M_data(epsilon)
    Sec 2.5: links spherical covering numbers to representation capacity; the step K_area << M_data implies over-clustering is qualitative, not shown for Locodec's actual tokens.
  • ad hoc to paper CFG-pathway-conflict hypothesis: AR drift is dominated by inconsistent amplification of overlapping acoustic cues across guidance paths
    Sec 4.2: the paper's own hypothesis; MP-ELD's multi-path design is justified by it, and the evidence is correlational grid-search behavior on an internal dataset.
  • ad hoc to paper Availability-bias to energy-bias training dynamic: higher dropout-retention probability for prefix dimensions forces higher learned energy there under a fixed spherical energy budget and rotation noise
    Sec 3.3: argued from training dynamics and verified in Fig 3, but the alternative (PDD as an extra bottleneck discarding detail) is not excluded; MCD degrades in Table 2.
  • domain assumption Evaluation assumption: WER/SIM/seg-SIM with the DiTAR-configured models capture the claimed qualities, and the internal ZH long-form set is representative
    Sec 5.1: metrics follow DiTAR; long-form conclusions rest on one internal ZH recording set with no public long-form comparator.

pith-pipeline@v1.3.0-daily-deepseek · 42867 in / 22473 out tokens · 233776 ms · 2026-08-03T08:29:43.892478+00:00 · methodology

0 comments
read the original abstract

Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation. Representations with higher frame rates or greater capacity can preserve more signal detail, but they also make streaming generation more vulnerable to distribution drift and AR error accumulation. Conversely, shorter and more compressed representations simplify AR modeling, but their limited bandwidth may discard important components and constrain the upper bound of reconstruction fidelity and generation quality. We ask whether a low-frame-rate, high-dimensional, high-bandwidth continuous representation can be co-designed with a streaming generation framework to support robust high-fidelity reconstruction, strong single-token predictability, and superior long-horizon stability. We decompose this goal into two coupled problems: what geometric and statistical properties a high-dimensional representation space should have, and how an AR continuous-token generator should be structured to resist error accumulation. Accordingly, we propose Locodec, a locally encoded codec that shapes its representation space to improve the interpolatability of a lower-dimensional core manifold and the identifiability of the native high-dimensional coordinates, thereby improving the predictability of high-dimensional high-bandwidth tokens. We also propose MP-ELD, a single-token AR flow-matching framework that uses multi-path information routing and residual classifier-free guidance to mitigate error accumulation. Experiments with 8-Hz, 768-dimensional tokens show that our design preserves reconstruction quality, improves single-token predictability, achieves competitive WER, and maintains stable long-form synthesis, without using external SSL/ASR models, pretrained text language models, or post-training stages.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

128 extracted references · 57 linked inside Pith

  1. [1]

    Stochastic interpolants: A unifying framework for flows and diffusions.Journal of Machine Learning Research, 26(209):1–80, 2025

    Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions.Journal of Machine Learning Research, 26(209):1–80, 2025

  2. [2]

    Building normalizing flows with stochastic interpolants.arXiv preprint arXiv:2209.15571, 2022

    Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants.arXiv preprint arXiv:2209.15571, 2022

  3. [3]

    The curse (s) of dimensionality.Nat Methods, 15(6):399–400, 2018

    Naomi Altman and Martin Krzywinski. The curse (s) of dimensionality.Nat Methods, 15(6):399–400, 2018

  4. [4]

    Seed-TTS: A family of high-quality versatile speech generation models.arXiv preprint arXiv:2406.02430, 2024

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-TTS: A family of high-quality versatile speech generation models.arXiv preprint arXiv:2406.02430, 2024

  5. [5]

    Common voice: A massively-multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. In Proceedings of the twelfth language resources and evaluation conference, pages 4218–4222, 2020

  6. [6]

    Latent space oddity: on the curvature of deep generative models

    Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11379, 2017

  7. [7]

    Geometrically enriched latent spaces.arXiv preprint arXiv:2008.00565, 2020

    Georgios Arvanitidis, Søren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces.arXiv preprint arXiv:2008.00565, 2020

  8. [8]

    Adaptive predictive coding of speech signals.Bell System Technical Journal, 49(8):1973–1986, 1970

    Bishnu S Atal and Manfred R Schroeder. Adaptive predictive coding of speech signals.Bell System Technical Journal, 49(8):1973–1986, 1970

  9. [9]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449–12460, 2020

  10. [10]

    Seed-music: A unified framework for high quality and controlled music generation

    Ye Bai, Haonan Chen, Jitong Chen, Zhuo Chen, Yi Deng, Xiaohong Dong, Lamtharn Hantrakul, Weituo Hao, Qingqing Huang, Zhongyi Huang, et al. Seed-music: A unified framework for high quality and controlled music generation. arXiv preprint arXiv:2409.09214, 2024

  11. [11]

    SPIE press, 1999

    Peter GJ Barten.Contrast sensitivity of the human eye and its effects on image quality. SPIE press, 1999

  12. [12]

    Location-relative attention mechanisms for robust long-form speech synthesis

    Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6194–6198. IEEE, 2020

  13. [13]

    The adaptive multirate wideband speech codec (AMR-WB).IEEE transactions on speech and audio processing, 10(8):620–636, 2002

    Bruno Bessette, Redwan Salami, Roch Lefebvre, Milan Jelinek, Jani Rotola-Pukkila, Janne Vainio, Hannu Mikkola, and Kari Jarvinen. The adaptive multirate wideband speech codec (AMR-WB).IEEE transactions on speech and audio processing, 10(8):620–636, 2002

  14. [14]

    Audiolm: a language modeling approach to audio generation

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31:2523–2533, 2023

  15. [15]

    Application of fourier analysis to the visibility of gratings.The Journal of physiology, 197(3):551, 1968

    Fergus W Campbell and John G Robson. Application of fourier analysis to the visibility of gratings.The Journal of physiology, 197(3):551, 1968

  16. [16]

    A discussion of semi-supervised learning and transduction

    Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. A discussion of semi-supervised learning and transduction. In Semi-supervised learning, pages 473–478. MIT Press, 2006

  17. [17]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advancesin Neural Information Processing Systems, 37:24081–24125, 2024

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advancesin Neural Information Processing Systems, 37:24081–24125, 2024

  18. [18]

    Minmo: A multimodal large language model for seamless voice interaction

    Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, et al. Minmo: A multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.06282, 2025

  19. [19]

    Flow matching on general geometries.arXiv preprint arXiv:2302.03660, 2023

    Ricky TQ Chen and Yaron Lipman. Flow matching on general geometries.arXiv preprint arXiv:2302.03660, 2023. 39

  20. [20]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022

  21. [21]

    Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024

  22. [22]

    On the importance of noise scheduling for diffusion models.arXiv preprint arXiv:2301.10972, 2023

    Ting Chen. On the importance of noise scheduling for diffusion models.arXiv preprint arXiv:2301.10972, 2023

  23. [23]

    Wavtts: Towards high-quality zero-shot tts via direct raw waveform modeling.arXiv preprint arXiv:2606.03455, 2026

    Wenxi Chen, Dongya Jia, Yushen Chen, Zhikang Niu, Yuzhe Liang, Xiquan Li, Ruiqi Yan, Ziyang Ma, Guanrou Yang, Sanyuan Chen, et al. Wavtts: Towards high-quality zero-shot tts via direct raw waveform modeling.arXiv preprint arXiv:2606.03455, 2026

  24. [24]

    Sac: Neural speech codec with semantic-acoustic dual-stream quantization

    Wenxi Chen, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, Shunshun Yin, Ming Tao, Xinsheng Wang, et al. Sac: Neural speech codec with semantic-acoustic dual-stream quantization. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3030–3048, 2026

  25. [25]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, JianZhao JianZhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6255–6271, 2025

  26. [26]

    Visqol v3: An open source production ready objective speech and audio metric

    Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O’Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In2020 twelfth international conference on quality of multimedia experience (QoMEX), pages 1–6. IEEE, 2020

  27. [27]

    Self-supervised learning with random- projection quantizer for speech recognition

    Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random- projection quantizer for speech recognition. InInternational Conference on Machine Learning, pages 3915–3924. PMLR, 2022

  28. [28]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

  29. [29]

    Simple and controllable music generation.Advances in neural information processing systems, 36:47704–47720, 2023

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation.Advances in neural information processing systems, 36:47704–47720, 2023

  30. [30]

    Variational autoencoders without the variation.arXiv preprint arXiv:2203.00645, 2022

    Gregory A Daly, Jonathan E Fieldsend, and Gavin Tabor. Variational autoencoders without the variation.arXiv preprint arXiv:2203.00645, 2022

  31. [31]

    Hyperspherical variational auto-encoders

    Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomczak. Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891, 2018

  32. [32]

    Pullback flow matching on data manifolds.arXiv preprint arXiv:2410.04543, 2024

    Friso de Kruiff, Erik Bekkers, Ozan Öktem, Carola-Bibiane Schönlieb, and Willem Diepeveen. Pullback flow matching on data manifolds.arXiv preprint arXiv:2410.04543, 2024

  33. [33]

    High fidelity neural audio compression.arXiv preprint arXiv:2210.13438, 2022

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression.arXiv preprint arXiv:2210.13438, 2022

  34. [34]

    Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

  35. [35]

    Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020

    Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020

  36. [36]

    Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

  37. [37]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024. 40

  38. [38]

    Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

  39. [39]

    E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts

    Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. In2024 IEEE spoken language technology workshop (SLT), pages 682–689. IEEE, 2024

  40. [40]

    Long-form music generation with latent diffusion.arXiv preprint arXiv:2404.10301, 2024

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion.arXiv preprint arXiv:2404.10301, 2024

  41. [41]

    Barewave: Waveform-native flow-matching text-to-speech.arXiv preprint arXiv:2606.09048, 2026

    Wei Fan, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang, and Nenghai Yu. Barewave: Waveform-native flow-matching text-to-speech.arXiv preprint arXiv:2606.09048, 2026

  42. [42]

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.Advances in Neural Information Processing Systems, 38:41819–41886, 2026

    Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.Advances in Neural Information Processing Systems, 38:41819–41886, 2026

  43. [43]

    Blessing of dimensionality: mathematical foundations of the statistical physics of data

    Alexander N Gorban and Ivan Yu Tyukin. Blessing of dimensionality: mathematical foundations of the statistical physics of data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2118):20170237, 2018

  44. [44]

    Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024

    Hao-Han Guo, Yao Hu, Kun Liu, Fei-Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024

  45. [45]

    Didispeech: A large scale mandarin speech corpus

    Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. Didispeech: A large scale mandarin speech corpus. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6968–6972. IEEE, 2021

  46. [46]

    Continuous speech separation using speaker inventory for long recording

    Cong Han, Yi Luo, Chenda Li, Tianyan Zhou, Keisuke Kinoshita, Shinji Watanabe, Marc Delcroix, Hakan Erdogan, John R Hershey, Nima Mesgarani, et al. Continuous speech separation using speaker inventory for long recording. In Interspeech, pages 3036–3040, 2021

  47. [47]

    beta-VAE: Learning basic visual concepts with a constrained variational framework

    Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id= Sy2fzU9gl

  48. [48]

    Simpler diffusion: 1.5 fid on imagenet512 with pixel-space diffusion

    Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans. Simpler diffusion: 1.5 fid on imagenet512 with pixel-space diffusion. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18062–18071, 2025

  49. [49]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrah- man Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021

  50. [50]

    Music transformer.arXiv preprint arXiv:1809.04281, 2018

    Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer.arXiv preprint arXiv:1809.04281, 2018

  51. [51]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

  52. [52]

    Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling

    Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Ruiqi Li, Ziang Zhang, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. In International Conference on Learning Representations, volume 2025, pages 93809–93826, 2025

  53. [53]

    Ditar: Diffusion transformer autoregressive modeling for speech generation.arXiv preprint arXiv:2502.03930, 2025

    Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, et al. Ditar: Diffusion transformer autoregressive modeling for speech generation.arXiv preprint arXiv:2502.03930, 2025

  54. [54]

    Diffrhythm 2: Efficient and high fidelity song generation via block flow matching.arXiv preprint arXiv:2510.22950, 2025

    Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, and Lei Xie. Diffrhythm 2: Efficient and high fidelity song generation via block flow matching.arXiv preprint arXiv:2510.22950, 2025. 41

  55. [55]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100, 2024

  56. [56]

    Hyperspherical latents improve continuous-token autoregressive generation.arXiv preprint arXiv:2509.24335, 2025

    Guolin Ke and Hui Xue. Hyperspherical latents improve continuous-token autoregressive generation.arXiv preprint arXiv:2509.24335, 2025

  57. [57]

    Mel-cepstral distance measure for objective speech quality assessment

    Robert Kubichek. Mel-cepstral distance measure for objective speech quality assessment. InProceedings of IEEE pacific rim conference on communications computers and signal processing, volume 1, pages 125–128. IEEE, 1993

  58. [58]

    High-fidelity audio compression with improved RVQGAN.Advances in Neural Information Processing Systems, 36:27980–27993, 2023

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN.Advances in Neural Information Processing Systems, 36:27980–27993, 2023

  59. [59]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11523–11532, 2022

  60. [60]

    Geometry-aware image flow matching

    Junho Lee, Kwanseok Kim, and Joonseok Lee. Geometry-aware image flow matching. arXiv preprint arXiv:2605.25294, 2026

  61. [61]

    Sequential data generation with groupwise diffusion process.arXiv preprint arXiv:2310.01400, 2023

    Sangyun Lee, Gayoung Lee, Hyunsu Kim, Junho Kim, and Youngjung Uh. Sequential data generation with groupwise diffusion process.arXiv preprint arXiv:2310.01400, 2023

  62. [62]

    REPA-E: Unlocking VAE for end-to-end tuning of latent diffusion transformers

    Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. REPA-E: Unlocking VAE for end-to-end tuning of latent diffusion transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18262–18272, 2025

  63. [63]

    Back to basics: Let denoising generative models denoise

    Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 36115–36125, 2026

  64. [64]

    Autoregressive image generation without vector quantization.Advances in Neural Information Processing Systems, 37:56424–56445, 2024

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization.Advances in Neural Information Processing Systems, 37:56424–56445, 2024

  65. [65]

    Shi Lian, Changtao Li, Bohan Li, Hankun Wang, Da Zheng, Junfeng Tian, Yufeng Ma, Colin Zhang, and Kai Yu. dots. tts technical report.arXiv preprint arXiv:2606.07080, 2026

  66. [66]

    Fish audio s2 technical report.arXiv preprint arXiv:2603.08823, 2026

    Shijia Liao, Yuxuan Wang, Songting Liu, Yifan Cheng, Ruoyi Zhang, Tianyu Li, Shidong Li, Yisheng Zheng, Xingwei Liu, Qingzheng Wang, et al. Fish audio s2 technical report.arXiv preprint arXiv:2603.08823, 2026

  67. [67]

    Semanticodec: An ultra low bitrate semantic audio codec for general sound.IEEE Journal of Selected Topics in Signal Processing, 18 (8):1448–1461, 2024

    Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanticodec: An ultra low bitrate semantic audio codec for general sound.IEEE Journal of Selected Topics in Signal Processing, 18 (8):1448–1461, 2024

  68. [68]

    Autoregressive diffusion transformer for text-to-speech synthesis

    Zhijun Liu, Shuai Wang, Sho Inoue, Qibing Bai, and Haizhou Li. Autoregressive diffusion transformer for text-to-speech synthesis. arXiv preprint arXiv:2406.05551, 2024

  69. [69]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022

  70. [70]

    Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation

    Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46–50. IEEE, 2020

  71. [71]

    Group communication with context codec for lightweight source separation

    Yi Luo, Cong Han, and Nima Mesgarani. Group communication with context codec for lightweight source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1752–1761, 2021

  72. [72]

    Gull: A generative multifunctional audio codec

    Yi Luo, Jianwei Yu, Hangting Chen, Rongzhi Gu, and Chao Weng. Gull: A generative multifunctional audio codec. arXiv preprint arXiv:2404.04947, 2024

  73. [73]

    Speech recognition and multi-speaker diarization of long conversations.arXiv preprint arXiv:2005.08072, 2020

    Huanru Henry Mao, Shuyang Li, Julian McAuley, and Garrison Cottrell. Speech recognition and multi-speaker diarization of long conversations.arXiv preprint arXiv:2005.08072, 2020. 42

  74. [74]

    Least squares generative adversarial networks

    Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. InProceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017

  75. [75]

    SampleRNN: An unconditional end-to-end neural audio generation model.arXiv preprint arXiv:1612.07837, 2016

    Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An unconditional end-to-end neural audio generation model.arXiv preprint arXiv:1612.07837, 2016

  76. [76]

    Autoregressive speech synthesis without vector quantization

    Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. Autoregressive speech synthesis without vector quantization. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1287–1300, 2025

  77. [77]

    Finite scalar quantization: VQ-VAE made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE made simple. InInternational Conference on Learning Representations, volume 2024, pages 51772–51783, 2024

  78. [78]

    Aligning latent geometry for spherical flow matching in image generation.arXiv preprint arXiv:2605.15193, 2026

    Tuna Han Salih Meral, Kaan Oktay, Hidir Yesiltepe, Adil Kaan Akan, and Pinar Yanardag. Aligning latent geometry for spherical flow matching in image generation.arXiv preprint arXiv:2605.15193, 2026

  79. [79]

    Sensitivity to changes in the intensity of white noise and its relation to masking and loudness

    George A Miller. Sensitivity to changes in the intensity of white noise and its relation to masking and loudness. The Journal of the Acoustical Society of America, 19(4):609–619, 1947

  80. [80]

    How should we extract discrete audio tokens from self-supervised models?arXiv preprint arXiv:2406.10735, 2024

    Pooneh Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. How should we extract discrete audio tokens from self-supervised models?arXiv preprint arXiv:2406.10735, 2024

Showing first 80 references.