Pith. sign in

REVIEW 3 major objections 5 minor 128 references

Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Autoregressive speech generation stays stable at 50 seconds when an 8-Hz, 768-dim spherical token space is shaped around a low-dimensional core and decoded by multi-path, time-scheduled residual guidance.

desk verdict A credible tokenizer-side contribution, but the load-bearing generator claim (multi-path routing) lacks a standard-ELD control and no artifacts are released. read the letter →

arxiv 2607.29363 v1 pith:2G45ZBSA submitted 2026-07-31 eess.AS cs.AIcs.LGcs.SD

Yi Luo , Rongzhi Gu , Jixun Yao This is my paper · ORCID
classification eess.AScs.AIcs.LGcs.SD
keywords autoregressivespeechgenerationcontinuoustokensflowmatchingtokenizerdesignlatentgeometryclassifier-freeguidancelong-formstabilitycodec
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autoregressive speech generation has to choose between long token sequences (expensive to generate, prone to drift) and compressed representations (which lose acoustic detail). This paper claims the trade-off can be broken by co-designing the tokenizer and the generator: an 8-Hz, 768-dimensional continuous spherical token whose space is organized around a lower-dimensional core manifold and given a prefix-heavy energy hierarchy, plus a flow-matching generator that routes local-continuity, self-consistency, and text-alignment information through separate residual guidance paths. The token space shaping preserves reconstruction quality while making single tokens easier to predict, and the multi-path guidance with a time schedule prevents the acoustic drift that normally accumulates in long generation. On a standard bilingual TTS benchmark the system reaches 0.95% word error rate for Chinese and 1.87% for English without external ASR/SSL models, pretrained text language models, or post-training stages, and it holds speaker similarity roughly steady across 50-second utterances. Why care: if this is right, high-bandwidth continuous tokens are viable AR targets, and the key to stability is representation geometry plus structured guidance, not external semantic supervision.

What carries the argument

The central object is the 8-Hz, 768-dimensional spherical continuous token. Three mechanisms carry the argument: (1) the spherical-cap area scale K_area(N,θ)=1/µ_N(θ)≈cosθ√(2πN)(sinθ)^(-(N-1)), an identity showing that dense interpolatability in the full high-dimensional sphere is exponentially impossible, which motivates constraining the token space around a lower-dimensional core; (2) postfix dimension dropout (PDD), which turns an availability bias into an energy hierarchy and thereby improves per-token identifiability under noise; and (3) multi-path residual classifier-free guidance, in which local-continuity, self-consistency, and alignment-consistency condition vectors are Gram-Schmidt

What would settle it

Train the same Locodec tokens and the same training budget with a single-path ELD generator using standard full-condition CFG with a null path; if that model maintains last-segment speaker similarity above roughly 0.65 at 50 seconds while matching the reported WER, the pathway-conflict hypothesis is falsified and the multi-path orthogonalization is not the cause of stability.

Watch

Extended reading notes

Core claim

The paper's central claim is that a low-frame-rate, high-dimensional continuous token can be a stable autoregressive target if the representation geometry and the generation framework are designed jointly. Locodec produces 8-Hz, 768-dimensional spherical tokens centered on a lower-dimensional interpolatable core manifold, and uses postfix dimension dropout so that availability bias becomes a coordinate-wise energy hierarchy, making the most reconstruction-critical dimensions the most identifiable under noise. MP-ELD predicts these tokens with a flow-matching decoder that separates local-continuity, self-consistency, and alignment-consistency conditioning into orthogonalized paths and combine

Load-bearing premise

The design assumes the main cause of long-horizon drift is that partially overlapping acoustic cues get amplified inconsistently when different guidance paths are combined; if the drift actually comes from exposure bias, high target-space entropy, or collapsed representations, the multi-path machinery is fixing a secondary cause.

Editorial extensions

If this is right

  • High-bandwidth continuous tokens are a practical AR target: the 8-Hz, 768-dim design reaches 0.95% WER on Chinese and 1.87% on English TTS evaluation without external ASR/SSL models, pretrained text LMs, or post-training.
  • Long-horizon stability becomes a guidance-design problem: with a delayed self-consistency schedule, last-segment speaker similarity stays near 0.67 at 50 seconds, while constant guidance collapses to about 0.40, so short-utterance evaluations miss the drift.
  • Token-space shaping changes learnability: adding the low-dimensional core and PDD lowers the generator's training loss and improves rollout stability; a moderate core dimension (32) gives the best WER/stability balance.
  • Residual CFG gives interpretable controls: alignment guidance mainly moves WER, self-consistency guidance mainly moves speaker similarity, and the bridge-time schedule trades the two off.
  • Low frame rate itself contributes a semantic inductive bias: each 8-Hz token spans about 125 ms, close to phoneme timing, helping competitive WER without semantic supervision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same shaped low-rate continuous-token recipe could transfer to other sequence modalities (image patches, video, long-context audio), since the reconstruction-capacity versus AR-stability tension is general.
  • Beyond the paper: the remaining speaker-similarity gap suggests a concrete next test — add a light semantic–acoustic factorization inside the continuous token space, still without SSL/ASR supervision, which could recover fine-grained similarity while keeping WER and stability gains.
  • Beyond the paper: if the pathway-conflict hypothesis is right, strong constant classifier-free guidance in autoregressive flow models is itself a principal cause of long-form drift, which would redirect exposure-bias fixes toward scheduled residual guidance rather than larger contexts or extra semantic tokens.
  • Beyond the paper: a dynamic CFG schedule that varies with AR step index, in addition to bridge time, should extend stability beyond 50 seconds; the paper leaves this untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper asks whether a low-frame-rate (8 Hz), high-dimensional (768-D) continuous spherical token can be a stable autoregressive speech-generation target. The authors propose Locodec, a locally encoded codec with rotation-based spherical corruption, postfix dimension dropout (PDD), and a learnable low-dimensional core manifold aligned via orthogonal projection/lifting and a bidirectional commitment loss. They propose MP-ELD, an encoder–LM–decoder flow-matching generator with three deliberately separated conditioning paths (local continuity, self-consistency, alignment), Gram–Schmidt orthogonalized residual conditioning, and residual classifier-free guidance with a bridge-time-dependent self-consistency schedule. Experiments cover eight tokenizer configurations and controlled MP-ELD training; results include reconstruction metrics, training-loss predictability curves, CFG WER/SIM sweeps, long-form segment-SIM curves, and Seed-TTS-eval comparisons. The headline results are competitive WER (0.95% ZH, 1.87% EN for the best configuration) without external SSL/ASR models or pretrained text LMs, and improved long-horizon stability with scheduled residual CFG.

Significance. The paper's analytical pieces are sound and largely self-contained: the spherical-cap covering argument (Eqs. 1–6), the chi-squared concentration motivating spherical latents, the under-stepping algebra (Eqs. 28–29), and the high-dimensional near-equivalence of SFM and VP paths are all correct. The experimental spine is also a strength: eight tokenizer configurations are compared with a fixed generator architecture and budget, all CFG settings are reported rather than cherry-picked, and the benchmark numbers use the public Seed-TTS-eval test set. If the central hypothesis is correct, the work demonstrates that geometric shaping of a continuous token space plus guided multi-path routing can substitute for external semantic supervision in AR TTS, which is an important and falsifiable claim. The main weakness is that the mechanism claimed to deliver the stability gain—multi-path information routing—is never isolated from the guidance schedule or model-capacity increase.

major comments (3)
  1. [§4.2; Eq. (63); Tables 4–5; Fig. 6] The central claim that multi-path routing is what mitigates error accumulation is not isolated experimentally. All reported MP-ELD results use the full three-path model with CFG extrapolation; the manuscript never compares against a standard ELD (one token encoder, one LM, one conditioning vector) under the same tokenizer, training budget, and residual-CFG schedule. Fig. 6 shows that the no-guidance full-condition setting (λmax_sc=λac=1) has WER>10% on the internal selection set, while every successful setting uses λac>1 or λmax_sc>1. Since the proposed residual CFG could in principle be applied to any two-condition ELD (e.g., full vs. local-only), the observed gains may come from the CFG schedule or from the extra parameter count (three encoders plus two LMs), not from pathway separation. A parameter-matched single-path baseline with the same residual schedule is required; without it, t
  2. [§3.3; Fig. 3; Table 3; Table 4] The paper claims that the PDD availability bias is converted through training into an energy bias, and that this energy hierarchy is what improves identifiability and predictability. The evidence shows a correlational pattern (energy profile and prefix-K reconstruction improve with K), but no ablation isolates the energy-bias mechanism from the more mundane effect of PDD as input dropout/regularization. For example, a uniform-random dimension-dropout variant (same expected retention but no prefix ordering) or a PDD variant trained without rotation corruption would test whether the ordered availability and the fixed spherical energy budget are necessary for the reported gains. This matters because PDD is one of the two main tokenizer contributions, and Table 4 attributes large long-form improvements to it (e.g., 32/× vs 32/✓ under CFG-S: fifth-segment SIM 0.419 vs 0.578).
  3. [§5.1; §5.3; Fig. 8; Table 4] The long-form stability claim rests entirely on an internal Chinese test set whose size, speaker composition, and recording conditions are not reported, and no public long-form benchmark is used. Since the headline 'long-horizon stability' is central and the Seed-TTS-eval utterances are only ~5 s, the paper should report the number of long-form utterances, the number of speakers, and ideally release the test set or an evaluation protocol. Fig. 8 shows 95% confidence intervals but no sample size; Table 4 reports long-form metrics without any uncertainty estimates.
minor comments (5)
  1. [§3.5] Typographical spacing issue: 'inthepresentsettings' appears as 'inthepresentsettings' in the text; please fix word spacing throughout.
  2. [Table 4] Several entries lack separators, e.g., '2.050.630', '9.630.748', '0.950.687'. These are hard to parse and should be formatted with commas or spaces.
  3. [Fig. 6] The caption says 'Darker points indicate configurations closer to the preferred low-WER, high-SIM region,' but the figure has no colorbar or explicit gray-scale legend. Please add one.
  4. [§4.2; Eq. (64)] When γ=0, s(τ)=τ^0 is undefined at τ=0 in the usual convention. Since the paper states s(0)=0, this corner case should be clarified (e.g., define s(τ)=1 for τ>0 and s(0)=0, or use a different schedule).
  5. [Fig. 5 caption] The notation 'd/PDD' is ambiguous: the caption says 'The notation d/PDD denotes the core dimension and whether PDD is used.' Please define the slash convention (e.g., 32/× vs 32/✓) explicitly, and use the same convention in Table 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the representation-shaping and stability claims are supported by external benchmarks and internal evaluations, and the paper's self-citations are methodological and non-load-bearing.

full rationale

I walked the claimed derivation chain. The analytical results (spherical-cap covering, Gaussian concentration, SFM/VP-path equivalence, MSE-optimal velocity magnitude, direction-only supervision) are derived from standard mathematics and do not assume the paper's conclusions. The tokenizer's energy hierarchy is induced by PDD and then verified empirically through coordinate-energy profiles and reconstruction from restricted representations, rather than being asserted by construction. Predictability is measured by MP-ELD training loss and by generation WER/SIM on the public Seed-TTS-eval benchmark, which was not used for fitting; CFG hyperparameters were tuned on internal medium/long-form sets and then evaluated on the public set, so the headline numbers are not fitted to their own test data. The few self-citations (e.g., [46, 70, 71, 72, 88]) are background or methodology references (long-sequence models, context-codec concept, STFT/GAN loss recipe) and do not carry the central argument. The paper explicitly labels the pathway-conflict mechanism as a hypothesis in Sec 4.2, and the absence of a standard-ELD ablation is a missing-control concern, not a circularity. No equation reduces to its own input by construction, and no predicted quantity is a renamed fitted parameter.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

No new physical entities (forces, particles, dimensions) are postulated. Identifiability, information routing, and the core manifold are properties or constructs of the proposed models, not freestanding entities. The central claims rest mainly on hand-set tokenizer hyperparameters (p=0.5, Beta(1,2), loss weights) and validation-selected design choices (d_core=32, CFG schedule), plus four domain assumptions that are plausible but not fully derived: the manifold hypothesis, the reconstruction-stable-basin capacity argument, the CFG-pathway-conflict cause of drift, and the availability-to-energy training dynamic.

free parameters (6)
  • PDD dropout keep-probability p = 0.5
    Sec 3.3: probability of keeping all 768 dimensions; hand-set; drives the prefix-to-postfix availability ordering that produces the energy hierarchy.
  • Spherical rotation corruption prior = Beta(1,2) scaled to [0, pi/2]
    Sec 3.2: angle theta=(pi/2)s with s~Beta(1,2); hand-chosen to bias toward small perturbations while allowing up to 90 degrees.
  • Tokenizer loss weights lambda_low, lambda_commit, lambda_lmag = 0.2 / 0.1 / 0.2
    Sec 4.1 defaults; hand-set; balance the low-dimensional path, commitment loss, and log-magnitude loss.
  • Core manifold dimension d_core = 32 (selected from 768, 256, 64, 32, 16)
    Sec 5.3: chosen by final MP-ELD training loss and generation checks; no bottleneck (768) or very small core (16) is worse.
  • CFG hyperparameters (lambda_sc_max, lambda_ac, gamma) = (2,2,1) long-form config; (2.5,2.5,1) best on CFG-selection set
    Sec 5.3: grid-searched on the internal medium-length set (40 configs); headline Table 5 numbers depend on the tuned configuration.
  • Inference velocity magnitude pi*R/2 and sphere radius R = R never reported
    Sec 3.5, Eq 34: the fixed magnitude replaces velocity-norm prediction; its value depends on the token sphere radius R, which the paper never specifies.
assumptions (7)
  • standard math Spherical cap covering asymptotics (Eqs 1-4): mu_N(theta) ~ (sin theta)^(N-1) / (cos theta sqrt(2 pi N))
    Laplace end-point approximation for the cap measure on S^(N-1); used to argue dense interpolatability of the 768-dim token sphere is impractical.
  • standard math High-dimensional concentration: chi-squared_N lies on a thin shell; uniform random vectors on S^(D-1) are nearly orthogonal with inner-product std O(D^-1/2)
    Sec 3.1 and 3.5: motivates spherical tokens and the SFM approximately equal to VP equivalence; accurate at D=768.
  • domain assumption Manifold hypothesis: real-world signals lie near a lower-dimensional manifold, and a well-shaped high-dim token space inherits this structure
    Sec 2.4: motivates the low-dim core manifold; no proof that Locodec's token space satisfies it, only indirect empirical support.
  • domain assumption Reconstruction-stable basins: each token has a decodable cap of radius theta, and K_area(d,rho) bounds the number of distinguishable signal states M_data(epsilon)
    Sec 2.5: links spherical covering numbers to representation capacity; the step K_area << M_data implies over-clustering is qualitative, not shown for Locodec's actual tokens.
  • ad hoc to paper CFG-pathway-conflict hypothesis: AR drift is dominated by inconsistent amplification of overlapping acoustic cues across guidance paths
    Sec 4.2: the paper's own hypothesis; MP-ELD's multi-path design is justified by it, and the evidence is correlational grid-search behavior on an internal dataset.
  • ad hoc to paper Availability-bias to energy-bias training dynamic: higher dropout-retention probability for prefix dimensions forces higher learned energy there under a fixed spherical energy budget and rotation noise
    Sec 3.3: argued from training dynamics and verified in Fig 3, but the alternative (PDD as an extra bottleneck discarding detail) is not excluded; MCD degrades in Table 2.
  • domain assumption Evaluation assumption: WER/SIM/seg-SIM with the DiTAR-configured models capture the claimed qualities, and the internal ZH long-form set is representative
    Sec 5.1: metrics follow DiTAR; long-form conclusions rest on one internal ZH recording set with no public long-form comparator.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens." pith.science (2026). https://pith.science/paper/2G45ZBSA

@misc{pith2026260729363,
  author       = {Pith},
  title        = {Pith review of: Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2G45ZBSA}},
  note         = {Machine review of arXiv:2607.29363}
}
read the original abstract

Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation. Representations with higher frame rates or greater capacity can preserve more signal detail, but they also make streaming generation more vulnerable to distribution drift and AR error accumulation. Conversely, shorter and more compressed representations simplify AR modeling, but their limited bandwidth may discard important components and constrain the upper bound of reconstruction fidelity and generation quality. We ask whether a low-frame-rate, high-dimensional, high-bandwidth continuous representation can be co-designed with a streaming generation framework to support robust high-fidelity reconstruction, strong single-token predictability, and superior long-horizon stability. We decompose this goal into two coupled problems: what geometric and statistical properties a high-dimensional representation space should have, and how an AR continuous-token generator should be structured to resist error accumulation. Accordingly, we propose Locodec, a locally encoded codec that shapes its representation space to improve the interpolatability of a lower-dimensional core manifold and the identifiability of the native high-dimensional coordinates, thereby improving the predictability of high-dimensional high-bandwidth tokens. We also propose MP-ELD, a single-token AR flow-matching framework that uses multi-path information routing and residual classifier-free guidance to mitigate error accumulation. Experiments with 8-Hz, 768-dimensional tokens show that our design preserves reconstruction quality, improves single-token predictability, achieves competitive WER, and maintains stable long-form synthesis, without using external SSL/ASR models, pretrained text language models, or post-training stages.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

128 extracted references · 57 linked inside Pith

  1. [1]

    Stochastic interpolants: A unifying framework for flows and diffusions.Journal of Machine Learning Research, 26(209):1–80, 2025

    Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions.Journal of Machine Learning Research, 26(209):1–80, 2025

    2025

  2. [2]

    Building normalizing flows with stochastic interpolants.arXiv preprint arXiv:2209.15571, 2022

    Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants.arXiv preprint arXiv:2209.15571, 2022

    arXiv 2022

  3. [3]

    The curse (s) of dimensionality.Nat Methods, 15(6):399–400, 2018

    Naomi Altman and Martin Krzywinski. The curse (s) of dimensionality.Nat Methods, 15(6):399–400, 2018

    2018

  4. [4]

    Seed-TTS: A family of high-quality versatile speech generation models.arXiv preprint arXiv:2406.02430, 2024

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-TTS: A family of high-quality versatile speech generation models.arXiv preprint arXiv:2406.02430, 2024

    arXiv 2024

  5. [5]

    Common voice: A massively-multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. In Proceedings of the twelfth language resources and evaluation conference, pages 4218–4222, 2020

    2020

  6. [6]

    Latent space oddity: on the curvature of deep generative models

    Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11379, 2017

    arXiv 2017

  7. [7]

    Geometrically enriched latent spaces.arXiv preprint arXiv:2008.00565, 2020

    Georgios Arvanitidis, Søren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces.arXiv preprint arXiv:2008.00565, 2020

    arXiv 2008

  8. [8]

    Adaptive predictive coding of speech signals.Bell System Technical Journal, 49(8):1973–1986, 1970

    Bishnu S Atal and Manfred R Schroeder. Adaptive predictive coding of speech signals.Bell System Technical Journal, 49(8):1973–1986, 1970

    1973

Show all 128 references
  1. [9]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449–12460, 2020

    2020

  2. [10]

    Seed-music: A unified framework for high quality and controlled music generation

    Ye Bai, Haonan Chen, Jitong Chen, Zhuo Chen, Yi Deng, Xiaohong Dong, Lamtharn Hantrakul, Weituo Hao, Qingqing Huang, Zhongyi Huang, et al. Seed-music: A unified framework for high quality and controlled music generation. arXiv preprint arXiv:2409.09214, 2024

    2024 arXiv

  3. [11]

    SPIE press, 1999

    Peter GJ Barten.Contrast sensitivity of the human eye and its effects on image quality. SPIE press, 1999

    1999

  4. [12]

    Location-relative attention mechanisms for robust long-form speech synthesis

    Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing...

    2020

  5. [13]

    The adaptive multirate wideband speech codec (AMR-WB).IEEE transactions on speech and audio processing, 10(8):620–636, 2002

    Bruno Bessette, Redwan Salami, Roch Lefebvre, Milan Jelinek, Jani Rotola-Pukkila, Janne Vainio, Hannu Mikkola, and Kari Jarvinen. The adaptive multirate wideband speech codec (AMR-WB).IEEE transactions on speech and audio processing, 10(8):620–636, 2002

    2002

  6. [14]

    Audiolm: a language modeling approach to audio generation

    Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and ...

    2023

  7. [15]

    Application of fourier analysis to the visibility of gratings.The Journal of physiology, 197(3):551, 1968

    Fergus W Campbell and John G Robson. Application of fourier analysis to the visibility of gratings.The Journal of physiology, 197(3):551, 1968

    1968

  8. [16]

    A discussion of semi-supervised learning and transduction

    Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. A discussion of semi-supervised learning and transduction. In Semi-supervised learning, pages 473–478. MIT Press, 2006

    2006

  9. [17]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advancesin Neural Information Processing Systems, 37:24081–24125, 2024

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advancesin Neural Information Processing Systems, 37:24081–24125, 2024

    2024

  10. [18]

    Minmo: A multimodal large language model for seamless voice interaction

    Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, et al. Minmo: A multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.06282, 2025

    2025 arXiv

  11. [19]

    Flow matching on general geometries.arXiv preprint arXiv:2302.03660, 2023

    Ricky TQ Chen and Yaron Lipman. Flow matching on general geometries.arXiv preprint arXiv:2302.03660, 2023. 39

    2023 arXiv

  12. [20]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):...

    2022

  13. [21]

    Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024

    Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024

    2024 arXiv

  14. [22]

    On the importance of noise scheduling for diffusion models.arXiv preprint arXiv:2301.10972, 2023

    Ting Chen. On the importance of noise scheduling for diffusion models.arXiv preprint arXiv:2301.10972, 2023

    2023 arXiv

  15. [23]

    Wavtts: Towards high-quality zero-shot tts via direct raw waveform modeling.arXiv preprint arXiv:2606.03455, 2026

    Wenxi Chen, Dongya Jia, Yushen Chen, Zhikang Niu, Yuzhe Liang, Xiquan Li, Ruiqi Yan, Ziyang Ma, Guanrou Yang, Sanyuan Chen, et al. Wavtts: Towards high-quality zero-shot tts via direct raw waveform modeling.arXiv preprint arXiv:2606.03455, 2026

    2026 arXiv

  16. [24]

    Sac: Neural speech codec with semantic-acoustic dual-stream quantization

    Wenxi Chen, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, Shunshun Yin, Ming Tao, Xinsheng Wang, et al. Sac: Neural speech codec with semantic-acoustic dual-stream quantization. In Proceedings of the 64th Annual Meeting of the Association for Computat...

    2026

  17. [25]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, JianZhao JianZhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volu...

    2025

  18. [26]

    Visqol v3: An open source production ready objective speech and audio metric

    Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O’Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In2020 twelfth international conference on quality of multimedia experience (QoMEX), pages 1–6. IEEE, 2020

    2020

  19. [27]

    Self-supervised learning with random- projection quantizer for speech recognition

    Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random- projection quantizer for speech recognition. InInternational Conference on Machine Learning, pages 3915–3924. PMLR, 2022

    2022

  20. [28]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  21. [29]

    Simple and controllable music generation.Advances in neural information processing systems, 36:47704–47720, 2023

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation.Advances in neural information processing systems, 36:47704–47720, 2023

    2023

  22. [30]

    Variational autoencoders without the variation.arXiv preprint arXiv:2203.00645, 2022

    Gregory A Daly, Jonathan E Fieldsend, and Gavin Tabor. Variational autoencoders without the variation.arXiv preprint arXiv:2203.00645, 2022

    2022 arXiv

  23. [31]

    Hyperspherical variational auto-encoders

    Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomczak. Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891, 2018

    2018 arXiv

  24. [32]

    Pullback flow matching on data manifolds.arXiv preprint arXiv:2410.04543, 2024

    Friso de Kruiff, Erik Bekkers, Ozan Öktem, Carola-Bibiane Schönlieb, and Willem Diepeveen. Pullback flow matching on data manifolds.arXiv preprint arXiv:2410.04543, 2024

    2024 arXiv

  25. [33]

    High fidelity neural audio compression.arXiv preprint arXiv:2210.13438, 2022

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression.arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  26. [34]

    Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  27. [35]

    Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020

    Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020

    2005 arXiv

  28. [36]

    Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    2025 arXiv

  29. [37]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024. 40

    2024 arXiv

  30. [38]

    Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025

    2025 arXiv

  31. [39]

    E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts

    Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. In2024 IEEE spoken language technology workshop (SLT), pages 682–689. ...

    2024

  32. [40]

    Long-form music generation with latent diffusion.arXiv preprint arXiv:2404.10301, 2024

    Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion.arXiv preprint arXiv:2404.10301, 2024

    2024 arXiv

  33. [41]

    Barewave: Waveform-native flow-matching text-to-speech.arXiv preprint arXiv:2606.09048, 2026

    Wei Fan, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang, and Nenghai Yu. Barewave: Waveform-native flow-matching text-to-speech.arXiv preprint arXiv:2606.09048, 2026

    2026 arXiv

  34. [42]

    Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.Advances in Neural Information Processing Systems, 38:41819–41886, 2026

    Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.Advances in Neural Information Pr...

    2026

  35. [43]

    Blessing of dimensionality: mathematical foundations of the statistical physics of data

    Alexander N Gorban and Ivan Yu Tyukin. Blessing of dimensionality: mathematical foundations of the statistical physics of data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2118):20170237, 2018

    2018

  36. [44]

    Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024

    Hao-Han Guo, Yao Hu, Kun Liu, Fei-Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024

    2024 arXiv

  37. [45]

    Didispeech: A large scale mandarin speech corpus

    Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. Didispeech: A large scale mandarin speech corpus. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),...

    2021

  38. [46]

    Continuous speech separation using speaker inventory for long recording

    Cong Han, Yi Luo, Chenda Li, Tianyan Zhou, Keisuke Kinoshita, Shinji Watanabe, Marc Delcroix, Hakan Erdogan, John R Hershey, Nima Mesgarani, et al. Continuous speech separation using speaker inventory for long recording. In Interspeech, pages 3036–3040, 2021

    2021

  39. [47]

    beta-VAE: Learning basic visual concepts with a constrained variational framework

    Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 201...

    2017

  40. [48]

    Simpler diffusion: 1.5 fid on imagenet512 with pixel-space diffusion

    Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans. Simpler diffusion: 1.5 fid on imagenet512 with pixel-space diffusion. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18062–18071, 2025

    2025

  41. [49]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrah- man Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 2...

    2021

  42. [50]

    Music transformer.arXiv preprint arXiv:1809.04281, 2018

    Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer.arXiv preprint arXiv:1809.04281, 2018

    2018 arXiv

  43. [51]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  44. [52]

    Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling

    Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Ruiqi Li, Ziang Zhang, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. In International Conference on Learning Representations, vo...

    2025

  45. [53]

    Ditar: Diffusion transformer autoregressive modeling for speech generation.arXiv preprint arXiv:2502.03930, 2025

    Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, et al. Ditar: Diffusion transformer autoregressive modeling for speech generation.arXiv preprint arXiv:2502.03930, 2025

    2025

  46. [54]

    Diffrhythm 2: Efficient and high fidelity song generation via block flow matching.arXiv preprint arXiv:2510.22950, 2025

    Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, and Lei Xie. Diffrhythm 2: Efficient and high fidelity song generation via block flow matching.arXiv preprint arXiv:2510.22950, 2025. 41

    2025

  47. [55]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100, 2024

    2024 arXiv

  48. [56]

    Hyperspherical latents improve continuous-token autoregressive generation.arXiv preprint arXiv:2509.24335, 2025

    Guolin Ke and Hui Xue. Hyperspherical latents improve continuous-token autoregressive generation.arXiv preprint arXiv:2509.24335, 2025

    2025

  49. [57]

    Mel-cepstral distance measure for objective speech quality assessment

    Robert Kubichek. Mel-cepstral distance measure for objective speech quality assessment. InProceedings of IEEE pacific rim conference on communications computers and signal processing, volume 1, pages 125–128. IEEE, 1993

    1993

  50. [58]

    High-fidelity audio compression with improved RVQGAN.Advances in Neural Information Processing Systems, 36:27980–27993, 2023

    Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN.Advances in Neural Information Processing Systems, 36:27980–27993, 2023

    2023

  51. [59]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11523–11532, 2022

    2022

  52. [60]

    Geometry-aware image flow matching

    Junho Lee, Kwanseok Kim, and Joonseok Lee. Geometry-aware image flow matching. arXiv preprint arXiv:2605.25294, 2026

    2026 arXiv

  53. [61]

    Sequential data generation with groupwise diffusion process.arXiv preprint arXiv:2310.01400, 2023

    Sangyun Lee, Gayoung Lee, Hyunsu Kim, Junho Kim, and Youngjung Uh. Sequential data generation with groupwise diffusion process.arXiv preprint arXiv:2310.01400, 2023

    2023 arXiv

  54. [62]

    REPA-E: Unlocking VAE for end-to-end tuning of latent diffusion transformers

    Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. REPA-E: Unlocking VAE for end-to-end tuning of latent diffusion transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18262–18272, 2025

    2025

  55. [63]

    Back to basics: Let denoising generative models denoise

    Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 36115–36125, 2026

    2026

  56. [64]

    Autoregressive image generation without vector quantization.Advances in Neural Information Processing Systems, 37:56424–56445, 2024

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization.Advances in Neural Information Processing Systems, 37:56424–56445, 2024

    2024

  57. [65]

    Shi Lian, Changtao Li, Bohan Li, Hankun Wang, Da Zheng, Junfeng Tian, Yufeng Ma, Colin Zhang, and Kai Yu. dots. tts technical report.arXiv preprint arXiv:2606.07080, 2026

    2026 arXiv

  58. [66]

    Fish audio s2 technical report.arXiv preprint arXiv:2603.08823, 2026

    Shijia Liao, Yuxuan Wang, Songting Liu, Yifan Cheng, Ruoyi Zhang, Tianyu Li, Shidong Li, Yisheng Zheng, Xingwei Liu, Qingzheng Wang, et al. Fish audio s2 technical report.arXiv preprint arXiv:2603.08823, 2026

    2026

  59. [67]

    Semanticodec: An ultra low bitrate semantic audio codec for general sound.IEEE Journal of Selected Topics in Signal Processing, 18 (8):1448–1461, 2024

    Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanticodec: An ultra low bitrate semantic audio codec for general sound.IEEE Journal of Selected Topics in Signal Processing, 18 (8):1448–1461, 2024

    2024

  60. [68]

    Autoregressive diffusion transformer for text-to-speech synthesis

    Zhijun Liu, Shuai Wang, Sho Inoue, Qibing Bai, and Haizhou Li. Autoregressive diffusion transformer for text-to-speech synthesis. arXiv preprint arXiv:2406.05551, 2024

    2024 arXiv

  61. [69]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022

    2022

  62. [70]

    Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation

    Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46–50. IEEE, 2020

    2020

  63. [71]

    Group communication with context codec for lightweight source separation

    Yi Luo, Cong Han, and Nima Mesgarani. Group communication with context codec for lightweight source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1752–1761, 2021

    2021

  64. [72]

    Gull: A generative multifunctional audio codec

    Yi Luo, Jianwei Yu, Hangting Chen, Rongzhi Gu, and Chao Weng. Gull: A generative multifunctional audio codec. arXiv preprint arXiv:2404.04947, 2024

    2024 arXiv

  65. [73]

    Speech recognition and multi-speaker diarization of long conversations.arXiv preprint arXiv:2005.08072, 2020

    Huanru Henry Mao, Shuyang Li, Julian McAuley, and Garrison Cottrell. Speech recognition and multi-speaker diarization of long conversations.arXiv preprint arXiv:2005.08072, 2020. 42

    2005 arXiv

  66. [74]

    Least squares generative adversarial networks

    Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. InProceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017

    2017

  67. [75]

    SampleRNN: An unconditional end-to-end neural audio generation model.arXiv preprint arXiv:1612.07837, 2016

    Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An unconditional end-to-end neural audio generation model.arXiv preprint arXiv:1612.07837, 2016

    2016 arXiv

  68. [76]

    Autoregressive speech synthesis without vector quantization

    Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. Autoregressive speech synthesis without vector quantization. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V...

    2025

  69. [77]

    Finite scalar quantization: VQ-VAE made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE made simple. InInternational Conference on Learning Representations, volume 2024, pages 51772–51783, 2024

    2024

  70. [78]

    Aligning latent geometry for spherical flow matching in image generation.arXiv preprint arXiv:2605.15193, 2026

    Tuna Han Salih Meral, Kaan Oktay, Hidir Yesiltepe, Adil Kaan Akan, and Pinar Yanardag. Aligning latent geometry for spherical flow matching in image generation.arXiv preprint arXiv:2605.15193, 2026

    2026 arXiv

  71. [79]

    Sensitivity to changes in the intensity of white noise and its relation to masking and loudness

    George A Miller. Sensitivity to changes in the intensity of white noise and its relation to masking and loudness. The Journal of the Acoustical Society of America, 19(4):609–619, 1947

    1947

  72. [80]

    How should we extract discrete audio tokens from self-supervised models?arXiv preprint arXiv:2406.10735, 2024

    Pooneh Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. How should we extract discrete audio tokens from self-supervised models?arXiv preprint arXiv:2406.10735, 2024

    2024 arXiv

  73. [81]

    Recognizing long-form speech using streaming end-to-end models

    Arun Narayanan, Rohit Prabhavalkar, Chung-Cheng Chiu, David Rybach, Tara N Sainath, and Trevor Strohman. Recognizing long-form speech using streaming end-to-end models. In2019 IEEE automatic speech recognition and understanding workshop (ASRU), pages 920–927. IEEE, 2019

    2019

  74. [82]

    Input perturbation reduces exposure bias in diffusion models.arXiv preprint arXiv:2301.11706, 2023

    Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Input perturbation reduces exposure bias in diffusion models.arXiv preprint arXiv:2301.11706, 2023

    2023 arXiv

  75. [83]

    Diffrhythm: Blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion

    Ziqian Ning, Huakang Chen, Yuepeng Jiang, Chunbo Hao, Guobin Ma, Shuai Wang, Jixun Yao, and Lei Xie. Diffrhythm: Blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion. arXiv preprint arXiv:2503.01183, 2025

    2025 arXiv

  76. [84]

    Scaling transformers for low-bitrate high-quality speech coding

    Julian Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, and Xubo Liu. Scaling transformers for low-bitrate high-quality speech coding. InInternationalConference on Learning Representations, volume 2025, pages 51997–52021, 2025

    2025

  77. [85]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  78. [86]

    Vibevoice technical report.arXiv preprint arXiv:2508.19205, 2025

    Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, et al. Vibevoice technical report.arXiv preprint arXiv:2508.19205, 2025

    2025 arXiv

  79. [87]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. InInternational conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  80. [88]

    Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis

    Desh Raj, Pavel Denisov, Zhuo Chen, Hakan Erdogan, Zili Huang, Maokui He, Shinji Watanabe, Jun Du, Takuya Yoshioka, Yi Luo, et al. Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis. In2021 IE...

    2021

  81. [89]

    Rolling diffusion models.arXiv preprint arXiv:2402.09470, 2024

    David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models.arXiv preprint arXiv:2402.09470, 2024

    2024 arXiv

  82. [90]

    Generalization in generation: A closer look at exposure bias

    Florian Schmidt. Generalization in generation: A closer look at exposure bias. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 157–167, 2019

    2019

  83. [91]

    Code-excited linear prediction (CELP): High-quality speech at very low bit rates

    Manfred Schroeder and B Atal. Code-excited linear prediction (CELP): High-quality speech at very low bit rates. In ICASSP’85. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 10, pages 937–940. IEEE, 1985

    1985

  84. [92]

    Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020. 43

    2002 arXiv

  85. [93]

    Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition

    Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, and Mike Seltzer. Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acousti...

    2021

  86. [94]

    Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  87. [95]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  88. [96]

    Geometry-aware autoencoders for metric learning and generative modeling on data manifolds

    Xingzhi Sun, Danqi Liao, Kincaid MacDonald, Yanlei Zhang, Guillaume Huguet, Guy Wolf, Ian Adelstein, Tim GJ Rudner, and Smita Krishnaswamy. Geometry-aware autoencoders for metric learning and generative modeling on data manifolds. In ICML 2024 Workshop on Geometry-grounded Rep...

    2024

  89. [97]

    An algorithm for intelligibility prediction of time–frequency weighted noisy speech.IEEE Transactions on audio, speech, and language processing, 19(7): 2125–2136, 2011

    Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligibility prediction of time–frequency weighted noisy speech.IEEE Transactions on audio, speech, and language processing, 19(7): 2125–2136, 2011

    2011

  90. [98]

    Gemma 4 technical report, 2026

    Gemma Team. Gemma 4 technical report, 2026. URLhttps://arxiv.org/abs/2607.02770

    2026 arXiv

  91. [99]

    Scaling text-to-image diffusion transformers with representation autoencoders

    Shengbang Tong, Boyang Zheng, Ziteng Wang, Bingda Tang, Nanye Ma, Ellis Brown, Jihan Yang, Rob Fergus, Yann LeCun, and Saining Xie. Scaling text-to-image diffusion transformers with representation autoencoders. arXiv preprint arXiv:2601.16208, 2026

    2026

  92. [100]

    Wavenet: A generative model for raw audio.arXiv preprint arXiv:1609.03499, 12(1), 2016

    Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalch- brenner, Andrew Senior, Koray Kavukcuoglu, et al. Wavenet: A generative model for raw audio.arXiv preprint arXiv:1609.03499, 12(1), 2016

    2016 arXiv

  93. [101]

    Neural discrete representation learning.Advancesinneuralinformation processing systems, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advancesinneuralinformation processing systems, 30, 2017

    2017

  94. [102]

    Neural codec language models are zero-shot text to speech synthesizers.arXiv preprint arXiv:2301.02111, 2023

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers.arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  95. [103]

    Pixnerd: Pixel neural field diffusion

    Shuai Wang, Ziteng Gao, Chenhui Zhu, Weilin Huang, and Limin Wang. Pixnerd: Pixel neural field diffusion. arXiv preprint arXiv:2507.23268, 2025

    2025 arXiv

  96. [104]

    Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens

    Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-tts: An efficient llm-based text-to-speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710, 2025

    2025 arXiv

  97. [105]

    Scaling speech tokenizers with diffusion autoencoders.arXiv preprint arXiv:2602.06602, 2026

    Yuancheng Wang, Zhenyu Tang, Yun Wang, Arthur Hinsvark, Yingru Liu, Yinghao Li, Kainan Peng, Junyi Ao, Mingbo Ma, Mike Seltzer, et al. Scaling speech tokenizers with diffusion autoencoders.arXiv preprint arXiv:2602.06602, 2026

    2026

  98. [106]

    Model of visual contrast gain control and pattern masking.Journal of the optical society of America A, 14(9):2379–2391, 1997

    Andrew B Watson and Joshua A Solomon. Model of visual contrast gain control and pattern masking.Journal of the optical society of America A, 14(9):2379–2391, 1997

    1997

  99. [107]

    Grounding continuous representations in geometry: Equivariant neural fields

    David Wessels, David Knigge, Riccardo Valperga, Samuele Papa, Sharvaree Vadgama, Efstratios Gavves, and Erik Bekkers. Grounding continuous representations in geometry: Equivariant neural fields. InInternational Conference on Learning Representations, volume 2025, pages 59774–5...

    2025

  100. [108]

    Step-audio 2 technical report.arXiv preprint arXiv:2507.16632, 2025

    Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, et al. Step-audio 2 technical report.arXiv preprint arXiv:2507.16632, 2025

    2025 arXiv

  101. [109]

    Next-gpt: Any-to-any multimodal llm.arXiv preprint arXiv:2309.05519, 2023

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm.arXiv preprint arXiv:2309.05519, 2023

    2023 arXiv

  102. [110]

    Fireredtts-2: Towards long conversational speech generation for podcast and chatbot.arXiv preprint arXiv:2509.02020, 2025

    Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, and Yao Hu. Fireredtts-2: Towards long conversational speech generation for podcast and chatbot.arXiv preprint arXiv:2509.02020, 2025. 44

    2025 arXiv

  103. [111]

    Longcat- audiodit: High-fidelity diffusion text-to-speech in the waveform latent space.arXiv preprint arXiv:2603.29339, 2026

    Detai Xin, Shujie Hu, Chengzuo Yang, Chen Huang, Guoqiao Yu, Guanglu Wan, and Xunliang Cai. Longcat- audiodit: High-fidelity diffusion text-to-speech in the waveform latent space.arXiv preprint arXiv:2603.29339, 2026

    2026

  104. [112]

    Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report.arXiv preprint arXiv:2509.17765, 2025

    2025 arXiv

  105. [113]

    Making reconstruction FID predictive of diffusion generation FID.arXiv preprint arXiv:2603.05630, 2026

    Tongda Xu, Mingwei He, Shady Abu-Hussein, Jose Miguel Hernandez-Lobato, Chunhang Zheng, Kai Zhao, Chao Zhou, Ya-Qin Zhang, and Yan Wang. Making reconstruction FID predictive of diffusion generation FID.arXiv preprint arXiv:2603.05630, 2026

    2026 arXiv

  106. [114]

    Moss-audio technical report.arXiv preprint arXiv:2606.01802, 2026

    Chen Yang, Chufan Yu, Hanfu Chen, Jie Zhu, Jingqi Chen, Ke Chen, Wenxuan Wang, Yang Wang, Yaozhou Jiang, Yi Jiang, et al. Moss-audio technical report.arXiv preprint arXiv:2606.01802, 2026

    2026 arXiv

  107. [115]

    Codec does matter: Exploring the semantic shortcoming of codec for audio language model

    Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference on Artificial Intelligence...

    2025

  108. [116]

    From slow bidirectional to fast autoregressive video diffusion models

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22963–22974, 2025

    2025

  109. [117]

    Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940, 2024

    Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think.arXiv preprint arXiv:2410.06940, 2024

    2024 arXiv

  110. [118]

    Pixeldit: Pixel diffusion transformers for image generation

    Yongsheng Yu, Wei Xiong, Weili Nie, Yichen Sheng, Shiqiu Liu, and Jiebo Luo. Pixeldit: Pixel diffusion transformers for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14273–14282, 2026

    2026

  111. [119]

    Yue: Scaling open foundation models for long-form music generation.arXiv preprint arXiv:2503.08638, 2025

    Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al. Yue: Scaling open foundation models for long-form music generation.arXiv preprint arXiv:2503.08638, 2025

    2025

  112. [120]

    What matters for diffusion-friendly latent manifold? prior-aligned autoencoders for latent diffusion.arXiv preprint arXiv:2605.07915, 2026

    Zhengrong Yue, Taihang Hu, Mengting Chen, Haiyu Zhang, Zihao Pan, Tao Liu, Zikang Wang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, et al. What matters for diffusion-friendly latent manifold? prior-aligned autoencoders for latent diffusion.arXiv preprint arXiv:2605.07915, 2026

    2026 arXiv

  113. [121]

    Soundstream: An end-to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495–507, 2021

    2021

  114. [122]

    Mimo-audio: Audio language models are few-shot learners.arXiv preprint arXiv:2512.23808, 2025

    Dong Zhang, Gang Wang, Jinlong Xue, Kai Fang, Liang Zhao, Rui Ma, Shuhuai Ren, Shuo Liu, Tao Guo, Weiji Zhuang, et al. Mimo-audio: Audio language models are few-shot learners.arXiv preprint arXiv:2512.23808, 2025

    2025

  115. [123]

    Speechtokenizer: Unified speech tokenizer for speech language models

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech language models. In International Conference on Learning Representations, volume 2024, pages 31798–31818, 2024

    2024

  116. [124]

    Diffusion transformers with representation autoencoders

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders. arXiv preprint arXiv:2510.11690, 2025

    2025 arXiv

  117. [125]

    Wavflow: Audio generation in waveform space.arXiv preprint arXiv:2605.18749, 2026

    Feiyan Zhou, Luyuan Wang, Shoufa Chen, Zhe Wang, Zhiheng Liu, Yuren Cong, Xiaohui Zhang, Fanny Yang, and Belinda Zeng. Wavflow: Audio generation in waveform space.arXiv preprint arXiv:2605.18749, 2026

    2026 arXiv

  118. [126]

    Voxcpm: Tokenizer-free tts for context-aware speech generation and true-to-life voice cloning

    Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Ziyang Wang, Runchuan Ye, Weiyue Sun, Jiancheng Gui, Kehan Li, et al. Voxcpm: Tokenizer-free tts for context-aware speech generation and true-to-life voice cloning. arXiv preprint arXiv:2509.24650, 2025

    2025

  119. [127]

    Voxcpm2 technical report.arXiv preprint arXiv:2606.06928, 2026

    Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Jiancheng Gui, Jiaheng Wu, Ziyang Wang, Xudong Shen, Runchuan Ye, et al. Voxcpm2 technical report.arXiv preprint arXiv:2606.06928, 2026

    2026 arXiv

  120. [128]

    Springer Science & Business Media, 2013

    Eberhard Zwicker and Hugo Fastl.Psychoacoustics: Facts and models, volume 22. Springer Science & Business Media, 2013. 45

    2013

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.