REVIEW 3 major objections 5 minor 128 references
Autoregressive speech generation stays stable at 50 seconds when an 8-Hz, 768-dim spherical token space is shaped around a low-dimensional core and decoded by multi-path, time-scheduled residual guidance.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 08:29 UTC pith:2G45ZBSA
load-bearing objection A credible tokenizer-side contribution, but the load-bearing generator claim (multi-path routing) lacks a standard-ELD control and no artifacts are released. the 3 major comments →
Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that a low-frame-rate, high-dimensional continuous token can be a stable autoregressive target if the representation geometry and the generation framework are designed jointly. Locodec produces 8-Hz, 768-dimensional spherical tokens centered on a lower-dimensional interpolatable core manifold, and uses postfix dimension dropout so that availability bias becomes a coordinate-wise energy hierarchy, making the most reconstruction-critical dimensions the most identifiable under noise. MP-ELD predicts these tokens with a flow-matching decoder that separates local-continuity, self-consistency, and alignment-consistency conditioning into orthogonalized paths and combine
What carries the argument
The central object is the 8-Hz, 768-dimensional spherical continuous token. Three mechanisms carry the argument: (1) the spherical-cap area scale K_area(N,θ)=1/µ_N(θ)≈cosθ√(2πN)(sinθ)^(-(N-1)), an identity showing that dense interpolatability in the full high-dimensional sphere is exponentially impossible, which motivates constraining the token space around a lower-dimensional core; (2) postfix dimension dropout (PDD), which turns an availability bias into an energy hierarchy and thereby improves per-token identifiability under noise; and (3) multi-path residual classifier-free guidance, in which local-continuity, self-consistency, and alignment-consistency condition vectors are Gram-Schmidt
Load-bearing premise
The design assumes the main cause of long-horizon drift is that partially overlapping acoustic cues get amplified inconsistently when different guidance paths are combined; if the drift actually comes from exposure bias, high target-space entropy, or collapsed representations, the multi-path machinery is fixing a secondary cause.
What would settle it
Train the same Locodec tokens and the same training budget with a single-path ELD generator using standard full-condition CFG with a null path; if that model maintains last-segment speaker similarity above roughly 0.65 at 50 seconds while matching the reported WER, the pathway-conflict hypothesis is falsified and the multi-path orthogonalization is not the cause of stability.
If this is right
- High-bandwidth continuous tokens are a practical AR target: the 8-Hz, 768-dim design reaches 0.95% WER on Chinese and 1.87% on English TTS evaluation without external ASR/SSL models, pretrained text LMs, or post-training.
- Long-horizon stability becomes a guidance-design problem: with a delayed self-consistency schedule, last-segment speaker similarity stays near 0.67 at 50 seconds, while constant guidance collapses to about 0.40, so short-utterance evaluations miss the drift.
- Token-space shaping changes learnability: adding the low-dimensional core and PDD lowers the generator's training loss and improves rollout stability; a moderate core dimension (32) gives the best WER/stability balance.
- Residual CFG gives interpretable controls: alignment guidance mainly moves WER, self-consistency guidance mainly moves speaker similarity, and the bridge-time schedule trades the two off.
- Low frame rate itself contributes a semantic inductive bias: each 8-Hz token spans about 125 ms, close to phoneme timing, helping competitive WER without semantic supervision.
Where Pith is reading between the lines
- Beyond the paper: the same shaped low-rate continuous-token recipe could transfer to other sequence modalities (image patches, video, long-context audio), since the reconstruction-capacity versus AR-stability tension is general.
- Beyond the paper: the remaining speaker-similarity gap suggests a concrete next test — add a light semantic–acoustic factorization inside the continuous token space, still without SSL/ASR supervision, which could recover fine-grained similarity while keeping WER and stability gains.
- Beyond the paper: if the pathway-conflict hypothesis is right, strong constant classifier-free guidance in autoregressive flow models is itself a principal cause of long-form drift, which would redirect exposure-bias fixes toward scheduled residual guidance rather than larger contexts or extra semantic tokens.
- Beyond the paper: a dynamic CFG schedule that varies with AR step index, in addition to bridge time, should extend stability beyond 50 seconds; the paper leaves this untested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper asks whether a low-frame-rate (8 Hz), high-dimensional (768-D) continuous spherical token can be a stable autoregressive speech-generation target. The authors propose Locodec, a locally encoded codec with rotation-based spherical corruption, postfix dimension dropout (PDD), and a learnable low-dimensional core manifold aligned via orthogonal projection/lifting and a bidirectional commitment loss. They propose MP-ELD, an encoder–LM–decoder flow-matching generator with three deliberately separated conditioning paths (local continuity, self-consistency, alignment), Gram–Schmidt orthogonalized residual conditioning, and residual classifier-free guidance with a bridge-time-dependent self-consistency schedule. Experiments cover eight tokenizer configurations and controlled MP-ELD training; results include reconstruction metrics, training-loss predictability curves, CFG WER/SIM sweeps, long-form segment-SIM curves, and Seed-TTS-eval comparisons. The headline results are competitive WER (0.95% ZH, 1.87% EN for the best configuration) without external SSL/ASR models or pretrained text LMs, and improved long-horizon stability with scheduled residual CFG.
Significance. The paper's analytical pieces are sound and largely self-contained: the spherical-cap covering argument (Eqs. 1–6), the chi-squared concentration motivating spherical latents, the under-stepping algebra (Eqs. 28–29), and the high-dimensional near-equivalence of SFM and VP paths are all correct. The experimental spine is also a strength: eight tokenizer configurations are compared with a fixed generator architecture and budget, all CFG settings are reported rather than cherry-picked, and the benchmark numbers use the public Seed-TTS-eval test set. If the central hypothesis is correct, the work demonstrates that geometric shaping of a continuous token space plus guided multi-path routing can substitute for external semantic supervision in AR TTS, which is an important and falsifiable claim. The main weakness is that the mechanism claimed to deliver the stability gain—multi-path information routing—is never isolated from the guidance schedule or model-capacity increase.
major comments (3)
- [§4.2; Eq. (63); Tables 4–5; Fig. 6] The central claim that multi-path routing is what mitigates error accumulation is not isolated experimentally. All reported MP-ELD results use the full three-path model with CFG extrapolation; the manuscript never compares against a standard ELD (one token encoder, one LM, one conditioning vector) under the same tokenizer, training budget, and residual-CFG schedule. Fig. 6 shows that the no-guidance full-condition setting (λmax_sc=λac=1) has WER>10% on the internal selection set, while every successful setting uses λac>1 or λmax_sc>1. Since the proposed residual CFG could in principle be applied to any two-condition ELD (e.g., full vs. local-only), the observed gains may come from the CFG schedule or from the extra parameter count (three encoders plus two LMs), not from pathway separation. A parameter-matched single-path baseline with the same residual schedule is required; without it, t
- [§3.3; Fig. 3; Table 3; Table 4] The paper claims that the PDD availability bias is converted through training into an energy bias, and that this energy hierarchy is what improves identifiability and predictability. The evidence shows a correlational pattern (energy profile and prefix-K reconstruction improve with K), but no ablation isolates the energy-bias mechanism from the more mundane effect of PDD as input dropout/regularization. For example, a uniform-random dimension-dropout variant (same expected retention but no prefix ordering) or a PDD variant trained without rotation corruption would test whether the ordered availability and the fixed spherical energy budget are necessary for the reported gains. This matters because PDD is one of the two main tokenizer contributions, and Table 4 attributes large long-form improvements to it (e.g., 32/× vs 32/✓ under CFG-S: fifth-segment SIM 0.419 vs 0.578).
- [§5.1; §5.3; Fig. 8; Table 4] The long-form stability claim rests entirely on an internal Chinese test set whose size, speaker composition, and recording conditions are not reported, and no public long-form benchmark is used. Since the headline 'long-horizon stability' is central and the Seed-TTS-eval utterances are only ~5 s, the paper should report the number of long-form utterances, the number of speakers, and ideally release the test set or an evaluation protocol. Fig. 8 shows 95% confidence intervals but no sample size; Table 4 reports long-form metrics without any uncertainty estimates.
minor comments (5)
- [§3.5] Typographical spacing issue: 'inthepresentsettings' appears as 'inthepresentsettings' in the text; please fix word spacing throughout.
- [Table 4] Several entries lack separators, e.g., '2.050.630', '9.630.748', '0.950.687'. These are hard to parse and should be formatted with commas or spaces.
- [Fig. 6] The caption says 'Darker points indicate configurations closer to the preferred low-WER, high-SIM region,' but the figure has no colorbar or explicit gray-scale legend. Please add one.
- [§4.2; Eq. (64)] When γ=0, s(τ)=τ^0 is undefined at τ=0 in the usual convention. Since the paper states s(0)=0, this corner case should be clarified (e.g., define s(τ)=1 for τ>0 and s(0)=0, or use a different schedule).
- [Fig. 5 caption] The notation 'd/PDD' is ambiguous: the caption says 'The notation d/PDD denotes the core dimension and whether PDD is used.' Please define the slash convention (e.g., 32/× vs 32/✓) explicitly, and use the same convention in Table 4.
Circularity Check
No significant circularity: the representation-shaping and stability claims are supported by external benchmarks and internal evaluations, and the paper's self-citations are methodological and non-load-bearing.
full rationale
I walked the claimed derivation chain. The analytical results (spherical-cap covering, Gaussian concentration, SFM/VP-path equivalence, MSE-optimal velocity magnitude, direction-only supervision) are derived from standard mathematics and do not assume the paper's conclusions. The tokenizer's energy hierarchy is induced by PDD and then verified empirically through coordinate-energy profiles and reconstruction from restricted representations, rather than being asserted by construction. Predictability is measured by MP-ELD training loss and by generation WER/SIM on the public Seed-TTS-eval benchmark, which was not used for fitting; CFG hyperparameters were tuned on internal medium/long-form sets and then evaluated on the public set, so the headline numbers are not fitted to their own test data. The few self-citations (e.g., [46, 70, 71, 72, 88]) are background or methodology references (long-sequence models, context-codec concept, STFT/GAN loss recipe) and do not carry the central argument. The paper explicitly labels the pathway-conflict mechanism as a hypothesis in Sec 4.2, and the absence of a standard-ELD ablation is a missing-control concern, not a circularity. No equation reduces to its own input by construction, and no predicted quantity is a renamed fitted parameter.
Axiom & Free-Parameter Ledger
free parameters (6)
- PDD dropout keep-probability p =
0.5
- Spherical rotation corruption prior =
Beta(1,2) scaled to [0, pi/2]
- Tokenizer loss weights lambda_low, lambda_commit, lambda_lmag =
0.2 / 0.1 / 0.2
- Core manifold dimension d_core =
32 (selected from 768, 256, 64, 32, 16)
- CFG hyperparameters (lambda_sc_max, lambda_ac, gamma) =
(2,2,1) long-form config; (2.5,2.5,1) best on CFG-selection set
- Inference velocity magnitude pi*R/2 and sphere radius R =
R never reported
axioms (7)
- standard math Spherical cap covering asymptotics (Eqs 1-4): mu_N(theta) ~ (sin theta)^(N-1) / (cos theta sqrt(2 pi N))
- standard math High-dimensional concentration: chi-squared_N lies on a thin shell; uniform random vectors on S^(D-1) are nearly orthogonal with inner-product std O(D^-1/2)
- domain assumption Manifold hypothesis: real-world signals lie near a lower-dimensional manifold, and a well-shaped high-dim token space inherits this structure
- domain assumption Reconstruction-stable basins: each token has a decodable cap of radius theta, and K_area(d,rho) bounds the number of distinguishable signal states M_data(epsilon)
- ad hoc to paper CFG-pathway-conflict hypothesis: AR drift is dominated by inconsistent amplification of overlapping acoustic cues across guidance paths
- ad hoc to paper Availability-bias to energy-bias training dynamic: higher dropout-retention probability for prefix dimensions forces higher learned energy there under a fixed spherical energy budget and rotation noise
- domain assumption Evaluation assumption: WER/SIM/seg-SIM with the DiTAR-configured models capture the claimed qualities, and the internal ZH long-form set is representative
read the original abstract
Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation. Representations with higher frame rates or greater capacity can preserve more signal detail, but they also make streaming generation more vulnerable to distribution drift and AR error accumulation. Conversely, shorter and more compressed representations simplify AR modeling, but their limited bandwidth may discard important components and constrain the upper bound of reconstruction fidelity and generation quality. We ask whether a low-frame-rate, high-dimensional, high-bandwidth continuous representation can be co-designed with a streaming generation framework to support robust high-fidelity reconstruction, strong single-token predictability, and superior long-horizon stability. We decompose this goal into two coupled problems: what geometric and statistical properties a high-dimensional representation space should have, and how an AR continuous-token generator should be structured to resist error accumulation. Accordingly, we propose Locodec, a locally encoded codec that shapes its representation space to improve the interpolatability of a lower-dimensional core manifold and the identifiability of the native high-dimensional coordinates, thereby improving the predictability of high-dimensional high-bandwidth tokens. We also propose MP-ELD, a single-token AR flow-matching framework that uses multi-path information routing and residual classifier-free guidance to mitigate error accumulation. Experiments with 8-Hz, 768-dimensional tokens show that our design preserves reconstruction quality, improves single-token predictability, achieves competitive WER, and maintains stable long-form synthesis, without using external SSL/ASR models, pretrained text language models, or post-training stages.
Reference graph
Works this paper leans on
-
[1]
Stochastic interpolants: A unifying framework for flows and diffusions.Journal of Machine Learning Research, 26(209):1–80, 2025
Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions.Journal of Machine Learning Research, 26(209):1–80, 2025
2025
-
[2]
Building normalizing flows with stochastic interpolants.arXiv preprint arXiv:2209.15571, 2022
Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants.arXiv preprint arXiv:2209.15571, 2022
Pith/arXiv arXiv 2022
-
[3]
The curse (s) of dimensionality.Nat Methods, 15(6):399–400, 2018
Naomi Altman and Martin Krzywinski. The curse (s) of dimensionality.Nat Methods, 15(6):399–400, 2018
2018
-
[4]
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-TTS: A family of high-quality versatile speech generation models.arXiv preprint arXiv:2406.02430, 2024
Pith/arXiv arXiv 2024
-
[5]
Common voice: A massively-multilingual speech corpus
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. In Proceedings of the twelfth language resources and evaluation conference, pages 4218–4222, 2020
2020
-
[6]
Latent space oddity: on the curvature of deep generative models
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. arXiv preprint arXiv:1710.11379, 2017
Pith/arXiv arXiv 2017
-
[7]
Geometrically enriched latent spaces.arXiv preprint arXiv:2008.00565, 2020
Georgios Arvanitidis, Søren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces.arXiv preprint arXiv:2008.00565, 2020
Pith/arXiv arXiv 2008
-
[8]
Adaptive predictive coding of speech signals.Bell System Technical Journal, 49(8):1973–1986, 1970
Bishnu S Atal and Manfred R Schroeder. Adaptive predictive coding of speech signals.Bell System Technical Journal, 49(8):1973–1986, 1970
1973
-
[9]
wav2vec 2.0: A framework for self-supervised learning of speech representations
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449–12460, 2020
2020
-
[10]
Seed-music: A unified framework for high quality and controlled music generation
Ye Bai, Haonan Chen, Jitong Chen, Zhuo Chen, Yi Deng, Xiaohong Dong, Lamtharn Hantrakul, Weituo Hao, Qingqing Huang, Zhongyi Huang, et al. Seed-music: A unified framework for high quality and controlled music generation. arXiv preprint arXiv:2409.09214, 2024
Pith/arXiv arXiv 2024
-
[11]
SPIE press, 1999
Peter GJ Barten.Contrast sensitivity of the human eye and its effects on image quality. SPIE press, 1999
1999
-
[12]
Location-relative attention mechanisms for robust long-form speech synthesis
Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6194–6198. IEEE, 2020
2020
-
[13]
The adaptive multirate wideband speech codec (AMR-WB).IEEE transactions on speech and audio processing, 10(8):620–636, 2002
Bruno Bessette, Redwan Salami, Roch Lefebvre, Milan Jelinek, Jani Rotola-Pukkila, Janne Vainio, Hannu Mikkola, and Kari Jarvinen. The adaptive multirate wideband speech codec (AMR-WB).IEEE transactions on speech and audio processing, 10(8):620–636, 2002
2002
-
[14]
Audiolm: a language modeling approach to audio generation
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31:2523–2533, 2023
2023
-
[15]
Application of fourier analysis to the visibility of gratings.The Journal of physiology, 197(3):551, 1968
Fergus W Campbell and John G Robson. Application of fourier analysis to the visibility of gratings.The Journal of physiology, 197(3):551, 1968
1968
-
[16]
A discussion of semi-supervised learning and transduction
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. A discussion of semi-supervised learning and transduction. In Semi-supervised learning, pages 473–478. MIT Press, 2006
2006
-
[17]
Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advancesin Neural Information Processing Systems, 37:24081–24125, 2024
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advancesin Neural Information Processing Systems, 37:24081–24125, 2024
2024
-
[18]
Minmo: A multimodal large language model for seamless voice interaction
Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, et al. Minmo: A multimodal large language model for seamless voice interaction. arXiv preprint arXiv:2501.06282, 2025
Pith/arXiv arXiv 2025
-
[19]
Flow matching on general geometries.arXiv preprint arXiv:2302.03660, 2023
Ricky TQ Chen and Yaron Lipman. Flow matching on general geometries.arXiv preprint arXiv:2302.03660, 2023. 39
Pith/arXiv arXiv 2023
-
[20]
Wavlm: Large-scale self-supervised pre-training for full stack speech processing
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022
2022
-
[21]
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers.arXiv preprint arXiv:2406.05370, 2024
Pith/arXiv arXiv 2024
-
[22]
On the importance of noise scheduling for diffusion models.arXiv preprint arXiv:2301.10972, 2023
Ting Chen. On the importance of noise scheduling for diffusion models.arXiv preprint arXiv:2301.10972, 2023
Pith/arXiv arXiv 2023
-
[23]
Wenxi Chen, Dongya Jia, Yushen Chen, Zhikang Niu, Yuzhe Liang, Xiquan Li, Ruiqi Yan, Ziyang Ma, Guanrou Yang, Sanyuan Chen, et al. Wavtts: Towards high-quality zero-shot tts via direct raw waveform modeling.arXiv preprint arXiv:2606.03455, 2026
Pith/arXiv arXiv 2026
-
[24]
Sac: Neural speech codec with semantic-acoustic dual-stream quantization
Wenxi Chen, Ruiqi Yan, Yushen Chen, Zhikang Niu, Ziyang Ma, Xiquan Li, Yuzhe Liang, Shunshun Yin, Ming Tao, Xinsheng Wang, et al. Sac: Neural speech codec with semantic-acoustic dual-stream quantization. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3030–3048, 2026
2026
-
[25]
F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching
Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, JianZhao JianZhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6255–6271, 2025
2025
-
[26]
Visqol v3: An open source production ready objective speech and audio metric
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O’Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In2020 twelfth international conference on quality of multimedia experience (QoMEX), pages 1–6. IEEE, 2020
2020
-
[27]
Self-supervised learning with random- projection quantizer for speech recognition
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random- projection quantizer for speech recognition. InInternational Conference on Machine Learning, pages 3915–3924. PMLR, 2022
2022
-
[28]
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.arXiv preprint arXiv:2311.07919, 2023
Pith/arXiv arXiv 2023
-
[29]
Simple and controllable music generation.Advances in neural information processing systems, 36:47704–47720, 2023
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation.Advances in neural information processing systems, 36:47704–47720, 2023
2023
-
[30]
Variational autoencoders without the variation.arXiv preprint arXiv:2203.00645, 2022
Gregory A Daly, Jonathan E Fieldsend, and Gavin Tabor. Variational autoencoders without the variation.arXiv preprint arXiv:2203.00645, 2022
Pith/arXiv arXiv 2022
-
[31]
Hyperspherical variational auto-encoders
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomczak. Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891, 2018
Pith/arXiv arXiv 2018
-
[32]
Pullback flow matching on data manifolds.arXiv preprint arXiv:2410.04543, 2024
Friso de Kruiff, Erik Bekkers, Ozan Öktem, Carola-Bibiane Schönlieb, and Willem Diepeveen. Pullback flow matching on data manifolds.arXiv preprint arXiv:2410.04543, 2024
Pith/arXiv arXiv 2024
-
[33]
High fidelity neural audio compression.arXiv preprint arXiv:2210.13438, 2022
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression.arXiv preprint arXiv:2210.13438, 2022
Pith/arXiv arXiv 2022
-
[34]
Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue.arXiv preprint arXiv:2410.00037, 2024
Pith/arXiv arXiv 2024
-
[35]
Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music.arXiv preprint arXiv:2005.00341, 2020
Pith/arXiv arXiv 2005
-
[36]
Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025
Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025
Pith/arXiv arXiv 2025
-
[37]
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024. 40
Pith/arXiv arXiv 2024
-
[38]
Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training.arXiv preprint arXiv:2505.17589, 2025
Pith/arXiv arXiv 2025
-
[39]
E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts
Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts. In2024 IEEE spoken language technology workshop (SLT), pages 682–689. IEEE, 2024
2024
-
[40]
Long-form music generation with latent diffusion.arXiv preprint arXiv:2404.10301, 2024
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion.arXiv preprint arXiv:2404.10301, 2024
Pith/arXiv arXiv 2024
-
[41]
Barewave: Waveform-native flow-matching text-to-speech.arXiv preprint arXiv:2606.09048, 2026
Wei Fan, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang, and Nenghai Yu. Barewave: Waveform-native flow-matching text-to-speech.arXiv preprint arXiv:2606.09048, 2026
Pith/arXiv arXiv 2026
-
[42]
Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.Advances in Neural Information Processing Systems, 38:41819–41886, 2026
Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.Advances in Neural Information Processing Systems, 38:41819–41886, 2026
2026
-
[43]
Blessing of dimensionality: mathematical foundations of the statistical physics of data
Alexander N Gorban and Ivan Yu Tyukin. Blessing of dimensionality: mathematical foundations of the statistical physics of data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2118):20170237, 2018
2018
-
[44]
Hao-Han Guo, Yao Hu, Kun Liu, Fei-Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications.arXiv preprint arXiv:2409.03283, 2024
Pith/arXiv arXiv 2024
-
[45]
Didispeech: A large scale mandarin speech corpus
Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. Didispeech: A large scale mandarin speech corpus. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6968–6972. IEEE, 2021
2021
-
[46]
Continuous speech separation using speaker inventory for long recording
Cong Han, Yi Luo, Chenda Li, Tianyan Zhou, Keisuke Kinoshita, Shinji Watanabe, Marc Delcroix, Hakan Erdogan, John R Hershey, Nima Mesgarani, et al. Continuous speech separation using speaker inventory for long recording. In Interspeech, pages 3036–3040, 2021
2021
-
[47]
beta-VAE: Learning basic visual concepts with a constrained variational framework
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id= Sy2fzU9gl
2017
-
[48]
Simpler diffusion: 1.5 fid on imagenet512 with pixel-space diffusion
Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans. Simpler diffusion: 1.5 fid on imagenet512 with pixel-space diffusion. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18062–18071, 2025
2025
-
[49]
Hubert: Self-supervised speech representation learning by masked prediction of hidden units
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrah- man Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021
2021
-
[50]
Music transformer.arXiv preprint arXiv:1809.04281, 2018
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer.arXiv preprint arXiv:1809.04281, 2018
Pith/arXiv arXiv 2018
-
[51]
Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024
Pith/arXiv arXiv 2024
-
[52]
Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling
Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Ruiqi Li, Ziang Zhang, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. In International Conference on Learning Representations, volume 2025, pages 93809–93826, 2025
2025
-
[53]
Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, et al. Ditar: Diffusion transformer autoregressive modeling for speech generation.arXiv preprint arXiv:2502.03930, 2025
arXiv 2025
-
[54]
Yuepeng Jiang, Huakang Chen, Ziqian Ning, Jixun Yao, Zerui Han, Di Wu, Meng Meng, Jian Luan, Zhonghua Fu, and Lei Xie. Diffrhythm 2: Efficient and high fidelity song generation via block flow matching.arXiv preprint arXiv:2510.22950, 2025. 41
arXiv 2025
-
[55]
Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100, 2024
Pith/arXiv arXiv 2024
-
[56]
Guolin Ke and Hui Xue. Hyperspherical latents improve continuous-token autoregressive generation.arXiv preprint arXiv:2509.24335, 2025
arXiv 2025
-
[57]
Mel-cepstral distance measure for objective speech quality assessment
Robert Kubichek. Mel-cepstral distance measure for objective speech quality assessment. InProceedings of IEEE pacific rim conference on communications computers and signal processing, volume 1, pages 125–128. IEEE, 1993
1993
-
[58]
High-fidelity audio compression with improved RVQGAN.Advances in Neural Information Processing Systems, 36:27980–27993, 2023
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN.Advances in Neural Information Processing Systems, 36:27980–27993, 2023
2023
-
[59]
Autoregressive image generation using residual quantization
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11523–11532, 2022
2022
-
[60]
Geometry-aware image flow matching
Junho Lee, Kwanseok Kim, and Joonseok Lee. Geometry-aware image flow matching. arXiv preprint arXiv:2605.25294, 2026
Pith/arXiv arXiv 2026
-
[61]
Sequential data generation with groupwise diffusion process.arXiv preprint arXiv:2310.01400, 2023
Sangyun Lee, Gayoung Lee, Hyunsu Kim, Junho Kim, and Youngjung Uh. Sequential data generation with groupwise diffusion process.arXiv preprint arXiv:2310.01400, 2023
Pith/arXiv arXiv 2023
-
[62]
REPA-E: Unlocking VAE for end-to-end tuning of latent diffusion transformers
Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. REPA-E: Unlocking VAE for end-to-end tuning of latent diffusion transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18262–18272, 2025
2025
-
[63]
Back to basics: Let denoising generative models denoise
Tianhong Li and Kaiming He. Back to basics: Let denoising generative models denoise. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 36115–36125, 2026
2026
-
[64]
Autoregressive image generation without vector quantization.Advances in Neural Information Processing Systems, 37:56424–56445, 2024
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization.Advances in Neural Information Processing Systems, 37:56424–56445, 2024
2024
-
[65]
Shi Lian, Changtao Li, Bohan Li, Hankun Wang, Da Zheng, Junfeng Tian, Yufeng Ma, Colin Zhang, and Kai Yu. dots. tts technical report.arXiv preprint arXiv:2606.07080, 2026
Pith/arXiv arXiv 2026
-
[66]
Fish audio s2 technical report.arXiv preprint arXiv:2603.08823, 2026
Shijia Liao, Yuxuan Wang, Songting Liu, Yifan Cheng, Ruoyi Zhang, Tianyu Li, Shidong Li, Yisheng Zheng, Xingwei Liu, Qingzheng Wang, et al. Fish audio s2 technical report.arXiv preprint arXiv:2603.08823, 2026
arXiv 2026
-
[67]
Semanticodec: An ultra low bitrate semantic audio codec for general sound.IEEE Journal of Selected Topics in Signal Processing, 18 (8):1448–1461, 2024
Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanticodec: An ultra low bitrate semantic audio codec for general sound.IEEE Journal of Selected Topics in Signal Processing, 18 (8):1448–1461, 2024
2024
-
[68]
Autoregressive diffusion transformer for text-to-speech synthesis
Zhijun Liu, Shuai Wang, Sho Inoue, Qibing Bai, and Haizhou Li. Autoregressive diffusion transformer for text-to-speech synthesis. arXiv preprint arXiv:2406.05551, 2024
Pith/arXiv arXiv 2024
-
[69]
A convnet for the 2020s
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022
2022
-
[70]
Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation
Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path RNN: Efficient long sequence modeling for time-domain single-channel speech separation. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46–50. IEEE, 2020
2020
-
[71]
Group communication with context codec for lightweight source separation
Yi Luo, Cong Han, and Nima Mesgarani. Group communication with context codec for lightweight source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1752–1761, 2021
2021
-
[72]
Gull: A generative multifunctional audio codec
Yi Luo, Jianwei Yu, Hangting Chen, Rongzhi Gu, and Chao Weng. Gull: A generative multifunctional audio codec. arXiv preprint arXiv:2404.04947, 2024
Pith/arXiv arXiv 2024
-
[73]
Huanru Henry Mao, Shuyang Li, Julian McAuley, and Garrison Cottrell. Speech recognition and multi-speaker diarization of long conversations.arXiv preprint arXiv:2005.08072, 2020. 42
Pith/arXiv arXiv 2005
-
[74]
Least squares generative adversarial networks
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. InProceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017
2017
-
[75]
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. SampleRNN: An unconditional end-to-end neural audio generation model.arXiv preprint arXiv:1612.07837, 2016
Pith/arXiv arXiv 2016
-
[76]
Autoregressive speech synthesis without vector quantization
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. Autoregressive speech synthesis without vector quantization. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1287–1300, 2025
2025
-
[77]
Finite scalar quantization: VQ-VAE made simple
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE made simple. InInternational Conference on Learning Representations, volume 2024, pages 51772–51783, 2024
2024
-
[78]
Tuna Han Salih Meral, Kaan Oktay, Hidir Yesiltepe, Adil Kaan Akan, and Pinar Yanardag. Aligning latent geometry for spherical flow matching in image generation.arXiv preprint arXiv:2605.15193, 2026
Pith/arXiv arXiv 2026
-
[79]
Sensitivity to changes in the intensity of white noise and its relation to masking and loudness
George A Miller. Sensitivity to changes in the intensity of white noise and its relation to masking and loudness. The Journal of the Acoustical Society of America, 19(4):609–619, 1947
1947
-
[80]
Pooneh Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. How should we extract discrete audio tokens from self-supervised models?arXiv preprint arXiv:2406.10735, 2024
Pith/arXiv arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.