Pith. sign in

REVIEW 4 major objections 6 minor 47 references

UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single distilled codebook stream can carry enough audio detail for an LLM to synthesize emotionally expressive speech end to end, this paper argues.

desk verdict A plausible engineering recipe with open code, but the flagship SOTA claim rests on a missing teacher baseline and unstandardized MOS. read the letter →

arxiv 2505.17426 v1 pith:AKSPIIEM submitted 2025-05-23 cs.SD cs.AIeess.AS

Rui Wang , Qianguo Sun , Tianrong Chen , Zhiyun Zeng , Junlong Wu , Jiaxing Zhang This is my paper
classification cs.SDcs.AIeess.AS
keywords text-to-speechneuralaudiocodecsingle-codebookdistillationlanguagemodelemotion-awarespeechsynthesisuniversalpretrainingdirectpreferenceoptimizationcodebookutilization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single-codebook audio codec, distilled from a multi-codebook codec, can carry the full acoustic and semantic content of speech, and that a large language model trained on this one token stream can synthesize emotionally expressive speech end to end. The authors introduce DistilCodec, a codec with 32,768 codes and near-total codebook utilization, trained on 100,000 hours of unlabeled universal audio, and UniTTS, a Qwen2.5-7B-based system that autoregresses text and audio together. If correct, TTS no longer needs to split audio into semantic and acoustic streams, and large pools of unlabeled audio can be used for pretraining. The paper reports that UniTTS reaches top emotional-expressiveness scores while preserving the LLM's text ability.

What carries the argument

The load-bearing mechanism is DMS (Distilling Multi-Codebook NAC to Single-Codebook NAC): train a teacher codec with residual/group vector quantization, then initialize a student codec's encoder and decoder from the teacher, replace the quantizer with a single 32,768-entry codebook of dimension 3,584, and retrain with the same GAN objectives. This lets the student inherit acoustic modeling capacity while learning a dense, balanced, single-stream discrete representation. UniTTS then concatenates that codebook with the word embedding of Qwen2.5-7B (matching the 3,584 dimension), yielding a vocabulary near 180,000, and trains on three autoregressive objectives: audio-only, text-only, and interleaved text-audio; a quality-score data filter and Linear Preference Optimization alignment complete the pipeline.

What would settle it

Measure reconstruction quality (PESQ, STOI, or a listening test) of the teacher codec and the distilled student on identical held-out speech: if the student's scores fall materially below the teacher's, the DMS inheritance is not preserving full audio information. A quick check is the paper's own Table 3, where DistilCodec scores 2.02 PESQ against the unified baselines X-codec2 (2.43) and BiCodec (2.51); confirming that gap on a matched teacher-student pair would settle the premise.

Watch

Extended reading notes

Core claim

The paper's central claim is that semantic-acoustic decoupling in neural audio codecs limits LLM-based TTS because semantic and acoustic information cannot be fully aligned, so the LLM never sees the complete audio. DistilCodec removes the decoupling by distilling an 8-residual, 4-group teacher codec into a student with one codebook of 32,768 codes and a dimension of 3,584, inheriting the teacher's encoder and decoder. With near-100% codebook utilization, the single stream is said to model prosody, timbre, emotion, and non-linguistic vocalizations directly, and it lets UniTTS pretrain on unlabeled audiobooks, speech, music, and sound effects. On the paper's evaluations, UniTTS-LPO reaches 4.60 in emotional expressiveness, above the comparison systems' 4.31, and 4.94 in naturalness, which the authors attribute to the holistic modeling of DistilCodec.

Load-bearing premise

The claim rests on the assumption that inheriting the teacher's encoder and decoder lets the single-codebook student preserve the full acoustic information of the multi-codebook teacher; if that transfer fails, the 'no decoupling' advantage is unsupported.

Editorial extensions

If this is right

  • A single 32,768-code stream can suffice for LLM-based TTS, so multi-stage decoding over multiple token streams is unnecessary.
  • Unlabeled universal audio can be folded into pretraining, so TTS data scale is no longer limited by paired text-audio data.
  • Interleaved text and audio prompts let one model handle TTS and potentially speech understanding without a separate semantic codec.
  • Emotional expressiveness such as laughter, crying, and subtle prosody can be generated from full-spectrum audio tokens rather than recovered by a separate acoustic model.
  • Codebook utilization near 100% suggests the distilled codec avoids the mode collapse that often plagues large codebooks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The codebook-dimension alignment (3,584 = Qwen2.5 embedding size) may be what lets UniTTS initialize its audio embedding layer from the codebook; an ablation randomizing that initialization would test whether the alignment is doing real work.
  • The paper observes modal competition in Stage 1 pretraining; a plausible next step it leaves open is an expert-routing architecture that isolates audio and text computation to avoid interference.
  • If the single-codebook stream truly carries complete audio information, the same DistilCodec should enable other audio tasks, such as speech-to-speech translation or audio captioning, by fine-tuning UniTTS's interleaved prompt format; this is testable but not claimed in the paper.
  • DistilCodec's bitrate (1,300 bps) is higher than some semantic-aligned codecs (800 bps for X-codec2); whether the fidelity gain justifies the rate difference in downstream LLM tasks is a comparison the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes two coupled components: DistilCodec, a single-codebook neural audio codec (32,768 codes, dimension 3584) trained by a procedure called DMS that initializes a student encoder/decoder from a multi-codebook TeacherCodec and retrains on 100,000 hours of universal audio, and UniTTS, a Qwen2.5-7B-based LLM that consumes DistilCodec tokens and is trained in three stages (pretraining, SFT, alignment via LPO). The authors claim that DistilCodec preserves comprehensive acoustic and semantic information in a single token stream, enabling end-to-end TTS without semantic/acoustic decoupling, and that UniTTS achieves state-of-the-art emotion-aware speech synthesis while also supporting interleaved text-audio prompts and substantially retaining the base LLM's text capabilities. The claims are supported by codebook perplexity/utilization measurements, objective codec metrics, MOS evaluations, TTS MOS comparisons, and CER ablations.

Significance. If the central premise holds, the contribution is substantial: a low-bitrate, single-stream tokenizer that carries both semantic and acoustic information would simplify LLM-based TTS pipelines and allow unlabeled universal audio to be used directly in pretraining, reducing dependence on paired text-audio data. The paper is transparent in several respects: it releases code and checkpoints, uses a large and diverse 100,000-hour audio corpus, reports negative results such as elevated pretraining loss and text-capability degradation, and includes careful ablation experiments on prompt format and instruction data. The universal-audio pretraining objective and the three-stage training recipe are reasonable and potentially reusable by the community. However, the load-bearing empirical premise of DMS — that parameter inheritance from a multi-codebook teacher preserves acoustic capacity in a single-codebook student — is not verified with a teacher-vs-student comparison, and the headline MOS claims lack statistical grounding. The significance of the contribution is therefore real but not yet established to the standard required by the claims.

major comments (4)
  1. [Algorithm 1; Section 4.2, Table 3] The central claim that DMS 'distills' a multi-codebook NAC into a single-codebook NAC while preserving full audio information is not supported by the reported evidence. Algorithm 1 copies only the teacher's encoder and decoder parameters into the student; the quantizer and factorized dense layers are trained from scratch (Table 9), and Step 4 uses the same DLF objective (Eq. 1) with no teacher-feature or teacher-output matching loss. No reconstruction metrics for TeacherCodec are reported anywhere. In Table 3, DistilCodec achieves PESQ 2.02 and UTMOS 3.75 at 1300 bps, below X-codec2 (2.43/4.13 at 800 bps), StableCodec (2.24/4.23 at 697 bps), and BiCodec (2.51/4.18 at 650 bps). Without a teacher-vs-student control, the reader cannot tell whether the student preserved the teacher's capacity or whether the teacher was already weaker. Please add TeacherCodec reconstruction metrics on the same benchmark and an explicit information-preservation check, such as teacher-feature matching or a downstream task comparison between teacher tokens and student tokens.
  2. [Section 4.3.2, Table 5] The 'state-of-the-art emotion-aware speech synthesis' claim rests on MOS scores reported without confidence intervals, rater counts, or significance tests. Table 5 lists UniTTS-LPO's emotional expressiveness as 4.60 versus 4.31 for IndexTTS and 4.29 for Fish-Speech, but with no uncertainty quantification the 0.29-point gap cannot be distinguished from rating noise. The same applies to the universal-audio MOS in Table 4. Please report the number of raters and clips, per-system means with 95% confidence intervals, and paired significance tests for the main comparisons, or explicitly present the table as a pilot evaluation.
  3. [Section 3.3.2, Eq. (4); Section 4.3.3, Table 6] The SFT data filter selects samples by quality(i) = DNSMOS(i) - CER(i) (Eq. 4 and Algorithm 3), and the ablation study supporting the claim that text-instruction data improves audio generation quality is evaluated primarily by CER (Table 6). Because the same core metric used for data selection is also used for evaluation, the measured improvement may partly reflect selection on the evaluation criterion rather than general audio quality. Please add held-out metrics orthogonal to the filter, such as WER with a different ASR, UTMOS, or human MOS on the ablation conditions, to break this partial circularity.
  4. [Appendix B.9, Table 21] The abstract's claim that UniTTS 'substantially preserves' Qwen2.5-7B's text capabilities is difficult to reconcile with the reported numbers: after stage-2 pretraining, MMLU drops from 74.26 to 52.44 and HumanEval from 77.44 to 14.63, with ARC-C and Winogrande also degraded. The authors acknowledge modal competition and missing code/math data, but the magnitude of the drop is large. Please either qualify the claim in the abstract and introduction or provide a text-only evaluation with confidence intervals that more directly demonstrates the intended degree of preservation.
minor comments (6)
  1. [Algorithm 1, Step 3] Step 3 initializes the student codebook as 35,768 rather than 32,768; please correct the typo.
  2. [Table 4] The GT average score is shown as '4.4.936'; this should be 4.936.
  3. [Table 12] The Multi-Scale Discriminator parameters duplicate the Multi-Period Discriminator's periods and kernel settings; a multi-scale discriminator should operate on different audio scales, not periods. Please correct or clarify the configuration.
  4. [Section 4.3.2] The sentence declaring 'state-of-the-art emotion-aware speech synthesis' does not note that UniTTS-LPO's Fidelity (4.80) ties with CosyVoice2 and is below F5-TTS (4.94); please qualify the claim to the specific dimensions and systems compared.
  5. [Table 6] The CER values are reported to four decimal places with no variance or significance test; please add bootstrap confidence intervals or per-sample standard deviations.
  6. [Appendix B.6] The evaluation criteria mention 'human-like speech/instrument production', but the systems evaluated are TTS models; please adapt the wording to speech synthesis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are benchmarked externally and are not definitionally tied to their inputs.

full rationale

Walking the paper's claimed derivation chain, I find no step where a prediction is equivalent by construction to its input or where a load-bearing premise rests on a self-citation. DistilCodec's reconstruction quality is evaluated against external codecs (Table 3) on LibriSpeech-Clean-Test using external metrics (PESQ, UTMOS, STOI), and its codebook utilization is measured on held-out test sets. The DMS parameter-inheritance scheme (Algorithm 1) is a training-initialization recipe, not a fitted quantity renamed as a prediction; whether it genuinely preserves teacher capacity is a validity question (the missing teacher baseline is a real experimental gap), not a circularity. UniTTS is compared against external TTS systems in Table 5, and its ablations vary prompt templates, instruction data, and pretraining stages. The SFT data filter (Algorithm 3) uses quality = DNSMOS - CER to rank training data, and CER is later reported, but the evaluation CER is computed on generated speech against reference text at inference time; the filter does not directly minimize that test metric, and the model is assessed on held-out prompts. The Spark-TTS citation [32] shares an author name with this paper but is used only as an external comparison baseline, not as evidence for a premise, so it is not load-bearing. The paper's own appended limitation statements (B.8, B.9) acknowledge insufficient pretraining and degraded text benchmarks, but acknowledging limitations or missing controls is not circular reasoning. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 3 invented entities

The central system depends on several hand-chosen capacities and data ratios. The codebook size, codebook dimension, teacher configuration, pretraining data mix, and SFT filter threshold are all selected by the authors. The DMS parameter-inheritance assumption and the sufficiency of a single 32,768-code codebook are asserted rather than derived. No new physical entities are introduced; DistilCodec and UniTTS are engineered systems with external benchmark handles.

free parameters (5)
  • codebook_size = 32768
    Chosen by hand; balances code capacity against sequence length. It is much larger than WavTokenizer's 4096 and smaller than X-codec2's 65536.
  • codebook_dimension = 3584
    Aligned to the Qwen2.5-7B embedding dimension so the codebook can initialize the audio embedding layer.
  • teacher_nac_configuration = 8 residual x 4 groups, 1024 codes, 512 dim
    Chosen as the multi-codebook teacher; no comparison against other teacher configurations is shown.
  • pretraining_data_mix = 322B tokens: 140B text, 82B aligned, 100B audio
    Hand-selected mixture; the paper does not ablate the ratio or test alternative mixes.
  • vad_threshold = 0.14
    Samples with VAD proportion above 0.14 are removed in SFT data filtering (Algorithm 3); no sensitivity analysis is provided.
assumptions (4)
  • standard math Bayes decomposition p(A,T) = p(A|T)p(T) = p(T|A)p(A) licenses the three pretraining tasks (audio AR, text AR, cross-modal conditional).
    Mathematically true, but it only justifies factorization, not that these tasks are sufficient or that a single LLM can optimize all three without modal competition (Section 3.3.1, Eq. 3).
  • domain assumption A single codebook with 32,768 codes can represent the full acoustic-semantic information needed for TTS.
    Core design assumption; evaluated only indirectly through reconstruction metrics and downstream MOS (Sections 3.2, 4.2).
  • ad hoc to paper Inheriting teacher encoder/decoder parameters into a student with a different quantizer preserves or improves reconstruction quality.
    Unique to the DMS recipe; no theoretical justification is given, and Table 3 shows DistilCodec below several baselines on PESQ and UTMOS (Algorithm 1).
  • domain assumption Universal audio autoregression improves later text-audio alignment.
    Claimed from the AB_SFT 18.18% versus UniTTS-SFT 3.43% CER comparison, which confounds data filtering, dataset scale, and prompt templates (Appendix B.10, Table 22).
invented entities (3)
  • DistilCodec independent evidence
    purpose: Single-codebook neural audio codec with 32,768 codes used to tokenize universal audio for the LLM.
    Evaluated against external codecs on LibriSpeech-Clean-Test (Table 3), and a public repository is claimed, giving a falsifiable quality handle.
  • UniTTS independent evidence
    purpose: Qwen2.5-7B-based TTS system that models audio, text, and cross-modal autoregression on interleaved prompts.
    Public checkpoints are claimed, and MOS/CER evaluations could be rerun, though no third-party benchmark or error bars are reported.
  • DMS (distilling multi-codebook NAC to single-codebook NAC)
    purpose: Training recipe that initializes student encoder/decoder from the teacher and retrains only the quantizer.
    No independent validation outside this paper; the only evidence is the paper's own DistilCodec evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information." pith.science (2026). https://pith.science/paper/AKSPIIEM

@misc{pith2026250517426,
  author       = {Pith},
  title        = {Pith review of: UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKSPIIEM}},
  note         = {Machine review of arXiv:2505.17426}
}
read the original abstract

The emergence of multi-codebook neutral audio codecs such as Residual Vector Quantization (RVQ) and Group Vector Quantization (GVQ) has significantly advanced Large-Language-Model (LLM) based Text-to-Speech (TTS) systems. These codecs are crucial in separating semantic and acoustic information while efficiently harnessing semantic priors. However, since semantic and acoustic information cannot be fully aligned, a significant drawback of these methods when applied to LLM-based TTS is that large language models may have limited access to comprehensive audio information. To address this limitation, we propose DistilCodec and UniTTS, which collectively offer the following advantages: 1) This method can distill a multi-codebook audio codec into a single-codebook audio codec with 32,768 codes while achieving a near 100\% utilization. 2) As DistilCodec does not employ a semantic alignment scheme, a large amount of high-quality unlabeled audio (such as audiobooks with sound effects, songs, etc.) can be incorporated during training, further expanding data diversity and broadening its applicability. 3) Leveraging the comprehensive audio information modeling of DistilCodec, we integrated three key tasks into UniTTS's pre-training framework: audio modality autoregression, text modality autoregression, and speech-text cross-modal autoregression. This allows UniTTS to accept interleaved text and speech/audio prompts while substantially preserving LLM's text capabilities. 4) UniTTS employs a three-stage training process: Pre-Training, Supervised Fine-Tuning (SFT), and Alignment. Source code and model checkpoints are publicly available at https://github.com/IDEA-Emdoor-Lab/UniTTS and https://github.com/IDEA-Emdoor-Lab/DistilCodec.

Figures

Figures reproduced from arXiv: 2505.17426 by the authors.

Figure 1
Figure 1. The UniTTS architecture consists of an ALM tokeniz [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The detailed network architecture of DistilCodec [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Training Diagram of DisitilCodec. Fig.3 illustrates the comprehensive training diagram of DisitilCodec. The configuration of optimizer parameters during the training of DistilCodec can be found in [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Training schema of UniTTS and DistilCodec. Distil [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Inference Prompt Template Algorithm 3 The data filtering algorithm based on quality scores Require: Text-Audio Alignment Dataset D(x, y) with N samples Ensure: scores 1: Initialize empty list scores = [] 2: for i ← 1 to N do 3: xi , yi = D(i) 4: dnsmos(i) = DNSMOS P.83…
Figure 6
Figure 6. Figure 6: Prompt configurations 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Stage 1 Pre-training Loss 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 19 canonical work pages

  1. [1]

    Seed-tts: A fami ly of high-quality versatile speech generation models

    Philip Anastassiou, Jiawei Chen, Jitong Chen, Y uanzhe C hen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A fami ly of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430 , 2024

    arXiv 2024

  2. [2]

    wav2vec 2.0: A framework for self-supervised learning of speech represen tations

    Alexei Baevski, Y uhao Zhou, Abdelrahman Mohamed, and Mi chael Auli. wav2vec 2.0: A framework for self-supervised learning of speech represen tations. Advances in neural informa- tion processing systems, 33:12449–12460, 2020

    work page 2020

  3. [3]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching

    Y ushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui W ang, Jian Zhao, Kai Y u, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024

    arXiv 2024

  4. [4]

    High fidelity neural audio compression

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Y ossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438 , 2022

    arXiv 2022

  5. [5]

    Moshi: a speech-text fou ndation model for real-time dialogue

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Améli e Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text fou ndation model for real-time dialogue. arXiv preprint arXiv:2410.00037 , 2024

    arXiv

  6. [6]

    Indextts: An industrial-level controllable and efficient zero-shot text-to-speech syste m

    Wei Deng, Siyi Zhou, Jingchen Shu, Jinchao Wang, and Lu Wa ng. Indextts: An industrial-level controllable and efficient zero-shot text-to-speech syste m. arXiv preprint arXiv:2502.05512 , 2025

    arXiv 2025

  7. [7]

    Cosyvoice: A scalable multil ingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Y ex in Y ang, Hangrui Hu, Siqi Zheng, Y ue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multil ingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407 , 2024

    arXiv 2024

  8. [8]

    Cosyvoice 2: Scalable s treaming speech synthesis with large language models

    Zhihao Du, Y uxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tia nyu Zhao, Zhifu Gao, Y exin Y ang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable s treaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117 , 2024

    arXiv 2024

Show all 47 references
  1. [9]

    Paraformer: Fast and accu- rate parallel transformer for non-autoregressive end-to- end speech recognition

    Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Y a n. Paraformer: Fast and accu- rate parallel transformer for non-autoregressive end-to- end speech recognition. arXiv preprint arXiv:2206.08317, 2022

    2022 arXiv

  2. [10]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten , Alex V aughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  3. [11]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yi cheng Gu, Hua Hua, Liwei Liu, Chen Y ang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language T echnology W ork- sho...

    2024

  4. [12]

    Hubert: Self-supervised spe ech representation learning by masked prediction of hidden units, 2021

    Wei-Ning Hsu, Benjamin Bolte, Y ao-Hung Hubert Tsai, Ku shal Lakhotia, Ruslan Salakhutdi- nov, and Abdelrahman Mohamed. Hubert: Self-supervised spe ech representation learning by masked prediction of hidden units, 2021

    2021

  5. [13]

    Wavtokenizer: an effi cient acoustic discrete codec tokenizer for audio language modeling

    Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Y ang, Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: an effi cient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532 , 2024

    2024 arXiv

  6. [14]

    Libriheavy: A 50,000 hours asr corpus with punctuation casing and con- text

    Wei Kang, Xiaoyu Y ang, Zengwei Y ao, Fangjun Kuang, Yifa n Y ang, Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: A 50,000 hours asr corpus with punctuation casing and con- text. In ICASSP 2024-2024 IEEE International Conference on Acousti cs, Speech and Signal Processing...

    2024

  7. [15]

    Scal ing laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown , Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scal ing laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  8. [16]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022–17033, 2020. 10

    2020

  9. [17]

    Single-codec: Single-codebook speech c odec towards high-performance speech generation

    Hanzhao Li, Liumeng Xue, Haohan Guo, Xinfa Zhu, Y uanjun Lv, Lei Xie, Y unlin Chen, Hao Yin, and Zhifei Li. Single-codec: Single-codebook speech c odec towards high-performance speech generation. arXiv preprint arXiv:2406.07422 , 2024

    2024 arXiv

  10. [18]

    Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthesis

    Shijia Liao, Y uxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, and Yijin Xing. Fish-speech: Leveraging large language models for advanced multilingual text-to-speech synthesis. arXiv preprint arXiv:2411.01156 , 2024

    2024 arXiv

  11. [19]

    Unitok: A unified tokenizer for visual generati on and understanding

    Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Y ang, Xin Y u, Zeh uan Y uan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generati on and understanding. arXiv preprint arXiv:2502.20321, 2025

    2025

  12. [20]

    Wenetspeech4tts: A 12, 800-hour mandarin tts corpus for large speech generation model benchmark

    Linhan Ma, Dake Guo, Kun Song, Y uepeng Jiang, Shuai Wang , Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, and Lei Xie. Wenetspeech4tts: A 12, 800-hour mandarin tts corpus for large speech generation model benchmark. arXiv preprint arXiv:2406.05763 , 2024

    2024 arXiv

  13. [21]

    Autoregressive speech synthesis without vector quanti- zation

    Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Y anqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al. Autoregressive speech synthesis without vector quanti- zation. arXiv preprint arXiv:2407.08551 , 2024

    2024 arXiv

  14. [22]

    Finite scalar quantization: Vq-vae made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and M ichael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505 , 2023

    2023 arXiv

  15. [23]

    Scaling transformers for low-bitrate high-quali ty speech coding

    Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zac k Zukowski, Zach Evans, and Xubo Liu. Scaling transformers for low-bitrate high-quali ty speech coding. arXiv preprint arXiv:2411.19842, 2024

    2024 arXiv

  16. [24]

    Loss-sensitive generative adversarial ne tworks on lipschitz densities, 2018

    Guo-Jun Qi. Loss-sensitive generative adversarial ne tworks on lipschitz densities, 2018

    2018

  17. [25]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Chri stine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International con- ference on machine learning , pages 28492–28518. PMLR, 2023

    2023

  18. [26]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dari o Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  19. [27]

    Direct preference optimization: Y our language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christ opher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Y our language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728–53741, 2023

    2023

  20. [28]

    Dnsmos p

    Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p. 835: A non-intrusive percep- tual objective speech quality metric to evaluate noise supp ressors. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 886–890. IEEE, 2022

    2022

  21. [29]

    Utmos: Utokyo-sarulab system for voicem os challenge 2022, 2022

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriya ma, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicem os challenge 2022, 2022

    2022

  22. [30]

    Neural discret e representation learning

    Aaron V an Den Oord, Oriol Vinyals, et al. Neural discret e representation learning. Advances in neural information processing systems , 30, 2017

    2017

  23. [31]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Y u Wu, Ziqiang Zhang, Long Z hou, Shujie Liu, Zhuo Chen, Y anqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111 , 2023

    2023 arXiv

  24. [32]

    Spark-tts: An efficient llm-based text-to- speech model with single-stream decoupled speech tokens

    Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Son gxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, et al. Spark-tts: An efficient llm-based text-to- speech model with single-stream decoupled speech tokens. arXiv preprint arXiv:2503.01710 , 2025

    2025 arXiv

  25. [33]

    Convnext v2: Co-designing and scaling conv nets with masked autoencoders, 2023

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Ch en, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling conv nets with masked autoencoders, 2023

    2023

  26. [34]

    Bigcodec: Pushing the limits of low-bitrate neural speech codec

    Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Sa ruwatari. Bigcodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377 , 2024

    2024 arXiv

  27. [35]

    An Y ang, Baosong Y ang, Beichen Zhang, Binyuan Hui, Bo Zh eng, Bowen Y u, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 techn ical report. arXiv preprint arXiv:2412.15115, 2024. 11

    2024 arXiv

  28. [36]

    Hifi-codec: Group-residual vector quantization for high fid elity audio codec

    Dongchao Y ang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Y uexian Zou. Hifi-codec: Group-residual vector quantization for high fid elity audio codec. arXiv preprint arXiv:2305.02765, 2023

    2023 arXiv

  29. [37]

    Codec does matter: Explo ring the semantic shortcoming of codec for audio language model

    Zhen Y e, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zhe qi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Explo ring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference on Artificial Intelligen...

    2025

  30. [38]

    Llasa: Scaling train-time a nd inference-time compute for llama-based speech synthesis

    Zhen Y e, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan, Jiahe Lei, Yi Peng, Haohe Liu, Yizhu Jin, Zheqi DAI, et al. Llasa: Scaling train-time a nd inference-time compute for llama-based speech synthesis. arXiv preprint arXiv:2502.04128 , 2025

    2025 arXiv

  31. [39]

    V ector-quan tized image modeling with improved vqgan

    Jiahui Y u, Xin Li, Jing Y u Koh, Han Zhang, Ruoming Pang, J ames Qin, Alexander Ku, Y uanzhong Xu, Jason Baldridge, and Y onghui Wu. V ector-quan tized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627 , 2021

    arXiv

  32. [40]

    Soundstream: An end-to-end neural audio codec

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skog lund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021

    2021

  33. [41]

    Speechtokenizer: Unified speech tokenizer for speech large language models

    Xin Zhang, Dong Zhang, Shimin Li, Y aqian Zhou, and Xipen g Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692 , 2023

    2023 arXiv

  34. [42]

    Scaling th e codebook size of vqgan to 100,000 with a utilization rate of 99%

    Lei Zhu, Fangyun Wei, Y anye Lu, and Dong Chen. Scaling th e codebook size of vqgan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406.11837 , 2024

    2024 arXiv

  35. [43]

    Autoregressive spe ech synthesis with next-distribution prediction

    Xinfa Zhu, Wenjie Tian, and Lei Xie. Autoregressive spe ech synthesis with next-distribution prediction. arXiv preprint arXiv:2412.16846 , 2024. A DistilCodec A.1 Model Structure of DistilCodec Figure 2: The detailed network architecture of DistilCodec . The detailed network a...

    2024

  36. [44]

    Fidelity: The audio accurately reproduces the original s ound characteristics, including timbre and pitch alignment with ground truth recordings

  37. [45]

    Stability: The audio playback exhibits no artifacts such as stuttering, frame skipping, or abrupt termination

  38. [46]

    Naturalness: The output demonstrates human-like speech /instrument production without robotic artifacts or unnatural prosody

  39. [47]

    16 Table 13: Multi STFT Discriminator parameter settings of Di stilCodec

    Emotional expressiveness: The audio effectively convey s intended emotional states (e.g., joy, sadness, anger) with appropriate vocal/instrumental cues . 16 Table 13: Multi STFT Discriminator parameter settings of Di stilCodec. Configuration Item V alue Number of STFT Discrimin...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.