Pith. sign in

REVIEW 4 major objections 5 minor 9 cited by

XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A low-bitrate speech codec can preserve both words and voice if the semantic and acoustic tasks stop sharing parameters.

desk verdict A plausible dual-tower codec whose semantic gains are confounded by the frozen Whisper encoder; the ablations never isolate the proposed training from the ASR backbone. read the letter →

arxiv 2506.23325 v2 pith:VJPQSGKQ submitted 2025-06-29 cs.SD cs.AIeess.AS

Yitian Gong , Luozhijie Jin , Ruifan Deng , Dong Zhang , Xin Zhang , Qinyuan Cheng , Zhaoye Fei , Shimin Li
show 1 more author
Xipeng Qiu
This is my paper
classification cs.SDcs.AIeess.AS
keywords speechcodeclow-bitrateaudiocompressionsemantictokensacousticresidualvectorquantizationLLM-basedASRlanguagemodelsdual-towerarchitecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that at roughly 1 kbps a speech codec can preserve both the linguistic content and the speaker's voice in one token stream, and that the usual semantic/acoustic tradeoff is caused by parameter sharing rather than by the bitrate itself. The proposed XY-Tokenizer separates semantic modeling from acoustic modeling, sharing only the quantizer, and trains in two stages: first with an LLM-based ASR objective plus reconstruction, then with GAN discriminators for fine acoustic detail. On the paper's evaluations, it reports an ASR probing word error rate of 0.13, better than distillation-based SpeechTokenizer at 0.20 and Mimi at 0.28, while speaker similarity reaches 0.83, close to the acoustic-only BigCodec's 0.84 at a similar bitrate. If these numbers hold, a speech LLM could use one token stream for both understanding and high-quality audio output.

What carries the argument

The central mechanism is the dual-tower encoder with an X-shaped pretraining stage and a Y-shaped post-training stage. In pretraining, a frozen semantic tower initialized from Whisper and a trainable acoustic tower are concatenated into a shared 8-level residual vector quantizer at 12.5 Hz, supervised jointly by an LLM ASR cross-entropy loss and a multi-scale mel-spectrogram reconstruction loss; post-training removes the semantic decoder and adds multi-period, multi-scale, and multi-scale-STFT discriminators. The machinery works by giving the semantic task its own frozen pathway so its gradients do not fight the reconstruction task, while the shared RVQ bottleneck forces both kinds of information into one compact token sequence.

What would settle it

Retrain XY-Tokenizer with the semantic tower frozen at random weights instead of Whisper weights; if the ASR probing WER stays near the reported 0.13, the multi-task training is doing the semantic work, while a large jump toward 0.5 would show the Whisper initialization is the source.

Watch

Extended reading notes

Core claim

The central claim is that a codec can resolve the semantic-acoustic conflict by making semantic modeling an explicit text-prediction task and keeping that task off the acoustic reconstruction path. XY-Tokenizer reports ASR probing WER of 0.13, surpassing SpeechTokenizer and Mimi, and speaker similarity of 0.83, comparable to BigCodec's 0.84 at 1.04 kbps. The authors attribute this to a frozen Whisper-initialized semantic encoder, an LLM-based ASR supervision loss instead of SSL-teacher distillation, and shared parameters confined to the RVQ module. If correct, the result means low bitrate does not force a choice between words and voice.

Load-bearing premise

The result depends on the ASR probing score measuring what the codec tokens preserve; if the frozen Whisper encoder itself is doing the recognition, the semantic score does not demonstrate the training paradigm.

Editorial extensions

If this is right

  • Speech LLMs could drop the separate semantic codec and drive both understanding and synthesis from one XY-Tokenizer stream at 1 kbps.
  • LLM-based ASR supervision outperforms SSL-teacher distillation for semantic codec training, per the paper's probing results.
  • Because the post-training GAN stage freezes the encoder and quantizer, adversarial refinement can improve perceptual quality without shifting representations away from text.
  • Reconstruction at speaker similarity 0.83 is close enough to BigCodec's 0.84 that low-bitrate codecs no longer need to sacrifice semantics for acoustic quality.
  • Ablations indicate shared parameters, not bitrate alone, drive the semantic/acoustic tradeoff, making parameter sharing a design axis for future codecs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: the semantic gain looks substantially inherited from Whisper; the paper's own ablation shows removing Whisper weights raises WER from 0.13 to 0.27, so the headline number is not purely learned by the multi-task paradigm.
  • Extension: the WER comparison might shift under a standardized SUPERB probe, since the paper upsamples low-frame-rate tokens and uses a two-layer LSTM on train-clean-100 rather than the default SUPERB recipe.
  • Extension: end-to-end speech-LLM tests, such as spoken QA or TTS from the tokens, would show whether the semantic and acoustic qualities survive when the tokens are consumed by a language model, which the paper does not evaluate.
  • Extension: varying how many RVQ layers are shared between the two towers could trade speaker similarity against WER, giving a controllable knob the paper leaves unexamined.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes XY-Tokenizer, a 1 kbps speech codec intended to preserve both semantic text content and acoustic speaker information. The architecture uses a two-channel encoder with a frozen Whisper-initialized semantic branch and a trainable Whisper-initialized acoustic branch, a shared 8-layer RVQ at 12.5 Hz, and a two-stage training procedure: a pre-training stage combining an LLM-based ASR loss with a multi-scale mel-spectrogram reconstruction loss, and a post-training GAN stage that fine-tunes the decoder with discriminators. The main empirical claim is that at ~1 kbps the system achieves ASR-probing WER of 0.13 and speaker similarity 0.83, thereby matching or exceeding semantic/audio codecs such as SpeechTokenizer, Mimi, and BigCodec. The paper also reports ablations on shared parameters, LLM trainability, Whisper initialization, and ASR supervision, and provides a GitHub link for code and models.

Significance. If the core comparative claims are reliable, the paper addresses a practical and timely problem: low-bitrate codecs typically trade off semantic alignability against acoustic fidelity, and a codec that preserves both at 1 kbps would be directly useful for speech language models. The paper's strengths include the scale of training (101k hours of Emilia), the clarity of the proposed two-stage training scheme, and the open release of code and models. The ablation studies also demonstrate a useful empirical point, namely that separating the semantic and acoustic pathways improves reconstruction metrics. However, the central quantitative claims rest on a non-standard ASR probing protocol and on a semantic branch that is initialized from a supervised ASR model (Whisper-small), so the headline WER and the conclusion that the proposed multi-task paradigm resolves the semantic-acoustic conflict need substantially stronger evidence before the contribution is established.

major comments (4)
  1. [Section 4.2 and Appendix D] The ASR probing protocol is non-standard and confounds the cross-codec WER comparison. The paper upsamples all quantized embeddings to a minimum of 50 Hz via replication and trains a two-layer LSTM CTC probe on LibriSpeech train-clean-100, evaluated on dev-clean. Because the codecs in Table 3 have native frame rates of 12.5, 50, 75, and 80 Hz, this upsampling changes the input sequence length non-uniformly, and the paper does not validate that the probe is comparable to the standard SUPERB setup. The authors should validate the probe against the standard SUPERB protocol and report results at native frame rates or at a common upsampled rate for all codecs, so that the reported WER ranking is not an artifact of the probing convention.
  2. [Section 3.1, Table 4, and Appendix F] The contribution of the frozen Whisper semantic branch is not isolated, so the semantic-alignment claim is partly circular. The full model concatenates a frozen Whisper-small semantic encoder with a trainable Whisper-initialized acoustic encoder. Table 4 shows that a single-channel Whisper-initialized trainable encoder already reaches the same probing WER of 0.13; Table 6 shows that removing Whisper initialization raises WER to 0.27; Table 7 shows that removing the LLM-based ASR supervision raises WER to 0.58. These ablations are consistent with the alternative explanation that the low WER is largely supplied by the Whisper initialization and the ASR loss, rather than by the dual-tower conflict-mitigation design. A controlled ablation that removes or replaces only the frozen semantic branch while keeping the two-channel structure and shared RVQ fixed is needed to separate the effects.
  3. [Section 4.3 and Table 3] The claim that XY-Tokenizer's reconstruction performance is comparable to BigCodec's is not supported across all reported metrics. While speaker similarity is close (0.83 vs 0.84), XY-Tokenizer is lower on PESQ-NB (3.00 vs 3.26), PESQ-WB (2.41 vs 2.68), and STOI (0.91 vs 0.93). No error bars, confidence intervals, or statistical significance tests are reported for any metric in Table 3, and several comparisons involve small differences (e.g., WER 0.13 vs Baichuan's 0.10). The authors should provide variance estimates and significance tests, especially for the headline comparisons that are central to the abstract and conclusion.
  4. [Section 3.1 and Table 2] The preliminary analysis of shared parameters is not a controlled experiment. The models in Table 2 (SpeechTokenizer-x1/x2/x3, Mimi-8, XCodec2.0) differ in architecture, training data, bitrate, and distillation weights, so the observed trend in SIM and WER cannot be attributed to the number of shared parameters alone. This weakens the stated motivation for the dual-tower design, and the authors should either provide a controlled comparison that varies only the amount of parameter sharing or soften the causal claim.
minor comments (5)
  1. [Section 4.2] The text states that all evaluations were conducted on the LibriSpeech test-clean subset, but the ASR probe is evaluated on LibriSpeech dev-clean; this inconsistency should be corrected.
  2. [Appendix G] The limitations section does not mention the potential circularity of using a frozen Whisper encoder in the semantic branch or the non-standard ASR probing protocol; both are the main threats to the central claims and should be acknowledged.
  3. [Table 5] The column 'LLM WER' is not clearly defined; the caption should state whether it is the WER of the LLM-decoded transcript on the same dev set used for the probing task and under what decoding settings.
  4. [Figure 2] The color-coding for 'frozen,' 'trainable,' and 'Whisper-initialized' components is important for understanding the architecture; a grayscale-safe legend or pattern distinction would improve readability.
  5. [Section 4.3] The paper reports no measure of variability (e.g., number of utterances, confidence intervals) for any metric in Table 3; at minimum, the evaluation set size and a note on statistical stability should be included.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the WER probe is an external downstream task, and the ablations separate the frozen-Whisper prior from the training signal.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs by construction. The semantic alignment claim is measured by an independently trained LSTM-CTC probing model on LibriSpeech, not by the codec's own Whisper-initialized semantic decoder; this probe is external to the codec and is applied identically to all baselines. The frozen Whisper encoder is an architectural component, but the paper explicitly ablates its contribution: Appendix F.1 shows WER degrades from 0.13 to 0.27 when Whisper weights are removed, and Appendix F.2 shows WER degrades from 0.13 to 0.58 when the LLM-based ASR supervision is removed. These ablations demonstrate that the training objective and the pretrained initialization are separable, so the reported WER is not simply the frozen Whisper encoder's own output restated as a prediction. The overlapping-author references (SpeechTokenizer, SpeechGPT, etc.) are used as baselines and background, not as load-bearing evidence for the effectiveness of XY-Tokenizer. The concern that some semantic alignment is inherited from Whisper is a legitimate attribution and novelty concern, but it is not a definitional circularity: the probe does not decode with Whisper, and the final quantized embeddings still require the adapter and RVQ to preserve the alignment. Therefore no step in the paper's argument is equivalent to its own input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rely on a handful of hand-chosen hyperparameters and several domain assumptions about the suitability of Whisper features, the ASR probing protocol, and the mel-spectrogram front-end. The paper introduces no new physical or mathematical entities; its contribution is architectural and experimental. The most fragile dependency is the frozen Whisper encoder, which appears to carry much of the semantic performance.

free parameters (3)
  • Loss weights lambda_asr, lambda_recon, lambda_commit, lambda_feat, lambda_adv = lambda_asr=20, lambda_recon=15, lambda_commit=1, lambda_feat=1, lambda_adv=1
    Chosen by hand, not derived. The paper does not report sensitivity analysis for these weights, so the stated balance between semantic and acoustic objectives is an empirical choice. Section 4.1.
  • RVQ codebook size and number of layers = 1024 entries, 8 layers, 1 kbps
    The architecture is set to achieve a target bitrate; this is a design choice rather than a fitted parameter, but the 12.5 Hz frame rate and 8-layer depth are chosen manually and directly determine the 1 kbps bitrate.
  • Probing hyperparameters for the ASR probe = 400,000 steps, batch size 4, max lr 1e-4
    The probe is trained once with fixed hyperparameters and no reported variation, so the WER depends on these choices. Reported in Section 4.2.
assumptions (4)
  • domain assumption Whisper's frozen encoder provides robust and sufficient semantic features for low-bitrate speech.
    The whole design depends on the claim that Whisper features retain text information after quantization and after the two-stage training. The paper provides preliminary evidence (Table 1), but that evidence comes from autoencoders without a quantizer, so it does not directly prove the assumption under low-bitrate quantization.
  • domain assumption The ASR probing task with a two-layer LSTM and upsampled features is a faithful measure of semantic alignment for all compared codecs.
    The paper admits it upsamples to a minimum 50 Hz frame rate (Section 4.2 and Appendix D), which is not the standard SUPERB setup. The probe is trained on train-clean-100 only, so the WER ranking could change with a stronger probe or different training data.
  • domain assumption Mel-spectrogram input at 100 Hz combined with RVQ at 12.5 Hz preserves enough information for both semantic and acoustic tasks.
    The choice of mel-spectrogram front-end and 12.5 Hz quantizer frame rate is presented without a comparison to alternative front-ends (e.g., raw waveform, different frame rates). The design assumes these are sufficient for the claimed dual performance.
  • domain assumption Speaker similarity (SIM), STOI, and PESQ on LibriSpeech test-clean are adequate measures of acoustic fidelity.
    These are standard metrics, but the paper does not include listening tests or out-of-domain evaluations. The claimed comparability to BigCodec rests on these particular metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs." pith.science (2026). https://pith.science/paper/VJPQSGKQ

@misc{pith2026250623325,
  author       = {Pith},
  title        = {Pith review of: XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJPQSGKQ}},
  note         = {Machine review of arXiv:2506.23325}
}
read the original abstract

Speech codecs serve as bridges between speech signals and large language models. An ideal codec for speech language models should not only preserve acoustic information but also capture rich semantic information. However, existing speech codecs struggle to balance high-quality audio reconstruction with ease of modeling by language models. In this study, we analyze the limitations of previous codecs in balancing semantic richness and acoustic fidelity. We propose XY-Tokenizer, a novel codec that mitigates the conflict between semantic and acoustic capabilities through multi-stage, multi-task learning. Experimental results demonstrate that XY-Tokenizer achieves performance in both semantic and acoustic tasks comparable to that of state-of-the-art codecs operating at similar bitrates, even though those existing codecs typically excel in only one aspect. Specifically, XY-Tokenizer achieves strong text alignment, surpassing distillation-based semantic modeling methods such as SpeechTokenizer and Mimi, while maintaining a speaker similarity score of 0.83 between reconstructed and original audio. The reconstruction performance of XY-Tokenizer is comparable to that of BigCodec, the current state-of-the-art among acoustic-only codecs, which achieves a speaker similarity score of 0.84 at a similar bitrate. Code and models are available at https://github.com/gyt1145028706/XY-Tokenizer.

Figures

Figures reproduced from arXiv: 2506.23325 by the authors.

Figure 1
Figure 1. Comparison of speech codecs in semantic and acoustic performance. The horizontal axis [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of XY-Tokenizer. The upper half depicts the pre-training stage, aligning XY-Tokenizer with text while preserving coarse acoustic features. The lower half illustrates the post￾training stage, modeling finer-grained acoustic features. Model architecture and training procedure are detailed in Section 3. Model Shared Parameters SIM ↑ WER ↓ SpeechTokenizer-x1 Encoder + Quantizer 0.65 0.34 Mimi-8 Encoder 0.73… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model

    cs.SD 2026-06 unverdicted novelty 7.0 of 10

    FlexiSLM is the first spoken language model supporting dynamic and controllable frame rates on speech input and output, outperforming fixed-rate 7B models at high quality and enabling faster inference at lower rates l...

  2. VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    VITA-QinYu is the first expressive end-to-end spoken language model supporting role-playing and singing alongside conversation, trained on 15.8K hours of data and outperforming prior models on expressiveness and conve...

  3. VERITAS: A Multi-Agent Co-Scientist for Verifiable Image-Derived Hypothesis Testing

    cs.MA 2026-04 unverdicted novelty 7.0 of 10

    VERITAS is a multi-agent system for verifiable hypothesis testing on multimodal clinical MRI datasets that achieves 81.4% verdict accuracy with frontier models and introduces an epistemic evidence labeling framework.

  4. dots.tts Technical Report

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    dots.tts reports SOTA benchmark results on Seed-TTS-Eval and other tests via continuous latent-space autoregressive modeling with three listed innovations and code release.

  5. Reducing Linguistic Hallucination in LM-Based Speech Enhancement via Noise-Invariant Acoustic-Semantic Distillation

    eess.AS 2026-05 unverdicted novelty 6.0 of 10

    L3-SE reduces linguistic hallucination in LM-based speech enhancement by distilling noise-invariant acoustic-semantic representations from noisy inputs to condition an autoregressive decoder-only language model.

  6. Why Your Tokenizer Fails in Information Fusion: A Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization

    eess.AS 2026-04 unverdicted novelty 6.0 of 10

    A timing-aware pre-quantization fusion approach integrates visual cues into audio tokenizers along the temporal axis, maintaining reconstruction quality while outperforming audio-only and prior multimodal baselines on...

  7. VERITAS: A Multi-Agent Co-Scientist for Verifiable Image-Derived Hypothesis Testing

    cs.MA 2026-04 conditional novelty 6.0 of 10

    A four-phase multi-agent co-scientist tests natural-language hypotheses on cardiac and glioma MRI and labels outcomes Supported, Refuted, Underpowered, or Invalid with an executable evidence trail.

  8. Qwen3-TTS Technical Report

    cs.SD 2026-01 unverdicted novelty 6.0 of 10

    Qwen3-TTS delivers state-of-the-art multilingual TTS performance with 3-second voice cloning, description control, and ultra-low-latency streaming via dual tokenizers and a dual-track LM architecture trained on over 5...

  9. AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation

    cs.SD 2025-09 conditional novelty 6.0 of 10

    A four-part benchmark plus a semantic/acoustic token taxonomy for comparing audio codecs, with correlation analysis across ten models.

Reference graph

Works this paper leans on

33 extracted references · 11 canonical work pages · cited by 8 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. 9 XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs arXiv preprint arXiv:2303.08774,

    arXiv

  2. [5]

    Moshi: a speech-text foundation model for real-time dialogue

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037,

    arXiv

  3. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pp. 4171–4186,

    work page 2019

  4. [9]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415,

    arXiv

  5. [10]

    Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, et al

    URL https://arxiv.org/abs/2106.07447. Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, et al. Wavchat: A survey of spoken dialogue models. arXiv preprint arXiv:2411.13577,

    arXiv

  6. [12]

    Baichuan-audio: A unified framework for end-to-end speech interaction

    Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, et al. Baichuan-audio: A unified framework for end-to-end speech interaction. arXiv preprint arXiv:2502.17239,

    arXiv

  7. [14]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

  8. [16]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He

    URL https://arxiv.org/ abs/2212.04356. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pp. 1–16. IEEE,

    arXiv

Show all 33 references
  1. [19]

    A short-time objective intelligibility measure for time-frequency weighted noisy speech

    Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In 2010 IEEE international conference on acoustics, speech and signal processing , pp. 4214–4217. IEEE,

    2010

  2. [20]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin

    URL https://arxiv.org/abs/2204.05409. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30,

    arXiv

  3. [21]

    Neural codec language models are zero-shot text to speech synthesizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111,

    arXiv

  4. [22]

    Towards audio language modeling–an overview

    Haibin Wu, Xuanjun Chen, Yi-Cheng Lin, Kai-wei Chang, Ho-Lam Chung, Alexander H Liu, and Hung-yi Lee. Towards audio language modeling–an overview. arXiv preprint arXiv:2402.13236,

    arXiv

  5. [23]

    Bigcodec: Pushing the limits of low-bitrate neural speech codec

    Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377, 2024a. Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low-bitrat...

    arXiv

  6. [24]

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi

    URL https://arxiv.org/abs/2502.04128. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507,

    arXiv

  7. [25]

    Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612,

    arXiv

  8. [26]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000,

    arXiv

  9. [27]

    Speechgpt-gen: Scaling chain-of-information speech generation

    Dong Zhang, Xin Zhang, Jun Zhan, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechgpt-gen: Scaling chain-of-information speech generation. arXiv preprint arXiv:2401.13527, 2024a. Xin Zhang, Xiang Lyu, Zhihao Du, Qian Chen, Dong Zhang, Hangrui Hu, Chaohong Tan, Tianyu Zhao, Yuxuan...

    arXiv

  10. [28]

    Each adapter consists of a 4-layer Transformer with a hidden dimension of 768, a feed-forward network (FFN) dimension of 3072, and 12 attention heads

    adapter modules at multiple components of the XY-Tokenizer. Each adapter consists of a 4-layer Transformer with a hidden dimension of 768, a feed-forward network (FFN) dimension of 3072, and 12 attention heads. Adapters are placed after the semantic encoder, before and after t...

    2016

  11. [29]

    with a hop size of 160 reconstructs the 16 kHz audio waveform. Discriminators To ensure high perceptual quality, we employ three discriminator models: multi- period discriminator (MPD) (Kong et al., 2020), multi-scale discriminator (MSD) (Kumar et al., 2019), and multi-scale s...

    2020

  12. [30]

    Addi- tionally, we train three variants of SpeechTokenizer using the official codebase, modifying only the RVQ layers and the distillation weight ( distill_loss_lambda). Specifically, we reduce the RVQ layers from 8 to 3 and train three versions: (1) RVQ-3 with distill_loss_la...

    2023

  13. [31]

    achieves high-quality audio generation with coherent long-term structure through coarse-to-fine token modeling. SpeechGPT (Zhang et al., 2023), the first end-to-end speech large language model, features strong instruction-following 5https://huggingface.co/fnlp/SpeechTokenizer/...

    2023

  14. [32]

    foreign language

    employs a multi-stream architecture that concurrently processes audio streams from both the user and the system (Moshi itself), supporting dynamic conversations with overlaps and interruptions, thereby achieving full-duplex dialogue. E.2 Speech Codecs Speech codecs play a vita...

    2021

  15. [33]

    first obtains a coarse Mel-spectrogram through multi-task learning and text alignment, then generates an enhanced Mel-spectrogram via conditional flow matching (Lipman et al., 2022), which is finally converted into waveforms using a pretrained vocoder (Kong et al., 2020). F Ad...

    2022

  16. [2001]

    V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis

    Hubert Siuzdak. V ocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.00814,

    arXiv

  17. [2006]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT...

    2024

  18. [2017]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206–5210. IEEE,

    2015

  19. [2019]

    Kimi-audio technical report

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504.18425,

    arXiv

  20. [2020]

    Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs

    Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processin...

    2001

  21. [2021]

    High fidelity neural audio compression

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438,

    arXiv

  22. [2022]

    doi: 10.1109/jstsp.2022.3188113

    ISSN 1941-0484. doi: 10.1109/jstsp.2022.3188113. URL http://dx.doi.org/10.1109/JSTSP.2022.3188113. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2...

    1941

  23. [2023]

    URL https://arxiv.org/abs/2306. 06546. Siddique Latif, Moazzam Shoukat, Fahad Shamshad, Muhammad Usama, Yi Ren, Heriberto Cuayáhuitl, Wenwu Wang, Xulong Zhang, Roberto Togneri, Erik Cambria, et al. Sparks of large audio models: A survey and outlook. arXiv preprint arXiv:2308.12792,

    arXiv

  24. [2024]

    W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training

    Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)...

    2021

  25. [2025]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.