Pith. sign in

REVIEW 3 major objections 3 minor 26 references

ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure

T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that phoneme structure in the representation entering a speech quantizer predicts how predictable the resulting tokens are for autoregressive models, and that a codec can preserve this structure while adding acoustic…

desk verdict Solid empirical paper with a genuinely useful matched diagnostic; the predictability claim rests on a probe quantizer rather than native tokens, but the TTS results and ablations make it worth serious review. read the letter →

arxiv 2608.08286 v1 pith:XGZQBGTK submitted 2026-08-08 eess.AS cs.SD

Zixiang Wan , Xusheng Yang , Zheng Wang , Peiji Yang This is my paper · ORCID
classification eess.AScs.SD
keywords speechcodecneuralaudiocompressiontokensautoregressivemodelingphonemestructureself-supervisedrepresentationssingle-codebookquantizationtext-to-speech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speech tokens that an autoregressive model can predict well and tokens that reconstruct waveforms well are usually different things. This paper establishes a diagnostic link: across 24 codec and self-supervised speech representations, the clarity of phoneme grouping before quantization strongly tracks next-token prediction accuracy under a matched probing protocol. The paper then builds ReLMCodec, a single-codebook codec that keeps frozen self-supervised features at the quantizer input, adds acoustic detail through a fixed-scale residual path, and uses a training-only teacher to refine post-quantization structure. At 650 and 800 bits per second, ReLMCodec improves both reconstruction and downstream text-to-speech intelligibility and speaker similarity relative to rate-matched baselines. The takeaway is that the design principle, not any single component, is the contribution.

What carries the argument

The load-bearing object is the pre-quantization representation\textemdash{}the continuous frame-level features that enter the quantizer before discrete code assignment\textemdash{}because the paper's whole diagnostic measures phoneme structure there. The matched probing vector quantizer (P-VQ) is the instrument that makes the comparison fair: every representation is quantized with the same 8,192-codeword probe and scored with the same autoregressive language model, yielding probe accuracy (P-ACC) and probe perplexity (P-PPL). PAPA is the architectural mechanism that translates the finding into a codec: it writes the quantizer input as $z_t = s_t + \alpha \Delta_t$ with frozen SSL anchor $s_t$ and $\alpha=0.1$, so the trainable branch can add acoustic detail without rewriting the SSL geometry. The JMAS loss is the refinement mechanism that shapes the quantized latent space toward the teacher's phoneme structure without adding inference-time cost.

What would settle it

Train the same autoregressive language model directly on ReLMCodec's native 8K and 64K token sequences and on X-Codec2's, then compare native next-token accuracy: the design is confirmed only if the native-token advantage at 800 bps resembles the probe advantage (9.65% versus 5.12%), since the paper reports P-ACC on a separately trained probing quantizer, not on the codec's own tokens.

Watch

Extended reading notes

Core claim

The central claim is that pre-quantization phoneme structure is a consistent predictor of autoregressive token predictability, with Spearman correlation 0.911 between KNN phoneme accuracy and probe next-token accuracy across 24 representations, and that this structure can be preserved inside a waveform codec without sacrificing reconstruction. ReLMCodec realizes the claim by composing three mechanisms: a frozen W2v-BERT 2.0 L17 anchor written with coefficient one into the quantizer input, a Pre-quantization Anchor-Preserving Adaptation (PAPA) that adds an acoustic residual at fixed scale $\alpha=0.1$, and a joint\textendash{}marginal alignment loss (JMAS) that pulls quantized latents toward WavLM-Large L24 teacher structure during training only. The result is an empirical frontier move at 650 bps and 800 bps: ReLMCodec@8K and ReLMCodec@64K beat rate-matched single-stream baselines on reconstruction metrics while raising probe next-token accuracy, and the tokens also improve downstream TTS word error rate and speaker similarity.

Load-bearing premise

The load-bearing premise is that the probing quantizer's next-token accuracy (P-ACC) measures the predictability of the codec's actual native tokens in downstream autoregressive speech models; if that proxy does not carry over, the design principle is not supported for real token sequences.

Editorial extensions

If this is right

  • If the correlation holds beyond the 24 evaluated representations, codec designers can screen candidate tokenizers by measuring pre-quantization phoneme KNN accuracy before training a full codec.
  • Single-codebook, single-stream codecs at 650 and 800 bps can achieve both strong reconstruction and predictable tokens, reducing the need for multi-stream residual quantizer interfaces in this regime.
  • Because the WavLM teacher and speaker-verification models are frozen and used only in training, ReLMCodec adds no inference-time computation beyond the SSL encoder, acoustic encoder, quantizer, and decoder.
  • Downstream TTS trained on ReLMCodec tokens shows lower word error rate and higher speaker similarity than rate-matched baselines, indicating the tokens carry usable linguistic and speaker information for generation.
  • The matched probing protocol itself is reusable: future codecs can be compared under identical quantizer capacity and language-model optimization rather than through their native token streams.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports P-ACC on a separately trained probing quantizer and, in Appendix B, states that P-ACC and native-token N-ACC are not directly comparable; a reader should not treat the 9.65% versus 5.12% figure as native-token evidence without checking native-token results.
  • Because the observational claim is limited to 50-Hz English LibriSpeech, the preserve\textendash{}control\textendash{}refine recipe may need re-validation for multilingual, noisy, or other-frame-rate speech before it is treated as a general design law.
  • The layer-dependence of phoneme structure suggests that selecting a teacher by final-layer accuracy alone is not generally safe; the right role assignment may need to be re-derived per encoder and per task.
  • A direct test of the design principle would replace the frozen SSL anchor with a trainable encoder of the same capacity: if P-ACC retention and KNN retention collapse, the anchor itself, rather than the added parameters, carries the predictability benefit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This manuscript introduces ReLMCodec, a single-codebook low-bitrate speech codec designed around the idea that pre-quantization phoneme structure predicts autoregressive token predictability. The authors first build a matched probing setup: 24 frozen codec and SSL representations are standardized, projected into a common 8,192-codeword probing vector quantizer (P-VQ), tokenized, and modeled by a shared Qwen2-1.5B LM; phoneme separability metrics are then correlated with probe accuracy (P-ACC) and perplexity (P-PPL). Based on this diagnosis, they instantiate a preserve-control-refine architecture: a frozen W2v-BERT 2.0 L17 main path at the quantizer input, a Pre-quantization Anchor-Preserving Adaptation (PAPA) residual module that adds acoustic detail while keeping the SSL anchor, and a training-only WavLM-Large L24 teacher via the JMAS loss. Evaluations report improved reconstruction at 650 and 800 bps relative to several released single-stream codecs, and improved downstream EmoVoice TTS intelligibility and speaker similarity for ReLMCodec tokens.

Significance. The paper's strengths are its controlled diagnostic (matched quantizer, LM, and evaluation protocol), bootstrap confidence intervals for the correlation analysis, extensive ablations (SSL-role, module, residual scale, quantizer), an efficiency benchmark, a human MOS study, and a plan to release code, checkpoints, and probing scripts. If the central frontier claim survives validation on native tokens, ReLMCodec would be a useful operating point for low-bitrate single-stream speech LMs, and the matched probing protocol would be a reusable measurement tool. The principal risk is that the predictability axis is measured on a separately trained P-VQ rather than on the codec's native tokens, and the paper does not currently supply the native-token measurements needed to close that gap.

major comments (3)
  1. [End-to-End Reconstruction and Predictability; Appendix B; Table A5] The headline frontier claim (e.g., the 650/800 bps comparison in Table 2 and the sentence in Section 4.1 that the matched P-VQ probe raises P-ACC from 5.12% to 9.65%) pairs reconstruction metrics computed on native tokens with predictability measured on P-VQ tokens. Appendix B explicitly states that P-ACC and N-ACC are not directly comparable, and Table A5 gives N-ACC only for ReLMCodec's two quantizers, with no baseline codec given an N-ACC value. The P-VQ has a feature-reconstruction objective, no waveform decoder, and a different codebook distribution than the native EMA-VQ, so the measured P-ACC gain may be a property of the probe rather than of the token sequences an autoregressive speech LM actually consumes. Please report N-ACC or native-token perplexity for FocalCodec, SemantiCodec, X-Codec2, and the ReLMCodec variants under a shared LM training recipe, and show that the P-ACC ordering tracks the native-token ordering across these systems. Until then, the design choices in Tables 3-4 and the alpha=0.1 selection in Figure 4 rest on a proxy of unvalidated validity.
  2. [Tables 2 and 6] The core empirical claim is a set of point estimates from single checkpoints: no confidence intervals, no multiple seeds, and no significance tests are reported for WER, SIM, PESQ, STOI, or UTMOS. Several margins are small (e.g., SIM 0.749 for both FocalCodec and ReLMCodec@8K; PESQ 2.17 versus 2.19 across alpha in Figure 4), and Table 2 compares released baseline checkpoints with the authors' trained ReLMCodec checkpoints, so training-recipe differences are confounded with architecture differences. Please provide bootstrap or multi-seed intervals for the headline comparisons, and consider matched-training baselines for at least FocalCodec and X-Codec2; at minimum, the baseline checkpoint provenance and training budgets should be stated in the main text.
  3. [Table A5 and Section 4.2] The quantizer ablation uses N-ACC to prefer EMA-VQ over FSQ, but the main design analyses (PAPA alpha, teacher role, module ablations) are all evaluated with P-ACC, and no P-ACC value is reported for the FSQ variant. This leaves the relationship between P-ACC and N-ACC untested even within a single codec architecture. A minimal check is to report P-ACC for both quantizers under the same P-VQ protocol with the same frozen encoder, together with N-ACC under a shared native-token LM; this would directly test whether the proxy is monotonic in native predictability for at least ReLMCodec.
minor comments (3)
  1. [Table 2] The ReLMCodec@64K row renders as '13.96' where the codebook count and WER appear concatenated; please fix the column separation so the row reads as Codebooks=1, WER=3.96.
  2. [Appendix Figure A5] The human MOS evaluation should report the number of listeners, the number of utterances rated, and inter-rater agreement; the current figure shows only rating distributions.
  3. [Equation (3)] Equation (3) uses J_R^T without defining the Jacobian or its dimensions; please define all symbols in the PAPA derivation.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: the diagnostic is transparently a matched P-VQ proxy, and the codec is validated on independent reconstruction and TTS benchmarks.

full rationale

The paper's derivation chain is empirical rather than definitional. The central diagnostic metric P-ACC is deliberately defined on a matched probing quantizer that bypasses native quantizers, and the paper explicitly disclaims comparability with native-token accuracy: Appendix B states that 'since P-VQ is trained for matched feature reconstruction whereas native quantizers use waveform objectives, P-ACC and N-ACC are not directly comparable.' The frontier claim is therefore transparently about P-VQ token sequences and does not secretly rename native-token predictability. There is a mild selection-in-the-loop: WavLM-Large L24 is chosen partly because it has the highest P-ACC among SSL representations, and the JMAS loss then aligns quantized latents toward that teacher, so the subsequent P-ACC gain is influenced by the diagnostic that motivated the design. However, ReLMCodec is also evaluated against released baseline checkpoints on independent reconstruction metrics, human H-MOS, and downstream TTS under a shared EmoVoice setup, so the central claim does not reduce to its own inputs. No load-bearing self-citation chain or imported uniqueness theorem is present; same-author citations (SpecTokenizer, PhoenixCodec) support only routine quantizer or baseline components. The P-ACC-to-N-ACC gap is a genuine validity limitation for claims about native-token predictability, but it is a proxy-validity concern rather than a circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a set of domain assumptions about the validity of phoneme-separability metrics, the P-VQ proxy for predictability, and the transferability of teacher structure. The only hand-fitted scalar that receives an ablation is α; the JMAS margins and loss weights are asserted without sensitivity analysis. No new physical or conceptual entities are introduced.

free parameters (5)
  • PAPA residual scale α = 0.1
    Hand-chosen and ablated in Figure 4 and Appendix D; smaller α improves P-ACC but degrades reconstruction, larger α improves PESQ marginally but lowers P-ACC.
  • JMAS frame margin m1 = 0.5
    Hand-chosen in Eq. 5; no ablation reported.
  • JMAS structure margin m2 = 0.25
    Hand-chosen in Eq. 6; no ablation reported.
  • JMAS loss weights λframe, λstruct = 1, 1
    Hand-chosen in Eq. 4; no ablation reported.
  • Reconstruction/commitment/perceptual loss weights = 45 / 1000 / 450
    Hand-chosen in Eq. 7 (Lmel, Lcommit, Lperc), standard in the codec literature; no sensitivity analysis.
assumptions (5)
  • domain assumption MFA-derived phoneme boundaries on LibriSpeech are a valid ground-truth phonetic segmentation for measuring phoneme structure.
    Used to compute KNN, Silhouette, Davies-Bouldin, V-measure (Appendix A). Errors in forced alignment propagate into all separability metrics.
  • domain assumption KNN phoneme accuracy on standardized, PCA-reduced pre-quantization features is a faithful operationalization of phoneme structure relevant to token predictability.
    KNN is the primary predictor in Table 1; the paper itself notes Silhouette and other metrics behave differently, so the choice of metric is load-bearing.
  • domain assumption P-ACC and P-PPL on a separately trained 8K P-VQ probe measure the autoregressive predictability of speech tokens for real language models.
    Appendix B states P-ACC and N-ACC are not directly comparable; the diagnostic and 'frontier' claims rely on this proxy.
  • domain assumption Aligning quantized latents to a frozen WavLM-Large L24 teacher with the JMAS pairwise-cosine loss transfers phoneme structure without explicitly optimizing token predictability.
    Assumed in 'Post-quantization Structure Refinement'; the mechanism is not derived, only validated by ablations.
  • domain assumption Frozen W2v-BERT 2.0 L17 features are a suitable reconstruction-compatible anchor and are not degraded by the residual adaptation in a way that harms downstream TTS.
    Chosen based on Figure 2 layer profiles; ablation Table 3 shows WavLM as main path hurts reconstruction, supporting the choice but only for these two encoders.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure." pith.science (2026). https://pith.science/paper/XGZQBGTK

@misc{pith2026260808286,
  author       = {Pith},
  title        = {Pith review of: ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGZQBGTK}},
  note         = {Machine review of arXiv:2608.08286}
}
read the original abstract

Neural speech codecs face a fundamental tension in the language-model era: tokens that support high-fidelity reconstruction are not necessarily easy for autoregressive models to predict. Our controlled analysis of diverse codec and self-supervised speech representations shows that clearer phoneme structure before discrete code assignment is consistently associated with easier autoregressive token prediction. Yet phoneme structure alone is insufficient for high-fidelity reconstruction, which also requires reconstruction-relevant acoustic detail. Guided by this observation, we introduce ReLMCodec, a low-bitrate single-codebook speech codec built upon a preserve--control--refine principle: it preserves the linguistic organization of frozen self-supervised learning (SSL) features at the quantizer input, controls reconstruction-driven drift through Pre-quantization Anchor-Preserving Adaptation (PAPA), and refines the quantized latent space with a training-only WavLM-Large L24 teacher to reduce phoneme-level token fragmentation. Together, these components allow acoustic detail to support waveform reconstruction while keeping the resulting token sequence predictable for autoregressive models. At 650 and 800 bps, ReLMCodec moves the empirical single-stream predictability--reconstruction frontier in our evaluations, with gains that carry over to downstream text-to-speech (TTS) synthesis in both intelligibility and speaker similarity.

Figures

Figures reproduced from arXiv: 2608.08286 by the authors.

Figure 2
Figure 2. Layer-wise KNN phoneme accuracy for four encoders; [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 1
Figure 1. Structure transfer from pre-quantization represen [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Overall architecture of ReLMCodec. initialization, optimizer, and schedule. Any suitable frozen encoder can provide the anchor after adjusting interface dimen￾sions. Single-Codebook Vector Quantization Our single-layer EMA-VQ uses 8,192 (8K) or 65,536 (64K) codewords at 50 tokens per second (TPS), yielding 650/800 bits per second (BPS). For utilization, we use 8-dimensional factorized codes with ℓ2 normalization [Ku… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: PAPA residual-scale ablation. variance standardization, or ℓ2 normalization, supporting preser￾vation rather than exact invariance [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 8 canonical work pages

  1. [5]

    LucaDellaLibera,FrancescoPaissan,CemSubakan,andMirco Ravanelli

    URL https://arxiv.org/abs/2410.00037. LucaDellaLibera,FrancescoPaissan,CemSubakan,andMirco Ravanelli. FocalCodec: Low-bitrate speech coding via focal modulation networks. InAdvances in Neural Information Processing Systems, volume 38, pages 23742–23767,

    arXiv

  2. [6]

    URL https://arxiv.org/abs/ 2603.05299. ZhihaoDu,QianChen,ShiliangZhang,KaiHu,HengLu,Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao,andZhijieYan.CosyVoice:Ascalablemultilingualzero- shottext-to-speechsynthesizerbasedonsupervisedsemantic tokens, 2024a. URL https://arxiv.org/abs/2407.05407. ZhihaoDu,ShiliangZhang,KaiHu,andSiqiZheng. FunCode...

    work page arXiv

  3. [7]

    URL https://arxiv.or g/abs/2505.17589. Yitian Gong, Luozhijie Jin, Kuangwei Chen, Dong Zhang, RuifanDeng,XiaoguiYang,XinZhang,ZhaoyeFei,Qinyuan Cheng,ShiminLi,andXipengQiu.XY-Tokenizer:Mitigating thesemantic-acousticconflictinlow-bitratespeechcodecs.In Proceedingsofthe64thAnnualMeetingoftheAssociationfor Computational Linguistics (Volume 1: Long Papers), ...

    arXiv

  4. [8]

    16 YiweiGuo,ZhihanLi,ChenpengDu,HankunWang,XieChen, and Kai Yu

    doi: 10.18653/v1/2026.acl-long.423. 16 YiweiGuo,ZhihanLi,ChenpengDu,HankunWang,XieChen, and Kai Yu. LSCodec: Low-bitrate and speaker-decoupled discrete speech codec. InProc. Interspeech 2025, pages 5018–5022,

    work page doi:10.18653/v1/2026.acl-long.423 2026

  5. [9]

    NadavHar-Tuv,OrTal,andYossiAdi

    doi: 10.21437/Interspeech.2025-1106. NadavHar-Tuv,OrTal,andYossiAdi. PAST:Phonetic-acoustic speech tokenizer,

    work page doi:10.21437/interspeech.2025-1106 2025

  6. [11]

    Harry Julian, Rachel Beeson, Lohith Konathala, Johanna Ulin, and Jiameng Gao

    doi: 10.18653/v1/2025.acl-long.937. Harry Julian, Rachel Beeson, Lohith Konathala, Johanna Ulin, and Jiameng Gao. Finite scalar quantization enables redun- dant and transmission-robust neural audio compression at low bit-rates,

    doi:10.18653/v1/2025.acl-long.937 2025

  7. [12]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae

    URL https://arxiv.org/abs/2509.09550. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-GAN: Generativeadversarialnetworksforefficientandhigh-fidelity speech synthesis. InAdvances in Neural Information Pro- cessing Systems, volume 33, pages 17022–17033,

    arXiv

  8. [13]

    Montreal forced aligner: Trainable text-speech alignment using Kaldi

    Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using Kaldi. InProc. Inter- speech 2017, pages 498–502,

    work page 2017

Show all 26 references
  1. [15]

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur

    doi: 10.21437/Interspeech.2024-2135. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: An ASR corpus based on public domainaudiobooks. In2015IEEEInternationalConference onAcoustics,SpeechandSignalProcessing(ICASSP),pages 5206–5210. IEEE,

    2024 doi

  2. [17]

    Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari

    URL https://arxiv.org/abs/2212.04356. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. UTMOS: UTokyo-SaruLab system for VoiceMOS challenge

    arXiv

  3. [18]

    arXiv preprint arXiv:2204.02152,

    arXiv

  4. [20]

    Laurin Wagner, Bernhard Thallinger, Miroslav Stankovic, and Mario Zusag

    URL https://arxiv.org/ abs/2506.00385. Laurin Wagner, Bernhard Thallinger, Miroslav Stankovic, and Mario Zusag. Content is what remains: Invariant speech tokenization from parallel utterances,

    arXiv

  5. [21]

    17 Zixiang Wan, Ziyue Qiu, Yiyang Liu, and Wei-Qiang Zhang

    URL https: //arxiv.org/abs/2607.19033. 17 Zixiang Wan, Ziyue Qiu, Yiyang Liu, and Wei-Qiang Zhang. Metadata-enhanced speech emotion recognition: Augmented residual integration and co-attention in two-stage fine-tuning. In2025IEEEInternationalConferenceonAcoustics,Speech and Si...

    arXiv

  6. [22]

    Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu

    doi: 10.1109/ICASSP55912.2026.11462082. Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. MaskGCT: Zero-shot text-to-speech with masked generative codec transformer. InInternational Confe...

    2026

  7. [23]

    Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari

    doi: 10.1 109/ICASSP48485.2024.10447464. Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. BigCodec: Pushing the limits of low-bitrate neural speech codec,

    2024

  8. [24]

    An Yang et al

    URL https://arxiv.org/abs/2511.05516. An Yang et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024a. doi: 10.48550/arXiv.2407.10671. DongchaoYang,SongxiangLiu,RongjieHuang,JinchuanTian, Chao Weng, and Yuexian Zou. HiFi-Codec: Group-residual vector quantization f...

  9. [25]

    DongchaoYang,HaohanGuo,YuanyuanWang,RongjieHuang, XiangLi,XuTan,XixinWu,andHelenMeng

    URL https://arxiv.org/abs/2305.02765. DongchaoYang,HaohanGuo,YuanyuanWang,RongjieHuang, XiangLi,XuTan,XixinWu,andHelenMeng. UniAudio1.5: Largelanguagemodel-drivenaudiocodecisafew-shotaudio task learner, 2024b. URL https://arxiv.org/abs/2406.10056. Dongchao Yang, Songxiang Liu,...

    arXiv

  10. [26]

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu

    doi: 10.1109/TA SLP.2021.3129994. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. SpeechTokenizer: Unified speech tokenizer for speech language models. InInternational Conference on Learning Representations,

    2021

  11. [2015]

    doi: 10.1109/ICASSP.2015.7178

    2015

  12. [2017]

    Pooneh Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli

    doi: 10.21437/Interspee ch.2017-1386. Pooneh Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. How should we extract discrete audio tokens from self-supervised models? InProc. Interspeech 2024, pages 2554–2558,

    2017 doi

  13. [2021]

    Qwen3-TTS technical report, 2026a

    HangruiHu,XinfaZhu,TingHe,DakeGuo,BinZhang,Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, et al. Qwen3-TTS technical report, 2026a. URL https://arxiv.org/abs/2601.15621. Jingbin Hu, Haoyu Zhang, Dake Guo, Qirui Zhan, Wenhao Li, Huakang Chen, Guobin Ma, Hanke Xi...

    arXiv

  14. [2022]

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour

    URL https://arxiv.org/abs/2210.13438. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: A speech-text foundation model for real- time dialogue,

    arXiv

  15. [2023]

    Hubert Siuzdak

    URL https://arxiv.org/abs/2312.05187. Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis,

    arXiv

  16. [2024]

    Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang,JianZhao,KaiYu,andXieChen

    URL https://arxiv.org/abs/ 2406.05370. Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang,JianZhao,KaiYu,andXieChen. F5-TTS:Afairytaler that fakes fluent and faithful speech with flow matching. In Proceedingsofthe63rdAnnualMeetingoftheAssociationfor Computational Lin...

    arXiv

  17. [2025]

    Yushen Chen, Kai Hu, Long Zhou, Shulin Feng, Xusheng Yang, Hangting Chen, and Xie Chen

    doi: 10.18653/v1/2025.acl-long.313. Yushen Chen, Kai Hu, Long Zhou, Shulin Feng, Xusheng Yang, Hangting Chen, and Xie Chen. AUV: Teaching audio universal vector quantization with single nested codebook. In2026IEEEInternationalConferenceonAcoustics,Speech andSignalProcessing(IC...

    2025 doi

  18. [2026]

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi

    URL https://arxiv.org/ abs/2604.12383. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.