Pith. sign in

REVIEW 3 major objections 5 minor 41 references

In codec-based self-supervised speech learning, the training language of the neural audio codec has little effect on downstream recognition, while the SSL pre-training language determines performance, meaning a single codec can be reused ac

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 00:03 UTC pith:KOIBEEX3

load-bearing objection Useful decomposition of codec-based SSL language sensitivity, but the headline claim about SSL pre-training language is confounded with domain; the codec-insensitivity result is the solid takeaway. the 3 major comments →

arxiv 2607.26350 v1 pith:KOIBEEX3 submitted 2026-07-28 cs.SD cs.CL

Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

Daigo Takizawa , Tomohiko Nakamura , Samuele Cornell , William Chen , Satoru Fukayama , Shinji Watanabe This is my paper
classification cs.SD cs.CL
keywords neural audio codecscodec-based self-supervised learninglanguage sensitivitycross-lingual speech representationdiscrete speech tokensautomatic speech recognitionspeech emotion recognitionself-supervised pre-training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a codec-based self-supervised speech system must be retrained when it moves to a new language, and it separates the question into two stages: the neural audio codec that turns waveforms into discrete tokens, and the SSL model trained on those tokens. By varying one stage's training language while holding the other fixed, across English, Japanese, and Chinese, it finds that the codec's training language barely matters, while matching the SSL pre-training language to the target language matters a lot. The upshot is that one audio codec can be reused across languages, but the language of the SSL pre-training data has to align with the downstream language.

Core claim

The paper reports a controlled decomposition of language sensitivity in codec-based SSL. When the SSL pre-training language is fixed and only the neural audio codec's training language is changed, downstream ASR and SER performance stay nearly constant in all three tested languages, even when the codec is trained on a totally different language from the downstream task. When the codec is fixed and only the SSL pre-training language changes, downstream performance swings strongly, with matched language generally winning for both ASR and SER. The authors conclude that language-specific structure is acquired primarily during SSL pre-training, not during codec training, and that a single acousti

What carries the argument

The central object is the two-stage codec-based SSL pipeline: a neural audio codec that converts waveforms into discrete tokens, and an SSL model trained on those tokens. The controlling instrument is the experimental design that changes one stage's training language while keeping the other fixed, together with the coefficient of variation (CoV) of per-language error rates normalized by a waveform-input baseline, which makes cross-language comparisons scale-free. The CoV values are the numbers that carry both conclusions: small for NAC-language changes, large for SSL-language changes.

Load-bearing premise

The conclusion that SSL sensitivity is driven by language rather than by corpus domain relies on the assumption that the three SSL pre-training corpora differ only in language: English comes from audiobooks, Japanese from broadcast TV, and Chinese from a mixed large-scale corpus, so domain mismatch could masquerade as a language effect.

What would settle it

Run codec-based SSL twice on the same language with SSL data drawn from two different domains—for example, English audiobooks versus English broadcast news—and measure the CoV between the two domain conditions. If that within-language CoV approaches the CoV observed between languages, the paper's attribution of sensitivity to language rather than domain would be refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Codec-based SSL pipelines can keep a single pretrained acoustic codec as a fixed front-end across languages, removing or deferring codec retraining costs.
  • Deployments should spend their language-specific effort on the SSL pre-training stage, since that is where cross-lingual degradation originates.
  • Improving codec tokens is unlikely by itself to fix cross-lingual performance gaps; the SSL pre-training data is the lever that matters.
  • Language-insensitive codecs make shared multilingual tokenizers feasible, so long as downstream SSL models are language-matched.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the language-vs-domain question: pre-train English SSL on a broadcast/news corpus and Japanese SSL on audiobook-style data; if the downstream gap reverses or vanishes, part of the reported language sensitivity is actually corpus-domain sensitivity.
  • The CoV normalization uses a separate waveform-input baseline per language; if those baselines differ in quality, some of the measured sensitivity could be normalization noise rather than SSL language effects.
  • The same staged-control design could be applied to non-speech audio—music or environmental sound—where the question of which stage carries category information is still open.
  • If the conclusion generalizes, a practical recipe for building multilingual codec-based speech models is to train one codec on available mixed audio and then train language-matched SSL models on top of it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies the language sensitivity of codec-based self-supervised speech learning (SSL), separately examining the neural audio codec (NAC) and the SSL model. The authors conduct controlled experiments in English, Japanese, and Chinese. RQ1 evaluates ASR/SER on NAC-reconstructed waveforms, comparing public codecs and DAC variants retrained on different language data. RQ2 fixes the NAC and varies the SSL pre-training language, reporting large coefficients of variation (CoV) in downstream ASR. RQ3 fixes SSL pre-training to the downstream language and varies the NAC training language, reporting small CoVs. The central claim is that downstream performance is insensitive to the NAC training language but strongly dependent on the SSL pre-training language, implying that a single NAC can be reused across languages while SSL pre-training data must be language-matched. Two downstream tasks (ASR and SER) and multiple test corpora per language are used.

Significance. If the central claim holds, the paper has a clear practical payoff: acoustic codecs need not be retrained for each language in codec-based SSL pipelines, while SSL pre-training data should be aligned with the target language. The experimental design is genuinely more systematic than prior English-centric codec studies: it decouples the two stages, retrains a codec under matched data amounts, and evaluates on both ASR and SER. The paper also has the virtue of using public recipes and reproducible components (HuBERT, DAC, ESPnet, fairseq). However, the strength of the claim is currently limited by a language–domain confound in RQ2 and by the fact that RQ3 retrains only one codec family selected after inspecting RQ1 results. These issues are fixable, but they need to be addressed before the conclusions can be accepted as stated.

major comments (3)
  1. [Section 5.2 / Table 3] The claim that downstream performance is 'strongly dependent on the SSL pre-training language' rests on RQ2, but the SSL pre-training corpora differ in both language and domain: English LibriSpeech is audiobooks, Japanese is in-house broadcast TV, and Chinese is WenetSpeech (multi-domain). The downstream test sets mirror this: EN LL-10h (Libri-Light audiobook), JP LTVS-100h (broadcast TV), ZH WS-100h (WenetSpeech subset). The largest WER/CER gaps occur exactly in these matched-domain cells (e.g., EN SSL on LL-10h: 12.3 vs 28.0/27.5 for JP/ZH; JP SSL on LTVS: 15.7 vs 20.4/18.4; ZH SSL on WS-100h: 16.9 vs 24.4/21.8). The cross-domain datasets (CSJ: 4.2–4.8; COJADS: 41.6–45.1; AS1: 4.8–5.5) show much smaller gaps. The statement in Section 5.2 that 'the effect is driven primarily by language differences rather than domain differences' is not supported by a clean contrast holding domain fixed
  2. [Sections 4.1 and 5.3] RQ3 concludes that retraining an acoustic NAC for each target language is 'generally unnecessary', but this conclusion is drawn from retraining only DAC, which was selected in RQ1 because it showed the lowest CoV among the public codecs. Selecting the codec on the outcome measure used in the main study risks understating NAC language sensitivity. The internal comparison among DAC training-language conditions is clean and shows small differences, but 'DAC' is one family. To support the general reuse claim, either retrain at least one additional codec family (e.g., EnCodec) under the same language-controlled conditions, or explicitly scope the RQ3 conclusion to DAC-like acoustic codecs.
  3. [Section 3.3 and Table 3] The CoV metric is computed over only three languages, each represented by a small number of datasets. The SER results include cross-validation standard deviations in Table 2 and Table 3, but these are not propagated into the CoV, and for ASR no uncertainty information is given. Because the key contrast is between RQ2 CoVs (12.49–42.99%) and RQ3 CoVs (1.38–5.77%), a simple permutation or bootstrap test, even over the three languages, would materially strengthen the claim that the difference is not attributable to sampling noise. Please report uncertainty intervals or a significance test for the CoVs in both research questions.
minor comments (5)
  1. [Table 1 / Section 3.2] The 'In-house Broadcast TV data (JP)' used for both NAC training and SSL pre-training is not a public dataset; please provide a description, a release link, or a clear statement of availability to facilitate reproducibility.
  2. [Section 5.1] The description of HuBERT clustering is brief. It says RQ2 uses MFCC-based two-stage clustering and RQ3 uses pseudo labels from language-specific HuBERT checkpoints, but it is not explicit whether the k-means cluster centers or the label inventory are shared across languages or defined per language. Please clarify.
  3. [Table 2 / Table 3] The top-line 'Waveform' rows use a dash in the 'Lang.' column, which is understandable but could be labeled '—' or 'N/A' for clarity. Also, the 'X/Y' notation for LL-10h and WS-100h in the table header is defined only in Section 4.1; consider adding a table note.
  4. [Section 3.2] The COJADS evaluation split is described only as 'selected to ensure regional balance'. A short description of the selection procedure or a reference to where this split is defined would improve reproducibility.
  5. [References] Several references have formatting artifacts (e.g., 'V oIP', 'Y .' with extra spaces). These are minor and do not affect content, but a cleanup pass would be beneficial.

Circularity Check

0 steps flagged

No circularity: all conclusions are direct measured comparisons; no fitted input is relabeled as a prediction, and no load-bearing self-citation chain forces the result.

full rationale

The paper does not contain a derivation chain in which an output is equivalent to an input by construction. RQ1–RQ3 are controlled comparisons of measured WER/CER/AR values, and the only formal equation (Eq. 1) defines CoV as a descriptive normalized statistic over those measured error rates, not as a fitted parameter or a predicted quantity. No parameter is estimated from a subset of the data and then used to predict a closely related outcome. The choice of DAC after observing its low CoV in RQ1 (Section 4.1: 'DAC achieved the smallest CoVs for both ASR and SER among the evaluated models. We therefore select DAC for subsequent analysis') is a model-selection decision that could understate NAC language sensitivity, but it is not circular reasoning: it does not by construction determine the RQ2/RQ3 results or define the paper's conclusion. The authors' self-citations (e.g., ESPnet-Codec, Codec2Vec, prior NAC-generalization work) are used as public toolkits, prior recipes, or corroborating observations, not as the sole justification for the central claim. The Section 5.2 inference that the SSL effect is 'driven primarily by language differences rather than domain differences' is under-supported given that each SSL corpus is tied to a different domain, but that is an external-validity/correctness concern about confounding, not circularity under the required evidentiary standard.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No free parameters or invented entities are introduced. Every reported value is a measured error rate/recall or a derived CoV; experimental controls such as 1056-hour and 960-hour sampling are not fitted parameters. The analysis rests on three domain assumptions, the most fragile being the language/domain confound in RQ2.

axioms (3)
  • domain assumption The CoV over baseline-normalized error rates (Eq. 1) is a meaningful scale-free measure of cross-lingual sensitivity.
    All sensitivity conclusions derive from this metric; it weights languages equally after normalization.
  • domain assumption Differences in RQ2 are attributable to SSL language rather than domain/dataset mismatch.
    English, Japanese, and Chinese SSL corpora come from different domains (audiobooks, broadcast TV, mixed web); the paper's defense in Section 5.2 is indirect.
  • domain assumption ASR and SER on English, Japanese, and Chinese generalize to other languages and downstream tasks.
    Only 3 languages and 2 tasks are tested; the practical recommendation about NAC retraining is an extrapolation.

pith-pipeline@v1.3.0-alltime-deepseek · 9528 in / 12530 out tokens · 132855 ms · 2026-08-01T00:03:22.815402+00:00 · methodology

0 comments
read the original abstract

Neural audio codecs (NACs) have become popular for obtaining speech representations as discrete tokens. Beyond compression, discrete tokens can be used to train self-supervised learning (SSL) models. Such models, referred to as codec-based SSL models, reduce data storage and computational cost, enabling scalable SSL pre-training. However, their language sensitivity remains unclear. When the language changes, codec-based SSL models may require retraining, which undermines their efficiency. In this paper, we present a systematic analysis of language sensitivity by varying either the NAC training language or the SSL pre-training language while keeping the other fixed. Experimental results show that downstream performance is insensitive to the NAC training language but strongly dependent on the SSL pre-training language. These findings suggest that a single NAC can be reused across languages, while aligning the SSL pre-training language with the target language is crucial.

Figures

Figures reproduced from arXiv: 2607.26350 by Daigo Takizawa, Samuele Cornell, Satoru Fukayama, Shinji Watanabe, Tomohiko Nakamura, William Chen.

Figure 1
Figure 1. Figure 1: Overview of our controlled experimental design to disentangle language sensitivity in codec-based SSL by de￾coupling the NAC and SSL pre-training stages. RQ1 eval￾uates downstream performance on NAC-reconstructed wave￾forms, RQ2 varies the SSL pre-training language with a fixed NAC, and RQ3 varies the NAC training language with matched SSL pre-training. when the pre-training language is misaligned with the… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 3 linked inside Pith

  1. [1]

    Introduction Self-supervised learning (SSL) is widely used to build univer- sal feature extractors for speech tasks such as automatic speech recognition (ASR) and speech emotion recognition (SER) [1– 3]. Despite this versatility, SSL typically requires unlabeled yet massive amounts of speech data and long training sched- ules, limiting the scalability of ...

  2. [2]

    ESPnet-Codec provided a unified benchmark for training and evaluating NACs arXiv:2607.26350v1 [cs.SD] 28 Jul 2026 and offered early empirical studies of codec-based SSL [8]

    Related Work Codec-based SSL has recently gained attention as a cost- efficient alternative to waveform-based SSL. ESPnet-Codec provided a unified benchmark for training and evaluating NACs arXiv:2607.26350v1 [cs.SD] 28 Jul 2026 and offered early empirical studies of codec-based SSL [8]. Codec2Vec showed that codec-based SSL can substantially re- duce sto...

    Pith/arXiv arXiv 2026

  3. [3]

    Experimental Design The experiments are designed to decouple language sensitiv- ity in codec-based SSL across the NAC and the SSL pre- training stage (Fig

    Evaluation Setup 3.1. Experimental Design The experiments are designed to decouple language sensitiv- ity in codec-based SSL across the NAC and the SSL pre- training stage (Fig. 1). For RQ1, we evaluate ASR/SER on NAC-reconstructed waveform using multiple publicly available NACs and Descript Audio Codec (DAC) [16] retrained under different training-langua...

  4. [4]

    Comparison across NACs To answer RQ1, we examined language sensitivity on NAC- reconstructed waveforms using the setup in Section 3

    Language Sensitivity in NAC-Reconstructed Waveforms 4.1. Comparison across NACs To answer RQ1, we examined language sensitivity on NAC- reconstructed waveforms using the setup in Section 3. We eval- uated five publicly available neural audio codecs with distinct training specifications: DAC [16], EnCodec [29], SpeechTok- enizer [30], X-Codec [31], and PAS...

  5. [5]

    Experimental Setup We now analyze the language sensitivity within codec-based SSLs to answer RQ2 and RQ3

    Language Sensitivity in Codec-Based SSLs 5.1. Experimental Setup We now analyze the language sensitivity within codec-based SSLs to answer RQ2 and RQ3. Unlike RQ1, which used re- constructed waveforms, codec-based SSL uses discrete tokens as inputs. Datasets.SSL pre-training used the language-specific corpora in Table 1. For RQ2, 960 hours of Japanese and...

  6. [6]

    Conclusion We dissected language sensitivity in codec-based SSL by sepa- rating the NAC and SSL pre-training stages. Across ASR and SER in English, Japanese, and Chinese, acoustic NACs exhib- ited limited language sensitivity and NAC training language had limited impact, whereas downstream performance was sensitive to SSL pre-training language. These resu...

  7. [7]

    This study is also supported by AIST policy-based bud- get project ’R&D on Generative AI Foundation Models for the Physical Domain’

    Acknowledgments This paper is based on results obtained from a project, Pro- grams for Bridging the gap between R&D and the IDeal so- ciety (society 5.0) and Generating Economic and social value (BRIDGE)/Practical Global Research in the AI × Robotics Services, implemented by the Cabinet Office, Government of Japan. This study is also supported by AIST pol...

  8. [8]

    The authors reviewed and edited the out- put as needed and take full responsibility for the content of the manuscript

    Generative AI Use Disclosure The authors used ChatGPT to improve the language and clarity of this manuscript. The authors reviewed and edited the out- put as needed and take full responsibility for the content of the manuscript

  9. [9]

    Self-supervised speech representation learning: A review,

    A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe, T. N. Sainath, and S. Watanabe, “Self-supervised speech representation learning: A review,”IEEE J. Sel. Top. Signal Process., vol. 16, no. 6, pp. 1179–1210, 2022

    2022

  10. [10]

    wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” inProc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, pp. 12 449–12 460

    2020

  11. [11]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhut- dinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 29, pp. 3451–3460, 2021

    2021

  12. [12]

    Reducing barriers to self-supervised learning: HuBERT pre- training with academic compute,

    W. Chen, X. Chang, Y . Peng, Z. Ni, S. Maiti, and S. Watan- abe, “Reducing barriers to self-supervised learning: HuBERT pre- training with academic compute,” inProc. INTERSPEECH, 2023, pp. 4404–4408

    2023

  13. [13]

    Fast- HuBERT: An efficient training framework for self-supervised speech representation learning,

    G. Yang, Z. Ma, Z. Zheng, Y . Song, Z. Niu, and X. Chen, “Fast- HuBERT: An efficient training framework for self-supervised speech representation learning,” inProc. IEEE Autom. Speech Recognit. Understanding Workshop, 2023

    2023

  14. [14]

    Towards efficient self-supervised repre- sentation learning in speech processing,

    L. Lugo and V . Vielzeuf, “Towards efficient self-supervised repre- sentation learning in speech processing,” inProc. Findings Assoc. Comput. Linguistics: EACL, 2024, pp. 340–346

    2024

  15. [15]

    Efficient training of self-supervised speech foundation models on a com- pute budget,

    A. T. Liu, Y .-C. Lin, H. Wu, S. Winkler, and H.-Y . Lee, “Efficient training of self-supervised speech foundation models on a com- pute budget,” inProc. IEEE Spoken Language Technology Work- shop, 2024, pp. 961–968

    2024

  16. [16]

    ESPnet-Codec: Comprehensive training and eval- uation of neural codecs for audio, music, and speech,

    J. Shi, J. Tian, Y . Wu, J.-W. Jung, J. Q. Yip, Y . Masuyama, W. Chen, Y . Wu, Y . Tang, M. Baali, D. Alharthi, D. Zhang, R. Deng, T. Srivastava, H. Wu, A. Liu, B. Raj, Q. Jin, R. Song, and S. Watanabe, “ESPnet-Codec: Comprehensive training and eval- uation of neural codecs for audio, music, and speech,” inProc. IEEE Spoken Language Technology Workshop, 2...

    2024

  17. [17]

    Codec2Vec: Self-supervised speech representation learning using neural speech codecs,

    W.-C. Tseng and D. Harwath, “Codec2Vec: Self-supervised speech representation learning using neural speech codecs,” in Proc. IEEE Autom. Speech Recognit. Understanding Workshop, 2025

    2025

  18. [18]

    Discrete audio tokens: More than a survey!

    P. Mousavi, G. Maimon, A. Moumen, D. Petermann, J. Shi, H. Wu, H. Yang, A. Kuznetsova, A. Ploujnikov, R. Marxer, B. Ramabhadran, B. Elizalde, L. Lugosch, J. Li, C. Subakan, P. Woodland, M. Kim, H. yi Lee, S. Watanabe, Y . Adi, and M. Ra- vanelli, “Discrete audio tokens: More than a survey!”Trans. Mach. Learn. Res., 2025

    2025

  19. [19]

    Exploration of language dependency for Japanese self-supervised speech rep- resentation models,

    T. Ashihara, T. Moriya, K. Matsuura, and T. Tanaka, “Exploration of language dependency for Japanese self-supervised speech rep- resentation models,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2023

    2023

  20. [20]

    Evaluating self- supervised speech models on a Taiwanese Hokkien corpus,

    Y .-H. Chou, K. Chang, M.-J. Wu, W. Ou, A. W.-H. Bi, C. Yang, B. Y . Chen, R.-W. Pai, P.-Y . Yeh, J.-P. Chiang, I.-T. Phoann, W. Chang, C. Cui, N. Chen, and J. Shi, “Evaluating self- supervised speech models on a Taiwanese Hokkien corpus,” in Proc. IEEE Autom. Speech Recognit. Understanding Workshop, 2023

    2023

  21. [21]

    How to learn a new language? An efficient solution for self-supervised learning mod- els unseen languages adaption in low-resource scenario,

    S.-H. Wang, Z.-C. Chen, J. Shi, M.-T. Chuang, G.-T. Lin, K.-P. Huang, D. Harwath, S.-W. Li, and H. yi Lee, “How to learn a new language? An efficient solution for self-supervised learning mod- els unseen languages adaption in low-resource scenario,”arXiv preprint arXiv:2411.18217, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    On the language and gen- der biases in PSTN, V oIP and neural audio codecs,

    K. Altwlkany, A. Kuric, and E. Lacic, “On the language and gen- der biases in PSTN, V oIP and neural audio codecs,” inProc. IN- TERSPEECH, 2025, pp. 1348–1352

    2025

  23. [23]

    Do neural codecs generalize? A controlled study across unseen languages and non-speech tasks,

    S.-H. Wang, J. Shi, J. Tian, H. Wu, and S. Watanabe, “Do neural codecs generalize? A controlled study across unseen languages and non-speech tasks,”arXiv preprint arXiv:2601.12205, 2026

    arXiv 2026

  24. [24]

    High-fidelity audio compression with improved RVQGAN,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved RVQGAN,” in Proc. Adv. Neural Inf. Process. Syst., vol. 36, 2023, pp. 27 980– 27 993

    2023

  25. [25]

    Libri-Light: A benchmark for ASR with limited or no supervi- sion,

    J. Kahn, M. Rivi `ere, W. Zheng, E. Kharitonov, Q. Xu, P. Mazar´e, J. Karadayi, V . Liptchinsky, R. Collobert, C. Fuegen, T. Likhoma- nenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, “Libri-Light: A benchmark for ASR with limited or no supervi- sion,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2020, pp. 7669–7673

    2020

  26. [26]

    Construction of a large-scale Japanese ASR corpus on TV recordings,

    S. Ando and H. Fujihara, “Construction of a large-scale Japanese ASR corpus on TV recordings,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2021, pp. 6948–6952

    2021

  27. [27]

    WenetSpeech: A 10000+ hours multi-domain Mandarin corpus for speech recognition,

    B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng, D. Wu, and Z. Peng, “WenetSpeech: A 10000+ hours multi-domain Mandarin corpus for speech recognition,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2022, pp. 6182–6186

    2022

  28. [28]

    AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline,

    H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline,” inProc. O-COCOSDA, 2017, pp. 1–5

    2017

  29. [29]

    Lib- riSpeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- riSpeech: An ASR corpus based on public domain audio books,” inProc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2015, pp. 5206–5210

    2015

  30. [30]

    Corpus of spontaneous Japanese: Its design and evaluation,

    K. Maekawa, “Corpus of spontaneous Japanese: Its design and evaluation,” inProc. ISCA & IEEE Workshop Spont. Speech Pro- cess. Recognit., 2003, pp. 7–12

    2003

  31. [31]

    Cor- pus of Japanese dialects (COJADS),

    National Institute for Japanese Language and Linguistics, “Cor- pus of Japanese dialects (COJADS),” https://www2.ninjal.ac.jp/ cojads/index.html, 2019, accessed: 2026-02-24

    2019

  32. [32]

    IEMOCAP: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: Interactive emotional dyadic motion capture database,”Lang. Re- sour. Evaluation, vol. 42, no. 4, pp. 335–359, 2008

    2008

  33. [33]

    Construction and anal- ysis of phonetically and prosodically balanced emotional speech database,

    E. Takeishi, T. Nose, Y . Chiba, and A. Ito, “Construction and anal- ysis of phonetically and prosodically balanced emotional speech database,” inProc. O-COCOSDA, 2016, pp. 16–21

    2016

  34. [34]

    EmotionTalk: An interactive chinese multimodal emotion dataset with rich an- notations,

    H. Sun, X. Wang, J. Zhao, S. Zhao, J. Zhou, H. Wang, J. He, A. Kong, X. Yang, Y . Wang, Y . Lin, and Y . Qin, “EmotionTalk: An interactive chinese multimodal emotion dataset with rich an- notations,”arXiv preprint arXiv:2505.23018, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    ESPnet: End-to-end speech processing toolkit,

    S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y . Unno, N. E. Y . Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduch- intala, and T. Ochiai, “ESPnet: End-to-end speech processing toolkit,” inProc. INTERSPEECH, 2018, pp. 2207–2211

    2018

  36. [36]

    SUPERB: Speech processing universal performance benchmark,

    S. wen Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, “SUPERB: Speech processing universal performance benchmark,” inProc. INTER- SPEECH, 2021, pp. 1194–1198

    2021

  37. [37]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”Trans. Mach. Learn. Res., 2023

    2023

  38. [38]

    SpeechTok- enizer: Unified speech tokenizer for speech language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “SpeechTok- enizer: Unified speech tokenizer for speech language models,” in Proc. Int. Conf. Learn. Representations, vol. 24, 2024, pp. 27–30

    2024

  39. [39]

    Codec does matter: Ex- ploring the semantic shortcoming of codec for audio language model,

    Z. Ye, P. Sun, J. Lei, H. Lin, X. Tan, Z. Dai, Q. Kong, J. Chen, J. Pan, Q. Liu, Y . Guo, and W. Xue, “Codec does matter: Ex- ploring the semantic shortcoming of codec for audio language model,” inProc. AAAI Conf. Artif. Intell., vol. 39, no. 24, 2025, pp. 25 697–25 705

    2025

  40. [40]

    PAST: Phonetic-acoustic speech tokenizer,

    N. Har-Tuv, O. Tal, and Y . Adi, “PAST: Phonetic-acoustic speech tokenizer,” inProc. INTERSPEECH, 2025, pp. 3509–3513

    2025

  41. [41]

    fairseq: A fast, extensible toolkit for sequence modeling,

    M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grang- ier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” inProc. NAACL-HLT Demonstrations, 2019, pp. 48– 53

    2019