Benchmarking Prosody Encoding in Discrete Speech Tokens

Daisuke Saito; Kentaro Onda; Nobuaki Minematsu; Satoru Fukayama

REVIEW 3 major objections 3 minor 49 references

Benchmarking Prosody Encoding in Discrete Speech Tokens

T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the prosodic information carried by discrete speech tokens can be measured by how much the token sequence changes when pitch and timing are artificially modified, and that this sensitivity can guide tokenizer design.

desk verdict A plausible empirical benchmark for prosody in discrete speech tokens, but the sensitivity metric needs a close look once a readable full text exists. read the letter →

arxiv 2508.11224 v1 pith:LMYVSNTK submitted 2025-08-15 cs.SD cs.CLeess.AS

Kentaro Onda , Satoru Fukayama , Daisuke Saito , Nobuaki Minematsu This is my paper

classification cs.SDcs.CLeess.AS

keywords discretespeechtokensprosodyencodingself-supervisedmodelsk-meansclusteringlanguageevaluationtokenization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish a practical benchmark for how much prosody survives when speech is converted into discrete tokens by self-supervised feature extractors and k-means clustering. Its central proposal is to measure a tokenizer's sensitivity to artificial prosody edits: if changing pitch or timing changes the resulting token sequence, the tokens are carrying prosodic information. The authors sweep choices such as the SSL model, the layer used, and the number of clusters, and use the sensitivity measure to derive design guidelines for discrete speech tokens. The reason to care is that speech language models are expected to produce prosodically natural output, but discretization choices are currently made heuristically and separately from downstream training.

What carries the argument

The central object is the sensitivity metric: the measured change in the discrete token sequence when a speech sample's prosody is artificially modified. High sensitivity is interpreted as strong prosodic encoding, low sensitivity as prosody lost in discretization. This metric carries the whole analysis, since it replaces end-to-end evaluation of a speech language model with a direct, token-level test of the discretizer. The benchmark uses it to compare configurations by systematically varying the self-supervised feature source, the quantization method, and the number of clusters.

What would settle it

Train a speech LM on high-scoring tokens from this benchmark and test whether its output preserves pitch contours and timing as judged by a prosody classifier or by listeners; if high sensitivity coexists with poor downstream prosody reproduction, the proxy fails. A second check: if the sensitivity score largely disappears when the words and phonemes are held strictly fixed, the measured signal may come from segmental rather than prosodic changes.

Watch

Extended reading notes

Core claim

The paper's central claim is that prosodic encoding in discrete speech tokens can be measured directly by looking at how token sequences react to controlled prosody manipulation, without needing a trained downstream model. Under this view, a tokenizer that changes its output when F0 or duration is edited is thereby shown to be preserving those prosodic cues; one that ignores the edit is discarding them. By applying this sensitivity test across different SSL model choices, quantization granularities, and cluster sizes, the paper aims to identify which configurations keep prosodic contrasts and which erase them. The resulting guidelines would let researchers choose tokenizer settings on the ba

Load-bearing premise

The load-bearing premise is that token-sequence changes under artificial F0 and duration edits are a faithful proxy for the prosodic information a downstream speech language model can actually use.

Editorial extensions

If this is right

Tokenizer settings can be screened for prosody preservation before expensive training of a speech language model.
Choices like the SSL model and cluster count become evidence-based: configurations that lose pitch or timing information can be rejected up front.
Discrete token benchmarks can include a prosody axis alongside content-based axes, so pseudo-text representations are compared on expressiveness as well as semantics.
Future tokenizers can be tested against the same sensitivity protocol, making the result a reusable evaluation rather than a one-off study.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: high sensitivity to prosody edits is likely necessary but not sufficient—a downstream speech LM may still fail to use the encoded cues if its training objective does not condition on prosody, so the benchmark measures the ceiling, not the realized behavior.
Editorial inference: the protocol could be split by prosodic dimension (pitch range, contour shape, speaking rate) to give each tokenizer a profile, which would reveal, for example, a configuration that keeps F0 but discards timing.
Editorial inference: rankings from this benchmark may shift on tonal languages, where pitch movements carry lexical meaning, or on expressive speech, so the design guidelines may need language- and style-specific re-evaluation.
Editorial inference: the same artificial-edits setup could be inverted to train tokenizers that are deliberately invariant to certain prosodic changes or deliberately sensitive to them, depending on the task.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Honest take: I only have the abstract in readable form; the full text on arXiv is a garbled mojibake dump, so this letter is an abstract-level judgment plus a flag for what to verify when a clean copy appears.

The paper wants to give practitioners a benchmark for how well discrete speech tokens preserve prosody. The abstract makes a fair point: SSL tokens are usually chosen heuristically, and the prosody question is underexplored. Running a sensitivity analysis across SSL models and cluster counts, with practical design guidelines at the end, is a useful contribution if the method holds up. I don't see anything in the abstract that smells like fabrication or a thought experiment dressed as measurement; it looks like a genuine empirical study.

What I can't verify is the full text. The reader's soundness score of 4 is honest unverifiability, not an identified flaw. The one substantive concern I want to put on record is the sensitivity metric itself. If you modify F0 or duration, you also shift frame alignment and segmental acoustics. If the token-change rate isn't normalized or controlled for that, the ranking of tokenizers could be about artifacts rather than prosody encoding. The stress-test note makes this exact point; I think it's a fair question, and the answer has to be in the method section. The abstract doesn't show that detail.

On the other hand, I wouldn't call it load-bearing from what's visible. The study could easily have done the right thing—alignment-invariant matching, phone-level controls, or a hold-out that separates prosodic from segmental change. Many speech benchmarks do. If they didn't, the guidelines are shaky; if they did, this is a solid measurement paper.

Does it deserve a referee? Yes. The topic is narrow but real, the abstract promises concrete output, and the risks are checkable. I'd send it to peer review with an explicit request for the confound analysis. I'd also tell the authors to fix whatever happened to the PDF, because no one can review a mojibake dump.

This is for the speech-LM and discrete-token crowd. I wouldn't cite it until I've read the clean version, but I'd put it on the reading list once that's out.

Referee Report

3 major / 3 minor

Summary. The paper addresses an important open problem in speech language modeling: how well discrete tokens obtained from SSL representations via k-means clustering preserve prosodic information. It proposes a benchmark that measures prosodic encoding by the sensitivity of token sequences to artificially modified prosody, compares tokenizers built from different SSL models and cluster counts, and aims to distill practical design guidelines. The abstract is clear and the research question is well motivated. However, the supplied full text is heavily corrupted (mojibake), so the method definition, experimental setup, equations, tables, and results are not readable. As a result, the central empirical claims cannot currently be verified, and the key validity question — whether the sensitivity metric isolates prosody from segmental and alignment confounds — is left open.

Significance. If the benchmark is valid, it would fill a real gap: prosody encoding in discrete speech tokens is understudied, and practitioners currently choose SSL models and cluster counts heuristically. A cheap, reliable sensitivity-based proxy would be useful for tokenizer design and for speech-language-model pretraining. The paper's stated goal — actionable guidelines — is valuable, and the comparison across SSL models and cluster counts is a sensible axis. However, the contribution is conditional on the metric actually measuring prosody rather than generic acoustic perturbation, and on the results being statistically robust. The current manuscript does not yet provide enough evidence for either condition. No code, data, or machine-checked supplementary material is visible; reproducibility cannot be assessed.

major comments (3)

[Abstract; all Methods/Results sections (unreadable due to encoding)] The manuscript as supplied contains a coherent abstract followed by an unreadable garbled body. The central metric — 'sensitivity to the artificially modified prosody' — is never defined in readable text: no equation, normalization, or implementation is visible. This is load-bearing because the claimed practical guidelines ('use SSL model X with K clusters') depend on the exact definition of the metric. Please provide a clean manuscript with the metric definition, the precise prosody modification recipes (dimensions, magnitudes, alignment handling), and the result tables, with section and equation numbers that can be checked.
[Abstract, 'sensitivity to the artificially modified prosody'] The proxy is at risk of confounds: F0 and duration modifications typically also alter segmental acoustics and frame alignment. If the metric counts token-sequence changes, a tokenizer with lower cluster count or different frame rate may show high sensitivity merely because token boundaries shift or phoneme identities change, not because prosody is encoded. The condition that must hold is that token changes under prosody modification are prosody-specific. To support the claim, include control conditions (e.g., prosody-flattened resynthesis vs. spectral-envelope perturbation) and an alignment-invariant normalization; additionally validate the sensitivity score against a downstream task such as question/statement classification or emphasis detection. Without such evidence, the ranking of SSL models and cluster counts may measure acoustic perturbation sensitivity rather than usable prosodic
[Results (unreadable); all tables/figures] No error bars, significance tests, or variance estimates are visible. The benchmark is described as 'comprehensive', so the design guidelines need to be robust to k-means initialization seeds, utterance-level variation, and corpus choice. Please report means and standard errors over utterances and repeated clusterings, and show that the ranking differences between SSL models and cluster counts exceed within-condition variability. Without this, the guidelines could overfit noise in a small or homogeneous corpus.

minor comments (3)

[Abstract] Please specify which prosodic dimensions are manipulated (F0, duration, energy, etc.) and how the modification is applied. The phrase 'artificially modified prosody' is too vague for a benchmark paper.
[Abstract] The sentence 'this study conducts a comprehensive analysis focusing on prosodic encoding based on their sensitivity' has an unclear antecedent for 'their'. Rephrase as 'the sensitivity of discrete tokens to ...'.
[Throughout] The full text is rendered in corrupted encoding, making sections, equations, and tables unintelligible. Even putting aside the scientific content, the manuscript needs to be re-uploaded with proper font/encoding so that the technical content is accessible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical benchmark, not a derivation, and its conclusions rest on measured token sensitivities rather than on fitted inputs or self-citations.

full rationale

The abstract and the only readable portions of the manuscript describe a measurement study: discrete tokens are produced by k-means on SSL features, prosody is artificially modified, and token-sequence changes are measured. There is no claimed first-principles derivation, no parameter fitted to data and then relabeled as a prediction, and no load-bearing self-citation that would force the result. The sensitivity-based operationalization of 'prosodic encoding' is an empirical proxy; whether it fully isolates prosody from segmental/alignment confounds is a validity question, not a circularity. The garbled full text prevents checking individual equations, but no specific equation or passage exhibiting X derivable from Y by construction could be identified. Accordingly, no circular step is flagged.

Assumptions & free parameters 3 free parameters · 2 assumptions · 0 invented entities

The paper is an empirical benchmark; it introduces no new theoretical entities. The free parameters are experimental design choices that bound the applicability of the guidelines, and the axioms are the domain-specific assumptions that make the sensitivity measure meaningful.

free parameters (3)

SSL model choice set = not specified
The set of self-supervised models (for example, wav2vec2.0, HuBERT, WavLM) is chosen by hand; the benchmark's conclusions only apply to the tested models, and missing a model could bias the guidelines.
Cluster count range = not specified
The number of k-means clusters (for example, 50, 100, 500, 1000) is chosen by hand; sensitivity to prosody likely varies with codebook size, so the guidelines depend on this range.
Prosody modification method = not specified
The way prosody is artificially altered (pitch shift, duration scaling, and so on) is chosen by hand; different modifications may highlight different aspects of prosodic encoding.

assumptions (2)

domain assumption Sensitivity of token sequences to artificial prosody modification reflects the amount of prosodic information preserved in the discrete tokens.
This is the core assumption of the benchmark's validity; the abstract states the analysis is based on this sensitivity but does not justify the proxy relationship to downstream tasks.
domain assumption The tested SSL models and cluster counts are representative of the space of discrete token designs.
The guidelines are meant to be practical, so the generalization from the tested conditions to other designs is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Benchmarking Prosody Encoding in Discrete Speech Tokens." pith.science (2026). https://pith.science/paper/LMYVSNTK

@misc{pith2026250811224,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Prosody Encoding in Discrete Speech Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMYVSNTK}},
  note         = {Machine review of arXiv:2508.11224}
}

read the original abstract

Recently, discrete tokens derived from self-supervised learning (SSL) models via k-means clustering have been actively studied as pseudo-text in speech language models and as efficient intermediate representations for various tasks. However, these discrete tokens are typically learned in advance, separately from the training of language models or downstream tasks. As a result, choices related to discretization, such as the SSL model used or the number of clusters, must be made heuristically. In particular, speech language models are expected to understand and generate responses that reflect not only the semantic content but also prosodic features. Yet, there has been limited research on the ability of discrete tokens to capture prosodic information. To address this gap, this study conducts a comprehensive analysis focusing on prosodic encoding based on their sensitivity to the artificially modified prosody, aiming to provide practical guidelines for designing discrete tokens.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 41 canonical work pages

[1]

write newline

" write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write newline " =0pt " write newline " " ALTinterwordstretchfactor * " " * write newli...
[2]

q U -Z4H 6 z) Q! A@ A VX q f hCyZ] ꍏPP'&

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

arXiv 1981
[3]

W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hu BERT : Self-supervised speech representation learning by masked prediction of hidden units,'' T-ASLP, vol. 29, pp. 3451--3460, 2021

work page 2021
[4]

Baevski, Y

A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in NeurIPS 2020, vol. 33, 2020, pp. 12\,449--12\,460

work page 2020
[5]

S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, M. Zeng, and F. Wei, ``Wav LM : Large-scale self-supervised pre-training for full stack speech processing,'' IEEE Journal of Selected Topics in Signal Processing, vol. 16, pp. 1505--1518, 2021. [Online]. Available: https://ap...

work page 2021
[6]

Chung, Y

Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, ``w2v- BERT : Combining contrastive learning and masked language modeling for self-supervised speech pre-training,'' 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244--250, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:237048255

work page 2021
[7]

Mohamed, H.-y

A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maal^^c3^^b8e, T. N. Sainath, and S. Watanabe, ``Self-supervised speech representation learning: A review,'' IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179--1210, 2022

work page 2022
[8]

Y. Guo, Z. Li, H. Wang, B. Li, C. Shao, H. Zhang, C. Du, X. Chen, S. Liu, and K. Yu, ``Recent advances in discrete speech tokens: A review,'' 2025. [Online]. Available: https://arxiv.org/abs/2502.06490

arXiv 2025

Show all 49 references

[9]

Lakhotia, E

K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux, ``On generative spoken language modeling from raw audio,'' TACL, vol. 9, pp. 1336--1354, 2021

2021
[10]

Zhang, S

D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu, ``Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities,'' in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 15\,757--15\,773

2023
[11]

Borsos, R

Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, ``Audiolm: A language modeling approach to audio generation,'' IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 31, p. 2523^^e2^...

2023
[12]

P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov et al., ``Audiopalm: A large language model that can speak and listen,'' arXiv preprint arXiv:2306.12925, 2023

2023 arXiv
[13]

Arora, K.-W

S. Arora, K.-W. Chang, C.-M. Chien, Y. Peng, H. Wu, Y. Adi, E. Dupoux, H.-Y. Lee, K. Livescu, and S. Watanabe, ``On the landscape of spoken language models: A comprehensive survey,'' 2025. [Online]. Available: https://arxiv.org/abs/2504.08528

2025 arXiv
[14]

Chang, J

X. Chang, J. Shi, J. Tian, Y. Wu, Y. Tang, Y. Wu, S. Watanabe, Y. Adi, X. Chen, and Q. Jin, ``The I nterspeech 2024 challenge on speech processing using discrete units,'' in Interspeech 2024, 2024, pp. 2559--2563

2024
[15]

Chang, B

X. Chang, B. Yan, Y. Fujita, T. Maekaku, and S. Watanabe, ``Exploration of efficient end-to-end ASR using discretized input from self-supervised learning,'' in Interspeech 2023, 2023, pp. 1399--1403

2023
[16]

Chang, B

X. Chang, B. Yan, K. Choi, J.-W. Jung, Y. Lu, S. Maiti, R. Sharma, J. Shi, J. Tian, S. Watanabe, Y. Fujita, T. Maekaku, P. Guo, Y.-F. Cheng, P. Denisov, K. Saijo, and H.-H. Wang, ``Exploring speech recognition, translation, and understanding with discrete speech units: A compa...

2024
[17]

Y. Yang, F. Shen, C. Du, Z. Ma, K. Yu, D. Povey, and X. Chen, ``Towards universal speech discrete tokens: A case study for ASR and TTS ,'' in ICASSP 2024, 2024, pp. 10\,401--10\,405

2024
[18]

Mousavi, J

P. Mousavi, J. Duret, S. Zaiem, L. Della Libera , A. Ploujnikov, C. Subakan, and M. Ravanelli, ``How should we extract discrete audio tokens from self-supervised models?'' in Interspeech 2024, 2024, pp. 2554--2558

2024
[19]

J. Shi, X. Ma, H. Inaguma, A. Sun, and S. Watanabe, `` MMM : Multi-layer multi-residual multi-stream discrete speech representation from self-supervised learning model,'' in Interspeech 2024, 2024, pp. 2569--2573

2024
[20]

F. Shen, Y. Guo, C. Du, X. Chen, and K. Yu, ``Acoustic bpe for speech generation with discrete tokens,'' in ICASSP 2024, 2024, pp. 11\,746--11\,750

2024
[21]

Nakamura, K

T. Nakamura, K. Choi, K. Hojo, Y. Bando, S. Fukayama, and S. Watanabe, ``Discrete speech unit extraction via independent component analysis,'' in SALMA: Speech and Audio Language Models - Architectures, Data Sources, and Training Paradigms, IEEE International Conference on Aco...

2025
[22]

I. Gat, F. Kreuk, T. Anh Nguyen, A. Lee, J. Copet, G. Synnaeve, E. Dupoux, and Y. Adi, ``Augmentation invariant discrete representation for generative spoken language modeling,'' in Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), E...

2023
[23]

Vashishth, H

S. Vashishth, H. Singh, S. Bharadwaj, S. Ganapathy, C. Asawaroengchai, K. Audhkhasi, A. Rosenberg, A. Bapna, and B. Ramabhadran, ``Stab: Speech tokenizer assessment benchmark,'' 2024. [Online]. Available: https://arxiv.org/abs/2409.02384

2024 arXiv
[24]

Yeh and H

S.-L. Yeh and H. Tang, ``Estimating the completeness of discrete speech units,'' in 2024 IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 415--422

2024
[25]

Zeghidour, A

N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, ``Soundstream: An end-to-end neural audio codec,'' IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495--507, 2021

2021
[26]

D \'e fossez, J

A. D \'e fossez, J. Copet, G. Synnaeve, and Y. Adi, ``High fidelity neural audio compression,'' Transactions on Machine Learning Research, 2023, featured Certification, Reproducibility Certification. [Online]. Available: https://openreview.net/forum?id=ivCd8z8zR2

2023
[27]

J. Shi, J. Tian, Y. Wu, J.-w. Jung, J. Q. Yip, Y. Masuyama, W. Chen, Y. Wu, Y. Tang, M. Baali et al., ``Espnet-codec: Comprehensive training and evaluation of neural codecs for audio, music, and speech,'' in 2024 IEEE Spoken Language Technology Workshop (SLT). 1em plus 0.5em m...

2024
[28]

Wu, H.-L

H. Wu, H.-L. Chung, Y.-C. Lin, Y.-K. Wu, X. Chen, Y.-C. Pai, H.-H. Wang, K.-W. Chang, A. Liu, and H.-y. Lee, ``Codec- SUPERB : An in-depth analysis of sound codec models,'' in Findings of the Association for Computational Linguistics: ACL 2024, L.-W. Ku, A. Martins, and V. Sri...

2024
[29]

Y. Guo, Z. Li, J. Li, C. Du, H. Wang, S. Wang, X. Chen, and K. Yu, ``vec2wav 2.0: Advancing voice conversion via discrete token vocoders,'' arXiv preprint arXiv:2409.01995, 2024

2024 arXiv
[30]

Osakuade and S

O. Osakuade and S. King, ``Do discrete self-supervised representations of speech capture tone distinctions?'' arXiv preprint arXiv:2410.19935, 2024

2024 arXiv
[31]

Kharitonov, A

E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T. A. Nguyen, M. Riviere, A. Mohamed, E. Dupoux, and W.-N. Hsu, ``Text-free prosody-aware generative spoken language modeling,'' in Proceedings of the 60th Annual Meeting of the Association for Computational Ling...

2022
[32]

Sicherman and Y

A. Sicherman and Y. Adi, ``Analysing discrete self supervised speech representation for spoken language modeling,'' in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1--5

2023
[33]

de Seyssel , M

M. de Seyssel , M. Lavechin, H. Titeux, A. Thomas, G. Virlet, A. S. Revilla, G. Wisniewski, B. Ludusan, and E. Dupoux, ``Prosaudit, a prosodic benchmark for self-supervised speech models,'' in Interspeech 2023, 2023, pp. 2963--2967

2023
[34]

de Seyssel, A

M. de Seyssel, A. D ' Avirro, A. Williams, and E. Dupoux, `` E mph A ssess : a prosodic benchmark on assessing emphasis transfer in speech-to-speech models,'' in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, a...

2024
[35]

Maimon, A

G. Maimon, A. Roth, and Y. Adi, ``Salmon: A suite for acoustic language model evaluation,'' in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1em plus 0.5em minus 0.4em IEEE, 2025, pp. 1--5

2025
[36]

Morise, F

M. Morise, F. Yokomori, and K. Ozawa, ``World: a vocoder-based high-quality speech synthesis system for real-time applications,'' IEICE TRANSACTIONS on Information and Systems, vol. 99, no. 7, pp. 1877--1884, 2016

2016
[37]

T. A. Nguyen, E. Kharitonov, J. Copet, Y. Adi, W.-N. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, and E. Dupoux, ``Generative spoken dialogue language modeling,'' Transactions of the Association for Computational Linguistics, vol. 11, pp. 250--266, 2023. [...

2023
[38]

K. Qian, Y. Zhang, H. Gao, J. Ni, C.-I. Lai, D. Cox, M. Hasegawa-Johnson, and S. Chang, ``Contentvec: An improved self-supervised speech representation by disentangling speakers,'' in International conference on machine learning. 1em plus 0.5em minus 0.4em PMLR, 2022, pp. 18\,...

2022
[39]

Baevski, W.-N

A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, ``Data2vec: A general framework for self-supervised learning in speech, vision and language,'' in International conference on machine learning. 1em plus 0.5em minus 0.4em PMLR, 2022, pp. 1298--1312

2022
[40]

Z. Ma, W. Wu, Z. Zheng, Y. Guo, Q. Chen, S. Zhang, and X. Chen, ``Leveraging speech ptm, text llm, and emotional tts for speech emotion recognition,'' in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 11\,146--11\,150

2024
[41]

Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, ``emotion2vec: Self-supervised pre-training for speech emotion representation,'' in Findings of the Association for Computational Linguistics: ACL 2024, L.-W. Ku, A. Martins, and V. Srikumar, Eds. 1em plus 0.5em min...

2024
[42]

Panayotov, G

V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: An ASR corpus based on public domain audio books,'' in ICASSP 2015, 2015, pp. 5206--5210

2015
[43]

K. Onda, K. Imoto, S. Fukayama, D. Saito, and N. Minematsu, `` Discrete Tokens Exhibit Interlanguage Speech Intelligibility Benefit: an Analytical Study Towards Accent-robust ASR Only with Native Speech Data ,'' in Interspeech 2025 , 2025 , pp. 221--225

2025
[44]

K. Wang, Q. Wu, L. Song, Z. Yang, W. Wu, C. Qian, R. He, Y. Qiao, and C. C. Loy, ``Mead: A large-scale audio-visual dataset for emotional talking-face generation,'' in Computer Vision ^^e2^^80^^93 ECCV 2020: 16th European Conference, Glasgow, UK, August 23^^e2^^80^^9328, 2020,...

2020 doi
[45]

Kando, Y

S. Kando, Y. Miyao, and S. Takamichi, `` Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models ,'' in Interspeech 2025 , 2025 , pp. 5728--5732

2025
[46]

Poeppel, ``The analysis of speech in different temporal integration windows: cerebral lateralization as ‘asymmetric sampling in time’,'' Speech Communication, vol

D. Poeppel, ``The analysis of speech in different temporal integration windows: cerebral lateralization as ‘asymmetric sampling in time’,'' Speech Communication, vol. 41, no. 1, pp. 245--255, 2003, the Nature of Speech Perception. [Online]. Available: https://www.sciencedirect...

2003
[47]

Garofolo, L

J. Garofolo, L. Lamel, W. Fisher, J. Fiscus, D. Pallett, N. Dahlgren, and V. Zue, ``Timit acoustic-phonetic continuous speech corpus,'' Linguistic Data Consortium, 11 1992

1992
[48]

K. Onda, Y. Kashiwagi, E. Tsunoo, H. Futami, and S. Watanabe, `` Differentiable K-means for Fully-optimized Discrete Token-based ASR ,'' in Interspeech 2025 , 2025 , pp. 1223--1227

2025
[49]

J. W. Yoon, S. M. Kim, and N. S. Kim, ``Mcr-data2vec 2.0: Improving self-supervised speech pre-training via model-level consistency regularization,'' in Interspeech 2023, 2023, pp. 2833--2837

2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write newline " =0pt " write newline " " ALTinterwordstretchfactor * " " * write newli...

[2] [2]

q U -Z4H 6 z) Q! A@ A VX q f hCyZ] ꍏPP'&

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

arXiv 1981

[3] [3]

W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hu BERT : Self-supervised speech representation learning by masked prediction of hidden units,'' T-ASLP, vol. 29, pp. 3451--3460, 2021

work page 2021

[4] [4]

Baevski, Y

A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in NeurIPS 2020, vol. 33, 2020, pp. 12\,449--12\,460

work page 2020

[5] [5]

S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, M. Zeng, and F. Wei, ``Wav LM : Large-scale self-supervised pre-training for full stack speech processing,'' IEEE Journal of Selected Topics in Signal Processing, vol. 16, pp. 1505--1518, 2021. [Online]. Available: https://ap...

work page 2021

[6] [6]

Chung, Y

Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, ``w2v- BERT : Combining contrastive learning and masked language modeling for self-supervised speech pre-training,'' 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244--250, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:237048255

work page 2021

[7] [7]

Mohamed, H.-y

A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maal^^c3^^b8e, T. N. Sainath, and S. Watanabe, ``Self-supervised speech representation learning: A review,'' IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179--1210, 2022

work page 2022

[8] [8]

Y. Guo, Z. Li, H. Wang, B. Li, C. Shao, H. Zhang, C. Du, X. Chen, S. Liu, and K. Yu, ``Recent advances in discrete speech tokens: A review,'' 2025. [Online]. Available: https://arxiv.org/abs/2502.06490

arXiv 2025