Pith. sign in

REVIEW 3 major objections 4 minor 38 references

Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An 80-millisecond token width with a 16k vocabulary matches fine-grained tokens while halving training data.

desk verdict Useful empirical sweep with a solid compute-saving result; the small accuracy gain needs error bars before you trust the exact optimum. read the letter →

arxiv 2505.17446 v2 pith:I32UCBHI submitted 2025-05-23 cs.CL cs.SDeess.AS

Shunsuke Kando , Yusuke Miyao , Shinnosuke Takamichi This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords speechtokenizationlanguagemodelsspokenunderstandingdiscreteunitsK-meansclusteringsegmentationwidthvocabularysizezero-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how two dials of speech tokenization—the width of fixed segments cut from self-supervised speech features and the size of the K-means vocabulary used to quantize each segment—affect a speech language model's zero-shot understanding. Training OPT-based models on LibriSpeech across 64 combinations of segment width and cluster size, the authors find that moderately coarse segmentation (about 80 ms) paired with a large cluster size (about 16,384) gives the best average accuracy on five spoken language understanding benchmarks. The most efficient best-performing setting reduces training data by 50% and training runtime by 70% relative to the conventional fine-grained small-vocabulary setting, with a slight accuracy gain. Variable-width segmentation by phonemes, syllables, or words does not clearly beat fixed-width segmentation. The takeaway is that tokenization choices are a free-lunch lever: a coarser, larger-vocabulary tokenizer can save substantial compute without hurting downstream SLU performance.

What carries the argument

The machinery is a three-stage tokenization pipeline: HuBERT layer-9 features (one vector per 20 ms) are cut into fixed-width windows of $N$ ms, mean-pooled per window, and quantized by a K-means codebook of size $K$; the resulting token sequences are deduplicated and used to train a causal OPT language model. The load-bearing relationship is that $N$ and $K$ trade off against each other: as $N$ grows, the pooled features cover more phonetic content, so the codebook must grow to retain the distinctions that downstream tasks rely on. The paper maps this tradeoff by training 64 SLMs in an $8 \times 8$ grid of $N$ and $K$ values and evaluating each on zero-shot SLU benchmarks.

What would settle it

Train the same 64-configuration grid with a different SSL encoder (e.g., WavLM) or a substantially larger SLM and check whether the best average accuracy still clusters around $(80, 2^{13})$ and whether $(80, 2^{14})$ still matches or beats $(20, 2^7)$; a shift of the optimum to a different $(N, K)$ region would refute the claimed general guidance.

Watch

Extended reading notes

Core claim

The paper's central claim is that, for SLMs trained on HuBERT-derived discrete units, the optimal tokenization is not the fine-grained 20 ms unit with a small vocabulary but a moderately coarse fixed segment of 80 ms combined with a large cluster size around $2^{13}$ to $2^{14}$. In the head-to-head comparison, the setting $(80, 2^{14})$ trains on 42M tokens instead of 87M and runs in 8.3 hours instead of 12.4 hours compared to $(20, 2^7)$, while improving the average zero-shot SLU accuracy from 0.65 to 0.67. The authors also find that larger segment widths require larger vocabularies to keep distinct categories, and that different benchmarks favor different settings—e.g., tSC prefers roughly $(40, 2^8)$—so combining multiple tokenizations may be beneficial. Variable-width segmentation by predicted phonemes, syllables, and words shows no consistent advantage over fixed-width segmentation, suggesting simpler tokenizers are preferable.

Load-bearing premise

The conclusions rest on a single model scale (a 12-layer OPT) trained on a single SSL encoder (HuBERT layer 9); if the optimal segment width and vocabulary size shift when the model or encoder changes, the recommended setting may not generalize.

Editorial extensions

If this is right

  • Using $(80, 2^{14})$ instead of $(20, 2^7)$ cuts training tokens by half and training runtime by 70% while slightly improving average zero-shot SLU accuracy, so tokenization choice is a direct compute lever.
  • Because larger $N$ demands larger $K$, practitioners can expect that any segmentation coarsening should be accompanied by a codebook size increase to avoid losing fine distinctions.
  • The absence of a clear win for variable-width linguistic segmentation over fixed-width segmentation implies that fixed-window tokenizers remain a competitive default for SLM training.
  • Benchmark-specific optima (for example tSC near $(40, 2^8)$, quantifiers near $(160, 2^{12})$) indicate that a single tokenization cannot maximize all SLU skills, motivating multi-token or multi-resolution training.
  • These results are limited to zero-shot SLU; speech synthesis and speech continuation are left unexplored, so the recommended setting may not carry over to generation tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the $N$–$K$ coupling reflects the number of distinct phonetic categories a window can contain, then a useful heuristic generalizing the paper is to scale $K$ roughly with the expected phonetic diversity per segment, which could let practitioners set $K$ without a full grid search.
  • The efficiency gain would likely compound at larger model scales, since sequence-length reduction shrinks the quadratic attention cost; a direct extension would test $(80, 2^{14})$ against $(20, 2^7)$ with a larger OPT or LLM.
  • The fixed-versus-variable comparison used unsupervised segmenters whose errors may blur the comparison; an oracle or forced-aligned segmentation would isolate whether linguistically aligned segments ever beat fixed windows.
  • Because the paper uses HuBERT layer 9 only, the 'coarse is better' result might be specific to that layer's abstraction level; probing layers 6 through 12 would show how sensitive the optimal $(N, K)$ is to the SSL layer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper studies how two tokenization choices — the segmentation width N and the K-means cluster size K — affect zero-shot spoken language understanding in speech language models. The authors pool HuBERT layer-9 representations into N-ms segments, quantize them with K-means models of varying K, train OPT language models under matched hyperparameters with three random seeds, and evaluate on sBLIMP, sWUGGY, pros-syntax, pros-lexical, and tSC. They report that the best configurations are centered around (80, 2^13), that (80, 2^14) reduces training data by 50% and training runtime by 70% relative to (20, 2^7) while slightly improving average accuracy, that variable-width segmentation shows no clear advantage over fixed-width segmentation, and that different benchmarks favor different settings.

Significance. If the accuracy ordering is reliable, this is a practically useful result: moderately coarse fixed-width segmentation paired with a larger cluster size can reduce SLM training cost substantially without degrading zero-shot SLU performance. The experiment is a systematic, well-controlled sweep (64 configurations, matched model size, three seeds, public code), which is a strength. However, the headline 0.02 average-accuracy gain is reported without error bars or significance tests and is selected post hoc from the same evaluation accuracies, so the claimed 'positive effect' is not statistically established in the present write-up. The efficiency benefit itself is measured directly and is robust even if the accuracy comparison is treated as a tie.

major comments (3)
  1. [§3.2, §4, Table 3] The central accuracy comparison is not statistically supported. Section 3.2 states that each SLM is trained with three random seeds, and Section 4 reports only the mean; no standard deviations, confidence intervals, or significance tests are provided anywhere. The claim that (80, 2^14) 'slightly improves' over (20, 2^7) rests on averages of 0.67 vs 0.65 in Table 3, a 0.02 difference comparable to the differences between adjacent cells in Figure 2. Because Section 4 identifies the best K for each N from the same evaluation accuracies across all 64 configurations, the comparison is affected by multiple comparisons and selection bias. The inclusion of sBLIMP, which Section 5.2 reports as near chance for every setting, makes the average even harder to interpret. Please report per-seed and per-task results with error bars, apply a paired test across seeds or tasks, or explicitly reframe the positive-effect conclusion as an exploratory trend.
  2. [§4, Table 3, Figure 2] The claimed optimum is not stable. The text says the best configurations are 'centered around (80, 2^13)', but Table 3 lists (80, 2^14) as the best setting for N = 80; from Figure 2, (80, 2^13) and (80, 2^14) both round to 0.67 average accuracy, so the exact optimum is within rounding error. The authors should acknowledge this flat performance surface and avoid presenting a single (N, K) as the uniquely best configuration.
  3. [§3.2, §5.1] The generality of the main claim is untested. All results use HuBERT layer 9 as the SSL encoder and a single OPT scale (12 layers, embedding size 1024), so the conclusion that 'moderately coarse segmentation and bigger cluster size' are beneficial may be specific to this configuration rather than a general property of speech tokenization. The qualitative phoneme/morpheme analogy in Section 5.1 is a post-hoc explanation and does not provide independent evidence. Either add at least one additional encoder or model scale, or explicitly limit the conclusion to the studied setup.
minor comments (4)
  1. [Title and §5.1] The title contains an unintended space in 'V ocabulary', and Section 5.1 uses 'morphome', which appears to be a typo for 'morpheme'; please correct both.
  2. [§5.2] The phrase 'significantly above chance' is used without any statistical test or confidence interval; please either provide the test or rephrase to avoid a statistical claim.
  3. [Table 1 and Table 3] The notation for powers of two is inconsistent: Table 1 uses '27' and '214' where '2^7' and '2^14' are meant; please standardize the notation throughout.
  4. [Figure 2] The heatmap axis labels and colorbar are small and the vertical axis is not explicitly labeled as 'segment width N (ms)'; larger fonts and explicit axis titles would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical observations from a 64-configuration sweep evaluated on external zero-shot benchmarks, with no fitted parameter renamed as a prediction.

full rationale

The paper's derivation chain is an empirical sweep, not a reduction from assumptions. Section 2 defines tokenization as (N, K): fixed-width mean pooling followed by K-means clustering and deduplication; Section 3 fixes HuBERT layer 9 and OPT hyperparameters matched to GSLM; Section 4 reports measured accuracies on five external zero-shot SLU benchmarks (sBLIMP, sWUGGY, pros-syntax, pros-lexical, tSC). The claim that 'best-performing configurations are centered around (80, 2^13)' and that (80, 2^14) reduces training data and runtime is a direct summary of measured grid results, not an output derived from its own definition. The phoneme/morpheme analogy in Section 5.1 is explicitly qualitative ('To discuss it qualitatively') and comes after the results; it is not used to derive or predict the optimum. Variable segmentation uses external tools (UnsupSeg, Sylber, GradSeg) and is compared against fixed-width segmentation, again empirically. No parameter is fitted to a subset and then reported as a prediction of closely related data: K-means models are trained on 100 hours of LibriSpeech and SLMs are evaluated on held-out benchmark pairs, not on the tokenizer training data. No load-bearing self-citation exists; the author team's own prior work is not used to justify the central premise. Concerns about missing error bars, multiple comparisons, and post-hoc selection of the best K are statistical-reliability issues, not circularity, because the reported optimum is a description of the observed grid rather than a prediction forced by construction. Therefore no circular step meets the quoted-reduction standard, and the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted in this empirical study; N and K are the experimental variables. The central claim rests on four domain assumptions: the fixed choice of HuBERT layer 9, the adequacy of mean pooling, the validity of the zero-shot SLU benchmarks, and the transferability of GSLM hyperparameters across all tokenizer configurations. No new entities are introduced.

assumptions (4)
  • domain assumption HuBERT layer-9 representations encode sufficient phonetic and prosodic information for mean-pooled tokenization to support SLU
    Section 2 fixes layer 9 as the sole SSL feature source; the entire study's conclusions are conditional on this encoder/layer choice.
  • domain assumption Mean pooling within N ms segments preserves the information needed for the five zero-shot SLU tasks
    Section 2 replaces each segment with the mean of its HuBERT frames; ordering information within a segment is discarded, yet the paper treats this as an innocuous aggregation for all N up to 280 ms.
  • domain assumption Zero-shot pairwise likelihood benchmarks (sBLIMP, sWUGGY, prosaudit, tSC) are valid proxies for spoken language understanding
    Section 3.3; the paper relies on these benchmarks, and notably sBLIMP is at chance across all settings while sSC was dropped for near-chance performance, weakening the breadth of this assumption.
  • domain assumption OPT hyperparameters copied from GSLM are appropriate for all 64 tokenization configurations
    Section 3.2 states hyperparameters were tuned to match GSLM; no per-configuration tuning is reported, so interactions between tokenizer statistics and LM hyperparameters are unexamined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models." pith.science (2026). https://pith.science/paper/I32UCBHI

@misc{pith2026250517446,
  author       = {Pith},
  title        = {Pith review of: Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I32UCBHI}},
  note         = {Machine review of arXiv:2505.17446}
}
read the original abstract

The purpose of speech tokenization is to transform a speech signal into a sequence of discrete representations, serving as the foundation for speech language models (SLMs). While speech tokenization has many options, their effect on the performance of SLMs remains unclear. This paper investigates two key aspects of speech tokenization: the segmentation width and the cluster size of discrete units. First, we segment speech signals into fixed/variable widths and pooled representations. We then train K-means models in multiple cluster sizes. Through the evaluation on zero-shot spoken language understanding benchmarks, we find the positive effect of moderately coarse segmentation and bigger cluster size. Notably, among the best-performing models, the most efficient one achieves a 50% reduction in training data and a 70% decrease in training runtime. Our analysis highlights the importance of combining multiple tokens to enhance fine-grained spoken language understanding.

Figures

Figures reproduced from arXiv: 2505.17446 by the authors.

Figure 1
Figure 1. Overview of our research. First, we extract contin￾uous speech representation from the SSL model. We add seg￾ments to the representation sequence by N ms and pooled them. We apply K-means clustering to pooled representations with the cluster size of K. By training SLMs in multiple settings of N and K, we explore the optimal choice for spoken language un￾derstanding. segment input speech into fixed or variable-width … view at source ↗
Figure 2
Figure 2. Main results of SLM performance on zero-shot SLU tasks [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example from sWUGGY where (20, 2 7 ) and (80, 2 14) can solve but (80, 2 7 ) fails. Differences in phoneme or unit are shown in bold. The first row shows the actual stim￾uli from the dataset and the rest shows unit sequences. Since the dataset does not include phonetic alignments, we annotated them by ourselves using Praat [22]. tween phoneme and morphome: combining a small number of phonemes produces a large num… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Results of variable segmentation on phoneme, syllable, and word levels. For comparison, we show the fixed width segmenta￾tion results of which N is the same as median of the distribution of variable segmentation. 2 7 2 8 2 9 2 10 2 11 2 12 2 13 2 14 clusters (K) 280 24…
Figure 5
Figure 5. Figure 5: sBLIMP accuracies split by task type. We show two results (ellipsis and quantifiers) which display unique tendency. might be solvable to some extent. We split the accuracy accord￾ing to the task and found that it is actually the case. We show two examples in [PITH_FUL…
Figure 2
Figure 2. Figure 2: This finding highlights the importance of combining [PITH_FULL_IMAGE:figures/full_fig_p004_2.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 35 canonical work pages

  1. [1]

    Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models

    Introduction With the recent breakthroughs in large language models for textual natural language processing, speech language models (SLMs) have emerged as a new paradigm for spoken language processing [1–4]. SLMs are built by training language mod- els on top of discrete speech representations (called “discrete units”). The process of converting a speech ...

    work page Pith review arXiv 2025

  2. [2]

    Throughout this study, we used HuBERT [7] as an SSL model and extracted representations from the ninth layer

    Tokenization Methods to be Explored As depicted in Figure 1, we first extract continuous speech rep- resentations of input speech using the SSL model. Throughout this study, we used HuBERT [7] as an SSL model and extracted representations from the ninth layer. On top of this representa- tion sequence, we performed speech tokenization in three steps

    work page

  3. [3]

    Segment a sequence by N ms and apply mean pooling

    work page

  4. [4]

    Apply K-means clustering with the cluster size K

    work page

  5. [5]

    Deduplicate units. (e.g. 54 54 54 88 88 3 → 54 88 3) Since each HuBERT representation corresponds to 20 ms speech, N is chosen as a multiple of 20. We experiment with eight values: {20, 40, 80, 120, 160, 200, 240, 280 }, where N = 20 corresponds to the original sequence. As N becomes larger, the resulting sequence length becomes shorter by the fac- tor of...

    work page

  6. [6]

    Dataset As a training set for SLM, we used LibriSpeech [17], a 960-hour English audiobook corpus

    Experimental Setup 3.1. Dataset As a training set for SLM, we used LibriSpeech [17], a 960-hour English audiobook corpus. Although this dataset is relatively small for SLM studies, our preliminary experiments showed that using a larger dataset (LibriLight [18]; 60k hours audio- book corpus) did not lead to performance improvements. A re- cent study on SLM...

    work page

  7. [7]

    Figure 2 shows results on fixed boundary settings

    Main Result For simplicity, we denote the configuration with segment width N and cluster size K as (N, K). Figure 2 shows results on fixed boundary settings. We observe that the best-performing configurations are centered around (80, 213). An exception is tSC, where the optimal set- ting appears to be around (40, 28) (if any), though the differ- ences in ...

    work page

  8. [8]

    yonder"

    Analysis 5.1. Effect of Larger N and K Overall, for larger N, accuracy tends to improve with increas- ing K. This observation is analogous to the relationship be- "yonder" "zonder" Y Z 0 20 40 60 80 100 120 140 160 180 200 220 240 (ms) AA AA ✔ (20, 27) 17 91 77 77 45 45 118 88 88 100100100 17 80 40 40 22 118 118 88 88 100100100 ✗ (80, 27) 28 54 32 37 54 3...

    work page 2016

Show all 38 references
  1. [9]

    We conducted mul- tiple speech tokenizations based on the combination of the fixed/variable segmentation and the cluster size

    Conclusion In this research, we explored the effect of speech tokeniza- tion on the SLU capabilities of SLMs. We conducted mul- tiple speech tokenizations based on the combination of the fixed/variable segmentation and the cluster size. Our experi- ment on fixed-width segmenta...

  2. [10]

    Acknowledgements This work was supported by JST ACT-X JPMJAX24C9

  3. [11]

    On Generative Spoken Language Modeling from Raw Audio,

    K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y . Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux, “On Generative Spoken Language Modeling from Raw Audio,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336–1354, 2021

    2021

  4. [12]

    Textually Pretrained Speech Language Models,

    M. Hassid, T. Remez, T. A. Nguyen, I. Gat, A. Conneau, F. Kreuk, J. Copet, A. Defossez, G. Synnaeve, E. Dupoux, R. Schwartz, and Y . Adi, “Textually Pretrained Speech Language Models,” in NeurIPS, Dec. 2023

    2023

  5. [13]

    Audiolm: A language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, “Audiolm: A language modeling approach to audio generation,” IEEE ACM Trans. Audio Speech Lang. Pro- cess., vol. 31, pp. 2523–2533,...

    2023

  6. [14]

    WavLLM: Towards Robust and Adaptive Speech Large Language Model,

    S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran, L. Liu, and F. Wei, “WavLLM: Towards Robust and Adaptive Speech Large Language Model,” inFindings of the Association for Computational Linguistics: EMNLP 2024 , Y . Al-Onaizan, M. Bansal,...

    2024

  7. [15]

    Representation Learning with Contrastive Predictive Coding,

    A. van den Oord, Y . Li, and O. Vinyals, “Representation Learning with Contrastive Predictive Coding,” Jan. 2019

    2019

  8. [16]

    Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Represen- tations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Represen- tations,” in Advances in Neural Information Processing Systems , vol. 33. Curran Associates, Inc., 2020, pp. 12 449–12 460

    2020

  9. [17]

    HuBERT: Self-Supervised Speech Rep- resentation Learning by Masked Prediction of Hidden Units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “HuBERT: Self-Supervised Speech Rep- resentation Learning by Masked Prediction of Hidden Units,” IEEE/ACM Transactions on Audio, Speech and Language Pro- cessing, vol. 29, pp. 3451–3460, Oct. 2021

    2021

  10. [18]

    The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,

    T. A. Nguyen, M. de Seyssel, P. Roz ´e, M. Rivi`ere, E. Kharitonov, A. Baevski, E. Dunbar, and E. Dupoux, “The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,” Dec. 2020

    2021

  11. [19]

    Generative Spoken Dialogue Language Model- ing,

    T. A. Nguyen, E. Kharitonov, J. Copet, Y . Adi, W.-N. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, and E. Dupoux, “Generative Spoken Dialogue Language Model- ing,” Transactions of the Association for Computational Linguis- tics, vol. 11, pp. 250–266, 2023

    2023

  12. [20]

    Direct Speech- to-Speech Translation With Discrete Units,

    A. Lee, P.-J. Chen, C. Wang, J. Gu, S. Popuri, X. Ma, A. Polyak, Y . Adi, Q. He, Y . Tang, J. Pino, and W.-N. Hsu, “Direct Speech- to-Speech Translation With Discrete Units,” inProceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1:...

    2022

  13. [21]

    Text-free prosody-aware generative spoken language modeling,

    E. Kharitonov, A. Lee, A. Polyak, Y . Adi, J. Copet, K. Lakhotia, T. A. Nguyen, M. Rivi`ere, A. Mohamed, E. Dupoux, and W. Hsu, “Text-free prosody-aware generative spoken language modeling,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguis...

    2022

  14. [22]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems 2017, I. Guyon, U. von Luxbu...

    2017

  15. [23]

    Self-Supervised Speech Representations are More Phonetic than Semantic,

    K. Choi, A. Pasad, T. Nakamura, S. Fukayama, K. Livescu, and S. Watanabe, “Self-Supervised Speech Representations are More Phonetic than Semantic,” in Proc. Interspeech, 2024, pp. 4578– 4582

    2024

  16. [24]

    Generative Spoken Language Model based on continuous word-sized audio tokens,

    R. Algayres, Y . Adi, T. Nguyen, J. Copet, G. Synnaeve, B. Sagot, and E. Dupoux, “Generative Spoken Language Model based on continuous word-sized audio tokens,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, H. Bouamor, J. Pino, an...

    2023

  17. [25]

    SyllableLM: Learn- ing Coarse Semantic Units for Speech Language Models,

    A. Baade, P. Peng, and D. Harwath, “SyllableLM: Learn- ing Coarse Semantic Units for Speech Language Models,” https://arxiv.org/abs/2410.04029v1, Oct. 2024

    2024 arXiv

  18. [26]

    Sylber: Syllabic Embedding Repre- sentation of Speech from Raw Audio,

    C. J. Cho, N. Lee, A. Gupta, D. Agarwal, E. Chen, A. W. Black, and G. K. Anumanchipalli, “Sylber: Syllabic Embedding Repre- sentation of Speech from Raw Audio,” Oct. 2024

    2024

  19. [27]

    Lib- rispeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An ASR corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015. IEEE, 2015, pp. 5206–5210

    2015

  20. [28]

    Libri-light: A benchmark for ASR with limited or no super- vision,

    J. Kahn, M. Rivi `ere, W. Zheng, E. Kharitonov, Q. Xu, P. Mazar´e, J. Karadayi, V . Liptchinsky, R. Collobert, C. Fuegen, T. Likhoma- nenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, “Libri-light: A benchmark for ASR with limited or no super- vision,” in 2020 IEEE Int...

    2020

  21. [29]

    OPT: open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. T. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “OPT: open pre-trained transformer language models,” CoRR, vol. a...

    2022 arXiv

  22. [30]

    ProsAudit, a prosodic benchmark for self-supervised speech models,

    M. De Seyssel, M. Lavechin, H. Titeux, A. Thomas, G. Vir- let, A. S. Revilla, G. Wisniewski, B. Ludusan, and E. Dupoux, “ProsAudit, a prosodic benchmark for self-supervised speech models,” in INTERSPEECH 2023. ISCA, Aug. 2023, pp. 2963– 2967

    2023

  23. [31]

    A corpus and cloze evaluation for deeper understanding of commonsense stories,

    N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. F. Allen, “A corpus and cloze evaluation for deeper understanding of commonsense stories,” in NAACL HLT 2016, K. Knight, A. Nenkova, and O. Rambow, Eds. The Association for Computational...

    2016

  24. [32]

    Praat: doing phonetics by com- puter [computer program]. version 6.4.27,

    P. Boersma and D. Weenink, “Praat: doing phonetics by com- puter [computer program]. version 6.4.27,” http://www.praat.org/, retrieved 27 January 2025

    2025

  25. [33]

    Martinet, Elements of General Linguistics, ser

    A. Martinet, Elements of General Linguistics, ser. Phoenix books. University of Chicago Press, 1966

    1966

  26. [34]

    Are Discrete Units Nec- essary for Spoken Language Modeling?

    T. A. Nguyen, B. Sagot, and E. Dupoux, “Are Discrete Units Nec- essary for Spoken Language Modeling?” IEEE Journal of Se- lected Topics in Signal Processing, vol. 16, no. 6, pp. 1415–1423, Oct. 2022

    2022

  27. [35]

    Spirit- lm: Interleaved spoken and written language model,

    T. A. Nguyen, B. Muller, B. Yu, M. R. Costa-juss `a, M. El- bayad, S. Popuri, P. Duquenne, R. Algayres, R. Mavlyutov, I. Gat, G. Synnaeve, J. Pino, B. Sagot, and E. Dupoux, “Spirit- lm: Interleaved spoken and written language model,” CoRR, vol. abs/2402.05755, 2024

    2024 arXiv

  28. [36]

    Multi- resolution hubert: Multi-resolution speech self-supervised learn- ing with masked unit prediction,

    J. Shi, H. Inaguma, X. Ma, I. Kulikov, and A. Y . Sun, “Multi- resolution hubert: Multi-resolution speech self-supervised learn- ing with masked unit prediction,” in The Twelfth International Conference on Learning Representations . OpenReview.net, 2024

    2024

  29. [37]

    Self-supervised contrastive learning for unsupervised phoneme segmentation,

    F. Kreuk, J. Keshet, and Y . Adi, “Self-supervised contrastive learning for unsupervised phoneme segmentation,” in Proc. In- terspeech, H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, pp. 3700–3704

    2020

  30. [38]

    Unsupervised word segmentation using temporal gradient pseudo-labels,

    T. S. Fuchs and Y . Hoshen, “Unsupervised word segmentation using temporal gradient pseudo-labels,” in IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP. IEEE, 2023, pp. 1–5

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.