Pith. sign in

REVIEW 3 major objections 6 minor 41 references

Differentiable K-means for Fully-optimized Discrete Token-based ASR

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Integrating differentiable k-means into discrete-token ASR lets the tokenizer, SSL encoder, and layer weights be trained jointly on the ASR objective, improving accuracy and phonetic purity.

desk verdict A promising but underspecified application of concrete k-means to discrete-token ASR: the WER gains are real and worth refereeing, but the paper never clarifies whether training is actually discrete, and alpha=0 leaves the k-means reconstruction objective out entirely. read the letter →

arxiv 2505.16207 v1 pith:HFOH2XG5 submitted 2025-05-22 cs.SD eess.AS

Kentaro Onda , Yosuke Kashiwagi , Emiru Tsunoo , Hayato Futami , Shinji Watanabe This is my paper · ORCID
classification cs.SDeess.AS
keywords differentiablek-meansdiscretespeechtokensself-supervisedlearningautomaticrecognitiontokenizationlayerweightingresynthesisphoneticinformation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that discrete speech tokens for ASR should be trained, not just clustered: replacing ordinary arg-min k-means with a differentiable version lets the tokenizer, the SSL encoder, and the layer-fusion weights all be optimized jointly against the ASR loss. On LibriSpeech-100, word error rate improves at every cluster size tested, with the biggest relative gains at 100 clusters (7.6/13.2 down to 5.4/9.2 for a single SSL layer). The learned tokens also carry cleaner phonetic information, measured by lower mean token error rate across speakers and shorter deduplicated sequences, and they improve resynthesis intelligibility. A careful reader would care because this points toward compact, task-specific discrete units that keep the efficiency of tokenization while narrowing the accuracy gap with continuous SSL features.

What carries the argument

The load-bearing object is the differentiable k-means operator: soft cluster-assignment probabilities $p(j \mid s_i) = \exp(-\sigma^2 \|s_i - \mu_j\|^2) \,/\, \sum_c \exp(-\sigma^2 \|s_i - \mu_c\|^2)$, sampled with Gumbel-Softmax at temperature $\tau$, so that the discrete argmax has a continuous surrogate for gradients. The total loss $L = L_{\mathrm{ASR}}(\mathrm{ASR}(\mathrm{DiffKM}(\mathrm{SSL}(X)), M), \theta_{\mathrm{asr}}) + \alpha L_{\mathrm{km}}$, with $\alpha=0$ in all reported experiments, routes the ASR error through the assignments to update the SSL parameters $\theta_{\mathrm{ssl}}$, centroids $M$, and in the multi-layer variant the weights $w$ over SSL layers. The k-means loss $L_{\mathrm{km}}$ is defined but deliberately unused, which makes the experiment a test of whether the ASR signal alone can shape the clusters.

What would settle it

During the full-finetune run on LibriSpeech-100 with 100 clusters, compute the mean entropy of the Gumbel-Softmax assignment distribution per frame and the fraction of centroids that receive at least one argmax assignment at the end of training; if the distribution is near-uniform or many centroids are dead, the reported gains cannot be attributed to the differentiable clustering mechanism.

Watch

Extended reading notes

Core claim

The central claim is that replacing standard arg-min k-means tokenization with a Gumbel-Softmax differentiable k-means operator lets the ASR objective backpropagate through the discrete assignment into the SSL encoder, the cluster centroids, and the layer weights, so the token vocabulary itself is optimized for recognition rather than for reconstruction fidelity. With the k-means reconstruction loss weighted at zero, full fine-tuning of the SSL model improves LibriSpeech-100 WER from 7.6/13.2 to 5.4/9.2 (single layer, 100 clusters) and from 8.4/18.3 to 5.3/8.2 (multi-layer, 100 clusters). The resulting tokens have lower mean token error rate across speakers who say the same sentence, shorter deduplicated token sequences, and better out-of-domain resynthesis intelligibility, which the paper reads as evidence that the tokens converge toward phoneme-level content.

Load-bearing premise

The load-bearing premise is that, with the reconstruction loss switched off ($\alpha=0$) and $\sigma^2$ fixed at 1.0, the Gumbel-Softmax soft assignments still pass gradients that push the SSL encoder and centroids toward useful phonetic structure instead of collapsing or staying too uniform.

Editorial extensions

If this is right

  • At 100 clusters, full fine-tuning brings single-layer WER from 7.6/13.2 to 5.4/9.2, and performance becomes nearly flat from 500 to 2000 clusters, so compact token vocabularies stop costing accuracy.
  • Multi-layer fusion with full fine-tuning reaches 5.3/8.2 WER, approaching the continuous-feature system (3.7/6.7), so discrete tokens narrow the gap with continuous SSL features.
  • TIMIT evaluation shows lower mean token error rate (28.8% vs 42.2% at 100 clusters) and shorter token sequences, indicating increased robustness to speaker and pronunciation variation.
  • Resynthesis with the optimized 100-cluster tokens improves out-of-domain WER from 31.4% to 6.1%, suggesting the units retain linguistic content needed for generation.
  • Improvements also hold on LibriSpeech-960 and the multilingual ML-SUPERB benchmark, so the effect is not limited to the small English corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors mention only in passing: the same differentiable tokenizer could be driven by other objectives, such as speech translation or TTS intelligibility, to shape tokens for those tasks.
  • Because freezing the SSL encoder while updating only centroids gives little gain, the paper's comparison suggests most of the benefit comes from fine-tuning the encoder itself; a straight-through hard-assignment baseline would isolate the contribution of the soft differentiable mechanism.
  • Setting $\alpha=0$ works better than including a reconstruction loss, which implies that clustering objectives can actively hurt downstream-shaped tokenization; sweeping $\alpha$ would make this tradeoff quantitative.
  • Task-optimized small vocabularies could be especially useful for LLM-based speech models, where shorter token sequences directly reduce memory and decoding cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes replacing the standard non-differentiable k-means tokenizer in discrete-token ASR with a differentiable variant built on the concrete/Gumbel-Softmax k-means of [15]. The tokenizer layer computes soft assignments (Eq. (1)) and Gumbel-Softmax samples (Eq. (2)), through which the ASR cross-entropy loss (and optionally a k-means reconstruction loss, Eq. (3)) can backpropagate into the SSL encoder parameters θ_ssl, the centroids M, and the layer-weighting vector w (Eqs. (4)-(6)). Experiments use WavLM-large as the SSL model and a joint CTC/attention encoder-decoder ASR in ESPnet, comparing three update regimes: ASR-only (baseline), ASR+centroids (freeze SSL), and ASR+centroids+SSL (full finetune), with α=0 so that the reconstruction loss is disabled. On LibriSpeech-100, full finetune improves WER from 7.6/13.2 to 5.4/9.2 (k=100, single layer) and from 8.4/18.3 to 5.3/8.2 (k=100, multi-layer), approaching the continuous-feature result of 3.7/6.6. Token analyses on TIMIT show lower TSL and MTER (interpreted as more phoneme-like, speaker-robust tokens) and higher NQE, with multi-layer tokens also improving PNMI. Resynthesis experiments with Tacotron2 show better content preservation (OOD WER 31.4 to 6.1 at k=100 multi-layer), and gains are reported on LibriSpeech-960 and ML-SUPERB.

Significance. If the mechanism is as claimed, the paper makes a useful step: it shows how to carry gradients from an ASR loss through a discrete-equivalent tokenization into a large SSL encoder, with learned layer weighting, and it provides evidence that jointly optimized tokens approach continuous-feature performance while remaining discrete, compact, and usable for generation. The evaluation is broad by the standards of the area, covering held-out test sets (LibriSpeech test-clean/other, TIMIT, LJSpeech/VCTK, ML-SUPERB), multiple metrics (WER, PNMI, NQE, TSL, MTER, MCD, F0 RMSE, UTMOS), and a clean baseline/freeze/full ablation that separates which parameters are updated. The paper builds on the public ESPnet toolkit and standard checkpoints, so the experiments are in principle reproducible, and the headline gains at k=100 are substantial and practically meaningful if they hold. The main issues requiring revision are methodological rather than existential: the training-time forward pass through the tokenizer is underspecified (hard vs.

major comments (3)
  1. [§2.2, Eqs. (1)–(2); §3.1.] The forward computation of the tokenizer is never defined, and this is load-bearing for the central claim of training a discrete token-based ASR. Eq. (2) outputs a continuous vector h_i, and the paper never states whether the ASR encoder consumes argmax h_i (token indices) or the soft vector h_i (e.g., an expectation over token embeddings) during training and inference. The sentence in §2.2 that Gumbel-Softmax "produces discrete outputs, i.e., arg max h_i" conflates the finite-temperature relaxation with its limit; at the initial temperature τ=2.0 the outputs are not one-hot. If the ASR consumes soft mixtures, the system is trained as a continuous-input model with a learned soft codebook, and the WER gains could come from the added flexibility of soft mixtures rather than from better discrete units; if the ASR consumes argmax indices, a straight-through estimator is silently required and Eq. (2) is not the distribution actually differentiated by the ASR loss. Section 3.1 reports only that τ was initialized to 2.0 and "gradually decreased," with no schedule, no final value, and no rule for producing the discrete tokens used at evaluation time (needed for the TIMIT clustering metrics and for the deduplicated token sequences used in resynthesis). Please specify the training- and inference-time forward passes, the temperature schedule, and the sampling/argmax rule for each experiment, and include some analysis of assignment sharpness (e.g., the entropy of h_i) to confirm that the trained tokenizer produces near-discrete assignments at the operating point.
  2. [§3.1 (α=0), §3.2, Table 1.] The paper does not isolate the contribution of the differentiable k-means formulation from the contribution of SSL fine-tuning. With α=0 (§3.1), the k-means reconstruction loss of Eq. (3) is never used, so the tokenizer is a distance-based soft quantizer driven solely by the ASR loss. Table 1 shows that updating only the centroids (freeze SSL) yields no gain over baseline at any cluster size (e.g., 7.3/12.6 vs. 7.6/13.2 at k=100), while essentially all of the gains appear when θ_ssl is fine-tuned. Given the claim that tokenization is "jointly optimized," the present experiments do not show whether the specific k-means structure of Eq. (1) matters for the reported gains. Please add (i) an ablation with α>0, (ii) a full-finetune condition with the centroids M frozen, and (iii) a comparison against an alternative differentiable quantizer (e.g., straight-through VQ-VAE-style codebook or a linear soft codebook) under otherwise identical SSL fine-tuning.
  3. [§3.2; Tables 1–5.] All reported results appear to be single runs without error bars or significance tests, yet the text makes equality and inequality claims that require them. In particular, §3.2 states that "there was no significant difference in performance between cluster sizes of 500, 1000, and 2000," and Table 1 contains baseline/freeze SSL differences as small as 0.1–0.4 WER, which is within typical run-to-run variation for these settings. For the headline tables, please report variance over multiple training seeds (or an equivalent estimate) and either significance tests or reworded claims for the "no difference" statements.
minor comments (6)
  1. [§3.4, Table 3.] At k=100, single-layer full finetune decreases PNMI from 0.4754 to 0.4561, yet the text describes the single-layer values as "almost the same"; please state this decrease explicitly and discuss whether it qualifies the phonetic-purity claim, which is currently carried by the multi-layer PNMI and by the TSL/MTER results.
  2. [§3.5, Table 4.] In Table 4 at k=2000, the OOD resynthesis WER slightly worsens under single-layer full finetune (14.0 to 14.9) and the ID WER is flat (2.7 to 2.6); the discussion of the "most balanced" configuration highlights only the k=100 rows and should also comment on this regression.
  3. [§3.3.] The single-layer full-finetune condition updates θ_ssl for 60 epochs, whereas the multi-layer condition updates it only for the last 30 epochs, so the comparison between single- and multi-layer results also varies the training schedule; the stated rationale for the 30-epoch choice ("w was effectively tuned") is asserted without supporting ablation data.
  4. [§3.3, §3.5, Eq. (5).] The claim that multi-layer full finetuning "can condense this information into a single layer" is not substantiated by any report of the learned layer weights w; showing w, or its evolution over training, would make the claim verifiable.
  5. [§3.1, §3.4, §3.6.] Several experimental definitions are incomplete: for MTER it is not stated whether token sequences are deduplicated before computing pairwise TER (TSL is defined on deduplicated sequences, so the relation between the two metrics should be clarified); the ML-SUPERB evaluation in §3.6 is described in one sentence with no details of the subset, recipe, or definition of "normal/few-shot"; and σ²=1.0 in Eq. (1) is given no sensitivity analysis despite high-dimensional WavLM features.
  6. [Throughout.] Typos and formatting issues to correct: "LibriSpecch-960" in the Table 5 caption, "freeeze SSL" in §3.3, "custer sizes" in §3.4, the inline "baseline) θasr, freeze SSL) θasr, w, and M, or full finetune) ..." enumeration in §3.3, and "pre-trained on the train clean 100" in §3.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: all central claims are validated against held-out external corpora and standard baselines, and the differentiable-k-means formulation is inherited from an external prior work rather than defined in terms of the conclusions.

full rationale

The paper's derivation chain is: Equ. (1) defines a soft assignment from SSL features to centroids, Equ. (2) applies Gumbel-Softmax sampling, Equ. (4) combines the ASR cross-entropy loss with a k-means reconstruction loss weighted by alpha, and Equ. (6) adds learnable layer weights. The paper explicitly sets alpha=0 and reports that a smaller alpha gave better ASR performance, so the final system is trained by the ASR objective through the differentiable assignments; there is no reconstruction loss being renamed as a prediction. The central claims (WER improvement, phonetic purity, and resynthesis quality) are all measured on held-out test sets: LibriSpeech test-clean/test-other, TIMIT, LJSpeech, VCTK, and ML-SUPERB. None of these external targets is inserted into Equ. (4) as a fitted quantity, so the improvements are not forced by construction. The k-means reconstruction loss Lkm is absent at alpha=0, and the paper honestly reports that NQE increases as a result, which further shows the reported gains are not a restatement of quantization error minimization. References [11,12,17,18,19,24,34] include authors who overlap with the present paper, but they are used for experimental configuration, layer selection, and as discrete-token baseline systems; they do not supply the mathematical derivation (which is credited to the external work [15]) and they are not invoked as a uniqueness theorem or as proof of the central claim. Thus no self-citation chain forces the result. The main caveat in the paper is an unspecified forward-pass detail: it does not clearly state whether the ASR consumes hard argmax tokens or the soft vectors h_i from Equ. (2), and alpha=0 means no explicit quantization objective is present. That is a completeness and correctness ambiguity about what exactly is optimized, but it is not circularity under the definition used here, because no equation in the paper reduces the claimed outcome to its own input by construction.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim depends on a small set of hyperparameters (cluster size, sigma, tau, alpha, schedule) and on several domain assumptions about SSL features and the metrics used to measure phonetic purity. No new entities are introduced. The most fragile choice is setting alpha=0, which removes the reconstruction loss and leaves the tokenizer to be shaped by the ASR objective alone.

free parameters (6)
  • Cluster size k = 100, 500, 1000, 2000 in LibriSpeech-100; 2000 in LibriSpeech-960 and ML-SUPERB
    Chosen per experiment; affects WER and token property metrics, and the reported advantage at small k is part of the central claim.
  • sigma^2 in Eq. (1) = 1.0 (fixed)
    Controls sharpness of the soft assignment probabilities; not learned or ablated.
  • Temperature tau in Eq. (2) = Initialized to 2.0, decreased during training; schedule not specified
    Controls the Gumbel-Softmax relaxation sharpness; the annealing schedule is a critical detail for the discrete approximation but is not reported.
  • alpha in Eq. (4) = 0
    The k-means reconstruction loss is completely disabled; chosen based on preliminary experiments, so the tokenizer is trained only by the ASR objective. This is a strong design choice that shapes all reported results.
  • Training schedule (epochs and phase boundaries) = 90 epochs total; first 30 freeze SSL and centroids; multi-layer full finetune updates SSL only in the last 30 epochs
    A manually chosen schedule with no ablation, though the paper claims it gives better accuracy.
  • Beam size for decoding = 5
    Chosen for consistent evaluation; the authors note it may slightly lower performance compared to other studies.
assumptions (6)
  • standard math Gumbel-Softmax (Eq. 2) is a valid differentiable surrogate for hard k-means assignments (Eq. 1).
    Technique from [15], cited as the source of the differentiable k-means formulation.
  • domain assumption WavLM-large layer 21 (and XLSR-1B layer 35) features contain sufficient information for ASR tokenization.
    Standard SSL practice; the paper follows [12] for feature extraction.
  • ad hoc to paper The ASR cross-entropy loss alone, without a reconstruction term (alpha=0), can drive centroids to form useful phonetic clusters.
    Set based on preliminary experiments, with no theoretical justification or analysis of centroid collapse.
  • domain assumption The weighted sum in Eq. (5) over all SSL layers is a sufficient fusion for tokenization.
    Common for continuous SSL features; the paper extends it to discrete tokenization without further validation.
  • domain assumption TSL and MTER measure phonetic purity rather than mere sequence compression after deduplication.
    The paper introduces these metrics as indicators of phoneme-like tokens, but does not control for the effect of sequence length on the error rate.
  • ad hoc to paper Multi-layer full finetune with SSL updates only in the last 30 epochs is beneficial.
    Claimed better than updating from the start, but no ablation is shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Differentiable K-means for Fully-optimized Discrete Token-based ASR." pith.science (2026). https://pith.science/paper/HFOH2XG5

@misc{pith2026250516207,
  author       = {Pith},
  title        = {Pith review of: Differentiable K-means for Fully-optimized Discrete Token-based ASR},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFOH2XG5}},
  note         = {Machine review of arXiv:2505.16207}
}
read the original abstract

Recent studies have highlighted the potential of discrete tokens derived from self-supervised learning (SSL) models for various speech-related tasks. These tokens serve not only as substitutes for text in language modeling but also as intermediate representations for tasks such as automatic speech recognition (ASR). However, discrete tokens are typically obtained via k-means clustering of SSL features independently of downstream tasks, making them suboptimal for specific applications. This paper proposes the use of differentiable k-means, enabling the joint optimization of tokenization and downstream tasks. This approach enables the fine-tuning of the SSL parameters and learning weights for outputs from multiple SSL layers. Experiments were conducted with ASR as a downstream task. ASR accuracy successfully improved owing to the optimized tokens. The acquired tokens also exhibited greater purity of phonetic information, which were found to be useful even in speech resynthesis.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 34 canonical work pages

  1. [15]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abili- ties,

    D. Zhang, S. Li, X. Zhang et al., “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abili- ties,” in Findings of the Association for Computational Linguis- tics: EMNLP 2023 , 2023, pp. 15 757–15 773

    work page 2023

  2. [1]

    Introduction The rapid advancement of Self-Supervised Learning (SSL) techniques has significantly influenced numerous speech pro- cessing tasks [1]. Pioneering SSL-based models such as Hu- BERT [2], wav2vec 2.0 [3], WavLM [4], and w2v-BERT [5] have demonstrated strong performance across diverse applica- tions including automatic speech recognition (ASR), ...

    work page Pith review arXiv 2025

  3. [2]

    text-like

    Discrete Token-based ASR with Differentiable K-means 2.1. Discrete token-based ASR Discrete token-based ASR has recently attracted attention as an alternative to using continuous features extracted from SSL models [11–13, 17, 18]. In this paradigm, each frame of the SSL output is clustered into discrete tokens, which then serve as input to a conventional ...

    work page

  4. [3]

    Experimental Setup Our experiments were conducted using ESPnet [24], follow- ing the basic configurations described in [11, 12]

    Experiments 3.1. Experimental Setup Our experiments were conducted using ESPnet [24], follow- ing the basic configurations described in [11, 12]. We used the joint CTC/attention-based encoder-decoder model [19] with WavLM-large1 as the upstream SSL. To avoid additional com- plexity in handling gradients, we did not perform deduplica- tion or subword model...

    work page

  5. [4]

    Since SSL fea- ture spaces are changed by finetuning, they are divided by the averaged L2 norm for normalization

    Normalized Quantization Error (NQE): The average Eu- clidean distance between the continuous SSL features of each frame and the corresponding cluster centroid. Since SSL fea- ture spaces are changed by finetuning, they are divided by the averaged L2 norm for normalization. As we use the ASR objec- tive (Equation (4) or (6)), which is different from the k-...

    work page

  6. [5]

    If the tokens are robust to sub- tle pronunciation variations and represent more phoneme-like symbols, the sequence increases the repetition and TSL will be shorter

    Token Sequence Length (TSL): The length of token se- quences after deduplication. If the tokens are robust to sub- tle pronunciation variations and represent more phoneme-like symbols, the sequence increases the repetition and TSL will be shorter. A lower value also indicates a higher efficiency in data compression

    work page

  7. [6]

    A lower value indicates the robustness of discrete tokens to non-linguistic information

    Mean Token Error Rate (MTER): The edit distance be- tween discrete token sequences from different utterances with the same transcriptions. A lower value indicates the robustness of discrete tokens to non-linguistic information. Unlike word error rate (WER) or character error rate (CER), which are cal- culated with the ground truth transcriptions, there is...

    work page 2000

  8. [7]

    As a result, ASR performance improved and the obtained tokens exhibited more phonetic characteristics with purer linguistic information

    Conclusion In this study, we introduced differentiable k-means to dis- crete token-based ASR to enable the joint optimization of to- kenization and the downstream task, along with SSL finetun- ing and weighted summation across SSL layers. As a result, ASR performance improved and the obtained tokens exhibited more phonetic characteristics with purer lingu...

    work page

Show all 41 references
  1. [8]

    Self-supervised speech representation learning: A review,

    A. Mohamed, H.-y. Lee, L. Borgholt et al. , “Self-supervised speech representation learning: A review,” IEEE Journal of Se- lected Topics in Signal Processing, vol. 16, no. 6, pp. 1179–1210, 2022

    2022

  2. [9]

    HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai et al. , “HuBERT: Self- supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021

    2021

  3. [10]

    wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associ...

    2020

  4. [11]

    WavLM: Large-scale self-supervised pre-training for full stack speech process- ing,

    S. Chen, C. Wang, Z. Chen et al. , “WavLM: Large-scale self-supervised pre-training for full stack speech process- ing,” IEEE Journal of Selected Topics in Signal Process- ing, vol. 16, pp. 1505–1518, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:239885872

    2021

  5. [12]

    w2v-BERT: Combining contrastive learning and masked language modeling for self- supervised speech pre-training,

    Y .-A. Chung, Y . Zhang, W. Han et al., “w2v-BERT: Combining contrastive learning and masked language modeling for self- supervised speech pre-training,” 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , pp. 244– 250, 2021. [Online]. Available: https:/...

    2021

  6. [13]

    SUPERB: Speech processing universal performance benchmark,

    S. wen Yang, P.-H. Chi, Y .-S. Chuang et al., “SUPERB: Speech processing universal performance benchmark,” in Interspeech 2021, 2021, pp. 1194–1198

    2021

  7. [14]

    The Interspeech 2024 challenge on speech processing using discrete units,

    X. Chang, J. Shi, J. Tian et al., “The Interspeech 2024 challenge on speech processing using discrete units,” in Interspeech 2024, 2024, pp. 2559–2563

    2024

  8. [16]

    V oxtlm: Unified decoder- only models for consolidating speech recognition, synthesis and speech, text continuation tasks,

    S. Maiti, Y . Peng, S. Choi et al. , “V oxtlm: Unified decoder- only models for consolidating speech recognition, synthesis and speech, text continuation tasks,” in ICASSP 2024. IEEE, 2024, pp. 13 326–13 330

    2024

  9. [17]

    Au- diopalm: A large language model that can speak and listen,

    P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen et al., “Au- diopalm: A large language model that can speak and listen,”arXiv preprint arXiv:2306.12925, 2023

    2023 arXiv

  10. [18]

    Ex- ploration of efficient end-to-end ASR using discretized input from self-supervised learning,

    X. Chang, B. Yan, Y . Fujita, T. Maekaku, and S. Watanabe, “Ex- ploration of efficient end-to-end ASR using discretized input from self-supervised learning,” in Interspeech 2023 , 2023, pp. 1399– 1403

    2023

  11. [19]

    Exploring speech recognition, translation, and understanding with discrete speech units: A com- parative study,

    X. Chang, B. Yan, K. Choi et al., “Exploring speech recognition, translation, and understanding with discrete speech units: A com- parative study,” inICASSP 2024, 2024, pp. 11 481–11 485

    2024

  12. [20]

    Towards universal speech discrete tokens: A case study for ASR and TTS,

    Y . Yang, F. Shen, C. Duet al., “Towards universal speech discrete tokens: A case study for ASR and TTS,” in ICASSP 2024, 2024, pp. 10 401–10 405

    2024

  13. [21]

    Estimating the completeness of discrete speech units,

    S.-L. Yeh and H. Tang, “Estimating the completeness of discrete speech units,” in 2024 IEEE Spoken Language Technology Work- shop (SLT), 2024, pp. 415–422

    2024

  14. [22]

    Deep clus- tering with concrete k-means,

    B. Gao, Y . Yang, H. Gouk, and T. M. Hospedales, “Deep clus- tering with concrete k-means,” in ICASSP 2020, 2020, pp. 4252– 4256

    2020

  15. [23]

    Neural discrete represen- tation learning,

    A. Van Den Oord, O. Vinyals et al. , “Neural discrete represen- tation learning,” Advances in neural information processing sys- tems, vol. 30, 2017

    2017

  16. [24]

    How should we extract dis- crete audio tokens from self-supervised models?

    P. Mousavi, J. Duret, S. Zaiem et al., “How should we extract dis- crete audio tokens from self-supervised models?” in Interspeech 2024, 2024, pp. 2554–2558

    2024

  17. [25]

    MMM: Multi-layer multi-residual multi-stream discrete speech represen- tation from self-supervised learning model,

    J. Shi, X. Ma, H. Inaguma, A. Sun, and S. Watanabe, “MMM: Multi-layer multi-residual multi-stream discrete speech represen- tation from self-supervised learning model,” in Interspeech 2024, 2024, pp. 2569–2573

    2024

  18. [26]

    Joint CTC-attention based end-to-end speech recognition using multi-task learning,

    S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning,” in ICASSP 2017, 2017, pp. 4835–4839

    2017

  19. [27]

    Sequence transduction with recurrent neural net- works,

    A. Graves, “Sequence transduction with recurrent neural net- works,” arXiv preprint arXiv:1211.3711, 2012

    2012 arXiv

  20. [28]

    Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer,

    Y . Peng, J. Tian, W. Chen et al., “Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer,” in Interspeech 2024, 2024, pp. 352–356

    2024

  21. [29]

    Comparing discrete and continuous space llms for speech recognition,

    Y . Xu, S.-X. Zhang, J. Yu, Z. Wu, and D. Yu, “Comparing discrete and continuous space llms for speech recognition,” inInterspeech 2024, 2024, pp. 2509–2513

    2024

  22. [30]

    Categorical reparameterization with gumbel-softmax,

    E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” in International Conference on Learning Representations, 2017. [Online]. Available: https://openreview. net/forum?id=rkE3y85ee

    2017

  23. [31]

    ESPnet: End-to-end speech processing toolkit,

    S. Watanabe, T. Hori, S. Karita et al., “ESPnet: End-to-end speech processing toolkit,” in Interspeech 2018, 2018, pp. 2207–2211

    2018

  24. [32]

    Lib- rispeech: An asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An asr corpus based on public domain audio books,” in ICASSP 2015, 2015, pp. 5206–5210

    2015

  25. [33]

    On generative spoken language modeling from raw audio,

    K. Lakhotia, E. Kharitonov, W.-N. Hsu et al. , “On generative spoken language modeling from raw audio,” Transactions of the Association for Computational Linguistics , vol. 9, pp. 1336–1354, 2021. [Online]. Available: https://aclanthology.org/ 2021.tacl-1.79

    2021

  26. [34]

    TIMIT acoustic-phonetic continuous speech corpus,

    J. S. Garofolo, L. F. Lamel, W. M. Fisher et al. , “TIMIT acoustic-phonetic continuous speech corpus,” 01 1993. [Online]. Available: https://cir.nii.ac.jp/crid/1881146593179904768

    1993

  27. [35]

    Natural TTS synthesis by con- ditioning wavenet on mel spectrogram predictions,

    J. Shen, R. Pang, R. J. Weiss et al., “Natural TTS synthesis by con- ditioning wavenet on mel spectrogram predictions,” in ICASSP 2018, 2018, pp. 4779–4783

    2018

  28. [36]

    The LJ speech dataset,

    K. Ito and L. Johnson, “The LJ speech dataset,” https://keithito. com/LJ-Speech-Dataset/, 2017

    2017

  29. [37]

    Parallel wavegan: A fast waveform generation model based on generative adversarial net- works with multi-resolution spectrogram,

    R. Yamamoto, E. Song, and J.-M. Kim, “Parallel wavegan: A fast waveform generation model based on generative adversarial net- works with multi-resolution spectrogram,” inICASSP 2020, 2020, pp. 6199–6203

    2020

  30. [38]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu et al. , “Robust speech recognition via large-scale weak supervision,” in Proceedings of the 40th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho et al. , Eds., vol. 202....

    2023

  31. [39]

    CSTR VCTK cor- pus: English multi-speaker corpus for cstr voice cloning toolkit,

    C. Veaux, J. Yamagishi, K. MacDonald et al., “CSTR VCTK cor- pus: English multi-speaker corpus for cstr voice cloning toolkit,” University of Edinburgh. The Centre for Speech Technology Re- search (CSTR), vol. 6, p. 15, 2017

    2017

  32. [40]

    UTMOS: UTokyo-SaruLab system for V oiceMOS challenge 2022,

    T. Saeki, D. Xin, W. Nakata et al. , “UTMOS: UTokyo-SaruLab system for V oiceMOS challenge 2022,” in Interspeech 2022 , 2022, pp. 4521–4525

    2022

  33. [41]

    ML-SUPERB: Multilingual speech universal performance benchmark,

    J. Shi, D. Berrebbi, W. Chen et al., “ML-SUPERB: Multilingual speech universal performance benchmark,” in Interspeech 2023, 2023, pp. 884–888

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.