Pith. sign in

REVIEW 4 major objections 5 minor 16 references

The paper claims a two-stage, layer-weighted training recipe makes discrete-token multilingual ASR nearly as accurate as continuous-feature ASR, cutting CER by 44 percent on ML-SUPERB for the XLS-R frontend.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Learning layer weights on continuous features and reusing them for discrete token extraction, plus fine-tuning XLS-R with extra data, yields 44% relative CER reduction on ML-SUPERB and tops the challenge's single-system leaderboard.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A useful but modest two-stage trick; headline 44% gain is confounded by LS960 fine-tuning and should not be taken at face value. the 4 major comments →

arxiv 2509.01900 v1 pith:A7JGEVXE submitted 2025-09-02 eess.AS

Multilingual Speech Recognition Using Discrete Tokens with a Two-step Training Strategy

Zehan Li , Yan Yang , Xueqing Li , Jian Kang , Xiao-Lei Zhang , Jie Li This is my paper
classification eess.AS
keywords discrete speech tokensmultilingual ASRself-supervised learningtwo-stage traininglayer-wise weightingXLS-RML-SUPERBk-means quantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to close the accuracy gap between automatic speech recognition (ASR) systems that use continuous self-supervised representations and systems that use discrete tokens (cluster indices). The proposed recipe has two parts: first, learn layer weights on continuous features; second, freeze those weights and use the weighted layer sum to extract discrete tokens for a downstream CTC/attention ASR model. A frontend fine-tuning step with extra LibriSpeech data is added to push word identity into the final layers. On the ML-SUPERB benchmark, the authors report a 44 percent relative CER reduction for XLS-R discrete tokens, bringing them within 8 percent of the same frontend's continuous-feature CER. This matters because discrete tokens are cheaper to store and fit naturally into language-model-style speech systems, provided they can match continuous features.

Core claim

The central claim is that the gap between discrete-token and continuous-representation ASR can be largely closed by a two-stage training strategy: in stage one, train a weighted sum of frontend layer outputs on a conventional continuous-feature ASR task; in stage two, freeze those weights, k-means-quantize the weighted representation into 2000-cluster discrete units, de-duplicate and subword-segment them, and train a discrete ASR model. The paper shows that fine-tuning the frontend first—using LibriSpeech 960 hours plus ML-SUPERB—changes layer-weight concentration from middle layers to final layers, matching supervised models like Whisper, and that this weighted sum then outperforms single-l

What carries the argument

weighted_sum: a trainable-weight softmax combination of all encoder block outputs (plus, for pre-trained models, the normalized final layer). The weights are learned with gradients flowing through continuous features in stage one, then frozen for discrete-token extraction in stage two, so the same layer weighting that helps continuous ASR guides which layers' clusters are used. The fine-tuning step, which breaks the auto-encoder-like behavior of the final layers by training with CTC on extra data, is what shifts the weight concentration to the last layers.

Load-bearing premise

The claimed lead over the WavLM baseline assumes the improvement comes from the proposed two-stage weighted-sum and fine-tuning recipe, not from the extra 960 hours of LibriSpeech used to fine-tune XLS-R but not the baseline.

What would settle it

Run the proposed pipeline on the challenge's provided training data only (no LibriSpeech 960 hours) and compare CER on ML-SUPERB test-1h against the same pipeline with LS960 and against the WavLM baseline; if the no-LS960 version loses most of the 44% relative reduction, the method's standalone contribution is small.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Discrete-token ASR with XLS-R on ML-SUPERB reaches 16.9% CER, within 8% of its continuous-feature result.
  • The 44% relative reduction over the untuned XLS-R discrete baseline also beats the challenge's WavLM baseline (26%) and took first place among single systems.
  • The same two-stage recipe works for supervised frontends like Whisper, though with smaller gains, suggesting it is not tied to self-supervised models alone.
  • Fine-tuning shifts informative layers to the final blocks, making the choice of which layer to quantize much less language-dependent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The headline gain versus WavLM is confounded with extra fine-tuning data (LibriSpeech 960 hours); a same-data comparison would be needed to isolate the method's independent contribution.
  • The learned layer weights could potentially be used to prune the frontend at inference, since the final layers dominate after fine-tuning, reducing compute for token extraction.
  • The same weight-transfer idea could apply to neural codec tokens (e.g., from discrete codebooks) beyond k-means cluster IDs, potentially improving language-model-based speech decoding.
  • If the fine-tuning shift to final layers is robust, token extraction could use a single high-weight layer with nearly the weighted-sum performance, simplifying deployment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a two-stage training strategy for multilingual ASR with discrete tokens. In stage 1, per-layer weights for a weighted summation of SSL representations are learned on a continuous-feature ASR task; in stage 2, these weights are frozen and used to extract discrete tokens via k-means, which then train a discrete-token E-Branchformer ASR model. The authors also fine-tune the XLS-R frontend on ML-SUPERB plus LibriSpeech 960h (train3) with CTC. On the ML-SUPERB test-1h set, they report 16.9% CER for XLS-R with character units and weighted-sum, a 44% relative reduction over a single-layer pre-trained XLS-R baseline, and claim first place among single-system results on the Interspeech2024 challenge leaderboard.

Significance. If the headline gains were supported by controlled experiments, the two-stage weighted-sum recipe would be a practically valuable method for closing the gap between discrete-token and continuous-feature multilingual ASR, while retaining the storage and efficiency benefits of discrete units. The paper uses a public benchmark, reproduces the challenge's WavLM baseline, and reports results in a transparent table format. The core idea—learning layer weights on continuous representations and transferring them to discrete-token extraction—is interesting and empirically testable. However, the paper's own ablations show the isolated contribution of weighted_sum is modest (3–18% relative reduction depending on configuration), and the headline 44% figure is confounded with additional fine-tuning data and tokenization changes. The significance of the work therefore depends on whether a controlled comparison can substantiate the claims.

major comments (4)
  1. [§3.4, Table 3; Abstract] The 44% relative reduction claimed in the abstract is computed by comparing xlsr layer-17 (CER 30.7) with xlsr_train3_char weighted_sum (CER 16.9). This contrast bundles the proposed weighted-sum with (i) fine-tuning on LS960 (train3) and (ii) switching from BPE to character units. The paper provides no ablation for xlsr_train3_char without weighted_sum, nor a single-layer row for this configuration. The available pairs show a much smaller isolated effect: xlsr 30.7→25.3 (17.6% relative), xlsr_train3 22.3→21.2 (4.9%), xlsr_train3_bpe6288 18.0→17.4 (3.3%). Thus the headline claim is not supported by a controlled comparison.
  2. [§2.2 vs §3.1] Section 2.2 states that 'a small amount of data' is used to fine-tune the frontend, but Section 3.1 defines train3 as ML-SUPERB + LS960 (960 hours). The WavLM baseline in Table 3 (CER 22.9) does not receive LS960 fine-tuning, so the claimed superiority over WavLM (CER 16.9) and part of the 44% gain are likely driven by the extra data. To support the central claim, the authors should include a baseline with train2-only fine-tuning (no LS960) using the same tokenization, and/or a WavLM frontend fine-tuned on the same data.
  3. [§3.3 and §3.4] The continuous-representation results in Table 2 use downstream training on train_2 only, while the frontends for xlsr_train3* are fine-tuned on train3. The discrete-token results in Table 3 use the same fine-tuned frontends. This experimental design is acceptable, but the paper should state explicitly that the additional LibriSpeech data is used only for frontend fine-tuning and not for downstream ASR model training. Currently, the text 'we only use train_2' appears only in §3.3 and could be misread as applying to the frontend as well.
  4. [§3.4, Figure 2] The weight distribution plot pads XLS-R-based models with zero weights to align with Whisper's 33 layers, which visually distorts the comparison. Since the weight distributions are used to justify the design of Eq. (1) (e.g., excluding the normalized final layer for fine-tuned models), the plot should show the actual distributions on a common axis without arbitrary zero-padding, or the authors should provide a quantitative comparison (e.g., entropy, layer index of max weight).
minor comments (5)
  1. [§3.2] Typo: 'Librispeeech' should be 'Librispeech'.
  2. [Eq. (1)] Clarify the index set for the fine-tuned case; also define F(h_l) more precisely and state why it is excluded after fine-tuning.
  3. [Table 3] No weighted_sum result is reported for WavLM. The claim that the method 'surpasses' WavLM should be qualified because WavLM is not given the proposed treatment (or LS960 fine-tuning).
  4. [§3.2] 'The default BPE with 6000 subword units fails to completely cover the characters' — coverage of characters is not the standard BPE failure mode; rephrase to describe the actual OOV issue.
  5. [§3.4] The paper does not report run-to-run variance. Given the small test-1h set, single-run CER differences of less than 1% may not be significant; consider reporting results over multiple seeds or noting this limitation.

Circularity Check

0 steps flagged

No significant circularity: the proposed two-stage transfer is an empirical pipeline evaluated on external benchmarks, not a derivation equivalent to its own inputs.

full rationale

This paper contains no circular reasoning of the kinds enumerated. The central mechanism is a two-stage pipeline: (1) trainable layer weights (λ_i) are optimized on continuous SSL representations for a downstream ASR task; (2) the same weights are frozen and used to form a weighted-sum representation from which k-means discrete tokens are extracted and used to train a separate discrete-ASR model. The discrete-token CERs in Table 3 are measured on held-out test sets (LibriSpeech and ML-SUPERB test-1h), so they are not equal by construction to the continuous-representation objective used to fit the weights. The result is a genuine transfer: no equation in the paper defines the discrete-token CER in terms of the continuous-stage loss. There are no load-bearing self-citations: all references are to external prior work (e.g., ML-SUPERB, k-means clustering, E-Branchformer), and no uniqueness theorem or ansatz is imported from the present authors' own previous publications. The method is also externally falsifiable because the evaluation uses public benchmarks and a publicly available pretrained frontend. Two non-circular concerns should be noted but do not affect the circularity score. First, Section 2 says 'a small amount of data' is used for fine-tuning while Section 3.1 uses the full LS960, an internal inconsistency. Second, the headline 44% relative reduction compares xlsr layer-17 (30.7) with xlsr_train3_char weighted_sum (16.9), which bundles the proposed weighted-sum recipe with LS960 fine-tuning, character targets, and BPE changes; the isolated weighted_sum contribution in Table 3 is much smaller (e.g., 25.3 vs 30.7 for the unfine-tuned xlsr). That is a validity/attribution problem, not a circularity problem. The derivation chain is self-contained and empirically grounded, so the score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The empirical result rests on fitting per-layer weights and tuning the BPE size, and on three domain assumptions: transferability of weights from continuous to discrete, comparability of evaluations that use extra fine-tuning data, and the adequacy of k-means discretization. No new physical or conceptual entities are introduced.

free parameters (3)
  • Per-layer weights lambda_i = Learned in stage 1 on continuous representations
    Defines the weighted_sum used for both continuous and discrete extraction; fitted to the continuous ASR task then frozen.
  • BPE vocabulary size = 6500 (or 6288)
    Hand-tuned after default 6000 produced <unk> tokens; affects fine-tuning and representation quality.
  • K-means cluster count = 2000
    Set as in prior work [15]; determines discrete token vocabulary.
axioms (3)
  • domain assumption Layer weights learned on continuous features transfer to discrete token extraction
    Core hypothesis of the two-stage method; if weights are not transferable, stage 2 provides no benefit. Empirically tested in Table 3 but not justified mathematically.
  • domain assumption Evaluation with extra fine-tuning data (LS960) is comparable to challenge baselines that do not use it
    The 44% reduction and WavLM comparison in Sections 3.1 and 3.4 assume adding LibriSpeech 960h does not invalidate the comparison.
  • domain assumption K-means with 2000 centroids on SSL representations yields discrete units with sufficient phonetic information for ASR
    Borrowed from prior work [15]; not re-derived.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Multilingual Speech Recognition Using Discrete Tokens with a Two-step Training Strategy." pith.science (2026). https://pith.science/paper/A7JGEVXE

@misc{pith2026250901900,
  author       = {Pith},
  title        = {Pith review of: Multilingual Speech Recognition Using Discrete Tokens with a Two-step Training Strategy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A7JGEVXE}},
  note         = {Machine review of arXiv:2509.01900}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Pre-trained models, especially self-supervised learning (SSL) models, have demonstrated impressive results in automatic speech recognition (ASR) task. While most applications of SSL models focus on leveraging continuous representations as features for training downstream tasks, the utilization of discrete units has gained increasing attention in recent years owing to its lower storage requirements and broader range of applications. In multilingual ASR tasks, representations at different layers of the model contribute differently to various languages, complicating the unification of discrete unit modeling. In this paper, we propose a two-stage training strategy to improve the discrete token performance of pre-trained models and narrow the gap with continuous representation performance. We validate our method on the XLS-R model following the settings of Interspeech2024 Speech Processing Using Discrete Speech Unit Challenge. Our method demonstrates a significant improvement on the ML-SUPERB dataset, achieving a 44% relative reduction on CER for the XLS-R model. This surpasses the previous baseline set by the WavLM model, which achieves a 26% relative reduction on CER. Furthermore, our method achieves the first place among all the single-system results on the leaderboard.

Figures

Figures reproduced from arXiv: 2509.01900 by Jian Kang, Jie Li, Xiao-Lei Zhang, Xueqing Li, Yan Yang, Zehan Li.

Figure 1
Figure 1. Figure 1: Two-stage discrete speech unit ASR system architecture. of similarity. In order to better utilize the representation differences of each layer and perform well on multilingual data, we choose to extract representations in the form of weighted summation, which is called weighted_sum in this paper. The weights are learned on a conventional downstream ASR task utilizing continuous representations in the first… view at source ↗
Figure 2
Figure 2. Figure 2: Weight distribution of different models on train_2. The XLSR pre-trained model has 25 layers of weights, whereas its fine-tuned version has 24 layers. To ensure alignment in the plot with the weights of the Whisper model (33 layers), we set the weights of the remaining layers in the XLSR-based models to 0 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 9 canonical work pages · 1 internal anchor

  1. [1]

    IEEE Journal of Selected Topics in Signal Processing 11(8), 1240–1253 (2017) 5 https://huggingface.co/microsoft/wavlm-large 8 Z

    Watanabe, S., Hori, T., Kim, S., Hershey, J.R., Hayashi, T.: Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing 11(8), 1240–1253 (2017) 5 https://huggingface.co/microsoft/wavlm-large 8 Z. Li et al

    work page 2017

  2. [2]

    In: Proc

    Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., Pang, R.: Conformer: Convolution-augmented Trans- former for Speech Recognition. In: Proc. Interspeech 2020. pp. 5036–5040 (2020). https://doi.org/10.21437/Interspeech.2020-3015

    doi:10.21437/interspeech.2020-3015 2020

  3. [3]

    In: Proc

    Gao, Z., Zhang, S., McLoughlin, I., Yan, Z.: Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition. In: Proc. In- terspeech 2022. pp. 2063–2067 (2022). https://doi.org/10.21437/Interspeech.2022- 9996

    doi:10.21437/interspeech.2022- 2022

  4. [4]

    Yao, Z., Guo, L., Yang, X., Kang, W., Kuang, F., Yang, Y., Jin, Z., Lin, L., Povey, D.: Zipformer: A faster and better encoder for automatic speech recognition (2024)

    work page 2024

  5. [5]

    Advances in Neural Information Processing Systems 33, 12449–12460 (2020)

    Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems 33, 12449–12460 (2020)

    work page 2020

  6. [6]

    IEEE/ACM Transactions on Audio, Speech, and Language Process- ing 29, 3451–3460 (2021)

    Hsu, W.N., Bolte, B., Tsai, Y.H.H., Lakhotia, K., Salakhutdinov, R., Mohamed, A.: Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing 29, 3451–3460 (2021)

    work page 2021

  7. [7]

    In: Interna- tional Conference on Machine Learning

    Baevski, A., Hsu, W.N., Xu, Q., Babu, A., Gu, J., Auli, M.: Data2vec: A general framework for self-supervised learning in speech, vision and language. In: Interna- tional Conference on Machine Learning. pp. 1298–1312. PMLR (2022)

    work page 2022

  8. [8]

    Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS

    Yang, Y., Shen, F., Du, C., Ma, Z., Yu, K., Povey, D., Chen, X.: Towards universal speech discrete tokens: A case study for asr and tts. arXiv preprint arXiv:2309.07377 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    arXiv preprint arXiv:2112.08352 (2021)

    Lee, A., Gong, H., Duquenne, P.A., Schwenk, H., Chen, P.J., Wang, C., Popuri, S., Adi, Y., Pino, J., Gu, J., et al.: Textless speech-to-speech translation on real data. arXiv preprint arXiv:2112.08352 (2021)

    Pith/arXiv arXiv 2021

  10. [10]

    arXiv preprint arXiv:2305.11000 (2023)

    Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., Qiu, X.: Speechgpt: Em- powering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000 (2023)

    Pith/arXiv arXiv 2023

  11. [11]

    arXiv preprint arXiv:2305.16107 (2023)

    Wang, T., Zhou, L., Zhang, Z., Wu, Y., Liu, S., Gaur, Y., Chen, Z., Li, J., Wei, F.: Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint arXiv:2305.16107 (2023)

    Pith/arXiv arXiv 2023

  12. [12]

    In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Wu, F., Kim, K., Watanabe, S., Han, K.J., McDonald, R., Weinberger, K.Q., Artzi, Y.: Wav2seq: Pre-training speech-to-text encoder-decoder models using pseudo languages. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2023)

    work page 2023

  13. [13]

    In: 2021 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU)

    Pasad, A., Chou, J.C., Livescu, K.: Layer-wise analysis of a self-supervised speech representation model. In: 2021 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU). pp. 914–921. IEEE (2021)

    work page 2021

  14. [14]

    In: Proc

    Shi, J., Berrebbi, D., Chen, W., Hu, E.P., Huang, W.P., Chung, H.L., Chang, X., Li, S.W., Mohamed, A., yi Lee, H., Watanabe, S.: ML-SUPERB: Multilingual Speech Universal PERformance Benchmark. In: Proc. INTERSPEECH 2023. pp. 884–888 (2023). https://doi.org/10.21437/Interspeech.2023-1316

    doi:10.21437/interspeech.2023-1316 2023

  15. [15]

    Chang, X., Yan, B., Fujita, Y., Maekaku, T., Watanabe, S.: Exploration of efficient end-to-end asr using discretized input from self-supervised learning (2023)

    work page 2023

  16. [16]

    In: 2022 IEEE Spoken Language Technology Workshop (SLT)

    Kim, K., Wu, F., Peng, Y., Pan, J., Sridhar, P., Han, K.J., Watanabe, S.: E- branchformer: Branchformer with enhanced merging for speech recognition. In: 2022 IEEE Spoken Language Technology Workshop (SLT). pp. 84–91 (2023). https://doi.org/10.1109/SLT54892.2023.10022656

    arXiv 2022

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.