Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTC

Jiancheng Sun; Qingzheng Wang; Shinji Watanabe; Yifan Peng

REVIEW 3 major objections 4 minor 40 references

Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTC

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that fine-tuning a multilingual speech foundation model on data augmented with extra Common Voice utterances and an auxiliary LID CTC loss achieves 86.9% LID accuracy and 15.6% CER on ML-SUPERB 2.0.

desk verdict Solid challenge-system report, but the headline numbers are dev-set selections and the LID CTC contribution is small; treat it as an engineering contribution, not a settled scientific claim. read the letter →

arxiv 2505.24200 v2 pith:TYDUPK6V submitted 2025-05-30 cs.SD cs.CLeess.AS

Qingzheng Wang , Jiancheng Sun , Yifan Peng , Shinji Watanabe This is my paper

classification cs.SDcs.CLeess.AS

keywords multilingualspeechlanguageidentificationautomaticrecognitionML-SUPERB2.0fine-tuningdataaugmentationLIDCTClosslow-resourcelanguages

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make multilingual speech models work better under scarce fine-tuning data. It claims that adding roughly 2,100 utterances per underperforming language from a public corpus and regularizing fine-tuning with an auxiliary LID CTC loss raises LID accuracy to 86.9% and cuts ASR CER to 15.6% on ML-SUPERB 2.0, a 14% relative LID gain and 30% relative CER reduction over the challenge baseline. The authors also find that the few-shot-language gains come mostly from data augmentation, while the LID CTC loss mainly helps dialect generalization. If true, this shows that a relatively cheap combination of data and an auxiliary objective can narrow the gap between high- and low-resource languages without changing the architecture.

What carries the argument

The load-bearing mechanism is an auxiliary CTC loss computed at selected intermediate encoder layers. For each utterance, a language-code sequence $I$ of the same length as the transcription is constructed with every symbol equal to the utterance's language code; this sequence is decoded from the hidden states $Z^l$ of layer $l$ via CTC alignment, producing $L_{\text{LID}}^l = -\log P(I \mid Z^l)$. The final objective mixes this with the primary ASR CTC loss as $L = (1-\beta)L_{\text{CTC}} + \beta \frac{1}{|M|}\sum_{l\in M} L_{\text{LID}}^l$, applied to a subset $M$ of the unfrozen layers. Assisting it is data augmentation: random utterance samples from Common Voice v20.0, on average 2,123 per language, are merged into the official training set to supply supervision for few-shot and underperforming languages. These two additions sit on top of a weighted-sum layer aggregation of upstream representations and a choice of partial-layer fine-tuning or LoRA for adaptation.

What would settle it

Run the final MMS-1B configuration (fine-tuned layers 25-36, LID CTC on layers 27, 30, 33, and 36, $\beta = 0.3$, augmented data) on the hidden ML-SUPERB 2.0 test set and compare LID accuracy and CER against the re-run baseline; if the test-set gains are much smaller than the reported 10.8-point LID gain and 6.7-point CER reduction, the development-set selection inflated the result.

Watch

Extended reading notes

Core claim

The central claim is that the right adaptation recipe for an encoder-only multilingual speech foundation model is to unfreeze a contiguous block of middle layers, inject additional labeled speech for weak languages, and supervise intermediate layers with a CTC loss that predicts the utterance's language code. With MMS-1B, that recipe reaches 86.9% LID accuracy and 15.6% CER on the standard development set, improving on the re-run baseline by 14% relative in LID and 30% relative in CER. The paper reports that on few-shot languages, augmentation alone accounts for most of the gain—for example, XEUS few-shot LID jumps 42.7 percentage points absolute—whereas the LID CTC loss contributes mainly to dialect robustness.

Load-bearing premise

The reported gains are measured on the development set, and that same set was used to choose the final layers, beta, and learning rate, so the paper assumes the development set predicts the hidden test set.

Editorial extensions

If this is right

Data augmentation is the main driver of few-shot gains: fine-tuning XEUS on augmented data improves few-shot LID accuracy by 42.7% and reduces few-shot CER by 17.2% absolutely.
The LID CTC loss improves dialect robustness for MMS-1B and XEUS when sufficient training data is present, such as a 10.3% absolute dialect LID gain for MMS-1B on augmented data.
LoRA on the Transformer-based MMS-1B outperforms full-layer fine-tuning, but the same LoRA configuration does not beat fine-tuning for E-Branchformer-based XEUS and OWSM-CTC.
Middle-layer fine-tuning is consistently better than bottom- or top-layer fine-tuning across MMS-1B, XEUS, and OWSM-CTC.
The combined recipe places the system second in the ML-SUPERB 2.0 Challenge and reduces the per-language spread of LID and CER results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the layer-dependent success of LoRA suggests that for E-Branchformer-based models, LoRA should also adapt the convolutional local-extractor branch, not just self-attention projections; a rank-constrained adaptation of those branches would be a cheap test.
Beyond the paper: because the LID CTC loss improves dialect sets, it may also improve generalization to unseen languages if the language-code vocabulary is expanded; one could test this by holding out several training languages entirely.
Beyond the paper: augmentation languages were selected by baseline underperformance rather than by validation loss, so selecting them by their development-set error contribution could yield larger gains per added utterance.
Beyond the paper: if the development-set results transfer to the hidden test set, the recipe is a practical drop-in for low-resource multilingual fine-tuning; if they do not, the main lesson is that dev-set hyperparameter selection, not the method, drove the improvement.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a competent challenge-system report and should be judged as one. The headline '86.9% LID / 15.6% CER, second place' is computed on the ML-SUPERB 2.0 development set, and the same dev set was used to pick fine-tuned layer ranges, LID CTC layer subsets, beta, and learning rate. So those numbers are a selected best-of-configurations estimate, not an unbiased forecast of test performance. The stress-test note gets that right.

What is genuinely useful: the systematic comparison of MMS-1B, XEUS, and OWSM-CTC under frozen, partial fine-tuning, and LoRA, with trainable parameter counts held roughly constant. The tables report not just full-set averages but worst-15 and few-shot performance, which is exactly where multilingual models struggle. The authors also re-ran the challenge baseline themselves rather than quoting prior numbers, and they are upfront in Section 4 that all evaluation is on dev sets.

The strongest empirical finding is no surprise, but it is cleanly shown: adding Common Voice data for a set of few-shot/underperforming languages produces large absolute gains in few-shot LID (e.g., XEUS few-shot LID from 0.0 to 42.7 with augmentation alone). The advertised LID CTC loss is a much smaller contributor—on MMS, DataAug alone gives 86.0 LID and DataAug* (with LID CTC) gives 86.9; on XEUS it is 84.8 vs 85.8. Its clearest benefit is on dialect LID for MMS and XEUS, while it actually hurts OWSM-CTC. The paper gives no analysis of why the upstream placement matters, and the layer selections look like a search over M and N rather than a motivated design.

The soft spots are real but not fatal. First, the dev-set selection issue: with fifteen-plus configurations in Table 1 and further choices in Table 2, the reported best is an upper-biased estimate, and the abstract's 'second place' claim is not verifiable without the held-out test. Second, no code, no seed, and only an average utterance count for augmentation ('an average of 2123 utterances')—enough to convey the idea but not to reproduce it. Third, no repeated runs, so run-to-run variance is unknown.

Who this is for: anyone working on low-resource multilingual ASR/LID or on the ML-SUPERB benchmark. It is a useful data point, not a methodological advance. I would send it to peer review as a challenge-system paper, with a request that the authors either run on the official test set or clearly label all numbers as dev-set and describe the selection procedure. I would not cite it as evidence for a general claim about LID CTC.

Referee Report

3 major / 4 minor

Summary. The paper describes a recipe for adapting three encoder-only speech foundation models (MMS-1B, XEUS, OWSM-CTC) to the ML-SUPERB 2.0 benchmark. It compares frozen-upstream training, partial fine-tuning of selected layer ranges, and LoRA, then adds two enhancements: extra Common Voice data for low-resource languages and an auxiliary LID CTC loss applied to a subset of fine-tuned layers. The reported best configuration achieves 86.9% LID accuracy and 15.6% CER on the standard development set, with additional gains on few-shot languages and mixed effects on dialect sets, and the abstract claims a 14% relative LID improvement, a 30% relative CER reduction, and second place in the Interspeech 2025 ML-SUPERB 2.0 Challenge.

Significance. If the results hold on held-out test data, the paper would provide a practical and broadly useful recipe for adapting large multilingual speech models under low-resource constraints, with a clear decomposition of the contributions from fine-tuning strategy, data augmentation, and LID-aware regularization. The comparison across three recent SFMs and three adaptation strategies is valuable, and the decision to re-run the challenge baseline in the same framework is a methodological strength. The main limitation is that all reported numbers, including the headline abstract claims, come from the development set that was also used for model selection, so the paper currently offers a promising recipe rather than a verified benchmark claim.

major comments (3)

[Section 4, 'General setup' and 'Fine-tuning with augmented data and LID CTC'] All headline results, including the abstract's 86.9% LID and 15.6% CER, are computed on the development set, and the same development set was used to select the fine-tuned layer ranges N, the LID CTC layer subsets M, the loss weight beta, the learning rate, and the final checkpoint. Because Table 1 and Table 2 report the best among many configurations chosen on this set, the reported numbers are a best-of-selection estimate rather than an unbiased evaluation. The paper should either report official test-set results, perform a proper train/dev/test split for model selection, or explicitly state and discuss this inflation risk in the abstract and Section 5.
[Abstract and Section 6, Conclusion] The claim of 'securing second place in the Interspeech 2025 ML-SUPERB 2.0 Challenge' is not supported by any reported test-set evaluation or official leaderboard entry. If this ranking comes from the challenge organizers' official test evaluation, those numbers and conditions should be reported; if it is inferred from development-set results, the claim should be removed or clearly qualified, since the development set was used for model selection.
[Section 3.2 and Section 4, 'Training setup'] The data augmentation is described only as random sampling of 'an average of 2123 utterances' per language, with no per-language counts, no sampling seed, and no release of code or augmented data. Combined with the lack of repeated runs and standard deviations for the main results, this prevents independent reproduction of the reported improvements. Please provide exact counts, seeds, and either a code release or a detailed data-preparation script.

minor comments (4)

[Section 3.2] There is a typo: 'Common V oice Corpus' should be 'Common Voice Corpus'.
[Section 4, 'General setup'] The footnote for the challenge baseline link contains a formatting artifact: 'mms 1b mlsuperb' should be a proper URL or model-card designation.
[Section 3.3 and Equation 5] The notation for the LID CTC loss uses frame index T for the upstream layer output Z^l, but earlier in the paper the frame length after subsampling is denoted T/kappa; with the subsampling factor in the downstream input layer, it should be clarified whether LID CTC is applied at the original or subsampled frame rate.
[Table 1 and Table 2] The tables would be easier to read if the 'DataAug*' notation were defined directly in the table caption, since the asterisk is only explained in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper is an empirical fine-tuning study with no first-principles claim that reduces to its inputs by construction.

full rationale

The paper contains no formal derivation chain: it reports empirical comparisons of fine-tuning strategies on ML-SUPERB 2.0. The auxiliary LID CTC loss (Eqs. 5-7) shares language-code labels with the primary ASR CTC target, but this is a multi-task regularization design, not an equivalence between the training objective and the reported dev-set evaluations. The dev-set evaluations use held-out utterances and ground-truth language labels, so the reported 86.9% LID accuracy and 15.6% CER are not forced by construction. Selection of fine-tuning layers, beta, and learning rate on the development set is a model-selection validity concern (possible overfitting to the dev set), not circularity. Citations to the authors' own XEUS and OWSM-CTC papers describe the upstream models being evaluated; those models are public checkpoints and the paper's contribution is the empirical comparison, so the self-citations are not load-bearing in a circular sense. No fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported via self-citation. Score 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and rests on the CTC objective, the representativeness of the development set, and the compatibility of the extra Common Voice data. No new entities or physical axioms are introduced.

free parameters (6)

Learning rate = 1e-4
Tuned over 1e-5, 3e-5, 1e-4, 3e-4 on the development set.
LID CTC loss weight beta = 0.3
Tuned over 3e-2, 1e-1, 3e-1 on the development set.
Fine-tuned layer ranges per model = MMS 25-36, XEUS 12-19, OWSM-CTC 8-13
Selected as the best rows in Table 1, which reports development set results.
LID CTC layer subsets M = MMS {27,30,33,36}, XEUS {14,17}, OWSM-CTC {10,13}
Chosen by even spacing within the fine-tuned layers; no tuning or justification is reported.
Augmented utterances per language = ~2123 average
Random sampling from Common Voice with exact per-language counts and seed undisclosed.
LoRA rank and scaling factor = 16 and 16
Fixed without an ablation study or sensitivity analysis.

assumptions (4)

standard math CTC conditional independence assumption
Equations 3 and 5 use the standard CTC objective, which assumes outputs are conditionally independent given hidden states.
domain assumption Development set is an unbiased proxy for the test set
Section 4 evaluates on the development set because the test set is unavailable, but the same dev set is used for model selection.
domain assumption Common Voice data is compatible with ML-SUPERB and has negligible overlap with the dev set
Section 3.2 adds random Common Voice utterances; any speaker or recording overlap with the dev set would inflate the few-shot gains.
domain assumption OWSM-CTC remains a fair comparison without its prefix tokens
Section 3.1.1 excludes prefix tokens and other components from OWSM-CTC, which changes the model from its intended design.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTC." pith.science (2026). https://pith.science/paper/TYDUPK6V

@misc{pith2026250524200,
  author       = {Pith},
  title        = {Pith review of: Improving Multilingual Speech Models on ML-SUPERB 2.0: Fine-tuning with Data Augmentation and LID-Aware CTC},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYDUPK6V}},
  note         = {Machine review of arXiv:2505.24200}
}

read the original abstract

Multilingual speech processing with self-supervised or supervised pre-trained Speech Foundation Models (SFM) has achieved strong performance on tasks like Language Identification (LID) and Automatic Speech Recognition (ASR). However, these models struggle with limited resources during fine-tuning. This paper enhances multilingual LID and ASR on ML-SUPERB 2.0 by exploring multiple strategies for adapting SFMs, including frozen upstream training, partial fine-tuning, and low-rank adaptation. Furthermore, we employ data augmentation to mitigate performance gaps in few-shot settings and introduce LID Connectionist Temporal Classification (CTC) loss for regularization. Our approach achieves a 14% relative improvement in LID accuracy and a 30% relative reduction in ASR CER over the baseline on ML-SUPERB 2.0, securing second place in the Interspeech 2025 ML-SUPERB 2.0 Challenge.

Figures

Figures reproduced from arXiv: 2505.24200 by the authors.

**Figure 1.** Model architecture. The upstream model is frozen, partially fine-tuned, or adapted with LoRA. The downstream model is fully trainable. Training uses CTC loss, with optional data augmentation and LID CTC loss. Dashed lines denote optional enhancements; solid lines represent core processes. 3.1.3. Fine-tuning upstream models While training only the downstream layers leverages pre-trained representations from the upst… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 36 canonical work pages

[1]

Introduction Multilingual speech processing, including tasks such as Lan- guage Identification (LID) and Automatic Speech Processing (ASR), is essential in a world with more than 7000 languages [1, p. 1]. Recent advancements in multilingual speech foun- dation models (SFM) based on self-supervised learning (SSL) have significantly expanded the scope of ro...

work page
[2]

Related studies Multilingual speech foundation models.Both supervised and self-supervised training strategies are used to build multilingual SFMs. Self-supervised pre-trained SFMs like Massively Mul- tilingual Speech (MMS) [5] and XEUS [6], trained on 1400+ and 4000+ languages respectively, achieve remarkable multi- lingual performance. Meanwhile, supervi...

work page
[3]

Proposed method 3.1. Training strategies with multilingual SFMs To evaluate the performance of the latest multilingual SFMs on the ML-SUPERB 2.0 benchmark, we select three representa- arXiv:2505.24200v2 [cs.SD] 3 Jun 2025 tive encoder-only SFMs as the upstream models: the SSL-based MMS [5] and XEUS [6], and supervised pre-trained OWSM- CTC [11]. The overa...

work page Pith review arXiv 2025
[4]

2 Models are evaluated on the development sets due to the unavailability of the test set

Experiments General setup.We use the ML-SUPERB 2.0 official dataset from the ML-SUPERB Challenge 2025, which includes a train- ing set (220 hours, 138 languages), standard development set (41 hours, 138 languages), and dialect development set (9 hours, 56 dialects). 2 Models are evaluated on the development sets due to the unavailability of the test set. ...

work page 2025
[5]

Training strategies with multilingual SFMs Table 1 shows LID and ASR results across SFMs and training strategies on ML-SUPERB 2.0

Results 5.1. Training strategies with multilingual SFMs Table 1 shows LID and ASR results across SFMs and training strategies on ML-SUPERB 2.0. On the standard set, MMS-1B with LoRA achieves the best performance (82.4% LID accu- racy, 16.3% CER). On the dialect set, fine-tuning the middle layers of MMS-1B achieves the highest LID accuracy (68.4%), while L...

work page
[6]

We evalu- ate MMS, XEUS, and OWSM-CTC under downstream training with frozen upstream, upstream fine-tuning, and LoRA

Conclusion In this paper, we explore strategies to improve multilingual LID and ASR performance on ML-SUPERB 2.0. We evalu- ate MMS, XEUS, and OWSM-CTC under downstream training with frozen upstream, upstream fine-tuning, and LoRA. Fur- thermore, we incorporate data augmentation and LID CTC loss for fine-tuning. Results show that MMS-1B outperforms XEUS a...

work page 2025
[7]

Acknowledgements Experiments of this work used the Bridges2 system at PSC and Delta system at NCSA through allocations CIS210014 and IRI120008P from the Advanced Cyberinfrastructure Coordi- nation Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296

work page
[8]

J. S. Peter K. Austin, Ed.,The Cambridge Handbook of Endan- gered Languages, ser. Cambridge Handbooks in Language and Linguistics. Cambridge University Press, 2011

work page 2011

Show all 40 references

[9]

XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,

A. Babu, C. Wang, A. Tjandra, K. Lakhotiaet al., “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale,” inProc. Interspeech, 2021, pp. 2278–2282

2021
[10]

ASR2K: Speech recognition for around 2000 languages without audio,

X. Li, F. Metze, D. R. Mortensen, A. W. Blacket al., “ASR2K: Speech recognition for around 2000 languages without audio,” in Proc. Interspeech, 2022, pp. 4885–4889

2000
[11]

Joint prediction and denoising for large-scale multilingual self-supervised learning,

W. Chen, J. Shi, B. Yan, D. Berrebbiet al., “Joint prediction and denoising for large-scale multilingual self-supervised learning,” inProc. ASRU, 2023, pp. 1–8

2023
[12]

Scaling speech technology to 1,000+ languages,

V . Pratap, A. Tjandra, B. Shi, P. Tomaselloet al., “Scaling speech technology to 1,000+ languages,”Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024

2024
[13]

Towards robust speech representation learning for thousands of languages,

W. Chen, W. Zhang, Y . Peng, X. Liet al., “Towards robust speech representation learning for thousands of languages,” in Proc. EMNLP, 2024, pp. 10 205–10 224

2024
[14]

mHuBERT- 147: A compact multilingual HuBERT model,

M. Zanon Boito, V . Iyer, N. Lagos, L. Besacieret al., “mHuBERT- 147: A compact multilingual HuBERT model,” inProc. Inter- speech, 2024, pp. 3939–3943

2024
[15]

A configurable multilingual model is all you need to recognize all languages,

L. Zhou, J. Li, E. Sun, and S. Liu, “A configurable multilingual model is all you need to recognize all languages,” inProc. ICASSP, 2021, pp. 6422–6426

2021
[16]

Robust speech recognition via large-scale weak supervision,

A. Radford, J. W. Kim, T. Xu, G. Brockmanet al., “Robust speech recognition via large-scale weak supervision,” inProc. ICML, 2023, pp. 28 492–28 518

2023
[17]

OWSM v3.1: Bet- ter and faster open Whisper-style speech models based on E- Branchformer,

Y . Peng, J. Tian, W. Chen, S. Aroraet al., “OWSM v3.1: Bet- ter and faster open Whisper-style speech models based on E- Branchformer,” inProc. Interspeech, 2024, pp. 352–356

2024
[18]

OWSM-CTC: An open encoder-only speech foundation model for speech recog- nition, translation, and language identification,

Y . Peng, Y . Sudo, M. Shakeel, and S. Watanabe, “OWSM-CTC: An open encoder-only speech foundation model for speech recog- nition, translation, and language identification,” inProc. ACL, 2024, pp. 10 192–10 209

2024
[19]

ML-SUPERB 2.0: Benchmarking multilingual speech models across modeling constraints, languages, and datasets,

J. Shi, S.-H. Wang, W. Chen, M. Barteldset al., “ML-SUPERB 2.0: Benchmarking multilingual speech models across modeling constraints, languages, and datasets,” inProc. Interspeech, 2024, pp. 1230–1234

2024
[20]

ML-SUPERB: Multilingual speech universal performance benchmark,

J. Shi, D. Berrebbi, W. Chen, H.-L. Chunget al., “ML-SUPERB: Multilingual speech universal performance benchmark,” inProc. Interspeech, 2023, pp. 884–888

2023
[21]

Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,

A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Con- nectionist temporal classification: labelling unsegmented se- quence data with recurrent neural networks,” inProc. ICML, 2006, pp. 369–376

2006
[22]

Deja-vu: Dou- ble feature presentation and iterated loss in deep transformer net- works,

A. Tjandra, C. Liu, F. Zhang, X. Zhanget al., “Deja-vu: Dou- ble feature presentation and iterated loss in deep transformer net- works,” inProc. ICASSP, 2020, pp. 6899–6903

2020
[23]

Intermediate loss regularization for CTC-based speech recognition,

J. Lee and S. Watanabe, “Intermediate loss regularization for CTC-based speech recognition,” inProc. ICASSP, 2021, pp. 6224–6228

2021
[24]

Relaxing the conditional indepen- dence assumption of CTC-based ASR by conditioning on inter- mediate predictions,

J. Nozaki and T. Komatsu, “Relaxing the conditional indepen- dence assumption of CTC-based ASR by conditioning on inter- mediate predictions,” inProc. Interspeech, 2021, pp. 3735–3739

2021
[25]

Improving massively mul- tilingual ASR with auxiliary CTC objectives,

W. Chen, B. Yan, J. Shi, Y . Penget al., “Improving massively mul- tilingual ASR with auxiliary CTC objectives,” inProc. ICASSP, 2023, pp. 1–5

2023
[26]

Massively multilin- gual ASR: A lifelong learning solution,

B. Li, R. Pang, Y . Zhang, T. N. Sainathet al., “Massively multilin- gual ASR: A lifelong learning solution,” inProc. ICASSP, 2022, pp. 6397–6401

2022
[27]

Making more of little data: Improving low-resource automatic speech recognition using data augmentation,

M. Bartelds, N. San, B. McDonnell, D. Jurafskyet al., “Making more of little data: Improving low-resource automatic speech recognition using data augmentation,” inProc. ACL, 2023, pp. 715–729

2023
[28]

Audio augmentation for speech recognition,

T. Ko, V . Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” inProc. Interspeech, 2015, pp. 3586–3589

2015
[29]

Specaugment: A simple data augmentation method for automatic speech recog- nition,

D. S. Park, W. Chan, Y . Zhang, C.-C. Chiuet al., “Specaugment: A simple data augmentation method for automatic speech recog- nition,” inProc. Interspeech, 2019, pp. 2613–2617

2019
[30]

SSHR: Leveraging self-supervised hierarchical representations for multilingual auto- matic speech recognition,

H. Xue, Q. Shao, K. Huang, P. Chenet al., “SSHR: Leveraging self-supervised hierarchical representations for multilingual auto- matic speech recognition,” inProc. ICME, 2024, pp. 1–6

2024
[31]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreitet al., “Attention is all you need,” inProc. NeurIPS, 2017, pp. 6000–6010

2017
[32]

E-Branchformer: Branch- former with enhanced merging for speech recognition,

K. Kim, F. Wu, Y . Peng, J. Panet al., “E-Branchformer: Branch- former with enhanced merging for speech recognition,” inProc. SLT, 2023, pp. 84–91

2023
[33]

Deep contextualized word representations,

M. E. Peters, M. Neumann, M. Iyyer, M. Gardneret al., “Deep contextualized word representations,” inProc. NAACL, 2018, pp. 2227–2237

2018
[34]

SU- PERB: Speech processing universal performance benchmark,

S. wen Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Laiet al., “SU- PERB: Speech processing universal performance benchmark,” in Proc. Interspeech, 2021, pp. 1194–1198

2021
[35]

LoRA: Low-rank adaptation of large language models,

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhuet al., “LoRA: Low-rank adaptation of large language models,” inProc. ICLR, 2022

2022
[36]

Common voice: A massively-multilingual speech corpus,

R. Ardila, M. Branson, K. Davis, M. Kohleret al., “Common voice: A massively-multilingual speech corpus,” inProc. LREC, 2020, pp. 4218–4222

2020
[37]

XLS-R: Self- supervised cross-lingual speech representation learning at scale,

A. Babu, C. Wang, A. Tjandra, K. Lakhotiaet al., “XLS-R: Self- supervised cross-lingual speech representation learning at scale,” inProc. Interspeech, 2022, pp. 2278–2282

2022
[38]

OWSM v4: Improving open Whisper-style speech models via data scaling and cleaning,

Y . Peng, S. Muhammad, Y . Sudo, W. Chenet al., “OWSM v4: Improving open Whisper-style speech models via data scaling and cleaning,” inProc. Interspeech, 2025

2025
[39]

ESPnet: End-to-end speech processing toolkit,

S. Watanabe, T. Hori, S. Karita, T. Hayashiet al., “ESPnet: End-to-end speech processing toolkit,” inProc. Interspeech, 2018, pp. 2207–2211

2018
[40]

Adam: A method for stochastic optimization,

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” inProc. ICLR (Poster), 2015

2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Introduction Multilingual speech processing, including tasks such as Lan- guage Identification (LID) and Automatic Speech Processing (ASR), is essential in a world with more than 7000 languages [1, p. 1]. Recent advancements in multilingual speech foun- dation models (SFM) based on self-supervised learning (SSL) have significantly expanded the scope of ro...

work page

[2] [2]

Related studies Multilingual speech foundation models.Both supervised and self-supervised training strategies are used to build multilingual SFMs. Self-supervised pre-trained SFMs like Massively Mul- tilingual Speech (MMS) [5] and XEUS [6], trained on 1400+ and 4000+ languages respectively, achieve remarkable multi- lingual performance. Meanwhile, supervi...

work page

[3] [3]

Proposed method 3.1. Training strategies with multilingual SFMs To evaluate the performance of the latest multilingual SFMs on the ML-SUPERB 2.0 benchmark, we select three representa- arXiv:2505.24200v2 [cs.SD] 3 Jun 2025 tive encoder-only SFMs as the upstream models: the SSL-based MMS [5] and XEUS [6], and supervised pre-trained OWSM- CTC [11]. The overa...

work page Pith review arXiv 2025

[4] [4]

2 Models are evaluated on the development sets due to the unavailability of the test set

Experiments General setup.We use the ML-SUPERB 2.0 official dataset from the ML-SUPERB Challenge 2025, which includes a train- ing set (220 hours, 138 languages), standard development set (41 hours, 138 languages), and dialect development set (9 hours, 56 dialects). 2 Models are evaluated on the development sets due to the unavailability of the test set. ...

work page 2025

[5] [5]

Training strategies with multilingual SFMs Table 1 shows LID and ASR results across SFMs and training strategies on ML-SUPERB 2.0

Results 5.1. Training strategies with multilingual SFMs Table 1 shows LID and ASR results across SFMs and training strategies on ML-SUPERB 2.0. On the standard set, MMS-1B with LoRA achieves the best performance (82.4% LID accu- racy, 16.3% CER). On the dialect set, fine-tuning the middle layers of MMS-1B achieves the highest LID accuracy (68.4%), while L...

work page

[6] [6]

We evalu- ate MMS, XEUS, and OWSM-CTC under downstream training with frozen upstream, upstream fine-tuning, and LoRA

Conclusion In this paper, we explore strategies to improve multilingual LID and ASR performance on ML-SUPERB 2.0. We evalu- ate MMS, XEUS, and OWSM-CTC under downstream training with frozen upstream, upstream fine-tuning, and LoRA. Fur- thermore, we incorporate data augmentation and LID CTC loss for fine-tuning. Results show that MMS-1B outperforms XEUS a...

work page 2025

[7] [7]

Acknowledgements Experiments of this work used the Bridges2 system at PSC and Delta system at NCSA through allocations CIS210014 and IRI120008P from the Advanced Cyberinfrastructure Coordi- nation Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296

work page

[8] [8]

J. S. Peter K. Austin, Ed.,The Cambridge Handbook of Endan- gered Languages, ser. Cambridge Handbooks in Language and Linguistics. Cambridge University Press, 2011

work page 2011