Pith. sign in

REVIEW 2 major objections 4 minor 63 references

The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new ASR challenge spans 149 languages and 93 dialects, and every submitted system beat the strongest self-supervised baselines.

desk verdict A solid challenge write-up whose core finding is believable, but the abstract's numbers don't match the body and the dialectal test set shares most of its corpora with the development set, so the inclusivity claims are weaker than advertised. read the letter →

arxiv 2509.07139 v1 pith:24WTOC4P submitted 2025-09-08 cs.CL eess.AS

William Chen , Chutong Meng , Jiatong Shi , Martijn Bartelds , Shih-Heng Wang , Hsiu-Hsuan Wang , Rafael Mosquera , Sara Hincapie
show 5 more authors
Dan Jurafsky Antonis Anastasopoulos Hung-yi Lee Karen Livescu Shinji Watanabe
This is my paper · ORCID
classification cs.CLeess.AS
keywords multilingualASRlanguageidentificationdialectrobustnessbenchmarkchallengeself-supervisedlearninginclusivespeechprocessingcharactererrorrateML-SUPERB2.0
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents the ML-SUPERB 2.0 Challenge, an ASR evaluation covering 149 languages and 93 accents and dialects, run on a hidden test set through an online server. It claims that all five submitted systems from three teams outperformed the strongest self-supervised baselines, with the best per-metric system cutting dialectal character error rate by 30.2 points and improving dialectal language identification by 23.0 points over the XEUS baseline. The authors see this as evidence that unconstrained community challenges can drive progress toward inclusive speech technology, while also showing that accented and dialectal speech remains the weak spot of current systems.

What carries the argument

The mechanisms are the challenge protocol and the metric suite. Participants upload model weights and inference code to an online evaluation server built on DynaBench, which runs inference on a hidden test set and returns only aggregate scores, preventing benchmark overfitting. Rankings are computed by averaging each system's rank on six metrics: Standard LID accuracy, Standard CER, CER standard deviation across languages, worst-15 CER, Dialectal LID accuracy, and Dialectal CER. The data pipeline also does linguistic normalization, such as merging Tagalog and Filipino, removing Norwegian due to conflation of written standards, and reconciling ISO codes, because label consistency is required for fair robustness metrics.

What would settle it

Take a random sample of utterances from the dialectal test set, have human annotators verify the dialect label and transcript, and recompute LID and CER on the cleaned subset; if the submitted systems' advantage over XEUS largely disappears, the claimed gains are an artifact of label noise.

Watch

Extended reading notes

Core claim

The paper's central claim is that a challenge with no restrictions on training data, architectures, or pretrained models, evaluated on a fully hidden test set spanning 149 languages and 93 language varieties, produces ASR systems that beat strong self-supervised baselines on every metric. The best submission per metric improved over XEUS by 12.4 in Standard LID accuracy, 19.3 in Standard CER, 5.8 in standard deviation of CER, 4.1 in worst-15-languages CER, 23.0 in Dialectal LID, and 30.2 in Dialectal CER. The paper also reports that supervised models like Whisper and OWSM degrade sharply on languages unseen in their training data, and that even the best challenge systems perform considerably worse on accented and dialectal data than on standard varieties.

Load-bearing premise

The whole evaluation stands on the source corpora's language, accent, and dialect labels and transcripts being correct; if those labels are wrong or noisy, the rankings and fairness conclusions do not follow.

Editorial extensions

If this is right

  • If the results hold, unconstrained shared tasks become a dependable mechanism for pushing multilingual and dialectal ASR beyond self-supervised fine-tuning baselines.
  • The large remaining gap on accented and dialectal speech means future benchmarks and models must treat dialect robustness as a separate objective, not a byproduct of language coverage.
  • The hidden-test-set server design blocks benchmark overfitting, so the measured gains are more likely to transfer to new speech data than gains from open test sets.
  • The use of average rank across six metrics makes leaderboard positions less sensitive to the different dynamic ranges of CER and LID accuracy.
  • Comparing supervised and self-supervised baselines shows that training-data language coverage is the dominant factor in cross-lingual performance, so scaling pretraining data may matter more than architecture choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dialectal gains may come less from novel architectures than from the freedom to curate external training data and ensemble models; a follow-up ablation separating data from modeling would test that.
  • Since the dialectal test set draws on a finite set of accent corpora, the challenge measures robustness to those particular varieties; a future round with entirely unseen accent sources would reveal how much of the gain is generic.
  • The seen-versus-unseen collapse of supervised models suggests the challenge could double as an audit tool for which languages are actually represented in a pretraining corpus.
  • Rank-based aggregation can hide large absolute-score gaps; reporting raw deltas alongside ranks would give a fuller picture of system differences.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper reports on the Interspeech 2025 ML-SUPERB 2.0 Challenge, an open ASR competition in which participants submit systems through a DynaBench-based server and are evaluated on a multilingual test set spanning 149 languages and a newly collected accented/dialectal test set spanning 93 varieties. The authors describe data cleaning decisions, challenge rules, and evaluation metrics (Standard LID/CER, language robustness StD/Worst-15 CER, and dialectal LID/CER), and compare five submissions from three teams against SSL and supervised baselines. They report that all submissions ranked above the best SSL baselines, with per-metric improvements over XEUS of 12.4 in Standard LID, 19.3 in Standard CER, 5.8 in StD, 4.1 in Worst-15 CER, 23.0 in Dialect LID, and 30.2 in Dialect CER.

Significance. If the reported results are taken at face value, the challenge makes a useful contribution: it is one of the broadest multilingual ASR evaluations to date, it includes an online evaluation server that keeps the test set hidden from participants, and it provides evidence that unconstrained community submissions can improve over standard SSL fine-tuning on both standard and dialectal test sets. Strengths include transparent documentation of data-cleaning decisions, a fixed ranking protocol based on average rank across six metrics, and release of the evaluation infrastructure. The empirical claim about dialectal robustness is weakened by the partial overlap between development and test sources, and the headline numbers in the abstract do not match Section 4.1, so the results as currently presented need revision before the significance can be fully assessed.

major comments (2)
  1. [Abstract and Section 4.1] The abstract states a 23% LID improvement and an 18% CER reduction on the general multilingual test set, and a 15.7% LID improvement on accented/dialectal data. Section 4.1 reports improvements over XEUS of 12.4 in Standard LID, 19.3 in Standard CER, 23.0 in Dialect LID, and 30.2 in Dialect CER, with no 15.7 value appearing anywhere in the paper. The abstract's 23% matches the Dialect LID number, not the general-set LID number; the 18% matches neither Standard CER (19.3) nor Dialect CER (30.2); and 15.7 is unexplained. This makes the headline result ambiguous and must be corrected so the reader knows which metric each number refers to.
  2. [Section 2.2 and Section 3.5] The dialectal development set is drawn from 9 corpora [25-33] and the hidden test set is drawn from the same 9 corpora plus 4 additional ones [34-37]. Because participants receive the development set (Section 3.5) and may use any data, 9 of the 13 test-source corpora are effectively public during system development, making the test set a near-perfect distributional proxy for the development set for those corpora. The reported dialectal gains (23.0 in LID, 30.2 in CER) may therefore overstate robustness to unseen varieties. Please report a per-corpus breakdown separating the 9 shared corpora from the 4 held-out corpora and discuss whether the aggregate conclusion holds on the 4 truly unseen corpora.
minor comments (4)
  1. [Section 3.2] Section 3.2 says participants are tasked with developing systems for 154 languages, while Section 2.1 and the introduction consistently say 149 languages; please align these numbers.
  2. [Abstract and Conclusion] The phrase '200+ languages, accents, and dialects' is imprecise: the paper evaluates 149 languages and 93 accents/dialects, which are not both languages; consider writing 'more than 200 language varieties and accents' or stating the two numbers explicitly.
  3. [Section 4.1] The claim that 'each team had a system submission that ranked 1st in at least 1 metric' would be easier to verify if the per-metric ranks of the five submissions were shown in a table; Figure 3 is hard to read at the level of individual metrics.
  4. [Section 3.7] The ranking is based on only seven systems (five submissions plus two baselines), and no confidence intervals or significance tests are reported; the authors should note that small rank differences are not necessarily meaningful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: independent external evaluation, fixed metrics, and transparent data provenance; dev/test corpus overlap is a validity concern, not a circular derivation.

full rationale

This paper is an empirical challenge report rather than a derivation chain. The central claims—that five submitted systems outperformed the SSL baselines on standard and dialectal metrics—are supported by blind inference on a hidden test set via the DynaBench server (Section 3.3), with metrics fixed in advance in Section 3.6 (LID ACC, CER, StD, Worst-15 CER, Dialectal LID, Dialectal CER). No parameter is fitted to the test data and no 'prediction' is derived from an input that already contains the outcome. Self-citations to earlier ML-SUPERB papers [11, 13, 14] are used as transparent data provenance and baseline construction references, not as load-bearing justification for the reported rankings. The skeptical concern that the dialectal development set shares 9 of 13 source corpora with the hidden test set (Section 2.2) is a benchmark-design or generalization-validity issue, and the abstract/Section 4.1 numerical discrepancy (23%/18% vs. 12.4/19.3) is a reporting inconsistency; neither constitutes circularity under the specified criteria. No equation reduction, fitted-input renaming, or uniqueness-imported-by-self-citation is present. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted. The challenge's empirical claims rest on external corpora, the aggregation metrics, and data-cleaning decisions. These are domain assumptions rather than adjustable constants, and they are stated in Sections 2 and 3.6.

assumptions (3)
  • domain assumption Transcripts and language or dialect labels in the source corpora are accurate.
    The reported CER and LID scores are computed directly from these labels; if the labels are wrong, the rankings and fairness conclusions are compromised. This assumption enters in Section 2 during data collection and in Section 3.6 during metric computation.
  • domain assumption Macro-averaged CER and LID across languages are a valid measure of model quality and inclusivity.
    The challenge ranking gives equal weight to each language and variety. This choice is made in Section 3.6, and it assumes per-language errors can be averaged without considering language difficulty or speaker populations.
  • domain assumption Merging and removal decisions for languages (for example merging Filipino and Tagalog, or removing Norwegian) correctly reflect linguistic identity.
    These curation decisions in Section 2.1 are necessary for fair cross-corpus evaluation, but they rely on the authors' linguistic judgments about which written standards and codes represent the same or different varieties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties." pith.science (2026). https://pith.science/paper/24WTOC4P

@misc{pith2026250907139,
  author       = {Pith},
  title        = {Pith review of: The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24WTOC4P}},
  note         = {Machine review of arXiv:2509.07139}
}
read the original abstract

Recent improvements in multilingual ASR have not been equally distributed across languages and language varieties. To advance state-of-the-art (SOTA) ASR models, we present the Interspeech 2025 ML-SUPERB 2.0 Challenge. We construct a new test suite that consists of data from 200+ languages, accents, and dialects to evaluate SOTA multilingual speech models. The challenge also introduces an online evaluation server based on DynaBench, allowing for flexibility in model design and architecture for participants. The challenge received 5 submissions from 3 teams, all of which outperformed our baselines. The best-performing submission achieved an absolute improvement in LID accuracy of 23% and a reduction in CER of 18% when compared to the best baseline on a general multilingual test set. On accented and dialectal data, the best submission obtained 30.2% lower CER and 15.7% higher LID accuracy, showing the importance of community challenges in making speech technologies more inclusive.

Figures

Figures reproduced from arXiv: 2509.07139 by the authors.

Figure 1
Figure 1. Overview of the challenge submission system. Participants upload their model weights and inference code to DynaBench, which runs the evaluation online and returns the evaluation metrics. observed in Serbian: FLEURS [22] had Latin transcripts while Common Voice [23] used Cyrillic. Normalizing these factors is particularly important for our challenge, as otherwise our met￾rics on fairness and robustness will be severe… view at source ↗
Figure 2
Figure 2. Comparison of different aggregation methods used to compute the final rankings across all metrics. Accuracies are inverted by subtracting from 100 when calculating the ag￾gregate raw scores, such that a lower score is better. Standard: This category mostly follows the original scoring system of ML-SUPERB’s multilingual ASR+LID task with two main metrics: average language identification accuracy (ACC) and ASR charact… view at source ↗
Figure 3
Figure 3. Raw scores/rankings of each system submission and the best 2 SSL model baselines on our challenge evaluation metrics [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 52 canonical work pages

  1. [1]

    standard

    Introduction In the past decade, studies on scaling end-to-end neural net- works have led to dramatic improvements in models for Au- tomatic Speech Recognition (ASR) [1, 2]. Importantly, ASR systems are no longer limited to solely the English language: state-of-the-art (SOTA) models achieve strong performance on over 50 languages [3–5]. However, these ben...

    work page 2025

  2. [2]

    We introduce a new challenge that evaluates multilingual ASR performance across 149 languages and 93 language varieties, representing the broadest coverage of any speech benchmark to date

    work page

  3. [3]

    We compare 5 submitted systems, which all out-performed our baseline systems, showing that community challenges can lead to better-performing systems

    work page

  4. [4]

    Despite these advancements, we find that SOTA ASR sys- tems continue to underperform on accented and dialectal speech

    work page

  5. [5]

    The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties

    Challenge Data 2.1. General Multilingual Data The data described in this section is designed to evaluate the general multilingual capabilities of ASR models across 149 languages. We obtain this data by combining previous ML- SUPERB benchmarks [11, 13, 14]. In doing so, we found sev- eral issues with the data used in these existing benchmarks or the corpor...

    work page Pith review arXiv 2025

  6. [6]

    The hidden test set con- tains data sourced from the same corpora as the development set along with 4 additional corpora [34–37]

    The development set contains data sourced from 9 accented or dialectal speech corpora [25–33]. The hidden test set con- tains data sourced from the same corpora as the development set along with 4 additional corpora [34–37]. While we list these datasets here for transparency, participants were not made aware of which datasets were used during the challeng...

    work page

  7. [7]

    "" 3Args: 4waveform (np.array): speech waveform 5Returns: 6pred_lid (str): ISO3 code of LID pred 7pred_asr (str): predicted transcript 8

    Challenge Task and Rules 3.1. Updates from Previous Challenges and Benchmarks The goal of this challenge is to encourage the development of ASR systems that are robust to languages, accents, and di- alects. Importantly, we avoid constraining participants to cer- tain datasets or modeling approaches. This is distinct from the goals of previous ML-SUPERB be...

    work page 2021

  8. [8]

    Calculate rankings for each model on each metric

    work page

Show all 63 references
  1. [9]

    Calculate the average ranking for each model across met- rics

  2. [10]

    Rank models by average ranking. We use the average rank of a system rather than the average of the raw metric scores to avoid the ranking being skewed by differences in the metrics’ dynamic ranges, which was effective in previous speech challenges [43, 44]. For example, the fi...

  3. [11]

    Since all of these models are self-supervised, we develop ASR systems via fine-tuning on the ML-SUPERB 2.0 public set [14]

    Benchmark and Submission Results Self-Supervised Models:We obtained baseline results with 7 systems based on SSL speech encoders: WavLM [45], XLSR- 53 [46], XLS-R 128 300M [47], XLS-R 128 1B [47], MMS 1B [17], w2v-BERT 2.0 [24], and XEUS [18]. Since all of these models are sel...

  4. [12]

    Superb@ slt 2022: Challenge on general- ization and efficiency of self-supervised speech representation learning,

    T.-h. Feng et al., “Superb@ slt 2022: Challenge on general- ization and efficiency of self-supervised speech representation learning,” inSLT, 2023, pp. 1096–1103

    2022

  5. [13]

    The challenge introduces a novel multilin- gual test suite of accented and dialect speech and uses new metrics to test the robustness of ASR systems

    Conclusion We propose the ML-SUPERB 2.0 Challenge, a novel speech processing challenge with the goal of inclusive ASR bench- marking for a large number (200+) of languages, accents, and dialects. The challenge introduces a novel multilin- gual test suite of accented and dialec...

  6. [14]

    Wav2vec 2.0: A framework for self- supervised learning of speech representations,

    A. Baevski et al., “Wav2vec 2.0: A framework for self- supervised learning of speech representations,”Proc. NeurIPS, vol. 33, pp. 12 449–12 460, 2020

    2020

  7. [15]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu et al., “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”TASLP, vol. 29, pp. 3451–3460, 2021

    2021

  8. [16]

    Robust speech recognition via large-scale weak supervision,

    A. Radford et al., “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning, 2023, pp. 28 492–28 518

    2023

  9. [17]

    Owsm v3. 1: Better and faster open whisper- style speech models based on e-branchformer,

    Y . Peng et al., “Owsm v3. 1: Better and faster open whisper- style speech models based on e-branchformer,”arXiv preprint arXiv:2401.16658, 2024

    2024 arXiv

  10. [18]

    Google usm: Scaling automatic speech recogni- tion beyond 100 languages,

    Y . Zhang et al., “Google usm: Scaling automatic speech recogni- tion beyond 100 languages,”arXiv preprint arXiv:2303.01037, 2023

    2023 arXiv

  11. [19]

    Self-supervised speech representations still struggle with african american vernacular english,

    K. Chang et al., “Self-supervised speech representations still struggle with african american vernacular english,” inINTER- SPEECH, 2024

    2024

  12. [20]

    Towards inclusive automatic speech recognition,

    S. Feng et al., “Towards inclusive automatic speech recognition,” Computer Speech & Language, vol. 84, p. 101 567, 2024

    2024

  13. [21]

    Findings of the IWSLT 2023 Evaluation Campaign,

    M. Agarwal et al., “Findings of the IWSLT 2023 Evaluation Campaign,” inProc. IWSLT, 2023

    2023

  14. [22]

    SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Genera- tive Capabilities,

    H.-S. Tsai et al., “SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Genera- tive Capabilities,” inACL, 2022

    2022

  15. [23]

    A V-SUPERB: A Multi-Task Evaluation Bench- mark for Audio-Visual Representation Models,

    Y . Tseng et al., “A V-SUPERB: A Multi-Task Evaluation Bench- mark for Audio-Visual Representation Models,” inICASSP, 2024

    2024

  16. [24]

    ML-SUPERB: Multilingual Speech Universal PERformance Benchmark,

    J. Shi et al., “ML-SUPERB: Multilingual Speech Universal PERformance Benchmark,” inProc. INTERSPEECH, 2023

    2023

  17. [25]

    Sada: Saudi audio dataset for arabic,

    S. Alharbi et al., “Sada: Saudi audio dataset for arabic,” in ICASSP, 2024, pp. 10 286–10 290

    2024

  18. [26]

    Findings of the 2023 ML-SUPERB Challenge: Pre- Training And Evaluation Over More Languages And Beyond,

    J. Shi et al., “Findings of the 2023 ML-SUPERB Challenge: Pre- Training And Evaluation Over More Languages And Beyond,” inASRU, 2023

    2023

  19. [27]

    ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets,

    J. Shi et al., “ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets,” inProc. INTERSPEECH, 2024

    2024

  20. [28]

    SUPERB: Speech Processing Universal PERfor- mance Benchmark,

    S. Yang et al., “SUPERB: Speech Processing Universal PERfor- mance Benchmark,” inInterspeech, 2021

    2021

  21. [29]

    Dynabench: Rethinking benchmarking in NLP,

    D. Kiela et al., “Dynabench: Rethinking benchmarking in NLP,” inNAACL, 2021

    2021

  22. [30]

    Scaling speech technology to 1,000+ lan- guages,

    V . Pratap et al., “Scaling speech technology to 1,000+ lan- guages,”JMLR, vol. 25, no. 97, pp. 1–52, 2024

    2024

  23. [31]

    Towards robust speech representation learning for thousands of languages,

    W. Chen et al., “Towards robust speech representation learning for thousands of languages,”arXiv preprint arXiv:2407.00837, 2024

    2024 arXiv

  24. [32]

    Artie bias corpus: An open dataset for detecting demographic bias in speech applications,

    J. Meyer et al., “Artie bias corpus: An open dataset for detecting demographic bias in speech applications,” English, inLREC, N. Calzolari et al., Eds., 2020, pp. 6462–6468

    2020

  25. [33]

    Speech Accent Archive,

    S. Weinberger, “Speech Accent Archive,”Retrieved from http://accent.gmu.edu, 2015

    2015

  26. [34]

    Towards measuring fairness in speech recognition: Fair-speech dataset,

    I.-E. Veliche et al., “Towards measuring fairness in speech recognition: Fair-speech dataset,”arXiv preprint arXiv:2408.12734, 2024

    2024 arXiv

  27. [35]

    Fleurs: Few-shot learning evaluation of uni- versal representations of speech,

    A. Conneau et al., “Fleurs: Few-shot learning evaluation of uni- versal representations of speech,” inSLT, 2023, pp. 798–805

    2023

  28. [36]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila et al., “Common voice: A massively-multilingual speech corpus,” inProc. LREC, 2020, pp. 4218–4222

    2020

  29. [37]

    Seamlessm4t-massively multilingual & mul- timodal machine translation,

    L. Barrault et al., “Seamlessm4t-massively multilingual & mul- timodal machine translation,”arXiv preprint arXiv:2308.11596, 2023

    2023 arXiv

  30. [38]

    Findings of the WMT 2021 shared task on large-scale multilingual machine translation,

    G. Wenzek et al., “Findings of the WMT 2021 shared task on large-scale multilingual machine translation,” inProc. WMT, L. Barrault et al., Eds., 2021, pp. 89–99

    2021

  31. [39]

    V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,

    C. Wang et al., “V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” inACL, 2021, pp. 993–1003

    2021

  32. [40]

    Open-source multi-speaker corpora of the English accents in the British isles,

    I. Demirsahin et al., “Open-source multi-speaker corpora of the English accents in the British isles,” eng, inLREC, N. Calzolari et al., Eds., 2020, pp. 6532–6541

    2020

  33. [41]

    Globe: A high-quality english corpus with global accents for zero-shot speaker adaptive text- to-speech,

    W. Wang, Y . Song, and S. Jha, “Globe: A high-quality english corpus with global accents for zero-shot speaker adaptive text- to-speech,” inInterspeech, 2024, pp. 1365–1369

    2024

  34. [42]

    L2-arctic: A non-native english speech corpus,

    G. Zhao et al., “L2-arctic: A non-native english speech corpus,” inInterspeech, 2018, pp. 2783–2787

    2018

  35. [43]

    Dogan-Schönberger, J

    P. Dogan-Schönberger, J. Mäder, and T. Hofmann,Swissdial: Parallel multidialectal corpus of spoken swiss german, 2021

    2021

  36. [44]

    Speech recognition for greek dialects: A challenging benchmark,

    S. Vakirtzian et al., “Speech recognition for greek dialects: A challenging benchmark,” inInterspeech, 2024, pp. 3974–3978

    2024

  37. [45]

    Interspeech 2018 low resource au- tomatic speech recognition challenge for indian languages,

    B. M. L. Srivastava et al., “Interspeech 2018 low resource au- tomatic speech recognition challenge for indian languages,” in 6th Workshop on Spoken Language Technologies for Under- Resourced Languages (SLTU 2018), 2018, pp. 11–14

    2018

  38. [46]

    Crowdsourcing Latin American Span- ish for low-resource text-to-speech,

    A. Guevara-Rukoz et al., “Crowdsourcing Latin American Span- ish for low-resource text-to-speech,” eng, inLREC, N. Calzolari et al., Eds., 2020, pp. 6504–6513

    2020

  39. [47]

    Leveraging data collection and un- supervised learning for code-switched tunisian arabic automatic speech recognition,

    A. A. B. Abdallah et al., “Leveraging data collection and un- supervised learning for code-switched tunisian arabic automatic speech recognition,” inICASSP, 2024, pp. 12 607–12 611

    2024

  40. [48]

    Casablanca: Data and models for multidialec- tal arabic speech recognition,

    B. Talafha et al., “Casablanca: Data and models for multidialec- tal arabic speech recognition,”arXiv preprint arXiv:2410.04527, 2024

    2024 arXiv

  41. [49]

    Automatic speech recognition datasets in Can- tonese: A survey and new dataset,

    T. Yu et al., “Automatic speech recognition datasets in Can- tonese: A survey and new dataset,” inLREC, N. Calzolari et al., Eds., 2022, pp. 6487–6494

    2022

  42. [50]

    These are run in azero-shot manner, as they are designed to be used out-of-the-box

    and Whisper Large v3 [3]. These are run in azero-shot manner, as they are designed to be used out-of-the-box. 4.1. Results Supervised vs Self-Supervised:Table 2 shows the scores of each supervised and SSL model. We find that SSL models gen- Table 3:CERs of supervised models wh...

  43. [51]

    Finnish dialect identification: The effect of audio and text,

    M. Hämäläinen et al., “Finnish dialect identification: The effect of audio and text,” inEMNLP, M.-F. Moens et al., Eds., 2021, pp. 8777–8783

    2021

  44. [52]

    Findings of the 2021 conference on ma- chine translation (WMT21),

    F. Akhbardeh et al., “Findings of the 2021 conference on ma- chine translation (WMT21),” inProc. WMT, L. Barrault et al., Eds

    2021

  45. [53]

    OWSM v3.1: Better and Faster Open Whisper- Style Speech Models based on E-Branchformer,

    Y . Peng et al., “OWSM v3.1: Better and Faster Open Whisper- Style Speech Models based on E-Branchformer,” inProc. IN- TERSPEECH, 2024

    2024

  46. [54]

    LLaMA: Open and efficient foundation lan- guage models,

    H. Touvron et al., “LLaMA: Open and efficient foundation lan- guage models,”arxiv:2302.13971, 2023

    2023 arXiv

  47. [55]

    Language models are few-shot learners,

    T. Brown et al., “Language models are few-shot learners,” in Proc. NeurIPS, vol. 33, 2020, pp. 1877–1901

    2020

  48. [56]

    The interspeech 2024 challenge on speech pro- cessing using discrete units,

    X. Chang et al., “The interspeech 2024 challenge on speech pro- cessing using discrete units,”arXiv preprint arXiv:2406.07725, 2024

    2024 arXiv

  49. [57]

    NeurIPS 2024 competition proposal: UR- GENT challenge,

    W. Zhang et al., “NeurIPS 2024 competition proposal: UR- GENT challenge,” inNeurIPS 2024 Competition Track, 2024

    2024

  50. [58]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”JSTSP, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  51. [59]

    Unsupervised cross-lingual represen- tation learning for speech recognition,

    A. Conneau et al., “Unsupervised cross-lingual represen- tation learning for speech recognition,”arXiv preprint arXiv:2006.13979, 2020

    2006 arXiv

  52. [60]

    Xls-r: Self-supervised cross-lingual speech rep- resentation learning at scale,

    A. Babu et al., “Xls-r: Self-supervised cross-lingual speech rep- resentation learning at scale,”arXiv preprint arXiv:2111.09296, 2021

    2021 arXiv

  53. [61]

    Attention is all you need,

    A. Vaswani et al., “Attention is all you need,” inNeurIPS 2017, 2017

    2017

  54. [62]

    Connectionist temporal classification: La- belling unsegmented sequence data with recurrent neural net- works,

    A. Graves et al., “Connectionist temporal classification: La- belling unsegmented sequence data with recurrent neural net- works,” inICML 2006, 2006, pp. 369–376

    2006

  55. [63]

    OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models,

    W. Chen et al., “OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models,”arXiv preprint arXiv:2502.10373, 2025

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.