Pith. sign in

REVIEW 2 major objections 6 minor 61 references

mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read mSTEB is a new benchmark that tests LLMs on speech in over 100 languages and on text in over 200, and it finds the worst performance among African and Americas/Oceania languages.

desk verdict Useful benchmark artifact with a real confound in the NLI regional result; worth engaging, but the headline gap needs a difficulty-matched check. read the letter →

arxiv 2506.08400 v3 pith:WL42RQ7Z submitted 2025-06-10 cs.CL cs.LGcs.SDeess.AS

Luel Hagos Beyene , Vivek Verma , Min Ma , Jesujoba O. Alabi , Fabian David Schmidt , Joyce Nakatumba-Nabende , David Ifeoluwa Adelani This is my paper
classification cs.CLcs.LGcs.SDeess.AS
keywords multilingualevaluationspeechrecognitionlow-resourcelanguageslanguageidentificationmachinetranslationnaturalinferenceaudiomodelsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces mSTEB, a unified evaluation benchmark that tests large language models (LLMs) on five speech tasks and five text tasks across more than 100 and 200 languages respectively. The tasks are language identification, topic classification, reading-comprehension question answering, translation, plus automatic speech recognition for speech and natural language inference for text. Evaluating five current LLMs, the authors report a wide performance gap between high-resource and low-resource languages, with the lowest scores concentrated in Africa and Americas/Oceania. The paper also finds that the open-weight speech model Qwen 2 Audio lags the proprietary models on every speech task, while the open-weight text model Gemma 3 27B is nearly competitive on text. If the benchmark is a fair instrument, these results make the uneven multilingual coverage of today's LLMs concrete and measurable, and point to speech understanding as a particularly weak spot for low-resource languages.

What carries the argument

The central object is the mSTEB benchmark itself, a task-aligned aggregation of existing datasets. Fleurs supplies the speech audio for language identification, ASR, and speech-to-text translation; SIB-Fleurs and Belebele-Fleurs provide audio topic classification and audio reading comprehension; Flores-200 supplies text translation and text LID; SIB-200 supplies text topic classification; Belebele supplies text reading comprehension; and the newly assembled Global NLI provides text NLI across 59 languages. By aligning tasks across modalities, the benchmark lets a reader compare a model's speech and text ability on the same kind of question, and to group results by region and language family. The same prompting and scoring protocols are applied to every model, with ChrF++ for translation and character error rate for ASR, so the numbers are directly comparable across systems.

What would settle it

Re-run the NLI evaluation using only the 33 languages with genre-balanced 600-sample test sets; if the Americas/Oceania region then no longer trails Europe (W, N, S) by roughly 35 points, the claimed NLI gap is a dataset-composition artifact rather than a model capability gap.

Watch

Extended reading notes

Core claim

On its own terms, mSTEB establishes a standardized cross-modal yardstick: the same evaluation recipe applied to speech and text, using Fleurs for the audio side and Flores-200, SIB-200, Belebele, and gathered NLI resources for the text side. Across this breadth, the authors find that proprietary models (Gemini 2.0 Flash, GPT-4o, GPT-4o Audio) consistently beat the open-weight models (Qwen 2 Audio, Gemma 3 27B), and that the largest gaps appear in Africa and Americas/Oceania in both modalities. The paper's central claim is that these regional and resource-level gaps reflect genuine under-representation of low-resource languages in LLM training data, and that closing them requires deliberate investment. The benchmark is offered as a tool for tracking such progress over time.

Load-bearing premise

The Global NLI scores treat all 59 languages as equally comparable, but only 33 of them share the same balanced, genre-matched sample design, so the apparent NLI weakness of Americas/Oceania may be an artifact of which datasets were included.

Editorial extensions

If this is right

  • Future LLM releases can be checked against a single 200+ language, two-modality benchmark, replacing English-first or few-language evaluation suites.
  • The large open-versus-proprietary gap on speech suggests that open-weight audio LLMs are not yet usable for most low-resource languages, a finding that should shape where open-model training effort goes.
  • Gemini 2.0 Flash beating SeamlessM4T v2 on ASR and S2TT indicates that general-purpose speech LLMs are becoming competitive with dedicated speech translation systems, even on low-resource languages.
  • The regional analysis identifies Africa and Americas/Oceania as the clearest priorities for data collection and model investment in both modalities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Global NLI's regional gaps are partly an artifact of its unbalanced design (only 33 of 59 languages are genre-matched), then mSTEB's Americas/Oceania NLI ranking may overstate model weakness; a balanced NLI subset across all 59 languages would be a direct test.
  • Because the paper excludes machine-translated datasets by design, mSTEB leaves out many existing low-resource resources; a variant that admits carefully audited pseudo-parallel data could extend coverage beyond 200 text languages.
  • The paper selects a single best prompt per task from an earlier multilingual benchmark study, so reported scores reflect prompt choice as well as model ability; re-running with multiple prompts would separate prompt incompatibility from true capability gaps.
  • The speech and text versions of 'the same' tasks use different source sentences (Fleurs audio vs Flores text), so the modality gap is approximate; a strictly parallel speech-text corpus would make the cross-modal comparison exact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. mSTEB introduces a massively multilingual benchmark that aggregates existing human-curated datasets (Flores-200, SIB-200, Belebele, FLEURS, FLEURS-SLU, and a newly assembled Global NLI) to evaluate LLMs on five speech tasks (LID, topic classification, RC-QA, ASR, S2TT) and five text tasks (LID, topic classification, RC-QA, MT, NLI) across more than 100 speech languages and 200 text languages. The authors evaluate two open-weight models (Qwen 2 Audio, Gemma 3 27B) and three proprietary models (GPT-4o, GPT-4o Audio, Gemini 2.0 Flash), along with task-specific baselines. The headline findings are a large performance gap between high- and low-resource languages, especially in Africa and Americas/Oceania, and that Qwen 2 Audio underperforms proprietary models on every speech task. The paper releases code, data, and a leaderboard.

Significance. If validated, mSTEB would be a valuable community resource: it is the first benchmark, to my knowledge, that jointly covers a broad set of speech and text tasks at this language scale, and it uses standard metrics (accuracy, CER, ChrF++) with several strong task-specific baselines. The paper is transparent about its composition criteria (human-curated data, regional coverage) and releases code, data, and a public leaderboard, which supports reproducibility. The empirical analyses by region, language family, and error type are informative and likely to be cited. The main contribution—a unified, massively multilingual, multimodal benchmark—fills a genuine gap, and the headline finding of a large low-resource gap is important if supported by the evidence.

major comments (2)
  1. [Section III (Global NLI) and Section V.A.2 / Table IV] The NLI regional comparison is confounded by dataset composition. Global NLI is assembled from 13 source datasets, and only 33 of 59 languages share the same balanced, genre-matched premise/hypothesis templates (XNLI, AfriXNLI, IndicXNLI, XNLI-ca, myXNLI). The remaining 26 languages, including 11 from AmericasNLI, come from datasets with different genres, content, and intrinsic difficulty. Table IV shows Americas/Oceania NLI accuracy of 46.1–49.1 versus 82.7–84.8 for Western Europe, and the paper uses this to support the claim that Americas/Oceania is the weakest region in text. However, the paper's own discussion in Section V.A.2 concedes that this gap may reflect the presence of truly low-resource AmericasNLI languages (e.g., Otomi, Wixarika) rather than model capability alone. Because the regional NLI scores are not directly comparable across regions, the headline claim about Americas/Oceania being the weakest text region rests on a confounded comparison. To make the claim load-bearing, the paper should either restrict the regional NLI analysis to the 33 languages that share the same parallel data, provide a per-dataset breakdown (e.g., AmericasNLI vs. the balanced subset) so that dataset difficulty can be disentangled from language difficulty, or explicitly moderate the claim and reframe the NLI result as exploratory and dataset-sensitive.
  2. [Section IV (Experimental Setup) and Section V (Results)] Every model-task pair is run exactly once with a single prompt, and no variance estimation or significance testing is reported. The paper states that it selected the best prompt from AfroBench and then ran each task only once. This means the reported differences—such as the ranking of GPT-4o Audio vs. Gemini 2.0 Flash on ASR, or the regional gaps in Table IV—are point estimates that could be sensitive to prompt wording and to stochastic decoding (especially for the API models). While the largest gaps (e.g., Qwen 2 Audio vs. Gemini 2.0 Flash on speech) are so wide that they are almost certainly robust, many smaller differences (e.g., 1–3 point gaps in Table IV) may not be meaningful. The paper should at least report confidence intervals for a subset of configurations (e.g., a few languages per region) or rerun with multiple prompts/seeds for key comparisons, or alternatively frame all comparative claims as provisional single-run observations. This is a standard expectation for a benchmark paper that makes strong comparative statements.
minor comments (6)
  1. [Section III (Global NLI)] The statement that 'For all languages except Romanian, they have an equal number of entailment, neutral, and contradiction labels' leaves the Romanian exception unexplained; please clarify the label distribution for Romanian and consider whether its 600-sample test set is comparable to the others.
  2. [Table I] The number of languages listed for LID (102) differs from ASR and S2TT (101). Please explain this discrepancy (e.g., a missing audio split for one language).
  3. [Section V.A.2] The sentence 'we observe the Americas region also has high performance across tasks including MT, this is not true for NLI where it achieved the lowest (46.8)' should specify that 46.8 is the Gemini score, not the GPT-4o score (which is 49.1); also cross-referencing Table IV would help.
  4. [Section V.A.2] The paper says 'indigeneous' (Section V.A.2); this should be 'indigenous'.
  5. [Section V.A.1] The claim that Gemini 2.0 Flash outperforms SeamlessM4T v2 on ASR and S2TT is made textually but the baseline numbers are only in Figure 1 (not in the tables). Please include the baseline values in Table III or an appendix table so the comparison is directly verifiable.
  6. [Section II (Related Work)] The novelty claim of being the 'first' unified speech/text benchmark at this scale would be strengthened by explicitly discussing other recent multilingual/multimodal benchmark efforts (e.g., XTREME-UP, SpeechGLUE, or other audio-LLM evaluation suites) and explaining how mSTEB differs.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: mSTEB is an aggregated benchmark whose scores are external model measurements, not quantities derived from its own definitions.

full rationale

The paper's contributions are dataset aggregation and zero-shot evaluation; there is no fitted parameter or derived quantity that is then presented as a prediction. The reported scores are obtained by running external models (GPT-4o, GPT-4o Audio, Gemini 2.0 Flash, Qwen 2 Audio, Gemma 3 27B) and task-specific baselines on fixed public test sets, so the performance gaps cannot reduce by construction to the benchmark's definition. Several component datasets and resources are authored by members of this group (SIB-200, AfriXNLI/IrokoBench, Fleurs-SLU, AfroBench), but they are used as independent evaluation resources; citations to them are not invoked to establish the paper's conclusions, and the evaluation numbers are not determined by those datasets' construction choices. The Global NLI comparability limitation (genre balance achieved for only 33 of 59 languages, with 11 Americas/Oceania languages from AmericasNLI) is a real validity concern for the regional NLI comparison, but it is a dataset-composition confound rather than a circularity: the Americas/Oceania weakness could partly reflect AmericasNLI's intrinsic difficulty, yet this does not make the benchmark's outputs equivalent to its inputs. Therefore no circular step is identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on the validity of its component datasets and on metric/prompt choices; these are domain assumptions rather than fitted parameters, and no new physical or mathematical entities are introduced.

assumptions (4)
  • domain assumption The source datasets (FLEURS, SIB-200, Belebele, Flores-200, FLEURS-SLU) provide valid human-created labels and aligned audio-text pairs across 100+ languages.
    This is the foundation of mSTEB; if these datasets contain systematic errors or machine-translated content, the benchmark's cross-lingual comparisons are unreliable. Stated in Section III.
  • domain assumption The Global NLI test set is comparable across the 59 languages despite being aggregated from 13 datasets with different source distributions and only 33 languages having genre-balanced, parallel items.
    Section III Global NLI paragraph; the authors acknowledge genre balance only for 33 languages, so cross-region gaps may partly reflect dataset difficulty.
  • domain assumption Character-level metrics (CER, ChrF++) are valid for comparing LLM performance across languages with different scripts and orthographies.
    Section IV, Evaluation metric; they cite prior work, but it is still an assumption that these metrics are equally fair across scripts.
  • domain assumption English prompts for zero-shot evaluation do not systematically disadvantage certain languages more than others.
    Section IV, LLMs evaluated; the authors rely on prior work showing English prompts work well, but if prompt language bias interacts with language resource level, the measured gap could be inflated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks." pith.science (2026). https://pith.science/paper/WL42RQ7Z

@misc{pith2026250608400,
  author       = {Pith},
  title        = {Pith review of: mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WL42RQ7Z}},
  note         = {Machine review of arXiv:2506.08400}
}
read the original abstract

Large Language models (LLMs) have demonstrated impressive performance on a wide range of tasks, including in multimodal settings such as speech. However, their evaluation is often limited to English and a few high-resource languages. For low-resource languages, there is no standardized evaluation benchmark. In this paper, we address this gap by introducing mSTEB, a new benchmark to evaluate the performance of LLMs on a wide range of tasks covering language identification, text classification, question answering, and translation tasks on both speech and text modalities. We evaluated the performance of leading LLMs such as Gemini 2.0 Flash and GPT-4o (Audio) and state-of-the-art open models such as Qwen 2 Audio and Gemma 3 27B. Our evaluation shows a wide gap in performance between high-resource and low-resource languages, especially for languages spoken in Africa and Americas/Oceania. Our findings show that more investment is needed to address their under-representation in LLMs coverage.

Figures

Figures reproduced from arXiv: 2506.08400 by the authors.

Figure 1
Figure 1. Average results by tasks. For the ASR, we reported “100 − CER” value. evaluating diverse set of languages [55] and better correlates with human judgment [56]. We provide other metrics such as BLEU and WER in the supplementary material. Group by regions: We categorize different languages into eight specific geographical regions reflecting the resource level between regions based on SIB-200 [9]. The languages are grou… view at source ↗
Figure 2
Figure 2. Confusion matrix for LID task per system, where the predictions over 102 languages had been aggregated into [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Fine-grained error analysis for proprietary LLMs wide gap in performance between open models and proprietary models for the speech modality while in the text modality, the gap is very small. Similarly, we find multimodal LLMs to be better than their task specific baselines on more challenging speech tasks such as ASR and S2TT. Finally, we find that the languages with the lowest performance in both modalities are fro… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 42 canonical work pages

  1. [1]

    Gpt-4 technical report,

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, and et al., “Gpt-4 technical report,” 2023. [Online]. Available: https: //api.semanticscholar.org/CorpusID:257532815

    work page 2023

  2. [2]

    Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J.-M. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, and et al., “Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning,” ArXiv, vol. abs/2501.12948, 2025. [Online]. Available: https://api.semanticscholar.org/CorpusID:275789950

    arXiv 2025

  3. [4]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919 , 2023

    arXiv 2023

  4. [5]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    arXiv 2024

  5. [6]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, M. Zejun, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” in The Twelfth International Conference on Learning Representations

    work page

  6. [7]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” ArXiv, vol. abs/2304.08485, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258179774

    arXiv 2023

  7. [8]

    Pangea: A fully open multilingual multimodal llm for 39 languages,

    X. Yue, Y . Song, A. Asai, S. Kim, J. de Dieu Nyandwi, S. Khanuja, A. Kantharuban, L. Sutawika, S. Ramamoorthy, and G. Neubig, “Pangea: A fully open multilingual multimodal llm for 39 languages,” ArXiv, vol. abs/2410.16153, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:273507992

    arXiv

  8. [9]

    SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects,

    D. I. Adelani, H. Liu, X. Shen, N. Vassilyev, J. O. Alabi, Y . Mao, H. Gao, and E.-S. A. Lee, “SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects,” in EACL. St. Julian’s, Malta: Association for Computational Linguistics, Mar. 2024, pp. 226–245. [Online]. Available: https://aclanthology.org/2024...

    work page 2024

Show all 61 references
  1. [10]

    Chatgpt mt: Competitive for high-(but not low-) resource languages,

    N. Robinson, P. Ogayo, D. R. Mortensen, and G. Neubig, “Chatgpt mt: Competitive for high-(but not low-) resource languages,” in WMT, 2023, pp. 392–418

    2023

  2. [11]

    Megaverse: Benchmarking large language models across languages, modalities, models and tasks,

    S. Ahuja, D. Aggarwal, V . Gumma, I. Watts, A. Sathe, M. Ochieng, R. Hada, P. Jain, M. Ahmed, K. Bali et al., “Megaverse: Benchmarking large language models across languages, modalities, models and tasks,” in NAACL, 2024, pp. 2598–2637

    2024

  3. [12]

    On the evaluation of speech foundation models for spoken language understanding,

    S. Arora, A. Pasad, C.-M. Chien, J. Han, R. Sharma, J.-w. Jung, H. Dhamyal, W. Chen, S. Shon, H.-Y . Lee et al. , “On the evaluation of speech foundation models for spoken language understanding,” in Findings of the Association for Computational Linguistics ACL 2024 , 2024, pp...

    2024

  4. [13]

    Towards robust speech representation learning for thousands of languages,

    W. Chen, W. Zhang, Y . Peng, X. Li, J. Tian, J. Shi, X. Chang, S. Maiti, K. Livescu, and S. Watanabe, “Towards robust speech representation learning for thousands of languages,” in EMNLP, 2024, pp. 10 205– 10 224

    2024

  5. [14]

    Ml-superb 2.0: Benchmarking multilingual speech models across modeling constraints, languages, and datasets,

    J. Shi, S.-H. Wang, W. Chen, M. Bartelds, V . Bannihatti Kumar, J. Tian, X. Chang, D. Jurafsky, K. Livescu, H.-y. Lee et al. , “Ml-superb 2.0: Benchmarking multilingual speech models across modeling constraints, languages, and datasets,” in Proc. Interspeech 2024 , 2024, pp. 1...

    2024

  6. [15]

    Unsupervised cross-lingual representation learning at scale,

    A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in ACL. Online: Association for Computational Linguistics, Jul. 2020, pp. 8440–

    2020

  7. [16]

    Deberta: Decoding- enhanced bert with disentangled attention,

    P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding- enhanced bert with disentangled attention,” in International Conference on Learning Representations , 2021. [Online]. Available: https: //openreview.net/forum?id=XPZIaotutsD

    2021

  8. [17]

    mT5: A massively multilingual pre- trained text-to-text transformer,

    L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mT5: A massively multilingual pre- trained text-to-text transformer,” in NAACL. Online: Association for Computational Linguistics, Jun. 2021, pp. 483–498. [Online]. Available: https://...

    2021

  9. [18]

    Xls-r: Self-supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” in Interspeech 2022, 2022, pp. 2278–2282

    2022

  10. [19]

    Towards robust speech representation learning for thousands of languages,

    W. Chen, W. Zhang, Y . Peng, X. Li, J. Tian, J. Shi, X. Chang, S. Maiti, K. Livescu, and S. Watanabe, “Towards robust speech representation learning for thousands of languages,” in EMNLP. Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 10 205–10 ...

    2024

  11. [20]

    SpiRit-LM: Interleaved spoken and written language model,

    T. A. Nguyen, B. Muller, B. Yu, M. R. Costa-jussa, M. Elbayad, S. Popuri, C. Ropers, P.-A. Duquenne, R. Algayres, R. Mavlyutov, I. Gat, M. Williamson, G. Synnaeve, J. Pino, B. Sagot, and E. Dupoux, “SpiRit-LM: Interleaved spoken and written language model,” TACL, vol. 13, pp. ...

    2025

  12. [21]

    No language left behind: Scaling human-centered machine translation,

    N. Team, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, and et al., “No language left behind: Scaling human-centered machine translation,” 2022. [Online]. Available: https://arxiv.org/abs/2207.04672

    2022 arXiv

  13. [22]

    The belebele benchmark: a parallel reading comprehension dataset in 122 language variants,

    L. Bandarkar, D. Liang, B. Muller, M. Artetxe, S. N. Shukla, D. Husa, N. Goyal, A. Krishnan, L. Zettlemoyer, and M. Khabsa, “The belebele benchmark: a parallel reading comprehension dataset in 122 language variants,” in ACL. Association for Computational Linguistics, 2024, p. ...

    2024 doi

  14. [23]

    Fleurs: Few-shot learning evaluation of universal representations of speech,

    A. Conneau, M. Ma, S. Khanuja, Y . Zhang, V . Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learning evaluation of universal representations of speech,” 2022 IEEE Spoken Language Technology Workshop (SLT) , pp. 798–805, 2022. [Online]. Available: htt...

    2022

  15. [24]

    Fleurs- slu: A massively multilingual benchmark for spoken language understanding,

    F. D. Schmidt, I. Vuli’c, G. Glavavs, and D. I. Adelani, “Fleurs- slu: A massively multilingual benchmark for spoken language understanding,” ArXiv, vol. abs/2501.06117, 2025. [Online]. Available: https://api.semanticscholar.org/CorpusID:275458371

    2025 arXiv

  16. [25]

    Speechtaxi: On multilingual semantic speech classification,

    L. Keller and G. Glavaš, “Speechtaxi: On multilingual semantic speech classification,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2025, pp. 1–5

    2025

  17. [26]

    A broad-coverage challenge corpus for sentence understanding through inference,

    A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in NAACL, 2018, pp. 1112–1122

    2018

  18. [27]

    XNLI: Evaluating cross-lingual sentence representations,

    A. Conneau, R. Rinott, G. Lample, A. Williams, S. Bowman, H. Schwenk, and V . Stoyanov, “XNLI: Evaluating cross-lingual sentence representations,” in EMNLP. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 2475–2485. [Online]. Available: https:...

    2018

  19. [28]

    Irokobench: A new benchmark for african languages in the age of large language models,

    D. I. Adelani, J. Ojo, I. A. Azime, J. Y . Zhuang, J. O. Alabi, X. He, M. Ochieng, S. Hooker, and et al., “Irokobench: A new benchmark for african languages in the age of large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2406.03368

    2025 arXiv

  20. [29]

    IndicXNLI: Evaluating multilingual inference for Indian languages,

    D. Aggarwal, V . Gupta, and A. Kunchukuttan, “IndicXNLI: Evaluating multilingual inference for Indian languages,” in EMNLP. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 10 994–11 006. [Online]. Available: https://aclanthology.org/ ...

    2022

  21. [30]

    AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages,

    A. Ebrahimi, M. Mager, A. Oncevay, V . Chaudhary, L. Chiruzzo, A. Fan, J. Ortega, R. Ramos, A. Rios, I. V . Meza Ruiz, G. Giménez-Lugo, E. Mager, G. Neubig, A. Palmer, R. Coto-Solano, T. Vu, and K. Kann, “AmericasNLI: Evaluating zero-shot natural language understanding of pret...

    2022

  22. [31]

    Building a data infrastructure for a mid-resource language: The case of Catalan,

    A. Gonzalez-Agirre, M. Marimon, C. Rodriguez-Penagos, J. Aula- Blasco, I. Baucells, C. Armentano-Oller, J. Palomar-Giner, B. Kulebi, and M. Villegas, “Building a data infrastructure for a mid-resource language: The case of Catalan,” in LREC-COLING 2024 . Torino, Italia: ELRA a...

    2024

  23. [32]

    SEACrowd: A multilingual multimodal data hub and benchmark suite for Southeast Asian languages,

    H. Lovenia, R. Mahendra, S. M. Akbar, L. J. V . Miranda, J. Santoso, E. Aco, A. Fadhilah, J. Mansurov, J. M. Imperial, and et al., “SEACrowd: A multilingual multimodal data hub and benchmark suite for Southeast Asian languages,” in EMNLP. Miami, Florida, USA: Association for C...

    2024

  24. [33]

    IndoNLI: A natural language inference dataset for Indonesian,

    R. Mahendra, A. F. Aji, S. Louvan, F. Rahman, and C. Vania, “IndoNLI: A natural language inference dataset for Indonesian,” in EMNLP. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 10 511–10 527. [Online]. Available: https:...

    2021

  25. [34]

    JGLUE: Japanese general language understanding evaluation,

    K. Kurihara, D. Kawahara, and T. Shibata, “JGLUE: Japanese general language understanding evaluation,” in LREC. Marseille, France: European Language Resources Association, Jun. 2022, pp. 2957–2966. [Online]. Available: https://aclanthology.org/2022.lrec-1.317

    2022

  26. [35]

    InferBR: A natural language inference dataset in Portuguese,

    L. Bencke, F. V . Pereira, M. K. Santos, and V . Moreira, “InferBR: A natural language inference dataset in Portuguese,” in LREC-COLING

  27. [36]

    Evaluation of sentence representations in Polish,

    S. Dadas, M. Perełkiewicz, and R. Po ´swiata, “Evaluation of sentence representations in Polish,” in LREC. Marseille, France: European Language Resources Association, May 2020, pp. 1674–1680. [Online]. Available: https://aclanthology.org/2020.lrec-1.207

    2020

  28. [37]

    Jampatoisnli: A jamaican patois natural language inference dataset,

    R. Armstrong, J. Hewitt, and C. D. Manning, “Jampatoisnli: A jamaican patois natural language inference dataset,” CoRR, vol. abs/2212.03419,

    arXiv

  29. [38]

    9050–9060

    Torino, Italia: ELRA and ICCL, May 2024, pp. 9050–9060. [Online]. Available: https://aclanthology.org/2024.lrec-main.793/

    2024

  30. [39]

    A Novel Cartography- Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus,

    E. G. Poesina, C. Caragea, and R. T. Ionescu, “A Novel Cartography- Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus,” in ACL. Association for Computational Linguistics, 2024

    2024

  31. [40]

    The flores-101 evalu- ation benchmark for low-resource and multilingual machine translation,

    N. Goyal, C. Gao, V . Chaudhary, P.-J. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Ranzato, F. Guzmán, and A. Fan, “The flores-101 evalu- ation benchmark for low-resource and multilingual machine translation,” TACL, vol. 10, pp. 522–538, 2022

    2022

  32. [41]

    Scaling speech technology to 1,000+ languages,

    V . Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi et al. , “Scaling speech technology to 1,000+ languages,” Journal of Machine Learning Re- search, vol. 25, no. 97, pp. 1–52, 2024

    2024

  33. [42]

    Klue: Korean language understanding evaluation,

    S. Park, J. Moon, S. Kim, W. I. Cho, J. Han, J. Park, C. Song, J. Kim, and et al., “Klue: Korean language understanding evaluation,” 2021

    2021

  34. [43]

    GlotLID: Language identification for low-resource languages,

    A. H. Kargaran, A. Imani, F. Yvon, and H. Schuetze, “GlotLID: Language identification for low-resource languages,” in Findings of ACL: EMNLP 2023 . Singapore: Association for Computational Linguistics, Dec. 2023, pp. 6155–6218. [Online]. Available: https: //aclanthology.org/20...

    2023

  35. [44]

    XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models,

    D. Liang, H. Gonen, Y . Mao, R. Hou, N. Goyal, M. Ghazvininejad, L. Zettlemoyer, and M. Khabsa, “XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models,” in EMNLP. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 13 142–13 152. [...

    2023

  36. [45]

    Self-distillation for model stacking unlocks cross-lingual nlu in 200+ languages,

    F. Schmidt, P. Borchert, I. Vuli ´c, and G. Glavaš, “Self-distillation for model stacking unlocks cross-lingual nlu in 200+ languages,” in Findings of ACL: EMNLP 2024 , 2024, pp. 6724–6743

    2024

  37. [46]

    Joint speech and text machine translation for up to 100 languages,

    L. Barrault, Y .-A. Chung, M. C. Meglioli, D. Dale, N. Dong, P.-A. Duquenne, H. Elsahar, H. Gong, K. Heffernan, J. Hoffman, and S. C. Team, “Joint speech and text machine translation for up to 100 languages,” Nature, vol. 637, no. 8046, pp. 587–593, Jan 2025. [Online]. Availab...

    2025 doi

  38. [47]

    LLM2vec: Large language models are secretly powerful text encoders,

    P. BehnamGhader, V . Adlakha, M. Mosbach, D. Bahdanau, N. Chapados, and S. Reddy, “LLM2vec: Large language models are secretly powerful text encoders,” in First Conference on Language Modeling , 2024. [Online]. Available: https://openreview.net/forum?id=IW1PR7vEBf

    2024

  39. [48]

    DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,

    P. He, J. Gao, and W. Chen, “DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/ forum?id=sE7-XhLxHA

    2023

  40. [49]

    A broad-coverage challenge corpus for sentence understanding through inference,

    A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in NAACL. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 1112–1122. [Online]. Available: https: //aclanthology.org/N18-1101/

    2018

  41. [50]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  42. [51]

    Afrobench: How good are large language models on african languages?

    J. Ojo, O. Ogundepo, A. Oladipo, K. Ogueji, J. Lin, P. Stenetorp, and D. I. Adelani, “Afrobench: How good are large language models on african languages?” 2025. [Online]. Available: https: //arxiv.org/abs/2311.07978

    2025 arXiv

  43. [52]

    Few-shot learning with multilingual language models,

    X. V . Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, R. Pasunuru, S. Shleifer, P. S. Koura, V . Chaudhary, B. O’Horo, J. Wang, L. Zettlemoyer, Z. Kozareva, M. T. Diab, V . Stoyanov, and X. Li, “Few-shot learning with multilingua...

    2021 arXiv

  44. [53]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023

    2023

  45. [54]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière et al. , “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786 , 2025

    2025 arXiv

  46. [55]

    Madlad-400: A multilingual and document-level large audited dataset,

    S. Kudugunta, I. Caswell, B. Zhang, X. Garcia, D. Xin, A. Kusupati, R. Stella, A. Bapna, and O. Firat, “Madlad-400: A multilingual and document-level large audited dataset,” Advances in Neural Information Processing Systems, vol. 36, pp. 67 284–67 296, 2023

    2023

  47. [56]

    Are LLMs breaking MT metrics? results of the WMT24 metrics shared task,

    M. Freitag, N. Mathur, D. Deutsch, C.-K. Lo, E. Avramidis, R. Rei, B. Thompson, F. Blain, T. Kocmi, J. Wang, D. I. Adelani, M. Buchicchio, C. Zerva, and A. Lavie, “Are LLMs breaking MT metrics? results of the WMT24 metrics shared task,” in WMT. Miami, Florida, USA: Association...

    2024

  48. [57]

    The state and fate of linguistic diversity and inclusion in the NLP world,

    P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The state and fate of linguistic diversity and inclusion in the NLP world,” in ACL. Online: Association for Computational Linguistics, Jul. 2020, pp. 6282–

    2020

  49. [58]

    chrF++: words helping character n-grams,

    M. Popovi ´c, “chrF++: words helping character n-grams,” in WMT, O. Bojar, C. Buck, R. Chatterjee, C. Federmann, Y . Graham, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, and J. Kreutzer, Eds. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 612–618....

    2017

  50. [2022]

    Available: https://doi.org/10.48550/arXiv.2212.03419

    [Online]. Available: https://doi.org/10.48550/arXiv.2212.03419

    doi arXiv

  51. [2024]

    Available: https://api.semanticscholar.org/CorpusID: 271571434

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 271571434

  52. [6293]

    Available: https://aclanthology.org/2020.acl-main.560/

    [Online]. Available: https://aclanthology.org/2020.acl-main.560/

    2020

  53. [8451]

    Available: https://aclanthology.org/2020.acl-main.747/

    [Online]. Available: https://aclanthology.org/2020.acl-main.747/

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.