{"id":"b88c83af-5f4d-4047-babc-72dc709d122f","arxiv_id":"2506.08400","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"mSTEB is a new 200+ language speech and text benchmark showing that LLMs perform substantially worse on low-resource African and Americas/Oceania languages, especially in speech tasks.","lead":"This paper introduces mSTEB, a benchmark that tests AI language models on speech and text tasks across more than 200 languages, using existing datasets plus a new natural-language-inference collection. The authors find large performance gaps: African and Americas/Oceania languages score far lower, and open speech models lag proprietary ones.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Global NLI's mixed dataset sources confound the Americas/Oceania weakness: the NLI regional gap may reflect AmericasNLI's intrinsic difficulty rather than model coverage, undermining a key part of the headline claim.","rationale":"The reader's weakest_assumption identifies exactly this issue: Global NLI's comparability across languages is load-bearing for the NLI results, and the Americas/Oceania region is over-represented by the potentially harder AmericasNLI dataset. The paper itself acknowledges the likely confound in Section V.A.2, and the NLI scores in Table IV show an unusually large gap for that region. This makes the concern concrete and testable, not a speculative appeal to dataset difficulty. The recommended concrete test is feasible with the released data: recompute regional averages on the balanced 33-language subset and on a difficulty-matched AmericasNLI subset. Because the rest of the benchmark (Flores, SIB-200, Belebele, Fleurs) is based on professionally translated or human-annotated data with fixed test sets, the broader high/low-resource gap is plausible and independently supported by prior work. The NLI issue narrows the 'especially Americas/Oceania' part of the claim but does not overturn the benchmark's utility or the overall gap. Therefore the reader's CONDITIONAL verdict remains appropriate, and no change to the verdict is needed.","tokens_in":17612,"tokens_out":4263,"duration_ms":52841,"concrete_test":"Restrict the Global NLI evaluation to the 33 genre-balanced, source-matched languages (XNLI, AfriXNLI, IndicXNLI, XNLI-ca, myXNLI) and recompute regional NLI accuracies for the three models in Table IV. Then add the AmericasNLI languages with an equal number of entailment/neutral/contradiction examples per language, matching the label distribution of the balanced subset. If Americas/Oceania is no longer the lowest region, or if the gap to Europe (W, N, S) shrinks to under 10 points, then the reported regional weakness is an artifact of dataset composition rather than model coverage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that Global NLI scores are comparable across the 59 languages, as described in Section III. The test set is balanced and genre-matched for only 33 languages; the remaining 26 come from source datasets with different genres, content, and difficulty. In particular, 11 of the 59 languages are from AmericasNLI, which contains truly low-resource indigenous languages such as Otomi and Wixarika, as noted in Section V.B.c. The paper's own discussion in Section V.A.2 concedes that Flores/SIB-200 cover only about five Americas/Oceania languages, whereas AmericasNLI includes harder ones. The headline claim that Americas/Oceania is the weakest region in text is driven largely by NLI: Americas/Oceania NLI accuracy is 46.1-49.1 versus 82.7-84.8 for Western Europe in Table IV. If AmericasNLI is intrinsically harder than the XNLI-style data used for the balanced languages, the regional gap is partly a dataset artifact rather than a pure model-capability gap. Because the central claim emphasizes 'especially for languages spoken in Africa and Americas/Oceania,' this confound is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"mSTEB introduces a massively multilingual benchmark that aggregates existing human-curated datasets (Flores-200, SIB-200, Belebele, FLEURS, FLEURS-SLU, and a newly assembled Global NLI) to evaluate LLMs on five speech tasks (LID, topic classification, RC-QA, ASR, S2TT) and five text tasks (LID, topic classification, RC-QA, MT, NLI) across more than 100 speech languages and 200 text languages. The authors evaluate two open-weight models (Qwen 2 Audio, Gemma 3 27B) and three proprietary models (GPT-4o, GPT-4o Audio, Gemini 2.0 Flash), along with task-specific baselines. The headline findings are a large performance gap between high- and low-resource languages, especially in Africa and Americas/Oceania, and that Qwen 2 Audio underperforms proprietary models on every speech task. The paper releases code, data, and a leaderboard.","tokens_in":17849,"tokens_out":6597,"duration_ms":69776,"significance":"If validated, mSTEB would be a valuable community resource: it is the first benchmark, to my knowledge, that jointly covers a broad set of speech and text tasks at this language scale, and it uses standard metrics (accuracy, CER, ChrF++) with several strong task-specific baselines. The paper is transparent about its composition criteria (human-curated data, regional coverage) and releases code, data, and a public leaderboard, which supports reproducibility. The empirical analyses by region, language family, and error type are informative and likely to be cited. The main contribution—a unified, massively multilingual, multimodal benchmark—fills a genuine gap, and the headline finding of a large low-resource gap is important if supported by the evidence.","major_comments":[{"comment":"The NLI regional comparison is confounded by dataset composition. Global NLI is assembled from 13 source datasets, and only 33 of 59 languages share the same balanced, genre-matched premise/hypothesis templates (XNLI, AfriXNLI, IndicXNLI, XNLI-ca, myXNLI). The remaining 26 languages, including 11 from AmericasNLI, come from datasets with different genres, content, and intrinsic difficulty. Table IV shows Americas/Oceania NLI accuracy of 46.1–49.1 versus 82.7–84.8 for Western Europe, and the paper uses this to support the claim that Americas/Oceania is the weakest region in text. However, the paper's own discussion in Section V.A.2 concedes that this gap may reflect the presence of truly low-resource AmericasNLI languages (e.g., Otomi, Wixarika) rather than model capability alone. Because the regional NLI scores are not directly comparable across regions, the headline claim about Americas/Oceania being the weakest text region rests on a confounded comparison. To make the claim load-bearing, the paper should either restrict the regional NLI analysis to the 33 languages that share the same parallel data, provide a per-dataset breakdown (e.g., AmericasNLI vs. the balanced subset) so that dataset difficulty can be disentangled from language difficulty, or explicitly moderate the claim and reframe the NLI result as exploratory and dataset-sensitive.","section":"Section III (Global NLI) and Section V.A.2 / Table IV"},{"comment":"Every model-task pair is run exactly once with a single prompt, and no variance estimation or significance testing is reported. The paper states that it selected the best prompt from AfroBench and then ran each task only once. This means the reported differences—such as the ranking of GPT-4o Audio vs. Gemini 2.0 Flash on ASR, or the regional gaps in Table IV—are point estimates that could be sensitive to prompt wording and to stochastic decoding (especially for the API models). While the largest gaps (e.g., Qwen 2 Audio vs. Gemini 2.0 Flash on speech) are so wide that they are almost certainly robust, many smaller differences (e.g., 1–3 point gaps in Table IV) may not be meaningful. The paper should at least report confidence intervals for a subset of configurations (e.g., a few languages per region) or rerun with multiple prompts/seeds for key comparisons, or alternatively frame all comparative claims as provisional single-run observations. This is a standard expectation for a benchmark paper that makes strong comparative statements.","section":"Section IV (Experimental Setup) and Section V (Results)"}],"minor_comments":[{"comment":"The statement that 'For all languages except Romanian, they have an equal number of entailment, neutral, and contradiction labels' leaves the Romanian exception unexplained; please clarify the label distribution for Romanian and consider whether its 600-sample test set is comparable to the others.","section":"Section III (Global NLI)"},{"comment":"The number of languages listed for LID (102) differs from ASR and S2TT (101). Please explain this discrepancy (e.g., a missing audio split for one language).","section":"Table I"},{"comment":"The sentence 'we observe the Americas region also has high performance across tasks including MT, this is not true for NLI where it achieved the lowest (46.8)' should specify that 46.8 is the Gemini score, not the GPT-4o score (which is 49.1); also cross-referencing Table IV would help.","section":"Section V.A.2"},{"comment":"The paper says 'indigeneous' (Section V.A.2); this should be 'indigenous'.","section":"Section V.A.2"},{"comment":"The claim that Gemini 2.0 Flash outperforms SeamlessM4T v2 on ASR and S2TT is made textually but the baseline numbers are only in Figure 1 (not in the tables). Please include the baseline values in Table III or an appendix table so the comparison is directly verifiable.","section":"Section V.A.1"},{"comment":"The novelty claim of being the 'first' unified speech/text benchmark at this scale would be strengthened by explicitly discussing other recent multilingual/multimodal benchmark efforts (e.g., XTREME-UP, SpeechGLUE, or other audio-LLM evaluation suites) and explaining how mSTEB differs.","section":"Section II (Related Work)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is written by several authors who also created some of the component datasets (SIB-200, AfriXNLI, FLEURS-SLU). I do not see a circularity problem in the evaluation itself, but the reader might perceive a conflict of interest; I suggest the authors add a brief statement of prior authorship of component datasets in the paper or the GitHub repo. The NLI confound is the primary technical issue; I recommend the authors either re-analyze the data to separate dataset effects from language effects or soften the Americas/Oceania claim. The single-run evaluation is a secondary concern that is likely acceptable if the authors add a robustness check for at least a subset of experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"mSTEB is a useful benchmark artifact. The contribution is the assembly: ten tasks across two modalities, 100+ languages in speech and 200+ in text, five models including open-weight and proprietary, and a public leaderboard. If you work on multilingual evaluation, this is a convenient reference instrument.\n\nWhat's genuinely new: the Global NLI corpus (an aggregation of 13 existing NLI datasets), the unified speech/text comparison on matched tasks, and the first zero-shot evaluation of GPT-4o Audio and Gemini 2.0 Flash at this breadth. The paper also does some things right: standard metrics, task-specific baselines (MMS-LID, SeamlessM4T, NLLB, GlotLID), and per-region/per-family analysis. The headline finding — large high/low-resource gaps, worse for Africa and Americas/Oceania — is plausible and consistent with prior work.\n\nThe soft spots. First, the Global NLI comparability problem is real and load-bearing for the NLI regional claim. Only 33 of 59 languages are balanced and genre-matched; the other 26, including 11 from AmericasNLI, come from datasets with different genres and difficulty. The paper itself concedes that Flores and SIB-200 cover few Americas/Oceania languages while AmericasNLI includes Otomi and Wixarika. So the −35 point NLI gap between Americas/Oceania and Western Europe likely reflects dataset difficulty as much as model capability. That does not sink the overall finding — the speech results and other text tasks also show the regional gap — but it weakens the NLI-specific conclusion.\n\nSecond, single-run evaluation without variance or significance tests is a genuine limitation for a benchmark meant to compare models. Prompt sensitivity and decoding variance matter, especially for the open-weight vs proprietary comparisons. Choosing one prompt per task from AfroBench is defensible, but the paper should state that as a caveat.\n\nThird, novelty is in the assembly, not the components. The abstract's claim that 'there is no standardized evaluation benchmark' is a bit strong given existing multilingual benchmarks. Still, the artifact is new and useful.\n\nCitation pattern is fine; self-citations point to the datasets actually used. No padding.\n\nWho is this for: anyone comparing LLMs across languages and modalities, and anyone designing multilingual benchmarks. It deserves a serious referee. I would ask for a difficulty-matched NLI re-analysis, per-language scores, and uncertainty estimates, but I would not desk-reject it.","headline":"Useful benchmark artifact with a real confound in the NLI regional result; worth engaging, but the headline gap needs a difficulty-matched check.","tokens_in":18399,"tokens_out":2427,"would_cite":true,"duration_ms":27084,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"mSTEB is a new benchmark that tests LLMs on speech in over 100 languages and on text in over 200, and it finds the worst performance among African and Americas/Oceania languages.","keywords":["multilingual evaluation","speech recognition","low-resource languages","language identification","machine translation","natural language inference","audio language models","benchmark"],"falsifier":"Re-run the NLI evaluation using only the 33 languages with genre-balanced 600-sample test sets; if the Americas/Oceania region then no longer trails Europe (W, N, S) by roughly 35 points, the claimed NLI gap is a dataset-composition artifact rather than a model capability gap.","tokens_in":17442,"feed_emoji":"🌐","tokens_out":11412,"duration_ms":112383,"temperature":0.7,"pith_summary":"The paper introduces mSTEB, a unified evaluation benchmark that tests large language models (LLMs) on five speech tasks and five text tasks across more than 100 and 200 languages respectively. The tasks are language identification, topic classification, reading-comprehension question answering, translation, plus automatic speech recognition for speech and natural language inference for text. Evaluating five current LLMs, the authors report a wide performance gap between high-resource and low-resource languages, with the lowest scores concentrated in Africa and Americas/Oceania. The paper also finds that the open-weight speech model Qwen 2 Audio lags the proprietary models on every speech task, while the open-weight text model Gemma 3 27B is nearly competitive on text. If the benchmark is a fair instrument, these results make the uneven multilingual coverage of today's LLMs concrete and measurable, and point to speech understanding as a particularly weak spot for low-resource languages.","feed_headline":"Benchmark finds LLMs weakest on African, Americas/Oceania languages","feed_subtitle":"mSTEB tests speech on 100+ and text on 200+ languages; open speech models lag far behind proprietary ones.","key_machinery":"The central object is the mSTEB benchmark itself, a task-aligned aggregation of existing datasets. Fleurs supplies the speech audio for language identification, ASR, and speech-to-text translation; SIB-Fleurs and Belebele-Fleurs provide audio topic classification and audio reading comprehension; Flores-200 supplies text translation and text LID; SIB-200 supplies text topic classification; Belebele supplies text reading comprehension; and the newly assembled Global NLI provides text NLI across 59 languages. By aligning tasks across modalities, the benchmark lets a reader compare a model's speech and text ability on the same kind of question, and to group results by region and language family. The same prompting and scoring protocols are applied to every model, with ChrF++ for translation and character error rate for ASR, so the numbers are directly comparable across systems.","core_discovery":"On its own terms, mSTEB establishes a standardized cross-modal yardstick: the same evaluation recipe applied to speech and text, using Fleurs for the audio side and Flores-200, SIB-200, Belebele, and gathered NLI resources for the text side. Across this breadth, the authors find that proprietary models (Gemini 2.0 Flash, GPT-4o, GPT-4o Audio) consistently beat the open-weight models (Qwen 2 Audio, Gemma 3 27B), and that the largest gaps appear in Africa and Americas/Oceania in both modalities. The paper's central claim is that these regional and resource-level gaps reflect genuine under-representation of low-resource languages in LLM training data, and that closing them requires deliberate investment. The benchmark is offered as a tool for tracking such progress over time.","pith_inferences":["If Global NLI's regional gaps are partly an artifact of its unbalanced design (only 33 of 59 languages are genre-matched), then mSTEB's Americas/Oceania NLI ranking may overstate model weakness; a balanced NLI subset across all 59 languages would be a direct test.","Because the paper excludes machine-translated datasets by design, mSTEB leaves out many existing low-resource resources; a variant that admits carefully audited pseudo-parallel data could extend coverage beyond 200 text languages.","The paper selects a single best prompt per task from an earlier multilingual benchmark study, so reported scores reflect prompt choice as well as model ability; re-running with multiple prompts would separate prompt incompatibility from true capability gaps.","The speech and text versions of 'the same' tasks use different source sentences (Fleurs audio vs Flores text), so the modality gap is approximate; a strictly parallel speech-text corpus would make the cross-modal comparison exact."],"forward_implications":["Future LLM releases can be checked against a single 200+ language, two-modality benchmark, replacing English-first or few-language evaluation suites.","The large open-versus-proprietary gap on speech suggests that open-weight audio LLMs are not yet usable for most low-resource languages, a finding that should shape where open-model training effort goes.","Gemini 2.0 Flash beating SeamlessM4T v2 on ASR and S2TT indicates that general-purpose speech LLMs are becoming competitive with dedicated speech translation systems, even on low-resource languages.","The regional analysis identifies Africa and Americas/Oceania as the clearest priorities for data collection and model investment in both modalities."],"supporting_citations":[{"why":"Supplies the 204-language Flores-200 devtest used for text LID and machine translation, and the English source for SIB-200.","marker":"[21]"},{"why":"Supplies the 102-language Fleurs audio set used for speech LID, ASR, and speech-to-text translation.","marker":"[23]"},{"why":"Supplies the 122-language Belebele reading comprehension questions used for RC-QA in both modalities.","marker":"[22]"},{"why":"Provides the 200+ language SIB-200 topic classification set for text, aligned to Flores sentences.","marker":"[9]"},{"why":"Provides Fleurs-SLU, the aligned audio counterparts for topic classification and reading comprehension, plus its cascaded baseline.","marker":"[24]"},{"why":"Contributes XNLI premise/hypothesis pairs that anchor the genre-balanced core of Global NLI.","marker":"[27]"},{"why":"Supplies AmericasNLI, the main source of truly low-resource Americas languages in Global NLI.","marker":"[30]"},{"why":"Provides MMS-LID, the task-specific speech language identification baseline that beats all LLMs.","marker":"[41]"},{"why":"Provides SeamlessM4T v2, the ASR and speech-to-text translation baseline that Gemini 2.0 Flash outperforms.","marker":"[42]"},{"why":"Provides GlotLID, the text language identification baseline trained on over 2,000 languages.","marker":"[43]"}],"fun_headline_variants":["mSTEB benchmark: LLMs lag on African and Americas/Oceania languages","Speech and text benchmark shows LLM gaps in Africa and Oceania","New mSTEB: open LLMs trail proprietary on low-resource languages","mSTEB: 200+ language eval reveals LLM weak spots in Africa and Americas","Benchmark: LLMs underperform on low-resource languages worldwide"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Global NLI scores treat all 59 languages as equally comparable, but only 33 of them share the same balanced, genre-matched sample design, so the apparent NLI weakness of Americas/Oceania may be an artifact of which datasets were included.","fun_headline_variants_meta":{"raw":{"variants":["mSTEB benchmark: LLMs lag on African and Americas/Oceania languages","Speech and text benchmark shows LLM gaps in Africa and Oceania","New mSTEB: open LLMs trail proprietary on low-resource languages","mSTEB: 200+ language eval reveals LLM weak spots in Africa and Americas","Benchmark: LLMs underperform on low-resource languages worldwide"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1469,"prompt_tokens":899,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":515,"tokens_out":570,"duration_ms":7112,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:11:30.057101+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the NLI evaluation using only the 33 languages with genre-balanced 600-sample test sets; if the Americas/Oceania region then no longer trails Europe (W, N, S) by roughly 35 points, the claimed NLI gap is a dataset-composition artifact rather than a model capability gap.","supporting_citations":[{"cited_title":"Fleurs: Few-shot learning evaluation of universal representations of speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the 102-language Fleurs audio set used for speech LID, ASR, and speech-to-text translation."},{"cited_title":"SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects,","cited_arxiv_id":null,"evidence_quote":"Provides the 200+ language SIB-200 topic classification set for text, aligned to Flores sentences."},{"cited_title":"XNLI: Evaluating cross-lingual sentence representations,","cited_arxiv_id":null,"evidence_quote":"Contributes XNLI premise/hypothesis pairs that anchor the genre-balanced core of Global NLI."},{"cited_title":"AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages,","cited_arxiv_id":null,"evidence_quote":"Supplies AmericasNLI, the main source of truly low-resource Americas languages in Global NLI."},{"cited_title":"Klue: Korean language understanding evaluation,","cited_arxiv_id":null,"evidence_quote":"Provides SeamlessM4T v2, the ASR and speech-to-text translation baseline that Gemini 2.0 Flash outperforms."},{"cited_title":"GlotLID: Language identification for low-resource languages,","cited_arxiv_id":null,"evidence_quote":"Provides GlotLID, the text language identification baseline trained on over 2,000 languages."}],"review_version":1}