Pith. sign in

REVIEW 17 cited by

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.03368 v2 pith:JGO6QKOC submitted 2024-06-05 cs.CL cs.AI

David Ifeoluwa Adelani , Jessica Ojo , Israel Abebe Azime , Jian Yun Zhuang , Jesujoba O. Alabi , Xuanli He , Millicent Ochieng , Sara Hooker
show 19 more authors
Andiswa Bukula En-Shiun Annie Lee Chiamaka Chukwuneke Happy Buzaaba Blessing Sibanda Godson Kalipe Jonathan Mukiibi Salomon Kabongo Foutse Yuehgoh Mmasibidi Setaka Lolwethu Ndolela Nkiruka Odu Rooweither Mabuya Shamsuddeen Hassan Muhammad Salomey Osei Sokhar Samb Tadesse Kebede Guge Tombekai Vangoni Sherman Pontus Stenetorp
This is my paper · ORCID
classification cs.CLcs.AI
keywords languagesafricanmodelsenglishhigh-resourceirokobenchlanguagellms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (\eg African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench -- a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and multi-choice knowledge-based question answering~(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings~(where test sets are translated into English) across 10 open and six proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages~(such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63\% of the best-performing proprietary model GPT-4o performance. In addition, machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, such as Gemma 2 27B and LLaMa 3.1 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding

    cs.CL 2024-12 conditional novelty 7.0 of 10

    A new multi-label emotion benchmark for four Ethiopian languages shows that fine-tuned encoder-only models outperform zero-shot and few-shot large language models, with large gaps between resource-rich and resource-po...

  2. AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages

    cs.CL 2026-01 conditional novelty 6.0 of 10

    Training-data mix—not base-model language coverage—was the main driver of improved African-language performance, with Qwen-3 bases gaining the most after continued pre-training.

  3. FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A new benchmark shows state-of-the-art LLMs perform poorly on three Taiwanese indigenous languages across MT, ASR, and summarization.

  4. Improving Multilingual Math Reasoning for African Languages

    cs.CL 2025-05 conditional novelty 6.0 of 10

    SFT on translated OpenMathInstruct data outperforms directly generated synthetic data for math in African languages, and combining both yields the best AfriMGSM scores.

  5. Voice of a Continent: Mapping Africa's Speech Technology Frontier

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A new benchmark and fine-tuned Simba models improve speech recognition, synthesis, and language identification across 61 African languages, but the claimed state of the art lacks comparisons to prior task-specific systems.

  6. Bridging the Data Provenance Gap Across Text, Speech and Video

    cs.AI 2024-12 conditional novelty 6.0 of 10

    A manual audit of nearly 4,000 text, speech, and video datasets finds AI training data increasingly comes from web and social media sources, carries hidden non-commercial restrictions, and remains Western-centric with...

  7. Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Human-translated benchmarks in eight African languages show GPT-4o accuracy is 12 to 20 percentage points below English, and fine-tuning on translated data recovers part of the gap.

  8. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    This paper quantifies Western-centric bias in MMLU, releases Global-MMLU across 42 languages with human-verified translations, and shows model rankings shift on culturally sensitive versus agnostic subsets.

  9. Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A six-language translated benchmark shows large language models are significantly less accurate on science and truthfulness questions in African languages than in English, and closed models beat open models.

  10. INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

    cs.CL 2024-11 conditional novelty 6.0 of 10

    INCLUDE is a multilingual benchmark of 197,243 exam questions from local sources that evaluates how well LLMs handle regional and cultural knowledge.

  11. Against 'softmaxing' culture

    cs.HC 2025-06 unverdicted novelty 5.0 of 10

    A position paper arguing that AI evaluations should shift from defining culture to understanding when culture becomes relationally valid.

  12. mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks

    cs.CL 2025-06 conditional novelty 5.0 of 10

    mSTEB is a new 200+ language speech and text benchmark showing that LLMs perform substantially worse on low-resource African and Americas/Oceania languages, especially in speech tasks.

  13. Lillama: Large Language Models Compression via Low-Rank Feature Distillation

    cs.LG 2024-12 conditional novelty 5.0 of 10

    Lillama compresses LLMs by SVD-initialized low-rank layers trained with a local Teacher plus Student activation distillation loss, achieving 20-40% parameter reduction with only 13 million calibration tokens.

  14. The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A continuously updated multilingual benchmark dashboard aggregates existing tasks to rank LLMs across up to 200 languages.

  15. Task-Oriented Dialog Systems for the Senegalese Wolof Language

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A Rasa-based Wolof task-oriented dialog system, trained on French MASSIVE data projected through an in-house French-Wolof machine translation system, achieves near-French intent classification but weaker slot filling.

  16. The Roles of English in Evaluating Multilingual Language Models

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Multilingual LM evaluations that mix English prompts with target-language text measure task performance boosted by an English interface, not true multilingual language understanding.

  17. Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages

    cs.CL 2025-06

Pith tools