Pith. sign in

REVIEW 4 major objections 4 minor 17 references

The paper claims that professionally curated local news media provide cleaner, more reliable training data for Yoruba, Kinyarwanda, and Amharic than conversational platforms, and that language detection models fail when text is code-switche

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Language detection models are near-perfect on clean news text in Yoruba, Kinyarwanda, and Amharic but perform poorly on code-switched Reddit posts, suggesting curated news data is more reliable for training.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Small, clearly-written study with a plausible practical recommendation, but the central quantitative claim about code-switching rests on unvalidated detector outputs and needs human labels before it can support the conclusion. the 4 major comments →

arxiv 2512.01557 v3 pith:DMUJZMDY submitted 2025-12-01 cs.CL

Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spaces

Edward Ajayi , Eudoxie Umwari , Mawuli Deku , Prosper Singadi , Jules Udahemuka , Bekalu Tadele , Chukuemeka Edeh This is my paper
classification cs.CL
keywords African languageslanguage identificationcode-switchingYorubaKinyarwandaAmhariclow-resource NLPdigital language representation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that for Yoruba, Kinyarwanda, and Amharic, the best available source of clean monolingual digital text is professionally edited local news, not social-media conversation. On a year of forum posts, the African languages appeared infrequently and mixed with English and other languages, while news sites yielded text that was almost entirely monolingual and prompted local-language engagement in comment sections. Language detection tools—both a dedicated African-language identifier and a large general-purpose model—performed near-perfectly on the news data but split widely and often flagged English on the forum data. The authors conclude that future African-language AI should be trained on curated news data for clean text and needs new models that treat code-switching as a valid, learnable phenomenon.

Core claim

The central discovery is a sharp asymmetry: the same detection tools that identify almost 100% of news headlines in Yoruba, Kinyarwanda, and Amharic find those languages only intermittently in the corresponding subreddits. A year of scraping produced 184, 165, and 833 posts respectively, with AfroLID assigning large shares to other African languages and Llama 3.3 70B labeling 69–97% of the posts as English. The authors' interpretation is that dedicated language communities on these conversational platforms are not writing monolingually; they are code-switching, and this is precisely the register that current detectors mishandle. In contrast, news media and their social comments show that whe

What carries the argument

The method pairs two language-detection tools—AfroLID, a neural identifier covering 517 African languages, and Llama 3.3 70B, a general-purpose multilingual LLM—with two contrastive corpora: one year of posts from Yoruba, Amharic, and Kinyarwanda subreddits, and headlines (plus Facebook comments) from local news outlets. The load-bearing contrast is the gap between the tools' near-perfect scores on the clean news text and their unstable, divergent scores on the conversational text. That divergence is used as evidence that conversational platforms are unreliable sources for monolingual training data and that code-switching is the real obstacle.

Load-bearing premise

The paper's picture of what is happening on the forums—how much English, how much code-switching—rests entirely on the output of AfroLID and Llama 3.3 70B; without human annotation of those posts, the measured 'struggle' could be the models' error rather than the data's code-switching.

What would settle it

Hand-annotate a sample of 100 posts from each of the three subreddits, sentence by sentence, with native speakers. If most sentences are judged to be Yoruba, Kinyarwanda, or Amharic with only embedded English words or phrases, the claim of English-heavy code-switching is falsified; if speakers agree with the models' English-majority judgment, the claim holds.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Clean, monolingual African-language corpora for training and evaluation can be assembled from local news outlets much more cheaply and reliably than by scraping conversational platforms.
  • Benchmarks built on news text alone will overstate real-world performance: detectors that score 100% on clean text can fall apart on everyday code-switched usage.
  • Social-media comment sections on local-language news posts may serve as a source of authentic conversational data, since local-language posts elicited local-language responses.
  • Language technology for these communities should treat code-switching as a first-class modelling target rather than as noise to be filtered out.
  • The negative feedback loop the paper describes—scarce data, poor models, users avoiding their language—can be interrupted at the data-collection stage by using curated news media.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pattern holds across other African languages, many existing web-crawled corpora are skewed toward formal registers, which would make models seem better than they are on spoken conversation.
  • A direct test of the paper's conclusion would be to fine-tune or prompt-tune a detector on the news data and evaluate on a human-annotated code-switched sample; the paper did not include such an evaluation, so the improvement remains a prediction.
  • The paper's own data suggests a practical shortcut for future collections: news organizations' Facebook comments appear to contain more spontaneous local-language conversation than dedicated subreddits, so these comment threads may be the closest available public proxy for conversational data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper examines the digital presence of Yoruba, Kinyarwanda, and Amharic by comparing Reddit posts with local news articles. It scrapes three subreddits (r/yoruba, r/amharic, r/Rwanda) and three news sources (BBC Yoruba, RBA, Fana), applies the AfroLID and Llama 3.3 70B language detectors, and manually annotates news article topics. The authors report that Reddit data are sparse and heavily code-switched/English-dominated, while news data are clean and monolingual; the detectors are near-perfect on news but 'struggle' on Reddit. The paper concludes that professionally curated news content is a more reliable source for training context-rich AI models for African languages than conversational platforms.

Significance. If supported, the paper would provide a useful empirical data point for low-resource African NLP: it quantifies the gap between conversational and professionally curated sources and documents the difficulty of language detection on mixed text. The manual topic annotation by native speakers is a genuine strength, as is the comparison of a specialized African-language detector with a general-purpose LLM. The claim itself is plausible and policy-relevant for dataset construction. However, the central empirical contrast is currently under-supported: the Reddit 'code-switching' claim is inferred from unvalidated detector outputs with incompatible label sets, and the news 'accuracy' is close to tautological because sources were selected precisely because they are known to publish in a target language. The paper contains no code, data, or uncertainty quantification, which limits its current value.

major comments (4)
  1. [§4.2, Tables 1–3] The central claim that detectors 'struggled' with Reddit code-switching is not directly supported by the measurements. No human gold labels were collected for Reddit posts; Tables 1–3 report the models' predicted label distributions, not accuracy. Because AfroLID has no English class while Llama outputs English for 69–97% of Reddit posts, the observed divergence is partly an artifact of incompatible label vocabularies. 'Struggled' is never defined—no precision, recall, or F1 is reported. This is load-bearing for the §5.4 conclusion about conversational platforms. I recommend human annotation of a stratified Reddit sample (native speakers were already used for topic classification), followed by per-language precision/recall/F1 and an explicit definition of code-switching.
  2. [§4.1.1, §5.1] The Kinyarwanda data come from r/Rwanda, which is a country subreddit, not a Kinyarwanda-language community. English dominance there is unsurprising and conflates national affiliation with language use. The §5.4 statement that platforms 'clearly dedicated to specific African linguistic communities' are unreliable overgeneralizes to the Rwandan case. The paper should either find a Kinyarwanda-focused subreddit or explicitly frame the result as 'r/Rwanda' rather than as evidence about Kinyarwanda usage.
  3. [§5.2.1, Tables 2 and 4] '100% accuracy' on news is close to tautological because each news source was selected for publishing in a known language (BBC Yoruba, RBA, Fana). A detector agreeing with source metadata does not validate the detector's general ability or establish that news is a 'robust' source beyond these three outlets. Report at least a human-verified sample of news posts, or evaluate on additional/held-out sources; otherwise the claim should be softened to 'detector predictions match source-language metadata on this sample.'
  4. [§5.1–5.3] The quantitative comparison lacks uncertainty quantification. Sample sizes are modest (e.g., 62–105 news articles; 165–833 Reddit posts), yet percentages are reported to two decimals without counts or confidence intervals. For example, Kinyarwanda news '97.14%' is 2 errors out of 105, which is not meaningfully different from 100% at this sample size. Report raw counts, Wilson intervals or bootstrap confidence intervals, and a significance test for the Reddit-vs-news contrast.
minor comments (4)
  1. [Abstract, §4.2] The abstract mentions a macro-classifier GlotLID, but GlotLID is never used or results reported; only AfroLID and Llama 3.3 70B appear in the methodology. Either include GlotLID results or remove the mention.
  2. [§4.1] No data availability statement, code, or scraping dates are provided; the Llama prompt and inference configuration are also omitted. This limits reproducibility for an empirical paper.
  3. [§5.1] The 'qualitative analysis' of Reddit posts is mentioned but no coding protocol, inter-annotator agreement, or example excerpts are given. Specify the procedure or remove the qualitative claim.
  4. [References] Reference [15] cites a GeeksforGeeks page for PRAW instead of the PRAW documentation/API; please cite the actual software. Also, the PRAW references [12]–[14] are listed as subreddit pages, which is acceptable but inconsistent with the style of [9]–[11]; please standardize.

Circularity Check

1 steps flagged

Reddit code-switching 'struggle' finding is inferred from the same detectors being evaluated; news-side conclusion partly restates the source-selection criterion.

specific steps
  1. self definitional [§5.2.2 (Table 3) and §5.4]
    "The LLM detected a significantly higher percentage of English content on Reddit compared to AfroLID, identifying 73.02% in r/Amharic, 97.45% in r/Rwanda, and 69.23% in r/Yoruba. While the LLM correctly identifies the primary non-English language in each subreddit (e.g., Yoruba at 28.21% in its respective subreddit), its high English classification rate suggests that it is categorizing much of the code-switched text and other languages as English, likely due to its broader, general-purpose training."

    The paper's evidence that the LLM 'struggles' on Reddit is the model's own predicted label distribution, and the code-switched 'ground truth' that would make that distribution an error is inferred from the same high-English output ('suggests that it is categorizing much of the code-switched text... as English'). No human-annotated gold labels are provided for Reddit text, and the two detectors have disjoint label vocabularies (AfroLID has no English class), so the observed divergence is partly a byproduct of tool design. The 'struggle' claim therefore is not independently measured; it reduces to an interpretive read of the very predictions being evaluated.

full rationale

The paper contains no fitted-parameter prediction and no self-citation chain, so most of the analysis is not circular. The news-side '100% accuracy' result is a weak check: because the sources were selected as channels that 'post primarily in local languages,' the source labels are known by construction, making the high agreement a sanity check rather than an independent validation of the claim that news media is a robust source. The more substantive circularity is on the Reddit side: the conclusion that code-switched Reddit data causes models to struggle is derived from the outputs of those very models, with no human-annotated reference labels. In particular, the LLM's high English rate is both the observation and the evidence that the model is misclassifying code-switched text. This makes the central comparative claim about detector failure on conversational data partially self-referential. I do not score higher because the paper's qualitative observations and raw post counts provide some independent grounding for the data-scarcity and code-switching narrative, and because the news-side comparison does use externally supplied source labels. The circularity is partial and localized to the Reddit-side performance inference, not a global derivation collapse.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters were fitted; the analysis relies on external LID models and manual topic annotation. The main implicit assumptions are that Reddit subreddits represent conversational platforms, news source labels are accurate, and model outputs can substitute for human annotation in characterizing Reddit code-switching.

axioms (4)
  • domain assumption Reddit subreddit posts (r/Yoruba, r/Amharic, r/Rwanda) are representative of conversational digital usage of these African languages.
    The paper uses the Reddit scrape as the conversational platform sample (§4.1.1, Figure 2) even though r/Rwanda is a country subreddit and the subreddits are small and translation-oriented (§5.1).
  • domain assumption News articles from BBC Yoruba, RBA, and Fana are gold-standard monolingual texts in the target languages.
    Used as the clean/curated baseline and as the labels for accuracy in Tables 2 and 4 (§4.1.2). No human verification of individual articles is described beyond source selection.
  • domain assumption Llama 3.3 70B's language output is a valid language detector for English and the target languages.
    The model is used as the detector that reports 69–97% English on Reddit (§4.2, Table 3), despite not officially supporting Yoruba, Kinyarwanda, or Amharic.
  • domain assumption Code-switching prevalence can be inferred from model output without human annotation.
    The paper's central 'struggles with code-switched Reddit posts' claim relies on the model's high English share rather than a labeled code-switching annotation (§5.2, Tables 1–3).

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spaces." pith.science (2026). https://pith.science/paper/DMUJZMDY

@misc{pith2026251201557,
  author       = {Pith},
  title        = {Pith review of: Language Diversity: Evaluating Language Usage and AI Performance on African Languages in Digital Spaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DMUJZMDY}},
  note         = {Machine review of arXiv:2512.01557}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This study examines the digital representation of African languages and the challenges this presents for current language detection tools. We evaluate their performance on Yoruba, Kinyarwanda, and Amharic. While these languages are spoken by millions, their online usage on conversational platforms is often sparse, heavily influenced by English, and not representative of the authentic, monolingual conversations prevalent among native speakers. This lack of readily available authentic data online creates a challenge of scarcity of conversational data for training language models. To investigate this, data was collected from subreddits and local news sources for each language. The analysis showed a stark contrast between the two sources. Reddit data was minimal and characterized by heavy code-switching. Conversely, local news media offered a robust source of clean, monolingual language data, which also prompted more user engagement in the local language on the news publishers' social media pages. Language detection models, including a macro-classifier (GlotLID), the specialized AfroLID, and a general-purpose LLM (Llama 3.3 70B), performed with near-perfect accuracy on the clean news data but struggled with the code-switched Reddit posts. The study concludes that professionally curated news content is a more reliable and effective source for training context-rich AI models for African languages than data from conversational platforms. It also highlights the need for future models that can process clean and code-switched text to improve the detection accuracy for African languages.

Figures

Figures reproduced from arXiv: 2512.01557 by Bekalu Tadele, Chukuemeka Edeh, Edward Ajayi, Eudoxie Umwari, Jules Udahemuka, Mawuli Deku, Prosper Singadi.

Figure 1
Figure 1. Figure 1: Workflow diagram of the study’s methodology [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of Reddit Posts and News Articles Count. Left: Number of Reddit Posts. Right: Number of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of BBC Yoruba and FANA Broadcasting Comments. Left: BBC Yoruba Comments. Right: [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 3 canonical work pages

  1. [1]

    Datasets Collection Framework for Low-Resourced Languages in South Africa

    Nontokozo M. Magangane, Skhumbuzo G. Zwane, and Matthew O. Adigun. “Datasets Collection Framework for Low-Resourced Languages in South Africa.” In:2024 Conference on Information Communications Technology & Society (ICTAS), Mar 2024, pp. 69–74. doi:10.1109/ICTAS59620.2024.10507140

    arXiv 2024

  2. [2]

    MasakhaNER: Named Entity Recognition for African Languages

    David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba ...

    2021

  3. [3]

    1st ed., Routledge, London, Jan 2023

    Christopher Moseley and Eda Derhemi.Endangered Languages in the 21st Century. 1st ed., Routledge, London, Jan 2023. doi:10.4324/9781003260288

    work page doi:10.4324/9781003260288 2023

  4. [4]

    Endangered Languages as Well as Causes of Language Extinction

    Elnura Xolmatova and Jamila Usuvaliyeva. “Endangered Languages as Well as Causes of Language Extinction.” Modern Science and Research, vol. 4, no. 4, Apr 2025, pp. 1195–1200

    2025

  5. [5]

    Protecting the Invisible: Exploring the Preservation of Endangered Languages Through New Media Technologies

    Mirya Nezvitskaya. “Protecting the Invisible: Exploring the Preservation of Endangered Languages Through New Media Technologies.” 2021

    2021

  6. [6]

    Social Media in Second and Foreign Language Teaching and Learning: Blogs, Wikis, and Social Networking

    Jonathon Reinhardt. “Social Media in Second and Foreign Language Teaching and Learning: Blogs, Wikis, and Social Networking.”Language Teaching, vol. 52, Jan 2019, pp. 1–39. doi:10.1017/S0261444818000356

    work page doi:10.1017/s0261444818000356 2019

  7. [7]

    African Languages – An Overview

    ScienceDirect. “African Languages – An Overview.” Accessed Aug. 11, 2025

    2025

  8. [8]

    Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments

    Tuka Alhanai, Aida Kasumovic, Mohammad M. Ghassemi, Andreas Zitzelberger, Jon M. Lundin, and Guillaume Chabot-Couture. “Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments.”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 27, Apr 2025, pp. 2780...

    work page doi:10.1609/aaai.v39i27.34996 2025

  9. [9]

    BBC News Yoruba: Abawole

    BBC News Yoruba. “BBC News Yoruba: Abawole.” Launched February 19, 2018. Accessed Aug. 11, 2025

    2018

  10. [10]

    RBA: All the Breaking News and the Stories Happening

    Rwanda Broadcasting Agency. “RBA: All the Breaking News and the Stories Happening.” Accessed Aug. 11, 2025

    2025

  11. [11]

    Front Page – Welcome to Fana Media Corporation S.C

    Fana Media Corporation. “Front Page – Welcome to Fana Media Corporation S.C.” Accessed Aug. 11, 2025

    2025

  12. [12]

    r/Yoruba – Reddit

    Reddit. “r/Yoruba – Reddit.” Accessed Aug. 11, 2025

    2025

  13. [13]

    r/Amharic – Reddit

    Reddit. “r/Amharic – Reddit.” Accessed Aug. 11, 2025

    2025

  14. [14]

    r/Rwanda – Reddit

    Reddit. “r/Rwanda – Reddit.” Accessed Aug. 11, 2025

    2025

  15. [15]

    Scraping Reddit Using Python

    GeeksforGeeks. “Scraping Reddit Using Python.” Accessed Aug. 11, 2025

    2025

  16. [16]

    AfroLID: A Neural Language Identification Tool for African Languages

    Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. “AfroLID: A Neural Language Identification Tool for African Languages.” arXiv:2210.11744 (cs.CL), 2022

    Pith/arXiv arXiv 2022

  17. [17]

    Langdetect: Language Detection Library Ported from Google’s Language-Detection

    “Langdetect: Language Detection Library Ported from Google’s Language-Detection.” Accessed Aug. 11, 2025. 9

    2025

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.