Pith. sign in

REVIEW 2 major objections 5 minor 29 references

Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language

T0 review · 2 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Pith's one-line read The authors introduce Sagalee, a 100-hour public speech corpus for Oromo built from 53,573 crowd-sourced utterances by 283 speakers, and report baseline word error rates of 15.32% (Conformer hybrid), 18.74% (Conformer CTC), and 10.82%…

desk verdict Sagalee is a genuinely useful public Oromo ASR corpus, but the baseline WERs may be optimistic because the train/dev/test split is never stated to be speaker-disjoint. read the letter →

arxiv 2502.00421 v1 pith:Y3YPCS6W submitted 2025-02-01 cs.CL cs.SDeess.AS

classification cs.CLcs.SDeess.AS
keywords SagaleeOromoASRAfaanOromoolow-resourcespeechrecognitioncrowd-sourcedcorpusConformerWhisperfine-tuningworderrorrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Sagalee, a publicly released 100-hour speech corpus for Oromo (Afaan Oromoo), a language with over 45 million speakers that had no usable public ASR dataset. The authors collected 53,573 read utterances from 283 speakers through a crowd-sourcing mobile app, with transcriptions drawn from news, books, and the Oromia regional constitution. To show the corpus works for speech recognition, they trained a Conformer model from scratch and fine-tuned Whisper large-v3, reporting word error rates of 15.32%, 18.74%, and 10.82% respectively. The point is to give Oromo a public, real-world-condition dataset and baselines so that ASR research for the language can start from a shared foundation.

What carries the argument

The load-bearing object is Sagalee itself: a crowd-sourced corpus built by (1) curating roughly 30,000 sentences from public texts, restricted to 4–25 words per sentence, (2) recording speakers through an Android app that includes a skip button for problematic sentences, and (3) logging speaker demographics such as gender, age, and dialect to capture phonetic and acoustic diversity. The baselines are produced with the WeNet toolkit, using a Conformer encoder-decoder with hybrid CTC-AED loss and a pure-CTC variant trained from scratch with a BPE unigram tokenizer, plus a fine-tuned Whisper large-v3 model.

What would settle it

If a speaker-verification check of the released audio finds that the same voices appear in both the training and test portions of Sagalee, the reported 10.82% WER would not measure recognition of new speakers; likewise, a sample re-transcription showing high mismatch rates would undercut the claim that the transcriptions can serve as ground truth.

Watch

Extended reading notes

Core claim

Sagalee is the first publicly available ASR dataset for Oromo that is suitable for training speech recognizers: it contains 100 hours of read speech from 283 speakers, recorded at 24 kHz in unconstrained acoustic environments, with 53,573 utterances and sentence-level transcriptions. The authors argue that previously released Oromo speech data—3 hours from 6 speakers and 17 hours from one male speaker, both intended for text-to-speech—cannot support ASR development. Their experiments establish that the dataset yields a reasonable baseline: a from-scratch Conformer with hybrid CTC-AED loss reaches 15.32% WER, a pure-CTC Conformer 18.74%, and fine-tuning Whisper large-v3 improves the WER to 10.82%.

Load-bearing premise

The crowd-sourced transcriptions are accurate enough to serve as ground truth, and the train/test split is speaker-disjoint, so the reported word error rates reflect generalization rather than memorization of individual speakers.

Editorial extensions

If this is right

  • With Sagalee released on GitHub, Oromo ASR researchers can train and compare models on a common corpus instead of relying on private or TTS-only data.
  • The 10.82% WER from fine-tuned Whisper shows that large multilingual pre-training transfers to Oromo, giving a strong start for downstream applications like voice assistants and dictation.
  • The 15.32% WER from a from-scratch Conformer indicates that roughly 100 hours of real-world Oromo speech is enough to train a usable recognizer, though not yet production quality.
  • The reported baselines give future work concrete numbers to beat, enabling straightforward evaluation of new architectures, data augmentation, or transfer-learning methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to verify whether the train/test split is speaker-disjoint; the paper does not state that speakers are exclusive across the split, so the reported WERs could partly reflect speaker identity leakage if utterances from the same person appear in both sets.
  • Because transcriptions come from crowd workers reading prepared sentences, an accuracy check against the original texts or a manual re-transcription of a random sample would quantify transcription error and give users a quality bound, a measure the paper does not report.
  • The dialect metadata collected at registration could support dialect-conditioned ASR or dialect-identification studies, a direction the paper does not explore.
  • A future release reporting per-speaker and per-dialect WERs would help identify which speakers or dialect groups are hardest to recognize, which the current aggregate numbers do not reveal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Sagalee, a crowdsourced Oromo speech corpus of approximately 100 hours of read audio from 283 speakers, with 53,573 utterances and paired transcriptions, released publicly. It also reports baseline ASR experiments: a from-scratch Conformer with hybrid CTC-AED loss reaches 15.32% WER, a from-scratch Conformer with pure CTC reaches 18.74% WER, and fine-tuned Whisper large-v3 reaches 10.82% WER. The central claim is that Sagalee is a usable public ASR resource for the under-represented Oromo language and that these WERs establish credible baselines.

Significance. If the dataset is made available as stated and the evaluation protocol is sound, this is a valuable contribution to low-resource speech processing: Oromo has very few public speech resources, and a 100-hour, multi-speaker corpus with train/dev/test separation and baseline results would be directly useful to the community. The data-collection procedure is described concretely, the dataset is non-circular relative to the baselines, and the internal ranking of the three models (hybrid Conformer better than pure-CTC Conformer; fine-tuned Whisper best) is consistent with standard ASR experience. The main significance hinges on whether the reported numbers measure true speaker-independent generalization.

major comments (2)
  1. [V (Experiments), dataset split] Section V states only that the dataset is "split into train, dev and test sets with 93.6hrs, 4.2hrs, and 2.4hrs size respectively," with no statement that the split is speaker-disjoint. Given 283 speakers and an average of about 192 utterances per speaker (Table III), an utterance-level random split would very likely place the same speaker in both training and test sets. If that occurred, the reported WERs of 15.32%, 18.74%, and 10.82% would not measure speaker-independent ASR generalization; they could be optimistically biased by the models memorizing speaker-specific acoustics and reading styles. This directly affects the paper's central claim that Sagalee is usable for ASR with credible baselines. The authors should either explicitly state that the split is speaker-disjoint and specify how it was constructed, or, if it is not, re-split the data with no speaker overlap and rerun the experiments.
  2. [IV.A.2 (Audio Recording) and V (Experiments)] The paper does not describe any transcription quality-control procedure beyond allowing speakers to skip sentences with spelling errors or difficult text. Since the crowd-sourced transcriptions serve as ground truth for the reported WERs, the absence of an independent verification step, a post-hoc correction procedure, or an inter-annotator agreement measure is a load-bearing gap: inaccurate transcriptions would inflate WER even with a perfect model, and the paper's claim that the dataset is "paired with transcriptions" as a usable ASR resource depends on transcription fidelity. The authors should describe their quality-control pipeline or explicitly acknowledge this as a limitation with a concrete estimate of transcription error.
minor comments (5)
  1. [II.B.1 (Writing System)] The phrase "digraphs knows as 'Qubee Dachaa'" contains a typo; it should be "known as."
  2. [Table I (Vowels)] The vowel table uses the notation "/5/" for the open central vowel, which appears to be a placeholder or rendering error; it should be a proper IPA symbol such as /ɐ/ or /a/.
  3. [V.B (Adaptation From Whisper)] The description of fine-tuning Whisper says the convolutional layers at the front of the encoder were "randomly initialized," but the reason for this choice is not explained; a sentence clarifying whether this was necessary for input feature compatibility or a deliberate regularization choice would improve reproducibility.
  4. [V.C (Result)] Table IV reports only WER percentages; reporting the number of utterances and speakers in the test set, as well as a confidence interval or significance test for the differences, would make the baselines more interpretable, especially since the test set appears to be only 2.4 hours.
  5. [IV.B (Data Statistics) and V (Experiments)] The paper reports per-split durations but not the number of utterances or speakers in each split; these numbers should be added so readers can assess the evaluation's statistical power.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported WERs are empirical measurements on a new public dataset, and no load-bearing step reduces to its own inputs.

full rationale

The paper's central claim is that Sagalee is a usable, publicly available ASR dataset for Oromo, supported by the dataset release and by measured baseline WERs. No step in the derivation chain is circular. The Conformer and Whisper results are empirical outcomes of training on the dataset; the WERs are not fixed by construction and are not derived from any fitted parameter that is later renamed as a prediction. The only hyperparameter search described is for the BPE vocabulary size, where the authors report trying several values and selecting nbpe=500; this is standard model selection and does not determine any reported WER by itself. The paper cites existing work for architectures and tooling, but these citations are not load-bearing for the dataset contribution: the related Oromo corpus [22] is mentioned only as prior context, not as the source of Sagalee or its baselines. There are no self-citations used to justify the central claim, no imported uniqueness theorem, and no ansatz smuggled in via citation. Potential concerns about speaker overlap in the train/dev/test split would be evaluation-validity issues rather than circularity and therefore fall outside this analysis. The paper is self-contained with respect to its main assertion: the dataset exists, is released, and its baselines are measured rather than derived.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on assumptions about transcription fidelity, orthography consistency, and evaluation tooling, plus standard model hyperparameters. No new physical or linguistic entities are postulated.

free parameters (3)
  • BPE vocabulary size (nbpe) = 500
    Selected by comparing 50, 100, 500, and 5000 on dev performance; affects tokenization and final WER, but not the dataset claim.
  • Conformer training epochs = 200
    Chosen by hand; no early stopping or dev-based selection reported.
  • Whisper fine-tuning learning rate = 1e-5
    Standard fine-tuning hyperparameter; not the central claim.
assumptions (4)
  • domain assumption Qubee orthography is used consistently in the transcriptions and is an adequate representation of Oromo phonology.
    Section II-B describes Qubee as the standard script; the dataset assumes transcriptions follow it.
  • domain assumption Crowd-sourced speakers read the displayed sentences as intended; self-reported dialect and demographic data are accurate.
    Section IV-A describes the recording process but no verification step for reading accuracy or metadata.
  • domain assumption The text sources (news, books, constitution) cover the intended domain and sentence-length filtering does not distort representativeness.
    Section IV-A1 describes curation of 30,000 sentences; no analysis of domain bias.
  • standard math WeNet's WER computation is a standard, valid measure of ASR performance for this dataset.
    The paper relies on the WeNet toolkit's evaluation without questioning its metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language." pith.science (2026). https://pith.science/paper/Y3YPCS6W

@misc{pith2026250200421,
  author       = {Pith},
  title        = {Pith review of: Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3YPCS6W}},
  note         = {Machine review of arXiv:2502.00421}
}
read the original abstract

We present a novel Automatic Speech Recognition (ASR) dataset for the Oromo language, a widely spoken language in Ethiopia and neighboring regions. The dataset was collected through a crowd-sourcing initiative, encompassing a diverse range of speakers and phonetic variations. It consists of 100 hours of real-world audio recordings paired with transcriptions, covering read speech in both clean and noisy environments. This dataset addresses the critical need for ASR resources for the Oromo language which is underrepresented. To show its applicability for the ASR task, we conducted experiments using the Conformer model, achieving a Word Error Rate (WER) of 15.32% with hybrid CTC and AED loss and WER of 18.74% with pure CTC loss. Additionally, fine-tuning the Whisper model resulted in a significantly improved WER of 10.82%. These results establish baselines for Oromo ASR, highlighting both the challenges and the potential for improving ASR performance in Oromo. The dataset is publicly available at https://github.com/turinaf/sagalee and we encourage its use for further research and development in Oromo speech processing.

Figures

Figures reproduced from arXiv: 2502.00421 by the authors.

Figure 1
Figure 1. Distribution of Sentences Length in Number of Words [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Mobile App Developed for Data Collection. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of Sagalee dataset across age groups, dialect, gender and [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 22 canonical work pages

  1. [1]

    Automatic speech recognition using advanced deep learning approaches: A survey,

    H. Kheddar, M. Hemis, and Y . Himeur, “Automatic speech recognition using advanced deep learning approaches: A survey,” Information Fusion, vol. 109, p. 102422, Sep. 2024. [Online]. Available: http://dx.doi.org/10.1016/j.inffus.2024.102422

  2. [2]

    Automatic speech recognition: a survey,

    M. Malik, M. K. Malik, K. Mehmood, and I. Makhdoom, “Automatic speech recognition: a survey,” Multimedia Tools and Applications , vol. 80, pp. 9411–9457, 2021

  3. [3]

    Conformer: Convolution- augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution- augmented transformer for speech recognition,” 2020

  4. [4]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. [Online]. Available: https://arxiv.org/abs/2212.04356

  5. [5]

    The development of written afan oromo and the appro- priateness of qubee, latin script, for afan oromo writing,

    W. Tegegne, “The development of written afan oromo and the appro- priateness of qubee, latin script, for afan oromo writing,” Historical Research Letter, vol. 28, 2016

  6. [6]

    Historical challenges in the development of the oromo language and some agenda for future research,

    F. Demie, “Historical challenges in the development of the oromo language and some agenda for future research,” The journal of Oromo studies, vol. 3, 1996. [Online]. Available: https://api.semanticscholar. org/CorpusID:155784010

  7. [7]

    The oromo orthography of shaykh bakri sapalo,

    R. J. Hayward and M. Hassan, “The oromo orthography of shaykh bakri sapalo,” Bulletin of the School of Oriental and African Studies, University of London , vol. 44, no. 3, pp. 550–566, 1981. [Online]. Available: http://www.jstor.org/stable/616613

  8. [8]

    Some aspects of oromo history that have been misunderstood,

    M. Hassen, “Some aspects of oromo history that have been misunderstood,” The journal of Oromo studies , vol. 1, 1994. [Online]. Available: https://api.semanticscholar.org/CorpusID:155351114

Show all 29 references
  1. [9]

    Qube afaan oromo: Reasons for choosing the latin script for developing an oromo alphabet,

    T. Gamta, “Qube afaan oromo: Reasons for choosing the latin script for developing an oromo alphabet,” The journal of Oromo studies , vol. 1,

  2. [10]

    Stroomer, A Comparative Study of Three Southern Oromo Dialects in Kenya: Phonology, Morphology and V ocabulary , ser

    H. Stroomer, A Comparative Study of Three Southern Oromo Dialects in Kenya: Phonology, Morphology and V ocabulary , ser. Cushitic language studies. Buske, 1987. [Online]. Available: https: //books.google.co.id/books?id=shoOAAAAY AAJ

  3. [11]

    Griefenow-Mewis, A Grammatical Sketch of Written Oromo , ser

    C. Griefenow-Mewis, A Grammatical Sketch of Written Oromo , ser. Grammatical analyses of african languages. R. K ¨oppe, 2001. [Online]. Available: https://books.google.com.hk/books?id=YuliAAAAMAAJ

  4. [12]

    A survey of multilingual models for automatic speech recognition,

    H. Yadav and S. Sitaram, “A survey of multilingual models for automatic speech recognition,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference . Marseille, France: European Language Resources Association, Jun. 2022, pp. 5071–5079. [Online]. Availabl...

  5. [13]

    Stable distillation: Regularizing continued pre-training for low-resource automatic speech recognition,

    A. Seth, S. Ghosh, S. Umesh, and D. Manocha, “Stable distillation: Regularizing continued pre-training for low-resource automatic speech recognition,” 2023. [Online]. Available: https://arxiv.org/abs/2312.12783

  6. [14]

    Knowledge distillation from multiple foundation models for end-to-end speech recognition,

    X. Yang, Q. Li, C. Zhang, and P. C. Woodland, “Knowledge distillation from multiple foundation models for end-to-end speech recognition,”

  7. [15]

    Effectiveness of self-supervised pre-training for speech recognition,

    A. Baevski, M. Auli, and A. Mohamed, “Effectiveness of self-supervised pre-training for speech recognition,” arXiv preprint arXiv:1911.03912 , 2019

  8. [16]

    Specaugment: A simple data augmentation method for automatic speech recognition,

    D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in Interspeech 2019. ISCA, Sep. 2019. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2680

  9. [17]

    Improving sequence-to-sequence speech recognition training with on-the-fly data augmentation,

    T.-S. Nguyen, S. St ¨uker, J. Niehues, and A. Waibel, “Improving sequence-to-sequence speech recognition training with on-the-fly data augmentation,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2020, pp. 7689– 7693

  10. [18]

    On-the- fly aligned data augmentation for sequence-to-sequence asr,

    T. K. Lam, M. Ohta, S. Schamoni, and S. Riezler, “On-the- fly aligned data augmentation for sequence-to-sequence asr,” in Interspeech 2021 . ISCA, Aug. 2021. [Online]. Available: http: //dx.doi.org/10.21437/Interspeech.2021-1679

  11. [19]

    Joint acoustic modeling of triphones and trigraphemes by multi-task learning deep neural net- works for low-resource speech recognition,

    D. Chen, B. Mak, C.-C. Leung, and S. Sivadas, “Joint acoustic modeling of triphones and trigraphemes by multi-task learning deep neural net- works for low-resource speech recognition,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . ...

  12. [20]

    Crowdsourcing parallel corpus for english-oromo neural machine translation using community engagement platform,

    S. Chala, B. Debisa, A. Diriba, S. Getachew, C. Getu, and S. Shiferaw, “Crowdsourcing parallel corpus for english-oromo neural machine translation using community engagement platform,” 2021. [Online]. Available: https://arxiv.org/abs/2102.07539

  13. [21]

    Afan oromo speech-based com- puter command and control: An evaluation with selected commands,

    T. Kebede, M. Getachew, and C. Kris, “Afan oromo speech-based com- puter command and control: An evaluation with selected commands,” Jimma Institute of Technology Electrical and Computer Engineering , vol. 2023, 2023

  14. [22]

    Large vocabulary read speech corpora for four Ethiopian languages: Amharic, Tigrigna, Oromo, and Wolaytta,

    S. T. Abate, M. Y . Tachbelie, M. Melese, H. Abera, T. Gebreselassie, W. Mulugeta, Y . Assabie, M. M. Beyene, S. Atinafu, and B. E. Seyoum, “Large vocabulary read speech corpora for four Ethiopian languages: Amharic, Tigrigna, Oromo, and Wolaytta,” in Proceedings of the F ourt...

  15. [23]

    Wenet: Production oriented streaming and non- streaming end-to-end speech recognition toolkit,

    Z. Yao, D. Wu, X. Wang, B. Zhang, F. Yu, C. Yang, Z. Peng, X. Chen, L. Xie, and X. Lei, “Wenet: Production oriented streaming and non- streaming end-to-end speech recognition toolkit,” in Interspeech 2021 , 2021, pp. 4054–4058

  16. [24]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garn...

  17. [25]

    Swish: a self-gated activation function,

    P. Ramachandran, B. Zoph, and Q. V . Le, “Swish: a self-gated activation function,” arXiv: Neural and Evolutionary Computing , 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:196158220

  18. [26]

    Neural machine translation of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , K. Erk and N. A. Smith, Eds. Berlin, Germany: Association ...

  19. [27]

    Combining simple but novel data augmentation methods for improving conformer asr,

    R. Damania, C. Homan, and E. Prud’hommeaux, “Combining simple but novel data augmentation methods for improving conformer asr,” in Interspeech 2022 , 2022, pp. 4890–4894

  20. [1993]

    Available: https://api.semanticscholar.org/CorpusID: 60149752

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 60149752

  21. [2023]

    Available: https://arxiv.org/abs/2303.10917

    [Online]. Available: https://arxiv.org/abs/2303.10917

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.