REVIEW 2 major objections 5 minor 29 references
Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language
T0 review · 2 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The authors introduce Sagalee, a 100-hour public speech corpus for Oromo built from 53,573 crowd-sourced utterances by 283 speakers, and report baseline word error rates of 15.32% (Conformer hybrid), 18.74% (Conformer CTC), and 10.82%…
desk verdict Sagalee is a genuinely useful public Oromo ASR corpus, but the baseline WERs may be optimistic because the train/dev/test split is never stated to be speaker-disjoint. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Sagalee itself: a crowd-sourced corpus built by (1) curating roughly 30,000 sentences from public texts, restricted to 4–25 words per sentence, (2) recording speakers through an Android app that includes a skip button for problematic sentences, and (3) logging speaker demographics such as gender, age, and dialect to capture phonetic and acoustic diversity. The baselines are produced with the WeNet toolkit, using a Conformer encoder-decoder with hybrid CTC-AED loss and a pure-CTC variant trained from scratch with a BPE unigram tokenizer, plus a fine-tuned Whisper large-v3 model.
What would settle it
If a speaker-verification check of the released audio finds that the same voices appear in both the training and test portions of Sagalee, the reported 10.82% WER would not measure recognition of new speakers; likewise, a sample re-transcription showing high mismatch rates would undercut the claim that the transcriptions can serve as ground truth.
Extended reading notes
Core claim
Sagalee is the first publicly available ASR dataset for Oromo that is suitable for training speech recognizers: it contains 100 hours of read speech from 283 speakers, recorded at 24 kHz in unconstrained acoustic environments, with 53,573 utterances and sentence-level transcriptions. The authors argue that previously released Oromo speech data—3 hours from 6 speakers and 17 hours from one male speaker, both intended for text-to-speech—cannot support ASR development. Their experiments establish that the dataset yields a reasonable baseline: a from-scratch Conformer with hybrid CTC-AED loss reaches 15.32% WER, a pure-CTC Conformer 18.74%, and fine-tuning Whisper large-v3 improves the WER to 10.82%.
Load-bearing premise
The crowd-sourced transcriptions are accurate enough to serve as ground truth, and the train/test split is speaker-disjoint, so the reported word error rates reflect generalization rather than memorization of individual speakers.
Editorial extensions
If this is right
- With Sagalee released on GitHub, Oromo ASR researchers can train and compare models on a common corpus instead of relying on private or TTS-only data.
- The 10.82% WER from fine-tuned Whisper shows that large multilingual pre-training transfers to Oromo, giving a strong start for downstream applications like voice assistants and dictation.
- The 15.32% WER from a from-scratch Conformer indicates that roughly 100 hours of real-world Oromo speech is enough to train a usable recognizer, though not yet production quality.
- The reported baselines give future work concrete numbers to beat, enabling straightforward evaluation of new architectures, data augmentation, or transfer-learning methods.
Reading between the lines
- One testable extension is to verify whether the train/test split is speaker-disjoint; the paper does not state that speakers are exclusive across the split, so the reported WERs could partly reflect speaker identity leakage if utterances from the same person appear in both sets.
- Because transcriptions come from crowd workers reading prepared sentences, an accuracy check against the original texts or a manual re-transcription of a random sample would quantify transcription error and give users a quality bound, a measure the paper does not report.
- The dialect metadata collected at registration could support dialect-conditioned ASR or dialect-identification studies, a direction the paper does not explore.
- A future release reporting per-speaker and per-dialect WERs would help identify which speakers or dialect groups are hardest to recognize, which the current aggregate numbers do not reveal.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Sagalee, a crowdsourced Oromo speech corpus of approximately 100 hours of read audio from 283 speakers, with 53,573 utterances and paired transcriptions, released publicly. It also reports baseline ASR experiments: a from-scratch Conformer with hybrid CTC-AED loss reaches 15.32% WER, a from-scratch Conformer with pure CTC reaches 18.74% WER, and fine-tuned Whisper large-v3 reaches 10.82% WER. The central claim is that Sagalee is a usable public ASR resource for the under-represented Oromo language and that these WERs establish credible baselines.
Significance. If the dataset is made available as stated and the evaluation protocol is sound, this is a valuable contribution to low-resource speech processing: Oromo has very few public speech resources, and a 100-hour, multi-speaker corpus with train/dev/test separation and baseline results would be directly useful to the community. The data-collection procedure is described concretely, the dataset is non-circular relative to the baselines, and the internal ranking of the three models (hybrid Conformer better than pure-CTC Conformer; fine-tuned Whisper best) is consistent with standard ASR experience. The main significance hinges on whether the reported numbers measure true speaker-independent generalization.
major comments (2)
- [V (Experiments), dataset split] Section V states only that the dataset is "split into train, dev and test sets with 93.6hrs, 4.2hrs, and 2.4hrs size respectively," with no statement that the split is speaker-disjoint. Given 283 speakers and an average of about 192 utterances per speaker (Table III), an utterance-level random split would very likely place the same speaker in both training and test sets. If that occurred, the reported WERs of 15.32%, 18.74%, and 10.82% would not measure speaker-independent ASR generalization; they could be optimistically biased by the models memorizing speaker-specific acoustics and reading styles. This directly affects the paper's central claim that Sagalee is usable for ASR with credible baselines. The authors should either explicitly state that the split is speaker-disjoint and specify how it was constructed, or, if it is not, re-split the data with no speaker overlap and rerun the experiments.
- [IV.A.2 (Audio Recording) and V (Experiments)] The paper does not describe any transcription quality-control procedure beyond allowing speakers to skip sentences with spelling errors or difficult text. Since the crowd-sourced transcriptions serve as ground truth for the reported WERs, the absence of an independent verification step, a post-hoc correction procedure, or an inter-annotator agreement measure is a load-bearing gap: inaccurate transcriptions would inflate WER even with a perfect model, and the paper's claim that the dataset is "paired with transcriptions" as a usable ASR resource depends on transcription fidelity. The authors should describe their quality-control pipeline or explicitly acknowledge this as a limitation with a concrete estimate of transcription error.
minor comments (5)
- [II.B.1 (Writing System)] The phrase "digraphs knows as 'Qubee Dachaa'" contains a typo; it should be "known as."
- [Table I (Vowels)] The vowel table uses the notation "/5/" for the open central vowel, which appears to be a placeholder or rendering error; it should be a proper IPA symbol such as /ɐ/ or /a/.
- [V.B (Adaptation From Whisper)] The description of fine-tuning Whisper says the convolutional layers at the front of the encoder were "randomly initialized," but the reason for this choice is not explained; a sentence clarifying whether this was necessary for input feature compatibility or a deliberate regularization choice would improve reproducibility.
- [V.C (Result)] Table IV reports only WER percentages; reporting the number of utterances and speakers in the test set, as well as a confidence interval or significance test for the differences, would make the baselines more interpretable, especially since the test set appears to be only 2.4 hours.
- [IV.B (Data Statistics) and V (Experiments)] The paper reports per-split durations but not the number of utterances or speakers in each split; these numbers should be added so readers can assess the evaluation's statistical power.
Circularity Check
No significant circularity: the reported WERs are empirical measurements on a new public dataset, and no load-bearing step reduces to its own inputs.
full rationale
The paper's central claim is that Sagalee is a usable, publicly available ASR dataset for Oromo, supported by the dataset release and by measured baseline WERs. No step in the derivation chain is circular. The Conformer and Whisper results are empirical outcomes of training on the dataset; the WERs are not fixed by construction and are not derived from any fitted parameter that is later renamed as a prediction. The only hyperparameter search described is for the BPE vocabulary size, where the authors report trying several values and selecting nbpe=500; this is standard model selection and does not determine any reported WER by itself. The paper cites existing work for architectures and tooling, but these citations are not load-bearing for the dataset contribution: the related Oromo corpus [22] is mentioned only as prior context, not as the source of Sagalee or its baselines. There are no self-citations used to justify the central claim, no imported uniqueness theorem, and no ansatz smuggled in via citation. Potential concerns about speaker overlap in the train/dev/test split would be evaluation-validity issues rather than circularity and therefore fall outside this analysis. The paper is self-contained with respect to its main assertion: the dataset exists, is released, and its baselines are measured rather than derived.
Assumptions & free parameters
free parameters (3)
- BPE vocabulary size (nbpe) =
500
- Conformer training epochs =
200
- Whisper fine-tuning learning rate =
1e-5
assumptions (4)
- domain assumption Qubee orthography is used consistently in the transcriptions and is an adequate representation of Oromo phonology.
- domain assumption Crowd-sourced speakers read the displayed sentences as intended; self-reported dialect and demographic data are accurate.
- domain assumption The text sources (news, books, constitution) cover the intended domain and sentence-length filtering does not distort representativeness.
- standard math WeNet's WER computation is a standard, valid measure of ASR performance for this dataset.
Cite this review
Pith. "Pith review of Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language." pith.science (2026). https://pith.science/paper/Y3YPCS6W
@misc{pith2026250200421,
author = {Pith},
title = {Pith review of: Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y3YPCS6W}},
note = {Machine review of arXiv:2502.00421}
}
read the original abstract
We present a novel Automatic Speech Recognition (ASR) dataset for the Oromo language, a widely spoken language in Ethiopia and neighboring regions. The dataset was collected through a crowd-sourcing initiative, encompassing a diverse range of speakers and phonetic variations. It consists of 100 hours of real-world audio recordings paired with transcriptions, covering read speech in both clean and noisy environments. This dataset addresses the critical need for ASR resources for the Oromo language which is underrepresented. To show its applicability for the ASR task, we conducted experiments using the Conformer model, achieving a Word Error Rate (WER) of 15.32% with hybrid CTC and AED loss and WER of 18.74% with pure CTC loss. Additionally, fine-tuning the Whisper model resulted in a significantly improved WER of 10.82%. These results establish baselines for Oromo ASR, highlighting both the challenges and the potential for improving ASR performance in Oromo. The dataset is publicly available at https://github.com/turinaf/sagalee and we encourage its use for further research and development in Oromo speech processing.
Figures
Reference graph
Works this paper leans on
-
[1]
Automatic speech recognition using advanced deep learning approaches: A survey,
H. Kheddar, M. Hemis, and Y . Himeur, “Automatic speech recognition using advanced deep learning approaches: A survey,” Information Fusion, vol. 109, p. 102422, Sep. 2024. [Online]. Available: http://dx.doi.org/10.1016/j.inffus.2024.102422
arXiv 2024
-
[2]
Automatic speech recognition: a survey,
M. Malik, M. K. Malik, K. Mehmood, and I. Makhdoom, “Automatic speech recognition: a survey,” Multimedia Tools and Applications , vol. 80, pp. 9411–9457, 2021
work page 2021
-
[3]
Conformer: Convolution- augmented transformer for speech recognition,
A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution- augmented transformer for speech recognition,” 2020
2020
-
[4]
Robust speech recognition via large-scale weak supervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022. [Online]. Available: https://arxiv.org/abs/2212.04356
arXiv 2022
-
[5]
W. Tegegne, “The development of written afan oromo and the appro- priateness of qubee, latin script, for afan oromo writing,” Historical Research Letter, vol. 28, 2016
work page 2016
-
[6]
Historical challenges in the development of the oromo language and some agenda for future research,
F. Demie, “Historical challenges in the development of the oromo language and some agenda for future research,” The journal of Oromo studies, vol. 3, 1996. [Online]. Available: https://api.semanticscholar. org/CorpusID:155784010
work page 1996
-
[7]
The oromo orthography of shaykh bakri sapalo,
R. J. Hayward and M. Hassan, “The oromo orthography of shaykh bakri sapalo,” Bulletin of the School of Oriental and African Studies, University of London , vol. 44, no. 3, pp. 550–566, 1981. [Online]. Available: http://www.jstor.org/stable/616613
work page 1981
-
[8]
Some aspects of oromo history that have been misunderstood,
M. Hassen, “Some aspects of oromo history that have been misunderstood,” The journal of Oromo studies , vol. 1, 1994. [Online]. Available: https://api.semanticscholar.org/CorpusID:155351114
work page 1994
Show all 29 references
-
[9]
Qube afaan oromo: Reasons for choosing the latin script for developing an oromo alphabet,
T. Gamta, “Qube afaan oromo: Reasons for choosing the latin script for developing an oromo alphabet,” The journal of Oromo studies , vol. 1,
-
[10]
Stroomer, A Comparative Study of Three Southern Oromo Dialects in Kenya: Phonology, Morphology and V ocabulary , ser
H. Stroomer, A Comparative Study of Three Southern Oromo Dialects in Kenya: Phonology, Morphology and V ocabulary , ser. Cushitic language studies. Buske, 1987. [Online]. Available: https: //books.google.co.id/books?id=shoOAAAAY AAJ
1987
-
[11]
Griefenow-Mewis, A Grammatical Sketch of Written Oromo , ser
C. Griefenow-Mewis, A Grammatical Sketch of Written Oromo , ser. Grammatical analyses of african languages. R. K ¨oppe, 2001. [Online]. Available: https://books.google.com.hk/books?id=YuliAAAAMAAJ
2001
-
[12]
A survey of multilingual models for automatic speech recognition,
H. Yadav and S. Sitaram, “A survey of multilingual models for automatic speech recognition,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference . Marseille, France: European Language Resources Association, Jun. 2022, pp. 5071–5079. [Online]. Availabl...
2022
-
[13]
Stable distillation: Regularizing continued pre-training for low-resource automatic speech recognition,
A. Seth, S. Ghosh, S. Umesh, and D. Manocha, “Stable distillation: Regularizing continued pre-training for low-resource automatic speech recognition,” 2023. [Online]. Available: https://arxiv.org/abs/2312.12783
2023 arXiv
-
[14]
Knowledge distillation from multiple foundation models for end-to-end speech recognition,
X. Yang, Q. Li, C. Zhang, and P. C. Woodland, “Knowledge distillation from multiple foundation models for end-to-end speech recognition,”
-
[15]
Effectiveness of self-supervised pre-training for speech recognition,
A. Baevski, M. Auli, and A. Mohamed, “Effectiveness of self-supervised pre-training for speech recognition,” arXiv preprint arXiv:1911.03912 , 2019
1911 arXiv
-
[16]
Specaugment: A simple data augmentation method for automatic speech recognition,
D. S. Park, W. Chan, Y . Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V . Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in Interspeech 2019. ISCA, Sep. 2019. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-2680
2019 doi
-
[17]
Improving sequence-to-sequence speech recognition training with on-the-fly data augmentation,
T.-S. Nguyen, S. St ¨uker, J. Niehues, and A. Waibel, “Improving sequence-to-sequence speech recognition training with on-the-fly data augmentation,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2020, pp. 7689– 7693
2020
-
[18]
On-the- fly aligned data augmentation for sequence-to-sequence asr,
T. K. Lam, M. Ohta, S. Schamoni, and S. Riezler, “On-the- fly aligned data augmentation for sequence-to-sequence asr,” in Interspeech 2021 . ISCA, Aug. 2021. [Online]. Available: http: //dx.doi.org/10.21437/Interspeech.2021-1679
2021 doi
-
[19]
Joint acoustic modeling of triphones and trigraphemes by multi-task learning deep neural net- works for low-resource speech recognition,
D. Chen, B. Mak, C.-C. Leung, and S. Sivadas, “Joint acoustic modeling of triphones and trigraphemes by multi-task learning deep neural net- works for low-resource speech recognition,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . ...
2014
-
[20]
Crowdsourcing parallel corpus for english-oromo neural machine translation using community engagement platform,
S. Chala, B. Debisa, A. Diriba, S. Getachew, C. Getu, and S. Shiferaw, “Crowdsourcing parallel corpus for english-oromo neural machine translation using community engagement platform,” 2021. [Online]. Available: https://arxiv.org/abs/2102.07539
2021 arXiv
-
[21]
Afan oromo speech-based com- puter command and control: An evaluation with selected commands,
T. Kebede, M. Getachew, and C. Kris, “Afan oromo speech-based com- puter command and control: An evaluation with selected commands,” Jimma Institute of Technology Electrical and Computer Engineering , vol. 2023, 2023
2023
-
[22]
Large vocabulary read speech corpora for four Ethiopian languages: Amharic, Tigrigna, Oromo, and Wolaytta,
S. T. Abate, M. Y . Tachbelie, M. Melese, H. Abera, T. Gebreselassie, W. Mulugeta, Y . Assabie, M. M. Beyene, S. Atinafu, and B. E. Seyoum, “Large vocabulary read speech corpora for four Ethiopian languages: Amharic, Tigrigna, Oromo, and Wolaytta,” in Proceedings of the F ourt...
2020
-
[23]
Wenet: Production oriented streaming and non- streaming end-to-end speech recognition toolkit,
Z. Yao, D. Wu, X. Wang, B. Zhang, F. Yu, C. Yang, Z. Peng, X. Chen, L. Xie, and X. Lei, “Wenet: Production oriented streaming and non- streaming end-to-end speech recognition toolkit,” in Interspeech 2021 , 2021, pp. 4054–4058
2021
-
[24]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garn...
2017
-
[25]
Swish: a self-gated activation function,
P. Ramachandran, B. Zoph, and Q. V . Le, “Swish: a self-gated activation function,” arXiv: Neural and Evolutionary Computing , 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:196158220
2017
-
[26]
Neural machine translation of rare words with subword units,
R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , K. Erk and N. A. Smith, Eds. Berlin, Germany: Association ...
2016
-
[27]
Combining simple but novel data augmentation methods for improving conformer asr,
R. Damania, C. Homan, and E. Prud’hommeaux, “Combining simple but novel data augmentation methods for improving conformer asr,” in Interspeech 2022 , 2022, pp. 4890–4894
2022
-
[1993]
Available: https://api.semanticscholar.org/CorpusID: 60149752
[Online]. Available: https://api.semanticscholar.org/CorpusID: 60149752
-
[2023]
Available: https://arxiv.org/abs/2303.10917
[Online]. Available: https://arxiv.org/abs/2303.10917
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.