REVIEW 3 major objections 6 minor 33 references
Fotheidil: an Automatic Transcription System for the Irish Language
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper presents Fotheidil, the first web-based transcription system for Irish, and claims that pseudo-labelled radio data plus a sequence-to-sequence text-restoration stage deliver substantial accuracy gains for a low-resource language.
desk verdict Useful Irish ASR systems paper with a real SSL story, but the headline comparison needs one controlled ablation before the SSL claim is solid. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are two. First, a Noisy-Student-Teacher-style semi-supervised loop: a supervised time-delay neural network acoustic model (M0) acts as teacher, decoding an unlabelled 3,230-hour radio corpus into lattices that are rescored and converted to pseudo-labels; the student model (M1) is trained on supervised plus pseudo-labelled data with SpecAugment spectral masking, and the M0-versus-M1 comparison carries the semi-supervised claim. Second, a transformer-based sequence-to-sequence model maps lower-cased, punctuation-free ASR-style text to rich transcriptions in one pass, restoring capitals, punctuation, digits and acronyms; it is compared against a word-level classifier whose output classes include second- and third-letter capitalisation to handle Irish initial mutation.
What would settle it
Train a model with the same architecture as the student on the 398-hour supervised set alone for 6 epochs with SpecAugment, decode the same five test sets, and compare word error rates with the student's numbers in Table 1; if the epoch-matched supervised model matches or beats the student, the observed gains cannot be attributed to the unlabelled data.
Extended reading notes
Core claim
The paper establishes that a substantial pool of untranscribed broadcast speech can be converted into effective training material for an Irish speech recogniser. A teacher acoustic model trained on 398 hours of labelled speech decodes 3,230 hours of radio audio; the decoding lattices are rescored with a 5-gram language model, the best paths become pseudo-labels, and a student model is trained on the combined data with SpecAugment noise. The student is claimed to be better not only on in-domain test sets but especially on out-of-domain data and on Ulster speech, the dialect least represented in the supervised training set. Independently, the paper claims that treating capitalisation and punctuation restoration as a sequence-to-sequence task yields better rich transcriptions than word-by-word classification, and that the restoration model remains useful when applied to actual recognition output.
Load-bearing premise
The comparison that carries the paper attributes the student model's improvement to the 3,230 hours of pseudo-labelled radio data, but the student is also trained for more epochs (6 versus 4) with the same noising scheme, so if longer training alone on the 398-hour supervised set produced the same gains, the semi-supervised claim would not be established.
Editorial extensions
If this is right
- If the semi-supervised result holds, untranscribed radio archives are a practical substitute for costly manual transcription when building speech recognisers for under-resourced languages.
- Because Ulster speech improved most, semi-supervised learning appears to counteract dialect bias when the unlabelled data is rich in the underrepresented dialect.
- RNNLM rescoring stacks with the improved acoustic model, so language-model and acoustic-model gains are complementary rather than redundant.
- The sequence-to-sequence C&PR model keeps rich-transcription word error below 9% on all test sets except CommonVoice, whose short fragmentary sentences form a distinct hard case.
- User corrections made through the Fotheidil interface can be fed back into the training data, supporting a community-driven improvement loop.
Reading between the lines
- Editorial inference: if the unlabelled corpus were enlarged or the loop iterated with the student as the next teacher, the reported relative gains for Ulster and out-of-domain speech could grow further; the paper notes this as future work rather than a demonstrated result.
- Editorial inference: the sequence-to-sequence restoration model's ability to output digits and acronyms suggests the same architecture could convert spoken-out units, currency or symbol names in dictation applications, a direction the paper mentions but does not test.
- Editorial inference: because the student model trains for more epochs than the teacher, the observed ASR gain should be read as provisional until an epoch-matched supervised baseline is run; the paper's design leaves that comparison open.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Fotheidil, a web-based automatic transcription system for Irish that integrates voice activity detection, speaker diarisation, a Kaldi TDNN-HMM ASR model, and a capitalisation/punctuation restoration (C&PR) module. The core ASR experiment compares a 398-hour supervised baseline (M0) with a student model (M1) trained semi-supervisedly on the same supervised data plus 3,230 hours of pseudo-labelled Irish radio speech, reporting relative WER reductions of 9% on MíleGlór, 2% on SS, 14% on CommonVoice, 7% on Fleurs, and 27% on a hand-labelled radio test set (HL), as well as a 17% relative improvement for Ulster speakers on MíleGlór. The C&PR work compares a sequence-to-sequence transformer with a token-level classifier, showing that the S2S model roughly halves WERpc on rich-transcription targets. The paper also describes the system interface and deployment details.
Significance. If the results hold, this is a useful demonstration of semi-supervised learning for a low-resource, dialect-rich language, and the S2S C&PR approach is a practical contribution for Irish text processing. The paper is commendably concrete: it gives detailed data-split tables, test-set descriptions, and model configurations that would allow reproduction, and it openly discloses the overlap between the HL test set and the unsupervised radio domain. The central machine-learning claim, however, currently rests on a comparison that confounds the pseudo-labelled data with changes in training epochs and augmentation, so the magnitude of the SSL-specific benefit is not yet established.
major comments (3)
- [§4.3.2, Tables 1–2] The headline comparison between M0 and M1 confounds the addition of the 3,230-hour pseudo-labelled set with the change in training schedule (4 vs 6 epochs) and with the augmentation setup, since M1 is described as trained 'with SpecAug' but the exact relationship to M0's augmentation is not specified. The paper itself notes in §4.3.2 that training duration and noising are not ablated. To support the central claim that semi-supervised learning is the cause of the WER reductions, the authors should train a matched control model on the 398-hour supervised set alone using the same number of epochs, the same augmentation pipeline, and the same optimisation settings as M1; if that control already reaches M1's WERs on CommonVoice, Fleurs, and HL, then the SSL conclusion would not be supported. Reporting this ablation, or explicitly redefining the claim as a combined effect of extra data and longer training, is essential.
- [§4.3.1–4.3.3, Table 1] The hand-labelled HL test set (0.7 hours, 198 utterances) is drawn from the same four radio programmes that generated the unsupervised pseudo-labelled data, with overlapping speakers and likely overlapping content. The 27% relative WER improvement on HL is therefore an in-domain, self-confirming measurement and should be presented separately from the out-of-domain gains rather than grouped with them in the abstract and conclusions. Moreover, because HL and CommonVoice are very small (0.7 h and 0.6 h, respectively), the reported differences may be within sampling noise; please provide bootstrap confidence intervals or significance tests for the WER differences in Tables 1 and 2.
- [§4.3.3, Table 2] The claim that semi-supervised learning improves the underrepresented Ulster dialect is not uniquely attributable to SSL: the unsupervised set contains 1,002 hours of Barrscéalta, an Ulster-dominant programme, so the 17% relative improvement for Ul speakers on MíleGlór could reflect additional exposure to Ulster speech in the pseudo-labelled data rather than a benefit of the semi-supervised mechanism per se. The controlled ablation requested in the first major comment would help to separate these effects; at minimum, the paper should acknowledge this conflation and soften the causal wording in the discussion.
minor comments (6)
- [§4.3.2] Please state explicitly whether M1 uses speed perturbation in addition to SpecAug, and give the SpecAug hyperparameters (e.g., time-mask width, frequency-mask width, number of masks) or a citation to the exact configuration used, so the two models can be compared.
- [§4.4.1] In the sentence introducing the evaluation datasets, 'MiléGlór' is a typo for 'MíleGlór'; please correct this throughout the paper.
- [§3.1] The word 'editting' in the dashboard description should be 'editing'.
- [References] The citation for the TDNN architecture, 'Peddinti et al., 2025', lists the year 2025 but the venue as Interspeech 2024; the original paper appears to be Interspeech 2015. Please correct the year, page numbers, and venue, and ensure the in-text citation matches.
- [§4.4.2] The validation metric 'CE-mean-words' is not defined; please explain what it measures or remove it if it is not needed for understanding the early-stopping procedure.
- [Table 5, §4.4.3] The sentence 'The final WERs are always lower than the accumulated WERs of the ASR and the S2S, suggesting that the degradation in the input text generated by the ASR does not impact in the performance of the S2S system' is difficult to interpret; please clarify how the 'accumulated' error is defined and what the comparison is intended to show.
Circularity Check
No circularity: the ASR and C&PR claims are empirical comparisons against held-out human references, not derivations from their own inputs.
full rationale
Fotheidil is an empirical systems paper; neither the semi-supervised ASR comparison nor the C&PR evaluation contains a step that reduces by construction to its inputs. The SSL claim is supported by held-out test sets (MG, SS, CV, FL, HL) with human-labelled references; pseudo-labels from the teacher M0 are used only as training targets for M1, while all reported WER numbers compare against human transcripts, so the evaluation is not defined by the fitted model's own outputs. The HL test set overlaps in speakers and content with the unsupervised radio data and is explicitly disclosed as such, and the M0-vs-M1 comparison is not fully controlled for training duration or SpecAug; both are experimental validity concerns (confounds), not circular reasoning. Self-citations to prior ABAIR work concern the pronunciation lexicon, data alignment protocol, and earlier dialect-bias findings; these are shared inputs or background for both models and are not the claim being derived. The C&PR experiment trains on the RT/NR/IN corpus split and evaluates on held-out FO/MG/CV/FL data; no fitted parameter is renamed as a prediction. There is no self-citation chain invoked to forbid alternatives, no uniqueness theorem, and no ansatz imported to make the conclusion true. Accordingly, no circular step can be quoted, and the paper is self-contained with respect to circularity.
Assumptions & free parameters
free parameters (2)
- Supervised/unsupervised alignment weighting =
1:1
- Training epochs for M1 vs M0 =
6 vs 4
assumptions (5)
- domain assumption Teacher model pseudo-labels on unlabelled audio are accurate enough to serve as training targets
- domain assumption Off-the-shelf Silero-VAD and Kaldi x-vector diarisation models work adequately on Irish broadcast audio
- domain assumption The hand-labelled HL test set (0.7h, 198 utterances) is representative of the unsupervised radio domain
- standard math Standard Kaldi TDNN-HMM and transformer training procedures are reliable as implemented
- domain assumption The 5-million-sentence text corpus is clean and representative for C&PR training
Cite this review
Pith. "Pith review of Fotheidil: an Automatic Transcription System for the Irish Language." pith.science (2026). https://pith.science/paper/RIQXHQQQ
@misc{pith2026250100509,
author = {Pith},
title = {Pith review of: Fotheidil: an Automatic Transcription System for the Irish Language},
year = {2026},
howpublished = {\url{https://pith.science/paper/RIQXHQQQ}},
note = {Machine review of arXiv:2501.00509}
}
read the original abstract
This paper sets out the first web-based transcription system for the Irish language - Fotheidil, a system that utilises speech-related AI technologies as part of the ABAIR initiative. The system includes both off-the-shelf pre-trained voice activity detection and speaker diarisation models and models trained specifically for Irish automatic speech recognition and capitalisation and punctuation restoration. Semi-supervised learning is explored to improve the acoustic model of a modular TDNN-HMM ASR system, yielding substantial improvements for out-of-domain test sets and dialects that are underrepresented in the supervised training set. A novel approach to capitalisation and punctuation restoration involving sequence-to-sequence models is compared with the conventional approach using a classification model. Experimental results show here also substantial improvements in performance. The system will be made freely available for public use, and represents an important resource to researchers and others who transcribe Irish language materials. Human-corrected transcriptions will be collected and included in the training dataset as the system is used, which should lead to incremental improvements to the ASR model in a cyclical, community-driven fashion.
Figures
Reference graph
Works this paper leans on
-
[1]
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218--4222, Marseille, France. European Language Resources A...
work page 2020
-
[2]
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449--12460
work page 2020
-
[3]
Mitchell DeHaven and Jayadev Billa. 2022. Improving low-resource speech recognition with pretrained speech models: Continued pretraining vs. semi-supervised training. arXiv preprint arXiv:2207.00659
arXiv 2022
-
[4]
Sebastian Egger, Tobias Hossfeld, Raimund Schatz, and Markus Fiedler. 2012. Waiting times in quality of experience for web based services. In 2012 Fourth International Workshop on Quality of Multimedia Experience, pages 86--96. IEEE
work page 2012
-
[5]
Miquel Espl \`a , Mikel Forcada, Gema Ram \' rez-S \'a nchez, and Hieu Hoang. 2019. P ara C rawl: Web-scale parallel corpora for the languages of the EU . In Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks, pages 118--119, Dublin, Ireland. European Association for Machine Translation
work page 2019
-
[6]
Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, Andr\' e F. T. Martins, and Alexandra Birch. 2018. Marian: Fast neural machine translation in C++ . In Proceedings of ACL 2018, System Demonstrations, pages 116--121, Melbourne, Australia...
work page 2018
-
[7]
Adam Kilgarriff, Michael Rundell, and Elaine U \' Dhonnchadha. 2006. Efficient corpus development for lexicography: building the new corpus for ireland. Language Resources and Evaluation, 40:127--152
work page 2006
-
[8]
Woojoo Kim, Shuping Xiong, and Zhuoqian Liang. 2017. Effect of loading symbol of online video on perception of waiting time. International Journal of Human--Computer Interaction, 33(12):1001--1009
work page 2017
Show all 33 references
-
[9]
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. Audio augmentation for speech recognition. In Proc. Interspeech 2015, pages 3586--3589
2015
-
[10]
Chia-Yu Li and Ngoc Thang Vu. 2024. Improving noisy student training for low-resource languages in end-to-end ASR using CycleGAN and inter-domain losses. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages@ LREC-COLING 2024, page...
2024
-
[11]
Liam Lonergan, Mengjie Qian, Neasa N \' Chiar \'a in, Christer Gobl, and Ailbhe N \' Chasaide. 2023 a . Cross-dialect lexicon optimisation for an endangered language ASR system: the case of Irish . In Proc. Interspeech 2023, pages 4865--4869
2023
-
[12]
Liam Lonergan, Mengjie Qian, Neasa Ní Chiaráin, Christer Gobl, and Ailbhe Ní Chasaide. 2024. Low-resource speech recognition and dialect identification of Irish in a multi-task framework. In The Speaker and Language Recognition Workshop (Odyssey 2024), pages 67--73
2024
-
[13]
Liam Lonergan, Mengjie Qian, Neasa Ní Chiaráin, Christer Gobl, and Ailbhe Ní Chasaide. 2023 b . Towards dialect-inclusive recognition in a low-resource language: are balanced corpora the answer? In Proc. Interspeech 2023, pages 5082--5086
2023
-
[14]
Min Ma, Yuma Koizumi, Shigeki Karita, Heiga Zen, Jason Riesa, Haruko Ishikawa, and Michiel Bacchiani. 2024. Fleurs-r: A restored multilingual speech corpus for generation tasks. In Proc. Interspeech 2024, pages 1835--1839
2024
-
[15]
Vimal Manohar, Hossein Hadian, Daniel Povey, and Sanjeev Khudanpur. 2018. Semi-supervised training of acoustic models using lattice-free MMI . In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4844--4848. IEEE
2018
-
[16]
Andy Murphy, Liam Lonergan, Mengjie Qian, Harald Berthelsen, Christoph Wendler, Neasa Ní Chiaráin , Ailbhe Ní Chasaide , and Christer Gobl. 2023. Abair & Éist: a demonstration of speech technologies for Irish . In 2nd Annual Meeting of the ELRA/ISCA SIG on Under-resourced Lang...
2023
-
[17]
Daniel Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc Le. 2020. Improved noisy student training for automatic speech recognition. In Proc. Interspeech 2020, pages 2817--2821
2020
-
[18]
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2015. Specaugment: A simple data augmentation method for automatic speech recognition. In Proc. Interspeech 2015, pages 2613--2617
2015
-
[19]
Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2025. A time delay neural network architecture for efficient modeling of long temporal contexts. In Proc. Interspeech 2024, pages 3214--3218
2025
-
[20]
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Belgium, Brussels. Association for Computational Linguistics
2018
-
[21]
Vasile P a i s and Dan Tufi s . 2022. Capitalization and punctuation restoration: a survey. Artif. Intell. Rev., 55(3):1681--1722
2022
-
[22]
Mengjie Qian, Harald Berthelsen, Liam Lonergan, Andy Murphy, Claire O'Neill, Claire Ní Chiaráin, Christer Gobl, and Ailbhe Ní Chasaide. 2022. Automatic speech recognition for Irish : testing lexicons and language models. In Proceedings of the 33rd Irish Signals and Systems Con...
2022
-
[23]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492--28518. PMLR
2023
-
[24]
Peter Reichl, Sebastian Egger, Raimund Schatz, and Alessandro D'Alconzo. 2010. The logarithmic nature of QoE and the role of the W eber- F echner law in QoE assessment. In 2010 IEEE International Conference on Communications, pages 1--5. IEEE
2010
-
[25]
Team Silero. 2024. Silero VAD : pre-trained enterprise-grade V oice A ctivity D etector ( VAD ), N umber D etector and L anguage C lassifier. https://github.com/snakers4/silero-vad
2024
-
[26]
David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. 2018. X-vectors: Robust DNN embeddings for speaker recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5329--5333. IEEE
2018
-
[27]
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...
2022 arXiv
-
[28]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc
2017
-
[29]
Yu Xi, Wen Ding, Kai Yu, and Junjie Lai. 2024. Semi-supervised learning for code-switching asr with large language model filter. arXiv preprint arXiv:2407.04219
2024 arXiv
-
[30]
Hainan Xu, Tongfei Chen, Dongji Gao, Yiming Wang, Ke Li, Nagendra Goel, Yishay Carmiel, Daniel Povey, and Sanjeev Khudanpur. 2018. A pruned RNNLM lattice-rescoring algorithm for automatic speech recognition. In 2018 IEEE international conference on acoustics, speech and signal...
2018
-
[31]
Yu Zhang, James Qin, Daniel Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc Le, and Yonghui Wu. 2020. Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2010.10504
2020 arXiv
-
[32]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[33]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.