{"id":"cc60ea07-f430-44f6-901f-77f4b0f580f0","arxiv_id":"2501.00509","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An Irish-language transcription web service whose ASR is improved by semi-supervised learning on 3,230 hours of radio speech, and whose punctuation/capitalisation restoration uses a sequence-to-sequence transformer.","lead":"Fotheidil is a free web tool that converts Irish speech into readable, punctuated text using off-the-shelf speaker detection plus Irish-trained speech recognition and text-restoration models. The paper reports that adding 3,230 hours of unlabelled radio speech through semi-supervised learning cuts word errors on out-of-domain tests by up to 14 percent, and that a translation-style model restores capitalisation and punctuation better than a classifier.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SSL gains are confounded by longer training and extra data scale; the key comparison lacks a controlled ablation of epochs, SpecAug, and pseudo-label weighting.","rationale":"The reader's verdict is CONDITIONAL, and the main condition — the training-duration/epoch confound — is exactly the load-bearing concern. The paper does not ablate epochs, SpecAug, or pseudo-label weighting, so the SSL conclusion is not fully supported. However, the reader's conditional verdict already captures this: the concern is addressable and does not contradict the existence of a deployed, useful system. The additional HL overlap and small-test-set noise further justify the conditional status but do not warrant rejection, especially given the paper's honest statement about no ablations and the plausibility of SSL gains in low-resource settings. The proposed controlled ablation would settle the concern, so the stress-test pass agrees with the reader and leaves the verdict UNCHANGED.","tokens_in":12143,"tokens_out":1441,"duration_ms":13077,"concrete_test":"Train a controlled student model M1_sup with the exact M1 recipe (6 epochs, SpecAug, same AM structure, lexicon, and LM) but using only the 398 h supervised set, and evaluate on MG, SS, CV, FL, HL and the Ulster split of MG. If M1_sup achieves WERs within 2% relative of M1 on out-of-domain and HL sets, the SSL contribution is not established. Additionally, compute bootstrap or matched-pair confidence intervals for the M0 vs M1 differences on HL (198 utterances) and CV (516 utterances) to verify the reported reductions are not within noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that semi-supervised learning with 3,230 h of unlabelled radio data improves the acoustic model over the 398 h supervised baseline (Section 4.3.2 and Tables 1-2). The comparison compares M0 (TDNN, 4 epochs, standard augmentation) against M1 (TDNN, 6 epochs, SpecAug, and a training set that is 8x larger because of the pseudo-labelled data). If longer training on the same supervised data, or simply the greater total number of gradient steps / larger effective corpus, accounts for a substantial share of the WER drop, then the 'SSL helps' conclusion is overstated. The paper explicitly notes that training duration and noising are not ablated (Section 4.3.2), and the reader's weakest assumption correctly identifies this. Moreover, the HL test set (0.7 h, n=198 utterances) overlaps in speakers and content with the radio shows that generated the pseudo-labels, so part of the 27% relative improvement could be in-domain memorization or a self-confirming loop. The claim that SSL improves underrepresented Ulster dialect is also not controlled: the unsupervised set contains 1002 h of Barrscéalta (Ulster), so the improvement could be data exposure rather than a SSL-specific effect. A concrete check is to train M1_controlled with the same 6-epoch schedule and SpecAug on the 398 h supervised set only; if WER on CV/FL/HL is close to M1, the SSL conclusion collapses. Also report bootstrap confidence intervals for the small test sets (HL, CV).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Fotheidil, a web-based automatic transcription system for Irish that integrates voice activity detection, speaker diarisation, a Kaldi TDNN-HMM ASR model, and a capitalisation/punctuation restoration (C&PR) module. The core ASR experiment compares a 398-hour supervised baseline (M0) with a student model (M1) trained semi-supervisedly on the same supervised data plus 3,230 hours of pseudo-labelled Irish radio speech, reporting relative WER reductions of 9% on MíleGlór, 2% on SS, 14% on CommonVoice, 7% on Fleurs, and 27% on a hand-labelled radio test set (HL), as well as a 17% relative improvement for Ulster speakers on MíleGlór. The C&PR work compares a sequence-to-sequence transformer with a token-level classifier, showing that the S2S model roughly halves WERpc on rich-transcription targets. The paper also describes the system interface and deployment details.","tokens_in":12441,"tokens_out":7232,"duration_ms":66543,"significance":"If the results hold, this is a useful demonstration of semi-supervised learning for a low-resource, dialect-rich language, and the S2S C&PR approach is a practical contribution for Irish text processing. The paper is commendably concrete: it gives detailed data-split tables, test-set descriptions, and model configurations that would allow reproduction, and it openly discloses the overlap between the HL test set and the unsupervised radio domain. The central machine-learning claim, however, currently rests on a comparison that confounds the pseudo-labelled data with changes in training epochs and augmentation, so the magnitude of the SSL-specific benefit is not yet established.","major_comments":[{"comment":"The headline comparison between M0 and M1 confounds the addition of the 3,230-hour pseudo-labelled set with the change in training schedule (4 vs 6 epochs) and with the augmentation setup, since M1 is described as trained 'with SpecAug' but the exact relationship to M0's augmentation is not specified. The paper itself notes in §4.3.2 that training duration and noising are not ablated. To support the central claim that semi-supervised learning is the cause of the WER reductions, the authors should train a matched control model on the 398-hour supervised set alone using the same number of epochs, the same augmentation pipeline, and the same optimisation settings as M1; if that control already reaches M1's WERs on CommonVoice, Fleurs, and HL, then the SSL conclusion would not be supported. Reporting this ablation, or explicitly redefining the claim as a combined effect of extra data and longer training, is essential.","section":"§4.3.2, Tables 1–2"},{"comment":"The hand-labelled HL test set (0.7 hours, 198 utterances) is drawn from the same four radio programmes that generated the unsupervised pseudo-labelled data, with overlapping speakers and likely overlapping content. The 27% relative WER improvement on HL is therefore an in-domain, self-confirming measurement and should be presented separately from the out-of-domain gains rather than grouped with them in the abstract and conclusions. Moreover, because HL and CommonVoice are very small (0.7 h and 0.6 h, respectively), the reported differences may be within sampling noise; please provide bootstrap confidence intervals or significance tests for the WER differences in Tables 1 and 2.","section":"§4.3.1–4.3.3, Table 1"},{"comment":"The claim that semi-supervised learning improves the underrepresented Ulster dialect is not uniquely attributable to SSL: the unsupervised set contains 1,002 hours of Barrscéalta, an Ulster-dominant programme, so the 17% relative improvement for Ul speakers on MíleGlór could reflect additional exposure to Ulster speech in the pseudo-labelled data rather than a benefit of the semi-supervised mechanism per se. The controlled ablation requested in the first major comment would help to separate these effects; at minimum, the paper should acknowledge this conflation and soften the causal wording in the discussion.","section":"§4.3.3, Table 2"}],"minor_comments":[{"comment":"Please state explicitly whether M1 uses speed perturbation in addition to SpecAug, and give the SpecAug hyperparameters (e.g., time-mask width, frequency-mask width, number of masks) or a citation to the exact configuration used, so the two models can be compared.","section":"§4.3.2"},{"comment":"In the sentence introducing the evaluation datasets, 'MiléGlór' is a typo for 'MíleGlór'; please correct this throughout the paper.","section":"§4.4.1"},{"comment":"The word 'editting' in the dashboard description should be 'editing'.","section":"§3.1"},{"comment":"The citation for the TDNN architecture, 'Peddinti et al., 2025', lists the year 2025 but the venue as Interspeech 2024; the original paper appears to be Interspeech 2015. Please correct the year, page numbers, and venue, and ensure the in-text citation matches.","section":"References"},{"comment":"The validation metric 'CE-mean-words' is not defined; please explain what it measures or remove it if it is not needed for understanding the early-stopping procedure.","section":"§4.4.2"},{"comment":"The sentence 'The final WERs are always lower than the accumulated WERs of the ASR and the S2S, suggesting that the degradation in the input text generated by the ASR does not impact in the performance of the S2S system' is difficult to interpret; please clarify how the 'accumulated' error is defined and what the comparison is intended to show.","section":"Table 5, §4.4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid systems paper with a clear practical contribution, but the central ASR claim needs a controlled ablation before it can be accepted as evidence that SSL itself helps. The HL-overlap and epoch confounds are disclosed but should be factored into the presentation and conclusions. If the ablation supports the current conclusions, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this is a useful applied paper that actually ships something—a working Irish transcription web service—and it contains two empirical studies worth engaging with. The strongest result is the consistent WER reduction from noisy-student semi-supervised training on 3,230 hours of unlabelled radio speech: 14% relative on CommonVoice, 7% on Fleurs, 27% on the hand-labelled radio set, and a 17% relative gain for Ulster speakers on MíleGlór. The C&PR study also has a genuinely useful finding: a sequence-to-sequence transformer clearly beats a word-level classifier when the target is true rich transcription, because it can handle digits and acronyms rather than only casing and punctuation.\n\nWhat's new here is the integration, not the individual methods. The system is the first web-based transcription pipeline for Irish with VAD, diarisation, Kaldi ASR, and C&PR, and the paper is unusually transparent about data composition, including per-dialect hour counts. That's real credit.\n\nThe soft spot is the central SSL comparison. M0 is trained for 4 epochs on 398 hours; M1 is trained for 6 epochs on roughly eight times that much data per epoch because the pseudo-labelled set is added. Longer training and more gradient steps could be doing a lot of the work, and the paper doesn't ablate the epoch count on the supervised set alone. The stress-test's proposed control—train a 6-epoch SpecAug model on the 398h set only—is exactly the experiment needed. Until that's done, the claim that SSL itself, rather than just more compute and more data, drives the gains is not fully pinned down. This is a confound, but it's fixable, not fatal.\n\nAlso worth noting: the HL test set overlaps in speakers and content with the pseudo-label source, so the large relative gain there is partly a domain-match effect. The paper discloses this. The test sets are small and there are no confidence intervals; adding bootstrap CIs would be cheap.\n\nOverall this is a solid systems paper that deserves a serious referee. I'd send it out, but ask for the epoch ablation and confidence intervals before acceptance. The likely audience—speech researchers working on low-resource languages and anyone building community transcription tools—will get real value from it.","headline":"Useful Irish ASR systems paper with a real SSL story, but the headline comparison needs one controlled ablation before the SSL claim is solid.","tokens_in":12993,"tokens_out":3558,"would_cite":true,"duration_ms":33196,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents Fotheidil, the first web-based transcription system for Irish, and claims that pseudo-labelled radio data plus a sequence-to-sequence text-restoration stage deliver substantial accuracy gains for a low-resource language.","keywords":["Irish language","automatic speech recognition","semi-supervised learning","Noisy Student Training","capitalisation and punctuation restoration","sequence-to-sequence transformer","low-resource languages","Fotheidil"],"falsifier":"Train a model with the same architecture as the student on the 398-hour supervised set alone for 6 epochs with SpecAugment, decode the same five test sets, and compare word error rates with the student's numbers in Table 1; if the epoch-matched supervised model matches or beats the student, the observed gains cannot be attributed to the unlabelled data.","tokens_in":11962,"feed_emoji":"🎙️","tokens_out":10931,"duration_ms":92608,"temperature":0.7,"pith_summary":"The paper aims to establish that a web-based transcription system for Irish can be built from a modular speech recogniser plus a text-restoration stage, and that two design choices carry most of the gain. The first is semi-supervised learning: a teacher model trained on 398 hours of supervised Irish speech decodes 3,230 hours of unlabelled radio broadcasts, and a student model trained on the combined pseudo-labelled data improves relative word error rate by 14% on CommonVoice, 7% on Fleurs, 27% on a hand-labelled radio test set, and 17% for Ulster speakers on MíleGlór. The second is a sequence-to-sequence transformer for capitalisation and punctuation restoration, which roughly halves word error rate on rich transcription targets compared with a word-classifier baseline. The broader goal is to make speech-to-text accessible for a minority language with scarce transcribed data and marked dialect differences.","feed_headline":"Unlabelled Irish radio speech improves speech-to-text by up to 27%","feed_subtitle":"A transformer also restores capitals, punctuation and digits, making Irish transcription practical for public use.","key_machinery":"The load-bearing objects are two. First, a Noisy-Student-Teacher-style semi-supervised loop: a supervised time-delay neural network acoustic model (M0) acts as teacher, decoding an unlabelled 3,230-hour radio corpus into lattices that are rescored and converted to pseudo-labels; the student model (M1) is trained on supervised plus pseudo-labelled data with SpecAugment spectral masking, and the M0-versus-M1 comparison carries the semi-supervised claim. Second, a transformer-based sequence-to-sequence model maps lower-cased, punctuation-free ASR-style text to rich transcriptions in one pass, restoring capitals, punctuation, digits and acronyms; it is compared against a word-level classifier whose output classes include second- and third-letter capitalisation to handle Irish initial mutation.","core_discovery":"The paper establishes that a substantial pool of untranscribed broadcast speech can be converted into effective training material for an Irish speech recogniser. A teacher acoustic model trained on 398 hours of labelled speech decodes 3,230 hours of radio audio; the decoding lattices are rescored with a 5-gram language model, the best paths become pseudo-labels, and a student model is trained on the combined data with SpecAugment noise. The student is claimed to be better not only on in-domain test sets but especially on out-of-domain data and on Ulster speech, the dialect least represented in the supervised training set. Independently, the paper claims that treating capitalisation and punctuation restoration as a sequence-to-sequence task yields better rich transcriptions than word-by-word classification, and that the restoration model remains useful when applied to actual recognition output.","pith_inferences":["Editorial inference: if the unlabelled corpus were enlarged or the loop iterated with the student as the next teacher, the reported relative gains for Ulster and out-of-domain speech could grow further; the paper notes this as future work rather than a demonstrated result.","Editorial inference: the sequence-to-sequence restoration model's ability to output digits and acronyms suggests the same architecture could convert spoken-out units, currency or symbol names in dictation applications, a direction the paper mentions but does not test.","Editorial inference: because the student model trains for more epochs than the teacher, the observed ASR gain should be read as provisional until an epoch-matched supervised baseline is run; the paper's design leaves that comparison open."],"forward_implications":["If the semi-supervised result holds, untranscribed radio archives are a practical substitute for costly manual transcription when building speech recognisers for under-resourced languages.","Because Ulster speech improved most, semi-supervised learning appears to counteract dialect bias when the unlabelled data is rich in the underrepresented dialect.","RNNLM rescoring stacks with the improved acoustic model, so language-model and acoustic-model gains are complementary rather than redundant.","The sequence-to-sequence C&PR model keeps rich-transcription word error below 9% on all test sets except CommonVoice, whose short fragmentary sentences form a distinct hard case.","User corrections made through the Fotheidil interface can be fed back into the training data, supporting a community-driven improvement loop."],"supporting_citations":[{"why":"Supplies the lattice-based semi-supervised training recipe (undeterminised decoding, rescoring, pseudo-labels) that the paper adapts for Irish.","marker":"Manohar et al. (2018)"},{"why":"Defines SpecAugment, the noise applied when training the student acoustic model.","marker":"Park et al. (2015)"},{"why":"Refines Noisy Student Training for ASR and motivates the student-noise design used here.","marker":"Park et al. (2020)"},{"why":"Reports the Noisy Student Training results that frame the paper's expectation that unlabelled data helps ASR.","marker":"Zhang et al. (2020)"},{"why":"Supplies the alignment protocol that prepares several supervised Irish corpora and the low-resource modular-ASR argument.","marker":"Lonergan et al. (2024)"},{"why":"Provides the CommonVoice Irish test set used as an out-of-domain evaluation.","marker":"Ardila et al. (2020)"},{"why":"Provides the Fleurs-R Irish test set used as a second out-of-domain evaluation.","marker":"Ma et al. (2024)"},{"why":"Defines the transformer architecture on which the sequence-to-sequence capitalisation and punctuation model is built.","marker":"Vaswani et al. (2017)"},{"why":"Provides the implementation used to train the sequence-to-sequence restoration model.","marker":"Junczys-Dowmunt et al. (2018)"},{"why":"Provides the normaliser that creates the normalised rich-transcript targets for the C&PR comparison.","marker":"Murphy et al. (2023)"}],"fun_headline_variants":["3,230 hours of unlabelled radio refine Irish speech recognition","Semi-supervised learning sharpens Irish ASR, especially for Ulster","Seq2seq wins for Irish capitalization and punctuation restoration","First web-based Irish transcription service leverages AI for speech and text","Teacher-student learning on radio improves out-of-domain Irish ASR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison that carries the paper attributes the student model's improvement to the 3,230 hours of pseudo-labelled radio data, but the student is also trained for more epochs (6 versus 4) with the same noising scheme, so if longer training alone on the 398-hour supervised set produced the same gains, the semi-supervised claim would not be established.","fun_headline_variants_meta":{"raw":{"variants":["3,230 hours of unlabelled radio refine Irish speech recognition","Semi-supervised learning sharpens Irish ASR, especially for Ulster","Seq2seq wins for Irish capitalization and punctuation restoration","First web-based Irish transcription service leverages AI for speech and text","Teacher-student learning on radio improves out-of-domain Irish ASR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000846,"raw_usage":{"total_tokens":3658,"prompt_tokens":897,"completion_tokens":2761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":2673}},"tokens_in":513,"tokens_out":2761,"duration_ms":20574,"temperature":1.0,"reasoning_tokens":2673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:48:46.372019+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model with the same architecture as the student on the 398-hour supervised set alone for 6 epochs with SpecAugment, decode the same five test sets, and compare word error rates with the student's numbers in Table 1; if the epoch-matched supervised model matches or beats the student, the observed gains cannot be attributed to the unlabelled data.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the lattice-based semi-supervised training recipe (undeterminised decoding, rescoring, pseudo-labels) that the paper adapts for Irish."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SpecAugment, the noise applied when training the student acoustic model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the alignment protocol that prepares several supervised Irish corpora and the low-resource modular-ASR argument."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CommonVoice Irish test set used as an out-of-domain evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the implementation used to train the sequence-to-sequence restoration model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the normaliser that creates the normalised rich-transcript targets for the C&PR comparison."}],"review_version":1}