REVIEW 4 major objections 4 minor 28 references
A Modular Part-of-Speech Tagger for Scottish Gaelic using spaCy
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A default spaCy pipeline, trained only on ARCOSG with no external embeddings, tags Scottish Gaelic at 88.6% fine-grained and 93.7% coarse-grained accuracy, matching the two specialised taggers published before it.
desk verdict A solid, honest baseline paper for Scottish Gaelic POS tagging whose main comparison claim is undermined by an internal tagset-size inconsistency; worth reviewing after a fix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is spaCy's Tok2Vec layer, a context-aware token encoder that builds numerical representations from local features such as word shape, spelling, and normalised form, together with spaCy's tagger component that maps these representations to PoS labels. This architecture lets the model learn grammatical information from subword and orthographic cues, so no stemmer, lemmatiser, or handwritten morphological rules are needed. The other essential pieces are the ARCOSG corpus itself, the fine-/coarse-grained tagset split, and the minimal Gaelic tokeniser exceptions from the spaCy language module, which keep tokenisation consistent with the corpus annotations.
What would settle it
Run the released pipeline on the exact held-out ARCOSG split and tag mapping used by the 2020 tagger, scoring every token including the ~0.4% currently labelled 'misaligned'; if fine-grained accuracy falls well below 90.7% or coarse accuracy below 94.7%, the claim of comparability fails.
Extended reading notes
Core claim
The central claim is that a spaCy-based tagger trained only on gold-standard ARCOSG labels, with minimal configuration and no external embeddings, achieves 88.6% accuracy on the full 215-tag tagset and 93.7% on the reduced 40-tag tagset. The paper presents these results as comparable to the 84% (fine) and 92% (coarse) reported in the 2016 evaluation and the 90.7% (fine) and 94.7% (coarse) of the 2020 Gaelic Linguistic Analyser, despite using far simpler, modular components. The contribution is positioned as the first spaCy-based Gàidhlig tagger and a reproducible baseline, with code released publicly.
Load-bearing premise
The comparability claim assumes that tagging accuracy measured across different tagset sizes (215 vs 246 fine-grained tags), different tokenisation conventions, and partially different training data can be meaningfully compared; if those settings are not commensurable, the central conclusion weakens.
Editorial extensions
If this is right
- Scottish Gaelic gains a publicly available, reproducible POS-tagging baseline inside the spaCy ecosystem that downstream tools can build on.
- The 5.1-point gap between the coarse and fine taggers suggests that most of the remaining error is tied to rare, fine-grained grammatical distinctions rather than to the pipeline's architecture.
- The same minimal recipe—corpus plus default spaCy pipeline, no external embeddings—may produce usable first taggers for other low-resource, morphologically complex languages.
- Future spaCy components such as a lemmatiser, dependency parser, or named-entity recogniser can be added incrementally to this pipeline, avoiding a from-scratch redesign.
Reading between the lines
- The comparability claim is sensitive to evaluation details that the paper reports inconsistently: the fine-grained tagset is given as 215 tags in Table 1 and 246 in Table 2, and one earlier tagger used additional training data, so a strict head-to-head on identical tags and splits could change the size of the reported gap.
- Because the pipeline uses no external embeddings, the 88.6% figure suggests ARCOSG alone carries enough morphological signal; adding self-supervised pre-training on unlabelled Gaelic text is a natural test of whether more data or a better architecture raises the ceiling.
- A confusion-matrix analysis of the fine-grained model would probably show errors concentrated among closely related rare tags, which could guide both annotation-scheme simplification and targeted data collection.
- Evaluating the same pipeline on a standard tagset such as Universal Dependencies would tell whether this lightweight approach transfers to other morphologically rich languages or to cross-lingual settings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a modular part-of-speech tagging pipeline for Scottish Gaelic built with the spaCy framework and trained on the Annotated Reference Corpus of Scottish Gaelic (ARCOSG). Two models are presented, one using a fine-grained tagset and one using a coarse-grained tagset, both trained with minimal preprocessing, no external embeddings, and largely default hyperparameters. The fine-grained model is reported to achieve 88.6% tagging accuracy and the coarse-grained model 93.7%, and the authors claim these results are comparable to the two previously published Gaelic taggers (Lamb et al. and Boizou et al.). The paper contributes a reproducible, lightweight baseline for a low-resource morphologically rich language and makes the code publicly available.
Significance. If the headline results were fully supported, the paper would provide a valuable and reproducible baseline for Scottish Gaelic POS tagging, showing that a general-purpose, off-the-shelf pipeline can perform competitively with more heavily engineered systems. The authors are transparent about some limitations, including a spaCy warning about the small training set and a ~0.4% token misalignment rate, and they release their code. However, the central 'comparable to previous taggers' claim is not currently established because of an internal inconsistency in the reported fine-grained tagset size (215 vs 246), an unexplained difference in training token counts between the two models, and the absence of any uncertainty quantification or significance testing. These issues must be resolved before the comparability claim can be accepted.
major comments (4)
- [Table 1, Table 2, Section 4.2, Section 8] There is a direct internal contradiction in the fine-grained tagset size. Table 1 and Sections 5 and 8 report 215 tags for the fine-grained model, while Table 2 labels the same model as '246 tags', and Section 4.2 describes the fine-grained tagset as '242 or 246 items'. The manuscript never explains where the 215-tag count comes from or whether tags were merged or excluded. Because the previous taggers used 246 tags and Boizou et al. additionally trained on data beyond ARCOSG, the accuracy of 88.6% is not directly comparable to the 84% and 90.7% reported for those systems unless the tagset definitions and training conditions match. The authors must clarify the exact tagset used, reconcile the numbers across tables and text, and either re-run the comparison under matched settings or explicitly qualify the comparability claim.
- [Section 5] The paper reports that about 0.4% of tokens are labelled 'misaligned' but does not state whether these tokens were excluded from the accuracy calculation. If they were excluded, the reported accuracies are computed on a subset of the test set; if they were included, tokenisation mismatches could have introduced spurious errors. The authors should explicitly describe the treatment of misaligned tokens and, ideally, report accuracy both with and without them to show the practical impact of the 0.4%.
- [Table 1, Section 5] Table 1 reports different training token counts for the two models (70,890 for the fine-grained model and 69,522 for the coarse-grained model) and different vocabulary sizes (7,160 vs 7,114), even though both models are said to be trained on the same ARCOSG corpus. The paper does not explain why the counts differ. If the coarse-grained model was trained on a subset of tokens because some fine-grained tags could not be mapped to the coarse tagset, the evaluation sets may also differ, which would confound the comparison between the two models and with prior work. The authors should explain the source of the token count discrepancy and confirm that both models were evaluated on the same test sentences.
- [Table 2, Section 6] The claim that the results are 'comparable' to previous taggers rests on a single 80/10/10 split with no confidence intervals, multiple seeds, or significance tests. Given the limited corpus size, differences such as 88.6% vs 84% (Lamb et al.) or 93.7% vs 94.7% (Boizou et al.) may be within the range of random variation. To support the headline claim, the authors should provide at least bootstrap confidence intervals or results over multiple splits, and, if feasible, a statistical comparison with the published numbers under matched evaluation settings.
minor comments (4)
- [Section 4.2] The citation [20] for 'prior studies' points to Lamb and Sinclair (2016) on word embedding models, which is not a POS tagging study; the intended reference is likely [18] or [19]. Please correct the citation.
- [Section 4.2] The phrase 'fine-grained (242 or 246 items)' and 'coarse-grained (40 or 41 items)' is ambiguous; the authors should give the exact tagset sizes and reconcile them with the numbers reported in Table 1.
- [Section 1] There is a typo in the sentence 'We now provide now a third tagger'; the word 'now' appears twice.
- [Abstract] The abstract uses 'Gaidhlig' without the accent in the first sentence, while the rest of the paper uses 'Gàidhlig'; please make the spelling consistent.
Circularity Check
No significant circularity: all reported accuracies are measured on a held-out ARCOSG split against published external benchmarks, and the sole self-citation is background material that does not support any central claim.
full rationale
The paper's derivation chain is a straightforward supervised-training experiment: it builds a spaCy tokeniser/Tok2Vec/tagger pipeline, trains it on an ARCOSG training split, and measures tagging accuracy on a held-out 10% test split. The headline numbers, 88.6% and 93.7%, are empirical observations from that held-out evaluation, not fitted parameters renamed as predictions, and no equation or definition in the paper makes a claimed result equal to an input by construction. The comparative claim against Lamb et al. and Boizou et al. rests on previously published accuracy figures from other work; although reference [19] shares a co-author with the present paper, those numbers are external, published results and are not derived from the present model, so this is not load-bearing self-citation. The only self-citation is [23], Barclay's morphology paper, cited in Section 2.1 merely as a background overview of Gaelic status and history; dropping it would not affect any experimental outcome. The paper itself also notes a spaCy warning about insufficient training examples in Section 5, which concerns model adequacy rather than circularity. The internal inconsistencies flagged by the reader, such as 215 tags in Table 1 versus 246 tags in Table 2 for the same fine-grained model, and the unreported treatment of 0.4% misaligned tokens, are correctness and reproducibility concerns about whether the comparison is commensurable, but they do not make the derivation circular. Accordingly, no circular step is present and the score is minimal.
Assumptions & free parameters
free parameters (3)
- spaCy Tok2Vec/tagger hyperparameters =
not reported, defaults per spaCy version
- Auxiliary punctuation tokenisation rules =
hand-crafted, not specified in detail
- 80/10/10 split seed =
not reported
assumptions (3)
- domain assumption ARCOSG annotations are gold-standard and consistent with spaCy tokenization
- domain assumption A single 10% random test split is representative of the corpus
- domain assumption Default spaCy tokenizer rules plus the Gaelic language module's exceptions give valid tokenisation for Gaelic text
Cite this review
Pith. "Pith review of A Modular Part-of-Speech Tagger for Scottish Gaelic using spaCy." pith.science (2026). https://pith.science/paper/J5HIGFTE
@misc{pith2026260804808,
author = {Pith},
title = {Pith review of: A Modular Part-of-Speech Tagger for Scottish Gaelic using spaCy},
year = {2026},
howpublished = {\url{https://pith.science/paper/J5HIGFTE}},
note = {Machine review of arXiv:2608.04808}
}
read the original abstract
Part-of-speech tagging for low-resource languages remains challenging due to limited annotated data, especially for linguistically complex languages. Gaidhlig (Scottish Gaelic) is a morphologically rich and endangered language with limited digital resources, making it suitable for examining a lightweight language processing approach. This paper describes using the modular spaCy Natural Language Processing framework to build part-of-speech taggers for Gaidhlig using the Annotated Reference Corpus of Scottish Gaelic. We train two models with minimal pre-processing and configuration: one using a fine-grained tagset and another using a reduced coarse-grained tagset. Both models are trained without external embeddings or pre-trained language models, using only supervised learning from the available corpus. The fine-grained model achieves 88.6% tagging accuracy, while the coarse-grained model achieves 93.7%. The results are comparable to those of the two previously published Gaidhlig taggers, indicating that simple, off-the-shelf language processing pipelines can demonstrate good performance in low-resource and morphologically complex linguistic settings.
Figures
Reference graph
Works this paper leans on
-
[1]
Universal dependencies for Scottish Gaelic: syntax
Colin Batchelor. Universal dependencies for Scottish Gaelic: syntax. InProceedings of the Celtic Language Technology Workshop, pages 7–15, 2019
work page 2019
-
[2]
An online linguistic analyser for Scottish Gaelic
Lo ¨ ıc Boizou and William Lamb. An online linguistic analyser for Scottish Gaelic. InHuman Language Technologies – The Baltic Perspective, pages 119–122. IOS Press, 2020
work page 2020
-
[3]
Ciprian Chelba.Statistical Language Modeling, pages 74–104. Wiley-Blackwell, John Wiley & Sons Ltd, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ United Kingdom, 2010. Parts of this chapter appeared in Computer Speech & Language, vol. 14, no. 4, pages 283-332, October 2000, and are used with the permission of Elsevier Limited
work page 2010
-
[4]
Alebachew Chiche and Betselot Yitagesu. Part of speech tagging: a systematic review of deep learning and machine learning approaches.Journal of Big Data, 9(1):10, 2022
work page 2022
-
[5]
Gaelic lemmatizer (spaCy-compatible NLP pipeline)
Scottish Gaelic Collab. Gaelic lemmatizer (spaCy-compatible NLP pipeline). https: //github.com/Scottish-Gaelic-Collab/lemmatizer gaelic, 2025. Accessed: 2026-06- 11
work page 2025
-
[6]
Developing an automatic part-of-speech tagger for Scottish Gaelic
Samuel Danso and William Lamb. Developing an automatic part-of-speech tagger for Scottish Gaelic. In John Judge, Teresa Lynn, Monica Ward, and Ois ´ ın Ruair ´ ı ´O Brian, editors,Proceedings of the Celtic Technology Workshop (CLTW 2014): A Workshop of the 25th International Conference on Computational Linguistics (COLING 2014), pages 1–5, Dublin, Ireland...
work page 2014
-
[7]
Developing au- tomatic speech recognition for Scottish Gaelic
Lucy Evans, William Lamb, Mark Sinclair, and Beatrice Alex. Developing au- tomatic speech recognition for Scottish Gaelic. InProceedings of the 4th Celtic language technology workshop within LREC2022, pages 110–120, 2022
work page 2022
-
[8]
Hedderich, Lukas Lange, Heike Adel, Jannik Str¨ otgen, and Diet- rich Klakow
Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Str¨ otgen, and Diet- rich Klakow. A survey on recent approaches for natural language processing in low-resource scenarios. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors,Proceedings ...
work page 2021
Show all 28 references
-
[9]
spaCy: Industrial-strength natural language processing in Python, 2020
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength natural language processing in Python, 2020. https: //doi.org/10.5281/zenodo.1212303
2020 doi
-
[10]
Martin.Speech and Language Processing: An In- troduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models
Daniel Jurafsky and James H. Martin.Speech and Language Processing: An In- troduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models. Stanford University (online manuscript re- leased January 6, 2026), 3rd edition, 2026....
2026
-
[11]
Language shift, bilingualism and the future of Britain’s Celtic languages.Philosophical Transactions of the Royal Society B: Biological Sciences, 365(1559):3855–3864, 2010
Anne Kandler, Roman Unger, and James Steele. Language shift, bilingualism and the future of Britain’s Celtic languages.Philosophical Transactions of the Royal Society B: Biological Sciences, 365(1559):3855–3864, 2010
2010
-
[12]
Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. https://arxiv.org/abs/2001.08361
2020 arXiv
-
[13]
A morphology-driven approach to NLP for a low-resource, highly complex language.Vietnam Journal of Computer Science, 1:19, 2025
Irakli Kardava. A morphology-driven approach to NLP for a low-resource, highly complex language.Vietnam Journal of Computer Science, 1:19, 2025. 12 P´ eter Stef´ an et al
2025
-
[14]
A practitioner’s guide to build- ing ASR models for low-resource languages: A case study on Scottish Gaelic
Ondˇ rej Klejch, William Lamb, and Peter Bell. A practitioner’s guide to build- ing ASR models for low-resource languages: A case study on Scottish Gaelic. In Proceedings of Interspeech 2025, pages 728–732, 2025
2025
-
[15]
Language death in the digital age.Language, 1(8), 2012
Andr´ as Kornai. Language death in the digital age.Language, 1(8), 2012
2012
-
[16]
Routledge, 2024
William Lamb.Scottish Gaelic: A comprehensive grammar. Routledge, 2024
2024
-
[17]
An- notated reference corpus of Scottish Gaelic (ARCOSG), 1997–2020, 2020
William Lamb, Sharon Arbuthnot, Susanna Naismith, and Samuel Danso. An- notated reference corpus of Scottish Gaelic (ARCOSG), 1997–2020, 2020. https: //github.com/Gaelic-Algorithmic-Research-Group/ARCOSG
1997
-
[18]
Developing an automatic part-of-speech tag- ger for Scottish Gaelic
William Lamb and Samuel Danso. Developing an automatic part-of-speech tag- ger for Scottish Gaelic. InProceedings of the First Celtic Language Technology Workshop, pages 1–5, 2014
2014
-
[19]
Evaluating a Gaelic Part-of- Speech Tagger and Reference Corpus
William Lamb, Samuel Danso, and Alistair Lawson. Evaluating a Gaelic Part-of- Speech Tagger and Reference Corpus. Technical report, University of Edinburgh, Edinburgh, Scotland, 2016
2016
-
[20]
Developing Word Embedding Models for Scot- tish Gaelic
William Lamb and Mark Sinclair. Developing Word Embedding Models for Scot- tish Gaelic. InActes de la conf´ erence conjointe JEP-TALN-RECITAL, volume 6, pages 31–41, 2016
2016
-
[21]
Heuristic sample selection to minimize reference standard training set for a part-of-speech tagger.Journal of the American Medical Informatics Association, 14(5):641–650, 2007
Kaihong Liu, Wendy Chapman, Rebecca Hwa, and Rebecca S Crowley. Heuristic sample selection to minimize reference standard training set for a part-of-speech tagger.Journal of the American Medical Informatics Association, 14(5):641–650, 2007
2007
-
[22]
Low-resource lan- guages: A review of past work and future challenges, 2020
Alexandre Magueresse, Vincent Carles, and Evan Heetderks. Low-resource lan- guages: A review of past work and future challenges, 2020. https://arxiv.org/abs/ 2006.07264
2020 arXiv
-
[23]
A Rule-Based Computational Model for G` aidhlig Morphology
Peter J Barclay. A Rule-Based Computational Model for G` aidhlig Morphology. In Proceedings of the 18th International Conference on Agents and Artificial Intelli- gence, volume 1, pages 683–693. SCITEPRESS – Science and Technology Publica- tions, Lda., March 2026
2026
-
[24]
PhD thesis, University of Glasgow, 2016
Susan Ross.The Standardisation of Scottish Gaelic Orthography 1750–2007. PhD thesis, University of Glasgow, 2016. https://theses.gla.ac.uk/7403/
2007
-
[25]
Data augmentation via dependency tree morphing for low-resource languages
G¨ ozde G¨ ul S ¸ahin and Mark Steedman. Data augmentation via dependency tree morphing for low-resource languages. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5004–5009, 2018
2018
-
[26]
Parsing mor- phologically rich languages: Introduction to the special issue.Computational lin- guistics, 39(1):15–22, 2013
Reut Tsarfaty, Djam´ e Seddah, Sandra K¨ ubler, and Joakim Nivre. Parsing mor- phologically rich languages: Introduction to the special issue.Computational lin- guistics, 39(1):15–22, 2013
2013
-
[27]
Lipton, and Yulia Tsvetkov
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual models: Findings and a meta-learning treatment. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Languag...
2020
-
[28]
Analyzing the understanding of morphologically complex words in large language models
Marion Weller-Di Marco and Alexander Fraser. Analyzing the understanding of morphologically complex words in large language models. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Re- sources and Evaluation (LREC-COLING 2024), pa...
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.