Pith. sign in

REVIEW 4 major objections 4 minor 28 references

A Modular Part-of-Speech Tagger for Scottish Gaelic using spaCy

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A default spaCy pipeline, trained only on ARCOSG with no external embeddings, tags Scottish Gaelic at 88.6% fine-grained and 93.7% coarse-grained accuracy, matching the two specialised taggers published before it.

desk verdict A solid, honest baseline paper for Scottish Gaelic POS tagging whose main comparison claim is undermined by an internal tagset-size inconsistency; worth reviewing after a fix. read the letter →

arxiv 2608.04808 v1 pith:J5HIGFTE submitted 2026-08-05 cs.CL

classification cs.CL
keywords low-resourcelanguagespart-of-speechtaggingScottishGaelicGàidhligspaCyARCOSGmorphologicallyrichTok2Vec
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a general-purpose, off-the-shelf NLP pipeline can serve as a competitive part-of-speech tagger for Scottish Gaelic, a morphologically rich and endangered language with a small annotated corpus. Two spaCy models are trained directly on the ARCOSG corpus, one with a 215-tag fine-grained tagset and one with a 40-tag coarse tagset, using no pre-trained embeddings and almost no preprocessing. The fine-grained model reaches 88.6% tagging accuracy and the coarse-grained model 93.7%. The authors argue these numbers are comparable to the two previously published Gàidhlig taggers, and that this provides a reproducible baseline for future low-resource NLP work on the language.

What carries the argument

The load-bearing component is spaCy's Tok2Vec layer, a context-aware token encoder that builds numerical representations from local features such as word shape, spelling, and normalised form, together with spaCy's tagger component that maps these representations to PoS labels. This architecture lets the model learn grammatical information from subword and orthographic cues, so no stemmer, lemmatiser, or handwritten morphological rules are needed. The other essential pieces are the ARCOSG corpus itself, the fine-/coarse-grained tagset split, and the minimal Gaelic tokeniser exceptions from the spaCy language module, which keep tokenisation consistent with the corpus annotations.

What would settle it

Run the released pipeline on the exact held-out ARCOSG split and tag mapping used by the 2020 tagger, scoring every token including the ~0.4% currently labelled 'misaligned'; if fine-grained accuracy falls well below 90.7% or coarse accuracy below 94.7%, the claim of comparability fails.

Watch

Extended reading notes

Core claim

The central claim is that a spaCy-based tagger trained only on gold-standard ARCOSG labels, with minimal configuration and no external embeddings, achieves 88.6% accuracy on the full 215-tag tagset and 93.7% on the reduced 40-tag tagset. The paper presents these results as comparable to the 84% (fine) and 92% (coarse) reported in the 2016 evaluation and the 90.7% (fine) and 94.7% (coarse) of the 2020 Gaelic Linguistic Analyser, despite using far simpler, modular components. The contribution is positioned as the first spaCy-based Gàidhlig tagger and a reproducible baseline, with code released publicly.

Load-bearing premise

The comparability claim assumes that tagging accuracy measured across different tagset sizes (215 vs 246 fine-grained tags), different tokenisation conventions, and partially different training data can be meaningfully compared; if those settings are not commensurable, the central conclusion weakens.

Editorial extensions

If this is right

  • Scottish Gaelic gains a publicly available, reproducible POS-tagging baseline inside the spaCy ecosystem that downstream tools can build on.
  • The 5.1-point gap between the coarse and fine taggers suggests that most of the remaining error is tied to rare, fine-grained grammatical distinctions rather than to the pipeline's architecture.
  • The same minimal recipe—corpus plus default spaCy pipeline, no external embeddings—may produce usable first taggers for other low-resource, morphologically complex languages.
  • Future spaCy components such as a lemmatiser, dependency parser, or named-entity recogniser can be added incrementally to this pipeline, avoiding a from-scratch redesign.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparability claim is sensitive to evaluation details that the paper reports inconsistently: the fine-grained tagset is given as 215 tags in Table 1 and 246 in Table 2, and one earlier tagger used additional training data, so a strict head-to-head on identical tags and splits could change the size of the reported gap.
  • Because the pipeline uses no external embeddings, the 88.6% figure suggests ARCOSG alone carries enough morphological signal; adding self-supervised pre-training on unlabelled Gaelic text is a natural test of whether more data or a better architecture raises the ceiling.
  • A confusion-matrix analysis of the fine-grained model would probably show errors concentrated among closely related rare tags, which could guide both annotation-scheme simplification and targeted data collection.
  • Evaluating the same pipeline on a standard tagset such as Universal Dependencies would tell whether this lightweight approach transfers to other morphologically rich languages or to cross-lingual settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper describes a modular part-of-speech tagging pipeline for Scottish Gaelic built with the spaCy framework and trained on the Annotated Reference Corpus of Scottish Gaelic (ARCOSG). Two models are presented, one using a fine-grained tagset and one using a coarse-grained tagset, both trained with minimal preprocessing, no external embeddings, and largely default hyperparameters. The fine-grained model is reported to achieve 88.6% tagging accuracy and the coarse-grained model 93.7%, and the authors claim these results are comparable to the two previously published Gaelic taggers (Lamb et al. and Boizou et al.). The paper contributes a reproducible, lightweight baseline for a low-resource morphologically rich language and makes the code publicly available.

Significance. If the headline results were fully supported, the paper would provide a valuable and reproducible baseline for Scottish Gaelic POS tagging, showing that a general-purpose, off-the-shelf pipeline can perform competitively with more heavily engineered systems. The authors are transparent about some limitations, including a spaCy warning about the small training set and a ~0.4% token misalignment rate, and they release their code. However, the central 'comparable to previous taggers' claim is not currently established because of an internal inconsistency in the reported fine-grained tagset size (215 vs 246), an unexplained difference in training token counts between the two models, and the absence of any uncertainty quantification or significance testing. These issues must be resolved before the comparability claim can be accepted.

major comments (4)
  1. [Table 1, Table 2, Section 4.2, Section 8] There is a direct internal contradiction in the fine-grained tagset size. Table 1 and Sections 5 and 8 report 215 tags for the fine-grained model, while Table 2 labels the same model as '246 tags', and Section 4.2 describes the fine-grained tagset as '242 or 246 items'. The manuscript never explains where the 215-tag count comes from or whether tags were merged or excluded. Because the previous taggers used 246 tags and Boizou et al. additionally trained on data beyond ARCOSG, the accuracy of 88.6% is not directly comparable to the 84% and 90.7% reported for those systems unless the tagset definitions and training conditions match. The authors must clarify the exact tagset used, reconcile the numbers across tables and text, and either re-run the comparison under matched settings or explicitly qualify the comparability claim.
  2. [Section 5] The paper reports that about 0.4% of tokens are labelled 'misaligned' but does not state whether these tokens were excluded from the accuracy calculation. If they were excluded, the reported accuracies are computed on a subset of the test set; if they were included, tokenisation mismatches could have introduced spurious errors. The authors should explicitly describe the treatment of misaligned tokens and, ideally, report accuracy both with and without them to show the practical impact of the 0.4%.
  3. [Table 1, Section 5] Table 1 reports different training token counts for the two models (70,890 for the fine-grained model and 69,522 for the coarse-grained model) and different vocabulary sizes (7,160 vs 7,114), even though both models are said to be trained on the same ARCOSG corpus. The paper does not explain why the counts differ. If the coarse-grained model was trained on a subset of tokens because some fine-grained tags could not be mapped to the coarse tagset, the evaluation sets may also differ, which would confound the comparison between the two models and with prior work. The authors should explain the source of the token count discrepancy and confirm that both models were evaluated on the same test sentences.
  4. [Table 2, Section 6] The claim that the results are 'comparable' to previous taggers rests on a single 80/10/10 split with no confidence intervals, multiple seeds, or significance tests. Given the limited corpus size, differences such as 88.6% vs 84% (Lamb et al.) or 93.7% vs 94.7% (Boizou et al.) may be within the range of random variation. To support the headline claim, the authors should provide at least bootstrap confidence intervals or results over multiple splits, and, if feasible, a statistical comparison with the published numbers under matched evaluation settings.
minor comments (4)
  1. [Section 4.2] The citation [20] for 'prior studies' points to Lamb and Sinclair (2016) on word embedding models, which is not a POS tagging study; the intended reference is likely [18] or [19]. Please correct the citation.
  2. [Section 4.2] The phrase 'fine-grained (242 or 246 items)' and 'coarse-grained (40 or 41 items)' is ambiguous; the authors should give the exact tagset sizes and reconcile them with the numbers reported in Table 1.
  3. [Section 1] There is a typo in the sentence 'We now provide now a third tagger'; the word 'now' appears twice.
  4. [Abstract] The abstract uses 'Gaidhlig' without the accent in the first sentence, while the rest of the paper uses 'Gàidhlig'; please make the spelling consistent.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: all reported accuracies are measured on a held-out ARCOSG split against published external benchmarks, and the sole self-citation is background material that does not support any central claim.

full rationale

The paper's derivation chain is a straightforward supervised-training experiment: it builds a spaCy tokeniser/Tok2Vec/tagger pipeline, trains it on an ARCOSG training split, and measures tagging accuracy on a held-out 10% test split. The headline numbers, 88.6% and 93.7%, are empirical observations from that held-out evaluation, not fitted parameters renamed as predictions, and no equation or definition in the paper makes a claimed result equal to an input by construction. The comparative claim against Lamb et al. and Boizou et al. rests on previously published accuracy figures from other work; although reference [19] shares a co-author with the present paper, those numbers are external, published results and are not derived from the present model, so this is not load-bearing self-citation. The only self-citation is [23], Barclay's morphology paper, cited in Section 2.1 merely as a background overview of Gaelic status and history; dropping it would not affect any experimental outcome. The paper itself also notes a spaCy warning about insufficient training examples in Section 5, which concerns model adequacy rather than circularity. The internal inconsistencies flagged by the reader, such as 215 tags in Table 1 versus 246 tags in Table 2 for the same fine-grained model, and the unreported treatment of 0.4% misaligned tokens, are correctness and reproducibility concerns about whether the comparison is commensurable, but they do not make the derivation circular. Accordingly, no circular step is present and the score is minimal.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

This is an empirical benchmark paper, so the ledger is dominated by data and configuration assumptions rather than invented entities. The central claim rests on ARCOSG being a correct gold standard, on spaCy's default tokenization being compatible with ARCOSG's annotation conventions (the authors report about 0.4% misalignment), and on a single unseeded 80/10/10 split being representative. The unenumerated spaCy configuration, the hand-written punctuation rules, and the missing split seed are the free parameters; no new theoretical entities are introduced.

free parameters (3)
  • spaCy Tok2Vec/tagger hyperparameters = not reported, defaults per spaCy version
    The paper states minimal tuning and default settings in most cases (Sections 4.1 and 6) but never enumerates architecture width, training iterations, dropout, or learning rate, so the exact configuration is an unstated degree of freedom for reproduction.
  • Auxiliary punctuation tokenisation rules = hand-crafted, not specified in detail
    Section 4.1 says the pipeline used 'some auxiliary rules for punctuation handling required to ensure compatibility with the annotation conventions used in ARCOSG'; these rules are a manual intervention that is not enumerated.
  • 80/10/10 split seed = not reported
    The single random split in Section 4.2 is not seeded or stratified, so the reported accuracies carry unknown variance across different splits of the corpus.
assumptions (3)
  • domain assumption ARCOSG annotations are gold-standard and consistent with spaCy tokenization
    The whole evaluation (Sections 4.2 and 5) treats ARCOSG labels as ground truth; the paper itself notes about 0.4% of tokens are 'misaligned', so the fit between annotation conventions and spaCy's tokenizer is imperfect.
  • domain assumption A single 10% random test split is representative of the corpus
    Section 5 evaluates on one 10% sample with no confidence interval, so the 88.6% and 93.7% figures are point estimates on one split.
  • domain assumption Default spaCy tokenizer rules plus the Gaelic language module's exceptions give valid tokenisation for Gaelic text
    Section 4.1 relies on the Cornell College Gaelic lang module [5] and shared default rules; no evaluation of tokenisation quality is given beyond the 0.4% misalignment note.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Modular Part-of-Speech Tagger for Scottish Gaelic using spaCy." pith.science (2026). https://pith.science/paper/J5HIGFTE

@misc{pith2026260804808,
  author       = {Pith},
  title        = {Pith review of: A Modular Part-of-Speech Tagger for Scottish Gaelic using spaCy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5HIGFTE}},
  note         = {Machine review of arXiv:2608.04808}
}
read the original abstract

Part-of-speech tagging for low-resource languages remains challenging due to limited annotated data, especially for linguistically complex languages. Gaidhlig (Scottish Gaelic) is a morphologically rich and endangered language with limited digital resources, making it suitable for examining a lightweight language processing approach. This paper describes using the modular spaCy Natural Language Processing framework to build part-of-speech taggers for Gaidhlig using the Annotated Reference Corpus of Scottish Gaelic. We train two models with minimal pre-processing and configuration: one using a fine-grained tagset and another using a reduced coarse-grained tagset. Both models are trained without external embeddings or pre-trained language models, using only supervised learning from the available corpus. The fine-grained model achieves 88.6% tagging accuracy, while the coarse-grained model achieves 93.7%. The results are comparable to those of the two previously published Gaidhlig taggers, indicating that simple, off-the-shelf language processing pipelines can demonstrate good performance in low-resource and morphologically complex linguistic settings.

Figures

Figures reproduced from arXiv: 2608.04808 by the authors.

Figure 1
Figure 1. Overview of the G`aidhlig PoS tagging pipeline. Most importantly, the spaCy framework provides a lang module containing language-specific information for the tokeniser. Thanks to the recent work of researchers at Cornell College [5], the G`aidhlig module now includes a list of stop￾words and tokeniser exceptions. The tokeniser also uses a rule set shared across most languages as a default in spaCy1 . In addition to … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

  1. [1]

    Universal dependencies for Scottish Gaelic: syntax

    Colin Batchelor. Universal dependencies for Scottish Gaelic: syntax. InProceedings of the Celtic Language Technology Workshop, pages 7–15, 2019

  2. [2]

    An online linguistic analyser for Scottish Gaelic

    Lo ¨ ıc Boizou and William Lamb. An online linguistic analyser for Scottish Gaelic. InHuman Language Technologies – The Baltic Perspective, pages 119–122. IOS Press, 2020

  3. [3]

    Wiley-Blackwell, John Wiley & Sons Ltd, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ United Kingdom, 2010

    Ciprian Chelba.Statistical Language Modeling, pages 74–104. Wiley-Blackwell, John Wiley & Sons Ltd, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ United Kingdom, 2010. Parts of this chapter appeared in Computer Speech & Language, vol. 14, no. 4, pages 283-332, October 2000, and are used with the permission of Elsevier Limited

  4. [4]

    Part of speech tagging: a systematic review of deep learning and machine learning approaches.Journal of Big Data, 9(1):10, 2022

    Alebachew Chiche and Betselot Yitagesu. Part of speech tagging: a systematic review of deep learning and machine learning approaches.Journal of Big Data, 9(1):10, 2022

  5. [5]

    Gaelic lemmatizer (spaCy-compatible NLP pipeline)

    Scottish Gaelic Collab. Gaelic lemmatizer (spaCy-compatible NLP pipeline). https: //github.com/Scottish-Gaelic-Collab/lemmatizer gaelic, 2025. Accessed: 2026-06- 11

  6. [6]

    Developing an automatic part-of-speech tagger for Scottish Gaelic

    Samuel Danso and William Lamb. Developing an automatic part-of-speech tagger for Scottish Gaelic. In John Judge, Teresa Lynn, Monica Ward, and Ois ´ ın Ruair ´ ı ´O Brian, editors,Proceedings of the Celtic Technology Workshop (CLTW 2014): A Workshop of the 25th International Conference on Computational Linguistics (COLING 2014), pages 1–5, Dublin, Ireland...

  7. [7]

    Developing au- tomatic speech recognition for Scottish Gaelic

    Lucy Evans, William Lamb, Mark Sinclair, and Beatrice Alex. Developing au- tomatic speech recognition for Scottish Gaelic. InProceedings of the 4th Celtic language technology workshop within LREC2022, pages 110–120, 2022

  8. [8]

    Hedderich, Lukas Lange, Heike Adel, Jannik Str¨ otgen, and Diet- rich Klakow

    Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Str¨ otgen, and Diet- rich Klakow. A survey on recent approaches for natural language processing in low-resource scenarios. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors,Proceedings ...

Show all 28 references
  1. [9]

    spaCy: Industrial-strength natural language processing in Python, 2020

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength natural language processing in Python, 2020. https: //doi.org/10.5281/zenodo.1212303

  2. [10]

    Martin.Speech and Language Processing: An In- troduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models

    Daniel Jurafsky and James H. Martin.Speech and Language Processing: An In- troduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, with Language Models. Stanford University (online manuscript re- leased January 6, 2026), 3rd edition, 2026....

  3. [11]

    Language shift, bilingualism and the future of Britain’s Celtic languages.Philosophical Transactions of the Royal Society B: Biological Sciences, 365(1559):3855–3864, 2010

    Anne Kandler, Roman Unger, and James Steele. Language shift, bilingualism and the future of Britain’s Celtic languages.Philosophical Transactions of the Royal Society B: Biological Sciences, 365(1559):3855–3864, 2010

  4. [12]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. https://arxiv.org/abs/2001.08361

  5. [13]

    A morphology-driven approach to NLP for a low-resource, highly complex language.Vietnam Journal of Computer Science, 1:19, 2025

    Irakli Kardava. A morphology-driven approach to NLP for a low-resource, highly complex language.Vietnam Journal of Computer Science, 1:19, 2025. 12 P´ eter Stef´ an et al

  6. [14]

    A practitioner’s guide to build- ing ASR models for low-resource languages: A case study on Scottish Gaelic

    Ondˇ rej Klejch, William Lamb, and Peter Bell. A practitioner’s guide to build- ing ASR models for low-resource languages: A case study on Scottish Gaelic. In Proceedings of Interspeech 2025, pages 728–732, 2025

  7. [15]

    Language death in the digital age.Language, 1(8), 2012

    Andr´ as Kornai. Language death in the digital age.Language, 1(8), 2012

  8. [16]

    Routledge, 2024

    William Lamb.Scottish Gaelic: A comprehensive grammar. Routledge, 2024

  9. [17]

    An- notated reference corpus of Scottish Gaelic (ARCOSG), 1997–2020, 2020

    William Lamb, Sharon Arbuthnot, Susanna Naismith, and Samuel Danso. An- notated reference corpus of Scottish Gaelic (ARCOSG), 1997–2020, 2020. https: //github.com/Gaelic-Algorithmic-Research-Group/ARCOSG

  10. [18]

    Developing an automatic part-of-speech tag- ger for Scottish Gaelic

    William Lamb and Samuel Danso. Developing an automatic part-of-speech tag- ger for Scottish Gaelic. InProceedings of the First Celtic Language Technology Workshop, pages 1–5, 2014

  11. [19]

    Evaluating a Gaelic Part-of- Speech Tagger and Reference Corpus

    William Lamb, Samuel Danso, and Alistair Lawson. Evaluating a Gaelic Part-of- Speech Tagger and Reference Corpus. Technical report, University of Edinburgh, Edinburgh, Scotland, 2016

  12. [20]

    Developing Word Embedding Models for Scot- tish Gaelic

    William Lamb and Mark Sinclair. Developing Word Embedding Models for Scot- tish Gaelic. InActes de la conf´ erence conjointe JEP-TALN-RECITAL, volume 6, pages 31–41, 2016

  13. [21]

    Heuristic sample selection to minimize reference standard training set for a part-of-speech tagger.Journal of the American Medical Informatics Association, 14(5):641–650, 2007

    Kaihong Liu, Wendy Chapman, Rebecca Hwa, and Rebecca S Crowley. Heuristic sample selection to minimize reference standard training set for a part-of-speech tagger.Journal of the American Medical Informatics Association, 14(5):641–650, 2007

  14. [22]

    Low-resource lan- guages: A review of past work and future challenges, 2020

    Alexandre Magueresse, Vincent Carles, and Evan Heetderks. Low-resource lan- guages: A review of past work and future challenges, 2020. https://arxiv.org/abs/ 2006.07264

  15. [23]

    A Rule-Based Computational Model for G` aidhlig Morphology

    Peter J Barclay. A Rule-Based Computational Model for G` aidhlig Morphology. In Proceedings of the 18th International Conference on Agents and Artificial Intelli- gence, volume 1, pages 683–693. SCITEPRESS – Science and Technology Publica- tions, Lda., March 2026

  16. [24]

    PhD thesis, University of Glasgow, 2016

    Susan Ross.The Standardisation of Scottish Gaelic Orthography 1750–2007. PhD thesis, University of Glasgow, 2016. https://theses.gla.ac.uk/7403/

  17. [25]

    Data augmentation via dependency tree morphing for low-resource languages

    G¨ ozde G¨ ul S ¸ahin and Mark Steedman. Data augmentation via dependency tree morphing for low-resource languages. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5004–5009, 2018

  18. [26]

    Parsing mor- phologically rich languages: Introduction to the special issue.Computational lin- guistics, 39(1):15–22, 2013

    Reut Tsarfaty, Djam´ e Seddah, Sandra K¨ ubler, and Joakim Nivre. Parsing mor- phologically rich languages: Introduction to the special issue.Computational lin- guistics, 39(1):15–22, 2013

  19. [27]

    Lipton, and Yulia Tsvetkov

    Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual models: Findings and a meta-learning treatment. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Languag...

  20. [28]

    Analyzing the understanding of morphologically complex words in large language models

    Marion Weller-Di Marco and Alexander Fraser. Analyzing the understanding of morphologically complex words in large language models. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Re- sources and Evaluation (LREC-COLING 2024), pa...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.