Pith. sign in

REVIEW 2 major objections 6 minor 29 references

GR-NLP-TOOLKIT: An Open-Source NLP Toolkit for Modern Greek

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper presents GR-NLP-TOOLKIT, an open-source Python library that claims state-of-the-art results on five core Modern Greek NLP tasks and reports outperforming spaCy and Stanza in benchmark comparisons.

desk verdict A genuinely useful, well-packaged Greek NLP toolkit, but the 'state-of-the-art' claim—especially for dependency parsing—is not yet backed by the evaluation. read the letter →

arxiv 2412.08520 v1 pith:JP6V5EZ6 submitted 2024-12-11 cs.CL cs.AIcs.SE

Lefteris Loukas , Nikolaos Smyrnioudis , Chrysa Dikonomaki , Spyros Barbakos , Anastasios Toumazatos , John Koutsikakis , Manolis Kyriakakis , Mary Georgiou
show 3 more authors
Stavros Vassos John Pavlopoulos Ion Androutsopoulos
This is my paper · ORCID
classification cs.CLcs.AIcs.SE
keywords ModernGreekNLPopen-sourcetoolkitGreek-BERTdependencyparsingnamedentityrecognitionGreeklishtransliterationpart-of-speechtaggingmorphological
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces GR-NLP-TOOLKIT, an open-source Python library that claims state-of-the-art results on five core Modern Greek NLP tasks: part-of-speech tagging, morphological tagging, dependency parsing, named entity recognition, and Greeklish-to-Greek transliteration. The claim matters because Modern Greek is poorly served by multilingual NLP tools, and the paper reports that its toolkit beats the widely used alternatives spaCy and Stanza in most benchmarks. The toolkit is built on a Greek-specialized transformer (Greek-BERT) for the four annotation tasks and a byte-level model (BYT5) for transliteration. If the reported results hold, researchers and developers get a single pip-installable package that outperforms the main open-source options for the language.

What carries the argument

The machinery is a Greek-BERT backbone (a transformer language model pretrained on Greek text) with task-specific classification heads: seventeen token-classification heads predict universal part-of-speech tags and morphological features, with one head per feature category, and a separate head handles part-of-speech. Dependency parsing uses a biaffine attention parser in which Greek-BERT's contextualized word embeddings replace the original BiLSTM encoder, and each word's head is chosen greedily by the highest arc score. Greeklish-to-Greek transliteration is handled by a BYT5 byte-level model that processes raw bytes, making it suited to text that mixes Greek and Latin alphabets. The whole toolkit is packaged for Python installation and exposed through a demo space and an HTTP API.

What would settle it

Run the dependency parser over the Greek Universal Dependencies test set and inspect every output for tree validity, counting sentences with cycles or multiple roots; if a substantial share of outputs are not trees, the reported UAS and LAS overstate the quality of dependency structures users would expect.

Watch

Extended reading notes

Core claim

The central claim is that a single open-source toolkit, created by fine-tuning Greek-BERT with task-specific heads, can outperform the widely used multilingual toolkits spaCy and Stanza on Modern Greek for part-of-speech tagging, morphological tagging, and dependency parsing, and that a BYT5-based transliterator provides state-of-the-art Greeklish-to-Greek conversion. On the Greek Universal Dependencies treebank, the toolkit reaches 0.98 micro-F1 and 0.97 macro-F1 for universal part-of-speech tagging, improves morphological tagging over both baselines in most categories, and reaches 0.94 UAS and 0.92 LAS for dependency parsing. On a Greek named-entity dataset with 18 entity types, it beats spaCy on all six shared entity types, and the Greeklish converter reproduces the BYT5 model of prior work that outperformed other evaluated methods aside from GPT-4. The paper frames this as a complete, ready-to-use solution for Modern Greek NLP.

Load-bearing premise

The paper assumes that greedily selecting each word's most probable head produces a usable dependency parser, even though the output can contain cycles and multiple roots and is not guaranteed to be a well-formed dependency tree.

Editorial extensions

If this is right

  • Greek NLP users get a single pip-installable package that covers four annotation tasks plus Greeklish-to-Greek conversion, with benchmark results above the main open-source alternatives.
  • A Greek-specialized transformer suffices as a backbone for these tasks even though it lowercases input and strips accents, so language-specific pretraining is worth the effort for under-resourced languages.
  • Because the Greeklish transliterator can be chained with the other components in one pipeline, Greeklish text can be normalized before downstream tagging and parsing.
  • The released demo space and HTTP API make the toolkit usable from other programming languages and in no-code settings, widening its potential user base.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Using a non-greedy decoder, such as maximum spanning tree inference, would guarantee well-formed dependency trees and could raise the reported parse scores; the greedy argmax is a conservative choice that likely understates the model's quality.
  • The transliterator's blind spot for code-switched Greeklish is a natural next target; the authors mention an older statistical model that already handles code switching and could be added as an option without retraining.
  • The same recipe—a specialized monolingual encoder plus task-specific heads—could serve as a template for other under-resourced languages that currently depend on multilingual models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents GR-NLP-TOOLKIT, an open-source Python toolkit for Modern Greek covering POS tagging, morphological tagging, dependency parsing, named entity recognition, and Greeklish-to-Greek transliteration. The core models are fine-tuned GreekBERT variants plus a ByT5-based transliteration module, with spaCy and Stanza as comparators on public Greek benchmarks. The paper additionally describes a HuggingFace demo space and a publicly available HTTP API. The stated central claim is state-of-the-art performance in all five tasks.

Significance. If the performance claims are reliable, this is a useful publicly released resource for a moderately low-resource language, with a straightforward pip install, open code, a demo, and an API. The evaluation is grounded in external public datasets and external baseline toolkits, which is a strength, and the release of the code and model infrastructure is a concrete contribution. However, the dependency-parsing evaluation appears to rest on a decoding protocol that is not standard UD tree decoding, which puts one of the five headline SOTA claims into question. The remaining claims are plausible but supported by a thin experimental section (single runs, no intervals, limited baselines). With the parser issue resolved and the evaluation documented more precisely, the resource would be a solid contribution to Greek NLP.

major comments (2)
  1. [§4.3 (arc-scoring equations and footnote 15)] The dependency parser as described cannot produce standard UD trees. The head-scoring model selects y_i^(arc) only among 'the other words of the sentence' and has no ROOT token, so the root word of a sentence has no representable attachment to ROOT (index 0). Moreover, greedy per-word argmax over heads can produce cycles and multiple roots, as footnote 15 implicitly acknowledges. As a result, the UAS/LAS figures in Table 5 are not comparable to those of Stanza and spaCy under the usual UD scoring conventions unless a non-standard protocol (e.g., excluding the root or evaluating only non-root tokens) was used; no such protocol is reported. The authors should either add a ROOT-scoring option and a tree-constrained decoder (e.g., Chu-Liu-Edmonds) or explicitly state and justify the scoring protocol, and recompute the comparison under identical conventions.
  2. [§4.1–4.3, Tables 2–5] The empirical support for the state-of-the-art claim is thin. All reported scores come from single runs without error bars, confidence intervals, or significance tests, and the exact train/dev/test splits and random seeds are not given. The NER comparison covers only the six entity types shared with spaCy, not the full 18 types the toolkit claims, and the POS comparison is effectively a tie with Stanza (micro-F1 0.98 vs 0.98). To support the central SOTA claim, the authors should report the evaluation protocol in enough detail to be reproducible, quantify variability, or temper the wording from 'state-of-the-art' to 'on par with or better than' where the data do not justify the stronger claim.
minor comments (6)
  1. [§4.2–4.3] The text says the models were trained and evaluated on 'the modern Greek part of the Universal Dependencies treebank' without naming whether this is GDT, GUD, or a concatenation, or which version and split were used; please specify.
  2. [§4, footnote 8] GreekBERT lowercases and strips accents; since morphological tagging and dependency parsing involve accent- and case-sensitive distinctions, the authors should briefly discuss how this preprocessing affects the reported scores.
  3. [§4.4] The Greeklish-to-Greek state-of-the-art claim is fully delegated to Toumazatos et al. (2024); either include a compact reproduction table or explicitly frame the claim as inherited from prior work rather than demonstrated in this paper.
  4. [Table 1] The GR-NLP-TOOLKIT row appears to contain more checkmarks than there are columns; please realign the table with the headers.
  5. [Appendix A.1] Table 6 reports only hyperparameter ranges; for reproducibility, please report the selected values for each task.
  6. [§4, paragraph 2] The citation 'citettoumazatos-etal-2024-still-all-greeklish-to-me' is malformed and should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the four evaluated tasks use external benchmarks and the Greeklish claim is backed by a separate published study.

full rationale

The paper's quantitative claims for POS tagging, morphological tagging, dependency parsing, and NER are benchmarked on external public datasets (the UD Greek treebank and the Bartziokas et al. NER corpus) against external toolkits (spaCy and Stanza). The models are fine-tuned on training splits, tuned on development splits, and reported on test splits, so no fitted quantity is renamed as a prediction. The Greeklish-to-Greek claim is not re-derived in this paper; it is attributed to a separately published, peer-reviewed study (Toumazatos et al., 2024) whose converter is reproduced in the toolkit. The overlapping authorship makes this a self-citation, but the cited study's evaluation on synthetic and real-life Greeklish, including comparisons against GPT-4 and statistical baselines, is externally falsifiable and does not depend on the present paper's fitted values. The dependency-parsing decoder in §4.3 greedily selects a head among 'the other words of the sentence' with no ROOT option, and footnote 15 concedes that a non-greedy decoder 'would also guarantee that the output is always a tree'; this is a correctness and comparability limitation rather than a circularity, because the reported UAS/LAS are still computed on an external treebank and not derived from the model's own assumptions. Self-citations to GreekBERT and to the group's BSc/MSc theses motivate design choices but are not load-bearing for the benchmark numbers. The 'state-of-the-art' wording overstates the evidence given that only two baselines are compared, but overclaiming relative to baselines is a scope concern, not circular reasoning. No step reduces by definition or by construction to its own input.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical quantities or ad hoc constants; it relies on standard fine-tuning of existing models. The main assumptions are about dataset quality, baseline representativeness, and the acceptability of the greedy parser.

assumptions (3)
  • domain assumption The Universal Dependencies Greek treebank and the Bartziokas et al. (2020) NER dataset provide gold-standard annotations for evaluation.
    The paper trains and evaluates on these datasets without auditing annotation quality, label distribution, or potential data leakage between train/dev/test.
  • domain assumption spaCy and Stanza represent the strongest available open-source Greek NLP toolkits, so outperforming them suffices for the 'state-of-the-art' claim.
    The comparison is limited to these two baselines; other Greek-specific systems (e.g., Prokopidis and Piperidis 2020) and large language models are not benchmarked, yet the paper generalizes its conclusion.
  • domain assumption The greedy decoder for dependency parsing is an acceptable inference procedure even though it does not guarantee a well-formed tree.
    Footnote 15 acknowledges the decoder may produce cycles or multiple roots, but the paper still reports parser accuracy and calls it state-of-the-art, implicitly assuming users accept such outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GR-NLP-TOOLKIT: An Open-Source NLP Toolkit for Modern Greek." pith.science (2026). https://pith.science/paper/JP6V5EZ6

@misc{pith2026241208520,
  author       = {Pith},
  title        = {Pith review of: GR-NLP-TOOLKIT: An Open-Source NLP Toolkit for Modern Greek},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JP6V5EZ6}},
  note         = {Machine review of arXiv:2412.08520}
}
read the original abstract

We present GR-NLP-TOOLKIT, an open-source natural language processing (NLP) toolkit developed specifically for modern Greek. The toolkit provides state-of-the-art performance in five core NLP tasks, namely part-of-speech tagging, morphological tagging, dependency parsing, named entity recognition, and Greeklishto-Greek transliteration. The toolkit is based on pre-trained Transformers, it is freely available, and can be easily installed in Python (pip install gr-nlp-toolkit). It is also accessible through a demonstration platform on HuggingFace, along with a publicly available API for non-commercial use. We discuss the functionality provided for each task, the underlying methods, experiments against comparable open-source toolkits, and future possible enhancements. The toolkit is available at: https://github.com/nlpaueb/gr-nlp-toolkit

Figures

Figures reproduced from arXiv: 2412.08520 by the authors.

Figure 1
Figure 1. An example of a Greek sentence written in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A dependency tree generated by GR-NLP￾TOOLKIT for a Greek sentence whose English transla￾tion is “Manchester United was defeated by Atletico Bilbao with a 2:3 score.” Figure from Smyrnioudis (2021). Tree drawn using SPACY’s visualizer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. shows an example of Greeklish-to-Greek [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 15 canonical work pages

  1. [1]

    Juli Bakagianni, Kanella Pouli, Maria Gavriilidou, and John Pavlopoulos. 2024. Towards S ystematic M onolingual NLP surveys: G en A of G reek NLP . arXiv preprint arXiv:2407.09861

    arXiv 2024

  2. [3]

    O'Reilly Media, Inc

    Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural L anguage P rocessing with P ython: A nalyzing T ext with the N atural L anguage T oolkit . " O'Reilly Media, Inc."

    work page 2009

  3. [4]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. https://doi.org/10.1162/tacl_a_00051 Enriching W ord V ectors with S ubword I nformation . Transactions of the Association for Computational Linguistics, 5:135--146

    doi:10.1162/tacl_a_00051 2017

  4. [5]

    Aimilios Chalamandaris, Athanassios Protopapas, Pirros Tsiakoulis, and Spyros Raptis. 2006. http://www.lrec-conf.org/proceedings/lrec2006/pdf/390_pdf.pdf All G reek to me! an automatic G reeklish to G reek transliteration system . In Proceedings of the Fifth International Conference on Language Resources and Evaluation ( LREC ' 06) , Genoa, Italy. Europea...

    work page 2006

  5. [6]

    Chu and T.-H Liu

    Y.-J. Chu and T.-H Liu. 1965. On the shortest arborescence of a directed graph. Science Sinica, 14:1396--1400

    work page 1965

  6. [7]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised C ross-lingual R epresentation L earning at S cale . In Proceedings of the 58th Annual Meeting of the Association for ...

    doi:10.18653/v1/2020.acl-main.747 2020

  7. [8]

    Dikonimaki

    C. Dikonimaki. 2021. A T ransformer-based natural language processing toolkit for G reek -- P art of speech tagging and dependency parsing. Technical report, BSc thesis, Department of Informatics, Athens University of Economics and Business. http://nlp.cs.aueb.gr/theses/dikonimaki_bsc_thesis.pdf

    work page 2021

  8. [9]

    Timothy Dozat, Peng Qi, and Christopher D. Manning. 2017. https://doi.org/10.18653/v1/K17-3002 S tanford ' s G raph-based N eural D ependency P arser at the C o NLL 2017 shared task . In Proceedings of the C o NLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies , pages 20--30, Vancouver, Canada. Association for Computational...

    doi:10.18653/v1/k17-3002 2017

Show all 29 references
  1. [10]

    J. Edmonds. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards B, 71(4):233--240

    1967

  2. [11]

    Voula Ghotsoulia, Elina Desypri, Maria Koutsombogera, Prokopis Prokopidis, and Haris Papageorgiou. 2007. Towards a F rame S emantics R esource for G reek. In Proceedings of The Sixth Workshop on Treebanks and Linguistic Theories (TLT 2007), Bergen, Norway. University of Bergen

    2007

  3. [12]

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. https://doi.org/10.5281/zenodo.1212303 spaCy : I ndustrial-strength N atural L anguage P rocessing in P ython

    2020 doi

  4. [13]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  5. [15]

    G localness

    Dimitris Koutsogiannis and Bessie Mitsikopoulou. 2017. https://doi.org/10.1111/j.1083-6101.2003.tb00358.x Greeklish and G reekness: T rends and D iscourses of “ G localness” . Journal of Computer-Mediated Communication, 9(1):JCMC918

    2017

  6. [16]

    Kyriakakis

    M. Kyriakakis. 2018. Exploring deep neural network models of syntax with a focus on G reek. Technical report, MSc thesis, Department of Informatics, Athens University of Economics and Business. http://nlp.cs.aueb.gr/theses/kiriakakis_msc_thesis.pdf

    2018

  7. [17]

    Ilya Loshchilov, Frank Hutter, et al. 2017. Fixing W eight D ecay R egularization in A dam. arXiv preprint arXiv:1711.05101, 5

    2017 arXiv

  8. [18]

    Sasha Luccioni, Yacine Jernite, and Emma Strubell. 2024. https://doi.org/10.1145/3630106.3658542 Power H ungry P rocessing: W atts D riving the C ost of AI D eployment? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT '24, page 85–...

    2024

  9. [19]

    Gud G reek- GUD : G reek U niversal D ependencies T reebank

    Stella Markantonatou, Vivian Stamou, and Socrates Vak. Gud G reek- GUD : G reek U niversal D ependencies T reebank. https://github.com/UniversalDependencies/UD_Greek-GUD

  10. [20]

    Harris Papageorgiou, Elina Desipri, Maria Koutsombogera, Kanella Pouli, and Prokopis Prokopidis. 2006. Adding M ulti-layer S emantics to the G reek D ependency T reebank. In Proceedings of The Fifth International Conference on Language and Evaluation (LREC-2006), Genoa, Italy. ELRA

    2006

  11. [22]

    Prokopis Prokopidis, Elina Desypri, Maria Koutsombogera, Haris Papageorgiou, and Stelios Piperidis. 2005. http://www.ilsp.gr/homepages/prokopidis/documents/gdt_tlt2005.pdf Theoretical and P ractical I ssues in the C onstruction of a G reek D ependency T reebank . In Proceeding...

    2005

  12. [23]

    Prokopis Prokopidis and Haris Papageorgiou. 2017. http://www.aclweb.org/anthology/W17-0413.pdf Universal D ependencies for G reek . In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 102--106, Gothenburg, Sweden. Association for Computatio...

    2017

  13. [24]

    Prokopis Prokopidis and Harris Papageorgiou. 2014. http://www.aclweb.org/anthology/W14-6109.pdf Experiments for D ependency P arsing of G reek . In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Ca...

    2014

  14. [25]

    Prokopis Prokopidis and Stelios Piperidis. 2020. https://doi.org/10.1145/3411408.3411430 A neural nlp toolkit for greek . In 11th Hellenic Conference on Artificial Intelligence, SETN 2020, page 125–128, New York, NY, USA. Association for Computing Machinery

    2020

  15. [26]

    Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. https://doi.org/10.18653/v1/2020.acl-demos.14 S tanza: A P ython N atural L anguage P rocessing T oolkit for M any H uman L anguages . In Proceedings of the 58th Annual Meeting of the Associatio...

    2020 doi

  16. [27]

    Smyrnioudis

    N. Smyrnioudis. 2021. A T ransformer-based natural language processing toolkit for G reek -- N amed entity recognition and multi-task learning. Technical report, BSc thesis, Department of Informatics, Athens University of Economics and Business. http://nlp.cs.aueb.gr/theses/sm...

    2021

  17. [28]

    Anastasios Toumazatos, John Pavlopoulos, Ion Androutsopoulos, and Stavros Vassos. 2024. https://aclanthology.org/2024.lrec-main.1330 Still A ll G reeklish to M e: G reeklish to G reek transliteration . In Proceedings of the 2024 Joint International Conference on Computational ...

    2024

  18. [29]

    Leon Voukoutis, Dimitris Roussis, Georgios Paraskevopoulos, Sokratis Sofianopoulos, Prokopis Prokopidis, Vassilis Papavasileiou, Athanasios Katsamanis, Stelios Piperidis, and Vassilis Katsouros. 2024. https://arxiv.org/abs/2407.20743 Meltemi: The first open L arge L anguage M ...

    2024 arXiv

  19. [30]

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. https://doi.org/10.1162/tacl_a_00461 B y T 5: Towards a T oken- F ree F uture with P re-trained B yte-to- B yte M odels . Transactions of the Association fo...

    2022 doi

  20. [31]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  21. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.