{"id":"975f948f-cbc8-44b7-a574-5ad55c6a9195","arxiv_id":"2412.08520","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GR-NLP-TOOLKIT is a pip-installable Greek NLP toolkit reporting improved scores over spaCy and Stanza on five tasks, built by fine-tuning GreekBERT and ByT5.","lead":"This paper presents an open-source Python toolkit for modern Greek that performs five core NLP tasks: part-of-speech tagging, morphological tagging, dependency parsing, named entity recognition, and Greeklish-to-Greek transliteration. It packages fine-tuned GreekBERT and ByT5 models into a pip-installable library with a demo and API, aiming to make Greek language processing easier for researchers and developers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dependency-parsing SOTA claim rests on a decoder with no ROOT candidate, so outputs cannot be standard UD trees and Table 5 scores may not be comparable.","rationale":"The reader's conditional verdict is appropriate, and the reader identified the general area of the parser's output validity. However, the more specific and more load-bearing problem is that the model as described has no ROOT token in its head-selection space, so it cannot represent the standard UD root relation at all. This is an internal inconsistency in the method description and in the evaluation protocol, and it directly threatens the dependency-parsing component of the central 'state-of-the-art' claim. The concrete check is feasible because the toolkit is open source and the evaluation data are public. If the check shows that root handling is present and standard scoring preserves the reported numbers, the concern would be resolved. If not, the paper should be revised to either add a constrained decoder or qualify the dependency-parsing claim. The weaker points about limited baselines and missing error bars remain secondary; they affect all tasks, but the parser issue is the sharpest and most testable flaw.","tokens_in":10768,"tokens_out":7634,"duration_ms":87297,"concrete_test":"Inspect the released parser source to determine whether a ROOT token is included in the candidate head set. Then rerun the Table 5 evaluation (a) with standard UD scoring that counts the root word's attachment to ROOT and (b) with a Chu-Liu/Edmonds constrained decoder. If UAS/LAS drop materially when root is counted, or if greedy outputs are not valid rooted trees, the dependency-parsing state-of-the-art claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of state-of-the-art dependency parsing (§4.3) rests on an unstated and likely nonstandard decoding/scoring protocol. The head classifier scores arcs only from 'other words of the sentence' and never from a ROOT token; the equations for y_i^(arc) and the training cross-entropy admit no ROOT value. A functional digraph in which every word must choose another word as head cannot be a rooted dependency tree: the root word's correct attachment to ROOT is not representable, and cycles will arise wherever greedy choices are locally wrong. Hence the reported UAS/LAS in Table 5 are either computed with root excluded (which would make them non-comparable to Stanza and spaCy, since those follow UD conventions) or are inconsistent with the model description. Footnote 15 acknowledges that a non-greedy decoder 'would also guarantee that the output is always a tree,' confirming that the current decoder need not produce trees. Thus the dependency-parsing state-of-the-art claim has not been demonstrated for standard tree parsing. The toolkit itself is open and useful, and the other task claims may be plausible, but this specific claim needs either a corrected decoder or a clearly stated and justified scoring protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GR-NLP-TOOLKIT, an open-source Python toolkit for Modern Greek covering POS tagging, morphological tagging, dependency parsing, named entity recognition, and Greeklish-to-Greek transliteration. The core models are fine-tuned GreekBERT variants plus a ByT5-based transliteration module, with spaCy and Stanza as comparators on public Greek benchmarks. The paper additionally describes a HuggingFace demo space and a publicly available HTTP API. The stated central claim is state-of-the-art performance in all five tasks.","tokens_in":10947,"tokens_out":6905,"duration_ms":75791,"significance":"If the performance claims are reliable, this is a useful publicly released resource for a moderately low-resource language, with a straightforward pip install, open code, a demo, and an API. The evaluation is grounded in external public datasets and external baseline toolkits, which is a strength, and the release of the code and model infrastructure is a concrete contribution. However, the dependency-parsing evaluation appears to rest on a decoding protocol that is not standard UD tree decoding, which puts one of the five headline SOTA claims into question. The remaining claims are plausible but supported by a thin experimental section (single runs, no intervals, limited baselines). With the parser issue resolved and the evaluation documented more precisely, the resource would be a solid contribution to Greek NLP.","major_comments":[{"comment":"The dependency parser as described cannot produce standard UD trees. The head-scoring model selects y_i^(arc) only among 'the other words of the sentence' and has no ROOT token, so the root word of a sentence has no representable attachment to ROOT (index 0). Moreover, greedy per-word argmax over heads can produce cycles and multiple roots, as footnote 15 implicitly acknowledges. As a result, the UAS/LAS figures in Table 5 are not comparable to those of Stanza and spaCy under the usual UD scoring conventions unless a non-standard protocol (e.g., excluding the root or evaluating only non-root tokens) was used; no such protocol is reported. The authors should either add a ROOT-scoring option and a tree-constrained decoder (e.g., Chu-Liu-Edmonds) or explicitly state and justify the scoring protocol, and recompute the comparison under identical conventions.","section":"§4.3 (arc-scoring equations and footnote 15)"},{"comment":"The empirical support for the state-of-the-art claim is thin. All reported scores come from single runs without error bars, confidence intervals, or significance tests, and the exact train/dev/test splits and random seeds are not given. The NER comparison covers only the six entity types shared with spaCy, not the full 18 types the toolkit claims, and the POS comparison is effectively a tie with Stanza (micro-F1 0.98 vs 0.98). To support the central SOTA claim, the authors should report the evaluation protocol in enough detail to be reproducible, quantify variability, or temper the wording from 'state-of-the-art' to 'on par with or better than' where the data do not justify the stronger claim.","section":"§4.1–4.3, Tables 2–5"}],"minor_comments":[{"comment":"The text says the models were trained and evaluated on 'the modern Greek part of the Universal Dependencies treebank' without naming whether this is GDT, GUD, or a concatenation, or which version and split were used; please specify.","section":"§4.2–4.3"},{"comment":"GreekBERT lowercases and strips accents; since morphological tagging and dependency parsing involve accent- and case-sensitive distinctions, the authors should briefly discuss how this preprocessing affects the reported scores.","section":"§4, footnote 8"},{"comment":"The Greeklish-to-Greek state-of-the-art claim is fully delegated to Toumazatos et al. (2024); either include a compact reproduction table or explicitly frame the claim as inherited from prior work rather than demonstrated in this paper.","section":"§4.4"},{"comment":"The GR-NLP-TOOLKIT row appears to contain more checkmarks than there are columns; please realign the table with the headers.","section":"Table 1"},{"comment":"Table 6 reports only hyperparameter ranges; for reproducibility, please report the selected values for each task.","section":"Appendix A.1"},{"comment":"The citation 'citettoumazatos-etal-2024-still-all-greeklish-to-me' is malformed and should be corrected.","section":"§4, paragraph 2"}],"recommendation":"major_revision","confidential_remarks":"The dependency-parsing flaw is the decisive issue, but it is fixable within a revision: the authors can add a ROOT candidate and a tree-constrained decoder, or clearly define and justify an alternative scoring protocol. I see no evidence of circularity or misrepresentation: the benchmarks and baselines are external, and the use of GreekBERT and two BSc theses from the same group is normal for a resource paper. The main recommendation hinges on the parser fix and on making the SOTA claims more precise given the thin experimental setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a real engineering contribution for Greek NLP. A pip-installable toolkit, a HuggingFace demo, an API, and the first Greeklish-to-Greek module in a mainstream pipeline. That's worth having. But the abstract's claim of 'state-of-the-art performance in five core NLP tasks' is not supported as written, and the dependency-parsing claim has a load-bearing flaw.\n\nWhat's actually new is the integration, not the models. The components are GreekBERT, ByT5, and the Dozat biaffine parser, all previously published; most of the fine-tuning comes from two BSc theses and an MSc thesis. The authors are honest about that provenance. The new work is packaging, refactoring, the Greeklish module, and the head-to-head comparison against spaCy and Stanza. That is a legitimate contribution on its own terms.\n\nThe biggest problem is dependency parsing. In §4.3, the head predictor scores arcs from a word to 'other words of the sentence'—there is no ROOT token in the candidate set. So the root word's correct attachment to ROOT is not even representable, and the greedy decoder can output cycles and multiple roots. Footnote 15 concedes as much by saying a non-greedy decoder 'would also guarantee that the output is always a tree.' The UAS/LAS in Table 5 are therefore either computed with the root excluded (making them non-comparable to Stanza and spaCy, which follow standard UD scoring) or under a nonstandard protocol. The sentence 'clearly provides state-of-the-art performance for this task too' is not supported. This is fixable—add a Chu-Liu/Edmonds decoder or model a ROOT token—but it needs fixing.\n\nThe rest of the evaluation is thinner than the claims. Only two baselines, no error bars or significance tests, and the exact data splits are not specified. For POS and morphological tagging, GR-NLP is roughly on par with Stanza, which is plausible; the interesting result is that a GreekBERT-based transformer beats fastText-based tools. But 'state-of-the-art' is a stretch when the comparison set is one or two tools from 2020. The NER comparison is only against spaCy because Stanza lacks NER, and the Greeklish module is a reproduction of prior work with the code-switching limitation stated openly.\n\nWho is this for? Anyone doing Greek NLP who wants a working pipeline, and anyone studying how toolkit papers should evaluate their claims. The paper deserves a serious referee, but the referee should send it back for a revised evaluation: fix the dependency parser or its scoring protocol, add variance across seeds, specify splits, and consider at least one more baseline. The toolkit is worth shipping; the numbers just aren't ready as stated.","headline":"A genuinely useful, well-packaged Greek NLP toolkit, but the 'state-of-the-art' claim—especially for dependency parsing—is not yet backed by the evaluation.","tokens_in":11561,"tokens_out":2901,"would_cite":false,"duration_ms":30744,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents GR-NLP-TOOLKIT, an open-source Python library that claims state-of-the-art results on five core Modern Greek NLP tasks and reports outperforming spaCy and Stanza in benchmark comparisons.","keywords":["Modern Greek NLP","open-source toolkit","Greek-BERT","dependency parsing","named entity recognition","Greeklish transliteration","part-of-speech tagging","morphological tagging"],"falsifier":"Run the dependency parser over the Greek Universal Dependencies test set and inspect every output for tree validity, counting sentences with cycles or multiple roots; if a substantial share of outputs are not trees, the reported UAS and LAS overstate the quality of dependency structures users would expect.","tokens_in":10568,"feed_emoji":"🇬🇷","tokens_out":7049,"duration_ms":61337,"temperature":0.7,"pith_summary":"This paper introduces GR-NLP-TOOLKIT, an open-source Python library that claims state-of-the-art results on five core Modern Greek NLP tasks: part-of-speech tagging, morphological tagging, dependency parsing, named entity recognition, and Greeklish-to-Greek transliteration. The claim matters because Modern Greek is poorly served by multilingual NLP tools, and the paper reports that its toolkit beats the widely used alternatives spaCy and Stanza in most benchmarks. The toolkit is built on a Greek-specialized transformer (Greek-BERT) for the four annotation tasks and a byte-level model (BYT5) for transliteration. If the reported results hold, researchers and developers get a single pip-installable package that outperforms the main open-source options for the language.","feed_headline":"Open-source Greek NLP toolkit beats rivals on five tasks","feed_subtitle":"One pip package delivers top scores in tagging, parsing, NER, and Greeklish-to-Greek conversion.","key_machinery":"The machinery is a Greek-BERT backbone (a transformer language model pretrained on Greek text) with task-specific classification heads: seventeen token-classification heads predict universal part-of-speech tags and morphological features, with one head per feature category, and a separate head handles part-of-speech. Dependency parsing uses a biaffine attention parser in which Greek-BERT's contextualized word embeddings replace the original BiLSTM encoder, and each word's head is chosen greedily by the highest arc score. Greeklish-to-Greek transliteration is handled by a BYT5 byte-level model that processes raw bytes, making it suited to text that mixes Greek and Latin alphabets. The whole toolkit is packaged for Python installation and exposed through a demo space and an HTTP API.","core_discovery":"The central claim is that a single open-source toolkit, created by fine-tuning Greek-BERT with task-specific heads, can outperform the widely used multilingual toolkits spaCy and Stanza on Modern Greek for part-of-speech tagging, morphological tagging, and dependency parsing, and that a BYT5-based transliterator provides state-of-the-art Greeklish-to-Greek conversion. On the Greek Universal Dependencies treebank, the toolkit reaches 0.98 micro-F1 and 0.97 macro-F1 for universal part-of-speech tagging, improves morphological tagging over both baselines in most categories, and reaches 0.94 UAS and 0.92 LAS for dependency parsing. On a Greek named-entity dataset with 18 entity types, it beats spaCy on all six shared entity types, and the Greeklish converter reproduces the BYT5 model of prior work that outperformed other evaluated methods aside from GPT-4. The paper frames this as a complete, ready-to-use solution for Modern Greek NLP.","pith_inferences":["Using a non-greedy decoder, such as maximum spanning tree inference, would guarantee well-formed dependency trees and could raise the reported parse scores; the greedy argmax is a conservative choice that likely understates the model's quality.","The transliterator's blind spot for code-switched Greeklish is a natural next target; the authors mention an older statistical model that already handles code switching and could be added as an option without retraining.","The same recipe—a specialized monolingual encoder plus task-specific heads—could serve as a template for other under-resourced languages that currently depend on multilingual models."],"forward_implications":["Greek NLP users get a single pip-installable package that covers four annotation tasks plus Greeklish-to-Greek conversion, with benchmark results above the main open-source alternatives.","A Greek-specialized transformer suffices as a backbone for these tasks even though it lowercases input and strips accents, so language-specific pretraining is worth the effort for under-resourced languages.","Because the Greeklish transliterator can be chained with the other components in one pipeline, Greeklish text can be normalized before downstream tagging and parsing.","The released demo space and HTTP API make the toolkit usable from other programming languages and in no-code settings, widening its potential user base."],"supporting_citations":[{"why":"Provides Greek-BERT, the transformer backbone for four of the five tasks.","marker":"Koutsikakis et al., 2020"},{"why":"Supplies the biaffine dependency parser architecture adapted with Greek-BERT embeddings.","marker":"Dozat et al., 2017"},{"why":"Source of the reproduced BYT5 Greeklish-to-Greek model and its prior state-of-the-art results.","marker":"Toumazatos et al., 2024"},{"why":"Stanza is the main comparison baseline for tagging and parsing.","marker":"Qi et al., 2020"},{"why":"spaCy is the main comparison baseline, especially for NER.","marker":"Honnibal et al., 2020"},{"why":"Provides the Greek Universal Dependencies treebank used for training and testing tagging and parsing.","marker":"Prokopidis and Papageorgiou, 2017"},{"why":"Supplies the Greek NER dataset with 18 entity types used for training and evaluation.","marker":"Bartziokas et al., 2020"},{"why":"Introduces BYT5, the byte-level model used for Greeklish-to-Greek transliteration.","marker":"Xue et al., 2022"}],"fun_headline_variants":["Open-source Greek NLP toolkit beats spaCy and Stanza on core tasks","Greek NLP toolkit: top scores for POS, parsing, NER, and Greeklish conversion","One pip install for state-of-the-art Modern Greek NLP toolkit","Free Greek NLP toolkit outperforms rivals on five core tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that greedily selecting each word's most probable head produces a usable dependency parser, even though the output can contain cycles and multiple roots and is not guaranteed to be a well-formed dependency tree.","fun_headline_variants_meta":{"raw":{"variants":["Open-source Greek NLP toolkit beats spaCy and Stanza on core tasks","Greek NLP toolkit: top scores for POS, parsing, NER, and Greeklish conversion","One pip install for state-of-the-art Modern Greek NLP toolkit","Free Greek NLP toolkit outperforms rivals on five core tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000707,"raw_usage":{"total_tokens":3161,"prompt_tokens":898,"completion_tokens":2263,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2185}},"tokens_in":514,"tokens_out":2263,"duration_ms":16275,"temperature":1.0,"reasoning_tokens":2185,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:43:35.064866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the dependency parser over the Greek Universal Dependencies test set and inspect every output for tree validity, counting sentences with cycles or multiple roots; if a substantial share of outputs are not trees, the reported UAS and LAS overstate the quality of dependency structures users would expect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the reproduced BYT5 Greeklish-to-Greek model and its prior state-of-the-art results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Greek Universal Dependencies treebank used for training and testing tagging and parsing."}],"review_version":1}