Pith. sign in

REVIEW 3 major objections 3 minor 28 references

GliLem: Leveraging GliNER for Contextualized Lemmatization in Estonian

T0 review · 3 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GliLem lifts Estonian lemmatization accuracy from 89% to 97.7%.

desk verdict Solid lemmatization result with a new application of open-vocabulary NER to scoring edit scripts; the open-vocabulary generalization claim is underevidenced and the IR half is thinner, but this deserves peer review. read the letter →

arxiv 2412.20597 v3 pith:UNSIYQ2E submitted 2024-12-29 cs.CL

Aleksei Dorkin , Kairit Sirts This is my paper · ORCID
classification cs.CL
keywords lemmatizationEstonianmorphologicaldisambiguationGliNERVabamorftransformationrulesinformationretrievalBM25
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GliLem combines the rule-based Estonian analyzer Vabamorf with GliNER, an open-vocabulary model built for named-entity recognition, to choose the correct lemma among Vabamorf's candidates. The paper shows this raises lemmatization accuracy on the Estonian UD treebank from about 89% with Vabamorf's built-in HMM disambiguator to 97.7%, leaving less than two percentage points to the oracle ceiling where the correct lemma is among the candidates. To show this matters beyond benchmark scores, the authors translate the English DBpedia-Entity dataset into Estonian and benchmark BM25 search with stemming, HMM-disambiguated lemmatization, and GliLem lemmatization. Proper lemmatization beats stemming by roughly 10% in retrieval metrics, and the improved disambiguation adds a smaller but consistent gain in recall, most visible at high retrieval depths. The paper argues that Vabamorf's weakness is not analysis but context-dependent candidate selection.

What carries the argument

The load-bearing mechanism is the transformation-rule label dressed as a natural-language entity for GliNER's cross-encoder. For each token, Vabamorf proposes one or more lemmas; each is converted via a shortest edit script into a compact string label such as 'remove the last letter', and the 'do nothing' rule is treated as the default unlabeled state instead of a scored entity. GliNER computes embeddings for candidate text spans and for these rule labels, then scores their pairwise similarity, letting the model rank only the handful of rules that Vabamorf actually generated rather than an open vocabulary. The result is a contextual re-ranking of a rule-based analyzer's candidates, with the oracle ceiling of 99.3% showing how much of the remaining error comes from the candidate set itself.

What would settle it

Construct a test set of Estonian UD sentences restricted to tokens for which Vabamorf proposes at least two distinct lemma candidates whose transformation rules differ, and require the model to pick the right one; if GliLem's accuracy on this subset is close to the HMM's rather than the reported 97.7%, the headline number is being carried by the 'do nothing' majority class. Alternatively, evaluate on a held-out domain such as the translated DBpedia-Entity corpus and compare token-level lemma accuracy against the UD test set; a large drop would indicate that the rule-scoring does not generalize beyond the training distribution.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an open-vocabulary span-and-label matching model pretrained for NER can be repurposed into a contextual disambiguator for a rule-based morphological analyzer. GliLem takes the transformation rules (shortest edit scripts, such as 'remove the last letter' or 'uppercase the first letter') that convert each surface form into each candidate lemma, feeds those rules as natural-language entity labels alongside the text, and scores which span-rule pair fits the context. On the Estonian UD EDT test set, this scores 0.977 accuracy compared with 0.892 for Vabamorf's HMM disambiguator, 0.966 for a pattern-based token classification lemmatizer, and 0.993 for the oracle that accepts any Vabamorf candidate. The authors conclude that the gap between Vabamorf and its oracle is mostly a disambiguation problem, not an analysis problem, and that the approach captures most of that gap.

Load-bearing premise

The method assumes that a model pretrained on named-entity recognition can be fine-tuned to score abstract word-edit scripts, such as 'remove the last letter', against word spans in Estonian, and that the resulting ranking of Vabamorf's candidates transfers reliably from NER entity types to arbitrary edit rules; if that transfer fails, the accuracy gain over the HMM disambiguator would not materialize.

Editorial extensions

If this is right

  • Vabamorf's built-in HMM disambiguator can be swapped for a neural disambiguator with no change to the analyzer, lifting lemmatization accuracy by roughly 8.5 points.
  • For Estonian lexical search, lemmatization with Vabamorf improves BM25 recall and success at 100 by about 7 to 11 points over stemming, with GliLem adding around 1 more point of recall at high k.
  • Because the most frequent transformation rule is 'do nothing', the per-token label space stays small, which makes the open-vocabulary scoring tractable despite a large global rule vocabulary.
  • The near-oracle accuracy means that further lemmatization gains for Vabamorf must come from improving the candidate-generation step, not just from better disambiguation.
  • The released Estonian translation of DBpedia-Entity provides the first IR benchmark for Estonian that lets researchers compare normalization methods under a standard retrieval setup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the NER-to-edit-script transfer holds, the same recipe should work for any morphologically rich language that has a rule-based analyzer and a UD treebank, provided the analyzer's candidate recall is already high.
  • The small IR gain from 89% to 97.7% lemmatization accuracy suggests BM25 is fairly tolerant of lemma errors, so the practical payoff of better disambiguation may concentrate in the first-stage retrieval of hybrid systems where recall at high k matters.
  • A direct test of label transfer: measure GliLem accuracy on tokens whose correct rule is absent from the training rule vocabulary; if accuracy holds there, the model is genuinely composing rules, and if it drops, it is memorizing rule-forms.
  • The paper's own observation that batch inference cannot vary labels per example means corpus-scale deployment is CPU-bound; a version allowing per-example label sets could make the approach practical for large collections.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes GliLem, a hybrid lemmatizer for Estonian in which the rule-based analyzer Vabamorf generates lemma candidates and a fine-tuned GliNER model scores candidate-specific transformation-rule strings against token spans. Training and evaluation use the Estonian UD 2.14 corpus; the main lemmatization result is 97.7% test accuracy versus 89.2% for Vabamorf's HMM disambiguator and a 99.3% oracle upper bound. The authors also introduce a machine-translated Estonian version of the DBpedia-Entity dataset and compare BM25 retrieval under stemming, Vabamorf lemmatization, and GliLem disambiguation, reporting that lemmatization improves over stemming and that GliLem gives small but consistent Recall gains at high k.

Significance. If the open-vocabulary mechanism is verified, the paper is a useful demonstration that a span-labeling NER model can be repurposed as a contextual disambiguator over analyzer candidates. The evaluation is careful in several respects: it uses a held-out UD test set, bootstrap confidence intervals, an oracle upper bound, and a token-classification baseline, and the authors release a demo, the translated IR dataset, and the code base. The main risk is that the advertised open-vocabulary advantage is not directly measured; the current evidence supports the accuracy of the system on this corpus, but not the generalization mechanism that motivates the method.

major comments (3)
  1. [Section 3, Table 1, Table 2] The paper motivates GliLem by the inability of pattern-based token classifiers to handle transformation rules unseen in training, but it never measures how often test-time transformation rules are absent from the training vocabulary. Table 1 shows that the top six rules account for 71.6% of training tokens, so the overall 97.7% accuracy in Table 2 could be driven by common rules even if novel rules are scored poorly. Please report the overlap between the training rule vocabulary and the gold and candidate rules on dev and test; stratification of accuracy by whether the gold rule was seen during training; and the number of test tokens for which Vabamorf proposes at least one rule not in the training vocabulary. Without these numbers, the open-vocabulary claim is unsupported.
  2. [Section 4.1, Section 4.3] The inference-time handling of the default 'do nothing' rule is not specified. Section 4.1 states that the model is trained without a 'do nothing' label and treats it as the default state, but the paper does not state the score threshold or selection rule that decides when no span is emitted. A threshold inherited from NER pretraining may be miscalibrated for transformation-rule scores, and the bootstrap confidence intervals in Table 2 do not cover this decision-rule uncertainty. Please give the exact prediction procedure, including whether the 'do nothing' rule is ever included among the entity strings at inference, and report accuracy as a function of the threshold on the dev set.
  3. [Section 4.2, Table 2] The token-classification baseline used for the comparison is an adapter-based variant rather than the original full fine-tuning approach of Dorkin and Sirts (2023), and the preprocessing and UD version also differ from that earlier work. Because the abstract claims an 'improvement over the token classification-based baseline', the comparison is confounded: the lower baseline score could reflect adapter capacity or preprocessing choices rather than the GliLem method. Please either report the original full-model accuracy on the same splits or retrain the baseline with the same backbone and comparable capacity, and state the result as a controlled comparison.
minor comments (3)
  1. [Table 1] The transformation-rule notation in Table 1 (e.g., '↓0;d¦', '↑0¦↓1;d¦') is never defined in the text; please add an explanation of the symbols and an example of how a rule is applied to a surface form to obtain a lemma.
  2. [Section 5.1] The translated DBpedia-Entity dataset is a released contribution, but its quality is only assessed by manual inspection of a small sample. A brief quantitative evaluation, such as a human-rated sample or a back-translation score, would make the resource more reliable and the IR conclusions easier to interpret.
  3. [Table 3, Section 5.4] The IR results are reported without confidence intervals or significance tests; differences of about one percentage point across 467 queries may be within noise, and for k=5 GliLem is worse than Vabamorf on MAP and Success@5. Please add uncertainty estimates and phrase the conclusion as metric-specific rather than as a general monotonic improvement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GliLem is trained and evaluated on held-out UD splits, and the reported gains are measured, not fitted or self-derived.

full rationale

The paper's central claim is that a GliNER-based disambiguator (GliLem) raises Vabamorf lemmatization accuracy from the HMM's 89% to 97.7% on the Estonian UD EDT test set. This is a direct supervised evaluation: transformation rules are derived from UD gold token/lemma pairs, the model is fine-tuned on the UD train split, and accuracy is computed on the held-out test split. No equation in the paper defines the predicted accuracy in terms of the training labels, and no fitted parameter is renamed as a prediction. The 'do nothing' default rule is treated as an unlabeled default rather than a scored class, but this is a modeling choice, not a circular derivation; at most it creates an underspecified inference threshold, which is a reproducibility concern. The paper cites the authors' earlier work (Dorkin and Sirts, 2023) for the HMM baseline and oracle figures, but these are external baselines, not premises that force the reported 97.7% result; moreover, the current paper independently evaluates the same Vabamorf disambiguator on the same test data (Table 2). The other self-citation (Dorkin and Sirts, 2024) is only contextual. The method's reliance on GliNER's pretrained span/entity representations generalizing to edit-script labels is an empirical assumption about transfer, not a formal circularity: the test result would be falsifiable if the transfer failed. The paper explicitly acknowledges limitations such as the lack of quantitative translation-quality evaluation and the noise in the translated IR corpus, but these affect external validity rather than circularity. There is no self-definitional reduction, no fitted input called a prediction, no load-bearing self-citation chain, and no imported uniqueness theorem. The derivation is self-contained against an external benchmark, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rely primarily on the transferability of a pretrained NER model to a different labeling task, on the quality of an unvalidated machine-translated IR benchmark, and on the coverage of Vabamorf's candidate set.

assumptions (4)
  • domain assumption The pre-trained GliNER model's span and entity representations generalize from NER entity types to arbitrary transformation-rule labels in Estonian.
    The whole method rests on this transfer; the paper asserts flexibility in Section 3 but does not provide a separate validation beyond the final accuracy.
  • domain assumption The NLLB-3B translation of DBpedia-Entity preserves enough query-document semantics for BM25 evaluation to be meaningful.
    Section 5.1 explicitly notes no quantitative quality evaluation; noise is acknowledged but assumed not to change relative rankings.
  • domain assumption Vabamorf's candidate generation covers the correct lemma for essentially all tokens (oracle >99%).
    Used to interpret the disambiguation gap; measured in Table 2 on the same test set, so it is empirically supported for this corpus.
  • domain assumption The transformation-rule representation (shortest edit scripts) is sufficient and generalizes beyond the training vocabulary.
    Based on Straka (2018) and Toporkov and Agerri (2024a), cited by the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GliLem: Leveraging GliNER for Contextualized Lemmatization in Estonian." pith.science (2026). https://pith.science/paper/UNSIYQ2E

@misc{pith2026241220597,
  author       = {Pith},
  title        = {Pith review of: GliLem: Leveraging GliNER for Contextualized Lemmatization in Estonian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UNSIYQ2E}},
  note         = {Machine review of arXiv:2412.20597}
}
read the original abstract

We present GliLem -- a novel hybrid lemmatization system for Estonian that enhances the highly accurate rule-based morphological analyzer Vabamorf with an external disambiguation module based on GliNER -- an open vocabulary NER model that is able to match text spans with text labels in natural language. We leverage the flexibility of a pre-trained GliNER model to improve the lemmatization accuracy of Vabamorf by 10% compared to its original disambiguation module and achieve an improvement over the token classification-based baseline. To measure the impact of improvements in lemmatization accuracy on the information retrieval downstream task, we first created an information retrieval dataset for Estonian by automatically translating the DBpedia-Entity dataset from English. We benchmark several token normalization approaches, including lemmatization, on the created dataset using the BM25 algorithm. We observe a substantial improvement in IR metrics when using lemmatization over simplistic stemming. The benefits of improving lemma disambiguation accuracy manifest in small but consistent improvement in the IR recall measure, especially in the setting of high k.

Figures

Figures reproduced from arXiv: 2412.20597 by the authors.

Figure 1
Figure 1. Schematic representation of the GliNER architecture applied to lemmatization. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 13 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Stefan Buttcher, Charles LA Clarke, and Gordon V Cormack. 2016. I nformation R etrieval: I mplementing and E valuating S earch E ngines . MIT Press

    work page 2016

  4. [4]

    Aleksei Dorkin and Kairit Sirts. 2023. https://aclanthology.org/2023.nodalida-1.28/ C omparison of C urrent A pproaches to L emmatization: A C ase S tudy in E stonian . In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 280--285, T \'o rshavn, Faroe Islands. University of Tartu Library

    work page 2023

  5. [5]

    Aleksei Dorkin and Kairit Sirts. 2024. https://doi.org/10.18653/v1/2024.starsem-1.33 S \ o najaht: Definition Embeddings and Semantic Search for Reverse Dictionary Creation . In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 410--420, Mexico City, Mexico. Association for Computational Linguistics

    doi:10.18653/v1/2024.starsem-1.33 2024

  6. [6]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 T he P ile: A n 800 GB D ataset of D iverse T ext for L anguage M odeling . ArXiv, abs/2101.00027

    arXiv 2020

  7. [7]

    Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021. https://doi.org/10.1007/978-3-030-72113-8_10 C omplementing L exical R etrieval with S emantic R esidual E mbedding . In Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28--April 1, 2021, Proceedings, Part I...

    doi:10.1007/978-3-030-72113-8_10 2021

  8. [8]

    Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. DB pedia- E ntity V 2: A T est C ollection for E ntity S earch. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 1265--1268. ACM

    work page 2017

Show all 28 references
  1. [9]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. https://proceedings.mlr.press/v97/houlsby19a.html P arameter- E fficient T ransfer L earning for NLP . In International confere...

    2019

  2. [10]

    Heiki-Jaan Kaalep and Tarmo Vaino. 2001. Complete Morphological Analysis in the Linguist’s Toolbox . Congressus Nonus Internationalis Fenno-Ugristarum Pars V, pages 9--16

    2001

  3. [11]

    Jenna Kanerva, Filip Ginter, Niko Miekka, Akseli Leino, and Tapio Salakoski. 2018. https://doi.org/10.18653/v1/K18-2013 Turku N eural P arser P ipeline: A n E nd-to- E nd S ystem for the C o NLL 2018 S hared T ask . In Proceedings of the CoNLL 2018 Shared Task: Multilingual pa...

    2018 doi

  4. [12]

    Jenna Kanerva, Filip Ginter, and Tapio Salakoski. 2021. https://doi.org/10.1017/S1351324920000224 Universal L emmatizer: A sequence-to-sequence model for lemmatizing U niversal D ependencies treebanks . Natural Language Engineering, 27(5):545--574

    2021 doi

  5. [13]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 D ense P assage R etrieval for O pen-domain Q uestion A nswering . In Proceedings of the 2020 Conference on E...

    2020 doi

  6. [14]

    Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. https://doi.org/10.1162/tacl_a_00405 R elevance-guided S upervision for O pen QA with C ol BERT . Transactions of the Association for Computational Linguistics, 9

    2021 doi

  7. [15]

    Dohyeon Lee, Seung-won Hwang, Kyungjae Lee, Seungtaek Choi, and Sunghyun Park. 2023. https://doi.org/10.18653/v1/2023.acl-long.746 O n C omplementarity O bjectives for H ybrid R etrieval . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisti...

    2023 doi

  8. [16]

    Xing Han Lù. 2024. http://arxiv.org/abs/2407.03618 BM25S: Orders of Magnitude Faster Lexical Search via Eager Sparse Scoring . arXiv preprint arXiv:2407.03618

    2024 arXiv

  9. [17]

    Siim Orasmaa, Timo Petmanson, Alexander Tkachenko, Sven Laur, and Heiki-Jaan Kaalep. 2016. https://aclanthology.org/L16-1390/ EstNLTK - NLP T oolkit for E stonian . In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, Fr...

    2016

  10. [18]

    Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. https://api.semanticscholar.org/CorpusID:41563977 Okapi at TREC-3 . Nist Special Publication Sp, 109:109

    1995

  11. [19]

    Milan Straka. 2018. https://doi.org/10.18653/v1/K18-2020 UDP ipe 2.0 P rototype at C o NLL 2018 UD S hared T ask . In Proceedings of the C o NLL 2018 S hared T ask: M ultilingual P arsing from R aw T ext to U niversal D ependencies , pages 197--207, Brussels, Belgium. Associat...

    2018 doi

  12. [20]

    Milan Straka, Jana Strakov \'a , and Jan Hajic. 2019. https://doi.org/10.18653/v1/W19-4212 UDPipe at SIGMORPHON 2019: Contextualized Embeddings, Regularization with Morphological Categories, Corpora Merging . SIGMORPHON 2019, page 95

    2019 doi

  13. [22]

    Nandan Thakur, Nils Reimers, Andreas R \"u ckl \'e , Abhishek Srivastava, and Iryna Gurevych. 2021. https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/65b9eea6e1cc6bb9f0cd2a47751a186f-Paper-round2.pdf BEIR : A H eterogenous B enchmark for Z ero-shot...

    2021

  14. [23]

    Olia Toporkov and Rodrigo Agerri. 2024 a . https://aclanthology.org/2024.lrec-main.572/ Evaluating S hortest E dit S cript M ethods for C ontextual L emmatization . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and E...

    2024

  15. [24]

    Olia Toporkov and Rodrigo Agerri. 2024 b . https://doi.org/10.1162/coli_a_00497 On the R ole of M orphological I nformation for C ontextual L emmatization . Computational Linguistics, 50(1)

    2024 doi

  16. [25]

    University of Tartu . 2018. https://doi.org/10.23673/PH6N-0144 U T R ocket

    2018 doi

  17. [26]

    Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. https://doi.org/10.18653/v1/2021.eacl-main.163 Applying the Transformer to Character-level Transduction . In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volu...

    2021 doi

  18. [27]

    Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2024. https://doi.org/10.18653/v1/2024.naacl-long.300 GL i NER : Generalist M odel for N amed E ntity R ecognition using B idirectional T ransformer . In Proceedings of the 2024 Conference of the North American...

    2024 doi

  19. [28]

    o zde Berk, Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agn \.e Bielinskien \.e , Krist \' n Bjarnad \'o ttir, Rogier Blokland, Victoria Bobicev, Lo \

    Daniel Zeman, Joakim Nivre, Mitchell Abrams, Elia Ackermann, No \"e mi Aepli, Hamid Aghaei, Z eljko Agi \'c , Amir Ahmadi, Lars Ahrenberg, Chika Kennedy Ajede, Salih Furkan Akkurt, Gabriel \.e Aleksandravi c i \=u t \.e , Ika Alfina, Avner Algom, Khalid Alnajjar, Chiara Alzett...

    2023

  20. [29]

    Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2023. https://arxiv.org/abs/2308.03279 U niversal NER : T argeted D istillation from L arge L anguage M odels for O pen N amed E ntity R ecognition . arXiv preprint arXiv:2308.03279

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.