{"id":"74956380-7456-4ea4-9357-9cf14b1c8ef4","arxiv_id":"2412.20597","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A hybrid system combining the rule-based analyzer Vabamorf with the GliNER span-labeling model reaches 97.7% lemmatization accuracy in Estonian and improves BM25 recall slightly over the default disambiguator.","lead":"Researchers built a new system, GliLem, that uses an open-vocabulary language model to pick the correct lemma of Estonian words, raising accuracy from 89% to 97.7% on a standard benchmark. The study also releases the first Estonian information retrieval test set, machine-translated from English, and shows that better lemmatization yields a small but consistent recall gain in BM25 search.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 97.7% accuracy is credible, but the 'open-vocabulary' claim needs support on unseen transformation rules.","rationale":"The reader's weakest assumption was that a model pretrained on NER can be fine-tuned to score abstract transformation-rule strings, with the paper providing empirical support but no separate analysis. I agree with that framing, and I sharpen it to a specific, testable failure mode: the model may only work for transformation rules that appear in training, and the decision rule for the 'do nothing' default is unspecified. The empirical 97.7% accuracy is well-supported by a held-out test set with bootstrap confidence intervals, so I do not believe the central number is wrong. However, the paper's stated novelty is the use of an open-vocabulary span-labeling model in a candidate-ranking setting, and that novelty is only justified if the model actually handles unfamiliar labels. The absence of any analysis of rule frequency or novelty, combined with the unspecified default threshold, means the generalization claim is not yet established. The reader's secondary concerns about IR error bars and baseline reproduction also remain, but they are not the focus of my stress-test. A concrete test on unseen rules and threshold sensitivity would settle whether the concern lands. I therefore keep the reader's CONDITIONAL verdict; the main claim is credible but would benefit from this clarification before full acceptance.","tokens_in":14422,"tokens_out":11569,"duration_ms":116846,"concrete_test":"On the UD test set, compute the gold transformation rule for each token and tag each rule as seen (appears in the training set) or unseen. Report the proportion of unseen rules, GliLem's accuracy on the seen subset, and its accuracy on the unseen subset. Additionally, sweep the similarity threshold used to accept a span-rule match (or the equivalent decision boundary for the 'do nothing' default) on the dev set and report how accuracy on the test set changes. If the unseen-rule accuracy is close to the overall accuracy and the result is insensitive to the threshold over a reasonable range, the open-vocabulary claim is supported; otherwise the concern stands.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central result depends on the claim that GliNER's pretrained span and entity embeddings generalize from NER entity types to arbitrary edit-script strings, enabling scoring of transformation rules that may never have been seen during fine-tuning. However, the paper does not analyze how often the gold transformation rule on the test set is absent from the training set, nor does it report accuracy on such novel rules. If novel rules are rare and the model performs near chance on them, then the 97.7% accuracy is driven by common rules, undermining the specific 'open vocabulary' motivation of the method. A second, related gap is the handling of the 'do nothing' default rule, which applies to 49.6% of training tokens. The authors train without an explicit 'do nothing' label and rely on the model emitting no span when the lemma equals the surface form. The paper never specifies the threshold or decision rule that operationalizes this default at inference. If a fixed threshold from NER pretraining is used, it may be miscalibrated for edit-script scoring, and the bootstrap confidence intervals in Table 2 do not capture this source of variability. These gaps do not invalidate the reported accuracy on the UD test set, but they leave the generalization mechanism of the method unverified, which is load-bearing for the paper's contribution beyond this one dataset.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GliLem, a hybrid lemmatizer for Estonian in which the rule-based analyzer Vabamorf generates lemma candidates and a fine-tuned GliNER model scores candidate-specific transformation-rule strings against token spans. Training and evaluation use the Estonian UD 2.14 corpus; the main lemmatization result is 97.7% test accuracy versus 89.2% for Vabamorf's HMM disambiguator and a 99.3% oracle upper bound. The authors also introduce a machine-translated Estonian version of the DBpedia-Entity dataset and compare BM25 retrieval under stemming, Vabamorf lemmatization, and GliLem disambiguation, reporting that lemmatization improves over stemming and that GliLem gives small but consistent Recall gains at high k.","tokens_in":14624,"tokens_out":7282,"duration_ms":74416,"significance":"If the open-vocabulary mechanism is verified, the paper is a useful demonstration that a span-labeling NER model can be repurposed as a contextual disambiguator over analyzer candidates. The evaluation is careful in several respects: it uses a held-out UD test set, bootstrap confidence intervals, an oracle upper bound, and a token-classification baseline, and the authors release a demo, the translated IR dataset, and the code base. The main risk is that the advertised open-vocabulary advantage is not directly measured; the current evidence supports the accuracy of the system on this corpus, but not the generalization mechanism that motivates the method.","major_comments":[{"comment":"The paper motivates GliLem by the inability of pattern-based token classifiers to handle transformation rules unseen in training, but it never measures how often test-time transformation rules are absent from the training vocabulary. Table 1 shows that the top six rules account for 71.6% of training tokens, so the overall 97.7% accuracy in Table 2 could be driven by common rules even if novel rules are scored poorly. Please report the overlap between the training rule vocabulary and the gold and candidate rules on dev and test; stratification of accuracy by whether the gold rule was seen during training; and the number of test tokens for which Vabamorf proposes at least one rule not in the training vocabulary. Without these numbers, the open-vocabulary claim is unsupported.","section":"Section 3, Table 1, Table 2"},{"comment":"The inference-time handling of the default 'do nothing' rule is not specified. Section 4.1 states that the model is trained without a 'do nothing' label and treats it as the default state, but the paper does not state the score threshold or selection rule that decides when no span is emitted. A threshold inherited from NER pretraining may be miscalibrated for transformation-rule scores, and the bootstrap confidence intervals in Table 2 do not cover this decision-rule uncertainty. Please give the exact prediction procedure, including whether the 'do nothing' rule is ever included among the entity strings at inference, and report accuracy as a function of the threshold on the dev set.","section":"Section 4.1, Section 4.3"},{"comment":"The token-classification baseline used for the comparison is an adapter-based variant rather than the original full fine-tuning approach of Dorkin and Sirts (2023), and the preprocessing and UD version also differ from that earlier work. Because the abstract claims an 'improvement over the token classification-based baseline', the comparison is confounded: the lower baseline score could reflect adapter capacity or preprocessing choices rather than the GliLem method. Please either report the original full-model accuracy on the same splits or retrain the baseline with the same backbone and comparable capacity, and state the result as a controlled comparison.","section":"Section 4.2, Table 2"}],"minor_comments":[{"comment":"The transformation-rule notation in Table 1 (e.g., '↓0;d¦', '↑0¦↓1;d¦') is never defined in the text; please add an explanation of the symbols and an example of how a rule is applied to a surface form to obtain a lemma.","section":"Table 1"},{"comment":"The translated DBpedia-Entity dataset is a released contribution, but its quality is only assessed by manual inspection of a small sample. A brief quantitative evaluation, such as a human-rated sample or a back-translation score, would make the resource more reliable and the IR conclusions easier to interpret.","section":"Section 5.1"},{"comment":"The IR results are reported without confidence intervals or significance tests; differences of about one percentage point across 467 queries may be within noise, and for k=5 GliLem is worse than Vabamorf on MAP and Success@5. Please add uncertainty estimates and phrase the conclusion as metric-specific rather than as a general monotonic improvement.","section":"Table 3, Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The central lemmatization accuracy result is credible and the evaluation is largely careful. The paper's distinctive claim, however, is the open-vocabulary generalization of the GliNER label-scoring mechanism, and that claim is currently unmeasured. I would urge the editor to require the rule-coverage and threshold analyses described in the major comments rather than reject the paper, because those analyses are feasible and directly address the load-bearing question."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the lemmatization result is credible and well evaluated—97.7% test accuracy on UD EDT with bootstrap CIs, an oracle upper bound, and a sensible token-classification baseline. The core trick, using GliNER's span-labeling cross-encoder to score Vabamorf's candidate edit scripts, is a genuinely new application of open-vocabulary NER, and it works. This paper deserves a serious referee.\n\nWhat it does well: the lemmatization evaluation is careful. They report CIs, compare against the HMM's 89.2%, and show GliLem gets within 1.6 points of the oracle. They also release the first Estonian IR dataset, translated from DBpedia-Entity, and show lemmatization beats stemming for BM25 by a wide margin, with small consistent recall gains for GliLem over the HMM at high k. The authors are honest about translation noise and computational cost.\n\nSoft spots, in proportion:\n\n- The 'open-vocabulary' claim is not actually demonstrated. The paper never reports how often the gold transformation rule on the test set is absent from the training set, nor accuracy on such novel rules. If novel rules are rare and the model mostly sees common edits, the 97.7% could be driven by memorization rather than generalization. That matters because the abstract's 'open vocabulary' framing is a key selling point. The fix is an analysis of rule coverage and per-rule accuracy. Not fatal, but necessary for the claim.\n\n- The default 'do nothing' rule is handled implicitly: they train without a label for it and rely on the model emitting no span. The paper never specifies the threshold or decision rule at inference. That's a reproducibility gap. A fixed threshold from NER pretraining might be miscalibrated for edit-script scoring, and the CIs don't capture that.\n\n- The IR experiments have no error bars or significance tests. The differences between Vabamorf and GliLem are small (1% in Recall@100), and on Success@5 and Success@100 GliLem is actually worse. The paper is appropriately cautious about this, but the lack of error bars makes it hard to know if the 'small consistent improvement' is signal or noise.\n\n- The token-classification baseline is adapter-based and uses different preprocessing and UD version than Dorkin and Sirts (2023), so the 1.2% gap is approximate. They acknowledge this, so it's a minor caveat.\n\nWho this is for: anyone working on lemmatization, morphologically rich languages, or Estonian NLP. The IR dataset alone is a useful community resource. The central claim holds up; the gaps are about generalization and reproducibility, not about the main result. I'd send it to review rather than desk-reject.","headline":"Solid lemmatization result with a new application of open-vocabulary NER to scoring edit scripts; the open-vocabulary generalization claim is underevidenced and the IR half is thinner, but this deserves peer review.","tokens_in":15186,"tokens_out":2687,"would_cite":true,"duration_ms":24610,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GliLem lifts Estonian lemmatization accuracy from 89% to 97.7%.","keywords":["lemmatization","Estonian","morphological disambiguation","GliNER","Vabamorf","transformation rules","information retrieval","BM25"],"falsifier":"Construct a test set of Estonian UD sentences restricted to tokens for which Vabamorf proposes at least two distinct lemma candidates whose transformation rules differ, and require the model to pick the right one; if GliLem's accuracy on this subset is close to the HMM's rather than the reported 97.7%, the headline number is being carried by the 'do nothing' majority class. Alternatively, evaluate on a held-out domain such as the translated DBpedia-Entity corpus and compare token-level lemma accuracy against the UD test set; a large drop would indicate that the rule-scoring does not generalize beyond the training distribution.","tokens_in":14202,"feed_emoji":"📖","tokens_out":6209,"duration_ms":51941,"temperature":0.7,"pith_summary":"GliLem combines the rule-based Estonian analyzer Vabamorf with GliNER, an open-vocabulary model built for named-entity recognition, to choose the correct lemma among Vabamorf's candidates. The paper shows this raises lemmatization accuracy on the Estonian UD treebank from about 89% with Vabamorf's built-in HMM disambiguator to 97.7%, leaving less than two percentage points to the oracle ceiling where the correct lemma is among the candidates. To show this matters beyond benchmark scores, the authors translate the English DBpedia-Entity dataset into Estonian and benchmark BM25 search with stemming, HMM-disambiguated lemmatization, and GliLem lemmatization. Proper lemmatization beats stemming by roughly 10% in retrieval metrics, and the improved disambiguation adds a smaller but consistent gain in recall, most visible at high retrieval depths. The paper argues that Vabamorf's weakness is not analysis but context-dependent candidate selection.","feed_headline":"Estonian lemmatizer jumps from 89% to 97.7% accuracy","feed_subtitle":"A named-entity model scores word-edit rules to pick the right lemma, and the gains show up in search results.","key_machinery":"The load-bearing mechanism is the transformation-rule label dressed as a natural-language entity for GliNER's cross-encoder. For each token, Vabamorf proposes one or more lemmas; each is converted via a shortest edit script into a compact string label such as 'remove the last letter', and the 'do nothing' rule is treated as the default unlabeled state instead of a scored entity. GliNER computes embeddings for candidate text spans and for these rule labels, then scores their pairwise similarity, letting the model rank only the handful of rules that Vabamorf actually generated rather than an open vocabulary. The result is a contextual re-ranking of a rule-based analyzer's candidates, with the oracle ceiling of 99.3% showing how much of the remaining error comes from the candidate set itself.","core_discovery":"On the paper's own terms, the central discovery is that an open-vocabulary span-and-label matching model pretrained for NER can be repurposed into a contextual disambiguator for a rule-based morphological analyzer. GliLem takes the transformation rules (shortest edit scripts, such as 'remove the last letter' or 'uppercase the first letter') that convert each surface form into each candidate lemma, feeds those rules as natural-language entity labels alongside the text, and scores which span-rule pair fits the context. On the Estonian UD EDT test set, this scores 0.977 accuracy compared with 0.892 for Vabamorf's HMM disambiguator, 0.966 for a pattern-based token classification lemmatizer, and 0.993 for the oracle that accepts any Vabamorf candidate. The authors conclude that the gap between Vabamorf and its oracle is mostly a disambiguation problem, not an analysis problem, and that the approach captures most of that gap.","pith_inferences":["If the NER-to-edit-script transfer holds, the same recipe should work for any morphologically rich language that has a rule-based analyzer and a UD treebank, provided the analyzer's candidate recall is already high.","The small IR gain from 89% to 97.7% lemmatization accuracy suggests BM25 is fairly tolerant of lemma errors, so the practical payoff of better disambiguation may concentrate in the first-stage retrieval of hybrid systems where recall at high k matters.","A direct test of label transfer: measure GliLem accuracy on tokens whose correct rule is absent from the training rule vocabulary; if accuracy holds there, the model is genuinely composing rules, and if it drops, it is memorizing rule-forms.","The paper's own observation that batch inference cannot vary labels per example means corpus-scale deployment is CPU-bound; a version allowing per-example label sets could make the approach practical for large collections."],"forward_implications":["Vabamorf's built-in HMM disambiguator can be swapped for a neural disambiguator with no change to the analyzer, lifting lemmatization accuracy by roughly 8.5 points.","For Estonian lexical search, lemmatization with Vabamorf improves BM25 recall and success at 100 by about 7 to 11 points over stemming, with GliLem adding around 1 more point of recall at high k.","Because the most frequent transformation rule is 'do nothing', the per-token label space stays small, which makes the open-vocabulary scoring tractable despite a large global rule vocabulary.","The near-oracle accuracy means that further lemmatization gains for Vabamorf must come from improving the candidate-generation step, not just from better disambiguation.","The released Estonian translation of DBpedia-Entity provides the first IR benchmark for Estonian that lets researchers compare normalization methods under a standard retrieval setup."],"supporting_citations":[{"why":"Supplies the rule-based analyzer whose lemma candidates GliLem re-ranks, along with the HMM disambiguator that serves as the main baseline.","marker":"Kaalep and Vaino, 2001"},{"why":"Contributes the GliNER architecture with span and entity embeddings and cross-encoder label matching that GliLem adapts.","marker":"Zaratiana et al., 2024"},{"why":"Provides the earlier measurements of Vabamorf's HMM and oracle accuracy and the pattern-based token classification baseline on Estonian.","marker":"Dorkin and Sirts, 2023"},{"why":"Introduces the transformation-rule (shortest edit script) representation that GliLem uses as natural-language labels.","marker":"Straka, 2018"},{"why":"Defines the DBpedia-Entity dataset and relevance judgments that the authors translate to Estonian for the IR benchmark.","marker":"Hasibi et al., 2017"},{"why":"Provides the NLLB translation model used to create the Estonian version of the DBpedia-Entity corpus.","marker":"NLLB Team et al., 2022"},{"why":"Supports the claim that shortest edit scripts generalize better than alternative rule-generation approaches, justifying the label design.","marker":"Toporkov and Agerri, 2024a"},{"why":"Supplies the Estonian Universal Dependencies EDT corpus and splits used to train and evaluate GliLem.","marker":"Zeman et al., 2023"},{"why":"Defines the BM25 retrieval algorithm used to measure the downstream IR impact of the lemmatization approaches.","marker":"Robertson et al., 1995"}],"fun_headline_variants":["Open-vocabulary NER improves Estonian lemmatization to 97.7%","From 89% to 97.7%: GliNER disambiguates Estonian lemmas","NER-based scorer for lemma selection: Estonian lemmatizer at 97.7%","Estonian lemmatizer hits 97.7% with NER-based disambiguation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a model pretrained on named-entity recognition can be fine-tuned to score abstract word-edit scripts, such as 'remove the last letter', against word spans in Estonian, and that the resulting ranking of Vabamorf's candidates transfers reliably from NER entity types to arbitrary edit rules; if that transfer fails, the accuracy gain over the HMM disambiguator would not materialize.","fun_headline_variants_meta":{"raw":{"variants":["Open-vocabulary NER improves Estonian lemmatization to 97.7%","From 89% to 97.7%: GliNER disambiguates Estonian lemmas","NER-based scorer for lemma selection: Estonian lemmatizer at 97.7%","Estonian lemmatizer hits 97.7% with NER-based disambiguation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00134,"raw_usage":{"total_tokens":5451,"prompt_tokens":951,"completion_tokens":4500,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":4400}},"tokens_in":567,"tokens_out":4500,"duration_ms":30607,"temperature":1.0,"reasoning_tokens":4400,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:15:58.633753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set of Estonian UD sentences restricted to tokens for which Vabamorf proposes at least two distinct lemma candidates whose transformation rules differ, and require the model to pick the right one; if GliLem's accuracy on this subset is close to the HMM's rather than the reported 97.7%, the headline number is being carried by the 'do nothing' majority class. Alternatively, evaluate on a held-out domain such as the translated DBpedia-Entity corpus and compare token-level lemma accuracy against the UD test set; a large drop would indicate that the rule-scoring does not generalize beyond the training distribution.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the rule-based analyzer whose lemma candidates GliLem re-ranks, along with the HMM disambiguator that serves as the main baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the earlier measurements of Vabamorf's HMM and oracle accuracy and the pattern-based token classification baseline on Estonian."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DBpedia-Entity dataset and relevance judgments that the authors translate to Estonian for the IR benchmark."},{"cited_title":"o zde Berk, Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agn \\.e Bielinskien \\.e , Krist \\' n Bjarnad \\'o ttir, Rogier Blokland, Victoria Bobicev, Lo \\","cited_arxiv_id":null,"evidence_quote":"Supplies the Estonian Universal Dependencies EDT corpus and splits used to train and evaluate GliLem."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the BM25 retrieval algorithm used to measure the downstream IR impact of the lemmatization approaches."}],"review_version":1}