REVIEW 4 major objections 5 minor 20 references
Context Aware Lemmatization and Morphological Tagging Method in Turkish
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper claims that adding word meaning from Turkish BERT to character-level LSTMs yields the first context-aware Turkish lemmatizer and beats SIGMORPHON 2019 baselines on IMST and most PUD metrics.
desk verdict A straightforward BERT+BiLSTM Turkish lemmatizer with new numbers, but its central 'first' claim is contradicted by its own references and the comparison is less fair than advertised. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is an encoder-decoder network with two parallel representation channels for each word: a character-level bidirectional LSTM builds a spelling vector from the word's letters, and a frozen-then-fine-tuned Turkish BERT model supplies a meaning vector from the surrounding sentence. In the sequenced variant, an additional embedding-plus-BiLSTM channel encodes the morphological tag predictions; the three vectors, or two in the separate variant, are projected through dense layers into one vector that initializes the decoder's hidden state. The decoder uses two forward LSTM layers with teacher forcing and Bahdanau attention over the character-level LSTM outputs to generate morphological tags or root-word letters one step at a time.
What would settle it
A controlled ablation that removes the BERT word-meaning vector from the model while keeping all other components and hyperparameters fixed, then reruns the same IMST and PUD splits, would settle whether context is the source of the gains; if the spelling-only variant matches the full model, the central claim that meaning sensitivity drives the improvement fails.
Extended reading notes
Core claim
The paper's central claim is that a Turkish-specific lemmatizer and morphological tagger that combines a word-meaning vector from a pretrained Turkish BERT model with a character-level bidirectional LSTM spelling representation outperforms the best results of the SIGMORPHON 2019 shared task on the Universal Dependencies Turkish IMST and PUD datasets in almost every reported metric. On IMST, the sequenced model, which feeds morphological tag predictions into the lemmatizer, reaches 98.13 percent lemmatization accuracy, 0.02 Levenshtein distance, 93.75 percent morphological tagging accuracy, and 98.24 F-1; the separate and sequenced models share the same morphological tagging numbers because the tagger is trained once and frozen. On PUD, the separate model reports 88.84 percent lemmatization accuracy, 0.12 Levenshtein distance, 89.66 percent morphological accuracy, and 96.41 F-1. The authors conclude that feeding morphological tags into lemmatization helps on IMST but hurts on PUD, and they present the work as the first Turkish lemmatization study that is sensitive to word meaning, while conceding that the comparison against the multilingual competition systems may be considered unfair.
Load-bearing premise
The claim of superiority rests on the assumption that comparing a Turkish-only model equipped with a large pretrained Turkish BERT to the multilingual SIGMORPHON 2019 systems is a meaningful basis for comparison; the paper itself acknowledges the comparison may be considered unfair.
Editorial extensions
If this is right
- On the IMST dataset, the sequenced model reports the highest score in all four SIGMORPHON metrics: 98.13 percent lemmatization accuracy, 0.02 Levenshtein distance, 93.75 percent morphological accuracy, and 98.24 morphological F-1.
- On the PUD dataset, the separate model beats the SIGMORPHON best in Levenshtein distance, morphological accuracy, and F-1, but both proposed models fall short on lemmatization accuracy (88.84 and 88.25 versus 89.03).
- Feeding predicted morphological tags into the lemmatizer helps on the larger IMST dataset but hurts on the smaller PUD dataset, so the choice between separate and sequenced architectures should depend on the target data.
- A word-meaning-sensitive lemmatizer can resolve ambiguous Turkish words such as 'oku' and 'bakan', whose roots change with sentence context, which spelling-only stemmers cannot do.
Reading between the lines
- Editorial inference: the reported gain on IMST may be substantially due to the pretrained Turkish BERT rather than the architectural novelty; an ablation that replaces BERT with a randomly initialized encoder would separate the two effects.
- Editorial inference: the PUD lemmatization differences are small enough that a paired significance test is needed before the blanket claim of superiority is accepted for that dataset.
- Editorial inference: because the architecture is language-agnostic apart from the BERT model and the morphological-tag lists, a plausible next step is applying the same two-channel design to other agglutinative languages that lack context-sensitive lemmatizers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a Turkish lemmatization and morphological tagging model that combines character-level bidirectional LSTM representations of word spelling with Turkish BERT (DistilBERT) representations of word meaning. Two model variants are described: a "separate" model in which the lemmatizer does not use morphological tag predictions, and a "sequenced" model in which the morphological tag model's outputs are fed into the lemmatizer. The models are trained on the SIGMORPHON 2019 versions of the Universal Dependencies Turkish IMST and PUD datasets, and the reported results are compared with the best SIGMORPHON 2019 results for those datasets. The abstract and conclusion claim that this is the first meaning-sensitive lemmatization study for Turkish and that the proposed models are superior to the SIGMORPHON results on both datasets and almost all metrics.
Significance. If the claims held, the paper would provide a useful reference point for context-aware lemmatization for Turkish and would demonstrate that a BERT-based character-sequence model can improve over shared-task baselines. The paper does use standard competition datasets and the SIGMORPHON evaluation code, which are appropriate and reproducible, and it honestly reports two model variants rather than only the best one. However, the significance is substantially undercut by two problems. First, the novelty claim is contradicted by the paper's own literature review, which describes prior context-sensitive neural lemmatizers evaluated on Turkish. Second, the empirical superiority claim is not established: the comparison is acknowledged to be potentially unfair, no significance tests are reported, and one of the headline metrics (PUD lemmatization accuracy) is actually lower than the SIGMORPHON baseline.
major comments (4)
- [Abstract, Section 2, Section 5] The paper's central novelty claim is internally contradicted by its own literature review. The abstract states that "there is no lemmatization study that is sensitive to word meaning in Turkish" and the conclusion repeats that this is the "first lemmatization model specifically focused on the Turkish language." Yet Section 2 describes Malaviya et al. [3] as a contextual neural lemmatizer that uses a sentence-level bidirectional LSTM and "shares its success in Turkish," and it describes Lematus [13] as a "Context Sensitive Neural Lemmatization" system evaluated on 20 languages. The paper explicitly acknowledges that in [3, 13] "other words in the sentence were also used, similar to our study." These citations directly defeat the claim of being first, and the claim cannot be repaired by a local revision without removing the paper's stated primary contribution.
- [Section 4.3, Tables 1 and 2] The paper's claim that the models "outperform the results of the SIGMORPHON competition in two datasets and in almost all categories" is not fully supported by the reported numbers. On the PUD dataset, the lemmatization accuracy of the separate model (88.84) and the sequenced model (88.25) are both below the SIGMORPHON best (89.03). The abstract's unqualified statement "our models were superior" is therefore inaccurate. In addition, no significance tests, confidence intervals, or repeated-run variances are reported for any difference; the observed gaps on IMST (e.g., 97.70 vs 96.84 lemmatization accuracy) and on PUD morphological accuracy (89.66 vs 87.63) could plausibly fall within run-to-run noise. The authors should either qualify the claim to the specific metrics where the proposed models are higher, or supply statistical support for the superiority claim.
- [Section 2 (comparison fairness) and Section 4.3] The comparison to the SIGMORPHON 2019 results is not a controlled evaluation of the contribution of BERT meaning vectors. The paper itself states that the comparison "may be considered unfair" because the competition systems were designed for multiple languages and did not use a large pretrained transformer, while the proposed model uses Turkish BERT. Because the baseline systems were trained under shared-task restrictions, the reported differences conflate the effect of the model architecture with the effect of access to a large pretrained language model. To support the superiority claim, the authors would need a fair baseline—for example, a context-aware lemmatizer with the same architecture but without BERT, or a BERT-based model trained under the same data and resource constraints as the competition—and significance testing over multiple runs.
- [Section 4.3, Tables 1 and 2] The paper selects the better of the two proposed variants per dataset when summarizing the results: the sequenced model is preferred on IMST and the separate model is preferred on PUD. If this choice is made after inspecting test-set results, the reported "best" outcomes overstate the expected performance of a single deployable model. The authors should either prespecify the variant to be used for each dataset or account for the selection over two models, for example by reporting the mean or by using a corrected significance test. This issue is load-bearing because the "almost all metrics" claim depends on cherry-picking the better variant for each dataset.
minor comments (5)
- [Section 4.3 and Data Availability] The Data Availability section says the datasets are "available at this repository" but no URL is provided; please supply the actual repository link.
- [Section 3] The description of the separate versus sequenced models is repetitive and confusing; for example, the text says "the lemmatization model takes morphological tags as input" in one paragraph and then describes the separate model as independent. A single crisp definition of the two variants, referring to Figure 1, would improve readability.
- [Section 4.2] The training parameters are incompletely specified: hidden sizes, number of LSTM layers, embedding dimensions, dropout, and the precise early-stopping criterion ("under a particular loss value") are not given. Since the paper claims a reproducible model, these details should be reported or the code should be linked.
- [Section 2] The sentence "In our study, the BERT model was used to determine the meaning of words in the sentence [13–16]" cites reference [13] (Lematus, 2018), which does not use BERT; the citation appears misplaced and should be corrected.
- [Tables 1 and 2] The table captions contain a typo ("T able 1"), and the text inconsistently refers to "split model and the merged model" when it means the separate and sequenced models.
Circularity Check
No circularity: the lemmatization and morphological-tagging predictions come from an external train/test evaluation, not from a fitted parameter relabeled as a prediction.
full rationale
The derivation chain is self-contained. The models are trained on the SIGMORPHON 2019 IMST and PUD training splits and evaluated with the shared task's benchmark code against the competition's published best results, which are external numbers. The two model variants (separate and sequenced) differ only in whether morphological-tag predictions are fed into the lemmatizer; both outcomes are reported, so no held-out metric is hidden or constructed. No parameter is fitted to the test set and then renamed a prediction, and no load-bearing claim rests on a self-citation; references [3] and [13] are prior external work by other authors. The paper does contain non-circular weaknesses: the abstract's 'first in the literature' novelty claim is contradicted by the paper's own literature-review descriptions of context-sensitive Turkish lemmatization in [3] and [13], and the PUD lemmatization accuracy in Table 2 (88.84 and 88.25) is below the SIGMORPHON baseline (89.03), so the unqualified 'superior' wording is not supported. The paper itself flags the comparison as possibly unfair. These are correctness and scope concerns, not circular reductions; the central prediction pipeline remains an honest train/test evaluation against external benchmarks.
Assumptions & free parameters
free parameters (6)
- learning_rate_stage1 =
1e-3
- learning_rate_stage2 =
1e-5
- epochs_stage1 =
128
- second_stage_epoch_cutoff =
48
- morphological_tag_order =
sequential/sorted
- morphological_tag_list_per_dataset =
different predetermined lists for IMST and PUD
assumptions (4)
- domain assumption Universal Dependencies IMST and PUD annotations are correct and appropriate for training and evaluation.
- domain assumption Summing BERT subword vectors yields a valid word-meaning vector.
- domain assumption Character-level BiLSTM representations are sufficient for spelling information in root and tag generation.
- domain assumption SIGMORPHON 2019 evaluation metrics correctly measure task success.
Cite this review
Pith. "Pith review of Context Aware Lemmatization and Morphological Tagging Method in Turkish." pith.science (2026). https://pith.science/paper/OZUYBIJ7
@misc{pith2026250102361,
author = {Pith},
title = {Pith review of: Context Aware Lemmatization and Morphological Tagging Method in Turkish},
year = {2026},
howpublished = {\url{https://pith.science/paper/OZUYBIJ7}},
note = {Machine review of arXiv:2501.02361}
}
read the original abstract
The smallest part of a word that defines the word is called a word root. Word roots are used to increase success in many applications since they simplify the word. In this study, the lemmatization model, which is a word root finding method, and the morphological tagging model, which predicts the grammatical knowledge of the word, are presented. The presented model was developed for Turkish, and both models make predictions by taking the meaning of the word into account. In the literature, there is no lemmatization study that is sensitive to word meaning in Turkish. For this reason, the present study shares the model and the results obtained from the model on Turkish lemmatization for the first time in the literature. In the present study, in the lemmatization and morphological tagging models, bidirectional LSTM is used for the spelling of words, and the Turkish BERT model is used for the meaning of words. The models are trained using the IMST and PUD datasets from Universal Dependencies. The results from the training of the models were compared with the results from the SIGMORPHON 2019 competition. The results of the comparisons revealed that our models were superior.
Reference graph
Works this paper leans on
-
[3]
Malaviya, C., Wu, S., Cotterell, R.: A Simple Joint Model for Improved Contextual Neural Lemmatization. arXiv. arXiv:1904.02306 [cs] (2020). http://arxiv.org/abs/1904.02306 Accessed 2022-09-21
work page Pith review arXiv 2020
-
[13]
Bergmanis, T., Goldwater, S.: Context Sensitive Neural Lemmati- zation with Lematus. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1391–1400. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://d...
-
[1]
In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp
M¨ uller, T., Cotterell, R., Fraser, A., Sch¨ utze, H.: Joint Lemmati- zation and Morphological Tagging with Lemming. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2268–2274. Association for Computational Linguis- tics, Lisbon, Portugal (2015). https://doi.org/10.18653/v1/D15-1272. https://aclanthology.org...
-
[2]
Kondratyuk, D., Gavenˇ ciak, T., Straka, M., Hajiˇ c, J.: LemmaTag: Jointly Tagging and Lemmatizing for Morphologically-Rich Languages with BRNNs. arXiv. arXiv:1808.03703 [cs] (2018). http://arxiv.org/abs/1808. 03703 Accessed 2022-09-21
arXiv 2018
-
[4]
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. arXiv:1810.04805 [cs] (2019). http://arxiv.org/abs/1810.04805 Accessed 2022-09-21
arXiv 2019
-
[5]
Long Short-Term Memory
-
[6]
McCarthy, A.D., Vylomova, E., Wu, S., Malaviya, C., Wolf-Sonkin, L., Nicolai, G., Kirov, C., Silfverberg, M., Mielke, S.J., Heinz, J., Cotterell, R., Hulden, M.: The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflection. In: Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonolog...
work page Pith review arXiv 2019
-
[7]
Nivre, J., de Marneffe, M.-C., Ginter, F., Hajiˇ c, J., Manning, C.D., Pyysalo, S., Schuster, S., Tyers, F., Zeman, D.: Universal Depen- dencies v2: An Evergrowing Multilingual Treebank Collection. arXiv. arXiv:2004.10643 [cs] (2020). http://arxiv.org/abs/2004.10643 Accessed 2022-09-21
arXiv 2020
Show all 20 references
-
[8]
http://snowball.tartarus
Snowball: A language for stemming algorithms. http://snowball.tartarus. org/texts/introduction.html Accessed 2022-09-21
2022
-
[9]
An algorithm for suffix stripping (1980)
1980
-
[10]
SiLahtaro˘ glu, O.D.-O.B.A.-G.: Lemmatizer: Akıllı T¨ urk¸ ce K¨ ok Bulma Y¨ ontemi. Turkish Studies-Information Technologies and Applied Sciences Springer Nature 2021 LATEX template Lemmatization and Morphological Tagging Method in Turkish 13 V olume 15 Issue 3(Volume 15 Issu...
2020 doi
-
[11]
Bili¸ sim Teknolojileri Der- gisi, 183–193 (2019)
Can, B.: LSTM A˘ gları ile T¨ urk¸ ce K¨ ok Bulma. Bili¸ sim Teknolojileri Der- gisi, 183–193 (2019). https://doi.org/10.17671/gazibtd.486042. Accessed 2022-09-21
2019 doi
-
[12]
Af, A., Ak, M.D.: Zemberek, an open source NLP framework for Turkic Languages, 9
-
[14]
In: Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonol- ogy, And Morphology, pp
Kondratyuk, D.: Cross-Lingual Lemmatization and Morphology Tag- ging with Two-Stage Multilingual BERT Fine-Tuning. In: Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonol- ogy, And Morphology, pp. 12–18. Association for Computational Lin- guistics, ...
2019 doi
-
[15]
Kondratyuk, D., Straka, M.: 75 Languages, 1 Model: Parsing Universal Dependencies Universally. arXiv. arXiv:1904.02099 [cs] (2019). http:// arxiv.org/abs/1904.02099 Accessed 2022-09-21
2019 arXiv
-
[16]
Straka, M., Strakov´ a, J., Hajiˇ c, J.: UDPipe at SIGMORPHON 2019: Con- textualized Embeddings, Regularization with Morphological Categories, Corpora Merging. arXiv. arXiv:1908.06931 [cs] (2019). http://arxiv.org/ abs/1908.06931 Accessed 2022-09-21
2019 arXiv
-
[17]
original-date: 2020-02-02T18:37:15Z (2022)
Schweter, S.: BERTurk. original-date: 2020-02-02T18:37:15Z (2022). https://github.com/stefan-it/turkish-bert Accessed 2022-09-21
2022
-
[18]
Bahdanau, D., Cho, K., Bengio, Y.: Neural Machine Translation by Jointly Learning to Align and Translate. arXiv. arXiv:1409.0473 [cs, stat] (2016). http://arxiv.org/abs/1409.0473 Accessed 2022-09-21
2016 arXiv
-
[19]
original-date: 2016-08-14T16:12:09Z (2022)
Stenstr¨ om, E.: CoNLL-U Parser. original-date: 2016-08-14T16:12:09Z (2022). https://github.com/EmilStenstrom/conllu Accessed 2022-09-21
2022
-
[20]
Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimiza- tion. arXiv. arXiv:1412.6980 [cs] (2017). http://arxiv.org/abs/1412.6980 Accessed 2022-09-21
2017 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.