{"id":"aba44de6-475d-4513-967d-29fbde4c2ffd","arxiv_id":"2501.02361","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A BiLSTM plus Turkish BERT lemmatizer and morphological tagger is tested on IMST and PUD, beating SIGMORPHON 2019 on most metrics but not on PUD lemmatization accuracy, while the paper's claim of being the first context-aware Turkish lemmatizer is contradicted by its own citations.","lead":"This paper describes a Turkish lemmatization and morphological tagging system that combines a character-level bidirectional LSTM with a pretrained Turkish BERT model. It claims to be the first meaning-sensitive Turkish lemmatizer and says it beats the best SIGMORPHON 2019 results, but the novelty claim is contradicted by the paper's own references and the comparison is not apples-to-apples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own references contradict its central novelty claim: Section 2 describes prior contextual neural lemmatizers with Turkish results, yet the abstract and conclusion assert \"first in the literature.\"","rationale":"The reader's weakest_assumption targets the fairness of the SIGMORPHON comparison, specifically the use of a pretrained Turkish BERT model. That concern is weaker than stated: SIGMORPHON 2019 systems themselves used pretrained contextual embeddings, including multilingual BERT, as the paper's own reference [14] shows. So the BERT-vs-no-BERT asymmetry is not as clear-cut as the reader suggests. However, the reader's rationale also flags the novelty contradiction, which I regard as the decisive issue: the paper's own Section 2 describes prior context-sensitive lemmatization work with Turkish results, directly undercutting the abstract's \"first time in the literature\" claim. The PUD lemmatization numbers in Table 2 (88.84/88.25 vs. 89.03) also make the abstract's unqualified \"superior\" wording inaccurate, and no significance testing is reported. Since the central claim is novelty plus superiority, and the novelty is self-refuted while the superiority is overstated, the correct verdict remains the reader's REJECT. I mark agreement 'partial' because the reader's named weakest assumption is not the same as my load-bearing concern, even though the reader's rationale mentions it.","tokens_in":8716,"tokens_out":7733,"duration_ms":72511,"concrete_test":"Read the published result tables of Malaviya et al. (2019, arXiv:1904.02306), cited as [3], and locate its Turkish (tr) results. The current paper states that [3] \"shares its success in Turkish\" and uses sentence context; if that statement is accurate, the abstract's \"no lemmatization study sensitive to word meaning in Turkish\" is directly refuted. As corroboration, also check whether Bergmanis and Goldwater's Lematus (NAACL 2018) reports Turkish results. This is a literature verification and requires no reimplementation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the abstract's assertion that \"there is no lemmatization study that is sensitive to word meaning in Turkish\" and that this work is \"the first time in the literature.\" That claim is internally contradicted by Section 2. In the Literature Review, the paper describes [3] (Malaviya et al.) as a study that \"shares its success in Turkish\" and uses sentence context via bidirectional LSTM to predict roots, and [13] (Lematus) as a context-sensitive neural lemmatizer evaluated on 20 languages, which the paper groups with Turkish-inclusive multilingual work. The paper then states \"in the two studies mentioned [3, 13] ... other words in the sentence were also used, similar to our study.\" If a prior system already used contextual information to lemmatize Turkish, the \"first meaning-sensitive Turkish lemmatizer\" claim is false regardless of whether it used BERT. This is not a matter of external consensus; it is a soundness failure within the paper's own cited evidence. A secondary empirical issue is that Table 2 shows the proposed models' PUD lemmatization accuracy (88.84/88.25) is below the SIGMORPHON baseline (89.03), so the unqualified \"superior\" wording in the abstract is also inaccurate; no significance test is reported for any of the differences. Because the primary contribution is novelty and the paper's own references defeat it, the central claim does not hold as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a Turkish lemmatization and morphological tagging model that combines character-level bidirectional LSTM representations of word spelling with Turkish BERT (DistilBERT) representations of word meaning. Two model variants are described: a \"separate\" model in which the lemmatizer does not use morphological tag predictions, and a \"sequenced\" model in which the morphological tag model's outputs are fed into the lemmatizer. The models are trained on the SIGMORPHON 2019 versions of the Universal Dependencies Turkish IMST and PUD datasets, and the reported results are compared with the best SIGMORPHON 2019 results for those datasets. The abstract and conclusion claim that this is the first meaning-sensitive lemmatization study for Turkish and that the proposed models are superior to the SIGMORPHON results on both datasets and almost all metrics.","tokens_in":8985,"tokens_out":4525,"duration_ms":42234,"significance":"If the claims held, the paper would provide a useful reference point for context-aware lemmatization for Turkish and would demonstrate that a BERT-based character-sequence model can improve over shared-task baselines. The paper does use standard competition datasets and the SIGMORPHON evaluation code, which are appropriate and reproducible, and it honestly reports two model variants rather than only the best one. However, the significance is substantially undercut by two problems. First, the novelty claim is contradicted by the paper's own literature review, which describes prior context-sensitive neural lemmatizers evaluated on Turkish. Second, the empirical superiority claim is not established: the comparison is acknowledged to be potentially unfair, no significance tests are reported, and one of the headline metrics (PUD lemmatization accuracy) is actually lower than the SIGMORPHON baseline.","major_comments":[{"comment":"The paper's central novelty claim is internally contradicted by its own literature review. The abstract states that \"there is no lemmatization study that is sensitive to word meaning in Turkish\" and the conclusion repeats that this is the \"first lemmatization model specifically focused on the Turkish language.\" Yet Section 2 describes Malaviya et al. [3] as a contextual neural lemmatizer that uses a sentence-level bidirectional LSTM and \"shares its success in Turkish,\" and it describes Lematus [13] as a \"Context Sensitive Neural Lemmatization\" system evaluated on 20 languages. The paper explicitly acknowledges that in [3, 13] \"other words in the sentence were also used, similar to our study.\" These citations directly defeat the claim of being first, and the claim cannot be repaired by a local revision without removing the paper's stated primary contribution.","section":"Abstract, Section 2, Section 5"},{"comment":"The paper's claim that the models \"outperform the results of the SIGMORPHON competition in two datasets and in almost all categories\" is not fully supported by the reported numbers. On the PUD dataset, the lemmatization accuracy of the separate model (88.84) and the sequenced model (88.25) are both below the SIGMORPHON best (89.03). The abstract's unqualified statement \"our models were superior\" is therefore inaccurate. In addition, no significance tests, confidence intervals, or repeated-run variances are reported for any difference; the observed gaps on IMST (e.g., 97.70 vs 96.84 lemmatization accuracy) and on PUD morphological accuracy (89.66 vs 87.63) could plausibly fall within run-to-run noise. The authors should either qualify the claim to the specific metrics where the proposed models are higher, or supply statistical support for the superiority claim.","section":"Section 4.3, Tables 1 and 2"},{"comment":"The comparison to the SIGMORPHON 2019 results is not a controlled evaluation of the contribution of BERT meaning vectors. The paper itself states that the comparison \"may be considered unfair\" because the competition systems were designed for multiple languages and did not use a large pretrained transformer, while the proposed model uses Turkish BERT. Because the baseline systems were trained under shared-task restrictions, the reported differences conflate the effect of the model architecture with the effect of access to a large pretrained language model. To support the superiority claim, the authors would need a fair baseline—for example, a context-aware lemmatizer with the same architecture but without BERT, or a BERT-based model trained under the same data and resource constraints as the competition—and significance testing over multiple runs.","section":"Section 2 (comparison fairness) and Section 4.3"},{"comment":"The paper selects the better of the two proposed variants per dataset when summarizing the results: the sequenced model is preferred on IMST and the separate model is preferred on PUD. If this choice is made after inspecting test-set results, the reported \"best\" outcomes overstate the expected performance of a single deployable model. The authors should either prespecify the variant to be used for each dataset or account for the selection over two models, for example by reporting the mean or by using a corrected significance test. This issue is load-bearing because the \"almost all metrics\" claim depends on cherry-picking the better variant for each dataset.","section":"Section 4.3, Tables 1 and 2"}],"minor_comments":[{"comment":"The Data Availability section says the datasets are \"available at this repository\" but no URL is provided; please supply the actual repository link.","section":"Section 4.3 and Data Availability"},{"comment":"The description of the separate versus sequenced models is repetitive and confusing; for example, the text says \"the lemmatization model takes morphological tags as input\" in one paragraph and then describes the separate model as independent. A single crisp definition of the two variants, referring to Figure 1, would improve readability.","section":"Section 3"},{"comment":"The training parameters are incompletely specified: hidden sizes, number of LSTM layers, embedding dimensions, dropout, and the precise early-stopping criterion (\"under a particular loss value\") are not given. Since the paper claims a reproducible model, these details should be reported or the code should be linked.","section":"Section 4.2"},{"comment":"The sentence \"In our study, the BERT model was used to determine the meaning of words in the sentence [13–16]\" cites reference [13] (Lematus, 2018), which does not use BERT; the citation appears misplaced and should be corrected.","section":"Section 2"},{"comment":"The table captions contain a typo (\"T able 1\"), and the text inconsistently refers to \"split model and the merged model\" when it means the separate and sequenced models.","section":"Tables 1 and 2"}],"recommendation":"reject","confidential_remarks":"The paper's stated primary contribution—being the first meaning-sensitive Turkish lemmatizer—is contradicted by its own literature review, and the empirical superiority claim rests on an acknowledged unfair comparison and lacks statistical support. These are central, not peripheral, issues. The manuscript may be more suitable as a technical report or an archival dataset/model release, but as a journal article carrying the current claims it does not meet the bar. I would encourage the authors to reframe the contribution around the specific BERT-based architecture and to add fair baselines and significance testing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper reports a clean enough BiLSTM+BERT lemmatizer and morphological tagger for Turkish with new numbers on IMST and PUD, but its central claim — that it is the first meaning-sensitive Turkish lemmatizer — is contradicted by its own literature review, and the main comparison is not as fair as the numbers suggest. I would not take the abstract at face value.\n\nWhat is actually new and done well: the authors build a reasonable encoder-decoder with character-level BiLSTM for spelling, Turkish BERT for word meaning, and optionally feed predicted morphological tags into the lemmatizer. They compare separate versus sequenced variants across four SIGMORPHON metrics on two Turkish treebanks, and they are honest that the comparison to multilingual SIGMORPHON systems 'may be considered unfair.' They also correctly note that injecting morphological tags helps on IMST but hurts on PUD. Those empirical results are new and the numbers are plausible. The code is not actually shared, despite the Data Availability statement being a vague form sentence.\n\nThe soft spots are larger than the paper concedes. First, the novelty claim is internally defeated. Section 2 describes [3] Malaviya et al. and [13] Lematus as context-sensitive lemmatizers with Turkish results, states that, like the present study, they used other words in the sentence, and then the abstract and conclusion still say there is no meaning-sensitive Turkish lemmatization study in the literature. That is an internal contradiction, not a matter of external opinion. Second, Table 2 shows the proposed models' PUD lemmatization accuracy (88.84/88.25) is below the SIGMORPHON baseline (89.03), so the unqualified 'superior' wording is inaccurate. There are no significance tests or repeated runs, and the fine-tuned external BERT gives the model an advantage over shared-task systems that could not use such resources. Minor issues: some duplicated sentences in Section 3 and sloppy reference formatting.\n\nWho is this for? Readers working on Turkish morphology might take the reported numbers as a reference point, but only after the claims are trimmed. The paper's main value is as a data point that a fairly standard BERT+BiLSTM setup gets about 97–98% lemmatization accuracy on IMST and around 89% on PUD. That is incrementally useful.\n\nOn peer review: it deserves a serious referee, mostly so the novelty claim is forced to be corrected or removed and so the comparison is analyzed properly. If I were the editor, I would send it out with a clear expectation of heavy revision.","headline":"A straightforward BERT+BiLSTM Turkish lemmatizer with new numbers, but its central 'first' claim is contradicted by its own references and the comparison is less fair than advertised.","tokens_in":9556,"tokens_out":2785,"would_cite":false,"duration_ms":24980,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding word meaning from Turkish BERT to character-level LSTMs yields the first context-aware Turkish lemmatizer and beats SIGMORPHON 2019 baselines on IMST and most PUD metrics.","keywords":["lemmatization","morphological tagging","Turkish","BERT","bidirectional LSTM","context-aware NLP","Universal Dependencies","SIGMORPHON 2019"],"falsifier":"A controlled ablation that removes the BERT word-meaning vector from the model while keeping all other components and hyperparameters fixed, then reruns the same IMST and PUD splits, would settle whether context is the source of the gains; if the spelling-only variant matches the full model, the central claim that meaning sensitivity drives the improvement fails.","tokens_in":8461,"feed_emoji":"🔤","tokens_out":11112,"duration_ms":96425,"temperature":0.7,"pith_summary":"This paper tries to establish that Turkish lemmatization and morphological tagging improve when a model is sensitive to word meaning, not just spelling. The authors combine character-level bidirectional LSTMs with a pretrained Turkish BERT model, and they train two variants on the Universal Dependencies IMST and PUD Turkish datasets: a separate model and a sequenced model that feeds predicted morphological tags into the lemmatizer. They report results that beat the best SIGMORPHON 2019 systems in almost all metrics, with the sequenced model reaching 98.13 percent lemmatization accuracy on IMST. If true, this would give Turkish a context-aware lemmatizer for the first time and would help applications that need to resolve ambiguous words whose root depends on sentence meaning. The authors note that the comparison may be considered unfair because the competition systems were built for many languages rather than Turkish only.","feed_headline":"Context-aware Turkish lemmatizer beats most SIGMORPHON 2019 scores","feed_subtitle":"Word meaning from Turkish BERT lifts lemmatization accuracy to 98.13% on IMST and beats most PUD metrics.","key_machinery":"The central mechanism is an encoder-decoder network with two parallel representation channels for each word: a character-level bidirectional LSTM builds a spelling vector from the word's letters, and a frozen-then-fine-tuned Turkish BERT model supplies a meaning vector from the surrounding sentence. In the sequenced variant, an additional embedding-plus-BiLSTM channel encodes the morphological tag predictions; the three vectors, or two in the separate variant, are projected through dense layers into one vector that initializes the decoder's hidden state. The decoder uses two forward LSTM layers with teacher forcing and Bahdanau attention over the character-level LSTM outputs to generate morphological tags or root-word letters one step at a time.","core_discovery":"The paper's central claim is that a Turkish-specific lemmatizer and morphological tagger that combines a word-meaning vector from a pretrained Turkish BERT model with a character-level bidirectional LSTM spelling representation outperforms the best results of the SIGMORPHON 2019 shared task on the Universal Dependencies Turkish IMST and PUD datasets in almost every reported metric. On IMST, the sequenced model, which feeds morphological tag predictions into the lemmatizer, reaches 98.13 percent lemmatization accuracy, 0.02 Levenshtein distance, 93.75 percent morphological tagging accuracy, and 98.24 F-1; the separate and sequenced models share the same morphological tagging numbers because the tagger is trained once and frozen. On PUD, the separate model reports 88.84 percent lemmatization accuracy, 0.12 Levenshtein distance, 89.66 percent morphological accuracy, and 96.41 F-1. The authors conclude that feeding morphological tags into lemmatization helps on IMST but hurts on PUD, and they present the work as the first Turkish lemmatization study that is sensitive to word meaning, while conceding that the comparison against the multilingual competition systems may be considered unfair.","pith_inferences":["Editorial inference: the reported gain on IMST may be substantially due to the pretrained Turkish BERT rather than the architectural novelty; an ablation that replaces BERT with a randomly initialized encoder would separate the two effects.","Editorial inference: the PUD lemmatization differences are small enough that a paired significance test is needed before the blanket claim of superiority is accepted for that dataset.","Editorial inference: because the architecture is language-agnostic apart from the BERT model and the morphological-tag lists, a plausible next step is applying the same two-channel design to other agglutinative languages that lack context-sensitive lemmatizers."],"forward_implications":["On the IMST dataset, the sequenced model reports the highest score in all four SIGMORPHON metrics: 98.13 percent lemmatization accuracy, 0.02 Levenshtein distance, 93.75 percent morphological accuracy, and 98.24 morphological F-1.","On the PUD dataset, the separate model beats the SIGMORPHON best in Levenshtein distance, morphological accuracy, and F-1, but both proposed models fall short on lemmatization accuracy (88.84 and 88.25 versus 89.03).","Feeding predicted morphological tags into the lemmatizer helps on the larger IMST dataset but hurts on the smaller PUD dataset, so the choice between separate and sequenced architectures should depend on the target data.","A word-meaning-sensitive lemmatizer can resolve ambiguous Turkish words such as 'oku' and 'bakan', whose roots change with sentence context, which spelling-only stemmers cannot do."],"supporting_citations":[{"why":"Defines the BERT architecture whose Turkish variant supplies the word-meaning vectors at the core of the context-aware model.","marker":"[4]"},{"why":"The SIGMORPHON 2019 shared task that defines the task, datasets, metrics, and best comparison results used in Tables 1 and 2.","marker":"[6]"},{"why":"The Universal Dependencies collection from which the IMST and PUD Turkish datasets are drawn.","marker":"[7]"},{"why":"Supplies the pretrained Turkish BERT model used as the frozen-then-fine-tuned meaning encoder.","marker":"[17]"},{"why":"The LSTM units used in the character-level spelling encoder and in the decoder for generating tags and root letters.","marker":"[5]"},{"why":"The Bahdanau attention mechanism used in the decoder over character-level encoder outputs.","marker":"[18]"},{"why":"A prior contextual joint lemmatization-and-tagging model whose finding that tag input does not always help motivates the separate-versus-sequenced design.","marker":"[3]"},{"why":"Context-sensitive neural lemmatizer used as the nearest prior contextual baseline for the comparison.","marker":"[13]"}],"fun_headline_variants":["Turkish BERT-powered lemmatizer tops SIGMORPHON 2019","First meaning-aware Turkish lemmatizer beats prior systems","Context-aware model sets new bar for Turkish lemmatization","BiLSTM + Turkish BERT achieve 98% lemmatization accuracy","Word meaning lifts Turkish lemmatization to 98.13%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of superiority rests on the assumption that comparing a Turkish-only model equipped with a large pretrained Turkish BERT to the multilingual SIGMORPHON 2019 systems is a meaningful basis for comparison; the paper itself acknowledges the comparison may be considered unfair.","fun_headline_variants_meta":{"raw":{"variants":["Turkish BERT-powered lemmatizer tops SIGMORPHON 2019","First meaning-aware Turkish lemmatizer beats prior systems","Context-aware model sets new bar for Turkish lemmatization","BiLSTM + Turkish BERT achieve 98% lemmatization accuracy","Word meaning lifts Turkish lemmatization to 98.13%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000537,"raw_usage":{"total_tokens":2619,"prompt_tokens":1025,"completion_tokens":1594,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":1498}},"tokens_in":641,"tokens_out":1594,"duration_ms":10509,"temperature":1.0,"reasoning_tokens":1498,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:13:27.978291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled ablation that removes the BERT word-meaning vector from the model while keeping all other components and hyperparameters fixed, then reruns the same IMST and PUD splits, would settle whether context is the source of the gains; if the spelling-only variant matches the full model, the central claim that meaning sensitivity drives the improvement fails.","supporting_citations":[{"cited_title":"The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflection","cited_arxiv_id":"1910.11493","evidence_quote":"The SIGMORPHON 2019 shared task that defines the task, datasets, metrics, and best comparison results used in Tables 1 and 2."},{"cited_title":"original-date: 2020-02-02T18:37:15Z (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained Turkish BERT model used as the frozen-then-fine-tuned meaning encoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The LSTM units used in the character-level spelling encoder and in the decoder for generating tags and root letters."},{"cited_title":"A Simple Joint Model for Improved Contextual Neural Lemmatization","cited_arxiv_id":"1904.02306","evidence_quote":"A prior contextual joint lemmatization-and-tagging model whose finding that tag input does not always help motivates the separate-versus-sequenced design."},{"cited_title":"In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp","cited_arxiv_id":null,"evidence_quote":"Context-sensitive neural lemmatizer used as the nearest prior contextual baseline for the comparison."}],"review_version":1}