{"id":"f9ce76db-f363-43e2-a513-355df6c175d5","arxiv_id":"1908.08528","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An unsupervised lemmatization method that clusters word forms with a distance measure combining Jaro-Winkler edit distance and FastText embedding cosine similarity surpasses simple baselines on most of 28 Universal Dependencies treebanks.","lead":"This paper proposes a fully unsupervised way to group inflected word forms into lemmas by combining string similarity with word-embedding similarity and clustering the results. It reports improvements over simple baselines on 23 of 28 multilingual datasets, though the method only outputs clusters, not final lemma labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 23/28 win count may be an artifact of token-based evaluation, which weights exactly the frequent word forms the method handles best.","rationale":"I agree with the reader's conditional assessment, but I identify a different load-bearing concern. The reader's weakest assumption is the K=3 stem-based hypercluster partition (Section 3.1), which prevents cross-prefix merges and caps the oracle upper bound below 100%. This is a genuine limitation, but it is explicitly acknowledged by the authors and does not by itself invalidate the claim of surpassing a weak baseline; even with the hypercluster constraint, the method can still beat 'form' or '5-prefix' on many datasets, as reported. In contrast, the token-based versus type-based evaluation choice is not acknowledged as a potential source of bias, and it bears directly on the '23 of 28' win count. Token-based weighting heavily favors high-frequency word forms, which are precisely the forms where the method has all its advantages: presence in the FastText 100k vocabulary, dense embeddings, subword similarity, and reliable OOV handling. Rare forms, which make up the long tail of the vocabulary, are more likely to be OOV and are assigned by a crude single-step rule that does little better than the baseline. A type-based v-measure would treat each word form equally, so the method's win rate over 28 datasets could decrease markedly. The paper's own oracle upper bound is also computed type-based? It is not clear; the evaluation section only says token-based for the main results. This omission makes the central claim fragile to a reasonable alternative metric. Since the concern is addressable by re-evaluation rather than by a fundamental flaw, I keep the reader's conditional verdict. I have not downgraded to reject because there is no internal inconsistency and the method likely does provide some real improvement; the paper simply needs to show that the headline result is not an artifact of the evaluation unit. I also note the reader's other concerns (lack of significance tests, cs_pdt inclusion in reported results) but these are secondary; the evaluation-weighting concern is the most load-bearing because it can flip the central claim. Agreement is partial because the reader identified hyperclusters as the weakest structural link, while I see the metric choice as more directly tied to the central empirical claim; both are legitimate and complementary.","tokens_in":71,"tokens_out":14451,"duration_ms":259301,"concrete_test":"Recompute the reported v-measure error (1-vmeasure) for all 28 treebanks using type-based weighting, where each distinct word form in the dev set contributes exactly one count, using the same clustering output (or the same clustering algorithm with the same hyperparameters) and the same gold lemmas. Then recount the number of datasets where the method beats the per-dataset baseline, and recompute the median error reduction. If the win count drops appreciably (e.g., to below 18 of 28) or the median error reduction becomes close to zero, the central claim is not robust to the choice of evaluation unit and should be substantially qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, surpassing the baseline on 23 of 28 datasets (Section 4.3, Table 2), rests on a token-based v-measure evaluation: repeated word forms are counted repeatedly (Section 4.1). This choice systematically favors the method because token frequency correlates with the conditions under which the method works well: frequent forms are more likely to be within the 100k FastText vocabulary, have reliable embedding similarity, and avoid the crude OOV single-step rule (Section 3). Rare forms, by contrast, are more likely to be handled by a nearest-cluster-or-singleton heuristic that is barely better than the 'form' baseline. Since the clustering unit is a word type, a type-based v-measure (each distinct form counted once) is an equally natural and arguably more appropriate evaluation of the clustering itself. Under type-based weighting, the method's advantage is likely to shrink because the many rare, hard forms contribute equally. The paper provides no justification for token-based evaluation and no type-based results, so we cannot tell whether the headline 23/28 count is robust to this reasonable alternative metric. The reader's focus on hyperclusters is a real limitation, but it is openly acknowledged and bounded by the oracle upper bound; the evaluation metric directly determines whether the central claim survives an equally defensible re-analysis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an unsupervised lemmatization method that clusters word-form types using a distance measure combining Jaro-Winkler edit distance and cosine similarity of FastText embeddings (Eq. 1), followed by agglomerative clustering with average linkage and a fixed distance threshold t=0.4. A crude K=3 prefix stem partitions the vocabulary into hyperclusters to make clustering tractable. The method is evaluated on 28 Universal Dependencies treebanks covering 23 languages, with token-based v-measure as the evaluation metric. The paper reports that the method beats a per-dataset baseline of 'form' or 'first 5 characters' on 23 of 28 datasets, with a median error reduction of 23%. The authors explicitly acknowledge several limitations: the hyperparameters were tuned on cs_pdt, the oracle upper bound is below 100% because of the prefix-stem constraint, and the method fails on isolating languages and on several difficult datasets.","tokens_in":5943,"tokens_out":9084,"duration_ms":84771,"significance":"The paper makes a simple, language-agnostic proposal that could be useful for low-resource languages where annotated data are unavailable but pretrained embeddings exist. Its evaluation is unusually broad, covering 28 treebanks and 23 languages, and the authors are transparent about limitations: they report an oracle upper bound, disclose that hyperparameters were tuned on Czech, and identify the worst-performing language groups. The source code is promised to be released. If the empirical claims withstand scrutiny, the paper provides a practical baseline and a useful analysis of where such embedding-plus-string clustering succeeds and fails. However, the headline 23/28 result is currently tied to a token-based evaluation choice and to a results table that includes the tuning treebank, so the central claim needs additional support before it can be accepted at face value.","major_comments":[{"comment":"The hyperparameters t=0.4, K=3, and N=100,000 were tuned on the cs_pdt treebank, and cs_pdt is included in the headline results in Table 2. This conflates tuning and evaluation. The authors should report the results with cs_pdt excluded (or explicitly designated as development) and verify that the 23/28 count and the 23% median error reduction are unchanged. The disclosure in Section 4.1 is honest, but it does not by itself resolve the problem for the reported numbers.","section":"§4.1, Table 2"},{"comment":"The evaluation is token-based, counting each occurrence of a word form, even though the clustering unit is a word-form type. Because the method's failures are likely concentrated in rare and out-of-vocabulary forms, which are handled by a crude nearest-cluster-or-singleton heuristic (Table 1), token-based weighting may inflate the method's apparent advantage over the 'form' baseline. The authors should report type-based v-measure results as well, or provide a task-level justification for token-based weighting. Without this, the central 23/28 empirical claim is not robust to an equally natural re-evaluation.","section":"§4.1, Table 2"},{"comment":"No significance tests, error bars, or confidence intervals are reported. Given the large negative outlier for ko kaist (-6392.8) and heterogeneous per-language effect sizes, the authors should add a sign test and a Wilcoxon signed-rank test over the 28 datasets, or bootstrap confidence intervals for the median error reduction, to substantiate the claim that the method beats the baseline beyond a few favorable datasets.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"Equation (1) is printed as 'dist(a, b) = 1 − J W(a, b) · cos(a, b) + 1 2', which is ambiguous; as written it could be read as 1 - JW·cos + 1/2. Add parentheses to make the intended shift clear: dist(a,b) = 1 - JW(a,b)·(cos(a,b)+1)/2.","section":"§2.3, Eq. (1)"},{"comment":"The error-reduction value for ko kaist is reported as -6392.8, but using the formula implied in Section 4.2 with the rounded values in the same row (baseline 0.14, ours 2.41, upper 0.11) gives approximately -7567. Please correct the typo or state the exact formula used.","section":"§4.2, Table 2"},{"comment":"The baseline is selected per dataset as the better of the 'form' and 'first 5 characters' baselines. Please state explicitly whether this selection was made on the evaluation split itself; if so, this makes the comparison conservative for the proposed method, which is a useful fact to report.","section":"§4.2, Table 2"},{"comment":"The 'Average' row is dominated by the Korean outlier and should probably be removed or replaced with the median. The text already notes that the average is not meaningful, but keeping the row invites misinterpretation.","section":"§4.3, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and the idea is simple but plausible. The main risks are the token-based evaluation and the inclusion of the tuning treebank in the headline results; both are fixable. If the authors provide the requested reanalysis, the paper could be a solid short contribution. The scope fits a general NLP or CL venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this is a simple, honest paper about unsupervised lemmatization. It clusters word forms using a distance that multiplies Jaro-Winkler string similarity with FastText cosine similarity, and reports beating a per-dataset 'form or first-5-chars' baseline on 23 of 28 UD treebanks. The method is not a breakthrough, and the authors say so themselves — they call it 'too weak' — but the experiment is broad and the result is plausible.\n\nThe new part is the product distance. Neither component is new, but combining them for lemmatization and testing across 28 datasets is a fair contribution. The paper also compares each distance alone and shows the combination helps. The evaluation is transparent in important places: they include an oracle upper bound that never reaches 100% because of their stem-based hyperclusters (first three characters), and they list failure cases like suppletion and homonymy.\n\nThe soft spots are mostly in the measurement. The headline 23/28 uses token-based v-measure, so frequent word forms count multiple times. That systematically favors the method, because frequent forms are the ones inside the FastText vocabulary with reliable embeddings. Rare forms are handled by a crude nearest-cluster-or-singleton rule, and under type-based evaluation they would get equal weight. The paper never reports type-based results, so we cannot tell if the win count survives. That is the strongest concern and it is easy to address.\n\nOther issues are minor. The threshold t=0.4, prefix length K=3, and vocabulary size N=100k were tuned on cs_pdt, which is still in the table; that is one dataset out of 28. The baseline is the better of two simple rules chosen on the same split, which is if anything conservative. There are no significance tests or error bars, but the effect is large on several languages.\n\nWho is this for? Anyone looking for a cheap unsupervised baseline for lemmatization or morphology clustering. It is not a state-of-the-art system. It deserves a serious referee, but the review should ask for type-based results and, if cheap, a confidence interval. I would not cite it in my own work in the next year, but I would bring it to a reading group to discuss evaluation choices.\n\nRecommendation: send it to peer review, with the request that the authors add a type-based re-analysis. That is the one number that could change the paper's conclusion.","headline":"Small, honest baseline paper; the token-based evaluation is the only thing that could sink it.","tokens_in":6472,"tokens_out":4607,"would_cite":false,"duration_ms":45516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A distance combining edit similarity and word-embedding similarity beats simple baselines for unsupervised lemmatization on 23 of 28 treebanks.","keywords":["unsupervised lemmatization","word form clustering","agglomerative clustering","word embeddings","Jaro-Winkler distance","cosine similarity","Universal Dependencies","v-measure"],"falsifier":"Rerun the released pipeline on the same 28 treebank datasets with the same pretrained embeddings and hyperparameters ($K=3$, $t=0.4$, average linkage). The central claim predicts positive error reduction on at least 23 datasets and a median near 23%; a faithful rerun yielding far fewer wins or a median near zero would show that the reported advantage of the combined distance does not hold.","tokens_in":5479,"feed_emoji":"🧩","tokens_out":12729,"duration_ms":112413,"temperature":0.7,"pith_summary":"Lemmatization normally needs annotated training data, but the paper sets out to show it can be done unsupervised by treating it as clustering of word forms. The proposed distance between two word forms multiplies a Jaro-Winkler string similarity, which emphasizes the beginnings of words where inflections rarely occur, with a shifted cosine similarity of pretrained word embeddings, used as a proxy for meaning. Agglomerative clustering with this distance beats the stronger of two simple baselines on 23 of 28 treebank datasets, with a median error reduction of 23%. This matters because only about 1% of the world's languages have annotated lemmatization data, and rule-based stemmers are either language-specific or merge distinct lemmas too coarsely, so a language-independent clustering method would extend lemmatization to low-resource languages.","feed_headline":"Unsupervised lemmatization beats baseline on 23 of 28 languages","feed_subtitle":"Edit distance plus word-embedding similarity cuts clustering error by a median of 23 percent, with no annotated data.","key_machinery":"The load-bearing object is the combined distance $dist(a,b) = 1 - JW(a,b)\\cdot(\\cos(a,b)+1)/2$: the paper multiplies a Jaro-Winkler string similarity (a string comparator that weights early-character matches more heavily) by the cosine similarity of pretrained fastText word embeddings, shifted from $[-1,1]$ to $[0,1]$. The multiplication is essential because the paper argues that both string and meaning similarity must hold for two forms to be considered inflections of the same lemma. The second mechanism is the hypercluster partition: word forms are grouped by the first three characters of their simplified form, and agglomerative clustering with average linkage runs inside each hypercluster, making the prefix a hard barrier that no merged cluster can cross. The threshold $t=0.4$ controls how eagerly clusters merge: too low leaves tense variants separate, too high merges derivationally related but distinct words.","core_discovery":"The paper's central claim is that inflections of the same word tend to be similar both in spelling and in meaning, so a distance that combines both signals can support unsupervised lemmatization. Concretely, the authors define $dist(a,b) = 1 - JW(a,b)\\cdot(\\cos(a,b)+1)/2$, where $JW$ is a Jaro-Winkler similarity over simplified word forms and $\\cos$ is the cosine similarity of subword-aware word embeddings. Applying average-linkage agglomerative clustering with this distance, stopping when the cluster distance exceeds $t=0.4$, and pre-partitioning the vocabulary into hyperclusters by the first three characters of each form, they report lower clustering error than the better of two baselines (form-as-lemma or five-character prefix) on 23 of 28 datasets. The median error reduction is 23%, with the largest gains on Slavic languages and losses on analytic languages such as Japanese and Korean, which have almost no inflection to exploit.","pith_inferences":["Beyond the paper: because the distance is multiplicative, a pair is merged only if it is close on both axes; on rare or out-of-vocabulary words, where embeddings are unreliable, the string component cannot rescue the product, so performance should degrade sharply.","Beyond the paper: varying the hypercluster stem length $K$ should trace a U-shaped error curve, and the optimal $K$ could be estimated per language from the distribution of prefix-sharing forms, making the method adaptive rather than fixed at $K=3$.","Beyond the paper: the same two-signal distance could be applied to other form-grouping tasks, such as discovering inflectional paradigms in a raw corpus or normalizing historical spellings, where form varies systematically and meaning is stable.","Beyond the paper: the failures on Japanese and Korean suggest a boundary condition: when the gold clusters are mostly singletons, any nonzero merge threshold creates error, so an unsupervised method needs a prior that separates inflectional from isolating languages."],"forward_implications":["An unsupervised lemmatizer for a new language can be built from an unannotated corpus plus pretrained embeddings, with no rule engineering or annotated data; the paper demonstrates this on 28 treebanks across 23 languages.","The combined distance is stronger than either component alone: the paper's comparison reports average error of 8.17 for Jaro-Winkler alone, 4.39 for cosine alone, and 3.77 for the product.","The method is most effective on suffixing, morphologically rich languages such as the Slavic ones, where error reductions reach about 50%, while analytic languages with little inflection are better handled by the simple form baseline.","Since word forms are clustered without context, homonymous forms cannot be separated; the paper identifies contextual embeddings as the natural next step."],"supporting_citations":[{"why":"Supplies the Jaro-Winkler string similarity, the string-side component of the combined distance.","marker":"Winkler, 1990"},{"why":"Establishes that word-embedding cosine similarity captures morphological, syntactic, and semantic similarity, motivating it as the meaning proxy.","marker":"Mikolov et al., 2013"},{"why":"Provides the pretrained subword-aware word embeddings used to compute cosine similarity for all 23 languages.","marker":"Grave et al., 2018"},{"why":"Supplies the average-linkage agglomerative clustering implementation in which the proposed distance is embedded.","marker":"Pedregosa et al., 2011"},{"why":"Defines the Universal Dependencies 2.3 treebanks that provide gold lemmas and the 28 evaluation datasets.","marker":"Nivre et al., 2018"}],"fun_headline_variants":["Embedding+edit distance clusters lemmas sans labels","Unsupervised lemma clustering wins on 23 of 28 datasets","No annotated data: meaning+string similarity lemmatizes","Word form clustering with embeddings: 23/28 wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every inflection of a lemma shares the same first three characters of the simplified word form; word forms that violate this land in different hyperclusters and can never be merged, so the correct clustering is unreachable for them.","fun_headline_variants_meta":{"raw":{"variants":["Embedding+edit distance clusters lemmas sans labels","Unsupervised lemma clustering wins on 23 of 28 datasets","No annotated data: meaning+string similarity lemmatizes","Word form clustering with embeddings: 23/28 wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000504,"raw_usage":{"total_tokens":2416,"prompt_tokens":859,"completion_tokens":1557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":1488}},"tokens_in":475,"tokens_out":1557,"duration_ms":14946,"temperature":1.0,"reasoning_tokens":1488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:36:48.492331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the released pipeline on the same 28 treebank datasets with the same pretrained embeddings and hyperparameters ($K=3$, $t=0.4$, average linkage). The central claim predicts positive error reduction on at least 23 datasets and a median near 23%; a faithful rerun yielding far fewer wins or a median near zero would show that the reported advantage of the combined distance does not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Universal Dependencies 2.3 treebanks that provide gold lemmas and the 28 evaluation datasets."}],"review_version":1}