Unsupervised Lemmatization as Embeddings-Based Word Clustering

Rudolf Rosa; Zden\v{e}k \v{Z}abokrtsk\'y

REVIEW 3 major objections 4 minor 14 references

Unsupervised Lemmatization as Embeddings-Based Word Clustering

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A distance combining edit similarity and word-embedding similarity beats simple baselines for unsupervised lemmatization on 23 of 28 treebanks.

desk verdict Small, honest baseline paper; the token-based evaluation is the only thing that could sink it. read the letter →

arxiv 1908.08528 v1 pith:HXECV7NI submitted 2019-08-22 cs.CL

Rudolf Rosa , Zdeněk Žabokrtský This is my paper · ORCID

classification cs.CL

keywords unsupervisedlemmatizationwordformclusteringagglomerativeembeddingsJaro-WinklerdistancecosinesimilarityUniversalDependenciesv-measure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Lemmatization normally needs annotated training data, but the paper sets out to show it can be done unsupervised by treating it as clustering of word forms. The proposed distance between two word forms multiplies a Jaro-Winkler string similarity, which emphasizes the beginnings of words where inflections rarely occur, with a shifted cosine similarity of pretrained word embeddings, used as a proxy for meaning. Agglomerative clustering with this distance beats the stronger of two simple baselines on 23 of 28 treebank datasets, with a median error reduction of 23%. This matters because only about 1% of the world's languages have annotated lemmatization data, and rule-based stemmers are either language-specific or merge distinct lemmas too coarsely, so a language-independent clustering method would extend lemmatization to low-resource languages.

What carries the argument

The load-bearing object is the combined distance $dist(a,b) = 1 - JW(a,b)\cdot(\cos(a,b)+1)/2$: the paper multiplies a Jaro-Winkler string similarity (a string comparator that weights early-character matches more heavily) by the cosine similarity of pretrained fastText word embeddings, shifted from $[-1,1]$ to $[0,1]$. The multiplication is essential because the paper argues that both string and meaning similarity must hold for two forms to be considered inflections of the same lemma. The second mechanism is the hypercluster partition: word forms are grouped by the first three characters of their simplified form, and agglomerative clustering with average linkage runs inside each hypercluster, making the prefix a hard barrier that no merged cluster can cross. The threshold $t=0.4$ controls how eagerly clusters merge: too low leaves tense variants separate, too high merges derivationally related but distinct words.

What would settle it

Rerun the released pipeline on the same 28 treebank datasets with the same pretrained embeddings and hyperparameters ($K=3$, $t=0.4$, average linkage). The central claim predicts positive error reduction on at least 23 datasets and a median near 23%; a faithful rerun yielding far fewer wins or a median near zero would show that the reported advantage of the combined distance does not hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that inflections of the same word tend to be similar both in spelling and in meaning, so a distance that combines both signals can support unsupervised lemmatization. Concretely, the authors define $dist(a,b) = 1 - JW(a,b)\cdot(\cos(a,b)+1)/2$, where $JW$ is a Jaro-Winkler similarity over simplified word forms and $\cos$ is the cosine similarity of subword-aware word embeddings. Applying average-linkage agglomerative clustering with this distance, stopping when the cluster distance exceeds $t=0.4$, and pre-partitioning the vocabulary into hyperclusters by the first three characters of each form, they report lower clustering error than the better of two baselines (form-as-lemma or five-character prefix) on 23 of 28 datasets. The median error reduction is 23%, with the largest gains on Slavic languages and losses on analytic languages such as Japanese and Korean, which have almost no inflection to exploit.

Load-bearing premise

The load-bearing premise is that every inflection of a lemma shares the same first three characters of the simplified word form; word forms that violate this land in different hyperclusters and can never be merged, so the correct clustering is unreachable for them.

Editorial extensions

If this is right

An unsupervised lemmatizer for a new language can be built from an unannotated corpus plus pretrained embeddings, with no rule engineering or annotated data; the paper demonstrates this on 28 treebanks across 23 languages.
The combined distance is stronger than either component alone: the paper's comparison reports average error of 8.17 for Jaro-Winkler alone, 4.39 for cosine alone, and 3.77 for the product.
The method is most effective on suffixing, morphologically rich languages such as the Slavic ones, where error reductions reach about 50%, while analytic languages with little inflection are better handled by the simple form baseline.
Since word forms are clustered without context, homonymous forms cannot be separated; the paper identifies contextual embeddings as the natural next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: because the distance is multiplicative, a pair is merged only if it is close on both axes; on rare or out-of-vocabulary words, where embeddings are unreliable, the string component cannot rescue the product, so performance should degrade sharply.
Beyond the paper: varying the hypercluster stem length $K$ should trace a U-shaped error curve, and the optimal $K$ could be estimated per language from the distribution of prefix-sharing forms, making the method adaptive rather than fixed at $K=3$.
Beyond the paper: the same two-signal distance could be applied to other form-grouping tasks, such as discovering inflectional paradigms in a raw corpus or normalizing historical spellings, where form varies systematically and meaning is stable.
Beyond the paper: the failures on Japanese and Korean suggest a boundary condition: when the gold clusters are mostly singletons, any nonzero merge threshold creates error, so an unsupervised method needs a prior that separates inflectional from isolating languages.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know: this is a simple, honest paper about unsupervised lemmatization. It clusters word forms using a distance that multiplies Jaro-Winkler string similarity with FastText cosine similarity, and reports beating a per-dataset 'form or first-5-chars' baseline on 23 of 28 UD treebanks. The method is not a breakthrough, and the authors say so themselves — they call it 'too weak' — but the experiment is broad and the result is plausible.

The new part is the product distance. Neither component is new, but combining them for lemmatization and testing across 28 datasets is a fair contribution. The paper also compares each distance alone and shows the combination helps. The evaluation is transparent in important places: they include an oracle upper bound that never reaches 100% because of their stem-based hyperclusters (first three characters), and they list failure cases like suppletion and homonymy.

The soft spots are mostly in the measurement. The headline 23/28 uses token-based v-measure, so frequent word forms count multiple times. That systematically favors the method, because frequent forms are the ones inside the FastText vocabulary with reliable embeddings. Rare forms are handled by a crude nearest-cluster-or-singleton rule, and under type-based evaluation they would get equal weight. The paper never reports type-based results, so we cannot tell if the win count survives. That is the strongest concern and it is easy to address.

Other issues are minor. The threshold t=0.4, prefix length K=3, and vocabulary size N=100k were tuned on cs_pdt, which is still in the table; that is one dataset out of 28. The baseline is the better of two simple rules chosen on the same split, which is if anything conservative. There are no significance tests or error bars, but the effect is large on several languages.

Who is this for? Anyone looking for a cheap unsupervised baseline for lemmatization or morphology clustering. It is not a state-of-the-art system. It deserves a serious referee, but the review should ask for type-based results and, if cheap, a confidence interval. I would not cite it in my own work in the next year, but I would bring it to a reading group to discuss evaluation choices.

Recommendation: send it to peer review, with the request that the authors add a type-based re-analysis. That is the one number that could change the paper's conclusion.

Referee Report

3 major / 4 minor

Summary. This paper proposes an unsupervised lemmatization method that clusters word-form types using a distance measure combining Jaro-Winkler edit distance and cosine similarity of FastText embeddings (Eq. 1), followed by agglomerative clustering with average linkage and a fixed distance threshold t=0.4. A crude K=3 prefix stem partitions the vocabulary into hyperclusters to make clustering tractable. The method is evaluated on 28 Universal Dependencies treebanks covering 23 languages, with token-based v-measure as the evaluation metric. The paper reports that the method beats a per-dataset baseline of 'form' or 'first 5 characters' on 23 of 28 datasets, with a median error reduction of 23%. The authors explicitly acknowledge several limitations: the hyperparameters were tuned on cs_pdt, the oracle upper bound is below 100% because of the prefix-stem constraint, and the method fails on isolating languages and on several difficult datasets.

Significance. The paper makes a simple, language-agnostic proposal that could be useful for low-resource languages where annotated data are unavailable but pretrained embeddings exist. Its evaluation is unusually broad, covering 28 treebanks and 23 languages, and the authors are transparent about limitations: they report an oracle upper bound, disclose that hyperparameters were tuned on Czech, and identify the worst-performing language groups. The source code is promised to be released. If the empirical claims withstand scrutiny, the paper provides a practical baseline and a useful analysis of where such embedding-plus-string clustering succeeds and fails. However, the headline 23/28 result is currently tied to a token-based evaluation choice and to a results table that includes the tuning treebank, so the central claim needs additional support before it can be accepted at face value.

major comments (3)

[§4.1, Table 2] The hyperparameters t=0.4, K=3, and N=100,000 were tuned on the cs_pdt treebank, and cs_pdt is included in the headline results in Table 2. This conflates tuning and evaluation. The authors should report the results with cs_pdt excluded (or explicitly designated as development) and verify that the 23/28 count and the 23% median error reduction are unchanged. The disclosure in Section 4.1 is honest, but it does not by itself resolve the problem for the reported numbers.
[§4.1, Table 2] The evaluation is token-based, counting each occurrence of a word form, even though the clustering unit is a word-form type. Because the method's failures are likely concentrated in rare and out-of-vocabulary forms, which are handled by a crude nearest-cluster-or-singleton heuristic (Table 1), token-based weighting may inflate the method's apparent advantage over the 'form' baseline. The authors should report type-based v-measure results as well, or provide a task-level justification for token-based weighting. Without this, the central 23/28 empirical claim is not robust to an equally natural re-evaluation.
[§4.2, Table 2] No significance tests, error bars, or confidence intervals are reported. Given the large negative outlier for ko kaist (-6392.8) and heterogeneous per-language effect sizes, the authors should add a sign test and a Wilcoxon signed-rank test over the 28 datasets, or bootstrap confidence intervals for the median error reduction, to substantiate the claim that the method beats the baseline beyond a few favorable datasets.

minor comments (4)

[§2.3, Eq. (1)] Equation (1) is printed as 'dist(a, b) = 1 − J W(a, b) · cos(a, b) + 1 2', which is ambiguous; as written it could be read as 1 - JW·cos + 1/2. Add parentheses to make the intended shift clear: dist(a,b) = 1 - JW(a,b)·(cos(a,b)+1)/2.
[§4.2, Table 2] The error-reduction value for ko kaist is reported as -6392.8, but using the formula implied in Section 4.2 with the rounded values in the same row (baseline 0.14, ours 2.41, upper 0.11) gives approximately -7567. Please correct the typo or state the exact formula used.
[§4.2, Table 2] The baseline is selected per dataset as the better of the 'form' and 'first 5 characters' baselines. Please state explicitly whether this selection was made on the evaluation split itself; if so, this makes the comparison conservative for the proposed method, which is a useful fact to report.
[§4.3, Table 2] The 'Average' row is dominated by the Korean outlier and should probably be removed or replaced with the median. The text already notes that the average is not meaningful, but keeping the row invites misinterpretation.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity: cs_pdt hyperparameter tuning contaminates one of 28 reported results; the distance measure and clustering are otherwise self-contained.

fitted input called prediction [Section 4.1 (Data) and Table 2; acknowledged in Section 4.3]
"We used the cs pdt treebank to tune the method and set its hyperparameters. ... The performance is especially good for Slavic languages (cs, hr, pl, sk, uk), where the error reduction is often around 50%. This is most probably due to the hyperparameters being tuned on the cs pdt treebank."

For cs_pdt, the reported v-measure is not an out-of-sample prediction: the global threshold t=0.4, stem length K=3, and vocabulary size N=100,000 were selected using this very treebank, and the threshold directly controls how aggressively clusters are merged. Including cs_pdt in Table 2 therefore lets one fitted dataset contribute to the headline '23 of 28' count; the independent count is at most 22 of 27. The paper openly discloses the tuning, so this is a minor, bounded circularity rather than a collapse of the method's central claim, which still rests on 22 independent datasets.

full rationale

The paper's central construction, Eq. (1), is a fixed product of Jaro-Winkler similarity and shifted cosine similarity; neither component is defined in terms of the gold lemmas, and no parameter of Eq. (1) is fit to labels. The clustering uses off-the-shelf agglomerative average linkage from scikit-learn. The stem-based hyperclusters and oracle upper bound are explicit limitations, not hidden inputs. The token-based v-measure choice is a validity/fairness concern but not circularity: it weights frequent forms more heavily, yet it does not make the output equal to any fitted quantity or cited result. The only load-bearing reduction is the tuning of t, K, and N on cs_pdt followed by reporting cs_pdt in Table 2; this is acknowledged in Sections 4.1 and 4.3 and lowers the independent support by exactly one dataset. No self-citation chain, imported uniqueness theorem, or ansatz-by-citation is used. Score 2 reflects this single minor tuned-on-test inclusion; the other 22 positive results and the component ablation in Table 3 are independent evidence.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on three fitted hyperparameters (t, K, N), all tuned on the Czech cs_pdt treebank. It also assumes that morphological inflections are suffixing, that FastText embeddings capture meaning similarity, and that a single global threshold transfers across languages. No new entities are introduced.

free parameters (3)

cluster distance threshold t = 0.4
Stopping threshold for agglomerative clustering; tuned on cs_pdt treebank and applied to all languages.
hypercluster prefix length K = 3
Number of initial characters used to define stems for pre-partitioning; tuned on cs_pdt.
vocabulary size N = 100,000
Top N most frequent words from FastText dictionary; tuned on cs_pdt; smaller values worsen OOV rates.

assumptions (4)

domain assumption Inflections of the same lemma tend to be similar in both string form and meaning.
Basis for combining Jaro-Winkler similarity and embedding cosine similarity (Section 2).
domain assumption Word embedding cosine similarity is a valid proxy for meaning similarity for this task.
Motivated by Mikolov et al. (2013); uses FastText embeddings trained on Wikipedia and Common Crawl (Section 2.2).
domain assumption Most languages are suffixing; the first K characters of a simplified form can serve as a stem.
Justifies hypercluster partition by first K=3 characters; cited WALS feature 26A (Section 3.1).
ad hoc to paper A single global threshold t=0.4 and K=3 trained on Czech transfer to all other languages.
Hyperparameters are tuned on cs_pdt and reused for all 28 datasets without per-language adjustment (Section 4.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Unsupervised Lemmatization as Embeddings-Based Word Clustering." pith.science (2026). https://pith.science/paper/HXECV7NI

@misc{pith2026190808528,
  author       = {Pith},
  title        = {Pith review of: Unsupervised Lemmatization as Embeddings-Based Word Clustering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXECV7NI}},
  note         = {Machine review of arXiv:1908.08528}
}

read the original abstract

We focus on the task of unsupervised lemmatization, i.e. grouping together inflected forms of one word under one label (a lemma) without the use of annotated training data. We propose to perform agglomerative clustering of word forms with a novel distance measure. Our distance measure is based on the observation that inflections of the same word tend to be similar both string-wise and in meaning. We therefore combine word embedding cosine similarity, serving as a proxy to the meaning similarity, with Jaro-Winkler edit distance. Our experiments on 23 languages show our approach to be promising, surpassing the baseline on 23 of the 28 evaluation datasets.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018
[4]

Dryer and Martin Haspelmath, editors

Matthew S. Dryer and Martin Haspelmath, editors. 2013. http://wals.info/ WALS Online . Max Planck Institute for Evolutionary Anthropology, Leipzig

work page 2013
[5]

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)

2018
[6]

Daniel Kondratyuk, Tom \' a s Gaven c iak, Milan Straka, and Jan Haji c . 2018. Lemmatag: Jointly tagging and lemmatizing for morphologically rich languages with BRNN s. In Proceedings of the Conference on Empirical Methods in Natural Language Processing EMNLP 2018 , pages 4921--4928, Stroudsburg, PA , USA . ACL 's special interest group on linguistic dat...

work page 2018
[7]

Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707--710

work page 1966
[8]

Julie Beth Lovins. 1968. Development of a stemming algorithm. Mech. Translat. & Comp. Linguistics, 11(1-2):22--31

work page 1968

Show all 14 references

[9]

David Mare c ek, Zhiwei Yu, Daniel Zeman, and Zden e k Z abokrtsk \'y . 2016. http://hdl.handle.net/11234/1-1743 Deltacorpus 1.1 . LINDAT / CLARIN digital library at the Institute of Formal and Applied Linguistics ( \'U FAL ), Faculty of Mathematics and Physics, Charles University

2016
[10]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111--3119

2013
[11]

Joakim Nivre et al. 2018. http://hdl.handle.net/11234/1-2895 Universal dependencies 2.3 . LINDAT / CLARIN digital library at the Institute of Formal and Applied Linguistics ( \'U FAL ), Faculty of Mathematics and Physics, Charles University

2018
[12]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine L...

2011
[13]

Martin F Porter. 2001. Snowball: A language for stemming algorithms

2001
[14]

William E. Winkler. 1990. http://www.amstat.org/sections/srms/Proceedings/papers/1990_056.pdf String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage. In Proceedings of the Section on Survey Research Methods (American Statistical Ass...

1990

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

arXiv 2018

[4] [4]

Dryer and Martin Haspelmath, editors

Matthew S. Dryer and Martin Haspelmath, editors. 2013. http://wals.info/ WALS Online . Max Planck Institute for Evolutionary Anthropology, Leipzig

work page 2013

[5] [5]

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)

2018

[6] [6]

Daniel Kondratyuk, Tom \' a s Gaven c iak, Milan Straka, and Jan Haji c . 2018. Lemmatag: Jointly tagging and lemmatizing for morphologically rich languages with BRNN s. In Proceedings of the Conference on Empirical Methods in Natural Language Processing EMNLP 2018 , pages 4921--4928, Stroudsburg, PA , USA . ACL 's special interest group on linguistic dat...

work page 2018

[7] [7]

Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707--710

work page 1966

[8] [8]

Julie Beth Lovins. 1968. Development of a stemming algorithm. Mech. Translat. & Comp. Linguistics, 11(1-2):22--31

work page 1968