Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing

Jana Strakov\'a; Jan Haji\v{c}; Milan Straka

REVIEW 2 major objections 5 minor 19 references

Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding BERT to a strong multilingual parser cuts tagging and dependency-parsing errors by up to a quarter and produces state-of-the-art results on the CoNLL 2018 benchmark.

desk verdict Large one-to-one comparison of contextualized embeddings that holds up; the SOTA claim needs a clearer statement of data version. read the letter →

arxiv 1908.07448 v1 pith:OAJQBLIP submitted 2019-08-20 cs.CL

Milan Straka , Jana Straková , Jan Hajič This is my paper · ORCID

classification cs.CL

keywords contextualizedembeddingsBERTFlairELMoUniversalDependenciesPOStagginglemmatizationdependencyparsing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper systematically measures how much three pretrained contextual word embedding methods improve POS tagging, lemmatization, and dependency parsing when plugged into a strong multilingual parser. Working with 89 treebanks in 54 languages from Universal Dependencies 2.3, the authors hold the UDPipe 2.0 architecture fixed and vary only the embedding source, so that any performance differences are attributable to the embeddings. They report that BERT gives the largest overall gains, while Flair and ELMo contribute more morphological and orthographic information, and the embeddings are complementary. The best single model reaches state-of-the-art scores on the official CoNLL 2018 Shared Task metrics, and an ensemble of all three embedding types sets a new state of the art on every metric.

What carries the argument

The workhorse is the UDPipe 2.0 multi-task network, which shares two bidirectional LSTM layers across a tagger, a lemmatizer, and a dependency parser with biaffine attention. The variable under study is the embedding source: pretrained FastText word vectors, end-to-end character-level word embeddings from bidirectional GRUs, and three contextual models—BERT (averaged over subwords and the last four layers), Flair (character-level language model states), and ELMo (bidirectional LSTM states). By keeping the network architecture and training procedure fixed and varying only these input features, the evaluation isolates the contribution of each embedding type.

What would settle it

Run the same UDPipe 2.0 system with and without BERT on the exact UD 2.2 treebanks and train/dev/test splits used in the CoNLL 2018 Shared Task, and compare against the official shared-task baselines; if the BERT system fails to beat those baselines on that identical data, the state-of-the-art claim is false.

Watch

Extended reading notes

Core claim

The central claim is that contextualized embeddings are a direct and large source of improvement in morphological tagging and syntactic parsing, not a small add-on. Adding BERT embeddings to a UDPipe 2.0 system that already uses word-level and character-level embeddings cuts relative error by 16.9% for universal POS tags, 12% for morphological features, 4.3% for lemmatization, and 14.5% for labeled dependency parsing across 72 treebanks. Across all 89 available treebanks, the full BERT+Flair system delivers a 15% relative error reduction in UPOS tagging and 11.5% in labeled dependency parsing, with lemmatization improving only 2.4%. On the CoNLL 2018 Shared Task setup, the addition of BERT yields state-of-the-art single-model results in UPOS, XPOS, UFeats, MLAS, and BLEX, and the ensemble is state-of-the-art in all metrics. The paper also identifies a division of labor: BERT captures syntax, while Flair and ELMo carry more morphological and orthographic information, so the embeddings are complementary.

Load-bearing premise

The state-of-the-art claim assumes that results on Universal Dependencies 2.3 treebanks are directly comparable with published CoNLL 2018 Shared Task scores computed on Universal Dependencies 2.2, even though treebank versions, tokenization, and data splits may differ.

Editorial extensions

If this is right

Adding BERT to a strong parser cuts relative error by roughly 15–25% for high-resource languages, with English treebanks showing the largest gains.
A single multilingual BERT model performs almost as well as a language-specific English BERT, suggesting that one pretrained model can serve many languages.
Flair and BERT are complementary: Flair recovers morphological detail and lemmatization accuracy that BERT lacks, and the combination beats either alone on every metric.
On English, ELMo alone outperforms Flair alone and BERT alone at POS tagging and lemmatization, while BERT wins in dependency parsing.
For the 13 treebanks whose languages are absent from BERT's pretraining, parsing still improves slightly, but lemmatization degrades, indicating that pretraining coverage is a limiting factor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the architecture is held constant, the measured differences are attributable to the embeddings; a similar ranking across tasks is plausible in other sequence-labeling settings such as named-entity recognition and semantic role labeling.
The small gain from adding FastText word vectors to BERT suggests contextual models largely subsume static word vectors, so future systems may not need per-language Wikipedia pretraining.
The near-linear relationship between pretraining corpus size and error reduction in the paper's figures could let a practitioner predict a new language's expected gain from BERT before running the full pipeline.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is the paper that gave the field the standard one-to-one comparison of BERT, Flair, and ELMo in a strong UDPipe 2.0 baseline across 54 languages. The comparative results are the contribution and they hold up; the SOTA claim is less cleanly supported than the authors' framing suggests, mostly because Table 5's comparison to CoNLL 2018 systems is not described in enough detail.

What's new: 89 corpora, three tasks, one fixed baseline, no finetuning, official evaluation script. The qualitative conclusions—BERT helps syntax most, Flair carries morphology and lemmatization, ELMo is competitive on English—are established by direct ablation and are now basically textbook. The per-language error reduction figures are useful, and the paper is honest about using a fixed architecture rather than tuning per embedding.

Soft spots: the biggest is the SOTA claim. The paper says state-of-the-art versus UD 2.2 CoNLL 2018 results, and Table 5 is captioned as CoNLL 2018 results; the authors also reuse the original UDPipe tokenization. That is strong evidence the comparison is on UD 2.2, not UD 2.3. But the evaluation section never explicitly says 'we trained and evaluated on UD 2.2 datasets and the official splits,' and with margins of 0.01–0.2 points, that matters. A referee should ask for a one-sentence clarification or released per-treebank scores. I do not think this is fatal; the one-to-one conclusions do not depend on it. Second, most results are single runs with no variance. The English experiments average three runs but do not report standard deviations, so small differences between embeddings on individual metrics could be noise. The main claims are large enough to survive, but the paper should say so. Third, the Flair comparison covers only the 23 treebanks with available embeddings, which is reasonable but weakens the '54 languages' headline.

Bottom line: this paper deserves a serious referee and will be cited. It is a solid empirical benchmark with one sloppy comparison. If I were handling it, I would ask for an explicit statement of the data version behind Table 5 and variance information for repeated runs, then accept.

Referee Report

2 major / 5 minor

Summary. This paper presents a large-scale empirical evaluation of three contextualized embedding methods (BERT, Flair, and ELMo) as additional inputs to the UDPipe 2.0 pipeline, which is a strong CoNLL 2018 Shared Task baseline. The evaluation covers up to 89 UD 2.3 treebanks in 54 languages and reports UPOS, XPOS, UFeats, lemmatization, UAS, LAS, MLAS, and BLEX scores, with macro-averaging across treebanks. The authors also compare their best configurations against published CoNLL 2018 Shared Task systems and claim state-of-the-art single-model performance on several metrics and state-of-the-art ensemble performance on all metrics, based on Table 5.

Significance. If the results hold, this is a valuable contribution: it is one of the broadest controlled comparisons of BERT, Flair, and ELMo under a fixed, non-fine-tuned baseline, and it quantifies gains across many languages and annotation layers. The strengths include the use of a fixed UDPipe 2.0 baseline, the official evaluation script, macro-averaging over treebanks, the explicit separation of languages not covered by BERT, and the reproducible setup of adding embeddings without fine-tuning. The main risk is that the headline state-of-the-art claim rests on a comparison in Table 5 that may mix UD 2.3 and UD 2.2 results, which would make the claim unsupported. The paper's broad benchmark results remain useful even if the SOTA claim is revised.

major comments (2)

[Section 4.3, Table 5] The state-of-the-art claim is based on comparing the new 'UDPipe 2.0 WE+CLE+BERT' rows with published CoNLL 2018 Shared Task systems, but the paper does not state which UD version was used for the new rows. The rest of the paper evaluates on UD 2.3, while the CoNLL 2018 Shared Task used UD 2.2. Section 4.3 only says that the authors reuse tokenization and segmentation from the original UDPipe 2.0 entry; it does not say that training and evaluation data were switched back to UD 2.2. If the new rows are UD 2.3 scores, the comparison with the published UD 2.2 systems is not same-data, and the claim of state-of-the-art results is not established. Please explicitly state that Table 5 was produced on UD 2.2 with the same train/dev/test splits as the shared task, or rerun the comparison on UD 2.2, or revise the SOTA claim to a comparison that is clearly marked as not strictly comparable.
[Table 5, HIT-SCIR row] The row 'HIT-SCIR Harbin (Che et al., 2018) w/o ensembling' reports only a single number (83.75) without a metric label, so the reader cannot determine whether this is LAS, UAS, or another metric. This makes it impossible to verify the single-model comparison for dependency parsing in the same table. Please complete this row with the metric name and, if space permits, the full set of metrics for that configuration.

minor comments (5)

[Section 4.5, Table 6] For the 13 treebanks whose languages are not included in the BERT multilingual model, the paper should explain what representation BERT produces for these languages (for example, whether it falls back to a default or UNK vector). This would help the reader interpret the mixed results reported for these treebanks.
[General] The paper states that it 'publish[es] a comparison,' but no URL or repository is given for the trained models, code, or evaluation scripts. Adding a release link would substantially support reproducibility.
[Tables 1, 3, 5] Several tables report macro-averages without variance or the number of runs, even for very close comparisons (for example, UPOS 96.32 vs. 96.23 in Table 5). Reporting standard deviations for at least the central configurations would make the state-of-the-art claims more robust.
[Abstract and Table 6] The abstract says '89 corpora in 54 languages,' while Table 6 says '89 treebanks'; please align the terminology throughout the paper.
[Figures 2 and 3] The fitted regression lines in Figures 2 and 3 are presented without R-squared values or confidence intervals; adding these would help the reader assess the strength of the claimed correlations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper is an empirical benchmark whose state-of-the-art claim rests on external shared-task scores, not on fitted inputs or author-imposed uniqueness.

full rationale

This paper makes no theoretical derivation. Its predictions are measured evaluation scores produced by running UDPipe 2.0 with externally published BERT, Flair, and ELMo embeddings on Universal Dependencies 2.3 data, evaluated with the official CoNLL 2018 script. No parameter is fitted to a target metric and then reported as that same metric; the improvement numbers are computed from held-out evaluation against treebank annotations. The self-citation to Straka (2018) describes the UDPipe 2.0 architecture and its CoNLL 2018 shared task entry; this is a baseline system, not an unverified premise that contains the conclusion. Table 5 compares new runs with published CoNLL 2018 results, and the associated UD 2.2 versus UD 2.3 comparability question is a validity concern about data versions, not circular reasoning. No step in the paper defines an input in terms of its output or renames a fitted parameter as a prediction. The finding is therefore no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper contributes an empirical measurement, so its central claims rest on dataset assumptions and experimental design choices, not on new free parameters or invented entities. The only numbers fitted to data are the learned weights of the UDPipe models, which are not part of a derivation and are not claimed as a new result.

assumptions (3)

domain assumption UD 2.3 treebank annotations and the official CoNLL 2018 evaluation script are the ground truth for measuring task performance.
The paper evaluates on Universal Dependencies 2.3 using the CoNLL 2018 evaluation script without questioning annotation quality or metric validity.
domain assumption The macro-average across treebanks is the appropriate aggregation for cross-lingual conclusions.
All conclusions about relative embedding performance are drawn from macro-averaged scores over 72 or 89 treebanks; treebanks are weighted equally regardless of language or size.
domain assumption Pretrained contextualized embedding models are used as frozen features with fixed pooling (average of last four BERT layers, full Flair and ELMo outputs), and this configuration is representative of each method's potential.
Section 3 fixes the pooling and states embeddings are never finetuned; the qualitative ranking of methods could depend on these choices.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing." pith.science (2026). https://pith.science/paper/OAJQBLIP

@misc{pith2026190807448,
  author       = {Pith},
  title        = {Pith review of: Evaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OAJQBLIP}},
  note         = {Machine review of arXiv:1908.07448}
}

read the original abstract

We present an extensive evaluation of three recently proposed methods for contextualized embeddings on 89 corpora in 54 languages of the Universal Dependencies 2.3 in three tasks: POS tagging, lemmatization, and dependency parsing. Employing the BERT, Flair and ELMo as pretrained embedding inputs in a strong baseline of UDPipe 2.0, one of the best-performing systems of the CoNLL 2018 Shared Task and an overall winner of the EPE 2018, we present a one-to-one comparison of the three contextualized word embedding methods, as well as a comparison with word2vec-like pretrained embeddings and with end-to-end character-level word embeddings. We report state-of-the-art results in all three tasks as compared to results on UD 2.2 in the CoNLL 2018 Shared Task.

Figures

Figures reproduced from arXiv: 1908.07448 by the authors.

**Figure 1.** UDPipe 2.0 architecture overview. tagging, lemmatization and dependency parsing. The system of Che et al. (2018) is one of the three winners of the CoNLL 2018 Shared Task. The authors employed manually trained ELMolike contextual word embeddings, reporting 7.9% error reduction in LAS parsing performance. 3 Methods Our baseline is the UDPipe 2.0 (Straka, 2018) participant system from the CoNLL 2018 Shared Task (Zema… view at source ↗

**Figure 2.** Relative error improvements on UD 2.3 treebanks wh [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Relative error improvements of the baseline with W [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual String Embeddings for Sequence Labeling . In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638--1649. Association for Computational Linguistics

work page 2018
[4]

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. http://aclweb.org/anthology/Q17-1010 Enriching Word Vectors with Subword Information . Transactions of the Association for Computational Linguistics, 5:135--146

work page 2017
[5]

Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. http://www.aclweb.org/anthology/K18-2005 Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies , pages 55--64, Brussels, Belgium. Ass...

work page 2018
[6]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

arXiv 2018
[7]

Timothy Dozat and Christopher D. Manning. 2016. http://arxiv.org/abs/1611.01734 Deep Biaffine Attention for Neural Dependency Parsing . CoRR, abs/1611.01734

arXiv 2016
[8]

Murhaf Fares, Stephan Oepen, Lilja vrelid, Jari Bj\"orne, and Richard Johansson. 2018. http://aclweb.org/anthology/K18-2002 The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, page...

work page 2018

Show all 19 references

[9]

Sepp Hochreiter and J \" u rgen Schmidhuber. 1997. Long S hort- T erm M emory. Neural Comput., 9(8):1735--1780

1997
[10]

Jenna Kanerva, Filip Ginter, Niko Miekka, Akseli Leino, and Tapio Salakoski. 2018. http://www.aclweb.org/anthology/K18-2013 Turku neural parser pipeline: An end-to-end system for the CoNLL 2018 shared task . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing fr...

2018
[11]

Black, and Isabel Trancoso

Wang Ling, Tiago Lu \' i s, Lu \' i s Marujo, Ram \' o n Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015. Finding F unction in F orm: C ompositional C haracter M odels for O pen V ocabulary W ord R epresentation. CoRR

2015
[12]

Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Haji c , Christopher Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proce...

2016
[13]

Joakim Nivre et al. 2018. http://hdl.handle.net/11234/1-2895 Universal dependencies 2.3 . LINDAT / CLARIN digital library at the Institute of Formal and Applied Linguistics ( \'U FAL ), Faculty of Mathematics and Physics, Charles University

2018
[14]

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: ...

2018
[15]

Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. http://www.aclweb.org/anthology/K18-2016 Universal dependency parsing from scratch . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies , pages 160--1...

2018
[16]

Milan Straka. 2018. UDPipe 2.0 Prototype at CoNLL 2018 UD Shared Task . In Proceedings of CoNLL 2018: The SIGNLL Conference on Computational Natural Language Learning, pages 197--207, Stroudsburg, PA, USA. Association for Computational Linguistics

2018
[17]

Gomez, Lukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. http://arxiv.org/abs/1706.03762 Attention is all you need . CoRR, abs/1706.03762

2017 arXiv
[18]

Daniel Zeman, Filip Ginter, Jan Haji c , Joakim Nivre, Martin Popel, and Milan Straka. 2018. CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dep...

2018
[19]

Daniel Zeman, Martin Popel, Milan Straka, Jan Haji c , Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh G \" o k rmak, Anna Nedoluzhko, Silvie Cinkov \' a , Jan Haji c jr., Jaroslava Hlav \' a c o...

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual String Embeddings for Sequence Labeling . In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638--1649. Association for Computational Linguistics

work page 2018

[4] [4]

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. http://aclweb.org/anthology/Q17-1010 Enriching Word Vectors with Subword Information . Transactions of the Association for Computational Linguistics, 5:135--146

work page 2017

[5] [5]

Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. http://www.aclweb.org/anthology/K18-2005 Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies , pages 55--64, Brussels, Belgium. Ass...

work page 2018

[6] [6]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

arXiv 2018

[7] [7]

Timothy Dozat and Christopher D. Manning. 2016. http://arxiv.org/abs/1611.01734 Deep Biaffine Attention for Neural Dependency Parsing . CoRR, abs/1611.01734

arXiv 2016

[8] [8]

Murhaf Fares, Stephan Oepen, Lilja vrelid, Jari Bj\"orne, and Richard Johansson. 2018. http://aclweb.org/anthology/K18-2002 The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, page...

work page 2018