Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

UDPipe at SIGMORPHON 2019: Contextualized Embeddings, Regularization with Morphological Categories, Corpora Merging

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A BERT-boosted UDPipe system achieves the best lemmatization accuracy in the SIGMORPHON 2019 shared task, 95.78 percent averaged over 107 corpora.

desk verdict The lemmatization win is credible; the morphological 'tightly second' is a leaderboard fact within selection noise, but the paper is honest about it and the ablations are useful. read the letter →

arxiv 1908.06931 v1 pith:UX47F7EE submitted 2019-08-19 cs.CL

Milan Straka , Jana Straková , Jan Hajič This is my paper · ORCID
classification cs.CL
keywords SIGMORPHON2019lemmatizationmorphologicalanalysiscontextualizedembeddingsBERTmulti-tasklearningregularizationcorporamerging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a modified UDPipe 2.0 pipeline, augmented with pretrained BERT embeddings, per-category morphological feature regularization, and merging of same-language corpora, produces the best lemmatization system in the SIGMORPHON 2019 shared task and a near-best morphological analyzer. The system reports 95.78 lemmatization accuracy averaged over 107 corpora, ahead of the second-best 95.00, and 93.19 morphological accuracy, just behind 93.23. The point of caring is that morphological analysis and lemmatization remain bottlenecks for downstream natural language processing, especially for morphologically rich and under-resourced languages, and the paper shows which generic additions actually transfer across many languages. The paper claims these gains come from three separable modifications, measured by ablation experiments.

What carries the argument

The load-bearing object is the lemma rule: a concatenation of a casing script and an edit script that converts a word form into its lemma by character-level prefix and suffix edits, with irregular forms handled by direct replacement. Predicting a lemma is thereby reduced to a multiclass classification over a finite rule set. The overall architecture is the UDPipe 2.0 multi-task network, which provides three shared bidirectional LSTM layers over word, character-level, and now BERT embeddings, with a whole-tag classifier and auxiliary per-category feature heads that add a weighted cross-entropy regularization term. Corpora merging contributes a second training set, and inference-time filtering to the target corpus's observed rules and tags recovers some of the loss from inconsistent annotation.

What would settle it

Run the same three ablations (BERT, feature regularization, merging) on a held-out set of languages from the same Universal Dependencies and Universal Morphology scheme, using the paper's model-selection protocol, and check whether the claimed ordering of gains, especially the large contribution of feature regularization, reproduces; alternatively, recompute the ablation table under the fixed three-model ensemble protocol instead of best-models-on-development to see whether the gains shrink.

Watch

Extended reading notes

Core claim

The central claim is that a multi-task neural tagger-lemmatizer can be pushed to state-of-the-art accuracy on 107 corpora by three changes: feeding frozen pretrained BERT representations into the shared encoder, adding auxiliary softmax heads that predict individual morphological features as regularizers while still predicting the whole tag at inference, and, for languages with several corpora, training a merged model but restricting its output to lemma rules and feature sets seen in the target corpus. The paper reports test-set lemmatization accuracy of 95.78, exceeding all submitted systems, and morphological accuracy of 93.19, second by 0.04 points. The ablations show that each modification contributes, with BERT helping morphological tagging more than lemma prediction and feature regularization being the strongest single addition.

Load-bearing premise

The results assume that choosing model configurations and ensembles by development-set accuracy generalizes to the test set; the paper's own competition entry shows this selection can overfit and underperform a fixed ensemble.

Editorial extensions

If this is right

  • The winning lemmatization accuracy of 95.78 shows that contextualized embeddings and feature-level regularization transfer across 66 languages without task-specific tuning.
  • The ablation results predict that BERT embeddings give their largest gain for morphological analysis, not for lemma generation.
  • Merged corpora help only when annotation is consistent; the paper's inference-time filtering is what makes the merged model usable in practice.
  • Selecting ensembles by development-set accuracy can overfit, as the competition entry's arbitrary nine-model selection scored lower on morphological accuracy than a fixed three-model ensemble.
  • A model without BERT would still place third in both tasks, indicating that the baseline UDPipe architecture itself is strong.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the development-to-test overfitting seen in the ensemble comparison also affects the ablation study's configuration choices, the reported per-modification gains may be optimistic on unseen corpora; a stricter cross-validation or fixed-protocol rerun could shrink them.
  • The lemma-rule formulation suggests a direct extension to inflection generation: the same casing-plus-edit-script classes could be run in reverse, from lemma to form, sharing the rule inventory.
  • Because the paper averages BERT layers without finetuning, a testable extension is whether finetuning the contextualized encoder, or using a language-family-specific BERT, widens the reported margins on morphologically rich languages.
  • The merging results imply a cheap diagnostic for annotation consistency: compare merged-model loss on each corpus's training data, and treat corpora where merging hurts most as candidates for reannotation or schema reconciliation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper describes the UDPipe-based system submitted by the authors to the SIGMORPHON 2019 Shared Task, Task 2 (contextual morphological analysis and lemmatization). The system extends UDPipe 2.0 with three modifications: (i) adding pretrained contextualized BERT embeddings as additional inputs, (ii) predicting individual morphological features as auxiliary regularization, and (iii) merging same-language corpora during training. On the official shared-task test set, the system reports first place in lemmatization with 95.78 accuracy (second best 95.00) and second place in morphological analysis with 93.19 accuracy (winning system 93.23). The paper presents a component-wise ablation (Table 3) showing that each modification improves accuracy, and a model-combination analysis (Table 4) that compares the competition entry with post-competition ensemble strategies, including a documented overfitting issue in development-based ensemble selection.

Significance. If the results hold, the paper makes a solid applied contribution: it shows that a strong UD parser architecture can be successfully adapted to contextual morphological analysis and lemmatization, with the lemmatization win being a clear, external result from the official shared-task evaluation. The ablation study is valuable because it isolates the effect of BERT, feature regularization, and corpus merging on a strong baseline, and the authors are transparent about the overfitting of their competition ensemble selection. The main weakness is that the morphological-analysis ranking is a close 0.04-point gap that the paper's own ensembling analysis suggests is within selection noise; no confidence intervals or significance tests are provided. The lemmatization claim (0.78-point margin) is far more robust than the morphological claim.

major comments (3)
  1. [Abstract and Section 4.1] The claim that the system 'placed tightly second' in morphological analysis is not statistically supported. The reported gap to first place is 0.04 percentage points (93.19 vs 93.23), while Table 4 shows that changing only the ensemble-selection rule moves morphological accuracy by 0.278 points (93.186 for the dev-selected arbitrary subset vs 93.464 for the per-configuration 3-model ensemble). On the paper's own numbers, the second-place ranking is indistinguishable from first or third. The authors should either report per-corpus variance, confidence intervals, or a significance test for this leaderboard comparison, or explicitly qualify the ranking as a point estimate that may not be robust to the documented selection noise.
  2. [Section 4.2 / Table 3] The ablation results are presented as single test-set values without variance. Given that Section 3.6 documents that development-based model selection overfits, especially for morphological prediction, the reader cannot rule out that the improvements attributed to BERT, feature regularization, and merging are partially selection artifacts. Since the authors already train three models per configuration, they should report mean and standard deviation over these random seeds, or perform a paired significance test across the 107 corpora, to make the contribution claims load-bearing.
  3. [Section 4.3 / Table 4] The claimed contribution of corpora merging is based on a single test-set comparison (e.g., morphological accuracy 93.148 for the regular model vs 93.322 for the regular plus merged model). This difference is within the spread that Table 4 shows for different ensemble choices, and the per-corpus deltas in Tables 5 and 6 include large negative values (e.g., Korean-PUD with a morphological accuracy decrease of 16.72 under merging). The authors should show that the merging improvement is consistent across corpora rather than driven by a few outliers, or at least report a significance test.
minor comments (5)
  1. [Abstract] There is a missing space in 'winning system's93.23' in the last sentence of the abstract.
  2. [Section 3.3] The sentence 'for Chinese use use the native Chinese model' contains a duplicated 'use'; it should read 'for Chinese we use the native Chinese model'.
  3. [Section 1] The phrase 'theThe 2018 Shared Task on Extrinsic Parser Evaluation' contains a duplicated definite article.
  4. [Table 3] The table would benefit from a legend or caption making explicit that a checkmark indicates use of the corresponding component, and that 'FT only' refers to the Grave et al. (2018) embeddings rather than the authors' pretrained embeddings. The current formatting is not self-explanatory.
  5. [Section 4.2] When discussing that the baseline would 'place third in both lemmatization and tagging accuracy', the authors should state explicitly that this comparison is against the official leaderboard on the test set, to avoid ambiguity about the evaluation split used for the ablation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported results are external shared-task test scores, and the listed contributions are evaluated by ablation against that external test set.

full rationale

The paper's load-bearing claims are official SIGMORPHON 2019 Task 2 test-set scores (lemmatization accuracy 95.78, morphological accuracy 93.19), computed by the shared-task organizers on held-out test data that is external to all training and development data. The three described improvements—BERT contextualized embeddings, regularization with individual morphological features, and corpora merging—are each evaluated in Table 3 and Table 4 as ablation comparisons on that same external test set. The feature-regularization auxiliary loss is used only during training and is not consulted at inference ('The predicted features are not used in any way during inference and act only as model regularization'), so it is not a fitted quantity masquerading as a prediction. The lemma rules are generated from the training data's form-lemma pairs and then predicted as a classification target; this is standard supervised learning, not a self-definitional reduction. Self-citations to UDPipe 2.0 (Straka 2018) and LemmaTag (Kondratyuk et al. 2018) describe the baseline architecture and a regularization precedent, but neither citation is invoked as a uniqueness theorem, as forbidden support for the central claim, or as a substitute for the external evaluation. The paper honestly documents that development-set ensemble selection can overfit (Section 3.6) and that the post-competition alternative selection changes morphological accuracy by more than the 0.04-point gap to the winner; this is a statistical robustness caveat about ranking precision, not a circularity. No equation, fitted parameter, or self-citation chain is shown to reproduce the headline results by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. The central empirical claims rest on standard pretrained artifacts (BERT, FastText, UDPipe 2.0), one tuned scalar (regularization weight w), hand choices for BERT aggregation, and per-corpus model selection on development data. The main additional axioms are about the UD annotation schema, the coverage of the lemma-rule encoding, the transfer of BERT embeddings across languages, and dev-test representativeness; the last is explicitly shown to be fragile.

free parameters (3)
  • Regularization weight w = 1 (0.5 and 2 also tested)
    Scales the averaged cross-entropy loss of auxiliary morphological feature predictions in Section 3.4; chosen using development data, with accuracy differences of 0.02 or less in Table 3.
  • BERT layer aggregation = average of the last 4 BERT layers
    Chosen by hand in Section 3.3; no comparison experiment is reported for this choice.
  • Per-corpus model selection (regular vs. merged vs. no-BERT) = selected per corpus on development data (Tables 5 and 6)
    Because merged models can hurt (e.g., Galician-TreeGal -11.16 morph accuracy), the choice of configuration is made per corpus on dev, not by a fixed rule.
assumptions (4)
  • domain assumption The Universal Dependencies feature schema is consistent enough across same-language corpora for merged training to be beneficial.
    Section 3.5 notes merged models can be worse due to systematically different annotation, so the assumption only holds for selected corpora after dev-based filtering.
  • domain assumption Lemma-rule edit scripts (prefix/root/suffix with longest shared root) can encode every valid form-to-lemma mapping; irregular mappings are replaced by direct replacement.
    Section 3.2 defines lemmatization as classification over these rules; if a language's morphology requires edits the script language cannot express, accuracy is capped.
  • domain assumption BERT frozen embeddings provide useful signal for languages that may be absent from BERT pretraining.
    Section 3.3 uses Multilingual BERT for non-English languages and Section 4.3 retains a no-BERT model because results might be misleading if the language was not present in the BERT training data.
  • domain assumption Development data is representative of test data for configuration and ensemble selection.
    Section 3.6 reports this assumption is violated for the competition entry's model selection, which overfit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UDPipe at SIGMORPHON 2019: Contextualized Embeddings, Regularization with Morphological Categories, Corpora Merging." pith.science (2026). https://pith.science/paper/UX47F7EE

@misc{pith2026190806931,
  author       = {Pith},
  title        = {Pith review of: UDPipe at SIGMORPHON 2019: Contextualized Embeddings, Regularization with Morphological Categories, Corpora Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UX47F7EE}},
  note         = {Machine review of arXiv:1908.06931}
}
read the original abstract

We present our contribution to the SIGMORPHON 2019 Shared Task: Crosslinguality and Context in Morphology, Task 2: contextual morphological analysis and lemmatization. We submitted a modification of the UDPipe 2.0, one of best-performing systems of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies and an overall winner of the The 2018 Shared Task on Extrinsic Parser Evaluation. As our first improvement, we use the pretrained contextualized embeddings (BERT) as additional inputs to the network; secondly, we use individual morphological features as regularization; and finally, we merge the selected corpora of the same language. In the lemmatization task, our system exceeds all the submitted systems by a wide margin with lemmatization accuracy 95.78 (second best was 95.00, third 94.46). In the morphological analysis, our system placed tightly second: our morphological analysis accuracy was 93.19, the winning system's 93.23.

Figures

Figures reproduced from arXiv: 1908.06931 by the authors.

Figure 1
Figure 1. The overall system architecture The lemmas are generated by classifying into a set of edit scripts which process input word form and produce lemmas by performing character￾level edits on the word prefix and suffix. The lemma classifier additionally takes the character￾level word embeddings as input. The lemmatiza￾tion is further described in Section 3.2. The input word embeddings are the same as in the UDPipe 2.0 (S… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Context Aware Lemmatization and Morphological Tagging Method in Turkish

    cs.CL 2025-01 reject novelty 3.0 of 10

    A BiLSTM plus Turkish BERT lemmatizer and morphological tagger is tested on IMST and PUD, beating SIGMORPHON 2019 on most metrics but not on PUD lemmatization accuracy, while the paper's claim of being the first conte...

Reference graph

Works this paper leans on

20 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. http://aclweb.org/anthology/Q17-1010 Enriching Word Vectors with Subword Information . Transactions of the Association for Computational Linguistics, 5:135--146

    2017

  4. [4]

    Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. http://www.aclweb.org/anthology/K18-2005 Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies , pages 55--64, Brussels, Belgium. Ass...

    2018

  5. [5]

    KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the P roperties of N eural M achine T ranslation: E ncoder- D ecoder A pproaches. CoRR

    work page 2014

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    arXiv 2018

  7. [7]

    Murhaf Fares, Stephan Oepen, Lilja vrelid, Jari Bj\"orne, and Richard Johansson. 2018. http://aclweb.org/anthology/K18-2002 The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, page...

    2018

  8. [8]

    Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)

    2018

Show all 20 references
  1. [9]

    Alex Graves and J \" u rgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, pages 5--6

    2005

  2. [10]

    Sepp Hochreiter and J \" u rgen Schmidhuber. 1997. Long S hort- T erm M emory. Neural Comput., 9(8):1735--1780

    1997

  3. [11]

    Daniel Kondratyuk, Tomas Gavenciak, Milan Straka, and Jan Hajic. 2018. https://aclanthology.info/papers/D18-1532/d18-1532 Lemmatag: Jointly tagging and lemmatizing for morphologically rich languages with brnns . In Proceedings of the 2018 Conference on Empirical Methods in Nat...

    2018

  4. [12]

    Black, and Isabel Trancoso

    Wang Ling, Tiago Lu \' i s, Lu \' i s Marujo, Ram \' o n Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015. Finding F unction in F orm: C ompositional C haracter M odels for O pen V ocabulary W ord R epresentation. CoRR

    2015

  5. [13]

    McCarthy, Miikka Silfverberg, Ryan Cotterell, Mans Hulden, and David Yarowsky

    Arya D. McCarthy, Miikka Silfverberg, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2018. https://www.aclweb.org/anthology/W18-6011 Marrying U niversal D ependencies and U niversal M orphology . In Proceedings of the Second Workshop on Universal Dependencies ( UDW 2018) , p...

    2018

  6. [14]

    Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Haji c , Christopher Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proce...

    2016

  7. [15]

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2018

  8. [16]

    Milan Straka. 2018. UDPipe 2.0 Prototype at CoNLL 2018 UD Shared Task . In Proceedings of CoNLL 2018: The SIGNLL Conference on Computational Natural Language Learning, pages 197--207, Stroudsburg, PA, USA. Association for Computational Linguistics

    2018

  9. [17]

    John Sylak-Glassman. 2016. The composition and use of the universal morphological feature schema (unimorph schema)

    2016

  10. [18]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. http://arxiv.org/abs/1706.03762 Attention is all you need . CoRR, abs/1706.03762

    2017 arXiv

  11. [19]

    Daniel Zeman, Filip Ginter, Jan Haji c , Joakim Nivre, Martin Popel, and Milan Straka. 2018. CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies . In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dep...

    2018

  12. [20]

    Daniel Zeman, Martin Popel, Milan Straka, Jan Haji c , Joakim Nivre, Filip Ginter, et al. 2017. CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies . In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal ...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.