Pith. sign in

REVIEW 3 major objections 5 minor 24 references

NameTag 3: A Tool and a Service for Multilingual/Multitagset NER

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One 355M model beats state of the art on 21 NER datasets

desk verdict Solid systems paper with a genuinely useful tool, but the SOTA headline doesn't match its own Table 3 and needs correcting before publication. read the letter →

arxiv 2506.05949 v2 pith:72HKRH37 submitted 2025-06-06 cs.CL

Jana Straková , Milan Straka This is my paper
classification cs.CL
keywords namedentityrecognitionmultilingualNLPmultitagsetlearningnestedentitiesfine-tunedlanguagemodelscross-lingualtransferRESTwebservicestate-of-the-artevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NameTag 3 is an open-source named entity recognition tool and cloud service built around a single fine-tuned 355M-parameter language model trained jointly on 21 corpora in 17 languages. The paper's central claim is that this one modestly sized model reaches state-of-the-art span-based $F_1$ on 21 test datasets in 15 languages and stays competitive on five more, outperforming widely used toolkits on every dataset where direct comparison is possible. The same design supports flat NER with three different entity tagsets and nested NER through a separate seq2seq head, with a 126M Czech model for nested entities. The paper also claims that this fine-tuned approach beats a 70B-parameter LLM by about 20 $F_1$ points on English CoNLL-2003 while running roughly 10,000 times faster, so the contribution is both an accuracy result and a practical service.

What carries the argument

The central machinery is multitagset fine-tuning: one shared Transformer encoder (XLM-R Large, 355M) with a distinct softmax classification head per tagset, so the encoder and all heads are trained jointly but only the head for the requested tagset runs at inference. Training batches are formed by square-root temperature sampling, where corpora are sampled proportionally to the square root of their sentence counts, and the optimization target is a macro span-based $F_1$ averaged uniformly across datasets; that choice is what lets small corpora contribute enough to keep 21 datasets balanced. For nested NER the softmax head is replaced by a seq2seq decoder (Straková et al., 2019) that generates linearized nested labels with hard attention on the current token, and a few pre-training epochs freeze the encoder so the decoder adapts before joint fine-tuning.

What would settle it

Retrain the 355M multilingual flat model with the same data but with uniform sentence sampling (no square-root temperature) and with per-dataset micro $F_1$ weighting, then compare all 21 test datasets against the reported baselines; if several of the narrow 1–3 point wins become losses, the central claim depends on the sampling choice and would need qualification.

Watch

Extended reading notes

Core claim

The core claim is that one joint multilingual model can replace a collection of per-language, per-tagset systems without sacrificing accuracy: NameTag 3's 355M flat model, with a separate softmax classification head for each of the CoNLL, UNER, and OntoNotes tagsets sharing one encoder, sets new state-of-the-art scores on datasets including Croatian UNER SET (95.55), Serbian UNER SET (97.10), Ukrainian Lang-uk (92.88), and Chinese UNER GSD (91.53), and remains within about two points of the best published results on Arabic, Dutch, German, Maghrebi, and Spanish. For nested NER, a 126M Czech model achieves 86.39 on CNEC 2.0, above the prior 83.44, while English ACE-2004 and ACE-2005 come within roughly half an $F_1$ point of the state of the art. The paper presents this as evidence that fine-tuning smaller open models is still the most resource-efficient route to accurate NER whenever training data exist.

Load-bearing premise

The claim that one model is state of the art on 21 datasets rests on the particular square-root batch-sampling scheme and the uniformly weighted macro $F_1$ objective; if a different sampling or weighting moved several of the 1–3 point margins, the state-of-the-art statement would weaken.

Editorial extensions

If this is right

  • One 355M model can serve 17 languages and three tagsets from a single checkpoint, so users no longer need per-language models for these languages.
  • The same jointly trained model outperforms popular toolkits on every dataset where both have comparable models, which makes it a drop-in replacement for those pipelines.
  • When training data are available, fine-tuning a small model is claimed to beat a 70B LLM by about 20 $F_1$ points while being more than 10,000 times faster, shifting the practical default away from prompting.
  • Nested NER for Czech reaches state of the art with a 126M model, and English nested results are near state of the art with a 355M model, so nested annotation can be served without a larger pipeline.
  • Cross-lingual transfer to languages not seen in training (for example Cebuano at 96.97 and Tagalog TRG at 97.78) suggests the model generalizes beyond its training languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to apply the same shared-encoder, per-task-head design to other sequence labeling tasks, such as part-of-speech tagging or chunking, where multilingual joint training could show similar gains.
  • The reported margins over prior state of the art are often only 1 to 3 $F_1$ points, so an ablation varying the square-root sampling temperature and the evaluation weighting would show whether the SOTA claim is robust or partly an artifact of the training schedule.
  • The strong zero-shot results on languages unseen during training suggest the model learns transferable span representations; probing those representations with more low-resource languages would clarify how far the transfer extends.
  • The nested seq2seq head is trained only on English and Czech; combining it with the multilingual flat encoder could provide nested NER for other languages as soon as nested corpora exist.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. NameTag 3 is an open-source named entity recognition tool and web service supporting flat and nested NER. The paper describes a single 355M-parameter multilingual XLM-R model with separate per-tagset classification heads for flat NER, trained jointly on 21 corpora in three tagsets, and a 126M seq2seq model for Czech nested NER. It evaluates the system on 21 in-domain flat NER datasets, 7 out-of-domain datasets, and 3 nested NER datasets, comparing against Stanza, SpaCy, leaderboard SOTA, and DeepSeek-R1/ChatGPT models, and reports throughput measurements. The central claim is that the system achieves state-of-the-art results on 21 test datasets in 15 languages.

Significance. If the empirical claims were fully supported, this would be a practically significant contribution: the code is open-source (MPL 2.0), models and a running web service are released, the single-model design supports multiple tagsets and cross-lingual transfer, and the LLM comparison with throughput measurements is a useful resource trade-off demonstration. The breadth of evaluation—28 flat datasets plus 3 nested datasets—is a strength, as is the explicit reproduction script for the LLM baselines. However, the headline SOTA count is not supported by the paper's own tables, and the absence of variance or significance information makes several sub-1-point margins unreliable. The tool itself is valuable, but the paper needs an accurate and precisely qualified statement of where it exceeds, matches, or trails published results.

major comments (3)
  1. [Abstract, §1, §5.1, Table 3] The claim that NameTag 3 'achieves state-of-the-art results on 21 test datasets in 15 languages' is directly contradicted by Table 3. For the Multi model, the cited SOTA baseline is higher on seven of the 21 in-domain rows: Arabic OntoNotes v5 (74.20 vs. 76.40), Dutch CoNLL-2002 (94.93 vs. 95.70), English OntoNotes v5 (90.19 vs. 92.07), English CoNLL-2003 (94.09 vs. 94.60), German CoNLL-2003 (87.48 vs. 88.38), Maghrebi UNER Arabizi (84.49 vs. 86.20), and Spanish CoNLL-2002 (90.29 vs. 90.40). Only eleven rows exceed the cited SOTA, and three rows have no prior SOTA, so at most fourteen of the 21 in-domain datasets are 'not below SOTA'. Adding the seven out-of-domain rows in Table 7, where Tagalog Ugnayan is also below SOTA, still gives only seventeen datasets above the cited baseline, not twenty-one. The language list is likewise inaccurate: English is listed as a SOTA language while both English datasets in Table 3 are below SOTA, and Tagalog is listed while Ugnayan is below SOTA. The headline must be corrected to report the actual number of datasets on which the model exceeds the cited baseline, and the criterion for calling a language a 'SOTA language' should be defined explicitly.
  2. [§5.1, Tables 3 and 7] The SOTA comparisons mix heterogeneous baselines and are partly circular. In Table 3, the Ukrainian baseline is 'NameTag 2' (110M), the authors' own previous system, which is not an independent state-of-the-art result. Other SOTA numbers come from different sources with very different parameter scales (e.g., 1117M–1853M for the Wang et al. rows vs. 355M for Mayhew et al.), making head-to-head SOTA comparisons informative only if the training data, tagsets, and evaluation splits are shown to be equivalent. For the Mayhew et al. baseline, the table caption says the better of the 'in-language' and 'all' models is reported, but the table does not state which setting corresponds to each row. The authors should either report a single consistent baseline regime or clearly tag each baseline's source, parameter count, and training setup.
  3. [§5.1, Tables 3 and 7] No variance or significance information is reported. The tables give a single F1 value per system, but several SOTA margins are close to or below 1 F1 point: Multi is 0.11 below SOTA on Spanish CoNLL-2002, 0.51 below on English CoNLL-2003, 0.49 above on Portuguese UNER Bosque, and 1.10 below SOTA on Tagalog Ugnayan. Without multiple seeds, confidence intervals, or a significance test, the exact list of 'SOTA datasets' is not robust. At minimum, the paper should state whether results are single runs or averaged, and it should soften the SOTA language to 'competitive with published SOTA' where margins are within a plausible run-to-run variation.
minor comments (5)
  1. [§4.1] The text says the training objective is a 'macro span-based F1 score with uniform weighting', while Table 3 reports span-based micro F1. Please clarify whether the macro objective is a multi-dataset average and how it relates to the micro F1 numbers reported in the results.
  2. [§5.1] In the paragraph following Table 4, '20 percent points higher' should be '20 percentage points higher'.
  3. [§3.1] The list of datasets mixes UNER, CoNLL, OntoNotes, CNEC, and Lang-uk corpora; a small table or column indicating the source benchmark and tagset for each row would improve readability and reduce ambiguity about the harmonization procedure.
  4. [Limitations] The Limitations section states that joint multilingual training makes NameTag 3 'less prone to biases' that affect monolingual models, but no bias evaluation is presented. This should be framed as a hope or design rationale, not as an empirical finding.
  5. [§5.2, Table 6] The caption of Table 6 would benefit from stating explicitly that the CNEC 2.0 row uses the original 46-type/4-container annotation, whereas the Czech row in Table 3 uses the harmonized 4-label CoNLL tagset; otherwise the two Czech results can be confused.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; the only circularity-adjacent items are two minor self-baselines (NameTag 2) that are not treated as authoritative. The SOTA mismatch between the abstract and Table 3 is a correctness/consistency issue, not circularity.

full rationale

This is an empirical systems paper, not a derivation. NameTag 3's central result is measured span-based micro F1 on held-out official test splits (Tables 3, 6, and 7), and the baseline numbers come from published leaderboards, the UniversalNER paper, or directly measured toolkits such as Stanza and SpaCy. There is no equation in which a claimed prediction is defined in terms of a fitted input: the training criterion uses square-root temperature sampling and a balanced objective, while evaluation is uniform span-based micro F1 on unseen test sections, so the evaluation numbers are not forced by the training procedure. The only circularity-adjacent elements are two baselines taken from the authors' own prior system: the Ukrainian flat SOTA '88.73 NameTag 2 110M' (Table 3) and the Czech nested SOTA '83.44 NameTag 2 110M' (Table 6). These are legitimate, falsifiable comparisons to a previously released system evaluated on the same test data, and NameTag 3 must actually surpass those measured scores (92.88 vs. 88.73; 86.39 vs. 83.44). They are therefore minor self-citations that are not load-bearing, exactly the case the rubric assigns score 2. The seq2seq head from Straková et al. (2019) is an architectural choice rather than an unverified premise used to forbid alternatives. The skeptic's point that the abstract claims SOTA on 21 datasets while Table 3 shows Multi below the cited SOTA on 7 of 21 in-domain rows is an internal-consistency and correctness concern about how 'SOTA' is counted (e.g., Mono vs. Multi, leaderboard selection, datasets with no prior SOTA), not a circularity. Under the hard rules, no specific reduction by construction or fitted-input-renamed-as-prediction can be quoted, so the honest finding is 0-2, with 2 reflecting the two minor non-load-bearing self-baselines.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper rests on standard supervised fine-tuning assumptions plus a choice of sampling strategy and hyperparameters. It introduces no new theoretical entities and its free parameters are all training choices, not fitted constants used as predictions.

free parameters (2)
  • Square-root temperature sampling exponent = 0.5 (fixed a priori)
    Downsampling/upsampling rate across the 21 corpora is chosen by the authors and is not swept; results may shift with this choice.
  • Training hyperparameters = See Table 2 (epochs, LR, batch size, warmup)
    Manual choices that affect reported F1, standard for fine-tuning papers.
assumptions (3)
  • domain assumption Stanza and SpaCy baselines are frozen legacy models; a comparison on their released checkpoints is a fair performance reference.
    Used in the claim that NameTag 3 beats both toolkits on all comparable datasets; the baseline versions are not fully specified.
  • domain assumption Public leaderboard SOTA numbers are directly comparable despite differing tag harmonization and evaluation scripts.
    Adopted when comparing to Mayhew et al. and other SOTA sources; the paper notes the UNER baselines come from specific tables but does not audit the scripts.
  • domain assumption The trained 355M multilingual model is fine-tuned from a pretrained language model that has no prior exposure to the test corpora.
    Not explicitly stated in the paper, but standard for fine-tuned PLMs; the paper does not name the exact PLM for the flat model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NameTag 3: A Tool and a Service for Multilingual/Multitagset NER." pith.science (2026). https://pith.science/paper/72HKRH37

@misc{pith2026250605949,
  author       = {Pith},
  title        = {Pith review of: NameTag 3: A Tool and a Service for Multilingual/Multitagset NER},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/72HKRH37}},
  note         = {Machine review of arXiv:2506.05949}
}
read the original abstract

We introduce NameTag 3, an open-source tool and cloud-based web service for multilingual, multidataset, and multitagset named entity recognition (NER), supporting both flat and nested entities. NameTag 3 achieves state-of-the-art results on 21 test datasets in 15 languages and remains competitive on the rest, even against larger models. It is available as a command-line tool and as a cloud-based service, enabling use without local installation. NameTag 3 web service currently provides flat NER for 17 languages, trained on 21 corpora and three NE tagsets, all powered by a single 355M-parameter fine-tuned model; and nested NER for Czech, powered by a 126M fine-tuned model. The source code is licensed under open-source MPL 2.0, while the models are distributed under non-commercial CC BY-NC-SA 4.0. Documentation is available at https://ufal.mff.cuni.cz/nametag, source code at https://github.com/ufal/nametag3, and trained models via https://lindat.cz. The REST service and the web application can be found at https://lindat.mff.cuni.cz/services/nametag/. A demonstration video is available at https://www.youtube.com/watch?v=-gaGnP0IV8A.

Figures

Figures reproduced from arXiv: 2506.05949 by the authors.

Figure 1
Figure 1. Visualization of the nested NER seq2seq decoder with hard attention on the current token. The example [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the flat NER classification [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 12 canonical work pages

  1. [1]

    Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. https://aclanthology.org/C18-1139/ Contextual string embeddings for sequence labeling . In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638--1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics

    work page 2018

  2. [2]

    Abdulrahman Aloraini, Juntao Yu, and Massimo Poesio. 2020. https://aclanthology.org/2020.crac-1.11/ Neural coreference resolution for A rabic . In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference, pages 99--110, Barcelona, Spain (online). Association for Computational Linguistics

    work page 2020

  3. [3]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  4. [4]

    George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) program-tasks, data, and evaluation . Proceedings of LREC, 2

    work page 2004

  5. [5]

    Matthew Honnibal and Ines Montani. 2017. spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing. To appear

    2017

  6. [6]

    Zhiheng Huang, Wei Xu, and Kai Yu. 2015. https://arxiv.org/abs/1508.01991 Bidirectional LSTM-CRF Models for Sequence Tagging . Preprint, arXiv:1508.01991

    arXiv 2015

  7. [7]

    Hongjun Li, Mingzhe Cheng, Zelin Yang, Liqun Yang, and Yansong Chua. 2023. Named entity recognition for chinese based on global pointer and adversarial training. Scientific Reports, 13(1):3242

    work page 2023

  8. [8]

    Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020. https://doi.org/10.18653/v1/2020.acl-main.45 Dice loss for data-imbalanced NLP tasks . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465--476, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.45 2020

Show all 24 references
  1. [9]

    Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 857--867. Association for Computational Linguistics

    2015

  2. [10]

    Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Suppa, Hila Gonen, Joseph Marvin Imperial, B \"o rje Karlsson, Peiqin Lin, Nikola Ljube s i \'c , Lester James Miranda, Barbara Plank, Arij Riabi, and Yuval Pinter. 2024. https://doi.org/10.18653/v1/2024.naacl-long.243 Universa...

    2024 doi

  3. [11]

    Aldrian Obaja Muis and Wei Lu. 2017. Labeling Gaps Between Words: Recognizing Overlapping Mentions with Mention Separators . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2608--2618. Association for Computational Linguistics

    2017

  4. [12]

    Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. https://aclanthology.org/W12-4501/ C o NLL -2012 shared task: Modeling multilingual unrestricted coreference in O nto N otes . In Joint Conference on EMNLP and C o NLL - Shared Task , pag...

    2012

  5. [13]

    Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. https://doi.org/10.18653/v1/2020.acl-demos.14 S tanza: A python natural language processing toolkit for many human languages . In Proceedings of the 58th Annual Meeting of the Association for Co...

    2020 doi

  6. [14]

    Magda S ev c \' i kov \' a , Zden e k Z abokrtsk \' y , and Old r ich Kr u za. 2007. Named Entities in Czech: Annotating Data and Developing NE Tagger . In Lecture Notes in Artificial Intelligence, Proceedings of the 10th International Conference on Text, Speech and Dialogue, ...

    2007

  7. [15]

    Yongliang Shen, Zeqi Tan, Shuhui Wu, Wenqi Zhang, Rongsheng Zhang, Yadong Xi, Weiming Lu, and Yueting Zhuang. 2023. https://doi.org/10.18653/v1/2023.acl-long.698 P rompt NER : Prompt locating and typing for named entity recognition . In Proceedings of the 61st Annual Meeting o...

    2023 doi

  8. [16]

    Jana Strakov \'a , Milan Straka, and Jan Haji c . 2014. https://doi.org/10.3115/v1/P14-5003 Open-source tools for morphology, lemmatization, POS tagging and named entity recognition . In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: Syste...

    2014 doi

  9. [17]

    Jana Strakov \'a , Milan Straka, and Jan Hajic. 2019. https://doi.org/10.18653/v1/P19-1527 Neural architectures for nested NER through linearization . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5326--5331, Florence, Italy....

    2019 doi

  10. [18]

    Tjong Kim Sang

    Erik F. Tjong Kim Sang. 2002. https://doi.org/10.3115/1118853.1118877 Introduction to the CoNLL-2002 Shared Task: Language-independent Named Entity Recognition . In Proceedings of the 6th Conference on Natural Language Learning - Volume 20, COLING-02, pages 1--4, Stroudsburg, ...

    2002

  11. [19]

    Tjong Kim Sang and Fien De Meulder

    Erik F. Tjong Kim Sang and Fien De Meulder. 2003. I ntroduction to the C o N L L -2003 S hared T ask: L anguage- I ndependent N amed E ntity R ecognition. In Proceedings of CoNLL-2003, pages 142--147. Edmonton, Canada

    2003

  12. [20]

    Rob van der Goot, Ahmet \"U st \"u n, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. https://doi.org/10.18653/v1/2021.eacl-demos.22 Massive choice, ample tasks ( M a C h A mp): A toolkit for multi-task learning in NLP . In Proceedings of the 16th Conference of the Euro...

    2021 doi

  13. [21]

    Bailin Wang and Wei Lu. 2018. Neural segmental hypergraphs for overlapping mention recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 204--214. Association for Computational Linguistics

    2018

  14. [22]

    Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. https://doi.org/10.18653/v1/2021.acl-long.206 Automated concatenation of embeddings for structured prediction . In Proceedings of the 59th Annual Meeting of the Association for Comp...

    2021 doi

  15. [23]

    Tingyu Xie, Qi Li, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2024. https://doi.org/10.18653/v1/2024.naacl-short.49 Self-improving for zero-shot named entity recognition with large language models . In Proceedings of the 2024 Conference of the North American Chapter of the Assoc...

    2024 doi

  16. [24]

    Zheng Yuan, Chuanqi Tan, Songfang Huang, and Fei Huang. 2022. https://doi.org/10.18653/v1/2022.findings-acl.250 Fusing heterogeneous factors with triaffine mechanism for nested named entity recognition . In Findings of the Association for Computational Linguistics: ACL 2022, p...

    2022 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.