REVIEW 4 major objections 6 minor 1 cited by
Nested Named-Entity Recognition on Vietnamese COVID-19: Dataset and Experiments
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read These experiments establish a manually annotated Vietnamese COVID-19 NER corpus of 10,271 sentences and 11,128 entities, with PhoBERT-large-CRF plus cross-sentence context scoring best at 82.34 F1 for top-level entities.
desk verdict A potentially useful Vietnamese COVID-19 nested NER dataset is presented, but the entity counts do not add up and the data is not released, so the contribution is currently unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the dataset itself: nested XML-style annotations with eleven entity types and up to four levels, converted into CoNLL-style files for training. The main mechanism for handling nesting is the joint tag, which merges a token's level tags into one composite tag such as HOS DATE+ISO DATE, allowing a standard sequence tagger to predict nested structure. The other mechanism that moves the scores is cross-sentence context: each input example begins with the target sentence and then packs the following sentences up to the model's length limit, which lets the model resolve date entities that are only specified in later sentences, such as an isolation date referred to as 'same day'.
What would settle it
Have independent Vietnamese-speaking annotators re-annotate a random sample of the test set with the same eleven-type schema and measure their agreement with the released labels; if agreement is low, for example below κ≈0.6, the reported F1 scores do not measure a stable ground truth. A secondary quick check is to reconcile the entity count: the abstract says 11,128 entities while Table 2 sums to 17,818.
Extended reading notes
Core claim
The central claim is that a manually produced Vietnamese COVID-19 NER dataset, built from medical reports, news sites, and public sources and cleaned and reviewed by hand, supports an eleven-type nested entity schema, and that a fine-tuned PhoBERT-large model with a CRF head and cross-sentence context is the strongest of the tested recognizers. On the test split, this model reaches F1 82.34 for Level-1 entities and 67.23 for Level-2 entities; per-type results are near-ceiling for NAME, AGE, GENDER, and ADDRESS (F1 95.65 to 99.77), while the overlapping date types score lower but still usable (SYM DATE 74.85, POS DATE 76.18, HOS DATE 77.45, ISO DATE 79.85). The paper treats the nesting order as carrying medical importance, with inner entities more important than outer ones, and attributes the best results to the combination of the pretrained model, the CRF layer, and cross-sentence context rather than to any single component.
Load-bearing premise
Everything rests on the manual annotations being correct and consistent, but the paper does not report any agreement measure between annotators or an independent check of the labels.
Editorial extensions
If this is right
- A reliable automatic extractor at these F1 levels could cut the hours spent manually reading each patient report, because high-confidence fields like name, age, gender, and address can be auto-filled and only ambiguous dates routed to human review.
- The dataset gives Vietnamese NLP a benchmark for nested NER with a clinical and epidemiological schema, complementing general-purpose Vietnamese NER resources.
- The five-way split of date entities makes the task more demanding than ordinary NER but also more directly useful for reconstructing patient timelines.
- The reported gap between Level-1 and Level-2 F1, about 15 points, quantifies how much harder nested recognition is on this schema.
- The joint-tag method shows that nested recognition can be handled by a single sequence tagger without a separate nested-decoding architecture.
Reading between the lines
- Editorial inference: if annotation guidelines and inter-annotator agreement are published, the same eleven-type schema could transfer to other Vietnamese clinical documents, not just COVID-19 reports.
- Editorial inference: the gain from cross-sentence context suggests that explicitly resolving temporal references such as 'same day' would close much of the remaining error on ISO, HOS, and POS dates.
- Editorial inference: because nesting order encodes medical priority, downstream applications could treat the hierarchy itself as structured output, generating for each patient a timeline of contact, quarantine, hospitalization, and confirmation dates.
- Editorial inference: a public release of the corpus with its nesting levels would let other researchers test whether the reported F1 numbers hold under independent annotation audits.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a manually annotated Vietnamese COVID-19 named-entity recognition dataset with nested entities, 11 entity types, up to four nesting levels, and 10,271 sentences. It describes the entity types, the three-phase annotation process, and evaluates BiLSTM-CRF, PhoBERT base/large, and CRF variants with and without cross-sentence context. The authors report that PhoBERTlarge-CRF with cross-sentence context yields the best results, with test F1 of 82.34 for Level-1 and 67.23 for Level-2 entities. The paper does not provide a link to the dataset or code.
Significance. If the dataset were released and its description corrected, it could be a reusable resource for Vietnamese clinical NER, particularly for fine-grained date and contact entities relevant to epidemic tracing. The comparative experiments provide a useful set of baselines, and the cross-sentence context idea is a sensible adaptation of prior work. However, the contribution's significance is currently limited by an unresolved internal inconsistency in the reported dataset size, the absence of any annotation-quality metrics, and the lack of a public data or code release. The circularity concern raised in the stress test does not land: all results are measured on a held-out test split, so there is no circularity in the evaluation.
major comments (4)
- [Abstract and Section 3.2 vs. Table 2] The abstract and Section 3.2 state that the dataset contains 10,271 sentences and 11,128 entities, but Table 2 reports per-type counts that sum to 17,818 and per-level counts (15,870 + 1,501 + 375 + 72) that also sum to 17,818. The intermediate Phase 1 and Phase 2 entity counts (6,481 and 7,069) do not sum to 11,128 either. Because no dataset URL or counting convention is specified, a reader cannot determine whether 11,128 refers to unique mentions, 17,818 to ENAMEX labels, or some other quantity. This internal inconsistency directly affects the central resource claim and must be resolved.
- [Section 3.2 (Annotation)] No inter-annotator agreement, annotation guideline, or adjudication procedure is reported. The paper states that the data was 're-checked manually' and 'reviewed' but gives no quantitative reliability measure such as Cohen's kappa. Since the models are trained and evaluated entirely against this manual annotation, label quality is an unverified load-bearing premise for the reported F1 scores.
- [Sections 4.3–4.4, Tables 3–6] Results are reported as means over five runs without standard deviations, confidence intervals, or significance tests. For example, in Table 5 the Level-1 F1 gap between PhoBERTlarge-CRF with cross-sentence context (82.34) and PhoBERTbase-CRF with cross-sentence context (81.89) is 0.45 points; without variance information it is impossible to judge whether this difference is meaningful. Please report per-run results or error bars and, if possible, significance tests.
- [Dataset availability] The paper provides no link, DOI, or repository for the dataset or the code used in the experiments. For a dataset contribution, this is a major omission: the announced resource cannot be downloaded, and the benchmark numbers cannot be independently reproduced. A public release (or a clear statement of access conditions) is needed for the claims to be verifiable.
minor comments (6)
- [Tables 5 and 6] The numeric entries are malformed in several places, with numbers concatenated together, e.g., '85.7883.9876.29' and '81.1278.63'. These tables need to be regenerated so that each value is clearly separated.
- [References] The reference list appears in full twice, after Section 5 and again after the first copy of the references; the duplicate should be removed.
- [Language and typos] The manuscript contains numerous English typos and grammatical errors (e.g., 'which be defined', 'recognization', 'takes place' in the abstract). A thorough proofreading pass is needed.
- [Section 4.1] The paper says sentences and words were segmented with both RDRsegmenter and Trankit, and then RDRsegmenter was chosen; please explain the selection criterion and why Trankit was discarded.
- [Table 7] It is unclear whether the per-entity results in Table 7 are for Level-1 entities only or include nested levels; please clarify the evaluation scope.
- [Section 3.1] The nested entity examples all use dates; no example is given for nested entities involving other types, and the statement that 'inside entity is more important than outside entity' is not operationalized in the evaluation.
Circularity Check
No circularity: the paper reports a new annotated dataset and standard held-out model evaluations; no prediction reduces to a fitted input or self-citation chain.
full rationale
The paper contains no theoretical derivation whose conclusion is equivalent to an assumption. Its central claims are (1) the construction of a manually annotated Vietnamese COVID-19 nested-NER dataset and (2) empirical F1 scores of BiLSTM and PhoBERT variants measured on a held-out test split. The model results in Tables 3–6 are standard evaluations against manually created labels, not quantities fitted from those labels and then relabeled as predictions. The two self-citations, [Lê et al., 2023] and [Nguyen-Phung et al., 2025], appear only in a related-work sentence noting other recent Vietnamese COVID-19 resources; they are not used to define entity types, to justify the dataset size, or to select the best model. The nested-entity handling (joint tags) and cross-sentence context follow external prior work (VLSP 2018, [Luoma and Pyysalo, 2020], [Minh, 2018]) and are not self-referential. The inconsistency between the abstract's '11128 entities' text and Table 2's sum of 17818 is a data-quality/verifiability defect, not circularity: it does not make any result true by construction. Because no load-bearing step reduces to its own input, the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Learning rate =
5e-5
- Batch size =
32
- Training epochs =
100
assumptions (3)
- domain assumption The manually assigned entity labels are correct and consistent.
- domain assumption PhoBERT representations transfer adequately to this Vietnamese medical and news domain.
- domain assumption Train, validation, and test splits are independent and representative.
Cite this review
Pith. "Pith review of Nested Named-Entity Recognition on Vietnamese COVID-19: Dataset and Experiments." pith.science (2026). https://pith.science/paper/TKQDPOXN
@misc{pith2026250421016,
author = {Pith},
title = {Pith review of: Nested Named-Entity Recognition on Vietnamese COVID-19: Dataset and Experiments},
year = {2026},
howpublished = {\url{https://pith.science/paper/TKQDPOXN}},
note = {Machine review of arXiv:2504.21016}
}
read the original abstract
The COVID-19 pandemic caused great losses worldwide, efforts are taken place to prevent but many countries have failed. In Vietnam, the traceability, localization, and quarantine of people who contact with patients contribute to effective disease prevention. However, this is done by hand, and take a lot of work. In this research, we describe a named-entity recognition (NER) study that assists in the prevention of COVID-19 pandemic in Vietnam. We also present our manually annotated COVID-19 dataset with nested named entity recognition task for Vietnamese which be defined new entity types using for our system.
Forward citations
Cited by 1 Pith paper
-
ViQA-COVID: COVID-19 Machine Reading Comprehension Dataset for Vietnamese
ViQA-COVID is a new Vietnamese COVID-19 reading comprehension dataset with 6,444 question-answer pairs, the first for Vietnamese with multi-span answers, benchmarked at 85.97% F1 by XLM-R large.
Reference graph
Works this paper leans on
-
[1]
[Bandaet al., 2020 ] Juan M. Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, Katya Artemova, Elena Tutubalina, and Gerardo Chowell. A large-scale covid-19 twitter chatter dataset for open sci- entific research – an international collaboration,
work page 2020
-
[3]
[Devlinet al., 2018 ] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing.arXiv preprint arXiv:1810.04805,
arXiv 2018
-
[5]
Bidirectional LSTM-CRF models for sequence tagging
[Huanget al., 2015 ] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991,
arXiv 2015
-
[7]
[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InProceed- ings of the 3rd International Conference for Learning Rep- resentations,
work page 2015
-
[8]
Improving clinical document understanding on covid-19 research with spark nlp,
[Kocaman and Talby, 2020] Veysel Kocaman and David Talby. Improving clinical document understanding on covid-19 research with spark nlp,
work page 2020
-
[9]
[Lˆeet al., 2023 ] Ngoc C. L ˆe, Hai-Chung Nguyen-Phung, Thuy Thu Tran, Ngoc-Uyen Thi Nguyen, Dang-Khoi Pham Nguyen, and Thanh-Huy Nguyen.On Natural Lan- guage Processing to Attack COVID-19 Pandemic: Experi- ences of Vietnam, pages 313–335. Springer International Publishing, Cham,
work page 2023
-
[11]
Trankit: A light-weight transformer-based toolkit for multilingual natural language processing
[Minh Van Nguyen and Nguyen, 2021] Amir Pouran Ben Veyseh Minh Van Nguyen, Viet Lai and Thien Huu Nguyen. Trankit: A light-weight transformer-based toolkit for multilingual natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,
work page 2021
-
[13]
PhoBERT: Pre-trained language models for Vietnamese
[Nguyen and Nguyen, 2020] Dat Quoc Nguyen and Anh Tuan Nguyen. PhoBERT: Pre-trained language models for Vietnamese. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042,
work page 2020
Show all 22 references
-
[14]
Vlsp 2016 shared task: Named entity recognition
[Nguyen and Vu, 2016] Huyen Nguyen and Luong Vu. Vlsp 2016 shared task: Named entity recognition. InPro- ceedings of Vietnamese Speech and Language Processing (VLSP),
2016
-
[19]
Tjong Kim Sang and Fien De Meulder
[Sang and Meulder, 2003] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language-independent named entity recognition. CoRR, cs.CL/0306050,
2003
-
[21]
Weld, Oren Etzioni, and Se- bastian Kohlmeier
[Wanget al., 2020 ] Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Bur- dick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong ...
2020
-
[22]
Extracting covid-19 events from twitter, 2020
[Zonget al., 2020 ] Shi Zong, Ashutosh Baheti, Wei Xu, and Alan Ritter. Extracting covid-19 events from twitter, 2020
2020
-
[2003]
COVID-19 Named Entity Recog- nition for Vietnamese
[Truonget al., 2021 ] Thinh Hung Truong, Mai Hoang Dao, and Dat Quoc Nguyen. COVID-19 Named Entity Recog- nition for Vietnamese. InProceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies,
2021
-
[2015]
Automatic text summariza- tion of covid-19 medical research articles using bert and gpt-2,
[Kieuvongngamet al., 2020 ] Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. Automatic text summariza- tion of covid-19 medical research articles using bert and gpt-2,
2020
-
[2016]
A fast and accu- rate vietnamese word segmenter.CoRR, abs/1709.06307,
[Nguyenet al., 2017 ] Dat Quoc Nguyen, Dai Quoc Nguyen, Thanh Vu, Mark Dras, and Mark Johnson. A fast and accu- rate vietnamese word segmenter.CoRR, abs/1709.06307,
2017 arXiv
-
[2017]
Vlsp shared task: Named entity recognition.Journal of Computer Science and Cy- bernetics, 34(4):283–294,
[Nguyenet al., 2019 ] Huyen Nguyen, Quyen Ngo, Luong Vu, Vu Tran, and Hien Nguyen. Vlsp shared task: Named entity recognition.Journal of Computer Science and Cy- bernetics, 34(4):283–294,
2019
-
[2018]
Weibo-cov: A large-scale covid-19 social media dataset from weibo,
[Huet al., 2020 ] Yong Hu, Heyan Huang, Anfan Chen, and Xian-Ling Mao. Weibo-cov: A large-scale covid-19 social media dataset from weibo,
2020
-
[2019]
Viqa-covid: Covid-19 machine reading comprehension dataset for vietnamese
[Nguyen-Phunget al., 2025 ] Hai-Chung Nguyen-Phung, Ngoc C L ˆe, Van-Chien Nguyen, Hang Thi Nguyen, and Thuy Phuong Thi Nguyen. Viqa-covid: Covid-19 machine reading comprehension dataset for vietnamese. arXiv preprint arXiv:2504.21017,
2025 arXiv
-
[2020]
[Chenet al., 2020 ] Q. Chen, A. Allot, and Z. Lu. Keep up with the latest coronavirus research.Nature, 579(7798):193,
2020
-
[2021]
A feature-based model for nested named-entity recognition at VLSP-2018 NER evaluation campaign.CoRR, abs/1803.08463,
[Minh, 2018] Pham Quang Nhat Minh. A feature-based model for nested named-entity recognition at VLSP-2018 NER evaluation campaign.CoRR, abs/1803.08463,
2018 arXiv
-
[2023]
Exploring cross-sentence contexts for named entity recognition with bert
[Luoma and Pyysalo, 2020] Jouni Luoma and Sampo Pyysalo. Exploring cross-sentence contexts for named entity recognition with bert. InProceedings of the 28th Models Level-1 Joint Prec. Recall F1 Prec. Recall F1 Bi-LSTM-CRF 66.06 75.31 70.38 65.21 68.51 66.82 PhoBERTbase 81.31 8...
2020
-
[2025]
fairseq: A fast, extensible toolkit for se- quence modeling
[Ottet al., 2019 ] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for se- quence modeling. InProceedings of NAACL-HLT 2019: Demonstrations,
2019
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.