Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Nested Named-Entity Recognition on Vietnamese COVID-19: Dataset and Experiments

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read These experiments establish a manually annotated Vietnamese COVID-19 NER corpus of 10,271 sentences and 11,128 entities, with PhoBERT-large-CRF plus cross-sentence context scoring best at 82.34 F1 for top-level entities.

desk verdict A potentially useful Vietnamese COVID-19 nested NER dataset is presented, but the entity counts do not add up and the data is not released, so the contribution is currently unverifiable. read the letter →

arxiv 2504.21016 v2 pith:TKQDPOXN submitted 2025-04-21 cs.CL cs.LG

classification cs.CLcs.LG
keywords nestednamedentityrecognitionVietnameseNLPCOVID-19datasetcontacttracingPhoBERTclinicaltextminingannotationcross-sentencecontext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to build an automatic information-extraction aid for Vietnam's COVID-19 contact-tracing effort, where patient interviews and movement reports are currently processed by hand. To that end, it introduces a manually annotated nested named-entity recognition dataset for Vietnamese: 10,271 sentences and 11,128 entities, labelled with eleven entity types covering patient demographics, five date types, contact persons, and locations, with nesting up to four levels. The paper then compares BiLSTM-CRF and PhoBERT-based models, adding a CRF layer and cross-sentence context, and reports that PhoBERT-large-CRF with cross-sentence context gives the best test F1 of 82.34 for outer-level entities and 67.23 for the second nesting level. A sympathetic reader would take away that automatic extraction of the dates and contacts needed for contact tracing is feasible in Vietnamese, at least for the high-frequency fields.

What carries the argument

The load-bearing object is the dataset itself: nested XML-style annotations with eleven entity types and up to four levels, converted into CoNLL-style files for training. The main mechanism for handling nesting is the joint tag, which merges a token's level tags into one composite tag such as HOS DATE+ISO DATE, allowing a standard sequence tagger to predict nested structure. The other mechanism that moves the scores is cross-sentence context: each input example begins with the target sentence and then packs the following sentences up to the model's length limit, which lets the model resolve date entities that are only specified in later sentences, such as an isolation date referred to as 'same day'.

What would settle it

Have independent Vietnamese-speaking annotators re-annotate a random sample of the test set with the same eleven-type schema and measure their agreement with the released labels; if agreement is low, for example below κ≈0.6, the reported F1 scores do not measure a stable ground truth. A secondary quick check is to reconcile the entity count: the abstract says 11,128 entities while Table 2 sums to 17,818.

Watch

Extended reading notes

Core claim

The central claim is that a manually produced Vietnamese COVID-19 NER dataset, built from medical reports, news sites, and public sources and cleaned and reviewed by hand, supports an eleven-type nested entity schema, and that a fine-tuned PhoBERT-large model with a CRF head and cross-sentence context is the strongest of the tested recognizers. On the test split, this model reaches F1 82.34 for Level-1 entities and 67.23 for Level-2 entities; per-type results are near-ceiling for NAME, AGE, GENDER, and ADDRESS (F1 95.65 to 99.77), while the overlapping date types score lower but still usable (SYM DATE 74.85, POS DATE 76.18, HOS DATE 77.45, ISO DATE 79.85). The paper treats the nesting order as carrying medical importance, with inner entities more important than outer ones, and attributes the best results to the combination of the pretrained model, the CRF layer, and cross-sentence context rather than to any single component.

Load-bearing premise

Everything rests on the manual annotations being correct and consistent, but the paper does not report any agreement measure between annotators or an independent check of the labels.

Editorial extensions

If this is right

  • A reliable automatic extractor at these F1 levels could cut the hours spent manually reading each patient report, because high-confidence fields like name, age, gender, and address can be auto-filled and only ambiguous dates routed to human review.
  • The dataset gives Vietnamese NLP a benchmark for nested NER with a clinical and epidemiological schema, complementing general-purpose Vietnamese NER resources.
  • The five-way split of date entities makes the task more demanding than ordinary NER but also more directly useful for reconstructing patient timelines.
  • The reported gap between Level-1 and Level-2 F1, about 15 points, quantifies how much harder nested recognition is on this schema.
  • The joint-tag method shows that nested recognition can be handled by a single sequence tagger without a separate nested-decoding architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if annotation guidelines and inter-annotator agreement are published, the same eleven-type schema could transfer to other Vietnamese clinical documents, not just COVID-19 reports.
  • Editorial inference: the gain from cross-sentence context suggests that explicitly resolving temporal references such as 'same day' would close much of the remaining error on ISO, HOS, and POS dates.
  • Editorial inference: because nesting order encodes medical priority, downstream applications could treat the hierarchy itself as structured output, generating for each patient a timeline of contact, quarantine, hospitalization, and confirmation dates.
  • Editorial inference: a public release of the corpus with its nesting levels would let other researchers test whether the reported F1 numbers hold under independent annotation audits.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a manually annotated Vietnamese COVID-19 named-entity recognition dataset with nested entities, 11 entity types, up to four nesting levels, and 10,271 sentences. It describes the entity types, the three-phase annotation process, and evaluates BiLSTM-CRF, PhoBERT base/large, and CRF variants with and without cross-sentence context. The authors report that PhoBERTlarge-CRF with cross-sentence context yields the best results, with test F1 of 82.34 for Level-1 and 67.23 for Level-2 entities. The paper does not provide a link to the dataset or code.

Significance. If the dataset were released and its description corrected, it could be a reusable resource for Vietnamese clinical NER, particularly for fine-grained date and contact entities relevant to epidemic tracing. The comparative experiments provide a useful set of baselines, and the cross-sentence context idea is a sensible adaptation of prior work. However, the contribution's significance is currently limited by an unresolved internal inconsistency in the reported dataset size, the absence of any annotation-quality metrics, and the lack of a public data or code release. The circularity concern raised in the stress test does not land: all results are measured on a held-out test split, so there is no circularity in the evaluation.

major comments (4)
  1. [Abstract and Section 3.2 vs. Table 2] The abstract and Section 3.2 state that the dataset contains 10,271 sentences and 11,128 entities, but Table 2 reports per-type counts that sum to 17,818 and per-level counts (15,870 + 1,501 + 375 + 72) that also sum to 17,818. The intermediate Phase 1 and Phase 2 entity counts (6,481 and 7,069) do not sum to 11,128 either. Because no dataset URL or counting convention is specified, a reader cannot determine whether 11,128 refers to unique mentions, 17,818 to ENAMEX labels, or some other quantity. This internal inconsistency directly affects the central resource claim and must be resolved.
  2. [Section 3.2 (Annotation)] No inter-annotator agreement, annotation guideline, or adjudication procedure is reported. The paper states that the data was 're-checked manually' and 'reviewed' but gives no quantitative reliability measure such as Cohen's kappa. Since the models are trained and evaluated entirely against this manual annotation, label quality is an unverified load-bearing premise for the reported F1 scores.
  3. [Sections 4.3–4.4, Tables 3–6] Results are reported as means over five runs without standard deviations, confidence intervals, or significance tests. For example, in Table 5 the Level-1 F1 gap between PhoBERTlarge-CRF with cross-sentence context (82.34) and PhoBERTbase-CRF with cross-sentence context (81.89) is 0.45 points; without variance information it is impossible to judge whether this difference is meaningful. Please report per-run results or error bars and, if possible, significance tests.
  4. [Dataset availability] The paper provides no link, DOI, or repository for the dataset or the code used in the experiments. For a dataset contribution, this is a major omission: the announced resource cannot be downloaded, and the benchmark numbers cannot be independently reproduced. A public release (or a clear statement of access conditions) is needed for the claims to be verifiable.
minor comments (6)
  1. [Tables 5 and 6] The numeric entries are malformed in several places, with numbers concatenated together, e.g., '85.7883.9876.29' and '81.1278.63'. These tables need to be regenerated so that each value is clearly separated.
  2. [References] The reference list appears in full twice, after Section 5 and again after the first copy of the references; the duplicate should be removed.
  3. [Language and typos] The manuscript contains numerous English typos and grammatical errors (e.g., 'which be defined', 'recognization', 'takes place' in the abstract). A thorough proofreading pass is needed.
  4. [Section 4.1] The paper says sentences and words were segmented with both RDRsegmenter and Trankit, and then RDRsegmenter was chosen; please explain the selection criterion and why Trankit was discarded.
  5. [Table 7] It is unclear whether the per-entity results in Table 7 are for Level-1 entities only or include nested levels; please clarify the evaluation scope.
  6. [Section 3.1] The nested entity examples all use dates; no example is given for nested entities involving other types, and the statement that 'inside entity is more important than outside entity' is not operationalized in the evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports a new annotated dataset and standard held-out model evaluations; no prediction reduces to a fitted input or self-citation chain.

full rationale

The paper contains no theoretical derivation whose conclusion is equivalent to an assumption. Its central claims are (1) the construction of a manually annotated Vietnamese COVID-19 nested-NER dataset and (2) empirical F1 scores of BiLSTM and PhoBERT variants measured on a held-out test split. The model results in Tables 3–6 are standard evaluations against manually created labels, not quantities fitted from those labels and then relabeled as predictions. The two self-citations, [Lê et al., 2023] and [Nguyen-Phung et al., 2025], appear only in a related-work sentence noting other recent Vietnamese COVID-19 resources; they are not used to define entity types, to justify the dataset size, or to select the best model. The nested-entity handling (joint tags) and cross-sentence context follow external prior work (VLSP 2018, [Luoma and Pyysalo, 2020], [Minh, 2018]) and are not self-referential. The inconsistency between the abstract's '11128 entities' text and Table 2's sum of 17818 is a data-quality/verifiability defect, not circularity: it does not make any result true by construction. Because no load-bearing step reduces to its own input, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central result is an empirical resource and benchmark. It relies on standard supervised machine learning assumptions, namely that labels are correct, splits are representative, and pretrained features transfer, rather than on new physical or mathematical postulates. No free parameters beyond ordinary training hyperparameters are introduced for the core claim, and no new theoretical entities are postulated.

free parameters (3)
  • Learning rate = 5e-5
    Chosen by hand for PhoBERT fine-tuning; a standard value, not fitted to the evaluation metric. It is a hyperparameter rather than a scientific constant.
  • Batch size = 32
    Chosen by hand; affects results but is not central to the scientific claim.
  • Training epochs = 100
    Selected with validation-based evaluation after each epoch; no explicit early stopping or best-epoch selection criterion is reported.
assumptions (3)
  • domain assumption The manually assigned entity labels are correct and consistent.
    Section 3.2 describes the annotation process but reports no inter-annotator agreement; all training and evaluation assume label reliability.
  • domain assumption PhoBERT representations transfer adequately to this Vietnamese medical and news domain.
    All best models rely on PhoBERT pretraining (Section 4.2); if the pretrained representations do not cover the domain vocabulary, the model results would not hold.
  • domain assumption Train, validation, and test splits are independent and representative.
    The paper does not describe de-duplication across splits or leakage checks after combining medical reports and crawled news (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Nested Named-Entity Recognition on Vietnamese COVID-19: Dataset and Experiments." pith.science (2026). https://pith.science/paper/TKQDPOXN

@misc{pith2026250421016,
  author       = {Pith},
  title        = {Pith review of: Nested Named-Entity Recognition on Vietnamese COVID-19: Dataset and Experiments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKQDPOXN}},
  note         = {Machine review of arXiv:2504.21016}
}
read the original abstract

The COVID-19 pandemic caused great losses worldwide, efforts are taken place to prevent but many countries have failed. In Vietnam, the traceability, localization, and quarantine of people who contact with patients contribute to effective disease prevention. However, this is done by hand, and take a lot of work. In this research, we describe a named-entity recognition (NER) study that assists in the prevention of COVID-19 pandemic in Vietnam. We also present our manually annotated COVID-19 dataset with nested named entity recognition task for Vietnamese which be defined new entity types using for our system.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ViQA-COVID: COVID-19 Machine Reading Comprehension Dataset for Vietnamese

    cs.CL 2025-04 conditional novelty 6.0 of 10

    ViQA-COVID is a new Vietnamese COVID-19 reading comprehension dataset with 6,444 question-answer pairs, the first for Vietnamese with multi-span answers, benchmarked at 85.97% F1 by XLM-R large.

Reference graph

Works this paper leans on

22 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, Katya Artemova, Elena Tutubalina, and Gerardo Chowell

    [Bandaet al., 2020 ] Juan M. Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, Katya Artemova, Elena Tutubalina, and Gerardo Chowell. A large-scale covid-19 twitter chatter dataset for open sci- entific research – an international collaboration,

  2. [3]

    Bert: Pre-training of deep bidirectional transformers for language understand- ing.arXiv preprint arXiv:1810.04805,

    [Devlinet al., 2018 ] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing.arXiv preprint arXiv:1810.04805,

  3. [5]

    Bidirectional LSTM-CRF models for sequence tagging

    [Huanget al., 2015 ] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991,

  4. [7]

    Kingma and Jimmy Ba

    [Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InProceed- ings of the 3rd International Conference for Learning Rep- resentations,

  5. [8]

    Improving clinical document understanding on covid-19 research with spark nlp,

    [Kocaman and Talby, 2020] Veysel Kocaman and David Talby. Improving clinical document understanding on covid-19 research with spark nlp,

  6. [9]

    [Lˆeet al., 2023 ] Ngoc C. L ˆe, Hai-Chung Nguyen-Phung, Thuy Thu Tran, Ngoc-Uyen Thi Nguyen, Dang-Khoi Pham Nguyen, and Thanh-Huy Nguyen.On Natural Lan- guage Processing to Attack COVID-19 Pandemic: Experi- ences of Vietnam, pages 313–335. Springer International Publishing, Cham,

  7. [11]

    Trankit: A light-weight transformer-based toolkit for multilingual natural language processing

    [Minh Van Nguyen and Nguyen, 2021] Amir Pouran Ben Veyseh Minh Van Nguyen, Viet Lai and Thien Huu Nguyen. Trankit: A light-weight transformer-based toolkit for multilingual natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,

  8. [13]

    PhoBERT: Pre-trained language models for Vietnamese

    [Nguyen and Nguyen, 2020] Dat Quoc Nguyen and Anh Tuan Nguyen. PhoBERT: Pre-trained language models for Vietnamese. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042,

Show all 22 references
  1. [14]

    Vlsp 2016 shared task: Named entity recognition

    [Nguyen and Vu, 2016] Huyen Nguyen and Luong Vu. Vlsp 2016 shared task: Named entity recognition. InPro- ceedings of Vietnamese Speech and Language Processing (VLSP),

  2. [19]

    Tjong Kim Sang and Fien De Meulder

    [Sang and Meulder, 2003] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language-independent named entity recognition. CoRR, cs.CL/0306050,

  3. [21]

    Weld, Oren Etzioni, and Se- bastian Kohlmeier

    [Wanget al., 2020 ] Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Bur- dick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong ...

  4. [22]

    Extracting covid-19 events from twitter, 2020

    [Zonget al., 2020 ] Shi Zong, Ashutosh Baheti, Wei Xu, and Alan Ritter. Extracting covid-19 events from twitter, 2020

  5. [2003]

    COVID-19 Named Entity Recog- nition for Vietnamese

    [Truonget al., 2021 ] Thinh Hung Truong, Mai Hoang Dao, and Dat Quoc Nguyen. COVID-19 Named Entity Recog- nition for Vietnamese. InProceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies,

  6. [2015]

    Automatic text summariza- tion of covid-19 medical research articles using bert and gpt-2,

    [Kieuvongngamet al., 2020 ] Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. Automatic text summariza- tion of covid-19 medical research articles using bert and gpt-2,

  7. [2016]

    A fast and accu- rate vietnamese word segmenter.CoRR, abs/1709.06307,

    [Nguyenet al., 2017 ] Dat Quoc Nguyen, Dai Quoc Nguyen, Thanh Vu, Mark Dras, and Mark Johnson. A fast and accu- rate vietnamese word segmenter.CoRR, abs/1709.06307,

  8. [2017]

    Vlsp shared task: Named entity recognition.Journal of Computer Science and Cy- bernetics, 34(4):283–294,

    [Nguyenet al., 2019 ] Huyen Nguyen, Quyen Ngo, Luong Vu, Vu Tran, and Hien Nguyen. Vlsp shared task: Named entity recognition.Journal of Computer Science and Cy- bernetics, 34(4):283–294,

  9. [2018]

    Weibo-cov: A large-scale covid-19 social media dataset from weibo,

    [Huet al., 2020 ] Yong Hu, Heyan Huang, Anfan Chen, and Xian-Ling Mao. Weibo-cov: A large-scale covid-19 social media dataset from weibo,

  10. [2019]

    Viqa-covid: Covid-19 machine reading comprehension dataset for vietnamese

    [Nguyen-Phunget al., 2025 ] Hai-Chung Nguyen-Phung, Ngoc C L ˆe, Van-Chien Nguyen, Hang Thi Nguyen, and Thuy Phuong Thi Nguyen. Viqa-covid: Covid-19 machine reading comprehension dataset for vietnamese. arXiv preprint arXiv:2504.21017,

  11. [2020]

    [Chenet al., 2020 ] Q. Chen, A. Allot, and Z. Lu. Keep up with the latest coronavirus research.Nature, 579(7798):193,

  12. [2021]

    A feature-based model for nested named-entity recognition at VLSP-2018 NER evaluation campaign.CoRR, abs/1803.08463,

    [Minh, 2018] Pham Quang Nhat Minh. A feature-based model for nested named-entity recognition at VLSP-2018 NER evaluation campaign.CoRR, abs/1803.08463,

  13. [2023]

    Exploring cross-sentence contexts for named entity recognition with bert

    [Luoma and Pyysalo, 2020] Jouni Luoma and Sampo Pyysalo. Exploring cross-sentence contexts for named entity recognition with bert. InProceedings of the 28th Models Level-1 Joint Prec. Recall F1 Prec. Recall F1 Bi-LSTM-CRF 66.06 75.31 70.38 65.21 68.51 66.82 PhoBERTbase 81.31 8...

  14. [2025]

    fairseq: A fast, extensible toolkit for se- quence modeling

    [Ottet al., 2019 ] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for se- quence modeling. InProceedings of NAACL-HLT 2019: Demonstrations,

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.