Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Coreference Resolution for Vietnamese Narrative Texts

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GPT-4 outperforms GPT-3.5 on Vietnamese coreference resolution, CoNLL F1 0.735 vs 0.478.

desk verdict The claimed coreference numbers don't measure what the paper says: models are given gold mention spans and only asked to cluster them, so the comparison is about mention clustering, not end-to-end coreference resolution. read the letter →

arxiv 2504.19606 v1 pith:YCYPT5XT submitted 2025-04-28 cs.CL

Hieu-Dai Tran , Duc-Vu Nguyen , Ngan Luu-Thuy Nguyen This is my paper · ORCID
classification cs.CL
keywords coreferenceresolutionVietnameseNLPlow-resourcelanguagesannotateddatasetfew-shotpromptingGPT-4evaluationCoNLLF1entityannotationguidelines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper works to establish that coreference resolution for Vietnamese, a low-resource language, can be tackled by prompting general-purpose large language models rather than training task-specific systems. To do this, it manually annotates 266 narrative Vietnamese texts (3 used for few-shot examples, 263 for evaluation) with entity mentions and gold coreference clusters, following written guidelines, and evaluates GPT-3.5-Turbo and GPT-4 under a few-shot prompt. The paper reports that GPT-4 reaches a CoNLL F1 of 0.735 against 0.478 for GPT-3.5-Turbo, with GPT-4 winning on every component metric and producing more format-consistent responses. If true, the dataset and protocol give Vietnamese NLP a reusable benchmark and show that a current large language model can partially close the annotation-resource gap for a low-resource language.

What carries the argument

The load-bearing mechanism is a pipeline that turns raw narrative text into indexed mentions and cluster tuples: texts are manually annotated with an open-source coreference annotation tool, converted to [mention]#tag form, grouped into gold_clusters as arrays of tuples, and then a few-shot prompt feeds three worked examples to the LLM so that it must return only the tuple list. The evaluation then runs the standard coreference metrics MUC, B-Cubed, and $\mathrm{CEAF}_{\phi}$, averaged into the CoNLL F1 score. This machinery lets the paper compare model outputs directly against human clusters without free-form text parsing, and its cluster-tuple format is what makes GPT-4's format consistency an explicit part of the measured reliability.

What would settle it

Have two or more independent annotators re-annotate a random sample of the 263 evaluation texts using the same guidelines, measure inter-annotator agreement, and recompute the score difference against each annotator's clusters; if agreement is low or the GPT-4 advantage shrinks under re-annotation, the reported gap is an artifact of the gold standard rather than a stable model difference.

Watch

Extended reading notes

Core claim

The central claim is that, given a small set of gold examples embedded in a prompt, GPT-4 performs Vietnamese coreference resolution markedly better than GPT-3.5-Turbo: on 263 held-out narrative texts it attains CoNLL F1 0.735 (MUC 0.858, B-Cubed 0.723, CEAF$\phi$ 0.625) versus GPT-3.5-Turbo's 0.478 (MUC 0.640, B-Cubed 0.474, CEAF$\phi$ 0.321), and its outputs adhere to the requested tuple format more consistently. The paper also claims that its manual annotation effort, with guidelines covering people and groups of people, adjective exclusion, and nested possessive mentions, yields a usable gold standard for Vietnamese narrative coreference.

Load-bearing premise

The manually annotated gold clusters are accurate, complete, and consistent enough to serve as ground truth, and the paper reports no inter-annotator agreement, so any annotation error directly shifts the reported F1 scores and the GPT-4-versus-GPT-3.5 gap.

Editorial extensions

If this is right

  • GPT-4 can serve as a practical few-shot baseline for Vietnamese coreference resolution, needing only a handful of annotated examples rather than a fine-tuned model.
  • The 263-text gold standard gives later work a benchmark against which future Vietnamese coreference systems can be measured.
  • The metric breakdown locates where improvements are needed, since GPT-4's weakest component is entity-level alignment (CEAF$\phi$ 0.625) rather than link or mention-level scoring.
  • The format-consistency finding implies that for LLM-based annotation pipelines, structured output reliability is part of the practical performance cost, not just clustering ability.
  • The annotation guidelines define a reproducible mention scheme for Vietnamese (humans and human groups, with nested possessive mentions), which later dataset efforts can adopt for comparability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's prompt-based comparison does not control for output formatting: GPT-3.5's extraneous text forced extra cleanup, so part of the F1 gap may reflect format adherence rather than coreference judgment; a structured-output or post-normalized comparison could isolate the linguistic difference.
  • Since no inter-annotator agreement is reported, the dataset's quality ceiling is unknown; if independent re-annotation shows high agreement, the resource could support fine-tuning smaller Vietnamese models that might rival GPT-4 at lower cost.
  • The entity definition excludes non-human entities, places, and objects, so the reported scores cover person-centric narratives only; extending the guidelines to locations, objects, and organizations would test whether the GPT-4 advantage persists on harder entity types.
  • The same few-shot protocol could be ported to other low-resource languages with similar pronoun systems, making the paper's pipeline a template for cross-lingual coreference evaluation without retraining.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript introduces a manually annotated Vietnamese coreference corpus: 266 narrative texts sampled from an earlier VnExpress collection, annotated for human-related mentions and clusters according to stated guidelines. The authors then evaluate GPT-3.5-Turbo and GPT-4 with a few-shot prompting protocol on 263 held-out texts, reporting CoNLL F1 scores of 0.478 and 0.735 respectively, and conclude that GPT-4 is significantly more accurate and more consistent for Vietnamese coreference resolution.

Significance. The dataset and annotation guidelines are potentially useful resources for low-resource Vietnamese NLP, and the paper is transparent about its prompting protocol and metric choices. The central empirical claim, however, is only partially supported by the experiment as described: because the test documents are pre-tagged with gold mentions, the benchmark measures the ability to cluster oracle mentions, not end-to-end coreference resolution. If the evaluation were redone on raw text, or the claims were carefully rescoped to clustering with gold mentions, the comparison could be informative; as it stands, the abstract's claim that GPT-4 is a 'more reliable tool for coreference resolution' overstates what Table 2 shows.

major comments (4)
  1. [Section 3 (prompt construction) and Table 2] The evaluation leaks gold mention information into the test input. In Section 3, after constructing gold_clusters, each remaining document is converted to indexed text such as '[Em]#1 ... [các bạn]#3 ...', and this indexed text is sent to the LLM as the input. The mention boundaries and tag indices are therefore taken directly from the authors' gold annotation, so the model never performs mention detection; it only assigns the provided tags to clusters. The CoNLL F1 scores in Table 2 thus measure clustering with oracle mentions, and they cannot support the abstract's claim that GPT-4 is a more reliable tool for coreference resolution, which is an end-to-end task. Please either re-run the evaluation with raw text and predicted mentions, or explicitly rescope the paper's claims and title to 'clustering with gold mentions' and adjust the conclusions accordingly.
  2. [Section 2.2 and Table 1] No inter-annotator agreement is reported. The gold standard was produced manually by volunteers using the SACR tool, but the paper does not state how many annotators worked on each text, whether any double annotation or adjudication occurred, or what agreement was achieved (e.g., mention-level agreement or Krippendorff's alpha). Since gold_clusters are the reference for all F1 scores, annotation noise directly biases both models' scores and the reported gap; without quality metrics the dataset cannot be fully assessed as a benchmark.
  3. [Section 4.2, Table 2] The reported scores come from a single evaluation run with no error bars. LLM API outputs are stochastic, and the paper itself notes that GPT-3.5-Turbo sometimes returned malformed or full-text outputs; a single sample per document cannot establish that GPT-4 'significantly outperforms' GPT-3.5-Turbo. Please report results over multiple runs (e.g., with different sampling seeds or temperatures), include per-metric variance or confidence intervals, and provide a statistical test of the difference.
  4. [Section 4.1 (Response Consistency) and Abstract] The response-consistency claim is anecdotal and not quantified. The bullet in Section 4.1 describes observations about irrelevant content and full-text outputs, but Table 2 contains no consistency metric, and no counts or rates are given. In addition, the post-processing needed to 'extract the final result' from GPT-3.5-Turbo outputs is not described; if nontrivial, it could affect the reported scores. Please define a measurable consistency metric (e.g., format-valid response rate across repeated calls) and document the post-processing pipeline.
minor comments (6)
  1. [Section 4.1 (CEAF equations)] The displayed CEAF recall formula appears to contain a typo: it is written as Σφ(Gi,Gi)/Σφ(Gi,Gi), which would always equal 1; the numerator should be the similarity of the aligned gold/predicted pair. Please correct the equations.
  2. [Figure 1] The caption says the process generates mention clusters 'from raw text using LLMs', but the input shown is already indexed with gold mention tags; the caption should say 'from gold-indexed text' or otherwise align with the actual protocol.
  3. [Section 2.2.2 and Section 5] The annotation guidelines cover only human-related entities (people, groups, organizations), yet the title and conclusion discuss 'coreference resolution' without this restriction; please qualify the scope as human-entity coreference throughout.
  4. [Section 3 (data availability)] The paper does not state the data release plan or provide a link to the annotated dataset or code, which is important for a resource paper; please include an availability statement.
  5. [Section 4.3] The case studies are illustrative and do not quantify how often the described error patterns occur; adding a small error analysis with counts would make the discussion more informative.
  6. [Table 1] Please clarify the units of 'Average length', 'Average mention', and 'Average entity' (per text) in the table caption or in the surrounding text.

Circularity Check

1 steps flagged · score 6.0 of 10

Gold mention tags from the manual annotation are baked into every test prompt, so the headline 'coreference resolution' result measures only clustering of oracle mentions.

  1. self definitional [Section 3, 'Prompting', third paragraph (gold_clusters construction) and numbered steps 1-2; results in Section 4.2, Table 2]
    "Before building the prompt, we also had to take a few steps to refine the annotated dataset to build the gold_clusters, which is the expected result that we want the LLMs to return. First, we'll format the text annotated with the format {M{#tag_number} mention} to [mention]{#tag_index}."

    The indexed text fed to the LLM is produced from the manually annotated gold spans: each [mention]#tag in the input is exactly a gold mention with a gold index. The model is asked to output only tuples of these pre-assigned tag indices, and is scored against gold_clusters over the same indices. By construction, mention detection is not evaluated: the model cannot miss a mention or invent a spurious mention, so mention-detection errors are impossible. The reported MUC, B-Cubed, CEAF, and CoNLL F1 scores therefore measure only the subproblem of grouping provided gold mentions, while the abstract and conclusion claim that GPT-4 is a more reliable tool for 'coreference resolution,' which centrally includes identifying mentions.

full rationale

This paper is an empirical benchmark rather than a derivation, and it contains no fitted parameters and no load-bearing self-citation chain; the GPT-4 versus GPT-3.5-Turbo comparison is a real measurement of a constrained clustering task. The circular component is structural: the same manual annotation that defines the gold_clusters is used to insert gold mention boundaries and tag indices into every test prompt, so the reported scores cannot include mention detection errors. The central claim about coreference resolution as 'identifying and linking' expressions is therefore only partially supported; the evaluated task is grouping pre-located, gold-indexed mentions. The clustering comparison itself retains independent empirical content and is not forced by the annotation, so a score of 6 rather than 8 is appropriate. The absence of inter-annotator agreement and the anecdotal 'response consistency' evidence are quality and reporting limitations, not circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's central claim does not depend on fitting any numeric parameters to data; the evaluation scores are measured, not optimized. The main assumptions are about the quality and representativeness of the gold annotations and the few-shot prompt, both of which are untested in the paper.

assumptions (3)
  • standard math CoNLL F1, MUC, B-Cubed, and CEAF metrics are appropriate for evaluating coreference resolution.
    The paper uses these standard metrics from the coreference literature without questioning their validity.
  • domain assumption The three few-shot examples are representative enough to teach the task to the models.
    The prompt performance depends on the examples, but no sensitivity analysis is provided.
  • domain assumption The random selection of 266 texts from the original 1,041 preserves the diversity of narrative styles.
    The paper does not describe the selection seed or compare characteristics between the subset and the full set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Coreference Resolution for Vietnamese Narrative Texts." pith.science (2026). https://pith.science/paper/YCYPT5XT

@misc{pith2026250419606,
  author       = {Pith},
  title        = {Pith review of: Coreference Resolution for Vietnamese Narrative Texts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YCYPT5XT}},
  note         = {Machine review of arXiv:2504.19606}
}
read the original abstract

Coreference resolution is a vital task in natural language processing (NLP) that involves identifying and linking different expressions in a text that refer to the same entity. This task is particularly challenging for Vietnamese, a low-resource language with limited annotated datasets. To address these challenges, we developed a comprehensive annotated dataset using narrative texts from VnExpress, a widely-read Vietnamese online news platform. We established detailed guidelines for annotating entities, focusing on ensuring consistency and accuracy. Additionally, we evaluated the performance of large language models (LLMs), specifically GPT-3.5-Turbo and GPT-4, on this dataset. Our results demonstrate that GPT-4 significantly outperforms GPT-3.5-Turbo in terms of both accuracy and response consistency, making it a more reliable tool for coreference resolution in Vietnamese.

Figures

Figures reproduced from arXiv: 2504.19606 by the authors.

Figure 1
Figure 1. The process of generating mention clusters from raw text using LLMs. The input document is processed [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. User interface of the Coreference Annotation Tool with SACR [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CORE-KG: An LLM-Driven Knowledge Graph Construction Framework for Human Smuggling Networks

    cs.CL 2025-06 conditional novelty 5.0 of 10

    CORE-KG reduces node duplication by 33.28% and legal noise by 38.37% versus a GraphRAG baseline on 20 human smuggling court cases, through type-aware LLM coreference resolution and domain-filtered extraction prompts.

Reference graph

Works this paper leans on

31 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference on language resources and evaluation workshop on linguistic coreference, pages 563--566

    work page 1998

  4. [4]

    Brown, Benjamin Mann, Nick Ryder, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  5. [5]

    Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. arXiv preprint arXiv:1609.08667

    arXiv 2016

  6. [6]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440--8451

    2020

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    arXiv 2018

  8. [8]

    Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723

    arXiv 2020

Show all 31 references
  1. [9]

    Ralph Grishman. 1997. Information extraction: Techniques and challenges. International Summer School on Information Extraction

    1997

  2. [10]

    Phu Gia Hoang, Canh Duc Luu, Khanh Quoc Tran, Kiet Van Nguyen, and Ngan Luu-Thuy Nguyen. 2023. Vihos: Hate speech spans detection for vietnamese. arXiv preprint arXiv:2301.10186

    2023 arXiv

  3. [11]

    Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90\ In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57--60

    2006

  4. [12]

    Daniel Jurafsky and James H. Martin. 2014. Speech and Language Processing. Prentice Hall

    2014

  5. [13]

    Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. arXiv preprint arXiv:1707.07045

    2017 arXiv

  6. [14]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th An...

    2020

  7. [15]

    Bill Yuchen Lin, Ziyi Wu, Sue Lee, Yichi Wang, and Xiang Ren. 2021. Few-shot learning with multilingual generative language models. arXiv preprint arXiv:2112.10668

    2021 arXiv

  8. [16]

    Pengfei Liu et al. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586

    2021 arXiv

  9. [17]

    Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the 2005 conference on empirical methods in natural language processing, pages 25--32

    2005

  10. [18]

    Ruslan Mitkov. 1998. A corpus-based approach to pronoun resolution. In Proceedings of the 17th international conference on Computational linguistics

    1998

  11. [19]

    Vincent Ng. 2010. Machine learning for coreference resolution: From local classification to global ranking. Proceedings of the ACL

    2010

  12. [20]

    Dat Quoc Nguyen, Dai Quoc Nguyen, Dang-Khoa Le-Tuan Nguyen, Son Bao Nguyen, and Son T. Pham. 2018. Vncorenlp: A vietnamese natural language processing toolkit. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:...

    2018

  13. [21]

    Nhu-Thanh Nguyen, Khoa Thi-Kim Phan, Duc-Vu Nguyen, and Ngan Luu-Thuy Nguyen. 2023. https://arxiv.org/abs/2312.07831 Abusive span detection for vietnamese narrative texts . arXiv preprint arXiv:2312.07831

    2023 arXiv

  14. [22]

    Benedikt Oberle. 2018. https://aclanthology.org/L18-1059 Sacr: A drag-and-drop based tool for coreference annotation . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA)

    2018

  15. [23]

    Massimo Poesio and Ron Artstein. 2008. Anaphora resolution: State of the art. In Proceedings of the ACL

    2008

  16. [24]

    Sameer Pradhan, Lance Ramshaw, Mitch Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1--27

    2011

  17. [25]

    Sameer Pradhan et al. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Proceedings of the CoNLL-2012

    2012

  18. [26]

    Alec Radford, Jeff Wu, Rewon Child, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog

    2019

  19. [27]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1--67

    2020

  20. [28]

    Altaf Rahman and Vincent Ng. 2012. Coreference resolution in a low-resource language: Hindi. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 41--46

    2012

  21. [29]

    Josef Steinberger and Karel Jezek. 2007. Text summarization within the information retrieval framework. Proceedings of the 7th International Conference on Text, Speech and Dialogue

    2007

  22. [30]

    Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6), pages 45--52

    1995

  23. [31]

    Jason Wei et al. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3859--3871

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.