{"id":"51a12c3c-fecf-4068-a243-a14c93bc066d","arxiv_id":"2504.19606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-4 outperforms GPT-3.5 on coreference resolution for a new, small Vietnamese narrative dataset annotated for human entities only.","lead":"The authors built a small Vietnamese coreference dataset from VnExpress narrative texts and tested how well GPT-4 and GPT-3.5 can find mentions and link them into clusters. GPT-4 scored a CoNLL F1 of 0.735 versus 0.478 for GPT-3.5, but the dataset is not released and annotation quality is unmeasured.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation leaks gold mentions into the prompt: test documents are pre-tagged with gold spans, so the reported F1 scores measure clustering with oracle mentions, not end-to-end coreference resolution.","rationale":"The paper's central claim is that GPT-4 significantly outperforms GPT-3.5 on Vietnamese coreference resolution. The load-bearing condition is that the evaluation actually measures coreference resolution. Section 3's indexing step injects gold mention boundaries into every test prompt, so the model is only asked to cluster tags. Mention detection, a core part of coreference resolution, is bypassed. This is more fundamental than the reader's annotation-quality concern: even a flawless gold standard would not fix the mismatch between the task tested and the task claimed. The reader's weakest assumption targets the gold standard's accuracy, but the oracle-mention design undermines the evaluation regardless of gold quality. The concrete test is a raw-text re-run; if the performance gap shrinks or disappears, the main comparison is an artifact of the protocol. The absence of inter-annotator agreement and error bars remains relevant, but the primary reason the current claim is unsupported is the prompt leakage of gold mentions.","tokens_in":10054,"tokens_out":6777,"duration_ms":69268,"concrete_test":"Take a random sample of 30 evaluation texts and re-run both models with the same few-shot examples but with raw text input (no [mention]#tag markup), instructing the model to return both mention spans and cluster labels; parse and score with MUC, B-Cubed, CEAFphi, and CoNLL F1. If end-to-end GPT-4 CoNLL F1 falls well below 0.735, or the gap over GPT-3.5 narrows, the Table 2 advantage is attributable to the oracle mentions.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3 states that before prompting, each test document is formatted into indexed text using the authors' manual annotation, producing text like '[Cha tôi]#1 rất yêu thương [gia đình]#2 ...' and the LLM is asked to output clusters of these tag indices. Thus every mention boundary and every mention index in the input comes from the gold clusters; the model is never required to detect mentions. The CoNLL F1 numbers in Table 2 consequently score only the ability to group provided mention tags into entities. A model that simply places all tags into one cluster would earn non-trivial MUC scores, and mention detection errors are impossible by construction. This is a mismatch with the central claim that GPT-4 is 'more reliable for coreference resolution'; at best the results support 'GPT-4 is better at clustering gold mentions.' The 'response consistency' claim is also anecdotal (Section 4.1), with no quantitative measure or error analysis. This problem is independent of annotation quality: even perfectly correct gold clusters would not make the reported scores a valid end-to-end coreference benchmark. Without raw-text prompting and predicted-mention evaluation, the headline comparison is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces a manually annotated Vietnamese coreference corpus: 266 narrative texts sampled from an earlier VnExpress collection, annotated for human-related mentions and clusters according to stated guidelines. The authors then evaluate GPT-3.5-Turbo and GPT-4 with a few-shot prompting protocol on 263 held-out texts, reporting CoNLL F1 scores of 0.478 and 0.735 respectively, and conclude that GPT-4 is significantly more accurate and more consistent for Vietnamese coreference resolution.","tokens_in":10213,"tokens_out":8098,"duration_ms":80858,"significance":"The dataset and annotation guidelines are potentially useful resources for low-resource Vietnamese NLP, and the paper is transparent about its prompting protocol and metric choices. The central empirical claim, however, is only partially supported by the experiment as described: because the test documents are pre-tagged with gold mentions, the benchmark measures the ability to cluster oracle mentions, not end-to-end coreference resolution. If the evaluation were redone on raw text, or the claims were carefully rescoped to clustering with gold mentions, the comparison could be informative; as it stands, the abstract's claim that GPT-4 is a 'more reliable tool for coreference resolution' overstates what Table 2 shows.","major_comments":[{"comment":"The evaluation leaks gold mention information into the test input. In Section 3, after constructing gold_clusters, each remaining document is converted to indexed text such as '[Em]#1 ... [các bạn]#3 ...', and this indexed text is sent to the LLM as the input. The mention boundaries and tag indices are therefore taken directly from the authors' gold annotation, so the model never performs mention detection; it only assigns the provided tags to clusters. The CoNLL F1 scores in Table 2 thus measure clustering with oracle mentions, and they cannot support the abstract's claim that GPT-4 is a more reliable tool for coreference resolution, which is an end-to-end task. Please either re-run the evaluation with raw text and predicted mentions, or explicitly rescope the paper's claims and title to 'clustering with gold mentions' and adjust the conclusions accordingly.","section":"Section 3 (prompt construction) and Table 2"},{"comment":"No inter-annotator agreement is reported. The gold standard was produced manually by volunteers using the SACR tool, but the paper does not state how many annotators worked on each text, whether any double annotation or adjudication occurred, or what agreement was achieved (e.g., mention-level agreement or Krippendorff's alpha). Since gold_clusters are the reference for all F1 scores, annotation noise directly biases both models' scores and the reported gap; without quality metrics the dataset cannot be fully assessed as a benchmark.","section":"Section 2.2 and Table 1"},{"comment":"The reported scores come from a single evaluation run with no error bars. LLM API outputs are stochastic, and the paper itself notes that GPT-3.5-Turbo sometimes returned malformed or full-text outputs; a single sample per document cannot establish that GPT-4 'significantly outperforms' GPT-3.5-Turbo. Please report results over multiple runs (e.g., with different sampling seeds or temperatures), include per-metric variance or confidence intervals, and provide a statistical test of the difference.","section":"Section 4.2, Table 2"},{"comment":"The response-consistency claim is anecdotal and not quantified. The bullet in Section 4.1 describes observations about irrelevant content and full-text outputs, but Table 2 contains no consistency metric, and no counts or rates are given. In addition, the post-processing needed to 'extract the final result' from GPT-3.5-Turbo outputs is not described; if nontrivial, it could affect the reported scores. Please define a measurable consistency metric (e.g., format-valid response rate across repeated calls) and document the post-processing pipeline.","section":"Section 4.1 (Response Consistency) and Abstract"}],"minor_comments":[{"comment":"The displayed CEAF recall formula appears to contain a typo: it is written as Σφ(Gi,Gi)/Σφ(Gi,Gi), which would always equal 1; the numerator should be the similarity of the aligned gold/predicted pair. Please correct the equations.","section":"Section 4.1 (CEAF equations)"},{"comment":"The caption says the process generates mention clusters 'from raw text using LLMs', but the input shown is already indexed with gold mention tags; the caption should say 'from gold-indexed text' or otherwise align with the actual protocol.","section":"Figure 1"},{"comment":"The annotation guidelines cover only human-related entities (people, groups, organizations), yet the title and conclusion discuss 'coreference resolution' without this restriction; please qualify the scope as human-entity coreference throughout.","section":"Section 2.2.2 and Section 5"},{"comment":"The paper does not state the data release plan or provide a link to the annotated dataset or code, which is important for a resource paper; please include an availability statement.","section":"Section 3 (data availability)"},{"comment":"The case studies are illustrative and do not quantify how often the described error patterns occur; adding a small error analysis with counts would make the discussion more informative.","section":"Section 4.3"},{"comment":"Please clarify the units of 'Average length', 'Average mention', and 'Average entity' (per text) in the table caption or in the surrounding text.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core evaluation issue is significant: as written, the benchmark is not end-to-end coreference resolution because gold mention spans are supplied in the prompt. The dataset contribution is still potentially valuable, but the current abstract and conclusions overstate what the experiment demonstrates. I would be willing to review a revised version that either re-runs the evaluation with raw-text inputs and predicted mentions or carefully rescopes the claims to clustering with oracle mentions, and that adds the missing annotation-agreement and runs-level statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one because it touches the low-resource coreference nerve, but the central number doesn't mean what the authors say it means. The test documents are pre-tagged with gold mention boundaries and indices; the LLM is given \"[Cha tôi]#1 rất yêu...\" and asked to output clusters of indices. So GPT-4's 0.735 CoNLL F1 is for clustering oracle mentions, not for detecting and resolving mentions from raw text. The paper claims GPT-4 is \"a more reliable tool for coreference resolution in Vietnamese,\" which is not supported by the experimental setup.\n\nCredit: they built a genuinely new annotated dataset of 266 Vietnamese narrative texts, focused on human entities, with explicit guidelines and a few-shot prompting protocol. That is a real resource if released. The use of MUC/B-cubed/CEAF is standard and appropriate for the sub-task they actually evaluate. And the result that GPT-4 clusters gold mentions better than GPT-3.5 is plausible and likely reproducible.\n\nSoft spots beyond the leakage: no inter-annotator agreement, so the gold standard's reliability is unknown; the dataset is not released; there is no error bar or multi-run statistics (single API call per document); the \"response consistency\" claim in §4.1 is anecdotal. Also, the paper does not survey prior Vietnamese coreference resources, which may be because there are none, but it should state that.\n\nThe annotation effort is worthwhile, and with a raw-text prompting setup, predicted mention evaluation, IAA numbers, and a released corpus, the comparison would be meaningful. As it stands, treat it as a pilot study on mention clustering, not a coreference benchmark. Worth a referee's time if the editor asks for major revision; I wouldn't desk-reject it, but the authors need to reframe and add the missing evidence.","headline":"The claimed coreference numbers don't measure what the paper says: models are given gold mention spans and only asked to cluster them, so the comparison is about mention clustering, not end-to-end coreference resolution.","tokens_in":10780,"tokens_out":2990,"would_cite":false,"duration_ms":28946,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4 outperforms GPT-3.5 on Vietnamese coreference resolution, CoNLL F1 0.735 vs 0.478.","keywords":["coreference resolution","Vietnamese NLP","low-resource languages","annotated dataset","few-shot prompting","GPT-4 evaluation","CoNLL F1","entity annotation guidelines"],"falsifier":"Have two or more independent annotators re-annotate a random sample of the 263 evaluation texts using the same guidelines, measure inter-annotator agreement, and recompute the score difference against each annotator's clusters; if agreement is low or the GPT-4 advantage shrinks under re-annotation, the reported gap is an artifact of the gold standard rather than a stable model difference.","tokens_in":1716,"feed_emoji":"🏷️","tokens_out":4230,"duration_ms":91153,"temperature":0.7,"pith_summary":"The paper works to establish that coreference resolution for Vietnamese, a low-resource language, can be tackled by prompting general-purpose large language models rather than training task-specific systems. To do this, it manually annotates 266 narrative Vietnamese texts (3 used for few-shot examples, 263 for evaluation) with entity mentions and gold coreference clusters, following written guidelines, and evaluates GPT-3.5-Turbo and GPT-4 under a few-shot prompt. The paper reports that GPT-4 reaches a CoNLL F1 of 0.735 against 0.478 for GPT-3.5-Turbo, with GPT-4 winning on every component metric and producing more format-consistent responses. If true, the dataset and protocol give Vietnamese NLP a reusable benchmark and show that a current large language model can partially close the annotation-resource gap for a low-resource language.","feed_headline":"GPT-4 tops GPT-3.5 on Vietnamese coreference: 0.735 vs 0.478","feed_subtitle":"A new 266-text Vietnamese benchmark has GPT-4 winning every coreference metric over GPT-3.5.","key_machinery":"The load-bearing mechanism is a pipeline that turns raw narrative text into indexed mentions and cluster tuples: texts are manually annotated with an open-source coreference annotation tool, converted to [mention]#tag form, grouped into gold_clusters as arrays of tuples, and then a few-shot prompt feeds three worked examples to the LLM so that it must return only the tuple list. The evaluation then runs the standard coreference metrics MUC, B-Cubed, and $\\mathrm{CEAF}_{\\phi}$, averaged into the CoNLL F1 score. This machinery lets the paper compare model outputs directly against human clusters without free-form text parsing, and its cluster-tuple format is what makes GPT-4's format consistency an explicit part of the measured reliability.","core_discovery":"The central claim is that, given a small set of gold examples embedded in a prompt, GPT-4 performs Vietnamese coreference resolution markedly better than GPT-3.5-Turbo: on 263 held-out narrative texts it attains CoNLL F1 0.735 (MUC 0.858, B-Cubed 0.723, CEAF$\\phi$ 0.625) versus GPT-3.5-Turbo's 0.478 (MUC 0.640, B-Cubed 0.474, CEAF$\\phi$ 0.321), and its outputs adhere to the requested tuple format more consistently. The paper also claims that its manual annotation effort, with guidelines covering people and groups of people, adjective exclusion, and nested possessive mentions, yields a usable gold standard for Vietnamese narrative coreference.","pith_inferences":["The paper's prompt-based comparison does not control for output formatting: GPT-3.5's extraneous text forced extra cleanup, so part of the F1 gap may reflect format adherence rather than coreference judgment; a structured-output or post-normalized comparison could isolate the linguistic difference.","Since no inter-annotator agreement is reported, the dataset's quality ceiling is unknown; if independent re-annotation shows high agreement, the resource could support fine-tuning smaller Vietnamese models that might rival GPT-4 at lower cost.","The entity definition excludes non-human entities, places, and objects, so the reported scores cover person-centric narratives only; extending the guidelines to locations, objects, and organizations would test whether the GPT-4 advantage persists on harder entity types.","The same few-shot protocol could be ported to other low-resource languages with similar pronoun systems, making the paper's pipeline a template for cross-lingual coreference evaluation without retraining."],"forward_implications":["GPT-4 can serve as a practical few-shot baseline for Vietnamese coreference resolution, needing only a handful of annotated examples rather than a fine-tuned model.","The 263-text gold standard gives later work a benchmark against which future Vietnamese coreference systems can be measured.","The metric breakdown locates where improvements are needed, since GPT-4's weakest component is entity-level alignment (CEAF$\\phi$ 0.625) rather than link or mention-level scoring.","The format-consistency finding implies that for LLM-based annotation pipelines, structured output reliability is part of the practical performance cost, not just clustering ability.","The annotation guidelines define a reproducible mention scheme for Vietnamese (humans and human groups, with nested possessive mentions), which later dataset efforts can adopt for comparability."],"supporting_citations":[{"why":"Defines the MUC link-based precision and recall used as one of the three CoNLL component metrics.","marker":"(Vilain et al., 1995)"},{"why":"Provides the B-Cubed mention-level clustering precision and recall used in the evaluation.","marker":"(Bagga and Baldwin, 1998)"},{"why":"Introduces the CEAF-phi entity-alignment scoring used as the third CoNLL component.","marker":"(Luo, 2005)"},{"why":"Defines CoNLL F1 as the average of MUC, B-Cubed, and CEAF-phi in the shared-task setup the paper follows.","marker":"(Pradhan et al., 2011)"},{"why":"Supplies the few-shot prompting paradigm the paper relies on to get LLM outputs from three annotated examples.","marker":"(Brown et al., 2020)"},{"why":"Provides the raw 1,041-text Vietnamese narrative corpus from which the paper samples its 266 texts.","marker":"(Nguyen et al., 2023)"},{"why":"Supplies the annotation tool used to build the gold mentions and clusters.","marker":"(Oberle, 2018)"}],"fun_headline_variants":["GPT-4 tops GPT-3.5 on Vietnamese coreference: 0.735 vs 0.478","Vietnamese coreference: GPT-4 hits 0.735 F1, GPT-3.5 only 0.478","New Vietnamese coreference benchmark: GPT-4 far exceeds GPT-3.5","GPT-4 beats GPT-3.5 by 0.257 F1 on Vietnamese coreference"],"cache_read_input_tokens":12928,"weakest_assumption_plain":"The manually annotated gold clusters are accurate, complete, and consistent enough to serve as ground truth, and the paper reports no inter-annotator agreement, so any annotation error directly shifts the reported F1 scores and the GPT-4-versus-GPT-3.5 gap.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 tops GPT-3.5 on Vietnamese coreference: 0.735 vs 0.478","Vietnamese coreference: GPT-4 hits 0.735 F1, GPT-3.5 only 0.478","New Vietnamese coreference benchmark: GPT-4 far exceeds GPT-3.5","GPT-4 beats GPT-3.5 by 0.257 F1 on Vietnamese coreference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3790,"prompt_tokens":857,"completion_tokens":2933,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":2826}},"tokens_in":473,"tokens_out":2933,"duration_ms":18603,"temperature":1.0,"reasoning_tokens":2826,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:47:40.797928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two or more independent annotators re-annotate a random sample of the 263 evaluation texts using the same guidelines, measure inter-annotator agreement, and recompute the score difference against each annotator's clusters; if agreement is low or the GPT-4 advantage shrinks under re-annotation, the reported gap is an artifact of the gold standard rather than a stable model difference.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MUC link-based precision and recall used as one of the three CoNLL component metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the B-Cubed mention-level clustering precision and recall used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the CEAF-phi entity-alignment scoring used as the third CoNLL component."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines CoNLL F1 as the average of MUC, B-Cubed, and CEAF-phi in the shared-task setup the paper follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the annotation tool used to build the gold mentions and clusters."}],"review_version":1}