{"id":"94f29f27-9e7d-4bef-b556-5d9fa59c0d79","arxiv_id":"2504.15052","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ChatGPT with a detailed prompt identifies and labels specialized translation errors at F1=0.71 on DeepL outputs, but at F1=0.50 on its own translations.","lead":"This study tests whether ChatGPT can spot and label errors in scientific abstracts translated into French, using a custom error typology. It finds moderately good performance on DeepL translations, but much weaker performance when ChatGPT evaluates its own translations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F1=0.707 is measured against one expert's gold standard using any-character-overlap matching, with no human-human agreement baseline; the 'partial substitute' claim is not yet established.","rationale":"The reader's conditional verdict is appropriate, and I agree with its weakest-assumption identification. The central applied claim is substitution for human annotation; that requires knowing how much two experts agree with each other. The paper's own limitations section acknowledges proprietary-model instability but not the absence of an inter-annotator baseline, and the overlap rule is presented as a practical choice rather than a validated proxy. The study is transparent and the DeepL long-prompt result is plausible; the issue is not internal inconsistency but the lack of a reference point that makes the headline F1 interpretable. A same-set second annotator plus exact-span recomputation would settle whether the concern lands. Because the reader already conditioned the verdict on this type of assumption, no verdict change is needed.","tokens_in":15960,"tokens_out":6174,"duration_ms":58500,"concrete_test":"Have a second professional translator independently annotate the same 35 DeepL translations using the same typology and manual. Compute pairwise expert-expert F1 with (a) the shared-character rule and (b) exact span matching, and compare both to ChatGPT's scores against each expert. Recompute the reported DeepL F1 with exact span matching and with single-label reference accuracy. If ChatGPT-human F1 is materially below expert-expert F1, or exact-match F1 drops well below 0.707, the 'partial substitute' claim should be downgraded. Also report the distribution of predicted span lengths to test whether long spans drive overlap-based true positives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (F1=0.707, precision 0.792, recall 0.653 for DeepL with the long prompt) is computed in Section 4.3 by counting a predicted error as correct if it shares at least one character with a reference error, and the reference is the annotation of a single professional translator using the authors' typology and manual. Two consequences follow. First, the overlap rule can inflate both precision and recall: a predicted span that touches any part of a gold error counts as a true positive, so long or loosely placed predicted spans can match several gold errors at once. The paper does not report predicted span lengths or an exact-match variant, and the reference errors carry up to six labels, which also makes the 64.1% label-accuracy figure easier to achieve than a single-label metric would. Second, no human-human agreement baseline is provided. If a second expert annotating the same 35 texts agrees with the first at F1=0.85, ChatGPT's 0.707 is a clear gap; if human-human agreement is around 0.70, ChatGPT is already within the range of human variability. Without this baseline, the applied claim that ChatGPT is a 'partial substitute' for human annotation is not interpretable. The self-evaluation comparison inherits the same gold-standard limitation and is further confounded because the DeepL and ChatGPT subcorpora use different source texts (35 vs 25 abstracts), so the reported self-evaluation penalty could reflect text difficulty rather than model bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a pilot study of using ChatGPT (GPT-4o) to annotate machine translations according to a specialized error typology. The authors built a corpus of French translations of NLP research abstracts: 35 texts translated by DeepL and 25 different texts translated by ChatGPT, each annotated by a single professional translator. They prompted ChatGPT with a long, detailed French prompt and a shorter variant, matched predicted error spans to the human reference by requiring at least one shared character, and computed document-level precision, recall, F1, and label-accuracy rates with bootstrap confidence intervals. The main results are that for DeepL translations the long prompt reaches F1 = 0.707 (precision 0.792, recall 0.653) with 64.1% correct error-label assignment; the short prompt has similar F1 but markedly lower label accuracy (46.9%); and for ChatGPT's own translations the long prompt yields F1 = 0.496 with 45.3% label accuracy. The authors interpret these results as evidence that LLM-based annotation is promising for specialized translation evaluation, that prompt detail mainly helps categorization, and that ChatGPT shows a clear self-evaluation penalty.","tokens_in":16244,"tokens_out":6177,"duration_ms":53524,"significance":"If the reported numbers survive robustness checks, this would be a useful contribution to the emerging literature on LLM-based translation evaluation, extending it to LSP translation and to fine-grained error annotation rather than quality scoring. The study has several strengths: the evaluation is anchored to an external human gold standard rather than to a fitted model, the authors report bootstrap confidence intervals for the main detection scores, they compare prompt variants, and they release the annotated corpus with a DOI, which is a concrete asset for future work. The paper is also transparent about the proprietary and non-reproducible nature of ChatGPT. However, the central quantitative claims currently rest on a lenient span-matching rule, a single-annotator gold standard, and a self-evaluation comparison that is confounded by different source texts. These issues are fixable within the scope of the manuscript, but they need to be addressed before the practical claim of a 'partial substitute' for human annotation can be considered established.","major_comments":[{"comment":"The one-character-overlap matching rule can materially inflate both precision and recall. A predicted span that overlaps several gold errors is counted as correct for each of them, and the evaluation only excludes the reverse case (a reference error matched by two predictions). This is especially consequential because reference errors range up to 81–103 characters and carry up to six labels. The paper does not report an exact-match or boundary-aware variant, predicted span lengths, or the number of multi-overlap cases. The 64.1% label-accuracy figure is also hard to interpret without knowing how often a predicted label could match by chance given the multiple reference labels. Please report at least one stricter span criterion (e.g., exact boundary match or token-level F1), the distribution of predicted span lengths and overlap counts, and label accuracy under single-label reference coding.","section":"§4.3, Eqs. (1)–(2) and footnote 13"},{"comment":"The reference annotations come from a single professional translator, and no inter-annotator agreement is reported. Since the typology is partly subjective, the headline F1 = 0.707 can only be interpreted relative to the human-human agreement level for this task: if two experts agree at F1 ≈ 0.85, then ChatGPT has a clear gap; if human agreement is around 0.70, ChatGPT is already within the range of human variability. Without this baseline, the statement that ChatGPT is a viable partial substitute for human annotation is not quantitatively interpretable. Please add an inter-annotator study on a subset of the corpus, or at minimum cite and use existing agreement data for this typology.","section":"§4.2 and §6"},{"comment":"The self-evaluation comparison is confounded by the use of different source texts in the two subcorpora: 35 source texts for DeepL and 25 different source texts for ChatGPT, with different error densities (399 vs. 193 gold errors). The reported drop from F1 = 0.707 to F1 = 0.496 could therefore reflect text difficulty or annotation differences rather than a self-evaluation bias. To support the paper's conclusion about self-assessment, please evaluate both translation systems on the same source texts, or at minimum have ChatGPT annotate DeepL translations of the same 25 source texts it translated itself, and report the controlled comparison.","section":"§4.4 and Table 1"},{"comment":"The claim that adding error definitions to the prompt improves categorization is based on the point estimates 64.1% vs. 46.9% label accuracy, but no confidence intervals or statistical tests are reported for label accuracy, and the F1 confidence intervals for the long and short prompts overlap substantially (0.707 ± 0.039 vs. 0.702 ± 0.053). Please provide per-document paired differences and bootstrap confidence intervals for label accuracy, and test whether the categorization difference is statistically reliable rather than attributable to prompt-instruction variability.","section":"§5 and Table 1"}],"minor_comments":[{"comment":"The model name is misspelled as \"ChatPT\" in the first experiment description; it should be \"ChatGPT\".","section":"§4.4"},{"comment":"The x-axis labels are run together in the caption and provide no legend keys for the three conditions; a box plot or per-document strip plot with clear labels would make the variability claims easier to verify.","section":"Figure 2"},{"comment":"The statement that the results \"seem likely\" to apply to other text types is not supported by the experimental design, since only NLP abstracts are tested; please soften or remove this generalization.","section":"§4.1"},{"comment":"The error typology is presented as a compact figure that is hard to read; please include a machine-readable table with all codes and definitions, especially the ten terminology subtypes, so that readers can check the labels used in the evaluation.","section":"Appendix A"},{"comment":"For reproducibility, please report the exact model version, access date, and decoding settings (temperature, top-p, number of runs) used for every ChatGPT call, since the paper itself notes that ChatGPT behavior fluctuates over time.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is well founded: the lenient matching rule and the single-annotator gold standard are the two load-bearing points that need to be addressed. The authors appear aware of many limitations, and the corpus release is a genuine asset, but the abstract and Section 6 currently state conclusions that are stronger than the evidence supports. I would ask for an inter-annotator baseline, an exact-match analysis, and a controlled self-evaluation comparison before reconsidering the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a solid, transparent pilot study, not an overclaiming one. On DeepL translations of NLP abstracts into French, GPT-4o with a detailed prompt finds errors at F1=0.707 (precision 0.79, recall 0.65) against one expert's gold standard, and correctly labels 64% of those. The same model with a shorter prompt does about the same at detection but drops to 47% label accuracy. On its own translations, F1 falls to 0.496 and more than half of its flagged errors are false positives. Those prompt-detail and self-evaluation effects are the genuinely new observations.\n\nThe work is also reproducible in a way a lot of LLM papers aren't: the corpus is released, the prompts are in the appendix, and the bootstrap CIs are reported. That's real value for translation-training researchers who want to build on it.\n\nThe soft spots are in the evaluation design, not the reporting. The gold standard comes from a single professional translator, and the paper doesn't provide a human-human agreement baseline. Without that, we can't tell whether ChatGPT's 0.707 is good or merely acceptable — if two experts agree at 0.85, there's a real gap; if they agree around 0.70, the model is already in human range. The matching rule adds to that uncertainty: any single shared character between a predicted and reference error counts as a hit, which can inflate both precision and recall. They do make sure one reference error isn't matched twice, but the rule is still lenient. And the self-evaluation comparison is confounded by different source sets — 35 DeepL abstracts against 25 ChatGPT abstracts — so part of the observed drop could be text difficulty rather than self-bias.\n\nThese are fixable problems, not fatal ones. The authors are appropriately modest in their conclusions and acknowledge the proprietary-model issue. The main missing pieces for the 'partial substitute' claim are a human baseline and an exact-match (or stricter overlap) variant. A referee should ask for those, plus a baseline metric like GEMBA on the same data, before accepting the headline numbers as strong.\n\nBottom line: it's a worthwhile first experiment with real data and a clear method. Bring it to peer review, and ask the authors to tighten the evaluation. I'd read it as a pilot, not a definitive result.","headline":"A transparent, small-scale pilot that shows prompt detail boosts GPT-4o's error-label accuracy on DeepL translations, but the 'self-evaluation is worse' finding is confounded and the headline F1 depends on a lenient overlap rule and a single annotator.","tokens_in":16733,"tokens_out":3367,"would_cite":false,"duration_ms":31036,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A detailed prompt lets ChatGPT catch seven in ten translation errors in specialised texts, but the model is markedly worse at finding errors in its own translations.","keywords":["machine translation evaluation","error annotation","large language models","ChatGPT","specialised translation","LSP translation","error typology","self-assessment"],"falsifier":"Take the same sixty translations and have several professional translators annotate them independently; if the new annotators disagree with the original reference as much as ChatGPT does, or if ChatGPT's agreement with a second annotator is no higher than the new annotator's agreement with the first, then the reported $F_1$ is measuring reference noise rather than model capability. A second check would recompute precision and recall requiring exact or boundary-matched error spans instead of one shared character, and observe how much $F_1$ drops.","tokens_in":15792,"feed_emoji":"📝","tokens_out":8233,"duration_ms":70475,"temperature":0.7,"pith_summary":"The paper asks whether a large language model can do fine-grained translation evaluation in a specialised domain: not just score a translation, but point at the specific words that are wrong and say what kind of error each one is. It tests ChatGPT (GPT-4o) with a customised error typology for French translations of NLP research abstracts, comparing the model's annotations with those of one professional translator. On translations produced by DeepL, the detailed prompt finds about seven of every ten reference errors at $F_1=0.707$ and assigns a correct error category to 64.1% of the errors it finds. Shortening the prompt barely changes error detection but drops correct categorisation to about 47%, showing that detailed definitions mainly help labelling, not spotting. When the same prompt is applied to ChatGPT's own translations, $F_1$ falls to 0.496 and more than half of its flagged errors are not real errors, evidence of a self-evaluation penalty.","feed_headline":"ChatGPT finds 7 in 10 translation errors, but misses its own","feed_subtitle":"On DeepL output it labels 64% of detected errors correctly; on its own translations accuracy collapses.","key_machinery":"The central machinery is a deliberately constructed French-language annotation prompt combined with a dedicated error typology. The typology, a fixed menu of over forty error categories with ten subtypes for terminology, defines what counts as an error and supplies the labels the model must choose from. The prompt chains the instructions together, attaches a 50-page annotation manual, asks for sentence-by-sentence output, and instructs the model to list each error span and its category. A matching rule completes the mechanism: a predicted error counts as a hit if it shares at least one character with a reference error, and a predicted label counts as correct if it matches any of the reference labels. This setup lets the authors compute document-level precision, recall, and $F_1$, and the long-versus-short prompt comparison isolates what the per-category definitions contribute.","core_discovery":"The discovery, on the paper's own terms, is that a prompt-only LLM can act as a partial substitute for a human error annotator in specialised translation: with the full prompt, ChatGPT reaches $F_1=0.707$ on DeepL output, with precision 0.792 and recall 0.653, and it labels 64.1% of detected errors with a category from the typology. A shorter prompt without per-category definitions yields nearly the same detection ($F_1=0.702$) but much worse labelling (46.9%), which the paper reads as evidence that spotting errors draws on knowledge the model already has, while categorising them benefits from explicit definitions. The same long prompt applied to ChatGPT's own translations scores $F_1=0.496$, labels only 45.3% of errors correctly, and produces false errors that make up more than half of its annotations, a weakness the paper attributes to self-assessment rather than to the prompt or typology. Together these results support the authors' claim that LLM error annotation is viable for external machine translations in specialised domains, with the caveat that the model cannot be trusted to evaluate its own output.","pith_inferences":["A multi-annotator gold standard would likely move the headline numbers: with one professional as the reference, part of ChatGPT's 'misses' and 'false positives' may be legitimate disagreements, so its true skill could be higher or lower than 0.707 depending on how the annotators differ.","The one-shared-character match rule makes the $F_1$ scores optimistic for real use; requiring exact or boundary-matched spans would lower them, and the gap between the two match rules would quantify how precisely ChatGPT localises errors.","The self-evaluation penalty could be an identity effect: a test that presents ChatGPT's own translations as coming from DeepL would show whether the drop comes from an inability to judge the text or from a bias toward its own output.","Because the model supplies explanations and corrections unprompted, the pedagogically useful unit may be the explanation rather than the error label; a classroom study could test whether students improve from the annotations even when one in three labels is wrong."],"forward_implications":["Error detection and error categorisation are separable abilities: deleting the per-category definitions leaves $F_1$ nearly unchanged (0.702 versus 0.707) but drops correct labelling from 64.1% to 46.9%, so evaluators who need labels must include definitions even though spotters do not need them.","A first-pass annotation workflow is within reach for specialised translation: a human reviewer would check the roughly 35% of gold errors ChatGPT misses and discard its false positives, rather than scan clean text for errors.","ChatGPT's own translations should not be quality-checked by ChatGPT itself: at $F_1=0.496$ with more than half of flagged errors being false, any self-reported error list would mislead a translator or a student.","Because the prompt contains only one sentence about the text domain, the authors expect the same prompt to transfer to specialised texts outside NLP, so the approach could be reused for legal, medical, or technical translation with little modification.","Downstream pedagogical use must tolerate label noise: since only 64.1% of detected errors receive a correct category, roughly one in three labels shown to students would be wrong unless a human filters them."],"supporting_citations":[{"why":"Defines the GEMBA zero-shot prompting baseline for LLM translation evaluation that this work extends to specialised error annotation.","marker":"Kocmi et al. (2023)"},{"why":"Introduced error-analysis prompting with the MQM typology and recommended against detailed error definitions, the exact recommendation the long-prompt experiment challenges.","marker":"Lu et al. (2024)"},{"why":"Shows AutoMQM can identify and classify MQM errors with prompted LLMs, establishing the error-annotation paradigm this paper adapts to LSP translation.","marker":"Fernandes et al. (2023)"},{"why":"Supplies GEMBA-MQM, a reference-free MQM error-annotation approach whose prompting design this paper's custom prompt builds on.","marker":"Kocmi and Federmann (2023)"},{"why":"Provides the MeLLANGE annotation framework that, alongside MQM, grounds the paper's customised error typology for specialised translation.","marker":"Castagnoli et al. (2011)"},{"why":"Documents the MeLLANGE framework's design, the source of the annotation guidelines this paper adapts.","marker":"Kübler (2008)"},{"why":"Motivates the error-annotation evaluation approach by showing large-scale human error annotation of machine translation.","marker":"Freitag et al. (2021)"},{"why":"Provides the bootstrap method used to compute the 95% confidence intervals reported for all precision, recall, and $F_1$ scores.","marker":"Efron and Tibshirani (1993)"},{"why":"Identifies GPT-4 as the model underlying ChatGPT in the experiments.","marker":"OpenAI et al. (2024)"},{"why":"Documents how ChatGPT's behaviour drifts over time, which the paper cites to qualify the reproducibility of its results.","marker":"Chen et al. (2024)"}],"fun_headline_variants":["ChatGPT tags DeepL errors well, stumbles on its own","Prompt detail improves error labeling, not detection","LLM error annotation: good for DeepL, poor for self","ChatGPT's translation error tagging fails self-assessment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation rests on a single professional translator's annotations, counted as correct whenever ChatGPT's error span shares one character with the reference error; if that annotator's judgment is idiosyncratic or the one-character rule is too lenient, the reported precision, recall, and labelling accuracy overstate how well ChatGPT would annotate translations in general.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT tags DeepL errors well, stumbles on its own","Prompt detail improves error labeling, not detection","LLM error annotation: good for DeepL, poor for self","ChatGPT's translation error tagging fails self-assessment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1533,"prompt_tokens":1027,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":440}},"tokens_in":643,"tokens_out":506,"duration_ms":4623,"temperature":1.0,"reasoning_tokens":440,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:33:48.716847+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same sixty translations and have several professional translators annotate them independently; if the new annotators disagree with the original reference as much as ChatGPT does, or if ChatGPT's agreement with a second annotator is no higher than the new annotator's agreement with the first, then the reported $F_1$ is measuring reference noise rather than model capability. A second check would recompute precision and recall requiring exact or boundary-matched error spans instead of one shared character, and observe how much $F_1$ drops.","supporting_citations":[{"cited_title":"u bler, and Alexandra Volanschi. 2011. https://u-paris.hal.science/hal-01135016 Designing a Learner Translator Corpus for Training Purposes . In Natalie K \\","cited_arxiv_id":null,"evidence_quote":"Provides the MeLLANGE annotation framework that, alongside MQM, grounds the paper's customised error typology for specialised translation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the bootstrap method used to compute the 95% confidence intervals reported for all precision, recall, and $F_1$ scores."}],"review_version":1}