Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Testing LLMs' Capabilities in Annotating Translations Based on an Error Typology Designed for LSP Translation: First Experiments with ChatGPT

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A detailed prompt lets ChatGPT catch seven in ten translation errors in specialised texts, but the model is markedly worse at finding errors in its own translations.

desk verdict A transparent, small-scale pilot that shows prompt detail boosts GPT-4o's error-label accuracy on DeepL translations, but the 'self-evaluation is worse' finding is confounded and the headline F1 depends on a lenient overlap rule and a single annotator. read the letter →

arxiv 2504.15052 v1 pith:RWPXN7NH submitted 2025-04-21 cs.CL eess.AS

Joachim Minder , Guillaume Wisniewski , Natalie Kübler This is my paper · ORCID
classification cs.CLeess.AS
keywords machinetranslationevaluationerrorannotationlargelanguagemodelsChatGPTspecialisedLSPtypologyself-assessment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a large language model can do fine-grained translation evaluation in a specialised domain: not just score a translation, but point at the specific words that are wrong and say what kind of error each one is. It tests ChatGPT (GPT-4o) with a customised error typology for French translations of NLP research abstracts, comparing the model's annotations with those of one professional translator. On translations produced by DeepL, the detailed prompt finds about seven of every ten reference errors at $F_1=0.707$ and assigns a correct error category to 64.1% of the errors it finds. Shortening the prompt barely changes error detection but drops correct categorisation to about 47%, showing that detailed definitions mainly help labelling, not spotting. When the same prompt is applied to ChatGPT's own translations, $F_1$ falls to 0.496 and more than half of its flagged errors are not real errors, evidence of a self-evaluation penalty.

What carries the argument

The central machinery is a deliberately constructed French-language annotation prompt combined with a dedicated error typology. The typology, a fixed menu of over forty error categories with ten subtypes for terminology, defines what counts as an error and supplies the labels the model must choose from. The prompt chains the instructions together, attaches a 50-page annotation manual, asks for sentence-by-sentence output, and instructs the model to list each error span and its category. A matching rule completes the mechanism: a predicted error counts as a hit if it shares at least one character with a reference error, and a predicted label counts as correct if it matches any of the reference labels. This setup lets the authors compute document-level precision, recall, and $F_1$, and the long-versus-short prompt comparison isolates what the per-category definitions contribute.

What would settle it

Take the same sixty translations and have several professional translators annotate them independently; if the new annotators disagree with the original reference as much as ChatGPT does, or if ChatGPT's agreement with a second annotator is no higher than the new annotator's agreement with the first, then the reported $F_1$ is measuring reference noise rather than model capability. A second check would recompute precision and recall requiring exact or boundary-matched error spans instead of one shared character, and observe how much $F_1$ drops.

Watch

Extended reading notes

Core claim

The discovery, on the paper's own terms, is that a prompt-only LLM can act as a partial substitute for a human error annotator in specialised translation: with the full prompt, ChatGPT reaches $F_1=0.707$ on DeepL output, with precision 0.792 and recall 0.653, and it labels 64.1% of detected errors with a category from the typology. A shorter prompt without per-category definitions yields nearly the same detection ($F_1=0.702$) but much worse labelling (46.9%), which the paper reads as evidence that spotting errors draws on knowledge the model already has, while categorising them benefits from explicit definitions. The same long prompt applied to ChatGPT's own translations scores $F_1=0.496$, labels only 45.3% of errors correctly, and produces false errors that make up more than half of its annotations, a weakness the paper attributes to self-assessment rather than to the prompt or typology. Together these results support the authors' claim that LLM error annotation is viable for external machine translations in specialised domains, with the caveat that the model cannot be trusted to evaluate its own output.

Load-bearing premise

The whole evaluation rests on a single professional translator's annotations, counted as correct whenever ChatGPT's error span shares one character with the reference error; if that annotator's judgment is idiosyncratic or the one-character rule is too lenient, the reported precision, recall, and labelling accuracy overstate how well ChatGPT would annotate translations in general.

Editorial extensions

If this is right

  • Error detection and error categorisation are separable abilities: deleting the per-category definitions leaves $F_1$ nearly unchanged (0.702 versus 0.707) but drops correct labelling from 64.1% to 46.9%, so evaluators who need labels must include definitions even though spotters do not need them.
  • A first-pass annotation workflow is within reach for specialised translation: a human reviewer would check the roughly 35% of gold errors ChatGPT misses and discard its false positives, rather than scan clean text for errors.
  • ChatGPT's own translations should not be quality-checked by ChatGPT itself: at $F_1=0.496$ with more than half of flagged errors being false, any self-reported error list would mislead a translator or a student.
  • Because the prompt contains only one sentence about the text domain, the authors expect the same prompt to transfer to specialised texts outside NLP, so the approach could be reused for legal, medical, or technical translation with little modification.
  • Downstream pedagogical use must tolerate label noise: since only 64.1% of detected errors receive a correct category, roughly one in three labels shown to students would be wrong unless a human filters them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A multi-annotator gold standard would likely move the headline numbers: with one professional as the reference, part of ChatGPT's 'misses' and 'false positives' may be legitimate disagreements, so its true skill could be higher or lower than 0.707 depending on how the annotators differ.
  • The one-shared-character match rule makes the $F_1$ scores optimistic for real use; requiring exact or boundary-matched spans would lower them, and the gap between the two match rules would quantify how precisely ChatGPT localises errors.
  • The self-evaluation penalty could be an identity effect: a test that presents ChatGPT's own translations as coming from DeepL would show whether the drop comes from an inability to judge the text or from a bias toward its own output.
  • Because the model supplies explanations and corrections unprompted, the pedagogically useful unit may be the explanation rather than the error label; a classroom study could test whether students improve from the annotations even when one in three labels is wrong.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a pilot study of using ChatGPT (GPT-4o) to annotate machine translations according to a specialized error typology. The authors built a corpus of French translations of NLP research abstracts: 35 texts translated by DeepL and 25 different texts translated by ChatGPT, each annotated by a single professional translator. They prompted ChatGPT with a long, detailed French prompt and a shorter variant, matched predicted error spans to the human reference by requiring at least one shared character, and computed document-level precision, recall, F1, and label-accuracy rates with bootstrap confidence intervals. The main results are that for DeepL translations the long prompt reaches F1 = 0.707 (precision 0.792, recall 0.653) with 64.1% correct error-label assignment; the short prompt has similar F1 but markedly lower label accuracy (46.9%); and for ChatGPT's own translations the long prompt yields F1 = 0.496 with 45.3% label accuracy. The authors interpret these results as evidence that LLM-based annotation is promising for specialized translation evaluation, that prompt detail mainly helps categorization, and that ChatGPT shows a clear self-evaluation penalty.

Significance. If the reported numbers survive robustness checks, this would be a useful contribution to the emerging literature on LLM-based translation evaluation, extending it to LSP translation and to fine-grained error annotation rather than quality scoring. The study has several strengths: the evaluation is anchored to an external human gold standard rather than to a fitted model, the authors report bootstrap confidence intervals for the main detection scores, they compare prompt variants, and they release the annotated corpus with a DOI, which is a concrete asset for future work. The paper is also transparent about the proprietary and non-reproducible nature of ChatGPT. However, the central quantitative claims currently rest on a lenient span-matching rule, a single-annotator gold standard, and a self-evaluation comparison that is confounded by different source texts. These issues are fixable within the scope of the manuscript, but they need to be addressed before the practical claim of a 'partial substitute' for human annotation can be considered established.

major comments (4)
  1. [§4.3, Eqs. (1)–(2) and footnote 13] The one-character-overlap matching rule can materially inflate both precision and recall. A predicted span that overlaps several gold errors is counted as correct for each of them, and the evaluation only excludes the reverse case (a reference error matched by two predictions). This is especially consequential because reference errors range up to 81–103 characters and carry up to six labels. The paper does not report an exact-match or boundary-aware variant, predicted span lengths, or the number of multi-overlap cases. The 64.1% label-accuracy figure is also hard to interpret without knowing how often a predicted label could match by chance given the multiple reference labels. Please report at least one stricter span criterion (e.g., exact boundary match or token-level F1), the distribution of predicted span lengths and overlap counts, and label accuracy under single-label reference coding.
  2. [§4.2 and §6] The reference annotations come from a single professional translator, and no inter-annotator agreement is reported. Since the typology is partly subjective, the headline F1 = 0.707 can only be interpreted relative to the human-human agreement level for this task: if two experts agree at F1 ≈ 0.85, then ChatGPT has a clear gap; if human agreement is around 0.70, ChatGPT is already within the range of human variability. Without this baseline, the statement that ChatGPT is a viable partial substitute for human annotation is not quantitatively interpretable. Please add an inter-annotator study on a subset of the corpus, or at minimum cite and use existing agreement data for this typology.
  3. [§4.4 and Table 1] The self-evaluation comparison is confounded by the use of different source texts in the two subcorpora: 35 source texts for DeepL and 25 different source texts for ChatGPT, with different error densities (399 vs. 193 gold errors). The reported drop from F1 = 0.707 to F1 = 0.496 could therefore reflect text difficulty or annotation differences rather than a self-evaluation bias. To support the paper's conclusion about self-assessment, please evaluate both translation systems on the same source texts, or at minimum have ChatGPT annotate DeepL translations of the same 25 source texts it translated itself, and report the controlled comparison.
  4. [§5 and Table 1] The claim that adding error definitions to the prompt improves categorization is based on the point estimates 64.1% vs. 46.9% label accuracy, but no confidence intervals or statistical tests are reported for label accuracy, and the F1 confidence intervals for the long and short prompts overlap substantially (0.707 ± 0.039 vs. 0.702 ± 0.053). Please provide per-document paired differences and bootstrap confidence intervals for label accuracy, and test whether the categorization difference is statistically reliable rather than attributable to prompt-instruction variability.
minor comments (5)
  1. [§4.4] The model name is misspelled as "ChatPT" in the first experiment description; it should be "ChatGPT".
  2. [Figure 2] The x-axis labels are run together in the caption and provide no legend keys for the three conditions; a box plot or per-document strip plot with clear labels would make the variability claims easier to verify.
  3. [§4.1] The statement that the results "seem likely" to apply to other text types is not supported by the experimental design, since only NLP abstracts are tested; please soften or remove this generalization.
  4. [Appendix A] The error typology is presented as a compact figure that is hard to read; please include a machine-readable table with all codes and definitions, especially the ten terminology subtypes, so that readers can check the labels used in the evaluation.
  5. [Limitations] For reproducibility, please report the exact model version, access date, and decoding settings (temperature, top-p, number of runs) used for every ChatGPT call, since the paper itself notes that ChatGPT behavior fluctuates over time.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported F1 scores are empirical measurements against an external human gold standard, not quantities equivalent to the paper's inputs by construction.

full rationale

The paper's central claims are empirical: ChatGPT is prompted to annotate machine translations, and its output is compared with the annotations of a professional translator who used the same error typology. There is no derivation chain in which a predicted quantity is defined in terms of the same quantity, and no parameter is fitted and then renamed as a prediction. The typology, prompt, and annotation manual are inputs that define the task rather than outputs derived from it; the fact that the typology is adapted from MQM and MeLLANGE (including the authors' own MeLLANGE work) is provenance, not a load-bearing self-citation. The use of any-character overlap and a single-annotator gold standard affects the validity and interpretability of the scores, but it does not make the scores circular: ChatGPT's annotations are not constructed from the reference annotations by the matching rule, they are independent model outputs. The trial-and-error choice of the prompt is a possible selection-bias concern, but the paper does not claim to derive the F1 result from that tuning set, and the reported F1 is not equal to any fitted quantity by construction. The self-evaluation comparison is likewise an external behavioral measurement of model bias, not a circular evaluation logic.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The main external inputs are the error typology, the expert gold annotation, and the behavior of a proprietary API. None are quantified with reliability measures, and the self-evaluation comparison uses different texts, so the central numerical claims carry medium risk.

free parameters (1)
  • error span matching threshold = at least one shared character
    The paper defines a reference error as correctly identified if it shares at least one character with a predicted error, as stated in Section 4.3. This hand-chosen rule can inflate agreement and affects all reported scores.
assumptions (3)
  • domain assumption The custom error typology, adapted from MQM and MeLLANGE, is adequate and complete for identifying translation errors in specialised texts.
    The entire annotation and evaluation rests on this typology, described in Section 3 and Figure 4. If the typology misses or merges error types relevant to LSP translation, the reported categorisation accuracy is not meaningful.
  • domain assumption The single professional translator's annotations are reliable ground truth.
    Only one expert annotated the corpus, as described in Section 4.2. No inter-annotator agreement is reported, so the gold standard may be idiosyncratic.
  • domain assumption GPT-4o with the tested prompts behaves consistently across the evaluation period.
    ChatGPT is a proprietary, versioned service; the paper notes performance fluctuations over time in the Limitations section, making exact replication difficult.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Testing LLMs' Capabilities in Annotating Translations Based on an Error Typology Designed for LSP Translation: First Experiments with ChatGPT." pith.science (2026). https://pith.science/paper/RWPXN7NH

@misc{pith2026250415052,
  author       = {Pith},
  title        = {Pith review of: Testing LLMs' Capabilities in Annotating Translations Based on an Error Typology Designed for LSP Translation: First Experiments with ChatGPT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RWPXN7NH}},
  note         = {Machine review of arXiv:2504.15052}
}
read the original abstract

This study investigates the capabilities of large language models (LLMs), specifically ChatGPT, in annotating MT outputs based on an error typology. In contrast to previous work focusing mainly on general language, we explore ChatGPT's ability to identify and categorise errors in specialised translations. By testing two different prompts and based on a customised error typology, we compare ChatGPT annotations with human expert evaluations of translations produced by DeepL and ChatGPT itself. The results show that, for translations generated by DeepL, recall and precision are quite high. However, the degree of accuracy in error categorisation depends on the prompt's specific features and its level of detail, ChatGPT performing very well with a detailed prompt. When evaluating its own translations, ChatGPT achieves significantly poorer results, revealing limitations with self-assessment. These results highlight both the potential and the limitations of LLMs for translation evaluation, particularly in specialised domains. Our experiments pave the way for future research on open-source LLMs, which could produce annotations of comparable or even higher quality. In the future, we also aim to test the practical effectiveness of this automated evaluation in the context of translation training, particularly by optimising the process of human evaluation by teachers and by exploring the impact of annotations by LLMs on students' post-editing and translation learning.

Figures

Figures reproduced from arXiv: 2504.15052 by the authors.

Figure 1
Figure 1. Example of a human reference annotation: each error is identified by its span (text written on an orange [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Distribution of precisions, recalls and F [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Example of annotation by ChatGPT. This figure shows the sentence-by-sentence annotation performed [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The error typology used in our experiments. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Prompt used on GPT-4o 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 11 canonical work pages

  1. [1]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Z...

    arXiv 2023

  2. [2]

    Sílvia Araújo and Micaela Aguiar. 2023. Comparing chatgpt's and human evaluation of scientific texts' translations from english to portuguese using popular automated translators notebook for the simpletext lab at clef 2023

    work page 2023

  3. [3]

    Lo \"i c Barrault, Magdalena Biesialska, Ond r ej Bojar, Marta R. Costa-juss \`a , Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljube s i \'c , Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 20...

    work page 2020

  4. [4]

    Siyi Cao and Linping Zhong. 2023. https://arxiv.org/abs/2309.01645 Exploring the effectiveness of chatgpt-based feedback compared with teacher feedback and self-feedback: Evidence from chinese to english translation . Preprint, arXiv:2309.01645

    work page Pith review arXiv 2023

  5. [5]

    u bler, and Alexandra Volanschi. 2011. https://u-paris.hal.science/hal-01135016 Designing a Learner Translator Corpus for Training Purposes . In Natalie K \

    Sara Castagnoli, Dragos Ciobanu, Kerstin Kunz, Natalie K \"u bler, and Alexandra Volanschi. 2011. https://u-paris.hal.science/hal-01135016 Designing a Learner Translator Corpus for Training Purposes . In Natalie K \"u bler, editor, Corpora, Language, Teaching, and Resources : From Theory to Practice. Bern: Peter Lang , volume Etudes Contrastives of Corpor...

    work page 2011

  6. [6]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2024. How Is ChatGPT s Behavior Changing Over Time ? Harvard Data Science Review, 6(2). Https://hdsr.mitpress.mit.edu/pub/y95zitmz

    work page 2024

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sashank Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bra...

    2024

  8. [8]

    Bradley Efron and Robert Tibshirani. 1993. An Introduction to the Bootstrap. Chapman and Hall/CRC, London

    work page 1993

Show all 31 references
  1. [9]

    Sabit Ekin. 2023. https://doi.org/10.36227/techrxiv.22683919.v2 Prompt engineering for chatgpt: A quick guide to techniques, tips, and best practices

    2023 doi

  2. [10]

    Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr \'e Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. https://doi.org/10.18653/v1/2023.wmt-1.100 The devil is in the errors: Leveraging large language models for f...

    2023 doi

  3. [11]

    Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. https://doi.org/10.1162/tacl_a_00437 Experts, errors, and context: A large-scale study of human evaluation for machine translation . Transactions of the Association for Comp...

    2021 doi

  4. [12]

    Sui He. 2024. https://aclanthology.org/2024.eamt-1.27/ Prompting C hat GPT for translation: A comparative analysis of translation brief and persona prompts . In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 316-...

    2024

  5. [13]

    Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. https://arxiv.org/abs/2301.08745 Is chatgpt a good translator? yes with gpt-4 as the engine . Preprint, arXiv:2301.08745

    2023 arXiv

  6. [14]

    Margaret King, Andrei Popescu-Belis, and Eduard Hovy. 2003. https://aclanthology.org/2003.mtsummit-papers.30/ FEMTI : creating and using a framework for MT evaluation . In Proceedings of Machine Translation Summit IX: Papers, New Orleans, USA

    2003

  7. [15]

    Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, To...

    2023 doi

  8. [16]

    Tom Kocmi, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Nov \'a k, Ma...

    2022

  9. [17]

    Tom Kocmi and Christian Federmann. 2023. https://aclanthology.org/2023.eamt-1.19/ Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193--203, Ta...

    2023

  10. [18]

    Natalie K \"u bler. 2008. https://u-paris.hal.science/hal-01217602 MeLLANGE Final Report . Intern report, Universit \'e Paris Diderot

    2008

  11. [19]

    Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. 2025. https://aclanthology.org/2025.coling-main.374/ MQM - APE : Toward high-quality error annotation predictors with automatic post-editing in LLM translation evaluators . In Proceedings of the 31st Internat...

    2025

  12. [20]

    Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2023. https://arxiv.org/pdf/2303.13809.pdf Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt . arXiv preprint

    2023 arXiv

  13. [21]

    Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2024. https://doi.org/10.18653/v1/2024.findings-acl.520 Error analysis prompting enables human-like translation evaluation in large language models . In Findings of the Association for Computational L...

    2024 doi

  14. [22]

    Wong, and Longyue Wang

    Chenyang Lyu, Zefeng Du, Jitao Xu, Yitao Duan, Minghao Wu, Teresa Lynn, Alham Fikri Aji, Derek F. Wong, and Longyue Wang. 2024. https://aclanthology.org/2024.lrec-main.120/ A paradigm shift: The future of machine translation lies with large language models . In Proceedings of ...

    2024

  15. [23]

    Qingsong Ma, Johnny Wei, Ond r ej Bojar, and Yvette Graham. 2019. https://doi.org/10.18653/v1/W19-5302 Results of the WMT 19 metrics shared task: Segment-level and strong MT systems pose big challenges . In Proceedings of the Fourth Conference on Machine Translation (Volume 2:...

    2019 doi

  16. [24]

    Benjamin Marie, Atsushi Fujita, and Raphael Rubino. 2021. https://doi.org/10.18653/v1/2021.acl-long.566 Scientific credibility of machine translation research: A meta-evaluation of 769 papers . In Proceedings of the 59th Annual Meeting of the Association for Computational Ling...

    2021 doi

  17. [25]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  18. [26]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  19. [27]

    Khapra, and Raj Dabre

    Ananya Sai B, Tanay Dixit, Vignesh Nagarajan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra, and Raj Dabre. 2023. https://doi.org/10.18653/v1/2023.acl-long.795 I ndic MT eval: A dataset to meta-evaluate machine translation metrics for I ndian languages . In Proceedings ...

    2023 doi

  20. [28]

    Sai Cheong Siu. 2023. Chatgpt and gpt-4 for professional translators: Exploring the potential of large language models in translation. SSRN Electronic Journal

    2023

  21. [29]

    Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.1036 Document-level machine translation with large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natu...

    2023 doi

  22. [30]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  23. [31]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.