Pith. sign in

REVIEW 3 major objections 5 minor 41 references

How Important is `Perfect' English for Machine Translation Prompts?

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For LLM machine translation, the choice of base prompt affects output quality more than realistic spelling, phrasing, or register errors in the user prompt, and spelling errors hurt more than phrase-level errors.

desk verdict Solid empirical study of prompt-error effects, but the headline variance claim needs a formal decomposition before it's fully supported. read the letter →

arxiv 2507.09509 v2 pith:UP4WBG47 submitted 2025-07-13 cs.CL

Patrícia Schmidtová , Niyati Bafna , Seth Aycock , Gianluca Vico , Wiktor Kamzela , Katharina Hämmerl , Vilém Zouhar This is my paper
classification cs.CL
keywords LLMmachinetranslationpromptrobustnessusererrorsL2EnglishchoiceevaluationsyntheticnoiseGEMBA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether imperfect English in user prompts actually hurts large-language-model translation and translation evaluation, and how much that matters compared with choosing a good prompt in the first place. Using seven synthetic error profiles (realistic spelling errors, phonetic misspellings, phrasal simplifications, register shifts, and combinations) applied at controlled intensities to four base prompts, the authors test six LLMs across three language pairs. They find that character-level errors degrade translation most, while phrase-level disfluencies have little effect or even help. Crucially, the variance in output quality caused by these realistic errors is smaller than the variance across different base prompts, so a well-chosen prompt matters more than perfect English in the prompt. Errors mostly hurt instruction-following—adding explanations, repeating source text, outputting off-target languages—rather than the translation itself.

What carries the argument

The central machinery is a family of controlled error augmenters that synthesize user-like noise at varying intensities, motivated by empirical studies of L1/L2 spelling errors (e.g., Cook 1997), LLM-generated phonetic, phrasal, and register variants, and combinations capturing 'L2' and 'lazy user' profiles, plus a uniform character-level control. To compare error types with different scales, the paper measures each augmented prompt's deviation from the base prompt by surface similarity (chrF) and semantic similarity (inner product of Sentence-BERT embeddings), then correlates translation and quality-estimation scores with these similarities across binned error intensities. This per-unit-error normalization is what lets the paper say spelling errors hurt more than phrasal errors at equivalent perturbation, and what supports the headline variance comparison.

What would settle it

Collect a corpus of real user or L2 English prompts with natural error rates, measure the actual orthographic error intensity per character, and rerun this study's augmentation and comparison; the central claim fails if the variance attributable to real errors then exceeds the variance attributable to prompt choice. Alternatively, hold the prompt fixed while injecting errors at measured real-user intensity and show that error-induced variance exceeds prompt-choice variance to refute the claim.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that for LLM-based machine translation, the variance in quality caused by realistic user errors is lower than the variance over the initial prompt choice, so perfect English for a given prompt matters less than selecting a good prompt. At equivalent perturbation intensity, spelling and character-level errors cause the largest degradation, while phrasal and register simplifications produce little or no drop and can even improve performance for some prompts. The paper further claims that error-augmented prompts degrade output mainly through worse instruction following—redundant text, off-target language, copying the prompt—not through lower core translation quality, and that LLMs can still translate prompts that are illegible to humans. A parallel experiment on GEMBA-style quality estimation shows a similar but weak transferable effect of orthographic errors on system-level correlation with human judgments.

Load-bearing premise

The load-bearing premise is that the synthetically generated errors—especially the hand-chosen intensity range p in [0, 0.4] for spelling errors and the LLM-generated phrasal and register edits—faithfully represent the type and severity of real users' errors, since the paper itself notes 'We used automatically generated errors rather than using error data from real learners'; if real spelling errors are more frequent or severe than modeled, the prompt-choice dominance could weaken.

Editorial extensions

If this is right

  • Users and practitioners get more from choosing a strong base prompt (or a short minimal one) than from polishing a single prompt to perfect English.
  • Spelling and typo robustness is the concrete bottleneck among realistic user errors; systems or preprocessors that correct character-level noise should yield the largest recovery.
  • Automatic MT evaluation under clean benchmarks may overstate real-world performance for L2 and casual users, mostly because of off-target and verbose outputs, not worse translations.
  • Quality estimation with LLM judges is also mildly sensitive to orthographic prompt errors, so evaluation setups should control prompt quality as well.
  • Heavily corrupted prompts still often yield valid translations, suggesting task recognition depends on preserved cues (source/target labels and source text) more than on full prompt legibility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If prompt choice dominates realistic errors, then automatic prompt search over a model-specific set of candidates is likely a more cost-effective robustness lever than spelling-correction preprocessing—a testable comparison the paper does not run.
  • The resilience of the shortest prompts hints at a length/complexity effect: shorter prompts have fewer characters to corrupt; a systematic ablation varying prompt length while fixing content could separate this from content effects.
  • Because the augmenters are synthetic and capped at p=0.4 for orthographic errors, the prompt-choice dominance may be an upper bound; collecting real learner-typing data and re-measuring could shift the balance.
  • The finding that models decode near-gibberish prompts by relying on language labels and source text suggests a cheap robustness test: strip all English instruction words and keep only placeholders, then see how much translation quality survives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper systematically perturbs English machine-translation prompts with synthetic user-like errors (orthographic, phonetic, phrasal, register, combined L2/lazy profiles, and a uniform control) and measures the effect on six LLMs across three WMT24 language pairs for both translation and GEMBA-based quality estimation. It reports that spelling-type errors degrade translation most, phrase-level simplifications have little or even positive effect, and that the dominant failure mode is reduced instruction-following rather than core translation quality. The authors release the translation dataset and code. The abstract's headline claim is that the variance in quality caused by these errors is lower than the variance over the initial prompt choice, implying that choosing a good prompt matters more than error-free English for a given prompt.

Significance. If the central variance claim is supported, the paper has a clear practical and fairness implication: for LLM-based MT, prompt selection may dominate realistic prompt-error effects, and robustness work should target both phenomena. The study is broad and externally anchored: six models, three language pairs, two automatic metrics (ChrF and COMET), human-judgment-based meta-evaluation for the QE task, plus qualitative analysis backed by output-length statistics and concrete examples. The released data and code are valuable for follow-up work. However, the headline variance comparison is not directly tested; it is inferred from correlations and a small hand-picked prompt set. The significance of the paper therefore depends on a formal variance decomposition that is currently missing.

major comments (3)
  1. [Abstract; §5.1] The central claim that 'the variance in quality caused by these errors is lower than the variance over the initial prompt choice' is never directly estimated. Figure 1 and Table 2 report correlations of quality with prompt similarity, but a correlation is not a variance decomposition, and no confidence intervals are given. With only four hand-picked WMT prompt templates plus one minimal prompt, the 'prompt choice' variance is a statistic over a non-random, small set; a wider prompt sample could change the comparison. I recommend adding an ANOVA or mixed-effects model that partitions total variance into prompt-choice, error-type, model, language, and residual components, with bootstrap intervals, and then framing the claim as holding for the tested prompt set and error ranges.
  2. [§3.1; Limitations] The error intensity is synthetic and bounded by hand-chosen ranges: orthographic error probability is capped at p=0.4, and phrasal and register errors have only two LLM-generated intensity levels. The Limitations explicitly state: 'We used automatically generated errors rather than using error data from real learners.' If real L2 user spelling errors are more frequent or more severe than modeled, the error-induced variance could exceed the prompt-choice variance and invert the headline. I ask for a robustness analysis with higher error intensities or a calibration to natural learner corpora, or at minimum a rephrasing of the abstract so that the dominance claim is explicitly conditional on the modeled error distribution.
  3. [Table 2; §5.2] The aggregated 'all prompts' correlations conceal strong prompt-by-error interactions that bear directly on the headline. For example, Prompt 3 shows r=-0.01 for orthographic errors and Prompt 4 shows r=-0.67 for phrasal errors, while other prompt/error combinations show positive or strongly negative correlations. Pooling across these heterogeneous prompts makes the 'prompt choice dominates' conclusion depend on the particular composition of the prompt set. I recommend reporting per-prompt variance components or interaction terms, and explicitly discussing whether any single prompt or error type reverses the aggregated ranking.
minor comments (5)
  1. [§5.4; Table 3] The segment-level negative correlation for QE Prompt 1 (-0.38) is likely an artifact of the strict no-retry setting that replaces malformed GEMBA outputs with 0; the paper acknowledges this, but the main text should state more prominently that the transferability claim rests on system-level correlations only.
  2. [Figure 1; Figures 5-9] The scatter plots and line plots would benefit from confidence bands or error bars; with roughly 500 segments per language pair and six models, sampling variability is large enough to affect the reported correlations.
  3. [§6] The conclusion states '4 language pairs' but the experiments use three language pairs (Czech-Ukrainian, German-English, English-Chinese); please correct this inconsistency.
  4. [Table 3] The row header 'Level' is ambiguous; it should be split into 'System-level' and 'Segment-level' rows.
  5. [Figure 5] The label 'Lexicophrasal' is a typo; it should be 'Lexical/Phrasal' (as used in Table 1).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are direct empirical observations evaluated against external benchmarks.

full rationale

This paper is an empirical study of how synthetic user-prompt errors affect LLM-based machine translation and quality estimation. Its central claims—that spelling errors hurt more than phrasal errors, that prompt choice matters more than most realistic errors, and that errors mostly degrade instruction-following rather than core translation ability—are supported by direct measurements against external benchmarks (COMET-22, ChrF, WMT24 human judgments for QE). No parameter is fitted to the target claim and then renamed a prediction: the 'variance in quality caused by errors vs. variance over prompt choice' comparison is a descriptive observation over the experimental grid, not the output of a fitted model. The error augmenters are synthetic and hand-tuned, which the Limitations explicitly acknowledge ('We used automatically generated errors rather than using error data from real learners'), but this is a threat to external validity, not circularity. Self-citations (e.g., sacrecomet and WMT24 findings) appear only as tooling or dataset references and are not load-bearing justifications of the conclusions. No step in the derivation chain reduces by construction to its own inputs, and no uniqueness or ansatz is imported from the authors' prior work. The paper is self-contained against external evaluation data, so no significant circularity is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims are empirical findings from controlled perturbation experiments, so the ledger contains no fitted parameters or invented entities. The only hand-chosen numeric inputs are the error intensity bounds (max p for orthographic and uniform errors), which shape the definition of 'realistic' error. Key domain assumptions are that COMET and ChrF measure translation quality, embedding similarity approximates semantic prompt distance, WMT24 human judgments serve as ground truth for QE meta-evaluation, LLM-generated errors resemble real user errors, and the five selected prompts represent the space of prompt choices.

free parameters (2)
  • p_max_orthographic = 0.4
    Upper bound of the spelling-error probability range chosen by hand (Section 3.1); the claim that errors are 'realistic' depends on this range.
  • p_max_uniform = 1.0
    Upper bound for random character noise (Appendix A.1); used as a stress test and contrast with realistic errors.
assumptions (5)
  • domain assumption COMET-22 and ChrF are valid automatic metrics of translation quality.
    Used throughout Sections 4 and 5 to measure LLM translation output; the paper notes COMET's limitations for off-target output.
  • domain assumption Inner product of all-MiniLM-L6-v2 embeddings measures semantic prompt similarity.
    Used in Section 3.2 to compare original and error-augmented prompts; this is an approximation of semantic distance.
  • domain assumption WMT24 human judgments are appropriate ground truth for meta-evaluating quality estimation.
    Used in Section 5.4 to compute system and segment level correlations for GEMBA.
  • domain assumption LLM-generated errors approximate real user errors.
    The paper acknowledges this in Limitations; all conclusions about error types depend on this realism.
  • domain assumption The four WMT24 prompts plus the minimal prompt represent the space of user prompt choices.
    Section 4; the prompt-choice-dominates claim relies on these prompts being representative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Important is `Perfect' English for Machine Translation Prompts?." pith.science (2026). https://pith.science/paper/UP4WBG47

@misc{pith2026250709509,
  author       = {Pith},
  title        = {Pith review of: How Important is `Perfect' English for Machine Translation Prompts?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UP4WBG47}},
  note         = {Machine review of arXiv:2507.09509}
}
read the original abstract

Large language models (LLMs) have achieved top results in recent machine translation evaluations, but they are also known to be sensitive to errors and perturbations in their prompts. We systematically evaluate how both humanly plausible and synthetic errors in user prompts affect LLMs' performance on two related tasks: Machine translation and machine translation evaluation. We provide both a quantitative analysis and qualitative insights into how the models respond to increasing noise in the user prompt. The prompt quality strongly affects the translation performance: With many errors, even a good prompt can underperform a minimal or poor prompt without errors. However, different noise types impact translation quality differently, with character-level and combined noisers degrading performance more than phrasal perturbations. Qualitative analysis reveals that lower prompt quality largely leads to poorer instruction following, rather than directly affecting translation quality itself. Further, LLMs can still translate in scenarios with overwhelming random noise that would make the prompt illegible to humans.

Figures

Figures reproduced from arXiv: 2507.09509 by the authors.

Figure 1
Figure 1. Changing model performance, as measured by COMET score (y-axis), across all error-augmented (orthographic errors) prompts and all models. The simi￾larity of each error-augmented prompt to the original is measured by the inner product of their sentence embed￾dings (x-axis). sensitive to errors in their prompts (Qiang et al., 2024, inter alia). Research publications and model evaluation se￾tups tend to use well-crafte… view at source ↗
Figure 2
Figure 2. Top: Machine translation pipeline. The origi￾nal prompt is augmented with an error, then filled with source language, target language, and source sentence, before being translated by an LLM. Bottom: Quality estimation pipeline. Translations are evaluated using the GPT Estimation Metric Based Assessment (GEMBA). instruction-following abilities (Zheng et al., 2023), and have since been successfully applied to transla￾… view at source ↗
Figure 3
Figure 3. Percentage of outputs in the target language, by language pair and model. Note that TowerInstruct does [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Changing model performance, in terms of system-level correlation (y-axis), across quality esti￾mation prompts augmented with orthographic errors, against semantic similarity of the error-augmented prompt to its original (x-axis). The results suggest only a weak trend …
Figure 5
Figure 5. Figure 5: Average performance (across models and languages) with respect to individual prompts and error types. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Sensitivity to error augmentation by language pair. Translation quality measured by ChrF (top) or COMET [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Sensitivity of individual models to prompt noising, for each language pair and by model. x-axis: Prompt [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Average length of generated outputs compared to the reference length. We compare the output lengths [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Sensitivity of QE outputs to perturbations by language pair. System-level correlation is measured against [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 14 canonical work pages

  1. [1]

    Duarte Alves, Nuno Guerreiro, Jo\ a o Alves, Jos \'e Pombal, Ricardo Rei, Jos \'e de Souza, Pierre Colombo, and Andre Martins. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.744 Steering large language models for machine translation with finetuning and in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2023, 1...

    doi:10.18653/v1/2023.findings-emnlp.744 2023

  2. [2]

    Niyati Bafna, Kenton Murray, and David Yarowsky. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1044 Evaluating large language models along dimensions of language variation: A systematik invesdigatiom uv cross-lingual generalization . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 18742--18762. Association fo...

    doi:10.18653/v1/2024.emnlp-main.1044 2024

  3. [3]

    Rachel Bawden and Fran c ois Yvon. 2023. https://aclanthology.org/2023.eamt-1.16/ Investigating the translation performance of a large multilingual language model: the case of BLOOM . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 157--170. European Association for Machine Translation

    work page 2023

  4. [4]

    Yonatan Belinkov and Yonatan Bisk. 2018. https://openreview.net/forum?id=BJ8vJebC- Synthetic and natural noise both break neural machine translation . In International Conference on Learning Representations

    work page 2018

  5. [5]

    Eleftheria Briakou, Zhongtao Liu, Colin Cherry, and Markus Freitag. 2024. http://arxiv.org/abs/2410.00863 On the implications of verbose LLM outputs: A case study in translation evaluation

    arXiv 2024

  6. [6]

    Vivian J Cook. 1997. https://www.tandfonline.com/doi/abs/10.1080/01434639708666335 L2 users and E nglish spelling . Journal of Multilingual and Multicultural Development, 18(6):474--488

    work page doi:10.1080/01434639708666335 1997

  7. [7]

    Arid Hasan, Imran Razzak, and Usman Naseem

    Krishno Dey, Prerona Tarannum, Md. Arid Hasan, Imran Razzak, and Usman Naseem. 2024. http://arxiv.org/abs/2410.13153 Better to ask in english: Evaluation of large language models on english, low-resource and cross-lingual settings

    arXiv 2024

  8. [8]

    Abhimanyu Dubey et al. 2024. https://doi.org/10.48550/arXiv.2407.21783 The Llama 3 herd of models

    arXiv doi:10.48550/arxiv.2407.21783 2024

Show all 41 references
  1. [9]

    Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andr \'e Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. https://doi.org/10.18653/v1/2023.wmt-1.100 The devil is in the errors: Leveraging large language models for f...

    2023 doi

  2. [10]

    Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. https://doi.org/10.18653/v1/2024.wmt-1.2 Ar...

    2024 doi

  3. [11]

    Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. https://doi.org/10.18653/v1/2023.wmt-1.51 Results of ...

    2023 doi

  4. [12]

    Smith, and Luke Zettlemoyer

    Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2024. http://arxiv.org/abs/2212.04037 Demystifying prompts in language models via perplexity estimation

    2024 arXiv

  5. [13]

    Google. 2024. https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/ Introducing gemini 2.0: Our new ai model for the agentic era

    2024

  6. [14]

    Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e F

    Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr \'e F. T. Martins. 2024. https://doi.org/10.1162/tacl\_a\_00683 xcomet: Transparent machine translation evaluation through fine-grained error detection . Transactions of the Association for ...

    2024 doi

  7. [15]

    Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. https://doi.org/10.48550/arXiv.2302.09210 How good are GPT models at machine translation? A comprehensive evaluation

    2023 doi arXiv

  8. [16]

    Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.211 Lost in the source language: How large language models evaluate the quality of machine translation . In Findings of the Association for Comput...

    2024 doi

  9. [17]

    Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, ...

    2024 doi

  10. [18]

    Tom Kocmi and Christian Federmann. 2023 a . https://doi.org/10.18653/v1/2023.wmt-1.64 GEMBA - MQM : Detecting translation quality error spans with GPT -4 . In Proceedings of the Eighth Conference on Machine Translation, 768--775. Association for Computational Linguistics

    2023 doi

  11. [19]

    Tom Kocmi and Christian Federmann. 2023 b . https://aclanthology.org/2023.eamt-1.19 Large language models are state-of-the-art evaluators of translation quality . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 193--203, Tamper...

    2023

  12. [20]

    Tom Kocmi, Vil \'e m Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popovi \'c , Mrinmaya Sachan, and Mariya Shmatova. 2024 b . https://doi.org/10.18653/v1/2024.wmt-1.131 Error span annotation: A balanced approach for human evaluation of machine tra...

    2024 doi

  13. [21]

    Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2024. https://doi.org/10.18653/v1/2024.findings-acl.520 Error analysis prompting enables human-like translation evaluation in large language models . In Findings of the Association for Computational L...

    2024 doi

  14. [22]

    Guerreiro, Ricardo Rei, Duarte M

    Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. http://ar...

    2024 arXiv

  15. [23]

    OpenAI. 2024. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o mini: Advancing cost-efficient intelligence

    2024

  16. [24]

    Ben Peters and Andre Martins. 2025. https://doi.org/10.18653/v1/2025.acl-long.122 Did translation models get more robust without anyone E ven noticing? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2445--245...

    2025 doi

  17. [25]

    Maja Popovi \'c . 2015. https://doi.org/10.18653/v1/W15-3049 chr F : character n-gram F -score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, 392--395. Association for Computational Linguistics

    2015 doi

  18. [26]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, 186--191. Association for Computational Linguistics

    2018 doi

  19. [27]

    Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, and Fred Blain. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.214 What do large language models need for machine translation evaluation? In Proceedings of the 2024 Con...

    2024 doi

  20. [28]

    Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, and Aram Galstyan. 2024. https://aclanthology.org/2024.findings-eacl.91/ Prompt perturbation consistency learning for robust language models . In Findings of the Association for Computa...

    2024

  21. [29]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022 a . https://aclanthology.org/2022.wmt-1.52/ COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In...

    2022

  22. [30]

    Guerreiro, Jo \ a o Alves, Pedro Henrique Martins, Patrick Fernandes, Helena Wu, Tania Vaz, Duarte Alves, Amin Farajian, Sweta Agrawal, Antonio Farinhas, Jos \'e G

    Ricardo Rei, Jose Pombal, Nuno M. Guerreiro, Jo \ a o Alves, Pedro Henrique Martins, Patrick Fernandes, Helena Wu, Tania Vaz, Duarte Alves, Amin Farajian, Sweta Agrawal, Antonio Farinhas, Jos \'e G. C. De Souza, and Andr \'e Martins. 2024. https://doi.org/10.18653/v1/2024.wmt-...

    2024 doi

  23. [31]

    Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e G

    Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos \'e G. C. de Souza , Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and Andr \'e F. T. Martins. 2022 b . https://aclanthology.org/2022.wmt-1.60 Cometkiwi: Ist-unba...

    2022

  24. [32]

    Nils Reimers and Iryna Gurevych. 2019. http://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  25. [33]

    Ayako Sato, Kyotaro Nakajima, Hwichan Kim, Zhousi Chen, and Mamoru Komachi. 2024. https://doi.org/10.18653/v1/2024.wmt-1.38 TMU - HIT `s submission for the WMT 24 quality estimation shared task: Is GPT -4 a good evaluator for machine translation? In Proceedings of the Ninth Co...

    2024 doi

  26. [34]

    Aarohi Srivastava and David Chiang. 2025. https://aclanthology.org/2025.wnut-1.6/ We're calling an intervention: Exploring fundamental hurdles in adapting language models to nonstandard text . In Proceedings of the Tenth Workshop on Noisy and User-generated Text, 45--56, Albuq...

    2025

  27. [35]

    Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. https://doi.org/10.18653/v1/2024.findings-acl.375 Mind your format: Towards consistent evaluation of in-context learning improvements . In Findings of the Association for Computational Linguistics: ACL 2024, 6287--6310. Associa...

    2024 doi

  28. [36]

    Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. https://openreview.net/forum?id=farT6XXntP A paradigm shift in machine translation: Boosting translation performance of large language models . In The Twelfth International Conference on Learning Representations

    2024

  29. [37]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2025 doi arXiv

  30. [38]

    Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. https://proceedings.mlr.press/v202/zhang23m/zhang23m.pdf Prompting large language model for machine translation: A case study . In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org

    2023

  31. [39]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Ab...

    2023

  32. [40]

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Gong, and Xing Xie. 2024. https://doi.org/10.1145/3689217.3690621 Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts . I...

    2024

  33. [41]

    Vil \'e m Zouhar, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. 2024. https://doi.org/10.18653/v1/2024.wmt-1.121 Pitfalls and outlooks in using COMET . In Proceedings of the Ninth Conference on Machine Translation, 1272--1288. Association for Computational Linguistics

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.