Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trends

Daimeng Wei; Hao Yang; Hengchao Shang; Jiaxin Guo; Jinlong Yang; Xiaoyu Chen; Zhiqiang Rao; Zongyao Li

REVIEW 2 major objections 7 minor 1 cited by

Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trends

T0 review · 2 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that neither d-Comet nor LLM-as-a-judge can fully align with human evaluation of document-level translation, and that LLM judges show bias, inaccuracy, and opacity.

desk verdict Competent survey of document-level MT evaluation whose one empirical claim about LLM-as-a-judge inaccuracy is an under-supported anecdote; useful as an entry point, not a research result. read the letter →

arxiv 2504.14804 v1 pith:ICBU6HSP submitted 2025-04-21 cs.CL cs.AI

Jiaxin GUO , Xiaoyu Chen , Zhiqiang Rao , Jinlong Yang , Zongyao Li , Hengchao Shang , Daimeng Wei , Hao Yang This is my paper · ORCID

classification cs.CLcs.AI

keywords AutomaticEvaluationMetricsDocument-levelTranslationd-CometLLM-as-a-judgeMachineReferenceDiversityBiasLargeLanguageModels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that automatic evaluation of document-level machine translation has not yet caught up with the translation systems it is meant to judge. Its central claim is that neither d-Comet, a context-aware extension of the Comet metric, nor LLM-as-a-judge, where a large language model scores or ranks translations, can fully align with human evaluation. The paper also identifies the reasons: reference translations lack diversity, document-level metrics depend on sentence-level alignment that LLM outputs often break, and LLM judges display self-preference bias, inaccuracy across translation lengths, and scores that are hard to interpret. A fair reader should care because evaluation metrics guide model development and system comparison, so a metric that rewards omission-heavy output can send development in the wrong direction. The paper closes by proposing future directions: less sentence-alignment dependence, multi-level and multi-granular evaluation, and evaluation models trained specifically for judging machine translation.

What carries the argument

The two load-bearing mechanisms are d-Comet and LLM-as-a-judge. d-Comet is the method of extending any pretrained sentence-level metric, such as Comet, to the document level by encoding surrounding context; in this paper it stands for the entire class of reference-based document-level metrics and their dependence on crisp sentence alignment. LLM-as-a-judge is the practice of using a large language model to score, rank, or select machine translation outputs; it stands for the reference-free branch of evaluation. The paper works by testing the implicit promises of these two mechanisms against practical constraints: alignment fragility for d-Comet, and bias, inaccuracy, and opacity for LLM-as-a-judge.

What would settle it

A reader could rerun the Table 1 comparison with a fixed prompt, a fixed judge model version, many repetitions with randomized output order, and human adjudication; if the omission-filled 525-token output no longer scores above the 802-token output, the paper's specific evidence for judge inaccuracy collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is a negative result about the current state of the field: no existing automatic metric is a reliable substitute for human evaluation of document-level translation. d-Comet is limited because it requires source, reference, and translation to be split into the same number of sentences and aligned sentence by sentence, while LLM translations often merge or re-segment sentences. LLM-as-a-judge is limited because it is biased toward its own outputs, can be inaccurate when translation lengths differ, and gives little interpretable justification for its scores. The paper supports the inaccuracy claim with a single scored example in Table 1: DeepSeek R1's 525-token translation, which has obvious omissions, received 8.5 from the LLM judge, while DeepSeek V3's 802-token translation, close in length to the 879-token reference, received 8. The paper therefore concludes that neither metric can fully align with human evaluation.

Load-bearing premise

The inaccuracy claim rests on a single scoring example in Table 1, where a shorter, omission-prone output scored higher than a fuller one; if that example is not representative, the paper's empirical case for LLM judge inaccuracy is unsupported.

Editorial extensions

If this is right

System comparisons built on these metrics may misreport quality, since an LLM judge can rank an omission-laden output above a more complete one.
d-Comet is reliable only when documents can be cleanly segmented and aligned sentence by sentence, which LLM-generated translations frequently violate.
Single-reference evaluation will under-reward valid alternative translations, making reported quality worse than actual quality for high-diversity LLM outputs.
Moving to multi-level, multi-granular evaluation with error-type identification, rather than one holistic score, is the paper's proposed route to more interpretable judgments.
Training a specialized evaluation model on human reasoning and scores is the paper's proposed route to more accurate and reliable judgments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If LLM judges systematically prefer shorter outputs, document-level leaderboards may be silently rewarding systems that compress and omit; this could be tested by scoring many outputs with controlled length ratios against human judgments.
The alignment-fragility diagnosis points to a concrete extension: evaluate documents directly on whole-document representations, or use automatic segmentation and alignment, so LLM re-segmentations no longer break the metric.
The paper's error-type proposal suggests a falsifiable design: judges that first identify omissions should penalize omission-laden outputs more than holistic judges do, and that difference could be measured on Table 1-style examples.
Because judge-model versions and prompts are not specified, the reported 8.5-versus-8 gap is probably unstable across judge versions; re-running with several judge models would quantify that instability.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read. The paper is a survey, not a research contribution. The tripartite taxonomy (traditional/model-based/LLM-based) and the challenges (reference diversity, sentence alignment dependency, judge bias/inaccuracy/interpretability) are all assembled from cited work, and the only new artifact is the Table 1 example. That's fine for an overview, and the prose is clear and accurate in most places. Someone entering document-level MT evaluation would get a reasonable map and a good pointer list.

The soft spot is exactly where the reader put it: Section 3.3. The claim that LLM-as-a-judge is 'inaccurate' rests on a single example where R1 (525 tokens) beat V3 (802 tokens) against an 879-token reference. That demonstrates nothing by itself. A shorter translation can be faithful; the table doesn't mark concrete omissions; and there is no judge model version, prompt, temperature, number of runs, position shuffling, or human adjudication. The paper's own Section 2.2 tells practitioners to shuffle and average, but the example doesn't show that was done. The surrounding sentence that 'longer texts are more prone to omissions' is also confusing, since the allegedly omission-prone output is the shorter one. So the abstract's 'inaccuracy' as a core challenge is not supported by the paper's evidence. To be fair, the bias claim is backed by a real cited study, and d-Comet's alignment dependency is a real limitation, so the overall message that automatic metrics have problems survives. The fix is easy: label Table 1 as a motivating anecdote or replace it with a small systematic study.

I don't see a novelty problem beyond what the reader flagged. The self-citations are to earlier MT systems, which is not circular. The survey's main weakness is overclaiming from one anecdote. If that's fixed, it's a decent overview. Honestly, I wouldn't cite it in my own work, but I'd be happy to see a revised version in print. It deserves a serious referee: the topic matters, the authors know it, and the one real flaw is fixable.

Referee Report

2 major / 7 minor

Summary. This manuscript is a short survey of automatic evaluation metrics for document-level machine translation. It organizes the field into reference-based and reference-free schemes, and into traditional (BLEU-style), model-based (COMET, BERTScore, d-COMET), and LLM-based (LLM-as-a-judge) metrics. It then lists four challenges: lack of reference diversity, d-COMET's dependence on sentence-level alignment, LLM-as-a-judge bias/inaccuracy/lack of interpretability, and the gap between automatic metrics and human evaluation. Finally it proposes future research directions such as reducing sentence-level dependency, multi-granular evaluation, and training specialized LRMs for MT evaluation. The paper's most specific empirical contribution is a single example (Table 1) intended to show LLM-as-a-judge inaccuracy.

Significance. If the challenge taxonomy and the empirical claim were properly supported, the paper would be a useful concise reference for practitioners entering document-level MT evaluation. The paper usefully brings together the distinction between sentence- and document-level evaluation, names the key failure modes, and cites the self-preference bias work [25] to substantiate the bias challenge. Its main novelty is the claim, based on the authors' own example, that LLM-as-a-judge is inaccurate for outputs of different lengths; because this claim is not adequately evidenced, the survey's central message is only partially supported. As a survey, it provides structure but not a systematic methodology or comparative synthesis.

major comments (2)

[Section 3.3, Table 1] The paper's central empirical claim that LLM-as-a-judge is 'highly inaccurate' rests on a single anecdote. The judge setup is not reported (prompt, model version, temperature, number of runs, position shuffling), although Section 2.2 states that multiple runs and shuffling are required. Moreover, the ranking is asserted as 'clearly unreasonable' without human adjudication or error annotation. The table itself shows only token counts and scores; no omissions are marked, so the 8.5 versus 8 score does not demonstrate an evaluation error. To support the claim, provide a controlled evaluation with human adjudication and full judge configuration, or explicitly downgrade the claim to an anecdotal observation.
[Section 3.3] The sentence 'using a Large Reasoning Model (LRM) for translation, longer texts are more prone to omissions' is contradicted by the accompanying example: the R1 output that allegedly has 'obvious omissions' is 525 tokens, shorter than the V3 output (802 tokens) and the reference (879 tokens), and no long-text comparison is conducted. Token-count disparity alone cannot distinguish omission from concise paraphrase. Consequently, the concluding sentence that inaccuracy is present 'especially when dealing with translation results of different lengths' is not a supported inference from the presented evidence.

minor comments (7)

[Title] The title contains stray spaces in 'T ranslation' and 'T rends'; these should be corrected throughout.
[Abstract and Section 5] The abstract lists 'bias, inaccuracy, and lack of interpretability' as challenges, but the conclusion lists only 'bias and lack of interpretability'; align the two lists.
[Section 2.2] The text says 'Comet includes several variants, such as Comet20 and Comet22' but does not specify precisely which models these abbreviations refer to; please expand or add citations.
[Section 2.2] The statement that 'in some machine translation research, the LLM-as-a-judge evaluation method has also been applied' would benefit from concrete citations to that research.
[Section 3.1] The claim that a single reference translation is insufficient for LLM-generated document-level translations is asserted without citations; supporting literature should be provided.
[References] Reference [26] ('Text style transfer back-translation') lacks a publication venue and DOI; the entry should be completed.
[Section 3.4] The claim about the discrepancy between metrics and human evaluation is not supported by citations to WMT Metrics shared-task results or similar studies; adding such references would strengthen the survey.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the survey makes no fitted predictions, and its self-citations are background references, not load-bearing evidence.

full rationale

This is a survey/overview paper rather than a derivation, so most circularity patterns do not apply. The central claims are (i) d-Comet depends on sentence-level alignment, (ii) LLM-as-a-judge exhibits bias, inaccuracy, and low interpretability, and (iii) neither metric fully aligns with human evaluation. Claim (i) restates the mechanism of the externally cited d-Comet method [22]; claim (ii) is supported by an external self-preference bias study [25] plus a single anecdotal experiment in Table 1; claim (iii) is a qualitative synthesis. No equation in the paper defines a metric in terms of its target, no fitted parameter is renamed as a prediction, and the paper does not invoke a uniqueness theorem. The self-citations ([8], [24], [26], [30]) are background references to the authors' MT system papers and do not carry the argument; reference [7], also by the first author, is cited only to name the LLM-as-a-judge paradigm, which is independently established. The Table 1 'inaccuracy' example is methodologically weak: the omission claim is inferred from a token-count disparity rather than human adjudication, the judge prompt, model version, and number of runs are unreported, and the accompanying sentence that 'longer texts are more prone to omissions' is inconsistent with the shorter R1 output being labeled omission-prone. However, weak evidence is a correctness or scope concern, not circularity, because the observation was not constructed from the conclusion. The paper has no derivation chain that reduces to its own inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new free parameters or entities. Its central claims depend on background assumptions about human evaluation as ground truth and on the completeness of its metric taxonomy; these are domain assumptions common in MT evaluation surveys.

assumptions (3)

domain assumption Human evaluation is the gold standard reference for translation quality
The paper repeatedly compares automatic metrics against human evaluation (Sections 1.2 and 3.4) without providing its own human agreement data or discussing the recognized subjectivity of human judgments.
domain assumption The taxonomy into traditional, model-based, and LLM-based metrics is complete and adequate for document-level translation evaluation
The survey organizes the field by this tripartite scheme (Section 2.2) but does not justify it as exhaustive or compare against alternative taxonomies from other surveys.
domain assumption LLM-as-a-judge practice of repeated averaging and random shuffling mitigates position and stochasticity bias
Section 3.3 states that multiple evaluations are averaged and results randomly shuffled, but no experiment in the paper validates that this mitigation is sufficient.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trends." pith.science (2026). https://pith.science/paper/ICBU6HSP

@misc{pith2026250414804,
  author       = {Pith},
  title        = {Pith review of: Automatic Evaluation Metrics for Document-level Translation: Overview, Challenges and Trends},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ICBU6HSP}},
  note         = {Machine review of arXiv:2504.14804}
}

read the original abstract

With the rapid development of deep learning technologies, the field of machine translation has witnessed significant progress, especially with the advent of large language models (LLMs) that have greatly propelled the advancement of document-level translation. However, accurately evaluating the quality of document-level translation remains an urgent issue. This paper first introduces the development status of document-level translation and the importance of evaluation, highlighting the crucial role of automatic evaluation metrics in reflecting translation quality and guiding the improvement of translation systems. It then provides a detailed analysis of the current state of automatic evaluation schemes and metrics, including evaluation methods with and without reference texts, as well as traditional metrics, Model-based metrics and LLM-based metrics. Subsequently, the paper explores the challenges faced by current evaluation methods, such as the lack of reference diversity, dependence on sentence-level alignment information, and the bias, inaccuracy, and lack of interpretability of the LLM-as-a-judge method. Finally, the paper looks ahead to the future trends in evaluation methods, including the development of more user-friendly document-level evaluation methods and more robust LLM-as-a-judge methods, and proposes possible research directions, such as reducing the dependency on sentence-level information, introducing multi-level and multi-granular evaluation approaches, and training models specifically for machine translation evaluation. This study aims to provide a comprehensive analysis of automatic evaluation for document-level translation and offer insights into future developments.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translation
cs.CL 2025-09 conditional novelty 5.0 of 10

An evaluation framework for document-level MT that aligns source and target sentences via dynamic programming and averages 1-4 chunk sliding-window scores, reporting 0.929 Pearson correlation with human MQM and enabli...

Reference graph

Works this paper leans on

31 extracted references · 8 canonical work pages · cited by 1 Pith paper

[25]

Wataoka, K., Takahashi, T., Ri, R.: Self-preference bia s in llm-as-a-judge (2024), https://arxiv.org/abs/2410.21819

arXiv 2024
[1]

Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D. , Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., T u, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H ., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Y...

arXiv doi:10.48550/arxiv.2309.16609 2023
[2]

In: Ku, L., Martins, A., Srikumar, V

Cui, M., Du, J., Zhu, S., Xiong, D.: Eﬃciently exploring la rge language models for document-level machine translation with in-con text learning. In: Ku, L., Martins, A., Srikumar, V. (eds.) Findings of the Asso ciation for Computational Linguistics, ACL 2024, Bangkok, Thailand an d virtual meet- ing, August 11-16, 2024. pp. 10885–10897. Association for C...

doi:10.18653/v1/2024.findings-acl.646 2024
[3]

CoRR abs/1810.04805 (2018), http://arxiv.org/abs/1810.04805

Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-t raining of deep bidirec- tional transformers for language understanding. CoRR abs/1810.04805 (2018), http://arxiv.org/abs/1810.04805

arXiv 2018
[4]

In : Zong, C., Xia, F., Li, W., Navigli, R

Fernandes, P., Yin, K., Neubig, G., Martins, A.F.T.: Meas uring and increas- ing context usage in context-aware machine translation. In : Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Proceedings of the 59th Annual Mee ting of the As- sociation for Computational Linguistics and the 11th Inter national Joint Con- ference on Natural Language Processing, ACL...

2021
[5]

In: I nui, K., Jiang, J., Ng, V., Wan, X

Ghazvininejad, M., Levy, O., Liu, Y., Zettlemoyer, L.: Ma sk-predict: Par- allel decoding of conditional masked language models. In: I nui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th I nterna- tional Joint Conference on Natural Language Processing, EM NLP-IJCNL...

doi:10.18653/v1/d19 2019
[6]

In: 6th International Confere nce on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, Apr il 30 - May 3, 2018, Conference Track Proceedings

Gu, J., Bradbury, J., Xiong, C., Li, V.O.K., Socher, R.: No n-autoregressive neural machine translation. In: 6th International Confere nce on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, Apr il 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net (2018), https://openreview.net/forum?id=B1l8BtlCb

work page 2018
[7]

: A survey on llm-as-a-judge (2025), https://arxiv.org/abs/2411.15594

Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, S., Zhang, K., Wang, Y., Gao, W., Ni, L., Guo, J. : A survey on llm-as-a-judge (2025), https://arxiv.org/abs/2411.15594

arXiv 2025

Show all 31 references

[8]

CoRR abs/2112.11640 (2021), https://arxiv.org/abs/2112.11640

Guo, J., Wang, M., Wei, D., Shang, H., Wang, Y., Li, Z., Yu, Z ., Wu, Z., Chen, Y., Su, C., Zhang, M., Lei, L., Tao, S., Yang, H.: Self-distillat ion mixup training for non-autoregressive neural machine translation. CoRR abs/2112.11640 (2021), https://arxiv.org/abs/2112.11640

2021 arXiv
[9]

(eds.) Proceedings of the Fourth Workshop on Discourse in Machine Translation, DiscoMT@EMN LP 2019, Hong Kong, China, November 3, 2019

Kim, Y., Tran, D.T., Ney, H.: When and why is document-leve l con- text useful in neural machine translation? In: Popescu-Bel is, A., Loái- ciga, S., Hardmeier, C., Xiong, D. (eds.) Proceedings of the Fourth Workshop on Discourse in Machine Translation, DiscoMT@EMN LP 2019, Ho...

2019 doi
[10]

Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: L arge language models are zero-shot reasoners (2023), https://arxiv.org/abs/2205.11916

2023 arXiv
[11]

CoRR abs/1907.11692 (2019), http://arxiv.org/abs/1907.11692

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Le vy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimiz ed BERT pretraining approach. CoRR abs/1907.11692 (2019), http://arxiv.org/abs/1907.11692

2019 arXiv
[12]

In: Moens, M., Huang, X., Specia, L., Yih, S.W

Lyu, X., Li, J., Gong, Z., Zhang, M.: Encouraging lexical translation consistency for document-level neural machine translation. In: Moens, M., Huang, X., Specia, L., Yih, S.W. (eds.) Proceedings of the 2021 Conference on Em pirical Methods in Natural Language Processing, EMNL...

2021 doi
[13]

ACM Comput

Maruf, S., Saleh, F., Haﬀari, G.: A survey on document-le vel neural machine trans- lation: Methods and evaluation. ACM Comput. Surv. 54(2), 45:1–45:36 (2022). https://doi.org/10.1145/3441691, https://doi.org/10.1145/3441691

2022 doi
[14]

CoRR abs/2303.08774 (2023)

OpenAI: GPT-4 technical report. CoRR abs/2303.08774 (2023). https://doi.org/10.48550/ARXIV.2303.08774, https://doi.org/10.48550/arXiv.2303.08774

2023 doi arXiv
[15]

In: Proceedings of the 40th A nnual Meeting of 10 Jiaxin GUO, Xiaoyu Chen et al

Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a metho d for automatic eval- uation of machine translation. In: Proceedings of the 40th A nnual Meeting of 10 Jiaxin GUO, Xiaoyu Chen et al. the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, US...

2002
[16]

In: Webber, B., Cohn, T., He, Y., Liu, Y

Rei, R., Stewart, C., Farinha, A.C., Lavie, A.: COMET: A n eural framework for MT evaluation. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds. ) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pr ocessing, EMNLP 2020, Online, November 16-20, 2020. pp...

2020 doi
[17]

Sennrich, R., Haddow, B., Birch, A.: Improving neural ma chine translation models with monolingual data (2016)

2016
[18]

In: Moens, M., Huang, X., Specia, L., Yih, S.W

Tan, X., Zhang, L., Zhou, G.: Coupling context modeling w ith zero pronoun recovering for document-level natural language ge neration. In: Moens, M., Huang, X., Specia, L., Yih, S.W. (eds.) Proceedin gs of the 2021 Conference on Empirical Methods in Natural Languag e Pro- cess...

2021 doi
[19]

, Joulin, A., Grave, E., Lample, G.: Llama: Open and eﬃcient foundation la nguage models

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lach aux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A. , Joulin, A., Grave, E., Lample, G.: Llama: Open and eﬃcient foundation la nguage models. CoRR abs/2302.13971 (2023). https://doi.org/10....

2023 doi arXiv
[20]

, Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E.M

Touvron, H., Martin, L., Stone, K., Albert, P., Almahair i, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., B lecher, L., Canton- Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J ., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V.,...

2023 doi arXiv
[21]

In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vish wanathan, S.V.N., Garnett, R

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jon es, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vish wanathan, S.V.N., Garnett, R. (eds.) Advances in Neural Information ...

2017
[22]

Vernikos, G., Thompson, B., Mathur, P., Federico, M.: Em barrassingly easy document-level mt metrics: How to convert any pretrained me tric into a document- level metric (2022), https://arxiv.org/abs/2209.13654

2022 arXiv
[23]

In: Palmer, M., Hwa, R., Riedel, S

Wang, L., Tu, Z., Way, A., Liu, Q.: Exploiting cross-sent ence context for neural machine translation. In: Palmer, M., Hwa, R., Riedel, S. (ed s.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Pr ocessing, EMNLP Automatic Evaluation Metrics for Do...

2017 doi
[24]

I n: Belz, A., Fan, A., Reiter, E., Sripada, Y

Wang, M., Guo, J., Wang, Y., Chen, Y., Su, C., Wei, D., Zhan g, M., Tao, S., Yang, H.: HI-CMLM: improve CMLM with hybrid decoder input. I n: Belz, A., Fan, A., Reiter, E., Sripada, Y. (eds.) Proceedings of the 14 th International Con- ference on Natural Language Generation, IN...

2021
[26]

Wei, D., Wu, Z., Shang, H., Li, Z., Wang, M., Guo, J., Chen, X., Yu, Z., Yang, H.: Text style transfer back-translation (2023)

2023
[27]

C oRR abs/2401.06468 (2024)

Wu, M., Vu, T., Qu, L., Foster, G.F., Haﬀari, G.: Adapting large language models for document-level machine translation. C oRR abs/2401.06468 (2024). https://doi.org/10.48550/ARXIV.2401.06468, https://doi.org/10.48550/arXiv.2401.06468

2024 doi arXiv
[28]

In: Koehn, P., Haddon, B., Kocmi, T., Monz, C

Wu, Y., Hu, G.: Exploring prompt engineering with GPT lan guage models for document-level machine translation: Insights a nd ﬁndings. In: Koehn, P., Haddon, B., Kocmi, T., Monz, C. (eds.) Proceed ings of the Eighth Conference on Machine Translation, WMT 2023, S in- gapore, Dec...

2023 doi
[29]

, Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai , S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X

Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li , C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J ., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, ...

2024 doi arXiv
[30]

, Wang, Y., Chen, Y., Su, C., Zhang, M., Lei, L., Tao, S., Yang, H.: Joint-train ing on symbiosis networks for deep nueral machine translation models

Yu, Z., Guo, J., Wang, M., Wei, D., Shang, H., Li, Z., Wu, Z. , Wang, Y., Chen, Y., Su, C., Zhang, M., Lei, L., Tao, S., Yang, H.: Joint-train ing on symbiosis networks for deep nueral machine translation models. CoRR abs/2112.11642 (2021), https://arxiv.org/abs/2112.11642

2021 arXiv
[31]

CoRR abs/1904.09675 (2019), http://arxiv.org/abs/1904.09675 Ϭ ϱ ϭϬ ϭϱ ϮϬ Ϯϱ ϯϬ ϯϱ ϰϬ ϰϱ ϱϬ Ϭ ϱ ϭϬ ϭϱ ϮϬ Ϯϱ ϯϬ ĂƚĂ ĂƚĂ

Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with BERT. CoRR abs/1904.09675 (2019), http://arxiv.org/abs/1904.09675 Ϭ ϱ ϭϬ ϭϱ ϮϬ Ϯϱ ϯϬ ϯϱ ϰϬ ϰϱ ϱϬ Ϭ ϱ ϭϬ ϭϱ ϮϬ Ϯϱ ϯϬ ĂƚĂ ĂƚĂ

2019 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [25]

Wataoka, K., Takahashi, T., Ri, R.: Self-preference bia s in llm-as-a-judge (2024), https://arxiv.org/abs/2410.21819

arXiv 2024

[2] [1]

Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D. , Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., T u, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H ., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Y...

arXiv doi:10.48550/arxiv.2309.16609 2023

[3] [2]

In: Ku, L., Martins, A., Srikumar, V

Cui, M., Du, J., Zhu, S., Xiong, D.: Eﬃciently exploring la rge language models for document-level machine translation with in-con text learning. In: Ku, L., Martins, A., Srikumar, V. (eds.) Findings of the Asso ciation for Computational Linguistics, ACL 2024, Bangkok, Thailand an d virtual meet- ing, August 11-16, 2024. pp. 10885–10897. Association for C...

doi:10.18653/v1/2024.findings-acl.646 2024

[4] [3]

CoRR abs/1810.04805 (2018), http://arxiv.org/abs/1810.04805

Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-t raining of deep bidirec- tional transformers for language understanding. CoRR abs/1810.04805 (2018), http://arxiv.org/abs/1810.04805

arXiv 2018

[5] [4]

In : Zong, C., Xia, F., Li, W., Navigli, R

Fernandes, P., Yin, K., Neubig, G., Martins, A.F.T.: Meas uring and increas- ing context usage in context-aware machine translation. In : Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Proceedings of the 59th Annual Mee ting of the As- sociation for Computational Linguistics and the 11th Inter national Joint Con- ference on Natural Language Processing, ACL...

2021

[6] [5]

In: I nui, K., Jiang, J., Ng, V., Wan, X

Ghazvininejad, M., Levy, O., Liu, Y., Zettlemoyer, L.: Ma sk-predict: Par- allel decoding of conditional masked language models. In: I nui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th I nterna- tional Joint Conference on Natural Language Processing, EM NLP-IJCNL...

doi:10.18653/v1/d19 2019

[7] [6]

In: 6th International Confere nce on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, Apr il 30 - May 3, 2018, Conference Track Proceedings

Gu, J., Bradbury, J., Xiong, C., Li, V.O.K., Socher, R.: No n-autoregressive neural machine translation. In: 6th International Confere nce on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, Apr il 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net (2018), https://openreview.net/forum?id=B1l8BtlCb

work page 2018

[8] [7]

: A survey on llm-as-a-judge (2025), https://arxiv.org/abs/2411.15594

Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, S., Zhang, K., Wang, Y., Gao, W., Ni, L., Guo, J. : A survey on llm-as-a-judge (2025), https://arxiv.org/abs/2411.15594

arXiv 2025