A rubric-plus-question-answering LLM framework for literary translation evaluation beats traditional MT metrics but still trails human agreement, especially on Korean honorifics.
Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Translating literary works has perennially stood as an elusive dream in machine translation (MT), a journey steeped in intricate challenges. To foster progress in this domain, we hold a new shared task at WMT 2023, the first edition of the Discourse-Level Literary Translation. First, we (Tencent AI Lab and China Literature Ltd.) release a copyrighted and document-level Chinese-English web novel corpus. Furthermore, we put forth an industry-endorsed criteria to guide human evaluation process. This year, we totally received 14 submissions from 7 academia and industry teams. We employ both automatic and human evaluations to measure the performance of the submitted systems. The official ranking of the systems is based on the overall human judgments. In addition, our extensive analysis reveals a series of interesting findings on literary and discourse-aware MT. We release data, system outputs, and leaderboard at http://www2.statmt.org/wmt23/literary-translation-task.html.
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls
A rubric-plus-question-answering LLM framework for literary translation evaluation beats traditional MT metrics but still trails human agreement, especially on Korean honorifics.