A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls

Alice Oh; Dongkeun Yoon; Jiwoo Choi; Seohyon Jung; Sheikh Shafayat; Woori Jang

REVIEW 3 major objections 5 minor 62 references

A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-step framework beats standard MT metrics for literary translation but trails humans on Korean honorifics.

desk verdict A careful, honest empirical study on LLM-based literary translation evaluation; the baseline comparison is weaker than the headline claim, but the core findings hold up and deserve a serious referee. read the letter →

arxiv 2412.01340 v3 pith:Y5JHNBBX submitted 2024-12-02 cs.CL

Sheikh Shafayat , Dongkeun Yoon , Woori Jang , Jiwoo Choi , Alice Oh , Seohyon Jung This is my paper · ORCID

classification cs.CL

keywords literarytranslationmachineevaluationLLM-as-a-judgerubric-basedquestion-answeringKoreanhonorificsEnglish-to-Koreaninterpretablemetrics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

To automate the evaluation of literary machine translation, this paper proposes a two-step framework that scores English-to-Korean translations on both general quality and story-specific literary fidelity. The first step, RULER, rates paragraphs with a human-crafted rubric covering lexical choice, honorifics, syntax/grammar, and content accuracy. The second step, VERSE, has one LLM generate story-specific literary questions and another verify whether the translation satisfies them. The paper reports that these fine-grained scores correlate with expert human judgments more strongly than traditional machine-translation metrics such as COMET, BERTScore, BLEURT, and GEMBA-MQM, though not as well as human annotators agree with each other. The authors present this as evidence that automated literary translation evaluation is feasible but not yet a replacement for human judgment, particularly on culturally loaded features like Korean honorifics.

What carries the argument

RULER is a four-criteria Likert-scale rubric (1–5) for lexical choice, proper use of Korean honorifics in dialogue, syntax and grammar, and content accuracy, based on the Multidimensional Quality Metrics (MQM) framework. VERSE is a two-agent verification module: an LLM generates story-specific literary questions from a short story summary, and a second LLM grades each question on a 1–3 scale. Together they turn 'quality of a literary translation' into a vector of interpretable scores, and the paper's key evidence is the correlation of those scores with expert human annotations.

What would settle it

Evaluate the same translations at matching granularity—for example, compute baseline metrics on full paragraphs by averaging or concatenating sentences, or run RULER/VERSE on single sentences—and check whether the reported correlation advantage over COMET, BERTScore, BLEURT, and GEMBA-MQM persists; if the advantage disappears or reverses, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that an automated two-stage pipeline—rubric-based rating (RULER) followed by story-specific question-answering verification (VERSE)—can evaluate literary machine translation in a fine-grained and interpretable way, and that it correlates with expert human judgment better than standard machine-translation metrics. Using 15 English short stories translated to Korean, with 725 aligned paragraphs and three literary experts as annotators, the authors find that the framework outperforms COMET, BERTScore, BLEURT, and GEMBA-MQM on correlation with human scores across all RULER criteria and on VERSE. The claim is qualified: the framework does not reach inter-human agreement, and the honorific category is the clearest failure mode, where LLM judges miss serious mistakes. The paper also reports that LLM evaluators tend to rate translations produced by other LLMs higher than human translations, and that chain-of-thought prompting does not improve evaluation accuracy while removing the rubric hurts more than removing the reference translation.

Load-bearing premise

The fair-comparison assumption: the baselines (COMET, BERTScore, BLEURT, GEMBA-MQM) are sentence-level metrics while RULER and VERSE evaluate paragraphs, and the paper itself says the comparison 'may not be entirely appropriate'; if the baseline numbers are not fairly obtained, the headline correlation advantage is not established.

Editorial extensions

If this is right

Literary translation quality can be scored as a profile of interpretable dimensions—lexical choice, honorifics, syntax, content, and story-specific literary criteria—rather than a single opaque number, giving translators and pipeline developers actionable feedback.
The framework can be used as an automated signal in agentic translation systems, letting developers test multiple translation attempts without running a human evaluation each time, though final screening for culturally sensitive features should remain human.
Rubrics are the stronger lever: removing the rubric degrades scores more than removing the reference translation, so designing explicit, language-specific evaluation guidelines matters more than supplying examples.
Chain-of-thought prompting and few-shot examples do not consistently help LLM evaluators in this domain, so simply giving the judge more reasoning time or examples is not the path to closing the gap with human agreement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The honorific failure mode likely extends to other register-rich languages (Japanese, Javanese, Thai); a testable prediction is that the gap between LLM judges and humans widens with the complexity of the target language's honorific system.
If the LLM-prefers-LLM bias is model-family-specific rather than a general quality confusion, then pairing translator and evaluator from the same model family could further inflate scores; this could be tested by cross-family evaluations.
The finding that reasoning prompts do not help suggests the bottleneck is missing cultural knowledge, not reasoning capability; injecting explicit honorific rules into the rubric might reduce the gap and is a concrete extension of RULER.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a careful, honest empirical study, and the new thing is real: a two-step pipeline for evaluating English-to-Korean literary translation, combining a rubric-based stage (RULER) with a QA-based story-specific stage (VERSE). Neither component is brand new, but the combination, applied to literary MT with culture-specific rubrics like Korean honorifics, is a legitimate contribution. The paper does the right reproducibility things—code and data are public, exact model versions are listed, ablations cover CoT, few-shot, and rubric removal, and inter-annotator agreement is reported. I trust the core numbers.

The softest spot is the baseline comparison. COMET, BERTScore, BLEURT, and GEMBA-MQM are sentence-level metrics; RULER and VERSE score whole paragraphs. The authors admit this in Section 5 and Appendix B.1, which is good, but it means the headline claim—'higher correlation with human judgment than traditional MT metrics'—rests on a comparison that is, at best, rough. If the sentence-level metrics were fed 6.8-sentence paragraphs without a stated aggregation or truncation rule, their scores could be artificially depressed. The stress-test concern is legitimate. That said, the authors flag it themselves, and the magnitude of the gap (e.g., Lexical tau 0.68 vs 0.23) is large enough that I suspect the direction would survive a fair comparison, though not necessarily the exact margins.

Other soft spots are minor: no confidence intervals or significance tests on the correlations; the honorifics analysis is dominated by the default score of 5 for non-dialogue paragraphs; and the VERSE question quality check was one expert on 50 questions. The mild circularity—rubric categories induced from observed LLM errors, and the same model generating and grading VERSE questions—is a design limitation, not a fatal flaw, and the authors note it.

The LLM self-preference finding (the judge rates LLM translations above human gold, with a small A/B check where a human picked gold 39/50 times) is interesting and worth following up.

Who this is for: people building LLM-as-a-judge metrics for long-form or literary text, and MT evaluation researchers generally. This is not a paper with a fatal flaw; it is a solid empirical contribution with an overly strong abstract given the baseline caveat. I would send it to peer review with a request for matched evaluation units, significance testing, and a toned-down claim. Conditional accept.

Referee Report

3 major / 5 minor

Summary. The paper proposes a two-step framework for evaluating English-to-Korean literary machine translation. Step 1 (RULER) uses a Likert-scale rubric with four criteria (lexical choice, honorifics, syntax/grammar, content accuracy), and Step 2 (VERSE) uses LLM-generated story-specific verification questions that a second LLM grades. The authors validate the framework against expert human annotations on 200 paragraphs from 10 translated short stories, compare it with COMET, BERTScore, BLEURT, and GEMBA-MQM, and report Kendall's tau, Spearman's rho, and MSE. They find that GPT-4o as an evaluator outperforms the traditional metrics but still falls short of inter-human agreement, particularly on Korean honorifics. They also observe that the evaluator LLM tends to rate LLM-generated translations higher than human reference translations. The paper includes ablations (chain-of-thought, few-shot, rubric removal, reference removal), exact model versions, and public code and data.

Significance. If the central comparison were properly controlled, the framework would be a useful contribution to literary MT evaluation: it offers fine-grained, interpretable dimensions (honorifics, style, imagery, etc.) and its negative results about LLM judges' limitations on culturally specific features are valuable to the community. The paper's strengths include reproducible model versions, released data and code, human checks of generated questions, and a concrete A/B test illustrating LLM bias against human translations. However, the headline claim of superiority over traditional MT metrics rests on a baseline comparison that the authors themselves concede is 'not entirely appropriate,' and the exact rubric prompts needed to reproduce RULER are not included. These issues are fixable within the manuscript's scope, so a major revision is appropriate.

major comments (3)

[Section 5, Baseline; Appendix B.1; Table 4] The comparison that supports the abstract's claim of 'higher correlation with human judgment than traditional machine translation metrics' is not apples-to-apples. The authors state that the baselines 'may not be entirely appropriate' because they are designed for sentence-level evaluation while RULER and VERSE operate on paragraphs, and Appendix B.1 notes that GEMBA-MQM and the framework's numbers are 'not really comparable' due to different error categories. The manuscript does not specify how BERTScore, BLEURT, COMET, and GEMBA-MQM were applied to the 200 annotated paragraphs, whether input-length limits caused truncation, how segment scores were aggregated to paragraph level, or whether the baselines were evaluated on the same 200 paragraphs. Table 4 also reports no confidence intervals or significance tests for the tau/rho differences, so the reported advantage (e.g., RULER Lexical tau=0.68 vs. BLEURT tau=0.23) could reflect a procedural handicap rather than genuine metric quality. Please provide the missing implementation details and matched evaluation units, or weaken the superiority claim accordingly.
[Section 4.1] The full RULER rubric is not included in the manuscript; the text says 'The detailed rubric for each criterion will be released with our code and data after acceptance.' Since RULER scores are obtained by prompting an LLM with this rubric, the reported correlations in Table 4 cannot be independently reproduced from the paper alone. The exact rubric wording, the VERSE question-generation prompt, and the grading prompts should be included in an appendix or in the released repository at the time of submission.
[Section 6.3, Table 6, and the 'What is Remembered' A/B test] The abstract's claim that 'LLMs tend to favor translations produced by other LLMs' is supported by only one external A/B test: a single human annotator on 50 paragraphs from one story. The aggregate scores in Table 6 are produced by the same GPT-4o evaluator whose bias is being demonstrated, so they do not independently establish the claim. Please strengthen this analysis with multiple stories, multiple annotators, and inter-annotator reliability for the A/B judgments, or present the finding as a preliminary observation rather than a general result.

minor comments (5)

[Table 4] MSE is reported for the proposed framework and LLM judges but omitted for COMET, BERTScore, BLEURT, and GEMBA-MQM; if these metrics produce scores on different scales, please state this explicitly so the reader knows the dash entries are not missing values.
[Section 5.2] The sentence 'language models are much better at correlating with human preference... than the traditional MT metrics' should be supported with a significance test or confidence interval; the raw tau differences in Table 4 may not be statistically distinguishable given 200 paragraphs.
[Section 6.3] Please clarify whether 'Claude-3.5 Sonnet' in the prose refers to 'claude-3-5-sonnet-20240620' in Table 9, since the translated model names in Tables 6 and 7 are abbreviated inconsistently.
[Section 5.3, ablation bullets] There is a typo: 'specially with honorifics' should be 'especially with honorifics'.
[Section 4.2 and Figure 2] The radial axes in Figures 2, 4, and 6 lack numerical tick labels, making it difficult to read exact percentages; consider adding gridlines or a legend with values.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the core validation rests on independent human annotations, and the self-referential design elements do not reduce the central claim.

full rationale

The paper's central claim is an empirical one: that the two-stage framework's scores correlate with expert human judgments better than traditional MT metrics do. The headline correlations in Table 4 are computed against human annotations that were collected independently of the model outputs; the rubric and questions were not fitted to those human scores. The RULER rubric was developed from a qualitative error analysis of machine translations, but the gold labels used for validation are the three literary experts' independent ratings, so the agreement is not an artifact of construction. The VERSE module does have a self-referential design: GPT-4o generates the verification questions and GPT-4o grades the answers, and the question taxonomy is also assigned by GPT-4o. However, the paper validates VERSE against human judgments and reports a human inspection of 50 generated questions, so the same-model generation does not by itself make the correlation with human judgment circular. The paper also explicitly concedes that the comparison with COMET, BERTScore, BLEURT, and GEMBA-MQM 'may not be entirely appropriate' because those are sentence-level metrics applied to paragraph-level texts (Section 5 and Appendix B.1). That is a genuine methodological weakness that threatens the fairness of the superiority claim, but it is not a circular reduction of the kind where a prediction is equivalent to its input by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameters renamed as predictions. The mild self-referentiality in the VERSE question generation and the shared annotators between rubric development and validation are limitations worth scrutiny, but they do not make the central empirical result circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework's inputs are mostly design choices and domain assumptions; no new physical or mathematical entities are introduced. The most consequential assumption is that expert human ratings are a stable ground truth, followed by the representativeness of the 15-story corpus and the validity of LLM-generated verification questions.

free parameters (4)

Default honorific score for non-dialogue paragraphs = 5
RULER assigns 5 automatically to paragraphs without dialogue or register expressions; this creates a strongly skewed label distribution and is central to the honorifics analysis in Section 5.1.
Number of VERSE questions per paragraph = 10
The paper states 'the language model evaluates 10 questions per paragraph' (Section 6.2); this hand-chosen count affects the granularity and robustness claims for Step 2.
RULER Likert scale range = 1 to 5
Hand-chosen rating granularity for the four rubric criteria (Section 4.1).
VERSE grading scale range = 1 to 3
Hand-chosen granularity for judging whether story-specific criteria are satisfied (Section 4.2).

assumptions (5)

domain assumption Expert human judgments are a valid ground truth for literary translation quality.
The entire validation compares model scores to three literary experts' ratings (Sections 3 and 5); if expert judgment is unstable, the framework has no benchmark.
domain assumption Inter-human agreement is the appropriate performance ceiling for automated evaluation.
The paper measures success as closing the gap to human-human agreement (Table 2 vs Table 4), which assumes that expert-level agreement is the correct target.
domain assumption The curated corpus of 15 English short stories and 725 paragraphs is representative of literary translation.
All experiments and conclusions are drawn from this corpus; the authors note in Limitations that this is only English-to-Korean and generalizability is untested.
domain assumption GPT-4o-generated story questions are valid evaluation criteria after one expert review of 50 items.
The VERSE questions were validated by a single expert on 50 randomly sampled questions (44 very helpful); there is no evidence the full question set covers all important literary qualities (Section 4.2).
standard math Rank correlations and MSE are sufficient statistics for comparing evaluator quality.
The analysis uses Kendall's tau, Spearman's rho, and MSE without confidence intervals or significance tests (Section 3).

how reviews work

0 comments

Cite this review

Pith. "Pith review of A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls." pith.science (2026). https://pith.science/paper/Y5JHNBBX

@misc{pith2026241201340,
  author       = {Pith},
  title        = {Pith review of: A 2-step Framework for Automated Literary Translation Evaluation: Its Promises and Pitfalls},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y5JHNBBX}},
  note         = {Machine review of arXiv:2412.01340}
}

read the original abstract

In this work, we propose and evaluate the feasibility of a two-stage pipeline to evaluate literary machine translation, in a fine-grained manner, from English to Korean. The results show that our framework provides fine-grained, interpretable metrics suited for literary translation and obtains a higher correlation with human judgment than traditional machine translation metrics. Nonetheless, it still fails to match inter-human agreement, especially in metrics like Korean Honorifics. We also observe that LLMs tend to favor translations generated by other LLMs, and we highlight the necessity of developing more sophisticated evaluation methods to ensure accurate and culturally sensitive machine translation of literary works.

Figures

Figures reproduced from arXiv: 2412.01340 by the authors.

**Figure 1.** The overview of our proposed framework: we evaluate translation of literary works in two stages. In the first step, we focus on semantic and linguistic features of the translation using a human-generated rubric. In the second step, we verify how much of the literary qualities have been captured through the translation by two-agent question answering, where one LLM generates a list of criteria and the other LLM veri… view at source ↗

**Figure 2.** Different stories require different kinds of attention to ensure that the literary essence of the work is preserved in the translation. The categories were found after classifying them in zero shot manner. See Section 4.2. The radial axis represents the percentage of questions belonging to that category for that particular story [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Confusion matrix of the zero-shot question classification. The category names are slightly abridged to fit in the figure. Notice that most apparent misclassifications are usually in a very closely related class and human inspection revealed that in most cases the misclassified class can also be considered valid. Clustering LLM Generated Questions To provide better interpretability for the LLM generated questions i… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Radar plot for scores (as percentage) across different categories of translation for each model in The Philosopher story. The radial axis begins at 40%. See [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Confusion matrices for Korean honorifics show strong agreement between human annotators (top row), who consistently mark some paragraphs as very poor (dark blue upper-left corner). In contrast, GPT-4o fails to capture this pattern (lighter upper-left corner, bottom row…

**Figure 6.** Figure 6: Fine-grained evaluation of 8 stories using VERSE. Notice that models score widely differently for different stories. The difference is more pronounced for less powerful model, for whom evaluator LLM is more sensitive. n refers to the number of questions belonging to th…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 26 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Anthropic . 2024 a . The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic.com/news/claude-3-family. Accessed: 2025-09-01

work page 2024
[4]

Anthropic . 2024 b . Introducing claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. Accessed: 2025-09-01

work page 2024
[5]

Anthropic . 2024 c . Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet. https://assets.anthropic.com/m/1cd9d098ac3e6467/original/Claude-3-Model-Card-October-Addendum.pdf. Accessed: 2025-09-01

work page 2024
[6]

Jacob Blakesley, editor. 2018. Sociologies of Poetry Translation: Emerging Perspectives. Bloomsbury Academic, London

work page 2018
[7]

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

arXiv 2023
[8]

Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, and Min Zhang. 2024. http://arxiv.org/abs/2408.09945 Benchmarking llms for translating classical chinese poetry:evaluating adequacy, fluency, and elegance

work page Pith review arXiv 2024

Show all 62 references

[9]

Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. 2023. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. arXiv preprint arXiv:2310.18235

2023 arXiv
[10]

Jaemin Cho, Abhay Zala, and Mohit Bansal. 2024. Visual programming for step-by-step text-to-image generation and evaluation. Advances in Neural Information Processing Systems, 36

2024
[11]

Pierre Colombo, Nuno Guerreiro, Ricardo Rei, Daan Van, Luisa Coheur, and Andr \'e Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics

2023
[12]

DeepSeek-AI et al. 2024. https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . arXiv preprint arXiv:2412.19437

2024 arXiv
[13]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

2019 doi
[14]

Hashimoto

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. http://arxiv.org/abs/2305.14387 Alpacafarm: A simulation framework for methods that learn from human feedback

2023 arXiv
[15]

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Fr \'e d \'e ric Blain, Daniel Deutsch, Craig Stewart, et al. 2023. Results of wmt23 metrics shared task: Metrics might be guilty but references are not innocent. In Proce...

2023
[16]

Google DeepMind and Google Research . 2024. https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical scale . arXiv preprint arXiv:2408.00118

2024 arXiv
[17]

Andrew Grattafiori and et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

2024 arXiv
[18]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, et al. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv:2501.12948

2025 arXiv
[19]

Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2024. Exploring human-like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229--246

2024
[20]

Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ...

2023
[21]

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.08745

2023 arXiv
[22]

Marzena Karpinska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. arXiv preprint arXiv:2304.03245

2023 arXiv
[23]

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Repr...

2023
[24]

Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Fables: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261

2024 arXiv
[25]

Jeremy Klemin. 2024. https://www.theatlantic.com/technology/archive/2024/01/literary-translation-artificial-intelligence/677038/ The last frontier of machine translation . The Atlantic

2024
[26]

Tom Kocmi and Christian Federmann. 2023. Gemba-mqm: Detecting translation quality error spans with gpt-4. arXiv preprint arXiv:2310.13988

2023 arXiv
[27]

Waltraud Kolb. 2023. ‘i am a bit surprised’: Literary translation and post-editing processes compared. In Computer-Assisted Literary Translation, pages 53--68. Routledge

2023
[28]

Mateusz Krubi \'n ski, Erfan Ghadery, Marie-Francine Moens, and Pavel Pecina. 2021. https://aclanthology.org/2021.wmt-1.58/ Just ask! evaluating machine translation by asking and answering questions . In Proceedings of the Sixth Conference on Machine Translation, pages 495--50...

2021
[29]

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470

2023 arXiv
[30]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR

2022
[31]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634

2023 arXiv
[32]

Lieve Macken, Bram Vanroy, Luca Desmet, and Arda Tezcan. 2022. Literary translation as a three-stage process: Machine translation, post-editing and revision. In 23rd Annual Conference of the European Association for Machine Translation, pages 101--110. European Association for...

2022
[33]

Valerie R Mariana. 2014. The Multidimensional Quality Metric (MQM) framework: A new framework for translation quality assessment. Brigham Young University

2014
[34]

Meta AI . 2024. Introducing llama 3.1: Our most capable models to date. https://ai.meta.com/blog/meta-llama-3-1/. Accessed: 2025-09-01

2024
[35]

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251

2023 arXiv
[36]

Jeremy Munday, Sara Ramos Pinto, and Jacob Blakesley. 2022. Introducing translation studies: Theories and applications. Routledge

2022
[37]

OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

2023 arXiv
[38]

OpenAI . 2024. Gpt-4o mini: Advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/. Accessed: 2025-09-01

2024
[39]

OpenAI. 2024. Hello gpt-4. https://openai.com/index/hello-gpt-4o/. Accessed: 2024-11-26

2024
[40]

OpenAI . 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2025-09-01

2024
[41]

OpenAI . 2025 a . Introducing gpt-4.1 in the api. https://openai.com/index/gpt-4-1/. Accessed: 2025-09-01

2025
[42]

OpenAI . 2025 b . Introducing o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/. Accessed: 2025-09-01

2025
[43]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[44]

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M \"u ller, Joe Penna, and Robin Rombach. 2024. https://openreview.net/forum?id=di52zR8xgf SDXL : Improving latent diffusion models for high-resolution image synthesis . In The Twelfth Internationa...

2024
[45]

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. https://proceedings.mlr.press/v139/ramesh21a.html Zero-shot text-to-image generation . In Proceedings of the 38th International Conference on Machine Learni...

2021
[46]

Ricardo Rei, Jos \'e GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e FT Martins. 2022. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Tr...

2022
[47]

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. arXiv preprint arXiv:2009.09025

2020 arXiv
[48]

Juan C Sager. 1998. What distinguishes major types of translation? The Translator, 4(1):69--89

1998
[49]

Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310.10076

2023 arXiv
[50]

Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696

2020 arXiv
[51]

Mary Snell-Hornby. 2006. The turns of translation studies. New Paradigsms or Shifting Viewpoints

2006
[52]

Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucía Tormo-Bañuelos, and Seungone Kim. 2024. http://arxiv.org/abs/2410.17578 Mm-eval: A multilingual meta-evaluation benchmark for llm-as-a-judge and ...

2024 arXiv
[53]

Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, and Mohit Iyyer. 2022. Exploring document-level literary machine translation with parallel paragraphs from world literature. arXiv preprint arXiv:2210.14250

2022 arXiv
[54]

Lawrence Venuti. 2017. The translator's invisibility: A history of translation. Routledge

2017
[55]

Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228

2020 arXiv
[56]

Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Liting Zhou, Chao-Hong Liu, Yufeng Ma, et al. 2023 a . Findings of the wmt 2023 shared task on discourse-level literary translation: A fresh orb in the cosmos of llms. arXiv preprint arXiv:2311.03127

2023 arXiv
[57]

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023 b . Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926

2023 arXiv
[58]

Minghao Wu, Yulin Yuan, Gholamreza Haffari, and Longyue Wang. 2024. (perhaps) beyond human translation: Harnessing multi-agent collaboration for translating ultra-long literary texts. arXiv preprint arXiv:2405.11804

2024 arXiv
[59]

Ran Zhang, Wei Zhao, and Steffen Eger. 2024. http://arxiv.org/abs/2410.18697 How good are llms for literary translation, really? literary translation evaluation with humans and llms

2024 arXiv
[60]

Weinberger, and Yoav Artzi

Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

2020
[61]

Xuan Zhang, Navid Rajabi, Kevin Duh, and Philipp Koehn. 2023. https://doi.org/10.18653/v1/2023.wmt-1.43 Machine translation with large language models: Prompting, few-shot learning, and fine-tuning with QL o RA . In Proceedings of the Eighth Conference on Machine Translation, ...

2023 doi
[62]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Anthropic . 2024 a . The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic.com/news/claude-3-family. Accessed: 2025-09-01

work page 2024

[4] [4]

Anthropic . 2024 b . Introducing claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet. Accessed: 2025-09-01

work page 2024

[5] [5]

Anthropic . 2024 c . Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet. https://assets.anthropic.com/m/1cd9d098ac3e6467/original/Claude-3-Model-Card-October-Addendum.pdf. Accessed: 2025-09-01

work page 2024

[6] [6]

Jacob Blakesley, editor. 2018. Sociologies of Poetry Translation: Emerging Perspectives. Bloomsbury Academic, London

work page 2018

[7] [7]

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

arXiv 2023

[8] [8]

Andong Chen, Lianzhang Lou, Kehai Chen, Xuefeng Bai, Yang Xiang, Muyun Yang, Tiejun Zhao, and Min Zhang. 2024. http://arxiv.org/abs/2408.09945 Benchmarking llms for translating classical chinese poetry:evaluating adequacy, fluency, and elegance

work page Pith review arXiv 2024