{"paper":{"title":"What Does LLM Refinement Actually Improve? A Systematic Study on Document-Level Literary Translation","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Document-level translation followed by segment-level refinement produces the most reliable gains in literary machine translation.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bill Byrne, Dawei Zhu, Felix Hieber, Ke Tran, Leonardo Ribeiro, Michael Denkowski, Shaomu Tan, Sony Trenous","submitted_at":"2026-05-13T11:27:32Z","abstract_excerpt":"Iterative self-refinement is a simple inference-time strategy for machine translation: an LLM revises its own translation over multiple inference-time passes. Yet document-scale refinement remains poorly understood: 1) which pipelines work best, 2) what quality dimensions improve, and 3) how refiners behave. In this paper, we present a systematic study of document-level literary translation, covering nine LLMs and seven language pairs. Across nine translation-refinement granularity combinations and five refinement strategies, we find a robust recipe: document-level MT followed by segment-level"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Across nine translation-refinement granularity combinations and five refinement strategies, we find a robust recipe: document-level MT followed by segment-level refinement yields strong and stable improvements. In contrast, document-level refinement often makes fewer edits and leads to smaller or less reliable gains.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that the observed patterns in refinement behavior and quality dimensions will generalize beyond the specific nine LLMs, seven language pairs, and literary texts tested in the study.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Document-level machine translation followed by segment-level LLM refinement provides the strongest and most stable improvements in literary translation quality, mainly enhancing fluency and style rather than adequacy.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Document-level translation followed by segment-level refinement produces the most reliable gains in literary machine translation.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"372853385b851e2261c878e020a453774d824a5efe24c83e2871acfc9be8a274"},"source":{"id":"2605.13368","kind":"arxiv","version":1},"verdict":{"id":"3fcbbbbb-1b0a-4772-a3aa-0a65cc3e7f7e","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T20:05:46.245306Z","strongest_claim":"Across nine translation-refinement granularity combinations and five refinement strategies, we find a robust recipe: document-level MT followed by segment-level refinement yields strong and stable improvements. In contrast, document-level refinement often makes fewer edits and leads to smaller or less reliable gains.","one_line_summary":"Document-level machine translation followed by segment-level LLM refinement provides the strongest and most stable improvements in literary translation quality, mainly enhancing fluency and style rather than adequacy.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that the observed patterns in refinement behavior and quality dimensions will generalize beyond the specific nine LLMs, seven language pairs, and literary texts tested in the study.","pith_extraction_headline":"Document-level translation followed by segment-level refinement produces the most reliable gains in literary machine translation."},"references":{"count":39,"sample":[{"doi":"10.18653/v1/2025.emnlp-main.1413","year":2025,"title":"Ademuyiwa, Andrew Caines, and Dietrich Klakow","work_id":"f5cdfa84-ebe3-406f-a41a-51b469c15afa","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"Ramakrishna Appicharla, Baban Gain, Santanu Pal, and Asif Ekbal. 2025. https://arxiv.org/abs/2506.07583 Beyond the sentence: A survey on context-aware machine translation with large language models . ","work_id":"b432bd14-f865-4e99-8b6c-f0a9cb78167c","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Eleftheria Briakou, Jiaming Luo, Colin Cherry, and Markus Freitag. 2024. Translating step-by-step: Decomposing the translation process for improved translation quality of long-form texts. In Proceedin","work_id":"9841de3c-da80-4737-b80f-9d75294aaca0","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2024. Iterative translation refinement with large language models. In Proceedings of the 25th Annual Conference of the European Associat","work_id":"f47937df-e0b3-4ed4-ab92-890fb6573160","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","ref_index":5,"cited_arxiv_id":"2501.12948","is_internal_anchor":true}],"resolved_work":39,"snapshot_sha256":"e19530d1d8c21450168f9fe2697d2abc006f47585c3d8fa2aff09c7644ac9a2f","internal_anchors":5},"formal_canon":{"evidence_count":2,"snapshot_sha256":"95a5339eab7da04dda0e7afef2637fd00aee7710a56653efa461b334e242abd8"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}