{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2026:NXG3YUB63DWKQRSY6GSLMD7BBZ","short_pith_number":"pith:NXG3YUB6","schema_version":"1.0","canonical_sha256":"6dcdbc503ed8eca84658f1a4b60fe10e4385a11635867779f50e2eb315dd9d0f","source":{"kind":"arxiv","id":"2605.13368","version":1},"attestation_state":"computed","paper":{"title":"What Does LLM Refinement Actually Improve? A Systematic Study on Document-Level Literary Translation","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Document-level translation followed by segment-level refinement produces the most reliable gains in literary machine translation.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bill Byrne, Dawei Zhu, Felix Hieber, Ke Tran, Leonardo Ribeiro, Michael Denkowski, Shaomu Tan, Sony Trenous","submitted_at":"2026-05-13T11:27:32Z","abstract_excerpt":"Iterative self-refinement is a simple inference-time strategy for machine translation: an LLM revises its own translation over multiple inference-time passes. Yet document-scale refinement remains poorly understood: 1) which pipelines work best, 2) what quality dimensions improve, and 3) how refiners behave. In this paper, we present a systematic study of document-level literary translation, covering nine LLMs and seven language pairs. Across nine translation-refinement granularity combinations and five refinement strategies, we find a robust recipe: document-level MT followed by segment-level"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2605.13368","kind":"arxiv","version":1},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CL","submitted_at":"2026-05-13T11:27:32Z","cross_cats_sorted":[],"title_canon_sha256":"7fba05e92e83f58cdca23df1e608365ebbd229b69ac5c092d3a21d155f305bb5","abstract_canon_sha256":"e73be54613a7714e9038f97eb3d2f8c92d91c21ebfcb89d540e737cc39d30ba1"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-18T02:44:48.028958Z","signature_b64":"PV0aICT/DB7aGDehjl+XujfO9KvW4jxPC18z50IYsU6XscGtBetUK5xQWtLTZpyT5JzYay9I23ZB04llNnU+Cw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"6dcdbc503ed8eca84658f1a4b60fe10e4385a11635867779f50e2eb315dd9d0f","last_reissued_at":"2026-05-18T02:44:48.028504Z","signature_status":"signed_v1","first_computed_at":"2026-05-18T02:44:48.028504Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"What Does LLM Refinement Actually Improve? A Systematic Study on Document-Level Literary Translation","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Document-level translation followed by segment-level refinement produces the most reliable gains in literary machine translation.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bill Byrne, Dawei Zhu, Felix Hieber, Ke Tran, Leonardo Ribeiro, Michael Denkowski, Shaomu Tan, Sony Trenous","submitted_at":"2026-05-13T11:27:32Z","abstract_excerpt":"Iterative self-refinement is a simple inference-time strategy for machine translation: an LLM revises its own translation over multiple inference-time passes. Yet document-scale refinement remains poorly understood: 1) which pipelines work best, 2) what quality dimensions improve, and 3) how refiners behave. In this paper, we present a systematic study of document-level literary translation, covering nine LLMs and seven language pairs. Across nine translation-refinement granularity combinations and five refinement strategies, we find a robust recipe: document-level MT followed by segment-level"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Across nine translation-refinement granularity combinations and five refinement strategies, we find a robust recipe: document-level MT followed by segment-level refinement yields strong and stable improvements. In contrast, document-level refinement often makes fewer edits and leads to smaller or less reliable gains.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that the observed patterns in refinement behavior and quality dimensions will generalize beyond the specific nine LLMs, seven language pairs, and literary texts tested in the study.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Document-level machine translation followed by segment-level LLM refinement provides the strongest and most stable improvements in literary translation quality, mainly enhancing fluency and style rather than adequacy.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Document-level translation followed by segment-level refinement produces the most reliable gains in literary machine translation.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"372853385b851e2261c878e020a453774d824a5efe24c83e2871acfc9be8a274"},"source":{"id":"2605.13368","kind":"arxiv","version":1},"verdict":{"id":"3fcbbbbb-1b0a-4772-a3aa-0a65cc3e7f7e","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T20:05:46.245306Z","strongest_claim":"Across nine translation-refinement granularity combinations and five refinement strategies, we find a robust recipe: document-level MT followed by segment-level refinement yields strong and stable improvements. In contrast, document-level refinement often makes fewer edits and leads to smaller or less reliable gains.","one_line_summary":"Document-level machine translation followed by segment-level LLM refinement provides the strongest and most stable improvements in literary translation quality, mainly enhancing fluency and style rather than adequacy.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that the observed patterns in refinement behavior and quality dimensions will generalize beyond the specific nine LLMs, seven language pairs, and literary texts tested in the study.","pith_extraction_headline":"Document-level translation followed by segment-level refinement produces the most reliable gains in literary machine translation."},"references":{"count":39,"sample":[{"doi":"10.18653/v1/2025.emnlp-main.1413","year":2025,"title":"Ademuyiwa, Andrew Caines, and Dietrich Klakow","work_id":"f5cdfa84-ebe3-406f-a41a-51b469c15afa","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"Ramakrishna Appicharla, Baban Gain, Santanu Pal, and Asif Ekbal. 2025. https://arxiv.org/abs/2506.07583 Beyond the sentence: A survey on context-aware machine translation with large language models . ","work_id":"b432bd14-f865-4e99-8b6c-f0a9cb78167c","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Eleftheria Briakou, Jiaming Luo, Colin Cherry, and Markus Freitag. 2024. Translating step-by-step: Decomposing the translation process for improved translation quality of long-form texts. In Proceedin","work_id":"9841de3c-da80-4737-b80f-9d75294aaca0","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2024. Iterative translation refinement with large language models. In Proceedings of the 25th Annual Conference of the European Associat","work_id":"f47937df-e0b3-4ed4-ab92-890fb6573160","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","ref_index":5,"cited_arxiv_id":"2501.12948","is_internal_anchor":true}],"resolved_work":39,"snapshot_sha256":"e19530d1d8c21450168f9fe2697d2abc006f47585c3d8fa2aff09c7644ac9a2f","internal_anchors":5},"formal_canon":{"evidence_count":2,"snapshot_sha256":"95a5339eab7da04dda0e7afef2637fd00aee7710a56653efa461b334e242abd8"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2605.13368","created_at":"2026-05-18T02:44:48.028585+00:00"},{"alias_kind":"arxiv_version","alias_value":"2605.13368v1","created_at":"2026-05-18T02:44:48.028585+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2605.13368","created_at":"2026-05-18T02:44:48.028585+00:00"},{"alias_kind":"pith_short_12","alias_value":"NXG3YUB63DWK","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"NXG3YUB63DWKQRSY","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"NXG3YUB6","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":0,"internal_anchor_count":0,"sample":[]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ","json":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ.json","graph_json":"https://pith.science/api/pith-number/NXG3YUB63DWKQRSY6GSLMD7BBZ/graph.json","events_json":"https://pith.science/api/pith-number/NXG3YUB63DWKQRSY6GSLMD7BBZ/events.json","paper":"https://pith.science/paper/NXG3YUB6"},"agent_actions":{"view_html":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ","download_json":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ.json","view_paper":"https://pith.science/paper/NXG3YUB6","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2605.13368&json=true","fetch_graph":"https://pith.science/api/pith-number/NXG3YUB63DWKQRSY6GSLMD7BBZ/graph.json","fetch_events":"https://pith.science/api/pith-number/NXG3YUB63DWKQRSY6GSLMD7BBZ/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ/action/timestamp_anchor","attest_storage":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ/action/storage_attestation","attest_author":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ/action/author_attestation","sign_citation":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ/action/citation_signature","submit_replication":"https://pith.science/pith/NXG3YUB63DWKQRSY6GSLMD7BBZ/action/replication_record"}},"created_at":"2026-05-18T02:44:48.028585+00:00","updated_at":"2026-05-18T02:44:48.028585+00:00"}