Testing the Deliteralization Hypothesis in Human and Machine Translation

Beno\^it Sagot; Malik Marmonier; Rachel Bawden

arxiv: 2605.25686 · v1 · pith:MUDGFB2K · submitted 2026-05-25 · cs.CL

Testing the Deliteralization Hypothesis in Human and Machine Translation

Malik Marmonier , Rachel Bawden , Beno\^it Sagot This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-29 21:55 UTCgrok-4.3pith:MUDGFB2K record.json open to challenge →

classification cs.CL

keywords deliteralization hypothesismachine translationlarge language modelsliterality measurementpost-editingiterative revisionWMT24++

0 comments

The pith

LLMs deliteralize their translations monotonically when prompted to revise their own output iteratively.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests the deliteralization hypothesis, the claim that translations grow less literal across successive drafts and revisions, to see whether it holds for large language models in addition to human translators. It measures literality with a Synthetic Literality Index built from six heuristics and applies the measure to human translations, NMT systems, and six LLMs across 54 language pairs on the WMT24++ dataset in three settings: direct translation, iterative self-revision, and post-editing. The results show that human output stays less literal than all tested machine systems but recent LLMs reduce the difference, that LLMs reduce literality steadily with each self-revision round, and that LLMs used as post-editors revise in the opposite direction from human post-editors.

Core claim

When prompted to iteratively revise their own output, LLMs deliteralize monotonically, providing the first evidence that the deliteralization hypothesis applies natively to LLM generation. Human translations remain significantly less literal than those of all tested MT systems, though recent LLMs narrow the gap. As post-editors, LLMs invert the revision triggers of human post-editors, tolerating literal drafts and targeting idiomatic human formulations for revision.

What carries the argument

The Synthetic Literality Index, a composite score from six heuristics that quantifies how literal a translation is relative to its source.

If this is right

Human translations remain significantly less literal than NMT or LLM output across the tested language pairs.
LLMs narrow but do not close the literality gap with human translators in direct translation.
LLM output becomes steadily less literal with each round of prompted self-revision.
LLMs used for post-editing revise away from literal drafts and toward more idiomatic human text, reversing the pattern seen in human post-editors.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Iterative self-revision prompting may be a practical route to more fluent LLM translations without additional training.
The opposite revision behavior of LLMs as post-editors suggests their internal preference for idiomatic phrasing differs from the literalness bias observed in NMT systems.
Extending the same index and revision protocol to other generation tasks could test whether deliteralization is specific to translation or a broader property of LLM output refinement.

Load-bearing premise

The Synthetic Literality Index constructed from six heuristics provides a valid, unbiased measure of literality that can be compared fairly across human translations, NMT output, and LLM output.

What would settle it

Demonstrating that the Synthetic Literality Index fails to correlate with direct human judgments of literality on the same WMT24++ sentences would undermine the reported differences and trends.

Figures

Figures reproduced from arXiv: 2605.25686 by Beno\^it Sagot, Malik Marmonier, Rachel Bawden.

**Figure 3.** Figure 3: Heuristic hit rates broken down across the nine [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Share of segments for which the initial human [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

**Figure 5.** Figure 5: Pairwise comparison of the systems and the human translators for task 1, illustrating the mean difference [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

**Figure 6.** Figure 6: Pairwise comparison of the systems for task [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗

**Figure 7.** Figure 7: Pairwise comparison of the systems for task 3 [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

**Figure 8.** Figure 8: Annotation interface [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗

**Figure 9.** Figure 9: Annotation guidelines. To ground our automatic heuristics in human intuition, we conducted a manual annotation campaign on a subset of our actual dataset. We focused on the en-fr_FR (English-French of France) data partition, evaluating pairs of translation hypotheses (such as the initial human translation, its postedited counterpart and MT candidates). Segments with a MetricX-24 score higher than 7.0 we… view at source ↗

**Figure 10.** Figure 10: Pairwise bootstrap mean SLI difference — Task 1 single translations (row [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗

**Figure 11.** Figure 11: Pairwise bootstrap mean SLI difference — Task 3 post-editions (row [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗

read the original abstract

The recent shift from dedicated NMT systems to general-purpose LLMs has reshaped machine translation, with LLMs reported to produce more fluent, less literal output than their predecessors. We test whether this shift extends to the deliteralization hypothesis, the long-standing claim from translation studies that translations become progressively less literal as they are drafted and revised. Using the WMT24++ dataset, we compare the literality of human translations and post-editions to that of two NMT systems and six LLMs across 54 language pairs and three tasks: direct translation, iterative self-revision, and post-editing of human drafts. Literality is measured via a validated Synthetic Literality Index built from six heuristics. We find that (i) human translations remain significantly less literal than those of all tested MT systems, though recent LLMs narrow the gap; (ii) when prompted to iteratively revise their own output, LLMs deliteralize monotonically, providing the first evidence that the hypothesis applies natively to LLM generation; and (iii) as post-editors, LLMs invert the revision triggers of human post-editors, tolerating literal drafts and targeting idiomatic human formulations for revision.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

LLMs show monotonic deliteralization on self-revision and invert human post-editing patterns, but the Synthetic Literality Index needs verification before the cross-system claims land.

read the letter

The paper's main new finding is that LLMs, when prompted for iterative self-revision, reduce literality step by step according to their index, which the authors flag as the first direct evidence the deliteralization hypothesis applies to LLM generation. They also report that LLMs as post-editors target idiomatic human text for change while tolerating literal drafts, the opposite of human behavior.

The work does a reasonable job scaling the test to 54 language pairs from WMT24++ and running the same comparisons on both NMT systems and multiple LLMs across direct translation, self-revision, and post-editing tasks. That breadth lets them place recent LLMs closer to human output than older NMT while still behind.

The soft spot is the Synthetic Literality Index itself. It is built from six heuristics and called validated, yet the abstract gives no detail on whether those heuristics were tested or tuned primarily on human data. If they encode assumptions that do not transfer cleanly to LLM output distributions, the monotonic trend and the inversion result could be metric artifacts rather than evidence about native LLM behavior. The large number of pairs also raises the usual multiple-comparison questions that need explicit handling.

This is for MT researchers who track fluency and literality claims from translation studies. A reader who wants empirical numbers on how LLMs revise would get concrete comparisons to think about.

I would send it for peer review. The question is straightforward, the dataset is external, and the scale is there; the metric validation is the part that needs referee attention rather than a reason to desk-reject.

Referee Report

3 major / 2 minor

Summary. The paper tests the deliteralization hypothesis (translations become less literal with drafting and revision) on the WMT24++ dataset across 54 language pairs. It compares literality of human translations/post-editions vs. two NMT systems and six LLMs using three tasks (direct translation, iterative self-revision, post-editing), with literality quantified by a Synthetic Literality Index from six heuristics. Key findings: (i) humans produce less literal output than MT systems (LLMs narrow the gap); (ii) LLMs deliteralize monotonically under iterative self-revision; (iii) LLMs as post-editors invert human revision patterns.

Significance. If the Synthetic Literality Index is shown to be valid and unbiased for cross-system comparison, the work would provide the first direct evidence that the deliteralization hypothesis applies natively to LLM generation, bridging translation studies and machine translation research. Strengths include the use of an external dataset (WMT24++), systematic comparison across many language pairs and systems, and three distinct tasks; the monotonicity result under self-revision is a falsifiable, testable claim.

major comments (3)

[§3] §3 (Synthetic Literality Index construction): the six heuristics must be shown to have been validated or cross-validated on LLM-generated text rather than solely on human data; without this, the index may embed human-specific assumptions about literality, making the monotonic deliteralization trend in finding (ii) potentially an artifact of the metric rather than evidence for native applicability to LLMs.
[Results] Results section (statistical analysis across 54 pairs): the reported significance tests for the three findings do not appear to include correction for multiple comparisons; with 54 language pairs and multiple tasks/systems, uncorrected p-values undermine the reliability of the cross-system and monotonicity claims.
[§4.2] §4.2 (iterative self-revision experiments): the monotonic deliteralization result for LLMs depends on the post-hoc choice of revision prompts and number of iterations; the manuscript should report sensitivity checks or pre-registered analysis plans to rule out that the observed trend is driven by these choices.

minor comments (2)

[Table 1] Table 1 or equivalent: clarify the exact composition of the six heuristics and their individual contributions to the index score.
[Abstract] Abstract and §2: the phrase 'validated Synthetic Literality Index' should be accompanied by a brief citation or pointer to the validation procedure even in the abstract.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their detailed and constructive feedback. We address each major comment below and commit to revisions that directly respond to the concerns raised.

read point-by-point responses

Referee: [§3] §3 (Synthetic Literality Index construction): the six heuristics must be shown to have been validated or cross-validated on LLM-generated text rather than solely on human data; without this, the index may embed human-specific assumptions about literality, making the monotonic deliteralization trend in finding (ii) potentially an artifact of the metric rather than evidence for native applicability to LLMs.

Authors: We appreciate this observation. The Synthetic Literality Index draws on heuristics validated in prior translation studies on human data. To directly address applicability to LLM output and rule out metric artifacts, the revised manuscript will include a new cross-validation subsection that applies the index to a held-out sample of LLM translations and reports agreement with human judgments on those outputs. revision: yes
Referee: [Results] Results section (statistical analysis across 54 pairs): the reported significance tests for the three findings do not appear to include correction for multiple comparisons; with 54 language pairs and multiple tasks/systems, uncorrected p-values undermine the reliability of the cross-system and monotonicity claims.

Authors: We agree that uncorrected tests across 54 language pairs require adjustment. The revision will recompute all reported significance tests with Benjamini-Hochberg FDR correction and will present both original and adjusted p-values, updating the results, tables, and discussion to reflect the more conservative thresholds. revision: yes
Referee: [§4.2] §4.2 (iterative self-revision experiments): the monotonic deliteralization result for LLMs depends on the post-hoc choice of revision prompts and number of iterations; the manuscript should report sensitivity checks or pre-registered analysis plans to rule out that the observed trend is driven by these choices.

Authors: The prompts and iteration schedule were selected after preliminary runs. While the study was not pre-registered, the revised version will add an explicit sensitivity analysis that re-runs the self-revision task with two alternative prompt templates and with iteration counts ranging from 1 to 5, confirming that the monotonic trend persists under these variations. revision: yes

Circularity Check

0 steps flagged

No significant circularity: empirical comparison on external dataset with fixed index

full rationale

The paper conducts an empirical comparison of literality across human translations, NMT outputs, and LLM generations on the external WMT24++ dataset using a pre-validated Synthetic Literality Index built from six fixed heuristics. No equations, predictions, or first-principles results reduce by construction to quantities fitted from the target data; the reported monotonic deliteralization under LLM self-revision is a direct measurement on held-out iterative outputs rather than a self-referential fit. Central claims rely on external benchmarks and a static metric, with no load-bearing self-citations or ansatzes that collapse the derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract only; no explicit free parameters, axioms, or invented entities are described.

pith-pipeline@v0.9.1-grok · 5740 in / 1135 out tokens · 26355 ms · 2026-06-29T21:55:13.013344+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages · 1 internal anchor

[1]

InProceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 72–81, Online

Statistical power and translationese in machine translation evaluation. InProceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 72–81, Online. Association for Computational Linguistics. Vladimir Ivir. 1981. Formal correspondence vs. trans- lation equivalence revisited.Poetics Today, 2(4):51– 59. Masoud Jal...

2020
[2]

No Language Left Behind: Scaling Human-Centered Machine Translation

Automatic detection of translated text and its impact on machine translation. InProceedings of Machine Translation Summit XII: Papers, Ottawa, Canada. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi- cient memory management for large language model serving with paged...

work page internal anchor Pith review Pith/arXiv arXiv 2023
[3]

quality” and “improvement,

Volume II : Articles courts, pages 205–214, Toulouse, France. ATALA. Yuming Zhai, Gabriel Illouz, and Anne Vilnat. 2020. Detecting non-literal translations by fine-tuning cross-lingual pre-trained language models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5944–5956, Barcelona, Spain (Online). International Com...

work page arXiv 2020

[1] [1]

InProceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 72–81, Online

Statistical power and translationese in machine translation evaluation. InProceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 72–81, Online. Association for Computational Linguistics. Vladimir Ivir. 1981. Formal correspondence vs. trans- lation equivalence revisited.Poetics Today, 2(4):51– 59. Masoud Jal...

2020

[2] [2]

No Language Left Behind: Scaling Human-Centered Machine Translation

Automatic detection of translated text and its impact on machine translation. InProceedings of Machine Translation Summit XII: Papers, Ottawa, Canada. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi- cient memory management for large language model serving with paged...

work page internal anchor Pith review Pith/arXiv arXiv 2023

[3] [3]

quality” and “improvement,

Volume II : Articles courts, pages 205–214, Toulouse, France. ATALA. Yuming Zhai, Gabriel Illouz, and Anne Vilnat. 2020. Detecting non-literal translations by fine-tuning cross-lingual pre-trained language models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5944–5956, Barcelona, Spain (Online). International Com...

work page arXiv 2020