REVIEW 3 major objections 4 minor 10 references
SAO WMT19 Test Suite: Machine Translation of Audit Reports
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Current machine translation systems, even the best in the 2019 news shared task, fail to preserve the identity of contracting parties in a sublease agreement; single-reference automatic metrics are practically useless for detecting this.
desk verdict Useful test suite, honest manual eval, and a party-identity finding that is probably real but needs an explicit source-side check. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing instrument is a manual, markable-based annotation protocol applied to one sublease agreement. The paper fixes a list of markables—named entities, dates, numbers, and document-specific legal terms such as 'tenant', 'lessee', 'supplement', 'equipment', and 'amenities'—and for each machine translation decides whether each occurrence is correct, wrong, missing, or clashing with another term. The decisive check is the term clash: 'tenant' and 'lessee' must map to two distinct Czech legal words ('nájemce' versus 'podnájemce') whenever both roles appear, otherwise a reader cannot tell which contracting party is which. This protocol sits on top of a trilingual parallel test suite of audit reports that the paper cleaned and released for reuse.
What would settle it
Translate a set of originally-English sublease agreements into Czech with these systems and check every mention of 'tenant' and 'lessee': if any system consistently uses two distinct terms (for example, 'nájemce' and 'podnájemce') for the two roles, the claim that even the best systems completely fail at party identity is refuted.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that the ceiling for current MT is not where sentence-level fluency suggests. In the sublease document, every one of the evaluated systems used a single Czech word, 'nájemce', for both 'tenant' and 'lessee', even though Czech has distinct terms ('nájemce' versus 'podnájemce') for the two roles in a sublease. Because a contract's meaning depends on knowing which party is which, all translations became effectively incomprehensible at the one point that matters most. The paper further shows that automatic scores, all computed against one reference translation, cannot flag this: the reference itself sometimes uses acceptable variants, and the clash is a consistency and factual error rather than an n-gram difference. Manual evaluation by domain professionals was required even to see the failure.
Load-bearing premise
The universal-failure conclusion rests on one sublease document and a small set of domain-expert annotations, with no formal measure of annotator agreement and a single annotator for the German directions.
Editorial extensions
If this is right
- High automatic scores on domain text should not be read as evidence that a system understands the domain; the audit-report translations scored near the top while hiding serious terminology errors.
- A contract-style document with two roles that must remain distinct is a cheap, high-signal probe for evaluating MT, since party identity is a semantic property that sentence-level metrics miss.
- Terminology consistency across a document is not enforced by current systems; translating the same source term differently, or two source terms identically, can invert the meaning of a legal text.
- For practical deployment in legal or audit settings, machine translation needs expert-in-the-loop correction or terminology lists, because neither automatic metrics nor unassisted MT output can guarantee role identity.
Reading between the lines
- The same party-identity failure should be expected in any target language that has distinct terms for landlord-tenant versus tenant-subtenant relationships; building a German version of the sublease test would be a direct check.
- A quantitative extension would be to define an 'entity-identity preservation' score from the markable annotations—the percentage of role mentions translated to the correct distinct term—and test whether existing reference-free semantic metrics correlate with it.
- Because the English source was itself translated from Czech, the reference translation carried a few errors; a cleaner test using an originally-English agreement with a vetted legal translation might make the systems' failure even sharper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents the SAO WMT19 test suite, a publicly released set of multilingual audit-report documents used as a WMT19 test suite for Czech, English, and German translation, and reports automatic and manual evaluations of the participating MT systems. The automatic scores show reasonable domain-level performance, while a manual evaluation with domain experts indicates that subtle terminological errors are missed by reference-based automatic metrics. In a separate analysis of one sublease agreement, the authors report that all tested systems translate both "tenant" and "lessee" as Czech "nájemce," collapsing the distinction between the contracting parties, and conclude that even the best systems completely fail to preserve the semantics of the agreement regarding party identity.
Significance. The paper's main contribution is a useful public test suite for a non-news domain, together with a detailed manual annotation protocol and a rare attempt to evaluate document-level semantic consistency in MT. The authors are commendably explicit about limitations: they report small sample sizes, standard deviations on automatic scores, the absence of reliable inter-annotator agreement, and the use of a single annotator for some language pairs. If the party-identity claim can be substantiated with a proper source-side validation, the work provides a concrete, falsifiable case that strong sentence-level NMT systems can destroy legally crucial role distinctions while scoring well on generic metrics. The public release of the test suite and annotation materials is a valuable resource for the community.
major comments (3)
- [Section 5, Table 16] The central empirical claim that all systems conflate the two contracting parties by translating both "tenant" and "lessee" as "nájemce" presupposes that the English source consistently uses "tenant" for one party and "lessee" for the other. Section 5 states that the English source "was in fact a (non-professional) translation from Czech," but the paper never checks the Czech original for how the parties are referred to and whether the English terms align with distinct Czech referents. If the non-professional English translation uses the two terms interchangeably, then an MT output that maps both to "nájemce" may be faithful to the source, and the observed clash would be inherited rather than introduced by MT. Please add a term concordance of the Czech original, the English source, and the occurrences counted in Table 16, and verify that every "clash" corresponds to a genuine role distinction in the source. If the check does not support the premise, the abstract and Section 7 claims must be weakened to a claim about preserving source ambiguity rather than party identity.
- [Section 4.4 and Section 5.4] The manual evaluation underlying the paper's stronger conclusions has very limited reliability evidence, which the authors acknowledge in Section 4.1 ("we did not collect enough annotations to reliably measure it"). Only three segments were double-annotated, with known disagreements, and all English-to-German and German-to-English evaluations were performed by a single annotator. For the sublease agreement, the annotation was described as "partially blind" and performed by the "main annotator" without a second annotation. This does not invalidate the findings, but it does mean that the abstract statement that automatic MT evaluation with one reference is "practically useless" is stronger than what the reliability evidence supports. I recommend reporting per-criterion agreement on the double-annotated segments, providing a second annotation of the party-reference marking in the sublease agreement, or explicitly restricting the automatic-evaluation claim to this test suite and to the specific error types studied rather than presenting it as a general conclusion.
- [Section 4.2 and Section 7] The paper argues that automatic evaluation with one reference is "practically useless" for detecting the fine-grained semantic errors it describes, but it never reports automatic scores for the sublease agreement, which is the document where the most severe error occurs. Without a direct comparison of automatic metric scores on that document against the manual error counts, the reader cannot assess whether the metrics fail specifically where the semantic distinction matters. I suggest either including the relevant automatic scores for the sublease agreement in Section 5, or adding a numerical comparison (e.g., correlation between automatic scores and expert judgments across the audit documents) to substantiate the general claim; otherwise the wording should be softened to "automatic metrics did not flag these errors in our evaluation."
minor comments (4)
- [Section 4.4.1] The sentence referring to "the main Findings of WMT19 paper" has no corresponding reference in the bibliography; please add the WMT19 Findings paper (Barrault et al., 2019).
- [Section 5.1] The sentence "The original Czech text was evaluated with all other WMT19 systems as if it was one of the systems" is unclear, since the original Czech text cannot be an output of an English-to-Czech system; please clarify whether it was used as a reference, a control condition, or something else.
- [Table 15] The header "Apartement in Question" contains a spelling error; it should be "Apartment in Question."
- [Section 5.2] There is a typo in "we demanded the this particular choice"; it should read "we demanded this particular choice."
Circularity Check
No circular derivation: the tenant/lessee claim is an empirical evaluation result; the paper's caveats concern annotation reliability and source-translation validity, not input-output circularity.
full rationale
The paper makes no formal derivation or fitted-parameter claim. Its central empirical observation — that all tested systems translate both 'tenant' and 'lessee' as 'nájemce' (Table 16, Section 5.4.2) — is an external measurement on released system outputs and a released test suite, not a quantity constructed from its own inputs. The evaluation criteria (Sections 4.1 and 5.2) are defined independently of the systems' outputs, and the markables are listed before reporting counts. No equation or fitted value is reused as a prediction. The self-citations (Bojar et al. 2018, Popel et al. 2019, Maršík and Bojar 2012) provide contextual background or describe the evaluated systems; none is the load-bearing justification for the tenant/lessee finding, which is directly observable in Table 16. The paper itself flags its limitations: inter-annotator agreement was not reliably measured (Section 4.1), English–German was evaluated by a single annotator (Section 4.4.2), and the sublease source is a non-professional translation from Czech (Section 5). These are validity or reliability concerns about the benchmark; they are not instances of a derivation reducing to its own input, so they do not raise the circularity score. The strongest claim is thus an empirical result with caveats, not a circular one.
Assumptions & free parameters
assumptions (3)
- domain assumption Manual evaluation by SAO employees is a valid measure of translation quality.
- domain assumption The sublease agreement sample is representative enough to generalize the 'all systems fail' conclusion.
- domain assumption The WMT19 systems evaluated are representative of current MT systems.
Cite this review
Pith. "Pith review of SAO WMT19 Test Suite: Machine Translation of Audit Reports." pith.science (2026). https://pith.science/paper/EKGM74E2
@misc{pith2026190901701,
author = {Pith},
title = {Pith review of: SAO WMT19 Test Suite: Machine Translation of Audit Reports},
year = {2026},
howpublished = {\url{https://pith.science/paper/EKGM74E2}},
note = {Machine review of arXiv:1909.01701}
}
read the original abstract
This paper describes a machine translation test set of documents from the auditing domain and its use as one of the "test suites" in the WMT19 News Translation Task for translation directions involving Czech, English and German. Our evaluation suggests that current MT systems optimized for the general news domain can perform quite well even in the particular domain of audit reports. The detailed manual evaluation however indicates that deep factual knowledge of the domain is necessary. For the naked eye of a non-expert, translations by many systems seem almost perfect and automatic MT evaluation with one reference is practically useless for considering these details. Furthermore, we show on a sample document from the domain of agreements that even the best systems completely fail in preserving the semantics of the agreement, namely the identity of the parties.
Figures
Reference graph
Works this paper leans on
-
[1]
Ond r ej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation ( WMT18 ) . In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Computational Linguistics
work page 2018
-
[2]
Gregor Leusch and Hermann Ney. 2008. BLEUSP, INVWER, CDER: Three improved MT evaluation measures . In NIST Metrics for Machine Translation Challenge, Waikiki, Honolulu, Hawaii
work page 2008
-
[3]
Ji r \' Mar s \' k and Ond r ej Bojar. 2012. TrTok: A Fast and Trainable Tokenizer for Natural Languages . Prague Bulletin of Mathematical Linguistics, 98:75--85
work page 2012
-
[4]
Arun Narayanan, Ananya Misra, Khe Chai Sim, Golan Pundak, Anshuman Tripathi, Mohamed Elfeky, Parisa Haghani, Trevor Strohman, and Michiel Bacchiani. 2018. https://doi.org/10.1109/SLT.2018.8639610 Toward domain-invariant speech recognition via large scale training . In 2018 IEEE Spoken Language Technology Workshop, SLT 2018, Athens, Greece, December 18-21,...
arXiv 2018
-
[5]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation . In Proceedings of ACL 2002 , pages 311--318, Philadelphia, Pennsylvania
work page 2002
-
[6]
Martin Popel, Dominik Mach\' a c ek, Michal Auersperger, Ond r ej Bojar, and Pavel Pecina. 2019. English-czech systems in wmt19: Document-level transformer. In Proceedings of the Fourth Conference on Machine Translation: Volume 2, Shared Task Papers
work page 2019
-
[7]
Maja Popovi\' c . 2015. http://aclweb.org/anthology/W15-3049 chrF: character n-gram F-score for automatic MT evaluation . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392--395, Lisbon, Portugal. ACL
work page 2015
-
[8]
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation . In Proceedings AMTA, pages 223--231
work page 2006
Show all 10 references
-
[9]
D\' a niel Varga, L\' a szl\' o N\' e meth, P\' e ter Hal\' a csy, Andr\' a s Kornai, Viktor Tr\' o n, and Viktor Nagy. 2005. Parallel corpora for medium density languages . In Proceedings of the Recent Advances in Natural Language Processing RANLP 2005, pages 590--596, Borove...
2005
-
[10]
Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. 2016. CharacTER : Translation edit rate on character level. In ACL 2016 First Conference on Machine Translation, Berlin, Germany
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.