A Test Suite and Manual Evaluation of Document-Level NMT at WMT19

Kate\v{r}ina Rysov\'a; Lucie Pol\'akov\'a; Magdal\'ena Rysov\'a; Ond\v{r}ej Bojar; Tom\'a\v{s} Musil

REVIEW 3 major objections 5 minor 18 references

A Test Suite and Manual Evaluation of Document-Level NMT at WMT19

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Giving neural machine translation systems multiple sentences of input did not yield measurable gains on three discourse-level phenomena in English-to-Czech translation at the 2019 WMT shared task.

desk verdict The test suite is a useful resource, but the headline no-difference claim rests on a word-order metric that flags the human reference too often to carry it. read the letter →

arxiv 1908.03043 v1 pith:S6SN6DXN submitted 2019-08-08 cs.CL

Kateřina Rysová , Magdaléna Rysová , Tomáš Musil , Lucie Poláková , Ondřej Bojar This is my paper · ORCID

classification cs.CL

keywords document-levelmachinetranslationtestsuitemanualevaluationdiscourseconnectivesAltLextopic-focusarticulationwordorderEnglishtoCzech

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents a manually annotated test suite for evaluating document-level translation quality of English-to-Czech neural machine translation (NMT) systems in the 2019 news translation shared task, focusing on three coherence-related phenomena: topic-focus articulation (subject word order), discourse connectives, and alternative lexicalizations of connectives (AltLexes). Evaluating five systems, including two context-aware models that received multiple sentences of input during decoding, the authors found that the context-aware systems did not outperform the sentence-by-sentence systems on these phenomena. Errors were most frequent in word order related to information structure, and the human reference translation was sometimes scored as worse than the automatic outputs, illustrating the difficulty of literal reference comparison. The suite is offered as a reusable resource for future document-level evaluation.

What carries the argument

The evaluation rests on a test suite of 101 documents from a parallel English-Czech treebank, with discourse-relation annotations taken from a discourse treebank. For connectives and AltLexes, trained annotators used a questionnaire to mark each occurrence as adequate and correctly placed, adequate but misplaced, harmlessly omitted, harmful omitted, or inadequate. For word order, the machinery is a proxy: English sentences whose subject is a noun with an indefinite article are automatically preselected, on the theory that such a subject is contextually non-bound and should therefore be placed after the predicate (as focus proper) in Czech. The annotators then judged whether the Czech translation preserved the non-bound status and placed it appropriately. This proxy carries the argument for information-structure sensitivity.

What would settle it

Re-annotate the same outputs with direct human judgments of contextual boundness and preferred word order for each subject, instead of the indefinite-article proxy; if the two measurements disagree substantially, the paper's word-order comparison across systems does not reflect actual information-structure handling.

Watch

Extended reading notes

Core claim

The central discovery is that, contrary to the authors' expectation, feeding a Transformer-based NMT system several sentences of context does not measurably improve its handling of followed document-level phenomena in English-to-Czech translation. All five evaluated systems translated AltLexes and discourse connectives at an adequate level in the vast majority of cases, with only minor differences among systems. The clearest shortcomings appeared in word order: systems tended to preserve the English order and did not adapt to Czech information structure, so contextually non-bound subjects were often not placed after the predicate as Czech requires. Because the context-aware systems did not outperform the others even here, the authors conclude that these systems already perform 'good enough' on the task, and that the remaining errors are idiosyncratic and difficult to predict.

Load-bearing premise

The word-order results rest on the assumption that an English subject with an indefinite article is always contextually non-bound and should therefore follow the predicate in Czech; this proxy was validated on only a subset of cases and its underlying theory was not independently tested.

Editorial extensions

If this is right

Context-aware decoding as currently implemented does not automatically improve discourse-level adequacy for English-to-Czech; AltLexes and connectives are already translated at near-reference quality by all observed systems.
The main differentiator of quality lies in word order and information-structure handling, where all systems—including context-aware ones—show comparable error rates.
Bilingual manual evaluation can rate a human reference translation below automatic output, because a looser, more natural translation can be judged as missing or misplacing the traced expressions.
Because errors appear individually and non-systematically, simple deterministic post-editing rules are unlikely to eliminate them.
The released test suite with discourse annotations gives future shared tasks a reusable instrument for measuring document-level phenomena.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the result generalizes, document-level NMT for English-to-Czech should shift from feeding adjacent sentences to modeling explicit discourse structure or longer-range coreference, since a few sentences of context gave no benefit.
The indefinite-article proxy could be directly validated against independently annotated topic-focus status in a larger corpus; such a study would establish whether the word-order comparison is an artifact of the proxy.
The absence of a context benefit may be specific to this language pair and system generation; applying the same suite to a language pair with freer word order could reveal larger context effects.
A scalable automatic metric for discourse-connective adequacy, tested against these manual annotations, would allow document-level evaluation to be automated, though the present study only provides the manual gold standard.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. The paper builds a usable English–Czech test suite for document-level MT evaluation and runs a careful manual evaluation of five WMT19 systems plus the PCEDT reference. That part is worth engaging. But the central negative finding—context-aware systems did not outperform sentence-level systems—rests on a word-order scoring rule that the paper itself applies inconsistently to the human reference, plus very small counts. As it stands, the claim is not established.

The concrete contribution is the reuse of 101 PCEDT documents with PDTB 3.0 discourse annotations for the WMT19 shared task, and the manual evaluation of AltLexes, connectives, and topic-focus placement. The annotation procedure is described in enough detail to replicate, the interface is shown, and pairwise inter-annotator agreement is reported (80% average). The authors are also candid about limitations. That is real work and a useful resource for English–Czech MT evaluation.

Soft spots, in order. Most important is the word-order metric in Section 6.1.1. The rule says a contextually non-bound English subject introduced by an indefinite article should appear post-verbally, ideally as focus proper, in Czech. The stress-test note is right: in the focus-proper task the human reference is marked “no” in 6 of 7 cases, and in placement it is marked “no” in 3 of 22, while Section 8 says this does not mean the reference is incorrect. A scoring rule that so often flags the human reference is measuring a preferred information-structure ordering, not translation adequacy. This matters because word order is the component where the context-aware systems look worst, so the no-gain conclusion is not safe. Second, the counts are tiny—often 0 to 5 per cell—and no significance testing is done, so “only minor differences” is more a description of the annotation than a measured finding. Third, the star tables in this version render only partially, and no direct link to the test suite is given, which weakens reuse.

Citation patterns are fine. Self-citations to the authors’ own Czech connective work are appropriate, and evaluating their own CUNI systems is disclosed and balanced by online-B and the reference.

Who is this for? People working on MT evaluation or Czech–English discourse. It deserves a serious referee, not because the conclusion is clearly right but because the resource and annotation are worth having. I’d send it out and ask for a revised version that releases the data, reports uncertainty, and either validates the word-order rule against acceptability judgments or softens the comparative conclusion to “no clear difference.”

Referee Report

3 major / 5 minor

Summary. The paper presents a manually annotated test suite for evaluating document-level English-to-Czech NMT at WMT19. The authors select 101 PCEDT documents with PDTB-style discourse annotations and use trained linguist annotators to evaluate five systems plus a human reference on three coherence-related phenomena: alternative lexicalizations of discourse connectives (AltLexes), discourse connectives, and topic-focus articulation realized through subject word order. They report inter-annotator agreement (80% overall), qualitative error analyses across linguistic levels, and quantitative comparisons. The central claim is that the two context-aware systems (CUNI-DocTransf-T2T and CUNI-DocTransf-Marian) did not outperform sentence-level systems on the observed phenomena, and that the reference translation was sometimes judged worse without being incorrect.

Significance. The test suite and manual annotation methodology are valuable resources for a difficult evaluation problem, and the paper is transparent about the annotation procedure, including the reporting of inter-annotator agreement. The qualitative inventory of error types (morphology, lexicon, syntax, semantics, discourse) is useful for system developers. The negative finding on context-aware systems is interesting and falsifiable, but as presented it rests on very small counts and on a word-order scoring rule that is not validated against acceptability; if the rule is confirmed by further adjudication, the result would be a meaningful contribution to the document-level MT debate. The authors deserve credit for explicitly noting that the reference translation being marked as 'worse' does not necessarily mean it is incorrect, although this admission creates a tension with the way the word-order metric is used.

major comments (3)

[6.1.1, 7.4, and 8] The word-order metric labels the human reference as a 'no' in 3/22 placement judgments and in 6/7 focus-proper judgments, while Section 8 states that this 'does not mean that the reference is incorrect.' This is an internal inconsistency in the validity of the metric: if the rule is a measure of translation adequacy, the reference should almost always satisfy it; if the rule encodes a strong FGD-based stylistic preference, then the 'no' labels for MT systems are not evidence of translation errors. Because the two context-aware systems have the worst 'no' counts in the placement table (CUNI-DocTransf-Marian 14/5 and CUNI-DocTransf-T2T 13/3), the central claim that context-aware systems did not outperform the others is not established for this component. The authors should validate the rule with independent native-speaker acceptability judgments on the contested cases, or reframe the word-order results as a descriptive measure of divergence from the FGD preference rather than as translation adequacy.
[7.2–7.4 and Tables 1–3] The quantitative comparisons are based on very small counts, with focus-proper 'yes/no' observations per system ranging from 0 to 5 and placement observations ranging from 3 to 17; the star charts in Tables 1 and 2 do not report raw frequencies. At these sample sizes, the finding that 'the systems performed with only a minor differences' is indistinguishable from a lack of statistical power, and the absence of a significant difference is not evidence of equivalence. The authors should report exact counts for every cell, and either provide an appropriate statistical test (e.g., Fisher's exact test) or explicitly label the comparison as descriptive. This is load-bearing for the abstract's claim that context-aware systems did not outperform the others.
[6.1.1 and 7.4] The word-order analysis assumes that an English subject noun with an indefinite article is contextually non-bound and therefore should appear after the predicate (or as focus proper) in Czech. This assumption is presented as given ('It is assumed'), but the annotators themselves only confirmed the contextual non-boundness for a subset of the automatically selected sentences (85 Yes, 10 No). Given that the whole word-order component depends on this proxy, the paper should report how many of the automatically selected sentences were judged by the annotators as non-bound, and how many of the 'no' placement labels co-occur with a 'no' from the annotators' own contextual-boundness judgment. Without this, the placement results conflate an invalid source-side assumption with a genuine target-side error.

minor comments (5)

[7.1] The inter-annotator agreement is reported as a percentage range and average, but the paper does not state whether this is raw percentage agreement or a chance-corrected measure such as Cohen's kappa; specifying the measure would make the agreement figure interpretable.
[7.2] The phrase 'There were 23 queries in average' should be corrected to 'on average', and the star charts in Tables 1 and 2 would be easier to interpret if exact counts were provided in parentheses next to each star rating.
[6.1.1] The paper cites Hajičová et al. (1998) for definitions of topic-focus articulation, but it does not justify the specific heuristic that an indefinite article on a subject noun marks contextual non-boundness; a brief justification or pointer to the relevant passage would strengthen the methodological basis.
[8] The sentence 'This can be attributed to the fact that the systems perform good enough on this task already' is informal; 'perform well enough' would be more appropriate in a journal-style report.
[5.3] The discussion of word order examples would be clearer if the problematic word order in each Czech example were explicitly highlighted (e.g., with underlining or a gloss), as the current text requires the reader to infer the intended issue.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is an independent manual assessment; self-citations are descriptive, not load-bearing.

full rationale

The paper is an evaluation study, not a derivation. Its central empirical claim—"Contrary to our assumptions, the two context-aware systems did not outperform the others in translating the followed document-level phenomena"—is a summary of manual annotations, not a consequence of a definition or a fitted parameter. The test suite is built from external resources (PCEDT, PDTB 3.0, and the Functional Generative Description literature by Sgall et al.), and the target side was validated by trained linguists with measured inter-annotator agreement. No parameter is fitted to a subset of outputs and then renamed as a prediction; the word-order rule is an explicit annotation criterion, and the paper itself acknowledges cases where the human reference receives a negative judgment, explicitly stating that this "does not mean that the reference is incorrect." Whether that rule is too strict is a validity or correctness concern, not circularity. Self-citations (Rysová & Rysová 2018; Rysová 2018) are used for descriptive background on connective positions and Czech multi-word connectives; they do not force the measured outcomes. The fact that the authors' own systems (CUNI-*) are among the evaluated systems is also not circular, since the outputs are externally produced WMT19 submissions judged on a fixed annotation protocol. The conclusions could have come out differently under the same framework, and indeed the paper reports that the reference sometimes scored worse, showing the annotation did not presuppose the result. No load-bearing step reduces to its own input.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central evaluation relies on the correctness of PDTB 3.0 discourse annotations, the Functional Generative Description theory of topic-focus articulation, and the automatic proxy for contextually non-bound subjects. No free parameters are fitted because the paper is an evaluation study.

assumptions (3)

domain assumption PDTB 3.0 annotations of discourse connectives and AltLexes in the English source are accurate and complete.
Section 2 describes extracting discourse annotations from PDTB 3.0 and using them as the gold standard for selecting connectives and AltLexes.
domain assumption The Functional Generative Description framework correctly predicts Czech word order relative to contextual boundness.
Section 6.1.1 adopts this framework for determining whether subjects are contextually bound and for judging correct placement.
ad hoc to paper An indefinite article on an English subject noun marks it as contextually non-bound.
Section 6.1.1 uses this heuristic to preselect sentences; annotators later validated the selection (85 Yes, 10 No), but the heuristic is a modeling choice specific to this study.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Test Suite and Manual Evaluation of Document-Level NMT at WMT19." pith.science (2026). https://pith.science/paper/S6SN6DXN

@misc{pith2026190803043,
  author       = {Pith},
  title        = {Pith review of: A Test Suite and Manual Evaluation of Document-Level NMT at WMT19},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6SN6DXN}},
  note         = {Machine review of arXiv:1908.03043}
}

read the original abstract

As the quality of machine translation rises and neural machine translation (NMT) is moving from sentence to document level translations, it is becoming increasingly difficult to evaluate the output of translation systems. We provide a test suite for WMT19 aimed at assessing discourse phenomena of MT systems participating in the News Translation Task. We have manually checked the outputs and identified types of translation errors that are relevant to document-level translation.

Figures

Figures reproduced from arXiv: 1908.03043 by the authors.

**Figure 1.** Screenshot of the annotation interface. annotator, or the function of AltLex – according to the original English annotation displayed. If yes, then whether its Czech translation is (2): • adequate and correctly placed,3 • adequate but incorrectly placed, • omitted and it does not harm the output • omitted and it harms the output • not adequate. The questionnaire for word order annotation is analogous, compare the de… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Ond r ej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation ( WMT18 ) . In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Computational Linguistics

2018
[4]

Liane Guillou, Christian Hardmeier, Preslav Nakov, Sara Stymne, J \"o rg Tiedemann, Yannick Versley, Mauro Cettolo, Bonnie Webber, and Andrei Popescu-Belis. 2016. Findings of the 2016 WMT shared task on cross-lingual pronoun prediction . In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, volume 2, pages 525--542

work page 2016
[5]

Jan Haji c , Eva Haji c ov \' a , Jarmila Panevov \' a , Petr Sgall, Ond r ej Bojar, Silvie Cinkov \' a , Eva Fu c \' kov \' a , Marie Mikulov \' a , Petr Pajas, Jan Popelka, Ji r \' Semeck \' y , Jana S indlerov \' a , Jan S t e p \' a nek, Josef Toman, Zde n ka Ure s ov \' a , and Zden e k Z abokrtsk \' y . 2012. Announcing Prague Czech-English Dependen...

work page 2012
[6]

Eva Haji c ov \'a , Barbara H Partee, and Petr Sgall. 1998. Topic-focus articulation, tripartite structures and semantic content. Kluwer, Dordrecht

work page 1998
[7]

Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity o...

work page 2018
[8]

M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn treebank . Computational Linguistics, 19:313--330

work page 1993

Show all 18 references

[9]

Michal Nov \'a k. 2016. Pronoun prediction with linguistic features and example weighing. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, volume 2, pages 602--608

2016
[10]

Martin Popel. 2018. Cuni transformer neural mt system for wmt18. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 482--487

2018
[11]

Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2010. Realization of discourse relations by other means: Alternative lexicalizations. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 1023--1031. Association for Computational Linguistics

2010
[12]

Rashmi Prasad, Eleni Miltsakaki, Nikhil Dinesh, Alan Lee, Aravind Joshi, Livio Robaldo, and Bonnie L Webber. 2007. The penn discourse treebank 2.0 annotation manual

2007
[13]

Magdal \'e na Rysov \'a and Kate r ina Rysov \'a . 2018. Primary and secondary discourse connectives: Constraints and preferences. Journal of Pragmatics, 130:16--32

2018
[14]

Magdaléna Rysová. 2018. Diskurzní konektory v češtině: Od centra k periferii. Institute of Formal and Applied Linguistics, Praha, Czechia

2018
[15]

Petr Sgall. 1967. Functional sentence perspective in a generative description. Prague studies in mathematical linguistics, 2(203-225)

1967
[16]

Petr Sgall, Eva Haji c ov \'a , and Jarmila Panevov \'a . 1986. The meaning of the sentence in its semantic and pragmatic aspects. Springer Science & Business Media

1986
[17]

Bonnie Webber. 2009. Genre distinctions for discourse in the Penn TreeBank . In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 674--682. ...

2009
[18]

Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual

2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Ond r ej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation ( WMT18 ) . In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Computational Linguistics

2018

[4] [4]

Liane Guillou, Christian Hardmeier, Preslav Nakov, Sara Stymne, J \"o rg Tiedemann, Yannick Versley, Mauro Cettolo, Bonnie Webber, and Andrei Popescu-Belis. 2016. Findings of the 2016 WMT shared task on cross-lingual pronoun prediction . In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, volume 2, pages 525--542

work page 2016

[5] [5]

Jan Haji c , Eva Haji c ov \' a , Jarmila Panevov \' a , Petr Sgall, Ond r ej Bojar, Silvie Cinkov \' a , Eva Fu c \' kov \' a , Marie Mikulov \' a , Petr Pajas, Jan Popelka, Ji r \' Semeck \' y , Jana S indlerov \' a , Jan S t e p \' a nek, Josef Toman, Zde n ka Ure s ov \' a , and Zden e k Z abokrtsk \' y . 2012. Announcing Prague Czech-English Dependen...

work page 2012

[6] [6]

Eva Haji c ov \'a , Barbara H Partee, and Petr Sgall. 1998. Topic-focus articulation, tripartite structures and semantic content. Kluwer, Dordrecht

work page 1998

[7] [7]

Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity o...

work page 2018

[8] [8]

M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn treebank . Computational Linguistics, 19:313--330

work page 1993