Pith. sign in

REVIEW 4 major objections 5 minor 23 references

Evaluation of NMT-Assisted Grammar Transfer for a Multi-Language Configurable Data-to-Text System

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that automatic grammar transfer by neural machine translation is error-free for the large majority of grammar units, making a one-time human review per project sufficient to keep generated text grammatical in many…

desk verdict An honest industry case study of NMT-plus-grammar-transfer for multilingual data-to-text, but the 'error-free in most cases' claim leans on edit counts that no baseline or inter-annotator agreement supports. read the letter →

arxiv 2501.16135 v1 pith:AEAICFEC submitted 2025-01-27 cs.CL

Andreas Madsack , Johanna Heininger , Adela Schneider , Ching-Yi Chen , Christian Eckard , Robert Wei{ss}graeber This is my paper · ORCID
classification cs.CL
keywords data-to-textgenerationneuralmachinetranslationgrammartransfermultilingualNLGrule-basedpost-editingevaluationgrammaticalitydependencyparsing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a rule-based multilingual data-to-text system can import grammatical configurations from a source language using machine translation plus a one-time human review, and that in most cases the transferred grammar is already correct. The authors report that human translators, working on generated basketball match reports in seven languages, changed only 19% of grammar units on average, and that the majority of changes fell into a few identifiable error types. The point of the claim is practical: if grammar transfer can be trusted after a single review, then generating correct text in many languages no longer requires a translator in the loop for each produced document. The paper also offers a post-editing protocol that classifies the kinds of grammatical changes translators make, intended as a diagnostic for where transfer fails.

What carries the argument

The central object is the grammar unit, a container of grammatical settings (lemma, case, number, tense, person, gender, and so on) attached to the variable parts of a sentence that a surface realizer needs in order to produce correct inflections. The transfer machinery works by marking text snippets that belong to such units in the source, running a dependency parse on the translated snippet in the target language, and re-deriving the unit's features from the parse tree with custom aggregation rules for nouns, pronouns, and verbs. The evaluation machinery is a post-editing protocol in which translators correct the automatically transferred units and every edit is tagged with a change category, turning the edit log into a per-feature error report.

What would settle it

Run the identical post-editing experiment twice on the same language pairs: once with the full grammar-transfer pipeline and once with plain NMT followed by a surface realizer that receives no transferred grammar features. If the edit rates and error categories are statistically indistinguishable, then the grammar-transfer component is not what keeps the edit rate low, and the paper's central claim would be falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's result is that the grammar-transfer step in their hybrid data-to-text pipeline is "error-free in most cases": across German, Spanish, French, Portuguese, Slovenian, Polish, and Chinese, 81% of grammar units were left unchanged by native-speaker translators after NMT translation plus automatic feature transfer. The errors that were fixed are largely attributable to specific, diagnosable mechanisms rather than to random translation noise; the most frequent German error, a genitive misanalysis of team names ending in 's', is traced to the dependency parser rather than to the translation model. The paper concludes that the transfer rate is not yet high enough for fully uncontrolled output, but that the remaining correction effort is small compared to reviewing every generated text, and that targeted improvements to the transfer rules are possible.

Load-bearing premise

The load-bearing premise is that the number and type of translator post-edits measure the quality of the grammar-transfer component specifically, rather than the underlying translation quality, surface realization choices, or each translator's personal editing style; the paper does not separate these influences or measure agreement between translators.

Editorial extensions

If this is right

  • If the 19% average edit rate holds more broadly, a single human review per translation project, rather than per generated text, is enough to keep grammar correct across languages.
  • The error taxonomy shows which components cause the remaining failures, so parser-specific fixes (such as handling team names ending in 's') can be targeted directly.
  • The post-editing protocol gives NLG developers a way to localize grammar errors by feature, language, and component.
  • The hybrid architecture keeps the hallucination avoidance and configurability of rule-based generation while gaining NMT's speed and fluency.
  • The approach implies that scaling a data-to-text system to a new language costs one translation-plus-review pass per project, not per output.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 19% figure likely mixes grammar-transfer errors with translation quality and editorial taste; a controlled ablation that disables the grammar-transfer step would isolate the component's true contribution.
  • The wide per-translator variation suggests edit counts are partly a measure of individual standards; reporting inter-annotator agreement would turn the error ranking into a firmer diagnostic.
  • The specific dependency-parser failure on English team names suggests that domain lexical data (team rosters, name-ending exceptions) fed into the transfer step could cut the edit rate further than generic parser improvements.
  • The same post-editing protocol could be run with a grammar-rich language such as German as the source, which the authors flag as future work and which would stress the feature-transfer assumptions more than English does.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes a hybrid multilingual data-to-text system in which grammar units (containers of grammatical settings) are translated from English into target languages by NMT and then post-processed with spaCy dependency parsing and custom aggregation, so that the grammar specifications themselves are transferred rather than only the surface text. The authors evaluate this grammar-transfer mechanism by asking 13 translators to post-edit system outputs on the SportSett:Basketball dataset and by categorizing the changes made to grammar units. They report that on average only 19% of grammar units were changed, interpret this as evidence that the automatic transfer was 'error-free in most cases,' and conclude that NMT-assisted grammar transfer can reduce human effort while human correction remains necessary for uncontrolled output. The paper also contributes a documented example of a German genitive/nominative error caused by the dependency parser and an appendix example of source, NMT, and post-edited text.

Significance. If its central claim were robustly established, the paper would offer a useful template for adding multilingual coverage to configurable rule-based NLG systems without a per-text human in the loop, and its post-editing methodology would be a practical evaluation instrument for grammar-level errors. The paper is honest about several limitations: it acknowledges the small participant count, the unfinished Polish annotation, the closed-source platform, and the resulting reproducibility limits. The specific error analyses, such as the German team-name 's' being parsed as genitive, are informative and concrete. However, the quantitative claim of 'error-free in most cases' is not yet supported by the data as presented, because the edit-rate evidence is not shown to be reliable or attributable to the grammar-transfer component. The paper also ships no code, data, or machine-checked artifacts, so the evaluation cannot currently be reproduced or audited externally.

major comments (4)
  1. [Section 5, Table 3] The percentage-based results in Table 3 are not quantitatively interpretable because the denominators are never reported. The table states that each percentage is relative to 'all grammar units of the language,' but the reader cannot tell how many grammar units were evaluated for German, French, Spanish, Slovenian, Polish, Portuguese, or Chinese. With seven German translators but only one translator for each of the other six languages, and with the Polish annotation explicitly unfinished, the table conflates individual editorial behavior with system-level quality. Please report per-language and per-participant grammar-unit counts, the number of edited units, and confidence intervals, and clearly separate or exclude the incomplete Polish data.
  2. [Section 6, with Section 5 and Figure 2] The sentence in Section 6 that 'the automatic transfer was error-free in most cases' is load-bearing, but it rests entirely on the 19% average edit rate reported in Section 5. This inference requires that an absent post-edit is evidence that a grammar unit was correctly transferred, and that each edit is evidence of a grammar-transfer error. Neither condition is established. No inter-annotator agreement is reported, and Figure 2 shows large variation within German: participant 2 made the minimum edits, while participant 9 made many more and, as the authors note, 'even adding more grammar units that were not necessary to get a grammatically correct rendered text.' Such optional additions show that edit counts include editorial preferences, not only grammatical errors. A threshold for 'error-free in most cases' should also be stated before the edit rate is cited as a success criterion.
  3. [Sections 4.2 and 5] The experiment has no ablation or baseline in which the grammar-transfer/dependency-adaptation step is disabled, and no comparison against an NMT-only condition. Because the grammar-transfer mechanism operates inside the same pipeline as NMT and surface realization, the observed post-edits cannot be attributed specifically to grammar transfer. The claim that the grammar-transfer feature works well therefore needs either a controlled comparison with grammar transfer turned off, or an error-class analysis that links each edit to a grammar-unit feature in a way that excludes translation and realization errors. Without such a contrast, the paper's positive result is a system-level usability observation rather than a component-level validation.
  4. [Section 6, Limitations paragraph] The authors themselves state that 'the limited number of participants hardly allows a profound quantitative evaluation' and that reproducibility is limited by the closed-source platform. These admissions are appropriate, but they undercut the quantitative use of Table 3 and Figure 2 elsewhere in the paper. The manuscript should explicitly frame the study as a pilot and avoid cross-language generalizations such as the claim that changes 'were as expected' for six languages when six of the seven languages have a single annotator and one of those annotations is incomplete.
minor comments (5)
  1. [Abstract and Section 6] The abstract says the generation phase scales 'without a human in the loop,' but Section 6 concludes that human correction of grammar units remains essential for uncontrolled output. Please align the wording so that the human-in-the-loop claim refers to the generation phase only and does not contradict the stated need for one-time translation review.
  2. [Table 2] The example shows English 'on Saturday' becoming German 'am Samstag,' but the table lists the preposition as 'an' without explaining the contraction with 'dem' to 'am'; a short gloss would make the transfer example easier to follow.
  3. [Figure 2] The figure's axes and units are not described in the text, so it is unclear whether the y-axis shows absolute edit counts, edits per grammar unit, or edits per statement. Please label the axes and state the unit in the caption.
  4. [Section 4.1] The dataset name is given as 'Sport-Sett:Basketball' but the URL uses 'sportsett_basketball'; please standardize the spelling and cite the dataset version used.
  5. [Appendix, Table 4] The appendix table relies on green text to indicate post-edits, but this formatting is not visible in a monochrome version; please add a marker such as bold or underline, and add a brief gloss for the German examples that contain obvious NMT artifacts (e.g., 'Rückprall' and 'helfen').

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the central result rests on external human post-edits, and the only self-citation is a non-load-bearing description of the proprietary platform.

full rationale

The paper contains no mathematical derivation whose output is equivalent to its input. The central claim in Section 6 that 'the automatic transfer was error-free in most cases' is an empirical summary of the Section 5 observation that 'on average only 19% of the grammar units were actually changed' by translators. The reference point is external: native-speaker translators performed post-edits on system output, and those edits were counted and categorized. No parameter of the grammar-transfer mechanism is fitted to the post-edit data, and no equation defines grammar-transfer quality in terms of the edit rate by construction. The self-citations (Weißgraeber and Madsack, 2017) describe the existing rule-based NLG platform but do not assert the evaluation result, so they are not load-bearing for the paper's conclusion. Concerns that edit counts may conflate NMT quality, surface realization, or individual editorial style, and the absence of inter-annotator agreement, are threats to the validity and reliability of the evaluation rather than evidence of circularity. The authors' own concession that the limited participant count 'hardly allows a profound quantitative evaluation' is a stated limitation, not a circular step.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters: it reports a system evaluation, not a fitted model. The main axioms are domain assumptions about the validity of post-editing as a measurement tool and about the accuracy of the dependency parser. No new entities are postulated.

assumptions (4)
  • domain assumption Grammatical acceptability judgments by native-speaker translators are a valid ground truth for measuring system output correctness.
    Section 4.2 defines the post-edit task; Section 6 acknowledges limited participants.
  • domain assumption Edits made during post-editing reflect system errors rather than translator preferences or stylistic choices.
    Figure 2 shows high variance in number of edits per participant, undermining this assumption.
  • ad hoc to paper Grammar units can be individually transferred and matched between source and target without considering larger syntactic context.
    Section 2.3 defines grammar units as containers; Section 5 matches 98% of units but does not measure whether context-independent transfer is sufficient.
  • domain assumption spaCy dependency parsing is accurate enough across all target languages for the grammar transfer to succeed.
    Section 5 reports that team names ending in 's' were misparsed as genitive in German, showing the parser is a failure point.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluation of NMT-Assisted Grammar Transfer for a Multi-Language Configurable Data-to-Text System." pith.science (2026). https://pith.science/paper/AEAICFEC

@misc{pith2026250116135,
  author       = {Pith},
  title        = {Pith review of: Evaluation of NMT-Assisted Grammar Transfer for a Multi-Language Configurable Data-to-Text System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AEAICFEC}},
  note         = {Machine review of arXiv:2501.16135}
}
read the original abstract

One approach for multilingual data-to-text generation is to translate grammatical configurations upfront from the source language into each target language. These configurations are then used by a surface realizer and in document planning stages to generate output. In this paper, we describe a rule-based NLG implementation of this approach where the configuration is translated by Neural Machine Translation (NMT) combined with a one-time human review, and introduce a cross-language grammar dependency model to create a multilingual NLG system that generates text from the source data, scaling the generation phase without a human in the loop. Additionally, we introduce a method for human post-editing evaluation on the automatically translated text. Our evaluation on the SportSett:Basketball dataset shows that our NLG system performs well, underlining its grammatical correctness in translation tasks.

Figures

Figures reproduced from arXiv: 2501.16135 by the authors.

Figure 1
Figure 1. Translation workflow, which involves adapting grammatical units during text translation to ensure proper [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Number of changes per participant This can be seen in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 16 canonical work pages

  1. [1]

    Eleftherios Avramidis, Aljoscha Burchardt, Christian Federmann, Maja Popovi \'c , Cindy Tscherwinka, and David Vilar. 2012. http://www.lrec-conf.org/proceedings/lrec2012/pdf/294_Paper.pdf Involving language professionals in the evaluation of machine translation . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( ...

    work page 2012

  2. [2]

    Wilker Aziz, Sheila Castilho, and Lucia Specia. 2012. http://www.lrec-conf.org/proceedings/lrec2012/pdf/985_Paper.pdf PET : a tool for post-editing and assessing machine translation . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( LREC '12) , pages 3982--3987, Istanbul, Turkey. European Language Resources Asso...

    work page 2012

  3. [3]

    Howcroft

    Anya Belz, Simon Mille, and David M. Howcroft. 2020. https://aclanthology.org/2020.inlg-1.24 Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing . In Proceedings of the 13th International Conference on Natural Language Generation, pages 183--194, Dublin, Ir...

    work page 2020

  4. [4]

    Daniel Braun, Kira Klimt, Daniela Schneider, and Florian Matthes. 2019. https://doi.org/10.18653/v1/W19-8651 S imple NLG - DE : Adapting S imple NLG 4 to G erman . In Proceedings of the 12th International Conference on Natural Language Generation, pages 415--420, Tokyo, Japan. Association for Computational Linguistics

    work page doi:10.18653/v1/w19-8651 2019

  5. [5]

    Glenn Carroll and Eugene Charniak. 1992. Two experiments on learning probabilistic dependency grammars from corpora. Technical report, Brown University, USA

    work page 1992

  6. [6]

    Robert Dale. 2020. https://doi.org/10.1017/S135132492000025X Natural language generation: The commercial state of the art in 2020 . Natural Language Engineering, 26(4):481–487

    work page doi:10.1017/s135132492000025x 2020

  7. [7]

    Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. https://doi.org/10.1162/tacl_a_00437 Experts, errors, and context: A large-scale study of human evaluation for machine translation . Transactions of the Association for Computational Linguistics, 9:1460--1474

    doi:10.1162/tacl_a_00437 2021

  8. [8]

    Albert Gatt and Ehud Reiter. 2009. https://aclanthology.org/W09-0613 S imple NLG : A realisation engine for practical applications . In Proceedings of the 12th E uropean Workshop on Natural Language Generation ( ENLG 2009) , pages 90--93, Athens, Greece. Association for Computational Linguistics

    work page 2009

Show all 23 references
  1. [9]

    Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ond r ej Du s ek, Chris Chinenye Emezue, Varun Gangal, Cri...

    2021

  2. [10]

    Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. http://arxiv.org/abs/2202.06935 Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text

    2022 arXiv

  3. [11]

    Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A

    David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. https://aclanthology.org/2020.inlg-1.23 Twenty years of confusion in human evaluation: NLG needs...

    2020

  4. [12]

    Juraj Juraska, Kevin Bowden, and Marilyn Walker. 2019. https://doi.org/10.18653/v1/W19-8623 V i GGO : A video game corpus for data-to-text generation in open-domain conversation . In Proceedings of the 12th International Conference on Natural Language Generation, pages 164--17...

    2019 doi

  5. [13]

    Ines Montani, Matthew Honnibal, Matthew Honnibal, Sofie Van Landeghem, Adriane Boyd, Henning Peters, Paul O'Leary McCann, jim geovedi, Jim O'Regan, Maxim Samsonov, György Orosz, Daniël de Kok, Marcus Blättermann, Duygu Altinok, Raphael Mitsch, Madeesh Kannan, Søren Lind Kristi...

    2023

  6. [14]

    Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, and Min-Yen Kan. 2020. https://doi.org/10.18653/v1/2020.acl-main.135 Semantic graphs for generating deep questions . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1463--14...

    2020 doi

  7. [15]

    Ehud Reiter, Robert Dale, and Zhiwei Feng. 2000. Building natural language generation systems, volume 33. MIT Press

    2000

  8. [16]

    Somayajulu Sripada, Ehud Reiter, and Lezan Hawizy. 2004. Evaluating an nlg system using post-editing. IJCAI International Joint Conference on Artificial Intelligence

    2004

  9. [17]

    Somayajulu Sripada, Ehud Reiter, and Lezan Hawizy. 2005. https://aclanthology.org/W05-1615 Evaluation of an NLG system using post-edit data: Lessons learnt . In Proceedings of the Tenth E uropean Workshop on Natural Language Generation ( ENLG -05) , Aberdeen, Scotland. Associa...

    2005

  10. [18]

    Carson T Sch \"u tze. 2016. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. Language Science Press

    2016

  11. [19]

    Craig Thomson, Ehud Reiter, and Somayajulu Sripada. 2020. https://aclanthology.org/2020.intellang-1.4 S port S ett:basketball - a robust and maintainable data-set for natural language generation . In Proceedings of the Workshop on Intelligent Information Processing and Natural...

    2020

  12. [20]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. http://arxiv.org/abs/1706.03762 Attention is all you need

    2017 arXiv

  13. [21]

    Robert Wei graeber and Andreas Madsack. 2017. https://doi.org/10.18653/v1/W17-3524 A working, non-trivial, topically indifferent NLG system for 17 languages . In Proceedings of the 10th International Conference on Natural Language Generation, pages 156--157, Santiago de Compos...

    2017 doi

  14. [22]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  15. [23]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.