{"id":"0aaa1d28-f0cf-40a1-9d12-6e0b071f5b2e","arxiv_id":"2501.16135","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a proprietary rule-based NLG system, NMT-assisted grammar transfer required post-editing of roughly 19% of grammar units across seven target languages in a small human evaluation.","lead":"A text generation system translates its grammar rules across languages with machine translation, then human editors patch the leftovers. In a small multilingual test on basketball game summaries, about one in five grammar units needed a manual correction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'error-free in most cases' claim depends on edit counts that are never shown to be reliable or attributable to the grammar-transfer component; no inter-annotator agreement or baseline is reported.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing gap: the low post-edit rate is interpreted as evidence of grammar-transfer quality, but the experiment does not control for translator variability, NMT output quality, or surface realization. The paper is an honest industrial case study with a concrete system description and worked examples in the appendix, but its headline claim is stronger than the evidence. The proposed controlled study would settle the attribution question directly: inter-annotator agreement tests whether an absent edit is a stable signal of correctness, and the ablation tests whether the grammar-transfer component is what produces the low edit rate. Given the proprietary platform and small participant count, the reader's CONDITIONAL verdict remains appropriate; I would not change it, but the authors should add such a study or soften the conclusion in revision.","tokens_in":7219,"tokens_out":7846,"duration_ms":72926,"concrete_test":"Run a controlled post-edit study on 30 randomly selected German statements: (a) with the full grammar-transfer pipeline and (b) with the spaCy dependency-adaptation step disabled, using two independent native-speaker translators per condition. Compute the per-grammar-unit edit counts and Cohen's kappa on the binary 'edited vs. not edited' label for each unit. If kappa falls below 0.6, or if the edit-rate difference between (a) and (b) is not significant, the 19% figure cannot be attributed to grammar-transfer correctness and the conclusion should be reframed; if kappa is high and the ablation increases edit counts substantially, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion in Section 6 — that automatic transfer was 'error-free in most cases' — rests entirely on the observation in Section 5 that on average only 19% of grammar units were changed by translators. This inference requires that an absent post-edit is evidence that a grammar unit was correctly transferred, and that the edits that do occur are caused by the grammar-transfer component. Neither condition is established. Section 5 reports no inter-annotator agreement, and Figure 2 shows edit counts varying widely even within German: participant 2 made the absolute minimum of edits, while participant 9 made many more and 'even adding more grammar units that were not necessary to get a grammatically correct rendered text.' Those optional additions mean edit counts include editorial style, not just grammatical errors. Because there is no ablation or baseline with the grammar-transfer/dependency-adaptation step disabled, the surviving edits cannot be separated from NMT translation quality or surface-realization choices. The authors themselves concede in Section 6 that the limited participant count 'hardly allows a profound quantitative evaluation.' The 19% figure is therefore not yet a stable measure of grammar-transfer correctness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a hybrid multilingual data-to-text system in which grammar units (containers of grammatical settings) are translated from English into target languages by NMT and then post-processed with spaCy dependency parsing and custom aggregation, so that the grammar specifications themselves are transferred rather than only the surface text. The authors evaluate this grammar-transfer mechanism by asking 13 translators to post-edit system outputs on the SportSett:Basketball dataset and by categorizing the changes made to grammar units. They report that on average only 19% of grammar units were changed, interpret this as evidence that the automatic transfer was 'error-free in most cases,' and conclude that NMT-assisted grammar transfer can reduce human effort while human correction remains necessary for uncontrolled output. The paper also contributes a documented example of a German genitive/nominative error caused by the dependency parser and an appendix example of source, NMT, and post-edited text.","tokens_in":7456,"tokens_out":3580,"duration_ms":37615,"significance":"If its central claim were robustly established, the paper would offer a useful template for adding multilingual coverage to configurable rule-based NLG systems without a per-text human in the loop, and its post-editing methodology would be a practical evaluation instrument for grammar-level errors. The paper is honest about several limitations: it acknowledges the small participant count, the unfinished Polish annotation, the closed-source platform, and the resulting reproducibility limits. The specific error analyses, such as the German team-name 's' being parsed as genitive, are informative and concrete. However, the quantitative claim of 'error-free in most cases' is not yet supported by the data as presented, because the edit-rate evidence is not shown to be reliable or attributable to the grammar-transfer component. The paper also ships no code, data, or machine-checked artifacts, so the evaluation cannot currently be reproduced or audited externally.","major_comments":[{"comment":"The percentage-based results in Table 3 are not quantitatively interpretable because the denominators are never reported. The table states that each percentage is relative to 'all grammar units of the language,' but the reader cannot tell how many grammar units were evaluated for German, French, Spanish, Slovenian, Polish, Portuguese, or Chinese. With seven German translators but only one translator for each of the other six languages, and with the Polish annotation explicitly unfinished, the table conflates individual editorial behavior with system-level quality. Please report per-language and per-participant grammar-unit counts, the number of edited units, and confidence intervals, and clearly separate or exclude the incomplete Polish data.","section":"Section 5, Table 3"},{"comment":"The sentence in Section 6 that 'the automatic transfer was error-free in most cases' is load-bearing, but it rests entirely on the 19% average edit rate reported in Section 5. This inference requires that an absent post-edit is evidence that a grammar unit was correctly transferred, and that each edit is evidence of a grammar-transfer error. Neither condition is established. No inter-annotator agreement is reported, and Figure 2 shows large variation within German: participant 2 made the minimum edits, while participant 9 made many more and, as the authors note, 'even adding more grammar units that were not necessary to get a grammatically correct rendered text.' Such optional additions show that edit counts include editorial preferences, not only grammatical errors. A threshold for 'error-free in most cases' should also be stated before the edit rate is cited as a success criterion.","section":"Section 6, with Section 5 and Figure 2"},{"comment":"The experiment has no ablation or baseline in which the grammar-transfer/dependency-adaptation step is disabled, and no comparison against an NMT-only condition. Because the grammar-transfer mechanism operates inside the same pipeline as NMT and surface realization, the observed post-edits cannot be attributed specifically to grammar transfer. The claim that the grammar-transfer feature works well therefore needs either a controlled comparison with grammar transfer turned off, or an error-class analysis that links each edit to a grammar-unit feature in a way that excludes translation and realization errors. Without such a contrast, the paper's positive result is a system-level usability observation rather than a component-level validation.","section":"Sections 4.2 and 5"},{"comment":"The authors themselves state that 'the limited number of participants hardly allows a profound quantitative evaluation' and that reproducibility is limited by the closed-source platform. These admissions are appropriate, but they undercut the quantitative use of Table 3 and Figure 2 elsewhere in the paper. The manuscript should explicitly frame the study as a pilot and avoid cross-language generalizations such as the claim that changes 'were as expected' for six languages when six of the seven languages have a single annotator and one of those annotations is incomplete.","section":"Section 6, Limitations paragraph"}],"minor_comments":[{"comment":"The abstract says the generation phase scales 'without a human in the loop,' but Section 6 concludes that human correction of grammar units remains essential for uncontrolled output. Please align the wording so that the human-in-the-loop claim refers to the generation phase only and does not contradict the stated need for one-time translation review.","section":"Abstract and Section 6"},{"comment":"The example shows English 'on Saturday' becoming German 'am Samstag,' but the table lists the preposition as 'an' without explaining the contraction with 'dem' to 'am'; a short gloss would make the transfer example easier to follow.","section":"Table 2"},{"comment":"The figure's axes and units are not described in the text, so it is unclear whether the y-axis shows absolute edit counts, edits per grammar unit, or edits per statement. Please label the axes and state the unit in the caption.","section":"Figure 2"},{"comment":"The dataset name is given as 'Sport-Sett:Basketball' but the URL uses 'sportsett_basketball'; please standardize the spelling and cite the dataset version used.","section":"Section 4.1"},{"comment":"The appendix table relies on green text to indicate post-edits, but this formatting is not visible in a monochrome version; please add a marker such as bold or underline, and add a brief gloss for the German examples that contain obvious NMT artifacts (e.g., 'Rückprall' and 'helfen').","section":"Appendix, Table 4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This is a self-evaluation of a proprietary commercial system by its developers, and the central quantitative claim rests on one translator per language for six of seven languages, an unfinished Polish annotation, and edit counts that are not shown to be attributable to the grammar-transfer component. The paper is nevertheless within scope for an application-oriented NLP venue if reframed as a pilot study with transparent denominators and a defensible attribution analysis. I would make the reporting of per-participant and per-language counts, the explicit pilot framing, and the release of anonymized post-edit data conditions of acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a cautious industry report, not a new scientific principle. The authors describe a hybrid pipeline: rule-based data-to-text with grammar units translated by NMT, then adjusted via dependency parsing (spaCy) to transfer grammatical features across languages. They evaluate it by having 13 translators post-edit generated texts on SportSett:Basketball, tracking changes at the grammar-unit level. The newness is the specific application of post-editing to grammar-transfer evaluation and the resulting error typology for one proprietary system.\n\nCredit where it is due: the system description is clear, the dataset is public, and the error typology (e.g., case changes from misparsed team names, compound-noun head marking) is practically useful. The appendix example shows exactly what kind of corrections are needed. The authors are also upfront about the closed platform and admit the participant count 'hardly allows a profound quantitative evaluation.' That honesty matters.\n\nNow the soft spots, and they are in the middle, not fatal. The headline claim that automatic transfer was 'error-free in most cases' rests on the 19% average edit rate. That number is not a stable measure of grammar-transfer correctness. There is no inter-annotator agreement, no baseline with the grammar-transfer step disabled, and Figure 2 shows high variance across translators even within German—participant 9 added grammar units that were not needed for grammatical correctness. So edit counts conflate grammatical errors, translation quality, and editorial style. Table 3 also lacks denominators and confidence intervals, and six of seven languages have a single translator, turning those percentages into anecdotes. The Polish annotation was unfinished and appears to still be in the table.\n\nThe good news is that the authors already concede the main limitation. A revision should reframe 'error-free in most cases' as 'most grammar units were left unchanged by one translator per language,' add per-language counts and confidence intervals, report inter-annotator agreement, and ideally include an ablation. None of that requires new theory.\n\nWho is this for? Practitioners building multilingual data-to-text systems and researchers working on human evaluation methodology for NLG. It deserves a serious referee—send it to peer review with a request for major revision, not a desk reject. If I were editing, I would find reviewers with NLG evaluation experience and push the authors to tighten the quantitative claims.","headline":"An honest industry case study of NMT-plus-grammar-transfer for multilingual data-to-text, but the 'error-free in most cases' claim leans on edit counts that no baseline or inter-annotator agreement supports.","tokens_in":7920,"tokens_out":1908,"would_cite":false,"duration_ms":19454,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that automatic grammar transfer by neural machine translation is error-free for the large majority of grammar units, making a one-time human review per project sufficient to keep generated text grammatical in many…","keywords":["data-to-text generation","neural machine translation","grammar transfer","multilingual NLG","rule-based NLG","post-editing evaluation","grammaticality","dependency parsing"],"falsifier":"Run the identical post-editing experiment twice on the same language pairs: once with the full grammar-transfer pipeline and once with plain NMT followed by a surface realizer that receives no transferred grammar features. If the edit rates and error categories are statistically indistinguishable, then the grammar-transfer component is not what keeps the edit rate low, and the paper's central claim would be falsified.","tokens_in":7072,"feed_emoji":"🏀","tokens_out":7604,"duration_ms":60166,"temperature":0.7,"pith_summary":"The paper claims that a rule-based multilingual data-to-text system can import grammatical configurations from a source language using machine translation plus a one-time human review, and that in most cases the transferred grammar is already correct. The authors report that human translators, working on generated basketball match reports in seven languages, changed only 19% of grammar units on average, and that the majority of changes fell into a few identifiable error types. The point of the claim is practical: if grammar transfer can be trusted after a single review, then generating correct text in many languages no longer requires a translator in the loop for each produced document. The paper also offers a post-editing protocol that classifies the kinds of grammatical changes translators make, intended as a diagnostic for where transfer fails.","feed_headline":"Grammar transfer survives translation: only 19% of units need fixing","feed_subtitle":"In a seven-language data-to-text system, neural translation plus one-time review left 81% of grammar units correct.","key_machinery":"The central object is the grammar unit, a container of grammatical settings (lemma, case, number, tense, person, gender, and so on) attached to the variable parts of a sentence that a surface realizer needs in order to produce correct inflections. The transfer machinery works by marking text snippets that belong to such units in the source, running a dependency parse on the translated snippet in the target language, and re-deriving the unit's features from the parse tree with custom aggregation rules for nouns, pronouns, and verbs. The evaluation machinery is a post-editing protocol in which translators correct the automatically transferred units and every edit is tagged with a change category, turning the edit log into a per-feature error report.","core_discovery":"On its own terms, the paper's result is that the grammar-transfer step in their hybrid data-to-text pipeline is \"error-free in most cases\": across German, Spanish, French, Portuguese, Slovenian, Polish, and Chinese, 81% of grammar units were left unchanged by native-speaker translators after NMT translation plus automatic feature transfer. The errors that were fixed are largely attributable to specific, diagnosable mechanisms rather than to random translation noise; the most frequent German error, a genitive misanalysis of team names ending in 's', is traced to the dependency parser rather than to the translation model. The paper concludes that the transfer rate is not yet high enough for fully uncontrolled output, but that the remaining correction effort is small compared to reviewing every generated text, and that targeted improvements to the transfer rules are possible.","pith_inferences":["The 19% figure likely mixes grammar-transfer errors with translation quality and editorial taste; a controlled ablation that disables the grammar-transfer step would isolate the component's true contribution.","The wide per-translator variation suggests edit counts are partly a measure of individual standards; reporting inter-annotator agreement would turn the error ranking into a firmer diagnostic.","The specific dependency-parser failure on English team names suggests that domain lexical data (team rosters, name-ending exceptions) fed into the transfer step could cut the edit rate further than generic parser improvements.","The same post-editing protocol could be run with a grammar-rich language such as German as the source, which the authors flag as future work and which would stress the feature-transfer assumptions more than English does."],"forward_implications":["If the 19% average edit rate holds more broadly, a single human review per translation project, rather than per generated text, is enough to keep grammar correct across languages.","The error taxonomy shows which components cause the remaining failures, so parser-specific fixes (such as handling team names ending in 's') can be targeted directly.","The post-editing protocol gives NLG developers a way to localize grammar errors by feature, language, and component.","The hybrid architecture keeps the hallucination avoidance and configurability of rule-based generation while gaining NMT's speed and fluency.","The approach implies that scaling a data-to-text system to a new language costs one translation-plus-review pass per project, not per output."],"supporting_citations":[{"why":"Supplies the SportSett:Basketball dataset used to generate the evaluation texts.","marker":"Thomson et al., 2020"},{"why":"Defines the three-stage NLG architecture (document planning, microplanning, surface realisation) that the system follows.","marker":"Reiter et al., 2000"},{"why":"Provides the Transformer architecture underlying the NMT component that translates source text.","marker":"Vaswani et al., 2017"},{"why":"Describes the earlier rule-based 17-language NLG system that this work extends with grammar transfer.","marker":"Weißgraeber and Madsack, 2017"},{"why":"Provides the dependency parser used to analyse translated grammar units and re-derive their features.","marker":"Montani et al., 2023"},{"why":"Established post-editing as an evaluation method for NLG systems, which the paper adapts to grammar units.","marker":"Sripada et al. (2005, 2004)"},{"why":"Frames the discussion of evaluation criteria and the need for standardised human evaluation in NLG.","marker":"Howcroft et al., 2020"}],"fun_headline_variants":["Grammar transfer: 81% correct, errors diagnosable","NMT grammar transfer leaves 81% of units intact","Translating grammar configs: 81% survive without fixes","Hybrid NLG: grammar transfer mostly error-free","Grammar transfer works: only 19% need human fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the number and type of translator post-edits measure the quality of the grammar-transfer component specifically, rather than the underlying translation quality, surface realization choices, or each translator's personal editing style; the paper does not separate these influences or measure agreement between translators.","fun_headline_variants_meta":{"raw":{"variants":["Grammar transfer: 81% correct, errors diagnosable","NMT grammar transfer leaves 81% of units intact","Translating grammar configs: 81% survive without fixes","Hybrid NLG: grammar transfer mostly error-free","Grammar transfer works: only 19% need human fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1143,"prompt_tokens":847,"completion_tokens":296,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":214}},"tokens_in":463,"tokens_out":296,"duration_ms":3342,"temperature":1.0,"reasoning_tokens":214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:42:02.139485+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical post-editing experiment twice on the same language pairs: once with the full grammar-transfer pipeline and once with plain NMT followed by a surface realizer that receives no transferred grammar features. If the edit rates and error categories are statistically indistinguishable, then the grammar-transfer component is not what keeps the edit rate low, and the paper's central claim would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SportSett:Basketball dataset used to generate the evaluation texts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the three-stage NLG architecture (document planning, microplanning, surface realisation) that the system follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the earlier rule-based 17-language NLG system that this work extends with grammar transfer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the dependency parser used to analyse translated grammar units and re-derive their features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Established post-editing as an evaluation method for NLG systems, which the paper adapts to grammar units."}],"review_version":1}