{"id":"036a6325-22f1-4260-be42-caadab918964","arxiv_id":"2506.07719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-tier multilingual grammatical error annotation framework reimplements errant on Stanza and applies it to five languages with minimal to deep customization.","lead":"This paper introduces a modular framework for multilingual grammatical error annotation that pairs a language-agnostic core (Missing, Replacement, Unnecessary) with structured language-specific extensions, reimplementing errant on Stanza. It demonstrates the framework on English, German, Czech, Korean, and Chinese, with the goal of more consistent GEC evaluation across languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-lingual reliability claim is not yet evidenced: validation relies on aggregate operation counts and selected examples, not error-level agreement with gold annotations.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the universal MRU core plus UD labels must be sufficient for accurate cross-lingual annotation. My review of the full text confirms that the paper provides no error-level validation for any non-English language. The English reproduction (Table 1) is encouraging but only demonstrates that the tool can approximate errant's system-level scores, not that it produces identical or equally correct annotations. The Czech/German comparison in Tables 2 and 3 is even weaker: operation counts and label-frequency lists can coincide despite total disagreement on individual edits. The paper's own Limitations section defers detailed analysis to future work, which is an explicit admission that the strongest claim is not yet substantiated. The proposed concrete test directly targets this gap by measuring error-level agreement against existing gold annotations. If the check passes, the framework's cross-lingual reliability is supported; if it fails, the paper should be revised to claim feasibility rather than reliability. I therefore recommend keeping the reader's CONDITIONAL verdict unchanged, with the condition being the required error-level validation. The concern is not that the framework is wrong in principle, but that its central claim currently rests on insufficient evidence.","tokens_in":12358,"tokens_out":3651,"duration_ms":46045,"concrete_test":"Compute edit-level agreement of the proposed system against the gold M2 annotations in the Czech (Náplava et al., 2022) and German (Boyd, 2018) development sets, reporting span-level precision/recall/F0.5 for both the M/R/U operation and the full error-type label (e.g., by matching corrected tokens and labels). Compare these numbers with the English reproduction-level F0.5 from Table 1. If error-level F0.5 is substantially below the English level, or if the system's labels disagree with gold on a majority of edits, the claim that universal POS/dependency modules produce reliable annotations without language-specific classifiers is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the framework produces 'accurate and interpretable annotations' across languages and can be 'readily extended' without language-specific modules. For this to hold, Stanza's UD POS and dependency labels must support error classification in each language, and the universal MRU annotations must match gold-standard error spans and types. Neither condition is tested at the error level. The English validation (Table 1) compares only aggregate F0.5 scores of a GEC system evaluated by our tool versus errant; similar system scores can hide large disagreement in individual edits. For Czech and German, Table 2 compares only total counts of M/R/U operations, and Table 3 compares top-10 label-frequency lists under different label schemes. Two annotation systems can have identical operation counts while disagreeing on nearly every edit span and type. The Czech and German examples are illustrative, and the Chinese/Korean sections are demonstrations without quantitative evaluation. The Limitations section explicitly states that 'a detailed analysis of annotation improvements... is left to future work.' Thus the load-bearing assumption that a shared MRU core plus UD labels is sufficient for accurate multilingual annotation remains unsupported. The claim should be weakened to feasibility, or validated with error-level agreement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a multilingual grammatical error annotation framework that reimplements errant using the Stanza toolkit, combining a language-agnostic MRU (Missing, Replacement, Unnecessary) core with structured language-specific extensions. The authors validate the English reimplementation by comparing T5-based GEC evaluation scores against the original errant, then apply the framework to German, Czech, Korean, and Chinese with varying levels of customization. The central claim is that this design provides consistency and flexibility across typologically diverse languages, enabling accurate and interpretable annotations without requiring full language-specific classifiers for European languages. The paper includes example annotations, operation-count comparisons, and top-label frequency comparisons, but no error-level agreement evaluation against gold annotations or prior adaptations.","tokens_in":12568,"tokens_out":2351,"duration_ms":30664,"significance":"If the framework's claims were fully supported, this would be a practical contribution to multilingual GEC evaluation: a single UD-based pipeline that can be extended to many languages while preserving comparability, with a publicly released codebase. The English reimplementation result is encouraging, reproducing errant's F0.5 to within 0.0002, and the two-tier typology is a sensible design that addresses real fragmentation in existing errant adaptations. However, the paper's cross-lingual validation is currently only at the level of aggregate operation counts and selected examples, which is insufficient to establish that the produced annotations are accurate and interpretable for the claimed range of languages. The explicit limitation statement that 'a detailed analysis of annotation improvements... is left to future work' further indicates that the current evidence is preliminary.","major_comments":[{"comment":"The Czech and German comparisons report only total operation counts and top-10 label frequencies. Two annotation systems can have identical operation counts while disagreeing on nearly every individual edit span and type, so the statement that the framework 'can replicate established annotation distributions' (end of §4.2) is not supported. Please compute edit-level agreement (e.g., precision/recall/F0.5 between the system's edits and the prior adaptations' edits on the same sentences, or Cohen's kappa on aligned edits) and report it alongside the aggregate numbers.","section":"§4.2, Tables 2–3"},{"comment":"The English validation demonstrates similar system-level F0.5 scores between the reimplementation and errant, but this does not directly validate the annotation output. Please report the overlap between the edits produced by the two toolkits on the same system outputs (again, edit-level precision/recall or exact-match agreement), since the paper's goal is annotation quality rather than only evaluation score reproduction.","section":"§4.1, Table 1"},{"comment":"The spelling classification in Algorithm 1 depends on thresholds α1 and α2 for phonetic and shape similarity, but the paper gives no values for these parameters, no description of how they were chosen, and no sensitivity analysis. The entire R:SPELL subtyping rests on these free parameters, and yet no evaluation is provided for the spelling classifier in any language. Please specify the thresholds and validate the spelling subtype decisions against a sample of gold annotations.","section":"§3.2, Algorithm 1"},{"comment":"The Limitations section states that 'a detailed analysis of annotation improvements... is left to future work,' which directly conflicts with the Conclusion's assertion that the system 'produces accurate and interpretable annotations in English' and 'generate[s] reliable annotations without requiring language-specific classification modules' for European languages. Please either provide the missing quantitative evidence (edit-level agreement, annotation-quality analysis) or explicitly weaken the conclusions to claims of feasibility and promising preliminary consistency, rather than established reliability.","section":"§5, Limitations"},{"comment":"The Korean and Chinese applications are presented as demonstrations of flexibility, but they include no quantitative evaluation of annotation quality. For the Chinese case in particular, retraining Stanza on an LTP-style segmentation may affect POS-tagging and dependency-parsing accuracy, yet no measurement of parser quality or downstream annotation consistency is provided. Since these sections are used to support the framework's adaptability across 'typologically diverse languages,' please add at least a small-scale evaluation (e.g., agreement with a manually annotated sample or with existing ChERRANT/KAGAS annotations) or label these sections explicitly as illustrative feasibility studies.","section":"§4.3–§4.4"}],"minor_comments":[{"comment":"The GitHub URL in the abstract (https://github.com/open-writing-evaluation/jp_errant_bea) does not match the URL given in the body text (http://open-writing-evaluation.github.io); please verify that both are the intended project location.","section":"Abstract and §1"},{"comment":"The table header lists 'T5' as the system, and the text cites Rothe et al. (2021), but the reference describes 'A Simple Recipe for Multilingual Grammatical Error Correction' rather than specifically naming T5; please clarify the connection between the cited work and the T5 model used.","section":"§4.1, Table 1"},{"comment":"The Czech example shows 'R:VERB AUX -> AUX VERB' while the text refers to distinguishing auxiliary and main verbs; using a consistent label format (e.g., arrows and spacing) across the paper would improve readability.","section":"Figure 4"},{"comment":"The figure contains two duplicate 'Correction:' lines with slightly different content (one with pinyin and one without); please remove the redundancy.","section":"Figure 6"},{"comment":"The top-10 label lists for Czech and German are presented side-by-side, but the label schemes differ (e.g., 'NOUN→NOUN' vs 'DET→DET' with explicit source-target pairs); please add a note explaining that these are not directly comparable re-labelings, as the text acknowledges, and consider aligning label formats where possible.","section":"Table 3"},{"comment":"The definition of 'Replacement' uses the pattern R:P1→P2, but the MRU core is described as classifying errors into three operations; please clarify how the R:P1→P2 pattern relates to the 'Replacement' category and to the later algorithm in §3.2.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is plausible and the English reimplementation is a useful baseline, but the cross-lingual evidence is currently too weak to support the stated conclusions. The authors should be asked to add edit-level agreement metrics or a manual evaluation, and to either specify the free parameters or remove the claim that the spelling subtypes are a validated contribution. The high self-citation rate and the fact that several cited works are from the same group are not themselves disqualifying, but they increase the need for independent validation against prior adaptations rather than only comparing with the authors' own frameworks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing: the English reproduction is the real contribution. Reimplementing errant with Stanza and getting T5 precision/recall/F0.5 within 0.002 points is not trivial, and it means the alignment and classification logic survived the port. The two-tier MRU-plus-extensions design is clean, and Algorithm 1 gives a reusable template for spelling, word-order, and word-boundary errors. The paper is honest about its own scope: the Limitations section says detailed analysis of annotation improvements is left to future work.\n\nThe soft spots are exactly where the evidence stops. The cross-lingual accuracy claim—that the framework produces accurate and interpretable annotations in German, Czech, Korean, and Chinese without language-specific modules—is supported only by aggregate operation counts and selected examples. Two annotation systems can have nearly identical M/R/U counts and disagree on almost every edit span and type. Tables 2 and 3 don't measure error-level agreement. The Chinese and Korean sections are demonstrations of flexibility, not evaluations. The alpha_1 and alpha_2 thresholds in Algorithm 1 are unspecified, so the spelling classification is not reproducible as written. And Stanza on learner text is a known risk that the paper acknowledges but doesn't quantify.\n\nThe circularity concern is real but not fatal. The framework is validated against errant and prior adaptations, several by the same group, rather than an independent gold standard. For a tooling paper that's acceptable as long as the claims stay modest. The abstract and conclusion overclaim relative to the evidence.\n\nWho should read it? Anyone building multilingual GEC evaluation, especially for languages without errant-style annotation. It's a useful base. It deserves peer review, but the referees should require error-level agreement numbers and specified thresholds. I wouldn't cite it for the multilingual claims yet; the English reproduction is the citable piece.","headline":"A solid errant-on-Stanza tool paper with a real English reproduction, but the multilingual accuracy claims outrun the evidence—needs error-level agreement.","tokens_in":13136,"tokens_out":3861,"would_cite":false,"duration_ms":40450,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single modular annotation framework—a language-agnostic Missing/Replacement/Unnecessary core with small per-language extensions, running on Stanza's Universal Dependencies labels—can reproduce errant's English…","keywords":["grammatical error annotation","grammatical error correction","errant","multilingual annotation","MRU framework","Universal Dependencies","Stanza","error typology"],"falsifier":"Run the framework on the first 1000 Czech sentences from Náplava et al. (2022) and the German development set from Boyd (2018), then compare its edits label-by-label against the gold M2 annotations. If per-label agreement is low even where total M/R/U counts match—especially for the most frequent gold categories like Czech DIACR and German DET:FORM—then the universal core is not actually reproducing established annotations; it is only producing similar totals. A second check is to tag uncorrected learner sentences with Stanza and compare against human-corrected POS tags; systematic tag noise on learner text would propagate directly into every downstream error label.","tokens_in":12149,"feed_emoji":"🌐","tokens_out":14563,"duration_ms":140074,"temperature":0.7,"pith_summary":"The paper argues that grammatical error annotation—the step that turns an original and a corrected sentence into labeled error types—can be made consistent across typologically diverse languages without giving up language-specific detail. It proposes a two-tier system: a universal core that classifies every edit as Missing, Replacement, or Unnecessary and attaches a part-of-speech label, plus optional modules for language-specific phenomena such as phonetic spelling errors, word order, word boundaries, and (for Korean) postpositions and honorifics. To support the claim, the authors reimplement the standard English errant toolkit on the multilingual Stanza pipeline, reproduce its scores on English ($F_{0.5}$ within 0.0002), then apply the same core to German and Czech with no language-specific code, add light templates for Korean, and retrain the tokenizer for Chinese. If the framework delivers what it claims, GEC systems could be evaluated and compared across many languages under one shared, interpretable annotation scheme instead of a patchwork of incompatible tools.","feed_headline":"Five languages share one grammar-error annotation core","feed_subtitle":"A universal error core with per-language tweaks aims for comparable GEC evaluation across languages.","key_machinery":"The load-bearing mechanism is the separation of the annotation pipeline into a shared alignment-and-classification core and an extensible per-language layer. The alignment core, inherited from errant, uses the linguistically enhanced Damerau-Levenshtein algorithm of Felice et al. (2016) to find edit spans between original and corrected sentences, weighting POS tags, lemmas, and character similarity rather than raw surface distance. The classification core is the MRU scheme: each edit receives one of three operations and a UD-based POS label, producing interpretable labels such as R:NOUN→NOUN or M:PRON. Language-specific extensions sit on top of this core: Algorithm 1 classifies replacement edits into spelling errors (phonetic, shape, or phonographic), R:WO when the same word set appears in a different order, and R:WB when merging tokens yields the same character sequence; Korean adds WB:M and WB:U spacing subtypes plus ADP, PART, and HON morpheme categories; Chinese swaps the segmentation scheme by retraining Stanza on an LTP-annotated treebank. The framework's scalability rests on Stanza's UD-based tokenization, POS tagging, and dependency parsing being available and consistent across target languages.","core_discovery":"The paper's central claim is that a grammatical error annotation framework with a language-agnostic core can match existing per-language errant adaptations while adding cross-linguistic consistency. The core is the MRU scheme: every edit is classified as Missing, Replacement, or Unnecessary and further specified by a Universal-Dependencies-style POS label, so that an edit errant would mark R:VERB:FORM becomes R:VERB→AUX and a missing relative pronoun becomes M:PRON instead of M:OTHER. Around this core sit structured, reusable extension templates—spelling errors divided into phonetic, shape-based, and combined phonographic types, plus word-order and word-boundary errors—which languages can adopt, refine, or replace. The authors demonstrate the claim empirically: the English reimplementation scores $F_{0.5}=0.5503$ versus errant's $0.5501$ on T5 outputs; the generic core applied with no customization to German and Czech yields near-identical operation counts to the existing adaptations (4897 vs 4915 total edits on the first 1000 Czech sentences); Korean needs only small templates for spacing and functional morphemes; and Chinese requires deeper work, retraining Stanza on an LTP-style segmentation to avoid misleading spans such as splitting 为什么 into separate tokens. The intended consequence is a single, interpretable annotation standard that can be rolled out to other languages—including the ten remaining MultiGEC-2025 languages—without re-engineering the pipeline.","pith_inferences":["Editorial extension: the real test of the universal core is per-label agreement, not just operation counts; computing label-level precision/recall against the existing Czech, German, and Greek errant-style gold would show whether the framework captures the same errors or merely similar totals.","Editorial extension: if Stanza's UD labels hold up on noisy learner text for the full 70+ language set, the architecture becomes a zero-configuration annotator for low-resource GEC languages; a natural first probe would be an agglutinative language like Hungarian or Finnish, which are far from the five demonstrated languages.","Editorial extension: the paper's suggestion of mapping its labels onto existing schemes points to an implicit interoperability promise; building those mapping tables would let existing per-language datasets be converted rather than re-annotated, a concrete next deliverable the paper does not itself supply.","Editorial extension: the Chinese experiment suggests that tokenization granularity, not classification logic, drives most cross-system disagreement for logographic languages; a testable corollary is that segmenting learner text the way a human would reduces spurious spans and apparent error counts."],"forward_implications":["The same pipeline, with no further language-specific code, can be applied to the other ten languages in the MultiGEC-2025 corpus, because Stanza already supplies UD tokenization and POS/dependency labels for them.","Error distributions and system evaluation scores become directly comparable across languages that share the MRU core, simplifying multilingual GEC benchmarks that today must reconcile incompatible annotation schemes.","Systems evaluated with this framework get more diagnostic feedback in some cases—a missing relative pronoun reads as M:PRON rather than M:OTHER, and passive constructions read as R:VERB→AUX—so error-type analysis is more informative than errant's.","A new language can adopt the framework incrementally: start with the generic core, add templates for spelling, word order, or word boundaries only where needed, and retrain the tokenizer only if segmentation demands it.","Because the framework emits standard M2 format, it can replace errant in existing evaluation workflows without changing the downstream scoring pipeline."],"supporting_citations":[{"why":"Defines the original errant framework and its 55 English error types, which the paper reimplements and extends to multiple languages.","marker":"Bryant et al., 2017"},{"why":"Supplies the linguistically enhanced alignment algorithm that identifies edit spans between original and corrected sentences.","marker":"Felice et al., 2016"},{"why":"Provides Stanza, the UD-based neural pipeline for tokenization, POS tagging, and dependency parsing across 70+ languages.","marker":"Qi et al., 2020"},{"why":"The German errant adaptation whose dataset and operation counts serve as the baseline for the generic European-language test.","marker":"Boyd, 2018"},{"why":"The Czech errant adaptation whose dataset and operation counts serve as the baseline for the generic European-language test.","marker":"Náplava et al., 2022"},{"why":"Frames the MultiGEC-2025 shared task, the twelve-language setting the framework aims to serve and the ten additional languages it claims to cover without customization.","marker":"Masciolini et al., 2025a"},{"why":"Defines the Universal POS tagset that gives the language-agnostic MRU core its shared label vocabulary.","marker":"Petrov et al., 2012"},{"why":"Provides the linguistically informed Chinese error typology and the GSD-based word boundaries used as the comparison point in the Chinese customization experiment.","marker":"Gu et al., 2025"},{"why":"The Korean KAGAS annotation scheme and priority rules that the Korean extension refines with spacing and morpheme-level categories.","marker":"Yoon et al., 2023"}],"fun_headline_variants":["One annotation core to match per-language tools across five languages","Universal grammar-error core with per-language tweaks","Five languages, one error core, same scores","Grammar-error annotation: one core, flexible extensions","Unified error annotation matches per-language systems"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that Stanza's off-the-shelf part-of-speech and dependency tags are reliable enough on learner-written text, and rich enough for any language's grammar, that a single rule-based core can produce correct error labels without language-specific classifiers.","fun_headline_variants_meta":{"raw":{"variants":["One annotation core to match per-language tools across five languages","Universal grammar-error core with per-language tweaks","Five languages, one error core, same scores","Grammar-error annotation: one core, flexible extensions","Unified error annotation matches per-language systems"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1821,"prompt_tokens":997,"completion_tokens":824,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":752}},"tokens_in":613,"tokens_out":824,"duration_ms":10094,"temperature":1.0,"reasoning_tokens":752,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:26:58.822029+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on the first 1000 Czech sentences from Náplava et al. (2022) and the German development set from Boyd (2018), then compare its edits label-by-label against the gold M2 annotations. If per-label agreement is low even where total M/R/U counts match—especially for the most frequent gold categories like Czech DIACR and German DET:FORM—then the universal core is not actually reproducing established annotations; it is only producing similar totals. A second check is to tag uncorrected learner sentences with Stanza and compare against human-corrected POS tags; systematic tag noise on learner text would propagate directly into every downstream error label.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the linguistically enhanced alignment algorithm that identifies edit spans between original and corrected sentences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Universal POS tagset that gives the language-agnostic MRU core its shared label vocabulary."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Korean KAGAS annotation scheme and priority rules that the Korean extension refines with spacing and morpheme-level categories."}],"review_version":1}