{"id":"9a5fc299-44d9-4fb4-9cb3-bad32fa53485","arxiv_id":"2507.11867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A bidirectional COLA-GEC framework produces gains on MuCGEC and CoNLL-14 but falls short of quoted baselines on BEA-19, FCGEC, and German.","lead":"This paper introduces a two-way knowledge transfer between grammatical acceptability judgment and grammatical error correction, using error correction data to train better acceptability models and acceptability scores to guide error correction. It reports improvements on several languages, though the headline 'state-of-the-art' claim is true on only two of the five tested benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported F0.5 scores are arithmetically inconsistent across Tables 3 and 6, so the claimed SOTA gains and 'consistent improvement' ablation cannot be verified as reported.","rationale":"The reader's weakest assumption identified data overlap between the COLA critic and the GEC training corpora as the key risk. That is a real concern about the interpretation of the gains, but it requires additional experiments to test and, even if true, does not necessarily invalidate the reported numbers themselves. A more immediate and decisive problem is that the reported F0.5 values are arithmetically inconsistent with the precision/recall pairs in the paper's own tables. The NaSGEC-Exam row in Table 6 contains F0.5 values that cannot be produced from the listed P and R by any correct computation, strongly suggesting a copying error. Moreover, the GECdi baselines differ between Table 3 and Table 6 for the two benchmarks where the paper claims new state-of-the-art results, so the magnitude of the supposed improvement is not stable. This is an internal inconsistency, not a disagreement with external consensus, and it directly affects the load-bearing empirical evidence for 'consistent enhancement' and for 'new best F0.5' on MuCGEC and CoNLL-14. A conditional verdict is appropriate pending corrected, reproducible tables; the central mechanism could still be valid, but the current write-up does not support it as presented.","tokens_in":9325,"tokens_out":6364,"duration_ms":76083,"concrete_test":"Recompute every F0.5 value in Tables 3 and 6 from the listed precision and recall using F0.5 = 1.25*P*R / (0.25*P + R), and flag any mismatch of more than 0.01 points. Specifically check the NaSGEC-Exam row of Table 6 and the GECdi baselines for MuCGEC and CoNLL-14 across Tables 3 and 6. If the printed F0.5 values cannot be reproduced from P and R, request the original evaluation logs and corrected tables before the empirical results can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on F0.5 values that are internally inconsistent. In Table 6, the NaSGEC-Exam row lists F0.5 = 67.33 for GEC+G-CoLA with P = 53.67 and R = 25.13, but the F0.5 formula (1.25*P*R / (0.25*P + R)) yields approximately 43.7, not 67.33. The adjacent GECdi column gives F0.5 = 79.32 with P = 62.7 and R = 31.54, which computes to about 52.3. These impossible values appear copied from the CoNLL-14 row, indicating a transcription error that calls the entire table into question. Separately, the GECdi baseline for MuCGEC is 48.61 in Table 3 but 49.77 in Table 6, and for CoNLL-14 it is 69.6 vs 70.11. The claimed improvement from adding G-Cola therefore varies by up to 1.16 F0.5 points depending on which table is used; the MuCGEC 'new best' margin shrinks from 1.30 to 0.14 points. Because the paper's headline claims depend on these numbers, the central claim is not verifiable from the reported data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes COLA-GEC, a bidirectional framework linking grammatical acceptability judgment (COLA) and grammatical error correction (GEC). It augments COLA training data with GEC error-correction pairs in multiple languages and introduces a dynamic loss that uses a COLA model's judgment of the GEC model's output to reweight the GEC training loss. The authors report state-of-the-art results on MuCGEC and CoNLL-14, consistent improvements in ablations, and an error analysis focusing on punctuation errors. The paper also provides code and detailed corpus statistics.","tokens_in":9631,"tokens_out":7468,"duration_ms":69438,"significance":"The cross-task transfer idea is timely and potentially valuable for both COLA and GEC, especially for low-resource languages. The authors provide reproducibility assets (a code link and corpus statistics) and a concrete ablation design. However, the main empirical claims are currently undermined by internal data inconsistencies and by a training-data overlap that clouds the interpretation of the COLA signal. If these issues are resolved, the framework could be a useful contribution.","major_comments":[{"comment":"The NaSGEC-Exam row in Table 6 reports F0.5 = 67.33 for GEC+G-CoLA with P=53.67 and R=25.13, and F0.5 = 79.32 for GECdi+G-CoLA with P=63.92 and R=32.10. Using the standard F0.5 formula (1.25*P*R)/(0.25*P+R), these values are approximately 43.7 and 52.3, respectively. The reported numbers appear to be copied from the CoNLL-14 row, corrupting the ablation claim of 'substantial gains' on NaSGEC-Exam.","section":"Section 5.4, Table 6"},{"comment":"Other F0.5 values in Table 6 are also arithmetically inconsistent with the reported P/R. For example, the CoNLL-14 base GEC row (P=63.02, R=41.26) yields F0.5 ≈ 57.0, not 54.08; the BEA-19 base row (P=64.41, R=51.11) yields ≈ 61.2, not 61.12; and the MuCGEC GECdi row (P=56.83, R=33.71) yields ≈ 50.0, not 49.77. The authors should either report exact computations or correct the numbers.","section":"Section 5.4, Table 6"},{"comment":"The baseline GECdi F0.5 for MuCGEC is 48.61 in Table 3 but 49.77 in Table 6; for CoNLL-14 it is 69.6 and 70.11, respectively. The claimed improvement from adding G-CoLA therefore changes by up to 1.16 F0.5 points depending on the table. The MuCGEC 'new best' margin shrinks from 1.30 to 0.14 points when using Table 6's baseline. The authors must clarify which table is correct and re-evaluate the SOTA claims.","section":"Section 5.1 vs. Section 5.4"},{"comment":"The COLA critic (G-Cola) is trained on the same GEC corpora used to train and fine-tune the GEC model (e.g., MuCGEC, FCGEC, NaSGEC for Chinese; W&I+LOCNESS, FCE, NUCLE for English; Falko/MERLIN for German). Because the critic has memorized the gold corrections for these training examples, its 'grammaticality' judgments may simply reflect that memorization rather than an independent grammatical signal. To validate the claimed mechanism, the authors should provide an ablation where the COLA model is trained on data disjoint from the GEC model's training data (e.g., a held-out GEC set) and show that the gains persist.","section":"Section 3.1, Tables 8-10"},{"comment":"The abstract and Section 5.2 state that the approach achieves state-of-the-art results on several multilingual benchmarks. In fact, the reported results are state-of-the-art only for MuCGEC and CoNLL-14; on BEA-19 the F0.5 of 74.03 is below Unsupervised GEC (76.5), on FCGEC the F0.5 of 58.6 is below EPOGEC (59.63), and on German the F0.5 of 70.80 is below MultimodelGEC (76.30). The claims should be scoped to the datasets where the method actually sets a new state of the art.","section":"Abstract and Section 5.2"}],"minor_comments":[{"comment":"The COLA_score definition is confusing: when Logits_0 > Logits_1 (the model predicts 'unacceptable'), the score approaches 1, meaning a high score indicates unacceptability. This contradicts the name 'COLA_score' and the text in Section 3.2 that describes the model as producing acceptability scores. Consider renaming it to a 'penalty score' or clarifying the sign convention.","section":"Section 3.2, Eq. (1)"},{"comment":"Some rows in Table 1 have dev/test sizes exceeding the training size (e.g., French: train 500, dev 521, test 521; Icelandic: train 500, dev 1,194, test 1,194). These are likely typos and should be corrected.","section":"Table 1"},{"comment":"There is a typo in 'HuggingFace'sBart-GermanWepretrainedonFalko' — missing spaces around 'Bart-German' and 'We'.","section":"Section 4, German Implementation"},{"comment":"The paper does not report variance or significance tests for the improvements, which is important because several margins are small (e.g., 0.14 F0.5 on MuCGEC when using the Table 6 baseline).","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The data inconsistencies are serious enough that the paper would be rejected if they cannot be corrected, but the underlying idea and the provided experimental scaffolding suggest the paper could be salvaged with a thorough revision that fixes the tables and addresses the training-data overlap concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Real talk: the bidirectional idea is sound and worth testing, but the paper's own numbers are not reliable enough to support the headline claims. The most useful thing you should know: Table 6 contains F0.5 values that are arithmetically impossible with the precision/recall shown. The NaSGEC-Exam rows appear copied from CoNLL-14; CoNLL-14 base GEC reads 54.08 but P=63.02/R=41.26 gives about 57.0; the German row is off by about a point. This doesn't disprove the approach, but it means the 'state-of-the-art' claims rest on unverified data.\n\nWhat is new: the two-directional combination is new as a system. Adding GEC-derived contrastive pairs to COLA training helps on Chinese, English, German, and Arabic; using the resulting critic as a dynamic loss weight for GEC is a clean adaptation of LCWL. The ablation design is reasonable, and the punctuation error analysis is a useful contribution.\n\nThe soft spots are real. First, the numbers above. Second, the critic and the GEC model are trained on the same GEC corpora (Tables 7-10). In the ablation, plain +CoLA often hurts or barely helps, while +G-CoLA helps; that pattern is exactly what you'd expect if the critic's gain comes partly from memorizing the gold corrections. The paper never tests a held-out separation, so the 'shared grammatical knowledge' mechanism is unproven. Third, the SOTA wording is stronger than the tables: on BEA-19, FCGEC, and German the model is behind other published systems. Fourth, gains are small and there are no significance tests; the Acc term in the loss and the GEC-pair filtering thresholds are underspecified.\n\nWho should read it: people working on GEC/COLA and cross-task augmentation. It is a useful systems paper, not a paradigm shift. I'd send it to review, but with a mandatory request for corrected tables, a leakage-controlled experiment, and a reproduced-baseline comparison. If those come back clean, the direction is publishable.","headline":"Useful bidirectional GEC/COLA idea, but impossible F0.5 values in Table 6 and shared critic/GEC training data make the headline claims unverified.","tokens_in":10163,"tokens_out":5723,"would_cite":false,"duration_ms":61750,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grammatical acceptability and error correction improve each other in a single bidirectional framework.","keywords":["grammatical error correction","grammatical acceptability judgment","bidirectional knowledge transfer","dynamic loss","label confidence weighted learning","multilingual evaluation","punctuation error analysis","dual-critic decoding"],"falsifier":"Retrain the G-Cola critic using only GEC error-correction pairs that were withheld from the GEC model's training corpora, keeping everything else fixed, and rerun the MuCGEC, FCGEC, and CoNLL-14 evaluations; if the F0.5 advantage over the dual-critic baseline shrinks or disappears, the central claim of shared-knowledge transfer is not supported.","tokens_in":9114,"feed_emoji":"🔄","tokens_out":5583,"duration_ms":58134,"temperature":0.7,"pith_summary":"The paper tries to show that grammatical acceptability judgment (COLA) and grammatical error correction (GEC) are mutually reinforcing: GEC error-correction pairs can be converted into acceptability training data that improves COLA models, and COLA acceptability signals can be folded into GEC training to push corrections toward acceptable output. It builds a bidirectional framework in which GEC-augmented critic models (G-Cola) are trained with the two tasks' data, and the critic's judgments enter the GEC loss through a dynamic weighting term. If the claim holds, the two tasks stop being developed in isolation and can share grammatical knowledge across languages, which matters because both are core to language-learning technology. The reported results include state-of-the-art F0.5 scores on MuCGEC and CoNLL-14 and consistent COLA gains across English, Chinese, German, and Arabic.","feed_headline":"Grammar-aware training lifts error correction to new best scores","feed_subtitle":"Acceptability scores steer GEC training by a dynamic loss, setting new F0.5 bests on MuCGEC and CoNLL-14.","key_machinery":"The central mechanism is a dynamic loss function built from Label Confidence Weighted Learning: the standard GEC cross-entropy loss at each timestep is multiplied by the square root of the COLA critic's accuracy times a sigmoid-transformed acceptability score, so corrections the critic deems unacceptable incur larger penalties. Carrying the argument on the other side is the data augmentation step that turns GEC error-correction pairs into binary acceptability judgments (corrected sentences labeled acceptable, error sentences unacceptable), which is what lets a single critic share grammatical knowledge between the tasks. The GEC model is then plugged into a dual-critic decoding framework, where the acceptability signal is combined with existing critics during inference.","core_discovery":"The paper claims that a bidirectional knowledge transfer between grammatical acceptability and grammatical error correction improves both tasks. On the acceptability side, converting GEC error-correction pairs into binary acceptability labels and adding them to existing COLA corpora improves accuracy and Matthews correlation across multiple languages, including a multilingual model that surpasses the previous state-of-the-art on the MELA benchmark. On the correction side, a COLA critic scores each candidate correction, and the score is multiplied into a label-confidence-weighted cross-entropy loss so that grammatically questionable corrections are penalized more heavily; combining this G-Cola GEC model with a dual-critic decoding framework yields new best F0.5 results on MuCGEC (49.91) and CoNLL-14 (70.84). The paper also reports an error analysis indicating that punctuation errors are a persistent bottleneck.","pith_inferences":["A direct consequence the paper leaves implicit: if acceptability signals transfer across languages, a critic trained on one language's GEC data might bootstrap GEC training for another language, lowering data requirements for low-resource GEC.","The dynamic loss depends on the critic's accuracy term, so one could test a curriculum that iteratively retrains the critic on the GEC model's own corrected outputs, potentially compounding gains.","Because the critic and GEC model share training corpora, a clean test of the mechanism would hold out entire GEC datasets from the critic's training and check whether the MuCGEC and CoNLL-14 gains survive; the paper does not perform this separation.","The punctuation bottleneck suggests a targeted synthetic augmentation: generating punctuation-specific error-correction pairs (or training a dedicated punctuation head) might recover the missing F0.5 more directly than generic acceptability fine-tuning."],"forward_implications":["GEC corpora can serve as cheap acceptability-judgment training data, improving COLA accuracy and MCC for English, Chinese, German, and Arabic without new linguistic annotation.","Acceptability-guided training raises GEC precision and F0.5 on CoNLL-14 and MuCGEC enough to set new state-of-the-art scores.","The benefit transfers to a low-resource language: the German G-Cola GECdi model far surpasses the text-only German rule/baseline system.","Ablations show that the gains are additive: adding GEC-enhanced G-Cola to both a base BART GEC model and a dual-critic GEC model improves F0.5 consistently across Chinese and English test sets.","Punctuation errors are a principal remaining weakness; removing them from BEA-19 dev evaluation raises the model's F0.5 from 72.25 to 74.40."],"supporting_citations":[{"why":"Establishes the original English acceptability corpus and benchmark that the paper augments with GEC data.","marker":"[21]"},{"why":"Provides the dual-critic decoding framework and the strong GEC baseline that COLA signals are added to.","marker":"[32]"},{"why":"Supplies the label confidence weighted learning mechanism whose loss is adapted to carry the COLA score.","marker":"[17]"},{"why":"Provides the multilingual acceptability benchmark and current SOTA model that G-Cola is compared against and surpasses.","marker":"[30]"},{"why":"Supplies the Chinese GEC test set on which the paper reports a new state-of-the-art F0.5.","marker":"[27]"},{"why":"Provides the English shared-task dataset whose dev set is used for the punctuation error analysis and whose test set is a reported benchmark.","marker":"[2]"},{"why":"Provides the Chinese acceptability corpus that is augmented with Chinese GEC data.","marker":"[7]"}],"fun_headline_variants":["Bidirectional grammar transfer lifts COLA and GEC","Dynamic loss uses acceptability scores to improve error correction","Acceptability critic steers GEC to new F0.5 bests on two benchmarks","Mutual knowledge transfer sets state-of-the-art in grammar tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the COLA critic supplies an independent grammaticality signal during GEC training even though it was trained on the same GEC corpora as the GEC model, so the reported transfer gains could be an artifact of data overlap rather than shared grammatical knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Bidirectional grammar transfer lifts COLA and GEC","Dynamic loss uses acceptability scores to improve error correction","Acceptability critic steers GEC to new F0.5 bests on two benchmarks","Mutual knowledge transfer sets state-of-the-art in grammar tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000456,"raw_usage":{"total_tokens":2232,"prompt_tokens":829,"completion_tokens":1403,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":1343}},"tokens_in":445,"tokens_out":1403,"duration_ms":13153,"temperature":1.0,"reasoning_tokens":1343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:00:13.837808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the G-Cola critic using only GEC error-correction pairs that were withheld from the GEC model's training corpora, keeping everything else fixed, and rerun the MuCGEC, FCGEC, and CoNLL-14 evaluations; if the F0.5 advantage over the dual-critic baseline shrinks or disappears, the central claim of shared-knowledge transfer is not supported.","supporting_citations":[{"cited_title":"Transactions of the Association for Computational Linguistics, 7, 625–641 (2019)","cited_arxiv_id":null,"evidence_quote":"Establishes the original English acceptability corpus and benchmark that the paper augments with GEC data."},{"cited_title":"In: Findings of the EMNLP 2023, pp","cited_arxiv_id":null,"evidence_quote":"Provides the dual-critic decoding framework and the strong GEC baseline that COLA signals are added to."},{"cited_title":"In: Proceedings of EMNLP 2024, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the label confidence weighted learning mechanism whose loss is adapted to carry the COLA score."},{"cited_title":"In: Proceedings of ACL 2024, pp","cited_arxiv_id":null,"evidence_quote":"Provides the multilingual acceptability benchmark and current SOTA model that G-Cola is compared against and surpasses."},{"cited_title":"In: Proceedings of NAACL (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the Chinese GEC test set on which the paper reports a new state-of-the-art F0.5."},{"cited_title":"In: Proceedings of BEA,(2019) pp","cited_arxiv_id":null,"evidence_quote":"Provides the English shared-task dataset whose dev set is used for the punctuation error analysis and whose test set is a reported benchmark."},{"cited_title":"Revisiting Acceptability Judgements","cited_arxiv_id":"2305.14091","evidence_quote":"Provides the Chinese acceptability corpus that is augmented with Chinese GEC data."}],"review_version":1}