Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

gec-metrics: A Unified Library for Grammatical Error Correction Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GEC-METRICS unifies ten grammatical error correction metrics and two meta-evaluation frameworks under one interface, and reports new results: first GJG15 scores for LLM-based metrics and an ensemble Spearman of 0.984 on SEEDA-E.

desk verdict A genuinely useful GEC metrics library, but the new LLM meta-evaluation results are about a modified metric, not the original one. read the letter →

arxiv 2505.19388 v1 pith:VIIRZNT4 submitted 2025-05-26 cs.CL

Takumi Goto , Yusuke Sakai , Taro Watanabe This is my paper · ORCID
classification cs.CL
keywords grammaticalerrorcorrectionevaluationmetricsmeta-evaluationunifiedlibraryreproducibilityLLM-basedERRANTSEEDA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the fragmentation of grammatical error correction (GEC) evaluation software—inconsistent interfaces, missing official implementations, and broken legacy code—has made fair system comparisons and meta-evaluation needlessly hard. It introduces GEC-METRICS, a library that reimplements ten public GEC metrics (ERRANT, PT-ERRANT, GoToScorer, GLEU, GREEN, SOME, Scribendi, IMPARA, LLM-S, LLM-E) and two meta-evaluation suites (GJG15 and SEEDA) behind a single API, CLI, and GUI, with verified agreement to original implementations where available. The library also adds analysis tools such as pairwise analysis, and the paper's own meta-evaluations yield the first GJG15 results for LLM-based metrics, showing they do not generalize as well as on SEEDA, and an ensemble of non-LLM metrics reaching Spearman 0.984 on SEEDA-E. The central promise is that a unified implementation makes evaluation transparent, reproducible, and extensible, so that new metrics can be developed and compared without re-deriving baselines.

What carries the argument

The load-bearing machinery is the library's interface design: an abstract Metric class and an abstract MetaEval class that define minimal required methods (e.g., score_sentence()), with every metric and meta-evaluation framework inheriting from them. A Config dataclass separates parameters (e.g., beta, language, model choice) from logic, YAML files make experiment settings reproducible, and the same scoring functions are callable programmatically (Python API), from the command line, or via a GUI. For sentence-level meta-evaluation, the paper also introduces pairwise analysis, which groups system pairs by the difference in human rankings and plots metric agreement, giving a more interpretable view than accuracy or Kendall's tau alone. A second key piece is the modified LLM-based metric pipeline: to make LLM-S and LLM-E applicable beyond SEEDA, the authors exclude surrounding context, change the instruction prompt, select up to five unique hypotheses by frequency among the systems, and use gpt-4o-mini-2024-07-18 instead of the original gpt-4-1106-preview.

What would settle it

Run the original authors' LLM-S and LLM-E implementations (with gpt-4-1106-preview, original prompt, and context) on the SEEDA or GJG15 datasets and compare the resulting system rankings to those produced by GEC-METRICS; a substantial divergence in rankings or scores would show the modifications changed the metric.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a single carefully designed library can faithfully reproduce all publicly available GEC evaluation metrics—verified to be consistent with official implementations—and that using this consistent implementation reveals new empirical facts about the metrics: LLM-based evaluators (GPT-4-S, GPT-4-E, Gemini-S, Qwen2.5-S) correlate well with human judgments on SEEDA but poorly on GJG15, GPT-4-E fails to reproduce the results reported in the original LLM-S/E paper, and averaging the rankings of seven non-LLM metrics yields the best system-level correlation (Spearman 0.984 on SEEDA-E). The library's claim is that these results are trustworthy because all metrics were run under the same interface, with shared configurations, rather than citing scores from disparate sources.

Load-bearing premise

The load-bearing premise is that the changes made to the LLM-based metrics—removing surrounding context, rewording the prompt, picking the five most common corrections, and using a cheaper model—do not change what the metric measures; if they do, the new GJG15 scores and the failure of GPT-4-E to reproduce earlier results would not tell us about the original metric.

Editorial extensions

If this is right

  • Researchers and system builders can compare GEC models by running any of the ten metrics on their own hypotheses rather than citing scores from prior papers, eliminating a major source of unfair and irreproducible evaluation.
  • The first GJG15 results for LLM-based metrics, if confirmed, indicate that strong correlation on SEEDA does not guarantee strong correlation on other meta-evaluation datasets, so LLM evaluators need dataset-specific validation before deployment.
  • The ensemble result (Spearman 0.984 on SEEDA-E) suggests that combining several complementary metrics—readily done in one library—can approach human agreement, pointing toward ensembling as a cheap way to strengthen evaluation.
  • Standardizing metric implementations through one package may accelerate research that uses metrics as components, such as reinforcement learning rewards, minimum Bayes risk decoding, and quality-estimation-based system combination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The modification to select hypotheses by frequency (rather than the original pre-sampled set) could bias LLM-based scores toward majority outputs, possibly masking rare but meaningful corrections; a testable extension would compare frequency-based selection with random selection on SEEDA to measure sensitivity.
  • Because the LLM-based metrics were evaluated with gpt-4o-mini instead of the original gpt-4-1106-preview, the reported GJG15 failure of GPT-4-E might reflect the weaker model rather than the approach; running the original model with the original prompt on GJG15 would separate these factors.
  • The pairwise analysis method is not limited to GEC; it could be applied to evaluate any sentence-level metric against human pairwise judgments in other generation tasks such as machine translation or summarization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces gec-metrics, a unified Python library for grammatical error correction (GEC) evaluation. It implements ten metrics (ERRANT, PT-ERRANT, GoToScorer, GLEU, GREEN, SOME, Scribendi, IMPARA, LLM-S, LLM-E) and two meta-evaluation frameworks (GJG15 and SEEDA), with CLI, Python API, and GUI interfaces, plus analysis and visualization tools. The authors claim that results from gec-metrics are consistent with official implementations for all publicly available metrics (Section 4). They present new meta-evaluation results on GJG15 and SEEDA, including first-time GJG15 scores for LLM-based metrics, an ensemble result, and pairwise analyses. The paper also describes modifications to the LLM-based metrics in Appendix C.

Significance. If the implementation is verified, the library fills a genuine need for a unified, reproducible evaluation framework in GEC, analogous to Hugging Face Evaluate in other fields. The open-source release, API design, meta-evaluation interface, and analysis scripts are valuable contributions. The first public implementations of LLM-based metrics and the new experimental results could be informative, but the strength of these contributions depends on the verification evidence and the careful interpretation of the LLM metric modifications. The paper ships a working library, but the key claim of consistency with official implementations is currently asserted rather than demonstrated.

major comments (3)
  1. [Section 4] The sentence "We have verified that the results obtained using GEC-METRICS are consistent with those from official implementations for all publicly available metrics" is a central claim of the paper, but no verification details are provided. There are no comparison tables, test cases, or scripts, and it is unclear which metrics have official implementations and what quantitative tolerance defines "consistent." Please provide a concrete verification procedure and results, such as a table comparing corpus-level and sentence-level scores from gec-metrics with the original implementations on a fixed set of inputs for each metric.
  2. [Section 5 and Appendix C] The modifications to LLM-S/LLM-E described in Appendix C -- removing contextual sentences, changing the instruction prompt, selecting up to five hypotheses by frequency instead of using the original pre-sampled set, and replacing gpt-4-1106-preview with gpt-4o-mini-2024-07-18 -- mean that the results for GPT-4-E, GPT-4-S, Gemini-S, and Qwen2.5-S in Table 2 are for modified variants, not for the original metrics. Consequently, the statement in Section 5 that "GPT-4-E fails to reproduce the results reported by (Kobayashi et al., 2024a)" is not a valid reproduction claim, because it compares a different model, prompt, context, and sampling setup to the original reported numbers. This conclusion should be removed or supported by an ablation that varies one factor at a time. Table 2 should also label these rows as "modified implementations" to avoid presenting them as the original LLM-S and LLM-E.
  3. [Section 5 and Appendix B] Because IMPARA's original fine-tuned weights are not publicly available, the authors retrained the model, as stated in Appendix B, and Table 2 reports results from this reproduction. The paper should explicitly disclose this in the experimental setup and in the Table 2 caption, and the Section 4 consistency claim should be qualified: it can only apply to metrics with official implementations, not to IMPARA or the modified LLM metrics. This distinction is important given that Table 1 shows substantial variation in previously reported IMPARA results across studies.
minor comments (5)
  1. [Section 5, Metric Ensemble] The ensemble result reporting Spearman 0.984 on SEEDA-E does not specify which metrics were included, how ties in average ranks were handled, or whether the correlation is system-level or sentence-level; please provide these details to make the experiment reproducible.
  2. [Table 2] Table 2 reports correlations without confidence intervals or significance tests; given the small number of systems and the noisiness of system-level correlation, please add bootstrap intervals or otherwise discuss the uncertainty of the reported values.
  3. [Section 4.1] The claim that the library enables "fair system comparisons" by ensuring a consistent implementation is overstated if some metrics are reimplementations (e.g., IMPARA) or modified from the original (LLM-based); consider softening the phrasing to acknowledge the provenance of each implementation.
  4. [Figure 3 and Section 2.1] In the list of n-gram groups, "Over Delete (OD), Over Insert (OD)" contains a typo; the second abbreviation should be "OI" (Over Insert).
  5. [Appendix B] There is a typo "exoerimental settings" in the first line, and the paper should specify exact versions of all dependencies (not just errant) to fully support reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical consistency checks against external code and human meta-evaluation datasets; LLM modifications are a fidelity concern, not a circular derivation.

full rationale

This paper is a systems and empirical contribution rather than a derivation. Its central claims are that GEC-METRICS reproduces official metric implementations and that it can run meta-evaluations against external human-annotation datasets (GJG15 and SEEDA). Neither claim defines a predicted quantity in terms of the data used to produce it: no parameter is fitted to GJG15/SEEDA labels and then reported as a prediction, and no metric score is constructed from the human rankings it is later correlated with. The same-author citations (Goto et al. 2024, 2025) are used for optional analysis and aggregation utilities, not as the load-bearing justification for the library's correctness. The Appendix C modifications to LLM-{S,E} (context removal, prompt change, frequency-based hypothesis selection, and gpt-4o-mini instead of gpt-4-1106-preview) do create a genuine fidelity threat to the claim that GPT-4-E 'fails to reproduce' Kobayashi et al. (2024a), but that is an external-validity and setup-mismatch issue, not a circularity: the GJG15 numbers are still empirical outputs of a stated procedure compared against external human judgments. Therefore no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the validity of meta-evaluation datasets, faithful re-implementation of ten metrics, and the equivalence of modified LLM prompts to the original procedure. No parameters were fitted to data to achieve the reported results.

assumptions (3)
  • domain assumption The meta-evaluation datasets GJG15 and SEEDA provide valid human judgments for comparing GEC metrics.
    Used as ground truth in Section 5 to compute correlations for all metrics.
  • domain assumption The implementations in gec-metrics faithfully reproduce the official implementations of the ten metrics.
    Claimed in Section 4 without detailed evidence in the paper.
  • ad hoc to paper Modifications to LLM-{S,E} in Appendix C preserve the properties of the original metric.
    Load-bearing for the new GJG15 results and the GPT-4-E failure conclusion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of gec-metrics: A Unified Library for Grammatical Error Correction Evaluation." pith.science (2026). https://pith.science/paper/VIIRZNT4

@misc{pith2026250519388,
  author       = {Pith},
  title        = {Pith review of: gec-metrics: A Unified Library for Grammatical Error Correction Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VIIRZNT4}},
  note         = {Machine review of arXiv:2505.19388}
}
read the original abstract

We introduce gec-metrics, a library for using and developing grammatical error correction (GEC) evaluation metrics through a unified interface. Our library enables fair system comparisons by ensuring that everyone conducts evaluations using a consistent implementation. Moreover, it is designed with a strong focus on API usage, making it highly extensible. It also includes meta-evaluation functionalities and provides analysis and visualization scripts, contributing to developing GEC evaluation metrics. Our code is released under the MIT license and is also distributed as an installable package. The video is available on YouTube.

Figures

Figures reproduced from arXiv: 2505.19388 by the authors.

Figure 1
Figure 1. System overview of GEC-METRICS. The sources are sentences containing grammatical errors, the hypotheses are their corrected version, and the ref￾erences are human-corrected sentences. Metric classes support both corpus-level and sentence-level evaluation. The MetaEval classes conducts meta-evaluation of met￾rics, by calculating correlations with human evaluation. These classes also provide analysis and visualize scr… view at source ↗
Figure 2
Figure 2. Examples of input/output for GEC evaluation. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Window-analysis results for IMPARA. The x [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Our modified instruction for LLM-S. experimental cost. We believe that not everyone can afford to use expensive models. C Our Modifications of the LLM-based Metrics As described in Section 5, we have made mod￾ifications to the LLM-based metric proposed by Kobayashi et …
Figure 7
Figure 7. Figure 7: GUI of GEC-METRICS . (a) is for metrics, and (b) is for meta-evaluation, which includes visualization of the analysis. They are actually combined on a single page. tion score assigned to each hypothesis is expanded across all systems that produced it. By selecting corr…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A reference-free GEC evaluation method that combines grammatical error detection pre-training with IMPARA's quality estimator achieves the highest sentence-level correlation with human judgments on SEEDA-S.

  2. Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Using TrueSkill aggregation instead of averaging raises the agreement of existing automatic GEC metrics with human system rankings on SEEDA.

Reference graph

Works this paper leans on

45 extracted references · 8 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. https://doi.org/10.18653/v1/D19-1435 Parallel iterative edit models for local sequence transduction . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing...

    doi:10.18653/v1/d19-1435 2019

  4. [4]

    Ond r ej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. https://aclanthology.org/W13-2201/ Findings of the 2013 W orkshop on S tatistical M achine T ranslation . In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1--...

    work page 2013

  5. [5]

    Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. https://doi.org/10.18653/v1/P17-1074 Automatic annotation and evaluation of error types for grammatical error correction . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793--805, Vancouver, Canada. Association for Computat...

    doi:10.18653/v1/p17-1074 2017

  6. [6]

    Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. https://doi.org/10.1162/coli_a_00478 Grammatical error correction: A survey of the state of the art . Computational Linguistics, pages 643--701

    doi:10.1162/coli_a_00478 2023

  7. [7]

    Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. 2024. https://doi.org/10.18653/v1/2024.emnlp-demo.37 mbrs: A library for minimum B ayes risk decoding . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 351--362, Miami, Florida, USA. Association for Computational L...

    doi:10.18653/v1/2024.emnlp-demo.37 2024

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

Show all 45 references
  1. [9]

    Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. https://aclanthology.org/C16-1079 Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments . In Proceedings of COLING 2016, the 26th International Conference on Computational Lin...

    2016

  2. [10]

    Peiyuan Gong, Xuebo Liu, Heyan Huang, and Min Zhang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.463 Revisiting grammatical error correction evaluation and beyond . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6891--690...

    2022 doi

  3. [11]

    Takumi Goto, Yusuke Sakai, and Taro Watanabe. 2025. https://arxiv.org/abs/2502.09416 Rethinking evaluation metrics for grammatical error correction: Why use a different evaluation process than human? Preprint, arXiv:2502.09416

    2025 arXiv

  4. [12]

    Takumi Goto, Justin Vasselli, and Taro Watanabe. 2024. https://arxiv.org/abs/2412.13110 Improving explainability of sentence-level metrics via edit-level attribution for grammatical error correction . Preprint, arXiv:2412.13110

    2024 arXiv

  5. [13]

    Takumi Gotou, Ryo Nagata, Masato Mita, and Kazuaki Hanawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.188 Taking the correction difficulty into account in grammatical error correction evaluation . In Proceedings of the 28th International Conference on Computational Lin...

    2020 doi

  6. [14]

    Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Edward Gillian. 2015. https://doi.org/10.18653/v1/D15-1052 Human evaluation of grammatical error correction systems . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 461--470, Lisb...

    2015 doi

  7. [15]

    Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. https://proceedings.neurips.cc/paper_files/paper/2006/file/f44ee263952e65b3610b8ba51229d1f9-Paper.pdf Trueskill : A bayesian skill rating system . In Advances in Neural Information Processing Systems, volume 19. MIT Press

    2006

  8. [16]

    Md Asadul Islam and Enrico Magnani. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.239 Is this the end of the gold standard? a straightforward reference-less grammatical error correction metric . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language...

    2021 doi

  9. [17]

    Masahiro Kaneko and Naoaki Okazaki. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.619 Reducing sequence length by predicting edit spans with large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10017--1002...

    2023 doi

  10. [18]

    Satoru Katsumata and Mamoru Komachi. 2020. https://doi.org/10.18653/v1/2020.aacl-main.83 Stronger baselines for grammatical error correction using a pretrained encoder-decoder model . In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Compu...

    2020 doi

  11. [19]

    Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024 a . https://aclanthology.org/2024.bea-1.6/ Large language models are state-of-the-art evaluator for grammatical error correction . In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Ap...

    2024

  12. [20]

    Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024 b . https://doi.org/10.1162/tacl_a_00676 Revisiting meta-evaluation for grammatical error correction . Transactions of the Association for Computational Linguistics, 12:837--855

    2024 doi

  13. [21]

    Shota Koyama, Ryo Nagata, Hiroya Takamura, and Naoaki Okazaki. 2024. https://aclanthology.org/2024.inlg-main.25/ n-gram F -score for evaluating grammatical error correction . In Proceedings of the 17th International Natural Language Generation Conference, pages 303--313, Tokyo...

    2024

  14. [22]

    Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoaki Okazaki. 2023. https://doi.org/10.18653/v1/2023.bea-1.18 Exploring effectiveness of GPT -3 in grammatical error correction: A study on performance and controllability in prompt-based methods . In Proceedings of the 18th Wor...

    2023 doi

  15. [23]

    Koki Maeda, Masahiro Kaneko, and Naoaki Okazaki. 2022. https://aclanthology.org/2022.coling-1.316 IMPARA : Impact-based metric for GEC using parallel data . In Proceedings of the 29th International Conference on Computational Linguistics, pages 3578--3588, Gyeongju, Republic o...

    2022

  16. [24]

    Courtney Napoles, Maria N a dejde, and Joel Tetreault. 2019. https://doi.org/10.1162/tacl_a_00282 Enabling robust grammatical error correction in new domains: Data sets, metrics, and analyses . Transactions of the Association for Computational Linguistics, 7:551--566

    2019 doi

  17. [25]

    Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. https://doi.org/10.3115/v1/P15-2097 Ground truth for grammatical error correction metrics . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Internat...

    2015 doi

  18. [26]

    Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2016. https://arxiv.org/abs/1605.02592 Gleu without tuning . Preprint, arXiv:1605.02592

    2016 arXiv

  19. [27]

    Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. https://doi.org/10.3115/v1/W14-1701 The C o NLL -2014 shared task on grammatical error correction . In Proceedings of the Eighteenth Conference on Computational Na...

    2014 doi

  20. [28]

    Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. https://aclanthology.org/W13-3601/ The C o NLL -2013 shared task on grammatical error correction . In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Sha...

    2013

  21. [29]

    Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. https://doi.org/10.18653/v1/2020.bea-1.16 GECT o R -- grammatical error correction: Tag, not rewrite . In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building ...

    2020 doi

  22. [30]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024....

    2024 arXiv

  23. [31]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  24. [32]

    Muhammad Reza Qorib and Hwee Tou Ng. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.785 System combination via quality estimation for grammatical error correction . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12746--12759...

    2023 doi

  25. [33]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwe...

    2025 arXiv

  26. [34]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  27. [35]

    Vyas Raina and Mark Gales. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-short.12 Minimum B ayes' risk decoding for system combination of grammatical error correction systems . In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd...

    2023 doi

  28. [36]

    Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. https://doi.org/10.18653/v1/2021.acl-short.89 A simple recipe for multilingual grammatical error correction . In Proceedings of the 59th Annual Meeting of the Association for Computatio...

    2021 doi

  29. [37]

    Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2017. https://aclanthology.org/I17-2062/ Grammatical error correction with neural reinforcement learning . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers),...

    2017

  30. [38]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lil...

    2025 arXiv

  31. [39]

    Leandro Von Werra, Lewis Tunstall, Abhishek Thakur, Sasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, and Helen Ngo. 2022. https://doi.org/10.18653/v1/2022.emnlp-demos.13 Evaluate & evaluation on the hub: Better best practices for d...

    2022 doi

  32. [40]

    Willard and Rémi Louf

    Brandon T. Willard and Rémi Louf. 2023. https://arxiv.org/abs/2307.09702 Efficient guided generation for large language models . Preprint, arXiv:2307.09702

    2023 arXiv

  33. [41]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 doi

  34. [42]

    Ryoma Yoshimura, Masahiro Kaneko, Tomoyuki Kajiwara, and Mamoru Komachi. 2020. https://doi.org/10.18653/v1/2020.coling-main.573 SOME : Reference-less sub-metrics optimized for manual evaluations of grammatical error correction . In Proceedings of the 28th International Confere...

    2020 doi

  35. [43]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf Bartscore: Evaluating generated text as text generation . In Advances in Neural Information Processing Systems, volume 34, p...

    2021

  36. [44]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  37. [45]

    Yike Zhao, Xiaoman Wang, Yunshi Lan, and Weining Qian. 2025. https://aclanthology.org/2025.coling-demos.5/ U nified GEC : Integrating grammatical error correction approaches for multi-languages with a unified framework . In Proceedings of the 31st International Conference on C...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.