Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that automatic grammatical error correction (GEC) metrics should be ranked using the same aggregation procedure humans use—TrueSkill on sentence-level pairwise comparisons—rather than by averaging sentence-level scores…

desk verdict Clear procedural fix for GEC evaluation with real empirical gains on SEEDA, though statistical evidence is thin and the core pairwise-conversion assumption needs direct validation. read the letter →

arxiv 2502.09416 v2 pith:YETMAQ2Z submitted 2025-02-13 cs.CL

Takumi Goto , Yusuke Sakai , Taro Watanabe This is my paper · ORCID
classification cs.CL
keywords grammaticalerrorcorrectionautomaticevaluationmetricsTrueSkillpairwisecomparisonsystemrankingmeta-evaluationSEEDAsentence-levelscores
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a simple procedural mismatch explains why automatic grammatical error correction (GEC) metrics appear to disagree with human judgments. Humans rank GEC systems by collecting sentence-level pairwise comparisons and aggregating them with a rating algorithm such as TrueSkill, while automatic metrics average sentence-level scores into corpus-level numbers and sort those. The paper proposes converting each metric's sentence-level scores into pairwise comparisons and running the same TrueSkill aggregation, and shows on the SEEDA benchmark that this improves correlation with human rankings for most of the tested metrics, sometimes substantially. If right, the result means existing metrics such as ERRANT and IMPARA were underestimated in prior meta-evaluations, and that future metrics should be designed to support accurate sentence-level relative comparisons.

What carries the argument

The central object is the aggregation step: instead of averaging per-sentence absolute scores and sorting, the method converts each metric's sentence-level scores into pairwise comparison results for every pair of systems on the same source sentence, then aggregates those comparisons with TrueSkill, the same Bayesian rating algorithm used to produce the human rankings in SEEDA. For N systems this yields N(N−1) comparisons per sentence, including ties. The mechanism works only to the extent that sentence-level scores are reliable enough that 'higher score on the same source' is a proxy for 'better correction,' which is why metrics trained for relative comparison, such as IMPARA, benefit most.

What would settle it

Take a meta-evaluation benchmark that records human pairwise judgments at the sentence level, and check how often the metric's higher-scored output loses the human pairwise comparison; if the metric's sentence-level score order disagrees with human pairwise choices on a substantial share of sentences, the proposed TrueSkill aggregation cannot be trusted to reveal human preferences.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the evaluation procedure, not just the metric, determines how well automatic GEC evaluation matches human preference. Using the SEEDA benchmark, whose human rankings were produced by applying TrueSkill to pairwise comparisons, the paper derives pairwise comparison outcomes from sentence-level metric scores and then applies TrueSkill to those outcomes. Across edit-based, n-gram-based, and sentence-level metrics, most show higher Pearson and Spearman correlations with human rankings under this procedure than under the conventional average-and-sort procedure. IMPARA in the SEEDA-S +Fluency setting exceeds the reported GPT-4 evaluator, and ERRANT improves by more than 0.2 in several configurations; n-gram metrics such as GLEU+ and GREEN do not improve, which the paper traces to unreliable sentence-level scores for short sentences. The paper's core statement is not that TrueSkill is the right aggregator, but that automatic evaluation should mirror whatever aggregation method the human evaluation used.

Load-bearing premise

The method assumes that comparing numerical metric scores for the same source sentence is equivalent to a human pairwise judgment, so when a metric gives a higher score to one correction than another, the higher-scored correction really is the better one; the paper offers only indirect evidence for this from one benchmark, with no per-sentence validation.

Editorial extensions

If this is right

  • Meta-evaluations that keep the average-and-sort procedure understate how well existing edit-based and sentence-level metrics align with human preference.
  • Metrics such as ERRANT and IMPARA become more competitive with, and in some settings surpass, LLM-based evaluators such as GPT-4 when evaluated the same way humans are ranked.
  • Developing sentence-level relative metrics, trained or tuned on pairwise comparisons, becomes a priority over corpus-level aggregate design.
  • The same principle transfers to other meta-evaluation datasets: follow whatever aggregation method the human evaluation used, such as Expected Wins, if that is what the dataset relies on.
  • n-gram metrics need sentence-level calibration before they can benefit from relative aggregation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to train or fine-tune GEC metrics directly on pairwise preference data and compare their TrueSkill-based rankings against those of metrics trained on absolute scores; the paper's reasoning predicts the pairwise-trained metric will win.
  • The pairwise conversion could amplify a metric's systematic sentence-level biases instead of removing them; checking agreement between metric score order and human pairwise judgment per sentence would tell whether the improvement generalizes beyond SEEDA.
  • The implicit recommendation for GEC researchers is to publish all compared system outputs, since the relative ranking method requires re-running all systems rather than citing previously reported corpus scores; this is a reproducibility-friendly side effect the paper notes explicitly.
  • If the same aggregation logic applies elsewhere, multilingual GEC and other sentence-level generation tasks with pairwise human evaluation could adopt the same procedure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes that automatic grammatical error correction (GEC) evaluation should aggregate sentence-level metric scores using the same procedure as human evaluation, specifically by converting metric scores into pairwise comparisons and applying TrueSkill, rather than averaging absolute scores across the corpus. The authors evaluate seven existing metrics on the SEEDA benchmark under both traditional averaging/summing and their proposed TrueSkill-based aggregation, reporting improved correlation with human rankings for most metrics, and finding that some BERT-based metrics (e.g., IMPARA) outperform reported GPT-4 evaluator results. The method is implemented in the open-source gec-metrics library.

Significance. The central observation—that human and automatic evaluation differ not only in the score source but also in the aggregation procedure, and that aligning this procedure can change system rankings—is an important and largely overlooked point in GEC evaluation. If the result holds, it would improve the practical utility of existing metrics without requiring new metric design, and would provide a concrete recommendation for future meta-evaluations. The paper is also useful as a reproducibility-oriented contribution: Appendix A gives detailed experimental settings, and the integration into gec-metrics lowers the barrier for adoption. The main weakness is that the load-bearing assumption—that numeric comparison of metric sentence-level scores faithfully represents human pairwise preference—is not directly validated; the reported correlation improvements are therefore suggestive but not conclusive.

major comments (4)
  1. [Section 2.2, Figure 1, Appendix A] The proposed procedure rests on converting sentence-level metric scores into pairwise comparison results by simple numeric comparison, as illustrated with the sentence scores 0.8, 0.7, and 0.9. This assumption is not validated at the sentence level. For edit-based metrics (ERRANT, PT-ERRANT), Appendix A states that when multiple references are available, the reference yielding the highest F0.5 is selected per sentence. Consequently, different systems may be scored against different references, and a higher score does not necessarily correspond to a better correction from a human perspective. The paper should provide direct per-sentence evidence that metric score differences align with human pairwise preferences, e.g., by computing pairwise agreement between metric decisions and SEEDA-S human judgments before any aggregation. Without such validation, the reported improvements could arise from the aggregator rather than from resolving a genuine procedural gap.
  2. [Section 2.2, Section 3.3, Table 1] Both the gold-standard human ranking and the proposed automatic ranking use TrueSkill. Because TrueSkill includes priors, tie handling, and rating dynamics, the shared aggregator can inflate rank correlations independently of the quality of the pairwise signals derived from metric scores. The paper does not control for this potential confound. A control experiment that applies the same pairwise-conversion to permuted or random sentence scores, or that swaps TrueSkill for another rating algorithm (e.g., Expected Wins or Bradley-Terry) for both human and automatic rankings, would clarify whether the improvement is due to procedural alignment or to the statistical properties of TrueSkill itself.
  3. [Table 1, Section 3.2] The meta-evaluations are based on a small number of systems (14 in the +Fluency configuration, fewer in Base), and no significance tests, confidence intervals, or bootstrap analyses are reported. Many of the observed differences, particularly for n-gram metrics (GLEU+, GREEN) where changes are small or even negative, and for edit-based metrics in the +Fluency column, are likely within noise. For example, GLEU+ Spearman drops from 0.902 to 0.846 in SEEDA-S Base, while GREEN improves slightly in one setting and worsens in others. The claim that the proposed method improves results for most metrics should be supported by error bars or a significance test, and the discussion should more honestly reflect the n-gram and +Fluency cases.
  4. [Section 3.3, Table 1, GPT-4 rows] The reported GPT-4 evaluation results are taken from Kobayashi et al. (2024a) and may not be directly comparable because of differences in prompts, decoding settings, or the exact TrueSkill implementation. The paper compares IMPARA and ERRANT against these externally reported numbers to support the claim that BERT-based metrics can outperform GPT-4. To make this comparison fair, the authors should either run GPT-4 under the exact same pairwise-conversion and TrueSkill pipeline used for the other metrics, or clearly state the limitations of comparing numbers from potentially different protocols.
minor comments (6)
  1. [Abstract] The phrase 'improves results for the most of metrics' should be corrected to 'for most of the metrics' or 'for the majority of metrics'.
  2. [Section 4] The text 'IMAPARA' is a typo and should be 'IMPARA'.
  3. [Section 1] 'integratedGEC-METRICS' is missing a space; it should read 'integrated into GEC-METRICS' or similar.
  4. [Table 1] The labels 'w/o TrueSkill' and 'w/ TrueSkill' are ambiguous; consider using 'conventional (average/sum)' and 'proposed (TrueSkill)' for clarity.
  5. [Section 3.1, GREEN description] The sentence 'GREEN uses the reference that yields the highest score is selected for each sentence' is ungrammatical; it should be 'GREEN selects the reference that yields the highest score for each sentence.'
  6. [Figure 2] The figure captions and legend are small and difficult to read, and the window-analysis method is only briefly described in Section 3.2; consider expanding the caption and using larger fonts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the proposed TrueSkill aggregation is an empirical procedure comparison, not a construction that encodes the SEEDA human rankings.

full rationale

The paper's derivation chain is: (1) SEEDA human gold rankings are produced by TrueSkill on human pairwise comparisons; (2) the paper converts each metric's sentence-level absolute scores into pairwise comparison results by numeric comparison; (3) it applies TrueSkill to those pseudo pairwise comparisons; and (4) it measures correlation with the SEEDA human rankings. None of these steps is definitionally circular. The metric scores that feed the proposed ranking are not derived from the SEEDA human rankings, and no parameter is fitted to SEEDA labels. The numeric-comparison conversion is an empirical assumption about metric scale, not a construction that forces the target correlation. The shared TrueSkill aggregator could inflate agreement through algorithmic artifacts, but that is a validity confound, not an equivalence by construction; the paper explicitly says that if a different meta-evaluation dataset used Expected Wins, one should use that instead, showing the aggregator is not secretly encoding SEEDA's human judgments. The only self-citations (Goto et al. 2025) are to the authors' GEC-METRICS implementation library used to run existing metrics; they are not load-bearing for the central claim, which rests on independent SEEDA human judgments. Therefore no specific circular step is established.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new free parameters or entities. It relies on three domain assumptions: the TrueSkill-aggregated human ranking as gold standard, the validity of converting sentence-level scores to pairwise comparisons, and the representativeness of the 14-system SEEDA benchmark. Metric-specific thresholds are imported from the original metric papers.

free parameters (3)
  • SOME sub-metric weights = 0.55, 0.43, 0.02
    Adopted from Yoshimura et al. (2020) and not fitted to SEEDA; these weights determine the SOME sentence scores that feed pairwise comparisons.
  • IMPARA BERT similarity threshold = 0.9
    Used in the paper's reimplementation of IMPARA; taken from Maeda et al. (2022), not tuned on SEEDA, but it changes which pairs count as similar.
  • Scribendi Score matching threshold = 0.8
    Adopted from Islam and Magnani (2021); controls the Levenshtein/token-sort ratio cutoff in the reference-free metric.
assumptions (3)
  • domain assumption Human evaluation rankings produced by TrueSkill on pairwise comparison results are the correct target (SEEDA gold standard).
    Section 2.1 establishes SEEDA's human ranking as the benchmark; the paper does not justify why TrueSkill-aggregated human preferences should be the objective over other aggregation schemes.
  • domain assumption Sentence-level metric scores can be converted into pairwise comparison results by simple numeric comparison.
    Section 2.2 and Figure 1 compare scores 0.8, 0.7, and 0.9 directly; this assumes the score scale is meaningful at the sentence level and that ordering corresponds to human preference.
  • domain assumption SEEDA with 14 systems is representative enough to support a general recommendation about evaluation aggregation.
    Section 3.2; no error bars or multiple benchmarks support generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?." pith.science (2026). https://pith.science/paper/YETMAQ2Z

@misc{pith2026250209416,
  author       = {Pith},
  title        = {Pith review of: Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YETMAQ2Z}},
  note         = {Machine review of arXiv:2502.09416}
}
read the original abstract

One of the goals of automatic evaluation metrics in grammatical error correction (GEC) is to rank GEC systems such that it matches human preferences. However, current automatic evaluations are based on procedures that diverge from human evaluation. Specifically, human evaluation derives rankings by aggregating sentence-level relative evaluation results, e.g., pairwise comparisons, using a rating algorithm, whereas automatic evaluation averages sentence-level absolute scores to obtain corpus-level scores, which are then sorted to determine rankings. In this study, we propose an aggregation method for existing automatic evaluation metrics which aligns with human evaluation methods to bridge this gap. We conducted experiments using various metrics, including edit-based metrics, n-gram based metrics, and sentence-level metrics, and show that resolving the gap improves results for the most of metrics on the SEEDA benchmark. We also found that even BERT-based metrics sometimes outperform the metrics of GPT-4. The proposed ranking method is integrated gec-metrics.

Figures

Figures reproduced from arXiv: 2502.09416 by the authors.

Figure 1
Figure 1. An overview of current human and automatic [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. shows the results of the window anal￾ysis for IMPARA and ERRANT measured on SEEDA-S and SEEDA-E, respectively. From Fig￾ure 2a, it can be seen that IMPARA particularly aligns with human evaluation in the lower ranks. The Pearson correlation coefficient also showed an improvement in the evaluation results for the top systems as well. Since the top systems include GEC systems that are largely rewritten, such as GPT-3.… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A reference-free GEC evaluation method that combines grammatical error detection pre-training with IMPARA's quality estimator achieves the highest sentence-level correlation with human judgments on SEEDA-S.

  2. gec-metrics: A Unified Library for Grammatical Error Correction Evaluation

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A new open-source library unifies ten GEC evaluation metrics and meta-evaluation frameworks, plus new empirical results including an ensemble that reaches 0.984 Spearman on SEEDA-E.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019. https://doi.org/10.18653/v1/D19-1435 Parallel iterative edit models for local sequence transduction . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing...

    doi:10.18653/v1/d19-1435 2019

  4. [4]

    Ond r ej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. https://aclanthology.org/W13-2201/ Findings of the 2013 W orkshop on S tatistical M achine T ranslation . In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1--...

    2013

  5. [5]

    Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. https://doi.org/10.18653/v1/P17-1074 Automatic annotation and evaluation of error types for grammatical error correction . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793--805, Vancouver, Canada. Association for Computat...

    doi:10.18653/v1/p17-1074 2017

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. https://aclanthology.org/C16-1079 Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pages 825--835, Osaka, Japan. The COLING 2016 Organ...

    2016

  8. [8]

    Peiyuan Gong, Xuebo Liu, Heyan Huang, and Min Zhang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.463 Revisiting grammatical error correction evaluation and beyond . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6891--6902, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics

    doi:10.18653/v1/2022.emnlp-main.463 2022

Show all 32 references
  1. [9]

    Takumi Goto, Yusuke Sakai, and Taro Watanabe. 2025. https://arxiv.org/abs/2505.19388 gec-metrics: A unified library for grammatical error correction evaluation . Preprint, arXiv:2505.19388

    2025 arXiv

  2. [10]

    Takumi Gotou, Ryo Nagata, Masato Mita, and Kazuaki Hanawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.188 Taking the correction difficulty into account in grammatical error correction evaluation . In Proceedings of the 28th International Conference on Computational Lin...

    2020 doi

  3. [11]

    Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Edward Gillian. 2015. https://doi.org/10.18653/v1/D15-1052 Human evaluation of grammatical error correction systems . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 461--470, Lisb...

    2015 doi

  4. [12]

    Ralf Herbrich, Tom Minka, and Thore Graepel. 2006. https://proceedings.neurips.cc/paper_files/paper/2006/file/f44ee263952e65b3610b8ba51229d1f9-Paper.pdf Trueskill : A bayesian skill rating system . In Advances in Neural Information Processing Systems, volume 19. MIT Press

    2006

  5. [13]

    Md Asadul Islam and Enrico Magnani. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.239 Is this the end of the gold standard? a straightforward reference-less grammatical error correction metric . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language...

    2021 doi

  6. [14]

    Masahiro Kaneko and Naoaki Okazaki. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.619 Reducing sequence length by predicting edit spans with large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10017--1002...

    2023 doi

  7. [15]

    Satoru Katsumata and Mamoru Komachi. 2020. https://doi.org/10.18653/v1/2020.aacl-main.83 Stronger baselines for grammatical error correction using a pretrained encoder-decoder model . In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Compu...

    2020 doi

  8. [16]

    Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024 a . https://aclanthology.org/2024.bea-1.6/ Large language models are state-of-the-art evaluator for grammatical error correction . In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Ap...

    2024

  9. [17]

    Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024 b . https://doi.org/10.1162/tacl_a_00676 Revisiting meta-evaluation for grammatical error correction . Transactions of the Association for Computational Linguistics, 12:837--855

    2024 doi

  10. [18]

    Shota Koyama, Ryo Nagata, Hiroya Takamura, and Naoaki Okazaki. 2024. https://aclanthology.org/2024.inlg-main.25/ n-gram F -score for evaluating grammatical error correction . In Proceedings of the 17th International Natural Language Generation Conference, pages 303--313, Tokyo...

    2024

  11. [19]

    Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoaki Okazaki. 2023. https://doi.org/10.18653/v1/2023.bea-1.18 Exploring effectiveness of GPT -3 in grammatical error correction: A study on performance and controllability in prompt-based methods . In Proceedings of the 18th Wor...

    2023 doi

  12. [20]

    Koki Maeda, Masahiro Kaneko, and Naoaki Okazaki. 2022. https://aclanthology.org/2022.coling-1.316 IMPARA : Impact-based metric for GEC using parallel data . In Proceedings of the 29th International Conference on Computational Linguistics, pages 3578--3588, Gyeongju, Republic o...

    2022

  13. [21]

    Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. https://doi.org/10.3115/v1/P15-2097 Ground truth for grammatical error correction metrics . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Internat...

    2015 doi

  14. [22]

    Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2016. https://arxiv.org/abs/1605.02592 Gleu without tuning . Preprint, arXiv:1605.02592

    2016 arXiv

  15. [23]

    Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. https://doi.org/10.3115/v1/W14-1701 The C o NLL -2014 shared task on grammatical error correction . In Proceedings of the Eighteenth Conference on Computational Na...

    2014 doi

  16. [24]

    Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. https://aclanthology.org/W13-3601/ The C o NLL -2013 shared task on grammatical error correction . In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Sha...

    2013

  17. [25]

    Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. https://doi.org/10.18653/v1/2020.bea-1.16 GECT o R -- grammatical error correction: Tag, not rewrite . In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building ...

    2020 doi

  18. [26]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  19. [27]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  20. [28]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  21. [29]

    Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. https://doi.org/10.18653/v1/2021.acl-short.89 A simple recipe for multilingual grammatical error correction . In Proceedings of the 59th Annual Meeting of the Association for Computatio...

    2021 doi

  22. [30]

    Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2014. https://doi.org/10.3115/v1/W14-3301 Efficient elicitation of annotations for human evaluation of machine translation . In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 1--11, Baltimore, M...

    2014 doi

  23. [31]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 doi

  24. [32]

    Ryoma Yoshimura, Masahiro Kaneko, Tomoyuki Kajiwara, and Mamoru Komachi. 2020. https://doi.org/10.18653/v1/2020.coling-main.573 SOME : Reference-less sub-metrics optimized for manual evaluations of grammatical error correction . In Proceedings of the 28th International Confere...

    2020 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.