Pith. sign in

REVIEW 3 major objections 7 minor 63 references

WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays

T0 review · 3 major / 7 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A modular LLM pipeline can score argumentative essays near human raters and produce feedback teachers largely endorse.

desk verdict Solid modular AWE system with a real evaluation bias: feedback approval rates were measured on the same 40 essays used to pick Claude 3.7. read the letter →

arxiv 2607.14524 v1 pith:MOWJLGMW submitted 2026-07-16 cs.AI cs.CL

Adnan Labib , Yixuan Huang , Jiahui Wu , John Maurice Gayed , Zheng Yuan , Qiao Wang This is my paper · ORCID
classification cs.AIcs.CL
keywords automatedwritingevaluationargumentativeessaysLLMscoringfeedbackgenerationmodularsystemsupervisedfine-tuningquadraticweightedkappagrammaticalerrorcorrection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that automated writing evaluation is better built as three separate modules—scoring, surface-level correction, and deep-level commentary—rather than as one model doing everything. On a private benchmark of 480 argumentative essays with official human scores, the scoring module reports a quadratic weighted kappa of 0.84 and a root mean square error of 0.44 on a 0-5 scale, which it calls state of the art. Human teachers approved 96.14% of surface-level edits, 93.03% of macro structural comments, and 94.69% of micro comments in a 40-essay sample. The paper also argues that for feedback generation, supervised fine-tuning was less effective than directly prompting a strong LLM, because the fine-tuned models produced unusable output. A sympathetic reader would care because the result suggests students could get immediate, layered, largely accurate feedback and teachers could offload routine correction.

What carries the argument

The load-bearing mechanism is the modular pipeline itself. Scoring uses a fine-tuned LLM with a rubric prompt. Surface-level feedback uses a directly prompted LLM with an explicit anti-overcorrection instruction. Deep-level feedback uses a few-shot prompted LLM split into two pipelines—macro comments aligned to paragraphs and micro comments anchored to highlighted text spans—with categories (argumentation, coherence, clarity, formality, grammar) derived from a thematic analysis of teacher annotations. The module choice is based on pilot comparisons of several LLMs on grammatical error correction and reasoning quality.

What would settle it

Take the 360-essay test subset and re-score it with the prior baseline model under identical conditions; if the baseline's QWK matches 0.84 or the gap reverses, the claimed state-of-the-art advantage disappears. Alternatively, have expert teachers mark every error and feedback-worthy span in a random sample of the 40 test essays and compute recall; if recall falls far below the reported 94-96% precision, the feedback module's apparent quality overstates how much it catches.

Watch

Extended reading notes

Core claim

WrAFT's central claim is that a modular architecture—assigning each evaluation task to a specialized model and prompt—yields both reliable scores and comprehensive feedback. For scoring, a fine-tuned commercial LLM trained on 120 essays reaches QWK 0.84 and RMSE 0.44 on 360 held-out essays, beating a prior fine-tuned baseline (QWK 0.78, RMSE 0.57). For feedback, direct prompting of a strong LLM produced 2,049 surface edits of which 96.14% were judged necessary and effective, and 831 deep-level comments of which 93-95% were judged effective, with high inter-rater agreement. The paper further claims that fine-tuned models for deep-level feedback failed outright—one exceeded its context window

Load-bearing premise

The scoring comparison assumes this system and the prior baseline were evaluated on the same partition of the same 480 essays; the paper does not explicitly confirm identical splits.

Editorial extensions

If this is right

  • If the scoring results hold, LLM-based scoring can exceed the QWK of 0.7 that the test's official scoring engine is expected to meet, making automated scoring viable in low-stakes and practice settings.
  • Teachers can shift effort from correcting grammar and mechanics to coaching argumentation and structure, since the system's surface module catches errors with high precision.
  • System designers may skip expensive supervised fine-tuning for feedback tasks and instead invest in prompt engineering and few-shot examples, since directly prompted models outperformed fine-tuned ones here.
  • A free interactive interface means these capabilities are available to students immediately, not just as a research benchmark.
  • The modular design gives institutions an automated reference score that could support consistency across multiple instructors in large writing courses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: the state-of-the-art scoring claim rests on comparability of the 120/360 train-test split with the prior baseline's split; the paper does not state the splits are identical, so the QWK gain from 0.78 to 0.84 could partly reflect partition luck. A direct same-split re-run would settle this.
  • Extension: because feedback evaluation measured precision only, the high approval percentages say nothing about recall—a system that misses most errors could still look good. A gold-standard error audit would quantify how much feedback students actually miss.
  • Extension: the finding that fine-tuned feedback models failed on formatting and context limits suggests that open-source fine-tuning for structured commentary may need an additional format-verification step, which the paper itself notes.
  • Extension: a natural next test is whether the same modular recipe transfers to source-based or genre-different writing, where the current system is explicitly not validated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper presents WrAFT, a modular automated writing evaluation system for argumentative TOEFL essays. It decomposes AWE into three modules: scoring (supervised fine-tuning of GPT-4o and LLaMA-3.3-70B-Instruct on 120 ETS-scored essays; tested on 360 held-out essays), surface-level feedback (direct prompting of GPT-4o for grammatical/mechanical edits and explanations), and deep-level feedback (few-shot prompting of Claude 3.7 for macro paragraph-level comments and micro comments on grammar, clarity, coherence, argumentation, and formality). The scoring module reports QWK 0.84 and RMSE 0.44 (GPT-4o), which the authors claim is state-of-the-art relative to their earlier baseline (Wang and Gayed 2024, QWK 0.78). Human evaluation found 96.14% of surface-level edits necessary and effective, and 93.03% (macro) and 94.69% (micro) of deep-level comments effective or necessary-and-effective. The paper also reports that supervised fine-tuning underperformed direct prompting for feedback generation, and it releases a public UI and code repository.

Significance. If the reported results hold, WrAFT would show that a modular LLM pipeline can score TOEFL argumentative essays with a QWK of 0.84 and RMSE 0.44 and, at the same time, produce surface and deep-level feedback that expert raters largely judge necessary and effective. That outcome is still uncommon in AWE research, which has concentrated on scoring. The paper has real strengths: a within-dataset scoring design with disjoint fine-tuning and test subsets; feedback evaluation by two raters with third-rater adjudication and Gwet's AC1 with confidence intervals; a GEC benchmark used to justify the surface-level model choice; candid reporting of the fine-tuning failures; and a publicly available codebase and UI. The headline claims, however, need qualification: the deep-level feedback approval rates come from a test set that was also used for model selection, the SOTA scoring claim depends on an unverified split-comparability assumption with a single baseline from the authors' own prior work, and the feedback evaluation is precision-only on a small two-prompt dataset. Several of these limitations are acknowledged in §6, but the most consequential one—selection on the evaluation set—is not.

major comments (3)
  1. [§4.3.1–4.3.3, §6] The deep-level feedback validation reuses the same 40-essay test set for model selection and for the reported approval rates. In §4.3.1, the authors compare LLaMA-3.3-70B, GPT-4o, and Claude 3.7 on those 40 essays, note that Claude 3.7 generates 628 comments versus 336/368, perform a qualitative review on three of the same essays, and decide to focus validation exclusively on Claude 3.7. The approval rates in §4.3.2 (93.03% macro) and §4.3.3 (94.69% micro) are then computed on exactly these 40 essays. The evaluation set thus informed model selection, so the reported rates are not held-out characterizations of the deployed pipeline but estimates conditional on a model selected using that same set. §6 acknowledges precision-only evaluation and missing recall but not this selection-on-test reuse. The claim that teachers largely endorse the feedback requires a held-out feedback evaluation on
  2. [§3.1, §4.1, Table 5] The SOTA scoring claim rests on a single baseline—the authors' own prior study, Wang and Gayed (2024). §3.1 states that the same dataset was used, but the paper does not state whether the 120/360 fine-tune/test partition matches the baseline's partition. If the baseline was evaluated on a different split, the QWK improvement from 0.78 to 0.84 in Table 5 could be attributable to split selection rather than model superiority. Since the baseline comes from the same group, the split should be documented or the baseline re-evaluated on the identical test subset. In addition, Gayed (2025), a more recent system built on the same dataset and described in §2.2, is not included in Table 5, so SOTA is established against a single 2024 comparator. The claim should be qualified, e.g., on this dataset and partition, versus the previously reported baseline.
  3. [§3.2, §4.3.1, §5, Introduction contribution 3] The third contribution—that supervised fine-tuning is less effective than direct prompting for feedback generation—is supported only by two specific implementation failures: the fine-tuned GPT-4o exceeded its 8,000-token context window on every inference, and the fine-tuned LLaMA-3.3-70B produced unparseable JSON (§4.3.1). Both are remediable engineering issues (e.g., constraining output length, adding format verification), not evidence of a general property of supervised fine-tuning for non-deterministic output. The fine-tuning configurations are uncontrolled or undocumented (parameters for GPT-4o are unknown), so the SFT-versus-prompting comparison is not a matched one. As stated, the contribution overgeneralizes. I recommend reframing it as a case report of this system's fine-tuning attempts, or adding a controlled comparison in which prompting and fine-tuned variants are matched on o
minor comments (7)
  1. [§4.3.1 vs §4.3.3] The number of Claude 3.7 micro comments is reported as 628 in §4.3.1 and 630 in §4.3.3; also, 596/630 = 94.60%, not the reported 94.69% necessary-and-effective rate. Please reconcile these figures.
  2. [Table 8] With two observed disagreements out of 579 judgments (cells 1 and 1), Gwet's AC1 cannot equal 1.00; it is approximately 0.9965. The reported AC1 = 1.00 with SE = 0.00 and CI [0.99, 1.00] appears inconsistent with the contingency table.
  3. [§4.2] The sentence '1985 edits deemed necessary (96.88%), out of which 1970 were deemed both necessary and effective (96.14%)' is ambiguous: 96.14% is 1970/2049 of all edits, not 1970/1985 (99.24%) of the necessary subset. Please clarify the denominator.
  4. [Appendix D.1] The deployed macro-feedback prompt contains a grammatical error: 'The conclusion part should efficient summaries the essay'. This is worth correcting in a teacher-facing system, and the appendix prompt should match the deployed prompt.
  5. [Appendix B] The scoring fine-tuning prompt includes a '[Complete rubrics here]' placeholder. The actual rubrics should be included or precisely cited for reproducibility, since the scoring and feedback claims depend on rubric-guided prompting.
  6. [§3.2, Table 5] The claim that LLaMA-3.3-70B-Instruct was 'the top model in instruction following' needs a dated citation to the leaderboard snapshot; 'Drawing on findings from these findings' is a wording error. The baseline row in Table 5 should also specify the model and split used in Wang and Gayed (2024).
  7. [§3.1, Table 2] It is not stated whether the 40 feedback-test essays overlap with the 120-essay scoring fine-tune subset or the 360-essay scoring test subset. Clarifying this would help readers assess the independence of the module evaluations.

Circularity Check

1 steps flagged · score 4.0 of 10

Deep-level feedback 'validation' reuses the 40-essay set that selected Claude 3.7; the scoring chain itself is non-circular.

  1. other [§4.3.1–§4.3.3 (Deep-level feedback module: preliminary check, macro and micro validation)]
    "Claude 3.7, however, generated 628 comments. To verify that this was a difference in quality and not just verbosity, the researchers conducted a qualitative review of three randomly selected essays. This comparison confirmed that the additional comments from Claude 3.7 were indeed more fine-grained and pedagogically necessary. Given the intensive human labor required for a full expert evaluation, a decision was made to focus validation efforts exclusively on the feedback from the most promising model, Claude 3.7."

    The 40-essay test subset defined in §3.1 (Table 2) was used twice: §4.3.1 uses it as the model-selection set, comparing comment counts of LLaMA-3.3-70B, GPT-4o, and Claude 3.7 on those exact essays and qualitatively reviewing three of the same essays; §4.3.2–4.3.3 then compute the reported 93.03% macro and 94.69% micro approval rates on exactly the same 40 essays. The reported model was selected for its output on this very set, so the stated human-approval figures characterize the chosen candidate on its own selection set. The evaluation input (the 40 essays) therefore entered the design twice, making the 'validation' of deep-level feedback a selection result rather than an unbiased estimate for unseen essays.

full rationale

The scoring chain is self-contained: 120 fine-tuning and 360 test essays are disjoint subsets of the same ETS dataset (§3.1, Table 2), and QWK 0.84 / RMSE 0.44 are computed against official benchmark scores external to the system; nothing in that derivation inverts its inputs. Surface-level approval (96.14%) rests on independent human judgments of ERRANT-aligned edits, and the surface model choice is justified by a separate GEC benchmark (Labib et al., 2027, BEA-2019 data), which is real external evidence. The only load-bearing reuse of evaluation data is in the deep-level feedback module: §4.3.1 selects Claude 3.7 by comparing the three prompted models on the same 40 essays later used for the reported approval rates, so those figures carry selection-on-test bias — though the human necessity/effectiveness judgments themselves are genuine discriminations (some comments were rated ineffective), so the numbers are not forced to a constant by construction. The SOTA scoring claim compares against Wang and Gayed (2024), the authors' own prior study, without stating that the fine-tune/test splits were identical; if the splits differ, the apparent 0.78→0.84 QWK gain may be partly a split artifact — a comparability caveat, not a tautology. Deep-level prompt categories (Table 4) were derived from the same teacher annotation scheme used to build the fine-tuning data, so category-conformant output is partly by design; only the external necessity/effectiveness ratings provide independent signal. Overall, no equation reduces to its own inputs; the central scoring result has independent content, so the finding is partial circularity (score 4), not a fully forced derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on proprietary ETS data, hand-designed prompts, fine-tuning choices, and human rater judgments rather than formal derivation. There are no invented physical or conceptual entities. The main ledger items are dataset splits, fine-tuning hyperparameters, and the unstated baseline split comparability.

free parameters (3)
  • Scoring fine-tuning subset size = 120 essays
    Chosen by score-based equal sampling; test subset is the remaining 360. This split determines the reported QWK/RMSE and is not theoretically motivated beyond balancing.
  • LLaMA fine-tuning epochs = 8
    Chosen for the LLaMA deep-feedback model; GPT-4o's fine-tuning parameters are unknown black-box defaults (§3.2).
  • Deep-level feedback annotation/test sizes = 90 essays for fine-tuning, 40 for test
    Hand-selected via equal sampling; affects the reliability of feedback approval rates and was partly determined by labor constraints (§3.1).
assumptions (4)
  • domain assumption ETS benchmark scores on the 480 TOEFL essays are an accurate gold standard for argumentative writing quality.
    Used throughout §3.1 and §4.1 as ground truth; if these scores are biased or noisy, the scoring QWK/RMSE claims lose meaning.
  • domain assumption Human teacher judgments of feedback necessity/effectiveness are reliable ground truth without a fully specified rubric.
    §3.3.2 and §4.2–4.3 rely on two raters plus a third for disagreement; no gold standard or recall metric is used.
  • ad hoc to paper The 120/360 train-test split is comparable to the split used by Wang and Gayed (2024) for the SOTA baseline.
    §3.1 and Table 5: the paper states the same dataset was used but never states the baseline split is identical.
  • ad hoc to paper The 40-essay feedback test set is representative enough to validate feedback after the best model was selected on that same set.
    §4.3.1: Claude 3.7 was chosen after a preliminary screen on the same 40 test essays used for final human evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays." pith.science (2026). https://pith.science/paper/MOWJLGMW

@misc{pith2026260714524,
  author       = {Pith},
  title        = {Pith review of: WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOWJLGMW}},
  note         = {Machine review of arXiv:2607.14524}
}
read the original abstract

This study presents WrAFT, a Writing Assessment and Feedback Tool, that delivers both accurate and reliable scores and effective comprehensive feedback to argumentative essays. WrAFT adopts a modular design by dividing automated writing evaluation (AWE) tasks into scoring, surface-level feedback, and deep-level feedback. In building the system, various Large Language Models (LLMs) have been evaluated, including LLaMA-3.3-70B-Instruct, GPT-4o, and Claude 3.7, through both direct prompting and supervised fine-tuning approaches. A proprietary dataset of 480 TOEFL Independent Writing essays with official benchmark scores was utilized. Benchmark-based evaluation shows that WrAFT achieves state-of-the-art performance in scoring, with a quadratic weighted kappa (QWK) of 0.84 and a root mean square error (RMSE) of 0.44 against official scores on a scale of 0-5. Human evaluation of system-generated feedback also reveals high approval ratings: 96.14 percent for surface-level feedback, 93.03 percent for deep-level macro feedback, and 94.69 percent for deep-level micro feedback. An interactive user interface has been developed for the system and is publicly available and free to use.

Figures

Figures reproduced from arXiv: 2607.14524 by the authors.

Figure 1
Figure 1. Example of a teacher comment in MS Word For testing of both surface-level and deep-level feedback, due to the ex￾tensive human labor in evaluating such feedback, only 40 essays were selected 9 [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Final system architecture 18 [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Screenshot of the surface-level feedback page - track changes [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Screenshot of the deep-level feedback page [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 4 canonical work pages

  1. [1]

    2003 , publisher=

    Clueless in Academe: How Schooling Obscures the Life of the Mind , author=. 2003 , publisher=

    2003

  2. [2]

    2005 , publisher=

    Education for Thinking , author=. 2005 , publisher=

    2005

  3. [3]

    2003 , publisher=

    Automated Essay Scoring: A Cross-disciplinary Perspective , author=. 2003 , publisher=

    2003

  4. [4]

    Journal of Technology, Learning, and Assessment , volume=

    Automated Essay Scoring with e-rater V.2 , author=. Journal of Technology, Learning, and Assessment , volume=

  5. [5]

    Automated Essay Scoring: A Cross-disciplinary Perspective , pages=

    Automated Scoring and Annotation of Essays with the Intelligent Essay Assessor , author=. Automated Essay Scoring: A Cross-disciplinary Perspective , pages=. 2003 , publisher=

    2003

  6. [6]

    Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics , pages=

    Modeling Argument Strength in Student Essays , author=. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics , pages=

  7. [7]

    BMC Medical Education , volume=

    Reliability of ChatGPT in Automated Essay Scoring for Dental Undergraduate Examinations , author=. BMC Medical Education , volume=

  8. [8]

    Heliyon , volume=

    ChatGPT's Capabilities in Providing Feedback on Undergraduate Argumentative Writing , author=. Heliyon , volume=

Show all 63 references
  1. [9]

    Computer Assisted Language Learning , volume=

    Effectiveness of Large Language Models in Automated Evaluation of Argumentative Essays , author=. Computer Assisted Language Learning , volume=

  2. [10]

    Asia TEFL , volume=

    Automated Writing Evaluation for ESL Learners: A Case Study of Pigai System , author=. Asia TEFL , volume=

  3. [11]

    arXiv preprint arXiv:2504.05736 , year=

    Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring , author=. arXiv preprint arXiv:2504.05736 , year=

    arXiv

  4. [12]

    Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications , pages=

    Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation , author=. Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications , pages=

  5. [13]

    arXiv preprint arXiv:2410.14202 , year=

    Rationale Behind Essay Scores: Enhancing S-LLM's Multi-Trait Essay Scoring with Rationale Generated by LLMs , author=. arXiv preprint arXiv:2410.14202 , year=

    arXiv

  6. [14]

    arXiv preprint arXiv:2403.06149 , year=

    Can Large Language Models Automatically Score Proficiency of Written Essays? , author=. arXiv preprint arXiv:2403.06149 , year=

    arXiv

  7. [15]

    BMC Medical Research Methodology , volume=

    A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples , author=. BMC Medical Research Methodology , volume=. 2013 , publisher=

    2013

  8. [16]

    British Journal of Mathematical and Statistical Psychology , volume=

    Computing inter-rater reliability and its variance in the presence of high agreement , author=. British Journal of Mathematical and Statistical Psychology , volume=. 2008 , publisher=

    2008

  9. [17]

    Language Teaching Research , volume=

    The value of written corrective feedback for migrant and international students , author=. Language Teaching Research , volume=. 2008 , publisher=

    2008

  10. [18]

    Language Learning , volume=

    The case against grammar correction in L2 writing classes , author=. Language Learning , volume=. 1996 , publisher=

    1996

  11. [19]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    ReFF: Reinforcing Format Faithfulness in Language Models Across Varied Tasks , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  12. [20]

    Evaluating Prompting Strategies for Grammatical Error Correction Based on Language Proficiency

    Zeng, Min and Kuang, Jiexin and Qiu, Mengyang and Song, Jayoung and Park, Jungyeul. Evaluating Prompting Strategies for Grammatical Error Correction Based on Language Proficiency. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Res...

    2024

  13. [21]

    Educational Measurement: Issues and Practice , volume=

    A framework for evaluation and use of automated scoring , author=. Educational Measurement: Issues and Practice , volume=. 2012 , publisher=

    2012

  14. [22]

    International Conference on Artificial Intelligence in Education , pages=

    Automated essay scoring incorporating multi-level semantic features , author=. International Conference on Artificial Intelligence in Education , pages=. 2023 , organization=

    2023

  15. [23]

    Automated Essay Scoring: A Reflection on the State of the Art

    Li, Shengjie and Ng, Vincent. Automated Essay Scoring: A Reflection on the State of the Art. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.18653/v1/2024.emnlp-main.991

    2024 doi

  16. [24]

    , author=

    Grading essays by computer: Progress report. , author=. Proceedings of the invitational Conference on Testing Problems , year=

  17. [25]

    TOEFL11: A corpus of non-native English , volume =

    Blanchard, Daniel and Tetreault, Joel and Higgins, Derrick and Cahill, Aoife and Chodorow, Martin , year =. TOEFL11: A corpus of non-native English , volume =. ETS Research Report Series , doi =

  18. [26]

    and Draxler, R

    Chai, T. and Draxler, R. R. , TITLE =. Geoscientific Model Development , VOLUME =. 2014 , NUMBER =

    2014

  19. [27]

    Physical Therapy , volume=

    The kappa statistic in reliability studies: Use, interpretation, and sample size requirements , author=. Physical Therapy , volume=. 2005 , publisher=

    2005

  20. [28]

    Biometrics , volume=

    The measurement of observer agreement for categorical data , author=. Biometrics , volume=. 1977 , publisher=

    1977

  21. [29]

    The Routledge international handbook of automated essay evaluation , pages=

    Introduction to automated essay evaluation , author=. The Routledge international handbook of automated essay evaluation , pages=. 2024 , publisher=

    2024

  22. [30]

    , author=

    Project Essay Grade: PEG. , author=. 2003 , publisher=

    2003

  23. [31]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  24. [32]

    Assessing Writing , volume=

    Investigating the Application of Automated Writing Evaluation to Chinese Undergraduate English Majors: A Case Study of WriteToLearn , author=. Assessing Writing , volume=. 2015 , publisher=

    2015

  25. [33]

    System , volume=

    L2 Student Engagement with Automated Feedback on Writing , author=. System , volume=. 2021 , publisher=

    2021

  26. [34]

    Ai magazine , volume=

    Automated essay evaluation: The Criterion online writing service , author=. Ai magazine , volume=

  27. [35]

    Computers in Human Behavior , volume=

    Stumping e-rater: challenging the validity of automated essay scoring , author=. Computers in Human Behavior , volume=. 2002 , publisher=

    2002

  28. [36]

    Assessing Writing , volume=

    Automated essay scoring: Psychometric guidelines and practices , author=. Assessing Writing , volume=. 2013 , publisher=

    2013

  29. [37]

    Assessing Writing , volume=

    On the relation between automated essay scoring and modern views of the writing construct , author=. Assessing Writing , volume=. 2013 , publisher=

    2013

  30. [38]

    Assessing Writing , volume=

    State-of-the-art automated essay scoring: Competition, results, and future directions from a United States demonstration , author=. Assessing Writing , volume=. 2014 , publisher=

    2014

  31. [39]

    Assessing Writing , volume=

    When ``the state of the art'' is counting words , author=. Assessing Writing , volume=. 2014 , publisher=

    2014

  32. [40]

    Research Methods in Applied Linguistics , volume=

    Exploring the potential of using an AI language model for automated essay scoring , author=. Research Methods in Applied Linguistics , volume=. 2023 , publisher=

    2023

  33. [41]

    Computers and Education: Artificial Intelligence , volume=

    Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability , author=. Computers and Education: Artificial Intelligence , volume=. 2024 , publisher=

    2024

  34. [42]

    Humanities and Social Sciences Communications , volume=

    Applying large language models for automated essay scoring for non-native Japanese , author=. Humanities and Social Sciences Communications , volume=. 2024 , publisher=

    2024

  35. [43]

    Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) , month = aug, year =

    Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring , author =. Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) , month = aug, year =. doi:10.18653/v1/K17-1017 , pages =

    2017 doi

  36. [44]

    Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =

    Jin, Cancan and He, Ben and Hui, Kai and Sun, Le , editor =. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =. doi:10.18653/v1/P18-1100 , pages =

    doi

  37. [45]

    A New Dataset and Method for Automatically Grading

    Yannakoudakis, Helen and Briscoe, Ted and Medlock, Ben , booktitle =. A New Dataset and Method for Automatically Grading. 2011 , address =

    2011

  38. [46]

    Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , month = jul, year =

    Grammar Error Correction Using Pseudo-Error Sentences and Domain Adaptation , author =. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , month = jul, year =

  39. [47]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =

    Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction , author =. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =. doi:10.18653/v1/P17-1074 , pages =

    doi

  40. [48]

    Transactions of the Association for Computational Linguistics , volume =

    Grammar Error Correction in Morphologically Rich Languages: The Case of Russian , author =. Transactions of the Association for Computational Linguistics , volume =. 2019 , address =. doi:10.1162/tacl_a_00251 , pages =

    2019 doi

  41. [49]

    Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019) , month = nov, year =

    Minimally-Augmented Grammatical Error Correction , author =. Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019) , month = nov, year =. doi:10.18653/v1/D19-5546 , pages =

    2019 doi

  42. [50]

    2024 , address =

    Katinskaia, Anisia and Yangarber, Roman , booktitle =. 2024 , address =

    2024

  43. [51]

    Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016) , month = nov, year =

    A Neural Approach to Automated Essay Scoring , author =. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016) , month = nov, year =

    2016

  44. [52]

    2025 , note =

    Anonymous , title =. 2025 , note =

    2025

  45. [53]

    Bryant, Christopher and Felice, Mariano and Andersen,. The. Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications , month =. 2019 , address =. doi:10.18653/v1/W19-4406 , pages =

    2019 doi

  46. [54]

    Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction , author=. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=. 2017 , organization=

    2017

  47. [55]

    arXiv preprint arXiv:1605.02592 , year=

    GLEU Without Tuning , author=. arXiv preprint arXiv:1605.02592 , year=

    arXiv

  48. [56]

    Revisiting Grammatical Error Correction Evaluation and Beyond

    Gong, Peiyuan and Liu, Xuebo and Huang, Heyan and Zhang, Min. Revisiting Grammatical Error Correction Evaluation and Beyond. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.463

    2022 doi

  49. [57]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , month = nov, year =

    Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric , author =. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , month = nov, year =

    2021

  50. [58]

    2025 , month = mar, url =

    Gayed, John Maurice , title =. 2025 , month = mar, url =

    2025

  51. [59]

    and Taslimipoor, Shiva and Yannakoudakis, Helen and Yuan, Zheng and Bryant, Christopher and Rei, Marek and Buttery, Paula

    Davis, Christopher and Caines, Andrew and Andersen, istein E. and Taslimipoor, Shiva and Yannakoudakis, Helen and Yuan, Zheng and Bryant, Christopher and Rei, Marek and Buttery, Paula. Prompting open-source and commercial language models for grammatical error correction of E n...

    2024 doi

  52. [60]

    Computational Linguistics , volume =

    Bryant, Christopher and Yuan, Zheng and Qorib, Muhammad Reza and Cao, Hannan and Ng, Hwee Tou and Briscoe, Ted , title =. Computational Linguistics , volume =. 2023 , month =. doi:10.1162/coli_a_00478 , url =

    2023 doi

  53. [61]

    M ulti-Class Grammatical Error Detection for Correction: A Tale of Two Systems

    Yuan, Zheng and Taslimipoor, Shiva and Davis, Christopher and Bryant, Christopher. M ulti-Class Grammatical Error Detection for Correction: A Tale of Two Systems. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021. doi:10.18653/v1/2021...

    2021 doi

  54. [62]

    Grammatical error correction using neural machine translation

    Yuan, Zheng and Briscoe, Ted. Grammatical error correction using neural machine translation. Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016. doi:10.18653/v1/N16-1042

    2016 doi

  55. [63]

    Multi-dimensional Evaluation of LLMs for Grammatical Error Correction

    Labib, Adnan and Wang, Qiao and Huang, Yixuan and Yuan, Zheng. Multi-dimensional Evaluation of LLMs for Grammatical Error Correction. Artificial Intelligence in Education. 2027

    2027

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.