Pith. sign in

REVIEW 1 major objections 2 minor 35 references

AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models

T0 review · 1 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A LoRA-adapted open-weight Gemma model scores TOEFL essays with 90.56 percent agreement to human raters.

desk verdict The paper ships public LoRA adapters and code for a Gemma-based essay scorer that hits RMSE 0.474 and QWK 0.828 on a 360-essay ETS hold-out, beating the LLaMA version and a prior GPT baseline, but the proprietary 120/360 split cannot be audited for leakage or selection effects. read the letter →

arxiv 2606.12801 v1 pith:ET7MNWMR submitted 2026-06-11 cs.CY

John Maurice Gayed This is my paper · ORCID
classification cs.CY
keywords automatedwritingevaluationLoRAadaptationlargelanguagemodelsessayscoringGemmamodelTOEFLessaysopen-sourcesystemquadraticweightedkappa
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that fine-tuning the Gemma-3-27B-it model with LoRA on a 120-essay subset of an ETS TOEFL dataset produces strong scoring performance on the remaining 360 essays. The resulting system reports a root mean square error of 0.474, quadratic weighted kappa of 0.828, and 90.56 percent agreement within half a point of human scores, beating both a larger LLaMA-3.3-70B model under identical adaptation and a prior GPT-3.5 baseline. A reader would care because the result indicates that accessible open-weight models can perform rubric-aligned essay evaluation without proprietary systems or large training sets. The work further reports that model scale does not reliably predict success under this method and that the same LoRA settings yield different outcomes across model families.

What carries the argument

LoRA adaptation of instruction-tuned large language models for producing rubric-aligned numeric scores on argumentative essays.

What would settle it

Evaluating the same fine-tuned models on a fresh collection of TOEFL essays drawn from a different source or on a different random split of the 480 essays and obtaining agreement rates well below the reported 90.56 percent would undermine the performance claim.

Watch

Extended reading notes

Core claim

The central claim is that the LoRA-adapted Gemma-3-27B-it model, fine-tuned on 120 TOEFL Independent Writing essays, achieves RMSE of 0.474, quadratic weighted kappa of 0.828, and 90.56 percent agreement within plus or minus 0.5 of human scores on the 360-essay test set, outperforming the identically adapted LLaMA-3.3-70B model and the fine-tuned GPT-3.5 baseline from prior work on the same data.

Load-bearing premise

The fixed 120-essay training split and 360-essay test split from the proprietary ETS dataset gives an unbiased measure of how well the adapted model generalizes.

Editorial extensions

If this is right

  • Open-weight LLMs can match or exceed proprietary fine-tuning for rubric-aligned scoring tasks.
  • Model scale is not a reliable predictor of downstream performance under LoRA adaptation.
  • Identical LoRA hyperparameters produce qualitatively different adaptation behaviors across model architectures.
  • The resulting scoring system runs on consumer-grade hardware and can be made publicly available with open adapters and code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This approach could allow smaller institutions to deploy automated essay feedback without depending on closed commercial APIs.
  • The architecture-specific differences in LoRA response point to a need for systematic model selection when adapting LLMs to other structured judgment tasks.
  • Extending the method to non-English essays or different rubrics would test whether the reported gains hold beyond the TOEFL Independent Writing prompt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 2 minor

Summary. The manuscript presents AiAWE, an open-source automated writing evaluation system that applies LoRA adaptation to instruction-tuned LLMs (primarily Gemma-3-27B-it, with comparison to LLaMA-3.3-70B) for scoring argumentative TOEFL essays. Using a proprietary ETS collection of 480 essays, the authors fine-tune on a 120-essay training subset and report results on the remaining 360-essay test set under matched inference settings: the Gemma model attains RMSE 0.474, QWK 0.828, and 90.56% agreement within ±0.5 of human scores, outperforming both the larger LLaMA model and a prior fine-tuned GPT-3.5 baseline on the same dataset. The work also releases the production system, LoRA adapters, and fine-tuning configurations.

Significance. If the reported metrics hold under the chosen split, the paper supplies concrete evidence that open-weight models under LoRA can reach rubric-aligned performance competitive with or superior to proprietary fine-tuning, while running on consumer hardware. The public release of adapters, code, and the deployed system constitutes a clear reproducibility asset for the AWE community. The secondary observations—that model scale is not a reliable predictor under identical LoRA settings and that adaptation behavior differs across architectures—are empirically grounded contributions.

major comments (1)
  1. [Dataset and experimental setup] Dataset and experimental setup (abstract and § on data): the 120-essay training subset is drawn from the 480-essay proprietary ETS collection with no description of the selection procedure (random, score-stratified, prompt-balanced, or otherwise). Because every headline metric (RMSE 0.474, QWK 0.828, 90.56% agreement) and the claim of outperforming the prior GPT-3.5 baseline rests exclusively on the 360-essay held-out set, the absence of an auditable partition leaves open the possibility of post-hoc selection effects or unintended leakage; this is load-bearing for the central empirical claims.
minor comments (2)
  1. [Abstract] Abstract: the phrase 'identical inference quantization' is used without stating the bit-width or method (e.g., 4-bit GPTQ), which would directly aid replication of the reported numbers.
  2. [Results and comparison] The manuscript would benefit from an explicit statement of whether the 360-essay test set matches the exact test partition used in the cited prior GPT-3.5 work, beyond the general claim of 'the same dataset'.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the careful and constructive review. We address the single major comment below and will incorporate the requested clarification in the revised manuscript.

read point-by-point responses

  1. Referee: [Dataset and experimental setup] Dataset and experimental setup (abstract and § on data): the 120-essay training subset is drawn from the 480-essay proprietary ETS collection with no description of the selection procedure (random, score-stratified, prompt-balanced, or otherwise). Because every headline metric (RMSE 0.474, QWK 0.828, 90.56% agreement) and the claim of outperforming the prior GPT-3.5 baseline rests exclusively on the 360-essay held-out set, the absence of an auditable partition leaves open the possibility of post-hoc selection effects or unintended leakage; this is load-bearing for the central empirical claims.

    Authors: We agree that the manuscript should have documented the partitioning procedure. The 120-essay training subset was obtained by simple random sampling (without replacement) from the full 480-essay collection prior to any model training, hyperparameter tuning, or evaluation; the remaining 360 essays formed the test set. No score stratification, prompt balancing, or post-hoc adjustments were applied. We will revise the data section to state this explicitly. Because the underlying ETS data remain proprietary, a fully public audit trail of the exact essay IDs is not feasible, but the pre-experiment random split eliminates the possibility of leakage or selection effects on the reported metrics. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: metrics are direct empirical comparisons on held-out data

full rationale

The paper's central claims consist of empirical performance numbers (RMSE 0.474, QWK 0.828, 90.56% agreement) obtained by running the fine-tuned model on a 360-essay held-out subset and comparing outputs to human scores. These quantities are computed externally against an independent reference and do not reduce to any internal equation, fitted parameter, or self-citation chain within the paper. The 120/360 split description and LoRA details are procedural but do not create a definitional loop or rename a fit as a prediction. No load-bearing self-citations, uniqueness theorems, or ansatzes appear in the reported derivation.

Assumptions & free parameters 1 free parameters · 1 assumptions · 0 invented entities

The paper relies on standard supervised fine-tuning assumptions and the representativeness of the proprietary ETS dataset; no new entities are postulated and hyperparameters follow common LoRA practice.

free parameters (1)
  • LoRA rank, alpha, and dropout
    Standard hyperparameters selected for the adaptation experiments rather than fitted directly to the reported test metrics.
assumptions (1)
  • domain assumption The 120-essay training subset is sufficient for effective LoRA adaptation that generalizes to the held-out test essays.
    Invoked when claiming the reported metrics demonstrate successful adaptation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models." pith.science (2026). https://pith.science/paper/ET7MNWMR

@misc{pith2026260612801,
  author       = {Pith},
  title        = {Pith review of: AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ET7MNWMR}},
  note         = {Machine review of arXiv:2606.12801}
}
read the original abstract

This study presents AiAWE, an open-source automated writing evaluation system that scores argumentative essays using a LoRA-adapted instruction-tuned large language model (Gemma-3-27B-it). Using a proprietary Educational Testing Service (ETS) dataset of 480 TOEFL Independent Writing essays, we fine-tune Gemma-3-27B and LLaMA-3.3-70B under identical LoRA configurations on a 120-essay training subset and evaluate on the remaining 360 essays under identical inference quantization. The fine-tuned Gemma model achieves a root mean square error of 0.474, a quadratic weighted kappa of 0.828, and an agreement rate of 90.56% within +/- 0.5 of the human score, outperforming both the larger LLaMA-3.3-70B model and the fine-tuned GPT-3.5 baseline reported in prior work on the same dataset. Three findings are of broader interest: open-weight LLMs can match or exceed proprietary fine-tuning for rubric-aligned scoring; model scale is not a reliable predictor of downstream performance under LoRA adaptation; and identical LoRA hyperparameters produce qualitatively different adaptation behaviors across architectures. The production system runs on a consumer-grade server and is publicly accessible at https://app.awade.gec.waseda.ac.jp. LoRA adapters, application code, and fine-tuning YAMLs are publicly available through their respective repositories.

Figures

Figures reproduced from arXiv: 2606.12801 by the authors.

Figure 1
Figure 1. illustrates this idea. During training, the original weights of the base model stay locked; gradients flow only into a small pair of adapter matrices injected into selected layers of the model. During inference, the frozen base and the trained adapter work together. This design has three practical consequences that matter for AES research. First, training is dramatically cheaper in time and memory, because only a ti… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 4 canonical work pages

  1. [1]

    zero-shot prompting , author=

    Effectiveness of large language models in automated evaluation of argumentative essays: finetuning vs. zero-shot prompting , author=. Computer Assisted Language Learning , pages=. 2024 , publisher=

    2024

  2. [2]

    Computer Assisted Language Learning , pages=

    Enhancing GPT-based automated essay scoring: the impact of fine-tuning and linguistic complexity measures , author=. Computer Assisted Language Learning , pages=. 2025 , publisher=

    2025

  3. [3]

    Assessing Writing , volume=

    Comparing GPT-based approaches in automated writing evaluation , author=. Assessing Writing , volume=. 2025 , publisher=

    2025

  4. [4]

    Atkinson, John and Palma, Daniel , journal=. An. 2025 , publisher=

    2025

  5. [5]

    Heliyon , volume=

    Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments , author=. Heliyon , volume=. 2024 , publisher=

    2024

  6. [6]

    Fine-tuning

    Latif, Ehsan and Zhai, Xiaoming , journal=. Fine-tuning. 2024 , publisher=

    2024

  7. [7]

    arXiv preprint arXiv:2407.01873 , year=

    Automated Text Scoring in the Age of Generative AI for the GPU-poor , author=. arXiv preprint arXiv:2407.01873 , year=

    work page arXiv

  8. [8]

    Exploring the potential of using an

    Mizumoto, Atsushi and Eguchi, Masaki , journal=. Exploring the potential of using an. 2023 , publisher=

    2023

Show all 35 references
  1. [9]

    Bernard, Fabrizio A. M. Davide , journal=. 2025 , publisher=

    2025

  2. [10]

    Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence (CSAI) , pages=

    On Automated Essay Grading using Large Language Models , author=. Proceedings of the 2024 8th International Conference on Computer Science and Artificial Intelligence (CSAI) , pages=. 2024 , organization=

    2024

  3. [11]

    From Prompting to Preference Optimization: A Comparative Study of

    Shibata, Ryo and Miyamura, Shinichi , journal=. From Prompting to Preference Optimization: A Comparative Study of

  4. [12]

    Applying large language models for automated essay scoring for non-native

    Hirao, Ryo and Arai, Satoru and Tsuboi, Yusuke , journal=. Applying large language models for automated essay scoring for non-native. 2024 , publisher=

    2024

  5. [13]

    Applied Sciences , volume=

    Large language model-powered automated assessment: a systematic review , author=. Applied Sciences , volume=. 2025 , publisher=

    2025

  6. [14]

    , author=

    A survey on deep learning-based automated essay scoring and feedback generation. , author=. Artif. Intell. Rev. , volume=

  7. [15]

    Artificial intelligence review , volume=

    An automated essay scoring systems: a systematic literature review , author=. Artificial intelligence review , volume=. 2022 , publisher=

    2022

  8. [16]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , journal=

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , journal=. 2022 , note=

    2022

  9. [17]

    , journal=

    Biderman, Dan and Portes, Jacob and Gonzalez Ortiz, Jose Javier and Paul, Mansheej and Greengard, Philip and Jennings, Connor and King, Daniel and Havens, Sam and Chiley, Vitaliy and Frankle, Jonathan and Blakeney, Cody and Cunningham, John P. , journal=

  10. [18]

    How Much is Too Much? Exploring

    Rathore, Darshita and Kumar, Vineet and Bansal, Chetna and Moitra, Anindya , journal=. How Much is Too Much? Exploring

  11. [19]

    arXiv preprint arXiv:2603.02224 , year=

    Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation , author=. arXiv preprint arXiv:2603.02224 , year=

  12. [20]

    Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke , journal=

  13. [21]

    Advances in Neural Information Processing Systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in Neural Information Processing Systems , volume=

  14. [22]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open Models Based on. arXiv preprint arXiv:2403.08295 , year=

    arXiv

  15. [23]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and others , journal=. The

  16. [24]

    An empirical study of

    Huang, Wei and Zheng, Xingyu and Ma, Xudong and others , journal=. An empirical study of. 2024 , publisher=

    2024

  17. [25]

    cpp Quantization on Llama-3.1-8B-Instruct , author=

    Which Quantization Should I Use? A Unified Evaluation of llama. cpp Quantization on Llama-3.1-8B-Instruct , author=. arXiv preprint arXiv:2601.14277 , year=

  18. [26]

    International Conference on Artificial Intelligence in Education , pages=

    A Web Application for a Cost-Effective Fine-Tuning of Open-Source LLMs in Education , author=. International Conference on Artificial Intelligence in Education , pages=. 2024 , organization=

    2024

  19. [27]

    Open-source

    Manchanda, Saurabh and Basu, Sarbajit , journal=. Open-source

  20. [28]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations) , pages=

    Llamafactory: Unified efficient fine-tuning of 100+ language models , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations) , pages=

  21. [29]

    Deng, Ruiqi and Benckendorff, Pierre , journal=. Does. 2025 , publisher=

    2025

  22. [30]

    A systematic review of large language models in

    Zhang, Yiming and Pang, Zhaoyang and Li, Jialong , journal=. A systematic review of large language models in. 2024 , publisher=

    2024

  23. [31]

    2013 , publisher=

    Blanchard, Daniel and Tetreault, Joel and Higgins, Derrick and Cahill, Aoife and Chodorow, Martin , booktitle=. 2013 , publisher=

    2013

  24. [32]

    2025 , howpublished=

    Gayed, John Maurice , title=. 2025 , howpublished=

    2025

  25. [33]

    Enhancing privacy-preserving deployable large language models for perioperative complication detection: a targeted strategy with

    Kus, Korbinian and others , journal=. Enhancing privacy-preserving deployable large language models for perioperative complication detection: a targeted strategy with. 2025 , publisher=

    2025

  26. [34]

    NPJ Digital Medicine , year=

    Enhancing privacy-preserving deployable large language models for perioperative complication detection: a targeted strategy with LoRA fine-tuning , author=. NPJ Digital Medicine , year=

  27. [35]

    Monitoring of scoring using the e-rater

    Wang, Zhen and von Davier, Alina A , journal=. Monitoring of scoring using the e-rater. 2014 , publisher=

    2014

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.