Investigating first-language bias in LLM-based automated essay scoring: A cross-prompt evaluation of an open-weight AI-model on TOEFL essays

John Maurice Gayed

REVIEW 2 major objections 4 minor 27 references

Investigating first-language bias in LLM-based automated essay scoring: A cross-prompt evaluation of an open-weight AI-model on TOEFL essays

T0 review · 2 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fine-tuned open-weight essay scorer assigns consistently higher raw scores to European-language writers than to East-Asian writers within the same proficiency band, a stable offset the paper cannot yet attribute to genuine writing quality

desk verdict A transparent, reproducible cross-prompt and L1-disaggregated AES evaluation whose 'bias' headline outruns the evidence; the missing prompt-conditioning check is a real soft spot. read the letter →

arxiv 2607.14605 v1 pith:VZWZWN45 submitted 2026-07-16 cs.CL cs.CY

John Maurice Gayed This is my paper · ORCID

classification cs.CLcs.CY

keywords automatedessayscoringfirst-languagebiascross-promptgeneralizationopen-weightlanguagemodelsproficiencybandclassificationfairnessfine-tuningwithin-bandscoreoffset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a small, inexpensively fine-tuned language model can grade essays on topics it never saw and whether it treats writers from different first-language backgrounds even-handedly. On 12,100 essays spanning eight unseen prompts, the model matched human proficiency bands 77.79% of the time, with errors almost always adjacent, and performance did not vary by prompt topic. The central finding is a systematic scoring offset: within every proficiency band, essays by European-language writers scored reliably higher than essays by East-Asian-language writers, and the ordering held across bands. The paper cannot determine whether that offset is a real signal of writing quality or a bias, but it is stable, directional, and not explained by the fine-tuning data's language mix. A sympathetic reader should care because aggregate accuracy can hide such offsets, and any deployed scorer should be audited by language group.

What carries the argument

The load-bearing device is the band-stratified within-band score analysis. Because the corpus only provides coarse band labels (low, medium, high) rather than the underlying human raw scores, the paper compares the model's fine-grained 0.5-point scores inside each band, ranks the eleven language groups by their mean within-band score, and checks that the ranking is stable across bands. The consistency of that ranking (Spearman ρ = 0.88–0.96) is what converts scattered per-language differences into a single directional offset, and the band stratification is what rules out the trivial explanation that overall accuracy differences are just an artifact of different language groups having differe

What would settle it

Get the raw averaged human scores for the essays (or commission an independent second rating of a representative sample) and compare human within-band ordering of language groups to the model's ordering. If human raters also give European-language writers higher scores within each band, the offset is signal; if the human ordering is flat or reversed, the offset is model bias.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a fine-tuned open-weight language model, trained on just 480 essays from two prompts and then applied with no modifications to 12,100 unseen essays from eleven first-language backgrounds and eight prompts, reproduces human proficiency bands 77.79% of the time, with errors almost always confined to adjacent bands. Its central discovery is a systematic within-band scoring offset: when the coarse proficiency band is held fixed, the model assigns reliably higher raw scores to essays from European-language backgrounds than to essays from East-Asian-language backgrounds, with the ranking of language groups consistent across all three bands (rank correla

Load-bearing premise

The load-bearing premise is that the three coarse proficiency bands hold true proficiency roughly constant within a band across first-language groups, so a within-band raw-score gap indicates differential model behavior rather than genuine language-group proficiency differences hidden inside the same band; the paper acknowledges this is untestable with the public corpus because the underlying human raw scores are unavailable.

Editorial extensions

If this is right

If the offset is real, aggregate accuracy figures such as 77.79% band agreement can mask a stable, directional scoring gap between language groups; deployed scoring systems should be audited separately for each first-language group.
The flat performance across eight unseen prompts, with no advantage for thematically related topics, implies that a model fine-tuned on two prompts can score new prompts without retraining, at least for this essay genre.
The same downward offset for East-Asian writers makes the model more accurate on their low-band essays (which it correctly keeps low) and less accurate on their high-band essays (which it pushes down), so a single tendency shows up as both higher and lower accuracy depending on the band.
The two most extreme errors—low-rated essays that the model scored high—are both truncated texts that read as proficient writing on the visible portion, suggesting even the model's largest disagreements may stem from incomplete corpus artifacts rather than misreading clearly weak essays.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the offset is driven by first-language-linked surface features such as rhetorical organization, sentence structure, or characteristic error patterns, then any fine-tuned essay scorer trained on human holistic scores may inherit this sensitivity; a direct test would hold essay content constant while varying such surface features and observing whether the model's score moves.
The German-language result converging with a finding from a separately trained proprietary model (which also shifted German essays upward) hints that at least part of the offset is a property of the writing itself rather than a quirk of this model; a multi-model comparison on the same corpus would settle whether the effect generalizes.
The within-band offset, if deemed bias after raw-score ground truth is obtained, could be corrected by per-language calibration; the paper's method of using a model's fine-grained scores against coarse labels gives a template for measuring such offsets in any grader where ground truth is coarse.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper before citing it as evidence of L1 bias in LLM essay scoring. It is not that. It is a careful, reproducible measurement of a single open-weight model's scoring behavior across the full TOEFL11 corpus, and the measured offset is real as a model-output fact. But the bias interpretation is underdetermined, and there is a checkable confound the author did not run.

What is genuinely new: the first all-11-L1 fairness analysis of a fine-tuned open-weight AES model with training prompts fully disjoint from all eight evaluation prompts. The model and adapter are fixed and public, inference is deterministic, and the paper reports confusion matrices, raw-score distributions, band-stratified accuracy, and the offset in enough detail to be audited. The cross-prompt generalization result (QWK 0.677–0.730, no thematic-proximity advantage) is solid and useful. The self-citation to Gayed (2026) is legitimate: the adapter is on Hugging Face and was validated on a separate held-out set, so this is not a circular argument.

The soft spots are in the interpretation, not the measurement. The title and abstract say 'first-language bias,' but Section 5.2 explicitly says the data cannot distinguish bias from genuine within-band proficiency differences. That is an honest limitation, but the framing still oversells. The more specific problem is the one the stress-test flags: the within-band raw-score analysis does not condition on prompt. TOEFL11 is observational, and L1 and prompt are not guaranteed independent. If European-language essays cluster on prompts the model scores leniently and East-Asian essays on prompts it scores harshly, the same offset would appear with no L1-specific behavior. The paper reports prompt-level QWK and agreement but not prompt-level raw-score means or L1×prompt cell counts, both of which are computable from the public corpus. This is not a fatal flaw—the offset is consistent across all three bands, which would be an odd coincidence under a pure prompt confound—but it is a missing control that a referee should require before the bias label sticks.

The band-granularity issue is also as acknowledged: the ETS labels are coarse, and the 'bias' reading depends on the untestable assumption that within-band proficiency is L1-independent. Small cells (German low n=15, Hindi low n=29) are flagged and treated cautiously. All of this is fixable without new data collection—recalibrate the claims, run the L1×prompt analysis, or get raw ETS scores.

This paper is for AES researchers and anyone doing fairness audits of LLM-based scoring. It is a strong validation-protocol exemplar and deserves a serious referee; the review should push for the prompt-conditioning analysis and a more measured title. I would bring it to a reading group as a case study in how to do transparent evaluation and how to avoid overinterpreting it.

Referee Report

2 major / 4 minor

Summary. The paper evaluates a fixed LoRA-adapted Gemma-3-27B-it model (public adapter, trained on 480 essays from two prompts) on the complete TOEFL11 corpus: 12,100 essays, 11 L1 groups, and 8 unseen prompts. It reports 77.79% exact band agreement and QWK = 0.702, with stable cross-prompt performance (QWK 0.677–0.730, no significant prompt effect) and a within-band raw-score offset in which European-language L1 groups receive higher scores than East-Asian L1 groups in every ETS band (gap +0.21 to +0.33; rank-order ρ = 0.88–0.96). The paper frames this as a systematic L1-linked scoring offset, while explicitly acknowledging in §5.2 and §5.4 that the offset could reflect genuine within-band proficiency differences rather than model bias.

Significance. The study has substantial strengths: it is a large, transparent, reproducible evaluation of an open-weight AES model, the band-stratified design avoids the band-composition confound, and the discussion is unusually candid about the limitations of the public TOEFL11 ground truth. The cross-prompt generalization result is a valuable addition to the sparse literature on strictly out-of-domain AES evaluation. If the L1 offset survives the prompt-stratified robustness check described below, it would be the first large-scale documentation of L1-correlated scoring differences in a fine-tuned open-weight LLM and would directly inform fairness audits for deploying AES systems. The paper ships a publicly available adapter and a deterministic inference pipeline, which are significant reproducibility assets.

major comments (2)

[§3.4.3, §4.3.2, Table 6] The central L1 offset analysis compares raw scores by L1 within ETS bands without conditioning on prompt. TOEFL11 is approximately balanced across prompts, but L1 × prompt cell counts are not reported. If European-language and East-Asian-language essays are not evenly distributed across the eight prompts, and if the model's raw-score level varies across prompts (plausible even though band-level QWK is stable in Table 6), the European–East-Asian gap could be an artifact of prompt composition rather than an L1-specific scoring behavior. The paper controls for band composition but not for L1 × prompt imbalance. A within-prompt, within-band analysis is necessary—for example, compute the European–East-Asian raw-score difference separately for each prompt and aggregate, or fit a mixed-effects model with prompt as a random effect. The data needed for this check are already in the authors' posse
[Title, RQ3, §5.2, §5.4] The title and RQ3 promise an investigation of first-language bias, and the abstract uses the phrase 'first-language bias' while the results and discussion carefully speak of a 'scoring offset.' Section 5.2 correctly states that the offset could reflect genuine within-band proficiency differences, and §5.4 concedes that the public corpus lacks raw ETS scores to disambiguate. As written, readers are invited to interpret the offset as bias even though the authors cannot rule out the null interpretation. The manuscript should either (a) reframe the central contribution as measuring a systematic L1-linked scoring offset with bias as one possible mechanism, and align the title/RQ3 accordingly, or (b) provide an additional analysis (e.g., an independent human rescoring of a representative sample) that could distinguish the two explanations. The current framing overstates the inference supported

minor comments (4)

[§4.3.3] The training-exposure analysis reports r = −0.43, p = .183, n = 11. The text claims this 'does not account for the offset,' but a non-significant correlation in a sample of 11 provides weak evidence against a training-exposure effect; the confidence interval for r would be wide. It would be more accurate to state that there is no statistically detectable relationship, and to report the confidence interval. The Korean/Spanish contrast is illustrative but not decisive on its own.
[§2.2] The sentence beginning 'The present study extends this line of work by introducing a stricter test of cross-prompt generalization, this study’s model was trained on two prompts...' is a run-on and should be split into two sentences.
[§3.4.3, §4.3.2] The Kruskal–Wallis and Mann–Whitney tests are reported without any multiple-comparison correction. Given the 11 L1 groups and the exploratory nature of the analysis, correction for multiple comparisons (or an explicit justification for not correcting) should be reported, particularly for the pairwise Mann–Whitney tests used in the regional comparisons.
[Table 7] The note under Table 7 is redundant with the header information but should be retained for emphasis on small cells. However, the paper would benefit from a supplementary table reporting the L1 × prompt cell counts for each ETS band, which is the same information required to address Major Comment 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model is a fixed artifact evaluated against an external benchmark, and the central L1-offset finding is a descriptive result with acknowledged underdetermination.

full rationale

The paper performs an external evaluation of a fixed, publicly released LoRA-adapted model against the TOEFL11 corpus. No parameter in this paper is fitted to the TOEFL11 data, and no quantity called a 'prediction' is derived from a fitted input. The central L1-offset result is a descriptive comparison of the model's raw scores within ETS proficiency bands, not a construction that builds the conclusion into its inputs. The only self-citation is Gayed (2026), which supplies the model, adapter, and inference configuration; this is not load-bearing in a circular sense because the adapter is publicly available on Hugging Face, the configuration is fixed before evaluation, and the paper's claims about L1 scoring behavior are measured on TOEFL11, not read off from the cited paper. The paper also explicitly acknowledges the key limitation that within-band proficiency is not exactly constant and that the data 'cannot fully distinguish' genuine within-band variation from model bias, which is an honest underdetermination rather than a circular derivation. The skeptic's concern about L1×prompt confounding is a validity threat internal to the design, not a circularity: the raw-score comparison does not condition on prompt, but this is a confound in an observational evaluation, not a case of a prediction reducing to a fitted parameter. No uniqueness theorem, ansatz-by-citation, or renaming of a known result is present. The evaluation is self-contained against an external benchmark and the findings are reported as measurements, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new mathematical objects or entities. Its claims rest on the validity of the ETS ground truth and on the assumption that within-band proficiency is comparable across L1 groups, which the paper itself flags as untestable with the public corpus.

assumptions (4)

domain assumption ETS band labels in TOEFL11 are a valid, reliable ground-truth anchor for essay proficiency.
The entire L1-offset analysis treats the ETS band labels as the reference; if human ratings are flawed, the 'offset' interpretation changes. Invoked in Sections 3.2 and 3.4.
domain assumption Within an ETS band, true writing proficiency is distributed comparably across L1 groups.
The claim that the within-band raw-score difference is a scoring offset (rather than genuine proficiency) requires this; the paper explicitly acknowledges it cannot verify this in §5.2 and §5.4.
domain assumption The LoRA adapter and inference configuration are identical to Gayed (2026) and the public adapter weights are the ones used.
Reproducibility of the measurement depends on this identity; stated in §3.1.
domain assumption Count of training essays per L1 is an adequate proxy for training-exposure effects.
Used to rule out training composition as an explanation in §4.3.3; other training-set properties correlated with L1 are not tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Investigating first-language bias in LLM-based automated essay scoring: A cross-prompt evaluation of an open-weight AI-model on TOEFL essays." pith.science (2026). https://pith.science/paper/VZWZWN45

@misc{pith2026260714605,
  author       = {Pith},
  title        = {Pith review of: Investigating first-language bias in LLM-based automated essay scoring: A cross-prompt evaluation of an open-weight AI-model on TOEFL essays},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VZWZWN45}},
  note         = {Machine review of arXiv:2607.14605}
}

read the original abstract

This study examines the cross-prompt generalization and first-language (L1) scoring effects of a LoRA-adapted open-weight large language model (Gemma-3-27B-it) applied to automated essay scoring. Using the identical model and inference configuration reported in "AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models" (Gayed, 2026), which was fine-tuned on 480 argumentative essays from two prompts, we evaluate scoring accuracy on the full TOEFL11 corpus: 12,100 essays written by test-takers from 11 first-language backgrounds across eight prompts, none of which were seen during training. The model's raw scores (0.5-5.0) are mapped to the same three proficiency bands (low, medium, high) used by ETS, enabling direct comparison. The model achieved an overall band agreement of 77.79% and a quadratic weighted kappa of 0.702, with adjacent-band agreement of 99.98%. Accuracy was stable across all eight unseen prompts, with no advantage for prompts thematically related to the training data, indicating robust cross-prompt generalization. However, the model exhibited a systematic, L1-linked scoring offset. Within every proficiency band, essays from European-language backgrounds received consistently higher scores than essays from East-Asian-language backgrounds, a pattern not attributable to the composition of the fine-tuning data. This is the first large-scale L1 fairness analysis of a fine-tuned open-weight LLM for automated essay scoring.

Figures

Figures reproduced from arXiv: 2607.14605 by the authors.

**Figure 1.** Heatmap of classification accuracy by first language and ETS band. Each cell shows the percentage of essays from one L1 group correctly classified within one band; darker red indicates higher accuracy, darker blue lower. L1 groups are arranged in three regional clusters (European, Other, East Asian); within each cluster, languages are ordered by overall agreement rate. Daggers mark cells whose band n is below 30, wh… view at source ↗

**Figure 2.** Within-band standardized scoring offset by first language, ordered from most positive to most negative. Positive values indicate that the model assigned essays from that L1 higher raw scores than the average for their ETS band; negative values indicate lower-than-average scoring. Bars are coloured by L1 region. The European versus East-Asian separation is clearly visible: all four European-language groups have posit… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 1 linked inside Pith

[1]

Exploring the potential of using an

Mizumoto, Atsushi and Eguchi, Masaki , journal=. Exploring the potential of using an. 2023 , publisher=

2023
[2]

A meta-analysis on the predictive validity of

Ihlenfeldt, Samuel Dale and Rios, Joseph A , journal=. A meta-analysis on the predictive validity of. 2023 , publisher=

2023
[3]

The interplay of task characteristics, linguistic complexity, and language proficiency in high-stakes

Riemenschneider, Anja and Weiss, Zarah and Schr. The interplay of task characteristics, linguistic complexity, and language proficiency in high-stakes. TESOL Quarterly , volume=. 2024 , publisher=

2024
[4]

2007 , publisher=

College writing and beyond: A new framework for university writing instruction , author=. 2007 , publisher=

2007
[5]

Applied Measurement in Education , volume=

Applying cognitive theory to the human essay rating process , author=. Applied Measurement in Education , volume=. 2020 , publisher=

2020
[6]

Frontiers in Education , volume=

Validity of comparative judgment scores: How assessors evaluate aspects of text quality when comparing argumentative texts , author=. Frontiers in Education , volume=. 2022 , organization=

2022
[7]

The impact of essay organization and overall quality on the holistic scoring of

Li, Junfei and Huang, Jinyan , journal=. The impact of essay organization and overall quality on the holistic scoring of. 2022 , publisher=

2022
[8]

Computer Assisted Language Learning , volume=

Effectiveness of large language models in automated evaluation of argumentative essays: Finetuning vs.\ zero-shot prompting , author=. Computer Assisted Language Learning , volume=. 2026 , publisher=

2026

Show all 27 references

[9]

Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in

Chen, Chi-Fen Emily and Cheng, Wei-Yuan Eugene , journal =. Beyond the design of automated writing evaluation: Pedagogical practices and perceived learning effectiveness in
[10]

Handbook of Automated Essay Evaluation , editor=

Implementation and applications of the intelligent essay assessor , author=. Handbook of Automated Essay Evaluation , editor=. 2013 , publisher=

2013
[11]

2013 , publisher=

Handbook of Automated Essay Evaluation , editor=. 2013 , publisher=

2013
[12]

Assessing writing with the tool for the automatic analysis of lexical sophistication (

Crossley, Scott A and Kyle, Kristopher , journal=. Assessing writing with the tool for the automatic analysis of lexical sophistication (. 2018 , publisher=

2018
[13]

Journal of Writing Research , volume=

Linguistic features in writing quality and development: An overview , author=. Journal of Writing Research , volume=. 2020 , doi=

2020
[14]

Handbook of Automated Essay Evaluation , editor=

Validity and reliability of automated essay scoring , author=. Handbook of Automated Essay Evaluation , editor=. 2013 , publisher=

2013
[15]

Artificial Intelligence Review , volume=

An automated essay scoring systems: A systematic literature review , author=. Artificial Intelligence Review , volume=. 2022 , publisher=

2022
[16]

IEEE Access , volume=

A systematic literature review: Are automated essay scoring systems competent in real-life education scenarios? , author=. IEEE Access , volume=. 2024 , publisher=

2024
[17]

arXiv preprint arXiv:2401.06431 , year=

From automation to augmentation: Large language models elevating essay scoring landscape , author=. arXiv preprint arXiv:2401.06431 , year=

arXiv
[18]

Large language models and automated essay scoring of

Pack, Austin and Barrett, Alex and Escalante, Juan , journal=. Large language models and automated essay scoring of. 2024 , publisher=

2024
[19]

Tate, Tamara P and Steiss, Jacob and Bailey, Drew and Graham, Steve and Moon, Youngsun and Ritchie, Daniel and Tseng, W and Warschauer, Mark , journal=. Can. 2024 , publisher=

2024
[20]

Gayed, John Maurice , journal=
[21]

Blanchard, Daniel and Tetreault, Joel and Higgins, Derrick and Cahill, Aoife and Chodorow, Martin , year=
[22]

2022 , note=

Hu, Edward J and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , journal=. 2022 , note=

2022
[23]

Comparing

Liu, Yingying and Lu, Xiaofei and Qi, Huilei , journal=. Comparing. 2025 , publisher=

2025
[24]

Enhancing

Liu, Yingying and Qi, Huilei and Lu, Xiaofei , journal=. Enhancing. 2025 , publisher=

2025
[25]

Applied Sciences , volume=

Large language model-powered automated assessment: A systematic review , author=. Applied Sciences , volume=
[26]

Fine-tuning

Latif, Ehsan and Zhai, Xiaoming , journal=. Fine-tuning. 2024 , publisher=

2024
[27]

2025 , note=

Gemma 3 technical report , author=. 2025 , note=

2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Exploring the potential of using an

Mizumoto, Atsushi and Eguchi, Masaki , journal=. Exploring the potential of using an. 2023 , publisher=

2023

[2] [2]

A meta-analysis on the predictive validity of

Ihlenfeldt, Samuel Dale and Rios, Joseph A , journal=. A meta-analysis on the predictive validity of. 2023 , publisher=

2023

[3] [3]

The interplay of task characteristics, linguistic complexity, and language proficiency in high-stakes

Riemenschneider, Anja and Weiss, Zarah and Schr. The interplay of task characteristics, linguistic complexity, and language proficiency in high-stakes. TESOL Quarterly , volume=. 2024 , publisher=

2024

[4] [4]

2007 , publisher=

College writing and beyond: A new framework for university writing instruction , author=. 2007 , publisher=

2007

[5] [5]

Applied Measurement in Education , volume=

Applying cognitive theory to the human essay rating process , author=. Applied Measurement in Education , volume=. 2020 , publisher=

2020

[6] [6]

Frontiers in Education , volume=

Validity of comparative judgment scores: How assessors evaluate aspects of text quality when comparing argumentative texts , author=. Frontiers in Education , volume=. 2022 , organization=

2022

[7] [7]

The impact of essay organization and overall quality on the holistic scoring of

Li, Junfei and Huang, Jinyan , journal=. The impact of essay organization and overall quality on the holistic scoring of. 2022 , publisher=

2022

[8] [8]

Computer Assisted Language Learning , volume=

Effectiveness of large language models in automated evaluation of argumentative essays: Finetuning vs.\ zero-shot prompting , author=. Computer Assisted Language Learning , volume=. 2026 , publisher=

2026