REVIEW 3 major objections 4 minor 1 cited by
Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper argues that five current LLMs, scoring Italian psychology essays with a four-criterion rubric, agree with human raters no better than chance on any criterion, and that their own scores are not reproducible across repeated runs.
desk verdict Useful empirical caution about LLM essay scoring, but the central validity claim leans on an unverified human benchmark. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by a four-criterion analytic rubric—Pertinence, Coherence, Originality, Feasibility—applied to a corpus of 67 Italian-language essays from a university psychology course. Each of the five LLMs scored all essays three times (three prompt replications), and three agreement statistics were used: Quadratic Weighted Kappa ($\kappa_q$) for human-LLM agreement, Kendall's coefficient of concordance $W$ for within-model stability across replications, and a second agreement analysis for inter-model convergence. These statistics are the load-bearing instrumentation: the conclusion of unsuitability rests entirely on the low $\kappa_q$ and weak $W$ values.
What would settle it
Have at least two independent trained human raters score the same 67 essays on the same four criteria and compute pairwise human-human Quadratic Weighted Kappa. If human-human agreement turns out to be as low as human-LLM agreement, the paper's attribution of the gap to LLM unreliability would not hold; if human agreement is high, the negative result against the LLMs would be confirmed.
Extended reading notes
Core claim
The study's claim is that, in a real higher-education setting, current LLMs cannot replicate human assessment of essay quality when the criteria call for discipline-specific understanding. Across 67 essays, five models, four rubric criteria, and three prompt replications, Quadratic Weighted Kappa values were low and non-significant, and within-model stability across replications was weak, with a median Kendall's $W$ below 0.30. The LLMs also showed systematic distortions, notably inflating the Coherence criterion, and they handled context-dependent dimensions inconsistently. Inter-model agreement was moderate only for Coherence and Originality and negligible for the other two criteria, which the authors read as evidence that agreement among models does not imply agreement with human judgment.
Load-bearing premise
The load-bearing premise is that the human scores used for comparison are themselves dependable; the paper does not report how well two human graders agreed on the same essays, so low human-LLM agreement could partly reflect human graders disagreeing with each other rather than the LLMs being wrong.
Editorial extensions
If this is right
- None of the five tested models would be a trustworthy standalone grader for open-ended academic essays, because human-LLM agreement is low and non-significant on all four rubric criteria.
- A model's score is not stable even for itself: repeated runs of the same essay produce weak within-model agreement, with a median Kendall's $W$ below 0.30.
- The tendency to inflate the Coherence criterion means LLM scores would systematically overstate one component of essay quality, distorting any composite grade built from them.
- Moderate inter-model agreement on Coherence and Originality does not rescue the approach, since agreement among models does not translate into agreement with human raters.
Reading between the lines
- The paper's negative result would be more decisive if human inter-rater reliability were high; a natural next experiment is to compute human-human agreement on the same 67 essays, because if human raters also disagree substantially, the rubric itself would be the limiting factor rather than the LLMs.
- A testable extension is to give the same models a rubric enriched with disciplinary examples and see whether human-LLM agreement rises; if it does, the deficiency may lie in prompt or rubric design rather than in a fixed model ceiling.
- Because all five models showed similar non-significance, the bottleneck may be shared across model families rather than specific to any one architecture, suggesting a comparison with instruction-tuned models trained on domain scoring data as a follow-up.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an empirical study of five large language models (Claude 3.5, DeepSeek v2, Gemini 2.5, GPT-4, and Mistral 24B) used to score 67 Italian-language psychology essays on a four-criterion rubric (Pertinence, Coherence, Originality, Feasibility). Each model scored all essays across three prompt replications to measure intra-model stability. The abstract reports consistently low and non-significant human-LLM agreement (Quadratic Weighted Kappa), weak within-model reliability across replications (median Kendall's W < 0.30), and mixed inter-model agreement, with moderate convergence for Coherence and Originality but negligible concordance for Pertinence and Feasibility. The authors conclude that current LLMs may struggle to replicate human judgment in tasks requiring disciplinary insight and contextual sensitivity, and that human oversight remains critical in interpretive domains.
Significance. If the finding is robust, the paper provides a useful, falsifiable negative result in the ongoing evaluation of LLMs for automated essay scoring. Its strengths include the real-world higher-education context, the use of multiple LLMs, and the use of established reliability metrics (Quadratic Weighted Kappa and Kendall's W). The study also has a clear and honest limitation statement ('Although limited in scope'), which is appropriate. However, the significance is undercut by the absence of reported human inter-rater reliability, which is necessary to interpret low human-LLM agreement as an LLM-specific deficiency. The small sample (N=67) and a single discipline/course also limit the generalizability, though these limits are acknowledged.
major comments (3)
- [Abstract] The abstract reports low and non-significant human-LLM agreement as the central validity evidence, but it does not report human inter-rater reliability for the four rubric criteria. Without evidence that human raters agree with one another, low human-LLM agreement could reflect an unreliable reference standard rather than an LLM deficiency. The paper must report human inter-rater reliability (e.g., multi-rater kappa or intraclass correlation) and, if it is poor, temper the conclusion that LLMs 'struggle to replicate human judgment.'
- [Abstract] The abstract reports 'non-significant' Quadratic Weighted Kappa values without giving point estimates, confidence intervals, or a discussion of statistical power. With only 67 essays, non-significance may result from low power rather than the absence of agreement. The paper should report the exact QWK values, 95% confidence intervals or bootstrap intervals, and the smallest effect size detectable with this sample size.
- [Abstract / Methods] The abstract does not specify the exact prompts used for the three replications, the precise model versions (e.g., GPT-4-turbo vs. GPT-4-base), or the generation parameters (temperature, max tokens). Since the study's reliability claim depends on the sensitivity of responses to prompt replication, full transparency about the prompt construction and the scoring procedure is essential for evaluating whether the low Kendall's W is a property of the models or of the prompt engineering.
minor comments (4)
- [Abstract] The terms 'reliability' and 'validity' are used in a general sense; the paper should state the specific definitions (e.g., reliability as consistency across replications, validity as agreement with human judgment) to avoid conflation.
- [Methods] The four rubric criteria (Pertinence, Coherence, Originality, Feasibility) are likely ordinal, but the paper should explicitly justify the use of Quadratic Weighted Kappa and specify the weighting scheme; if the criteria are treated as nominal, a different agreement coefficient would be appropriate.
- [Results] The abstract reports 'median Kendall's W < 0.30' without clarifying whether the median is computed across the five models or across the four criteria; the paper should present the full distribution, including per-model and per-criterion values, and discuss possible range restriction in the score distributions that may deflate W.
- [Conclusion] The claim that 'current LLMs may struggle to replicate human judgment in tasks requiring disciplinary insight and contextual sensitivity' goes beyond the direct evidence of low agreement; the authors should restrict the conclusion to what the data show (low agreement and low replication reliability) and label the 'disciplinary insight' interpretation as one plausible explanation among others.
Circularity Check
No circularity: the paper is an empirical benchmark comparing LLM scores to human rubric scores; the main limitation (missing human inter-rater reliability) is a validity concern, not circular reasoning.
full rationale
This paper is an empirical comparison study, not a derivation, so equation-level circularity does not apply. The derivation chain is: five off-the-shelf LLMs score 67 Italian essays on a four-criterion rubric across three prompt replications; the paper then computes human-LLM agreement (Quadratic Weighted Kappa), within-model stability (Kendall's W), and inter-model concordance. Low human-LLM agreement and weak within-model reliability are measured outputs, not conclusions forced by the inputs. The conclusion that LLMs 'may struggle to replicate human judgment' is an interpretation of those measurements, and the operational criterion 'replicate human judgment' is directly instantiated as agreement with human raters; this is criterion-validity testing, not a self-definitional circle. No fitted parameter is relabeled as a prediction: the models are not calibrated or trained on the 67 essays, and their scores are generated responses, not fitted values. No load-bearing self-citation or imported uniqueness theorem appears in the provided text. The main weakness is that the paper treats human scores as the reference standard without reporting human inter-rater reliability for the four rubric criteria. This is a genuine evidentiary and interpretive gap, because low human-LLM agreement cannot be cleanly attributed to LLM-specific unreliability if human raters themselves disagree substantially. However, that is a validity-assumption concern about the benchmark, not circularity: the paper never defines LLM quality in terms of its own outputs in a way that guarantees the observed result, and the within-model reliability finding is independent of the human benchmark. Under the rule that an unverified external benchmark is a correctness risk rather than a circularity finding, no circular step can be exhibited, and the honest score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Human essay scores constitute a reliable and valid reference standard for comparing LLM grading.
- domain assumption The 67 essays from one psychology course are representative enough to support general statements about LLMs in higher education.
- domain assumption Three prompt replications suffice as an estimate of within-model stability.
Cite this review
Pith. "Pith review of Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education." pith.science (2026). https://pith.science/paper/GCY35L6A
@misc{pith2026250802442,
author = {Pith},
title = {Pith review of: Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education},
year = {2026},
howpublished = {\url{https://pith.science/paper/GCY35L6A}},
note = {Machine review of arXiv:2508.02442}
}
read the original abstract
This study investigates the reliability and validity of five advanced Large Language Models (LLMs), Claude 3.5, DeepSeek v2, Gemini 2.5, GPT-4, and Mistral 24B, for automated essay scoring in a real world higher education context. A total of 67 Italian-language student essays, written as part of a university psychology course, were evaluated using a four-criterion rubric (Pertinence, Coherence, Originality, Feasibility). Each model scored all essays across three prompt replications to assess intra-model stability. Human-LLM agreement was consistently low and non-significant (Quadratic Weighted Kappa), and within-model reliability across replications was similarly weak (median Kendall's W < 0.30). Systematic scoring divergences emerged, including a tendency to inflate Coherence and inconsistent handling of context-dependent dimensions. Inter-model agreement analysis revealed moderate convergence for Coherence and Originality, but negligible concordance for Pertinence and Feasibility. Although limited in scope, these findings suggest that current LLMs may struggle to replicate human judgment in tasks requiring disciplinary insight and contextual sensitivity. Human oversight remains critical when evaluating open-ended academic work, particularly in interpretive domains.
Forward citations
Cited by 1 Pith paper
-
Comparative Validation of GPT-4o-mini and Teacher Mean Scores for Automated Scoring of Music Analysis Responses: Single-Pass Deployment, Repeatability, and Strategy-Specific Bias
GPT-4o-mini's scoring of music analysis essays agrees only moderately with teacher mean scores, with strategy-specific bias: Fs+CoT under-scores, RAG over-scores, and self-consistency is repeatable but weakly accurate...
Reference graph
Works this paper leans on
-
[1]
������������ ������� �������� ���������� ���� ������ ������������ ��� ����� �������� ��������� ������ ������ ���� � �������� ������ ���� �������� �������� ������ ��������� ����� �� ���� � ������ ������ ���� � ������� ���������� ������ ����������� ������ ���� ��������� ������������ �� ��� �� ��� ���� ���������� ��� ������ ����� ������� ���� � ������ ������...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.