The reviewed record of science sign in
Pith

arxiv: 2606.08400 · v1 · pith:2SAASMUE · submitted 2026-06-07 · cs.SE · cs.AI· cs.CL

Impacts of Histories and Models on LLM Grading: A Study in Advanced Software Engineering Courses

Qilin Zhou , Zhuo Wang , Yue Li , W.K. Chan This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-27 18:25 UTCgrok-4.3pith:2SAASMUErecord.jsonopen to challenge →

classification cs.SE cs.AIcs.CL
keywords LLM gradinggrading consistencyinteraction historysoftware engineering educationmodel driftacademic assessmenthuman alignmentgrading workflow
0
0 comments X

The pith

Continuous interaction history drives LLMs to drift systematically away from human expert scores when grading student work.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper studies whether large language models can reliably grade graduate research reports in an advanced software engineering course, using a case study of 180 student submissions. It tests a proposed human-aligned workflow with two models and measures how well their scores match human experts while tracking consistency within and across models. The work shows that LLMs offer some workload reduction potential but produce model-specific consistency levels and inter-model disagreements, with ongoing chat history causing grading standards to shift away from human benchmarks over time. Simple averaging of multiple models does not fix the alignment problem. These patterns matter because they point to risks of unfairness if LLMs are applied without controls in academic settings.

Core claim

In evaluations of Grok and GPT on 180 submissions, the models display distinct intra-model consistency and notable inter-model inconsistencies; continuous interaction history produces systematic drift in their grading standards that moves them further from human expert scores, while ensemble methods fail to improve that alignment.

What carries the argument

Continuous interaction history supplied to the LLM across successive grading prompts, which incrementally shifts the model's internal grading criteria.

If this is right

  • LLMs can lower grading workload for educators but demand targeted operational practices to limit unfairness.
  • Indiscriminate LLM use for grading risks introducing systemic disparities across students.
  • Different LLMs vary in their grading consistency, so model selection affects reliability.
  • Ensemble approaches alone do not resolve misalignment with human judgment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Resetting conversation context between individual submissions could prevent the observed drift in other grading contexts.
  • The same history-driven drift pattern may appear when LLMs grade work in fields outside software engineering.
  • Longer sequences of submissions could amplify the divergence, suggesting a need to monitor or cap session length.

Load-bearing premise

Human expert scores on the 180 submissions form an unbiased and stable reference point against which LLM outputs can be compared to detect drift.

What would settle it

Running the same grading tasks with fresh, history-free prompts for every submission and observing whether the systematic drift from human scores disappears.

Figures

Figures reproduced from arXiv: 2606.08400 by Qilin Zhou, W.K. Chan, Yue Li, Zhuo Wang.

Figure 3
Figure 3. Figure 3: P-values in the Wilcoxon signed-rank test across paired attempts and models. before conducting the pairwise Wilcoxon signed-rank test and calculating ICC and Hit@k. C. Study Results and Analysis 1) Answer to RQ1: We first analyze the consistency within and across models. As illustrated in the ICC heatmap ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
read the original abstract

Graduate-level research reading report assessment creates a substantial labor burden for educators. While large language models (LLMs) hold great potential for automating academic grading, their reliability for this specialized task remains understudied, particularly regarding grading consistency, the lack of which represents a primary obstacle to educational fairness. This paper proposes a human-aligned LLM-assisted grading workflow and presents a case study based on 180 student submissions from a graduate advanced software engineering course. We evaluate two mainstream LLMs, Grok and GPT, in terms of grading consistency and alignment with human scores. We find LLMs exhibit distinct levels of intra-model consistency and significant inter-model grading inconsistencies, while simple ensemble approaches cannot improve alignment with human evaluation. Critically, continuous interaction history drives systematic drift in models' grading standards away from human expert scores. Our findings demonstrate LLMs' potential in reducing grading workload for educators in graduate education, while highlighting that indiscriminate LLM grading may introduce systemic unfairness, suggesting that specific operational practices are required to mitigate such disparities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper reports an empirical case study using two LLMs (Grok and GPT) to grade 180 student research-reading reports from a graduate advanced software engineering course. It evaluates intra-model consistency, inter-model alignment, the effect of simple ensembles, and—most centrally—the impact of continuous interaction history on grading standards relative to human expert scores. The authors conclude that LLMs can reduce educator workload but that history induces systematic drift away from human benchmarks, that inter-model inconsistencies exist, and that specific operational practices are needed to avoid unfairness.

Significance. If the history-drift result holds after proper controls, the work is significant for the growing literature on LLM use in education: it supplies real-course ecological data (180 submissions) and isolates a concrete operational variable (interaction history) that can produce measurable divergence from human judgment. The explicit comparison of two production LLMs and the negative result on simple ensembles are useful negative findings. The manuscript does not yet supply the methodological transparency (prompt details, statistical tests, human reliability metrics) required to treat the drift claim as robust.

major comments (2)
  1. [Abstract] Abstract and (presumed) Methods section: the central claim that continuous interaction history produces systematic drift away from human expert scores treats the human scores on the 180 submissions as a fixed, low-variance reference. No inter-rater reliability statistic (Cohen’s κ, ICC, or equivalent), number of graders, blinding protocol, or aggregation rule is reported. Without these, any observed divergence could be attributable to human label noise rather than model history.
  2. [Abstract] Abstract: the reported findings on grading consistency, inter-model inconsistency, and ensemble ineffectiveness are presented without any description of the prompting regime, temperature, few-shot examples, or controls for prompt variation. This omission makes it impossible to determine whether the observed effects are properties of the models and history or artifacts of unstandardized prompting.
minor comments (1)
  1. The abstract refers to “simple ensemble approaches” without defining the aggregation method (score averaging, majority vote on letter grades, etc.). A brief operational definition would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on methodological transparency. We address the two major comments below and will revise the manuscript to incorporate the requested details.

read point-by-point responses

  1. Referee: [Abstract] Abstract and (presumed) Methods section: the central claim that continuous interaction history produces systematic drift away from human expert scores treats the human scores on the 180 submissions as a fixed, low-variance reference. No inter-rater reliability statistic (Cohen’s κ, ICC, or equivalent), number of graders, blinding protocol, or aggregation rule is reported. Without these, any observed divergence could be attributable to human label noise rather than model history.

    Authors: We agree that the absence of these details limits the strength of the drift claim. The current manuscript refers to 'human expert scores' without further elaboration on the grading process. We will add a subsection in Methods describing the human grading protocol, including the number of graders, use of any rubric, blinding status, and aggregation method (or lack thereof). If inter-rater reliability data were not collected, we will explicitly note this and discuss its implications for interpreting divergence from the human reference. revision: yes

  2. Referee: [Abstract] Abstract: the reported findings on grading consistency, inter-model inconsistency, and ensemble ineffectiveness are presented without any description of the prompting regime, temperature, few-shot examples, or controls for prompt variation. This omission makes it impossible to determine whether the observed effects are properties of the models and history or artifacts of unstandardized prompting.

    Authors: We acknowledge that the manuscript lacks a full account of the prompting setup. We will insert a new Methods subsection that documents the exact prompting regime for each experiment, including temperature values, system/user prompt templates, any few-shot examples, and any steps taken to standardize or vary prompts. This addition will allow readers to evaluate whether the reported consistency, inconsistency, and ensemble results are robust to prompting choices. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical observational study

full rationale

The paper reports an empirical case study on 180 student submissions, comparing LLM grading outputs (with and without interaction history) against human expert scores for consistency and alignment. No equations, parameters, derivations, or predictive models are defined or fitted. The reported 'drift' consists of direct empirical measurements of divergence from the human benchmark as history length increases; this does not reduce to any self-definition, fitted input renamed as prediction, or self-citation chain. The study is self-contained as an observational report against an external human reference, with no load-bearing uniqueness theorems or ansatzes imported from prior author work.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on the domain assumption that human expert scores are the appropriate reference standard and that the 180 submissions plus the chosen prompting regime are representative of typical graduate-level grading tasks.

axioms (1)
  • domain assumption Human expert scores constitute a stable and unbiased ground truth for grading quality
    All alignment and drift measurements are defined relative to these human scores.

pith-pipeline@v0.9.1-grok · 5714 in / 1253 out tokens · 28183 ms · 2026-06-27T18:25:43.505269+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references

  1. [1]

    Right answer, wrong score: Uncovering the inconsistencies of LLM evaluation in multiple -choice question answering,

    F. M. Molfese, L. Moroni, L. Gioffre´, A. Scire`, S. Conia, and R. Navigli, “Right answer, wrong score: Uncovering the inconsistencies of LLM evaluation in multiple -choice question answering,” in Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for C...

    2025

  2. [2]

    Exploring llm prompting strategies for joint essay scoring and feedback generation,

    M. Stahl, L. Biermann, A. Nehring, and H. Wachsmuth, “Exploring llm prompting strategies for joint essay scoring and feedback generation,” in Proceedings of the 19th workshop on innovative use of NLP for building educational applications (BEA 2024), 2024, pp. 283–298

    2024

  3. [3]

    Llms in automated essay evaluation: A case study,

    M. Kostic, H. F. Witschel, K. Hinkelmann, and M. Spahic -Bogdanovic, “Llms in automated essay evaluation: A case study,” in Proceedings of the AAAI Symposium Series, vol. 3, no. 1, 2024, pp. 143–147

    2024

  4. [4]

    Can llms grade open response reading comprehension questions? an empirical study using the roars dataset,

    O. Henkel, L. Hills, B. Roberts, and J. McGrane, “Can llms grade open response reading comprehension questions? an empirical study using the roars dataset,” International journal of artificial intelligence in education, vol. 35, no. 2, pp. 651–676, 2025

    2025

  5. [5]

    Cograder: Transforming instructors’ assessment of project reports through col - laborative llm integration,

    Z. Chen, J. Wang, Y. Li, H. Li, C. Shi, R. Zhang, and H. Qu, “Cograder: Transforming instructors’ assessment of project reports through col - laborative llm integration,” in Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, 2025, pp. 1– 18

    2025

  6. [6]

    Development of a formative assessment system for reports using large language models and rubrics,

    H. Matsukawa and C. Iwasaki, “Development of a formative assessment system for reports using large language models and rubrics,” in 2024 International Symposium on Educational Technology (ISET) . IEEE, 2024, pp. 34–38

    2024

  7. [7]

    Addressing inconsistencies in grading practices,

    T. R. Guskey, “Addressing inconsistencies in grading practices,” Phi Delta Kappan, vol. 105, no. 8, pp. 52–57, 2024

    2024

  8. [8]

    Does context matter? ContextualJudgeBench for evaluating LLM -based judges in contextual settings,

    A. Xu, S. Bansal, Y. Ming, S. Yavuz, and S. Joty, “Does context matter? ContextualJudgeBench for evaluating LLM -based judges in contextual settings,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Vienna, Austria: Association for ...

    2025

  9. [9]

    What is context engineering?

    IBM, “What is context engineering?” https://www.ibm.com/think/topics/ context-engineering

  10. [10]

    Grok 4.1 fast and agent tools api,

    xAI, “Grok 4.1 fast and agent tools api,” 2025, accessed: 2026 -04-30. [Online]. Available: https://x.ai/blog/grok-4-1-fast

    2025

  11. [11]

    Introducing gpt-oss,

    OpenAI, “Introducing gpt-oss,” 2025, accessed: 2026-04-30. [Online]. Available: https://openai.com/index/introducing-gpt-oss/

    2025

  12. [12]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  13. [13]

    A guideline of selecting and reporting intraclass correlation coefficients for reliability research,

    T. K. Koo and M. Y. Li, “A guideline of selecting and reporting intraclass correlation coefficients for reliability research,” Journal of chiropractic medicine, vol. 15, no. 2, pp. 155–163, 2016

    2016

  14. [14]

    Efficient solutions for an intriguing failure of LLMs: Long context window does not mean LLMs can analyze long sequences flawlessly,

    P. Hosseini, I. Castro, I. Ghinassi, and M. Purver, “Efficient solutions for an intriguing failure of LLMs: Long context window does not mean LLMs can analyze long sequences flawlessly,” in Proceedings of COLING 2025, 2025

    2025