pith. sign in

arxiv: 2606.24973 · v1 · pith:FQ2YU5RXnew · submitted 2026-06-23 · 💻 cs.CL · cs.AI· cs.CY· cs.LG

LLM Performance on a Real, Double-Marked GCSE Benchmark

Malachy Fox , Kavi Samra , Paul Jung This is my paper

Pith reviewed 2026-06-26 00:20 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.CYcs.LG
keywords large language modelsautomated markingGCSE examsinter-rater agreementeducational assessmenthandwritten responsesmock exams
0
0 comments X

The pith

Top LLMs agree with GCSE examiner consensus more closely than the two examiners agree with each other.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds a benchmark from over thirty thousand real double-marked GCSE mock exam answers across five subjects and uses it to measure how well off-the-shelf large language models match human examiners. It reports that the strongest models reach higher agreement with the final consensus score than the two human markers reach with each other, and that this holds for essay writing as well as for messy handwritten mathematics. A reader would care because the result bears directly on whether current models can already serve as reliable, low-cost markers for high-stakes national exams taken by sixteen-year-olds.

Core claim

On the 32,534-response double-marked GCSE mock dataset, the best-performing large language models agree with the examiner consensus at levels that exceed the agreement observed between the two human examiners themselves. The same models perform strongly on subjective English essay tasks and on complex handwritten mathematics scripts, with agreement that stays roughly uniform across the score range and shows little dependence on model size.

What carries the argument

Inter-examiner agreement on the double-marked responses, used as the reference standard for judging LLM marking reliability.

If this is right

  • Models can mark subjective tasks such as English essays at agreement levels comparable to or above human markers.
  • Performance on handwritten mathematics scripts remains high despite messy real student work.
  • Agreement does not vary strongly with model size, allowing smaller, cheaper models to deliver similar results.
  • Uniform agreement near the examiner line suggests the models do not systematically over- or under-score particular bands.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the pattern holds on live papers, automated marking could reduce the variability that currently exists between different human markers.
  • The same benchmark method could be applied to other national exams or to continuous classroom assessment.
  • Widespread use of smaller models that match the top performers would lower the cost barrier to large-scale adoption.

Load-bearing premise

Close agreement with the examiner consensus on this mock-exam set is a sufficient proxy for reliable performance on future live GCSE papers.

What would settle it

A side-by-side comparison of the same models against multiple independent markers on a fresh set of unreleased live GCSE papers.

Figures

Figures reproduced from arXiv: 2606.24973 by Kavi Samra, Malachy Fox, Paul Jung.

Figure 1
Figure 1. Figure 1: The generic marking prompt: a single multimodal message (instruction, question, mark scheme, student answer, and any canvas image) returns a structured integer mark. We report the average of examiner-model agreement against examiner-examiner agreement. For a model the mean agreement with the two examiners is RA = 1 2 [QWK(AI, E1) + QWK(AI, E2)] , the mean QWK between the model and each examiner. The refere… view at source ↗
Figure 2
Figure 2. Figure 2: shows each best-in-subject model’s mean agreement, RA, against the agreement between the two examiners, RH [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: English: each model’s mean QWK with the two examiners (higher = better), best first. Dashed line and band are the examiner-examiner QWK and its 95% CI; a bar reaching the band marks as consistently as a second examiner. Figures 5 and 6 share this layout. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: English essays, signed error per model (AI − consensus, % of max mark), most accurate at the top. Dashed line is neutral; left is harsh, right lenient. 6 Science and Maths Marking The majority of tested models were found to mark the sciences and Maths as consistently as a second examiner, with Maths showing larger gaps in performance between models. Top models in Science achieve positive deltas approaching… view at source ↗
Figure 5
Figure 5. Figure 5: Maths: model mean QWK with the two examiners against the examiner-examiner line (cf [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Science (Biology, Chemistry, Physics combined): model mean QWK with the two examiners against the examiner￾examiner line (cf [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Mean QWK vs cost to mark 1,000 papers (USD list price, log scale), for English and Maths+Science. Each point is a model; top-left is best (high agreement, low cost). Dashed line and band are the examiner line and its 95% CI. 8 Discussion and Conclusion This benchmark shows that the LLMs available today, run under a single generic prompt at minimum reasoning effort, work as a reliable second marker. Additio… view at source ↗
read the original abstract

We introduce a dataset of 32,534 double-marked real student responses to GCSE mock exams (GCSEs are the UK's national exams, taken at age ~16), spanning 328 questions across five subjects and including handwritten work. We test whether off-the-shelf large language models agree with examiners as closely as the two examiners agree with each other. We find that models overwhelmingly agree well with the examiner consensus across subjects, with the top performing models agreeing more closely with examiners than examiners agree with each other. Models achieve high scores for subjective tasks like English essay marking, as well as handling complex and messy handwritten Maths paper scripts. Agreement is uniform near the examiner line, and not massively discriminated by model size, providing cost-effective automated marking solutions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper introduces a dataset of 32,534 double-marked real GCSE mock exam responses spanning 328 questions across five subjects (including handwritten work) and evaluates off-the-shelf LLMs on agreement with the examiner consensus. It reports that top models agree more closely with the consensus than the two examiners agree with each other, with strong performance even on subjective English essays and messy Maths scripts, and that agreement is uniform and not strongly dependent on model size.

Significance. The dataset itself constitutes a valuable public resource for benchmarking automated marking on authentic, double-marked student work. If the central empirical comparison can be shown to be free of aggregation bias, the results would supply concrete evidence on the current reliability of LLMs for high-stakes educational assessment tasks that include both objective and subjective components.

major comments (1)
  1. [Abstract] Abstract: the central claim that 'top performing models agreeing more closely with examiners than examiners agree with each other' rests on a comparison of model-to-consensus agreement versus pairwise examiner agreement. Because the consensus is an aggregate of the two examiner marks, any model whose per-response error variance is comparable to that of the examiners will exhibit strictly smaller expected deviation to the consensus by construction (var(m − (e1+e2)/2) = 1.5σ² versus var(e1−e2) = 2σ²). The manuscript must therefore report model agreement against each individual examiner mark (not only the consensus) and demonstrate that the per-examiner deviations remain smaller; without this control the numerical superiority is non-diagnostic.
minor comments (1)
  1. [Abstract] Abstract: no information is supplied on the precise agreement metric (e.g., exact agreement, Cohen’s κ, mean absolute error), the statistical test used to compare the two quantities, data exclusion criteria, or handling of edge cases such as zero-mark or maximum-mark responses.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their careful review and for identifying this important statistical nuance in our central claim. We address the major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that 'top performing models agreeing more closely with examiners than examiners agree with each other' rests on a comparison of model-to-consensus agreement versus pairwise examiner agreement. Because the consensus is an aggregate of the two examiner marks, any model whose per-response error variance is comparable to that of the examiners will exhibit strictly smaller expected deviation to the consensus by construction (var(m − (e1+e2)/2) = 1.5σ² versus var(e1−e2) = 2σ²). The manuscript must therefore report model agreement against each individual examiner mark (not only the consensus) and demonstrate that the per-examiner deviations remain smaller; without this control the numerical superiority is non-diagnostic.

    Authors: We agree that this is a valid statistical concern and that the comparison to the consensus alone is not fully diagnostic without the requested control. The dataset contains the separate marks from each examiner, so the additional per-examiner analyses are feasible. In the revised manuscript we will report model agreement metrics (MAE, correlation, etc.) against examiner 1 and examiner 2 individually, directly alongside the inter-examiner agreement figures. The abstract and results sections will be updated to present these comparisons and to qualify the central claim accordingly. We view this as a substantive improvement. revision: yes

Circularity Check

0 steps flagged

No circularity: direct empirical comparison on external benchmark

full rationale

The paper reports measured agreement rates between LLMs and an examiner consensus on a fixed, externally double-marked dataset of 32,534 responses. No equations, fitted parameters, or derivations are present; the central claim is a head-to-head numerical comparison of two observed quantities (model-to-consensus vs. inter-examiner pairwise) against real marks. This is self-contained against the external data and does not reduce any reported result to a quantity defined inside the paper by construction. No self-citations or ansatzes are invoked as load-bearing steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are required; the work rests on standard inter-rater agreement measurement applied to a new empirical dataset.

pith-pipeline@v0.9.1-grok · 5657 in / 1019 out tokens · 22531 ms · 2026-06-26T00:20:24.871193+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 7 canonical work pages · 1 internal anchor

  1. [1]

    Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220

    1968

  2. [2]

    Landis, J. R. & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data.Biometrics, 33(1), 159–174

    1977

  3. [3]

    Shermis, M. D. & Burstein, J. (Eds.) (2013).Handbook of Automated Essay Evaluation: Current Applications and New Directions. Routledge

    2013

  4. [4]

    Automated Student Assessment Prize (ASAP)

    Hewlett Foundation (2012). Automated Student Assessment Prize (ASAP). Kaggle Competition. https://www. kaggle.com/c/asap-aes

    2012

  5. [5]

    & Wilson, J

    Huang, Y . & Wilson, J. (2025). Evaluating LLM-Based Automated Essay Scoring: Accuracy, Fairness, and Validity.Proceedings of AIME-CON (Works in Progress), 71–83. 8 LLM Performance on a Real, Double-Marked GCSE Benchmark

    2025

  6. [6]

    Taghipour, K. & Ng, H. T. (2016). A Neural Approach to Automated Essay Scoring.EMNLP 2016, 1882–1891

    2016

  7. [7]

    & Yang, J

    Dong, F., Zhang, Y . & Yang, J. (2017). Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring.CoNLL 2017, 153–162

    2017

  8. [8]

    X., Zhang, K., Wang, Y

    Xiao, C., Ma, W., Song, Q., Xu, S. X., Zhang, K., Wang, Y . & Fu, Q. (2024). Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs.arXiv:2401.06431

    work page arXiv 2024

  9. [9]

    Marking Consistency Metrics: An Update

    Ofqual (2018). Marking Consistency Metrics: An Update. https://assets.publishing.service.gov.uk/ media/5bfbfd70e5274a0fb775cca3/Marking_consistency_metrics_-_an_update_-_FINAL64492. pdf

    2018

  10. [10]

    Teacher Workload Survey 2019: Research Report

    Department for Education (2019). Teacher Workload Survey 2019: Research Report. https://www.gov.uk/ government/publications/teacher-workload-survey-2019

    2019

  11. [11]

    & Onishchuk, D

    Kortemeyer, G., Nöhl, J. & Onishchuk, D. (2024). Grading Assistance for a Handwritten Thermodynamics Exam using AI: An Exploratory Study.Phys. Rev. Phys. Educ. Res., 20, 020144

    2024

  12. [12]

    & Lan, A

    Caraeni, A., Scarlatos, A. & Lan, A. (2024). Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams. arXiv:2411.05231

    work page arXiv 2024

  13. [13]

    & Elsayed, T

    Mansour, W., Albatarni, S., Eltanbouly, S. & Elsayed, T. (2024). Can Large Language Models Automatically Score Proficiency of Written Essays?arXiv:2403.06149

    work page arXiv 2024

  14. [14]

    & Elsayed, T

    Eltanbouly, S., Albatarni, S. & Elsayed, T. (2025). TRATES: Trait-Specific Rubric-Assisted Cross-Prompt Essay Scoring.arXiv:2505.14577

    work page arXiv 2025

  15. [15]

    & Zhai, X

    Latif, E., Fang, L., Ma, P. & Zhai, X. (2024). Knowledge Distillation of Large Language Models for Automatic Scoring of Science Assessments.arXiv:2312.15842

    work page arXiv 2024

  16. [16]

    K., Knill, K

    Bannò, S., Vydana, H. K., Knill, K. M. & Gales, M. J. F. (2024). Can GPT-4 do L2 analytic assessment? arXiv:2404.18557

    work page arXiv 2024

  17. [17]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

    Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M. & Gao, J. (2024). MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts.ICLR 2024; arXiv:2310.02255. 9

    work page internal anchor Pith review Pith/arXiv arXiv 2024