Pith. sign in

REVIEW 3 major objections 6 minor 24 references

Open-source LLMs can grade UML class diagrams nearly as consistently as teaching assistants, criterion by criterion.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 23:48 UTC pith:NGFOVCLX

load-bearing objection Credible open-model numbers and a useful per-criterion failure map for UML grading, but the “close to a TA” claim is unanchored without any inter-rater reliability. the 3 major comments →

arxiv 2603.16357 v2 pith:NGFOVCLX submitted 2026-03-17 cs.CY cs.SE

Beyond Grading Accuracy: Exploring Alignment of TAs and LLMs

Matthijs Jansen op de Haar , Nacir Bouali , Faizan Ahmed This is my paper
classification cs.CY cs.SE
keywords AI-Assisted GradingAutogradingTeaching AssistantLarge Language ModelsUMLClass DiagramMixed-InitiativeOpen-Source Models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Universities face growing numbers of computer science students and heavier grading loads for subjective modeling work such as UML class diagrams. This paper asks whether free, non-proprietary large language models can share that load without surrendering transparency or privacy. The authors build a pipeline that turns student diagrams into natural-language descriptions, feeds them to six open-source models with a detailed 40-point rubric, and compares each criterion score against the grades teaching assistants already assigned. On 92 real exam diagrams the best models reach roughly 85–89 percent per-criterion accuracy and total-score correlations up to 0.80, matching earlier results that used commercial models. The paper shows that the models are strong on simple entity checks, weaker on multiplicities and multi-hop relationships, and often slightly harsher than humans. Those patterns point to a practical mixed-initiative workflow in which the model handles the easy, high-agreement criteria and the TA focuses on the hard ones, cutting workload while keeping human judgment in the loop.

Core claim

On 92 student UML class diagrams scored with a 40-criterion rubric, open-source LLMs reach per-criterion accuracy up to 88.56 percent and total-score Pearson correlation up to 0.798 (mean absolute error about 3.2 points out of 40) with teaching-assistant grades—performance that matches or exceeds prior proprietary-model benchmarks on the same task setup—and are therefore ready to serve as agents inside a mixed-initiative grading pipeline.

What carries the argument

A three-stage grading pipeline: a robust JSON-to-natural-language parser that tolerates student labeling errors, a prompt that forces the model to justify every criterion score, and a criterion-by-criterion comparison against TA grades that surfaces both accuracy and systematic bias.

Load-bearing premise

That teaching-assistant scores are a stable enough reference for “alignment” and for the claim of near-TA performance, even though the diagrams were split among three TAs and no inter-rater reliability is reported.

What would settle it

Double-grade a random subset of the same 92 diagrams with a second independent TA (or the original three TAs on overlapping sets) and show that model–TA agreement is statistically indistinguishable from, or clearly worse than, TA–TA agreement on the same criteria.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • High-agreement criteria such as class presence (≈97 percent accuracy) can be auto-scored, removing roughly 40 percent of the rubric from manual review.
  • Low-agreement criteria (late-mentioned entities, multiplicities, 3-hop relationships) can be automatically flagged for mandatory TA inspection.
  • Open-source mid-size models (24–30 B parameters) can be hosted locally, satisfying cost and data-privacy constraints common in universities.
  • The same pipeline and mixed-initiative pattern can be extended to other UML diagram types once the parser and rubric are adapted.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Without a published human–human baseline, the “close to a TA” claim remains relative rather than absolute; future deployments should report that baseline as a standard calibration step.
  • The observed harshness bias and positional neglect of late entities suggest simple prompt or post-processing fixes (e.g., re-ordering requirements or a small score offset) that could raise agreement without changing the model.
  • If the mixed-initiative loop is closed—TAs editing model scores and those edits feeding back into prompt refinement—the same 92-diagram set could become a living calibration corpus for successive model generations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper evaluates open-source LLMs for grading student UML class diagrams via an improved JSON-to-natural-language parsing pipeline and a modified prompt that elicits per-criterion scores plus clarifications. On 92 exam diagrams scored with a 40-criterion rubric (0/0.5/1), six open-weight models are compared to TA grades at both total-score and per-criterion levels. Reported results include per-criterion accuracy up to 88.56%, total-score Pearson correlation up to r=0.798 (MAE≈3.22/40), a systematic under-grading bias for most models, and lower agreement on multiplicities, late-introduced entities, and multi-hop relationships. The authors argue these models are suitable agents in a mixed-initiative grading workflow that automates high-agreement criteria while flagging hard ones for TAs.

Significance. If the alignment numbers hold under a proper human baseline, the work is practically significant for large CS courses: it shows open-source models can match or exceed prior proprietary-model benchmarks on the same UML grading task while remaining deployable under university cost and privacy constraints. Strengths include temperature-0 deterministic evaluation, external (non-LLM) score aggregation, an improved multiplicity-aware parser, explicit per-criterion analysis rather than diagram-level scores only, and a concrete mixed-initiative roadmap. The category-level error analysis (classes vs associations vs multiplicities; positional and 3-hop weaknesses) is useful beyond the specific models tested.

major comments (3)
  1. [§3.2 / Abstract / Conclusion] §3 and §3.2, and the abstract/conclusion claim that models achieve performance “close to that of a TA” and are therefore suitable mixed-initiative agents: the 92 diagrams were split across three TAs who only “coordinated” for consistency; no inter-rater reliability, double-grading, or pairwise TA–TA agreement is reported for any criterion or for totals. The paper itself states TA grades “do not represent an absolute ground truth.” Without a measured human–human baseline, model–TA Pearson of 0.798 / accuracy of 88.56% cannot be interpreted relative to the noise floor of human grading and may simply track TA–TA disagreement. This is load-bearing for the leap from “agrees with the assigned TA on this split” to “close to a TA / ready for mixed-initiative use.” Either report IRR (even on a re-graded subset) or substantially soften the “close to a TA” language and treat TA scores strictly as o
  2. [§4 / Abstract] §4.1 and the “optimal model” discussion (abstract and §4): selecting the best LLM per criterion after seeing the results produces an oracle upper bound, not a deployable system. Presenting it as achieving performance “close to that of a TA” risks overstating what any single model (or fixed ensemble chosen without hold-out) can deliver. Clarify that this is an exploratory ceiling analysis and report a proper cross-validated or held-out ensemble if the claim is retained.
  3. [§3 / §4.1] Comparison to Bouali et al. (2025) (§1, §3, §4.1): the same dataset, rubric, and largely the same prompt family are reused, with parser and prompt modifications. The claim of “substantial improvement over previous work while using only open-source models” therefore confounds model choice with pipeline changes. A controlled ablation (old parser/prompt vs new, on the same open-source models, or new pipeline on the prior proprietary models) is needed to attribute gains; otherwise the improvement claim should be limited to “open-source models under the improved pipeline reach prior proprietary levels.”
minor comments (6)
  1. [Keywords] Keywords list GPT, Claude, Gemini while the study evaluates only open-source models; align keywords with the actual models (GLM, DeepSeek, Mistral, MiniMax, Gemma, Qwen).
  2. [§3.3.1 / Figure 1] Figure 1 caption and §3.3.1: the parser’s multiplicity reassignment heuristics are central but only described narratively; a short pseudocode box or appendix example of a mis-placed label before/after correction would aid reproducibility.
  3. [§5.6] §5.6 Eq. (1) for KV-cache memory is correct in form but sits somewhat loosely relative to the grading results; either tie it to a concrete context-length measurement for the UML prompt or move to an appendix on deployment.
  4. [Table 2 / §4.1] Table 2 reports Pearson and MAE only; adding per-model per-criterion accuracy (or at least mean accuracy ± CI) in the main text would make the 85–89% range easier to verify against Figure 3.
  5. [Abstract / §4] Minor wording: abstract says “Pearson correlation coefficient of up to 0.78” while Table 2 lists 0.798 for GLM 4.7; use consistent rounding. Also “criterion” vs “criteria” agreement is uneven in a few places (§4.3).
  6. [§3.3.1] The paper notes a new UTML version that would fix free-text association issues but was not used; state explicitly whether any of the 92 diagrams contain free-text labels that the current parser cannot bind, and how many.

Circularity Check

0 steps flagged

Empirical comparison of LLM outputs to independently assigned TA grades on held student work; self-extension of the authors' prior pipeline is methodological continuity, not circular derivation of the reported accuracies or correlations.

full rationale

The paper's load-bearing claims are quantitative measurements: per-criterion accuracy (up to 88.56%), Pearson r (up to 0.798), and MAE on 92 diagrams scored against a fixed 40-criterion rubric by three TAs. These numbers are obtained by running six open-source LLMs on a natural-language parse of student JSON, then comparing the resulting 0/0.5/1 scores to the TA scores via standard external metrics (accuracy, Pearson, MAE). Totals are summed outside the models; temperature is fixed at zero; no free parameters are fitted to the evaluation set and then re-used as 'predictions.' The only self-reference is reuse of the dataset, rubric, and base prompt from Bouali et al. (2025) (overlapping authors), with documented parser and prompt changes. That is ordinary methodological continuity for an incremental empirical study, not a self-definitional loop, fitted-input-as-prediction, uniqueness theorem imported from the same authors, or ansatz smuggled via citation. The 'optimal model' that picks the best LLM per criterion is an explicit post-hoc upper-bound exploration, not a claimed first-principles derivation. Consequently there is no reduction of the central performance claims to their own inputs by construction. (The separate validity concern that TA–TA reliability is unreported is outside the scope of circularity analysis.)

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claim is empirical agreement between LLMs and TAs under a fixed rubric and a JSON→text parser. Load-bearing premises are domain conventions (UML class-diagram grading, TA scores as reference), experimental controls (temperature 0, 40 binary/half-point criteria), and engineering choices (parser heuristics, prompt edits). No physical constants or deep theoretical free parameters; the main “invented” constructs are the improved parser pipeline and the post-hoc per-criterion “optimal model.”

free parameters (3)
  • LLM sampling temperature = 0
    Fixed to 0 for deterministic outputs (§3.3); choice affects reproducibility of generations but is not fit to maximize reported accuracy.
  • Per-criterion score scale {0, 0.5, 1} = 0 / 0.5 / 1
    Rubric discretization chosen by course staff (§3.2); defines the agreement metric space for all reported accuracies.
  • Parser multiplicity/label association heuristics
    Rules that reassign misplaced multiplicities and labels from student JSON (§3.3.1) are hand-designed; they change the natural-language input the LLMs see and thus the measured alignment.
axioms (4)
  • domain assumption TA-assigned rubric scores are a useful reference for measuring LLM grading alignment, even if not absolute ground truth.
    Stated explicitly in §3; entire evaluation is model–TA agreement, not correctness against a gold design.
  • domain assumption A natural-language rendering of student JSON plus rubric text is a sufficient interface for LLMs to grade UML class diagrams comparably to visual TA inspection of PNGs.
    Core of the pipeline in §3.3; modality mismatch (PNG for TAs, text for LLMs) is accepted without a controlled same-input ablation.
  • ad hoc to paper Open-weight models evaluated at temperature 0 with the modified prompt are representative of deployable university-local grading agents.
    Model selection by popularity/cost/openness (§3.3); deployment discussion in §5.6 assumes mid-scale local hosting feasibility.
  • standard math Standard correlation/accuracy/MAE metrics on independent criterion scores quantify suitability for mixed-initiative use.
    Pearson r, MAE, per-criterion accuracy used throughout §4 without novel statistical machinery.
invented entities (2)
  • Improved JSON-to-natural-language UML parser with multiplicity reassignment no independent evidence
    purpose: Normalize messy student label placement so LLM prompts better reflect intended diagram structure.
    Described in §3.3.1 as an advance over Bouali et al.; performance gains cannot be fully attributed because the prior system did not handle these cases the same way.
  • Per-criterion “optimal model” (best LLM chosen per rubric item) no independent evidence
    purpose: Upper-bound demonstration that ensemble-by-criterion selection can approach TA-level aggregate performance.
    Introduced in abstract/findings as combining best model per criterion; requires oracle knowledge of which model wins each criterion and is not a single deployable grader.

pith-pipeline@v1.1.0-grok45 · 15432 in / 3535 out tokens · 43698 ms · 2026-07-13T23:48:19.052671+00:00 · methodology

0 comments
read the original abstract

In this paper, we investigate the potential of open-source Large Language Models (LLMs) for grading Unified Modeling Language (UML) class diagrams. In contrast to existing work, which primarily evaluates proprietary LLMs, we focus on non-proprietary models, making our approach suitable for universities where transparency and cost are critical. Additionally, existing studies assess performance over complete diagrams rather than individual criteria, offering limited insight into how automated grading aligns with human evaluation. To address these gaps, we propose a grading pipeline in which student-generated UML class diagrams are independently evaluated by both teaching assistants (TAs) and LLMs. Grades are then compared at the level of individual criteria. We evaluate this pipeline through a quantitative study of 92 UML class diagrams from a software design course, comparing TA grades against assessments produced by six open-source LLMs. Performance is measured across individual criteria, highlighting areas where LLMs diverge from human graders. Our results show per-criterion accuracy of up to 88.56\% and a Pearson correlation coefficient of up to 0.78, representing a substantial improvement over previous work while using only open-source models. The models achieve performance close to that of a TA, suggesting a possible path toward a mixed-initiative grading system, where TAs are aided in their grading. Our findings demonstrate that open-source LLMs can effectively support UML class diagram grading by explicitly identifying alignment with grading criteria. The proposed pipeline provides a practical approach to managing increasing workloads with growing student counts.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 5 canonical work pages

  1. [1]

    Faizan Ahmed, Nacir Bouali, and Marcus Gerhold. 2024. Teaching Assistants as Assessors: An Experience Based Narrative. InProceedings of the 16th International Conference on Computer Supported Education, CSEDU 2024 (International Confer- ence on Computer Supported Education, CSEDU - Proceedings), Oleksandra Poquet, Alejandro Ortega-Arranz, Olga Viberg, Ire...

    work page doi:10.5220/0012624200003693 2024

  2. [2]

    Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Warriem, and Prajish Prasad. 2023. Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. 292–298. doi:10.1145/3587102.3588852

    doi:10.1145/3587102.3588852 2023

  3. [3]

    Weiyi Bian, Omar Alam, and Jörg Kienzle. 2019. Automated grading of class diagrams. In2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). IEEE, 700–709

    2019

  4. [4]

    Weiyi Bian, Omar Alam, and Jörg Kienzle. 2020. Is automated grading of models effective? assessing automated grading of class diagrams. InProceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems(Virtual Event, Canada)(MODELS ’20). Association for Computing Machinery, New York, NY, USA, 365–376. doi:10.114...

    doi:10.1145/3365438.3410944 2020

  5. [5]

    Weiyi Bian, Omar Alam, and Jörg Kienzle. 2021. Automated grading of class diagrams. InProceedings of the 22nd International Conference on Model Driven Engineering Languages and Systems Companion(Munich, Germany)(MODELS ’19 Companion). IEEE Press, 700–709. doi:10.1109/MODELS-C.2019.00106 Beyond Grading Accuracy: Exploring Alignment of TAs and LLMs

    work page doi:10.1109/models-c.2019.00106 2021

  6. [6]

    Nacir Bouali, Marcus Gerhold, Tosif {Ul Rehman}, and Faizan Ahmed. 2025. To- ward Automated UML Diagram Assessment: Comparing LLM-Generated Scores with Teaching Assistants. InProceedings of the 17th International Conference on Computer Supported Education, CSEDU 2025 (International Conference on Com- puter Supported Education, CSEDU - Proceedings, Vol. 1)...

    work page doi:10.5220/0013481900003932 2025

  7. [7]

    Younes Boubekeur, Gunter Mussbacher, and Shane McIntosh. 2020. Automatic assessment of students’ software models using a simple heuristic and machine learning. InProceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. 1–10

    2020

  8. [8]

    Corbett, Kenneth R

    Albert T. Corbett, Kenneth R. Koedinger, and John R. Anderson. 1997. Intelligent Tutoring Systems. InHandbook of Human-Computer Interaction. Elsevier, 849–

    1997

  9. [9]

    doi:10.1016/B978-044481862-1.50103-X

    work page doi:10.1016/b978-044481862-1.50103-x

  10. [10]

    Sarah Foss, Tatiana Urazova, and Ramon Lawrence. 2022. Automatic generation and marking of UML database design diagrams. InProceedings of the 53rd ACM Technical Symposium on Computer Science Education-Volume 1. 626–632

    2022

  11. [11]

    Sarah Foss, Tatiana Urazova, and Ramon Lawrence. 2022. Learning UML data- base design and modeling with AutoER. InProceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. 42–45

    2022

  12. [12]

    Thomas P Hogan and John C Norcross. 2012. Undergraduates as Teaching Assistants.Effective college and university teaching: Strategies and tactics for the new professoriate(2012), 197

    2012

  13. [13]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Pittsburgh, Pennsylvania, USA)(CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030

    doi:10.1145/302979.303030 1999

  14. [14]

    Jeuring, and Bastiaan Heeren

    Hieke Keuning, J.T. Jeuring, and Bastiaan Heeren. 2018. A systematic literature review of automated feedback generation for programming exercises.ACM Transactions on Computing Education19, 1 (2018). doi:10.1145/3231711

    doi:10.1145/3231711 2018

  15. [15]

    Sushant Kumar, Sumit Datta, Vishakha Singh, Deepanwita Datta, Sanjay Ku- mar Singh, and Ritesh Sharma. 2024. Applications, Challenges, and Future Directions of Human-in-the-Loop Learning.IEEE Access12 (2024), 75735–75760. doi:10.1109/ACCESS.2024.3401547

    doi:10.1109/access.2024.3401547 2024

  16. [16]

    Nguyen-Thinh Le. 2016. A Classification of Adaptive Feedback in Educational Systems for Programming.Systems4, 2 (2016), 22. doi:10.3390/systems4020022

    work page doi:10.3390/systems4020022 2016

  17. [17]

    Gyeong-Geon Lee, Ehsan Latif, Xuansheng Wu, Ninghao Liu, and Xiaoming Zhai. 2024. Applying large language models and chain-of-thought for automatic scoring.Computers and Education: Artificial Intelligence6 (2024), 100213. doi:10. 1016/j.caeai.2024.100213

    arXiv 2024

  18. [18]

    Dominic Lohr, Hieke Keuning, and Natalie Kiesler. 2025. You’re (Not) My Type- Can LLMs Generate Feedback of Specific Types for Introductory Programming Tasks?Journal of Computer Assisted Learning41, 1 (2025), e13107

    2025

  19. [19]

    Cheng Luo, Zefan Cai, Hanshi Sun, Jinqi Xiao, Bo Yuan, Wen Xiao, Junjie Hu, Jiawei Zhao, Beidi Chen, and Anima Anandkumar. 2025. HeadInfer: Memory- Efficient LLM Inference by Head-wise Offloading. arXiv:2502.12574 [cs.LG] https://arxiv.org/abs/2502.12574

    Pith/arXiv arXiv 2025

  20. [20]

    Alfredo Milani, Valentina Franzoni, Emanuele Florindi, Assel Omarbekova, Gulmira Bekmanova, and Banu Yergesh. 2025. When AI Is Fooled: Hidden Risks in LLM-Assisted Grading.Education Sciences15, 11 (2025). doi:10.3390/ educsci15111419

    2025

  21. [21]

    2018.Assessing and responding to the growth of computer science undergraduate enrollments

    National Academies of Sciences, Medicine, Division on Engineering, Physical Sciences, Computer Science, Telecommunications Board, Global Affairs, Board on Higher Education, and Committee on the Growth of Computer Science Under- graduate Enrollments. 2018.Assessing and responding to the growth of computer science undergraduate enrollments. National Academies Press

    2018

  22. [22]

    Dave R Stikkolorum, Peter van der Putten, Caroline Sperandio, and Michel Chaudron. 2019. Towards Automated Grading of UML Class Diagrams with Machine Learning.BNAIC/BENELEARN2491 (2019)

    2019

  23. [23]

    Tobias Stähle, Matthijs Jansen op de Haar, Sophia Boyer, Rita Sevastjanova, Arpit Narechania, and Mennatallah El-Assady. 2025. A Design Space for Intelligent Agents in Mixed-Initiative Visual Analytics. arXiv:2512.23372 [cs.HC] https: //arxiv.org/abs/2512.23372

    arXiv 2025

  24. [24]

    Yu, and Qingsong Wen

    Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. 2024. Large Language Models for Education: A Survey and Outlook. arXiv:2403.18105 [cs.CL] https://arxiv.org/abs/2403.18105

    Pith/arXiv arXiv 2024