REVIEW 3 major objections 6 minor 18 references
Automated Feedback on Student-Generated UML and ER Diagrams Using Large Language Models
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read DUET is a prototype LLM-based tool that converts instructor and student UML and ER diagrams into PlantUML text, compares them, and generates structured feedback for both students and educators.
desk verdict A clear, honest prototype for LLM-generated feedback on UML/ER diagrams, but the load-bearing conversion/comparison accuracy is never measured and the six-interview evaluation only captures perceptions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a four-step pipeline: upload of a reference and a student diagram, conversion of both images into PlantUML text by GPT-4o, structural comparison of the two text representations by a smaller LLM such as Mistral-7B-Instruct-v0.3, and generation of two categories of feedback from that comparison, one aimed at students and one at educators. The conversion step is what makes the approach feasible, since the authors argue that LLMs process textual diagram representations more reliably than raw images, and the comparison step deliberately restricts itself to objective differences before any feedback wording is produced.
What would settle it
Run DUET on a corpus of, say, fifty student-drawn UML and ER diagrams whose errors have been annotated by two instructors, then measure whether the tool's identified differences match the annotations; the central claim fails if the image-to-PlantUML step misreads a substantial share of legible hand-drawn diagrams or if the tool's agreement with the rubric is no better than chance.
Extended reading notes
Core claim
The paper's central claim is that a two-stage LLM pipeline can turn diagram-based modeling exercises into a scalable, personalized feedback loop. DUET converts an instructor reference diagram and a student-submitted UML or ER diagram into PlantUML text with GPT-4o, has a smaller LLM such as Mistral-7B-Instruct-v0.3 compare the two text representations, and then generates structured feedback: reflective hints for the student and misconception insights for the educator. The authors position DUET not as an assessment engine but as a supplementary formative tool that gives instant, repeatable practice opportunities and aggregates signals about common student misunderstandings. The supporting evidence is qualitative: semi-structured interviews with two lecturers and four teaching assistants who saw accessibility, scalability, and learning support as strengths, while flagging nondeterministic output, reliability, and cheating risk as limitations.
Load-bearing premise
The entire pipeline rests on the assumption that GPT-4o reliably turns any student diagram, including messy hand-drawn ones, into correct PlantUML code, and that comparing those text versions captures the real structural differences between diagrams.
Editorial extensions
If this is right
- Students in large courses could get immediate feedback on diagram drafts before submitting, reducing dependence on teaching assistant availability.
- Educators could aggregate feedback data across many submissions to identify frequent misconceptions and adjust their teaching accordingly.
- The same pipeline could accept a single reference solution and many student submissions, scaling formative assessment without adding instructor labor.
- Feedback designed as reflective hints rather than right or wrong labels could encourage iterative self-correction and deeper engagement with modeling concepts.
- DUET could be used as a pre-lab preparation activity that helps students arrive at tutorials with more refined questions and clearer design reasoning.
Reading between the lines
- Editorial inference: the same image-to-text-then-compare pipeline could generalize to other diagram notations such as BPMN, SysML, or architecture diagrams, since nothing in the architecture is UML-specific beyond the prompt templates.
- Editorial inference: if conversion reliability improves, the educator-facing view could evolve into a real-time misconception dashboard, an application the interviews only gesture at.
- Editorial inference: a direct quantitative study comparing DUET's feedback with teaching assistant feedback on identical submissions, scored by a third expert, would test the core assumption that the interviews could not.
- Editorial inference: limiting the student view to hints rather than corrections is a deliberate pedagogical choice; a version that leaked 'correct answer' wording would undermine the reflective-learning goal the paper emphasizes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents DUET (Diagrammatic UML & ER Tutor), a prototype web-based tool that uses a multi-stage LLM pipeline to provide automated feedback on student-generated UML and ER diagrams. In Step 1, users upload an instructor-provided reference diagram and a student diagram. Step 2 uses GPT-4o to convert both images into PlantUML textual representations. Step 3 uses a smaller LLM, such as Mistral-7B-Instruct-v0.3, to compare the two PlantUML representations and produce a list of structural differences. Step 4 uses the same smaller LLM to generate structured, reflective feedback for students and analytical insights for educators. The evaluation is a qualitative study based on semi-structured interviews with two university lecturers and four teaching assistants, asking about potential use cases, pros, cons, and suggested features. The participants identified strengths including accessibility, scalability, and instant feedback, and limitations including reliability concerns and potential for misuse. The authors conclude that DUET represents a promising direction for integrating LLMs into modeling education and offers a foundation for future classroom integration and empirical evaluation.
Significance. If the pipeline works reliably, DUET addresses a real need: scalable, personalized feedback for UML and ER diagram exercises in large courses, where manual feedback from instructors and TAs does not scale. The paper is honest about its scope, explicitly positioning DUET as a functional prototype and acknowledging the small sample and hypothetical scenarios in the evaluation. The source code and supplementary materials are made available in a GitHub repository, which supports reproducibility. The design is transparent: the workflow is described step by step, and the prompts are predefined and externally configurable. The main weakness is that the central functional claim—that DUET provides accurate and therefore useful feedback—is not measured anywhere in the evaluation. The study relies entirely on participants' perceptions of a demonstration, not on correctness checks of the generated differences or feedback. Given the paper's own cited evidence that GPT-4o-based UML assessment matched human instructors in only about 50% of cases and identified only 25% of class relationships correctly, the unvalidated conversion and comparison stages are the load-bearing risk in this work.
major comments (3)
- [Section 4.1-4.2] The evaluation does not measure the accuracy of the core pipeline: the image-to-PlantUML conversion (Section 3.2, Step 2) and the structural comparison (Section 3.2, Step 3) are never checked against ground truth. The six participants reacted to a demonstration of the workflow, not to output produced for their own students' diagrams, and the paper reports no verification that the generated difference lists or feedback were correct. Since the paper's central claim is that DUET "provides structured feedback based on the differences," it is load-bearing to show that those differences are accurate. I recommend adding a small quantitative study on real student diagrams (e.g., 20-30 diagrams) with human-coded ground truth for classes, attributes, relationships, and multiplicities, reporting conversion fidelity and agreement between generated feedback and instructor judgments.
- [Section 3.2, Step 2] The manuscript explicitly acknowledges that "the quality of the extracted PlantUML code may vary based on image quality, diagram clarity, handwriting legibility (for hand-drawn diagrams), and adherence to standard notations," but it provides no evidence about how this variation affects feedback quality. This is not a hypothetical concern: the paper's own related work (Section 2.3) cites Wang et al. (2024) showing GPT-4-based UML assessments matched human instructors in about 50% of cases and identified only 25% of class relationships correctly, and GPT-4o still made minor mistakes in hand-drawn-to-PlantUML conversion. Since DUET explicitly accepts hand-drawn student diagrams, the unverified first stage is the most likely point of failure, and the "promising direction" claim currently rests on this assumption.
- [Section 4.2 and Section 5] The interview questions in Table 1 ask about "potential use cases," "pros," and "cons," so the participants' responses are perceptions of the demonstration, not evidence of the tool's effectiveness in a real classroom setting. The conclusion's assertion that "the results raise concerns about the accuracy and interpretability of LLM-generated feedback" attributes accuracy concerns to the interviewees, but the paper never measures accuracy directly. The conclusion should explicitly distinguish between perceived limitations (interviewee concerns) and measured limitations (none reported), and the claim that DUET "offers a foundation for future classroom integration" should be softened until the accuracy of the pipeline is demonstrated.
minor comments (6)
- [Section 2.2] The statement that "LLMs still lack the ability to extract high-detail information from images effectively" is too strong when the same paper cites, in Section 2.3, studies in which GPT-4 produced near-perfect PlantUML conversions with no syntax errors over 36 generations; the sentence should be qualified to reflect the mixed evidence.
- [Section 4.1] The evaluation section does not state explicitly that no students participated, even though DUET's primary target includes students; this is a material limitation that should be stated in the methods paragraph.
- [Section 3.2] The workflow describes model selection but does not specify which LLM configuration (e.g., GPT-4o for conversion and Mistral-7B-Instruct-v0.3 for comparison) produced the demonstration shown to interview participants; the paper should state the exact models and prompts used in the evaluation.
- [Section 3 and 4] The paper provides a GitHub link for source code and supplementary materials, but it does not include the predefined prompt templates or a concrete example of the student-view and educator-view feedback output; adding one representative input-output example would substantially improve reproducibility.
- [References] Reference [13] (Rivera et al.) concerns transformer-driven feedback for student program planning, not diagram analysis; the connection to the present work is not immediate, and the paper should either cite more directly relevant work on LLM feedback in modeling education or add a sentence explaining the analogy.
- [Figure 2] The workflow figure abbreviates the pipeline with ellipses and does not clearly show the student-facing and educator-facing feedback views; the caption or the figure should label the two distinct outputs from Step 4.
Circularity Check
No significant circularity: DUET is a tool prototype with a qualitative evaluation, and no claimed prediction or derivation reduces to its inputs.
full rationale
The paper does not present a derivation chain in which an output quantity is constructed from, or fitted to, the same data it claims to predict. DUET is a multi-stage LLM pipeline whose claimed contribution is a workflow and prototype; the evaluation consists of semi-structured interviews with six participants reacting to a demonstration, and the paper explicitly frames this as preliminary. The load-bearing claims about LLM accuracy are not asserted as measured results; instead, the paper cites external studies (e.g., Wang et al. 2024) that report accuracies, and it states in Section 3.2 that extraction quality may vary with image quality and handwriting, acknowledging the unvalidated stage. No equation, fitted parameter, or uniqueness theorem is used to force a conclusion. The only self-citation is reference [13] (Rivera, Steinmaurer, et al.), used in the introduction to support the general claim that LLMs can provide adaptive feedback and engage in dialogues; this is background support for motivation, not a load-bearing premise of the tool's design or evaluation. Consequently, there is no circular step, and the central claim does not reduce to its own inputs.
Assumptions & free parameters
assumptions (3)
- domain assumption GPT-4o can convert diagram images, including hand-drawn ones, into syntactically correct PlantUML code.
- domain assumption A smaller LLM such as Mistral-7B can accurately compare two PlantUML representations and identify structural differences that are pedagogically relevant.
- domain assumption Structural differences such as missing classes, changed attributes, and wrong relationships are a sufficient basis for reflective learning feedback.
Cite this review
Pith. "Pith review of Automated Feedback on Student-Generated UML and ER Diagrams Using Large Language Models." pith.science (2026). https://pith.science/paper/IY3LUIVU
@misc{pith2026250723470,
author = {Pith},
title = {Pith review of: Automated Feedback on Student-Generated UML and ER Diagrams Using Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/IY3LUIVU}},
note = {Machine review of arXiv:2507.23470}
}
read the original abstract
UML and ER diagrams are foundational in computer science education but come with challenges for learners due to the need for abstract thinking, contextual understanding, and mastery of both syntax and semantics. These complexities are difficult to address through traditional teaching methods, which often struggle to provide scalable, personalized feedback, especially in large classes. We introduce DUET (Diagrammatic UML & ER Tutor), a prototype of an LLM-based tool, which converts a reference diagram and a student-submitted diagram into a textual representation and provides structured feedback based on the differences. It uses a multi-stage LLM pipeline to compare diagrams and generate reflective feedback. Furthermore, the tool enables analytical insights for educators, aiming to foster self-directed learning and inform instructional strategies. We evaluated DUET through semi-structured interviews with six participants, including two educators and four teaching assistants. They identified strengths such as accessibility, scalability, and learning support alongside limitations, including reliability and potential misuse. Participants also suggested potential improvements, such as bulk upload functionality and interactive clarification features. DUET presents a promising direction for integrating LLMs into modeling education and offers a foundation for future classroom integration and empirical evaluation.
Figures
Reference graph
Works this paper leans on
-
[1]
Aakash Ahmad, Muhammad Waseem, Peng Liang, Mahdi Fahmideh, Mst Shamima Aktar, and Tommi Mikkonen. 2023. Towards human-bot collabo- rative software architecting with chatgpt. InProceedings of the 27th international conference on evaluation and assessment in software engineering. 279–285
work page 2023
-
[2]
Seiko Akayama, Birgit Demuth, Timothy C Lethbridge, Marion Scholz, Perdita Stevens, and Dave R Stikkolorum. 2013. Tool use in software modelling educa- tion. InProceedings of the Educators’ Symposium co-located with ACM/IEEE 16th International Conference on Model Driven Engineering Languages and Systems (MODELS 2013)
work page 2013
-
[3]
Pasquale Ardimento, Lerina Aversano, Mario Luca Bernardi, Vito Alessandro Carella, Marta Cimitile, and Michele Scalera. 2023. UML Miner: A Tool for Mining UML Diagrams. In2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). 30–34. doi:10.1109/ MODELS-C59198.2023.00014
arXiv 2023
-
[4]
Javier Cámara, Javier Troya, Lola Burgueño, and Antonio Vallecillo. 2023. On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML.Software and Systems Modeling22, 3 (2023), 781–793
work page 2023
-
[5]
Stanislav Chren, Barbora Buhnova, Martin Macak, Lukas Daubner, and Bruno Rossi. 2019. Mistakes in UML Diagrams: Analysis of Student Projects in a Software Engineering Course. In2019 IEEE/ACM 41st International Conference on Sebastian Gürtl, Gloria Schimetta, David Kerschbaumer, Michael Liut, and Alexander Steinmaurer Software Engineering: Software Enginee...
arXiv 2019
-
[6]
Aaron Conrardy and Jordi Cabot. 2024. From image to uml: first results of image based uml diagram generation using llms.arXiv preprint arXiv:2404.11376(2024)
arXiv 2024
-
[7]
Gabriele De Vito, Fabio Palomba, Carmine Gravino, Sergio Di Martino, and Filomena Ferrucci. 2023. Echo: An approach to enhance use case quality ex- ploiting large language models. In2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 53–60
work page 2023
-
[8]
Rutwa Engineer, Volodymyr Yaremchuk, Eren Suner, Omar Khamis, Alex Apos- tolu, and Arthur Ng. 2024. UML Mentor: A Tool for Interactive and Collaborative Software Design Education. InProceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 2(Virtual Event, NC, USA)(SIGCSE Vir- tual 2024). Association for Computing Machinery, New York...
Show all 18 references
-
[9]
John Erickson and Keng Siau. 2007. Can UML be simplified? Practitioner use of UML in separate domains. (2007)
2007
-
[10]
Giacomo Garaccione, Riccardo Coppola, Luca Ardito, and Marco Torchiano. 2025. Gamification of conceptual modeling education: an analysis of productivity and students’ perception.Software Quality Journal33, 1 (2025), 1–19
2025
-
[11]
Marian Petre. 2013. UML in practice. In2013 35th International Conference on Software Engineering (ICSE). 722–731. doi:10.1109/ICSE.2013.6606618
2013
-
[12]
Rebecca Reuter, Theresa Stark, Yvonne Sedelmaier, Dieter Landes, Jürgen Mottok, and Christian Wolff. 2020. Insights in Students’ Problems during UML Modeling. In2020 IEEE Global Engineering Education Conference (EDUCON). 592–600. doi:10. 1109/EDUCON45650.2020.9125110
2020
-
[13]
Elijah Rivera, Alexander Steinmaurer, Kathi Fisler, and Shriram Krishnamurthi
-
[14]
Joseph Romeo, Marco Raglianti, Nagy Csaba, and Michele Lanza. 2025. UML is back. Or is it?. InICSE 2025 47th International Conference on Software Engineering
2025
-
[15]
Cigdem Sengul, Rumyana Neykova, and Giuseppe Destefanis. 2024. Software engineering education in the era of conversational AI: current trends and future directions.Frontiers in Artificial Intelligence7 (2024), 1436350
2024
-
[16]
Beian Wang, Chong Wang, Peng Liang, Bing Li, and Cheng Zeng. 2024. How LLMs aid in UML modeling: an exploratory study with novice analysts. In2024 IEEE International Conference on Software Services Engineering (SSE). IEEE, 249– 257
2024
-
[17]
Chong Wang, Beian Wang, Peng Liang, and Jie Liang. 2024. Assessing UML Models by ChatGPT: Implications for Education.arXiv preprint arXiv:2412.17200 (2024)
2024
-
[2024]
InProceedings of the 2024 on Innovation and Technology in Computer Science Education V
Iterative Student Program Planning using Transformer-Driven Feedback. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1(Milan, Italy)(ITiCSE 2024). Association for Computing Machinery, New York, NY, USA, 45–51. doi:10.1145/3649217.3653607
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.