{"id":"724480ba-b43a-486b-87d8-183b092217f1","arxiv_id":"2507.23470","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DUET uses a two-stage LLM pipeline to convert and compare UML and ER diagrams and generate automated student feedback, but its only evaluation so far is six qualitative interviews.","lead":"DUET is a prototype tool that turns student-drawn UML and ER diagrams into text and uses large language models to compare them with an instructor's reference, generating structured feedback. The paper reports only qualitative interview results from six educators and teaching assistants, not measurements of feedback accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DUET's core accuracy is unmeasured: the image-to-PlantUML conversion and the smaller-LLM comparison are never validated, despite cited prior work showing GPT-4o's UML analysis can be unreliable. The 'promising direction' claim therefore rests on an untested assumption.","rationale":"The reader's conditional verdict is appropriate. I agree with the weakest_assumption: the conversion and comparison stages are the load-bearing components of the tool's value proposition. My stress-test confirms that no quantitative evidence exists for either stage. The paper itself is transparent about this, citing variability in extraction quality and participant concerns about reliability, but transparency does not substitute for measurement. The related work section provides direct evidence that GPT-4o's UML analysis is far from perfect, which raises the prior probability that DUET's pipeline will make errors on realistic hand-drawn student diagrams. The evaluation cannot mitigate this because interviewees evaluated a demonstration, not actual outputs on their students' work. That said, the central claim is deliberately modest ('promising direction', 'foundation for future work'), and the authors include explicit limitations and future validation plans. A rejection would overstate the problem; the claim is not that DUET is proven effective, only that it is a plausible prototype worth studying. The concrete test I propose would either convert the conditional acceptability into an evidence-backed one or force a revision of the claim to focus on the pipeline's limitations and the need for human oversight. Since the reader already assigned CONDITIONAL, my verdict recommendation is UNCHANGED: the concern is real but does not move the verdict; it sharpens the condition under which the paper should be considered acceptable.","tokens_in":7386,"tokens_out":3090,"duration_ms":30421,"concrete_test":"Build a benchmark from real student submissions: collect 20-30 UML/ER diagrams (half hand-drawn, half digital) from an existing intro databases/SE course, each paired with an instructor reference solution. Have two human experts independently transcribe each diagram into PlantUML and annotate structural differences against the reference. Run DUET on each pair. Compute (a) conversion fidelity: precision/recall/F1 of extracted classes/entities, attributes, and relationships against the expert transcriptions, and (b) feedback accuracy: precision/recall/F1 of DUET's reported differences against the expert difference annotations. Also run each pair 3 times to measure output consistency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single most load-bearing assumption is that DUET's two-stage LLM pipeline (GPT-4o image-to-PlantUML conversion in Step 2, then a smaller LLM such as Mistral-7B for structural comparison in Step 3) produces sufficiently accurate difference lists for feedback to be useful. The paper provides no accuracy measurement for either stage. Section 2.3 cites evidence that GPT-4o's UML assessment matched human instructors only about 50% of the time and identified only 25% of class relationships correctly (Wang et al., 2024); related work also documents persistent minor errors in hand-drawn-to-PlantUML conversion. Since students may submit hand-drawn diagrams of varying legibility, and Section 3.2 itself concedes that extraction quality may vary with image quality and handwriting, the failure mode is not hypothetical. The evaluation in Section 4 consists of six participants reacting to a demonstration of the workflow, not to real output on their own students' diagrams; they did not verify the correctness of the generated feedback. Thus, the 'promising direction' claim currently rests on an unverified pipeline, and the participants' own reliability concerns (Section 4.2) cannot be checked against any quantitative evidence. If the first LLM misreads a relationship or the comparison LLM hallucinates a difference, the entire feedback is misleading to learners. This is an empirical gap, not an internal inconsistency, but it is the load-bearing place where the central claim could fail.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents DUET (Diagrammatic UML & ER Tutor), a prototype web-based tool that uses a multi-stage LLM pipeline to provide automated feedback on student-generated UML and ER diagrams. In Step 1, users upload an instructor-provided reference diagram and a student diagram. Step 2 uses GPT-4o to convert both images into PlantUML textual representations. Step 3 uses a smaller LLM, such as Mistral-7B-Instruct-v0.3, to compare the two PlantUML representations and produce a list of structural differences. Step 4 uses the same smaller LLM to generate structured, reflective feedback for students and analytical insights for educators. The evaluation is a qualitative study based on semi-structured interviews with two university lecturers and four teaching assistants, asking about potential use cases, pros, cons, and suggested features. The participants identified strengths including accessibility, scalability, and instant feedback, and limitations including reliability concerns and potential for misuse. The authors conclude that DUET represents a promising direction for integrating LLMs into modeling education and offers a foundation for future classroom integration and empirical evaluation.","tokens_in":7643,"tokens_out":3963,"duration_ms":38748,"significance":"If the pipeline works reliably, DUET addresses a real need: scalable, personalized feedback for UML and ER diagram exercises in large courses, where manual feedback from instructors and TAs does not scale. The paper is honest about its scope, explicitly positioning DUET as a functional prototype and acknowledging the small sample and hypothetical scenarios in the evaluation. The source code and supplementary materials are made available in a GitHub repository, which supports reproducibility. The design is transparent: the workflow is described step by step, and the prompts are predefined and externally configurable. The main weakness is that the central functional claim—that DUET provides accurate and therefore useful feedback—is not measured anywhere in the evaluation. The study relies entirely on participants' perceptions of a demonstration, not on correctness checks of the generated differences or feedback. Given the paper's own cited evidence that GPT-4o-based UML assessment matched human instructors in only about 50% of cases and identified only 25% of class relationships correctly, the unvalidated conversion and comparison stages are the load-bearing risk in this work.","major_comments":[{"comment":"The evaluation does not measure the accuracy of the core pipeline: the image-to-PlantUML conversion (Section 3.2, Step 2) and the structural comparison (Section 3.2, Step 3) are never checked against ground truth. The six participants reacted to a demonstration of the workflow, not to output produced for their own students' diagrams, and the paper reports no verification that the generated difference lists or feedback were correct. Since the paper's central claim is that DUET \"provides structured feedback based on the differences,\" it is load-bearing to show that those differences are accurate. I recommend adding a small quantitative study on real student diagrams (e.g., 20-30 diagrams) with human-coded ground truth for classes, attributes, relationships, and multiplicities, reporting conversion fidelity and agreement between generated feedback and instructor judgments.","section":"Section 4.1-4.2"},{"comment":"The manuscript explicitly acknowledges that \"the quality of the extracted PlantUML code may vary based on image quality, diagram clarity, handwriting legibility (for hand-drawn diagrams), and adherence to standard notations,\" but it provides no evidence about how this variation affects feedback quality. This is not a hypothetical concern: the paper's own related work (Section 2.3) cites Wang et al. (2024) showing GPT-4-based UML assessments matched human instructors in about 50% of cases and identified only 25% of class relationships correctly, and GPT-4o still made minor mistakes in hand-drawn-to-PlantUML conversion. Since DUET explicitly accepts hand-drawn student diagrams, the unverified first stage is the most likely point of failure, and the \"promising direction\" claim currently rests on this assumption.","section":"Section 3.2, Step 2"},{"comment":"The interview questions in Table 1 ask about \"potential use cases,\" \"pros,\" and \"cons,\" so the participants' responses are perceptions of the demonstration, not evidence of the tool's effectiveness in a real classroom setting. The conclusion's assertion that \"the results raise concerns about the accuracy and interpretability of LLM-generated feedback\" attributes accuracy concerns to the interviewees, but the paper never measures accuracy directly. The conclusion should explicitly distinguish between perceived limitations (interviewee concerns) and measured limitations (none reported), and the claim that DUET \"offers a foundation for future classroom integration\" should be softened until the accuracy of the pipeline is demonstrated.","section":"Section 4.2 and Section 5"}],"minor_comments":[{"comment":"The statement that \"LLMs still lack the ability to extract high-detail information from images effectively\" is too strong when the same paper cites, in Section 2.3, studies in which GPT-4 produced near-perfect PlantUML conversions with no syntax errors over 36 generations; the sentence should be qualified to reflect the mixed evidence.","section":"Section 2.2"},{"comment":"The evaluation section does not state explicitly that no students participated, even though DUET's primary target includes students; this is a material limitation that should be stated in the methods paragraph.","section":"Section 4.1"},{"comment":"The workflow describes model selection but does not specify which LLM configuration (e.g., GPT-4o for conversion and Mistral-7B-Instruct-v0.3 for comparison) produced the demonstration shown to interview participants; the paper should state the exact models and prompts used in the evaluation.","section":"Section 3.2"},{"comment":"The paper provides a GitHub link for source code and supplementary materials, but it does not include the predefined prompt templates or a concrete example of the student-view and educator-view feedback output; adding one representative input-output example would substantially improve reproducibility.","section":"Section 3 and 4"},{"comment":"Reference [13] (Rivera et al.) concerns transformer-driven feedback for student program planning, not diagram analysis; the connection to the present work is not immediate, and the paper should either cite more directly relevant work on LLM feedback in modeling education or add a sentence explaining the analogy.","section":"References"},{"comment":"The workflow figure abbreviates the pipeline with ellipses and does not clearly show the student-facing and educator-facing feedback views; the caption or the figure should label the two distinct outputs from Step 4.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest, well-structured, and clearly positioned as a prototype paper, which fits venues such as ITiCSE, SIGCSE, or a modeling-education workshop. The single most important gap is the complete absence of any accuracy measurement for the conversion and comparison stages. Because the paper itself cites prior evidence that GPT-4o-based UML analysis is unreliable, the unvalidated pipeline is a genuine risk to the central claim rather than a stylistic concern. The qualitative study is a fair start, but the conclusion should not be accepted without either a small ground-truth accuracy evaluation or a substantially narrowed claim. I would be willing to see a revised version that adds even a modest accuracy study on a small set of real diagrams."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a pleasant paper to read: the authors describe a concrete prototype (DUET) that uses GPT-4o to turn diagram images into PlantUML and a smaller LLM (Mistral-7B) to compare student and reference diagrams and produce structured feedback for students and educators. The code is on GitHub, the architecture is simple and clearly explained, and the related work is honestly cited—including the Wang et al. findings that GPT-4o's UML grading matched instructors only about half the time and spotted just 25% of class relationships. That is exactly the right context for a prototype that depends on that conversion and comparison.\n\nWhat is genuinely new is the specific two-stage pipeline applied to both UML and ER diagrams, with separate feedback views for students and instructors. The interview study with six participants (two lecturers, four TAs) is methodologically transparent and the participants' stated concerns—reliability, non-determinism, misuse—are sensible and align with the cited evidence.\n\nThe soft spot is exactly what the stress-test flagged, and the reader's conditional verdict is right: the central functional claim is never measured. There is no check of whether GPT-4o's PlantUML output is structurally correct, and no check of whether the smaller LLM's difference list is accurate. The paper cites prior work suggesting both stages can fail, so this is not a hypothetical worry. The evaluation interviews were based on a demonstration, not on real student diagrams or real feedback quality; the six participants could not verify output correctness. To be fair, the authors do not overclaim. They call DUET a prototype and say the study involved hypothetical scenarios, and the conclusion explicitly frames this as a foundation for future empirical evaluation. So the gap is empirical rather than a logical contradiction, but it is load-bearing: 'promising direction' is an untested assumption at this point.\n\nI would send this to review rather than desk-reject it. It is a well-written, honest tool paper with open-source code and a sensible qualitative evaluation design. A serious referee should push for a small accuracy study—even a few dozen diagrams with known ground truth—or a clear reframing as a design/UX study of perceived usefulness. For a CS education venue at workshop level, it is likely fine with revisions; for a fuller venue, the missing validation needs to be addressed. I would bring it to a reading group as an example of honest prototyping, and I would cite it if I were working on LLM-based diagram feedback, with the caveat that its effectiveness claims are unverified.","headline":"A clear, honest prototype for LLM-generated feedback on UML/ER diagrams, but the load-bearing conversion/comparison accuracy is never measured and the six-interview evaluation only captures perceptions.","tokens_in":8210,"tokens_out":2595,"would_cite":true,"duration_ms":26078,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DUET is a prototype LLM-based tool that converts instructor and student UML and ER diagrams into PlantUML text, compares them, and generates structured feedback for both students and educators.","keywords":["Large Language Models","Automated Feedback","Unified Modeling Language","Entity-Relationship Diagram","CS Education","PlantUML","Diagram Comparison","Student Support"],"falsifier":"Run DUET on a corpus of, say, fifty student-drawn UML and ER diagrams whose errors have been annotated by two instructors, then measure whether the tool's identified differences match the annotations; the central claim fails if the image-to-PlantUML step misreads a substantial share of legible hand-drawn diagrams or if the tool's agreement with the rubric is no better than chance.","tokens_in":7179,"feed_emoji":"🤖","tokens_out":3727,"duration_ms":36142,"temperature":0.7,"pith_summary":"This paper asks whether large language models can provide useful automated feedback on student-generated UML and ER diagrams, where classes are large and individualized feedback is scarce. To answer it, the authors built DUET, a prototype that converts both the instructor's reference diagram and the student's submission into PlantUML text, lets a small LLM compare the two, and returns structured, reflective feedback in separate views for students and educators. The paper argues that this approach is a promising direction for integrating LLMs into modeling education, not a replacement for human teaching staff. Six educators and teaching assistants interviewed about the prototype saw concrete benefits in accessibility and scalability, while raising concerns about reliability and potential misuse. If the pipeline works as described, DUET would let students practice and revise independently and give instructors a window into common misconceptions at scale.","feed_headline":"LLM tutor turns UML and ER diagrams into instant feedback","feed_subtitle":"DUET converts reference and student diagrams to text, compares them, and coaches both learners and instructors.","key_machinery":"The load-bearing mechanism is a four-step pipeline: upload of a reference and a student diagram, conversion of both images into PlantUML text by GPT-4o, structural comparison of the two text representations by a smaller LLM such as Mistral-7B-Instruct-v0.3, and generation of two categories of feedback from that comparison, one aimed at students and one at educators. The conversion step is what makes the approach feasible, since the authors argue that LLMs process textual diagram representations more reliably than raw images, and the comparison step deliberately restricts itself to objective differences before any feedback wording is produced.","core_discovery":"The paper's central claim is that a two-stage LLM pipeline can turn diagram-based modeling exercises into a scalable, personalized feedback loop. DUET converts an instructor reference diagram and a student-submitted UML or ER diagram into PlantUML text with GPT-4o, has a smaller LLM such as Mistral-7B-Instruct-v0.3 compare the two text representations, and then generates structured feedback: reflective hints for the student and misconception insights for the educator. The authors position DUET not as an assessment engine but as a supplementary formative tool that gives instant, repeatable practice opportunities and aggregates signals about common student misunderstandings. The supporting evidence is qualitative: semi-structured interviews with two lecturers and four teaching assistants who saw accessibility, scalability, and learning support as strengths, while flagging nondeterministic output, reliability, and cheating risk as limitations.","pith_inferences":["Editorial inference: the same image-to-text-then-compare pipeline could generalize to other diagram notations such as BPMN, SysML, or architecture diagrams, since nothing in the architecture is UML-specific beyond the prompt templates.","Editorial inference: if conversion reliability improves, the educator-facing view could evolve into a real-time misconception dashboard, an application the interviews only gesture at.","Editorial inference: a direct quantitative study comparing DUET's feedback with teaching assistant feedback on identical submissions, scored by a third expert, would test the core assumption that the interviews could not.","Editorial inference: limiting the student view to hints rather than corrections is a deliberate pedagogical choice; a version that leaked 'correct answer' wording would undermine the reflective-learning goal the paper emphasizes."],"forward_implications":["Students in large courses could get immediate feedback on diagram drafts before submitting, reducing dependence on teaching assistant availability.","Educators could aggregate feedback data across many submissions to identify frequent misconceptions and adjust their teaching accordingly.","The same pipeline could accept a single reference solution and many student submissions, scaling formative assessment without adding instructor labor.","Feedback designed as reflective hints rather than right or wrong labels could encourage iterative self-correction and deeper engagement with modeling concepts.","DUET could be used as a pre-lab preparation activity that helps students arrive at tutorials with more refined questions and clearer design reasoning."],"supporting_citations":[{"why":"Establishes PlantUML as the most used text-based UML tool, motivating the decision to convert diagrams into textual form for LLM processing.","marker":"[14]"},{"why":"Shows that GPT-4 can convert hand-drawn UML diagrams to PlantUML with very few syntax errors, grounding the diagram recognition stage.","marker":"[6]"},{"why":"Reports that GPT-4 outperformed other multimodal LLMs in converting hand-drawn UML diagrams, supporting the choice of GPT-4o.","marker":"[4]"},{"why":"Documents LLM accuracy on identifying classes, operations, attributes, and relationships in student UML models, informing the expected strengths and weaknesses of the comparison step.","marker":"[16]"},{"why":"Reports that GPT-4 grading matched a human instructor in about half of cases, which the authors use to argue that human oversight remains necessary.","marker":"[17]"},{"why":"Shows transformer-driven feedback used for student program planning, providing a precedent for LLM-generated formative feedback in computing education.","marker":"[13]"},{"why":"Describes an existing automated evaluation tool with gamified live feedback that increased diagram correctness, serving as a comparison baseline for DUET's contribution.","marker":"[10]"},{"why":"Depicts an existing UML evaluation tool that relies on manual feedback from educators, highlighting the gap DUET aims to close.","marker":"[3]"}],"fun_headline_variants":["LLM tutor delivers instant feedback on UML and ER diagrams","DUET: AI that converts diagrams to text for personalized coaching","AI-driven feedback for student UML and ER diagrams","Automated diagram feedback tool uses LLMs to coach students"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire pipeline rests on the assumption that GPT-4o reliably turns any student diagram, including messy hand-drawn ones, into correct PlantUML code, and that comparing those text versions captures the real structural differences between diagrams.","fun_headline_variants_meta":{"raw":{"variants":["LLM tutor delivers instant feedback on UML and ER diagrams","DUET: AI that converts diagrams to text for personalized coaching","AI-driven feedback for student UML and ER diagrams","Automated diagram feedback tool uses LLMs to coach students"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3061,"prompt_tokens":909,"completion_tokens":2152,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2086}},"tokens_in":525,"tokens_out":2152,"duration_ms":16220,"temperature":1.0,"reasoning_tokens":2086,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:41:54.266640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DUET on a corpus of, say, fifty student-drawn UML and ER diagrams whose errors have been annotated by two instructors, then measure whether the tool's identified differences match the annotations; the central claim fails if the image-to-PlantUML step misreads a substantial share of legible hand-drawn diagrams or if the tool's agreement with the rubric is no better than chance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes PlantUML as the most used text-based UML tool, motivating the decision to convert diagrams into textual form for LLM processing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports that GPT-4 outperformed other multimodal LLMs in converting hand-drawn UML diagrams, supporting the choice of GPT-4o."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents LLM accuracy on identifying classes, operations, attributes, and relationships in student UML models, informing the expected strengths and weaknesses of the comparison step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports that GPT-4 grading matched a human instructor in about half of cases, which the authors use to argue that human oversight remains necessary."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows transformer-driven feedback used for student program planning, providing a precedent for LLM-generated formative feedback in computing education."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes an existing automated evaluation tool with gamified live feedback that increased diagram correctness, serving as a comparison baseline for DUET's contribution."}],"review_version":1}