{"id":"72f56c68-81a0-4e9a-8dd1-e753a4f22ab6","arxiv_id":"2411.17976","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Multimodal GPTs can turn UML class, sequence, and hand-drawn activity diagrams into working Python code, and can reverse engineer code back into diagrams, demonstrated across several software engineering tasks.","lead":"This paper explores whether multimodal AI assistants, such as GPT-4, can use UML diagrams and other visual models as inputs for generating, updating, and summarizing code. The authors present a set of software engineering use cases where diagrams are combined with natural language prompts to guide code generation and reverse engineering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'watershed' claim lacks a text-only baseline: visual structure may not be load-bearing.","rationale":"Reading the paper in good faith, it is a clear demonstration of plausible use cases, with reproducible supplementary material and honest reporting of some failures (PlantUML syntax errors in Section 4.3, missed cardinality constraints in Section 4.1). These are real contributions. The reader's CONDITIONAL verdict is appropriate because the strong 'watershed' and 'first' claims rest on a small set of selected examples rather than systematic evaluation. My concern sharpens the external-validity gap into a specific missing control: the paper never tests whether the visual structure in the diagrams actually matters, as opposed to the textual content embedded in them. This is directly load-bearing for the title claim about the 'importance of visual modelling languages.' The reader's weakest assumption concerned reliability of semantic extraction; my concern is related but distinct, focusing on whether even reliable extraction is necessary. I therefore partially agree. The proposed text-only baseline test would settle the question: if text-only prompting matches the multimodal results, the central benefit claim fails; if text-only prompting is worse, the visual modality is shown to matter. Since the reader already requested systematic evaluation and baselines, the verdict remains CONDITIONAL with no adjustment needed.","tokens_in":10719,"tokens_out":4580,"duration_ms":42226,"concrete_test":"Re-run the Section 4.1 and Section 4.3 use cases with a text-only prompt that fully serialises the diagram content: for Figure 1, provide a PlantUML source or a structured textual list of classes, attributes, operations, associations with multiplicities and labels; for Figure 8, provide a step-by-step textual description or PlantUML of the activity diagram. Use the same model (e.g., Copilot or GPT-4o) with default settings and chain the same prompts. Compare generated Python code and reverse-engineered diagrams against the multimodal outputs using fidelity criteria such as presence of all classes/methods, correct handling of cardinality constraints, and behavioural equivalence. If the text-only variant achieves equal or better fidelity, the visual modality is not load-bearing for the claimed use cases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that multimodal GPTs make visual modelling languages important in generative software engineering (Abstract, Section 5). The key asserted advantage is that \"diagram-based prompting may be advantageous, in terms of information transfer efficiency, compared to pure natural language prompting\" (Section 5). Yet no experiment compares diagram input against a text-only serialisation of the same diagram content. In Section 4.1, Figure 1's relevant content (classes, attributes, operations, associations with multiplicity labels) is fully expressible as text; in Section 4.3, the hand-drawn activity diagram can be described step by step. If GPT-4 performs equally well given a text-only description, the demonstrated outcomes stem from reading textual labels, not from parsing the visual layout, cardinality notation, or spatial relationships. The paper's own observation in Section 4.1 that the reconstructed class diagram only partly matches the original because cardinality constraints are not fully captured shows that image understanding is imperfect. Without an ablation or baseline, the paper asserts rather than demonstrates that visual structure itself contributes value. This is load-bearing because the title and strongest claim rest on the visual modality adding something beyond what text-only prompting already provides.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that multimodal GPT models such as GPT-4 mark a 'watershed' for generative software engineering because they can accept prompts that mix images of UML diagrams with natural language. The author presents a portfolio of use cases: generating Python classes from a class diagram, implementing dynamic behaviour from a sequence diagram, generating code from a hand-drawn activity diagram, applying design patterns from a class diagram, and building larger applications (an expression evaluator, tic-tac-toe, and checkers) guided by UML diagrams. The paper also demonstrates reverse engineering of PlantUML diagrams from code and reports partial successes and failures. The central claim is that diagram-based prompting is advantageous over pure natural language prompting in information-transfer efficiency, and that these use cases open new research directions in software engineering and generative AI.","tokens_in":10902,"tokens_out":3624,"duration_ms":35263,"significance":"If the central claim were rigorously established, the paper would open a useful direction for software engineering: visual modelling languages could become a practical interface to generative code assistants, and reverse-engineered diagrams could support code summarisation and documentation. The paper has concrete strengths: it provides documented prompt chains, includes a supplementary Gemini-based Jupyter notebook for reproducibility, and candidly reports cases where the generated output only partially matches the input diagram. However, the evidence is anecdotal, lacks a text-only baseline, and does not quantify errors or selection. The significance is therefore conditional; the paper is better read as an experience report than as a demonstration that visual structure itself is load-bearing.","major_comments":[{"comment":"The central claim that 'diagram-based prompting may be advantageous, in terms of information transfer efficiency, compared to pure natural language prompting' is never tested against a text-only serialisation of the same diagram content. In Section 4.1 and Section 4.3, the relevant content of Figure 1 and Figure 8 (classes, attributes, operations, multiplicities, activity steps) can be fully expressed as text, yet no comparison shows whether the visual layout, cardinality notation, or spatial relationships contribute anything beyond the textual labels. Because the title and abstract rest on the visual modality adding value, this missing baseline is load-bearing.","section":"Abstract, Section 5"},{"comment":"The paper reports that the reconstructed class diagram 'only partly matches the original design' because cardinality constraints are not fully captured. This is an admission that image understanding is imperfect, but the work does not quantify how often or in what ways the multimodal model fails to extract intended semantics from diagram images. Without such analysis, the assumption that the workflow reliably depends on the diagram as a technical language remains unsupported.","section":"Section 4.1, paragraph after Figure 2"},{"comment":"The evaluation is anecdotal and lacks systematic measurement. There are no error counts, no repeated runs, no comparison conditions, and no explicit acceptance criteria. For example, Section 4.3 mentions PlantUML syntax errors in the generated activity diagram, and Section 4.5 reports ten corrective prompts for the checkers application, including game-logic errors such as a piece not being removed after a jump; these are treated as inconsequential, but they directly bear on the reliability of the proposed workflow and should be analysed rather than dismissed.","section":"Sections 4.3 and 4.5"},{"comment":"The methodology does not specify how the presented use cases were selected or whether less successful attempts were omitted. This makes it impossible to infer the success rate of the proposed approach or to rule out selection bias. The authors should disclose the full set of attempted interactions or explicitly state the selection criteria, or alternatively frame the paper as an exploratory demonstration rather than an evaluation.","section":"Section 3"}],"minor_comments":[{"comment":"The prompt text contains a typo: 'Extend the UML diagram by using the Adapter design patter' should read 'pattern'.","section":"Section 4.4"},{"comment":"References to 'the SM' are vague; the supplementary material should include explicit pointers to file names, notebook cells, and line numbers for each claimed output so that readers can verify the reported interactions.","section":"Sections 4.1-4.5"},{"comment":"The claim that 'no other work has investigated similar use cases' is difficult to verify and should be scoped with a literature cut-off date or a more careful formulation, especially since the paper itself cites related work on image-based UML generation by Conrardy and Cabot.","section":"Abstract and Section 5"},{"comment":"The text says the reconstructed class diagram 'only partly matches' the original but does not provide a visual diff or a list of mismatches; adding such a comparison would help readers judge the severity of the discrepancy.","section":"Section 4.1"},{"comment":"The description of 'Voting/self-consistency' and other prompting techniques is brief and would benefit from concrete examples tied to the use cases that follow.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an experience report with useful documentation and a reproducibility notebook, but the title and abstract claim more than the evidence supports. The missing text-only baseline is a substantial gap because the 'importance of visual modelling languages' claim depends on showing that the visual channel adds value beyond the textual labels. I would encourage the editor to request a major revision that either adds a controlled comparison or substantially tempers the claims to match the anecdotal nature of the study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper does something real: it shows, across several worked examples, that a multimodal LLM can take a UML diagram as input and generate working code, and can also reverse-engineer code back into PlantUML diagrams. That particular bundle—diagram-to-code and code-to-diagram in one workflow—is not something I have seen assembled before, and the paper's literature positioning is careful enough that I believe the 'first' claims are plausible. Second, the strong title and the 'watershed' framing are not backed by the evidence. The paper argues that diagram-based prompting beats pure text prompting on 'information transfer efficiency,' but it never runs the comparison. A text-only serialisation of the same diagram content would be a cheap control, and without it the central claim that the visual structure itself is load-bearing is asserted, not demonstrated.\n\nWhat the paper does well: it is clearly written, the use cases are genuinely instructive, and the supplementary material ships a Gemini pipeline plus chat logs, which is more than most vision papers bother with. The author also mentions failures in passing—cardinality constraints not fully captured in the reconstructed class diagram, minor PlantUML syntax errors in the activity diagram, and multiple error-correction rounds in the checkers game. That honesty is a point in the paper's favour, even though those same failures are then waved away rather than analysed.\n\nWhere the soft spots are: the evidence is anecdotal. There is no systematic evaluation, no count of how many prompts succeeded or failed, no baseline against text-only prompting, and no analysis of the failure modes. The checkers case, in particular, needed ten prompts just to fix errors after nine implementation prompts; that is not a sign of a dependable technical language, it is a demo of a patient user and a forgiving model. The paper's own observation that the code did not fully capture all cardinality constraints shows the image understanding is imperfect. The 'watershed' language and the claim that the visual channel is 'intensely high-resolution' come from Tufte and a general compression argument, not from any measurement in this study.\n\nWho this is for: people working on LLM-based SE tooling, especially those interested in multimodal prompting, will get value from the worked examples and the identified research gaps. It deserves a serious referee, but it should be revised before acceptance: add at least one text-only baseline, report errors systematically, and temper the 'first' and 'watershed' language to match what is actually demonstrated. The core idea is worth engaging; the current packaging oversells it.","headline":"A useful bundle of UML-to-code demonstrations, but the 'watershed' claim needs a text-only baseline before it carries weight.","tokens_in":11431,"tokens_out":1539,"would_cite":false,"duration_ms":17187,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal GPTs let developers turn UML diagrams and hand-drawn sketches directly into working Python code.","keywords":["multimodal GPT","UML diagrams","code generation","software engineering","generative AI","visual modelling languages","diagram-based prompting","reverse engineering"],"falsifier":"A concrete check would be to run the same seven use-case prompts against a text-only language model given a natural-language description of each diagram (listing the classes, relations, cardinalities, and control flow) and compare the resulting code with the diagram-guided output on the same correctness criteria; if the text-only baseline matches or beats the diagram-guided version, the paper's claim that diagrams add unique value collapses. A second, cheaper check is to audit diagram-to-code outputs on a corpus of diagrams with explicit labels and cardinality constraints and count how often the generated code misses those constraints, as the paper itself reports for the reconstructed class diagram.","tokens_in":10493,"feed_emoji":"📐","tokens_out":11738,"duration_ms":87131,"temperature":0.7,"pith_summary":"The paper aims to establish that multimodal GPTs—models that accept both images and text—are a turning point for the interaction between software engineering and generative AI. The author argues that UML diagrams and hand-drawn sketches, which developers already use for communication, can act as a high-resolution and denotative channel for guiding code generation, code summarisation, and software design. Seven use cases in the paper show a multimodal model translating class, sequence, state, and activity diagrams into working Python code, and also reverse-engineering code back into PlantUML diagrams. If the claim is right, developers gain a practical way to combine visual modelling languages with generative AI, without needing diagrams to be complete specifications—the reason earlier code-generation CASE tools struggled. The paper positions these demonstrations as the first investigation of SE tasks via multimodal GPTs prompted with a mix of diagrams and natural language.","feed_headline":"UML diagrams plus GPT-4 write working Python code","feed_subtitle":"Seven use cases show GPT-4 turning UML and hand-drawn diagrams into code and summaries.","key_machinery":"The central mechanism is the multimodal diagram-and-text prompt chain: a chat in which the user attaches an image of a UML or hand-drawn diagram together with a short text instruction, and each successive prompt builds on the previous outputs. The paper's key move is to treat visual modelling languages as denotative technical languages—they mean one thing and one thing only—in contrast to connotative natural language, and to let the diagram carry the structural information while the text carries the task instruction. PlantUML notation serves as the round-trip format: code can be produced from a diagram, and a diagram can be regenerated from code, which is what allows the author to validate the generated code and to summarise behaviour visually. The demonstrations rely on GPT-4 (via an image-capable chatbot) and an equivalent image-capable pipeline for reproducibility, with all parameters left at defaults and no role or context instructions.","core_discovery":"The paper's central claim is that the advent of multimodal GPTs, such as GPT-4, may represent a watershed in the interplay between software engineering and generative AI, because such models accept image and text inputs and can therefore be prompted with a mix of diagrams and natural language. The core discovery, demonstrated across seven use cases, is that a multimodal GPT can use UML class diagrams, sequence diagrams, state diagrams, and hand-drawn activity diagrams to guide the implementation of Python classes, method behaviour, and complete applications, and can also generate PlantUML diagrams that summarise the structure and behaviour of code. In the author's telling, the model extracts the intended semantics from the diagrams—including relationship labels, states, and even the control flow of a hand-drawn bubble-sort activity diagram—and converts them into working code, while the reverse process produces diagrams that, in one instance, only partly capture the original cardinality constraints. The paper reports this as the first application of multimodal GPTs to software development, software design, and diagram-level code generation, and to code summarisation that returns a diagram.","pith_inferences":["If this workflow becomes dependable, the practical incentive to maintain live UML diagrams rises, potentially reviving modelling practices that declined after earlier code-generation tools failed.","The claims suggest a directly testable hypothesis: on paired diagram-versus-text descriptions of the same design, multimodal prompting should outperform text-only prompting on structural tasks such as class generation and behaviour specification.","The paper's own admission that cardinality constraints were not fully captured points to a natural hardening step: coupling diagram prompting with a verification pass that regenerates a diagram from the produced code and diffs it against the source diagram.","A further extension the author leaves implicit is treating the diagram itself as a test oracle—generating unit tests from the diagram's stated constraints and using them to validate the model's code."],"forward_implications":["Developers can attach UML class, sequence, state, or activity diagrams to a multimodal GPT and get working Python code, so diagrams shift from post-hoc documentation to input specifications.","Code summarisation can take the form of PlantUML diagrams, giving a visual, high-resolution view of a system's structure and behaviour that complements textual summaries.","Design patterns can be injected visually: attaching a diagram of a pattern such as Adapter steers the model's design decisions without a lengthy textual description.","Complex applications (the paper shows tic tac toe and checkers) can be built incrementally through short, chained prompts that start from a class diagram and feed error stack traces back to the model.","Because diagrams compress information relative to natural language, diagram-based prompting is expected to be more information-efficient for SE tasks than text-only prompting."],"supporting_citations":[{"why":"describes GPT-4, the multimodal model whose image-and-text input the paper's use cases exploit.","marker":"[OpenAI, 2024]"},{"why":"provides the empirical finding that developers favour informal hand-drawn diagrams and rarely use UML, motivating the search for a tool to make sketches useful.","marker":"[Baltes and Diehl, 2014]"},{"why":"survey of generative software engineering that identifies the software-design sub-task as a research gap the paper aims to address.","marker":"[Huang et al., 2024]"},{"why":"systematic literature review establishing that existing LLM-for-SE studies rely on text-based data, the gap the multimodal diagram-based approach claims to fill.","marker":"[Hou et al., 2024]"},{"why":"source of the library class-diagram case study used in Section 4.1 for static modelling.","marker":"[Stevens and Pooley, 2006]"},{"why":"defines design patterns, including the Adapter pattern used in Section 4.4 to demonstrate visual few-shot prompting.","marker":"[Gamma et al., 1994]"},{"why":"earlier work building UML class diagrams in PlantUML via ChatGPT text prompts, which the paper extends to image input and code generation.","marker":"[C´amara et al., 2023]"}],"fun_headline_variants":["GPT-4 reads UML and hand-drawn diagrams to write code","Multimodal GPT-4: from diagrams to working software","Seven use cases show GPT-4 turning diagrams into code","UML and sketches: GPT-4 powers code generation","GPT-4 interprets diagrams to generate Python code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the multimodal model can reliably extract the intended meaning from diagram images—including connection labels, cardinality constraints, and hand-drawn notation—so the workflow is only as dependable as the model's image understanding.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 reads UML and hand-drawn diagrams to write code","Multimodal GPT-4: from diagrams to working software","Seven use cases show GPT-4 turning diagrams into code","UML and sketches: GPT-4 powers code generation","GPT-4 interprets diagrams to generate Python code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3228,"prompt_tokens":811,"completion_tokens":2417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":2335}},"tokens_in":427,"tokens_out":2417,"duration_ms":14797,"temperature":1.0,"reasoning_tokens":2335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:38:02.299775+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to run the same seven use-case prompts against a text-only language model given a natural-language description of each diagram (listing the classes, relations, cardinalities, and control flow) and compare the resulting code with the diagram-guided output on the same correctness criteria; if the text-only baseline matches or beats the diagram-guided version, the paper's claim that diagrams add unique value collapses. A second, cheaper check is to audit diagram-to-code outputs on a corpus of diagrams with explicit labels and cardinality constraints and count how often the generated code misses those constraints, as the paper itself reports for the reconstructed class diagram.","supporting_citations":[{"cited_title":"Large language models for software engineering: A systematic literature review.ACM Trans","cited_arxiv_id":null,"evidence_quote":"systematic literature review establishing that existing LLM-for-SE studies rely on text-based data, the gap the multimodal diagram-based approach claims to fill."}],"review_version":1}