{"id":"fddf67fa-28a4-4ca1-b155-42ea16925514","arxiv_id":"2506.15695","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SimuGen uses specialized LLM agents and a Simulink block database to convert diagram images into executable MATLAB/Simulink code, reporting 94.5% average accuracy on nine examples.","lead":"SimuGen is a multi-agent AI system that reads a Simulink block diagram image and writes MATLAB code to rebuild the same simulation model. The paper reports 94.5% average structural accuracy on nine hand-picked examples, but the evaluation has gaps that make the number hard to trust.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94.5% reproduction accuracy rests on a recall-only metric that ignores extra blocks and connections, so the evaluation cannot distinguish faithful reconstruction from over-generation.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing issue: Eq. (1) is recall-only and cannot penalize over-generation, and the small hand-picked task set further weakens the empirical claim. I agree with that reading. The paper has genuine strengths: the multi-agent workflow is clearly described, full prompts are included, the public code link and a worked case study support reproducibility, and the architecture is plausible. However, the headline 94.5% figure is not robustly supported because the metric measures how much of the ground truth is recovered, not how closely the generated model matches it. The concrete test above would settle whether the concern actually changes the reported number. Since the reader already assigned CONDITIONAL, my stress-test does not move the verdict; it reinforces the need for precision-aware metrics and repeated trials before the central claim is accepted at face value.","tokens_in":18546,"tokens_out":4452,"duration_ms":45861,"concrete_test":"For the same nine saved generated models, recompute accuracy with precision-aware denominators, e.g., (1/2)(|Bmatch|/|BGT ∪ Bgen| + |Cmatch|/|CGT ∪ Cgen|), and separately report block precision and connection precision. If average precision is materially below 94.5%, the recall-only metric is inflating the headline; if precision is near 100% on all nine tasks, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim depends on Eq. (1), which defines accuracy as (|Bmatch|/|BGT| + |Cmatch|/|CGT|)/2. Because the denominators contain only ground-truth elements, any extra blocks or connections in Bgen/Cgen are invisible to the score; a model that adds hallucinated blocks or wires can still score highly as long as the GT elements are present. The Unit Test Reviewer (Section 3.3) checks internal consistency with Simulink standards, not fidelity to the input diagram, and the Executor only requires that the generated code runs, not that its behavior matches the intended simulation. The Bipolar Transistor case study in Appendix E illustrates the issue: the Block Builder's code connects the RConn2 ports of the Voltage-Controlled Voltage Source and Current-Controlled Current Source to Electrical Reference, connections that were not in the Investigator's extracted connection list. Such additions are not penalized by Eq. (1). Combined with only nine self-selected tasks and no repeated runs, the reported 94.5% average is not a demonstrated measure of reproduction accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SimuGen is a multi-agent framework that takes a Simulink block-diagram image as input and produces executable MATLAB/Simulink code via six specialized agents: Investigator, Unit Test Reviewer, Block Builder, Executor, Debug Locator, and Report Writer, supported by a database of 50 Simulink block descriptions. The paper evaluates the system on nine hand-selected Simulink models spanning four application domains. With o4-mini as the visual Investigator and o3-mini for the remaining agents, the authors report an average reproduction accuracy of 94.5%, defined by Eq. (1) as the unweighted average of ground-truth block recall and ground-truth connection recall. Ablation results removing the Unit Test Reviewer and/or Debug Locator are presented in Table 4, and a detailed Bipolar Transistor case study is provided in Appendix E.","tokens_in":18695,"tokens_out":10521,"duration_ms":93784,"significance":"If the reported numbers were reliable, SimuGen would be a useful step toward automating Simulink model construction from diagrams, an area with little prior work. The modular agentic design, the inclusion of full prompts in Appendix C, the detailed Bipolar Transistor trace in Appendix E, and the public code repository are genuine strengths. However, the evaluation as written does not establish the headline accuracy: the metric ignores spurious blocks and connections, the ablation table is internally inconsistent, the results come from single runs on a small self-selected set, and the case study shows the pipeline adding connections that were not extracted from the diagram. The framework is promising, but the quantitative claims need re-evaluation before the paper can be accepted.","major_comments":[{"comment":"The accuracy metric in Eq. (1) is recall-only for both blocks and connections: the denominators are |B_GT| and |C_GT|, so extra or spurious elements in the generated model are never penalized. The Executor (§3.3) only checks that the code runs, not that the resulting model matches the diagram. Appendix E shows this is not hypothetical: the Block Builder's final Bipolar Transistor code contains connections that are absent from the Investigator's extracted connection list, including 'Voltage-Controlled Voltage source/RConn2' and 'Current-Controlled Current source/RConn2' to 'Electrical Reference/LConn1', and 'RLoad/LConn1' to both controlled-source LConn2 ports. Whether or not these additions happen to match the ground truth, the reported accuracy cannot distinguish faithful reconstruction from over-generation. I recommend reporting precision or F1 per block and per connection, along with Bmatch, Cmatch, and false-positive counts for each task.","section":"§4.1, Eq. (1), and Appendix E"},{"comment":"The ablation table is internally inconsistent. Summing the 'w/o Unit Test Reviewer' column gives (100 + 73.3 + 95.5 + 100 + 92.8 + 61.65 + 95.3 + 96.15 + 91.25) / 9 = 89.55%, not the reported 86.21%. The 'w/o Debug Locator' average of 83.9% and the 'w/o All' average of 51.7% are recovered only if the '–' entries are counted as 0, but the table and text never state that a dash denotes a failed run with accuracy 0. These arithmetic and reporting issues directly affect the claim in §4.3 that the Unit Test Reviewer and Debug Locator have a synergistic effect; the claim needs corrected arithmetic and an explicit policy for failed runs.","section":"§4.3, Table 4"},{"comment":"All quantitative results come from a single run per task on nine hand-selected examples. There are no error bars, no repeated trials, and no held-out diagrams; Table 2 reports exactly one number per task. Because the selection of tasks and the development of the prompts and database are not described in a way that permits external validation, the 94.5% average is a point estimate with unknown run-to-run variability and unknown generalization to new diagrams. I recommend at least three to five independent runs per task with mean and standard deviation, and an evaluation on diagrams not used during development.","section":"§4.1–§4.2, Tables 1–2"},{"comment":"The paper's own contribution statement says the Unit Test Reviewer cannot directly assess the correctness of connections, and the Executor only verifies that code runs. The Debug Locator in Appendix E compounds this problem: it reports that the generated code 'follows Implementation Details exactly' and sets Investigator_error=true because two lines target the same LConn2 port. However, the code at that point contains RLoad/LConn1-to-CCCS/LConn2 and RLoad/LConn1-to-VCVS/LConn2 connections that are not in the Investigator's connection list, so the alleged discrepancy was introduced by the Block Builder, not the Investigator. This breaks the modular error attribution that the framework relies on and means execution success is not evidence of reproduction fidelity.","section":"§3.3, §3.4, and Appendix E"}],"minor_comments":[{"comment":"The heading 'Alblation' should be 'Ablation', and the text contains typos such as 'MEanwhile' and 'SimGen' (in Appendix E) that should be corrected.","section":"§4.3"},{"comment":"The text says 'As illustrated in Figure 2' when describing common LLM errors, but Figure 2 is the architecture overview; the errors appear to be shown in Figure 1.","section":"§2.2"},{"comment":"The sentence 'Overall, from Table 3, we observe...' should refer to Table 4, which is where the ablation results are presented.","section":"§4.3"},{"comment":"The dash '–' is used without a legend; the table should explicitly state whether a dash indicates a failed run (accuracy 0), a not-run condition, or a missing measurement.","section":"Table 4"},{"comment":"The anonymous link to the full report should be replaced with a stable reference or removed, since the report is part of the review artifact and the link may not persist.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is plausible as a systems contribution, but the evaluation needs substantial strengthening before it can support the claims made in the abstract and conclusion. The Table 4 arithmetic inconsistency is the kind of error that should be caught by the authors before resubmission. I did not attempt to run the public code, so reproducibility remains unverified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: SimuGen is a real engineering attempt and the agent decomposition is sensible, but the headline 94.5% reproduction accuracy is not yet supported. The metric in Eq. (1) is recall-only: extra blocks and wires in the generated model do not lower the score. The Bipolar Transistor case study in Appendix E actually shows the Block Builder adding RConn2 connections that were not in the Investigator's connection list; those additions cost nothing under the metric. So the evaluation cannot distinguish faithful reconstruction from over-generation.\n\nWhat is genuinely new: this is the first end-to-end pipeline I have seen that takes a Simulink diagram image and outputs executable matlab.engine code with a reviewer/debug loop. The prompts and the 50-block database are included, the code is public, and the case study is concrete. Anyone wanting to build on this has a real starting point. The authors also honestly acknowledge that the Unit Test Reviewer cannot check connection correctness directly, only standards compliance—that is a useful design observation.\n\nSoft spots, in order of importance. First, the accuracy metric. It needs precision or an F1-style score over blocks and connections, or at least a false-positive count; otherwise the main number is ambiguous. Second, there is no quantitative baseline beyond their own ablations. The ablation table is also internally inconsistent: Table 4's 'w/o Unit Test Reviewer' average is 86.21, but averaging the nine rows gives 89.55. That is the kind of arithmetic a referee would catch. Third, nine hand-picked tasks, no repeated runs, no error bars. Fourth, the Executor only checks that code runs, not that simulated behavior matches the intended model, so successful execution is a weak correctness criterion.\n\nNone of this proves the system does not work; it is a credible engineering contribution that needs a more honest evaluation. The case study is suggestive, not confirmatory. The citation pattern looks fine—prior Simulink LLM work, SWE-agent, and ChatDev are all acknowledged—and the public artifacts are a real plus.\n\nWho is this for: people building LLM agents for graphical or model-based engineering tools, and anyone thinking about evaluation metrics for code-generation agents. It deserves a serious referee because the claims are checkable and the artifact is available, but I would send it back for major revision: fix Eq. (1), add precision or false-positive reporting, include a real baseline, run each task multiple times, correct Table 4, and ideally validate on held-out diagrams rather than the nine used to tune the workflow.","headline":"SimuGen is a plausible engineering system with a sensible agent decomposition, but the recall-only metric and thin evaluation do not support the 94.5% reproduction accuracy claim.","tokens_in":19272,"tokens_out":2856,"would_cite":false,"duration_ms":27928,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-agent pipeline reads Simulink diagram images and produces executable simulation code, reporting 94.5% average reproduction accuracy.","keywords":["SimuGen","Simulink model generation","multi-agent LLM framework","block diagram understanding","multimodal LLM","simulation code generation","unit test reviewer","debug locator"],"falsifier":"Re-run the nine tasks scoring both missing and extra elements (for instance with F1) or feed the pipeline diagrams it has not seen, including diagrams with subsystems; if the score drops well below 94.5%, the headline number was an artifact of counting only ground-truth elements.","tokens_in":18292,"feed_emoji":"🧩","tokens_out":8489,"duration_ms":78787,"temperature":0.7,"pith_summary":"SimuGen is an attempt to close a gap the authors identify: large language models can write text code but fail to produce reliable Simulink simulation code from text prompts alone, because Simulink's two-dimensional block structure and its engineering semantics are underrepresented in pretraining. The paper proposes a multi-agent, multimodal framework that takes a Simulink diagram image as input, reads the blocks and connections visually, retrieves structured descriptions of 50 block types from a domain database, and generates executable Simulink construction code through specialized agents. On nine hand-selected simulation tasks spanning general, physical, electrical, and automotive modeling, the framework reports an average reproduction accuracy of 94.5%, with the Unit Test Reviewer and Debug Locator contributing a combined accuracy gain that exceeds either alone. If this result holds, it would make diagram-to-simulation conversion, and the documentation that goes with it, automatable rather than a manual reverse-engineering chore.","feed_headline":"Agents read block diagrams and rebuild Simulink models at 94.5%","feed_subtitle":"Text-only LLMs fail at Simulink; adding the diagram image plus a block database lifts reproduction accuracy.","key_machinery":"The load-bearing mechanism is the feedback loop between the Investigator and the Unit Test Reviewer, backed by a retrieval database of 50 block definitions that supply library paths, port names, and parameter rules. The Reviewer checks eight conformance criteria, including duplicate connections, disciplined input/output ports, formatting, and complete port coverage, and sends failures back to the Investigator; the Debug Locator then parses runtime errors and routes blame either to the Block Builder's code or back to the Investigator's connection description. The reported accuracy is the average of two recall ratios, block matches over ground-truth blocks and connection matches over ground-truth connections, so the loop's job is to make the generated model contain the ground-truth elements.","core_discovery":"On the paper's own terms, the discovery is that a diagram image plus structured block knowledge is enough for a multimodal LLM to reconstruct an executable Simulink model: the investigator agent extracts blocks and connections from the image, the unit test reviewer enforces eight standards-conformance checks on the proposed wiring, the block builder emits construction code using only add_block and add_line, the executor runs it, and the debug locator decides whether a failure is a code bug or a flaw in the investigator's description before looping back. With the multimodal investigator o4-mini and the reasoning agents o3-mini, the system reports 94.5% average reproduction accuracy over nine ground-truth models, versus 83.2% with GPT-4.1 as investigator. The ablation results argue that the reviewer and debugger are not interchangeable conveniences: removing both drops accuracy to 51.7%, and the reviewer is useful even though it cannot directly verify whether a connection is semantically correct.","pith_inferences":["Because the accuracy metric divides only by ground-truth block and connection counts, a generated model that adds spurious elements can still score 100%; the 94.5% figure would likely move if the metric penalized false positives or if the system faced held-out diagrams.","The nine-task benchmark is hand-selected and each diagram is paired with a textual explanation, so the natural next stress test is a larger, uncurated set of diagrams, especially ones with subsystems, which the investigator prompt explicitly forbids.","The reviewer's eight rules could be extracted into a standalone, LLM-free linter for Simulink connection descriptions, giving the same standards enforcement without per-task agent cost.","One could also reverse the pipeline: feed SimuGen's generated reports back as training data to teach smaller models the mapping from diagram layouts to wiring, addressing the data scarcity the paper cites."],"forward_implications":["If the reported accuracy holds, an engineer can hand SimuGen a screenshot of an existing Simulink diagram and receive runnable construction code plus a four-part simulation report.","The system should be applied where diagrams are small to medium in size: the paper observes accuracy near 100% for models with fewer than about 10 connections and degradation beyond that.","The ablation implies that any practical deployment needs both a standards checker and a runtime debugger; removing both makes most complex tasks fail.","At roughly $0.17 and 275 seconds per model, the pipeline is cheap enough to regenerate models repeatedly during debugging."],"supporting_citations":[{"why":"Establishes the data-scarcity baseline by generating simple diagrams from only 400 Simulink models.","marker":"[23]"},{"why":"Documents text-to-Simulink failure modes and synthetic-data needs in automotive modeling.","marker":"[1]"},{"why":"Inspires the unit-test-reviewer design by treating review as a structured agent-computer interface.","marker":"[26]"},{"why":"Documents long-context degradation that motivates splitting work across agents.","marker":"[13]"},{"why":"Motivates the multi-agent collaboration design for long-context reasoning.","marker":"[28]"},{"why":"Supplies the GPT-4 family as the multimodal reasoning backbone behind the agents.","marker":"[2]"},{"why":"Provides the GPT-4o system card underpinning the multimodal vision capabilities.","marker":"[9]"},{"why":"Defines Simulink and its block/connection semantics as the target domain.","marker":"[20]"}],"fun_headline_variants":["Multimodal agents read block diagrams to rebuild Simulink models","Diagram plus domain knowledge lifts Simulink accuracy to 94.5%","Visual context boosts LLM Simulink code from 83% to 94.5%","Agentic pipeline turns block images into executable Simulink models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported score counts only what is missing, never what is extra: a model that adds spurious blocks and wires can still score perfectly.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal agents read block diagrams to rebuild Simulink models","Diagram plus domain knowledge lifts Simulink accuracy to 94.5%","Visual context boosts LLM Simulink code from 83% to 94.5%","Agentic pipeline turns block images into executable Simulink models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1197,"prompt_tokens":927,"completion_tokens":270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":188}},"tokens_in":543,"tokens_out":270,"duration_ms":3274,"temperature":1.0,"reasoning_tokens":188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:21:50.762233+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the nine tasks scoring both missing and extra elements (for instance with F1) or feed the pipeline diagrams it has not seen, including diagrams with subsystems; if the score drops well below 94.5%, the headline number was an artifact of counting only ground-truth elements.","supporting_citations":[{"cited_title":"Slgpt: Using transfer learning to directly generate simulink model files and find bugs in the simulink toolchain","cited_arxiv_id":null,"evidence_quote":"Establishes the data-scarcity baseline by generating simple diagrams from only 400 Simulink models."},{"cited_title":"Generative artificial intelligence for model-based graphical programming in automotive function development","cited_arxiv_id":null,"evidence_quote":"Documents text-to-Simulink failure modes and synthetic-data needs in automotive modeling."},{"cited_title":"Long-context llms struggle with long in-context learning","cited_arxiv_id":null,"evidence_quote":"Documents long-context degradation that motivates splitting work across agents."},{"cited_title":"Chain of agents: Large language models collaborating on long-context tasks","cited_arxiv_id":null,"evidence_quote":"Motivates the multi-agent collaboration design for long-context reasoning."},{"cited_title":"Simulink - simulation and model-based design, 2025","cited_arxiv_id":null,"evidence_quote":"Defines Simulink and its block/connection semantics as the target domain."}],"review_version":1}