{"id":"a558e4a5-7965-4abe-80eb-4b2bd3ca6096","arxiv_id":"2507.08719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.","lead":"The paper introduces M2-Coder, a 7B multimodal model that takes UML diagrams and flowcharts together with text prompts to generate code in 10 programming languages. It also contributes a 13.1M-sample instruction dataset and a 300-problem benchmark, and shows its model beating some larger open-weight vision-language models on that benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"0.6-point edge over Qwen2.5-VL-72B is within binomial noise; headline outperformance claim lacks variance estimates.","rationale":"The paper makes a credible contribution: a large synthetic multimodal instruction-tuning dataset, a two-stage training recipe, and a new multilingual benchmark with careful human verification. The ablation (Model ⑦ vs ①) supports the central claim that the training data improves code generation. However, the specific headline that a 7B model exceeds Qwen2.5-VL-72B rests on a 2-problem difference out of 300, which is within binomial sampling noise. The reader's verdict already marks the paper CONDITIONAL and mentions missing error bars, so no verdict change is needed, but this statistical fragility should be elevated from a side note to a primary condition for accepting the cross-scale comparison. The benchmark-representativeness concern is real but secondary; even if the benchmark were perfectly representative, the reported margin cannot support the superiority claim as stated.","tokens_in":35081,"tokens_out":5331,"duration_ms":64428,"concrete_test":"Use the per-problem scores in Figure 17 to compute a cluster bootstrap over the 30 unique problems: resample problems with replacement, for each resample compute the mean M2-Coder minus Qwen2.5-VL-72B Pass@1 across all languages, and report the 95% percentile interval. If the interval includes 0, the claimed outperformance over Qwen2.5-VL-72B is not statistically established. Alternatively, rerun M2-Coder training with 5 random seeds and report mean±std on M2EVAL; if the 25.3 figure varies by more than 2 points, the headline difference is within training noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 4's marquee result is M2-Coder-7B at 25.3% vs Qwen2.5-VL-72B at 24.7% — a 0.6-point gap, i.e., 2 problems out of 300. With 30 unique concepts (one shared diagram per concept, translated to 10 languages) and a single greedy decoding run per model, this margin is sampling noise. The standard error of a 25% pass rate on 300 items is about 2.5 points; the difference is well under one SE. Figure 17's per-problem totals show M2-Coder and Qwen2.5-VL-72B differ by only 2 aggregated points (76 vs 74). The claim that M2-Coder 'exceeds Qwen2.5-VL-72B' and the implied superiority over 70B+ models is therefore unsupported without confidence intervals or multiple runs. The larger ablation gains (25.3 vs 12.0 for Qwen2-VL-7B base) are more robust, but the central comparison to a 72B model is the headline. No variance, seeds, or significance tests are reported anywhere in the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes M2-Coder, a 7B-parameter vision-language model for code generation from UML diagrams and flowcharts ('visual workflows'), trained in two stages on a newly constructed 13.1M-instance instruction-tuning corpus M2C-Instruct. It also introduces M2Eval, a 300-problem benchmark covering 10 programming languages built from 30 concepts, each with one LLM-generated diagram shared across languages. On M2Eval, M2-Coder-7B reports a 25.3 average Pass@1, higher than all evaluated models of comparable size and nominally higher than Qwen2.5-VL-72B (24.7). The authors argue this demonstrates the value of large-scale multimodal instruction tuning and exposes remaining weaknesses of current LMMs in visual understanding and instruction following.","tokens_in":35312,"tokens_out":8519,"duration_ms":90101,"significance":"The resources are potentially valuable: M2C-Instruct is large and multilingual, M2Eval is execution-based with human-verified translations and test cases, and the two-stage SFT recipe gives a substantial improvement over the Qwen2-VL-7B base (25.3 vs 12.0). The paper also reports detailed annotation protocols, payment information, and plans to release code and data, which support reproducibility. However, the central comparative claim against larger models is not statistically supported (a 2-problem difference out of 300), and the benchmark/training distribution overlap means the advantage over other models may partly reflect style matching rather than general multimodal coding ability. With added uncertainty quantification and a robustness analysis against diagram style, the contribution would be a useful step for multimodal code generation.","major_comments":[{"comment":"The headline result that M2-Coder-7B outperforms Qwen2.5-VL-72B rests on a 0.6-point gap (25.3 vs 24.7 average Pass@1), which in Figure 17 corresponds to 76 versus 74 correct (problem, language) pairs out of 300. With a single greedy decoding run and no confidence intervals, the standard error of a 25% pass rate on 300 items is about 2.5 points, so this difference is not distinguishable from sampling noise. The authors should report multiple runs or a paired significance test (e.g., McNemar's test on the 300 items) before claiming superiority, or explicitly soften the claim to 'comparable within noise.' As written, contribution bullet 3 ('competitiveness with larger 70B+ LMMs') is not supported by this comparison, although the larger gaps against 7B baselines (e.g., Qwen2-VL-7B at 12.0) are more robust.","section":"§4.2, Table 4, Figure 17"},{"comment":"M2EVAL and the diagram subset of M2C-INSTRUCT are generated by the same LLM pipeline: Qwen2.5-Coder produces PlantUML/Mermaid diagrams from a code problem and then rewrites the prompt so that key details appear only in the diagram. Because M2-Coder is trained on M2C-INSTRUCT, it is evaluated on the same synthetic diagram style that it saw during training, while all comparison models encounter that style for the first time. This distributional overlap is a plausible source of M2-Coder's advantage beyond actual multimodal coding skill. The paper should (i) explicitly discuss this overlap, (ii) measure sensitivity to diagram style, e.g., by testing on human-created or out-of-pipeline diagrams, and (iii) report whether the performance gap persists when comparison models are given additional diagram examples or when M2-Coder is evaluated on a held-out style. Without such analysis, the comparative results in Table 4 are open to an overfitting explanation.","section":"§2.2, §3.2, Appendices C.2, D.2"},{"comment":"The effective visual diversity of M2EVAL is 30 unique diagrams (one per concept, shared across the 10 language versions), so the 300-problem count overstates the independent sample size for the visual component. Per-language Pass@1 scores are computed on 30 items, giving standard errors around 8-9 percentage points; the per-language differences in Table 4 (e.g., C# 26.7 vs 16.7) are therefore not meaningful without uncertainty estimates. The authors should report confidence intervals or cluster the analysis by the 30 concepts (e.g., bootstrap by concept) when comparing models.","section":"§2.3, Figure 17"}],"minor_comments":[{"comment":"The naming is inconsistent: the abstract and contributions use 'MM-Coder', 'MMc-Instruct', and 'MMEval', while the body and tables use M2-Coder, M2C-INSTRUCT, and M2EVAL. Please unify the terminology.","section":"Abstract and throughout"},{"comment":"The M2EVAL row lists '#Train 13.1M' and a checkmark under 'Instruct', which conflates the 300-problem evaluation benchmark with the 13.1M-instance M2C-INSTRUCT training corpus. M2EVAL should be listed as evaluation-only, with the training-corpus statistics attributed to M2C-INSTRUCT.","section":"Table 3"},{"comment":"The evaluation protocol mixes greedy decoding for 'ordinary' models with temperature 0.6 sampling for 'thinking' models, and no seed averaging is reported for the sampling-based results. This makes the Pass@1 values for thinking models (e.g., Doubao1.5-thinking-pro, QVQ, Kimi) not directly comparable to greedy scores; please either use a uniform protocol or report both sets of numbers and justify the choice.","section":"§4.1"},{"comment":"The 0.0 scores for the text-only models DeepSeek-V3 and Qwen2.5-Coder are expected by construction: §2.2 states that the curation process removes from the prompt all details that are moved into the diagram, making the prompt alone insufficient. This should not be presented as evidence that diagrams are generally necessary for code generation in real-world settings.","section":"§4.2 and §2.2"},{"comment":"The ablation table omits the score for Qwen2-VL-Base without any fine-tuning (model ③ shows dashes). Without this baseline, the ablation cannot separate the effect of stage-1 SFT from the base model's own capability or inability to follow the instruction format; please add the missing baseline or explain why it cannot be evaluated.","section":"Table 5"},{"comment":"There is a typo ('Futhermore') in Appendix D.1, and Figure 6's 'data efficiency' analysis lacks a description of how many checkpoints were used and whether the trend is based on a single run. Please clarify the experimental design behind this figure.","section":"Appendix D.1 and Figure 6"},{"comment":"The paper does not report a contamination check between M2Eval and the training corpora of the evaluated models. Since M2C-INSTRUCT is derived from GitHub code and LLM-generated problems, the possibility that M2Eval's prototype problems or solutions appear in pretraining corpora should be addressed, e.g., by a fuzzy-match or near-duplicate analysis.","section":"Evaluation methodology"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's contribution is promising, but the headline comparison to 72B models is overclaimed relative to the statistical power of the benchmark. I recommend major revision, not rejection, because the core data and training pipeline are likely sound and the statistical and robustness issues are addressable. The editor may also want to consider whether the benchmark/training distribution overlap is sufficiently disclosed for the target venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is the first attempt I have seen to train and evaluate code LMMs on UML/flowchart 'visual workflow' inputs across ten languages. The two-stage SFT recipe is clearly described, and the ablation in Table 5 is the strongest evidence in the paper — full two-stage SFT on the Qwen2-VL-7B base more than doubles the base (25.3 vs 12.0 Pass@1), and the individual stages each add something. That is a real, replicable training signal, and the M2C-INSTRUCT pipeline (cross-modal rendering plus diagram synthesis with Qwen2.5-Coder) is detailed enough to reconstruct.\n\nThe weak spots are mostly in the evaluation rather than the method. The headline claim that a 7B model beats Qwen2.5-VL-72B rests on a 25.3 vs 24.7 average — a two-problem difference out of 300. With no confidence intervals, no seeds, and no repeated runs anywhere in the paper, that gap is sampling noise. The claim should be rephrased as 'comparable to'. Second, M2EVAL is produced with the same Qwen2.5-Coder diagram-generation pipeline that generated the training data, so the benchmark is in-distribution by construction. Human curation helps, and the authors' check that text-only models get 0.0 without the diagram shows the diagram is necessary, but it does not show the diagram style generalizes to real human-drawn UML. I would need an independent validation set to trust the absolute numbers. Third, the benchmark is small: 30 concepts, one shared diagram per concept, 300 problems total. Per-language 30 is thin, and 30 visual inputs is very thin. Finally, thinking models are decoded at temperature 0.6 while regular models are greedy; that inconsistency can bias cross-model comparisons.\n\nNone of these issues kill the paper — the training recipe is plausibly useful and the ablation is solid. But the evaluation needs variance estimates, a less overclaimed framing, and ideally a test set of independently drawn diagrams. Also, the artifacts are not actually released despite the GitHub URL; that is the biggest practical obstacle for the community. I would not desk-reject this. I would send it to a serious venue with the expectation of heavy revision, and I would treat the results as evidence that the two-stage instruction-tuning recipe works, not that M2-Coder is at parity with 70B+ models. The paper is useful for people working on multimodal code generation and visual-spec benchmarks, and less relevant for a general reading group.","headline":"Solid two-stage SFT recipe for diagram-conditioned code gen, undermined by an in-distribution 300-problem benchmark and a noisy headline comparison to a 72B model.","tokens_in":35871,"tokens_out":5694,"would_cite":false,"duration_ms":59623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7-billion-parameter model trained to read UML diagrams and flowcharts alongside text outperforms much larger models, including a 72-billion-parameter one, on a new 10-language code-generation benchmark.","keywords":["multimodal code generation","visual workflow","UML diagrams","flowcharts","instruction tuning","multilingual code evaluation","M2EVAL benchmark","Pass@1"],"falsifier":"Re-run M2EVAL's 30 problems with the same content but diagrams re-rendered in a different style, layout, or drawing tool, or replaced by hand-drawn versions. If M2-Coder-7B's Pass@1 falls sharply while GPT-4o's holds, the gains come from style matching rather than diagram reading. As a second check, give text-only models such as Qwen2.5-Coder-32B the diagram content rewritten as plain text inside the prompt: if they then match or beat M2-Coder, the model's advantage is the extra information the diagram carries, not the visual channel itself.","tokens_in":34887,"feed_emoji":"💻","tokens_out":11065,"duration_ms":108452,"temperature":0.7,"pith_summary":"This paper argues that code generation gets better when the model can see the software design: UML diagrams and flowcharts, which the authors call Visual Workflow, carry information that text prompts alone do not. To test the idea, the authors built M2C-INSTRUCT, a 13.1-million-sample instruction-tuning corpus, and M2EVAL, a new benchmark of 300 problems across 10 programming languages in which key details are deliberately placed only inside the diagram. Their 7-billion-parameter model, M2-Coder, reaches 25.3 average Pass@1 — the share of generated programs that pass all test cases — beating every similarly sized model and exceeding the 72-billion-parameter Qwen2.5-VL at 24.7. The result matters because software teams routinely communicate through diagrams, and a small open model that can read them would turn design intent directly into working code.","feed_headline":"7B model that reads UML diagrams beats 72B vision rival","feed_subtitle":"Trained on 13.1M visual-workflow examples, it scores 25.3 Pass@1 across 10 languages; text-only models score zero.","key_machinery":"The load-bearing object is the Visual Workflow: UML class and sequence diagrams and Mermaid-rendered flowcharts generated from code by an LLM, then manually refined and re-embedded into problems. Two mechanisms carry the argument. The first is the diagram-necessity construction principle: when a text problem is converted into a multimodal problem, information is removed from the prompt and re-encoded inside the diagram, so the image is the unique carrier of facts the code must reflect. The second is the two-stage instruction-tuning recipe: stage 1 aligns the model with rendered code images and OCR-heavy inputs at scale, and stage 2, with the vision tower frozen, sharpens diagram comprehension and instruction following on shorter, higher-quality problems. M2EVAL applies the same necessity principle in reverse, stripping prompts until they are unsolvable without the diagram, and scores model outputs by executing them against test cases in ten languages.","core_discovery":"The central claim is that large-scale multimodal instruction tuning on visual workflow data turns a standard vision-language model into a multilingual code generator that uses design diagrams. Starting from Qwen2-VL-7B, the authors fine-tune in two stages: first on 12.9 million cross-modal problems, in which source code is rendered as syntax-highlighted images alongside diagram problems, then on 168 thousand high-quality diagram problems whose essential information is encoded only in the image. On M2EVAL, the resulting M2-Coder-7B scores 25.3 average Pass@1 across ten programming languages, above every other open-weight model evaluated up to 400 billion parameters, including Qwen2.5-VL-72B at 24.7, and close to proprietary leaders such as GPT-4o at 49.7. The paper also shows that text-only code models score 0.0 when the diagram is withheld, evidence that the diagrams are genuinely load-bearing, and that the two-stage recipe beats either stage alone (10.0 and 18.0 individually, 25.3 combined).","pith_inferences":["Because M2EVAL and M2C-INSTRUCT share the same LLM-assisted diagram pipeline, I would expect Pass@1 to drop on hand-drawn or professionally authored diagrams; a re-rendered dev set would quantify how much of the result is genuine visual understanding.","The diagram-necessity trick is portable: the same strip-text-and-re-encode-in-image recipe could be applied to UI mockups, database schemas, or API specifications to build multimodal training data for other software-engineering tasks.","Stage-1 data volume keeps helping, so the 13.1-million-instance budget is not exhausted; a cheaper question worth testing is whether a much smaller, diagram-only corpus could capture most of the gain, since stage 2 alone already contributes a large share.","If the gains transfer across rendering styles, then code-from-diagram generation could be plugged into industrial design tools that already export Mermaid or PlantUML, giving the approach a natural deployment path."],"forward_implications":["A 7-billion-parameter open-weight model can post 25.3 average Pass@1 on multilingual multimodal code generation, above most models an order of magnitude larger, making diagram-reading a small-model capability rather than a frontier-model one.","Even the strongest evaluated model, GPT-4o, reaches only about 50 percent, so M2EVAL leaves clear headroom; the paper names precise visual capture, instruction following, and design-pattern knowledge as the open bottlenecks.","Text-only code models score 0.0 when the diagram is withheld, so software workflows that communicate through diagrams will need multimodal components rather than text-only code LLMs.","Design-pattern problems are the hardest category for every model, and strictly typed languages (C#, Scala, Swift) trail scripting languages (Python, PHP, JavaScript), pointing to where data and evaluation effort should go next.","The dataset, not the base model, drives most of the gain: stage-2 fine-tuning alone moves Qwen2-VL-Instruct from 12.0 to 16.7, and the full two-stage pipeline reaches 25.3."],"supporting_citations":[{"why":"Provides the Qwen2-VL-7B vision-language base model that M2-Coder is initialized from and fine-tuned in two stages.","marker":"[76]"},{"why":"Generates the 12.9M question-answer pairs and the Mermaid/PlantUML diagrams that make up M2C-INSTRUCT, and also serves as a text-only control model.","marker":"[27]"},{"why":"Contributes the OSS-Instruct source corpus and the instruction-tuning methodology that the stage-2 diagram-problem synthesis follows.","marker":"[80]"},{"why":"Supplies the Evol-CodeAlpaca dataset, the second source corpus for stage-2 multimodal diagram problem synthesis.","marker":"[49]"},{"why":"Its preprocessing pipeline filters and cleans the GitHub source data used for M2C-INSTRUCT stage 1.","marker":"[35]"},{"why":"Defines the Pass@k metric used to report Pass@1 on M2EVAL.","marker":"[15]"},{"why":"Provides the multilingual execution environment and language runtime setup on which M2EVAL's 10-language test-case evaluation is built.","marker":"[13]"},{"why":"MMCode, an earlier Python-only multimodal code benchmark, is the comparison point that establishes M2EVAL's new multilingual visual-workflow task type.","marker":"[34]"}],"fun_headline_variants":["Code from diagrams: 7B model beats 72B vision rival","UML-aware 7B model outperforms 72B in code generation","Text-only code models score zero when diagrams matter","7B model reads flowcharts, codes in 10 languages, beats 72B","Multimodal instruction tuning lifts 7B code model above 72B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark that measures success, M2EVAL, is built with the same LLM-generated Mermaid/PlantUML diagram pipeline as the training data, so the reported Pass@1 numbers could reflect mastery of synthetic diagram styles rather than of real-world design documents; with only 30 unique concepts, each diagram is also reused across ten languages.","fun_headline_variants_meta":{"raw":{"variants":["Code from diagrams: 7B model beats 72B vision rival","UML-aware 7B model outperforms 72B in code generation","Text-only code models score zero when diagrams matter","7B model reads flowcharts, codes in 10 languages, beats 72B","Multimodal instruction tuning lifts 7B code model above 72B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000489,"raw_usage":{"total_tokens":2419,"prompt_tokens":968,"completion_tokens":1451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":1355}},"tokens_in":584,"tokens_out":1451,"duration_ms":14822,"temperature":1.0,"reasoning_tokens":1355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:10:37.628825+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run M2EVAL's 30 problems with the same content but diagrams re-rendered in a different style, layout, or drawing tool, or replaced by hand-drawn versions. If M2-Coder-7B's Pass@1 falls sharply while GPT-4o's holds, the gains come from style matching rather than diagram reading. As a second check, give text-only models such as Qwen2.5-Coder-32B the diagram content rewritten as plain text inside the prompt: if they then match or beat M2-Coder, the model's advantage is the extra information the diagram carries, not the visual channel itself.","supporting_citations":[],"review_version":1}