{"id":"0d1ee4af-4eea-4d2b-9bcc-f1c0b1a9f692","arxiv_id":"2507.11356","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new dataset and head-to-head comparison of nine process model representations with LLMs finds Mermaid best for general use and BPMN text best for generation.","lead":"This paper compares nine ways of writing down business process models for large language models, using a new dataset of 55 descriptions. It finds Mermaid is the most convenient all-around notation, while BPMN text produces the most accurate generated models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The PMG ranking is not identified as load-bearing concern; the weakest point is Table 4's author-scored grading, which drives the headline Mermaid claim and is explicitly acknowledged as subjective.","rationale":"The reader's weakest_assumption focused on the conversion subset for branching PMRs (Section 3.3), which is indeed a genuine limitation for the PMG comparison. However, I see the more load-bearing threat in the suitability ranking: the headline finding ('Mermaid is the most suitable PMR') is derived from a single-row average of six author-assigned grades with no reliability evidence, while the PMG comparison, though subset-restricted, is at least computed from objective PME similarity scores and explicitly reports coverage losses. The conversion bias could be mitigated by subset-matched evaluation, but the author-graded Table 4 is more dangerous because it is the entire evidence for the Mermaid claim and is openly acknowledged as subjective. Hence a conditional verdict is appropriate: accept the dataset and PMG methodology as a useful contribution, but require either inter-rater validation or a sensitivity analysis before the suitability ranking is treated as established. The reader deserves partial agreement: they identified a real weakness (the conversion subset), but the primary threat to the central claim is the non-validated subjective scoring of Table 4, which they did not emphasize as the leading concern.","tokens_in":9195,"tokens_out":1111,"duration_ms":13295,"concrete_test":"Independently re-score Table 4 with two additional annotators (or a rubric-based re-scoring by the paper's authors blinded to the original grades) and recompute the average ranks. If the top-ranked PMR changes or the gap between Mermaid and Graphviz closes below the noise level, the headline suitability claim is not stable. A minimal analytical check: perturb each grade by ±1 in the direction most favorable to Graphviz and compute whether Mermaid remains first (it does not under several plausible perturbations, e.g., lowering Mermaid's visualizability to 4 and raising Graphviz's usability to 4).","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim \"Mermaid achieves the highest overall score across six PMo criteria\" rests almost entirely on Table 4, whose six grades (1-5 per PMR) are assigned by the authors based on qualitative judgment. The paper itself concedes: \"our evaluation of PMRs for PMo with LLMs (Table 4), while incorporating quantitative metrics, remains primarily subjective, as it is based on the assessment of the authors\" (Section 5). The averaging in Table 4 is then used to rank Mermaid first (avg. 4.00) over Graphviz (3.67) and BPMN process (3.50). No inter-rater reliability, sensitivity analysis, or alternative weighting is reported. Many component scores are plausibly contestable: e.g., Mermaid gets 5/5 for token compactness while POWL code also gets 5; BPMN text gets 1/5 for visualizability despite being convertible to BPMN; JSON branches receives a +1 bonus for JSON schema support but BPMN text (also schema-structured XML) does not. Because the headline \"best PMR\" claim is a ranking derived from these hand-assigned grades, a small change in one or two subjective grades could flip the top result. The PMG result (BPMN text best) is more robust because it is based on computed PME similarity, though its comparison is on a restricted subset of convertible models; that subset issue is real but secondary, since the paper explicitly reports coverage limitations and the PMG task is separately scoped.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the PMo Dataset of 55 process descriptions paired with ground-truth models in nine process model representations (PMRs), and reports a comparative evaluation of these PMRs for LLM-based process modeling. Suitability is assessed along six author-defined criteria (token compactness, expressivity, human readability, visualization, usability, extensibility), and process model generation (PMG) performance is measured under a standardized prompting setup with LLaMA-3.3-70b using element counts and PME similarity. The paper concludes that Mermaid is the most suitable PMR for PMo with LLMs, while BPMN text yields the best PMG results in terms of process element similarity.","tokens_in":9506,"tokens_out":2996,"duration_ms":37815,"significance":"The paper targets a real gap: PMRs used in LLM-based process modeling have not been systematically compared, and existing PMG evaluations are fragmented. The PMo Dataset and the public code and prompts are useful community assets, and the use of an open-source LLM with a standardized prompting protocol is a methodological strength. The PMG comparison, based on computed PME similarity with semantic matching, is more objective than the suitability grading and produces a concrete, falsifiable ranking. However, the headline suitability conclusion rests on subjective author-assigned grades, and the PMG conclusion is supported by a single run on a restricted subset of the data, so the empirical evidence is currently thinner than the abstract implies.","major_comments":[{"comment":"The central claim that Mermaid is the most suitable PMR for PMo with LLMs rests on the six grade columns of Table 4, whose values are assigned by the authors. The paper itself concedes in Section 5 that this evaluation \"remains primarily subjective.\" No inter-rater reliability, sensitivity analysis, or alternative weighting is reported. Several individual grades are contestable: for example, BPMN text receives 1/5 for visualization despite being convertible to BPMN, JSON branches receives a schema-support bonus while BPMN text, which is XML-based, does not, and Mermaid and POWL code both receive 5/5 for token compactness. Because the average scores are close (Mermaid 4.00, Graphviz 3.67, BPMN process 3.50), modest changes to one or two grades could change the ranking. I ask the authors to provide a robustness analysis, such as a sensitivity sweep over grades or a structured multi-expert grading protocol with agreement metrics, before the Mermaid conclusion can be considered supported.","section":"Section 4.1, Table 4; Section 5"},{"comment":"The PMG result that BPMN text is best is computed on a restricted subset: Section 3.3 states that for branching PMRs, two thirds of the BPMN models cannot be converted. The PME similarity scores for BPMN text and JSON branches are therefore based on roughly one third of the dataset, and these models may be systematically simpler than the non-convertible ones. The paper does not report the PME scores of the other PMRs on the same subset, so the comparison may conflate PMR quality with subset difficulty. Additionally, Table 6 reports a single run per prompt with no variance estimates or significance tests; with temperature 0.2 and one run, the differences between BPMN text (0.54), JSON branches (0.53), and Mermaid (0.48) may not be stable. I recommend re-evaluating all PMRs on the intersection of convertible models and reporting results over multiple seeds with confidence intervals or a significance test.","section":"Section 3.3, Table 6"},{"comment":"The ground-truth PMR models are produced by automatic conversion from BPMN, with validation by converting back to BPMN, but the paper does not report any quantitative or manual verification that the conversions preserve the process semantics relevant for evaluation. The paper also notes that unsupported information is ignored (e.g., conditions in POWL code) and that models are excluded when conversion would lose significant information. Since PME similarity in Section 3.5 is computed against these converted ground truths, errors or omissions introduced during conversion directly affect the reported scores. I ask the authors to present a conversion-quality check, such as a sample of manually inspected conversions or a comparison of element coverage before and after conversion, to support the assumption that the converted ground truths are faithful.","section":"Section 3.3"}],"minor_comments":[{"comment":"There is a duplicated word in the sentence \"a PMR should should also support direct graphical visualization.\"","section":"Section 3.4"},{"comment":"The column header \"Vizualisable\" is misspelled; it should be \"Visualizable\" or \"Visualization capability\" for consistency with the text.","section":"Table 4"},{"comment":"The sentence \"BPMN text, in particular, achieves the strongest similarity with ground truth elements for both raw numbers and semantic content\" conflates two different measurements: element counts (Table 5) and PME similarity (Table 6). Please phrase this as referring to both element-count differences and PME similarity scores separately.","section":"Section 5"},{"comment":"The distinction between \"BPMN\" and \"BPMN process\" is clear in the text, but Table 2 would benefit from a note that the difference is the presence or absence of the BPMN diagram definition, as stated in Section 3.2.","section":"Section 3.2, Table 2"},{"comment":"The semantic matching threshold of 0.7 is stated as \"experimentally defined\" without a reference or local justification. Since this threshold affects all PME similarity scores, a brief explanation or a sensitivity check would help the reader assess its impact.","section":"Section 3.5"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful dataset contribution and addresses a relevant comparative question, but the two headline claims are currently supported by evidence of different quality: the Mermaid suitability claim is explicitly subjective, and the BPMN text PMG claim lacks statistical grounding and is computed on a restricted subset. A major revision that adds robustness analyses would make the contribution suitable for publication. I would also encourage the authors to consider reporting results on the common subset of all PMRs to improve comparability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the dataset and the head-to-head comparison. Prior work each picked one PMR and evaluated it in isolation; this paper assembles 55 process descriptions paired with models in nine PMRs, releases the code and data, and runs them through one prompting setup. That is a real contribution and the field needed it. The paper is also honest about its own limits, which counts for something.\n\nWhat works well: the dataset construction is thoughtful (filtering out problematic sources, grading choices in Mangler), the PMG evaluation uses a reasonable semantic-matching pipeline, and the finding that branching PMRs mitigate gateway under-generation is credible and consistent with the element-count tables. BPMN text's top PME similarity score is computed, not hand-waved.\n\nNow the soft spots, in proportion. The Mermaid claim rests almost entirely on Table 4, a set of 1-5 grades assigned by the authors with no inter-rater reliability or sensitivity analysis. The paper itself concedes the evaluation 'remains primarily subjective.' Shifting one grade in Table 4 by a single point could move Mermaid below Graphviz. That is load-bearing for RQ2. The stress-test note has this right. The PMG result (RQ3) is more robust because it is metric-based, but it is still a single LLM, a single run per prompt, no variance or significance tests, and the branching PMRs are evaluated on only one third of the dataset due to conversion loss. That subset issue is real and could bias the comparison if the convertible models are systematically simpler. The semantic-matching threshold (0.7) is inherited from prior work without local validation, a minor point. Also minor: JSON branches gets a +1 usability bonus for JSON schema support while BPMN text, equally schema-structured XML, does not.\n\nNone of this sinks the paper. The limitations are acknowledged, the PMG claims are not overreaching, and the dataset will be reused regardless. But the two headline conclusions deserve different confidence levels: BPMN text for generation is a reasonable empirical finding; Mermaid as 'most suitable' is a well-reasoned opinion, not a measured fact.\n\nWho this is for: anyone working on LLM-based process modeling or building benchmarks for BPM. It deserves a serious referee. The requested revisions should include repeated runs or at least variance estimates, a sensitivity analysis of Table 4, and a subset-matched PMG comparison to address the conversion-loss confound.","headline":"Useful first head-to-head benchmark for process model representations in LLM pipelines, but the 'Mermaid is best' headline rests on author-scored grades; the PMG comparison is the more solid result.","tokens_in":10024,"tokens_out":1449,"would_cite":true,"duration_ms":19239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents the first head-to-head comparison of process model representations for LLM-based process modeling, concluding that Mermaid is the most suitable overall while BPMN text yields the most faithful generated models.","keywords":["process model representation","large language models","process model generation","business process modeling","PMo dataset","BPMN","Mermaid","PME similarity"],"falsifier":"Re-run the PMG experiment on a dataset in which all 55 descriptions include swimlanes or message flows representable in every PMR, and check whether BPMN text still has the highest PME similarity; if its lead disappears on the full set, the reported superiority is an artifact of the evaluable subset.","tokens_in":9003,"feed_emoji":"🧩","tokens_out":7194,"duration_ms":73917,"temperature":0.7,"pith_summary":"Process modeling with large language models has grown quickly, but each research group has used its own output notation, evaluation set, and metric, so no one knew whether one process model representation is better than another. This paper tries to answer that question directly: it builds the PMo Dataset, 55 gold-standard process descriptions each paired with a model in nine different representations, and compares all nine under one standardized prompting and scoring protocol. On the paper's own terms, the answer is split: Mermaid is the most suitable all-round representation for LLM-based process modeling, while BPMN text gives the most faithful process model generation when measured by element similarity. The authors argue that choosing a representation is itself a design decision that changes both how usable a model format is and how well an LLM can reproduce a process from text.","feed_headline":"Mermaid tops the format test; BPMN text tops generation","feed_subtitle":"Nine-format LLM study finds output format changes both usability and generation fidelity.","key_machinery":"The load-bearing mechanism is the PMo Dataset plus its conversion-and-evaluation pipeline. Fifty-five expert-validated BPMN models are automatically converted into nine process model representations (BPMN, BPMN process, Graphviz, Mermaid, PME, Simplified XML, POWL code, BPMN text, and JSON branches), where a PMR is a notation for expressing a process model, such as XML, JSON, Python code, or diagram syntax. Every PMG experiment then runs under one standardized prompt template with LLaMA-3.3-70b, and outputs are scored by element counts and by PME similarity, a two-step measure that semantically matches generated elements to ground-truth elements using sentence embeddings and then computes the Dice–Sørensen coefficient between the sets.","core_discovery":"The central discovery is a comparative result rather than a new algorithm: across six suitability criteria (token compactness, expressiveness, human readability, visualizability, usability, and extensibility), Mermaid scores highest on average, driven by its combination of extreme compactness and direct visualization; yet when the same 55 descriptions are given to an open LLM with identical prompts, the branching formats BPMN text and JSON branches yield models with clearly higher PME similarity to the ground truth, with BPMN text best overall. The paper also uncovers a systematic under-generation tendency: generated models have roughly eight fewer nodes than the ground truth on average, exclusive gateways are cut by about half, and parallel gateways by about two thirds. Representations that natively encode branching structure partially counteract this drop, which the authors take as evidence that the representation itself shapes generation fidelity.","pith_inferences":["The paper's two-thirds conversion failure means BPMN text's PMG advantage is measured only on the roughly one-third of models that are representable in branching formats; extending the comparison to swimlanes and message flows could shrink or reverse the gap.","A direct testable extension would be to prompt the same LLM to generate both Mermaid and BPMN text for identical descriptions and convert both outputs to BPMN before scoring, isolating the representation effect from the dataset-subset effect.","The suitability grades for six criteria are author-assigned; using the released dataset to collect multi-expert ratings would turn the Mermaid result from a claim into an independently reproducible ranking."],"forward_implications":["New or existing PMo tools should consider Mermaid as the default working representation for interactive modeling, because it combines the best compactness with direct visualization.","PMG systems should prefer branching representations such as BPMN text or JSON branches as generation targets, since these partially offset the LLM's tendency to omit gateways.","Process modeling pipelines can be staged by representation: use Mermaid for elicitation and communication, then switch to BPMN text for faithful generation from text.","Future PMG evaluations should report the target PMR explicitly, because the representation alone changes element counts and similarity scores by a large margin.","The PMo Dataset and the unified protocol provide a common benchmark on which future PMG approaches can be compared against each other."],"supporting_citations":[{"why":"Introduced ConverMod, the conversational PMG approach that used Graphviz and Mermaid as PMRs, supplied the PET-7 evaluation set, and reported user preference for Mermaid.","marker":"[9]"},{"why":"Introduced ProMoAI and the POWL code PMR, Python code that is executed to build a POWL model, one of the nine representations compared here.","marker":"[14]"},{"why":"Introduced BPMN-chatbot and the JSON branches PMR, a compact branching format designed for schema adherence during generation.","marker":"[10]"},{"why":"Introduced MAO and the BPMN text PMR, which this paper identifies as the best PMG target by element similarity.","marker":"[16]"},{"why":"Provided the PME representation and the semantic-matching-plus-Dice–Sørensen evaluation method used to score generated models.","marker":"[19]"},{"why":"Supplied the PMo Benchmark, one of the five gold-standard data sources in the PMo Dataset, and an evaluation framework for LLMs on PMo.","marker":"[12]"},{"why":"Supplied the Mangler dataset, the largest source of textual descriptions paired with expert-validated BPMN models used in the PMo Dataset.","marker":"[17]"},{"why":"Supplied the BPMN for Research dataset, used both as a data source for PMo and as the evaluation set in the MAO baseline.","marker":"[3]"}],"fun_headline_variants":["Mermaid best for modeling, BPMN text best for generation","Study: Mermaid tops usability, BPMN text tops fidelity","Nine formats tested: Mermaid wins usability, BPMN text wins accuracy","LLMs: Mermaid for modeling, BPMN text for generation","Format matters: Mermaid vs BPMN text in LLM process modeling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the automatic conversion of the ground-truth BPMN models into all nine representations is faithful, and that the subset of models convertible into the branching PMRs, about one third of the dataset, is representative enough for a fair PMG comparison.","fun_headline_variants_meta":{"raw":{"variants":["Mermaid best for modeling, BPMN text best for generation","Study: Mermaid tops usability, BPMN text tops fidelity","Nine formats tested: Mermaid wins usability, BPMN text wins accuracy","LLMs: Mermaid for modeling, BPMN text for generation","Format matters: Mermaid vs BPMN text in LLM process modeling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000152,"raw_usage":{"total_tokens":1174,"prompt_tokens":885,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":193}},"tokens_in":501,"tokens_out":289,"duration_ms":3056,"temperature":1.0,"reasoning_tokens":193,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:09:25.933244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the PMG experiment on a dataset in which all 55 descriptions include swimlanes or message flows representable in every PMR, and check whether BPMN text still has the highest PME similarity; if its lead disappears on the full set, the reported superiority is an artifact of the evaluable subset.","supporting_citations":[{"cited_title":"Conversational Process Modeling: Can Generative AI Empower Domain Experts in Creating and Redesigning Process Models?","cited_arxiv_id":"2304.11065","evidence_quote":"Introduced ConverMod, the conversational PMG approach that used Graphviz and Mermaid as PMRs, supplied the PET-7 evaluation set, and reported user preference for Mermaid."},{"cited_title":"In: BPM2024 (Sep 2024)","cited_arxiv_id":null,"evidence_quote":"Introduced BPMN-chatbot and the JSON branches PMR, a compact branching format designed for schema adherence during generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplied the BPMN for Research dataset, used both as a data source for PMo and as the evaluation set in the MAO baseline."}],"review_version":1}