REVIEW 3 major objections 5 minor 1 cited by
What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper presents the first head-to-head comparison of process model representations for LLM-based process modeling, concluding that Mermaid is the most suitable overall while BPMN text yields the most faithful generated models.
desk verdict Useful first head-to-head benchmark for process model representations in LLM pipelines, but the 'Mermaid is best' headline rests on author-scored grades; the PMG comparison is the more solid result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the PMo Dataset plus its conversion-and-evaluation pipeline. Fifty-five expert-validated BPMN models are automatically converted into nine process model representations (BPMN, BPMN process, Graphviz, Mermaid, PME, Simplified XML, POWL code, BPMN text, and JSON branches), where a PMR is a notation for expressing a process model, such as XML, JSON, Python code, or diagram syntax. Every PMG experiment then runs under one standardized prompt template with LLaMA-3.3-70b, and outputs are scored by element counts and by PME similarity, a two-step measure that semantically matches generated elements to ground-truth elements using sentence embeddings and then computes the Dice–Sørensen coefficient between the sets.
What would settle it
Re-run the PMG experiment on a dataset in which all 55 descriptions include swimlanes or message flows representable in every PMR, and check whether BPMN text still has the highest PME similarity; if its lead disappears on the full set, the reported superiority is an artifact of the evaluable subset.
Extended reading notes
Core claim
The central discovery is a comparative result rather than a new algorithm: across six suitability criteria (token compactness, expressiveness, human readability, visualizability, usability, and extensibility), Mermaid scores highest on average, driven by its combination of extreme compactness and direct visualization; yet when the same 55 descriptions are given to an open LLM with identical prompts, the branching formats BPMN text and JSON branches yield models with clearly higher PME similarity to the ground truth, with BPMN text best overall. The paper also uncovers a systematic under-generation tendency: generated models have roughly eight fewer nodes than the ground truth on average, exclusive gateways are cut by about half, and parallel gateways by about two thirds. Representations that natively encode branching structure partially counteract this drop, which the authors take as evidence that the representation itself shapes generation fidelity.
Load-bearing premise
The comparison assumes the automatic conversion of the ground-truth BPMN models into all nine representations is faithful, and that the subset of models convertible into the branching PMRs, about one third of the dataset, is representative enough for a fair PMG comparison.
Editorial extensions
If this is right
- New or existing PMo tools should consider Mermaid as the default working representation for interactive modeling, because it combines the best compactness with direct visualization.
- PMG systems should prefer branching representations such as BPMN text or JSON branches as generation targets, since these partially offset the LLM's tendency to omit gateways.
- Process modeling pipelines can be staged by representation: use Mermaid for elicitation and communication, then switch to BPMN text for faithful generation from text.
- Future PMG evaluations should report the target PMR explicitly, because the representation alone changes element counts and similarity scores by a large margin.
- The PMo Dataset and the unified protocol provide a common benchmark on which future PMG approaches can be compared against each other.
Reading between the lines
- The paper's two-thirds conversion failure means BPMN text's PMG advantage is measured only on the roughly one-third of models that are representable in branching formats; extending the comparison to swimlanes and message flows could shrink or reverse the gap.
- A direct testable extension would be to prompt the same LLM to generate both Mermaid and BPMN text for identical descriptions and convert both outputs to BPMN before scoring, isolating the representation effect from the dataset-subset effect.
- The suitability grades for six criteria are author-assigned; using the released dataset to collect multi-expert ratings would turn the Mermaid result from a claim into an independently reproducible ranking.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the PMo Dataset of 55 process descriptions paired with ground-truth models in nine process model representations (PMRs), and reports a comparative evaluation of these PMRs for LLM-based process modeling. Suitability is assessed along six author-defined criteria (token compactness, expressivity, human readability, visualization, usability, extensibility), and process model generation (PMG) performance is measured under a standardized prompting setup with LLaMA-3.3-70b using element counts and PME similarity. The paper concludes that Mermaid is the most suitable PMR for PMo with LLMs, while BPMN text yields the best PMG results in terms of process element similarity.
Significance. The paper targets a real gap: PMRs used in LLM-based process modeling have not been systematically compared, and existing PMG evaluations are fragmented. The PMo Dataset and the public code and prompts are useful community assets, and the use of an open-source LLM with a standardized prompting protocol is a methodological strength. The PMG comparison, based on computed PME similarity with semantic matching, is more objective than the suitability grading and produces a concrete, falsifiable ranking. However, the headline suitability conclusion rests on subjective author-assigned grades, and the PMG conclusion is supported by a single run on a restricted subset of the data, so the empirical evidence is currently thinner than the abstract implies.
major comments (3)
- [Section 4.1, Table 4; Section 5] The central claim that Mermaid is the most suitable PMR for PMo with LLMs rests on the six grade columns of Table 4, whose values are assigned by the authors. The paper itself concedes in Section 5 that this evaluation "remains primarily subjective." No inter-rater reliability, sensitivity analysis, or alternative weighting is reported. Several individual grades are contestable: for example, BPMN text receives 1/5 for visualization despite being convertible to BPMN, JSON branches receives a schema-support bonus while BPMN text, which is XML-based, does not, and Mermaid and POWL code both receive 5/5 for token compactness. Because the average scores are close (Mermaid 4.00, Graphviz 3.67, BPMN process 3.50), modest changes to one or two grades could change the ranking. I ask the authors to provide a robustness analysis, such as a sensitivity sweep over grades or a structured multi-expert grading protocol with agreement metrics, before the Mermaid conclusion can be considered supported.
- [Section 3.3, Table 6] The PMG result that BPMN text is best is computed on a restricted subset: Section 3.3 states that for branching PMRs, two thirds of the BPMN models cannot be converted. The PME similarity scores for BPMN text and JSON branches are therefore based on roughly one third of the dataset, and these models may be systematically simpler than the non-convertible ones. The paper does not report the PME scores of the other PMRs on the same subset, so the comparison may conflate PMR quality with subset difficulty. Additionally, Table 6 reports a single run per prompt with no variance estimates or significance tests; with temperature 0.2 and one run, the differences between BPMN text (0.54), JSON branches (0.53), and Mermaid (0.48) may not be stable. I recommend re-evaluating all PMRs on the intersection of convertible models and reporting results over multiple seeds with confidence intervals or a significance test.
- [Section 3.3] The ground-truth PMR models are produced by automatic conversion from BPMN, with validation by converting back to BPMN, but the paper does not report any quantitative or manual verification that the conversions preserve the process semantics relevant for evaluation. The paper also notes that unsupported information is ignored (e.g., conditions in POWL code) and that models are excluded when conversion would lose significant information. Since PME similarity in Section 3.5 is computed against these converted ground truths, errors or omissions introduced during conversion directly affect the reported scores. I ask the authors to present a conversion-quality check, such as a sample of manually inspected conversions or a comparison of element coverage before and after conversion, to support the assumption that the converted ground truths are faithful.
minor comments (5)
- [Section 3.4] There is a duplicated word in the sentence "a PMR should should also support direct graphical visualization."
- [Table 4] The column header "Vizualisable" is misspelled; it should be "Visualizable" or "Visualization capability" for consistency with the text.
- [Section 5] The sentence "BPMN text, in particular, achieves the strongest similarity with ground truth elements for both raw numbers and semantic content" conflates two different measurements: element counts (Table 5) and PME similarity (Table 6). Please phrase this as referring to both element-count differences and PME similarity scores separately.
- [Section 3.2, Table 2] The distinction between "BPMN" and "BPMN process" is clear in the text, but Table 2 would benefit from a note that the difference is the presence or absence of the BPMN diagram definition, as stated in Section 3.2.
- [Section 3.5] The semantic matching threshold of 0.7 is stated as "experimentally defined" without a reference or local justification. Since this threshold affects all PME similarity scores, a brief explanation or a sensitivity check would help the reader assess its impact.
Circularity Check
No circularity: the comparison is empirical and its scoring limitations are explicitly disclosed.
full rationale
This paper is an empirical comparative study rather than a derivation, so the main circularity patterns do not apply. The headline conclusions—Mermaid best suited for PMo and BPMN text best for PMG—are supported by measured length statistics, element-coverage computations, and PME similarity scores computed against ground-truth models originating from independent datasets (Mangler, PMo Benchmark, PET-7, BPMN for research, CCC19). The suitability ranking in Table 4 relies on grades assigned by the authors, but the paper explicitly acknowledges this subjectivity in Section 5 as a limitation, which makes it a disclosed weakness rather than a hidden circular step. No load-bearing self-citations are present: the reference list contains no works by the present authors (Brissard, Cuppens, Zouaq). The PME semantic-matching threshold of 0.7 is inherited from prior work [19] and is not re-fitted to this dataset, so it is not a fitted input presented as a prediction. The restricted coverage for branching PMRs (two thirds of BPMN models cannot be converted) is a genuine external-validity concern but is openly reported and does not constitute circular reasoning. Consequently, the paper does not reduce to its own inputs by construction.
Assumptions & free parameters
free parameters (1)
- Semantic matching threshold =
0.7
assumptions (4)
- domain assumption The six requirements (token compactness, expressivity, human readability, visualization, usability, extensibility) are the relevant criteria for LLM-based PMo.
- domain assumption The nine selected PMRs are representative of PMRs used in current PMG approaches.
- ad hoc to paper Automatic BPMN-to-PMR conversions preserve process semantics despite information loss.
- domain assumption LLaMA-3.3-70b with standard prompting is representative for evaluating PMG performance across PMRs.
Cite this review
Pith. "Pith review of What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models." pith.science (2026). https://pith.science/paper/PKKJUL5J
@misc{pith2026250711356,
author = {Pith},
title = {Pith review of: What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/PKKJUL5J}},
note = {Machine review of arXiv:2507.11356}
}
read the original abstract
Large Language Models (LLMs) are increasingly applied for Process Modeling (PMo) tasks such as Process Model Generation (PMG). To support these tasks, researchers have introduced a variety of Process Model Representations (PMRs) that serve as model abstractions or generation targets. However, these PMRs differ widely in structure, complexity, and usability, and have never been systematically compared. Moreover, recent PMG approaches rely on distinct evaluation strategies and generation techniques, making comparison difficult. This paper presents the first empirical study that evaluates multiple PMRs in the context of PMo with LLMs. We introduce the PMo Dataset, a new dataset containing 55 process descriptions paired with models in nine different PMRs. We evaluate PMRs along two dimensions: suitability for LLM-based PMo and performance on PMG. \textit{Mermaid} achieves the highest overall score across six PMo criteria, whereas \textit{BPMN text} delivers the best PMG results in terms of process element similarity.
Figures
Forward citations
Cited by 1 Pith paper
-
Assessing the Business Process Modeling Competences of Large Language Models
Open-source LLMs can produce BPMN process models that rival human experts on syntax and readability, but they lag on semantic accuracy and frequently generate invalid BPMN-XML.
Reference graph
Works this paper leans on
-
[1]
Apaydin, K., Zisgen, Y.: Local Large Language Models for Business Process Modeling. In: Delgado, A., Slaats, T. (eds.) Process Mining Workshops. pp. 605–609. Springer Nature Switzerland, Cham (2025). https://doi.org/10.1007/978-3-031-82225-4_44
-
[2]
PET: An Annotated Dataset for Process Extraction from Natural Language Text
Bellan, P., van der Aa, H., Dragoni, M., Ghidini, C., Ponzetto, S.P.: PET: An Annotated Dataset for Process Extraction from Natural Language Text (Jun 2022). https://doi.org/10. 48550/arXiv.2203.04860 12 A. Brissard et al
work page Pith review arXiv doi:10.48550/arxiv.2203.04860 2022
-
[3]
Camunda: BPMN for Research Dataset (Dec 2015)
work page 2015
-
[4]
Fettke, P., Houy, C.: Evaluating the Process Modeling Abilities of Large Language Models – Preliminary Foundations and Results (Mar 2025). https://doi.org/10.48550/arXiv.2503.13520
work page Pith review arXiv doi:10.48550/arxiv.2503.13520 2025
-
[5]
In: Modellierung 2024 Satellite Events
Forell, M., Schüler, S.: Modeling meets Large Language Models. In: Modellierung 2024 Satellite Events. p. 10.18420/modellierung2024. Gesellschaft für Informatik e.V. (2024)
-
[6]
https://doi.org/10.48550/arXiv.2307.09923
Grohs,M.,Abb,L.,Elsayed,N.,Rehse,J.R.:LargeLanguageModelscanaccomplishBusiness Process Management Tasks (Jul 2023). https://doi.org/10.48550/arXiv.2307.09923
-
[7]
Software and Systems Modeling21(5), 1877–1906 (Oct 2022)
Ivanchikj, A., Serbout, S., Pautasso, C.: Live process modeling with the BPMN Sketch Miner. Software and Systems Modeling21(5), 1877–1906 (Oct 2022). https://doi.org/10. 1007/s10270-022-01009-w
work page 1906
-
[8]
Ivanov, S., Kalenkova, A., Aalst, van der, W.: BPMNDiffViz : A tool for BPMN models comparison. In: Daniel, F., Zugal, S. (eds.) Proceedings of the Demo Session of the 13th International Conference on Business Process Management (BPM 2015, Innsbruck, Austria, August 31-September 3, 2015). pp. 35–39. CEUR Workshop Proceedings (2015)
work page 2015
Show all 19 references
- [9]
-
[10]
In: BPM2024 (Sep 2024)
Kopke, J., Safan, A.: Efficient LLM-Based Conversational Process Modeling. In: BPM2024 (Sep 2024)
2024
-
[11]
https://doi.org/10.48550/arXiv.2408.08892
Kourani, H., Berti, A., Hennrich, J., Kratsch, W., Weidlich, R., Li, C.Y., Arslan, A., Schuster, D., van der Aalst, W.M.P.: Leveraging Large Language Models for Enhanced Process Model Comprehension (Sep 2024). https://doi.org/10.48550/arXiv.2408.08892
- [12]
- [13]
- [14]
-
[15]
In: 2023 International Joint Conference on Neural Networks (IJCNN)
Li,X.,Ni,L.,Li,R.,Liu,J.,Zhang,M.:MaD:ADatasetforInterview-basedBPMinBusiness Process Management. In: 2023 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (Jun 2023). https://doi.org/10.1109/IJCNN54540.2023.10191898
2023
- [16]
-
[17]
https://doi.org/10.5281/zenodo.7783492
Mangler, J., Klievtsova, N.: Textual Process Descriptions and Corresponding BPMN Models (Mar 2023). https://doi.org/10.5281/zenodo.7783492
2023 doi
-
[18]
https://doi.org/10.4121/UUID: C923AF09-CE93-44C3-ACE0-C5508CF103AD
Munoz-Gama, J., de la Fuente, R.R., Sepúlveda, M.M., Fuentes, R.R.: Confor- mance Checking Challenge 2019 (CCC19) (Feb 2019). https://doi.org/10.4121/UUID: C923AF09-CE93-44C3-ACE0-C5508CF103AD
2019 doi
- [19]
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.