REVIEW 3 major objections 6 minor 30 references
Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM
T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The paper reports a qualified yes: a modeling tool largely written by an AI coding agent can be trusted, but only when correctness is enforced by executable external checks and a domain expert in the loop.
desk verdict A credible single-case experience report: the measured dev-mining numbers and taxonomy are genuinely new, and the trust-in-scaffolding claim is plausible and well-argued, but neither the label reliability nor the oracle sensitivity is actually measured — treat the headline ratios as provisional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the oracle-based validation strategy placed at model-transformation boundaries. PM4Py-UCM's pipeline is a chain of transformations — process tree to UCM object model to interchange file and back — where a wrong condition or label can produce a file that opens cleanly yet encodes incorrect behavior. The paper makes these boundaries mechanically checkable using golden-file oracles (known-good interchange fixtures), metamorphic tests (comparing outputs across related inputs), deterministic round-trips against a reference editor, and metamodel conformance checks. A second, reflexive-mining apparatus classifies every human turn from agent-session transcripts using a taxon
What would settle it
Have two independent human coders apply the published taxonomy to the 317 substantive turns and measure agreement with the paper's gold labels; if the fix-to-feature ratio departs materially from 2.3:1 or the correction rate from 18%, the paper's quantitative case fails, even though its qualitative pattern of fix-heavy, boundary-concentrated development may survive.
Extended reading notes
Core claim
The paper's core claim is that an AI-agent-written modeling tool can be made trustworthy, but the trust must be engineered: transformations between the process tree, the UCM object model, and the interchange format are exactly where errors become silent, producing files that open cleanly yet encode wrong behavior. PM4Py-UCM's validation strategy pairs every feature with an executable oracle — hand-computed fixtures, algebraic invariants, metamorphic transforms, byte-deterministic round-trips, and differential comparison against reference metric functions — so that subtle semantic corruption surfaces as visible test failure. Measured on the development record, this strategy corresponds to a t
Load-bearing premise
The quantitative backbone relies on the LLM-proposed, author-sample-reviewed gold labels being an accurate reference standard for classifying turns; the paper acknowledges these labels are not independently human-coded, so the 2.3:1 and 18% figures could shift if the labels are systematically wrong.
Editorial extensions
If this is right
- Teams commissioning agent-built modeling tools should budget for a refinement loop: fixes outnumbered features 2.3:1, so iteration, not first-shot generation, is the expected workflow.
- Agent weaknesses concentrate at format and renderer boundaries — external interchange formats and visual fidelity accounted for the highest fix rates (78–83%) — so verification should target those boundaries first.
- A domain expert in the loop is indispensable: the 18% correction rate recorded expert repairs of semantic errors, not generic oversight.
- Tests can keep pace with feature growth under agent-driven development — the suite grew from 108 to 691 test functions in lockstep with releases — so 'test as you go' is feasible.
- Independent analyzers corroborated the internal oracles: 90.6% line coverage, low average complexity, and no medium/high-severity security findings, suggesting quality was not sacrificed for velocity.
Reading between the lines
- The paper's own threat analysis implies the exact quantitative ratios depend on the reliability of LLM-proposed, author-sample-reviewed gold labels; an obvious extension is to re-label the corpus with two independent human coders and estimate inter-rater agreement.
- If the oracle-based pattern transfers, the scarce skill for AI-assisted modeling-tool development shifts from writing transformation code to designing executable oracles and metamodel conformance checks.
- A testable extension suggested by the paper is to give the agent an explicit 'definition of done' enumerating auxiliary artifacts (docs, notebooks, save/load schemas) to recover the 9.8% cross-cutting consistency tax.
- The qualitative results — fix-heavy development and error concentration at external-format boundaries — are more likely to generalize than the specific ratios, but replication on other tools and less-expert developers is needed to establish transferability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports on the agent-assisted development of PM4Py-UCM, an open-source Python tool that mines Use Case Map models from event logs and exports them to jUCMNav. The author mined 18 Claude Code sessions, 374 original human turns (317 substantive), 151 commits, and test growth from 108 to 691 functions, classifying turns with a 10-intent/5-flag taxonomy. Headline results: fixes outnumber features 2.3:1, 18% of turns correct agent errors, 9.8% are cross-cutting consistency work, and tool actions per turn are about 28:1. Independent analyzers report 90.6% line coverage, no medium/high-severity Bandit findings, 2/2 architecture contracts, and stable complexity. The paper argues that trust in the resulting tool came not from the agent but from executable verification scaffolding: golden-file oracles, metamorphic tests, byte-deterministic XMI round-trips, and differential tests, plus a domain expert in the loop. It offers a reflexive-mining toolkit and lessons for AI-assisted EM tooling.
Significance. The paper is honest about its single-case scope and ships a reproducible, privacy-preserving toolkit, published gold labels and aggregates, and genuinely external third-party analyzers (pytest-cov, radon, Bandit, import-linter, pip-audit). If the labeling is reliable, the 2.3:1 fix:feature ratio and 18% correction rate provide useful descriptive benchmarks for agent-assisted development. The paper also makes a valuable distinction between code quality (coverage, security, architecture) and semantic correctness of model transformations, a distinction often missing in reports on LLM-generated software. However, the central causal claim — that the verification scaffolding turned silent semantic corruption into visible failures — is asserted rather than demonstrated, and the quantitative backbone depends on an unvalidated reference standard. These gaps are fixable and do not invalidate the qualitative patterns, but they currently limit the strength of the conclusions.
major comments (3)
- [Sect. 5.4, 6 (Lesson 3), 9] The central conclusion — that 'mechanical checkability' turned silent semantic corruption into visible failures — is not empirically demonstrated. The paper lists golden-file oracles, metamorphic tests, byte-deterministic round-trips, and differential checks, but reports no mutation testing or fault seeding on the transformation chain, and jUCMNav is not part of the automated test suite. A byte-deterministic round-trip verifies internal consistency, not semantic conformity; a metamorphic relation verifies only the chosen property. Thus the oracles' sensitivity to the specific failure mode in Sect. 3 (a wrong condition/label yielding a .jucm file that opens cleanly but executes incorrectly) is unmeasured. Please report a mutation/fault-injection score, or at least an explicit list of injected semantic mutants and which oracles caught them, before claiming the scaffolding 'turned' corrupti
- [Sect. 5.2, Table 2, Sect. 7] The quantitative backbone (2.3:1 fix:feature, 18% corrects_agent, 9.8% cross_cutting) depends on gold labels that are 'LLM-proposed and expert-verified (through a sample)' rather than independently human-coded, with no reported inter-rater reliability. Sect. 5.2 admits the reported comparison 'measures the baseline's inadequacy, not the reference's reliability.' Since the labeler is the same model family that wrote the code and the expert is the author, self-preference and confirmation bias are unchecked. Please add a second human coder and report agreement (e.g., Cohen's kappa or per-class precision/recall), or explicitly reclassify these ratios as exploratory and remove them from the evidential load of the conclusion.
- [Sect. 5.5, Table 4] The independent static analyzers are a genuine strength, but the statement that they 'corroborate our internal oracle-based validation' overreaches. Coverage, complexity, maintainability, Bandit findings, and import-linter contracts measure code quality and structural hygiene; they do not measure whether the oracles detect semantic corruption in model transformations. The corroboration is therefore indirect and should be framed accordingly. If the causal claim is kept, the evidence must come from oracle sensitivity (e.g., mutation testing), not from these quality metrics.
minor comments (6)
- [Title/Abstract] Typographical issues: 'onPM4Py-UCM' and 'Use Case Maps·Process' are missing spaces. Please fix.
- [Sect. 4 IV] The prompt text begins with 'Y ou classify' — typo for 'You classify'.
- [Sect. 5] The ≈28:1 tool-action-to-turn ratio uses 374 original turns, while most other proportions use 317 substantive turns. State the denominator explicitly at the point of the ratio to avoid ambiguity.
- [Sect. 5, footnote 5] The 65-hour active-time figure depends on the 10-minute inter-event gap threshold; a sensitivity check (e.g., 5/15/20 minutes) would strengthen the claim that this is a conservative floor.
- [Sect. 5.1 vs Sect. 7] Sect. 5.1 calls the 18% corrects_agent rate a 'conservative lower bound,' while Sect. 7 calls it an 'upper bound' on genuine agent error (because ambiguous instructions may be folded in). These statements are not contradictory if interpreted precisely, but the wording should be reconciled to avoid confusion.
- [Table 3] The 'jUCMNav export/round-trip' row has only N=6 turns; the 83% fix rate is a small-sample descriptive statistic. Consider adding a caution or confidence statement when discussing this as evidence that the agent 'struggled most' with external-format fidelity.
Circularity Check
No significant circularity: the trust argument is anchored to external oracles and independent analyzers; the gold-label self-reference is an acknowledged validity threat, not a circular reduction.
full rationale
This is an experience report, not a derivation chain with predictions. The central claim (Sect. 9) — that trust in PM4Py-UCM came from mechanical checks and expert verification rather than agent self-report — is supported by external evidence: byte-deterministic XMI round-trips, golden-file oracles, metamorphic tests, differential comparisons against PM4Py's metric functions, and independent third-party analyzers (pytest-cov, radon, Bandit, import-linter, pip-audit; Sect. 5.5). None of these reduce to the conclusion by construction: the oracles test against a reference metamodel/editor and hand-computed fixtures, and the static metrics are computed by external tools on the final code. The paper's quantitative development measures (2.3:1 fix:feature ratio, 18% corrects_agent) do depend on LLM-proposed, sample-reviewed labels, and Sect. 5.2 explicitly states that the comparison 'measures the baseline's inadequacy, not the reference's reliability,' while Sect. 7 lists the single-developer/analyst and LLM-assisted labeling as validity threats. This is a measurement-reliability limitation, not a circularity: the labels are not used to define the trust conclusion, and the qualitative pattern (fix-heavy development, error concentration at XMI/visualization boundaries) is presented as a case-study hypothesis rather than a forced result. The prior-work citations ([2], [4]) are for inherited capabilities and are treated as given, not as evidence for the trust claim; the jUCMNav reference editor is an external artifact, not an output of this paper. No equation or fitted parameter is renamed as a prediction, and no uniqueness theorem is invoked. The skeptic's concern that oracle sensitivity is unmeasured (no mutation testing) is a correctness/evidence gap, not circularity. Overall score 0.
Assumptions & free parameters
free parameters (2)
- active-time inter-event gap threshold =
10 minutes
- substantive-turn inclusion rules (resume-duplicate collapse, interrupt/attachment artifact removal) =
48 duplicates collapsed; 374 → 317 turns
assumptions (4)
- ad hoc to paper The LLM-produced, author-sample-reviewed gold labels are an accurate enough reference standard to support the reported proportions (fix 33.8%, corrects_agent 18%, cross_cutting 9.8%)
- domain assumption Agent-session transcripts plus git history and test snapshots faithfully represent the development effort
- domain assumption The inductive miner's block-structured process trees map cleanly to the UCM metamodel subset, making the discovery chain semantics-preserving
- domain assumption jUCMNav's .jucm (XMI) file format and the ITU-T Z.151 metamodel are the authoritative oracle targets for correctness
invented entities (1)
-
Turn-classification taxonomy with 10 primary intents and 5 cross-cutting flags (cross_cutting, corrects_agent, user_incorrect, scope_change, rework)
independent evidence
Cite this review
Pith. "Pith review of Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM." pith.science (2026). https://pith.science/paper/3DEWRDGH
@misc{pith2026260728825,
author = {Pith},
title = {Pith review of: Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM},
year = {2026},
howpublished = {\url{https://pith.science/paper/3DEWRDGH}},
note = {Machine review of arXiv:2607.28825}
}
read the original abstract
Enterprise-modeling (EM) tools are often complex and hard to extend. Yet, users may want to explore new EM features and capabilities that currently do not exist. AI coding agents can help here by enabling the development of new capabilities and entire tools, but whether we can trust a modeling-language tool an LLM largely wrote remains a question. This paper reports on the AI-assisted construction of PM4Py-UCM, an open-source tool that mines Use Case Map (UCM) models from event logs. PM4Py-UCM's capabilities include some expected from process mining tools (e.g., performance heat-maps and dashboards) and distinctive ones (e.g., mined executable scenarios/variants, and model decomposition). We mined the development record itself, composed of 18 agent sessions (374 human turns and 10,328 tool actions over 65 hours), 151 commits, 20 releases, and a test suite grown from 108 to 691 test functions, in order to characterize, in a single in-depth case, how the tool was built with an agent (Claude Code), complemented by an independent static assessment of the resulting code (coverage, complexity, maintainability, security, architecture). We contribute a reproducible, privacy-preserving toolkit and taxonomy that classify human turns and flag cross-cutting consistency work, agent corrections, and retracted requests. Up to version 0.7.4, fixes outnumber features 2.3:1, with ~18% of turns for correcting agent errors. Feature waves dragged a measurable tail of documentation/test/notebook consistency work, and tests grew lockstep with features. We finally present lessons learned, centered on making model transformations mechanically checkable, and the oracle-based validation strategy that closed the "the agent said it works" gap, for responsibly engineering EM tooling with AI.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
van der Aalst, W.: Process Mining: Data Science in Action, pp. 3–23. Springer (2016).https://doi.org/10.1007/978-3-662-49851-4_1
-
[2]
In: 18th System Analysis and Modelling Conference (SAM ’26)
Amyot, D.: On process mining executable use case maps: Concurrency-aware sce- nario synthesis with variant- and data-driven conditions. In: 18th System Analysis and Modelling Conference (SAM ’26). ACM (2026)
2026
-
[3]
Zenodo (2026).https://doi.org/10.5281/zenodo.21611544
Amyot, D.: PM4Py-UCM reflexive development-mining and quality toolkit (repli- cation package). Zenodo (2026).https://doi.org/10.5281/zenodo.21611544
-
[4]
Towards Process Mining Use Case Map Models with PM4Py-UCM
Amyot, D.: Towards process mining use case map models with PM4Py-UCM. In: IEEE 34th Int. Requirements Engineering Conference Workshops (REW). IEEE CS (2026).https://doi.org/10.48550/arXiv.2606.04350
work page Pith review arXiv doi:10.48550/arxiv.2606.04350 2026
-
[5]
Amyot, D., Akhigbe, O., Baslyman, M., Ghanavati, S., Ghasemi, M., Hassine, J., Lessard, L., Mussbacher, G., Shen, K., Yu, E.: Combining goal modelling with business process modelling: Two decades of experience with the user requirements notation standard. Enterprise Modelling and Information Systems Architectures (EMISAJ)17, 2:1–38 (2022).https://doi.org/...
-
[6]
Software Impacts17(2023).https://doi.org/10.1016/j.simpa.2023.100556
Berti, A., van Zelst, S., Schuster, D.: PM4Py: A process mining library for Python. Software Impacts17(2023).https://doi.org/10.1016/j.simpa.2023.100556
arXiv 2023
-
[7]
Bhati, H.: Agentic AI in the software development lifecycle: Architecture, empirical evidence, and the reshaping of software engineering (2026),https://arxiv.org/ abs/2604.26275
arXiv 2026
-
[8]
IEEE Transactions on Software Engineering24(12), 1131–1155 (1998).https://doi
Buhr, R.J.A.: Use case maps as architectural entities for complex systems. IEEE Transactions on Software Engineering24(12), 1131–1155 (1998).https://doi. org/10.1109/32.738343
Show all 30 references
-
[9]
Software & Systems Modeling22(3), 781–793 (2023).https://doi.org/10.1007/ s10270-023-01105-5
Cámara, J., Troya, J., Burgueño, L., Vallecillo, A.: On the assessment of gen- erative AI in modeling tasks: An experience report with ChatGPT and UML. Software & Systems Modeling22(3), 781–793 (2023).https://doi.org/10.1007/ s10270-023-01105-5
2023
-
[10]
ACM Computing Surveys51(1), 4:1–4:27 (2018).https://doi.org/10.1145/3143561
Chen, T.Y., Kuo, F.C., Liu, H., Poon, P.L., Towey, D., Tse, T.H., Zhou, Z.Q.: Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys51(1), 4:1–4:27 (2018).https://doi.org/10.1145/3143561
2018 doi
-
[11]
Management Science (2026).https://doi.org/10.1287/mnsc.2025
Cui,K.Z.,Demirer,M.,Jaffe,S.,Musolff,L.,Peng,S.,Salz,T.:Theeffectsofgener- ative AI on high-skilled work: Evidence from three field experiments with software developers. Management Science (2026).https://doi.org/10.1287/mnsc.2025. 00535
2026 doi
-
[12]
Dong, Y., Jiang, X., Qian, J., Wang, T., Zhang, K., Jin, Z., Li, G.: A survey on code generation with LLM-based agents (2025),https://arxiv.org/abs/2508.00083
2025 arXiv
-
[13]
Enterprise Modelling and Information Systems Architectures (EMISAJ)8(1), 4–25 (2013).https://doi.org/10.18417/ emisa.8.1.1
Fill, H.G., Karagiannis, D.: On the conceptualisation of modelling methods us- ing the ADOxx meta modelling platform. Enterprise Modelling and Information Systems Architectures (EMISAJ)8(1), 4–25 (2013).https://doi.org/10.18417/ emisa.8.1.1
2013
-
[14]
Software & Systems Modeling13(3), 941–962 (2014).https://doi.org/10.1007/s10270-012-0273-9
Frank,U.:Multi-perspectiveenterprisemodeling:Foundationalconcepts,prospects and future research challenges. Software & Systems Modeling13(3), 941–962 (2014).https://doi.org/10.1007/s10270-012-0273-9
2014 doi
-
[15]
Business & Information Systems Engineering68, 59–83 (2026)
Hörner, L.F., Möller, M., Reichert, M.: Automatically generating BPMN 2.0 pro- cess models from natural language process descriptions: Challenges, framework, quality assessment. Business & Information Systems Engineering68, 59–83 (2026). https://doi.org/10.1007/s12599-025-0098...
2026 doi
-
[16]
ACM Transactions on Software Engineering and Methodology33(8) (2024).https://doi.org/10.1145/3695988
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., Wang, H.: Large language models for software engineering: A systematic liter- ature review. ACM Transactions on Software Engineering and Methodology33(8) (2024).https://doi.org/10.1145/3695988
2024 doi
-
[17]
iso.org/obp/ui/#iso:std:iso-iec:15909:-1:en
ISO: ISO/IEC 15909-1:2019 – Systems and software engineering – High-level Petri nets – Part 1: Concepts, definitions and graphical notation (2019),https://www. iso.org/obp/ui/#iso:std:iso-iec:15909:-1:en
2019
-
[18]
ITU-T: Recommendation Z.151 (10/2018): User requirements notation (URN) — language definition. Tech. rep., International Telecommunication Union (2018), https://www.itu.int/rec/T-REC-Z.151-201810-I/en
2018
-
[19]
Ji, Y., Wang, J., Zhou, Y., Chen, F., Yang, H.: An exploratory study on LLM- generatedcodeandcommentsincoderepositories.JournalofSystemsandSoftware 242, 113026 (2026).https://doi.org/10.1016/j.jss.2026.113026
2026
-
[20]
International Journal of Information Technology and Computer Science14(5), 1–14 (2022).https: //doi.org/10.5815/ijitcs.2022.05.01
Kesici, C.A., Ozkan, N., Taşkesenlioglu, S., Erdogan, T.G.: A systematic liter- ature review of studies comparing process mining tools. International Journal of Information Technology and Computer Science14(5), 1–14 (2022).https: //doi.org/10.5815/ijitcs.2022.05.01
2022 doi
-
[21]
Apress Berkeley, USA (2025).https://doi.org/10.1007/979-8-8688-1826-4
Khorasani, M., Abdou, M., Fernández Hernández, J.: Streamlit for Web Devel- opment: Build and Scale Secure Python-Powered Apps with Streamlit. Apress Berkeley, USA (2025).https://doi.org/10.1007/979-8-8688-1826-4
2025 doi
-
[22]
Enterprise Modelling and Information Systems Architec- tures (EMISAJ)17, 3–1 (2022).https://doi.org/10.18417/emisa.17.3
Laue, R., Mayr, H.C., Thalheim, B.: 100 years of graphical business process mod- elling: Guest editorial. Enterprise Modelling and Information Systems Architec- tures (EMISAJ)17, 3–1 (2022).https://doi.org/10.18417/emisa.17.3
2022 doi
-
[23]
In: Application and Theory of Petri Nets and Concurrency
Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Discovering block-structured process models from event logs - a constructive approach. In: Application and Theory of Petri Nets and Concurrency. LNCS, vol. 7927, pp. 311–329. Springer (2013).https://doi.org/10.1007/978-3-642...
2013 doi
-
[24]
International Journal of Data Science and Analytics15(4), 407–420 (2023).https: //doi.org/10.1007/s41060-022-00379
Loyola-González, O.: Process mining: software comparison, trends, and challenges. International Journal of Data Science and Analytics15(4), 407–420 (2023).https: //doi.org/10.1007/s41060-022-00379
2023 doi
-
[25]
OMG: Business process model and notation (BPMN), version 2.0.2 (2014),https: //www.omg.org/spec/BPMN/2.0.2
2014
-
[26]
Journal of Machine Learning Research12, 2825–2830 (2011),http: //scikit-learn.sourceforge.net
Pedregosa, F., Varoquaux, G., Gramfort, A., et al.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research12, 2825–2830 (2011),http: //scikit-learn.sourceforge.net
2011
-
[27]
In: System Analysis and Modeling (SAM 2006)
Roy, J.F., Kealey, J., Amyot, D.: Towards integrated tool support for the user requirements notation. In: System Analysis and Modeling (SAM 2006). LNCS, vol. 4320, pp. 198–215. Springer (2006).https://doi.org/10.1007/11951148_13
2006 doi
-
[28]
The Enterprise Engineering Series, Springer (2014).https://doi.org/10.1007/978-3-662-43725-4
Sandkuhl, K., Stirna, J., Persson, A., Wißotzki, M.: Enterprise Modeling: Tackling Business Challenges with the 4EM Method. The Enterprise Engineering Series, Springer (2014).https://doi.org/10.1007/978-3-662-43725-4
2014 doi
-
[29]
Information and Software Technology178, 107610 (2025).https://doi.org/10.1016/j.infsof
Sergeyuk, A., Golubev, Y., Bryksin, T., Ahmed, I.: Using AI-based coding assis- tants in practice: State of affairs, perceptions, and ways forward. Information and Software Technology178, 107610 (2025).https://doi.org/10.1016/j.infsof. 2024.107610
2025
-
[30]
Computers in Industry122,103265(2020).https://doi.org/10.1016/j.compind.2020.103265
Vernadat, F.: Enterprise modelling: Research review and outlook. Computers in Industry122,103265(2020).https://doi.org/10.1016/j.compind.2020.103265
2020
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.