REVIEW 4 major objections 5 minor 23 references
LLM-based Vulnerability Discovery in Business Process Documentation
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Off-the-shelf LLMs, without fine-tuning, can find contradictions and subtle numeric errors in business documentation and automatically turn ERP user guides into executable Petri nets.
desk verdict Plausible system, weak evaluation: the same LLM that writes the test data shouldn't be the one scoring it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the Documentation Processing Toolkit (DPT), a set of LLM-driven stages built around a shared idea: break documents into chunks, prompt a general-purpose LLM to classify or extract, then feed structured output into a formal-model downstream. For contradiction detection, the paper pairs chunks, asks the model whether two passages are inconsistent, and uses logprob outputs on single yes/no tokens (LLM-as-classifier) to cull unlikely pairs and avoid checking all quadratic many pairs. For work instructions, a single prompt asks for a brief explanation of any inconsistency or the response 'none'. For ERP guides, the pipeline normalizes PDFs to markdown, performs structure-aware hierarchical chunking, extracts references and user-action summaries, and synthesizes JSON for a Petri net simulation toolchain. The repeated finding is that chunk size and model scale, not task-specific training, drive accuracy.
What would settle it
Run the work-instruction flagging prompt on a set of genuinely human-authored erroneous instructions from an equipment manual or field service report (errors like wrong pressure conversions or contradictory torque limits). If accuracy on those instructions falls clearly below the reported 0.96, the claim that the approach generalizes from synthetic instructions to real business documentation is falsified.
Extended reading notes
Core claim
The central claim is that the hypothesis that LLMs can be effective for uncovering documentation contradictions buried deeply in business text is valid, and that the same models also flag subtle numeric errors in short work instructions and reconstruct executable process structure from real user guides. In the contradiction task, GPT-OSS:120B with a 512-token chunk size performed best, accuracy rose as chunk size shrank and as model size grew, and small models failed. In the work-instruction task, the LLM achieved precision 0.96 and accuracy 0.96 over 100 synthetic instructions, even catching one error the authors had missed in a 'clean' instruction. In the ERP pipeline, all tested backends extracted references and action summaries with high fidelity, while the largest non-quantized model produced the most coherent Petri-net-compatible JSON; quantized models were faster but sometimes structurally inconsistent. Together these results support the claim that LLMs significantly accelerate the transition from unstructured enterprise documentation to rigorous, machine-readable formats.
Load-bearing premise
The evaluation data is assumed to resemble real business documentation errors: the contradiction corpus is not business text (which the authors note may give the LLM an unfair advantage), and the erroneous work instructions were generated by the same family of large language model that was later scored, so if real human-authored documentation errors look different, the reported accuracies may not transfer.
Editorial extensions
If this is right
- Contradiction detection in business documentation can be made practical by choosing smaller chunk sizes (512 tokens in this study) and larger models, at the cost of more chunk pairs to check.
- Quantized models are a viable throughput option for extraction tasks, but structured workflow synthesis (Petri net generation) still favors a larger, non-quantized model.
- The same general-purpose LLM, without fine-tuning, can handle three different document-processing jobs (contradiction classification, work-instruction flagging, and process extraction) with the same style of prompting.
- The extracted Petri-net JSON can serve as input to simulation, verification, and auditing tools, closing the gap between natural-language documentation and formal logic-based fault detection.
Reading between the lines
- A natural next experiment is to test the contradiction-detection step with contradiction pairs rewritten in full business prose (torque limits, tolerance ranges, sign-off rules) rather than encyclopedia-style sentences, to see whether the accuracy advantage holds without the lexical-context cue.
- The logprob-based LLM-as-classifier culling could be made cheaper still by pre-filtering chunk pairs with embedding similarity, reserving the LLM for pairs that are semantically close but lexically different.
- A two-tier deployment is implied by the results: a fast quantized model for broad extraction and a large non-quantized model to verify structured workflow output, which could bring the accuracy of the pipeline to controlled-information environments at acceptable throughput.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes MIRABELLE/DPT, a pipeline that uses off-the-shelf LLMs to process business-process documentation. It reports three experiments: (i) contradiction detection by injecting Stanford Contradiction Corpora pairs into a host document and asking LLMs to identify inconsistent chunk pairs; (ii) classification of 100 synthetic work instructions, half with injected subtle errors, generated by GPT-OSS:250B and later scored by the same model; and (iii) an end-to-end pipeline that converts Chapter 15 of the Oracle Purchasing User Guide into CPN-Py Petri Net JSON. The paper reports high accuracy and precision for (i) and (ii), and qualitative success with quantified runtimes for (iii), and concludes that LLMs significantly accelerate formalization and vulnerability discovery.
Significance. If the empirical claims transferred to real enterprise documentation, the system would be practically valuable: it combines document ingestion, LLM-based extraction, graph/Petri-net formalization, and downstream logic analysis in one pipeline, and it compares quantized and non-quantized backends on realistic runtimes. The paper is transparent about several limitations (SCC is not business text; work instructions are synthetic), and the pipeline engineering is described in enough detail to replicate. However, the current evaluation does not yet support the transfer claim: the two error-detection experiments use synthetic distributions that are either generated by the same model under test or assembled from non-business text, and the Petri-net experiment has no quantitative accuracy assessment. The contribution is therefore an initial system demonstration rather than a validated empirical result.
major comments (4)
- [§3.2, Table 1] The headline precision/accuracy of 0.96 is computed on a corpus of 100 work instructions that were generated by GPT-OSS:250B, the same model that is then evaluated at temperature 0.1. This does not establish transfer to real, human-authored erroneous work instructions: the model's ability to find errors injected according to its own generation priors may reflect consistency with those priors rather than general error-detection ability. Since the claims in §4 about 'high accuracy in detecting nuanced conversion and range errors within work instructions' rest entirely on this dataset, the evaluation is load-bearing. I request a rerun on human-authored or independently collected erroneous work instructions, with per-category (conversion, range, unit) results and error bars, or a clear reframing of the result as a self-consistency check.
- [§3.1, Figure 3] The contradiction-detection experiment injects SCC pairs into an unspecified 'host document' and reports only a figure without exact accuracy values in the text. The authors themselves state that SCC text is 'not pure business text' and 'might give the LLM an unfair advantage,' so the experiment does not support the strong conclusion that 'LLMs can be effective for uncovering documentation contradictions buried deeply in business text.' Moreover, no baseline, precision/recall breakdown, or statistical characterization is provided. Please specify the host document and injection protocol, report exact per-condition accuracies (and ideally precision/recall/F1), and re-evaluate on real SOP/user-guide text or on business-style contradiction pairs before claiming validity.
- [§3.3, Table 2] The Petri-net extraction experiment reports processing times but no quantitative evaluation of correctness for the generated CPN-Py JSON. Statements such as 'high fidelity', 'most accurate and coherent', and 'usable but occasionally incomplete' are not backed by any metric, ground-truth comparison, or human evaluation protocol. As a result, the conclusion that the pipeline 'significantly accelerates the transition from unstructured enterprise documentation to rigorous, machine-readable formats' is not yet supported. I request a scoring scheme for generated workflows (e.g., structural alignment with a reference model, place/transition precision and recall, or expert ratings on a defined scale) and a per-model accuracy table to accompany the runtime table.
- [§3.2, Table 1 vs text] The numbers in §3.2 and Table 1 are inconsistent. The text says the LLM 'correctly validating all the clean instructions' and that precision was '1.0 (or 0.96 if we let the zero-day error ride)', then reports accuracy '0.98' for the error set, while Table 1 lists both Precision and Accuracy as 0.96. The definitions of precision and accuracy are also not given (e.g., accuracy over all 100 instructions, or over the 50 faulty ones only). Please reconcile the reported scores and define the metrics precisely; this affects the credibility of the central 0.96 claim.
minor comments (5)
- [§3.1, §3.2] Section 3.1 refers to GPT-OSS:120B in the model list and Figure 3, while §3.2 uses GPT-OSS:250B and the scaling discussion mentions GPT-OSS (250B parameters); clarify whether these are two distinct checkpoints or a typo.
- [Figure 3] The x-axis labels '1..10' and the plotted series are not defined in the caption; please state what each run/series represents and add replicate counts or error bars.
- [§3.2] The phrase 'zero-day error' is informal and undefined; replace it with standard terminology such as 'undetected annotation error' or 'false positive against the original annotation.'
- [Abstract, §3.3] The abstract claims 'attributed graph representations' and 'formal logic approaches' that are processed downstream, but the reported experiments stop at Petri-net JSON generation; clarify the scope so readers do not expect end-to-end vulnerability reports in this paper.
- [Figure 1] The caption spells 'MIRABELE' while the text uses 'MIRABELLE'; fix the typo.
Circularity Check
No circularity found: the Section 3.2 same-model synthetic evaluation is a validity concern, not a by-construction reduction.
full rationale
The paper's claims are empirical evaluations rather than derivations, and none of the reported results is equivalent to its own inputs by construction. Section 3.1 tests contradiction detection using the independent Stanford Contradiction Corpora injected into a host document; the accuracy is an observed outcome, and the authors themselves note that SCC text is not pure business text, which is an acknowledged limitation rather than a circular step. Section 3.2 uses 100 synthetic work instructions generated by GPT-OSS:250B and then evaluates GPT-OSS:250B on them, creating a same-distribution confound for any transfer claim to real work instructions. However, this is not circular in the defined sense: no parameter is fitted and then renamed as a prediction, and the evaluation outcome was not forced (the model missed one injected error and also found an additional 'zero-day' issue in a clean instruction). Section 3.3 is a qualitative pipeline demonstration on the real Oracle Purchasing User Guide, with no quantitative accuracy claim that reduces to its input. No load-bearing self-citation or imported uniqueness theorem appears; the cited prior work is not used to forbid alternatives. The synthetic-data limitation belongs to external validity and correctness risk, not circularity, per the review rules.
Assumptions & free parameters
free parameters (3)
- chunk_size =
512, 1024, 2048, 4096 tokens
- temperature =
0.1
- contradiction_probability_threshold =
not specified
assumptions (5)
- domain assumption Business process vulnerabilities can be detected from documentation text alone, without access to running systems.
- ad hoc to paper Injecting Stanford Contradiction Corpora pairs into a host document produces a valid proxy for contradictions in real business documentation.
- ad hoc to paper Synthetic work instructions generated by GPT-OSS:250B, after hand-checking, are representative of real erroneous work instructions.
- domain assumption LLM-extracted Petri nets from ERP user guides are correct without comparison to a gold standard.
- standard math Logprob-based LLM-as-classifier output provides a valid estimate of contradiction probability.
Cite this review
Pith. "Pith review of LLM-based Vulnerability Discovery in Business Process Documentation." pith.science (2026). https://pith.science/paper/YLSXA4P4
@misc{pith2026260804271,
author = {Pith},
title = {Pith review of: LLM-based Vulnerability Discovery in Business Process Documentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/YLSXA4P4}},
note = {Machine review of arXiv:2608.04271}
}
read the original abstract
Just like software and hardware, business processes are susceptible to vulnerabilities that can lead to product quality issues, delays, and increased costs. Business process vulnerabilities can arise from a variety of sources, including conflicting requirements, ambiguous documentation, invalid measurement spec-ifications, omission of quality checks, or implementations that differ from speci-fications. MIRABELLE is a system that identifies and characterizes business logic (BL) vulnerabilities from available business process representations, in-cluding ISO 9000/9001 documentation, user guides, work instructions, and pro-cess execution logs. MIRABELLE leverages recent advances in AI/ML to pro-cess available business process documentation and generate attributed graph rep-resentations of the business logic that can be processed using both graph and for-mal logic approaches for identifying potential vulnerabilities. However, extract-ing the business logic (e.g., operation execution sequences, decisions, input/out-put resources) from mostly natural language artifacts is challenging due to the required domain expertise, inherent process complexity, and the sometimes very large volumes of information. This paper focuses on our experimentation with Large Language Models (LLMs) and their role within MIRABELLE. We report on the performance of several LLMs across vital stages of vulnerability detection, from grammatical and technical error-flagging in short phrasings, to complete process structure recovery and extraction.
Reference graph
Works this paper leans on
-
[1]
Finding contradictions in text
Marneffe M., Rafferty A., Manning C.: “Finding contradictions in text”. 46th Annual Meet- ing of the Assoc. for Computational Linguistics and Human Language Technology Confer- ence, Ohio (2008)
work page 2008
-
[2]
ProMoAI: process modeling with generative AI
Kourani H. et al. "ProMoAI: process modeling with generative AI." arXiv preprint arXiv:2403.04327 (2024)
arXiv 2024
-
[3]
MetaBPL: Fault Detection in Business Logic Systems
Katsios G. et al. "MetaBPL: Fault Detection in Business Logic Systems." Human Aspects of Advanced Manufacturing, Production Management and Process Control 184 (2025): 102
work page 2025
-
[4]
Process model generation from natural language text
Friedrich F., Mendling J, and Puhlmann F. "Process model generation from natural language text." International conference on advanced information systems engineering . Berlin, Hei- delberg: Springer Berlin Heidelberg, 2011
work page 2011
-
[5]
A qualitative analysis of the state of the art in pro- cess extraction from text
Bellan, P., Dragoni M., and Ghidini C. "A qualitative analysis of the state of the art in pro- cess extraction from text." Proceedings of the AIxIA 2020 Discussion Papers Workshop . Vol. 2776. CEUR-WS. org, 2020
work page 2020
-
[6]
POWL: partially ordered workflow language
Kourani H., and J. van Zelst. S., "POWL: partially ordered workflow language." Interna- tional Conf. on Business Process Management. Cham: Springer Nature Switzerland, 2023
work page 2023
-
[7]
Structured Extraction from Business Process Diagrams Using Vision-Language Models
Deka P., and Devereux B. "Structured Extraction from Business Process Diagrams Using Vision-Language Models." arXiv preprint arXiv:2511.22448 (2025)
arXiv 2025
-
[8]
Abstractions, scenarios, and prompt def- initions for process mining with llms: A case study
Berti, A., Schuster D., and MP van der Aalst, W. "Abstractions, scenarios, and prompt def- initions for process mining with llms: A case study." International conference on business process management. Cham: Springer Nature Switzerland, 2023
work page 2023
Show all 23 references
-
[9]
Gpt-4 technical report
Achiam, J. et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023)
2023 arXiv
-
[10]
Process Mining Tools,
Celonis Inc., "Process Mining Tools," [Online]. Available: https://celonis.com
-
[11]
SAP Signavio,
SAP, "SAP Signavio," [Online]. Available: https://www.signavio.com/
-
[12]
Process MIning Workbench
ProM Tools. "Process MIning Workbench." https://promtools.org/prom-6-12/
-
[13]
Zeebe YAML workflows
Camunda, “Zeebe YAML workflows ”, [Online]. Available: https://unsupported.docs.ca- munda.io/0.25/docs/components/zeebe/yaml-workflows/. LLM-Based Vulnerability Discovery in Business Process Documentation 15
-
[14]
Process Model Generation from Natural Lan- guage Text
Friedrich F., Mendling J., and Puhlmann F. “Process Model Generation from Natural Lan- guage Text.” In Int’l. Conference on Advanced Information Systems Engineering, 2011
2011
-
[15]
A Concept for Generating Business Process Models from Natural Language Description
Honkisz K., Kluza K., and Wiśniewski P., “A Concept for Generating Business Process Models from Natural Language Description.” In 11th International Conference on Knowledge Science, Engineering and Management , KSEM 2018, Changchun, China, Au- gust 17–19, 2018
2018
-
[16]
A Machine Translation Like Approach to Generate Business Process Model from Textual Description
Sonbol R., Rebdawi G., and Ghneim N. “A Machine Translation Like Approach to Generate Business Process Model from Textual Description.”, SN Computer Science, 4(3), Mar 2023, https://doi.org/10.1007/s42979-023-01742-z
2023 doi
-
[17]
Detecting Data-Flow Errors in BPMN 2.0
Stackelberg S.V., Putze S., Mülle J., Böhm K., “Detecting Data-Flow Errors in BPMN 2.0.” Open Journal of Information Systems, 1(2), (2014): 1-19
2014
- [18]
-
[19]
Petri Net Based Data-Flow Error Detection and Correction Strategy for Busi- ness Processes
Liu C. et al., "Petri Net Based Data-Flow Error Detection and Correction Strategy for Busi- ness Processes" , IEEE Access, vol. 8, pp. 43265 - 43276, 2020, doi: 10.1109 / ACCESS.2020.2976124
2020
-
[20]
Applying Cognitive Principles to Model-Finding Output: The Positive Value of Negative Information
Dyer T., Nelson T., Fisler K., and Krishnamurthi S., “Applying Cognitive Principles to Model-Finding Output: The Positive Value of Negative Information.” ACM SIGPLAN con- ference on Object-Oriented Programming Systems, Languages & Applications, 2022
2022
-
[21]
Business Process Model Notation 2.0,
Object Management Group, "Business Process Model Notation 2.0," [Online]. Available: http://www.omg.org/spec/BPMN/2.0/
-
[22]
Extract, define, canonicalize: An LLM -based framework for knowledge graph construction
Zhang, B., & Soh, H. “Extract, define, canonicalize: An LLM -based framework for knowledge graph construction”. Proc. Empirical Methods in Natural Language Processing (2024)
2024
-
[23]
Language models are few -shot learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., “Language models are few -shot learners”. Ad- vances in Neural Information Processing Systems, 33, 1877–1901, 2020. Peraton RITM #: RITM0533635
1901
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.