REVIEW 4 major objections 7 minor 26 references
Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation
T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A retrieval-augmented general-purpose LLM detects undesired process behavior—inserts, skips, repeats, replacements, and swaps—better than fine-tuned LLMs.
desk verdict Useful first RAG-for-deviation-detection paper, but the ablation undercuts the central claim: on the two large synthetic corpora, log-context-only is nearly as good, so the RAG component's contribution is only demonstrated on the tiny P2P log. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is a retrieval augmented generation (RAG) pipeline, which supplies a general-purpose LLM with task-related knowledge at inference time instead of baking it in through fine-tuning. An offline component generates conforming and deviating traces from process models, embeds them with a pre-trained sentence embedding model, and stores them in a knowledge base. An online component embeds each incoming trace, retrieves the most similar traces by cosine similarity, and fills a prompt template with the retrieved examples, the trace, and log-derived context such as frequent traces and activities. A parser then forces the LLM's output into a structured form representing the five deviation patterns.
What would settle it
Replace the retrieved examples in the prompt with random traces from the knowledge base on the same evaluation logs; if detection F1 does not drop substantially, the retrieval mechanism is not carrying the load.
Extended reading notes
Core claim
The paper claims that retrieval augmented generation can replace resource-intensive fine-tuning for detecting undesired process behavior. The approach builds a knowledge base from a large process model collection by generating desired traces and traces deliberately contaminated with five known deviation patterns, embedding each trace, and storing the known deviations alongside it. For each trace in a new event log, it retrieves the most similar knowledge-base traces by cosine similarity and prompts a general-purpose LLM with the task description, the trace itself, frequently occurring activities and traces from the event log, and the retrieved labeled examples. In the reported experiments, this outperforms fine-tuned anomaly detectors as well as vanilla LLM prompting, with the retrieved examples mattering most on complex, manually crafted deviations.
Load-bearing premise
The whole method rests on the assumption that a general LLM, shown a few retrieved example traces from other processes, can correctly transfer which behaviors are undesired to the new process at hand.
Editorial extensions
If this is right
- Organizations can detect deviations in event logs without maintaining a dedicated process model of the target process.
- RAG with explicit log context provides a computationally cheap alternative to fine-tuning, cutting training from hundreds of GPU-hours to roughly an hour of offline knowledge-base population.
- The approach locates complex undesired fragments spanning any number of activities, not just pairs of activities as in constraint-checking baselines.
- Log context alone already helps detect missing activities, and retrieval adds the largest benefit on more realistic, manually crafted deviations.
- The same pipeline can be applied to other process mining tasks that need knowledge transfer from similar processes.
Reading between the lines
- The method's success hinges on embedding cosine similarity selecting genuinely relevant examples, so a better retrieval metric or domain-aware embeddings could further improve transfer and perhaps fix the weak detection of swaps.
- The paper hints that combining fine-tuning with RAG may outperform either alone; a natural extension would be testing that combination on the same deviation-pattern task.
- Retrieval-based knowledge transfer could generalize to process improvement, predictive monitoring, or root-cause explanation, where labeled examples from similar processes are also scarce.
- Because swaps remain the hardest pattern and fine-tuning helps there, ordering information may need explicit contrastive examples or a specialized prompt structure to be learned reliably.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a RAG-based approach to detect undesired process behavior in event logs without a process model or fine-tuning. An offline component builds a knowledge base from a collection of process models by generating conforming and deviating traces and embedding them; an online component retrieves similar traces from the knowledge base, combines them with frequent-trace/activity log context, and prompts a general-purpose LLM to output deviations. The evaluation compares against fine-tuned baselines (xSemAD, DABL) and vanilla LLM baselines on three test collections: SAP-SAM-test, PROP, and a small P2P log, reporting precision/recall/F1 per pattern type, plus a qualitative analysis on BPI Challenge 2019.
Significance. If the approach's performance holds, it would be a valuable contribution: a model-free, fine-tuning-free conformance-checking alternative that is more computationally efficient than fine-tuning and can handle multi-activity deviations. The paper provides a public repository, a rigorous hyperparameter optimization, and a comparison with multiple baselines, which are strengths. However, the evidence that RAG—rather than the embedded log context—is the source of improvement is limited, and the evaluation has methodological weaknesses (no error bars on main results, synthetic test logs generated with the same procedure as the knowledge base, lenient scoring). These issues need to be addressed before the contribution can be fully assessed.
major comments (4)
- [Section 4.3, Table 2] The central claim of the paper is that RAG—retrieving labeled examples from other processes—enables accurate deviation detection. The ablation in Section 4.3, Table 2 does not support this attribution. On SAP-SAM-test, the Log Context baseline without any retrieved examples achieves F1 scores of 0.79/0.39/0.80/0.51/0.19/0.90 for inserted/skipped/repeated/replaced/swapped/conforming, while the full approach achieves 0.79/0.38/0.83/0.52/0.23/0.91; the differences are at most 0.04 and the full approach is worse on skipped. On PROP, the full approach is clearly better only for inserted (0.79 vs 0.64) and is worse for skipped (0.36 vs 0.39). The only dataset where RAG provides a substantial improvement is P2P, a manually curated log with 58 traces. Because the abstract's headline conclusion attributes the benefit to RAG, the evaluation cannot distinguish a RAG effect from a log-context/prompt-engineering effect. Please provide stronger evidence for the RAG contribution (e.g., realistic multi-process logs, retrieval-quality analysis) or reframe the contribution as a log-context-augmented LLM prompting approach.
- [Section 4.3, Tables 2 and 3] The main results are reported as point estimates with no error bars or statistical tests. Only the P2P log, which contains 58 traces, has standard deviations, and these are based on only three seeds; they show non-negligible variability (e.g., inserted recall ±0.05 for the proposed approach and inserted precision ±0.24 for the Log Context baseline). Given that the differences between Log Context and the full approach on SAP-SAM are ≤0.04 F1, these differences are likely within run-to-run variance. Please report variance across the individual logs or repeated runs, and assess the statistical significance of the observed differences.
- [Section 4.1 and Section 3.2] The synthetic test logs for SAP-SAM-test and PROP are generated with the same deviation-generation procedure as the knowledge base: the same five pattern types, the same random selection of fragments, the same retry-and-validate logic, and the same share of deviating traces. This means that the retrieval examples and the test traces come from the same synthetic distribution, which may make the retrieval task easier than it would be on real-world deviations. The paper does not discuss this as a potential source of optimistic bias. The P2P log is the only realistic test collection, and it is very small. Please add an explicit discussion of this limitation and, if possible, validate on data generated with a different procedure or on additional realistic logs.
- [Section 4.2, Metrics] The scoring rule grants partial credit on the activity level and accepts alternative pattern interpretations (e.g., a repetition as an insertion, a replacement as a skip plus insertion). This is a defensible activity-level evaluation, but it likely inflates absolute F1 scores. More importantly, the comparison with xSemAD and DABL relies on converting their outputs to insertions and skips, even though these approaches were not designed to produce such patterns; this conversion may not reflect their intended semantics. Please report results under a stricter exact-match scoring as a sensitivity analysis, and clarify whether the relative ranking of approaches is robust to the scoring rule.
minor comments (7)
- [Section 3.1, notation] The symbol A is used both for the universe of activities and for the set of activities of a process model (A ⊆ A); please use distinct symbols to avoid confusion.
- [Section 4.2, hyperparameter optimization] The text states that the best hyperparameters are highlighted in bold, but the selected values are not explicitly listed. Please state the chosen values (e.g., 3 frequent traces, 5 retrieved examples, 10% relative frequency) in the text.
- [Section 4.2, Metrics] The description of partial FPs and FNs is incomplete; please specify how partial credit is assigned when a detected deviation overlaps only partially with a ground-truth deviation.
- [Table 2] The visual grouping of the SAP-SAM-test and PROP panels is unclear because the pattern-type rows are not separated by dataset; consider restructuring the table into separate panels per dataset.
- [Section 4.1] Duplicating traces to reach a minimum of 100 traces introduces duplicate entries that may not be independent; please discuss the potential impact on the evaluation.
- [Section 5, limitations] The phrase 'to an extend' should be 'to an extent'.
- [Section 5, conclusion] The statement 'RAG improved the results, especially in more complex processes' is stronger than what Table 2 supports; only the P2P log shows a clear improvement. Please reword to be more precise.
Circularity Check
No significant circularity: the empirical pipeline is self-contained, and the limited RAG benefit over log context is a support weakness, not a definitional or self-citation circularity.
full rationale
The paper contains no formal derivation chain whose output is equivalent to its inputs by construction. The knowledge base is populated from SAP-SAM-train, while the main test logs are generated from the disjoint SAP-SAM-test split plus two external collections (PROP and P2P), so the retrieval examples are not drawn from the test processes. Hyperparameters were selected on 100 held-out process models, not on the test sets, and no fitted parameter is relabeled as a prediction. The RAG component is explicitly ablated by the Log Context baseline, which removes retrieved examples; this baseline nearly matches the full approach on most synthetic patterns. That is an empirical weakness in attributing gains to RAG, not circularity, because the compared conditions differ in a real system component and the outcome is not forced by construction. The stated transfer assumption ('assuming that the LLM can transfer this knowledge to the process at hand') is an openly declared limitation rather than a hidden input. Self-citations (e.g., refs. [13], [25]) appear only as contextual references to the authors' prior conformance-checking work and the SAP-SAM dataset; they do not carry the central claim. No uniqueness theorem or externally imported result is used to exclude alternatives. The limitations section also acknowledges that findings on generated logs and the manually designed P2P log may not generalize, which further shows the evaluation is presented as evidence, not as a self-referential proof.
Assumptions & free parameters
free parameters (7)
- Number of most frequent traces included in prompt =
not visible (tuned via grid search over [0,1,2,3,4,5])
- Number of retrieved relevant trace examples =
not visible (tuned via grid search over [0,3,5,8,10,15])
- Relative frequency threshold for activities in log context =
not visible (tuned via grid search over [0%,10%,20%])
- Share of deviating traces in synthetic log generation =
55%
- Maximum number of deviations per trace =
3
- Length range of affected fragments =
1 to 3 activities
- Retry limit for valid deviation generation =
10
assumptions (5)
- domain assumption An LLM can transfer knowledge of undesired behavior from retrieved traces of other processes to a previously unseen process.
- domain assumption Frequent activities and frequent traces in an event log indicate desired behavior.
- domain assumption Synthetic traces generated by adding random insertions, skips, repetitions, replacements, and swaps to model executions are representative of real undesired process behavior.
- domain assumption Embedding similarity via cosine similarity in Nomic Embed space corresponds to relevance for deviation transfer.
- domain assumption The five deviation patterns (inserted, skipped, repeated, replaced, swapped) constitute a complete and meaningful categorization of undesired control-flow.
Cite this review
Pith. "Pith review of Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation." pith.science (2026). https://pith.science/paper/KU7MLA44
@misc{pith2026250522041,
author = {Pith},
title = {Pith review of: Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/KU7MLA44}},
note = {Machine review of arXiv:2505.22041}
}
read the original abstract
Conformance checking techniques detect undesired process behavior by comparing process executions that are recorded in event logs to desired behavior that is captured in a dedicated process model. If such models are not available, conformance checking techniques are not applicable, but organizations might still be interested in detecting undesired behavior in their processes. To enable this, existing approaches use Large Language Models (LLMs), assuming that they can learn to distinguish desired from undesired behavior through fine-tuning. However, fine-tuning is highly resource-intensive and the fine-tuned LLMs often do not generalize well. To address these limitations, we propose an approach that requires neither a dedicated process model nor resource-intensive fine-tuning to detect undesired process behavior. Instead, we use Retrieval Augmented Generation (RAG) to provide an LLM with direct access to a knowledge base that contains both desired and undesired process behavior from other processes, assuming that the LLM can transfer this knowledge to the process at hand. Our evaluation shows that our approach outperforms fine-tuned LLMs in detecting undesired behavior, demonstrating that RAG is a viable alternative to resource-intensive fine-tuning, particularly when enriched with relevant context from the event log, such as frequent traces and activities.
Figures
Reference graph
Works this paper leans on
-
[1]
van der Aa, H., Rebmann, A., Leopold, H.: Natural language-based detection of semantic execution anomalies in event logs. Inf Syst 102, 101824 (2021)
work page 2021
-
[2]
van der Aalst, W.M.: What makes a good process model? lessons learned from process min- ing. SoSyM 11(4), 557–569 (2012)
work page 2012
-
[3]
Bernardi, M.L., Casciani, A., Cimitile, M., Marrella, A.: Conversing with business process- aware large language models: the BPLLM framework. J Intell Inf Syst pp. 1–23 (2024)
work page 2024
-
[4]
Böhmer, K., Rinderle-Ma, S.: Multi instance anomaly detection in business process execu- tions. In: BPM. pp. 77–93. Springer (2017) Detecting Undesired Behavior with RAG 17
work page 2017
-
[5]
Böhmer, K., Rinderle-Ma, S.: Mining association rules for anomaly detection in dynamic process runtime behavior and explaining the root cause to users. Inf Syst 90, 101438 (2020)
work page 2020
-
[6]
Busch, K., Kampik, T., Leopold, H.: xsemad: Explainable semantic anomaly detection in event logs using sequence-to-sequence models. In: BPM. pp. 309–327. Springer (2024)
work page 2024
-
[7]
Carmona, J., van Dongen, B., Solti, A., Weidlich, M.: Conformance checking. Springer (2018)
work page 2018
-
[8]
Caspary, J., Rebmann, A., van der Aa, H.: Does this make sense? machine learning-based de- tection of semantic anomalies in business processes. In: BPM. pp. 163–179. Springer (2023)
work page 2023
Show all 26 references
-
[9]
In: S-BPM ONE
Dunzer, S., Stierle, M., Matzner, M., Baier, S.: Conformance checking: A state-of-the-art literature review. In: S-BPM ONE. p. 1–10. ACM (2019)
2019
-
[10]
In: BPMDS
Estrada-Torres, B., del Río-Ortega, A., Resinas, M.: Mapping the landscape: Exploring large language model applications in business process management. In: BPMDS. pp. 22–
-
[11]
In: CAISE
Friedrich, F., Mendling, J., Puhlmann, F.: Process model generation from natural language text. In: CAISE. pp. 482–496. Springer (2011)
2011
-
[12]
Trans Softw Eng 44(3), 262–290 (2017)
García-Bañuelos, L., Van Beest, N., Dumas, M., La Rosa, M., Mertens, W.: Complete and interpretable conformance checking of business processes. Trans Softw Eng 44(3), 262–290 (2017)
2017
-
[13]
Grohs, M., van der Aa, H., Rehse, J.R.: Beyond log and model moves in conformance check- ing: Discovering process-level deviation patterns. In: BPM. pp. 381–399. Springer (2024)
2024
-
[14]
Inf Sys 127, 102461 (2025), https://www
Grohs, M., Pfeiffer, P., Rehse, J.R.: Proactive conformance checking: An approach for predicting deviations in business processes. Inf Sys 127, 102461 (2025), https://www. sciencedirect.com/science/article/pii/S0306437924001194
2025
-
[15]
Guan, W., Cao, J., Gao, J., Zhao, H., Qian, S.: Dabl: Detecting semantic anomalies in busi- ness processes using large language models (2024), https://arxiv.org/abs/2406.15781
2024
-
[16]
J Inf Sys 38(1), 67–89 (2024)
Hosseinpour, M., Jans, M.: Auditors’ categorization of process deviations. J Inf Sys 38(1), 67–89 (2024)
2024
-
[17]
In: ICPM Workshops
Lahann, J., Pfeiffer, P., Fettke, P.: Lstm-based anomaly detection of process instances: bench- mark and tweaks. In: ICPM Workshops. pp. 229–241. Springer (2022)
2022
-
[18]
de Leoni, M., Maggi, F.M., van der Aalst, W.M.P.: Aligning event logs and declarative pro- cess models for conformance checking. In: BPM. pp. 82–97. Springer (2012)
2012
-
[19]
Adv Neural Inf Process Sys 33, 9459–9474 (2020)
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V ., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv Neural Inf Process Sys 33, 9459–9474 (2020)
2020
-
[20]
In: International Conference on Business Process Management
Monti, F., Leotta, F., Mangler, J., Mecella, M., Rinderle-Ma, S.: Nl2processops: towards llm-guided code generation for process execution. In: International Conference on Business Process Management. pp. 127–143. Springer (2024)
2024
-
[21]
Springer (2016)
Munoz-Gama, J.: Conformance Checking and Diagnosis in Process Mining: Comparing Ob- served and Modeled Processes. Springer (2016)
2016
-
[22]
Inf Syst 103, 101458 (2019)
Nolle, T., Luettgen, S., Seeliger, A., Mühlhäuser, M.: Binet: Multi-perspective business pro- cess anomaly classification. Inf Syst 103, 101458 (2019)
2019
-
[23]
Nussbaum, Z., Morris, J.X., Duderstadt, B., Mulyar, A.: Nomic embed: Training a repro- ducible long context text embedder (2024), https://arxiv.org/abs/2402.01613
2024 arXiv
-
[24]
In: ICPM
Rebmann, A., Schmidt, F.D., Glavaš, G., van Der Aa, H.: Evaluating the ability of llms to solve semantics-aware process mining tasks. In: ICPM. pp. 9–16 (2024)
2024
-
[25]
In: ICPM Workshops
Sola, D., Warmuth, C., Schäfer, B., Badakhshan, P., Rehse, J.R., Kampik, T.: SAP Sig- navio Academic Models: a large process model dataset. In: ICPM Workshops. pp. 453–465. Springer (2022)
2022
-
[26]
https://doi.org/10.34740/KAGGLE/M/3301, https://www
Team, G.: Gemma2 (2024). https://doi.org/10.34740/KAGGLE/M/3301, https://www. kaggle.com/m/3301
2024 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.