Pith. sign in

REVIEW 4 major objections 7 minor 26 references

Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A retrieval-augmented general-purpose LLM detects undesired process behavior—inserts, skips, repeats, replacements, and swaps—better than fine-tuned LLMs.

desk verdict Useful first RAG-for-deviation-detection paper, but the ablation undercuts the central claim: on the two large synthetic corpora, log-context-only is nearly as good, so the RAG component's contribution is only demonstrated on the tiny P2P log. read the letter →

arxiv 2505.22041 v1 pith:KU7MLA44 submitted 2025-05-28 cs.LG

classification cs.LG
keywords processminingconformancecheckingretrievalaugmentedgenerationlargelanguagemodelsdeviationpatternseventlogsanomalydetectionsemantic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conformance checking normally needs a process model of the desired behavior, and model-less alternatives either rely on statistical infrequency or on fine-tuning large language models, which is costly and often fails to generalize. This paper proposes a third path: retrieve labeled examples of desired and undesired behavior from other processes and hand them to a general-purpose LLM along with context from the event log itself. The central claim is that this retrieval-augmented setup detects insertions, skips, repetitions, replacements, and swaps more accurately than fine-tuned LLMs, at a fraction of the training cost. If true, organizations without up-to-date process models can still locate deviations in their event logs cheaply.

What carries the argument

The engine is a retrieval augmented generation (RAG) pipeline, which supplies a general-purpose LLM with task-related knowledge at inference time instead of baking it in through fine-tuning. An offline component generates conforming and deviating traces from process models, embeds them with a pre-trained sentence embedding model, and stores them in a knowledge base. An online component embeds each incoming trace, retrieves the most similar traces by cosine similarity, and fills a prompt template with the retrieved examples, the trace, and log-derived context such as frequent traces and activities. A parser then forces the LLM's output into a structured form representing the five deviation patterns.

What would settle it

Replace the retrieved examples in the prompt with random traces from the knowledge base on the same evaluation logs; if detection F1 does not drop substantially, the retrieval mechanism is not carrying the load.

Watch

Extended reading notes

Core claim

The paper claims that retrieval augmented generation can replace resource-intensive fine-tuning for detecting undesired process behavior. The approach builds a knowledge base from a large process model collection by generating desired traces and traces deliberately contaminated with five known deviation patterns, embedding each trace, and storing the known deviations alongside it. For each trace in a new event log, it retrieves the most similar knowledge-base traces by cosine similarity and prompts a general-purpose LLM with the task description, the trace itself, frequently occurring activities and traces from the event log, and the retrieved labeled examples. In the reported experiments, this outperforms fine-tuned anomaly detectors as well as vanilla LLM prompting, with the retrieved examples mattering most on complex, manually crafted deviations.

Load-bearing premise

The whole method rests on the assumption that a general LLM, shown a few retrieved example traces from other processes, can correctly transfer which behaviors are undesired to the new process at hand.

Editorial extensions

If this is right

  • Organizations can detect deviations in event logs without maintaining a dedicated process model of the target process.
  • RAG with explicit log context provides a computationally cheap alternative to fine-tuning, cutting training from hundreds of GPU-hours to roughly an hour of offline knowledge-base population.
  • The approach locates complex undesired fragments spanning any number of activities, not just pairs of activities as in constraint-checking baselines.
  • Log context alone already helps detect missing activities, and retrieval adds the largest benefit on more realistic, manually crafted deviations.
  • The same pipeline can be applied to other process mining tasks that need knowledge transfer from similar processes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's success hinges on embedding cosine similarity selecting genuinely relevant examples, so a better retrieval metric or domain-aware embeddings could further improve transfer and perhaps fix the weak detection of swaps.
  • The paper hints that combining fine-tuning with RAG may outperform either alone; a natural extension would be testing that combination on the same deviation-pattern task.
  • Retrieval-based knowledge transfer could generalize to process improvement, predictive monitoring, or root-cause explanation, where labeled examples from similar processes are also scarce.
  • Because swaps remain the hardest pattern and fine-tuning helps there, ordering information may need explicit contrastive examples or a specialized prompt structure to be learned reliably.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a RAG-based approach to detect undesired process behavior in event logs without a process model or fine-tuning. An offline component builds a knowledge base from a collection of process models by generating conforming and deviating traces and embedding them; an online component retrieves similar traces from the knowledge base, combines them with frequent-trace/activity log context, and prompts a general-purpose LLM to output deviations. The evaluation compares against fine-tuned baselines (xSemAD, DABL) and vanilla LLM baselines on three test collections: SAP-SAM-test, PROP, and a small P2P log, reporting precision/recall/F1 per pattern type, plus a qualitative analysis on BPI Challenge 2019.

Significance. If the approach's performance holds, it would be a valuable contribution: a model-free, fine-tuning-free conformance-checking alternative that is more computationally efficient than fine-tuning and can handle multi-activity deviations. The paper provides a public repository, a rigorous hyperparameter optimization, and a comparison with multiple baselines, which are strengths. However, the evidence that RAG—rather than the embedded log context—is the source of improvement is limited, and the evaluation has methodological weaknesses (no error bars on main results, synthetic test logs generated with the same procedure as the knowledge base, lenient scoring). These issues need to be addressed before the contribution can be fully assessed.

major comments (4)
  1. [Section 4.3, Table 2] The central claim of the paper is that RAG—retrieving labeled examples from other processes—enables accurate deviation detection. The ablation in Section 4.3, Table 2 does not support this attribution. On SAP-SAM-test, the Log Context baseline without any retrieved examples achieves F1 scores of 0.79/0.39/0.80/0.51/0.19/0.90 for inserted/skipped/repeated/replaced/swapped/conforming, while the full approach achieves 0.79/0.38/0.83/0.52/0.23/0.91; the differences are at most 0.04 and the full approach is worse on skipped. On PROP, the full approach is clearly better only for inserted (0.79 vs 0.64) and is worse for skipped (0.36 vs 0.39). The only dataset where RAG provides a substantial improvement is P2P, a manually curated log with 58 traces. Because the abstract's headline conclusion attributes the benefit to RAG, the evaluation cannot distinguish a RAG effect from a log-context/prompt-engineering effect. Please provide stronger evidence for the RAG contribution (e.g., realistic multi-process logs, retrieval-quality analysis) or reframe the contribution as a log-context-augmented LLM prompting approach.
  2. [Section 4.3, Tables 2 and 3] The main results are reported as point estimates with no error bars or statistical tests. Only the P2P log, which contains 58 traces, has standard deviations, and these are based on only three seeds; they show non-negligible variability (e.g., inserted recall ±0.05 for the proposed approach and inserted precision ±0.24 for the Log Context baseline). Given that the differences between Log Context and the full approach on SAP-SAM are ≤0.04 F1, these differences are likely within run-to-run variance. Please report variance across the individual logs or repeated runs, and assess the statistical significance of the observed differences.
  3. [Section 4.1 and Section 3.2] The synthetic test logs for SAP-SAM-test and PROP are generated with the same deviation-generation procedure as the knowledge base: the same five pattern types, the same random selection of fragments, the same retry-and-validate logic, and the same share of deviating traces. This means that the retrieval examples and the test traces come from the same synthetic distribution, which may make the retrieval task easier than it would be on real-world deviations. The paper does not discuss this as a potential source of optimistic bias. The P2P log is the only realistic test collection, and it is very small. Please add an explicit discussion of this limitation and, if possible, validate on data generated with a different procedure or on additional realistic logs.
  4. [Section 4.2, Metrics] The scoring rule grants partial credit on the activity level and accepts alternative pattern interpretations (e.g., a repetition as an insertion, a replacement as a skip plus insertion). This is a defensible activity-level evaluation, but it likely inflates absolute F1 scores. More importantly, the comparison with xSemAD and DABL relies on converting their outputs to insertions and skips, even though these approaches were not designed to produce such patterns; this conversion may not reflect their intended semantics. Please report results under a stricter exact-match scoring as a sensitivity analysis, and clarify whether the relative ranking of approaches is robust to the scoring rule.
minor comments (7)
  1. [Section 3.1, notation] The symbol A is used both for the universe of activities and for the set of activities of a process model (A ⊆ A); please use distinct symbols to avoid confusion.
  2. [Section 4.2, hyperparameter optimization] The text states that the best hyperparameters are highlighted in bold, but the selected values are not explicitly listed. Please state the chosen values (e.g., 3 frequent traces, 5 retrieved examples, 10% relative frequency) in the text.
  3. [Section 4.2, Metrics] The description of partial FPs and FNs is incomplete; please specify how partial credit is assigned when a detected deviation overlaps only partially with a ground-truth deviation.
  4. [Table 2] The visual grouping of the SAP-SAM-test and PROP panels is unclear because the pattern-type rows are not separated by dataset; consider restructuring the table into separate panels per dataset.
  5. [Section 4.1] Duplicating traces to reach a minimum of 100 traces introduces duplicate entries that may not be independent; please discuss the potential impact on the evaluation.
  6. [Section 5, limitations] The phrase 'to an extend' should be 'to an extent'.
  7. [Section 5, conclusion] The statement 'RAG improved the results, especially in more complex processes' is stronger than what Table 2 supports; only the P2P log shows a clear improvement. Please reword to be more precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical pipeline is self-contained, and the limited RAG benefit over log context is a support weakness, not a definitional or self-citation circularity.

full rationale

The paper contains no formal derivation chain whose output is equivalent to its inputs by construction. The knowledge base is populated from SAP-SAM-train, while the main test logs are generated from the disjoint SAP-SAM-test split plus two external collections (PROP and P2P), so the retrieval examples are not drawn from the test processes. Hyperparameters were selected on 100 held-out process models, not on the test sets, and no fitted parameter is relabeled as a prediction. The RAG component is explicitly ablated by the Log Context baseline, which removes retrieved examples; this baseline nearly matches the full approach on most synthetic patterns. That is an empirical weakness in attributing gains to RAG, not circularity, because the compared conditions differ in a real system component and the outcome is not forced by construction. The stated transfer assumption ('assuming that the LLM can transfer this knowledge to the process at hand') is an openly declared limitation rather than a hidden input. Self-citations (e.g., refs. [13], [25]) appear only as contextual references to the authors' prior conformance-checking work and the SAP-SAM dataset; they do not carry the central claim. No uniqueness theorem or externally imported result is used to exclude alternatives. The limitations section also acknowledges that findings on generated logs and the manually designed P2P log may not generalize, which further shows the evaluation is presented as evidence, not as a self-referential proof.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. Its contribution rests on several evaluative assumptions: that synthetic deviations resemble real deviations, that embedding similarity matches transferability, and that frequent log behavior proxies desired behavior. Free parameters are mainly prompt and data-generation choices tuned or set by hand.

free parameters (7)
  • Number of most frequent traces included in prompt = not visible (tuned via grid search over [0,1,2,3,4,5])
    Tuned on a validation set of 100 process models; directly influences the prompt content and the LLM's reference for desired behavior.
  • Number of retrieved relevant trace examples = not visible (tuned via grid search over [0,3,5,8,10,15])
    Controls how many knowledge base traces with known deviations are inserted into the prompt; central to the RAG mechanism.
  • Relative frequency threshold for activities in log context = not visible (tuned via grid search over [0%,10%,20%])
    Determines which frequent activities are reported to the LLM; affects the log context part of the prompt.
  • Share of deviating traces in synthetic log generation = 55%
    Hand-chosen to create balanced data; affects both knowledge base and test log composition.
  • Maximum number of deviations per trace = 3
    Hand-chosen; defines the synthetic ground truth complexity.
  • Length range of affected fragments = 1 to 3 activities
    Hand-chosen; constrains the synthetic deviation patterns.
  • Retry limit for valid deviation generation = 10
    Hand-chosen to guarantee termination when selecting deviations that are actual deviations per the model.
assumptions (5)
  • domain assumption An LLM can transfer knowledge of undesired behavior from retrieved traces of other processes to a previously unseen process.
    Stated in the abstract and Section 3.1; the core premise of the RAG approach. If false, the method cannot outperform fine-tuning.
  • domain assumption Frequent activities and frequent traces in an event log indicate desired behavior.
    Stated in Section 3.3 and acknowledged in Section 5 as an assumption that might not always hold; used as log context in the prompt.
  • domain assumption Synthetic traces generated by adding random insertions, skips, repetitions, replacements, and swaps to model executions are representative of real undesired process behavior.
    Used in both knowledge base population (Section 4.2) and test log generation (Section 4.1). The alignment between generation procedure and evaluation procedure is a potential source of bias.
  • domain assumption Embedding similarity via cosine similarity in Nomic Embed space corresponds to relevance for deviation transfer.
    Section 3.3 uses cosine similarity to retrieve similar traces; no validation that embedding proximity predicts transferability of deviations.
  • domain assumption The five deviation patterns (inserted, skipped, repeated, replaced, swapped) constitute a complete and meaningful categorization of undesired control-flow.
    Adopted from Hosseinpour and Jans [16]; the paper does not justify completeness beyond citing the source.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation." pith.science (2026). https://pith.science/paper/KU7MLA44

@misc{pith2026250522041,
  author       = {Pith},
  title        = {Pith review of: Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KU7MLA44}},
  note         = {Machine review of arXiv:2505.22041}
}
read the original abstract

Conformance checking techniques detect undesired process behavior by comparing process executions that are recorded in event logs to desired behavior that is captured in a dedicated process model. If such models are not available, conformance checking techniques are not applicable, but organizations might still be interested in detecting undesired behavior in their processes. To enable this, existing approaches use Large Language Models (LLMs), assuming that they can learn to distinguish desired from undesired behavior through fine-tuning. However, fine-tuning is highly resource-intensive and the fine-tuned LLMs often do not generalize well. To address these limitations, we propose an approach that requires neither a dedicated process model nor resource-intensive fine-tuning to detect undesired process behavior. Instead, we use Retrieval Augmented Generation (RAG) to provide an LLM with direct access to a knowledge base that contains both desired and undesired process behavior from other processes, assuming that the LLM can transfer this knowledge to the process at hand. Our evaluation shows that our approach outperforms fine-tuned LLMs in detecting undesired behavior, demonstrating that RAG is a viable alternative to resource-intensive fine-tuning, particularly when enriched with relevant context from the event log, such as frequent traces and activities.

Figures

Figures reproduced from arXiv: 2505.22041 by the authors.

Figure 1
Figure 1. Overview of our approach for detecting undesired behavior [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Excerpt from prompt tem￾plate. Boldness indicates dynamic parts that are adjusted per trace. Detection. The second step of the online compo￾nent detects undesired behavior in t by feeding the prompt into an LLM and parsing the output into a structured format. LLM. We use a general-purpose LLM, designed to generate accurate responses to a wide range of questions. This LLM is not fine-tuned for the specific task at ha… view at source ↗
Figure 3
Figure 3. Exemplary parsed output [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

  1. [1]

    Inf Syst 102, 101824 (2021)

    van der Aa, H., Rebmann, A., Leopold, H.: Natural language-based detection of semantic execution anomalies in event logs. Inf Syst 102, 101824 (2021)

  2. [2]

    SoSyM 11(4), 557–569 (2012)

    van der Aalst, W.M.: What makes a good process model? lessons learned from process min- ing. SoSyM 11(4), 557–569 (2012)

  3. [3]

    J Intell Inf Syst pp

    Bernardi, M.L., Casciani, A., Cimitile, M., Marrella, A.: Conversing with business process- aware large language models: the BPLLM framework. J Intell Inf Syst pp. 1–23 (2024)

  4. [4]

    Böhmer, K., Rinderle-Ma, S.: Multi instance anomaly detection in business process execu- tions. In: BPM. pp. 77–93. Springer (2017) Detecting Undesired Behavior with RAG 17

  5. [5]

    Inf Syst 90, 101438 (2020)

    Böhmer, K., Rinderle-Ma, S.: Mining association rules for anomaly detection in dynamic process runtime behavior and explaining the root cause to users. Inf Syst 90, 101438 (2020)

  6. [6]

    Busch, K., Kampik, T., Leopold, H.: xsemad: Explainable semantic anomaly detection in event logs using sequence-to-sequence models. In: BPM. pp. 309–327. Springer (2024)

  7. [7]

    Springer (2018)

    Carmona, J., van Dongen, B., Solti, A., Weidlich, M.: Conformance checking. Springer (2018)

  8. [8]

    Caspary, J., Rebmann, A., van der Aa, H.: Does this make sense? machine learning-based de- tection of semantic anomalies in business processes. In: BPM. pp. 163–179. Springer (2023)

Show all 26 references
  1. [9]

    In: S-BPM ONE

    Dunzer, S., Stierle, M., Matzner, M., Baier, S.: Conformance checking: A state-of-the-art literature review. In: S-BPM ONE. p. 1–10. ACM (2019)

  2. [10]

    In: BPMDS

    Estrada-Torres, B., del Río-Ortega, A., Resinas, M.: Mapping the landscape: Exploring large language model applications in business process management. In: BPMDS. pp. 22–

  3. [11]

    In: CAISE

    Friedrich, F., Mendling, J., Puhlmann, F.: Process model generation from natural language text. In: CAISE. pp. 482–496. Springer (2011)

  4. [12]

    Trans Softw Eng 44(3), 262–290 (2017)

    García-Bañuelos, L., Van Beest, N., Dumas, M., La Rosa, M., Mertens, W.: Complete and interpretable conformance checking of business processes. Trans Softw Eng 44(3), 262–290 (2017)

  5. [13]

    Grohs, M., van der Aa, H., Rehse, J.R.: Beyond log and model moves in conformance check- ing: Discovering process-level deviation patterns. In: BPM. pp. 381–399. Springer (2024)

  6. [14]

    Inf Sys 127, 102461 (2025), https://www

    Grohs, M., Pfeiffer, P., Rehse, J.R.: Proactive conformance checking: An approach for predicting deviations in business processes. Inf Sys 127, 102461 (2025), https://www. sciencedirect.com/science/article/pii/S0306437924001194

  7. [15]

    Guan, W., Cao, J., Gao, J., Zhao, H., Qian, S.: Dabl: Detecting semantic anomalies in busi- ness processes using large language models (2024), https://arxiv.org/abs/2406.15781

  8. [16]

    J Inf Sys 38(1), 67–89 (2024)

    Hosseinpour, M., Jans, M.: Auditors’ categorization of process deviations. J Inf Sys 38(1), 67–89 (2024)

  9. [17]

    In: ICPM Workshops

    Lahann, J., Pfeiffer, P., Fettke, P.: Lstm-based anomaly detection of process instances: bench- mark and tweaks. In: ICPM Workshops. pp. 229–241. Springer (2022)

  10. [18]

    de Leoni, M., Maggi, F.M., van der Aalst, W.M.P.: Aligning event logs and declarative pro- cess models for conformance checking. In: BPM. pp. 82–97. Springer (2012)

  11. [19]

    Adv Neural Inf Process Sys 33, 9459–9474 (2020)

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V ., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv Neural Inf Process Sys 33, 9459–9474 (2020)

  12. [20]

    In: International Conference on Business Process Management

    Monti, F., Leotta, F., Mangler, J., Mecella, M., Rinderle-Ma, S.: Nl2processops: towards llm-guided code generation for process execution. In: International Conference on Business Process Management. pp. 127–143. Springer (2024)

  13. [21]

    Springer (2016)

    Munoz-Gama, J.: Conformance Checking and Diagnosis in Process Mining: Comparing Ob- served and Modeled Processes. Springer (2016)

  14. [22]

    Inf Syst 103, 101458 (2019)

    Nolle, T., Luettgen, S., Seeliger, A., Mühlhäuser, M.: Binet: Multi-perspective business pro- cess anomaly classification. Inf Syst 103, 101458 (2019)

  15. [23]

    Nussbaum, Z., Morris, J.X., Duderstadt, B., Mulyar, A.: Nomic embed: Training a repro- ducible long context text embedder (2024), https://arxiv.org/abs/2402.01613

  16. [24]

    In: ICPM

    Rebmann, A., Schmidt, F.D., Glavaš, G., van Der Aa, H.: Evaluating the ability of llms to solve semantics-aware process mining tasks. In: ICPM. pp. 9–16 (2024)

  17. [25]

    In: ICPM Workshops

    Sola, D., Warmuth, C., Schäfer, B., Badakhshan, P., Rehse, J.R., Kampik, T.: SAP Sig- navio Academic Models: a large process model dataset. In: ICPM Workshops. pp. 453–465. Springer (2022)

  18. [26]

    https://doi.org/10.34740/KAGGLE/M/3301, https://www

    Team, G.: Gemma2 (2024). https://doi.org/10.34740/KAGGLE/M/3301, https://www. kaggle.com/m/3301

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.