{"id":"e7359c2f-5f27-45c2-b1df-4b0a71e68c3a","arxiv_id":"2505.22041","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A RAG-based approach that supplies an LLM with retrieved process examples and event log context detects undesired control-flow deviations at levels comparable to or better than fine-tuned LLMs, without fine-tuning.","lead":"This paper proposes a retrieval-augmented generation (RAG) approach to detect undesired behavior in business process event logs without a process model or fine-tuned large language model. It claims to outperform fine-tuned LLMs by giving a general-purpose LLM access to similar process examples and frequent traces from the log at hand.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RAG's contribution is not established: on SAP-SAM/PROP, the log-context-only baseline matches the full approach on most patterns; only the small P2P log shows a clear RAG benefit.","rationale":"The reader's CONDITIONAL verdict is well-founded. My stress-test isolates one specific load-bearing weakness: the paper's central claim attributes the performance gain to RAG, but the controlled comparison the authors already include (Log Context baseline) shows that on the two large synthetic test collections, the full approach is within 0.04 F1 of the no-RAG baseline for most patterns and slightly worse for skipped on PROP. The only strong RAG advantage is on the 58-trace P2P log. Thus the retrieval/transfer assumption—identified by the reader as the weakest assumption—is not merely unvalidated; the paper's own results suggest it may play a minor role. The synthetic generation shared between KB and test logs is a related concern, but the decisive issue is attribution: the incremental benefit of retrieval is not demonstrated. The proposed random-retrieval ablation would settle whether similarity-based selection contributes at all. Since the full approach still outperforms the fine-tuned baselines and there is some evidence of transfer on P2P, the appropriate outcome remains conditional acceptance with a request for this analysis (or a reframed contribution).","tokens_in":14672,"tokens_out":8314,"duration_ms":93630,"concrete_test":"Replace the similarity-based retrieval in the online component with randomly sampled knowledge-base traces (keeping the same number and the same desired/undesired label balance), while keeping log context identical, and rerun on SAP-SAM-test, PROP, and P2P. If F1 does not drop significantly relative to similarity-based retrieval on the large sets, the claimed benefit of retrieval is not due to relevant example selection; if it drops only on P2P, the transfer claim rests on a single small log.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RAG—supplying the LLM with retrieved examples from other processes—is what enables accurate deviation detection and outperforms fine-tuning. The paper's own ablation does not support this attribution. Table 2 shows that on SAP-SAM-test, the Log Context baseline (no retrieved examples) achieves F1 of 0.79/0.38/0.80/0.51/0.19/0.90 for inserted/skipped/repeated/replaced/swapped/conforming, while the full approach achieves 0.79/0.38/0.83/0.52/0.23/0.91; the differences are ≤0.04. On PROP, the full approach even scores lower on skipped (0.36 vs 0.39). The only dataset where the full approach substantially beats Log Context is P2P, a manually curated log with 58 traces (e.g., inserted 0.75 vs 0.46, skipped 0.73 vs 0.41). Thus the evidence that retrieved examples transfer knowledge from other processes is confined to a single small log; on the large synthetic test collections the RAG component contributes little or nothing. Because the abstract's headline conclusion attributes the gain to RAG, the current evaluation cannot distinguish a RAG effect from a prompt-engineering/log-context effect. This is load-bearing: if log context alone is sufficient, the paper's novelty is not RAG but context injection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a RAG-based approach to detect undesired process behavior in event logs without a process model or fine-tuning. An offline component builds a knowledge base from a collection of process models by generating conforming and deviating traces and embedding them; an online component retrieves similar traces from the knowledge base, combines them with frequent-trace/activity log context, and prompts a general-purpose LLM to output deviations. The evaluation compares against fine-tuned baselines (xSemAD, DABL) and vanilla LLM baselines on three test collections: SAP-SAM-test, PROP, and a small P2P log, reporting precision/recall/F1 per pattern type, plus a qualitative analysis on BPI Challenge 2019.","tokens_in":15023,"tokens_out":10268,"duration_ms":94561,"significance":"If the approach's performance holds, it would be a valuable contribution: a model-free, fine-tuning-free conformance-checking alternative that is more computationally efficient than fine-tuning and can handle multi-activity deviations. The paper provides a public repository, a rigorous hyperparameter optimization, and a comparison with multiple baselines, which are strengths. However, the evidence that RAG—rather than the embedded log context—is the source of improvement is limited, and the evaluation has methodological weaknesses (no error bars on main results, synthetic test logs generated with the same procedure as the knowledge base, lenient scoring). These issues need to be addressed before the contribution can be fully assessed.","major_comments":[{"comment":"The central claim of the paper is that RAG—retrieving labeled examples from other processes—enables accurate deviation detection. The ablation in Section 4.3, Table 2 does not support this attribution. On SAP-SAM-test, the Log Context baseline without any retrieved examples achieves F1 scores of 0.79/0.39/0.80/0.51/0.19/0.90 for inserted/skipped/repeated/replaced/swapped/conforming, while the full approach achieves 0.79/0.38/0.83/0.52/0.23/0.91; the differences are at most 0.04 and the full approach is worse on skipped. On PROP, the full approach is clearly better only for inserted (0.79 vs 0.64) and is worse for skipped (0.36 vs 0.39). The only dataset where RAG provides a substantial improvement is P2P, a manually curated log with 58 traces. Because the abstract's headline conclusion attributes the benefit to RAG, the evaluation cannot distinguish a RAG effect from a log-context/prompt-engineering effect. Please provide stronger evidence for the RAG contribution (e.g., realistic multi-process logs, retrieval-quality analysis) or reframe the contribution as a log-context-augmented LLM prompting approach.","section":"Section 4.3, Table 2"},{"comment":"The main results are reported as point estimates with no error bars or statistical tests. Only the P2P log, which contains 58 traces, has standard deviations, and these are based on only three seeds; they show non-negligible variability (e.g., inserted recall ±0.05 for the proposed approach and inserted precision ±0.24 for the Log Context baseline). Given that the differences between Log Context and the full approach on SAP-SAM are ≤0.04 F1, these differences are likely within run-to-run variance. Please report variance across the individual logs or repeated runs, and assess the statistical significance of the observed differences.","section":"Section 4.3, Tables 2 and 3"},{"comment":"The synthetic test logs for SAP-SAM-test and PROP are generated with the same deviation-generation procedure as the knowledge base: the same five pattern types, the same random selection of fragments, the same retry-and-validate logic, and the same share of deviating traces. This means that the retrieval examples and the test traces come from the same synthetic distribution, which may make the retrieval task easier than it would be on real-world deviations. The paper does not discuss this as a potential source of optimistic bias. The P2P log is the only realistic test collection, and it is very small. Please add an explicit discussion of this limitation and, if possible, validate on data generated with a different procedure or on additional realistic logs.","section":"Section 4.1 and Section 3.2"},{"comment":"The scoring rule grants partial credit on the activity level and accepts alternative pattern interpretations (e.g., a repetition as an insertion, a replacement as a skip plus insertion). This is a defensible activity-level evaluation, but it likely inflates absolute F1 scores. More importantly, the comparison with xSemAD and DABL relies on converting their outputs to insertions and skips, even though these approaches were not designed to produce such patterns; this conversion may not reflect their intended semantics. Please report results under a stricter exact-match scoring as a sensitivity analysis, and clarify whether the relative ranking of approaches is robust to the scoring rule.","section":"Section 4.2, Metrics"}],"minor_comments":[{"comment":"The symbol A is used both for the universe of activities and for the set of activities of a process model (A ⊆ A); please use distinct symbols to avoid confusion.","section":"Section 3.1, notation"},{"comment":"The text states that the best hyperparameters are highlighted in bold, but the selected values are not explicitly listed. Please state the chosen values (e.g., 3 frequent traces, 5 retrieved examples, 10% relative frequency) in the text.","section":"Section 4.2, hyperparameter optimization"},{"comment":"The description of partial FPs and FNs is incomplete; please specify how partial credit is assigned when a detected deviation overlaps only partially with a ground-truth deviation.","section":"Section 4.2, Metrics"},{"comment":"The visual grouping of the SAP-SAM-test and PROP panels is unclear because the pattern-type rows are not separated by dataset; consider restructuring the table into separate panels per dataset.","section":"Table 2"},{"comment":"Duplicating traces to reach a minimum of 100 traces introduces duplicate entries that may not be independent; please discuss the potential impact on the evaluation.","section":"Section 4.1"},{"comment":"The phrase 'to an extend' should be 'to an extent'.","section":"Section 5, limitations"},{"comment":"The statement 'RAG improved the results, especially in more complex processes' is stronger than what Table 2 supports; only the P2P log shows a clear improvement. Please reword to be more precise.","section":"Section 5, conclusion"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on process mining and LLMs, but the central claim needs to be reined in. The paper is the first to apply RAG to semantic anomaly detection in event logs, and it is properly built: a clear pipeline, five deviation patterns, public code and data, and a comparison against two fine-tuned baselines (xSemAD, DABL) plus two vanilla-LLM controls. On the manually curated P2P log, the full approach beats the log-context-only baseline by large margins on inserted, skipped, repeated, and swapped deviations. That is real evidence that retrieved examples can transfer across processes.\n\nThe soft spot is exactly what the stress-test note identifies. On SAP-SAM-test and PROP, the log-context-only baseline is within 0.04 F1 of the full approach for most patterns; on PROP, the full approach is actually worse on skipped. So the contribution of the RAG component itself is only demonstrated on one small log (58 traces). The abstract's \"outperforms fine-tuned LLMs\" is defensible against DABL and xSemAD, but the \"RAG is a viable alternative\" framing is not fully backed by the ablation. A reader cannot tell from the data whether retrieval is doing the work or whether the frequent-trace and activity context is the active ingredient.\n\nOther weaknesses are secondary but real. No error bars are reported for the main SAP-SAM/PROP results. The metrics grant partial credit and accept alternative pattern interpretations, which inflates scores. And because the knowledge base and the test logs are generated with the same synthetic deviation procedure, retrieval quality may be artificially high; the P2P result partly reassures, but it is one small log.\n\nI would not take the headline at face value, but the paper is honest about limitations, and the experiments are reproducible. The authors noticed the log-context effect themselves and only claim that RAG helps \"especially in more complex processes.\" So this is a solid incremental contribution with an overstated abstract, not a flawed method. A serious referee could push them to reframe the claim and add variance and error analysis for the main results. I would want to see that revision before citing it as evidence for RAG's value in this task. Send it to peer review; the core idea is new enough to deserve referee time.","headline":"Useful first RAG-for-deviation-detection paper, but the ablation undercuts the central claim: on the two large synthetic corpora, log-context-only is nearly as good, so the RAG component's contribution is only demonstrated on the tiny P2P log.","tokens_in":15486,"tokens_out":1778,"would_cite":true,"duration_ms":20471,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrieval-augmented general-purpose LLM detects undesired process behavior—inserts, skips, repeats, replacements, and swaps—better than fine-tuned LLMs.","keywords":["process mining","conformance checking","retrieval augmented generation","large language models","deviation patterns","event logs","anomaly detection","semantic anomaly detection"],"falsifier":"Replace the retrieved examples in the prompt with random traces from the knowledge base on the same evaluation logs; if detection F1 does not drop substantially, the retrieval mechanism is not carrying the load.","tokens_in":1239,"feed_emoji":"🔍","tokens_out":1335,"duration_ms":55277,"temperature":0.7,"pith_summary":"Conformance checking normally needs a process model of the desired behavior, and model-less alternatives either rely on statistical infrequency or on fine-tuning large language models, which is costly and often fails to generalize. This paper proposes a third path: retrieve labeled examples of desired and undesired behavior from other processes and hand them to a general-purpose LLM along with context from the event log itself. The central claim is that this retrieval-augmented setup detects insertions, skips, repetitions, replacements, and swaps more accurately than fine-tuned LLMs, at a fraction of the training cost. If true, organizations without up-to-date process models can still locate deviations in their event logs cheaply.","feed_headline":"Retrieval beats fine-tuning for detecting bad process behavior","feed_subtitle":"A general LLM with retrieved examples catches insertions, skips, repeats, replacements, and swaps without any fine-tuning.","key_machinery":"The engine is a retrieval augmented generation (RAG) pipeline, which supplies a general-purpose LLM with task-related knowledge at inference time instead of baking it in through fine-tuning. An offline component generates conforming and deviating traces from process models, embeds them with a pre-trained sentence embedding model, and stores them in a knowledge base. An online component embeds each incoming trace, retrieves the most similar traces by cosine similarity, and fills a prompt template with the retrieved examples, the trace, and log-derived context such as frequent traces and activities. A parser then forces the LLM's output into a structured form representing the five deviation patterns.","core_discovery":"The paper claims that retrieval augmented generation can replace resource-intensive fine-tuning for detecting undesired process behavior. The approach builds a knowledge base from a large process model collection by generating desired traces and traces deliberately contaminated with five known deviation patterns, embedding each trace, and storing the known deviations alongside it. For each trace in a new event log, it retrieves the most similar knowledge-base traces by cosine similarity and prompts a general-purpose LLM with the task description, the trace itself, frequently occurring activities and traces from the event log, and the retrieved labeled examples. In the reported experiments, this outperforms fine-tuned anomaly detectors as well as vanilla LLM prompting, with the retrieved examples mattering most on complex, manually crafted deviations.","pith_inferences":["The method's success hinges on embedding cosine similarity selecting genuinely relevant examples, so a better retrieval metric or domain-aware embeddings could further improve transfer and perhaps fix the weak detection of swaps.","The paper hints that combining fine-tuning with RAG may outperform either alone; a natural extension would be testing that combination on the same deviation-pattern task.","Retrieval-based knowledge transfer could generalize to process improvement, predictive monitoring, or root-cause explanation, where labeled examples from similar processes are also scarce.","Because swaps remain the hardest pattern and fine-tuning helps there, ordering information may need explicit contrastive examples or a specialized prompt structure to be learned reliably."],"forward_implications":["Organizations can detect deviations in event logs without maintaining a dedicated process model of the target process.","RAG with explicit log context provides a computationally cheap alternative to fine-tuning, cutting training from hundreds of GPU-hours to roughly an hour of offline knowledge-base population.","The approach locates complex undesired fragments spanning any number of activities, not just pairs of activities as in constraint-checking baselines.","Log context alone already helps detect missing activities, and retrieval adds the largest benefit on more realistic, manually crafted deviations.","The same pipeline can be applied to other process mining tasks that need knowledge transfer from similar processes."],"supporting_citations":[{"why":"defines retrieval augmented generation and motivates supplying LLMs with task-related knowledge not seen during training.","marker":"[19]"},{"why":"supplies the five deviation pattern types and the manually crafted purchase-to-pay event log used as test data.","marker":"[16]"},{"why":"provides the pre-trained sentence embedding model used to embed and retrieve traces.","marker":"[23]"},{"why":"supplies the general-purpose LLM that performs detection and output parsing.","marker":"[26]"},{"why":"provides the large collection of process models used to populate the knowledge base and generate synthetic test logs.","marker":"[25]"},{"why":"serves as the fine-tuned sequence-to-sequence baseline for semantic anomaly detection.","marker":"[6]"},{"why":"serves as the fine-tuned LLM baseline that detects the same deviation patterns.","marker":"[15]"}],"fun_headline_variants":["RAG beats fine-tuning for process anomaly detection","No fine-tuning: retrieval catches process deviations","Retrieval-augmented LLM tops fine-tuning on process errors","Process flaws detected without resource-heavy fine-tuning","RAG outscores fine-tuning in spotting process misbehavior"],"cache_read_input_tokens":17664,"weakest_assumption_plain":"The whole method rests on the assumption that a general LLM, shown a few retrieved example traces from other processes, can correctly transfer which behaviors are undesired to the new process at hand.","fun_headline_variants_meta":{"raw":{"variants":["RAG beats fine-tuning for process anomaly detection","No fine-tuning: retrieval catches process deviations","Retrieval-augmented LLM tops fine-tuning on process errors","Process flaws detected without resource-heavy fine-tuning","RAG outscores fine-tuning in spotting process misbehavior"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1453,"prompt_tokens":905,"completion_tokens":548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":472}},"tokens_in":521,"tokens_out":548,"duration_ms":5890,"temperature":1.0,"reasoning_tokens":472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:16:09.299440+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the retrieved examples in the prompt with random traces from the knowledge base on the same evaluation logs; if detection F1 does not drop substantially, the retrieval mechanism is not carrying the load.","supporting_citations":[{"cited_title":"Adv Neural Inf Process Sys 33, 9459–9474 (2020)","cited_arxiv_id":null,"evidence_quote":"defines retrieval augmented generation and motivates supplying LLMs with task-related knowledge not seen during training."},{"cited_title":"J Inf Sys 38(1), 67–89 (2024)","cited_arxiv_id":null,"evidence_quote":"supplies the five deviation pattern types and the manually crafted purchase-to-pay event log used as test data."},{"cited_title":"In: ICPM Workshops","cited_arxiv_id":null,"evidence_quote":"provides the large collection of process models used to populate the knowledge base and generate synthetic test logs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"serves as the fine-tuned sequence-to-sequence baseline for semantic anomaly detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"serves as the fine-tuned LLM baseline that detects the same deviation patterns."}],"review_version":1}