{"id":"e109a838-f629-46f2-bce2-2ac00b988b43","arxiv_id":"2504.21074","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuned Llama-3 and Mistral reach macro F1 0.69 to 0.88 and fitness 0.80 to 0.84 on five new semantics-aware process mining benchmarks, while few-shot in-context learning stays near random.","lead":"This paper defines five semantics-aware process mining tasks that require understanding what activities mean, and publishes benchmark datasets for each. It reports that fine-tuned open-source LLMs (Llama-3, Mistral) clearly outperform in-context learning, with macro F1 scores of 0.69 to 0.88 and fitness scores of 0.80 to 0.84.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-standard validity is the load-bearing assumption: the sap-sam-derived 'true' process models are admitted (Sec. 7.3.1, Sec. 9) to contain semantically incorrect behavior, so the reported F1/fitness scores do not yet establish semantic understanding; a human-validated gold re-scoring is required.","rationale":"The reader's weakest_assumption is correct and is the right center of gravity. I verified it against the text: the gold standard is generated entirely from sap-sam diagrams, the only filter is sound block-structuredness (Sec. 4.1.1), and the authors explicitly acknowledge semantically incorrect models in Sec. 7.3.1 and promise a quality-assured collection in Sec. 9. This is not an external objection; it is the paper's own stated limitation, so it must be weighed in the verdict. An additional construct-validity problem exists for S-NAP: the paper itself observes (Sec. 7.1.3) that with parallel activities the next activity is 'indeterminable... as there are multiple valid options,' yet the gold standard assigns a single next activity from one allowed execution. This means 0.69 macro F1 is not a clean semantic-understanding score. I focus the headline on the gold-standard issue because it affects all five tasks and is directly conceded. The relative ordering of methods is still informative: fine-tuned models beat ICL, random, and RoBERTa on the same labels. But the central claim's absolute wording ('strong performance... across a broad range of process types and industries') requires valid golds. A conditional acceptance with the proposed human-gold re-scoring—exactly the quality-assured corpus the authors already plan—is the appropriate verdict. No change to the reader's CONDITIONAL verdict is needed.","tokens_in":21240,"tokens_out":4361,"duration_ms":47658,"concrete_test":"Manually validate the semantic correctness of the gold model for a stratified random sample of 200 test-set instances (40 per task) by two independent process-mining experts who see only the activity set, not the sap-sam gold; retain only instances where both experts judge the gold semantically plausible, then recompute all FT Llama and FT Mistral scores on that subset. If scores fall materially below the reported 0.79 / 0.88 / 0.69 / 0.80 / 0.83, the absolute claim is an artifact of flawed gold standards; if scores hold, the concern is refuted. Include the paper's own travel-reconciliation and trip-information examples to verify whether semantically reasonable LLM outputs are being penalized by invalid golds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that fine-tuned LLMs 'achieve strong performance' on semantics-aware tasks presupposes that each task's gold standard encodes the correct semantics of the process. That precondition is not secured. The corpus (Sec. 4.1.1) is built from sap-sam BPMN diagrams filtered only for sound block-structured workflow nets; this is a syntactic-soundness filter, not a semantic-quality filter. The paper itself concedes in Sec. 7.3.1 that 'the process models in the employed collection do not always represent semantically correct models,' citing a travel-reconciliation gold tree that makes verification, payment, authorization, archiving, and acceptance all parallel—behavior the authors call 'clearly problematic.' Since every label (T-SAD/A-SAD valid/anomalous, S-NAP next activity, S-DFD/S-PTD gold graphs/trees) is derived from these models, high agreement with the gold can mean learning the (sometimes nonsensical) sap-sam collection rather than understanding business semantics. The Sec. 9 plan to redo evaluation on a 'quality-assured model collection' is an admission that current gold standards are not trustworthy for absolute measurement. Relative results (FT > ICL > random; Llama/Mistral > RoBERTa) may survive a re-run, but the headline absolute scores—F1 0.79/0.88/0.69 and fitness 0.80–0.84—are not yet evidence of semantic correctness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines five semantics-aware process mining tasks (T-SAD, A-SAD, S-NAP, S-DFD, S-PTD), constructs benchmarking datasets from a large corpus of sap-sam BPMN models, and evaluates open-source LLMs (Llama-3-8B, Mistral-7B) under both in-context learning and supervised fine-tuning, with RoBERTa and rule-based/random baselines. The central empirical finding is that ICL performs poorly on these tasks, whereas fine-tuned LLMs achieve macro F1 scores of 0.79 (T-SAD), 0.88 (A-SAD), and 0.69 (S-NAP), and fitness scores of 0.80–0.84 on the two generation tasks, consistently outperforming fine-tuned RoBERTa on classification and exceeding the random baseline on generation.","tokens_in":21509,"tokens_out":3379,"duration_ms":37034,"significance":"If the benchmark gold standards are valid, this is a valuable contribution: it is the first systematic comparison of fine-tuned open LLMs against encoder and rule-based baselines on five well-defined semantics-aware process mining tasks. The paper ships public datasets, training scripts, and evaluation code, uses process-level split to avoid leakage, and reports three-seed variance. The relative ranking (fine-tuning > ICL > random; LLMs > RoBERTa) is robust across tasks and seeds. The main weakness is that the gold standards themselves are asserted, not demonstrated, to be semantically correct; the paper's own admission of problematic models in the corpus undermines the absolute performance claims as evidence of semantic understanding. The results are still useful as a benchmark of agreement with a plausible process-model collection, but the headline claim of 'strong performance' on semantics-aware tasks is not yet secured.","major_comments":[{"comment":"The load-bearing assumption of the evaluation is that the sap-sam-derived gold standards correctly encode the semantics of the underlying processes. The corpus creation filters only for sound block-structured workflow nets, which is a syntactic quality filter; it does not establish semantic correctness. The paper itself states in Section 7.3.1 that 'the process models in the employed collection do not always represent semantically correct models', giving a travel-reconciliation example where the gold tree allows verification, payment, authorization, archiving, and acceptance in parallel. Section 9's plan to redo evaluation on a 'quality-assured model collection' confirms that the current gold standards are not trustworthy for absolute measurement. Since every label and gold graph/tree in T-SAD, A-SAD, S-NAP, S-DFD, and S-PTD is derived from these models, the reported F1 and fitness scores do not by themselves establish semantic understanding. Please either re-score the test set on a human-validated subset of models or provide an analysis showing that the semantic errors in the gold standards do not systematically inflate or deflate the reported scores.","section":"Sections 4.1.1, 7.3.1, and 9"},{"comment":"The in-depth analysis further illustrates the benchmark-validity problem. The paper shows a prescription-fulfillment example where the generated tree differs from the gold tree in the ordering of 'Check insurance coverage' and 'Fill prescription', and a travel process where the LLM's sequential tree is described as 'not ideal either'. These cases demonstrate that a low fitness score is not necessarily a semantic error, and a high fitness score is not necessarily semantic correctness. The discussion of per-process-type and per-industry fitness should therefore be interpreted as agreement with a specific gold collection rather than as evidence of domain-level semantic competence. I recommend reporting results on the subset of models that pass a semantic-quality review and separately quantifying the fraction of gold models with identified semantic problems.","section":"Section 7.3.1 (and Table 5/6)"},{"comment":"The abstract and conclusion state that fine-tuned LLMs 'achieve strong performance' on semantics-aware tasks. Given the gold-standard validity issue, the safe claim is that they achieve strong agreement with the gold-standard models derived from sap-sam. The distinction matters because the paper's own examples show that multiple reasonable models can exist for the same activity set, and the gold standard is not always one of them. I ask that the absolute performance claims be tempered or, ideally, accompanied by a human-evaluation study on a sample of outputs, so that 'strong performance' refers to semantic correctness rather than benchmark agreement.","section":"Sections 7.1.2 and 7.2.2"}],"minor_comments":[{"comment":"The text refers to 'Mistral-2 in its 7 billion parameter version', but the cited model is Mistral-7B-Instruct-v0.2; please correct the model name for reproducibility and consistency.","section":"Section 6, Large Language Models"},{"comment":"The 'Retail' row appears twice with identical values (0.88, 9.03); one of these rows should be removed.","section":"Table 6"},{"comment":"The task abbreviation 'S-DPT' is used instead of 'S-PTD' in the S-PTD dataset description and in the discussion of fine-tuning results; please standardize the abbreviation.","section":"Sections 4.2.2 and 7.2.2"},{"comment":"The text mentions 'S-SPT variations'; this appears to be a typo for 'S-PTD'.","section":"Section 7.2.1"},{"comment":"There are inconsistent spellings of 'RoBERTa' (e.g., 'RoBER T a', 'RobERTa' in Tables 3 and 7 and running text); these should be unified.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well within the scope of the venue and the authors have made the data and code publicly available, which is commendable. The primary risk is benchmark validity rather than novelty or execution. I do not see a conflict-of-interest concern beyond the declared SAP affiliation of the corresponding author; the use of sap-sam is disclosed and appropriate. If the authors can re-score on a quality-assured subset or otherwise quantify the impact of semantically incorrect gold models, the paper could become a strong reference benchmark for semantics-aware process mining."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is one of the more careful LLM-for-process-mining benchmark papers I have seen: five tasks, large gold-standard datasets, model-based splits, three seeds, code and data released, and a consistent relative result that fine-tuning beats ICL and RoBERTa. Second, the load-bearing assumption—that the gold-standard models actually encode correct process semantics—is not secured, and the authors say so themselves in Sections 7.3.1 and 9. That makes the absolute scores weaker evidence than the abstract implies.\n\nWhat is new: the S-DFD and S-PTD generation tasks are genuinely new relative to the ICPM 2024 predecessor, and the systematic fine-tuning comparison across open LLMs is a real step forward. I especially appreciate that they do not use an LLM judge; they score against explicit gold standards with macro F1 and footprint fitness. The data construction is transparent: from sap-sam BPMN diagrams, filtering for sound block-structured workflow nets, deduplicating by activity set. The reproducibility story is strong, and the self-citations point to the actual predecessor paper and to public artifacts, so I do not see a citation problem.\n\nWhere it gets soft. The corpus filter is syntactic, not semantic. A BPMN diagram can be sound and block-structured and still describe nonsense, and the paper's own travel-reconciliation example—all five activities parallel: verify, authorize, archive, accept—is exactly that. Since every label comes from these models, high F1 can mean learning the quirks of sap-sam rather than understanding business semantics. The authors acknowledge this and plan a quality-assured collection, but that plan belongs before the headline claim, not after. S-NAP is also partially ill-posed: with parallel branches, multiple next activities are valid, and the single gold label is an artifact of one execution. They admit this too, and it explains the lower score, but it means the benchmark needs a different evaluation protocol, like top-k or valid-set matching, not just a disclaimer. Finally, there is no human baseline, so we cannot calibrate how good 0.79–0.88 F1 or 0.80–0.84 fitness really is.\n\nNone of this kills the paper. The relative ordering—fine-tuned decoder LLMs over RoBERTa over ICL over random—is consistent and should survive a re-run on cleaned gold standards, and the modest margins (2–6 points over RoBERTa) are probably the honest headline. I would send this to serious peer review, and as an editor I would ask for the quality-assured gold re-scoring and a human baseline before acceptance. In the meantime, it is a useful benchmark paper worth citing for the tasks and datasets, with the semantic-correctness caveat attached.","headline":"A genuinely useful benchmark and fine-tuning study for semantics-aware process mining, with relative results that hold up; the absolute 'semantic understanding' claim outruns the gold-standard validity.","tokens_in":22081,"tokens_out":2233,"would_cite":true,"duration_ms":24303,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned large language models can solve semantics-aware process mining tasks, while off-the-shelf or few-shot LLMs largely cannot.","keywords":["process mining","large language models","semantics-aware process mining","semantic anomaly detection","next activity prediction","process discovery","fine-tuning","in-context learning"],"falsifier":"Recompute the evaluation on a quality-assured subset: take a random sample of test models, have process experts judge whether each gold-standard trace, relation, and process tree is semantically correct, and rerun the fine-tuned models only on instances with expert-verified gold standards. If the fine-tuned LLMs no longer beat the encoder baseline or the random baseline on that subset, the claim that fine-tuning unlocks semantic process understanding would be refuted.","tokens_in":21016,"feed_emoji":"🔄","tokens_out":8862,"duration_ms":86365,"temperature":0.7,"pith_summary":"This paper tries to establish that large language models can carry out process mining tasks that require understanding what activities mean, not just counting how often one activity follows another. The authors define five such tasks: judging whether a whole trace is anomalous, judging whether one activity may eventually follow another, predicting the next activity from a prefix, discovering a directly-follows graph from an activity set, and discovering a process tree from an activity set. They build large benchmark datasets for all five by unfolding a corpus of process models into all allowed execution sequences and deriving gold labels. Their experiments show that the same open LLMs that score near chance with few-shot prompting jump to macro F1 scores of 0.79, 0.88, and 0.69 on the three classification tasks and to fitness scores around 0.80 to 0.84 on the two discovery tasks once fine-tuned, outperforming a fine-tuned encoder-only language model on every classification task.","feed_headline":"Fine-tuned LLMs solve semantic process-mining tasks","feed_subtitle":"With task-specific tuning, open LLMs beat encoder baselines on anomaly detection, next-step prediction, and process discovery.","key_machinery":"The argument is carried by a benchmark-construction pipeline plus a fine-tuning protocol. The corpus starts from a large public collection of English business-process diagrams; each diagram is converted into a sound block-structured workflow net, and each net is unfolded into all its allowed activity sequences, with loops executed at most once. Those sequences are the source of gold labels for the five tasks: trace-level and relation-level semantic anomaly detection, next-activity prediction, directly-follows graph discovery, and process-tree discovery; the same nets are translated into process trees as gold standards for the process-tree task. On the modeling side, the load-bearing mechanism is low-rank adaptation (LoRA) of decoder LLMs: classification is cast as constrained generation over the allowed class tokens, and discovery as free generation of edge lists or tree notation, so both kinds of tasks can be fine-tuned without updating all billions of parameters. Output quality is measured by footprint-based fitness, which compares the behavioral footprints of predicted and gold directly-follows graphs or process trees.","core_discovery":"The central claim is that the process knowledge needed for semantics-aware process mining is already present in large language models but is not accessible through prompting alone; supervised fine-tuning unlocks it. With in-context learning, the models hover at or near a random baseline on trace-level and activity-pair anomaly detection (F1 0.44 to 0.53) and manage only weak next-activity prediction (best F1 0.32), while their discovered graphs and trees score fitness 0.52 to 0.61. After fine-tuning, Llama-3 and Mistral reach macro F1 of 0.79 on trace-level anomaly detection, 0.88 on activity-pair anomaly detection, and 0.69 on semantic next-activity prediction, and fitness 0.80 to 0.81 on directly-follows graph discovery and 0.83 to 0.84 on process-tree discovery. The fine-tuned LLMs outperform fine-tuned RoBERTa on every classification task, and on the discovery tasks they learn the few global ordering constraints of a process more effectively than the many local pairwise relations.","pith_inferences":["Beyond the paper: fine-tuning on all five tasks jointly, as the authors propose as future work, may produce a single multi-task model whose semantic competence transfers across tasks, potentially reducing the data needed for new process mining tasks.","Beyond the paper: the paper's own observation that some gold-standard models are not semantically correct, such as a travel-reconciliation model that allows payment verification after payment approval, means the absolute fitness and F1 numbers likely underestimate LLM capability on a cleaner benchmark; the fine-tuning-versus-prompting gap is the more robust finding.","Beyond the paper: a natural testable extension is to plug a fine-tuned semantic scorer into an existing frequency-based next-activity predictor and check whether rejecting semantically invalid predictions improves accuracy on real logs."],"forward_implications":["A fine-tuned LLM can perform semantic checks without access to historical event logs, since inputs are only activity labels plus a trace, relation, or activity set; this makes the approach usable for processes with scarce or missing logs.","The near-random few-shot results imply that task-specific fine-tuning, not prompt engineering, is the practical condition for deploying LLMs on such process mining tasks.","Because fine-tuned LLMs surpass a fine-tuned encoder on all classification tasks, decoder LLMs should be the default choice for semantics-aware process mining rather than smaller bidirectional encoders.","On process-tree discovery the LLMs learn global process structure well enough to reach fitness 0.83 to 0.84, which suggests that LLM-generated process trees could serve as semantic models for conformance checking against observed traces."],"supporting_citations":[{"why":"Supplies the source collection of process diagrams from which the behavior corpus and all gold standards are derived.","marker":"[17]"},{"why":"Provides the rule-based semantic anomaly detection approach used as a classification baseline.","marker":"[1]"},{"why":"Establishes the encoder-based machine-learning approach that fine-tuned LLMs are compared against on activity-pair anomaly detection.","marker":"[2]"},{"why":"Supplies the formal definitions of process trees and directly-follows footprints on which the tasks and evaluation are built.","marker":"[9]"},{"why":"Provides the translation from block-structured workflow nets to process trees used to build the process-tree gold standard.","marker":"[18]"},{"why":"Defines the RoBERTa encoder baseline that fine-tuned LLMs outperform across all classification tasks.","marker":"[21]"},{"why":"Supplies low-rank adaptation, the parameter-efficient method that makes fine-tuning of the 7 to 8 billion parameter decoder LLMs feasible.","marker":"[27]"},{"why":"Defines footprint-based fitness, the conformance measure used to score both process discovery tasks.","marker":"[28]"}],"fun_headline_variants":["Fine-tuned LLMs crack semantic process mining","LLM fine-tuning unlocks process mining smarts","Semantics-aware process mining: fine-tuning is key","Fine-tuned LLMs outdo encoders on process tasks","Prompting alone fails; fine-tuning wins process tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation assumes the gold-standard process models created from the source diagrams correctly encode the semantics of the underlying processes, so that a trace, relation, or generated model matching the gold standard is semantically right; the paper itself notes in Section 7.3.1 that some gold standards are not semantically correct.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLMs crack semantic process mining","LLM fine-tuning unlocks process mining smarts","Semantics-aware process mining: fine-tuning is key","Fine-tuned LLMs outdo encoders on process tasks","Prompting alone fails; fine-tuning wins process tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1424,"prompt_tokens":980,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":596,"tokens_out":444,"duration_ms":4728,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:14:12.466246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the evaluation on a quality-assured subset: take a random sample of test models, have process experts judge whether each gold-standard trace, relation, and process tree is semantically correct, and rerun the fine-tuned models only on instances with expert-verified gold standards. If the fine-tuned LLMs no longer beat the encoder baseline or the random baseline on that subset, the claim that fine-tuning unlocks semantic process understanding would be refuted.","supporting_citations":[{"cited_title":"In: ICPM Workshops, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the source collection of process diagrams from which the behavior corpus and all gold standards are derived."},{"cited_title":"Information Systems 102, 101824 (2021)","cited_arxiv_id":null,"evidence_quote":"Provides the rule-based semantic anomaly detection approach used as a classification baseline."},{"cited_title":"In: BPM, pp","cited_arxiv_id":null,"evidence_quote":"Establishes the encoder-based machine-learning approach that fine-tuned LLMs are compared against on activity-pair anomaly detection."},{"cited_title":"In: Process Mining Handbook, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the formal definitions of process trees and directly-follows footprints on which the tasks and evaluation are built."},{"cited_title":"In: Application and The- ory of Petri Nets and Concurrency: 34th International Conference, PETRI NETS 2013, Milan, Italy, June 24-28, 2013","cited_arxiv_id":null,"evidence_quote":"Provides the translation from block-structured workflow nets to process trees used to build the process-tree gold standard."},{"cited_title":"56, (2018)","cited_arxiv_id":null,"evidence_quote":"Defines footprint-based fitness, the conformance measure used to score both process discovery tasks."}],"review_version":1}