PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks

Alessandro Berti; Humam Kourani; Wil M.P. van der Aalst

PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.13244 v1 pith:RKUUGUES submitted 2024-07-18 cs.CL cs.DB

Alessandro Berti , Humam Kourani , Wil M.P. van der Aalst This is my paper · ORCID

classification cs.CLcs.DB

keywords llmstasksminingmodelsprocessbenchmarkadequatebiases

verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments

read the original abstract

Large Language Models (LLMs) have the potential to semi-automate some process mining (PM) analyses. While commercial models are already adequate for many analytics tasks, the competitive level of open-source LLMs in PM tasks is unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive benchmark for PM focusing on domain knowledge (process-mining-specific and process-specific) and on different implementation strategies. We focus also on the challenges in creating such a benchmark, related to the public availability of the data and on evaluation biases by the LLMs. Overall, we observe that most of the considered LLMs can perform some process mining tasks at a satisfactory level, but tiny models that would run on edge devices are still inadequate. We also conclude that while the proposed benchmark is useful for identifying LLMs that are adequate for process mining tasks, further research is needed to overcome the evaluation biases and perform a more thorough ranking of the competitive LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Beyond Generalist LLMs: Specialist Agentic Systems for Structured Code Workflow Execution
cs.SE 2026-07 conditional novelty 6.0 of 10

A specialist BPMN-to-agent pipeline beats general-purpose coding agents on tool-use accuracy, latency, and token cost for deterministic business workflows.
On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks
cs.DB 2025-04 conditional novelty 6.0 of 10

Fine-tuned Llama-3 and Mistral reach macro F1 0.69 to 0.88 and fitness 0.80 to 0.84 on five new semantics-aware process mining benchmarks, while few-shot in-context learning stays near random.
Revisiting Predictive Process Monitoring in the Age of Foundation Models: A Comparative Study of Sequence, Tabular, and LLM Approaches
cs.LG 2026-07 conditional novelty 5.0 of 10

On five business-process logs, trained-from-scratch sequence models beat fine-tuned LLMs and tabular foundation models at next-activity prediction, while tabular models are competitive for remaining-time and next-even...
Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis
cs.DB 2024-11 conditional novelty 5.0 of 10

A benchmark of 20 business processes and 16 large language models finds Claude-3.5-Sonnet produces the highest-quality process models, and suggests that output optimization improves weaker models.
No AI Without PI! Object-Centric Process Mining as the Enabler for Generative, Predictive, and Prescriptive Artificial Intelligence
cs.AI 2025-07 conditional novelty 4.0 of 10

The paper asserts that object-centric process mining, renamed Process Intelligence, is the missing grounding layer for AI applied to operational processes.

Pith tools