Pith. sign in

REVIEW 2 major objections 5 minor 35 references

Identifying Process Improvement Opportunities through Process Execution Benchmarking

T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A process-mining technique that turns event-log comparison into concrete change recommendations.

desk verdict A cleanly built process-mining pipeline for proposing activity replacements, but the performance-impact claim is explicitly unevaluated, so the prescriptive promise outruns the evidence. read the letter →

arxiv 2504.16215 v1 pith:RLB7HZNU submitted 2025-04-22 cs.SE

Luka Abb , Majid Rafiei , Timotheus Kampik , Jana-Rebecca Rehse This is my paper · ORCID
classification cs.SE
keywords processminingbenchmarkingimprovementactivityreplacementeventlogsbehavioralfootprintsprescriptiveanalyticsperformance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Commercial process-mining tools tell an organization how its process performance compares with peers, but not what to do about a gap. This paper proposes a prescriptive benchmarking technique: given an "own" event log and a "benchmark" event log from the same process type, it finds activities whose behavior in the own log matches activities in the benchmark log, groups compatible replacements into process changes, and scores each change for feasibility and expected performance impact. The intended practical payoff is a manager-ready list of concrete modifications, such as "create the purchase order the way the faster branch does," with numbers attached. In synthetic evaluation the matching step finds most intended replacements (recall 0.901, precision 0.831), and the feasibility score separates technique-suggested changes from a random baseline (0.802 vs 0.580).

What carries the argument

The load-bearing mechanism is the behavioral footprint: the row of the activity-relation matrix that records, for each activity in the log, whether it is in strict order, exclusive, or interleaving with every other activity. Footprints are made noise-tolerant by thresholded support scores ($s_\#$ and $s_\parallel$), then matched across logs only on the activities present in both logs. Compatibility of replacements is handled by a compatibility graph whose fully connected subgraphs (maximal cliques, plus single nodes) become the process changes; feasibility uses Levenshtein-based trace alignment to the benchmark log, and performance impact is the frequency-weighted mean difference in the common performance measure between affected own traces and their closest aligned benchmark traces.

What would settle it

Generate a paired synthetic scenario in which activity $b$ has the same performance as activity $a$, but an unrelated third activity that co-varies with $b$ in the benchmark log is the true cause of faster cases. If the technique's $\Delta\pi$ for the replacement $(a_1,b_2)$ is substantially positive in this scenario, the performance-impact score is attributing causality it cannot see. The same experiment can be run with a real log pair where the better-performing branch differs from the worse one by several known changes, checking whether the recommended replacements match the actual causal changes.

Watch

Extended reading notes

Core claim

The paper claims that process execution benchmarking can be made prescriptive rather than merely descriptive. Using behavioral relations (strict order, exclusiveness, interleaving order) computed from traces in both logs, it constructs one behavioral footprint per activity and declares an activity from the benchmark log a plausible replacement for an own-log activity when their footprints agree on all activities present in both logs. Pairwise replacements that do not conflict (no activity is replaced twice) are combined into process changes, each receiving a feasibility score based on edit similarity between the modified own variants and their closest benchmark variants, and an expected performance impact computed as the mean performance difference between affected own traces and their closest benchmark counterparts. The authors' claim is that this pipeline converts a benchmark event log into evidence-based recommendations for process improvement, which is what existing high-level benchmarking dashboards do not provide.

Load-bearing premise

The approach assumes that the measured performance difference between an own process variant and its closest variant in the benchmark log is caused by the activities being replaced; if other differences between the variants drive the performance gap, the impact estimates are not evidence for improvement.

Editorial extensions

If this is right

  • Process managers receive a sortable list of activity-replacement options, each with a feasibility score in [0,1] and an estimated per-case performance impact, so benchmarking output can feed directly into improvement initiatives.
  • Because the technique uses only event logs, it can be applied to internal benchmarking across branches of the same organization without hand-built process models.
  • If the technique is right, the matching step transfers best practices: activities already executed in the benchmark log become candidates to replace slower own-log activities that occupy analogous behavioral positions.
  • The noise-tolerant relation scores should make recommendations robust to infrequent traces, so a single erroneous case does not by itself create a false replacement.
  • The evaluation's recall of 0.901 and precision of 0.831 suggest the approach can recover most intended replacements while keeping false alarms limited in synthetic logs of the tested shape.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The performance-impact score attributes the entire performance difference between an own variant and its closest benchmark match to the replaced activities; a controlled experiment that varies only the replacement while holding other activities equal would test whether that attribution holds.
  • The same compatibility-graph machinery could be extended beyond 1:1 replacements to n:m exchanges (e.g., replacing an entire choice construct), at higher computational cost; this is a natural next test.
  • The technique's reliance on standardized activity names could be relaxed by adding semantic matching on labels, which would widen applicability to cross-organization logs where naming conventions differ.
  • A testable extension is to apply the pipeline to real paired logs from branches with known best-practice differences and check whether the recommended replacements correspond to the changes the better-performing branch actually made.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a prescriptive process execution benchmarking technique that, given an 'own' event log and a benchmark event log, identifies behaviorally plausible activity replacements, combines them into compatible sets of process changes, and scores each change by feasibility (edit-distance-based alignment to benchmark variants) and estimated performance impact (average performance difference between affected own variants and their closest benchmark matches). The technique is evaluated on 1,000 synthetic log pairs, reporting matching precision 0.831 and recall 0.901, and a feasibility score of 0.802 against a random baseline of 0.580; a case study on an SAP Signavio purchasing log illustrates the output. The paper explicitly omits an evaluation of the performance impact measure in Section 5.1.

Significance. If the performance-impact step were validated, the paper would make a useful contribution to prescriptive process mining: it is, to my knowledge, a novel end-to-end pipeline for turning cross-organizational event-log comparison into concrete activity-replacement suggestions, and the transparent, reproducible implementation (code released on GitLab) plus the synthetic ground-truth evaluation of the matching step are strengths. The feasibility score's separation from a random baseline indicates the behavioral filtering is not vacuous. However, the central prescriptive claim—that the recommendations improve process performance—rests entirely on the unvalidated performance-impact estimate, so the significance as a prescriptive tool is currently not demonstrated.

major comments (2)
  1. [Section 4.5] The expected performance impact Δπ(a1,b2) is computed as the frequency-weighted mean of (π̄(μ(v'1)) − π̄(v1)) over affected own variants, where μ(v'1) is the closest benchmark variant to the modified variant v'1. This attributes the entire performance gap between the own variant and its closest benchmark match to the replaced activity (or activities in a change), without controlling for other differences such as insertions/deletions of additional activities, resource or case-mix effects, or variant-frequency differences between the logs. Because the closest match is chosen on control-flow edit distance, the performance contrast is not a controlled comparison, and this attribution is the load-bearing assumption for the paper's prescriptive claim.
  2. [Section 5.1] The manuscript states that evaluating the performance assessment 'would not be informative' and omits it. This is a load-bearing gap: the matching and feasibility results do not validate the performance-impact step, and without that step the output is a list of behaviorally plausible changes with no evidence that they improve performance. I ask the authors to add a synthetic experiment with injected performance effects (e.g., make the replacement activity systematically faster in the benchmark log than the original in the own log, with known effect sizes) and assess whether the ranking by Δπ recovers the true improvements, and to add at least one control comparison (e.g., using random or alternative matchings) to test the attribution.
minor comments (5)
  1. [Section 4.3] The text says that 'fully connected subgraphs' (cliques) of the compatibility graph are the output, but Figure 4 labels the output as 'Max. Comp. Replacements' and shows only two maximal cliques; please clarify whether all cliques or only maximal cliques are returned, as this changes the number of process changes presented to the user.
  2. [Section 5.1, Table 2] The table reports only means over the 1,000 log pairs; please add standard deviations or confidence intervals for precision, recall, and feasibility, since these metrics are likely to vary with the random generation of models and logs.
  3. [Section 4.1] The exclusiveness and interleaving thresholds exc and int are user-set with no guidance beyond 'close to 1'; a brief sensitivity analysis showing how matching precision/recall vary with these thresholds would strengthen the reproducibility of the technique.
  4. [Section 5.2, Table 3] The table's formatting is ambiguous: the last two rows appear to describe process changes consisting of two replacement options each, but the column layout makes it difficult to see which activities are combined; please reformat for readability.
  5. [Section 5.2] The case study text concludes that 'creating a PO in the SCM leads to considerably slower process executions' based on the Δπ values, but since the performance-impact measure is exactly the unvalidated component and the case study has no ground truth, the wording overstates the evidence; please rephrase as illustrative output interpretation or add validation first.

Circularity Check

1 steps flagged · score 6.0 of 10

Performance-impact estimate is definitional: Δπ is the raw benchmark-vs-own performance gap, and the paper omits its evaluation as 'not informative'.

  1. self definitional [Section 4.5, Δπ definition; Section 5.1, experiment evaluation note]
    "The expected performance impact of a process change can then be approximated as the average difference in π between the traces in L1 where the to-be-replaced activities are executed and their corresponding (closest match) traces in L2 where the replacing activities are executed. [...] By design, our approach involves identifying these exact traces in the benchmark log through alignments and calculating a performance difference to the original ones (see Section 4.5). This evaluation would therefore not be informative w.r.t. its efficacy."

    Δπ(a1,b2) is defined as the frequency-weighted mean of π̄(μ(v1′)) − π̄(v1) over affected variants; this is exactly the observed performance gap between the own variants and their closest benchmark matches. The paper's central claim—that the technique 'recommends targeted process changes to improve process performance'—is therefore supported only by this gap. The evaluation section explicitly declines to test the performance assessment because the test would reproduce the same trace-identification-and-difference computation ('not informative'). So the 'expected performance impact' is the input/alignment result renamed as a prediction, not an independently derived or validated estimate; the recommendation carries no evidence beyond the raw benchmark-vs-own difference.

full rationale

The core matching step is genuinely validated: synthetic ground truth (1,000 model pairs with recorded replacements) yields recall 0.901 and precision 0.831, and the feasibility score is compared to a random baseline (0.802 vs 0.580). These parts are not circular. The circularity is concentrated in Section 4.5's performance-impact measure: by definition it is the average performance difference between affected own variants and their closest benchmark matches, and Section 5.1 states that an evaluation would be 'not informative' because the method is exactly that computation. Since the prescriptive claim ('expected to improve performance') is operationalized solely by this measure, the paper's main improvement advice reduces to identifying and renaming a performance gap. No load-bearing self-citation or imported uniqueness theorem appears; the issue is definitional, not bibliographic. Score 6 reflects a partial but central circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; behavioral footprints, compatibility graphs, and trace alignments are constructions from existing process mining concepts. The main assumptions are the same-process-type, standardized-name, common-performance-measure, and behavioral-equivalence premises, plus the edit-distance-as-feasibility heuristic.

free parameters (3)
  • exclusiveness threshold (exc) = not reported; recommended close to 1
    User-set threshold in Section 4.1 controlling when the exclusiveness score is strong enough to classify two activities as exclusive; no value or sensitivity analysis is reported.
  • interleaving threshold (int) = not reported; recommended close to 1
    User-set threshold in Section 4.1 controlling when the interleaving score is strong enough to classify two activities as interleaving; no value or sensitivity analysis is reported.
  • case-study variant filter = 5% least common variants removed
    Section 5.2 filters out the 5% least common variants before applying the technique; this preprocessing choice is not justified or varied.
assumptions (5)
  • domain assumption Both event logs capture the same process type.
    Section 4, Assumption (1): otherwise it would be hard to improve one process using data-driven insights from the other.
  • domain assumption Activity names are standardized, so same name means same process step.
    Section 4, Assumption (2): realistic only in environments like SAP with standardized reference process models.
  • domain assumption Both logs contain at least one common case-level performance measure.
    Section 4, Assumption (3): the performance impact estimate requires a common performance measure pi(t).
  • domain assumption Identical behavioral footprints on shared activities imply that activities implement the same process step.
    Section 4.2: an activity in L2 is treated as a replacement for an activity in L1 iff their behavioral footprints are identical. This is the core matching heuristic.
  • domain assumption A close edit-distance match in the benchmark log indicates that a modified variant is feasible.
    Section 4.4: feasibility is computed as the average edit similarity between modified own variants and their closest benchmark variants.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Identifying Process Improvement Opportunities through Process Execution Benchmarking." pith.science (2026). https://pith.science/paper/RLB7HZNU

@misc{pith2026250416215,
  author       = {Pith},
  title        = {Pith review of: Identifying Process Improvement Opportunities through Process Execution Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RLB7HZNU}},
  note         = {Machine review of arXiv:2504.16215}
}
read the original abstract

Benchmarking functionalities in current commercial process mining tools allow organizations to contextualize their process performance through high-level performance indicators, such as completion rate or throughput time. However, they do not suggest any measures to close potential performance gaps. To address this limitation, we propose a prescriptive technique for process execution benchmarking that recommends targeted process changes to improve process performance. The technique compares an event log from an ``own'' process to one from a selected benchmark process to identify potential activity replacements, based on behavioral similarity. It then evaluates each proposed change in terms of its feasibility and its estimated performance impact. The result is a list of potential process modifications that can serve as evidence-based decision support for process improvement initiatives.

Figures

Figures reproduced from arXiv: 2504.16215 by the authors.

Figure 1
Figure 1. Running example own and benchmark versions of a (simplified) purchas [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. High-level overview of the proposed technique and intermediary results [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Matching activities between footprint matrices to find behaviorally plau [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Matches, compatibility graph, and maximal compatible replacements from [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 35 canonical work pages

  1. [1]

    IEEE Transactions on Knowledge and Data Engineering 16(9), 1128–1142 (2004)

    van der Aalst, W., Weijters, T., Maruster, L.: Workflow mining: discovering process models from event logs. IEEE Transactions on Knowledge and Data Engineering 16(9), 1128–1142 (2004)

    work page 2004

  2. [2]

    Wiley International Reviews: Data Mining and Knowledge Discovery 2(2), 182–192 (2012)

    van der Aalst, W., Adriansyah, A., van Dongen, B.: Replaying history on process models for conformance checking and performance analysis. Wiley International Reviews: Data Mining and Knowledge Discovery 2(2), 182–192 (2012)

    work page 2012

  3. [3]

    In: IEEE International Enterprise Distributed Object Computing Conference

    Aksu, U., Reijers, H.A.: How business process benchmarks enable organizations to improve performance. In: IEEE International Enterprise Distributed Object Computing Conference. pp. 197–208 (2020)

    work page 2020

  4. [4]

    In: Advanced Information Systems Engineering

    Ballambettu, N.P., Suresh, M.A., Bose, R.P.J.C.: Analyzing process variants to un- derstand differences in key performance indices. In: Advanced Information Systems Engineering. pp. 298–313. Springer, Cham (2017)

    work page 2017

  5. [5]

    Business Process Management Journal 22(1), 56–74 (2016)

    Bisogno, S., Calabrese, A., Gastaldi, M., Levialdi Ghiron, N.: Combining modelling and simulation approaches: How to measure performance of business processes. Business Process Management Journal 22(1), 56–74 (2016)

    work page 2016

  6. [6]

    In: Hull, R., Mendling, J., Tai, S

    Bose, R.P.J.C., van der Aalst, W.: Trace alignment in process mining: Opportu- nities for process diagnostics. In: Hull, R., Mendling, J., Tai, S. (eds.) Business Process Management. pp. 227–242. Springer, Cham (2010)

    work page 2010

  7. [7]

    In: Enterprise, Business-Process and Information Systems Model- ing

    Buijs, J.C.A.M., Reijers, H.A.: Comparing business process variants using models and event logs. In: Enterprise, Business-Process and Information Systems Model- ing. pp. 154–168. Springer, Cham (2014)

    work page 2014

  8. [8]

    In: BPM Demos

    Burattin, A.: PLG2: Multiperspective process randomization with online and of- fline simulations. In: BPM Demos. CEUR (2016)

    work page 2016

Show all 35 references
  1. [9]

    Benchmarking 9, 244–255 (2002)

    Carpinetti, L., de Melo, A.: What to benchmark? a systematic approach and cases. Benchmarking 9, 244–255 (2002)

    2002

  2. [10]

    Celonis: How does process mining work? (2024), https://www.celonis.com/ process-mining/how-does-process-mining-work/ , accessed: 05.12.2024

    2024

  3. [11]

    Organization Science 12(3), 346–371 (2001)

    Cramton, C.D.: The mutual knowledge problem and its consequences for dispersed collaboration. Organization Science 12(3), 346–371 (2001)

    2001

  4. [12]

    Enterprise Resource Planning Series, Prentice Hall PTR, Upper Saddle River (1997)

    Curran, T., Keller, G., Ladd, A.: SAP R/3 Business Blueprint: Understanding the Business Process Reference Model. Enterprise Resource Planning Series, Prentice Hall PTR, Upper Saddle River (1997)

    1997

  5. [13]

    Dees, M., de Leoni, M., van der Aalst, W.M.P., Reijers, H.A.: What if process predictions are not followed by good recommendations? In: BPM Industry Forum. pp. 61–72. CEUR (2019)

    2019

  6. [14]

    In: BPM Workshops

    van Dongen, B.F., Adriansyah, A.: Process mining: Fuzzy clustering and perfor- mance visualization. In: BPM Workshops. pp. 158–169. Springer, Cham (2010)

    2010

  7. [15]

    Business Process Management Journal 27(2), 369– 387 (2021)

    Grisold, T., Mendling, J., Otto, M., vom Brocke, J.: Adoption, use and management of process mining in practice. Business Process Management Journal 27(2), 369– 387 (2021)

    2021

  8. [16]

    In: Business Information Systems

    Gr¨ oger, C., Schwarz, H., Mitschang, B.: Prescriptive analytics for recommendation- based business process optimization. In: Business Information Systems. pp. 25–37. Springer, Cham (2014)

    2014

  9. [17]

    Kubrak, K., Milani, F., Nolte, A., Dumas, M.: Prescriptive process monitoring: Quo vadis? PeerJ Computer Science 8, e1097 (2022)

    2022

  10. [18]

    In: BPM Workshops

    K¨ uster, J.M., Koehler, J., Ryndina, K.: Improving business process models with reference models in business-driven development. In: BPM Workshops. pp. 35–44. Springer, Cham (2006) 16 Luka Abb, Majid Rafiei, Timotheus Kampik, and Jana-Rebecca Rehse

    2006

  11. [19]

    In: BPM Work- shops

    Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Discovering block-structured process models from event logs containing infrequent behaviour. In: BPM Work- shops. pp. 66–78. Springer, Cham (2014)

    2014

  12. [20]

    Information Systems 102, 101724 (2021)

    Leemans, S.J., van der Aalst, W.M., Brockhoff, T., Polyvyanyy, A.: Stochastic process mining: Earth movers’ stochastic conformance. Information Systems 102, 101724 (2021)

    2021

  13. [21]

    In: Business Process Management

    Leopold, H., Niepert, M., Weidlich, M., Mendling, J., Dijkman, R., Stuckenschmidt, H.: Probabilistic optimization of semantic process model matching. In: Business Process Management. pp. 319–334. Springer, Cham (2012)

    2012

  14. [22]

    Cybernetics and Control Theory 10(8), 707–710 (1966)

    Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Cybernetics and Control Theory 10(8), 707–710 (1966)

    1966

  15. [23]

    Progress in Artificial Intelligence 11(3), 275–290 (2022)

    Park, G., van der Aalst, W.M.P.: Action-oriented process mining: bridging the gap between insights and actions. Progress in Artificial Intelligence 11(3), 275–290 (2022)

    2022

  16. [24]

    In: International Conference on Process Mining

    Park, G., Van Der Aalst, W.M.: Realizing a digital twin of an organization using action-oriented process mining. In: International Conference on Process Mining. pp. 104–111 (2021)

    2021

  17. [25]

    ACM Transactions on Management Information Systems 5(4), 1–18 (2015)

    Partington, A., Wynn, M., Suriadi, S., Ouyang, C., Karnon, J.: Process mining for clinical processes: A comparative analysis of four australian hospitals. ACM Transactions on Management Information Systems 5(4), 1–18 (2015)

    2015

  18. [26]

    Jour- nal of Knowledge Management 9(3), 18–35 (2005)

    Riege, A.: Three-dozen knowledge-sharing barriers managers must consider. Jour- nal of Knowledge Management 9(3), 18–35 (2005)

    2005

  19. [27]

    SAP Signavio: Process performance indicators in SAP Signavio Process In- sights (2024), https://community.sap.com/t5/technology-blogs-by-sap/ process-performance-indicators-in-sap-signavio-process-insights-discovery/ ba-p/13744670, accessed: 05.12.2024

    2024

  20. [28]

    In: Handbook on Business Process Management 2: Strategic Alignment, Governance, People and Culture

    Scheer, A.W., Hoffmann, M.: The process of business process management. In: Handbook on Business Process Management 2: Strategic Alignment, Governance, People and Culture. pp. 351–380. Springer, Cham (2015)

    2015

  21. [29]

    Information Sciences 657, 119958 (2024)

    Schuster, D., Zerbato, F., van Zelst, S.J., van der Aalst, W.M.: Defining and visu- alizing process execution variants from partially ordered event data. Information Sciences 657, 119958 (2024)

    2024

  22. [30]

    Stein Dani, V., Leopold, H., van der Werf, J.M.E.M., Beerepoot, I., Reijers, H.A.: From process mining insights to process improvement: All talk and no action? In: Cooperative Information Systems. pp. 275–292. Springer, Cham (2024)

    2024

  23. [31]

    Sage (2002)

    Szulanski, G.: Sticky knowledge: Barriers to knowing in the firm. Sage (2002)

    2002

  24. [32]

    Knowledge-Based Systems 211, 106557 (2021)

    Taymouri, F., Rosa, M.L., Dumas, M., Maggi, F.M.: Business process variant anal- ysis: Survey and classification. Knowledge-Based Systems 211, 106557 (2021)

    2021

  25. [33]

    ACM Transactions on Intelligent Systems and Technology 10(4) (2019)

    Verenich, I., Dumas, M., Rosa, M.L., Maggi, F.M., Teinemaa, I.: Survey and cross- benchmark comparison of remaining time prediction methods in business pro- cess monitoring. ACM Transactions on Intelligent Systems and Technology 10(4) (2019)

    2019

  26. [34]

    In: Advanced Information Systems En- gineering

    Weidlich, M., Dijkman, R., Mendling, J.: The icop framework: Identification of correspondences between process models. In: Advanced Information Systems En- gineering. pp. 483–498. Springer, Cham (2010)

    2010

  27. [35]

    In: Advanced Information Systems Engineering

    Weidlich, M., Mendling, J., Weske, M.: A foundational approach for managing process variability. In: Advanced Information Systems Engineering. pp. 267–282. Springer, Cham (2011)

    2011

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.