Rethinking BPS: A Utility-Based Evaluation Framework

Han van der Aa; Heiner Stuckenschmidt; Keyvan Amiri Elyasi; Konrad \"Ozdemir; Lukas Kirchdorfer

REVIEW 2 major objections 5 minor 25 references

Rethinking BPS: A Utility-Based Evaluation Framework

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Business process simulation should be evaluated by whether simulated data trains predictive models as well as real data does, not by how closely simulated and real logs resemble each other under Earth Mover's Distance.

desk verdict Solid critique of EMD-based BPS evaluation and a promising utility framework, but the headline claim about separating accuracy from data complexity is undercut by an unnormalized utility loss. read the letter →

arxiv 2505.22316 v1 pith:W4BVXUAG submitted 2025-05-28 cs.LG

Konrad Özdemir , Lukas Kirchdorfer , Keyvan Amiri Elyasi , Han van der Aa , Heiner Stuckenschmidt This is my paper

classification cs.LG

keywords businessprocesssimulationminingpredictivemonitoringutility-basedevaluationEarthmover'sdistanceeventlogstrainonsynthetictestreal

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard way of evaluating business process simulation models measures the wrong thing. Standard practice compares a simulated event log to a hold-out test log using Earth Mover's Distance-based metrics, which treats simulation as a forecasting problem and can punish a faithful model when process behavior drifts, or reward a degenerate model that only matches mean arrival rates. The paper proposes evaluating a simulation model by its utility: train predictive process monitoring models on simulated data and on real training data, then measure how much performance on a hold-out test log differs between the two. If simulated data trains models as well as real data, the simulation has captured the as-is process. The result is an assessment that separates model accuracy from inherent data complexity and points to the specific process perspective that is poorly modeled.

What carries the argument

The central object is the utility-loss vector, computed by comparing predictive process monitoring performance on real versus simulated training data. The downstream tasks are next activity prediction (control flow), next role prediction (resource), and next processing time, next waiting time, and remaining time (temporal and congestion), evaluated on a hold-out test log with both a Transformer and an LSTM architecture. The formal instrument that carries the critique of existing evaluation is the identity for Wasserstein-1 distance on sorted count sequences, which shows that sorting removes temporal structure and makes the metric outlier-sensitive and mean-favoring. The utility-loss vector does the constructive work: because each task targets one process perspective, a large loss in one coordinate identifies which aspect of the process the simulation model fails to preserve.

What would settle it

Take a process with strong drift between train and test, deliberately corrupt a BPS model by randomizing activity order while keeping marginal counts, and compute utility loss. If real-data PPM performance on the test log is already near floor, the corrupt model's utility loss will also be near zero, so the framework would fail to flag a clearly wrong simulation model.

Watch

Extended reading notes

Core claim

The central claim is that a BPS model's quality is the utility of its simulated event log as training data for downstream predictive tasks. Formally, the utility loss of model $S$ is the element-wise absolute difference $\mathrm{UtilityLoss}(S) = |\mathcal{M}(L_{\text{train}}) - \mathcal{M}(L_{\text{sim}})|$ between performance vectors on the hold-out test log, averaged over PPM architectures and over five tasks spanning control flow, resource, temporal, and congestion perspectives. The paper also gives theoretical and empirical evidence against the standard metric: the Wasserstein-1 distance between binned count sequences collapses to $W_1(P,Q) = \frac{1}{B}\sum_{i=1}^B |x_{(i)} - y_{(i)}|$, so it sorts away temporal order and biases toward estimates near the mean. A concrete example shows a pattern-neglecting estimate can receive a perfect W1 score while a pattern-recognizing estimate is penalized, and an experiment shows a mean-only arrival model can beat a distribution-matching simulator under the standard metric. On the paper's own terms, the utility framework discerns model accuracy from data complexity, evaluates temporal relationships rather than absolute timestamps, and localizes which process perspective a simulator fails to capture.

Load-bearing premise

The load-bearing premise is that the hold-out test log is a valid benchmark for comparing training-data utility: if the test period is so drifted that models trained on either real or simulated data perform at floor, the absolute utility loss becomes artificially small and can mask model inaccuracy.

Editorial extensions

If this is right

The standard EMD-based comparison can rank a mean-only simulation above a distribution-matching one; under the utility framework, models are ranked by whether their simulated behavior supports realistic predictions on downstream tasks.
When process behavior drifts between train and test periods, a faithful simulation of the training period is no longer penalized for failing to forecast the drift, so simulator accuracy and data complexity are separated.
A near-zero utility loss on one downstream task localizes the process perspective a model captures well, while a large loss on next processing time, waiting time, or remaining time points to specific temporal or congestion failures.
The benchmark results suggest that the deep-learning-based simulator's temporal modeling does not clearly beat parameterized distributions, which changes which BPS design choices look justified.
Because utility loss is measured per task and per perspective, the framework can support targeted improvement: a simulator with good control-flow but poor resource losses can be repaired in the resource component rather than retrained wholesale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Utility loss as an absolute difference is scale-dependent: normalizing by the real-data performance level would let users compare across event logs and avoid the floor effect under extreme test-period drift.
Since a model that simply replays the training log would have near-zero utility loss, the framework needs a companion check for diversity or non-triviality before it can serve as a standalone acceptance test.
The same logic could be inverted for data augmentation: simulated logs with small utility loss relative to real data could be selected as training augmentation for predictive process monitoring, linking simulation quality to prediction gains.
The reported lack of correlation between next processing time and next waiting time errors suggests these are driven by different mechanisms; a direct test would correlate the two error vectors across many event logs and simulator variants.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the good news. The paper correctly identifies two real problems with the current standard: it treats simulation as forecasting, and the W1/EMD metrics discard temporal order. Theorem 1 is right—W1 on binned count sequences is computed on sorted values, so a pattern-neglecting estimate can score better than a faithful one. The example is persuasive. Applying the TSTR idea to BPS with a task set covering control flow, resources, timing, and congestion is a genuine novelty, and the synthetic experiment (Section 5.1) is a proper sanity check: known modifications produce utility loss in the expected perspectives, and the near-zero loss for calendar shifts shows the framework measures relative temporal structure. Code and data being public is a plus.

Now the soft spots. The utility loss as defined in Eq. (3) is the absolute difference between metric values on the hold-out test set. It is not normalized by what real-data performance is on that test set. When the test period is strongly drifted, a model trained on real data already performs poorly; any simulated log will also produce a poor model, and the absolute difference can be tiny even for a simulator that ignores the process. The P2P result (ASim NPP utility loss 0.60 against real MAE 67.11) lives exactly in that regime. The paper's own caveat about copying L_train addresses one degenerate case, but not the broader floor effect. So the claim that the framework separates model accuracy from data complexity is not yet established. The benchmark also lacks significance testing; differences between BPS approaches are reported without confidence intervals or paired tests, so it is hard to know which results are meaningful.

These issues are fixable. Normalizing utility loss by real-data performance, or evaluating on a stationarity control, would address the floor effect. The framework is well designed and the idea is worth developing.

I would send this to peer review. It is a serious contribution that advances the evaluation discussion, even though the central evidence is weaker than presented. A good referee will ask for exactly these controls.

Referee Report

2 major / 5 minor

Summary. The paper argues that the standard EMD-based evaluation of business process simulation (BPS) models has two flaws: an objective mismatch that treats simulation as forecasting, and a metric shortfall because Wasserstein-1 sorting destroys temporal structure. The authors propose a utility-based framework in which a simulated log is assessed by training predictive process monitoring (PPM) models on simulated and real training logs and comparing their performance on a hold-out test log. The utility loss (Eq. 3) is the element-wise absolute difference between the performance vectors of models trained on real and simulated data. The framework is evaluated in a synthetic experiment with known model modifications and in a benchmark of three BPS approaches across eight event logs. The paper claims the framework identifies which process perspective is inadequately captured and separates model accuracy from data complexity.

Significance. If the framework is validated, it would provide a valuable new direction for BPS evaluation, replacing log-distance comparisons with a downstream utility perspective that is closer to how simulation is actually used. The theoretical demonstration that W1 sorting removes temporal order (Theorem 1 and Example 1) is clear and correct, and the synthetic experiment with externally anchored modifications is a good design choice. The release of code and data is another strength. However, the central claim that the framework separates model accuracy from data complexity is not yet established, because the utility loss is unnormalized and lacks a no-skill baseline, leaving a floor-effect failure mode in drifted test periods. The benchmark results, especially for the P2P log, are interpreted more strongly than the current metric supports.

major comments (2)

[Section 4, Eq. (3); Section 5.2, Table 4] The unnormalized absolute difference in Eq. (3) makes the central claim vulnerable to a floor effect in drifted test periods. In the P2P log (Table 3 reports CT-train 12.14 vs CT-test 30.82 days), the PPM model trained on real Ltrain already has high MAE on the test log (NPP 67.11 min), so a model trained on any simulated log Lsim will also perform poorly, and the absolute difference can be small even if Lsim encodes a degenerate process. The P2P row in Table 4 illustrates this: ASim's NPP utility loss is 0.60 against a real-trained MAE of 67.11, and the paper interprets this as evidence that ASim captures processing times. Without a no-skill baseline (e.g., predicting the training mean) or normalization by the real-trained performance level, this small loss does not exclude the floor-effect explanation. The Section 4 caveat about approaches that copy Ltrain addresses only a trivial failure mode, not this broader scaling problem. I recommend reporting utility loss relative to M(Ltrain) or adding a baseline, and re-examining the P2P-based conclusion.
[Section 5.1, Table 2] The synthetic experiment lacks a direct control showing that the ground-truth LoanGT model has near-zero utility loss. Table 2 reports the predictive performance of LoanGT (NAP 0.71, NRP 0.75, NPP 67.60, etc.) but not the utility loss of LoanGT relative to a PPM trained on real data. Without this comparison, the reader cannot verify that an accurate simulator is assigned a low utility loss. The modifications are then interpreted as producing losses in specific perspectives, but the absence of the real-trained performance baseline makes it impossible to separate true signal from the floor effect described above. Please include the real-trained PPM performance and the LoanGT utility loss in the table or discussion.

minor comments (5)

[Section 3.3, Theorem 1] The formal proof of Theorem 1 is provided only via an external repository link; for a self-contained paper, the proof should be included in an appendix.
[Section 5.2, Table 4] The DSim rows show 'NA' for the NRP task; the text should explain why DeepSimulator does not produce role predictions.
[Section 5.1, Table 2] There is a typographical issue in 'LoanDU R' which should be 'LoanDUR'.
[Section 4, Eq. (2)] The notation M(L) is written as a function of only L, but the result depends on the hold-out test log Ltest; please clarify the dependence, for example by writing M_{Ltest}(L).
[Table 3] The log names 'C. 1000' and 'C. 2000' are not self-explanatory; please define them in the text or caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the utility-loss metric is an operational proposal validated by externally constructed modifications, not a self-referential derivation.

full rationale

The central derivation is not circular. The paper proposes UtilityLoss(S) = |M(Ltrain) - M(Lsim)| as a definitional evaluation metric, then validates it externally: Experiment 1 modifies a known simulation model in ways whose expected effects on control-flow, resource, temporal, and congestion perspectives are stated in advance, and the observed utility losses align with those expectations. This provides an outside anchor independent of the metric itself. The P2P drift example, where real-trained PPM errors are high and ASim's utility loss is near zero, is an interpretive claim about separating model accuracy from data complexity, not an equation that forces the conclusion by construction. The acknowledged 'copy the log' loophole is a scope restriction, not a hidden fit. Self-citations are not load-bearing: AgentSimulator [13] is a peer-reviewed baseline used empirically, and the framework's ideas are explicitly traced to the external TSTR literature [11]. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from author-only work, and no known result is merely relabeled. The unnormalized nature of Eq. (3) under strong test-period drift is a legitimate validity concern about sensitivity to floor effects, but it is not circularity: the metric does not reduce to its own inputs by definition. Thus the paper's claimed derivation chain is self-contained against external benchmarks and contains no circular steps.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The framework introduces no new physical or conceptual entities; it defines a new metric (utility loss) based on existing PPM tasks. The core axioms are about the validity of TSTR as an evaluation strategy in the process mining domain.

assumptions (3)

domain assumption Predictive process monitoring performance on a hold-out test log is a valid proxy for the utility of the training data.
The entire framework rests on the premise that if PPM models trained on simulated and real data perform similarly on a test log, the simulation captures task-relevant patterns. This is stated in Section 4.
domain assumption The simulated log should mirror the training log (the as-is process), not the test log.
The paper argues that BPS evaluation should assess how well the model captures the training distribution, and criticizes the standard practice for comparing to a drifted test log. This normative claim is central to the framework's design (Section 3.2).
domain assumption The selected five downstream tasks adequately cover the four process perspectives relevant for BPS evaluation.
The framework's diagnostic power depends on the tasks capturing control flow, resource, temporal, and congestion aspects. The paper acknowledges this is a limitation and may not capture all aspects (Section 6).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Rethinking BPS: A Utility-Based Evaluation Framework." pith.science (2026). https://pith.science/paper/W4BVXUAG

@misc{pith2026250522316,
  author       = {Pith},
  title        = {Pith review of: Rethinking BPS: A Utility-Based Evaluation Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W4BVXUAG}},
  note         = {Machine review of arXiv:2505.22316}
}

read the original abstract

Business process simulation (BPS) is a key tool for analyzing and optimizing organizational workflows, supporting decision-making by estimating the impact of process changes. The reliability of such estimates depends on the ability of a BPS model to accurately mimic the process under analysis, making rigorous accuracy evaluation essential. However, the state-of-the-art approach to evaluating BPS models has two key limitations. First, it treats simulation as a forecasting problem, testing whether models can predict unseen future events. This fails to assess how well a model captures the as-is process, particularly when process behavior changes from train to test period. Thus, it becomes difficult to determine whether poor results stem from an inaccurate model or the inherent complexity of the data, such as unpredictable drift. Second, the evaluation approach strongly relies on Earth Mover's Distance-based metrics, which can obscure temporal patterns and thus yield misleading conclusions about simulation quality. To address these issues, we propose a novel framework that evaluates simulation quality based on its ability to generate representative process behavior. Instead of comparing simulated logs to future real-world executions, we evaluate whether predictive process monitoring models trained on simulated data perform comparably to those trained on real data for downstream analysis tasks. Empirical results show that our framework not only helps identify sources of discrepancies but also distinguishes between model accuracy and data complexity, offering a more meaningful way to assess BPS quality.

Figures

Figures reproduced from arXiv: 2505.22316 by the authors.

**Figure 1.** Inter-arrival time histograms for the BPIC12W log. and ‘Test’ exhibit similar inter-arrival distributions (resembling an exponential distribution), ‘Simulated Mean’ collapses into a degenerate distribution, concentrating all mass at a single value. From a practitioner’s perspective, the choice is clear: the mean-based approach is unsuitable for meaningful simulation, whereas AgentSimulator reasonably approximates… view at source ↗

**Figure 2.** Overview of our proposed BPS evaluation framework. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 19 canonical work pages

[1]

In: vom Brocke, J., Rosemann, M

van der Aalst, W.M.P.: Business process simulation survival guide. In: vom Brocke, J., Rosemann, M. (eds.) Handbook on Business Process Management 1, Introduc- tion, Methods, and Information Systems, 2nd Ed, pp. 337–370. International Hand- books on Information Systems, Springer (2015)

work page 2015
[2]

arXiv preprint:2104.00721 (2021)

Bukhsh, Z.A., Saeed, A., Dijkman, R.M.: Processtransformer: Predictive business process monitoring with transformer network. arXiv preprint:2104.00721 (2021)

arXiv 2021
[3]

Decision Support Systems134 (2020)

Camargo, M., Dumas, M., González-Rojas, O.: Automated discovery of business process simulation models from event logs. Decision Support Systems134 (2020)

work page 2020
[4]

Camargo, M., Dumas, M., Rojas, O.G.: Learning accurate LSTM models of business processes. In: BPM. Springer (2019)

work page 2019
[5]

PeerJ Comput

Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci.7, e577 (2021)

work page 2021
[6]

In: CAiSE

Camargo, M., Dumas, M., Rojas, O.G.: Learning accurate business process simula- tion models from event logs via automated process discovery and deep learning. In: CAiSE. Springer (2022)

work page 2022
[7]

Information Systems127, 102447 (2025)

Chapela-Campa, D., Benchekroun, I., Baron, O., Dumas, M., Krass, D., Senderovich, A.: A framework for measuring the quality of business process sim- ulation models. Information Systems127, 102447 (2025)

work page 2025
[8]

CEUR Workshop Proceedings, vol

Dumas, M.: Constructing digital twins for accurate and reliable what-if business process analysis. CEUR Workshop Proceedings, vol. 2938, pp. 23–27 (2021)

work page 2021

Show all 25 references

[9]

Springer (2013)

Dumas, M., Rosa, M.L., Mendling, J., Reijers, H.A.: Fundamentals of Business Process Management. Springer (2013)

2013
[10]

In: Proceedings of the 26th international database engineered applications symposium

Endres, M., Mannarapotta Venugopal, A., Tran, T.S.: Synthetic data generation: A comparative study. In: Proceedings of the 26th international database engineered applications symposium. pp. 94–102 (2022)

2022
[11]

arXiv e-prints arXiv:1706.02633 (2017)

Esteban, C., Hyland, S.L., Rätsch, G.: Real-valued (Medical) Time Series Gener- ation with Recurrent Conditional GANs. arXiv e-prints arXiv:1706.02633 (2017)

2017 arXiv
[12]

In: ICCS

Khodyrev, I., Popova, S.: Discrete modeling and simulation of business processes using event logs. In: ICCS. Elsevier (2014)

2014
[13]

In: ICPM

Kirchdorfer, L., Blümel, R., Kampik, T., van der Aa, H., Stuckenschmidt, H.: Agentsimulator: An agent-based approach for data-driven business process simula- tion. In: ICPM. pp. 97–104. IEEE (2024)

2024
[14]

Springer (2022)

López-Pintado, O., Dumas, M.: Business process simulation with differentiated resources: Does it make a difference? In: BPM. Springer (2022)

2022
[15]

In: ICPM

López-Pintado, O., Murashko, S., Dumas, M.: Discovery and simulation of data- aware business processes. In: ICPM. IEEE (2024) Rethinking BPS: A Utility-Based Evaluation Framework 17

2024
[16]

In: ICPM

Meneghello, F., Francescomarino, C.D., Ghidini, C.: Runtime integration of ma- chine learning and simulation for business processes. In: ICPM. IEEE (2023)

2023
[17]

Meneghello, F., Francescomarino, C.D., Ghidini, C., Ronzani, M.: Runtime inte- gration of machine learning and simulation for business processes: Time and decision mining predictions. Inf. Syst.128, 102472 (2025)

2025
[18]

In: Proc

Okimura, I., Reid, M., Kawano, M., Matsuo, Y.: On the impact of data augmen- tation on downstream performance in natural language processing. In: Proc. of 3rd WS- Insights from Neg. Results in NLP. Assoc. for Comp. Linguistics (2022)

2022
[19]

Rozinat, A., Mans, R.S., Song, M., van der Aalst, W.M.P.: Discovering simulation models. Inf. Syst.34(3), 305–327 (2009)

2009
[20]

In: CVPR WS (2018)

Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., Birchfield, S.: Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In: CVPR WS (2018)

2018
[21]

Graduate studies in mathematics, American Mathematical Society (2003)

Villani, C., Society, A.M.: Topics in Optimal Transportation. Graduate studies in mathematics, American Mathematical Society (2003)

2003
[22]

Wang, Q., Gao, J., Lin, W., Yuan, Y.: Learning from synthetic data for crowd counting in the wild (2019)

2019
[23]

In: Beygelzimer, A., Hsu, D

Weed, J., Berthet, Q.: Estimation of smooth densities in wasserstein distance. In: Beygelzimer, A., Hsu, D. (eds.) Proceedings of the Thirty-Second Conference on Learning Theory. vol. 99, pp. 3118–3119. PMLR (25–28 Jun 2019)

2019
[24]

In: NeurIPS

Yoon, J., Jarrett, D., van der Schaar, M.: Time-series generative adversarial net- works. In: NeurIPS. vol. 32. Curran Associates, Inc. (2019)

2019
[25]

Özdemir, K., Kirchdorfer, L., Amiri Eliasi, K., Van der Aa, H., Stuckenschmidt, H.: Codebase: Rethinking business process simulation: A utility-based evaluation framework (May 2025).https://doi.org/10.5281/zenodo.15489551

2025 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

In: vom Brocke, J., Rosemann, M

van der Aalst, W.M.P.: Business process simulation survival guide. In: vom Brocke, J., Rosemann, M. (eds.) Handbook on Business Process Management 1, Introduc- tion, Methods, and Information Systems, 2nd Ed, pp. 337–370. International Hand- books on Information Systems, Springer (2015)

work page 2015

[2] [2]

arXiv preprint:2104.00721 (2021)

Bukhsh, Z.A., Saeed, A., Dijkman, R.M.: Processtransformer: Predictive business process monitoring with transformer network. arXiv preprint:2104.00721 (2021)

arXiv 2021

[3] [3]

Decision Support Systems134 (2020)

Camargo, M., Dumas, M., González-Rojas, O.: Automated discovery of business process simulation models from event logs. Decision Support Systems134 (2020)

work page 2020

[4] [4]

Camargo, M., Dumas, M., Rojas, O.G.: Learning accurate LSTM models of business processes. In: BPM. Springer (2019)

work page 2019

[5] [5]

PeerJ Comput

Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci.7, e577 (2021)

work page 2021

[6] [6]

In: CAiSE

Camargo, M., Dumas, M., Rojas, O.G.: Learning accurate business process simula- tion models from event logs via automated process discovery and deep learning. In: CAiSE. Springer (2022)

work page 2022

[7] [7]

Information Systems127, 102447 (2025)

Chapela-Campa, D., Benchekroun, I., Baron, O., Dumas, M., Krass, D., Senderovich, A.: A framework for measuring the quality of business process sim- ulation models. Information Systems127, 102447 (2025)

work page 2025

[8] [8]

CEUR Workshop Proceedings, vol

Dumas, M.: Constructing digital twins for accurate and reliable what-if business process analysis. CEUR Workshop Proceedings, vol. 2938, pp. 23–27 (2021)

work page 2021