Pith. sign in

REVIEW 4 major objections 5 minor 24 references

ProCause: Generating Counterfactual Outcomes to Evaluate Prescriptive Process Monitoring Methods

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ProCause generates counterfactual outcomes to evaluate prescriptive process monitoring methods.

desk verdict Practical extension of RealCause with ensemble and LSTM support; the ensemble claim is plausible and well-tested on a simulator, but the simulator is the authors' own and real-world ranking validity is untested. read the letter →

arxiv 2509.00797 v1 pith:ZDWLENVU submitted 2025-08-31 cs.LG cs.AIstat.ME

Jakob De Moor , Hans Weytjens , Johannes De Smedt This is my paper
classification cs.LGcs.AIstat.ME
keywords PrescriptiveProcessMonitoringCausalInferenceCounterfactualOutcomeGenerationMiningTARNetMeta-learnersLSTMEvaluationMethodology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prescriptive process monitoring recommends real-time interventions in business processes, but evaluating a recommendation policy is hard because the outcomes of actions not taken are never observed. The paper introduces ProCause, a generative framework that trains causal learners on historical event logs to produce these missing counterfactual outcomes, and then uses those outcomes to score and rank PresPM methods. The central claim is that averaging three causal-learner architectures—S-learner, T-learner, and TARNet—into an ensemble yields more reliable evaluations than the single TARNet model used by the established RealCause approach. The paper also shows that sequential models such as LSTMs can improve evaluation when intervention timing matters, and that the generated treatment and outcome variables statistically match real event logs. A practitioner adopting the ensemble as a default would get more stable comparisons of PresPM methods without needing to know the unknowable causal structure of their data.

What carries the argument

The central mechanism is the outcome-distribution estimator used as an evaluator: given a prefix of a case (a partial trace), a treatment level, and covariates, ProCause fits the conditional outcome distribution P(Y|T,X) using either MLPs or LSTMs under three causal learners—S-learner (one model with treatment as input), T-learner (separate models per treatment level), and TARNet (shared representation with treatment-specific heads)—and can average samples from the three learners' outcome distributions to form an ensemble. The ensemble is what buffers against the fundamental problem of causal inference: because the true data-generating process is unobservable, no single learner can be truste

What would settle it

Run the same ProCause pipeline on a second PresPM simulator or on logged data from a randomized intervention trial with known counterfactual outcomes, comparing ensemble versus TARNet+MLP across the same confounding levels. If TARNet's Kendall tau or Wasserstein distance is equal to or better than the ensemble's on the timing-based intervention, the central claim that the ensemble is consistently more reliable fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an ensemble of three causal-inference learners—S-learner, T-learner, and TARNet—provides consistently strong counterfactual outcome generation for evaluating PresPM methods across interventions, confounding levels, and assessment types, whereas any single learner, including the TARNet used by RealCause, performs well only under favorable data conditions. The paper further claims that incorporating LSTMs as base models improves evaluation accuracy for interventions with time-varying treatment decisions, and that on the BPIC12 and BPIC17 datasets the generated treatment and outcome distributions are statistically indistinguishable from th

Load-bearing premise

The SimBank simulator faithfully represents real-world prescriptive process monitoring data, so evaluation accuracy measured on it carries over to actual deployments.

Editorial extensions

If this is right

  • If the ensemble is adopted as the default evaluator, PresPM comparisons no longer depend on guessing which causal learner matches the unknowable data-generating process.
  • For interventions with timing decisions, LSTM-based evaluators improve outcome estimation and ranking, so evaluation frameworks should offer sequential base models rather than relying solely on MLPs.
  • An evaluator's absolute accuracy (distance to the true outcome distribution) does not imply its ranking accuracy and vice versa, so evaluations should report both metrics separately.
  • Because generated variables on BPIC12 and BPIC17 pass statistical realism tests, the ensemble can support practical evaluation in real-world settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ensemble averages sampled outcomes from three learners, so systematic errors shared by all three (e.g., all three failing on the same rare treatment group) would persist in the average.
  • The weak ranking agreement for the timing-based intervention suggests that model-driven evaluation may need architectures that explicitly model event order, or a fundamentally different ground-truth source such as logged randomized interventions, to reach full reliability.
  • The ensemble idea likely transfers to other evaluation problems where counterfactual outcomes are generated from observational data, such as off-policy evaluation in reinforcement learning or clinical decision support, whenever the true data-generating process is unknown.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ProCause, a generative approach for evaluating Prescriptive Process Monitoring (PresPM) methods. ProCause extends the RealCause framework by supporting both MLP and LSTM base models and by integrating three causal-inference learner architectures (S-learner, T-learner, TARNet) plus an ensemble that averages their counterfactual outcomes. The method is assessed on the authors' SimBank simulator, where ground truth is known, using Wasserstein distance for absolute outcome accuracy and Kendall's tau for ranking accuracy of PresPM methods. A real-world validation on BPIC12 and BPIC17 compares generated treatment and outcome variables with observed ones via statistical tests. The main claims are that the ensemble option gives the most consistent evaluation reliability and that LSTM-based models can help when treatment timing matters. The paper includes an openly accessible implementation.

Significance. If the central claims held, ProCause would be a useful, practical upgrade to RealCause for PresPM evaluation, and the comparison of absolute versus relative evaluation performance is a valuable conceptual contribution. The simulator-based experiment with known ground truth, the attempt to vary confounding, and the release of code are strengths. However, the evidence for the headline 'ensemble is most reliable' claim is incomplete: it rests on a single, author-built simulator, lacks inferential statistics, and the real-world validation does not test ranking accuracy. The paper's own reporting of weak ranking agreement for one intervention further tempers the conclusions. The contribution is incremental but potentially relevant to the process-mining evaluation community.

major comments (4)
  1. [Section VI.A / Fig. 3] The central claim that the ensemble is 'most consistently strong' is not backed by inferential statistics. The figure shows only shaded standard errors, and across two interventions and six confounding levels there are no significance tests or confidence intervals for differences between learners or between MLP/LSTM. With this limited number of configurations, overlapping error bars leave open the possibility that the ensemble's advantage is within noise. Please add paired tests across the 12 setups (e.g., bootstrap paired differences or a signed-rank test) or report effect sizes.
  2. [Sections V.B, VI.B, VIII] External validity of the main claim rests entirely on SimBank, the authors' own simulator from prior work. The paper acknowledges 'reliance on a single simulator' (Section VIII) but does not address the resulting risk that the DGP's structure favors the ensemble's averaging behavior. The real-world validation (Section VI.B) tests only whether generated T and Y have similar marginal distributions to the real test set; it does not test whether ProCause ranks PresPM methods correctly on real data. The abstract's 'ensuring a more reliable evaluation' is therefore not empirically established for real deployments. A second independent simulator or a semi-synthetic setup with known counterfactuals (or a placebo-type test on real logs) would materially strengthen the claim.
  3. [Section VI.A.3 / Section VII] The manuscript itself reports weak Kendall's tau agreement for the 'time contact HQ' intervention, which is one of the two main SimBank interventions and the one with the strongest temporal component. Since correct ranking of PresPM methods is the primary practical use of an evaluator, this weak agreement substantially limits the general claim that ProCause provides 'a robust and reliable framework'. The conclusion should be qualified to state when the method is reliable and when it is not, rather than concluding that ProCause 'offers a promising step towards more comprehensive evaluations' without this caveat.
  4. [Section V.C / Table I] The realism validation is presented in a way that overstates support. The text says 'in every case, at least one of these configurations produced a p-value above commonly accepted significance thresholds', but several reported p-values are clearly below 0.05 (BPI12 LSTM: Y KS=0.03, Y ES=0.01; BPI17 LSTM: Y KS=0.01). Selecting the favorable configuration after the fact is not a valid test. In addition, no multiple-testing correction is applied, and the multivariate tests on (T,Y,X) are admitted to lose power due to high-dimensional X, yielding p-values near 1. The validation section should use a pre-specified acceptance rule (e.g., both base models must pass, or a combined test) and should report the low-power limitation explicitly in the interpretation of the results.
minor comments (5)
  1. [Section V.B / Fig. 3] Figure 3 is extremely dense; the 'single setup' background points are hard to distinguish from the aggregated lines, especially in grayscale. Consider separate panels per aggregation or a supplementary table.
  2. [Section V.B.1.a] The statement that 'the maximum possible network size resulting from this tuning is identical across setups' needs a precise definition of the hyperparameter search space (ranges for hidden dimension, learning rate, batch size) to make this claim checkable.
  3. [Section V.C / Table I] Report p-values with consistent precision; '0.00' should be given as e.g. '<0.005'. Also note which exact test is one-sided/two-sided.
  4. [Section VI.A.3] The sentence 'model rankings under the set interest rate intervention are generally in agreement with the true ranking' would benefit from a numerical summary, since the figure alone does not show the actual ranking tables; the repository link is helpful but should be complemented by an in-paper table.
  5. [Section VII] Figure 4 is described as a probability density of WD; the empirical bars and smoothed lines are informative, but the figure caption should state the bandwidth or smoothing method used.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the ensemble result is measured against an external simulator, and the self-citation/benchmark dependence is an acknowledged generalization caveat, not a definitional reduction.

full rationale

ProCause's derivation chain is not circular in the sense the analyzer is asked to check. The learners (S-learner, T-learner, TARNet, ensemble) are defined in Section IV.B from standard CI architectures; none is defined in terms of the evaluation metrics (Wasserstein distance, Kendall's tau), and no parameter is fitted to the ground-truth rankings and then reported as a prediction. The central empirical claim—that the ensemble offers the most consistent absolute and relative evaluation accuracy—is obtained by training evaluators on a SimBank training sample and measuring their outputs against SimBank-generated ground truth on a fresh 1,000-case test set. The simulator is an external DGP relative to ProCause: SimBank's assumptions do not include 'the ensemble is best', so the measured superiority is not built in by construction. The paper is transparent about the relevant self-citations: Section II.B.1 says 'both are prior works of the authors, including the simulator used in our experiments below', and Section V.B says 'We use SimBank as our simulator [7]'. It also explicitly acknowledges the resulting limitation in Section VIII: 'A first key limitation is reliance on a single simulator, though we mitigated this by varying DGPs using different interventions and confounding levels.' This is a genuine external-validity concern—the ranking accuracy is only demonstrated on the authors' own simulator, and the real-data analysis tests distributional realism of generated T and Y rather than real-world ranking accuracy. The paper even flags the low power of the high-dimensional multivariate test in the Section VI.B footnote. These are evidentiary weaknesses, not circular reductions: the empirical result is conditional on SimBank's realism, but it is not equivalent to the input by definition. No load-bearing step reduces to a self-citation that itself asserts the target result. Accordingly, the circularity score is low, reflecting only the minor self-citation/benchmark dependence that limits generalization rather than the logical derivation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The method relies on standard CI assumptions and on the trustworthiness of the authors' own simulator, SimBank, as ground truth.

assumptions (3)
  • domain assumption Standard causal inference assumptions: ignorability, positivity, consistency, no interference
    Section IV.A.2 states CATE identification requires these untestable assumptions.
  • ad hoc to paper SimBank is a correct and realistic simulation of bank loan processes with accurate ground-truth outcomes
    SimBank is the authors' own simulator (cite [7]) and is the only source of ground truth; all conclusions about evaluator accuracy depend on its realism.
  • domain assumption The chosen outcome distribution families (e.g., Bernoulli, mixed distribution with sigmoidal flow) are flexible enough to approximate the true data-generating process
    Section IV.B.1 specifies distribution types; if misspecified, the evaluator's estimates will be biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ProCause: Generating Counterfactual Outcomes to Evaluate Prescriptive Process Monitoring Methods." pith.science (2026). https://pith.science/paper/ZDWLENVU

@misc{pith2026250900797,
  author       = {Pith},
  title        = {Pith review of: ProCause: Generating Counterfactual Outcomes to Evaluate Prescriptive Process Monitoring Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZDWLENVU}},
  note         = {Machine review of arXiv:2509.00797}
}
read the original abstract

Prescriptive Process Monitoring (PresPM) is the subfield of Process Mining that focuses on optimizing processes through real-time interventions based on event log data. Evaluating PresPM methods is challenging due to the lack of ground-truth outcomes for all intervention actions in datasets. A generative deep learning approach from the field of Causal Inference (CI), RealCause, has been commonly used to estimate the outcomes for proposed intervention actions to evaluate a new policy. However, RealCause overlooks the temporal dependencies in process data, and relies on a single CI model architecture, TARNet, limiting its effectiveness. To address both shortcomings, we introduce ProCause, a generative approach that supports both sequential (e.g., LSTMs) and non-sequential models while integrating multiple CI architectures (S-Learner, T-Learner, TARNet, and an ensemble). Our research using a simulator with known ground truths reveals that TARNet is not always the best choice; instead, an ensemble of models offers more consistent reliability, and leveraging LSTMs shows potential for improved evaluations when temporal dependencies are present. We further validate ProCause's practical effectiveness through a real-world data analysis, ensuring a more reliable evaluation of PresPM methods.

Figures

Figures reproduced from arXiv: 2509.00797 by the authors.

Figure 1
Figure 1. S-learner, T-learner, and TARNet architectures using neural networks; [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The experimental setup of our assessment of ProCause using a simulator. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The results of our assessment of ProCause using a simulator, displaying both absolute performance in WD (lower is better), and relative performance [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The probability density of the WD over the test set between true and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages

  1. [1]

    2171–2173

    van der Aalst, W.M.P.: Process Mining, pp. 2171–2173. Springer US (2009). https://doi.org/10.1007/978-3-662-49851-4

    doi:10.1007/978-3-662-49851-4 2009

  2. [2]

    In: Advanced Information Systems Engineering

    Bozorgi, Z.D., Dumas, M., Rosa, M.L., Polyvyanyy, A., Shoush, M., Teinemaa, I.: Learning when to treat business processes: Prescriptive process monitoring with causal inference and reinforcement learning. In: Advanced Information Systems Engineering. vol. 13901. Springer (2023)

    work page 2023

  3. [3]

    In: BPM Forum

    Branchi, S., Di Francescomarino, C., Ghidini, C., Massimo, D., Ricci, F., Ronzani, M.: Learning to act: A reinforcement learning approach to recom- mend the best next activities. In: BPM Forum. vol. 458. Springer (2022)

    work page 2022

  4. [4]

    In: 36th Conference on NeurIPS Datasets and Benchmarks Track

    Crabb´e, J., Curth, A., Bica, I., van der Schaar, M.: Benchmarking heteroge- neous treatment effect models through the lens of interpretability. In: 36th Conference on NeurIPS Datasets and Benchmarks Track. vol. 35 (2022)

    work page 2022

  5. [5]

    In: NeurIPS Track on Datasets and Benchmarks

    Curth, A., Svensson, D., Weatherall, J., van der Schaar, M.: Really doing great at estimating cate? a critical look at ml benchmarking practices in treatment effect estimation. In: NeurIPS Track on Datasets and Benchmarks. vol. 1 (2021)

    work page 2021

  6. [6]

    Information Systems 116 (2023)

    Dasht Bozorgi, Z., Teinemaa, I., Dumas, M., La Rosa, M., Polyvyanyy, A.: Prescriptive process monitoring based on causal effect estimation. Information Systems 116 (2023)

    work page 2023

  7. [7]

    SimBank: from Simulation to Solution in Prescriptive Process Monitoring

    De Moor, J., Weytjens, H., De Smedt, J., De Weerdt, J.: Simbank: from simulation to solution in prescriptive process monitoring. In: BPM Forum (2025), https://arxiv.org/abs/2506.14772, accepted for publication, to appear

    work page Pith review arXiv 2025

  8. [8]

    https://doi.org/10.4121/uuid:3926db30-f712-4394-aebc-75976070e91f

    van Dongen, B.: Bpi challenge 2012 (2012). https://doi.org/10.4121/uuid:3926db30-f712-4394-aebc-75976070e91f

    doi:10.4121/uuid:3926db30-f712-4394-aebc-75976070e91f 2012

Show all 24 references
  1. [9]

    https://doi.org/10.4121/uuid:5f3067df-f10b-45da-b98b-86ae4c7a310b

    van Dongen, B.: Bpi challenge 2017 (2017). https://doi.org/10.4121/uuid:5f3067df-f10b-45da-b98b-86ae4c7a310b

    2017 doi

  2. [10]

    Journal of the American Statistical Association 81 (1986)

    Holland, P.W.: Statistics and causal inference. Journal of the American Statistical Association 81 (1986)

    1986

  3. [11]

    In: ICML

    Huang, C.W., Krueger, D., Lacoste, A., Courville, A.: Neural autoregressive flows. In: ICML. vol. 80. PMLR (2018)

    2018

  4. [12]

    Cambridge University Press (2015)

    Imbens, G.W., Rubin, D.B.: Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press (2015)

    2015

  5. [13]

    National Academy of Sciences 116 (2017)

    K ¨unzel, S., Sekhon, J., Bickel, P., Yu, B.: Meta-learners for estimating heterogeneous treatment effects using machine learning. National Academy of Sciences 116 (2017)

    2017

  6. [14]

    Neal, B., Huang, C.W., Raghupathi, S.: Realcause: Realistic causal inference benchmarking (2021), https://openreview.net/forum?id=m28E5RN64hi

    2021

  7. [15]

    In: 39th ICML

    Parikh, H., Varjao, C., Xu, L., Tchetgen, E.T.: Validating causal inference methods. In: 39th ICML. vol. 162. PMLR (2022)

    2022

  8. [16]

    IEEE Transactions on Services Computing 16 (2023)

    Rama-Maneiro, E., Vidal, J.C., Lama, M.: Deep learning for predictive business process monitoring: Review and benchmark. IEEE Transactions on Services Computing 16 (2023)

    2023

  9. [17]

    In: 34th ICML

    Shalit, U., Johansson, F.D., Sontag, D.: Estimating individual treatment effect: generalization bounds and algorithms. In: 34th ICML. vol. 70. PMLR (2017)

    2017

  10. [18]

    K ¨unstliche Intelligenz 39 (2024)

    Shoush, M., Dumas, M.: Prescriptive process monitoring under resource constraints: A reinforcement learning approach. K ¨unstliche Intelligenz 39 (2024)

    2024

  11. [19]

    In: Process Mining Workshops

    Shoush, M., Dumas, M.: Prescriptive process monitoring under resource constraints: A causal inference approach. In: Process Mining Workshops. vol. 433. Springer (2022)

    2022

  12. [20]

    In: BPM Forum

    Shoush, M., Dumas, M.: When to intervene? prescriptive process monitoring under uncertainty and resource constraints. In: BPM Forum. vol. 458. Springer (2022)

    2022

  13. [21]

    Data & Knowledge Engineering 155 (2025)

    Shoush, M., Dumas, M.: White box specification of intervention policies for prescriptive process monitoring. Data & Knowledge Engineering 155 (2025)

    2025

  14. [22]

    In: 38th NeurIPS

    de Vassimon Manela, D., Battaglia, L., Evans, R.J.: Marginal causal flows for validation and inference. In: 38th NeurIPS. vol. 37 (2024)

    2024

  15. [23]

    ACM Trans

    Verenich, I., Dumas, M., Rosa, M.L., Maggi, F.M., Teinemaa, I.: Survey and cross-benchmark comparison of remaining time prediction methods in business process monitoring. ACM Trans. Intell. Syst. Technol. 10 (2019)

    2019

  16. [24]

    reinforcement learning

    Weytjens, H., Verbeke, W., De Weerdt, J.: Timed process interventions: Causal inference vs. reinforcement learning. In: BPM Workshops. vol. 492. Springer (2024)

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.