REVIEW 4 major objections 5 minor 32 references
Feature Attribution-Based Explainability Analysis of Deep Learning Models in Predictive Process Monitoring
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that cutting traces into control-flow segments and computing SHAP values per segment gives a middle ground between event-level and trace-level explanations for deep-learning outcome predictions.
desk verdict A practical control-flow-aware segmentation for segment-level SHAP in PPM, with honest limitations—but the real-world phase-boundary evidence is partly circular. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the control-flow-aware segmentation algorithm: for each trace, it reads off the sequence of directly-follows transition probabilities from a normalized directly-follows-relation matrix and detects change points by minimizing the sum of segment costs \(C(s)=\sum_{i=1}^{m-1} -\log T[a_i,a_{i+1}]\) (with penalty \(\gamma\) for unseen transitions) plus \(\$\beta$ k\) for the number of change points, solved with the PELT algorithm. Positions where the transition-probability profile shifts are interpreted as boundaries between process phases. These segments then become the features in a KernelSHAP computation, so each explanation unit is a locally coherent control-flow region rather than a single event.
What would settle it
Run the segmentation on a synthetic log built from two process variants that share the same activity labels but use different transition probabilities, with phase boundaries known in advance; if the pooled directly-follows matrix places change points that do not match the true boundaries, or if the entropy reduction comes only from splitting traces into very short segments, the phase-boundary interpretation is refuted. A simpler check is to shuffle the transition-probability signal within each trace and see whether the discovered segments and segment-level SHAP values change as much as they do for the true ordering.
Extended reading notes
Core claim
The central claim is that segment-level SHAP explanations computed over control-flow-aware segments are a useful middle ground between event-level and trace-level explanations for predictive process monitoring. The method defines a segment by identifying change points in a trace where the empirical directly-follows transition probabilities shift, using a negative-log-probability cost with a penalty for extra boundaries (via the PELT algorithm). It then treats each segment as one coalition in KernelSHAP, with a baseline built from the training distribution. On the synthetic SimBank log the detected boundaries agree with ground-truth change points (RandIndex above 0.9, F1 scores of 0.885 and 0.874); on BPIC17, top segmentation patterns cover large fractions of accepted and canceled cases, entropic relevance drops from 37.81 to 6.68, and perturbation-based faithfulness metrics (relative prediction change is high for top-attributed segments, low for low-attributed ones) show the attributed segments genuinely influence predictions. The paper also reports the boundary condition: in a high-variability log like BPIC15, the same algorithm fragments traces into many case-specific segments, so the phase interpretation weakens.
Load-bearing premise
The method assumes that the places where the usual next-activity probabilities change mark real boundaries between process stages; if those changes are noise or arbitrary variability, the segments do not have the process meaning the explanations claim.
Editorial extensions
If this is right
- On logs with moderate control-flow variability, analysts can see which phase of a case pushes the prediction—for example, repeated offer creation pushes BPIC17 predictions toward cancellation.
- Segment-level explanations are less faithful than event-level ones but more comprehensible, because they group coherent events and reduce interleaving of high- and low-attribution events.
- Because segments act as coalitions in SHAP, the number of explanation features drops from trace length to number of segments, cutting the computational cost of attribution.
- Segmentation can be validated against ground truth when known change points exist, and the same evaluation framework transfers to other logs via pattern coverage and entropic relevance.
- The benefit is conditional: high-variability logs such as BPIC15 yield case-specific fine-grained segments, so conclusions at the segment level only generalize for logs with identifiable phase structure.
Reading between the lines
- A natural extension is to feed the detected segment boundaries, or the counts and types of segments, back into the predictive model as engineered features; that would test whether the phase structure itself carries outcome signal beyond the activities.
- The same directly-follows cost could guide counterfactual generation, treating segment boundaries as the points where an intervention could switch a predicted outcome; this application is not explored in the paper.
- Because the directly-follows-relation matrix is pooled across all cases, the method may conflate different process variants; estimating variant-specific transition probabilities before segmentation could restore phase meaning in high-variability logs.
- A prefix-length sensitivity analysis would clarify whether the discovered phases are stable as a case progresses, since the 30-event prefix used here may cut off later decisive phases.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a method for segment-level SHAP explanations of deep LSTM outcome predictors in predictive process monitoring. It introduces a control-flow-aware segmentation algorithm that uses directly-follows transition probabilities and the PELT change-point detection algorithm, defining segment cost as the sum of negative log directly-follows probabilities and selecting change points via a penalized objective. Segment-level SHAP values are then computed with KernelSHAP over the resulting segments. The method is evaluated on a synthetic SimBank dataset with two hand-picked cases with known change points, and on BPIC15 and BPIC17, using segmentation quality metrics (RandIndex, F1, entropic relevance) and perturbation-based faithfulness metrics (RPCI/RPCU), together with qualitative analyses of BPIC17 segmentation patterns. The paper claims that segment-level explanations offer a useful middle ground between event-level and trace-level explanations and that the method reveals recurring trace segments associated with accepted and canceled cases.
Significance. If the central claim were established, the paper would make a useful contribution by addressing a real gap in PPM explainability: event-level attributions are fragmented and trace-level attributions lose control-flow context, while segment-level attributions can preserve locally coherent process behavior and reduce computational cost. The paper has notable strengths: the code is made publicly available, the synthetic evaluation uses explicit ground-truth change points, the method is compared against event-level and distribution-based segmentation baselines, and the faithfulness evaluation uses perturbation-based metrics. The writing is generally clear and the limitation for high-variability logs is acknowledged. However, the evidence for the central claim—that the discovered segments correspond to meaningful process phases—is currently weak and partly circular, so the main contribution is not yet convincingly validated.
major comments (4)
- [Section 6.1, Table 2, and Eq. (1)-(3)] The entropic relevance (ER) reduction from 37.81 to 6.68 for BPIC17 is not independent evidence that the segmentation captures process phases. ER is defined in terms of the same directly-follows transition probabilities that the segmentation cost in Eq. (1) minimizes, and ER is evaluated per segment using transitions within that segment, so finer segmentation trivially lowers ER. A single-transition segment has zero entropy. The reported reduction is therefore largely mechanical and overlaps with the optimization objective. Please provide a non-circular validation, for example synthetic logs with known phases, segmentation evaluated against a held-out transition matrix, or a comparison that controls for the number of segments.
- [Section 5.1, Table 1] The synthetic evaluation uses only two hand-picked samples selected precisely because their change points are clearly defined. This does not provide sufficient evidence that the DFR-shift criterion reliably identifies process phases across realistic process variants, noise levels, or unseen transitions. There is also no comparison against the distribution-based baseline or a per-event baseline on the same synthetic data, and no sensitivity analysis with respect to the penalty parameter beta. Please extend the synthetic evaluation to a larger, systematically varied set of traces and include baseline comparisons.
- [Section 6.2, Table 3, and Section 7] The RPC evaluation validates the faithfulness of SHAP values conditional on the chosen segments, but it does not validate that the segments themselves are meaningful process phases. Since the top-k segments are selected by the same SHAP values whose faithfulness is being tested, RPCI/RPCU is an internal consistency check and cannot support the statement in Section 7 that the identified regions are 'influential rather than arbitrary' in the process-level sense. Independent validation of phase boundaries is needed before segment-level SHAP values can be claimed to carry process-level meaning.
- [Section 4.1 and Section 7] The paper assumes that positions where the directly-follows cost changes 'can be interpreted as boundaries between different process phases.' Section 7 concedes that high-variability logs such as BPIC15 produce fine-grained, case-specific segmentation, which is the failure mode of this assumption. The scope of the central claim should therefore be stated more carefully: at present, the evidence supports only logs with moderate control-flow variability, and the paper should either test another such log or explicitly restrict its claims in the abstract and conclusion.
minor comments (5)
- [Table 1 and Section 5.1] The activity names are inconsistent: Section 5.1 refers to 'skip_contact_hq' while Table 1 labels the sample as 'skipped_hq'; please align the notation.
- [Section 5.2] The choice of 40% trace-length coverage for the top-k segments is not justified; please report sensitivity to this threshold or provide a rationale.
- [Table 3] The table does not include any statistical comparison between strategies; paired tests or confidence intervals would help assess whether the observed RPCI/RPCU differences are meaningful.
- [Section 6.3, Figure 3] The comparison of event-level and segment-level SHAP values in Figure 3 is described only briefly; additional caption detail on how the segment values are derived from event values would improve reproducibility.
- [Throughout] There are several minor grammatical issues and unclear cross-references (e.g., 'the former contains administrative processes' after the BPIC15/BPIC17 introduction could be misread); a thorough language edit is recommended.
Circularity Check
Real-world phase-boundary evidence is circular: ER reduction restates the DFR segmentation objective; only synthetic ground truth is independent.
-
self definitional
[Sections 4.1 (Eqs. 1-2), 5.2, 6.1 (Table 2)]
"To assess whether a candidate segment is coherent, we assign it a cost based on the directly-follows probabilities. ... C(s) = sum_{i=1}^{m-1} c(a_i,a_{i+1}) ... c(a_i,a_{i+1}) = -log T[a_i,a_{i+1}] if T[a_i,a_{i+1}]>0, gamma otherwise. ... ER quantifies the number of bits needed to encode traces based on the directly-follows transition probabilities within each segment. Segments with more deterministic transitions yield lower ER scores, indicating that the segmentation captures coherent patterns that require fewer bits to describe."
The cost minimized by PELT is the sum of negative log directly-follows probabilities, and ER is defined as the encoding cost under those same directly-follows transition probabilities within each segment. The reported ER drop on BPIC17 (37.81 to 6.68) therefore restates the segmentation objective rather than validating that the detected change points are process phases. Because ER is computed per segment from within-segment transitions, any split at a rare transition lowers the metric mechanically: a single-event segment has no transitions and zero entropy. The conclusion that moderate-variability logs exhibit 'clearly identifiable phases' thus rests on the synthetic ground truth, not on the real-world ER evidence.
full rationale
The central circularity is confined to the real-world segmentation evaluation. The algorithm's objective (Eqs. 1-2) is a sum of negative log directly-follows probabilities, and the entropic relevance metric used in Table 2 is defined from the same DFR transition probabilities within each segment; the reported ER reduction is therefore the optimization objective resurfacing as a validation number, not independent evidence that the change points are process-phase boundaries. Since ER is computed per segment, a finer partition lowers it automatically, with single-event segments contributing zero transition entropy. The synthetic SimBank experiments provide a genuinely independent check: RandIndex, Precision, Recall and F1 against planted change points (Table 1) do not depend on the DFR cost and show strong recovery, so the method is not wholly circular. The SimBank reference is co-authored, but it is a generative benchmark with externally fixed change points rather than a self-validating theorem, so it supplies real support. The RPC faithfulness evaluation, by contrast, selects the segments to perturb using the SHAP values under test; this is an internal consistency check rather than external confirmation of process meaning. The paper's Section 7 acknowledgment that high-variability logs (BPIC15) produce fragmented, case-specific segmentation is consistent with this reading: the phase-boundary interpretation is an assumption about DFR shifts, and the real-world evidence for it collapses into the objective being optimized. Overall, partial circularity: the real-world phase-boundary claim rests on a metric that reduces to the segmentation cost, while the method retains independent support only from the synthetic ground truth.
Assumptions & free parameters
free parameters (4)
- PELT penalty beta =
not reported (selected by grid search or AIC)
- gamma (unseen transition penalty) =
not reported
- Prefix length =
30 events
- Top-k coverage for RPC =
40% of trace
assumptions (5)
- domain assumption Directly-follows transition probabilities estimated from the event log reflect true control-flow regularities.
- domain assumption Shifts in DFR probabilities mark meaningful process phase boundaries.
- domain assumption KernelSHAP with a mean/mode baseline yields valid attributions for dependent sequential features.
- standard math PELT exactly minimizes the penalized cost in Eq. (3).
- domain assumption RPC is a valid faithfulness measure for feature attributions.
Cite this review
Pith. "Pith review of Feature Attribution-Based Explainability Analysis of Deep Learning Models in Predictive Process Monitoring." pith.science (2026). https://pith.science/paper/NFEAXWS6
@misc{pith2026260717783,
author = {Pith},
title = {Pith review of: Feature Attribution-Based Explainability Analysis of Deep Learning Models in Predictive Process Monitoring},
year = {2026},
howpublished = {\url{https://pith.science/paper/NFEAXWS6}},
note = {Machine review of arXiv:2607.17783}
}
read the original abstract
Predictive process monitoring supports the optimization and control of operational business processes by forecasting the future state or outcome of ongoing cases. While deep neural networks have achieved strong performance for these tasks by modeling sequential dependencies in event logs, their black-box nature limits trust and practical adoption. Feature attribution methods are often used to address this, but applying them directly poses a dilemma: event-level attributions impose high computational complexity for long traces, while explanations based on aggregated trace representations often fail to capture the underlying control-flow dynamics. To address this issue, we propose a local post-hoc explainability method for deep neural networks in outcome prediction. The method relies on a control-flow-aware segmentation algorithm that partitions a trace into meaningful segments and supports the computation of segment-level SHAP explanations. This makes it possible to identify which parts of a trace influence a prediction and which change points steer the case toward the predicted outcome. We assess the proposed segmentation method on a synthetic dataset with known process logic, where meaningful change points can be explicitly verified, and we demonstrate its usefulness on real-world event logs from a loan application process and an administrative process of a Dutch municipality.
Figures
Reference graph
Works this paper leans on
-
[1]
Artificial Intelli- gence298(2021).https://doi.org/10.1016/j.artint.2021.103502
Aas, K., Jullum, M., Løland, A.: Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artificial Intelli- gence298(2021).https://doi.org/10.1016/j.artint.2021.103502
arXiv 2021
-
[2]
In: Proceedings of the 36th International Conference on Neural In- formation Processing Systems
Agarwal, C., Krishna, S., Saxena, E., Pawelczyk, M., Johnson, N., Puri, I., Zitnik, M., Lakkaraju, H.: Openxai: towards a transparent evaluation of post hoc model explanations. In: Proceedings of the 36th International Conference on Neural In- formation Processing Systems. NIPS ’22, Curran Associates Inc., Red Hook, NY, USA (2022)
work page 2022
-
[3]
In: XAI World Conference (2025)
Baer, G., Grau, I., Zhang, C., Van Gorp, P.: Class-dependent perturbation effects in evaluating time series attributions. In: XAI World Conference (2025)
work page 2025
-
[4]
In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining
Bento, J.a., Saleiro, P., Cruz, A.F., Figueiredo, M.A., Bizarro, P.: Timeshap: Explaining recurrent models through sequence perturbations. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. p. 2565–2573. KDD ’21, Association for Computing Machinery, New York, NY, USA (2021).https://doi.org/10.1145/3447548.3467166
arXiv 2021
-
[5]
Data Mining and Knowledge Discov- ery (2025).https://doi.org/10.1007/s10618-025-01117-3
Buliga, A., Di Francescomarino, C., Ghidini, C., Donadello, I., Maggi, F.M.: Guid- ing the generation of counterfactual explanations through temporal background knowledge for predictive process monitoring. Data Mining and Knowledge Discov- ery (2025).https://doi.org/10.1007/s10618-025-01117-3
-
[6]
Explaining Time Series Predictions with Dynamic Masks
Crabbé, J., Van Der Schaar, M.: Explaining time series predictions with dynamic masks. In: ICML (2021).https://doi.org/10.48550/arXiv.2106.05303
work page Pith review arXiv doi:10.48550/arxiv.2106.05303 2021
-
[7]
In: BPM (2025).https://doi.org/ 10.1007/978-3-032-02929-4_10
De Moor, J., Weytjens, H., De Smedt, J., De Weerdt, J.: Simbank: from simulation to solution in prescriptive process monitoring. In: BPM (2025).https://doi.org/ 10.1007/978-3-032-02929-4_10
-
[8]
Deldari, S., Smith, D.V., Sadri, A., Salim, F.: Espresso: Entropy and shape aware time-series segmentation for processing heterogeneous sensor data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies4(3) (2020).https://doi.org/10.1145/3411832
Show all 32 references
-
[9]
In: ICDM (2022)
Doddaiah, R., Parvatharaju, P., Rundensteiner, E., Hartvigsen, T.: Class-specific explainability for deep time series classifiers. In: ICDM (2022)
2022
-
[10]
Algorithms (2022)
El-khawaga, G., Abu-Elkheir, M., Reichert, M.: Xai in the context of predictive process monitoring: An empirical analysis framework. Algorithms (2022)
2022
-
[11]
In: ICPM (2020)
Galanti, R., Coma-Puig, B., de Leoni, M., Carmona, J., Navarin, N.: Explainable predictive process monitoring. In: ICPM (2020)
2020
-
[12]
Cell Reports Physical Science (2026).https://doi.org/10.1016/j.xcrp
Hamed, H., Reis, A.C., Choobar, B.G., Pang, Q., Killick, R., Safari, M.: Change- point detection as a light data-driven approach to battery state-of-health predic- tion. Cell Reports Physical Science (2026).https://doi.org/10.1016/j.xcrp. 2026.103157
2026
-
[13]
In: DAS- FAA (2024)
Jiang, G., Zhuang, F., Song, B., Zhu, Y., Sun, Y., Wang, W., Wang, D.: Seqshap: subsequence level shapley value explanations for sequential predictions. In: DAS- FAA (2024)
2024
-
[14]
Journal of the American Statistical Association 107(500) (2012).https://doi.org/10.1080/01621459.2012.737745 Feature Attribution-Based Explainability Analysis in PPM 17
Killick, R., Fearnhead, P., Eckley, I.A.: Optimal detection of changepoints with a linear computational cost. Journal of the American Statistical Association 107(500) (2012).https://doi.org/10.1080/01621459.2012.737745 Feature Attribution-Based Explainability Analysis in PPM 17
2012
-
[15]
Journal of the Royal Statistical Society Series B: Statistical Methodology86(2) (2024)
Li, J., Fearnhead, P., Fryzlewicz, P., Wang, T.: Automatic change-point detection in time series via deep learning. Journal of the Royal Statistical Society Series B: Statistical Methodology86(2) (2024)
2024
-
[16]
In: ICLR (2024)
Liu, Z., Zhang, Y., Wang, T., Wang, Z., Luo, D., Du, M., Wu, M., Wang, Y., Chen, C., Fan, L., Wen, Q.: Explaining time series via contrastive and locally sparse perturbations. In: ICLR (2024)
2024
-
[17]
In: Proceedings of the 31st International Conference on Neural Information Processing Systems
Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. p. 4768–4777. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017).https://doi.org/10.48550/arXi...
-
[18]
Journal of Biomedical Informatics144, 104438 (2023)
Nayebi, A., Tipirneni, S., Reddy, C.K., Foreman, B., Subbian, V.: Windowshap: An efficient framework for explaining time-series classifiers based on shapley values. Journal of Biomedical Informatics144, 104438 (2023)
2023
-
[19]
Information Systems139(2026).https://doi.org/10.1016/j.is.2026
Peeperkorn, J., De Smedt, J., De Weerdt, J.: Model-driven stochastic trace clus- tering. Information Systems139(2026).https://doi.org/10.1016/j.is.2026. 102697
2026 doi
-
[20]
In: ICPM (2020).https: //doi.org/10.1109/ICPM49681.2020.00024
Polyvyanyy, A., Moffat, A., García-Bañuelos, L.: An entropic relevance measure for stochastic conformance checking in process mining. In: ICPM (2020).https: //doi.org/10.1109/ICPM49681.2020.00024
2020
-
[21]
In: BPM (2020).https: //doi.org/10.1007/978-3-030-58638-6_9
Rizzi, W., Di Francescomarino, C., Maggi, F.M.: Explainability in predictive pro- cess monitoring: When understanding helps improving. In: BPM (2020).https: //doi.org/10.1007/978-3-030-58638-6_9
2020 doi
-
[22]
In: ECML-PKDD (2021).https://doi.org/10.1007/978-3-030-93736-2_40
Schlegel, U., Vo, D.L., Keim, D.A., Seebacher, D.: Ts-mule: Local interpretable model-agnostic explanations for time series forecast models. In: ECML-PKDD (2021).https://doi.org/10.1007/978-3-030-93736-2_40
2021 doi
-
[23]
The Bell system tech- nical journal27(3) (1948)
Shannon, C.E.: A mathematical theory of communication. The Bell system tech- nical journal27(3) (1948)
1948
-
[24]
In: International Conference on Artificial Intelligence and Statistics (2022)
Sivill, T., Flach, P.: Limesegment: Meaningful, realistic time series explanations. In: International Conference on Artificial Intelligence and Statistics (2022)
2022
-
[25]
IEEE TSC (2025).https://doi.org/10.1109/TSC.2025.3609837
Stevens, A., Ouyang, C., De Smedt, J., Moreira, C.: Generating feasible and plausi- ble counterfactual explanations for outcome prediction of business processes. IEEE TSC (2025).https://doi.org/10.1109/TSC.2025.3609837
2025
- [26]
-
[27]
In: CAiSE (2017).https://doi.org/10.1007/ 978-3-319-59536-8_30
Tax, N., Verenich, I., La Rosa, M., Dumas, M.: Predictive business process moni- toring with lstm neural networks. In: CAiSE (2017).https://doi.org/10.1007/ 978-3-319-59536-8_30
2017
-
[28]
ACM TKDD13(2) (2019).https: //doi.org/10.1145/3301300
Teinemaa, I., Dumas, M., Rosa, M.L., Maggi, F.M.: Outcome-oriented predictive process monitoring: Review and benchmark. ACM TKDD13(2) (2019).https: //doi.org/10.1145/3301300
2019 doi
-
[29]
Signal processing167(2020).https://doi.org/10.1016/j
Truong, C., Oudre, L., Vayatis, N.: Selective review of offline change point de- tection methods. Signal processing167(2020).https://doi.org/10.1016/j. sigpro.2019.107299
2020
-
[30]
Nature Machine Intelligence5(3) (2023).https://doi.org/10.1038/s42256-023-00620-w
Turbé, H., Bjelogrlic, M., Lovis, C., Mengaldo, G.: Evaluation of post-hoc inter- pretability methods in time-series classification. Nature Machine Intelligence5(3) (2023).https://doi.org/10.1038/s42256-023-00620-w
2023 doi
-
[31]
In: ICSOC (2021)
Velmurugan, M., Ouyang, C., Moreira, C., Sindhgatta, R.: Evaluating stability of post-hoc explanations for business process predictions. In: ICSOC (2021)
2021
-
[32]
lstm (with at- tention)
Weytjens, H., De Weerdt, J.: Process outcome prediction: Cnn vs. lstm (with at- tention). In: BPM (2020).https://doi.org/10.1007/978-3-030-66498-5_24
2020 doi
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.