REVIEW 3 major objections 6 minor 28 references
An Uncertainty-Aware ED-LSTM for Probabilistic Suffix Prediction
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Probabilistic suffix prediction—sampling many futures from an uncertainty-aware LSTM—can beat the single most-likely suffix and gives calibrated remaining-time distributions.
desk verdict Useful application-level combination of known uncertainty techniques, but the headline calibration claim is contradicted by the paper's own PIT plots. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the U-ED-LSTM, an encoder-decoder LSTM in which each continuous event attribute is predicted as a mean and a log-variance pair and each categorical attribute as mean logits and a log-variance vector; the training losses are the negative log-likelihoods of those distributions, so the model learns an input-dependent noise level. Epistemic uncertainty is injected by Monte Carlo dropout with a fresh variational dropout mask per rollout, while aleatoric uncertainty is captured by the learned per-step variances. The MC suffix sampling algorithm then autoregressively draws continuous values from Normal distributions (or Log-Normal distributions for strictly positive durations) and categorical classes from softmaxed Gaussian logits, stopping at an end-of-sequence token or a maximum length, and 1000 such rollouts approximate the suffix posterior. The paper also tested a Log-Normal variant as Setting 3 and reports that it is sensitive to extreme sampled outliers.
What would settle it
Build a small synthetic event log with a known generative process so the exact conditional suffix distribution can be computed for every prefix, then compare the model's 1000-sample predictive distribution to the truth with a probability integral transform (PIT) histogram over many prefixes; a systematic U-shape or slope would show that the per-step distributional assumptions do not capture the true posterior.
Extended reading notes
Core claim
The paper claims that suffix prediction should be framed as distribution prediction rather than point prediction. Its central empirical discovery is that on four real-life and one artificial event log, the mean over 1000 Monte-Carlo-sampled suffixes can outperform the autoregressively decoded most-likely suffix on remaining-time mean absolute error (MAE) and suffix-length MAE, especially for short prefixes and long suffixes, while the model's remaining-time predictions are calibrated well enough that probability integral transform (PIT) plots show no systematic location bias. For event-label accuracy, the most-likely suffix is usually better on low-variability logs, whereas the probabilistic suffix is competitive or better on high-variability logs such as Sepsis and BPIC-17. The paper concludes from these results that probabilistic suffix prediction is effective and can be used for advanced applications such as confidence interval estimation.
Load-bearing premise
The method assumes that the per-event Normal or Log-Normal distributions and the per-class logit distributions, combined through 1000 autoregressive rollouts, honestly represent the true distribution of possible suffixes; if single-step errors compound during sampling, the resulting distribution can drift away from reality.
Editorial extensions
If this is right
- Process analysts can extract confidence intervals for remaining time directly from the 1000 sampled suffixes rather than treating one predicted trace as ground truth.
- Probabilistic aggregation is most valuable in high-variability settings, where short prefixes and long suffixes make the greedy most-likely suffix least reliable.
- Because the decoder outputs distributions for every event attribute, the same sampled suffix set can support questions about activity order, suffix length, and continuous attributes without retraining separate models.
- The PIT calibration procedure gives process-mining teams a concrete way to test whether a predictor's uncertainty estimates are trustworthy before using them in decisions.
Reading between the lines
- Editorial inference: if the mean over sampled suffixes beats the mode mainly because remaining-time distributions are skewed, probabilistic aggregation should pay off most in human-driven, high-variance logs and least in fully automated, low-variance ones.
- Editorial inference: the Setting 3 log-normal outliers suggest a direct testable extension—clip predicted variances during sampling or replace the log-normal head with a quantile or mixture head, then rerun the same MAE and PIT comparisons.
- Editorial inference: the empirical cumulative distribution functions behind the PIT plots could be queried directly for operational statements such as 'probability this case exceeds a 30-day deadline,' a use the paper names but does not evaluate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes probabilistic suffix prediction for business processes, in which an encoder-decoder LSTM (U-ED-LSTM) is trained with MC dropout for epistemic uncertainty and learned loss attenuation for aleatoric uncertainty, and a Monte Carlo suffix sampling algorithm generates an approximate distribution over suffixes. The approach is evaluated on four real-life and one artificial event log under three hyperparameter settings, comparing aggregated probabilistic predictions with greedy most-likely suffix predictions on suffix length, Damerau-Levenshtein similarity, and remaining-time MAE, and assessing calibration of remaining-time predictions via PIT plots. The central claims are that probabilistic suffix prediction can outperform most-likely suffix prediction, that the U-ED-LSTM has reasonable predictive performance, and that the model's predictions are well calibrated.
Significance. If the calibration claim were supported, the paper would make a useful contribution to predictive process monitoring: a calibrated distribution over suffixes would enable confidence-interval estimation and better handling of process variability. The manuscript has concrete strengths: it provides a public implementation, evaluates on multiple real-world datasets, reports three hyperparameter settings, and compares against existing suffix-prediction methods. However, the calibration evidence is central to the paper's novelty, and the paper's own PIT analysis contradicts the stated conclusions. The value of the contribution therefore depends on resolving an internal inconsistency rather than on a disagreement with existing consensus.
major comments (3)
- [Abstract; Section 4.2] The calibration claim is contradicted by the paper's own PIT results. Section 4.2 defines a uniform PIT density as perfect calibration, a U-shape as underdispersion, a bell shape as overdispersion, and a slope or boundary peak as systematic bias. Figures 4 and 5 show a bell shape for Helpdesk, peaks at 1 for Sepsis and BPIC-17, a peak at 0 for PCR, and a peak near 0.2 for Repair, all of which the paper itself classifies as miscalibration. Nevertheless, the text states that 'our approach shows no systematic bias across all data sets' and the abstract concludes that 'the model's predictions are well calibrated.' This internal inconsistency directly undermines the central claim that probabilistic suffix prediction provides a calibrated distribution suitable for confidence-interval estimation, and it must be resolved before the headline conclusion can be accepted.
- [Section 3.1; Algorithm 1] There is an inconsistency between the dropout used for training and the dropout used for sampling. Section 3.1 states that variational dropout is applied in the U-ED-LSTM, with the same mask applied across time steps, and Figure 1 indicates this. Algorithm 1, however, samples the decoder weights with 'NaiveDropout(W_dec,p)' at each step (line 10). Since MC dropout as a Bayesian approximation relies on using the same dropout mechanism during inference as during training, and since naive dropout is known to be ineffective in recurrent networks, the epistemic-uncertainty quantification is not implemented as described. The authors need to correct the algorithm or justify why this discrepancy does not affect the uncertainty estimates.
- [Section 4.1, Table 4] The Setting 3 results illustrate that the assumed sampling distributions are fragile: the log-normal variant produces extreme outliers, e.g., a remaining-time sum MAE of 15,262 seconds for PCR versus roughly 160 seconds in Settings 1 and 2, and the text acknowledges that such outliers 'greatly influence the aggregated mean of all sampled suffixes.' This is not a peripheral issue, because the MC suffix sampler draws from these per-event distributions; pathological draws imply that the approximated suffix distribution is not reliable. This observation reinforces the calibration problem identified in the previous major comment and should be addressed explicitly in the evaluation.
minor comments (6)
- [Section 4.1] There is an unresolved placeholder 'Figure??' in the BPIC-17 discussion, and the accompanying figure reference should be corrected.
- [Table 3] Several numeric entries are concatenated without separators (e.g., '13.741.481.05' and '0.10.35 0.830.86'), which makes the table very difficult to read and should be reformatted.
- [Abstract; Section 1] The list in the abstract contains grammatical issues: 'probabilistic suffix prediction can outperform most likely suffix prediction, the U-ED-LSTM has reasonable predictive performance, and ii)' is not a well-formed parallel list, and the same problem appears in the introduction.
- [Algorithm 1] The symbols 'EOS' and the function 'GetActivity' are used without prior definition; the termination condition is unclear if the dataset does not contain an EOS activity, and the authors should state how the stopping criterion is defined for datasets without an explicit end marker.
- [Section 4.2] The PIT formula is not numbered, and the wording 'no systematic bias' appears immediately before a list of biased PIT shapes; the text should be aligned with the evidence presented in Figures 4 and 5.
- [Figures 2 and 3] The artificial dataset is called 'Repair' in the text and tables but 'Repair Shop' in Figure 3; the naming should be harmonized, and the y-axis label 'Remaining time MAE (days)' in Figure 3(b) should be checked against the Repair dataset's time unit.
Circularity Check
No circularity: the paper is an empirical benchmark; the calibration overstatement is an internal-consistency problem, not a circular derivation.
full rationale
The paper makes no derivational claim that reduces to its own inputs by construction or by self-citation. The U-ED-LSTM is trained using standard negative log-likelihood losses (Eqs. 3-5) adapted from external references, and the MC suffix sampling algorithm autoregressively draws from the learned per-event distributions against held-out ground-truth suffixes on external event logs. No fitted parameter is renamed as a prediction, no uniqueness theorem from the authors is invoked, and the loss functions do not encode the evaluation metrics. The calibration analysis uses PIT values computed on test data; the conclusion that predictions are 'well calibrated' is an interpretive statement, not a quantity forced by the training objective. One substantive non-circularity issue is that the paper's own PIT taxonomy (uniform = calibrated; U-shape, bell-shape, peaks, and slopes = miscalibration) conflicts with the non-uniform PIT histograms in Figs. 4-5, undermining the abstract's calibration claim. This is a correctness/falsification concern about the evidence, not a circularity of the derivation chain.
Assumptions & free parameters
free parameters (10)
- MC dropout probability p =
0.1
- Sequence length S =
5
- Teacher forcing probability =
0.8 initial, decayed after 20% of epochs
- Hidden size =
128
- Number of MC samples T =
1000
- Learning rate =
1e-4 to 1e-6 per dataset
- Number of training epochs =
200 for setting 1, 100 for settings 2 and 3
- L2 regularization lambda =
1e-4
- Embedding dimension D =
min(600, round(1.6*(K+2)^0.56))
- Maximum case length padding cutoff =
Excludes top 1.5% longest cases
assumptions (5)
- domain assumption MC dropout is a valid variational approximation to the posterior over LSTM weights.
- domain assumption Per-event continuous attributes follow Normal or Log-Normal noise and categorical logits follow independent Normal noise.
- ad hoc to paper Autoregressive sampling with T=1000 approximates the true suffix distribution.
- domain assumption The case-level 65/15/20 split yields independent and identically distributed test cases.
- domain assumption Pooled PIT values can be read as a single calibration diagnostic.
Cite this review
Pith. "Pith review of An Uncertainty-Aware ED-LSTM for Probabilistic Suffix Prediction." pith.science (2026). https://pith.science/paper/5MD2PJJQ
@misc{pith2026250521339,
author = {Pith},
title = {Pith review of: An Uncertainty-Aware ED-LSTM for Probabilistic Suffix Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/5MD2PJJQ}},
note = {Machine review of arXiv:2505.21339}
}
read the original abstract
Suffix prediction of business processes forecasts the remaining sequence of events until process completion. Current approaches focus on predicting the most likely suffix, representing a single scenario. However, when the future course of a process is subject to uncertainty and high variability, the expressiveness of such a single scenario can be limited, since other possible scenarios, which together may have a higher overall probability, are overlooked. To address this limitation, we propose probabilistic suffix prediction, a novel approach that approximates a probability distribution of suffixes. The proposed approach is based on an Uncertainty-Aware Encoder-Decoder LSTM (U-ED-LSTM) and a Monte Carlo (MC) suffix sampling algorithm. We capture epistemic uncertainties via MC dropout and aleatoric uncertainties as learned loss attenuation. This technical report presents a comprehensive evaluation of the probabilistic suffix prediction approach's predictive performance and calibration under three different hyperparameter settings, using four real-life and one artificial event log. The results show that: i) probabilistic suffix prediction can outperform most likely suffix prediction, the U-ED-LSTM has reasonable predictive performance, and ii) the model's predictions are well calibrated.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Fieguth, Xiaochun Cao, Abbas Khosravi, U
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul W. Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges.Inf. Fusion, 76:243–297, 2021. doi: 10.1016/J.INFFUS.20...
-
[2]
Mixture density networks
Christopher M Bishop. Mixture density networks. 1994
1994
-
[3]
Learning accurate LSTM models of business processes
Manuel Camargo, Marlon Dumas, and Oscar González Rojas. Learning accurate LSTM models of business processes. InBusiness Process Management - BPM, pages 286–302, 2019. doi: 10.1007/978-3-030-26619-6\_19
-
[4]
Paolo Ceravolo, Marco Comuzzi, Jochen De Weerdt, Chiara Di Francescomarino, and Fab- rizio Maria Maggi. Predictive process monitoring: concepts, challenges, and future research directions.Process Science, 1(1):2, 2024
work page 2024
-
[5]
Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. InInternational Conference on Machine Learning - ICML, pages 794–803, 2018
work page 2018
-
[6]
Predicting process behaviour using deep learning.Decis
Joerg Evermann, Jana-Rebecca Rehse, and Peter Fettke. Predicting process behaviour using deep learning.Decis. Support Syst., 100:129–140, 2017. doi: 10.1016/J.DSS.2017.04.003
-
[7]
Dropoutasabayesianapproximation: Representingmodel uncertainty in deep learning
YarinGalandZoubinGhahramani. Dropoutasabayesianapproximation: Representingmodel uncertainty in deep learning. InInternational Conference on Machine Learning - ICML, pages 1050–1059, 2016. URLhttp://proceedings.mlr.press/v48/gal16.html
work page 2016
-
[8]
A theoretically grounded application of dropout in recurrent neural networks
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. InAdvances in Neural Information Processing Systems - 20 NeurIPS, pages 1019–1027, 2016. URLhttps://proceedings.neurips.cc/paper/2016/ hash/076a0c97d09cf1a0ec3e19c7f2529f2b-Abstract.html
work page 2016
Show all 28 references
-
[9]
A direct data aware LSTM neural network architecture for complete remaining trace and runtime prediction.IEEE Trans
Björn Rafn Gunnarsson, Seppe vanden Broucke, and Jochen De Weerdt. A direct data aware LSTM neural network architecture for complete remaining trace and runtime prediction.IEEE Trans. Serv. Comput., 16(4):2330–2342, 2023. doi: 10.1109/TSC.2023.3245726
2023
-
[10]
Long Short-Term Memory.Neural Computation, 9(8):1735–1780, 1997
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory.Neural Computation, 9(8):1735–1780, 1997. doi: 10.1162/neco.1997.9.8.1735
1997 doi
-
[11]
Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods.Mach
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods.Mach. Learn., 110(3):457–506, 2021. doi: 10.1007/S10994-021-05946-3
2021 doi
-
[12]
What uncertainties do we need in bayesian deep learn- ing for computer vision? InAdvances in Neural Information Processing Systems - NeurIPS, pages 5574–5584, 2017
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learn- ing for computer vision? InAdvances in Neural Information Processing Systems - NeurIPS, pages 5574–5584, 2017. URLhttps://proceedings.neurips.cc/paper/2017/ hash/2650d6089a6d640c5e85b2b88265dc2b-...
2017
-
[13]
van Dongen
István Ketykó, Felix Mannhardt, Marwan Hassani, and Boudewijn F. van Dongen. What aver- ages do not tell: predicting real life processes with sequential deep learning. InACM/SIGAPP Symposium on Applied Computing - SAC, pages 1128–1131, 2022. doi: 10.1145/3477314. 3507179
2022 doi
-
[14]
Distributional regression for data analysis.Annual Review of Statistics and Its Application, 11, 2024
Nadja Klein. Distributional regression for data analysis.Annual Review of Statistics and Its Application, 11, 2024. doi: 10.1146/annurev-statistics-040722-053607
2024 doi
-
[15]
Mm-pred: A deep predictive model for multi attribute event sequence
Li Lin, Lijie Wen, and Jianmin Wang. Mm-pred: A deep predictive model for multi attribute event sequence. InSIAM International Conference on Data Mining - SDM, pages 118–126,
-
[16]
Pre- dictive monitoring of business processes
Fabrizio Maria Maggi, Chiara Di Francescomarino, Marlon Dumas, and Chiara Ghidini. Pre- dictive monitoring of business processes. InAdvanced Information Systems Engineering - CAiSE, pages 457–472. Springer, 2014. doi: 10.1007/978-3-319-07881-6\_31
2014 doi
-
[17]
Augmenting post-hoc explanations for predictive process monitoring with uncertainty quantification via conformalized monte carlo dropout.Data Knowl
Nijat Mehdiyev, Maxim Majlatow, and Peter Fettke. Augmenting post-hoc explanations for predictive process monitoring with uncertainty quantification via conformalized monte carlo dropout.Data Knowl. Eng., 156:102402, 2025. doi: 10.1016/J.DATAK.2024.102402
2025
-
[19]
Bayesian network based predictions of business processes
Stephen Pauwels and Toon Calders. Bayesian network based predictions of business processes. InBusiness Process Management Forum - BPM Forum, pages 159–175. Springer, 2020. doi: 10.1007/978-3-030-58638-6\_10. 21
2020 doi
-
[20]
Uncertainty in predictive process monitoring
Pietro Portolani, Alessandro Brusaferri, Andrea Ballarino, and Matteo Matteucci. Uncertainty in predictive process monitoring. InInformation Processing and Management of Uncertainty in Knowledge-Based Systems - IPMU,pages547–559, 2022. doi: 10.1007/978-3-031-08974-9\_44
2022 doi
-
[21]
Exploit- ing recurrent graph neural networks for suffix prediction in predictive monitoring.Computing, 106(9):3085–3111, 2024
EfrénRama-Maneiro, JuanCarlosVidal, ManuelLama, andPabloMonteagudo-Lago. Exploit- ing recurrent graph neural networks for suffix prediction in predictive monitoring.Computing, 106(9):3085–3111, 2024. doi: 10.1007/S00607-024-01315-9
2024 doi
-
[22]
Simon Rauch, Christian M. M. Frey, Ludwig Zellner, and Thomas Seidl. Process-aware bayesian networks for sequential event log queries. InInternational Conference on Process Mining - ICPM, pages 161–168, 2024. doi: 10.1109/ICPM63005.2024.10680678
2024
-
[23]
Deepar: Probabilistic forecasting with autoregressive recurrent networks.International Journal of Forecasting, 36 (3):1181–1191, 2020
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks.International Journal of Forecasting, 36 (3):1181–1191, 2020. doi: 0.1016/j.ijforecast.2019.07.001
2020
-
[24]
Predictive business process monitoring with LSTM neural networks
Niek Tax, Ilya Verenich, Marcello La Rosa, and Marlon Dumas. Predictive business process monitoring with LSTM neural networks. InAdvanced Information Systems Engineering - CAiSE, pages 477–492, 2017. doi: 10.1007/978-3-319-59536-8\_30
2017 doi
-
[25]
Farbod Taymouri, Marcello La Rosa, and Sarah M. Erfani. A deep adversarial model for suffix and remaining time prediction of event sequences. InSIAM International Conference on Data Mining - SDM, pages 522–530, 2021. doi: 10.1137/1.9781611976700.59
2021 doi
-
[26]
Learning uncertainty with artificial neural networks for predictive process monitoring.Appl
Hans Weytjens and Jochen De Weerdt. Learning uncertainty with artificial neural networks for predictive process monitoring.Appl. Soft Comput., 125:109134, 2022. doi: 10.1016/J.ASOC. 2022.109134
2022
-
[27]
Brecht Wuyts, Seppe K. L. M. vanden Broucke, and Jochen De Weerdt. Sutran: an encoder- decoder transformer for full-context-aware suffix prediction of business processes. InInterna- tional Conference on Process Mining - ICPM, pages 17–24, 2024. doi: 10.1109/ICPM63005. 2024.10680671
2024
-
[28]
Deep and confident prediction for time series at uber
Lingxue Zhu and Nikolay Laptev. Deep and confident prediction for time series at uber. In IEEE International Conference on Data Mining Workshops - ICDM, pages 103–110, 2017. doi: 10.1109/ICDMW.2017.19. 22
2017 doi
-
[2019]
doi: 10.1137/1.9781611975673.14
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.