{"id":"ba36ae0e-7582-4fb2-bbf9-489a1ee3691e","arxiv_id":"2412.04914","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A proof-of-concept that transfers distribution-level demographic parity metrics and a Wasserstein composite loss to outcome-oriented predictive process monitoring, with a tunable accuracy-fairness trade-off.","lead":"This paper applies fairness measurement and training techniques to predictive process monitoring, where machine learning forecasts the final outcome of ongoing business processes. It demonstrates distribution-based demographic parity metrics and a Wasserstein fairness loss that trade predictive accuracy against group fairness on simulated hiring, lending, and renting logs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Batch-level Wasserstein estimates may not reflect population divergence; prefix correlation and fixed batch size leave Experiment 2's fairness generalization unsecured.","rationale":"The paper's central claim is that distribution-based independence can be both measured and optimized in predictive process monitoring, as demonstrated by the Pareto fronts in Experiment 2. This demonstration stands or falls on whether optimizing the minibatch Wasserstein loss actually improves population-level parity on unseen cases. The reader's weakest assumption correctly identifies the batch-size dependence of the IPM estimate; the prefix-level minibatch structure makes this issue more severe than in typical i.i.d. settings, because correlated prefixes reduce the effective independent sample size per batch and make the protected group's batch-level statistics noisier. The authors provide code and additional results in the repository, and they explicitly acknowledge batch-size limitations in Section 6, which supports a conditional rather than a reject verdict. The concern is directly testable by a batch-size and case-level aggregation ablation, so the appropriate verdict remains CONDITIONAL and does not change from the reader's assessment.","tokens_in":19721,"tokens_out":12185,"duration_ms":134735,"concrete_test":"Retrain the hiring_high Experiment 2 models for lambda values 0, 0.25, and 0.5 with batch sizes 128, 256, 512, and full-batch (or 2048), and evaluate all models on both (a) the original test prefix set and (b) a test set aggregated to one randomly sampled prefix per case, recomputing ABPC and ABCC in both cases. If test fairness varies substantially across batch sizes, or if case-level ABPC/ABCC is much worse than prefix-level ABPC/ABCC, the batch-level IPM estimate is not faithful and the generalization claim fails. Report AUC alongside each fairness value to ensure comparisons are made at matched predictive performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (5) optimizes LIPM computed as the Sinkhorn-approximated Wasserstein distance on training minibatches of prefix-outcome pairs, with batch size fixed at 512 for all IPM runs (Section 4.2). The load-bearing assumption is that this minibatch statistic is a faithful, unbiased proxy for the population-level divergence between protected groups' propensity distributions, and that minimizing it transfers to the held-out test set. Three concrete problems threaten this assumption. First, prefixes from the same case are highly correlated: Table 1 shows roughly 4-5 prefixes per training case, so a batch of 512 prefixes contains far fewer than 512 independent cases, and since the protected group is only about 20% of cases, the effective minority sample per batch is around 20-25 independent cases. Second, the empirical Wasserstein distance is positively biased at finite sample sizes, and the entropy-regularized Sinkhorn approximation adds further bias; gradients w.r.t. this biased statistic can be dominated by within-case correlation and batch-composition noise rather than true group divergence. Third, Section 6 itself recommends sufficiently large batch sizes or full batch training, but provides no sensitivity analysis to show the reported Pareto fronts are stable. If the optimization tracks batch-level noise, the test-time ABPC/ABCC reductions in Figure 4 could be specific to the particular prefix composition of the test set and would not support the paper's general claim that the composite loss reliably reduces demographic parity violations in OOPPM.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses group fairness in outcome-oriented predictive process monitoring (OOPPM). It adopts independence (demographic parity) as the fairness criterion, evaluates threshold-based ΔDP metrics and distribution-based metrics ABPC and ABCC, and proposes a composite loss L_total = (1-λ)L_BCE + λL_IPM, using the Sinkhorn-approximated Wasserstein distance as the IPM. Two experiments on synthetic event logs are reported: Experiment 1 measures fairness metrics on LSTM classifiers with and without the sensitive attribute; Experiment 2 sweeps λ from 0 to 0.5 on three high-bias logs and reports Pareto fronts between AUC and ABPC/ABCC. The paper concludes that distribution-based metrics are useful threshold-independent indicators and that the composite loss enables a tunable fairness–accuracy trade-off.","tokens_in":19988,"tokens_out":8922,"duration_ms":84400,"significance":"The contribution is conceptually useful for the PPM community: distribution-based demographic parity metrics have not been systematically explored in OOPPM, and the proposed loss is simple, model-agnostic, and implemented in PyTorch with code and full results released on GitHub. If the empirical claims are robust, the framework gives practitioners a straightforward way to audit and steer group fairness in outcome prediction. However, the evidence as presented is preliminary: the evaluation metric ABCC is closely aligned with the training objective, the batch-size assumption is unvalidated, and the single-run results lack statistical grounding. These gaps currently limit the strength of the claims.","major_comments":[{"comment":"The ABCC metric, defined as the integrated absolute difference of the empirical CDFs, is the population counterpart of the Wasserstein-1 distance used as L_IPM in Eq. (5). The paper itself notes in §3.2 that 'using Wasserstein as IPM loss is more aligned with optimizing ABCC than with optimizing ABPC.' Therefore, the observed decrease in ABCC with increasing λ in Fig. 4 is expected by construction and should not be presented as independent evidence of fairness improvement. The ABPC results, which are not directly optimized, provide the more meaningful non-circular evidence. Please rebalance the presentation accordingly—for example, report ABPC and threshold-based ΔDP at several thresholds as the primary fairness metrics, and treat ABCC as a check of the training objective rather than as an independent evaluation.","section":"§3.2, §5.3, Fig. 4"},{"comment":"The reliability of the minibatch Sinkhorn Wasserstein estimates is load-bearing for Experiment 2 but is not validated. Section 4.2 fixes the batch size at 512 'to ensure reliable computation of batch-level statistics', and Section 6 acknowledges that these losses 'rely on batch-level statistics' and recommends sufficiently large batch sizes or full-batch training, yet no sensitivity analysis is provided. Since Table 1 indicates that only about 9–30% of cases belong to the protected group and cases contribute multiple correlated prefixes, a batch of 512 prefixes contains a very small number of effectively independent protected-group samples. The empirical Wasserstein distance and its Sinkhorn approximation are biased and high-variance in this regime. Please provide a batch-size sensitivity analysis (e.g., 256, 512, 1024, full batch) and show that the Pareto fronts in Fig. 4 are stable; otherwise the observed test-time reductions could reflect optimization of batch-level noise.","section":"§4.2, §6, Fig. 4"},{"comment":"Experiment 2 is conducted as single runs without repeated seeds, and the Pareto fronts in Fig. 4 are derived from one train/test split. Because the AUC differences across λ are small (e.g., roughly 0.05 in hiring_high), it is not clear that the apparent trade-off is not a random artifact of model initialization and data sampling. Please repeat the λ sweep with multiple random seeds and report means with standard deviations or confidence intervals for AUC, ABPC, and ABCC (and ideally ΔDP at several thresholds) so that the shape of the Pareto front can be assessed statistically.","section":"§5.3"},{"comment":"The paper does not compare the proposed IPM-based composite loss with existing fairness interventions in PPM, notably the adversarial debiasing approach of [19] or a simple post-processing baseline such as per-group threshold adjustment. Since the contribution is a training-time mitigation method, at least one such baseline would help calibrate its practical value and rule out that the observed trade-off is specific to this LSTM-plus-Wasserstein configuration. Even a proof-of-concept experiment would be strengthened by including a reference method.","section":"§5.3, Related Work"}],"minor_comments":[{"comment":"The phrase 'composite loss function existing of binary cross-entropy' should be 'consisting of'.","section":"Abstract and §1"},{"comment":"The validation set is created by splitting 20% of the prefix-outcome pairs from the training samples rather than at the case level; because multiple prefixes come from the same case, this introduces case-level leakage between training and validation, affecting early stopping and threshold tuning. Please clarify whether a case-level split was considered or state why it is unnecessary here.","section":"§4.1"},{"comment":"The bandwidth selection for the KDE used to estimate ABPC is not reported; please add the bandwidth selection method or the code reference.","section":"§3.1"},{"comment":"Notation is inconsistent: ΔDPc, ΔDPt_b, ΔDP0.5_b, and ΔDPopt_b are all used; unify the notation and always specify the threshold value.","section":"Eqs. (1)–(2), Table 2"},{"comment":"There are several rendering artifacts in the text (e.g., 'F airness' in the title/header, 'T raining (& V alidation) Set' in Table 1, and 'Y ers' in the Section 6 text) that should be corrected in the final version.","section":"Throughout"},{"comment":"The legend entries appear as '=0.00', '=0.05', etc., without the λ symbol; please ensure the legend and captions are consistent so that readers can interpret the λ values.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a proof-of-concept and is generally clearly written. The code and data release are commendable. My main concern is that the empirical evaluation does not yet disentangle the effect of optimizing the IPM from the effect of selecting an evaluation metric that is almost identical to the training objective. The batch-size sensitivity and repeated-seed issues are also important for the paper's claims. I would encourage the editor to seek a revision that addresses these points, after which the paper could be acceptable for a specialized venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This paper is a clean proof-of-concept, not a conceptual advance. It takes two known pieces—Han et al.'s threshold-free distributional parity metrics (ABPC/ABCC) and the Wasserstein IPM loss from the fair-representation literature—and ports them to outcome-oriented predictive process monitoring with LSTMs on simulated logs. That application is genuinely new in the PPM subfield, and the paper is honest about what it does and doesn't show.\n\nWhat it does well: the framing of threshold-free metrics is motivated clearly (Figure 1 is a nice toy example), the experimental setup is controlled and reproducible, and the authors ship code and data. They also flag their own limitations: low F1, artificial logs, calibration concerns, and the reliance of distribution-based losses on batch-level statistics. That level of transparency is welcome.\n\nThe soft spots are real but not fatal. Most importantly, every result is a single run with no standard deviations or repeated seeds, so we have no idea how stable the Pareto fronts in Figure 4 are. There is also no comparison against existing PPM fairness mitigation, like the adversarial debiasing in [19] or the bias-mitigation survey in [34]. Given that the paper's novelty is the application, that baseline comparison matters. The stress-test concern about batch size is legitimate: prefixes from the same case are highly correlated, so a batch of 512 prefixes contains far fewer independent cases, and the minority group is small. The authors themselves recommend large batches but provide no sensitivity analysis. I don't think this breaks the core demonstration—the Pareto fronts are consistent across three logs and both metrics—but it does mean the quantitative claims should be read as suggestive, not definitive. Also, part of the ABCC decrease is by construction because the Wasserstein loss is aligned with ABCC; the paper admits this. The ABPC decrease is the more independent evidence, and it does move in the expected direction.\n\nWho is this for? PPM practitioners who want off-the-shelf fairness metrics and a tunable fairness knob, and researchers working at the intersection of process mining and fair ML. It deserves a serious referee. The revision request should ask for repeated runs with error bars, at least one baseline method, and a batch-size sensitivity study. The central idea is sound and the presentation is honest; the empirics just need to be hardened.","headline":"Solid proof-of-concept applying known distributional fairness metrics and a Wasserstein IPM loss to outcome-oriented predictive process monitoring; useful for PPM practitioners, but the experiments need repeated runs and baselines before the claims are secure.","tokens_in":20534,"tokens_out":2293,"would_cite":false,"duration_ms":24023,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a Wasserstein term to the training loss reduces demographic parity violations in predictive process monitoring, with a tunable fairness–accuracy trade-off along a Pareto front.","keywords":["predictive process monitoring","group fairness","demographic parity","independence","Wasserstein distance","integral probability metrics","ABPC","ABCC"],"falsifier":"Train the same LSTM with the composite loss on a large real-world event log while varying batch size from 64 to 2048 at a fixed $\\lambda$; if the reduction in test-set ABPC/ABCC between $\\lambda=0$ and $\\lambda=0.5$ shrinks or vanishes at small batch sizes, the batch-statistics assumption is load-bearing and the claim does not generalize beyond the controlled setup.","tokens_in":19504,"feed_emoji":"⚖️","tokens_out":5778,"duration_ms":52136,"temperature":0.7,"pith_summary":"Predictive process monitoring models trained on biased historical event logs can carry demographic bias into their predictions. This paper argues that group fairness through independence—predictions unaffected by sensitive group membership—can be both measured and optimized within this setting. It brings threshold-independent distribution metrics (area between density curves and area between cumulative curves) to process monitoring, and proposes a composite loss that mixes binary cross-entropy with a Sinkhorn-approximated Wasserstein distance, weighted by a hyperparameter lambda. In controlled experiments on three simulated event logs, increasing the fairness weight moves models along a Pareto front between AUC and demographic parity violations, demonstrating a tunable trade-off. If the claim holds, practitioners gain a model-agnostic way to audit and steer fairness without committing to a classification threshold.","feed_headline":"Adding a Wasserstein term to the loss cuts demographic parity gaps","feed_subtitle":"Threshold-free parity metrics make the fairness cost visible, and one knob λ tunes the trade-off.","key_machinery":"The load-bearing component is the composite loss $L_{\\text{total}}=(1-\\lambda)L_{\\text{BCE}}+\\lambda L_{\\text{IPM}}$ with the Wasserstein distance (Earth Mover's Distance), computed via the Sinkhorn approximation on minibatches of size 512, serving as $L_{\\text{IPM}}$. Because the Wasserstein distance equals the area between cumulative distribution functions, the training objective aligns directly with the ABCC fairness metric. The evaluation machinery consists of the threshold-independent metrics ABPC (estimated by kernel density estimation) and ABCC (from empirical CDFs), integrated over $[0,1]$.","core_discovery":"The central discovery is that distribution-based independence can be integrated directly into the training objective of an outcome-oriented predictive process monitoring classifier. The paper trains an LSTM with the composite loss $L_{\\text{total}}=(1-\\lambda)L_{\\text{BCE}}+\\lambda L_{\\text{IPM}}$, where $L_{\\text{IPM}}$ is the Sinkhorn-approximated Wasserstein distance between the propensity distributions of the two protected groups. Sweeping $\\lambda$ from 0 to 0.5 produces a Pareto front between AUC and the threshold-independent parity metrics ABPC and ABCC on the hiring, lending, and renting simulated logs: higher $\\lambda$ lowers parity violations at the cost of some predictive performance. The paper also shows that single-threshold metrics like $\\Delta\\text{DP}^{0.5}_b$ can report zero bias while the propensity distributions remain highly separated, motivating the distribution-level view.","pith_inferences":["The paper's own caveat that process fairness must improve in parallel implies that deploying independence-trained models as early interventions could shift rather than remove bias; an audit of separation and sufficiency would then be necessary, an extension the authors explicitly flag.","The slight AUC improvements at small $\\lambda$ on some logs suggest the IPM term acts as a regularizer; a direct test would compare the same Pareto-front protocol with MMD or KL as the IPM.","The fixed batch size of 512 is a testable lever: varying batch size while holding $\\lambda$ fixed should move the Pareto front if batch-statistics fidelity is the limiting factor."],"forward_implications":["PPM teams can audit any propensity-outputting model for demographic parity with ABPC and ABCC without choosing a decision threshold first.","The $\\lambda$ parameter lets stakeholders select an operating point on the fairness–accuracy frontier that matches their regulatory risk tolerance.","Because the loss is model-agnostic, the same composite objective can be layered onto transformer-based PPM models or other gradient-trained classifiers.","Distribution-level metrics expose parity violations that $\\Delta\\text{DP}$ at a single threshold can hide, such as identical means with opposite-shaped group densities."],"supporting_citations":[{"why":"Supplies the ABPC and ABCC distribution-level metrics and the argument for retiring threshold-dependent $\\Delta$DP.","marker":"[13]"},{"why":"Shows how an integral probability metric can enforce demographic parity, the basis for the IPM loss.","marker":"[15]"},{"why":"Provides the simulated event logs (hiring, lending, renting) used in the controlled experiments.","marker":"[28]"},{"why":"Supplies the Sinkhorn-approximated Wasserstein implementation used inside the training loop.","marker":"[33]"},{"why":"Introduces Sinkhorn distances, the approximation technique that makes the Wasserstein loss computationally feasible.","marker":"[6]"},{"why":"Provides the specific event log artifact containing the protected feature used to define the groups.","marker":"[27]"},{"why":"The LSTM architecture that carries the experiments, chosen for its standard use in PPM.","marker":"[14]"},{"why":"Documents the inherent fairness–accuracy trade-off that the Pareto-front analysis is built to navigate.","marker":"[5]"}],"fun_headline_variants":["Wasserstein loss for fairer process predictions","Distribution-based fairness loss tunes the cost","One knob to trade accuracy for group parity","Threshold-free metrics expose bias hidden by cutoffs","Predictive monitoring gets a fairer training objective"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The demonstration collapses if the Sinkhorn-approximated Wasserstein distance computed on 512-sample training minibatches is not a faithful estimate of the population-level divergence between the protected groups' prediction distributions.","fun_headline_variants_meta":{"raw":{"variants":["Wasserstein loss for fairer process predictions","Distribution-based fairness loss tunes the cost","One knob to trade accuracy for group parity","Threshold-free metrics expose bias hidden by cutoffs","Predictive monitoring gets a fairer training objective"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000712,"raw_usage":{"total_tokens":3182,"prompt_tokens":901,"completion_tokens":2281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2213}},"tokens_in":517,"tokens_out":2281,"duration_ms":15922,"temperature":1.0,"reasoning_tokens":2213,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:08:27.486755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same LSTM with the composite loss on a large real-world event log while varying batch size from 64 to 2048 at a fixed $\\lambda$; if the reduction in test-set ABPC/ABCC between $\\lambda=0$ and $\\lambda=0.5$ shrinks or vanishes at small batch sizes, the batch-statistics assumption is load-bearing and the claim does not generalize beyond the controlled setup.","supporting_citations":[{"cited_title":"Transactions on Machine Learning Research (2023), ISSN 2835-8856","cited_arxiv_id":null,"evidence_quote":"Supplies the ABPC and ABCC distribution-level metrics and the argument for retiring threshold-dependent $\\Delta$DP."},{"cited_title":"In: Proceedings of the 39th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol","cited_arxiv_id":null,"evidence_quote":"Shows how an integral probability metric can enforce demographic parity, the basis for the IPM loss."},{"cited_title":"A Collection of Simulated Event Logs for Fairness Assessment in Process Mining","cited_arxiv_id":"2306.11453","evidence_quote":"Provides the simulated event logs (hiring, lending, renting) used in the controlled experiments."},{"cited_title":"In: Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol","cited_arxiv_id":null,"evidence_quote":"Supplies the Sinkhorn-approximated Wasserstein implementation used inside the training loop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the specific event log artifact containing the protected feature used to define the groups."},{"cited_title":"Neural Comput","cited_arxiv_id":null,"evidence_quote":"The LSTM architecture that carries the experiments, chosen for its standard use in PPM."},{"cited_title":"In: Proceedings of the 23rd acm sigkdd interna- tional conference on knowledge discovery and data mining, pp","cited_arxiv_id":null,"evidence_quote":"Documents the inherent fairness–accuracy trade-off that the Pareto-front analysis is built to navigate."}],"review_version":1}