{"id":"f1bcd2a7-61cd-4399-8de2-60c279cdf82f","arxiv_id":"2607.17783","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A control-flow-aware trace segmentation algorithm enables segment-level SHAP explanations for deep-learning outcome prediction in predictive process monitoring.","lead":"The authors propose a way to explain deep-learning predictions of business process outcomes by splitting each process trace into segments at points where activity-transition patterns change, and then computing SHAP importance values per segment. The method is tested on a synthetic loan process and two real event logs, showing that segment-level explanations can reveal which parts of a case push a prediction toward acceptance or cancellation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world segmentation evaluation is circular: the ER metric rewards fragmentation and overlaps with the segmentation objective, so the claim that DFR shift points are process-phase boundaries lacks non-circular support.","rationale":"The reader's weakest assumption identified that DFR probability shifts may not correspond to meaningful process phase boundaries. My concern sharpens this: the real-world evaluation of segmentation quality uses ER, which is structurally biased toward finer segmentation and is computed from the same transition probabilities the segmentation optimizes. This is a specific internal-circularity problem, not just an assumption that could fail. However, the reader's verdict is already CONDITIONAL and their rationale explicitly notes that 'the ER evaluation overlaps with the segmentation objective.' Therefore my analysis does not move the verdict; it reinforces the need for the condition. The central claim remains plausible as a method proposal, but its empirical support is weaker than the paper presents. A synthetic stress test with known phases and rare in-phase transitions would directly test whether the method detects phase boundaries rather than transition rarities, and a random-segmentation ER control would test whether the ER result is meaningful.","tokens_in":11864,"tokens_out":5968,"duration_ms":57819,"concrete_test":"Generate a synthetic log with known phase boundaries and deliberately include low-probability exception transitions within each phase (e.g., 2-5% of transitions). Run the proposed PELT segmentation with the paper's AIC-selected beta and compute precision/recall against true phase boundaries across many traces. Also compare ER reduction of the proposed segmentation to a random segmentation with the same number of segments per trace. If precision/recall drops due to spurious in-phase splits, or random segmentation achieves comparable ER reduction, then the claimed phase-boundary detection and the ER-based evaluation are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, discovered segments must correspond to meaningful process phases. The real-world evidence for this relies on entropic relevance (ER) in Section 6.1 and Table 2: after segmentation, ER drops from 37.81 to 6.68 for BPIC17. But ER measures encoding cost under directly-follows transition probabilities (ref [20]), and the segmentation cost in Eq. (1) is the sum of negative log DFR probabilities. The segmentation therefore directly minimizes the quantity ER measures. In addition, ER is evaluated per segment using transitions within that segment, so finer segmentation trivially lowers ER (a single-transition segment has zero entropy). The reported ER reduction is an artifact of the optimization and segment count, not independent evidence that boundaries align with process phases. The only ground-truth validation uses two hand-picked SimBank samples (Table 1), which does not rule out spurious splits at rare in-phase transitions. Section 7 concedes that high-variability logs such as BPIC15 yield fine-grained, case-specific segmentation, exactly the failure mode of the DFR-shift assumption. Without a non-circular ground-truth check, segment-level SHAP values cannot be asserted to carry process-level meaning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for segment-level SHAP explanations of deep LSTM outcome predictors in predictive process monitoring. It introduces a control-flow-aware segmentation algorithm that uses directly-follows transition probabilities and the PELT change-point detection algorithm, defining segment cost as the sum of negative log directly-follows probabilities and selecting change points via a penalized objective. Segment-level SHAP values are then computed with KernelSHAP over the resulting segments. The method is evaluated on a synthetic SimBank dataset with two hand-picked cases with known change points, and on BPIC15 and BPIC17, using segmentation quality metrics (RandIndex, F1, entropic relevance) and perturbation-based faithfulness metrics (RPCI/RPCU), together with qualitative analyses of BPIC17 segmentation patterns. The paper claims that segment-level explanations offer a useful middle ground between event-level and trace-level explanations and that the method reveals recurring trace segments associated with accepted and canceled cases.","tokens_in":12134,"tokens_out":3839,"duration_ms":35977,"significance":"If the central claim were established, the paper would make a useful contribution by addressing a real gap in PPM explainability: event-level attributions are fragmented and trace-level attributions lose control-flow context, while segment-level attributions can preserve locally coherent process behavior and reduce computational cost. The paper has notable strengths: the code is made publicly available, the synthetic evaluation uses explicit ground-truth change points, the method is compared against event-level and distribution-based segmentation baselines, and the faithfulness evaluation uses perturbation-based metrics. The writing is generally clear and the limitation for high-variability logs is acknowledged. However, the evidence for the central claim—that the discovered segments correspond to meaningful process phases—is currently weak and partly circular, so the main contribution is not yet convincingly validated.","major_comments":[{"comment":"The entropic relevance (ER) reduction from 37.81 to 6.68 for BPIC17 is not independent evidence that the segmentation captures process phases. ER is defined in terms of the same directly-follows transition probabilities that the segmentation cost in Eq. (1) minimizes, and ER is evaluated per segment using transitions within that segment, so finer segmentation trivially lowers ER. A single-transition segment has zero entropy. The reported reduction is therefore largely mechanical and overlaps with the optimization objective. Please provide a non-circular validation, for example synthetic logs with known phases, segmentation evaluated against a held-out transition matrix, or a comparison that controls for the number of segments.","section":"Section 6.1, Table 2, and Eq. (1)-(3)"},{"comment":"The synthetic evaluation uses only two hand-picked samples selected precisely because their change points are clearly defined. This does not provide sufficient evidence that the DFR-shift criterion reliably identifies process phases across realistic process variants, noise levels, or unseen transitions. There is also no comparison against the distribution-based baseline or a per-event baseline on the same synthetic data, and no sensitivity analysis with respect to the penalty parameter beta. Please extend the synthetic evaluation to a larger, systematically varied set of traces and include baseline comparisons.","section":"Section 5.1, Table 1"},{"comment":"The RPC evaluation validates the faithfulness of SHAP values conditional on the chosen segments, but it does not validate that the segments themselves are meaningful process phases. Since the top-k segments are selected by the same SHAP values whose faithfulness is being tested, RPCI/RPCU is an internal consistency check and cannot support the statement in Section 7 that the identified regions are 'influential rather than arbitrary' in the process-level sense. Independent validation of phase boundaries is needed before segment-level SHAP values can be claimed to carry process-level meaning.","section":"Section 6.2, Table 3, and Section 7"},{"comment":"The paper assumes that positions where the directly-follows cost changes 'can be interpreted as boundaries between different process phases.' Section 7 concedes that high-variability logs such as BPIC15 produce fine-grained, case-specific segmentation, which is the failure mode of this assumption. The scope of the central claim should therefore be stated more carefully: at present, the evidence supports only logs with moderate control-flow variability, and the paper should either test another such log or explicitly restrict its claims in the abstract and conclusion.","section":"Section 4.1 and Section 7"}],"minor_comments":[{"comment":"The activity names are inconsistent: Section 5.1 refers to 'skip_contact_hq' while Table 1 labels the sample as 'skipped_hq'; please align the notation.","section":"Table 1 and Section 5.1"},{"comment":"The choice of 40% trace-length coverage for the top-k segments is not justified; please report sensitivity to this threshold or provide a rationale.","section":"Section 5.2"},{"comment":"The table does not include any statistical comparison between strategies; paired tests or confidence intervals would help assess whether the observed RPCI/RPCU differences are meaningful.","section":"Table 3"},{"comment":"The comparison of event-level and segment-level SHAP values in Figure 3 is described only briefly; additional caption detail on how the segment values are derived from event values would improve reproducibility.","section":"Section 6.3, Figure 3"},{"comment":"There are several minor grammatical issues and unclear cross-references (e.g., 'the former contains administrative processes' after the BPIC15/BPIC17 introduction could be misread); a thorough language edit is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of process mining / explainability venues and the proposed idea is sensible, but the core validation of segmentation quality is currently circular or too limited. The fixes needed—additional synthetic experiments, baseline comparisons, and a non-circular real-world validation—appear feasible within the scope of a revision. I would not recommend rejection, provided the authors address the load-bearing concerns about ER circularity and the generality of the phase-boundary claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the genuinely new piece is using directly-follows transition probabilities as the cost signal for PELT change-point detection and treating the resulting segments as SHAP coalitions. That combination is not in the PPM explainability literature, and it addresses a real gap: event-level attributions are fragmented, trace-level ones lose control-flow context. The authors ship code and data, and the synthetic SimBank check with known change points is the right idea, giving F1 around 0.89. RPC results on BPIC17 move in the expected direction for faithful attributions, and the paper explicitly admits that BPIC15's high variability leads to case-specific fine-grained segmentation and that event-level attributions are slightly more faithful. The limitations section is unusually candid.\n\nSoft spots, in order. The ER reduction in Table 2 is not independent evidence for meaningful phase boundaries. Eq. (1) minimizes negative log DFR probabilities; ER measures encoding cost under the same DFR probabilities. Minimizing the objective you then evaluate with is circular, and ER will favor fragmentation, since a single-transition segment has zero entropy. The synthetic validation uses only two hand-picked samples, so it cannot rule out spurious splits at rare but in-phase transitions. There is no trace-level SHAP baseline, even though the paper's central claim is a middle ground between event and trace granularity. And RPC compares perturbations of SHAP-selected segments, which is an internal consistency check, not a ground-truth test of segment semantics. None of these is fatal: the method can still be useful, and the authors seem aware of the limits. But as published, the claim that DFR shift points are process-phase boundaries rests mostly on assumption plus synthetic evidence from two samples.\n\nWho this is for: researchers in explainable predictive process monitoring who want a practical segment-level attribution alternative. The paper deserves a serious referee; I would send it out and ask for a non-circular evaluation—more simulated logs with known change points, a trace-level baseline, and either human annotation or a process-model-based check of segment boundaries. The core idea is sound enough to justify the effort.","headline":"A practical control-flow-aware segmentation for segment-level SHAP in PPM, with honest limitations—but the real-world phase-boundary evidence is partly circular.","tokens_in":12645,"tokens_out":2358,"would_cite":true,"duration_ms":22033,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that cutting traces into control-flow segments and computing SHAP values per segment gives a middle ground between event-level and trace-level explanations for deep-learning outcome predictions.","keywords":["predictive process monitoring","explainability","SHAP","trace segmentation","change point detection","directly-follows relation","LSTM","event logs"],"falsifier":"Run the segmentation on a synthetic log built from two process variants that share the same activity labels but use different transition probabilities, with phase boundaries known in advance; if the pooled directly-follows matrix places change points that do not match the true boundaries, or if the entropy reduction comes only from splitting traces into very short segments, the phase-boundary interpretation is refuted. A simpler check is to shuffle the transition-probability signal within each trace and see whether the discovered segments and segment-level SHAP values change as much as they do for the true ordering.","tokens_in":11710,"feed_emoji":"🧩","tokens_out":11369,"duration_ms":88582,"temperature":0.7,"pith_summary":"The paper tries to establish that deep-learning outcome predictions in business processes can be explained at the level of process phases rather than individual events or whole traces. To do this it cuts each trace into segments at points where the directly-follows transition probabilities between activities change, then computes SHAP values with segments as the explanation units. The authors argue these segment-level explanations preserve local control-flow context, avoid the fragmentation of event-level attributions, and avoid the information loss of trace-level aggregation. On a synthetic loan log with known change points, the segmentation recovers the true boundaries with high agreement, and on the BPIC17 loan log it surfaces recurring segments tied to acceptance and cancellation. The payoff would be that stakeholders can see which concrete part of a running case steers a prediction, not just which isolated activities matter.","feed_headline":"Trace segments reveal why AI predicts process outcomes","feed_subtitle":"Segment-level SHAP keeps the control-flow context, so analysts see which part of a running case drives the forecast.","key_machinery":"The load-bearing object is the control-flow-aware segmentation algorithm: for each trace, it reads off the sequence of directly-follows transition probabilities from a normalized directly-follows-relation matrix and detects change points by minimizing the sum of segment costs \\(C(s)=\\sum_{i=1}^{m-1} -\\log T[a_i,a_{i+1}]\\) (with penalty \\(\\gamma\\) for unseen transitions) plus \\(\\$\\beta$ k\\) for the number of change points, solved with the PELT algorithm. Positions where the transition-probability profile shifts are interpreted as boundaries between process phases. These segments then become the features in a KernelSHAP computation, so each explanation unit is a locally coherent control-flow region rather than a single event.","core_discovery":"The central claim is that segment-level SHAP explanations computed over control-flow-aware segments are a useful middle ground between event-level and trace-level explanations for predictive process monitoring. The method defines a segment by identifying change points in a trace where the empirical directly-follows transition probabilities shift, using a negative-log-probability cost with a penalty for extra boundaries (via the PELT algorithm). It then treats each segment as one coalition in KernelSHAP, with a baseline built from the training distribution. On the synthetic SimBank log the detected boundaries agree with ground-truth change points (RandIndex above 0.9, F1 scores of 0.885 and 0.874); on BPIC17, top segmentation patterns cover large fractions of accepted and canceled cases, entropic relevance drops from 37.81 to 6.68, and perturbation-based faithfulness metrics (relative prediction change is high for top-attributed segments, low for low-attributed ones) show the attributed segments genuinely influence predictions. The paper also reports the boundary condition: in a high-variability log like BPIC15, the same algorithm fragments traces into many case-specific segments, so the phase interpretation weakens.","pith_inferences":["A natural extension is to feed the detected segment boundaries, or the counts and types of segments, back into the predictive model as engineered features; that would test whether the phase structure itself carries outcome signal beyond the activities.","The same directly-follows cost could guide counterfactual generation, treating segment boundaries as the points where an intervention could switch a predicted outcome; this application is not explored in the paper.","Because the directly-follows-relation matrix is pooled across all cases, the method may conflate different process variants; estimating variant-specific transition probabilities before segmentation could restore phase meaning in high-variability logs.","A prefix-length sensitivity analysis would clarify whether the discovered phases are stable as a case progresses, since the 30-event prefix used here may cut off later decisive phases."],"forward_implications":["On logs with moderate control-flow variability, analysts can see which phase of a case pushes the prediction—for example, repeated offer creation pushes BPIC17 predictions toward cancellation.","Segment-level explanations are less faithful than event-level ones but more comprehensible, because they group coherent events and reduce interleaving of high- and low-attribution events.","Because segments act as coalitions in SHAP, the number of explanation features drops from trace length to number of segments, cutting the computational cost of attribution.","Segmentation can be validated against ground truth when known change points exist, and the same evaluation framework transfers to other logs via pattern coverage and entropic relevance.","The benefit is conditional: high-variability logs such as BPIC15 yield case-specific fine-grained segments, so conclusions at the segment level only generalize for logs with identifiable phase structure."],"supporting_citations":[{"why":"Supplies the PELT change-point detection algorithm that minimizes the penalized segmentation cost.","marker":"[14]"},{"why":"Provides the baseline event vector and perturbation scheme used to compute segment-level SHAP values.","marker":"[4]"},{"why":"Defines the distribution-based segmentation baseline against which the proposed method is compared.","marker":"[13]"},{"why":"Generates the synthetic loan-process log with known change points used as ground truth for segmentation quality.","marker":"[7]"},{"why":"Supplies the outcome labeling strategies used to prepare the BPIC15 and BPIC17 logs.","marker":"[28]"},{"why":"Defines entropic relevance, the metric used to show that segments are more compressible and structured.","marker":"[20]"},{"why":"Provides the LSTM sequence model architecture that the method explains.","marker":"[27]"}],"fun_headline_variants":["Segment-level SHAP explains which trace parts drive predictions","Control-flow segments uncover why AI predicts process outcomes","Local SHAP on trace segments spots control-flow change points","Trace segmentation pinpoints prediction-driving process phases","Segment attributions reveal control-flow dynamics in forecasts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the places where the usual next-activity probabilities change mark real boundaries between process stages; if those changes are noise or arbitrary variability, the segments do not have the process meaning the explanations claim.","fun_headline_variants_meta":{"raw":{"variants":["Segment-level SHAP explains which trace parts drive predictions","Control-flow segments uncover why AI predicts process outcomes","Local SHAP on trace segments spots control-flow change points","Trace segmentation pinpoints prediction-driving process phases","Segment attributions reveal control-flow dynamics in forecasts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000314,"raw_usage":{"total_tokens":1797,"prompt_tokens":974,"completion_tokens":823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":750}},"tokens_in":590,"tokens_out":823,"duration_ms":7461,"temperature":1.0,"reasoning_tokens":750,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:34:24.468481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the segmentation on a synthetic log built from two process variants that share the same activity labels but use different transition probabilities, with phase boundaries known in advance; if the pooled directly-follows matrix places change points that do not match the true boundaries, or if the entropy reduction comes only from splitting traces into very short segments, the phase-boundary interpretation is refuted. A simpler check is to shuffle the transition-probability signal within each trace and see whether the discovered segments and segment-level SHAP values change as much as they do for the true ordering.","supporting_citations":[{"cited_title":"In: DAS- FAA (2024)","cited_arxiv_id":null,"evidence_quote":"Defines the distribution-based segmentation baseline against which the proposed method is compared."},{"cited_title":"In: BPM (2025).https://doi.org/ 10.1007/978-3-032-02929-4_10","cited_arxiv_id":null,"evidence_quote":"Generates the synthetic loan-process log with known change points used as ground truth for segmentation quality."},{"cited_title":"In: CAiSE (2017).https://doi.org/10.1007/ 978-3-319-59536-8_30","cited_arxiv_id":null,"evidence_quote":"Provides the LSTM sequence model architecture that the method explains."}],"review_version":1}