{"id":"5daa24d4-a513-4e20-9c28-07e5915e7633","arxiv_id":"2501.10543","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"FORLAPS combines offline Q-learning with process-aware augmentation and fine-tuning, claiming 31% resource time savings and 23% process time reduction in prescriptive process monitoring.","lead":"This paper proposes FORLAPS, an offline reinforcement learning method with reward shaping and data augmentation to recommend activity orderings in business processes. The authors report 31% resource time savings and 23% process time reduction on an energy regulator's event log, but the evaluation has gaps that make these numbers hard to trust.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 31%/23% KPI savings rest on an underspecified replay simulator; without an explicit model of how reordering changes durations, approval outcomes, and activity cancellation, the central performance claim is unestablished.","rationale":"The reader's weakest_assumption pinpoints the same load-bearing issue: the KPI evaluation is a counterfactual replay, and the paper does not define the simulator. I agree. The reason this is the single most consequential concern is that the abstract and conclusion make the quantitative claim (31% resource savings, 23% time-span reduction) the primary evidence for FORLAPS outperforming LSTM and PFI. If the replay semantics are unspecified, those numbers cannot be checked or reproduced, and no amount of Q-value convergence plots or Damerau-Levenshtein similarity analysis establishes the KPI claim. The paper's internal caveat that sequential reordering may add delays, together with the process description involving cascading failures and up to 10 revisions, makes the counterfactual assumptions especially nontrivial: a realistic simulator would need to model revision loops, resource contention, and order-dependent approval outcomes. Other issues are real but secondary: the malformed semantic-validity formula (Eq. 4) undermines the formal description of augmentation, and the conclusion's phrase \"online fine-tuning\" contradicts Section 2.3's fully offline fine-tuning. However, even if those were fixed, the central performance comparison would still rest on the unvalidated replay. I would keep the reader's REJECT verdict: the paper's main claim is not supported by the evidence as presented, though a clearly specified and data-calibrated simulator could make the framework evaluable. My concrete test—recomputing resource savings under a no-skipping control—would distinguish a genuine process improvement from an artifact of implicit simulator assumptions.","tokens_in":16442,"tokens_out":8344,"duration_ms":80519,"concrete_test":"Release the replay simulator as executable code with a formal description of the MDP transition used for evaluation, then rerun the Table 3 comparison under an explicit counterfactual control: for each of the 15,000 test traces, apply the learned policy to reorder the activities but force every activity in the original trace to be executed exactly once with its original duration and original approval outcome (no activity skipping, no order-dependent outcome changes). Under this control, the resource-time KPI must be identically zero regardless of policy. If the published 31% resource saving does not collapse to zero under this control, the extra savings must be attributed to unstated skipping/outcome-changing mechanics; if it does collapse, the headline KPI claim depends wholly on counterfactual assumptions that have not been validated against the energy-regulator log.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines the two KPIs as differences between original traces and traces \"reordered according to the RL optimal policy,\" and the reported numbers in Table 3 come from replaying 15,000 test traces. The paper never specifies the replay simulator: no transition function, no duration model, no approval-outcome model, no cancellation/revision semantics. Yet the claimed savings are only physically possible under specific counterfactual assumptions — for instance, that an early disapproval can cause later activities to be skipped, and that doing so does not merely defer the disapproved work into a revision. The paper's own description of the process says up to 10 revisions occur and activities are partially parallel; Section 3 also concedes that sequential reordering may add delays. A simulator that does not model resubmission loops or resource contention cannot justify the 31% resource-time and 23% time-span figures. Because the central claim is a quantitative comparison against LSTM/PFI on these KPIs, an unspecified simulator makes the headline result non-reproducible. The Damerau-Levenshtein analysis in Section 4.3 is a separate robustness check on sequence similarity; it does not measure resource time or process span and therefore does not rescue the KPI claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FORLAPS, a five-step prescriptive process monitoring framework that combines offline reinforcement learning with a state-dependent (length-aware) reward shaping mechanism and a process-aware data augmentation technique. The authors apply the framework to a real-life event log from a Canadian energy regulator and report 31% savings in resource time spent and 23% reduction in process time span, comparing favorably against LSTM and Permutation Feature Importance baselines. They also report improved Q-value convergence on nine public datasets and validate the recommended sequences using Damerau-Levenshtein distance at both trace and prefix levels. The central claim is that FORLAPS reliably identifies optimal execution paths and provides context-sensitive next-activity recommendations without online environment interaction.","tokens_in":16682,"tokens_out":7784,"duration_ms":75339,"significance":"If the empirical claims were substantiated, the paper would offer a practical, data-driven approach to prescriptive process monitoring in data-scarce, high-stakes settings, and the integration of length-aware reward shaping with process-aware augmentation is a sensible direction. The authors provide source code on GitHub and evaluate on a diverse set of datasets (Sepsis, BPIC 2015/2017/2019, Traffic Fines), which is a strength. However, the headline results rest on an unspecified replay simulator, and the Q-value improvements are partly circular because the augmentation is described as selectively increasing Q-values. As written, the contribution is not fully established, and the evaluation protocol needs substantial clarification before the claims can be accepted.","major_comments":[{"comment":"The replay simulation that produces the 31% resource-time and 23% time-span savings is not defined. The text says \"we replayed 15,000 traces from the test set of event logs using the learned optimal policy,\" but no transition function, duration model, activity outcome (approval/disapproval) model, or cancellation/revision semantics are specified. The process in Section 2.1 is described as having up to 10 revisions and partially parallel activities, and Section 3 itself concedes that sequential reordering may add delays. Without an explicit simulator, the KPI computations are not reproducible and the comparison against LSTM and PFI baselines is uninterpretable. This is the core empirical claim, so the paper must provide a precise replay model and re-run the evaluation.","section":"Section 3, Table 3"},{"comment":"Reporting Q-value increases as performance improvements is circular. The abstract and Section 2.3 state that the augmentation technique \"selectively increases the average estimated Q-values in sampled batches,\" and Section 3 then reports a 44% Q-value improvement during fine-tuning, while Section 4.2 reports \"outperformed the Offline RL agent by 28 percentage points on the Sepsis dataset and 33 percentage points on the BPIC_2017 dataset\" as superiority. If the augmentation is intentionally designed to raise Q-values, higher Q-values are not independent evidence of a better policy. The paper should use externally defined metrics (e.g., expected return under a fixed simulator, success rate, or downstream KPIs) to compare policies, and should either drop the Q-value-based superiority claims or present them only as convergence diagnostics.","section":"Sections 2.3, 3, and 4.2"},{"comment":"Equation (4) is malformed: it uses the subset symbol \"⊆\" for trace membership, mixes universal and existential quantifiers with an unclear implication scope, and does not define the validity predicate V(σ', b) that it claims to formalize. As written, the equation cannot be implemented or checked, yet it is the formal basis for the \"process semantic check\" that the augmentation is said to satisfy. This needs to be rewritten as a precise logical condition over traces, business rules, and the augmented log.","section":"Section 2.3, Eq. (4)"},{"comment":"The paper contradicts itself on whether fine-tuning is offline or online. Section 2.3 says \"we enhance policy learning through a fine-tuning phase conducted entirely in the offline setting\" and the method is \"fully compliant with an offline RL paradigm,\" but the Conclusion (Section 5) describes FORLAPS as \"a hybrid technique that combines offline pre-training with online fine-tuning.\" This is not a minor wording issue: it changes the nature of the method and its claimed applicability to settings where online interaction is expensive or infeasible. The authors must unambiguously state the operating mode and ensure all sections are consistent.","section":"Section 5 vs. Section 2.3"}],"minor_comments":[{"comment":"The column header \"Opt%\" is never defined, and the table reports a \"96% confidence interval,\" which is unusual; please define the statistic and justify the confidence level.","section":"Section 3, Table 3"},{"comment":"The introductory paragraph contains a duplicated discussion of manufacturing environments: \"Manufacturing environments face similar challenges...\" is followed by \"Similarly, manufacturing environments rely on well-ordered production activities...\" Please remove the redundancy.","section":"Section 1, Introduction"},{"comment":"The phrase \"80'%\" contains a stray apostrophe and should be \"80%.\"","section":"Section 4.1"},{"comment":"The sentence \"BPIC_2019 showed that LSTMs lightly outperformed offline RL\" likely means \"slightly outperformed\"; please correct the typo.","section":"Section 4.3"},{"comment":"Equation (5) is not a correct optimal-policy equation: it mixes an argmax over actions with an argmax over policies and omits the expectation over transitions. Please rewrite it using standard notation.","section":"Section 2.3, Eq. (5)"},{"comment":"The reward function is written as $-r|s_t|\\mathcal{I}(s_t,a_t)$ in Eq. (2) but as $reward \\leftarrow -r \\cdot |s_t| \\cdot \\mathcal{I}(s_t,a_t)$ in Algorithm 1; please align the notation and clarify whether the indicator function depends on both state and action or on the outcome only.","section":"Section 2.2, Eq. (2) and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely problem, and the authors provide a wide set of experiments plus open-source code. However, the evaluation protocol is the main bottleneck: the replay simulator and baseline comparison procedure must be fully specified before the headline KPI claims can be assessed. The Q-value circularity and the offline/online contradiction also need to be resolved. I recommend inviting a revision that addresses these points, and I would like to see the authors explicitly provide the simulator definition and the code for the baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the method is a reasonable combination of known pieces, but the headline numbers don't hold up as reported because the replay simulator behind them is never defined, and the paper contradicts itself on whether the fine-tuning is offline or online.\n\nThe genuinely new part is the specific recipe: tabular offline Q-learning with state-dependent reward shaping, plus process-aware data augmentation that respects precedence and co-occurrence rules, plus a fine-tuning phase. Applying it to nine public datasets is more breadth than most prior prescriptive monitoring work, and the Damerau-Levenshtein distance as a sequence-level robustness check is a sensible addition. The prefix-level analysis is a nice touch too.\n\nThe soft spots are serious, though. Section 3 defines the two KPIs as differences between original traces and traces \"reordered according to the RL optimal policy,\" and Table 3 reports 31% resource savings and 23% time-span savings from replaying 15,000 traces. The paper never specifies that replay simulation: no transition function, no duration model, no approval-outcome model, no handling of the up-to-10 revisions mentioned in the process description. Without that, the central quantitative claim is not reproducible. The Q-value improvement is partly circular: the augmentation is advertised as selectively increasing the average estimated Q-values, and those higher Q-values are then reported as evidence of success. Equation 4 is malformed; it doesn't parse as a logical constraint. And the conclusion says the method combines offline pre-training with online fine-tuning, while Section 2.3 explicitly says the fine-tuning happens entirely in the offline setting and is fully compliant with an offline RL paradigm. That is a direct contradiction, not a stylistic slip.\n\nWho should read it: researchers in prescriptive process monitoring who want to see the combination of reward shaping and augmentation tried on public benchmarks. They should treat the 31%/23% numbers as unverified until the simulator is released. The Damerau-Levenshtein results are a useful comparison, but they don't measure resource time or process span.\n\nFor peer review: I'd send it to referees, but with the expectation of major revision. The idea is salvageable, and the flaws are fixable — but only if the authors either specify the counterfactual simulator in detail or drop the KPI claims and reframe around the sequence-similarity validation.","headline":"The combination of offline Q-learning, reward shaping, and process-aware augmentation is genuinely new, but the headline KPI savings are unestablished because the replay simulator is never specified and the paper contradicts itself on offline vs online fine-tuning.","tokens_in":17218,"tokens_out":3389,"would_cite":false,"duration_ms":32290,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A five-step offline reinforcement learning framework claims 31% resource-time savings and 23% shorter process spans by reordering activities in business workflows.","keywords":["prescriptive process monitoring","offline reinforcement learning","process-aware data augmentation","reward shaping","best next activity","Q-learning","event logs","process mining"],"falsifier":"Replay the learned policy on the energy-regulator event log while keeping each activity's real per-activity duration and order-dependent approval probabilities; if reordered traces do not reduce resource time by 31% and process span by 23% (or if they increase either), the central claim collapses.","tokens_in":16177,"feed_emoji":"📊","tokens_out":5675,"duration_ms":50524,"temperature":0.7,"pith_summary":"This paper proposes FORLAPS, a five-step framework that uses offline reinforcement learning with a state-dependent reward function to reorder activities in business processes, and claims it cuts resource time by 31% and total process span by 23% on a real event log from a Canadian energy regulator. The key idea is to penalize a disapproved activity more heavily the later in the trace it occurs, so the learned policy front-loads risky tasks and short-circuits cascading failures. To compensate for scarce historical data, the paper adds process-aware data augmentation that preserves business rules, enabling offline fine-tuning without online interaction. If the reported savings hold, the framework provides a practical decision-support tool for prescriptive process monitoring that goes beyond prediction to recommend optimal next steps.","feed_headline":"Offline RL cuts process resource time by 31%","feed_subtitle":"A five-step framework learns optimal next actions from event logs, also cutting process span 23%.","key_machinery":"The central object is the state-dependent reward-shaping function $R(s_t,a_t)=-r\\,|s_t|\\,\\mathbb{I}(s_t,a_t)$, which returns a penalty proportional to the number of activities already executed when the chosen activity leads to a KPI violation. This length-aware penalty is what makes the Q-learning agent prefer to surface risky activities early. The second load-bearing component is the process-aware data augmentation: timestamps are shifted by up to 10% of inter-activity time, non-critical activities are selectively removed from completed groups, and the synthetic traces are filtered against business-rule constraints, allowing the pre-trained Q-function to be fine-tuned offline as if exploration had continued.","core_discovery":"On its own terms, the paper establishes that a Q-learning agent over process states, trained with a length-aware reward $R(s_t,a_t)=-r\\,|s_t|\\,\\mathbb{I}(s_t,a_t)$ and fine-tuned on augmented traces, learns to recommend activity orders that outperform both a multi-task LSTM next-action recommender and a permutation-feature-importance baseline. When 15,000 test-set traces are replayed with the learned policy, the paper reports 31% savings in resource time spent and a 23% reduction in process time span, with an average 2.66 fewer days of resource time and 9.69 fewer days per case compared with a baseline of 8.52 days per activity and 40.56 days per case. The framework also yields a ranked set of best next activities rather than a single action, and its robustness is assessed with Damerau–Levenshtein distance across nine public event logs from healthcare, finance, permits, and operations.","pith_inferences":["The 31% and 23% figures depend on a replay counterfactual that the paper never fully specifies; if activity durations or approval outcomes are order-dependent in the real process, the realized savings could differ even if the Q-learning converged correctly.","The same reward-shaping principle — penalizing late failure linearly in trace length — could be ported to other continuous KPIs such as cost or defect counts, as long as the state encodes those dimensions.","A natural next test is a real A/B trial where the policy's reorderings are applied to a random subset of cases; if early disapproval does not reliably cancel downstream work, the resource-time gain shrinks.","The analogy to fast-fail in manufacturing suggests the framework may transfer well beyond business process mining, to any sequential task where late-stage failure is disproportionately expensive."],"forward_implications":["Applied to new applications at the energy regulator, the learned policy could reduce resource time spent on reviews by roughly a third and total approval time by about a quarter, if the replay assumptions hold.","Because fine-tuning is fully offline, the same recipe transfers to domains where online interaction is expensive, such as healthcare treatment pathways and permit processing.","The policy does not prescribe one rigid sequence; it outputs a ranked list of next-best activities, so decision-makers can choose among near-optimal actions under operational constraints.","The Damerau–Levenshtein validation indicates the framework both reproduces traces that lead to desired outcomes and avoids traces that lead to undesired ones, across different prefix lengths.","Performance is strongest when the event log has dense state coverage relative to its number of activities; sparse logs such as BPIC_2015 are an explicit boundary of the method."],"supporting_citations":[{"why":"Supplies the LSTM-based next-best-action baseline that FORLAPS is compared against for KPI savings.","marker":"Weinzierl et al., 2020"},{"why":"Provides the permutation-feature-importance baseline and the Sepsis dataset outcome definition used in evaluation.","marker":"Vazifehdoostirani et al., 2023"},{"why":"Framework for offline reinforcement learning that motivates the offline setting and the fine-tuning approach.","marker":"Levine et al., 2020"},{"why":"Comparison point for offline RL with generative augmentation; the paper contrasts its process-aware augmentation with this approach.","marker":"Bozorgi et al., 2023"},{"why":"Alternative outcome-oriented prescriptive monitoring via temporal logic, used as a reference for the BPIC_2015 and Traffic Fines outcome definitions.","marker":"Donadello et al., 2023"},{"why":"Comparison of RL and causal inference for process interventions, framing the timed-intervention baseline the paper distinguishes from next-best activity recommendation.","marker":"Weytjens et al., 2023"}],"fun_headline_variants":["Adaptive RL prescribes optimal process paths, cutting resource time 31%","FORLAPS: RL framework saves 31% resource time, 23% process span","Context-aware RL boosts process efficiency: 31% less resource time","Reinforcement learning prescribes next best steps, saving 31% time","New RL method cuts process time by 31% and span by 23%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The savings numbers assume that replaying 15,000 traces with the learned policy is a valid counterfactual: activities can be reordered freely, their durations and approval outcomes are order-independent, and an early disapproval allows later activities to be skipped.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive RL prescribes optimal process paths, cutting resource time 31%","FORLAPS: RL framework saves 31% resource time, 23% process span","Context-aware RL boosts process efficiency: 31% less resource time","Reinforcement learning prescribes next best steps, saving 31% time","New RL method cuts process time by 31% and span by 23%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000577,"raw_usage":{"total_tokens":2761,"prompt_tokens":1025,"completion_tokens":1736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":1633}},"tokens_in":641,"tokens_out":1736,"duration_ms":10539,"temperature":1.0,"reasoning_tokens":1633,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:08:33.545116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replay the learned policy on the energy-regulator event log while keeping each activity's real per-activity duration and order-dependent approval probabilities; if reordered traces do not reduce resource time by 31% and process span by 23% (or if they increase either), the central claim collapses.","supporting_citations":[],"review_version":1}