Pith. sign in

REVIEW 3 major objections 4 minor 36 references

A Procedural Framework for Assessing the Desirability of Process Deviations

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that process analysts can replace ad-hoc judgment about whether a deviation is problematic, acceptable, or beneficial with a five-step framework that assigns every deviation to one of seven mutually exclusive…

desk verdict A genuinely new procedural scaffold for deviation triage, but the correctness/completeness claim rests on qualitative self-report rather than demonstrated reliability. read the letter →

arxiv 2506.11525 v1 pith:UXQH32VL submitted 2025-06-13 cs.SE

Michael Grohs , Nadine Cordes , Jana-Rebecca Rehse This is my paper · ORCID
classification cs.SE
keywords processminingconformancecheckingdeviationdesirabilityproceduralframeworkbusinessmanagementactionrecommendationqualitativeevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Process mining can find where real executions differ from a prescribed process model, but it cannot say whether those differences are harmless, costly, or beneficial. This paper attempts to close that gap by turning deviation desirability into a repeatable procedure: a five-step framework that guides an analyst through eleven input factors, in a fixed order, to place each deviation into one of seven mutually exclusive categories. The categories are deliberately action-linked, so the output is not just a label but a recommendation to filter out, ignore, prevent, or adopt the behavior. A sympathetic reader would care because the current alternative is ad-hoc expert judgment, which is slow, hard to reproduce across analysts, and difficult to justify. The paper's evaluation with eight practitioners is presented as evidence that the framework achieves its two claimed properties: correct and complete components, and usefulness for orientation.

What carries the argument

The central object is the procedural framework itself: a five-step decision flow in which eleven input factors act as knockout criteria, meaning a negative answer can assign a final category and stop the flow, and seven output categories are each tied to an action recommendation. The load-bearing design is the order of the steps. First comes data quality; then model correctness and suitability; then case type and control; then compliance, outcome, performance, and standardization; and finally reaction effectiveness and cost. That ordering lets an analyst stop early whenever a deviation is a false alarm or an exception, saving a full impact analysis. The framework's two-part structure separates per-case representational problems from process-wide desirability, and it is this structure that the paper claims is correct, complete, and useful.

What would settle it

A controlled application of the framework in a domain not used in its development, such as healthcare or public-sector permitting, where analysts encounter a deviation whose deciding consideration is not one of the eleven factors, or where two analysts applying the same steps to the same traces land in different final categories because of the impact trade-off; finding an unmodelled factor that changes a category, or systematic inter-analyst disagreement, would contradict the completeness claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a deviation's desirability can be assessed procedurally rather than impressionistically. The paper claims that every deviation that conformance checking flags can be routed through two assessment parts: an individual-level part that first excludes false alarms caused by the log, false alarms caused by the model, and justified exceptions, and an aggregated-level part that evaluates true deviations collectively, weighing compliance, outcome, performance, and standardization under both direct and risk-and-opportunity perspectives. If the deviation has positive or negative impact, the analyst then weighs reaction effectiveness against reaction cost, yielding the final categories: false alarm (log), false alarm (model), exception, neutral deviation, positive deviation, negative deviation, or reaction-inefficient deviation. That final assignment is the paper's contribution: a taxonomy plus a procedural path to it.

Load-bearing premise

The framework stands on the assumption that the eleven input factors and their fixed order capture everything an analyst needs to judge desirability, and that analysts can reliably make the qualitative trade-offs the steps demand.

Editorial extensions

If this is right

  • If the framework is correct, an analyst can apply the five steps to any conformance deviation and end with one of seven categories, making desirability assessments more replicable across analysts.
  • The early knockout checks allow many deviations to be set aside as false alarms or exceptions without a full impact analysis, reducing analysis effort in practice.
  • Each category carries a recommended action, so the assessment output maps directly to decisions: filter out, ignore, prevent, or adopt.
  • The framework is claimed to be process-agnostic, so a single checklist could be reused across different business domains rather than being rebuilt for each process.
  • The aggregated-level step brings deviation frequency and collective behavior into desirability assessment, connecting single-case analysis to process-wide improvement efforts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One extension left implicit is that the knockout structure is effectively a decision tree, so the framework could be translated into a software wizard or automated classifier that asks the eleven factors in order; the paper does not specify that translation.
  • The paper's completeness claim is only as strong as the absence of reported gaps from eight evaluators, so a field study in domains beyond procurement, such as healthcare or public administration, would be the real test of the eleven-factor set.
  • Because step four leaves the trade-off between compliance and performance to the analyst, the framework likely standardizes the procedure more than the outcome; different organizations could weight those factors differently and arrive at different final categories.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents a procedural framework for assessing the desirability of process deviations, developed through a literature review (32 papers) and qualitative interviews and a focus group. The framework consists of two analysis parts and five steps, using eleven input factors as knockout criteria to assign seven output categories (false alarm log, false alarm model, exception, neutral, positive, negative, reaction-inefficient), each with an action recommendation. The authors claim that the framework's components are correct and complete (P1) and that it is useful for process analysts (P2), and they evaluate it through a deviation assessment task with eight practitioners. The paper includes supplementary materials and explicitly acknowledges limitations in quantitative evaluation.

Significance. If validated, the framework would fill a genuine gap: conformance checking identifies deviations but not their desirability. The work usefully synthesizes scattered literature and provides a structured, actionable procedure. The authors are transparent about their qualitative method and provide open supplementary materials. However, the current evidence for P1 is limited: there is no inter-rater reliability analysis, no gold-standard correctness test, and the completeness claim rests on the absence of reported missing factors from eight evaluators. The paper's honest acknowledgment of these limitations is a strength, but it also means the central claims are not yet fully supported.

major comments (3)
  1. [Section 5.1] Section 5.1 states that the evaluation did not examine whether participants arrive at a certain outcome because 'interpretative assumptions render classification outcomes incomparable.' Consequently, the paper provides no reliability evidence for the central claim that the framework categorizes deviations into mutually exclusive categories. The only support for P1 is the absence of reported gaps from eight practitioners, which is a weak basis for claiming correctness and completeness. At minimum, the authors should report a comparison of participants' final categories against the researchers' expected categories, or temper the P1 claim to a plausibility argument.
  2. [Section 4.2, Evaluate impact] The Evaluate impact step considers four input factors (compliance, outcome, performance, and standardization) but provides no prescriptive aggregation rule. The paper states in Section 6 that trade-offs among these factors 'are typically discussed with the management of the process.' This means the framework underdetermines the final desirability category whenever the factors conflict, as in the paper's own invoice example where time savings are weighed against compliance violations. The authors should either specify a decision rule or explicitly characterize the framework as an analytical aid rather than a deterministic classifier.
  3. [Section 6, Application Challenges] The paper concedes 'there might be ambiguity within output categories, meaning that, e.g., a deviation might be an exception in some cases whereas it is a negative deviation in other cases.' This directly conflicts with the abstract's claim of 'mutually exclusive desirability categories.' The authors should clarify whether mutual exclusivity is guaranteed by the procedure or is merely a design goal, and discuss how ambiguity should be resolved in practice.
minor comments (4)
  1. [Section 4.1] The phrase 'afalse alarm (log)' in the running text should be corrected to 'a false alarm (log)'.
  2. [Figure 2] Figure 2 is visually dense; a legend that clearly distinguishes input factors, assessment steps, output categories, and action recommendations would improve readability.
  3. [Section 5.2] The statement that 'the experts mostly converged on similar assessments' would be more informative if the paper reported the level of agreement or disagreement, for example by listing the categories assigned by each participant to a sample deviation.
  4. [Section 6] The repeated phrase 'out of scope' used for several expert suggestions could be replaced with a more structured discussion of the boundary conditions of the framework.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the framework is a qualitative synthesis and its evaluation limitations are explicitly acknowledged, not definitional reductions.

full rationale

This paper contains no mathematical derivation chain, so the classic circularity patterns do not arise: there are no fitted parameters renamed as predictions, no equations equivalent by construction, and no uniqueness theorems imported from the authors' prior work. The framework's five steps, eleven input factors, and seven output categories are assembled from a literature review of 32 papers, six development interviews, and a researcher focus group, and are presented as a conceptual procedure rather than as a formal result derived from prior theorems. The only self-citation, [Grohs et al. 2024], is used to supply high-level deviation patterns as evaluation stimuli; this does not justify the framework's correctness and is not load-bearing. The evaluation supports P1 and P2 through qualitative practitioner feedback, and the paper explicitly concedes in Section 5.1 that classification outcomes were not compared ('we did not evaluate whether participants arrive at a certain outcome') and in Section 6 that 'there might be ambiguity within output categories.' Those concessions weaken the empirical support for mutual exclusivity and reliability, but they are acknowledged limitations of a qualitative evaluation, not a circular reduction of the framework to its own inputs. No specific equation, fitted value, or self-citation chain can be exhibited that forces the claimed result, so no circularity is established.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The framework introduces no numbers fitted to data. Its conceptual choices (eleven factors, seven categories, five steps, knockout ordering) are grounded in a documented qualitative process but are not formally derived. The central claims rest on the assumptions that the factor set and ordering are complete and that analysts can apply them reliably, both of which are only weakly tested.

assumptions (3)
  • domain assumption The 11 input factors and their ordering constitute a complete and correct basis for desirability assessment.
    Stated as property P1 in Sections 3 and 4; not proven, only supported by literature review and small expert feedback.
  • domain assumption Process analysts can reliably make the qualitative judgments required (e.g., outweighing impacts, reaction effectiveness versus cost).
    The framework leaves trade-offs and factor weighting to the analyst; Sections 4.2 and 6 note high cognitive load and need for domain knowledge.
  • domain assumption Expert self-report during interviews is a valid indicator of framework quality.
    Evaluation in Section 5 relies on practitioner feedback rather than objective outcome measures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Procedural Framework for Assessing the Desirability of Process Deviations." pith.science (2026). https://pith.science/paper/UXQH32VL

@misc{pith2026250611525,
  author       = {Pith},
  title        = {Pith review of: A Procedural Framework for Assessing the Desirability of Process Deviations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UXQH32VL}},
  note         = {Machine review of arXiv:2506.11525}
}
read the original abstract

Conformance checking techniques help process analysts to identify where and how process executions deviate from a process model. However, they cannot determine the desirability of these deviations, i.e., whether they are problematic, acceptable or even beneficial for the process. Such desirability assessments are crucial to derive actions, but process analysts typically conduct them in a manual, ad-hoc way, which can be time-consuming, subjective, and irreplicable. To address this problem, this paper presents a procedural framework to guide process analysts in systematically assessing deviation desirability. It provides a step-by-step approach for identifying which input factors to consider in what order to categorize deviations into mutually exclusive desirability categories, each linked to action recommendations. The framework is based on a review and conceptualization of existing literature on deviation desirability, which is complemented by empirical insights from interviews with process analysis practitioners and researchers. We evaluate the framework through a desirability assessment task conducted with practitioners, indicating that the framework effectively enables them to streamline the assessment for a thorough yet concise evaluation.

Figures

Figures reproduced from arXiv: 2506.11525 by the authors.

Figure 1
Figure 1. Overview of framework development phases [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Procedural framework for deviation desirability assessment [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Example deviation from the invoice-to-cash process [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages

  1. [1]

    Adamo, G., Borgo, S., Di Francescomarino, C., Ghidini, C., Guarino, N., Sanfilippo, E.M.: Business process activity relationships: Is there anything beyond arrows? In: BPM Forum. pp. 53–70. Springer (2018)

    work page 2018

  2. [2]

    In: ICSC

    Adriansyah, A., van Dongen, B.F., Zannone, N.: Controlling break-the-glass through align- ment. In: ICSC. pp. 606–611. IEEE (2013)

    work page 2013

  3. [3]

    In: CO- DASPY

    Alizadeh, M., Zannone, N.: Risk-based analysis of business process executions. In: CO- DASPY . pp. 130–132. ACM (2016)

    work page 2016

  4. [4]

    CAIS 34, 1041–1066 (2014)

    Alter, S.: Theory of workarounds. CAIS 34, 1041–1066 (2014)

    work page 2014

  5. [5]

    In: AMCIS

    Andrade, E., van der Aa, H., Leopold, H., Alter, S., Reijers, H.A.: Factors leading to business process noncompliance and its positive and negative effects: Empirical insights from a case study. In: AMCIS. pp. 1–10. AIS (2016)

    work page 2016

  6. [6]

    In: BPMDS EMMSAD

    Andree, K., Bano, D., Weske, M.: Beyond temporal dependency: An ontology-based ap- proach to modeling causal structures in business processes. In: BPMDS EMMSAD. pp. 152–166. Springer (2023)

    work page 2023

  7. [7]

    In: EDOC

    Andree, K., Pufahl, L.: Am i allowed to change an activity relationship? - a metamodel for business process redesign. In: EDOC. pp. 18–35. Springer (2025)

    work page 2025

  8. [8]

    In: BPM Workshops

    van Beest, N.R., Weber, I.: Behavioral classification of business process executions at run- time. In: BPM Workshops. pp. 339–353. Springer (2017)

    work page 2017

Show all 36 references
  1. [9]

    CAIS 37, 205–224 (2015)

    vom Brocke, J., Simons, A., Riemer, K., Niehaves, B., Plattfaut, R., Cleven, A.: Standing on the shoulders of giants: Challenges and recommendations of literature search in information systems research. CAIS 37, 205–224 (2015)

    2015

  2. [10]

    Springer (2018)

    Carmona, J., van Dongen, B., Solti, A., Weidlich, M.: Conformance Checking - Relating Processes and Models. Springer (2018)

    2018

  3. [11]

    In: Handbook on Business Process Management and Digital Transformation, pp

    Depaire, B.: Process deviation analysis. In: Handbook on Business Process Management and Digital Transformation, pp. 220–249. Edward Elgar (2024)

    2024

  4. [12]

    In: BPM Workshops

    Depaire, B., Swinnen, J., Jans, M., Vanhoof, K.: A process deviation analysis framework. In: BPM Workshops. pp. 701–706. Springer (2013)

    2013

  5. [13]

    Dumas, M., Maggi, F.M.: Enabling Process Innovation via Deviance Mining and Predictive Monitoring, pp. 145–154. Springer (2015) Framework for Assessing Deviation Desirability 17

    2015

  6. [14]

    Dunzer, S., Liessmann, A., Stierle, M., Matzner, M.: Conceptualizing business process vio- lations. In: PLC. pp. 1–14. CEUR-WS (2024)

    2024

  7. [15]

    In: S-BPM ONE

    Dunzer, S., Stierle, M., Matzner, M., Baier, S.: Conformance checking: A state-of-the-art literature review. In: S-BPM ONE. p. 1–10. ACM (2019)

    2019

  8. [16]

    Grohs, M., van der Aa, H., Rehse, J.R.: Beyond log and model moves in conformance check- ing: Discovering process-level deviation patterns. In: BPM. pp. 381–399. Springer (2024)

    2024

  9. [17]

    Int J Acc Inf Syst 32, 44–58 (2019)

    Jans, M., Hosseinpour, M.: How active learning and process mining can act as continuous auditing catalyst. Int J Acc Inf Syst 32, 44–58 (2019)

    2019

  10. [18]

    In: EDOC

    Knoblich, S., Pfahlsberger, L., Mendling, J.: Conceptualizing change activities in process mining. In: EDOC. pp. 155–168. Springer (2025)

    2025

  11. [19]

    Koschmider, A., Kaczmarek, K., Krause, M., van Zelst, S.J.: Demystifying noise and outliers in event logs: Review and future directions. In: BPM. pp. 123–135. Springer (2022)

    2022

  12. [20]

    Sage (2014)

    Krueger, R.A., Casey, M.A.: Focus Groups: A Practical Guide for Applied Research. Sage (2014)

    2014

  13. [21]

    In: ICPM

    Laghmouch, M., Jans, M., Depaire, B.: Classifying process deviations with weak supervi- sion. In: ICPM. pp. 89–96. IEEE (2020)

    2020

  14. [22]

    J Inf Syst 30(2), 135–157 (2016)

    Li, P., Chan, D.Y ., Kogan, A.: Exception prioritization in the continuous auditing environ- ment: A framework and experimental evaluation. J Inf Syst 30(2), 135–157 (2016)

    2016

  15. [23]

    In: CAiSE

    Maggi, F.M., Di Francescomarino, C., Dumas, M., Ghidini, C.: Predictive monitoring of business processes. In: CAiSE. pp. 457–472. Springer (2014)

    2014

  16. [24]

    In: CAiSE

    Maggi, F.M., Montali, M., Peñaloza, R.: Probabilistic conformance checking based on declarative process models. In: CAiSE. pp. 86–99. Springer (2020)

    2020

  17. [25]

    Theoretical Foundation, Basic Procedures and Software Solution

    Mayring, P.: Qualitative Content Analysis. Theoretical Foundation, Basic Procedures and Software Solution. Gesis (2014)

    2014

  18. [26]

    Inf Organ 17(1), 2–26 (2007)

    Myers, M.D., Newman, M.: The qualitative interview in is research: Examining the craft. Inf Organ 17(1), 2–26 (2007)

    2007

  19. [27]

    In: BPM Workshops

    Outmazgin, N.: Exploring workaround situations in business processes. In: BPM Workshops. pp. 426–437. Springer (2013)

    2013

  20. [28]

    In: BPM Forum

    Outmazgin, N., van der Waal, W., Beerepoot, I., Hadar, I., van de Weerd, I., Soffer, P.: From automatic workaround detection to process improvement: A case study. In: BPM Forum. pp. 372–390. Springer (2023)

    2023

  21. [29]

    In: BPM Forum

    Park, G., Benzin, J.V ., van der Aalst, W.M.P.: Detecting context-aware deviations in process executions. In: BPM Forum. pp. 190–206. Springer (2022)

    2022

  22. [30]

    Sage (2015)

    Saldaña, J.: The Coding Manual for Qualitative Researchers. Sage (2015)

    2015

  23. [31]

    In: EDOC

    Stertz, F., Mangler, J., Rinderle-Ma, S.: Data-driven improvement of online conformance checking. In: EDOC. pp. 187–196. IEEE (2020)

    2020

  24. [32]

    In: BPM Workshops

    Swinnen, J., Depaire, B., Jans, M., Vanhoof, K.: A process deviation analysis-a case study. In: BPM Workshops. pp. 87–98. Springer (2012)

    2012

  25. [33]

    Bus Inf Syst Eng pp

    van der Waal, W., van de Weerd, I., Beerepoot, I., Lu, X., Kappen, T., Haitjema, S., Reijers, H.: Putting the SWORD to the test: Finding workarounds with process mining. Bus Inf Syst Eng pp. 1–20 (2024)

    2024

  26. [34]

    J Bus Ana 5(1), 76–100 (2022)

    Weinzierl, S., Wolf, V ., Pauli, T., Beverungen, D., Matzner, M.: Detecting temporal workarounds in business processes – a deep-learning-based method for analysing event log data. J Bus Ana 5(1), 76–100 (2022)

    2022

  27. [35]

    J Biomed Inform 85, 155–167 (2018)

    Yang, S., Sarcevic, A., Farneth, R.A., Chen, S., Ahmed, O.Z., Marsic, I., Burd, R.S.: An ap- proach to automatic process deviation detection in a time-critical clinical process. J Biomed Inform 85, 155–167 (2018)

    2018

  28. [36]

    In: IPMU

    Zhang, S., Genga, L., Dekker, L., Nie, H., Lu, X., Duan, H., Kaymak, U.: Towards multi- perspective conformance checking with aggregation operations. In: IPMU. pp. 215–229. Springer (2020)

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.