Pith. sign in

REVIEW 3 major objections 4 minor 25 references

AI-Enhanced Business Process Automation: A Case Study in the Insurance Domain Using Object-Centric Process Mining

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LLM deployed in production at an insurer identified 1,034 claim parts over five months versus 68 identified by human handlers, a 1,420% increase, while the investigation stage became the new bottleneck.

desk verdict A real production case study worth reading, but the headline 1420% scaling figure is not demonstrated because it compares raw AI predictions to human reports without measuring AI precision. read the letter →

arxiv 2504.17295 v1 pith:F4BFWPIB submitted 2025-04-24 cs.AI

Shahrzad Khayatbashi , Viktor Sjölind , Anders Gran{aa}ker , Amin Jalali This is my paper · ORCID
classification cs.AI
keywords AI-DrivenAutomationBusinessProcessReengineeringDigitalTransformationManagementObject-CentricMiningLargeLanguageModelsClaimsProcessingScalability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Faced with rising claim volumes, an insurance company replaced a manual, knowledge-intensive screening step with a large language model that scans claim descriptions and notes and predicts which claims contain claim parts requiring special handling. The paper uses object-centric process mining on a production event log covering 3,743 claims over five months to compare the old and new process variants running in parallel. It reports that the AI identified 1,034 claim parts while human handlers identified 68, a 1,420% scale-up, and that the ratio of claims with claim parts rose from 1.82% to 27.62%, in line with what business experts expected. The same analysis shows the next step did not scale: only 26 investigation cases were created, because the number of investigators had been sized for the lower volume. The paper's central point is that AI can remove one bottleneck only to create another, and that object-centric logs make that bottleneck shift visible and attributable to each process variant.

What carries the argument

The central object is the Object-Centric Event Log (OCEL 2.0), a data format in which each event can relate to several object types instead of a single case; here the object types are Customer, Claim, Claim Note, Claim Part, AI Model, Claim Handler, and Claim Part Investigator. Two AI-specific activities, scan claim and predict claim part, record when the model reads a claim's description and notes and when it flags a claim part. Analysis proceeds through Object-Centric Directly-Follows Graphs (OC-DFGs) and Object-Centric Petri Nets, with drill-down, unfolding, and flattening operations used to separate human-only from AI-involved paths. The mechanism that carries the argument is the ability to filter and unfold the log by role and by AI involvement, which lets the authors attribute each investigation case to humans, the AI, or both; the paper also notes a limitation, that OCEL 2.0 cannot represent expired object-to-object relations, so the log keeps only the most recent claim note before each scan.

What would settle it

Re-extract the same 3,743 claims with full claim-note history, or with alternative snapshots of the notes available at scan time, and recompute the AI and human identification counts and the attribution of the 26 investigation cases; if those numbers move materially, the 1,420% scaling figure and the 3/18/5 complementarity are not robust.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a production large language model can scale claim-part identification far beyond human throughput while maintaining acceptable quality, and that object-centric process mining can quantify the consequences. The model identified claim parts for 1,034 of 3,743 claims (27.62%), versus 68 claims (1.82%) identified by human handlers, whom the company's baseline analysis estimated miss 30% of true claim parts. The paper reports a 1,420% scaling of the identification step and notes that among the 26 claim parts opened for investigation, 18 were found by both AI and humans, 5 were found only by AI, and 3 were found only by humans. It then argues that this success exposed a new constraint: claim part investigators opened only 26 cases, so the downstream investigation stage, not identification, now limits the claims management process. The paper concludes that scaling a single step does not automatically add business value and that end-to-end process redesign is needed when AI is introduced.

Load-bearing premise

The load-bearing premise is that the single most recent claim note retained in the log before each AI scan faithfully represents the claim text the AI actually read in production, so that the counts of AI-identified and human-identified claim parts are accurate rather than artifacts of log extraction.

Editorial extensions

If this is right

  • Automating a single bottleneck step in a business process shifts rather than removes the constraint; downstream capacity must be planned in the same reengineering effort.
  • Object-centric event logs make it possible to evaluate AI and human process variants during a gradual transition, because outcomes can be attributed to each variant by filtering on activity and object type.
  • Flattened, single-case graphs are more useful than full object-centric models for communicating results to business stakeholders, even when the object-centric representation is what enables the analysis.
  • Recall, rather than F1-score, is the right quality target when missed cases are far more costly than false positives, as in claim-part identification.
  • If the AI's identification capacity is used fully, the organization must hire or train more investigators, automate parts of the investigation step, or redesign the workflow, or the added detections will not turn into handled cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same pattern likely generalizes to other AI-screening applications, such as medical triage, fraud alerting, or content moderation: a high-recall model increases queue volume at the human review stage, and the new queue location is visible only if the log preserves the object relations that connect screening outputs to downstream work.
  • A direct test of the paper's interpretation would be to measure investigation cycle time and investigator utilization before and after deployment; a growing queue with rising wait times would confirm the bottleneck shift, while a flat queue would suggest the 26-investigation count reflects business selection criteria rather than capacity constraints.
  • Because the extracted log keeps only one claim note per claim before the AI scan, re-running the same comparison on a temporal event knowledge graph that preserves note history would show whether the 1,034-versus-68 contrast and the 3/18/5 complementarity counts depend on the snapshot assumption.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a real-world case study at If P&C Insurance in which a GPT-4o-based LLM was deployed in production to identify insurance claim parts, replacing a manual identification process that had been identified as a scalability bottleneck. The authors follow a combined BPR, CRISP-DM, and OCPM2 methodology, extract an OCEL 2.0 event log over five months covering 3,743 claims, and use object-centric and flat process mining to compare human and AI identification behavior. Their central empirical claim is that the AI scaled claim-part identification by roughly 1420%: humans reported 68 claim parts (1.82% of claims) while the AI predicted 1,034 claim parts (27.62% of claims). They further report that downstream claim-part investigation became the new bottleneck, with only 26 investigation cases created, and that among those 26 cases, AI alone identified 5, humans alone identified 3, and both identified 18. The paper also discusses lessons learned about process-mining visualization and stakeholder communication, and concludes that AI-driven automation can shift bottlenecks rather than eliminate them.

Significance. If the empirical counts are reliable, the paper provides a valuable, rare, real-world demonstration of using object-centric process mining to quantify the side effects of introducing an LLM into a knowledge-intensive insurance process. The observation that AI can remove one bottleneck only to create another, and that OCPM can make this trade-off visible from production event logs, is practically important and would be of interest to the BPM and process-mining communities. The paper is transparent about several limitations, including the OCEL 2.0 restriction on representing expired relations and the difficulty of communicating object-centric models to stakeholders. The main strength is the concrete, company-scale empirical setting and the honest reporting of downstream constraints. However, the central quantitative claim about a 1420% scaling depends on treating raw LLM predictions as equivalent to human-identified claim parts without a production precision measurement, which limits the force of the conclusions as currently stated.

major comments (3)
  1. [Section 3.3, Q2 and Fig. 6; Section 3.2, Table 1] The 1420% scaling figure compares 1,034 pCP events (raw LLM predictions) with 68 rCP events (hand-reported claim parts). The pCP events are model outputs, not validated claim parts, and the paper provides no production precision estimate for the 1,034 predictions. The only production-level validation reported is that AI was involved in 23 of the 26 investigation cases, which is a recall-like figure for the investigated subset, not a precision measure for all predictions. Moreover, the numbers are internally tension-prone: if the stated 70% human recall baseline applies in production, then 68 human reports imply roughly 97 true claim parts, so the AI's 1,034 predictions would imply a very high false-positive rate; conversely, if 1,034 is the correct count of true claim parts, then human production recall would be 68/1034 ≈ 6.6%, far below the 70% baseline. The phrase "confirmed by the business" does not quantify precision. The authors should either reframe the claim as "candidate lead generation" or provide a production precision evaluation (e.g., a random sample of the 1,034 predictions reviewed by investigators) before asserting that the process has scaled by 1420%.
  2. [Section 3.3, Implementation and Log Extraction] The extracted OCEL 2.0 log retains only the most recent claim note prior to the scan activity because OCEL 2.0 cannot represent expired object-to-object relations. Since the AI in production reads claim descriptions and notes, and the log is the basis for the pCP counts and the Q3/Q4 complementarity results, it is unclear whether the documented 1,034 predictions and the 3/5/18 breakdown reflect the data the AI actually used or an artifact of this one-note snapshot. The authors acknowledge the limitation, but they should clarify whether the pCP event counts come directly from the production AI system (which would be unaffected by the OCEL note-filtering) or from the reconstructed OCEL, and they should discuss how sensitive the reported ratios are to this filtering choice.
  3. [Section 3.3, Q3/Q4 and Fig. 9] The complementarity findings (AI missed 3, humans missed 5, both identified 18) are computed only over the 26 investigation cases, which is a small, deliberately selected subset of the 3,743 claims (investigators choose cases based on business criteria). As reported, the numbers describe the investigated set, not the full population of claims or all identified claim parts. The paper should state this limitation directly in the Q3/Q4 analysis and temper any conclusion about AI's general added value in identification, since the 26 cases are not a random or representative sample of the claims processed during the study period.
minor comments (4)
  1. [Section 3.3, Fig. 4] The figure shows pCP with frequency 1,069 and a loop, while the text correctly notes that the incoming flow shows 1,034 unique predictions; adding an explicit label for the unique prediction count on the figure would reduce ambiguity.
  2. [References] Reference [14] contains the typo "Accpeted" and reference [17] says "Accepted in BPMDS 2025" without a year; these should be corrected for consistency.
  3. [Section 3.3, Implementation and Log Extraction] The sentence "This limitation does not exist in all OCED formats" appears to contain a typo; it should presumably be "OCEL" or "object-centric event data formats."
  4. [Section 3.1] The company name is rendered with a typographical oddity: "If P&C Insurance , a property..." has a stray space before the comma; this should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the scaling ratio is a direct count of production events rather than a fitted or self-cited prediction.

full rationale

The paper's central quantitative claim, the 1420% scaling figure, is an arithmetic ratio of two independently recorded production event counts from the extracted OCEL 2.0 log: 1034 AI predictions (pCP) versus 68 human reports (rCP) over the same 3743 claims. Neither count is a fitted parameter, an optimized value, or a quantity defined in terms of the other; they are observed frequencies from the company's event data, and the conclusion that AI shifted the bottleneck to investigation is based on the further observed count of 26 investigation cases. The Q3/Q4 complementarity results (3, 5, 18) are computed under the paper's explicit operationalization that an investigated claim part counts as truly identified; that definition is disclosed, and the findings were confirmed against source systems, so no conclusion is forced by the definition itself. The self-citations to OCPM2, OLAP operations, tEKG, and drill-down/unfolding appear as methodological scaffolding and as a statement of an OCEL 2.0 limitation; none of these citations supplies a theorem or fitted value that determines the empirical outcome. The reviewer's concern that production precision of the LLM was not measured is a legitimate external-validity threat to the interpretation of 1034 as 'true' claim parts, but it is not a circularity: the paper's own derivation does not equate the conclusion with an input by construction. No load-bearing step reduces to a self-citation or to a fitted parameter renamed as a prediction. Under the hard requirement that circularity be exhibited by specific reduction, none is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical case study. The central results depend on data fidelity and ground truth, not on fitted coefficients; the main burden is the assumption that the extracted OCEL, the expert-confirmed percentages, and the offline model evaluation all reflect the production reality.

assumptions (4)
  • domain assumption The OCEL 2.0 log, with the one-note-per-claim simplification, faithfully represents the executed process for the purpose of comparing human and AI identification.
    Section 3.3, Implementation and Log Extraction: the authors retain only the most recent note prior to the scan activity because OCEL 2.0 cannot represent expired object relations; the analysis assumes this lossy representation does not bias the measured counts.
  • domain assumption The temporal data warehouse and the operational KPI extraction capture all relevant claim, note, claim part, and investigation events without omissions or misattributions.
    Section 3.3, Implementation and Log Extraction: extraction required combining a temporal warehouse with an operational system; the verification matrix in Fig. 3 checks relations but not completeness of events.
  • domain assumption Business experts' confirmation that the human identification rate (1.82%) is too low and the AI rate (27.62%) matches expectations is a valid external ground truth.
    Sections 3.3 Q1 and Q2: the percentages are validated by 'business experts we interviewed'; no independent quantitative audit is described beyond stakeholder discussion.
  • domain assumption The offline evaluation of GPT-4o version 5 (81% recall, 81% precision on English) transfers to the production deployment with GPT-4o-0806 over the 5-month window.
    Section 3.2, Table 1: production model behavior is not separately measured against ground truth; the paper assumes the deployed model performs at the level of the version selected in offline evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI-Enhanced Business Process Automation: A Case Study in the Insurance Domain Using Object-Centric Process Mining." pith.science (2026). https://pith.science/paper/F4BFWPIB

@misc{pith2026250417295,
  author       = {Pith},
  title        = {Pith review of: AI-Enhanced Business Process Automation: A Case Study in the Insurance Domain Using Object-Centric Process Mining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4BFWPIB}},
  note         = {Machine review of arXiv:2504.17295}
}
read the original abstract

Recent advancements in Artificial Intelligence (AI), particularly Large Language Models (LLMs), have enhanced organizations' ability to reengineer business processes by automating knowledge-intensive tasks. This automation drives digital transformation, often through gradual transitions that improve process efficiency and effectiveness. To fully assess the impact of such automation, a data-driven analysis approach is needed - one that examines how traditional and AI-enhanced process variants coexist during this transition. Object-Centric Process Mining (OCPM) has emerged as a valuable method that enables such analysis, yet real-world case studies are still needed to demonstrate its applicability. This paper presents a case study from the insurance sector, where an LLM was deployed in production to automate the identification of claim parts, a task previously performed manually and identified as a bottleneck for scalability. To evaluate this transformation, we apply OCPM to assess the impact of AI-driven automation on process scalability. Our findings indicate that while LLMs significantly enhance operational capacity, they also introduce new process dynamics that require further refinement. This study also demonstrates the practical application of OCPM in a real-world setting, highlighting its advantages and limitations.

Figures

Figures reproduced from arXiv: 2504.17295 by the authors.

Figure 1
Figure 1. The followed research methodology based on BPR [ [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The conceptual model illustrating the identified object types and their relation [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The verification matrix showing the number of relations between activities and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The discovered Directly-Follows Graph (DFG) of claim part identification pro [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The discovered DFG depicting human performance in identifying Claim Parts [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: The discovered DFG depicting AI performance in identifying Claim Parts. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: The discovered OC-DFG separating cases identified by humans alone ( [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: The discovered OC-DFG separating cases identified by AI alone ( [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Number of investigation created based on AI & Human identification. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Analyzing interconnected processes: using object-centric process mining to analyze procure- ment processes

    Alessandro Berti, Urszula Jessen, Gyunam Park, Majid Rafiei, and Wil MP van der Aalst. Analyzing interconnected processes: using object-centric process mining to analyze procure- ment processes. International Journal of Data Science and Analytics, pages 1–23, 2023

    work page 2023

  3. [3]

    Ocel (object-centric event log) 2.0 specification

    Alessandro Berti, Istv ´an Koren, Jan Niklas Adams, Gyunam Park, Benedikt Knopp, Nina Graves, Majid Rafiei, Lukas Liß, Leah Tacke Genannt Unterberg, Yisong Zhang, et al. Ocel (object-centric event log) 2.0 specification. arXiv preprint arXiv:2403.01975, 2024

    arXiv 2024

  4. [4]

    Limiting variety by standardizing and controlling knowledge intensive processes

    Ilia Bider and Amin Jalali. Limiting variety by standardizing and controlling knowledge intensive processes. In 2016 IEEE 20th International Enterprise Distributed Object Com- puting Workshop (EDOCW), pages 1–9. IEEE, 2016

    work page 2016

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877– 1901, 2020

    1901

  6. [6]

    Exploratory data analysis (eda)

    Estelle Camizuli and Emmanuel John Carranza. Exploratory data analysis (eda). The ency- clopedia of archaeological sciences, pages 1–7, 2018

    work page 2018

  7. [7]

    Crisp-dm 1.0: Step-by-step data mining guide

    Pete Chapman, Julian Clinton, Randy Kerber, Thomas Khabaza, Thomas Reinartz, Colin Shearer, Rudiger Wirth, et al. Crisp-dm 1.0: Step-by-step data mining guide. SPSS inc, 9(13):1–73, 2000

    work page 2000

  8. [8]

    About us

    If P&C Insurance. About us. https://www.if-insurance.com/about-if/about-us, 2025. Ac- cessed: 2025-03-13

    work page 2025

Show all 25 references
  1. [9]

    Aspect mining in business process management

    Amin Jalali. Aspect mining in business process management. In Perspectives in Business Informatics Research: 13th International Conference, BIR 2014, Lund, Sweden, September 22-24, 2014. Proceedings 13, pages 246–260. Springer, 2014

    2014

  2. [10]

    Object type clustering using markov directly-follow multigraph in object-centric process mining

    Amin Jalali. Object type clustering using markov directly-follow multigraph in object-centric process mining. IEEE Access, 10:126569–126579, 2022

    2022

  3. [11]

    Transforming event knowledge graph to object-centric event logs: A comparative study for multi-dimensional process analysis

    Shahrzad Khayatbashi, Olaf Hartig, and Amin Jalali. Transforming event knowledge graph to object-centric event logs: A comparative study for multi-dimensional process analysis. In International Conference on Conceptual Modeling, pages 220–238. Springer, 2023

    2023

  4. [12]

    Transforming object-centric event logs to temporal event knowledge graphs

    Shahrzad Khayatbashi, Olaf Hartig, and Amin Jalali. Transforming object-centric event logs to temporal event knowledge graphs. In International Conference on Business Process Management, pages 300–313. Springer, 2024

    2024

  5. [13]

    Advancing object-centric process mining with multi-dimensional data operations

    Shahrzad Khayatbashi, Najmeh Miri, and Amin Jalali. Advancing object-centric process mining with multi-dimensional data operations. Springer, 2025

    2025

  6. [14]

    OLAP operations for object-centric process mining

    Shahrzad Khayatbashi, Najmeh Miri, and Amin Jalali. OLAP operations for object-centric process mining. In Accpeted in CAiSE Forum 2025. Springer, 2025

    2025

  7. [15]

    Overstock problems in a purchase-to-pay process: An object-centric process mining case study

    Dina Kretzschmann, Gyunam Park, Alessandro Berti, and Wil MP van der Aalst. Overstock problems in a purchase-to-pay process: An object-centric process mining case study. In International Conference on Advanced Information Systems Engineering , pages 347–359. Springer, 2024

    2024

  8. [16]

    Uncovering patterns in object-centric process mining: An approach using drill-down and roll-up techniques

    Najmeh Miri and Amin Jalali. Uncovering patterns in object-centric process mining: An approach using drill-down and roll-up techniques. In Pari Delir Haghighi, Michal Gregu ˇs, Gabriele Kotsis, and Ismail Khalil, editors, Information Integration and Web Intelligence , pages 49...

    2025

  9. [17]

    OCPM 2: Ex- tending the Process Mining Methodology for Object-Centric Event Data Extraction, 2025

    Najmeh Miri, Shahrzad Khayatbashi, Jelena Zdravkovic, and Amin Jalali. OCPM 2: Ex- tending the Process Mining Methodology for Object-Centric Event Data Extraction, 2025. Accepted in BPMDS 2025. 16 Shahrzad Khayatbashi et al

    2025

  10. [18]

    Business process reengineering: A theoretical framework and an integrated model

    Jaideep Motwani, Ashok Kumar, James Jiang, and Mohamed Youssef. Business process reengineering: A theoretical framework and an integrated model. International Journal of Operations & Production Management, 18(9/10):964–977, 1998

    1998

  11. [19]

    Json mode - openai api

    OpenAI. Json mode - openai api. https://platform.openai.com/docs/guides/structured- outputs, 2025. Accessed: 2025-03-16

    2025

  12. [20]

    Structured outputs - openai api

    OpenAI. Structured outputs - openai api. https://platform.openai.com/docs/guides/structured- outputs, 2025. Accessed: 2025-02-27

    2025

  13. [21]

    Analyzing an after- sales service process using object-centric process mining: A case study

    Gyunam Park, Sevde Aydin, C ¨uneyt U˘gur, and Wil MP van der Aalst. Analyzing an after- sales service process using object-centric process mining: A case study. In International Conference on Process Mining, pages 406–418. Springer, 2023

    2023

  14. [22]

    Object-centric process mining: dealing with divergence and con- vergence in event data

    Wil MP van der Aalst. Object-centric process mining: dealing with divergence and con- vergence in event data. In Software Engineering and Formal Methods: 17th International Conference, SEFM 2019, Oslo, Norway, September 18–20, 2019, Proceedings 17, pages 3–

    2019

  15. [23]

    Discovering object-centric petri nets

    Wil MP van der Aalst and Alessandro Berti. Discovering object-centric petri nets. Funda- menta informaticae, 175(1-4):1–40, 2020

    2020

  16. [24]

    van Eck, Xixi Lu, Sander J

    Maikel L. van Eck, Xixi Lu, Sander J. J. Leemans, and Wil M. P. van der Aalst. PM 2: A process mining project methodology. In Jelena Zdravkovic, Marite Kirikova, and Paul Johannesson, editors, Advanced Information Systems Engineering, pages 297–313. Springer International Publ...

    2015

  17. [25]

    Towards a theory on process automation effects

    Hoang Vu, Jennifer Haase, Henrik Leopold, and Jan Mendling. Towards a theory on process automation effects. In International Conference on Business Process Management , pages 285–301. Springer, 2023

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.