REVIEW 3 major objections 5 minor 1 cited by
On three standard automation tasks, RPA beat an LLM computer-use agent on speed and reliability, while the agent cut development time from hours to minutes.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
In head-to-head tests on three standard automation challenges, RPA bots beat LLM computer-use agents on speed and reliability, while the agents needed far less development time.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Useful early empirical anchor, but the abstract oversells reliability and the P3 task versions may not be like-for-like. the 3 major comments →
Are LLM Agents the New RPA? A Comparative Study with RPA Across Enterprise Workflows
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On the paper's own terms, the central discovery is a quantified trade-off, not a single winner. In the two processes that were fully benchmarked, the RPA tool completed runs in 53.9 and 20 seconds on average with 10/10 successes, while the LLM computer-use agent took 109.8 and 202.8 seconds with 9/10 and 6/10 successes, and in a third process the agent could not finish the full task at all. Yet the agent achieved its first successful run after only 10–15 minutes of prompt writing, compared with roughly 38–240 minutes of RPA development, and it correctly recognized fields and OCR text even when page layouts shifted. The authors reject their hypothesis that AACU is faster, find no statisticall
What carries the argument
The load-bearing object is the controlled benchmark protocol: three standard RPA Challenge tasks (dynamic web-form data entry, threshold-based monitoring, invoice OCR extraction) run ten times per technology where possible, measuring execution time, success count, and time-to-first-successful-run. AACU is represented by a public-beta computer-use agent that receives screenshots and emits mouse and keyboard actions; RPA is represented by a commercial automation studio. Welch's t-test on run times, Fisher's exact test on success rates, and descriptive comparison of development times carry the quantitative argument.
Load-bearing premise
The invoice task was shortened to a single page for the LLM agent, and the paper does not state clearly whether the RPA baseline ran the same shortened version or the full challenge, so the P3 speed and reliability comparison may not be like-for-like.
What would settle it
Rerun P2 and P3 with byte-identical task definitions for both technologies (same pages, same row counts, same output format), and also run the full, unshortened invoice task with the agent using a larger context window. If the agent then matches or beats RPA on execution time and success rate, the paper's rejection of its first two hypotheses would be overturned.
If this is right
- For stable, repetitive, high-volume workflows, RPA remains the stronger choice on current evidence; AACU is not yet a drop-in replacement.
- For lightweight, infrequent automations, the agent's very low setup time can make automation economically viable where scripting effort previously blocked it.
- AACU's demonstrated ability to read shifting form layouts and OCR content points to tasks where RPA's rigid selectors fail.
- Hybrid designs, with an agent orchestrating and RPA executing critical steps, become an obvious next step for combining quick setup with reliable runs.
- The paper's failure cases (session freezes, lost context on long tasks) make context length and state persistence the key levers for closing the reliability gap.
Where Pith is reading between the lines
- If the measured gap is mostly beta-tool instability rather than a fundamental LLM limit, rerunning these benchmarks on newer computer-use models could shrink or reverse the speed and reliability differences.
- The reported ~$0.28 cost per invoice run, combined with near-zero setup time, implies an economic crossover: for processes that run rarely, an agent may be cheaper end-to-end even though each execution is slower.
- Because the invoice task was shortened to the first page only for the agent, the paper defines AACU's current viability as limited to small-context tasks; testing longer document sets would show whether the reliability gap grows with task length.
- A direct test that holds the task definition byte-identical and deliberately changes page layouts would separate the agent's perceptual strengths from its orchestration weaknesses.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript compares UiPath RPA with Anthropic's Computer Use Agent (referred to as AACU) on three standardized RPA challenge tasks: data entry (P1), stock market monitoring (P2), and invoice extraction (P3). For each process, the authors report execution time, success rate, and development time, and use Welch t-tests and Fisher exact tests to evaluate hypotheses H1 (speed), H2 (reliability), and H3 (development time). The results show that RPA is significantly faster on P2 and P3, that reliability differences are not statistically significant, and that AACU appears to require less development time. Important caveats are present: P1 has only a single AACU run, P3 was shortened for AACU, and development time is measured as a single observation per process. The paper concludes that RPA outperforms AACU in speed and reliability and that AACU offers significantly reduced development time and better flexibility.
Significance. If the results are fully supported, this paper would provide useful early empirical evidence on a topic largely driven by vendor claims. The use of well-known benchmark processes, a version-pinned agent environment, a public repository, and explicit discussion of limitations are genuine strengths. The main value is as a preliminary, field-experiment-style benchmark rather than a definitive evaluation. However, two load-bearing issues must be addressed: the abstract/conclusion overstate reliability despite non-significant Fisher tests, and the description of P3 leaves open the possibility that RPA and AACU ran tasks with different scope. After revision, including clarification or re-running, this could be an acceptable archival contribution.
major comments (3)
- [Sections 3 and 4, Process 3] The description of P3 is ambiguous about whether the UiPath baseline ran the same task as the AACU. Section 3 is headed 'RPA challenge Invoices – Simplified for AACU' and states 'we limited the scope of this challenge to the first page only, which contains four invoices.' Section 4 states P3 'was shortened to fit within the current context limitations of the agent.' The paper never states explicitly that the UiPath implementation was also restricted to the first page. Because P3 is one of only two processes with full 10-run measurements for both technologies, and it underlies the largest speed difference (20 s vs 202.8 s) and the only near-significant reliability comparison (p=0.0866), a task mismatch would confound the H1 and H2 conclusions. Please state explicitly that both technologies ran exactly the same scope. If they did not, rerun the comparison on a matched task or remove P3 fro
- [Section 5, H2; Abstract and Conclusion] The Fisher exact tests in Table 3 are p=1.0 for P2 and p=0.0866 for P3; the Results section correctly says 'we cannot confirm that the AACU is either better or worse than RPA in terms of reliability based on the current data.' However, the Abstract and Section 7 claim that 'RPA outperforms AACU in ... reliability' and 'RPA outperformed AACU in both speed and reliability.' This is an overstatement of non-significant evidence. The reliability conclusions should be rephrased to say that the observed success rates favored RPA but the differences were not statistically significant at the 0.05 level.
- [Section 5, Development Time Analysis; Conclusion] The claim that 'AACU significantly reduces development time' is not supported by the data. Table 4 reports one measurement per process, and P1 has no AACU development time. The authors themselves note that 'no statistical test could be applied to validate Hypothesis 3.' The word 'significantly' should be removed or replaced with 'descriptively' or 'in this limited sample.' This affects the Abstract, Section 5, and Section 7.
minor comments (5)
- [Table 1] The column headers contain typos: 'Succesful' and 'Unsuccesful' should be 'Successful' and 'Unsuccessful.' The word 'challange' appears in several places (e.g., Section 3 headings).
- [Section 5, Table 2] The p-value for P3 is reported as '0.0000118'; for consistency and readability, it should be expressed as '<0.0001' or '1.2e-5.' Also, the degrees of freedom for the Welch t-tests are not reported, which would help readers assess the critical values.
- [Section 5, H1] The text says 'leading to the rejection of the null hypothesis' after finding RPA is faster. The null hypothesis should be stated explicitly (e.g., equal mean execution times) to avoid ambiguity about what is being rejected.
- [References and Related Work] The citation style is inconsistent: some references are cited as author-year (e.g., Chakraborti et al., 2022) while others are numeric [10,11]. Please use one consistent style. Also, the statement in Section 2 that 'the term agentic automation does not currently yield any relevant literature focused specifically on business process automation' is contradicted by later citations such as ProAgent [12].
- [Appendix 1] The appendix contains a typo: 'invoice management systém' should be 'invoice management system.' Minor language editing throughout would improve readability.
Circularity Check
No circularity: the study is a direct empirical measurement comparison with no fitted parameters, derived quantities, or self-citation chains.
full rationale
This paper is an empirical benchmark study comparing RPA (UiPath) with an LLM computer-use agent (AACU) on three standardized tasks. The central claims—RPA is faster and more reliable, AACU requires less development time—are supported by directly measured outcomes (execution time, success/failure counts, development time). There are no fitted parameters, no statistical models whose predictions are compared to the data used to fit them, and no quantities that are defined in terms of the outcomes they purport to explain. The only model involvement is using Claude to rewrite its own prompt for Process 3; this is an input configuration choice, not a fitted parameter, and it does not define or predetermine the measured success metric. The paper contains no load-bearing self-citations: the references are to external prior work, industry documentation, and public challenges; none of the paper's conclusions are justified by citing the authors' own prior results. The significant concern noted in the manuscript—that Process 3 was 'Simplified for AACU' and shortened to the first page—is an experimental validity and fairness issue regarding whether RPA and AACU ran identical tasks, not a circularity issue. It concerns confounded comparison, not reduction of the conclusion to its inputs by definition. Therefore the appropriate circularity score is 0.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption The three rpachallenge.com tasks are representative of typical enterprise RPA workflows.
- domain assumption The chosen versions (UiPath 2023.4.0, claude-sonnet-4-20250514) represent their technology classes.
- domain assumption The P3 task, shortened to the first invoice page, is comparable across technologies.
- domain assumption A single successful run defines development time and is a meaningful proxy for development effort.
Cite this review
Pith. "Pith review of Are LLM Agents the New RPA? A Comparative Study with RPA Across Enterprise Workflows." pith.science (2026). https://pith.science/paper/FSNDIJSQ
@misc{pith2026250904198,
author = {Pith},
title = {Pith review of: Are LLM Agents the New RPA? A Comparative Study with RPA Across Enterprise Workflows},
year = {2026},
howpublished = {\url{https://pith.science/paper/FSNDIJSQ}},
note = {Machine review of arXiv:2509.04198}
}
read the original abstract
The emergence of large language models (LLMs) has introduced a new paradigm in automation: LLM agents or Agentic Automation with Computer Use (AACU). Unlike traditional Robotic Process Automation (RPA), which relies on rule-based workflows and scripting, AACU enables intelligent agents to perform tasks through natural language instructions and autonomous interaction with user interfaces. This study investigates whether AACU can serve as a viable alternative to RPA in enterprise workflow automation. We conducted controlled experiments across three standard RPA challenges data entry, monitoring, and document extraction comparing RPA (via UiPath) and AACU (via Anthropic's Computer Use Agent) in terms of speed, reliability, and development effort. Results indicate that RPA outperforms AACU in execution speed and reliability, particularly in repetitive, stable environments. However, AACU significantly reduces development time and adapts more flexibly to dynamic interfaces. While current AACU implementations are not yet production-ready, their promise in rapid prototyping and lightweight automation is evident. Future research should explore multi-agent orchestration, hybrid RPA-AACU architectures, and more robust evaluation across industries and platforms.
Figures
Forward citations
Cited by 1 Pith paper
-
Plover: Steering GUI Agents through Plan-Centric Interaction
An expert repairing visible plans rescued 23 of 26 failed GUI automation runs, turning 17 into full and 6 into partial successes.
Reference graph
Works this paper leans on
-
[1]
Chakraborti, T., Rizk, Y., Isahagian, V., Aksar, B., Fuggitti, F.: From Natural Lan- guage to Workflows: Towards Emergent Intelligence in Robotic Process Automa- tion. In: Marrella, A., Matulevicius, R., Gabryelczyk, R., Axmann, B., Vuksic, V.B., Gaaloul, W., Stemberger, M.I., Ko, A., and Lu, Q. (eds.) Business Process Management: Blockchain, Robotic Proc...
doi:10.1007/9 2022
-
[2]
Jansen, J.A., Manukyan, A., Al Khoury, N., Akalin, A.: Leveraging large language models for data analysis automation. PLoS ONE. 20, e0317084 (2025). https://doi.org/10.1371/journal.pone.0317084
-
[3]
Towards Intelligent Robotic Process Automation for BPMers
Agostinelli, S., Marrella, A., Mecella, M.: Towards Intelligent Robotic Process Automation for BPMers, https://arxiv.org/abs/2001.00804, (2020). https://doi.org/10.48550/ARXIV.2001.00804
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2001.00804 2001
-
[4]
Enterprise Information Systems
Wewerka, J., Reichert, M.: Robotic process automation - a systematic mapping study and classification framework. Enterprise Information Systems. 17, 1986862 (2023). https://doi.org/10.1080/17517575.2021.1986862
arXiv 2023
-
[5]
Enriquez, J.G., Jimenez-Ramirez, A., Dominguez-Mayo, F.J., Garcia-Garcia, J.A.: Robotic Process Automation: A Scientific and Industrial Systematic Mapping Study. IEEE Access. 8, 39113 –39129 (2020). https://doi.org/10.1109/ACCESS.2020.2974934
-
[6]
Syed, R., Suriadi, S., Adams, M., Bandara, W., Leemans, S.J.J., Ouyang, C., ter Hofstede, A.H.M., van de Weerd, I., Wynn, M.T., Reijers, H.A.: Robotic Process 14 P. Průcha. M. Matoušková and J. Strnad Automation: Contemporary themes and challenges. Computers in Industry. 115, 103162 (2020). https://doi.org/10.1016/j.compind.2019.103162
-
[7]
Journal of Computer Information Systems
Hughes, L., Dwivedi, Y.K., Malik, T., Shawosh, M., Albashrawi, M.A., Jeon, I., Dutot, V., Appanderanda, M., Crick, T., De’, R., Fenwick, M., Gunaratnege, S.M., Jurcys, P., Kar, A.K., Kshetri, N., Li, K., Mutasa, S., Samothrakis, S., Wade, M., Walton, P.: AI Agents and Agentic Systems: A Multi-Expert Analysis. Journal of Computer Information Systems. 1 –29...
arXiv 2025
-
[8]
Arthroscopy: The Journal of Arthroscopic & Related Surgery
Woo, J.J., Yang, A.J., Olsen, R.J., Hasan, S.S., Nawabi, D.H., Nwachukwu, B.U., Williams, R.J., Ramkumar, P.N.: Custom Large Language Models Improve Ac- curacy: Comparing Retrieval Augmented Generation and Artificial Intelligence Agents to Noncustom Model s for Evidence -Based Medicine. Arthroscopy: The Journal of Arthroscopic & Related Surgery. 41, 565 -...
-
[9]
Martins, A., Londral, A., L. Nunes, I., V. Lapão, L.: Unlocking human -like con- versations: Scoping review of automation techniques for personalized healthcare interventions using conversational agents. International Journal of Medical Infor- matics. 185, 105385 (2024). https://doi.org/10.1016/j.ijmedinf.2024.105385
arXiv 2024
-
[10]
Gaurav Samdani, Kabita Paul, Flavia Saldanha: Agentic AI in the Age of Hyper - Automation. World J. Adv. Eng. Technol. Sci. 8, 416 –427 (2023). https://doi.org/10.30574/wjaets.2023.8.1.0042
-
[11]
Sapkota, R., Roumeliotis, K.I., Karkee, M.: AI Agents vs. Agentic AI: A Concep- tual Taxonomy, Applications and Challenges, https://arxiv.org/abs/2505.10468, (2025). https://doi.org/10.48550/ARXIV.2505.10468
-
[12]
https://doi.org/10.48550/ARXIV.2311.10751
Ye, Y., Cong, X., Tian, S., Cao, J., Wang, H., Qin, Y., Lu, Y., Yu, H., Wang, H., Lin, Y., Liu, Z., Sun, M.: ProAgent: From Robotic Process Automation to Agentic Process Automation, https://arxiv.org/abs/2311.10751, (2023). https://doi.org/10.48550/ARXIV.2311.10751
-
[13]
Lamanna, C.: Announcing new computer use in Microsoft Copilot Studio for UI automation, https://www.microsoft.com/en -us/microsoft-copilot/blog/copilot-stu- dio/announcing-computer-use-microsoft-copilot-studio-ui-automation/
-
[14]
Xiong: Invoice Extraction with OCR, https://github.com/MaxineXiong/Invoice - Extraction-OCR-Challenge-RPA
-
[15]
RPA Challenge Invoice Extraction in UiPath | Automation OCR Challenge | Ui- Path RPA. (2020)
work page 2020
-
[16]
London School of Economics and Political Science, LSE Library (2015)
Willcocks, L.P., Lacity, M., Craig, A.: The IT function and robotic process auto- mation. London School of Economics and Political Science, LSE Library (2015)
work page 2015
-
[17]
Syed, R., Wynn, M.T.: Robotic process automation: a review of the state -of-the- art. In: Grefen, P. and Vanderfeesten, I. (eds.) Handbook on Business Process Ma- nagement and Digital Transformation. pp. 333 –362. Edward Elgar Publishing (2024). https://doi.org/10.4337/9781802206098.00024
-
[18]
In: 2024 47th MIPRO ICT and Electronics Conven- tion (MIPRO)
Perković, G., Drobnjak, A., Botički, I.: Hallucinations in LLMs: Understanding and Addressing Challenges. In: 2024 47th MIPRO ICT and Electronics Conven- tion (MIPRO). pp. 2084 –2088. IEEE, Opatija, Croatia (2024). https://doi.org/10.1109/MIPRO60963.2024.10569238. Are LLM Agents the New RPA? 15
arXiv 2024
-
[19]
In: 2025 IEEE Con- ference on Secure and Trustworthy Machine Learning (SaTML)
Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G.J., Wong, E.: Jailbrea- king Black Box Large Language Models in Twenty Queries. In: 2025 IEEE Con- ference on Secure and Trustworthy Machine Learning (SaTML). pp. 23–42. IEEE, Copenhagen, Denmark (2 025). https://doi.org/10.1109/Sa- TML64287.2025.00010
-
[20]
Martino, A., Iannelli, M., Truong, C.: Knowledge Injection to Counter Large Lan- guage Model (LLM) Hallucination. In: Pesquita, C., Skaf -Molli, H., Efthymiou, V., Kirrane, S., Ngonga, A., Collarana, D., Cerqueira, R., Alam, M., Trojahn, C., and Hertling, S. (eds.) The Semantic Web: ESWC 2023 Satellite Events. pp. 182–
work page 2023
-
[21]
https://doi.org/10.48550/ARXIV.2402.02047
Spiess, C., Gros, D., Pai, K.S., Pradel, M., Rabin, M.R.I., Alipour, A., Jha, S., Devanbu, P., Ahmed, T.: Calibration and Correctness of Language Models for Code, https://arxiv.org/abs/2402.02047, (2024). https://doi.org/10.48550/ARXIV.2402.02047
-
[22]
https://doi.org/10.48550/ARXIV.2308.05391
Schwartz, S., Yaeli, A., Shlomov, S.: Enhancing Trust in LLM-Based AI Automa- tion Agents: New Considerations and Future Challenges, https://ar- xiv.org/abs/2308.05391, (2023). https://doi.org/10.48550/ARXIV.2308.05391
-
[23]
van der Aalst, W.M.P., Bichler, M., Heinzl, A.: Robotic Process Automation. Bus Inf Syst Eng. 60, 269–272 (2018). https://doi.org/10.1007/s12599-018-0542-4
-
[24]
eds: Robotic process automation: management, techno- logy, applications
Czarnecki, C., Fettke, P. eds: Robotic process automation: management, techno- logy, applications. De Gruyter Oldenbourg, Berlin ; Boston (2021). Appendix 1 INVOICE DATA EXTRACTION TASK ENVIRONMENT: - Web browser (Firefox) with invoice management systém - Text editor (gedit) with example.csv file open OBJECTIVE: Extract and compile invoice information int...
work page 2021
-
[26]
Click invoice download button for each ID
-
[27]
Open the downloaded invoice document
-
[28]
Extract the following fields: • Invoice Number • Invoice Date (DD-MM-YYYY format) • Company Name 16 P. Průcha. M. Matoušková and J. Strnad • Total Amount Due OUTPUT FORMAT (CSV): ID,DueDate,InvoiceNo,InvoiceDate,CompanyName,TotalDue SAMPLE DATA: 5jef1y8yx4t8yupbpo3fzg,25 -02-2019,10021,13-02-2019,Sit Amet Corp.,1234.40 g11rzk7loegnitbclwnh8k,06 -04-2019,2...
work page 2019
-
[185]
https://doi.org/10.1007/978 -3- 031-43458-7_34
Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978 -3- 031-43458-7_34
doi:10.1007/978 2023
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.