REVIEW 4 major objections 5 minor 27 references
From Chaos to Automation: Enabling the Use of Unstructured Data for Robotic Process Automation
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read UNDRESS combines fuzzy regex, named-entity recognition, and an LLM so that RPA can extract fields from unstructured invoices and resumes, with perfect accuracy on pattern-based fields in the reported tests.
desk verdict A sensible hybrid RPA pipeline undermined by an evaluation that doesn't compare the whole system to its parts—on a key field the hybrid actually loses to the LLM alone. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
UNDRESS (UNstructured Document REtrieval SyStem) is the central object: a two-module pipeline whose text-extraction module converts scans and PDFs into clean text (preprocessing, OCR, spell correction, then LLM-based text cleanup), and whose information-retrieval module turns a user query into a field value by escalating through fuzzy regular expressions, named-entity recognition, and finally an LLM prompt. The escalation order is the load-bearing mechanism. It is designed so that each stage covers the cases the previous stage cannot handle, which is what lets the whole system reach high accuracy across document types while keeping most extractions on simpler, controllable components.
What would settle it
Run the same hybrid cascade and single-technique baselines on a larger multi-company corpus of real invoices and resumes with independently verified ground truth; the central claim fails if the hybrid does not beat the best single technique on most fields, or if its accuracy collapses on layouts outside the original datasets.
Extended reading notes
Core claim
The discovery is that the ordering of techniques is itself a design choice that pays off. Instead of handing every field to an LLM, the system tries the simplest interpretable matcher first: fuzzy regular expressions for well-patterned fields such as e-mail, phone numbers, addresses, dates, and IBANs; then named-entity recognition for persons, organizations, and languages; and only then an LLM extraction prompt for semantic fields such as education, skills, job title, invoice number, and seller. In the reported results, this escalation beats single-technique baselines: on the ambiguous 'Language' field, the LLM alone confuses the language of the document with the languages a candidate speaks, while NER gets it right, and on invoice dates fuzzy regex achieves perfect precision where the LLM does not. The paper reads these results as evidence that a solution based solely on LLMs is not necessary for high-performing document automation, and that a balanced hybrid strategy can match it while being simpler, more interpretable, and less dependent on a model's whims.
Load-bearing premise
The results rest on the assumption that 400 synthetic resumes and 100 invoices from one Dutch lab-services company represent the range of unstructured documents RPA will meet, and that the manually corrected OCR output is error-free ground truth.
Editorial extensions
If this is right
- RPA bots can be aimed at scanned invoices, resumes, and other free-form PDFs, with no requirement that users first convert the documents into structured tables.
- Fields with fixed patterns (e-mail, address, phone number, IBAN) are extracted at or near perfect accuracy by simple, interpretable rules, so the LLM is not needed for the bulk of typical documents.
- On ambiguous semantic fields the technique order matters: fuzzy regex and NER can outperform the LLM, so a hybrid route can beat any single technique across a mixed corpus.
- Document-heavy processes such as invoice processing, financial reporting, and HR resume screening become viable RPA targets, expanding automation beyond structured-input workflows.
- Because the LLM is used only as a fallback, the approach lowers cost, energy use, and hallucination risk relative to an LLM-only extractor.
Reading between the lines
- Beyond the paper's claims, the same routing principle should transfer to other document classes: extract anything matchable by a pattern or named entity before invoking a generative model, so the cost per successful extraction scales with field difficulty rather than document volume.
- A natural next experiment would instrument the pipeline to log which component produced each correct value; that would let practitioners compute the actual cost and latency contributed by the LLM fallback and tune the cascade per document type.
- The 'Language' failure suggests a testable fix the paper does not try: adding field-specific context or schema hints to the LLM prompt should resolve the ambiguity between document language and candidate languages.
- Because the invoice ground truth is itself OCR-derived and manually corrected, a stronger evaluation would compare the full pipeline to independent human transcription of the same scans, separating OCR errors from retrieval errors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents UNDRESS, a pipeline that combines OCR-based text extraction with a sequential information-retrieval cascade (fuzzy regular expressions, then named entity recognition, then an LLM) to let RPA platforms retrieve fields from unstructured documents. The authors evaluate text-extraction quality with Jaccard similarity on 400 generated resumes and 100 real invoices, measure per-field retrieval accuracy, precision, and recall on 40 documents per dataset, and report a qualitative study with five RPA developers. The central claim is that this hybrid cascade enables RPA to work effectively with unstructured data and outperforms single-technique approaches.
Significance. The practical motivation is strong: RPA platforms generally require structured input, and a modular, extensible extraction system could broaden their scope. The paper's strengths include a clear system architecture, a public code repository, and component-wise evaluation against external ground truth. If the effectiveness claim were properly supported, this would be a useful engineering contribution to the RPA and document-processing communities. However, the current evidence does not establish the central claim: there is no end-to-end comparison of complete pipelines, the fixed cascade is worse than its own LLM component on a key invoice field, and prompt selection was performed on the test set. These issues are load-bearing and require additional analysis or a substantial tempering of the claims.
major comments (4)
- [Section 4.4, Tables 2–3, Figures 3–4] The conclusion that the hybrid approach outperforms methods focused on a single technique (Section 6) is not supported because the evaluation never compares complete pipelines. Tables 2 and 3 report per-field accuracy for whichever component fired, and Figures 3 and 4 compare individual components only on individual fields. No aggregate accuracy, precision, or recall is reported for UNDRESS as a whole versus an LLM-only, fuzzy-regex-only, or NER-only pipeline. The 'Technique' column therefore reflects the routing of the cascade, not a controlled comparison. Please add end-to-end results over the full test sets, ideally with confidence intervals.
- [Section 4.4, Table 3, Figure 4, and Section 3] The fixed execution order described in Section 3 (fuzzy regex first, then NER, then LLM) can make the hybrid worse than its own LLM component. For the invoice 'Total amount' field, Table 3 reports full-pipeline accuracy of 0.65 via fuzzy regex, while Figure 4 reports LLM-only accuracy of 0.88 on the same field. Because the LLM is consulted only when fuzzy regex and NER find nothing, a wrong fuzzy match is never corrected by the LLM. Section 5 acknowledges problems with duplicate 'Total amount' labels but does not reconcile this with the reliability claim; this is a load-bearing issue for the contribution.
- [Section 4.3, Table 1] The text-extraction prompt was selected on the evaluation set: the authors tested three prompts on the test documents and kept the one with the highest average Jaccard index before adding tokenization and lemmatization. This is test-set optimization, so the reported indices (0.99 for resumes, 0.81 for invoices) and the downstream retrieval numbers are optimistic estimates. Please use a held-out validation set for prompt selection or explicitly report the selection procedure as a limitation.
- [Section 4.1 and Section 4.4] The evaluation corpus is narrow: 100 invoices from a single Dutch company and 400 synthetically generated resumes, with retrieval measured on only 40 documents per field. No confidence intervals or significance tests are reported for accuracies such as 0.65 versus 0.88. The abstract and conclusion claim that UNDRESS is effective for unstructured documents generally, which goes beyond what this sample can support. Please either broaden the evaluation or restrict the claims to the studied document types and report statistical uncertainty.
minor comments (5)
- [Section 4.3] The text contains a typo: 'avarage' should be 'average', and 'pdfminder' appears to be a typo for 'pdfminer'.
- [Section 4.4 and References] The author name is spelled inconsistently as 'Keturis et al.' in the text and 'Kerutis et al.' in the reference list; please standardize the spelling.
- [Figure 4] The value '3·10−2' in the figure is difficult to read and the bar legend is not self-explanatory; please use decimal labels and a clear legend for the three methods.
- [Section 5] The statement 'we are the first to use an LLM integrated hybrid approach in RPA context' is a strong novelty claim that is not supported by the related-work review; please soften it or provide a systematic search as evidence.
- [Tables 2–3] Reporting the number of documents in which each component actually fired would help the reader interpret the per-field accuracies, since the current tables only show the final routing and outcome.
Circularity Check
No circularity: UNDRESS is an empirical pipeline evaluation; effectiveness claims are checked against independent ground-truth data, and no fitted parameter or load-bearing self-citation makes the conclusion equivalent to its inputs.
full rationale
The paper contains no mathematical derivation in which an output quantity is defined in terms of the quantity being predicted. Each component of UNDRESS (fuzzy regex, spaCy NER, GPT-3.5) is an off-the-shelf tool, and the system is evaluated by comparing extracted values against external ground truth: generated-resume text from the PDF generator itself, and invoice text manually verified from Adobe Acrobat Pro output. No parameter is fitted to a subset of the data and then reported as a prediction of a closely related quantity; prompt selection on the test set is a methodological optimism concern, not circularity. The citations to prior work by co-authors (e.g., Gao et al. [8], Syed et al. [22]) provide background and context, but they are not invoked as unverified premises that force the paper's central claim. The claim that the hybrid approach outperforms single-technique methods may be under-supported by the reported per-field results, but under-support is a soundness issue, not a circularity issue. Therefore no circular step can be quoted or reduced to the paper's own inputs.
Assumptions & free parameters
free parameters (2)
- LLM prompt template and postprocessing =
Prompt 2 ('Correct spelling mistakes...') plus tokenization/lemmatization
- Temperature of GPT-3.5-turbo =
0.1
assumptions (6)
- standard math Precision/recall and Jaccard similarity are appropriate evaluation metrics for the claims.
- domain assumption The 100 LabSC invoices and 400 generated resumes represent unstructured documents in RPA practice.
- domain assumption Adobe Acrobat Pro OCR with manual correction yields correct ground truth for the invoice dataset.
- domain assumption Tesseract OCR, after the preprocessing steps, produces text accurate enough for the downstream components.
- domain assumption GPT-3.5-turbo corrects and formats text as prompted, and later extracts fields correctly.
- domain assumption The nine hand-written fuzzy regex patterns cover the static fields in both datasets.
Cite this review
Pith. "Pith review of From Chaos to Automation: Enabling the Use of Unstructured Data for Robotic Process Automation." pith.science (2026). https://pith.science/paper/VOS36CZ6
@misc{pith2026250711364,
author = {Pith},
title = {Pith review of: From Chaos to Automation: Enabling the Use of Unstructured Data for Robotic Process Automation},
year = {2026},
howpublished = {\url{https://pith.science/paper/VOS36CZ6}},
note = {Machine review of arXiv:2507.11364}
}
read the original abstract
The growing volume of unstructured data within organizations poses significant challenges for data analysis and process automation. Unstructured data, which lacks a predefined format, encompasses various forms such as emails, reports, and scans. It is estimated to constitute approximately 80% of enterprise data. Despite the valuable insights it can offer, extracting meaningful information from unstructured data is more complex compared to structured data. Robotic Process Automation (RPA) has gained popularity for automating repetitive tasks, improving efficiency, and reducing errors. However, RPA is traditionally reliant on structured data, limiting its application to processes involving unstructured documents. This study addresses this limitation by developing the UNstructured Document REtrieval SyStem (UNDRESS), a system that uses fuzzy regular expressions, techniques for natural language processing, and large language models to enable RPA platforms to effectively retrieve information from unstructured documents. The research involved the design and development of a prototype system, and its subsequent evaluation based on text extraction and information retrieval performance. The results demonstrate the effectiveness of UNDRESS in enhancing RPA capabilities for unstructured data, providing a significant advancement in the field. The findings suggest that this system could facilitate broader RPA adoption across processes traditionally hindered by unstructured data, thereby improving overall business process efficiency.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
IEEE Access 10, 78398–78413 (2022)
Arslan,H.:Endtoendinvoiceprocessingapplicationbasedonkeyfieldsextraction. IEEE Access 10, 78398–78413 (2022)
work page 2022
-
[2]
In: 2021 4th International Conference on Information and Computer Technologies (ICICT)
Baidya, A.: Document analysis and classification: A Robotic Process Automation (RPA) and machine learning approach. In: 2021 4th International Conference on Information and Computer Technologies (ICICT). pp. 33–37 (2021)
work page 2021
-
[3]
Expert Systems with Applications210, 118487 (2022)
Barducci, A., Iannaccone, S., La Gatta, V., Moscato, V., Sperlì, G., Zavota, S.: An end-to-end framework for information extraction from Italian resumes. Expert Systems with Applications210, 118487 (2022)
work page 2022
-
[4]
Baviskar, D., Ahirrao, S., Potdar, V., Kotecha, K.: Efficient automated processing of the unstructured documents using artificial intelligence: A systematic literature review and future directions. IEEE Access9(art. no. 9402739), 72894–72936 (2021)
work page 2021
-
[5]
Engineering Management in Production and Services 14, 1 – 12 (2022)
Costa, D.S., Mamede, H.S., da Silva, M.M.: Robotic Process Automation (RPA) adoption: A systematic literature review. Engineering Management in Production and Services 14, 1 – 12 (2022)
work page 2022
-
[6]
In: Research Anthol- ogy on Cross-Disciplinary Designs and Applications of Automation, pp
Doguc, O.: Robot Process Automation (RPA) and its future. In: Research Anthol- ogy on Cross-Disciplinary Designs and Applications of Automation, pp. 35–58. IGI Global (2022)
work page 2022
-
[7]
IEEE Access 8, 39113–39129 (2020)
Enríquez, J.G., Jiménez-Ramírez, A., Domínguez-Mayo, F.J., García-García, J.A.: Robotic Process Automation: A scientific and industrial systematic mapping study. IEEE Access 8, 39113–39129 (2020)
work page 2020
-
[8]
Gao, J., van Zelst, S.J., Lu, X., van der Aalst, W.M.P.: Automated Robotic Process Automation: A self-learning approach. In: OTM Conferences. Lecture Notes in Computer Science, vol. 11877, pp. 95–112. Springer (2019)
work page 2019
Show all 27 references
-
[9]
Gruzauskas, V., Ragavan, D.: Robotic Process Automation for document process- ing:Acasestudyofalogisticsserviceprovider.JournalofManagement 36,119–126 (2020)
2020
-
[10]
unstructured data
IBM: Structured vs. unstructured data. https://www.ibm.com/blog/ structured-vs-unstructured-data/ (2021), accessed on: December 5, 2024
2021
-
[11]
In: Large Language Models: A Deep Dive: Bridging Theory and Practice, pp
Kamath, U., Keenan, K., Somers, G., Sorenson, S.: LLM challenges and solutions. In: Large Language Models: A Deep Dive: Bridging Theory and Practice, pp. 219–
-
[12]
In: International Conference on Information and Software Tech- nologies
Kerutis, V., Calneryte, D.: Intelligent invoice documents processing employing RPA technologies. In: International Conference on Information and Software Tech- nologies. pp. 235–247. Springer International Publishing (October 2022)
2022
-
[13]
In: 2020 Chinese Automation Congress (CAC)
Ling, X., Gao, M., Wang, D.: Intelligent document processing based on RPA and machine learning. In: 2020 Chinese Automation Congress (CAC). pp. 1349–1353 (2020)
2020
-
[14]
In: Lecture Notes in Networks and Systems, vol
Major, A., Ngalamou, L.: Improving accuracy and efficiency in document handling for business processes. In: Lecture Notes in Networks and Systems, vol. 815 LNNS, pp. 543–562 (2023)
2023
-
[15]
Martínez-Rojas, A., Rodríguez-Ruíz, A., Enríquez, J.G., Ramirez, A.J.: What’s behind the screen? Unveiling UI hierarchies in process-related UI logs. In: BPM. Lecture Notes in Computer Science, vol. 14940, pp. 256–272. Springer (2024)
2024
-
[16]
OpenAI: Best practices for prompt engineering with the OpenAI API (2024), https://help.openai.com/en/articles/ 6654000-best-practices-for-prompt-engineering-with-the-openai-api , accessed: 2024-11-29
2024
-
[17]
In: Jeena Jacob, I., Gonzalez-Longatt, F., Kolandapalayam Shanmugam, S., Izonin, I
Pudasaini, S., Shakya, S., Lamichhane, S., Adhikari, S., Tamang, A., Adhikari, S.: Application of NLP for information extraction from unstructured documents. In: Jeena Jacob, I., Gonzalez-Longatt, F., Kolandapalayam Shanmugam, S., Izonin, I. (eds.) Expert Clouds and Applicatio...
2022
-
[18]
In: New Trends in Business Information Systems and Technology, Studies in Systems, Decision and Control, vol
Pustulka, E., Hanne, T.: Text mining innovation for business. In: New Trends in Business Information Systems and Technology, Studies in Systems, Decision and Control, vol. 294. Springer, Cham (2021)
2021
-
[19]
In: 2022 IEEE 20th Student Conference on Research and Development (SCOReD)
Rohaime, N., Abdul Razak, N., Thamrin, N., Shyan, C.: Integrated invoicing solu- tion: A Robotic Process Automation with AI and OCR approach. In: 2022 IEEE 20th Student Conference on Research and Development (SCOReD). pp. 30–33 (2022)
2022
-
[20]
In: 2021 6th International Conference on Recent Trends on Electronics, Informa- tion, Communication and Technology (RTEICT)
Roopesh, N., Babu, C.: Robotic Process Automation for resume processing system. In: 2021 6th International Conference on Recent Trends on Electronics, Informa- tion, Communication and Technology (RTEICT). pp. 180–184 (2021)
2021
-
[21]
Journal of Integrated Science and Technology11(4), 561–561 (2023)
Sreekrishna, M., Jacob, T.: Systematic mapping in improving the extraction of can- cer pathology information using RPA orchestration. Journal of Integrated Science and Technology11(4), 561–561 (2023)
2023
-
[22]
Computers in Industry115, 103162 (2020)
Syed, R., Suriadi, S., Adams, M., Bandara, W., Leemans, S.J.J., Ouyang, C., ter Hofstede, A.H.M., van de Weerd, I., Wynn, M.T., Reijers, H.A.: Robotic Process Automation: Contemporary themes and challenges. Computers in Industry115, 103162 (2020)
2020
-
[23]
arXiv preprint arXiv:2403.03344 (2024)
Vartziotis, T., Dellatolas, I., Dasoulas, G., Schmidt, M., Schneider, F., Hoffmann, T., Keckeisen, M.: Learn to code sustainably: An empirical study on LLM-based green code generation. arXiv preprint arXiv:2403.03344 (2024)
2024 arXiv
-
[24]
In: 2020 IEEE 24th International Enterprise Distributed Object Computing Conference (EDOC)
Wewerka, J., Dax, S., Reichert, M.: A user acceptance model for Robotic Pro- cess Automation. In: 2020 IEEE 24th International Enterprise Distributed Object Computing Conference (EDOC). pp. 97–106 (Oct 2020)
2020
-
[25]
Procedia Computer Science192, 2432–2439 (2021), knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021
Wosiak, A.: Automated extraction of information from Polish resume documents in the IT recruitment process. Procedia Computer Science192, 2432–2439 (2021), knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021
2021
-
[26]
ICIC Express Letters 14(6), 593–603 (2020)
Zaman, G., Mahdin, H., Hussain, K., Rahman, A.: Information extraction from semi and unstructured data sources: A systematic literature review. ICIC Express Letters 14(6), 593–603 (2020)
2020
-
[274]
Springer Nature Switzerland (2024) Enabling the Use of Unstructured Data for RPA 15
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.