Pith. sign in

REVIEW 4 major objections 5 minor 27 references

From Chaos to Automation: Enabling the Use of Unstructured Data for Robotic Process Automation

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read UNDRESS combines fuzzy regex, named-entity recognition, and an LLM so that RPA can extract fields from unstructured invoices and resumes, with perfect accuracy on pattern-based fields in the reported tests.

desk verdict A sensible hybrid RPA pipeline undermined by an evaluation that doesn't compare the whole system to its parts—on a key field the hybrid actually loses to the LLM alone. read the letter →

arxiv 2507.11364 v1 pith:VOS36CZ6 submitted 2025-07-15 cs.IR cs.SE

classification cs.IRcs.SE
keywords RoboticProcessAutomationunstructureddatainformationextractionlargelanguagemodelsnamedentityrecognitionfuzzyregularexpressionsdocumentprocessingtext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UNDRESS is a proposed pipeline that gives Robotic Process Automation systems access to unstructured documents, the roughly 80 percent of enterprise data that does not arrive in neat tables or fixed forms. The paper's central claim is that a hybrid route — fuzzy regular expressions first, named-entity recognition second, and a large-language-model prompt only as a fallback — extracts business fields from scanned invoices and free-form resumes more reliably than any single technique used alone. In the reported evaluation, the system achieves perfect accuracy on address, e-mail, phone, and language fields, near-perfect accuracy on job title and education, and strong accuracy on invoice number and seller, with text-extraction similarity of 0.99 for resumes and 0.81 for invoices. The point of showing this is practical: if the hybrid pipeline holds up, RPA can automate document-heavy processes that today require structured input, and it can do so while reserving expensive, error-prone LLM calls for the few fields that actually need semantic understanding.

What carries the argument

UNDRESS (UNstructured Document REtrieval SyStem) is the central object: a two-module pipeline whose text-extraction module converts scans and PDFs into clean text (preprocessing, OCR, spell correction, then LLM-based text cleanup), and whose information-retrieval module turns a user query into a field value by escalating through fuzzy regular expressions, named-entity recognition, and finally an LLM prompt. The escalation order is the load-bearing mechanism. It is designed so that each stage covers the cases the previous stage cannot handle, which is what lets the whole system reach high accuracy across document types while keeping most extractions on simpler, controllable components.

What would settle it

Run the same hybrid cascade and single-technique baselines on a larger multi-company corpus of real invoices and resumes with independently verified ground truth; the central claim fails if the hybrid does not beat the best single technique on most fields, or if its accuracy collapses on layouts outside the original datasets.

Watch

Extended reading notes

Core claim

The discovery is that the ordering of techniques is itself a design choice that pays off. Instead of handing every field to an LLM, the system tries the simplest interpretable matcher first: fuzzy regular expressions for well-patterned fields such as e-mail, phone numbers, addresses, dates, and IBANs; then named-entity recognition for persons, organizations, and languages; and only then an LLM extraction prompt for semantic fields such as education, skills, job title, invoice number, and seller. In the reported results, this escalation beats single-technique baselines: on the ambiguous 'Language' field, the LLM alone confuses the language of the document with the languages a candidate speaks, while NER gets it right, and on invoice dates fuzzy regex achieves perfect precision where the LLM does not. The paper reads these results as evidence that a solution based solely on LLMs is not necessary for high-performing document automation, and that a balanced hybrid strategy can match it while being simpler, more interpretable, and less dependent on a model's whims.

Load-bearing premise

The results rest on the assumption that 400 synthetic resumes and 100 invoices from one Dutch lab-services company represent the range of unstructured documents RPA will meet, and that the manually corrected OCR output is error-free ground truth.

Editorial extensions

If this is right

  • RPA bots can be aimed at scanned invoices, resumes, and other free-form PDFs, with no requirement that users first convert the documents into structured tables.
  • Fields with fixed patterns (e-mail, address, phone number, IBAN) are extracted at or near perfect accuracy by simple, interpretable rules, so the LLM is not needed for the bulk of typical documents.
  • On ambiguous semantic fields the technique order matters: fuzzy regex and NER can outperform the LLM, so a hybrid route can beat any single technique across a mixed corpus.
  • Document-heavy processes such as invoice processing, financial reporting, and HR resume screening become viable RPA targets, expanding automation beyond structured-input workflows.
  • Because the LLM is used only as a fallback, the approach lowers cost, energy use, and hallucination risk relative to an LLM-only extractor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the same routing principle should transfer to other document classes: extract anything matchable by a pattern or named entity before invoking a generative model, so the cost per successful extraction scales with field difficulty rather than document volume.
  • A natural next experiment would instrument the pipeline to log which component produced each correct value; that would let practitioners compute the actual cost and latency contributed by the LLM fallback and tune the cascade per document type.
  • The 'Language' failure suggests a testable fix the paper does not try: adding field-specific context or schema hints to the LLM prompt should resolve the ambiguity between document language and candidate languages.
  • Because the invoice ground truth is itself OCR-derived and manually corrected, a stronger evaluation would compare the full pipeline to independent human transcription of the same scans, separating OCR errors from retrieval errors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents UNDRESS, a pipeline that combines OCR-based text extraction with a sequential information-retrieval cascade (fuzzy regular expressions, then named entity recognition, then an LLM) to let RPA platforms retrieve fields from unstructured documents. The authors evaluate text-extraction quality with Jaccard similarity on 400 generated resumes and 100 real invoices, measure per-field retrieval accuracy, precision, and recall on 40 documents per dataset, and report a qualitative study with five RPA developers. The central claim is that this hybrid cascade enables RPA to work effectively with unstructured data and outperforms single-technique approaches.

Significance. The practical motivation is strong: RPA platforms generally require structured input, and a modular, extensible extraction system could broaden their scope. The paper's strengths include a clear system architecture, a public code repository, and component-wise evaluation against external ground truth. If the effectiveness claim were properly supported, this would be a useful engineering contribution to the RPA and document-processing communities. However, the current evidence does not establish the central claim: there is no end-to-end comparison of complete pipelines, the fixed cascade is worse than its own LLM component on a key invoice field, and prompt selection was performed on the test set. These issues are load-bearing and require additional analysis or a substantial tempering of the claims.

major comments (4)
  1. [Section 4.4, Tables 2–3, Figures 3–4] The conclusion that the hybrid approach outperforms methods focused on a single technique (Section 6) is not supported because the evaluation never compares complete pipelines. Tables 2 and 3 report per-field accuracy for whichever component fired, and Figures 3 and 4 compare individual components only on individual fields. No aggregate accuracy, precision, or recall is reported for UNDRESS as a whole versus an LLM-only, fuzzy-regex-only, or NER-only pipeline. The 'Technique' column therefore reflects the routing of the cascade, not a controlled comparison. Please add end-to-end results over the full test sets, ideally with confidence intervals.
  2. [Section 4.4, Table 3, Figure 4, and Section 3] The fixed execution order described in Section 3 (fuzzy regex first, then NER, then LLM) can make the hybrid worse than its own LLM component. For the invoice 'Total amount' field, Table 3 reports full-pipeline accuracy of 0.65 via fuzzy regex, while Figure 4 reports LLM-only accuracy of 0.88 on the same field. Because the LLM is consulted only when fuzzy regex and NER find nothing, a wrong fuzzy match is never corrected by the LLM. Section 5 acknowledges problems with duplicate 'Total amount' labels but does not reconcile this with the reliability claim; this is a load-bearing issue for the contribution.
  3. [Section 4.3, Table 1] The text-extraction prompt was selected on the evaluation set: the authors tested three prompts on the test documents and kept the one with the highest average Jaccard index before adding tokenization and lemmatization. This is test-set optimization, so the reported indices (0.99 for resumes, 0.81 for invoices) and the downstream retrieval numbers are optimistic estimates. Please use a held-out validation set for prompt selection or explicitly report the selection procedure as a limitation.
  4. [Section 4.1 and Section 4.4] The evaluation corpus is narrow: 100 invoices from a single Dutch company and 400 synthetically generated resumes, with retrieval measured on only 40 documents per field. No confidence intervals or significance tests are reported for accuracies such as 0.65 versus 0.88. The abstract and conclusion claim that UNDRESS is effective for unstructured documents generally, which goes beyond what this sample can support. Please either broaden the evaluation or restrict the claims to the studied document types and report statistical uncertainty.
minor comments (5)
  1. [Section 4.3] The text contains a typo: 'avarage' should be 'average', and 'pdfminder' appears to be a typo for 'pdfminer'.
  2. [Section 4.4 and References] The author name is spelled inconsistently as 'Keturis et al.' in the text and 'Kerutis et al.' in the reference list; please standardize the spelling.
  3. [Figure 4] The value '3·10−2' in the figure is difficult to read and the bar legend is not self-explanatory; please use decimal labels and a clear legend for the three methods.
  4. [Section 5] The statement 'we are the first to use an LLM integrated hybrid approach in RPA context' is a strong novelty claim that is not supported by the related-work review; please soften it or provide a systematic search as evidence.
  5. [Tables 2–3] Reporting the number of documents in which each component actually fired would help the reader interpret the per-field accuracies, since the current tables only show the final routing and outcome.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: UNDRESS is an empirical pipeline evaluation; effectiveness claims are checked against independent ground-truth data, and no fitted parameter or load-bearing self-citation makes the conclusion equivalent to its inputs.

full rationale

The paper contains no mathematical derivation in which an output quantity is defined in terms of the quantity being predicted. Each component of UNDRESS (fuzzy regex, spaCy NER, GPT-3.5) is an off-the-shelf tool, and the system is evaluated by comparing extracted values against external ground truth: generated-resume text from the PDF generator itself, and invoice text manually verified from Adobe Acrobat Pro output. No parameter is fitted to a subset of the data and then reported as a prediction of a closely related quantity; prompt selection on the test set is a methodological optimism concern, not circularity. The citations to prior work by co-authors (e.g., Gao et al. [8], Syed et al. [22]) provide background and context, but they are not invoked as unverified premises that force the paper's central claim. The claim that the hybrid approach outperforms single-technique methods may be under-supported by the reported per-field results, but under-support is a soundness issue, not a circularity issue. Therefore no circular step can be quoted or reduced to the paper's own inputs.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central claim rests on evaluation assumptions rather than on mathematical axioms: synthetic-data representativeness, manual ground truth, OCR quality, and reliable LLM behavior. No new physical or conceptual entities are introduced; the system is a composition of existing components.

free parameters (2)
  • LLM prompt template and postprocessing = Prompt 2 ('Correct spelling mistakes...') plus tokenization/lemmatization
    Selected by comparing average Jaccard scores on the same test datasets (Section 4.3), making reported text-extraction scores in-sample rather than predictions.
  • Temperature of GPT-3.5-turbo = 0.1
    Chosen by hand for deterministic responses (Section 4.3); affects output variability.
assumptions (6)
  • standard math Precision/recall and Jaccard similarity are appropriate evaluation metrics for the claims.
    Used in Section 4.2 without alternative metrics; accuracy counting for multi-label fields is ambiguous.
  • domain assumption The 100 LabSC invoices and 400 generated resumes represent unstructured documents in RPA practice.
    Section 4.1; resumes are synthetic with randomized layouts, invoices come from one company, so generalizability is unverified.
  • domain assumption Adobe Acrobat Pro OCR with manual correction yields correct ground truth for the invoice dataset.
    Section 4.3; no inter-annotator check or independent verification is reported.
  • domain assumption Tesseract OCR, after the preprocessing steps, produces text accurate enough for the downstream components.
    Section 3 'OCR Engine'; no OCR error measurement on the actual datasets is reported.
  • domain assumption GPT-3.5-turbo corrects and formats text as prompted, and later extracts fields correctly.
    Sections 3 and 4.3; the authors note the LLM often formats despite instructions, so this assumption is only partially met.
  • domain assumption The nine hand-written fuzzy regex patterns cover the static fields in both datasets.
    Section 3 'Fuzzy Regular Expressions'; low accuracy on 'Total amount' (0.65) suggests some patterns are not robust.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Chaos to Automation: Enabling the Use of Unstructured Data for Robotic Process Automation." pith.science (2026). https://pith.science/paper/VOS36CZ6

@misc{pith2026250711364,
  author       = {Pith},
  title        = {Pith review of: From Chaos to Automation: Enabling the Use of Unstructured Data for Robotic Process Automation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VOS36CZ6}},
  note         = {Machine review of arXiv:2507.11364}
}
read the original abstract

The growing volume of unstructured data within organizations poses significant challenges for data analysis and process automation. Unstructured data, which lacks a predefined format, encompasses various forms such as emails, reports, and scans. It is estimated to constitute approximately 80% of enterprise data. Despite the valuable insights it can offer, extracting meaningful information from unstructured data is more complex compared to structured data. Robotic Process Automation (RPA) has gained popularity for automating repetitive tasks, improving efficiency, and reducing errors. However, RPA is traditionally reliant on structured data, limiting its application to processes involving unstructured documents. This study addresses this limitation by developing the UNstructured Document REtrieval SyStem (UNDRESS), a system that uses fuzzy regular expressions, techniques for natural language processing, and large language models to enable RPA platforms to effectively retrieve information from unstructured documents. The research involved the design and development of a prototype system, and its subsequent evaluation based on text extraction and information retrieval performance. The results demonstrate the effectiveness of UNDRESS in enhancing RPA capabilities for unstructured data, providing a significant advancement in the field. The findings suggest that this system could facilitate broader RPA adoption across processes traditionally hindered by unstructured data, thereby improving overall business process efficiency.

Figures

Figures reproduced from arXiv: 2507.11364 by the authors.

Figure 1
Figure 1. Business-relevant documents, such as invoices, are often captured in un￾structured or semi-structured formats (e.g., PDFs or images). Extracting key information like invoice number, total amount, and other essential data is cru￾cial for completing business processes. This figure illustrates this problem using two examples of invoices from LabSC with the relevant information extracted. According to IBM, unstructured … view at source ↗
Figure 2
Figure 2. System Design Spell Checker We use the Python spell checker library to correct common OCR errors, such as confusing “l” with “i”. Words starting with capital letters, often names, are excluded from correction to avoid misprocessing. The spell checker is too simple to handle these cases properly and returns “None”, which is undesirable since we want to retain those names. OpenAI - Text Extraction Finally, we use an O… view at source ↗
Figure 4
Figure 4. Accuracy results per extrac￾tion method for invoice fields 4.5 Results - Qualitative Evaluation The primary goal of the qualitative evaluation was to assess the value of UN￾DRESS for RPA developers in their work activities. To achieve this, we demon￾strated the system to five RPA developers: three from a large academic hospital in Utrecht, the Netherlands, and two from an RPA consultancy in Amsterdam, the Netherland… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Survey results for RPA De￾velopers by statement disagree, disagree, neutral, agree, strongly agree) on the following statements, which focused on ease of use, applicability, and novelty: 1. The graphical user interface of UNDRESS is intuitive and easy to navigate (Ease…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 26 canonical work pages

  1. [1]

    IEEE Access 10, 78398–78413 (2022)

    Arslan,H.:Endtoendinvoiceprocessingapplicationbasedonkeyfieldsextraction. IEEE Access 10, 78398–78413 (2022)

  2. [2]

    In: 2021 4th International Conference on Information and Computer Technologies (ICICT)

    Baidya, A.: Document analysis and classification: A Robotic Process Automation (RPA) and machine learning approach. In: 2021 4th International Conference on Information and Computer Technologies (ICICT). pp. 33–37 (2021)

  3. [3]

    Expert Systems with Applications210, 118487 (2022)

    Barducci, A., Iannaccone, S., La Gatta, V., Moscato, V., Sperlì, G., Zavota, S.: An end-to-end framework for information extraction from Italian resumes. Expert Systems with Applications210, 118487 (2022)

  4. [4]

    IEEE Access9(art

    Baviskar, D., Ahirrao, S., Potdar, V., Kotecha, K.: Efficient automated processing of the unstructured documents using artificial intelligence: A systematic literature review and future directions. IEEE Access9(art. no. 9402739), 72894–72936 (2021)

  5. [5]

    Engineering Management in Production and Services 14, 1 – 12 (2022)

    Costa, D.S., Mamede, H.S., da Silva, M.M.: Robotic Process Automation (RPA) adoption: A systematic literature review. Engineering Management in Production and Services 14, 1 – 12 (2022)

  6. [6]

    In: Research Anthol- ogy on Cross-Disciplinary Designs and Applications of Automation, pp

    Doguc, O.: Robot Process Automation (RPA) and its future. In: Research Anthol- ogy on Cross-Disciplinary Designs and Applications of Automation, pp. 35–58. IGI Global (2022)

  7. [7]

    IEEE Access 8, 39113–39129 (2020)

    Enríquez, J.G., Jiménez-Ramírez, A., Domínguez-Mayo, F.J., García-García, J.A.: Robotic Process Automation: A scientific and industrial systematic mapping study. IEEE Access 8, 39113–39129 (2020)

  8. [8]

    In: OTM Conferences

    Gao, J., van Zelst, S.J., Lu, X., van der Aalst, W.M.P.: Automated Robotic Process Automation: A self-learning approach. In: OTM Conferences. Lecture Notes in Computer Science, vol. 11877, pp. 95–112. Springer (2019)

Show all 27 references
  1. [9]

    Gruzauskas, V., Ragavan, D.: Robotic Process Automation for document process- ing:Acasestudyofalogisticsserviceprovider.JournalofManagement 36,119–126 (2020)

  2. [10]

    unstructured data

    IBM: Structured vs. unstructured data. https://www.ibm.com/blog/ structured-vs-unstructured-data/ (2021), accessed on: December 5, 2024

  3. [11]

    In: Large Language Models: A Deep Dive: Bridging Theory and Practice, pp

    Kamath, U., Keenan, K., Somers, G., Sorenson, S.: LLM challenges and solutions. In: Large Language Models: A Deep Dive: Bridging Theory and Practice, pp. 219–

  4. [12]

    In: International Conference on Information and Software Tech- nologies

    Kerutis, V., Calneryte, D.: Intelligent invoice documents processing employing RPA technologies. In: International Conference on Information and Software Tech- nologies. pp. 235–247. Springer International Publishing (October 2022)

  5. [13]

    In: 2020 Chinese Automation Congress (CAC)

    Ling, X., Gao, M., Wang, D.: Intelligent document processing based on RPA and machine learning. In: 2020 Chinese Automation Congress (CAC). pp. 1349–1353 (2020)

  6. [14]

    In: Lecture Notes in Networks and Systems, vol

    Major, A., Ngalamou, L.: Improving accuracy and efficiency in document handling for business processes. In: Lecture Notes in Networks and Systems, vol. 815 LNNS, pp. 543–562 (2023)

  7. [15]

    Martínez-Rojas, A., Rodríguez-Ruíz, A., Enríquez, J.G., Ramirez, A.J.: What’s behind the screen? Unveiling UI hierarchies in process-related UI logs. In: BPM. Lecture Notes in Computer Science, vol. 14940, pp. 256–272. Springer (2024)

  8. [16]

    OpenAI: Best practices for prompt engineering with the OpenAI API (2024), https://help.openai.com/en/articles/ 6654000-best-practices-for-prompt-engineering-with-the-openai-api , accessed: 2024-11-29

  9. [17]

    In: Jeena Jacob, I., Gonzalez-Longatt, F., Kolandapalayam Shanmugam, S., Izonin, I

    Pudasaini, S., Shakya, S., Lamichhane, S., Adhikari, S., Tamang, A., Adhikari, S.: Application of NLP for information extraction from unstructured documents. In: Jeena Jacob, I., Gonzalez-Longatt, F., Kolandapalayam Shanmugam, S., Izonin, I. (eds.) Expert Clouds and Applicatio...

  10. [18]

    In: New Trends in Business Information Systems and Technology, Studies in Systems, Decision and Control, vol

    Pustulka, E., Hanne, T.: Text mining innovation for business. In: New Trends in Business Information Systems and Technology, Studies in Systems, Decision and Control, vol. 294. Springer, Cham (2021)

  11. [19]

    In: 2022 IEEE 20th Student Conference on Research and Development (SCOReD)

    Rohaime, N., Abdul Razak, N., Thamrin, N., Shyan, C.: Integrated invoicing solu- tion: A Robotic Process Automation with AI and OCR approach. In: 2022 IEEE 20th Student Conference on Research and Development (SCOReD). pp. 30–33 (2022)

  12. [20]

    In: 2021 6th International Conference on Recent Trends on Electronics, Informa- tion, Communication and Technology (RTEICT)

    Roopesh, N., Babu, C.: Robotic Process Automation for resume processing system. In: 2021 6th International Conference on Recent Trends on Electronics, Informa- tion, Communication and Technology (RTEICT). pp. 180–184 (2021)

  13. [21]

    Journal of Integrated Science and Technology11(4), 561–561 (2023)

    Sreekrishna, M., Jacob, T.: Systematic mapping in improving the extraction of can- cer pathology information using RPA orchestration. Journal of Integrated Science and Technology11(4), 561–561 (2023)

  14. [22]

    Computers in Industry115, 103162 (2020)

    Syed, R., Suriadi, S., Adams, M., Bandara, W., Leemans, S.J.J., Ouyang, C., ter Hofstede, A.H.M., van de Weerd, I., Wynn, M.T., Reijers, H.A.: Robotic Process Automation: Contemporary themes and challenges. Computers in Industry115, 103162 (2020)

  15. [23]

    arXiv preprint arXiv:2403.03344 (2024)

    Vartziotis, T., Dellatolas, I., Dasoulas, G., Schmidt, M., Schneider, F., Hoffmann, T., Keckeisen, M.: Learn to code sustainably: An empirical study on LLM-based green code generation. arXiv preprint arXiv:2403.03344 (2024)

  16. [24]

    In: 2020 IEEE 24th International Enterprise Distributed Object Computing Conference (EDOC)

    Wewerka, J., Dax, S., Reichert, M.: A user acceptance model for Robotic Pro- cess Automation. In: 2020 IEEE 24th International Enterprise Distributed Object Computing Conference (EDOC). pp. 97–106 (Oct 2020)

  17. [25]

    Procedia Computer Science192, 2432–2439 (2021), knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021

    Wosiak, A.: Automated extraction of information from Polish resume documents in the IT recruitment process. Procedia Computer Science192, 2432–2439 (2021), knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021

  18. [26]

    ICIC Express Letters 14(6), 593–603 (2020)

    Zaman, G., Mahdin, H., Hussain, K., Rahman, A.: Information extraction from semi and unstructured data sources: A systematic literature review. ICIC Express Letters 14(6), 593–603 (2020)

  19. [274]

    Springer Nature Switzerland (2024) Enabling the Use of Unstructured Data for RPA 15

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.