pith. sign in

arxiv: 2605.21082 · v1 · pith:PK7W636Knew · submitted 2026-05-20 · 💻 cs.AI

AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions

Minghao Chen , Xinyi Hu , Zhou Yu , Yufei Yin This is my paper

Pith reviewed 2026-05-21 04:25 UTC · model grok-4.3

classification 💻 cs.AI
keywords GUI automationLLM agentsRobotic Process AutomationReActcode synthesisretrieval-augmented generationtoken efficiency
0
0 comments X

The pith

AutoRPA distills ReAct LLM GUI interactions into reusable RPA functions that reduce token usage by 82 to 96 percent.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces AutoRPA to handle repetitive GUI tasks more efficiently than repeated LLM calls under the ReAct paradigm. It uses a translator agent to turn specific ReAct actions into more general soft-coded procedures. A builder agent then creates robust RPA functions by applying retrieval-augmented generation across several recorded trajectories. A hybrid repair process tests the code in execution and falls back to ReAct reasoning when needed for fixes. Experiments in multiple GUI settings show the resulting functions complete similar tasks with far lower token consumption and greater reusability.

Core claim

AutoRPA automatically distills the decision logic of ReAct-style agents into robust RPA functions. It does so through a translator-builder pipeline where the translator converts hard-coded ReAct actions into soft-coded procedures and the builder synthesizes RPA functions via retrieval-augmented generation over multiple trajectories. A hybrid repair strategy refines the code by combining direct RPA execution with ReAct-based fallback for iterative improvement. This produces functions that solve similar tasks across GUI environments while cutting token usage by 82 to 96 percent.

What carries the argument

Translator-builder pipeline that converts ReAct actions into RPA functions using retrieval-augmented generation over multiple trajectories plus hybrid repair.

If this is right

  • RPA functions handle repetitive GUI tasks without repeated LLM reasoning at each step.
  • Token consumption drops 82 to 96 percent relative to pure ReAct execution.
  • Runtime efficiency and reusability increase for automation scripts across environments.
  • The same pipeline supports iterative refinement without full manual code rewriting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same distillation approach could be tested on agent frameworks other than ReAct for GUI work.
  • Production systems might adopt the generated functions to lower per-task LLM costs in high-volume settings.
  • Adding more detailed logging of failure modes during hybrid repair could yield even more reliable code.

Load-bearing premise

The translator can reliably generalize hard-coded ReAct steps into soft-coded procedures and the builder can synthesize RPA functions that keep the original decision logic intact across similar tasks.

What would settle it

A generated RPA function that fails on a new but similar GUI task variant which the original ReAct agent completes successfully, or one that shows no token reduction when run.

Figures

Figures reproduced from arXiv: 2605.21082 by Minghao Chen, Xinyi Hu, Yufei Yin, Zhou Yu.

Figure 1
Figure 1. Figure 1: Comparison of GUI automation paradigms. (a) ReAct-style LLM agents achieve high flexibility but incur substantial per￾instance costs, unsuitable for repetitive tasks. (b) Traditional RPA offers efficiency for repetitive tasks but requires manual scripting. (c) AutoRPA automatically synthesizes robust, low-cost RPA functions for arbitrary task types from LLM agent interactions. To this end, we propose AutoR… view at source ↗
Figure 2
Figure 2. Figure 2: AutoRPA Overview: For a task in the target task type, AutoRPA explores and repairs bugs using the ReAct agent, while a translator agent converts the resulting actions into soft-coded actions. A builder then generates the RPA function based on the simplified trajectory from the trajectory bank. The newly generated code will be verified on the seen tasks. If it fails, it will be analyzed and repaired; if suc… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of Code Generation and Refinement: Based on the initial exploration trajectory, the generated RPA code failed to generalize across scenarios. Through the verification and hybrid repair processes, the builder agent can improve the robustness of the RPA code. tering the first failure task g∗. Rather than directly request￾ing the builder to debug, we introduce an analyzer agent to analyze the bre… view at source ↗
Figure 4
Figure 4. Figure 4: Testing success rate and token consumption of different methods with GPT-5 on WebArena (Reddit). Implementation Details. We employ GPT-4o, GPT-4.1, or GPT-5 as the LLM backbone of our agents. For Android￾World, we additionally evaluate with Claude-4.5-sonnet as the backbone to demonstrate that our method can bene￾fit from better backbones (results are provided in the Ap￾pendix). During the building stage, … view at source ↗
Figure 5
Figure 5. Figure 5: The success rate curve of varying building task numbers with GPT-4.1 on AndroidWorld. the environment to generate reasonable code-style plans or actions. In contrast, thanks to the ReAct exploration phase, AutoRPA can generate robust RPA code for all task types with only one demonstration on a simple task (“click-button”). 4.2. Ablation Study More experimental results on token costs during building, compar… view at source ↗
Figure 6
Figure 6. Figure 6: The task type-level performance of ReAct† , AutoRPA (code only), and AutoRPA with GPT-4.1 in MiniWoB++ [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Code generated by different agent frameworks for tic-tac-toe. To facilitate a clear comparison of the core differences, we present a streamlined version of the code, generated by AutoRPA, that preserves all essential decision logic. E. Case study of AutoRPA E.1. Generated Code: AutoRPA vs. Prior Methods We use the tic-tac-toe task type in MiniWoB++ as a case study to compare the skill code produced by prio… view at source ↗
Figure 8
Figure 8. Figure 8: ReAct Trajectory in AndroidWorld – MarkorCreateNote 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Hybrid Repair Process in AndroidWorld – MarkorCreateNote Python code : Initial RPA Code on MarkorCreateNote ### Func Description: # Create a new note in Markor with a specified name and content. ### Params Description: # - file_name (Optional[str]): The name of the note to create (with or without extension). # - text (Optional[str]): The content to write into the note. ### Example Usage: # create_markor_no… view at source ↗
read the original abstract

Large Language Model (LLM) based agents have demonstrated proficiency in multi-step interactions with graphical user interfaces (GUIs). While most research focuses on improving single-task performance, practical scenarios often involve repetitive GUI tasks for which invoking LLM reasoning repeatedly, i.e., the ReAct paradigm, is inefficient. Prior to LLMs, traditional Robotic Process Automation (RPA) offers runtime efficiency but demands significant manual effort to develop and maintain. To bridge this gap, we propose AutoRPA, a framework that automatically distills the decision logic of ReAct-style agents into robust RPA functions. AutoRPA introduces two core innovations: (1) A translator-builder pipeline, where a translator agent converts hard-coded ReAct actions into soft-coded procedures, and a builder agent synthesizes robust RPA functions via retrieval-augmented generation over multiple trajectories; (2) A hybrid repair strategy during code verification, combining RPA execution with ReAct-based fallback for iterative refinement. Experiments across multiple GUI environments demonstrate that RPA functions generated by AutoRPA successfully solve similar tasks while reducing token usage by 82% to 96%, significantly improving runtime efficiency and reusability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes AutoRPA, a framework to automatically distill reusable RPA functions from ReAct-style LLM GUI interactions. It introduces a translator agent to convert hard-coded ReAct actions into soft-coded procedures and a builder agent that synthesizes RPA code via retrieval-augmented generation over multiple trajectories, plus a hybrid repair strategy that combines RPA execution with ReAct fallback during verification. Experiments across GUI environments are reported to show that the generated RPA functions solve similar tasks while achieving 82-96% token reduction, improving runtime efficiency and reusability.

Significance. If the empirical claims hold under rigorous held-out evaluation, the work offers a practical bridge between the flexibility of LLM agents and the efficiency of traditional RPA for repetitive GUI tasks. The translator-builder pipeline with RAG-based synthesis represents a concrete technical step toward reusable code generation from interaction traces, which could reduce reliance on repeated LLM calls in production settings.

major comments (2)
  1. [Experiments] Experiments section: The headline claim that RPA functions 'successfully solve similar tasks' while delivering 82-96% token reduction rests on an unstated assumption that evaluation tasks are distinct from the trajectories supplied to the builder's RAG component. No held-out split, trajectory diversity metric, or overlap analysis is described; without this, success rates and efficiency gains may reflect retrieval of near-identical procedures rather than distillation of reusable decision logic.
  2. [§3.2] §3.2 (Builder agent) and hybrid repair description: The hybrid repair step, which falls back to ReAct calls during verification, risks masking incompleteness in the synthesized RPA function. The paper should quantify how often the final RPA code executes without fallback and report separate metrics for pure-RPA success versus hybrid success to substantiate the reusability claim.
minor comments (2)
  1. [Abstract] Abstract and §4: The phrase 'multiple GUI environments' is used without naming the specific platforms, task distributions, or number of trajectories per environment; adding these details would improve reproducibility.
  2. [§3.1] Notation in the pipeline description: The distinction between 'hard-coded ReAct actions' and 'soft-coded procedures' is introduced without a formal definition or example; a small illustrative table would clarify the translator's role.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. We address each major comment point by point below. We agree that the experimental reporting requires clarification and additional metrics to more rigorously support claims of generalization and reusability, and we will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Experiments] Experiments section: The headline claim that RPA functions 'successfully solve similar tasks' while delivering 82-96% token reduction rests on an unstated assumption that evaluation tasks are distinct from the trajectories supplied to the builder's RAG component. No held-out split, trajectory diversity metric, or overlap analysis is described; without this, success rates and efficiency gains may reflect retrieval of near-identical procedures rather than distillation of reusable decision logic.

    Authors: We acknowledge that the manuscript does not explicitly describe a held-out split or provide quantitative overlap analysis between RAG trajectories and evaluation tasks. This is a valid concern for distinguishing true distillation from retrieval. In the revised version, we will expand the Experiments section to detail the trajectory collection process, introduce a held-out test split, report a trajectory diversity metric (e.g., average semantic or sequence similarity), and present results on strictly held-out tasks to demonstrate generalization beyond near-identical procedures. revision: yes

  2. Referee: [§3.2] §3.2 (Builder agent) and hybrid repair description: The hybrid repair step, which falls back to ReAct calls during verification, risks masking incompleteness in the synthesized RPA function. The paper should quantify how often the final RPA code executes without fallback and report separate metrics for pure-RPA success versus hybrid success to substantiate the reusability claim.

    Authors: We agree that aggregate success rates without isolating the hybrid repair's contribution could obscure the standalone quality of the synthesized RPA functions. In the revision, we will add explicit reporting of: (1) the percentage of verification runs that complete without ReAct fallback; (2) pure-RPA success rates on the evaluation tasks; and (3) a direct comparison of pure-RPA versus hybrid success to better substantiate the reusability of the generated code. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical validation of AutoRPA pipeline stands independent of inputs

full rationale

The paper's central claims rest on experimental outcomes across GUI environments, where RPA functions generated via the translator-builder pipeline and hybrid repair are shown to solve similar tasks with 82-96% token reduction. No equations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the derivation; the translator converting ReAct actions to procedures and the builder using RAG over trajectories are presented as methodological steps whose success is measured externally rather than presupposed by definition. The evaluation on similar tasks is framed as a test of reusability and efficiency, without any reduction of the reported results to the synthesis trajectories by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Based solely on the abstract, the paper introduces an engineering framework rather than new mathematical axioms or physical entities; no free parameters or invented entities are identifiable from the given text.

pith-pipeline@v0.9.0 · 5734 in / 1032 out tokens · 33277 ms · 2026-05-21T04:25:28.798238+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · 2 internal anchors

  1. [1]

    Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents

    [Online]. https://openai.com/blog/ computer-using-agent . Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P ., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P ., Christiano, P . F., Leike, J., and Lowe, R. J. Training language models to follow inst...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  2. [2]

    Large Language Model-Brained GUI Agents: A Survey

    URL https://api.semanticscholar. org/CorpusID:280699844. Zhang, C., He, S., Qian, J., Li, B., Li, L., Qin, S., Kang, Y ., Ma, M., Liu, G., Lin, Q., et al. Large language model- brained gui agents: A survey. arXiv:2411.18279, 2024. Zhang, C., Y ang, Z., Liu, J., Li, Y ., Han, Y ., Chen, X., Huang, Z., Fu, B., and Y u, G. Appagent: Multimodal agents as smar...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Extract all goal-relevant info dont miss anything useful

    Analyze Input: Carefully examine all input. Extract all goal-relevant info dont miss anything useful. Directly infer and record any obvious conclusions

    work page

  4. [4]

    Otherwise, update completed tasks

    Evaluate Progress: Check if the goal is achieved; if so, stop. Otherwise, update completed tasks

    work page

  5. [5]

    Identify and obtain any missing information

    Devise Plan: Break the goal into efficient, non-redundant steps. Identify and obtain any missing information

    work page

  6. [6]

    Adjust the plan if elements are missing or inaccessible

    Execute & Adjust: Analyze the UI info to decide actions. Adjust the plan if elements are missing or inaccessible

    work page

  7. [7]

    Error Handling: Retry once on failure; if it still fails, choose an alternative

    work page

  8. [8]

    Nothing Happens

    Generate Next Action: Choose the next logical action that advances the goal. [Guidelines] Follow these guidelines: - After you output the action, the action will be executed. The results of each action and the new observations will be printed to you at next step. - Maintain a holistic view by identifying the specific steps required to complete the task us...

    work page

  9. [9]

    Compare Screenshots: Focus on differences related to the highlighted element in the 'before' screenshot and the executed code

    work page

  10. [10]

    Verify Purpose: Check if the executed code aligns with its intended purpose (reason for code) and if the highlighted element meets expectations

    work page

  11. [11]

    Compare Code: Confirm that the expected code matches the executed code; if not, identify discrepancies

    work page

  12. [12]

    Assess Outcome: Determine if the executed code met the intended goal

    work page

  13. [13]

    [Guidelines] - If actions like `answer` or `wait` do not change the screen, assume success

    Highlight Findings: Note key insights for future actions. [Guidelines] - If actions like `answer` or `wait` do not change the screen, assume success. - If no change occurs, clearly state the failure and possible reasons. - Rely primarily on screenshot analysis. - Focus on actionable insights; avoid redundant details. - For file-related operations, make su...

    work page

  14. [14]

    - Clearly explain why this specific step failed (e.g., incorrect actions, misinterpretation of UI, planners inaccurate decision-making)

    Analyze Trajectory: - Identify and pinpoint exactly which step in the trajectory led to failure. - Clearly explain why this specific step failed (e.g., incorrect actions, misinterpretation of UI, planners inaccurate decision-making). - Highlight key decision points and provide specific reasoning behind each critical action

    work page

  15. [15]

    - Highlight any misjudgments or missed opportunities for correction

    Root Cause Analysis (RCA): - Clearly state the underlying cause(s) of the failure. - Highlight any misjudgments or missed opportunities for correction

    work page

  16. [16]

    Formulate Corrective Guidelines: - Propose clear, actionable guidelines or improvements for avoiding similar failures in future attempts

    work page

  17. [17]

    - Highlight the reasoning behind critical decisions and their role in the task's success

    Summary Generation: - Focus on key actions that directly contributed to the goal, showing how each step led to the next. - Highlight the reasoning behind critical decisions and their role in the task's success. - Write a single coherent paragraph in natural language, emphasizing the causal relationships between actions. [GUIDELINES] - Avoid generic or unr...

    work page

  18. [18]

    The revised logic maintains the same intended behavior as the original hardcoded action

    work page

  19. [19]

    content_description

    If indexing is not required, do not use the find_element method. [Index Replacement] You need to use this function to replace the hardcoded `index` value with the index variable generated by the ` find_element()`. ### Get Element Index env_op.find_element(**kwargs) -> int # Use this function to find an element in the UI list using filtering criteria and r...

    work page

  20. [20]

    irrelevant steps, even if the task succeeded, to improve efficiency

    Analyze Trajectories: - Review the execution history for beneficial vs. irrelevant steps, even if the task succeeded, to improve efficiency. - Perform a Root Cause Analysis (RCA) on failed trajectories to identify the exact reasons for failure. - Compare successful and failed trajectories, highlighting the differences or weaknesses that need improvement. ...

    work page

  21. [21]

    Ensure the code handles all cases

    Generate Optimized Skill Code: - Wrap the code in a reusable function (e.g., def function_name():) with generic parameters. Ensure the code handles all cases. - Structure the code based on the High-Level Plan. - Implement clear error handling with assertions to identify issues, avoiding internal error catching. Error handling is external. - Do not alter k...

    work page

  22. [22]

    goal completed

    Enhance Generalization: - Improve logging, readability, and maintainability. - Ensure the code is general, reusable, and applicable to similar tasks. F .4. Prompts for Executor and Analyzer Agents Listing 6. System Prompts for RPA Executor Agent [Role] You are an expert in extracting task parameters for Android RPA functions. Your task is to accurately ex...

    work page