Pith. sign in

REVIEW 4 major objections 6 minor 50 references

Cybernaut: a single human demonstration, converted into a parameterized SOP, plus robust hidden-element identification and embedding-based consistency checking, lifts internal web-automation success from 72.00% to 88.68%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A demonstration-to-SOP framework plus robust element identification and a trace similarity metric improves enterprise web automation success rates on an internal benchmark, with a fine-tuned consistency classifier reaching 84.7% accuracy.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A useful industrial recipe for reliable web agents, but the headline 23.2% improvement is not yet externally verifiable because the internal benchmark mixes demonstrations into the setup. the 4 major comments →

arxiv 2508.16688 v1 pith:2ODL4GZM submitted 2025-08-21 cs.SE cs.AI

Cybernaut: Towards Reliable Web Automation

Ankur Tomar , Hengyue Liang , Indranil Bhattacharya , Natalia Larios , Francesco Carbone This is my paper
classification cs.SE cs.AI
keywords web automationLLM agentsdemonstration learningstandard operating proceduresDOM element recognitionexecution consistencySiamese embeddingenterprise web interfaces
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cybernaut tackles a practical gap: LLM web agents work well on polished public sites but fail on the messy, poorly structured internal websites typical of enterprises. The paper claims that three engineering mechanisms close most of that gap: converting a single human demonstration into a parameterized Standard Operating Procedure, reliably identifying hidden interactive HTML elements via LLM-assisted signature extraction, and scoring execution consistency with a fine-tuned embedding model. On an internal benchmark of 117 tasks across 25 domains, the combination lifts task success from 72.00% (the open-source baseline agent) to 88.68%, a 23.2% relative improvement; the consistency model separates consistent from inconsistent runs with 84.7% accuracy at an 81.1% threshold. If true, this makes enterprise web automation practical enough for repeated operational workflows, and gives operators a quantitative way to know when an agent is drifting.

Core claim

The paper's central claim is that reliability in web automation is achievable not by making the agent smarter but by feeding it procedural knowledge extracted from one human demonstration and by hardening its perception of the page. Concretely, Cybernaut claims that adding a demonstration-derived SOP accounts for a 13.9-point accuracy gain over the baseline (72.00% to 82.02%), and that additionally fixing critical element detection adds another 6.66 points, reaching 88.68%. It further claims that a Siamese embedding model fine-tuned on 816 labeled trace pairs can distinguish consistent from inconsistent executions with 84.7% accuracy (87.3% F1) at an 81.1% threshold, with consistent runs ave

What carries the argument

Three mechanisms carry the argument. First, a browser extension records a human's clicks and inputs as JSON; an LLM prompt converts that trace into a generalizable SOP template with placeholders such as language, year, and sort key, discarding irrelevant actions like CAPTCHA-solving. Second, critical element identification works in three stages—presence verification (XPath/identifier lookup, falling back to LLM semantic matching against the current HTML snapshot), key-value signature assignment (LLM selects stable attributes and Playwright checks uniqueness), and configuration persistence—so hidden or dynamically renamed elements stay reachable. Third, execution consistency is formalized as

Load-bearing premise

The load-bearing premise is that evaluating tasks of the same type as the recorded demonstrations is a fair zero-shot test; if evaluation instances overlap with demonstrated instances, the 88.68% figure measures recipe following rather than generalization.

What would settle it

Record demonstrations only for, say, form-filling tasks, then evaluate Cybernaut on a held-out set of data-extraction, table-QnA, and complex-reasoning tasks; split the 117-task internal benchmark by category so no demonstrated task type appears in the evaluation set. If accuracy on held-out types stays near the 72% baseline instead of 88.68%, the reported improvement is recipe following rather than generalization.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Internal knowledge-operation tasks—retrieving ratings, filling forms, querying tables—can be automated at 88.68% success instead of 72%, cutting manual effort on repetitive workflows.
  • Because the SOP is parameterized, one demonstration generalizes across inputs (e.g., any ASIN, any date range) rather than being a brittle hard-coded replay.
  • The consistency score gives a runtime signal: at threshold 81.1%, agents that drift from reference patterns are distinguishable from reliable ones, enabling monitoring and adaptive re-prompting.
  • The framework is agent-agnostic; replacing the underlying browser agent should preserve the SOP, element-signature, and consistency components.
  • The approach is currently scoped to linear, non-branching tasks, so its reliability claims apply to that class of workflows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An immediate test the paper leaves implicit: withhold demonstrations for one whole task category and measure accuracy on that category. If accuracy drops toward 72%, the 88.68% result is recipe-following on known procedure types, not generalization to unseen workflows.
  • The 30-point gap between consistent (90.97%) and inconsistent (60.60%) cosine similarities suggests the consistency score could serve as an early-warning drift detector during execution, not just a post-hoc metric; the paper lists this as future work, so it is an extension.
  • Because the consistency feature vector (goal, action, element attributes) is defined at the step level, the same metric could be applied to compare two different agents on the same task or to monitor how an agent changes after a website redesign, not only to compare repeated runs of Cybernaut.
  • The public-benchmark result without SOPs (80.3% vs 82.2%) implies critical-element handling generalizes to well-designed sites, while SOPs matter most on messy internal sites; a controlled study varying only the SOP component would isolate where the gain actually comes from.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Cybernaut, a framework for enterprise web automation built on three components: (1) an LLM-based generator that converts a single user demonstration (recorded as a browser JSON trace) into a reusable Standard Operating Procedure (SOP), (2) a critical-element identification mechanism that extracts stable HTML attribute signatures from demonstration traces and persists them in configuration files, and (3) a consistency metric that compares execution traces using a Siamese embedding network fine-tuned on manually labeled trace pairs. The authors report 88.68% task success on an internal benchmark of 117 tasks across 25 internal domains, versus 72.00% for the Browser-Use baseline, and 84.7% accuracy for the consistency model on a validation set. They also report results on the public WebVoyager benchmark, where Cybernaut achieves 80.3% versus 82.2% for Browser-Use.

Significance. If the claims were supported, the work would be practically useful for enterprise settings where tasks are repetitive and web interfaces are poorly structured. The idea of turning a single human demonstration into an SOP and then using stable element signatures derived from that demonstration is a reasonable engineering contribution, and the attempt to define a quantitative consistency metric is a useful step. The paper also has a strength in that it provides a concrete comparison on a public benchmark (WebVoyager) in addition to its internal data. However, the paper's central claims rest on an internal benchmark whose construction, demonstration/evaluation overlap, and statistical reliability are not adequately documented. The consistency metric is evaluated on the same validation set used for threshold selection, and the reported gains are not accompanied by error bars or significance tests. As presented, the headline numbers are not externally verifiable, and the internal evaluation does not isolate the framework's contribution from the information provided by demonstrations of the same task types.

major comments (4)
  1. [§3.1, §4.1, §4.2.1, Table 1] The headline claim of a 23.2% improvement (72.00% to 88.68%) is confounded by the relationship between the demonstrations used to generate SOPs and the evaluation tasks. §3.1 calls the internal dataset a 'zero-shot setting', but §4.1 states that 'recorded demonstrations along with task definitions were then processed by an LLM to generate structured SOPs', and §3.2 says the LLM 'processes this demonstration to generate a generalizable SOP template'. The paper never states whether the evaluation task instances are disjoint from the demonstrated instance for each task type, nor how many demonstrations were used per task type. If the evaluation tasks overlap with the demonstrated tasks or differ only in placeholders, then the 88.68% number measures recipe-following, not generalization, and the comparison to Browser-Use (which receives only the natural-language task) is not like-for-like. Th
  2. [§4.2.1, Table 1] The paper states that adding critical element handling yields 'an additional 9.3% gain' over SOP-only, but the numbers in Table 1 show 82.02% → 88.68%, which is 6.66 percentage points (an 8.1% relative improvement). This arithmetic inconsistency suggests the evaluation reporting is not rigorous. The same section also claims a 13.9% improvement for SOP alone (72.00% → 82.02%), which is consistent, but the 'additional 9.3%' is not. Please correct the wording and provide the exact percentage-point and relative changes, and clarify whether these numbers come from the same runs.
  3. [§4.1, §4.2.2, Eq. (3)] The consistency model's reported 84.7% accuracy and 87.3% F1 are measured on a validation set, and the prediction threshold of 81.1% was selected on that same validation set. This is a form of threshold tuning on the test data, so the reported numbers are optimistic estimates of the model's true discrimination performance. No separate test set or cross-validation is described. The fine-tuning dataset is also small (816 pairs from 80 executions across only 8 task types), and results from a single training run are reported without variance. The claim that Cybernaut 'identifies consistent execution patterns with 84.7% accuracy' is therefore not established.
  4. [§4.3, Table 4] The WebVoyager evaluation is asymmetric: the paper explicitly states that no demonstrations were generated for WebVoyager, and Cybernaut there trails Browser-Use (80.3% vs 82.2%). This is informative, but it also strengthens the concern that the internal benchmark gains come from the demonstration-derived SOPs rather than from a generally superior agent. The paper should state clearly that the WebVoyager result is a lower bound for the full framework or, alternatively, should report the WebVoyager result with SOPs enabled on tasks where demonstrations could be defined. Without this, the 'generalizable solution' claim in the conclusion is not supported.
minor comments (6)
  1. [§4.1] The text says 'Clause 3.7' where it presumably means 'Claude 3.7' (the LLM used). This typo should be corrected.
  2. [§5] The conclusion contains a duplicated word: 'a robust and and generalizable solution'. Please fix.
  3. [General] All internal benchmark numbers are reported without error bars or significance tests. At minimum, report the number of runs per task and the variance across runs, or state that each task was run once.
  4. [§3.1] The internal dataset is not public, and the paper does not describe a plan to release it. Since the central claims rely on this dataset, the absence of a public benchmark or detailed task descriptions makes independent verification impossible.
  5. [§4.2.2, Table 2] The 'out-of-the-box' model uses a threshold of 99.8% to achieve 71.1% accuracy. This is an unusual way to report a baseline and should be explained: if the default threshold is 0.5, why is a 99.8% threshold used? Otherwise, the comparison is misleading.
  6. [Appendix C, Listing 3] The sample SOP output contains a placeholder `<country>` in the final instruction, although the input parameters only include language, year, and sort_by. This suggests the SOP generation may introduce variables that are not defined in the input parameter set; please reconcile.

Circularity Check

0 steps flagged

No significant circularity: the paper's headline numbers are empirical measurements, not deductions from fitted inputs; the demonstration/SOP overlap is a possible confound, not a circular reduction.

full rationale

The paper's claimed derivation chain is: (1) user demonstrations are converted to SOPs via an LLM; (2) SOPs plus critical-element handling are executed by a Browser-Use-based agent; (3) the internal benchmark success rate rises from 72.00% to 88.68%; (4) a Siamese embedding model is fine-tuned on 816 manually labeled trace pairs and reports 84.7% validation accuracy. None of these steps defines its output in terms of the quantity being predicted. The 23.2% improvement is a measured difference, not a fitted parameter. The consistency score (Eq. 3) is computed by cosine similarity over embeddings; the embeddings are trained on human labels, so the 84.7% accuracy is a standard supervised evaluation, not a self-fulfilling metric. The paper does not cite any uniqueness theorem or prior work by the same authors as the load-bearing justification. The main evaluation concern is that the internal benchmark may not be disjoint from the demonstrations used to create SOPs: Section 4.1 says 'recorded demonstrations along with task definitions were then processed by an LLM to generate structured SOPs,' and Section 4.2.1 attributes the gain to SOP integration, while Section 4.3 says no demonstrations were generated for WebVoyager, where Cybernaut trails baseline (80.3% vs 82.2%). This asymmetry is a plausible confound, and the paper's use of 'zero-shot' in Section 3.1 is inconsistent with providing demonstrations. However, a confound is not a circular reduction: the paper never states that the same task instances are used for both demonstration and evaluation, and Section 3.2 explicitly speaks of 'each new task instance i' that populates placeholders for a generalizable SOP. Without evidence that the demonstration instances are identical to the test instances, the improvement cannot be exhibited as equivalent to the input by construction. The threshold 81.1% in Table 2 may be selected on the validation set, which is a methodological weakness, but it does not make the reported accuracy a prediction forced by the model definition. Therefore the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central empirical claims rest on two classes of fitted quantities: the embedding model and the classification threshold for the consistency metric, plus several domain assumptions about demonstrations, benchmark representativeness, annotation reliability, element-signature persistence, and the baseline. No new physical or formal entities are introduced.

free parameters (2)
  • Fine-tuned Siamese embedding weights = all-distilroberta-v1 fine-tuned with contrastive loss on 816 labeled trace pairs, 3 epochs, lr 5e-5
    The consistency score C(E1,E2) = Sim(Embed(f(E1)), Embed(f(E2))) (Equation 3) depends on Embed, which is trained on the authors' own labels; the 84.7% accuracy is evaluated on data from the same annotation scheme.
  • Consistency classification threshold = 81.1% for fine-tuned model; 99.8% for base model
    The threshold is chosen on the validation set (Section 4.2.2, Table 2), and the reported accuracy and F1 correspond to that optimized threshold.
axioms (5)
  • domain assumption User demonstrations encode the optimal or sufficient step sequence for a task.
    Section 3.2 asserts users are 'uniquely positioned' to know the optimal sequence; if a demo omits steps or records quirks, the SOP inherits them.
  • domain assumption The 117 internal tasks across 25 domains are representative of enterprise web automation workloads.
    Section 3.1 constructs the benchmark from the authors' internal sites; no external validation of representativeness is provided.
  • domain assumption Manual labeling of trace pairs as functionally equivalent or not is reliable ground truth.
    Section 4.1 says labels were assigned based on the 'spirit of execution'; no inter-annotator agreement or independent adjudication is reported.
  • domain assumption Stable element signatures extracted from demonstration data persist across executions.
    Section 3.3 assumes key-value attribute signatures remain unique and valid on later page versions; dynamic sites may invalidate them, which the presence-verification step tries to mitigate.
  • domain assumption Browser-Use at 72% is an appropriate state-of-the-art baseline for internal tasks.
    Table 1 compares only against Browser-Use; no other web agents or tuned baselines are reported, so the 'SOTA baseline' label is unverified.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Cybernaut: Towards Reliable Web Automation." pith.science (2026). https://pith.science/paper/2ODL4GZM

@misc{pith2026250816688,
  author       = {Pith},
  title        = {Pith review of: Cybernaut: Towards Reliable Web Automation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ODL4GZM}},
  note         = {Machine review of arXiv:2508.16688}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The emergence of AI-driven web automation through Large Language Models (LLMs) offers unprecedented opportunities for optimizing digital workflows. However, deploying such systems within industry's real-world environments presents four core challenges: (1) ensuring consistent execution, (2) accurately identifying critical HTML elements, (3) meeting human-like accuracy in order to automate operations at scale and (4) the lack of comprehensive benchmarking data on internal web applications. Existing solutions are primarily tailored for well-designed, consumer-facing websites (e.g., Amazon.com, Apple.com) and fall short in addressing the complexity of poorly-designed internal web interfaces. To address these limitations, we present Cybernaut, a novel framework to ensure high execution consistency in web automation agents designed for robust enterprise use. Our contributions are threefold: (1) a Standard Operating Procedure (SOP) generator that converts user demonstrations into reliable automation instructions for linear browsing tasks, (2) a high-precision HTML DOM element recognition system tailored for the challenge of complex web interfaces, and (3) a quantitative metric to assess execution consistency. The empirical evaluation on our internal benchmark demonstrates that using our framework enables a 23.2% improvement (from 72% to 88.68%) in task execution success rate over the browser_use. Cybernaut identifies consistent execution patterns with 84.7% accuracy, enabling reliable confidence assessment and adaptive guidance during task execution in real-world systems. These results highlight Cybernaut's effectiveness in enterprise-scale web automation and lay a foundation for future advancements in web automation.

Figures

Figures reproduced from arXiv: 2508.16688 by Ankur Tomar, Francesco Carbone, Hengyue Liang, Indranil Bhattacharya, Natalia Larios.

Figure 1
Figure 1. Figure 1: Architectural overview of Cybernaut’s workflow pipeline, illustrating the transformation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: 3 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of tasks across categories and internal website types in the benchmarking [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Critical element identification approach [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: 1. Presence Verification: At current page, we attempt to locate element using XPath and identifiers. If no exact match is found, this means the state has changed and XPaths are not valid anymore. To get the HTML code of element, we submit both the recorded element attributes and the current HTML snapshot to an LLM to perform semantic matching and return HTML code of element that might match this element. 2… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 34 canonical work pages · 2 internal anchors

  1. [1]

    Computer-Using Agent

    OpenAI. Computer-Using Agent. https://openai.com/index/ computer-using-agent/, 2025. 1

    work page 2025

  2. [2]

    Computer Use (Beta)

    Anthropic. Computer Use (Beta). https://docs.anthropic.com/en/docs/ agents-and-tools/computer-use , October 2024. 1

    work page 2024

  3. [3]

    Browser Use: Enable AI to control your browser

    Magnus Müller and Gregor Žuniˇc. Browser Use: Enable AI to control your browser. GitHub,

    work page

  4. [4]

    Nova Act

    Amazon. Nova Act. https://nova.amazon.com/act, 2025. 2

    work page 2025

  5. [5]

    STEVE: A Step Verification Pipeline for Computer-use Agent Training

    Fanbin Lu, Zhisheng Zhong, Ziqin Wei, Shu Liu, Chi-Wing Fu, and Jiaya Jia. STEVE: A Step Verification Pipeline for Computer-use Agent Training. arXiv preprint arXiv:2503.12532, 2025. 3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    WebV oyager: Building an End-to-End Web Agent with Large Multimodal Models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebV oyager: Building an End-to-End Web Agent with Large Multimodal Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) , pages 6864–6890, Bangkok, Thailand, August 2024. Association ...

    work page 2024

  7. [7]

    Mind2Web: Towards a Generalist Agent for the Web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a Generalist Agent for the Web. In Advances in Neural Information Processing Systems, volume 36, pages 28091–28114. Curran Associates, Inc., 2023. 3

    work page 2023

  8. [8]

    VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks.arXiv preprint arXiv:2401.13649,

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks.arXiv preprint arXiv:2401.13649,

    Pith/arXiv arXiv

  9. [9]

    AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv preprint arXiv:2405.14573, 2025. 3

    Pith/arXiv arXiv 2025

  10. [10]

    CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents

    Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin, Jianbo Deng, Philip Torr, Bernard Ghanem, and Guohao Li. CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. arXiv preprint arXiv:2407.01511, 2024. 3

    Pith/arXiv arXiv 2024

  11. [11]

    Beyond Browsing: API-Based Web Agents

    Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond Browsing: API-Based Web Agents. arXiv preprint arXiv:2410.16464, 2025. 2

    Pith/arXiv arXiv 2025

  12. [12]

    WebLINX: Real-World Website Navigation with Multi-Turn Dialogue

    Xing Han Lù, Zdenˇek Kasner, and Siva Reddy. WebLINX: Real-World Website Navigation with Multi-Turn Dialogue. arXiv preprint arXiv:2402.05930, 2024. 2 10

    arXiv 2024

  13. [13]

    GPT-4V(ision) is a Generalist Web Agent, if Grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a Generalist Web Agent, if Grounded. arXiv preprint arXiv:2401.01614, 2024. 2

    Pith/arXiv arXiv 2024

  14. [14]

    Large Language Models Empowered Personalized Web Agents

    Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, and Tat-Seng Chua. Large Language Models Empowered Personalized Web Agents. In Proceedings of the ACM on Web Conference 2025, pages 198–215. Association for Computing Machinery, 2025

    work page 2025

  15. [15]

    GUI Agents with Foundation Models: A Comprehensive Survey

    Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang, Chuhan Wu, Yasheng Wang, Ruiming Tang, and Jianye Hao. GUI Agents with Foundation Models: A Comprehensive Survey. arXiv preprint arXiv:2411.04890, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    AutoDroid: LLM-powered Task Automation in Android

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. AutoDroid: LLM-powered Task Automation in Android. arXiv preprint arXiv:2308.15272, 2024. 3

    Pith/arXiv arXiv 2024

  17. [17]

    DroidBot-GPT: GPT-powered UI Automation for Android

    Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li. DroidBot-GPT: GPT-powered UI Automation for Android. arXiv preprint arXiv:2304.07061, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Mapping Natural Language Instructions to Mobile UI Action Sequences

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping Natural Language Instructions to Mobile UI Action Sequences. arXiv preprint arXiv:2005.03776, 2020. 3

    Pith/arXiv arXiv 2005

  19. [19]

    CogAgent: A Visual Language Model for GUI Agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. CogAgent: A Visual Language Model for GUI Agents. arXiv preprint arXiv:2312.08914, 2024. 3

    Pith/arXiv arXiv 2024

  20. [20]

    AppAgent: Multimodal Agents as Smartphone Users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. AppAgent: Multimodal Agents as Smartphone Users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , Article 70, pages 1–20. Association for Computing Machinery, 2025

    work page 2025

  21. [21]

    Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. arXiv preprint arXiv:2401.16158, 2024. 3

    Pith/arXiv arXiv 2024

  22. [22]

    Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs. arXiv preprint arXiv:2404.05719, 2024. 3

    Pith/arXiv arXiv 2024

  23. [23]

    OmniParser for Pure Vision Based GUI Agent

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. OmniParser for Pure Vision Based GUI Agent. arXiv preprint arXiv:2408.00203, 2024. 3

    Pith/arXiv arXiv 2024

  24. [24]

    Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining

    Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, and Yueting Zhuang. Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining. arXiv preprint arXiv:2412.10342, 2025. 3

    Pith/arXiv arXiv 2025

  25. [25]

    Yu, and Qing Li

    Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S. Yu, and Qing Li. A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models.arXiv preprint arXiv:2503.23350, 2025. 1

    Pith/arXiv arXiv 2025

  26. [26]

    A Survey of Demonstration Learning

    André Correia and Luís A. Alexandre. A Survey of Demonstration Learning. arXiv preprint arXiv:2303.11191, 2023. 2 11 A Appendix A: User demonstration to step-by-step SOP instructions To convert user demonstrations into step-by-step SOP instructions, we utilize an LLM that takes as input a high-level task definition and a browser-recorded execution trace i...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [28]

    For the first step in your SOP: - Use both website name and the exact URL in your instruction

    work page

  28. [29]

    - Use the intention and purpose behind to guide your composition of SOP

    For the second step and onwards in your SOP: - Look holistically at the demo and identify the intention and goal behind each browsing action and step recorded. - Use the intention and purpose behind to guide your composition of SOP. - If navigating to a specific website is a critical action to achieve the goal , always include the web page name and the ex...

    work page

  29. [30]

    - Do not come up with an SOP from your memory

    Your SOP should only include the knowledge that can be drawn from the demo replay within <demo> tags. - Do not come up with an SOP from your memory

    work page

  30. [31]

    Examples of such steps are: - Steps related to solving CAPTCHA

    Exclude steps within <demo> tages that are unrelated to the task within < task_for_sop> tags. Examples of such steps are: - Steps related to solving CAPTCHA. - Steps related to close pop up windows. - Mouse clicks on non-interactable elements such as background, or plain text. </requirement> You should follow the formatting instructions below to provide a...

    work page

  31. [32]

    Using <sop> tags to include all contents below

    work page

  32. [33]

    Restate the task description content within <task_for_sop> tags above, now using <task> tags

    work page

  33. [34]

    Explanation

    <task_for_sop> may be a general version of the <demo> example. Please identify proper input parameters so that <task_for_sop> can be faithfully represented. - Format the input parameters as .json within <input_param> tags. E.g., { input_param_1: "Explanation"}. - If no input parameter needed, output an empty dict within <input_param> tags . I.e., {}

    work page

  34. [35]

    - Only provide the instructions within <instructions-step-by-step> tags

    Document your final SOP within <instructions-step-by-step> tags. - Only provide the instructions within <instructions-step-by-step> tags. No need to explain within <instructions-step-by-step> tags. - Use proper referece to the input parameters identified. E.g., using < INPUT_PARAMETER_1> - Using numbered points to organize your sop. </format_instructions>...

    work page

  35. [36]

    Do not open any URL from your memory

    Only navigate the websites you are in. Do not open any URL from your memory. </guardrails> output_format: | Below are the output formatting requirements: <output_format>

    work page

  36. [37]

    No answer found

    Provide your final answer to the task within <final_answer> tags. - If you cannot find the answer related to the <task> from your browsing activities, say "No answer found". Do not make up one yourself based on your prior knowledge

    work page

  37. [38]

    language

    Provide your thoughts within <thoughts> tags. </output_format> prompt_head: | You are a browsing agent and are asked to perform a web browsing task described within <task> tags below. You are ask to act like a human associate who will strictly follow the instructions within <instructions-step-by-step> tags, using the appropriate input parameter values pro...

    work page 2020

  38. [39]

    Navigate to https://www.imdb.com/?ref_=nv_home

    work page

  39. [40]

    Click on the "All" dropdown menu at the top of the page (usually located in the search bar area)

    work page

  40. [41]

    Advanced Search

    Click on "Advanced Search" from the dropdown menu

    work page

  41. [42]

    Expand Release date

    In the Advanced Search page, first click on Expand all to access all search filters and then click on "Expand Release date" section

    work page

  42. [43]

    <year>-01

    Enter "<year>-01" and "<year>-12" in the "Enter release date above" field

    work page

  43. [44]

    <language>

    In the language search field, type "<language>" (e.g., "Japanese")

    work page

  44. [45]

    <language>

    Select the checkbox for "<language>" from the dropdown results

    work page

  45. [46]

    See results

    Click on the "See results" button

    work page

  46. [47]

    On the results page, click on the "Sort by" dropdown menu

    work page

  47. [48]

    <sort_by>

    Select "<sort_by>" from the dropdown options (e.g., "USER_RATING_COUNT")

    work page

  48. [49]

    If needed, click on the sort order button to change from ascending to descending order

    work page

  49. [50]

    The page will now display the top 20 movies from <country> in <language> language with the highest number of ratings in <year> </instructions-step-by-step> </sop> Listing 3: User Demonstration translated to a sample robust SOP 19

    work page

  50. [2024]

    https://github.com/browser-use/browser-use . 1, 3

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.